drivers: Add sensors v2 API

An addition of a new header for the V2 sensors API. The API attempts
to cover the most general use cases for sensors as well as include a
few more advanced yet still common use cases. This API does NOT
attempt to enumerate all of the sensor functionalities. It is believed
that going into too much detail simply makes writing the drivers too
difficult. Instead, applications requiring highly specific uses should
be making direct calls to the sensor via the attached bus.
The purpose of the driver is to abstract away common functionalities
between sensors. If highly specific functionality is required by the
application, then it is likely that a different sensor cannot be used
as a drop-in replacement. It is then assumed that driver API will not
be used by such an application and that the application will make
more direct API calls. These direct calls can still be provided in
Zephyr by the individual driver's header instead of under the common
API.
Fixes #13718
Signed-off-by: Yuval Peress<EMAIL_ADDRESS>

This is a port of https://docs.google.com/document/d/1UsElJJHRudeWXb-tYwXA6qt9gF6PH8wwS7PjhqO1MLI/edit# into a PR as discussed in the API meeting

Set as a draft until I can clear the build issues.

This PR is ready for review. The only failing check is the typedef which is required to abstract away the float (I'll be working on moving that code to a separate PR with a math library so please feel free to ignore it here).

I'm really looking forward to this being merged (I develop computer hardware...) however... aren't the sensor_channel enums a bit specific to the EC world?

Zephyr has an existing sensor device driver APIs, also there is a list of existing sensor device drivers available in Zephyr today.
I wonder what is the relationship between this proposed V2 API and the existing Zephyr sensor device API:

The V2 API will be a replacement of Zephyr sensor device API? meaning Zephyr sensor device API will be deprecated.
Or it will become a duplication of Zephyr sensor device API? meaning developer can select 1 of 2 to use in Zephyr.
Or it is developed on top of Zephyr sensor device API? meaning both APIs are available in Zephyr for different layers of applications to use

Also I would like to understand what is the impact from this PR to the existing sensor device drivers in Zephyr

Something we should also consider is if we should define specific SI units per sensor type (which I believe the Android API does), and if so then those should be documented here in Zephyr rather than pointing people to an external reference we don't control.
One shortcoming of the current API beyond the SI units is also that the underlying C type used is usually fixed to floating point, which may be wasteful, and has a specific unit scale that isn't always a natural fit for every sensor (KG versus milligrams, etc.).
I tried to come up with two 32-bit values that could encapsulate in a flexible manner all of this information (SI unit type, C type used, and unit scale):

Source: https://github.com/microbuilder/linaro_sensor_pipeline/blob/main/include/step/measurement/measurement.h

 *    3                   2                   1
 *  1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |              Flags            |  Ext. M Type  |  Base M Type  | <- Filter
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |     C Type    | Scale Factor  |         SI Unit Type          | <- Unit
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |   Source ID   | S Cnt | V | F |        Payload Length         | <- SrcLen
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |                      Timestamp (optional)                     |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |                                                               |
 * |                            Payload                            |
 * |                                                               |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *
 *            1
 *  5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * | Res | TSt | CMP | Encod |  DF | <- Flags
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *     |    |     |      |      |
 *     |    |     |      |      +-------- Data Format (CBOR, etc.)
 *     |    |     |      +--------------- Encoding (BASE64, BASE45, etc.)
 *     |    |     +---------------------- Compression (LZ4, etc.)
 *     |    +---------------------------- Timestamp
 *     +--------------------------------- Reserved (version flag?)

Where C types, for example, have the following 8-bit values:

Source: https://github.com/microbuilder/linaro_sensor_pipeline/blob/main/include/step/measurement/unit.h

/**
 * @brief C type used to represent a measurement in memory (8-bits).
 *
 * @note All types are little-endian.
 *
 * Memory map:
 *   - 0 = Undefined
 *   - 0x10..0x4F = Standard C types
 *   - 0x50..0x7F = Reserved
 *   - 0x80..0x8F = Range types (unit interval, percent, etc.)
 *   - 0x90..0xEF = Reserved
 *   - 0xF0..0xFE = User-defined types
 */
enum step_mes_unit_ctype {
	STEP_MES_UNIT_CTYPE_UNDEFINED                   = 0x00,

	/* 0x10..0x4F: Standard C types. */
	STEP_MES_UNIT_CTYPE_IEEE754_FLOAT32             = 0x10,
	STEP_MES_UNIT_CTYPE_IEEE754_FLOAT64             = 0x11,
	STEP_MES_UNIT_CTYPE_IEEE754_FLOAT128            = 0x12,
	STEP_MES_UNIT_CTYPE_S8                          = 0x13,
	STEP_MES_UNIT_CTYPE_S16                         = 0x14,
	STEP_MES_UNIT_CTYPE_S32                         = 0x15,
	STEP_MES_UNIT_CTYPE_S64                         = 0x16,
	STEP_MES_UNIT_CTYPE_S128                        = 0x17,
	STEP_MES_UNIT_CTYPE_U8                          = 0x18,
	STEP_MES_UNIT_CTYPE_U16                         = 0x19,
	STEP_MES_UNIT_CTYPE_U32                         = 0x1A,
	STEP_MES_UNIT_CTYPE_U64                         = 0x1B,
	STEP_MES_UNIT_CTYPE_U128                        = 0x1C,
	STEP_MES_UNIT_CTYPE_BOOL                        = 0x1D,
	STEP_MES_UNIT_CTYPE_COMPLEX_32                  = 0x30,
	STEP_MES_UNIT_CTYPE_COMPLEX_64                  = 0x31,

	/* 0x80..0x8F: Range types (unit interval, percent, etc.) */
	/** @brief 0..1.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT32. */
	STEP_MES_UNIT_CTYPE_RANG_UNIT_INTERVAL_32       = 0x80,
	/** @brief 0..1.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT64. */
	STEP_MES_UNIT_CTYPE_RANG_UNIT_INTERVAL_64       = 0x81,
	/** @brief 0.0..100.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT32. */
	STEP_MES_UNIT_CTYPE_RANG_PERCENT_32             = 0x82,
	/** @brief 0.0..100.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT64. */
	STEP_MES_UNIT_CTYPE_RANG_PERCENT_64             = 0x83,

	/* 0xF0..0xFE: User-defined types. */
	STEP_MES_UNIT_CTYPE_USER_1                      = 0xF0,
	STEP_MES_UNIT_CTYPE_USER_2                      = 0xF1,
	STEP_MES_UNIT_CTYPE_USER_3                      = 0xF2,
	STEP_MES_UNIT_CTYPE_USER_4                      = 0xF3,
	STEP_MES_UNIT_CTYPE_USER_5                      = 0xF4,
	STEP_MES_UNIT_CTYPE_USER_6                      = 0xF5,
	STEP_MES_UNIT_CTYPE_USER_7                      = 0xF6,
	STEP_MES_UNIT_CTYPE_USER_8                      = 0xF7,
	STEP_MES_UNIT_CTYPE_USER_9                      = 0xF8,
	STEP_MES_UNIT_CTYPE_USER_10                     = 0xF9,
	STEP_MES_UNIT_CTYPE_USER_11                     = 0xFA,
	STEP_MES_UNIT_CTYPE_USER_12                     = 0xFB,
	STEP_MES_UNIT_CTYPE_USER_13                     = 0xFC,
	STEP_MES_UNIT_CTYPE_USER_14                     = 0xFD,
	STEP_MES_UNIT_CTYPE_USER_15                     = 0xFE,

	STEP_MES_UNIT_CTYPE_MAX                         = 0xFF
};

Followed by a 16-bit SI unit type:
/**
 * @brief Standard SI units (16-bits).
 *
 * @note Base and derived SI units can be represented in 8-bits, while
 * combined units require 16-bits to represent.
 *
 * Memory map:
 *   - 0              = Undefined unit
 *   - 0x0001..0x000F = Reserved
 *   - 0x0010..0x001F = SI base units
 *   - 0x0020..0x003F = SI derived units
 *   - 0x0040..0x007F = Reserved
 *   - 0x0080..0x008F = Unitless values (percent, interval, etc.)
 *   - 0x0090..0x00FF = Reserved
 *   - 0x0100..0x0FFF = SI combined units (generic)
 *   - 0x1000..0w7FFF = Base type specific units, in groups of 0xFF.
 *   - 0x8000..0xFEFF = Reserved
 *   - 0xFF00..0xFFFE = User defined units
 */
enum step_mes_unit_si {
	STEP_MES_UNIT_SI_UNDEFINED              = 0,

	/* 0x0010..0x001F (16-31): SI Base Units. */
	/** @brief A, electric current */
	STEP_MES_UNIT_SI_AMPERE                 = 0x10,
	/** @brief cd, Luminous intensity */
	STEP_MES_UNIT_SI_CANDELA                = 0x11,
	/** @brief K, thermodynamic temp. */
	STEP_MES_UNIT_SI_KELVIN                 = 0x12,
	/** @brief kg, mass */
	STEP_MES_UNIT_SI_KILOGRAM               = 0x13,
	/** @brief m, length */
	STEP_MES_UNIT_SI_METER                  = 0x14,
	/** @brief mol, Amount of substance */
	STEP_MES_UNIT_SI_MOLE                   = 0x15,
	/** @brief s, time */
	STEP_MES_UNIT_SI_SECOND                 = 0x16,

	/* 0x0020..0x003F (32-63): SI Derived Units. */
	/** @brief Bq, radionucl. activity, 1/s */
	STEP_MES_UNIT_SI_BECQUEREL              = 0x20,
	/** @brief C, electric charge, A*s */
	STEP_MES_UNIT_SI_COULOMB                = 0x21,
	/** @brief Degrees C, Celsius temp, K */
	STEP_MES_UNIT_SI_DEGREE_CELSIUS         = 0x22,
	/** @brief F, elec. capaticance, C/V or s^4*A^2/m^2*kg */
	STEP_MES_UNIT_SI_FARAD                  = 0x23,
	/** @brief Gy, Absorbed dose, J/kg or m^2/s^2 */
	STEP_MES_UNIT_SI_GRAY                   = 0x24,
	/** @brief H, electric inductance, Wb/A or kg*m^2/s^2*A^2 */
	STEP_MES_UNIT_SI_HENRY                  = 0x25,
	/** @brief Hz, frequency, 1/s */
	STEP_MES_UNIT_SI_HERTZ                  = 0x26,
	/** @brief J, energy, work, N*m or kg*m^2/s^2 */
	STEP_MES_UNIT_SI_JOULE                  = 0x27,
	/** @brief kat, Catamytic activ., mol/s */
	STEP_MES_UNIT_SI_KATAL                  = 0x28,
	/** @brief lm, Luminous flux, cd*sr */
	STEP_MES_UNIT_SI_LUMEN                  = 0x29,
	/** @brief lx, illuminance, lm/m^2 */
	STEP_MES_UNIT_SI_LUX                    = 0x2A,
	/** @brief N, force, m*kg/s^2 */
	STEP_MES_UNIT_SI_NEWTON                 = 0x2B,
	/** @brief electric resistence, V/A or kg*m^2/s^3*A^2*/
	STEP_MES_UNIT_SI_OHM                    = 0x2C,
	/** @brief Pa, pressure, stress, N/m^2 or kg/m*s^2 */
	STEP_MES_UNIT_SI_PASCAL                 = 0x2D,
	/** @brief rad, plane angle, m/m */
	STEP_MES_UNIT_SI_RADIAN                 = 0x2E,
	/** @brief S, electric conductance, A/V */
	STEP_MES_UNIT_SI_SIEMENS                = 0x2F,
	/** @brief Sv, Dose equivalent, J/kg or m^2/s^2 */
	STEP_MES_UNIT_SI_SIEVERT                = 0x30,
	/** @brief sr, solid angle, m^2/m^2 */
	STEP_MES_UNIT_SI_STERADIAN              = 0x31,
	/** @brief T, magn. flux dens., Wb/m^2 or kg/A*s^2*/
	STEP_MES_UNIT_SI_TESLA                  = 0x32,
	/** @brief V, elec. poten. diff, W/A or kg*m^2/A*s^3 */
	STEP_MES_UNIT_SI_VOLT                   = 0x33,
	/** @brief W, power, radiant flux, J/s or kg*m^2/s^3 */
	STEP_MES_UNIT_SI_WATT                   = 0x34,
	/** @brief Wb, magnetic flux, V*s or kg*m^2/s^2*A */
	STEP_MES_UNIT_SI_WEBER                  = 0x35,
...

And then an optional 8-bit scale factor from the default scale:
/**
 * @brief Standard SI scales/powers.
 */
enum step_mes_si_scale {
	STEP_MES_SI_SCALE_YOTTA = 24,   /**< Y: Septillion */
	STEP_MES_SI_SCALE_ZETTA = 21,   /**< Z: Sextillion */
	STEP_MES_SI_SCALE_EXA   = 18,   /**< E: Quintillion */
	STEP_MES_SI_SCALE_PETA  = 15,   /**< P: Quadrillion */
	STEP_MES_SI_SCALE_TERA  = 12,   /**< T: Trillion */
	STEP_MES_SI_SCALE_GIGA  = 9,    /**< G: Billion */
	STEP_MES_SI_SCALE_MEGA  = 6,    /**< M: Million */
	STEP_MES_SI_SCALE_KILO  = 3,    /**< k: Thousand */
	STEP_MES_SI_SCALE_HECTO = 2,    /**< h: Hundred */
	STEP_MES_SI_SCALE_DECA  = 1,    /**< da: Ten*/
	STEP_MES_SI_SCALE_NONE  = 0,    /**< One */
	STEP_MES_SI_SCALE_DECI  = -1,   /**< d: Tenth */
	STEP_MES_SI_SCALE_CENTI = -2,   /**< c: Hundredth */
	STEP_MES_SI_SCALE_MILLI = -3,   /**< m: Thousandth */
	STEP_MES_SI_SCALE_MICRO = -6,   /**< u: Millionth */
	STEP_MES_SI_SCALE_NANO  = -9,   /**< n: Billionth */
	STEP_MES_SI_SCALE_PICO  = -12,  /**< p: Trillionth */
	STEP_MES_SI_SCALE_FEMTO = -15,  /**< f: Quadrillionth */
	STEP_MES_SI_SCALE_ATTO  = -18,  /**< a: Quintillionth */
	STEP_MES_SI_SCALE_ZEPTO = -21,  /**< z: Sextillionth */
	STEP_MES_SI_SCALE_YOCTO = -24,  /**< y: Septillionth */
};

This is a very different approach that what is propsosed here, following Android's sensor APIs, but the code above comes out of bumping into shortcomings in the real world with the current approach in Zephyr and Android's APIs.
Allowing default SI units and scale is one thing, but we should also allow those values to be represented with an efficient underlying C type, and enable flexibility on unit scale to make best used of limited SRAM and maximum precision in value representations.

Something we should also consider is if we should define specific SI units per sensor type (which I believe the Android API does), and if so then those should be documented here in Zephyr rather than pointing people to an external reference we don't control.
One shortcoming of the current API beyond the SI units is also that the underlying C type used is usually fixed to floating point, which may be wasteful, and has a specific unit scale that isn't always a natural fit for every sensor (KG versus milligrams for a load cell, etc.).
I tried to come up with two 32-bit values that could encapsulate in a flexible manner all of this information (SI unit type, C type used, and unit scale):

Source: https://github.com/microbuilder/linaro_sensor_pipeline/blob/main/include/step/measurement/measurement.h

 *    3                   2                   1
 *  1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |              Flags            |  Ext. M Type  |  Base M Type  | <- Filter
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |     C Type    | Scale Factor  |         SI Unit Type          | <- Unit
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |   Source ID   | S Cnt | V | F |        Payload Length         | <- SrcLen
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |                      Timestamp (optional)                     |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |                                                               |
 * |                            Payload                            |
 * |                                                               |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *
 *            1
 *  5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * | Res | TSt | CMP | Encod |  DF | <- Flags
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 *     |    |     |      |      |
 *     |    |     |      |      +-------- Data Format (CBOR, etc.)
 *     |    |     |      +--------------- Encoding (BASE64, BASE45, etc.)
 *     |    |     +---------------------- Compression (LZ4, etc.)
 *     |    +---------------------------- Timestamp
 *     +--------------------------------- Reserved (version flag?)

Where C types, for example, have the following 8-bit values:

Source: https://github.com/microbuilder/linaro_sensor_pipeline/blob/main/include/step/measurement/unit.h

/**
 * @brief C type used to represent a measurement in memory (8-bits).
 *
 * @note All types are little-endian.
 *
 * Memory map:
 *   - 0 = Undefined
 *   - 0x10..0x4F = Standard C types
 *   - 0x50..0x7F = Reserved
 *   - 0x80..0x8F = Range types (unit interval, percent, etc.)
 *   - 0x90..0xEF = Reserved
 *   - 0xF0..0xFE = User-defined types
 */
enum step_mes_unit_ctype {
	STEP_MES_UNIT_CTYPE_UNDEFINED                   = 0x00,

	/* 0x10..0x4F: Standard C types. */
	STEP_MES_UNIT_CTYPE_IEEE754_FLOAT32             = 0x10,
	STEP_MES_UNIT_CTYPE_IEEE754_FLOAT64             = 0x11,
	STEP_MES_UNIT_CTYPE_IEEE754_FLOAT128            = 0x12,
	STEP_MES_UNIT_CTYPE_S8                          = 0x13,
	STEP_MES_UNIT_CTYPE_S16                         = 0x14,
	STEP_MES_UNIT_CTYPE_S32                         = 0x15,
	STEP_MES_UNIT_CTYPE_S64                         = 0x16,
	STEP_MES_UNIT_CTYPE_S128                        = 0x17,
	STEP_MES_UNIT_CTYPE_U8                          = 0x18,
	STEP_MES_UNIT_CTYPE_U16                         = 0x19,
	STEP_MES_UNIT_CTYPE_U32                         = 0x1A,
	STEP_MES_UNIT_CTYPE_U64                         = 0x1B,
	STEP_MES_UNIT_CTYPE_U128                        = 0x1C,
	STEP_MES_UNIT_CTYPE_BOOL                        = 0x1D,
	STEP_MES_UNIT_CTYPE_COMPLEX_32                  = 0x30,
	STEP_MES_UNIT_CTYPE_COMPLEX_64                  = 0x31,

	/* 0x80..0x8F: Range types (unit interval, percent, etc.) */
	/** @brief 0..1.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT32. */
	STEP_MES_UNIT_CTYPE_RANG_UNIT_INTERVAL_32       = 0x80,
	/** @brief 0..1.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT64. */
	STEP_MES_UNIT_CTYPE_RANG_UNIT_INTERVAL_64       = 0x81,
	/** @brief 0.0..100.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT32. */
	STEP_MES_UNIT_CTYPE_RANG_PERCENT_32             = 0x82,
	/** @brief 0.0..100.0 inclusive, STEP_MES_UNIT_CTYPE_FLOAT64. */
	STEP_MES_UNIT_CTYPE_RANG_PERCENT_64             = 0x83,

	/* 0xF0..0xFE: User-defined types. */
	STEP_MES_UNIT_CTYPE_USER_1                      = 0xF0,
	STEP_MES_UNIT_CTYPE_USER_2                      = 0xF1,
	STEP_MES_UNIT_CTYPE_USER_3                      = 0xF2,
	STEP_MES_UNIT_CTYPE_USER_4                      = 0xF3,
	STEP_MES_UNIT_CTYPE_USER_5                      = 0xF4,
	STEP_MES_UNIT_CTYPE_USER_6                      = 0xF5,
	STEP_MES_UNIT_CTYPE_USER_7                      = 0xF6,
	STEP_MES_UNIT_CTYPE_USER_8                      = 0xF7,
	STEP_MES_UNIT_CTYPE_USER_9                      = 0xF8,
	STEP_MES_UNIT_CTYPE_USER_10                     = 0xF9,
	STEP_MES_UNIT_CTYPE_USER_11                     = 0xFA,
	STEP_MES_UNIT_CTYPE_USER_12                     = 0xFB,
	STEP_MES_UNIT_CTYPE_USER_13                     = 0xFC,
	STEP_MES_UNIT_CTYPE_USER_14                     = 0xFD,
	STEP_MES_UNIT_CTYPE_USER_15                     = 0xFE,

	STEP_MES_UNIT_CTYPE_MAX                         = 0xFF
};

Followed by a 16-bit SI unit type:
/**
 * @brief Standard SI units (16-bits).
 *
 * @note Base and derived SI units can be represented in 8-bits, while
 * combined units require 16-bits to represent.
 *
 * Memory map:
 *   - 0              = Undefined unit
 *   - 0x0001..0x000F = Reserved
 *   - 0x0010..0x001F = SI base units
 *   - 0x0020..0x003F = SI derived units
 *   - 0x0040..0x007F = Reserved
 *   - 0x0080..0x008F = Unitless values (percent, interval, etc.)
 *   - 0x0090..0x00FF = Reserved
 *   - 0x0100..0x0FFF = SI combined units (generic)
 *   - 0x1000..0w7FFF = Base type specific units, in groups of 0xFF.
 *   - 0x8000..0xFEFF = Reserved
 *   - 0xFF00..0xFFFE = User defined units
 */
enum step_mes_unit_si {
	STEP_MES_UNIT_SI_UNDEFINED              = 0,

	/* 0x0010..0x001F (16-31): SI Base Units. */
	/** @brief A, electric current */
	STEP_MES_UNIT_SI_AMPERE                 = 0x10,
	/** @brief cd, Luminous intensity */
	STEP_MES_UNIT_SI_CANDELA                = 0x11,
	/** @brief K, thermodynamic temp. */
	STEP_MES_UNIT_SI_KELVIN                 = 0x12,
	/** @brief kg, mass */
	STEP_MES_UNIT_SI_KILOGRAM               = 0x13,
	/** @brief m, length */
	STEP_MES_UNIT_SI_METER                  = 0x14,
	/** @brief mol, Amount of substance */
	STEP_MES_UNIT_SI_MOLE                   = 0x15,
	/** @brief s, time */
	STEP_MES_UNIT_SI_SECOND                 = 0x16,

	/* 0x0020..0x003F (32-63): SI Derived Units. */
	/** @brief Bq, radionucl. activity, 1/s */
	STEP_MES_UNIT_SI_BECQUEREL              = 0x20,
	/** @brief C, electric charge, A*s */
	STEP_MES_UNIT_SI_COULOMB                = 0x21,
	/** @brief Degrees C, Celsius temp, K */
	STEP_MES_UNIT_SI_DEGREE_CELSIUS         = 0x22,
	/** @brief F, elec. capaticance, C/V or s^4*A^2/m^2*kg */
	STEP_MES_UNIT_SI_FARAD                  = 0x23,
	/** @brief Gy, Absorbed dose, J/kg or m^2/s^2 */
	STEP_MES_UNIT_SI_GRAY                   = 0x24,
	/** @brief H, electric inductance, Wb/A or kg*m^2/s^2*A^2 */
	STEP_MES_UNIT_SI_HENRY                  = 0x25,
	/** @brief Hz, frequency, 1/s */
	STEP_MES_UNIT_SI_HERTZ                  = 0x26,
	/** @brief J, energy, work, N*m or kg*m^2/s^2 */
	STEP_MES_UNIT_SI_JOULE                  = 0x27,
	/** @brief kat, Catamytic activ., mol/s */
	STEP_MES_UNIT_SI_KATAL                  = 0x28,
	/** @brief lm, Luminous flux, cd*sr */
	STEP_MES_UNIT_SI_LUMEN                  = 0x29,
	/** @brief lx, illuminance, lm/m^2 */
	STEP_MES_UNIT_SI_LUX                    = 0x2A,
	/** @brief N, force, m*kg/s^2 */
	STEP_MES_UNIT_SI_NEWTON                 = 0x2B,
	/** @brief electric resistence, V/A or kg*m^2/s^3*A^2*/
	STEP_MES_UNIT_SI_OHM                    = 0x2C,
	/** @brief Pa, pressure, stress, N/m^2 or kg/m*s^2 */
	STEP_MES_UNIT_SI_PASCAL                 = 0x2D,
	/** @brief rad, plane angle, m/m */
	STEP_MES_UNIT_SI_RADIAN                 = 0x2E,
	/** @brief S, electric conductance, A/V */
	STEP_MES_UNIT_SI_SIEMENS                = 0x2F,
	/** @brief Sv, Dose equivalent, J/kg or m^2/s^2 */
	STEP_MES_UNIT_SI_SIEVERT                = 0x30,
	/** @brief sr, solid angle, m^2/m^2 */
	STEP_MES_UNIT_SI_STERADIAN              = 0x31,
	/** @brief T, magn. flux dens., Wb/m^2 or kg/A*s^2*/
	STEP_MES_UNIT_SI_TESLA                  = 0x32,
	/** @brief V, elec. poten. diff, W/A or kg*m^2/A*s^3 */
	STEP_MES_UNIT_SI_VOLT                   = 0x33,
	/** @brief W, power, radiant flux, J/s or kg*m^2/s^3 */
	STEP_MES_UNIT_SI_WATT                   = 0x34,
	/** @brief Wb, magnetic flux, V*s or kg*m^2/s^2*A */
	STEP_MES_UNIT_SI_WEBER                  = 0x35,
...

And then an optional 8-bit scale factor from the default scale:
/**
 * @brief Standard SI scales/powers.
 */
enum step_mes_si_scale {
	STEP_MES_SI_SCALE_YOTTA = 24,   /**< Y: Septillion */
	STEP_MES_SI_SCALE_ZETTA = 21,   /**< Z: Sextillion */
	STEP_MES_SI_SCALE_EXA   = 18,   /**< E: Quintillion */
	STEP_MES_SI_SCALE_PETA  = 15,   /**< P: Quadrillion */
	STEP_MES_SI_SCALE_TERA  = 12,   /**< T: Trillion */
	STEP_MES_SI_SCALE_GIGA  = 9,    /**< G: Billion */
	STEP_MES_SI_SCALE_MEGA  = 6,    /**< M: Million */
	STEP_MES_SI_SCALE_KILO  = 3,    /**< k: Thousand */
	STEP_MES_SI_SCALE_HECTO = 2,    /**< h: Hundred */
	STEP_MES_SI_SCALE_DECA  = 1,    /**< da: Ten*/
	STEP_MES_SI_SCALE_NONE  = 0,    /**< One */
	STEP_MES_SI_SCALE_DECI  = -1,   /**< d: Tenth */
	STEP_MES_SI_SCALE_CENTI = -2,   /**< c: Hundredth */
	STEP_MES_SI_SCALE_MILLI = -3,   /**< m: Thousandth */
	STEP_MES_SI_SCALE_MICRO = -6,   /**< u: Millionth */
	STEP_MES_SI_SCALE_NANO  = -9,   /**< n: Billionth */
	STEP_MES_SI_SCALE_PICO  = -12,  /**< p: Trillionth */
	STEP_MES_SI_SCALE_FEMTO = -15,  /**< f: Quadrillionth */
	STEP_MES_SI_SCALE_ATTO  = -18,  /**< a: Quintillionth */
	STEP_MES_SI_SCALE_ZEPTO = -21,  /**< z: Sextillionth */
	STEP_MES_SI_SCALE_YOCTO = -24,  /**< y: Septillionth */
};

This measurement-centric approach is different than what is propsosed in this PR, following Android's sensor APIs, but the code above comes out of bumping into shortcomings in the real world with the current approach in Zephyr and Android's APIs, and it's worth considering pros and cons of various ways of modelling sensor data.
Allowing default SI units and scale is one thing, but we should also allow those values to be represented with an efficient underlying C type, and enable flexibility on unit scale to make best used of limited SRAM and maximum precision in value representations.

I'm currently working on a sample migration of the bmi160 to this as well as the test suite that will demonstrate how this API will be used and how it should be easy enough to migrate from v1 to v2 piece by piece

Some random thoughts:

floating point is great, but only if it is optional. 99% of sensors are fixed point.
USB HID has sensor usage tables that include a ton of standard sensor definitions that are then adopted by Windows and Linux. I think those tables are a better default option than the EC-centric ones proposed here initially. Some sensor types may be missing, but those could be implemented separately and then proposed to the USB folks.
Personal opinion, but I think a successful sensor API should be somewhat compatible with some kind of embedded C scientific library types. Perfect example is https://github.com/zephyrproject-rtos/zscilib that already contains most of what people will need to build robust data processing and sensor fusion

* USB HID has sensor usage tables that include a ton of standard sensor definitions that are then adopted by Windows and Linux. I think those tables are a better default option than the EC-centric ones proposed here initially. Some sensor types may be missing, but those could be implemented separately and then proposed to the USB folks.

Wasn't aware of this, but I'll have a look. Thanks for the heads up that there is an HID list.

* Personal opinion, but I think a successful sensor API should be somewhat compatible with some kind of embedded C scientific library types. Perfect example is https://github.com/zephyrproject-rtos/zscilib that already contains most of what people will need to build robust data processing and sensor fusion

Glad to see someone is potentially using zscilib. ;P To be fair, though, I ended up requiring double-precision floats for some specialty calculations in it (svd, pinv, etc.), so it's not going to be small ... but it's easy to go from fixed point or small integer-based sensor data from the driver to float32 or float64 for processing after the fact.

Some random thoughts:

floating point is great, but only if it is optional. 99% of sensors are fixed point.

Right, the math library here allowed fixed/float depending on CONFIG_FPU so it should work for either use case.

USB HID has sensor usage tables that include a ton of standard sensor definitions that are then adopted by Windows and Linux. I think those tables are a better default option than the EC-centric ones proposed here initially. Some sensor types may be missing, but those could be implemented separately and then proposed to the USB folks.

I'm still considering this and have had several meetings with some of the folk here at Zephyr to discuss. The big picture here is to leverage this API to build a central sensor subsystem that's compatible with Android's CHRE. This would allow application developers to simply focus on their use cases Application/nanoapps instead of the math required to generate the data they want.

Personal opinion, but I think a successful sensor API should be somewhat compatible with some kind of embedded C scientific library types. Perfect example is https://github.com/zephyrproject-rtos/zscilib that already contains most of what people will need to build robust data processing and sensor fusion

I'll take a look at zscilib. The current focus was to be able to build a sensor subsystem (which may end up using zscilib) to make the application development easier.

@ddavidebor @microbuilder I took a look at zscilib. It looks great but I have some concerns of tying the base sensor API too closely to the library:

I'll just use the example of accelerometers but this is true for a multitude of other sensors. Most sensors don't use SI as the default unit. Adding the additional conversion (while not expensive) seems unnecessary. This API (and Android's embedded API for CHRE) keeps that conversion up to the application (the Application Processor in the case of Android).
I couldn't find the image size impact of zscilib. For the same reason as above, I don't want to say that anyone using sensors needs to use a library. zscilib should become 1p if we were to do that.

Thoughts?

@yperess

I think this is not correct. Accelerometers use m/s^2. Almost all sensors can be easily described with a combination of base SI units.
I agree, it should be optional. Perhaps required to be implemented for all drivers, but optional. The non-SI value could be called “raw”.

I think SI units are the only way to make sensors easily interchangeable. All accelerometers give m/s^2 as an output. As long as the values are normalised to SI units, any accelerometer could in theory be swapped just by changing the devicetree.

@yperess

I think this is not correct. Accelerometers use m/s^2. Almost all sensors can be easily described with a combination of base SI units.
Really? Every data sheet I looked at for accelerometers only lists things in g. I know it's just a scale issue to get it over to SI, but I think it gets confusing for implementors of the driver. If you can link some examples of sensors datasheets that use m/s^2 I'll be happy to see them.

I agree, it should be optional. Perhaps required to be implemented for all drivers, but optional. The non-SI value could be called “raw”.

I think SI units are the only way to make sensors easily interchangeable. All accelerometers give m/s^2 as an output. As long as the values are normalised to SI units, any accelerometer could in theory be swapped just by changing the devicetree.

I don't see SI as the only way to make them interchangable. At the end of the day, any standard would work. I'm 100% with you that SI is nice, I use it daily. My concern is that every sensor I've worked with doesn't use it. So I would want to find a standard that matches as many of the existing sensors instead of force the change.
At the end of the day, I don't feel very strongly about the units here to argue it. If that's the only concern I'll be happy to swap it over.

@yperess Accelerometers do tend to be quite standard and are documented based on 'g's, but other similar classes of devices are more inconsistant like gyros (DPS or radians?), magnetometers (Gauss, Tesla?), etc. The small translation cost of standardising on specific SI units and scales far outweighs the translation cost in my opinion, in that consumers no longer need to care about the source of the data they're consuming.
Any accelerometer being essential interoperable with other accelerometers (etc.) is an important goal for a v2 API IMO, particularly given a context like today with high unpredictability around stock levels and device manufacturing.
As per zscilib, I don't think we need to concern ourselves too much with considerations of zscilib at the driver level BEYOND having a standard unit and scale, and if that lines up with what zscilib expects all the better, but I wouldn't bend over backwards to align the two since any translation cost before feeding something into zscilib will be a drop in the bucket.
Concerning CHRE, I'm interested in it as a pipeline, and I understand the Android ecosystem is important, but I don't think we should be wedding the Zephyr sensor subsystem entirely to CHRE. Some sort of translation layer between the sensor API and CHRE likely makes more sense to make room for other use cases and alternatives.

:) Quick counter example from the Android source code with accels in m/s^2: https://github.com/aosp-mirror/platform_hardware_libhardware/blob/master/include/hardware/sensors.h#L318= ... this code, however, is quite old, and I think there are some improvements to be made on many of the assumptions here, where we can learn from experience.
If we are going to settle on something, I don't think we should cherry-pick, and should just go all in on SI units and fixed scales. It will make any sort of calculations and sensor fusion much easier further down the pipeline, and put the burden of translating raw units to SI on the driver author, who is the best placed person to write that code and handle edge cases appropriately (such as handling changes in gain settings, etc.).

@microbuilder I'm sold on leaving the units at SI. I'll update the BMI example in this PR. Thanks for entertaining this discussion.

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Concerning CHRE, I'm interested in it as a pipeline, and I understand the Android ecosystem is important, but I don't think we should be wedding the Zephyr sensor subsystem entirely to CHRE. Some sort of translation layer between the sensor API and CHRE likely makes more sense to make room for other use cases and alternatives.

Agree. We want to support CHRE, but not require it for every Zephyr sensor application.

Zephyr has an existing sensor device driver APIs, also there is a list of existing sensor device drivers available in Zephyr today.
I wonder what is the relationship between this proposed V2 API and the existing Zephyr sensor device API:

The V2 API will be a replacement of Zephyr sensor device API? meaning Zephyr sensor device API will be deprecated.
Or it will become a duplication of Zephyr sensor device API? meaning developer can select 1 of 2 to use in Zephyr.
Or it is developed on top of Zephyr sensor device API? meaning both APIs are available in Zephyr for different layers of applications to use

Also I would like to understand what is the impact from this PR to the existing sensor device drivers in Zephyr

@yperess any input on this, seems you missed it.

@nashif @KeHIntel Sorry, I did miss that.

Zephyr has an existing sensor device driver APIs, also there is a list of existing sensor device drivers available in Zephyr today.
I wonder what is the relationship between this proposed V2 API and the existing Zephyr sensor device API:

The V2 API will be a replacement of Zephyr sensor device API? meaning Zephyr sensor device API will be deprecated.
Yes, the existing API will become deprecated. This PR adds an example for how this can be done gracefully by migrating an implementation to V2 and keeping the V1 supported until the migration can be completed.

Or it will become a duplication of Zephyr sensor device API? meaning developer can select 1 of 2 to use in Zephyr.
Or it is developed on top of Zephyr sensor device API? meaning both APIs are available in Zephyr for different layers of applications to use

Also I would like to understand what is the impact from this PR to the existing sensor device drivers in Zephyr

The impact will be minimal at first. Once this merges, the plan is to start working on a sensor subsystem that will leverage these APIs and provide common sensor computations and features along with virtual sensors. As we do this, we'll start migrating existing V1 sensor implementations (similar to how this PR migrates the BMI160). New sensors being added to Zephyr should only implement V2. Eventually, V1 will be deprecated and we'll migrate the remaining sensors in that list over. The migration is pretty simple and didn't take long in this example. The V2 API is a superset that will allow for much easier integration with the sensor subsystem as well as all the features that are needed to build the ChromeOS EC (and theoretically Android's EC)

@yperess any input on this, seems you missed it.

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Concerning CHRE, I'm interested in it as a pipeline, and I understand the Android ecosystem is important, but I don't think we should be wedding the Zephyr sensor subsystem entirely to CHRE. Some sort of translation layer between the sensor API and CHRE likely makes more sense to make room for other use cases and alternatives.

Agree. We want to support CHRE, but not require it for every Zephyr sensor application.

Makes a lot of sense, I'll update the PR to reflect that right now. @microbuilder I believe that Maureen makes a good point here about providing the native values from the sensors API and utility functions to convert these to SI values.

@nashif @KeHIntel Sorry, I did miss that.

Zephyr has an existing sensor device driver APIs, also there is a list of existing sensor device drivers available in Zephyr today.
I wonder what is the relationship between this proposed V2 API and the existing Zephyr sensor device API:

The V2 API will be a replacement of Zephyr sensor device API? meaning Zephyr sensor device API will be deprecated.
Yes, the existing API will become deprecated. This PR adds an example for how this can be done gracefully by migrating an implementation to V2 and keeping the V1 supported until the migration can be completed.

Or it will become a duplication of Zephyr sensor device API? meaning developer can select 1 of 2 to use in Zephyr.
Or it is developed on top of Zephyr sensor device API? meaning both APIs are available in Zephyr for different layers of applications to use

Also I would like to understand what is the impact from this PR to the existing sensor device drivers in Zephyr

The impact will be minimal at first. Once this merges, the plan is to start working on a sensor subsystem that will leverage these APIs and provide common sensor computations and features along with virtual sensors. As we do this, we'll start migrating existing V1 sensor implementations (similar to how this PR migrates the BMI160). New sensors being added to Zephyr should only implement V2. Eventually, V1 will be deprecated and we'll migrate the remaining sensors in that list over. The migration is pretty simple and didn't take long in this example. The V2 API is a superset that will allow for much easier integration with the sensor subsystem as well as all the features that are needed to build the ChromeOS EC (and theoretically Android's EC)

@yperess any input on this, seems you missed it.

Just want to clarify, for V1 sensor implementation, it is actually referring to the existing Chrome EC sensor drivers, right?
Then looks to me this V2 API (if merged) will become a duplicated implementation in Zephyr with existing Zephyr sensor device driver APIs, which will bring unnecessary confusion to Zephyr community especially to people who are not working on Chrome, CHRE, or Android.
In my opinion, the standardization of Zephyr sensing API shall be un-biased and flexible enough to support any type of usages, from edge, IOT, to client, from Chrome, Android to Linux and Windows.

@nashif @KeHIntel Sorry, I did miss that.

Zephyr has an existing sensor device driver APIs, also there is a list of existing sensor device drivers available in Zephyr today.
I wonder what is the relationship between this proposed V2 API and the existing Zephyr sensor device API:

The V2 API will be a replacement of Zephyr sensor device API? meaning Zephyr sensor device API will be deprecated.
Yes, the existing API will become deprecated. This PR adds an example for how this can be done gracefully by migrating an implementation to V2 and keeping the V1 supported until the migration can be completed.

Or it will become a duplication of Zephyr sensor device API? meaning developer can select 1 of 2 to use in Zephyr.
Or it is developed on top of Zephyr sensor device API? meaning both APIs are available in Zephyr for different layers of applications to use

Also I would like to understand what is the impact from this PR to the existing sensor device drivers in Zephyr

The impact will be minimal at first. Once this merges, the plan is to start working on a sensor subsystem that will leverage these APIs and provide common sensor computations and features along with virtual sensors. As we do this, we'll start migrating existing V1 sensor implementations (similar to how this PR migrates the BMI160). New sensors being added to Zephyr should only implement V2. Eventually, V1 will be deprecated and we'll migrate the remaining sensors in that list over. The migration is pretty simple and didn't take long in this example. The V2 API is a superset that will allow for much easier integration with the sensor subsystem as well as all the features that are needed to build the ChromeOS EC (and theoretically Android's EC)

@yperess any input on this, seems you missed it.

Just want to clarify, for V1 sensor implementation, it is actually referring to the existing Chrome EC sensor drivers, right?

Yes

Then looks to me this V2 API (if merged) will become a duplicated implementation in Zephyr with existing Zephyr sensor device driver APIs, which will bring unnecessary confusion to Zephyr community especially to people who are not working on Chrome, CHRE, or Android.
In my opinion, the standardization of Zephyr sensing API shall be un-biased and flexible enough to support any type of usages, from edge, IOT, to client, from Chrome, Android to Linux and Windows.

I 100% agree, and I'm sorry if I made it sound like that's not my goal. The existing API is missing key aspects needed to run the platforms you listed above. Extending that API will be very cumbersome as it's lacking FIFO logic, timestamps, and other features. This API is intended to better serve anyone looking to get more out of the sensors (not just the above mentioned platforms). The exiting API cannot in its current state support these larger platforms (at least from my experience and trials).

@KeHIntel I'm about to push a change to this in response to Maureen's comments that will simplify a fair amount of this API's blocking reading calls.

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Fair point. But I'd argue that a function to apply or return the SI conversion coefficient from raw should also be mandatory since there are too many edge cases that the driver author is the best placed person to deal with.

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Fair point. But I'd argue that a function to apply or return the SI conversion coefficient from raw should also be mandatory since there are too many edge cases that the driver author is the best placed person to deal with.

Agree. Things like ranges and offsets have to be taken care by the driver of the sensor has them. Returning raw values could be optional.

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Fair point. But I'd argue that a function to apply or return the SI conversion coefficient from raw should also be mandatory since there are too many edge cases that the driver author is the best placed person to deal with.

Agree. Things like ranges and offsets have to be taken care by the driver if the sensor has them.

@microbuilder @ddavidebor The latest push on this PR includes this function sensor_sample_to_si() which you can see used in the test. I didn't want to invest too much time into it yet so there are a lot of optimizations that can be made still. Please let me know if this pattern works for you. There will need to be some memoization to avoid having extra bus reads for every conversion, but the concept I think fits well with what @MaureenHelm was saying to return raw data and allow for an easy SI conversion

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Fair point. But I'd argue that a function to apply or return the SI conversion coefficient from raw should also be mandatory since there are too many edge cases that the driver author is the best placed person to deal with.

Yes, I agree

@yperess in your example driver (for which, thanks!) it looks like IRQ handling etc is the responsibility of the driver, not the application. Do you expect that to be a hard-and-fast general rule, or are you still anticipating situations where applications might need to configure their own ISRs?

@yperess in your example driver (for which, thanks!) it looks like IRQ handling etc is the responsibility of the driver, not the application. Do you expect that to be a hard-and-fast general rule, or are you still anticipating situations where applications might need to configure their own ISRs?

Right so I'm trying to keep a clear separation of the user facing API (which is what this PR is meant for) and the driver developer APIs which will come next. If you'd like to dive a bit into my thinking for those I'm happy to give a few bullet points.

ISR registration should remain as is. There are several considerations for this since it was brought up in the Google doc first, but the biggest one for me is that it's how things are currently done and I don't see a big benefit in forcing applications to take over this. The only thing that I would change is how the handler is done (next bullet).
Handlers are done in different ways right now. Some drivers hand their own threads that the ISR notifies and reading is done there. Others post to the kernel work queue. I believe that there's a middle ground to be had there. I would/will suggest a single thread that will be responsible for all of these, it can be thought of a sensors only work queue.

I'm not yet sure how exactly this would look like, it unfortunately blurs a bit the line between the driver and a sensor subsystem. This is why my original proposal left the ISR up to the application, though I think that will hit too much resistance. The most likely approach I will be suggesting (I can try to add this to this PR) is to have an API for this common sensor subsystem. Then using a Kconfig allow the application or a sensor subsystem to override it. The API will be very very simple. It'll basically just look a bit like the k_work_submit where drivers can rely on some "work submit" function to exist. The default one will simply implement a work queue and callback to the handler of the driver with possibly a nicer API since we can cast some of the structs for the caller.
Let me know if you'd like me to include all of these changes in this PR.

@teburd I've update the API but I'm having second thoughts. The big difference is that the data struct returned by the API now simply uses a raw buffer and the helper breaks it up into 3 separate values depending on the resolution and range. See struct below. After committing this and pushing I immediately regret it. I believe that this is going too far actually. The FIFO internals are unique to the sensor. Some sensors have headers in the FIFO and these headers can mean different things. Returning the full DMA copied FIFO isn't actually useful in the driver context. I am now in the camp that if an application requires the 100% raw data then it most likely needs to know the sensor model and is taking on the responsibility to parse that data itself.
struct sensor_raw_sample_data {
	/** */
	uint16_t data_buffer_size;

	/** The base timestamp, in nanoseconds. */
	uint64_t base_timestamp;

	/** The number of valid elements in the 'data' array. */
	uint16_t data_count;

	/** Pointer to the start of the sensor type array in 'data' */
	uint8_t *sensor_type;

	/** The raw data from the sensor */
	uint8_t data[1];
};

The following is a sketch demonstrating the issue raised at the API meeting about the data rate, scale, or resolution changing after the FIFO was sampled. Note that a change while there's data in the FIFO is irrelevant because every hardware descriptor says that the FIFO should be flushed after configuration changes.
Blank diagram - Page 2.pdf
The next sketch demonstrates how the driver can use a simple lock to avoid the issue.
Blank diagram - Page 1.pdf
Another point of discussion was that some application may require additional features from these present in the new API. I would argue that if any further customization is needed, that should be done in the specific sensor's header. For example my_sensor.h could add a function such as:
int my_sensor_unique_functionality(const struct device *dev, ...);

UPDATE:
After talking with @teburd  I believe we reached an understanding on the raw API:

The default API will read the FIFO and parse it into the separate uint32_t values.
When developers enable the raw API we'll just expose the same function in the API sensor_raw_reading_to_sensor_data

Doing this will allow the pipeline to look like:
ISR              -> schedule a HW FIFO read
Read the HW FIFO -> schedule for processing
Start processing -> Convert raw HW FIFO to`struct sensor_data`

UPDATE: After talking with @teburd I believe we reached an understanding on the raw API:

The default API will read the FIFO and parse it into the separate uint32_t values.
When developers enable the raw API we'll just expose the same function in the API sensor_raw_reading_to_sensor_data

Doing this will allow the pipeline to look like:
ISR              -> schedule a HW FIFO read
Read the HW FIFO -> schedule for processing
Start processing -> Convert raw HW FIFO to`struct sensor_data`

This is definitely much closer to what I was looking for personally. I would ask why bother dealing with anything but this flow, but I might be alone on that front.
I think the questions that remain for me are mostly around the numerical format. Does it make sense to enable a numerical format decision at conversion time...
e.g. convert from raw to something like CMSIS's q7/q15/q31 or a scaled integer alongside a more common f32 (enabling subsequent usage of cmsis dsp with desired on Arm or custom dsp functions doing integer math)
That might require type tagged unions and that would be just fine by me. Even if you can't choose in the first version of this if it can be added after the fact, and is considered, that would be really quite nice.
In almost all my own applications what I'd actually do prior to using Arm even is convert to scaled integers so I could do integer math and filtering. That's not that uncommon to consider for DSP applications or doing things on a cortex-m0/m3/softcore where you aren't likely to have a FPU.
What I mean by scaled integer is using a unit that effectively shifts over the decimal position to enable integer math in a sensible way. e.g. mutiply/divide the scale into a unit where integer math keeps most of the precision you care about, uint8_t representing +4/-4 of X -> uint16_t as micro/milli/centi X

UPDATE: After talking with @teburd I believe we reached an understanding on the raw API:

The default API will read the FIFO and parse it into the separate uint32_t values.
When developers enable the raw API we'll just expose the same function in the API sensor_raw_reading_to_sensor_data

Doing this will allow the pipeline to look like:
ISR              -> schedule a HW FIFO read
Read the HW FIFO -> schedule for processing
Start processing -> Convert raw HW FIFO to`struct sensor_data`

This is definitely much closer to what I was looking for personally. I would ask why bother dealing with anything but this flow, but I might be alone on that front.

I'm actually very curious about this flow now too. On Chrome we just do the following steps:

grab the timestamp / schedule the FIFO read
Read the HW FIFO and convert to SI units data using fp_t and schedule additional processing
Perform the additional processing which includes timestamp spreading and gesture detection

In your proposal we would offload some of the work from step 2 into step 3 which I guess would allow us to have finer grain priority on thread scheduling right? Reading the HW FIFO can be a higher priority than processing the data. I'd be curious to try this flow as I believe you're right and it makes more sense.

I think the questions that remain for me are mostly around the numerical format. Does it make sense to enable a numerical format decision at conversion time...
e.g. convert from raw to something like CMSIS's q7/q15/q31 or a scaled integer alongside a more common f32 (enabling subsequent usage of cmsis dsp with desired on Arm or custom dsp functions doing integer math)

I think if this were done in a similar way as how I've set up the math utils I would be very happy with it. Basically (to save you time parsing this PR) I just switch based on CONFIG_FPU whether typedef (.*) fp_t is a float or int32_t. We could just as well increase this switch to support additional types. So I think that we can do this in a follow-up and other people can introduce different representations as they see fit.

That might require type tagged unions and that would be just fine by me. Even if you can't do that in the first version of this API, if it can be added after the fact, and is considered, that would be really quite nice. Even better if it can do it without blowing up the code rom size by deciding at compile time which conversion functions to use.
In almost all my own applications what I'd actually do prior to using Arm even is convert to scaled integers so I could do integer math and filtering. That's not that uncommon to consider for DSP applications or doing things on a cortex-m0/m3/softcore where you aren't likely to have a FPU.
What I mean by scaled integer is using a unit that effectively shifts over the decimal position to enable integer math in a sensible way. e.g. mutiply/divide the scale into a unit where integer math keeps most of the precision you care about, uint8_t representing +4/-4 of X -> uint16_t as micro/milli/centi X

What you're suggesting is basically what the fp_t is doing here. It just uses integer arithmetic with 16 bits on either side. I'm more than happy to parameterize it using a Kconfig, but I didn't want this PR to be about the math lib. My thinking is:

Get this merged
Clean up the math util/lib
Add tools to make writing drivers easier (these will include standardizing how interrupts are handled and how FIFOs are read)
Update documentation to tell everyone how to use the V2 APIs and tools
Start migrating drivers as needed to the V2 APIs

Sensor drivers should provide information to the application about how to convert raw data to SI units so the application can choose when to do the conversion. The application may want to do some signal processing in raw fixed-point units and wait to convert the result to SI units to minimize loss of precision or reduce computational complexity. A downward sample rate converter, for example, would be more computationally efficient to convert only decimated output samples to standard units.

Fair point. But I'd argue that a function to apply or return the SI conversion coefficient from raw should also be mandatory since there are too many edge cases that the driver author is the best placed person to deal with.

Agree. Things like ranges and offsets have to be taken care by the driver if the sensor has them.

@microbuilder @ddavidebor The latest push on this PR includes this function sensor_sample_to_si() which you can see used in the test. I didn't want to invest too much time into it yet so there are a lot of optimizations that can be made still. Please let me know if this pattern works for you. There will need to be some memoization to avoid having extra bus reads for every conversion, but the concept I think fits well with what @MaureenHelm was saying to return raw data and allow for an easy SI conversion

I have some different opinion on SI unit, for discussion:

some SI unit is too big, like Tesla. Earth's magnetic field is just 50~60uT, most applications using magnetometer will not use Tesla in calculation.
some sensor‘s output cannot be converted to SI unit, like color (unit is CIE chromaticity X, Y), SAR (unit is specific absorption ratio), etc.
some sensor's raw data needs complicated algorithm to be converted to SI unit, like radar, LiDAR, converting their raw data to distance is obviously not the work for sensor device driver
some smart sensor's raw data is just output the label or number, like HALL, step counter; some smart sensor's output can be even more complicated, like RFID scan result, medical sensor outputs, concept of SI unit does not fit to these sensors.

In summary, I think the support to SI unit is not necessary, as long as we have a clear definition for the sensor value unit for each sensor channel or type, and every driver follows it, we should be fine.

In summary, I think the support to SI unit is not necessary, as long as we have a clear definition for the sensor value unit for each sensor channel or type, and every driver follows it, we should be fine.

I agree, my first pass of this didn't use SI but just a reasonable standard unit. I'd be happy to add an API for the driver as some variation of
int sensor_raw_to_type_standard(
    const struct device *sensor,
    const struct sensor_data *in,
    int reading_offset,
    struct sensor_three_axis_data *out
);

I agree, my first pass of this didn't use SI but just a reasonable standard unit. I'd be happy to add an API for the driver as some variation of

I'm curious what a 'reasonable standard unit' might be that isn't SI? Imperial? ;P If we're going to do raw to anything standard, I feel like there is an enormous burden of proof to justify something other than SI, even if that conversion itself is optional and secondary.

I agree, my first pass of this didn't use SI but just a reasonable standard unit. I'd be happy to add an API for the driver as some variation of

I'm curious what a 'reasonable standard unit' might be that isn't SI? Imperial? ;P If we're going to do raw to anything standard, I feel like there is an enormous burden of proof to justify something other than SI, even if that conversion itself is optional and secondary.

I would say that what they're trying to avoid is something like a device using fixed point 16 bits to either side of the decimal. then using radians seems wasteful since you have 16 bits to represent numbers that will usually be well under 10. Where as using degrees makes better use of the fixed point math. Regardless, I don't know that we even need a standard. Since the API returns the raw numbers, applications can convert this to any units they want. We can provide some common utilities like in this PR. Or, we can have them as a part of the driver (which would increase efficiency but put the burden on the driver implementor)

@teburd I've updated the API once more. I think this should be the final or at least near final update. Here's the summary of the changes:

I removed sensor_get_range() and sensor_get_resolution() and replaced them with sensor_get_scale(). Now you can call this get scale function when you get your sample or in the callback of the FIFO read. This can be queued up with the data for later processing.
I updated the sensor_sample_to_three_axis_data to make use of this struct, the scaling is now done using this cached value instead of the current one. This function is not optimized, but demonstrates how a single sample can be converted. The final units are in whatever units the sensor reports in sensor_get_scale(). Clients can make further adjustments after that. The enum for the range types is extensible so people can add new values to it.

Let me know if you'd like me to include all of these changes in this PR.

@yperess no, that all sounds fine as long as it's out of the application developer's hands. Thanks for confirming.

@teburd I'm getting close to another version of the FIFO API that should match what we discussed last. I wanted to get your thoughts on the following:

The callback returns a raw buffer (uint8_t[] + a header with basic size information)
The API exposes an iterator that can be set ontop of that buffer (pointer to the buffer + an offset size_t)
Each driver would implement an iterator_next to move the iterator to the next sample
Each driver would implement an iterator_get_sensor_type to get the sensor type of the current iterator position
Each driver would implement an iterator_get_reading to convert the current iterator position to a sensor_*_data depending on the type returned from step 4

These obviously depend on the struct sensor_scale_metadata to be able to convert the current iterator to the reading struct, but the API is already provided for that. Since all the sample range and resolution is in that metadata I believe that iterator_get_reading can be done without involving the driver. While iterator_next will need the metadata as an argument but will still need to be a part of the driver because the FIFO headers will likely be chip specific and cannot be made generic (unless we add that to the metadata). Similarly, getting the sensor type involves parsing the FIFO header so that will be a driver API. This would look something like:
/** Global function to convert the iterator position into a sensor_*_data value
int sensor_raw_reading_iterator_get_reading(
    const struct device *sensor,
    const struct sensor_raw_data_iterator *iterator,
    const struct sensor_scale_metadata *scale_metadata,
    void *sensor_data
);

/** Driver function to update the offset in the iterator */
typedef int (*sensor_raw_reading_iterator_next_t)(
    const struct device *sensor,
    const struct sensor_scale_metadata *scale_metadata,
    struct sensor_raw_data_iterator *iterator
);

/** Driver function to get the sensor type from the current iterator */
typedef int (*sensor_raw_reading_iterator_get_sensor_type_t)(
    const struct device *sensor,
    const struct sensor_raw_data_iterator *iterator,
    uint32_t *sensor_type
);

If I'm understanding this idea correctly, then I think something like the following example might be possible
struct sensor_data
while (sensor_raw_reading_iterator_next(sensor, metadata, iter)) {
  sensor_raw_reading_iterator_get_type(sensor, itetaror, reading_type);
  if (reading_type == SENSOR_TYPE_ACCELEROMETER || reading_type == SENSOR_TYPE_GYROSCOPE || reading_type == SENSOR_TYPE_MAGNETOMETER) {
    sensor_raw_reading_iterator_get_reading(sensor, iterator, metadata, value3i);
    printk("sensor value is ...\n");
  } else {
    printk("I don't know what to do with this, could be temperature\n");
  }
}

I think that makes some sense and seems pretty reasonable.
I'd think really long and hard about if there are ways you can avoid syscalls and branches in the tight loop. That sort of thing adds up. When you want dynamic functionality without branching, function pointers are your friends.
e.g. provide what you want in advance, entirely perhaps, setting a function pointer per type.
struct sensor_three_axis_data sensor3_data[3];
struct sensor_float_data sensor_tmp_data;
struct sensor_readings readings[4] = {
  {
   /* .read_fn =  hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_ACCELEROMETER,
   .sensor_data = &sensor3_data[0],
  },
  {
   /* .read_fn = hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_GYROSCOPE,
   .sensor_data = &sensor3_data[1],
  },
  {
   /* .read_fn = hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_MAGNETOMETER,
   .sensor_data = &sensor3_data[2],
  },
  {
   /* .read_fn = hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_GYRO_TEMP,
   .sensor_data = &sensor_tmp_data,
  }
};

/* the sensor knows the FIFO packet stride (e.g. 20 bytes)
 * as well as which function for each sensor type to use to decode the data
 * given the current format, it also knows if the above is invalid
 *
 * e.g. perhaps I've setup 20bit resolution for tdk 42688, where data is split up across
 * the FIFO packet. No big deal, the driver selects the *correct* decode fn and sets the
 * read_fn pointer.
 */
sensor_setup_iterator(sensor, iterator, readings, // ALL PARAMS NEEDED TO SETUP ITERATOR HERE);

/* no longer syscalls, iterator next jumps the fifo packet stride,
 * calls the read_fn for each reading, no syscalls or branching involved
 */
while(sensor_iterator_next(iterator)) {
  readings[0].x /* setup 0 to be accelerometer */
  readings[1].x /* setup 1 to be gyro */
  readings[2].x /* setup 2 to be magnetometer */
  readings[4].value /* setup 4 to be gyro temp */
}

I hope that helps! The key above, if you want to go this route (I'd recommend it) is using function pointers setup ahead of the loop iteration to avoid all the syscall overhead.
That only works though if the buffer contains uniform data. So maybe others can comment here on that.
@microbuilder I'm especially looking at what you think!

@teburd Sorry, have had my hands full with release schedule and deadlines for some pending conferences. Will dig into this again next week to give some feedback.

If I'm understanding this idea correctly, then I think something like the following example might be possible
struct sensor_data
while (sensor_raw_reading_iterator_next(sensor, metadata, iter)) {
  sensor_raw_reading_iterator_get_type(sensor, itetaror, reading_type);
  if (reading_type == SENSOR_TYPE_ACCELEROMETER || reading_type == SENSOR_TYPE_GYROSCOPE || reading_type == SENSOR_TYPE_MAGNETOMETER) {
    sensor_raw_reading_iterator_get_reading(sensor, iterator, metadata, value3i);
    printk("sensor value is ...\n");
  } else {
    printk("I don't know what to do with this, could be temperature\n");
  }
}

I think that makes some sense and seems pretty reasonable.
I'd think really long and hard about if there are ways you can avoid syscalls and branches in the tight loop. That sort of thing adds up. When you want dynamic functionality without branching, function pointers are your friends.
e.g. provide what you want in advance, entirely perhaps, setting a function pointer per type.
struct sensor_three_axis_data sensor3_data[3];
struct sensor_float_data sensor_tmp_data;
struct sensor_readings readings[4] = {
  {
   /* .read_fn =  hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_ACCELEROMETER,
   .sensor_data = &sensor3_data[0],
  },
  {
   /* .read_fn = hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_GYROSCOPE,
   .sensor_data = &sensor3_data[1],
  },
  {
   /* .read_fn = hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_MAGNETOMETER,
   .sensor_data = &sensor3_data[2],
  },
  {
   /* .read_fn = hidden function pointer here, setup in the next call */
   .sensor_type = SENSOR_TYPE_GYRO_TEMP,
   .sensor_data = &sensor_tmp_data,
  }
};

/* the sensor knows the FIFO packet stride (e.g. 20 bytes)
 * as well as which function for each sensor type to use to decode the data
 * given the current format, it also knows if the above is invalid
 *
 * e.g. perhaps I've setup 20bit resolution for tdk 42688, where data is split up across
 * the FIFO packet. No big deal, the driver selects the *correct* decode fn and sets the
 * read_fn pointer.
 */
sensor_setup_iterator(sensor, iterator, readings, // ALL PARAMS NEEDED TO SETUP ITERATOR HERE);

/* no longer syscalls, iterator next jumps the fifo packet stride,
 * calls the read_fn for each reading, no syscalls or branching involved
 */
while(sensor_iterator_next(iterator)) {
  readings[0].x /* setup 0 to be accelerometer */
  readings[1].x /* setup 1 to be gyro */
  readings[2].x /* setup 2 to be magnetometer */
  readings[4].value /* setup 4 to be gyro temp */
}

I hope that helps! The key above, if you want to go this route (I'd recommend it) is using function pointers setup ahead of the loop iteration to avoid all the syscall overhead.
That only works though if the buffer contains uniform data. So maybe others can comment here on that.
EDIT
This also reduces the API call count for iterators to effectively one call which is nice, even if that call involves some additional branching to setup the function pointers.
@microbuilder I'm especially looking at what you think!

Thanks Tom, after we spoke last I made some changes to the iterator, the main difference between what I just uploaded and what you have here is that the sensor driver returns the iterator API. The reason for that is that iterating could require parsing the FIFO headers which is sensor specific. The key is that once the iterator is created, we don't need a syscall to actually iterate. Please let me know if you can think of any ways in which this might not work. Here's the workflow I'm envisioning:

Applications will get the FIFO API via sensor_get_fifo_iterator_api() (only needed once per sensor)
Interrupt comes in, we grab the timestamp and schedule reading the FIFO
We read the FIFO into struct sensor_raw_data and call the callback set by the application
During the callback

Applications might grab the current struct sensor_scale_metadata (I say may because some applications might never change this value so they can just grab it once at boot).
Applications will schedule the FIFO data for processing (either copy the raw data or set a new raw buffer)

During processing, create an iterator via SENSOR_FIFO_ITERATOR_OF()
In pseudo code (some of these code segments would be in different functions and run on work queues, but in this order):

struct sensor_raw_data raw_data;
struct sensor_three_axis_data three_axis_data;
struct sensor_scale_metadata accel_scale;
struct sensor_fifo_iterator_api iterator_api;
struct sensor_fifo_iterator it;

sensor_get_fifo_iterator_api(sensor, &api);
sensor_get_scale(sensor, SENSOR_TYPE_ACCELEROMETER, &accel_scale);

// Assuming raw_data.header.reading_count > 0
it = SENSOR_FIFO_ITERATOR_OF(&raw_data, &accel_scale);

do {
  uint32_t sensor_type = 0;

  iterator_api.get_sensor_type(it, &sensor_type);

  switch (sensor_type) {
  case SENSOR_TYPE_ACCELEROMETER:
    iterator_api.read(it, &three_axis_data);
    // Access SI unit data in three_axis_data
    break;
  }
} while (iterator_api.next(it);

I added support for flushing the hardware FIFO as well as more documentation.

@teburd any thoughts on the latest patch?

I think its looking pretty good overall, I need to spend some time next week experimenting with it a bit at this point, I can only stare at diffs for so long

@yperess in your example driver (for which, thanks!) it looks like IRQ handling etc is the responsibility of the driver, not the application. Do you expect that to be a hard-and-fast general rule, or are you still anticipating situations where applications might need to configure their own ISRs?

Right so I'm trying to keep a clear separation of the user facing API (which is what this PR is meant for) and the driver developer APIs which will come next. If you'd like to dive a bit into my thinking for those I'm happy to give a few bullet points.

ISR registration should remain as is. There are several considerations for this since it was brought up in the Google doc first, but the biggest one for me is that it's how things are currently done and I don't see a big benefit in forcing applications to take over this. The only thing that I would change is how the handler is done (next bullet).
Handlers are done in different ways right now. Some drivers hand their own threads that the ISR notifies and reading is done there. Others post to the kernel work queue. I believe that there's a middle ground to be had there. I would/will suggest a single thread that will be responsible for all of these, it can be thought of a sensors only work queue.

I'm not yet sure how exactly this would look like, it unfortunately blurs a bit the line between the driver and a sensor subsystem. This is why my original proposal left the ISR up to the application, though I think that will hit too much resistance. The most likely approach I will be suggesting (I can try to add this to this PR) is to have an API for this common sensor subsystem. Then using a Kconfig allow the application or a sensor subsystem to override it. The API will be very very simple. It'll basically just look a bit like the k_work_submit where drivers can rely on some "work submit" function to exist. The default one will simply implement a work queue and callback to the handler of the driver with possibly a nicer API since we can cast some of the structs for the caller.
Let me know if you'd like me to include all of these changes in this PR.

Multiple threads are serious problem as each thread requires memory used for stack. For our needs we have created sensor manager (https://github.com/nrfconnect/sdk-nrf/blob/main/subsys/caf/modules/sensor_manager.c) that would be capable of sampling multiple sensors. It's far from being perfect but does it's job. There were some sensor api limitations (the lack of streaming was one of these) we missed on the way.
Note that this might not be useful for everybody - it can happen that some sensor's handling might be time critical in which trigger from the sensor should be handled separately. This is why I would suggest to leave the senor api with ability to trigger directly and create a higher level component that is able to set and handle multiple triggers (for other, non time critical sensors).

Some potential first driver implementation ideas to validate the API on maybe two fronts, first the FIFO API front: TDK 42688-P (32KHz 20Bit 6 axis sensor with big FIFO)
And the unique sensor with many similar channels fronts:
AS7343 14 Channel Spectral Sensor (light and near light wavelengths)
It would also be great to see a driver that covers atmospheric data, BME280 is a nice one.
The last sort of thing I'd like to consider is the trigger source of reads may be...

GPIO Interrupt from the sensor to read
Chained from another sensors GPIO trigger
A timer
Ad hoc reading (read my sensor right now)

For some reason I can't reply directly on the thread for https://github.com/zephyrproject-rtos/zephyr/pull/44098#discussion_r910828244. @pdunaj the callback doesn't necessarily need to do any conversion. In some cases which @teburd brought to my attention, you might want to take the raw FIFO data and send it off chip for processing. So, yes, likely a lot of applications will want to take the raw data and convert it to SI or some similar units which are easier to read, but the API remains flexible to also account for high throughput applications.
It's also easy to imagine applications that read the struct sensor_scale_metadata which tells them the range and resolution of the data samples. Then, during the sample processing, they don't want to throw away any data under 2g (for accelerometers). They can:

Configure the sensor's sample rate, streaming mode, range, and resolution
Read the scale metadata
Calculate the raw data threshold. For example say we have 16 bit resolution, and a 4g range. That means that a 2g threshold will be 4g (using xx + yy + z*z and the threshold squared to avoid a sqrt call). A 4g reading is 1/2 of the range so that's just (2 << (range - 1) ) - 1.
Now when we process the reading, we can use simple integer arithmetic to filter low value samples.

And a unique sensor with many similar channels: AS7343 14 Channel Spectral Sensor (light and near light wavelengths)

So I've been giving this a lot of thought today. Please correct me if I'm off on any aspects of this and we can iterate on it. After reading the data sheet for the AS7343 I see that the 14 channels are defined by various ranges in wave length. That means that CHANNEL_0 on this device might not map to the same range as another device. This brings up the issue of using something like SENSOR_TYPE_SPECTRAL_CHANNEL_0 is invalid. Similarly, creating specific ranges doesn't make sense as it gets too noisy and doesn't scale (something like SENSOR_TYPE_SPECTRAL_395NM_415NM).
So my current thinking is a bit out there but it'll help align upstream and downstream drivers without the need for a custom API. Lets assume we have the following:
# zephyr,spectometer_range_map.yaml

description: Wavelength mapping for spectrometers
compatible: "zephyr,spectometer-range-map"

child-binding:
  description: Mapping of sensor type to wavelength in nanometers
  properties:
    channel:
      type: string
      description: The channel to configure
      enum:
        - "SENSOR_TYPE_SPECTROMETER_CHANNEL_0"
        - "SENSOR_TYPE_SPECTROMETER_CHANNEL_1"
        - "SENSOR_TYPE_SPECTROMETER_CHANNEL_2"
        - "SENSOR_TYPE_SPECTROMETER_CHANNEL_3"
        - ...
    wave-length-min:
      type: int
      description: The minimum wavelength in nanometers
    wave-length-max:
      type: int
      description: The maximum wavelength in nanometers

The sensor_types.h would now include:
#if DT_HAS_COMPAT_STATUS_OKAY(zephyr_spectometer_range_map)

/* Assert there's only one node for the mapping */
BUILD_ASSERT(DT_NUM_INST_STATUS_OKAY(zephyr_spectometer_range_map) == 1);

#ifdef __cplusplus__
#define SPECTOMETER_DECLARE_CHANNEL(id) \
    constexpr uint32_t _CONCAT(DT_STRING_UPPER_TOKEN(id, channel), _MIN) = DT_PROP(id, wave_length_min); \
    constexpr uint32_t _CONCAT(DT_STRING_UPPER_TOKEN(id, channel), _MAX) = DT_PROP(id, wave_length_max);
#else
#define SPECTOMETER_DECLARE_CHANNEL(id) \
    static const uint32_t _CONCAT(DT_STRING_UPPER_TOKEN(id, channel), _MIN) = DT_PROP(id, wave_length_min); \
    static const uint32_t _CONCAT(DT_STRING_UPPER_TOKEN(id, channel), _MAX) = DT_PROP(id, wave_length_max);
#endif

DT_FOREACH_CHILD_STATUS_OKAY(
    DT_INST(0, zephyr_spectometer_range_map),
    SPECTOMETER_DECLARE_CHANNEL);

#undef SPECTOMETER_DECLARE_CHANNEL

#endif

Next, the driver would be implemented using something like:
/* This can be optimized I'm sure but it's much more readable this way for the sake of an example */
static int read_data(const struct device *sensor, uint32_t sensor_type, void *data) {
  switch (sensor_type) {
    case SENSOR_TYPE_SPECTROMETER_CHANNEL_0:
      return read_data_in_range(
          SENSOR_TYPE_SPECTROMETER_CHANNEL_0_MIN,
          SENSOR_TYPE_SPECTROMETER_CHANNEL_0_MAX);
    ...
  }
}

Thoughts?

And a unique sensor with many similar channels: AS7343 14 Channel Spectral Sensor (light and near light wavelengths)

So I've been giving this a lot of thought today. Please correct me if I'm off on any aspects of this and we can iterate on it. After reading the data sheet for the AS7343 I see that the 14 channels are defined by various ranges in wave length. That means that CHANNEL_0 on this device might not map to the same range as another device. This brings up the issue of using something like SENSOR_TYPE_SPECTRAL_CHANNEL_0 is invalid. Similarly, creating specific ranges doesn't make sense as it gets too noisy and doesn't scale (something like SENSOR_TYPE_SPECTRAL_395NM_415NM).
So my current thinking is a bit out there but it'll help align upstream and downstream drivers without the need for a custom API. Lets assume we have the following:
[snip]
Thoughts?

I don't think you should focus that much on the specific device (AS7343). It was just mentioned as an example. What I'd really like to see is a way for any sensor v2 device to support multiple channels of the same measurement type, e.g. temperature sensor with multiple temperature sensors, all of them exactly equal in functionality, but different inputs on the sensor (similar as to how the e.g. the ADC API allows for ADCs with more than one channel).
The support for this needs to be generic so that applications accessing the sensor data does not have to care about whether the hardware has 7 different sensor devices with one temperature sensor input each, or one sensor device with 7 temperature sensor inputs.
This is one of the identified limitations of the current sensor v1 API. Please see https://github.com/zephyrproject-rtos/zephyr/issues/13718. It would be nice to hear your view on how many of the then identified shortcomings are addressed by the proposed v2 API.

wave-length-min:
  type: int
  description: The minimum wavelength in nanometers
wave-length-max:
  type: int
  description: The maximum wavelength in nanometers

@yperess I think a more common way to describe this would be to define the peak wavelength in nm and then the half maximum wavelength as a delta, so 450 nm peak (avg), and 50 nm half maximum, implying a spread from 425-475 nm from half to peak sensitivity. The AMS data sheet describes the later as “Full Width Half Maximum [nm]”.

And a unique sensor with many similar channels: AS7343 14 Channel Spectral Sensor (light and near light wavelengths)

So I've been giving this a lot of thought today. Please correct me if I'm off on any aspects of this and we can iterate on it. After reading the data sheet for the AS7343 I see that the 14 channels are defined by various ranges in wave length. That means that CHANNEL_0 on this device might not map to the same range as another device. This brings up the issue of using something like SENSOR_TYPE_SPECTRAL_CHANNEL_0 is invalid. Similarly, creating specific ranges doesn't make sense as it gets too noisy and doesn't scale (something like SENSOR_TYPE_SPECTRAL_395NM_415NM).
So my current thinking is a bit out there but it'll help align upstream and downstream drivers without the need for a custom API. Lets assume we have the following:
[snip]
Thoughts?

I don't think you should focus that much on the specific device (AS7343). It was just mentioned as an example. What I'd really like to see is a way for any sensor v2 device to support multiple channels of the same measurement type, e.g. temperature sensor with multiple temperature sensors, all of them exactly equal in functionality, but different inputs on the sensor (similar as to how the e.g. the ADC API allows for ADCs with more than one channel).
The support for this needs to be generic so that applications accessing the sensor data does not have to care about whether the hardware has 7 different sensor devices with one temperature sensor input each, or one sensor device with 7 temperature sensor inputs.
This is one of the identified limitations of the current sensor v1 API. Please see #13718. It would be nice to hear your view on how many of the then identified shortcomings are addressed by the proposed v2 API.

If your goal is to make it so that reading a channel is the same as a different device when they match some configuration then
I think the way to solve that is to use the sensor subsystem as a wrapping layer. That's exactly what the subsystem is designed to do (abstracting away what's a device vs a channel of a device). I don't think we should solve it at this layer as it would complicate the API too much. In the sensor subsystem model this would look like:
int selected_sensor_index = -1;
struct sensor_subsys_info *sensors;
int sensor_count;

sensor_subsys_list(&sensors, &sensor_count);

// Search sensors for SENSOR_SUBSYS_TYPE_SPECTROMETER
for (int i = 0; i < sensor_count; ++i) {
  if (sensors[i].type != SENSOR_SUBSYS_TYPE_SPECTROMETER) {
    continue;
  }
  if (sensors[i].min_range >= 375 && sensors[i].max_range <= 425) {
    selected_sensor_index = i;
    break;
  }

I've done a board port of the tdk robokit1 and icm42688 for the old sensor API, working on updating it to use this API with the FIFO interface to give it a twirl

I hope its ok, but its easier if I just kind of take notes and write them here as I go...
I quickly noted sensor_read_data and I guess I'd rather see it passed in a self descriptive array of readings to fetch and return in a blocking manner rather than what it does today. The problem I see here is the current method requires each sensor type to have its own spi/i2c transaction to fetch the data. This is a lot of overhead and means that the readings may not be in the same sample period. While many implementation in V1 do this, fetch with SENSOR_CHAN_ALL could easily be optimized to be done in a single transaction, this API leaves no room for that.
There's also this implied meaning of accelerometer/gyro that there are 3 axis. There are 1 axis accel/gyro sensors. This is probably less of a problem but still kind of quirky to implement for a 1 axis sensor. Or if for example someone is only using 1 or 2 axis of a 3 axis sensor to save power.
I think I like the sensor_scale_meta struct, just starting in on that part, though there's a few nits there...
For the gyro range its usually degrees or radians per second, not just degrees or radians, the define needs a slight rename to DEG_PER_SEC or RAD_PER_SEC for the scale units.
I guess I would've expected a min/max as part of the scale as well, rather than a single range value. This is usually fine for accel/gyros that center on 0 but not every sensor does that.
I'm also noting a missing ability to configure the sensors interrupts in any way. V1 has the idea of set_trigger() which this is missing entirely I believe. This sensor has 2 interrupt lines that can be configured to catch taps, steps, and more. I can't use that at all with this API, where as the v1 API I could at least configure it kind of sort of maybe.
That's where I got today, tomorrow I plan on trying to finish the non-streaming part of the API and begin in on the FIFO API part.

Some notes about the FIFO interface so far and I think at this point, while I'm still working through implementing everything, there's some clearly identifiable points to discuss.

The callback context isn't clear for the process data callback. It could be ISR, thread, or work queue.
From the flow diagrams there needs to be a lock around the process callback to ensure that a sensor ISR doesn't trigger another read to the same buffer and that the iterator object remains valid.
The iterator API itself I think could be simpler and more useful

The order of the sensor type I guess would be device dependent, and lead to confusing code around it trying to figure out which samples belong to which sample period.
The iteration also going over not just fifo packets by sensor types also means more logic is needed in the iterator itself to figure that out, I guess with modulo operators.
The first call to next() isn't specified, does the iterator start on an invalid item and on the first next call to to the first valid one? That would be nicest for a while(iter_next(iter)) { } style loop.
There's no count of samples in the iterator
There's no way to get ad-hoc samples from the buffer with the iterator which means filtering requires the buffer be decoded and copied before doing a FIR/IIR/Kalman/Mahony/Pattern matching process.

The lifetime of the iterator and data is limited to the scope of the process callback

This means the data can't be moved across a thread of processes while being interpretted/worked on, it must be decoded into a larger buffer first or risk blocking the sensor from reading the next buffer.
Something closer to a queue of available buffers to read into, and a queue of buffers ready to process would be nicer where there's no chance of blocking the sensor or subsequent processing elements. There's no requirement to decode and copy the buffer immediately.

@yperess might be worth discussing some of this in the dev review meeting this week? (9/8) if that makes sense.
I think there's a generally common issue here from my perspective. My 6 axis + temp + timestamp capable sensor can't have readings viewed as one sampling in the same period with this API.