Redesign the pixel encoding procedure

Something that has been growing in my mind for a few days is the idea of changing the way colors are encoded into pixel friendly formats. The current procedure may look like this:
let linear = Rgb::new(1.0, 0.3, 0.6);
let srgb = Srgb::from_linear(linear);
let pixel: [u8; 3] = srgb.to_pixel();

or, using a shortcut:
let linear = Rgb::new(1.0, 0.3, 0.6);
let pixel: [u8; 3] = Srgb::linear_to_pixel(linear);

There's also the case where a simple custom gamma exponent can be used:
let linear = Rgb::new(1.0, 0.3, 0.6);
let pixel: [u8; 3] = GammaRgb::linear_to_pixel(linear, 2.2);

This is fine and all, but it's not really easy to use in a general way:

There's no common trait or type for encoded colors.
The alpha component is baked into the color type.
GammaRgb has a variable gamma exponent, which makes it very different from other encodings, and impossible to fully express as a simple type.

This problem is tightly connected to #31, where a general RGB type is proposed. The way GammaRgb is structures interferes with that proposal, since it has an additional gamma component, besides the usual red, green and blue. I'm therefore proposing that we redesign the way pixel encoding works to make it more friendly towards the idea of a more generalized RGB type. I would also like to extend this to other RGB related color spaces, such as HSV, HSL, Luma and HWB. These can all be seen as input types and they are, often enough, defined in terms of already encoded RGB.
The initial proposal is to keep linear and encoded types separate. The purpose of this library is to strongly encourage a linear work flow and this can easily be achieved by only implementing operations on the linear variant. There was a proposal, by @sidred, to have marker type parameters, such as Rgb<Linear, Srgb, ...> and Rgb<Gamma, Srgb, ...>, but recent problems with such marker parameters, as well as the additional complexity of extra type parameters, makes that method less attractive.
The separation of linear and encoded colors, using two different structs, will come with a bit of duplication, but the duplication will mainly be restricted to construction and fields. The linear variants will implement the operation traits, while the encoded variants will be focused on packing and converting data. Here is an example of how it may look when combined with #31:
pub trait RgbEncoding {
    ///Encode a color component.
    fn encode<T: Float>(value: T) -> T;

    ///Decode a color component.
    fn decode<T: Float>(value: T) -> T;
}

pub trait RgbPrimaries {
    //get red, green, blue primaries, etc.
}

///Marker type, which both defines a set of primaries and an encoding
pub struct Srgb;

impl  RgbEncoding for Srgb {
    //...
}

impl  RgbPrimaries for Srgb {
    //...
}

struct LinearRgb<Prim: RgbPrimaries = Srgb, ...> {
    //..
    primaries: PhantomType<Prim>,
}

struct Rgb<Enc: RgbEncoding = Srgb, ...> {
    //..
    primaries: PhantomType<Enc>,
}

Encodings and primaries are often strongly connected to each other, which is why the same type can be used for both. The problem with this is that it's hard to decouple them, but that's hopefully rare enough to make this more good than bad. The only problem left is the GammaRgb. Its variable gamma component makes it incompatible with the above pattern, and having floating point values as type parameters is nothing but a (bad?) dream at this stage. We can, however, make more assumptions and add more escape hatches. I'm therefore proposing the addition of the probably most common gamma exponent, 2.2, as an encoding type, and the extension of the above RgbEncoding that makes it possible for an encoding to map to any set of primaries:
pub trait RgbEncoding {
    type Primaries: RgbPrimaries;

    ///Encode a color component.
    fn encode<T: Float>(value: T) -> T;

    ///Decode a color component.
    fn decode<T: Float>(value: T) -> T;
}

///Gamma exponent 2.2, with any primaries.
struct Gamma22<P: RgbPrimaries>;

impl<P: RgbPrimaries> RgbEncoding for Primaries<P>
    type Primaries = P;

    //...
}

impl  RgbEncoding for Srgb {
    type Primaries = Srgb;
    //...
}

fn main() {
    let encoded = Rgb::<Gamma22<Srgb>>::new(...);
    let linear = encoded.to_linear(); //Gives LinearRgb<Srgb>
}

The preferred primaries can then be fetched from the associated Primaries type when converting from encoded to linear. Now, what about those other gamma values? That's where we need escape hatches. One such escape hatch could be to keep the old GammaRgb type around. It may be a bit unwieldy, but it works. An other is to just have a straight-to-pixel conversion, like the *::linear_to_pixel function in the current design. More suggestions are welcome here.
The last question is what encoded types are allowed to do. My proposal is "basically nothing", since I'm still strongly in favor of keeping them strictly separated from linear types. This is to minimize any risk of mistaking non-linear values for linear and forcing the user to make an explicit conversion between them. The minimal set of functions I can think of at the moment is:
impl<E: RgbEncoding, ...> Rgb<E, ...> {
    fn new(red: T, green: T, blue: T) -> Rgb<E, ...> {...}
    fn new_u8(red: u8, green: u8, blue: u8) -> Rgb<E, ...> {...}
    fn from_pixel<P: RgbPixel<T>>(pixel: P) -> Rgb<E, ...> {...}
    fn to_pixel<P: RgbPixel<T>>(&self) -> P {...}
    fn from_linear<C: Into<LinearRgb<E::Primaries, ...>>(color: C) -> Rgb<E, ...> {...}
    fn to_linear<C: From<LinearRgb<E::Primaries, ...>>(&self) -> C {...}

    //Useful addition
    fn linear_to_pixel<C: Into<LinearRgb<E::Primaries, ...>, P: RgbPixel<T>>(color: C) -> P {
        Rgb<E, ...>::from_linear(color).to_pixel()
    }
}

These can, of course, be tweaked and changed. Some of these functions would be nice as a general encoding trait, but that's quite complex and may be better discussed at a later stage.
Something this proposal hasn't touched, so far, is what to do with the alpha component. I think it would be best to make use of the Alpha type, but I haven't figured out exactly how, so that question remains open.
An other open question is type aliases for common RGB types, such as AdobeRgb. One idea is to not alias sRGB, and only implement new for Rgb<Srgb, D65, T>, similar to how it's done with only D65 today. An other is to alias all of them and not pick a default.

A few things
Is it necessary to separate RgbEncoding and Rgb Primaries? They both are properties of the specific Rgb Color space. Example
///Represents the tristimulus values for the Rgb primaries
pub trait RgbVariant<Wp, T>
    where T: Float,
        Wp: WhitePoint<T>,
{

    fn red() -> Yxy<Wp, T>;

    fn green() -> Yxy<Wp, T>;

    fn blue() -> Yxy<Wp, T>;

    ///The default gamma value for the rgb color space
    fn gamma() -> T;

    fn into_gamma(inp: Rgb<Wp, T>) -> Rgb<Wp, T>{
        let gamma = Self::gamma();
        let inv_gamma = if gamma.is_normal() { T::one() / gamma } else { T::one() };
        Rgb::with_wp(inp.red.powf(inv_gamma), inp.green.powf(inv_gamma), inp.blue.powf(inv_gamma) )
    }

    fn into_linear(inp: Rgb<Wp, T>) -> Rgb<Wp, T> {
        let gamma = Self::gamma();
        Rgb::with_wp(inp.red.powf(gamma), inp.green.powf(gamma), inp.blue.powf(gamma) )
    }

    fn mat3_from_primaries() -> Mat3<T> {
       ....
    }

    ///Geneartes to Rgb to Xyz transformation matrix for the given white point
    fn rgb_to_xyz_matrix() -> Mat3<T> {
      ...
    }

    ///Geneartes the Xyz to Rgb transformation matrix for the given white point
    fn xyz_to_rgb_matrix() -> Mat3<T> {
       ...
    }

}

In this case Srgb can override the default gamma conversion with a custom conversion and for all other colors we just need to provide the default gamma value.

The preferred primaries can then be fetched from the associated Primaries type when converting from encoded to linear.

What do primaries have to do with gamma encoding?

Now, what about those other gamma values? That's where we need escape hatches. One such escape hatch could be to keep the old GammaRgb type around. It may be a bit unwieldy, but it works.

If we have the encoding and decoding on the specifc rgb color space, then there will be no need to have the GammaRgb type. The Rgb type above will hold the gamma encoded values and LinearRgb will hold the linear values. Am I missing some use case that you have in mind?
I need to think a bit more about the pixel type.

Am I missing some use case that you have in mind?

I'm trying to keep things open for customization, and dynamically chosen gamma exponent can't be represented as a type. That was the whole point with the GammaRgb type, and it's also the point of separating primaries from encoding. Why assume that primaries and encoding are always tightly coupled? Some custom RGB variant may use the same primaries as sRGB, but a different encoding. Others may just want to encode with some gamma exponent and be done with it. Some primaries may not even have an associated encoding. Separating the definition of primaries and encoding makes these situations easier.

What do primaries have to do with gamma encoding?

Very good question. Answer: Nothing, really. They are just paired together in an RGB standard, together with a white point, and may be anything. The RgbEncoding trait could also be called RgbStandard. That's also what your RgbVariant seem to be, but the difference is that the RgbEncoding doesn't assume one single set of primaries for an encoding.

One issue with having Gamma and Linear types separate is you cannot prevent mistakes like this
let rgb = RgbLinear::new();
let val = rgb.into_linear();// should error or return the same value

This can be avoided if the gamma or linear is part of the trait.

Into_linear shouldn't even be implemented for LinearRgb.

Into_linear shouldn't even be implemented for LinearRgb.

I was confusing the trait implementation with the struct implementation
So we can just have 2 traits. Having fn gamma() on the Gamma trait will make GammaRgb redundant
pub trait Gamma {
    fn gamma() -> T;

    ///Encode a color component.
    fn to_gamma<T: Float>(value: T) -> T {
      let gamma = Self::gamma();
      let inv_gamma = if gamma.is_normal() { T::one() / gamma } else { T::one()}
      value.powf(inv_gamma)
   }

    ///Decode a color component.
    fn to_linear<T: Float>(value: T) -> T {
      let gamma = Self::gamma();
      value.powf(inv_gamma)
   }
}

pub trait RgbPrimaries {
    //get red, green, blue primaries, etc.
}

Having an impl like this will take care of the standard converison linear gamma conversions.
impl<R: Gamma + RgbPrimaries> LinearRgb<R> {
    to_linear() {
     R::to_linear() // will take care of the default Srgb, Adbode Rgb conversions from the traits
  }
}

Why not implement Blend etc on the gamma encoded value by converting to Linear, atleast for the known types.
What about from and into color? Do we add from_rgb_linear and assume existing from_rgb converts to the gamma value?

So we can just have 2 traits. Having fn gamma() on the Gamma trait will make GammaRgb redundant

That's basically what this proposal is all about, but without assuming exponential encoding, or tight coupling between encoding and primaries. It's common, but we don't have to assume it.

Having an impl like this will take care of the standard converison linear gamma conversions.

Does that convert from linear to linear?

Why not implement Blend etc on the gamma encoded value by converting to Linear, atleast for the known types.

Well, hmm... I suppose they could. They were mainly meant to be temporary states, but that doesn't stop them from jumping to and from being linear, just like with premultiplied alpha.

What about from and into color? Do we add from_rgb_linear and assume existing from_rgb converts to the gamma value?

I'm not sure if they should be concerned by the encoded variants at all, but then I don't know what happens if we add things like non-linear HSV...

I'm not sure if they should be concerned by the encoded variants at all, but then I don't know what happens if we add things like non-linear HSV...

Lets take an example of converting D50 to D65
let c: Rgb<Srgb, D50, _> = Rgb::new(0.5,0.5,0.5)

let o: Rgb<Srgb, D65, _> = c.adapt_into();

That would be a lot easier than
// Might rrquire more type annotations / intermediate variables
let o: Rgb<Srgb, D65, _> = c.into_linear().adapt_into().to_gamma();

Also for IntoColor and AdaptInto to work for both Rgb and LinearRgb, you would need a trait bound of  Rc: RgbPrimaries + Gamma.
So having a custom Gamma   Rgb<CustomGamma, Wp, T> will not mean that   custom_rgb.into_xyz() will work. You would need to convert it into a known linear or gamma encoded Rgb for it works. So from a  usability point of view I am not sure separating RgbVariants into Gamma and Primaries, gets us anything.
If the user has just a separate gamma, wouldn't it be easier to let users convert it into a known color directly or implement the entire RgbVariant trait? That way all other operations will work easily and will be a more explicit error.
Do we need Hsl, LinearHsl, Hwb, LinearHwb, Hsv, LinearHsv as well?

Having an impl like this will take care of the standard converison linear gamma conversions.
Does that convert from linear to linear?

LinearRgb<Srgb, D50,_> to LinearRgb<adobeRgb, D50,_> would be implemented via chromatic adpatation, not from and into color.

Would it have to? The white point doesn't change.

Yeah, I guess these will need to be taken care of in the from_rgb and into_rgb by conversion via xyz .

We need to rethink the conversions. Right now From and Into only handle different colors (same white point and rgb variant). Chromatic adaptation handles different whitepoints and uses From and Into for conversions.
We need to handle Rgb<Srgb,> to Hsl<AdobeRgb> and Rgb<AdobeRgb>. May ne another trait or within the from and Into.

Sorry, I somehow missed the other post.
I'm not sure I see what the problem is. IntoColor can't be infinite. We have to draw a line somewhere, so I'm thinking that maybe it should just be limited to the linear types. Anything can use it anyway, and the reverse conversion can be done with FromColor. They don't have to take the encoding into account at all. It's not interesting when defining the linear RGB, since it's not encoded.
Separating primaries and encoding will simply separate two independent, but often coupled, concepts. One is part of the definition of the color space, and the other tells us how to store the values. The only thing they really have in common is that standardized encodings comes with an RGB space definition, including primaries, but any primaries can really be used with any encoding.
I want to model this independence in a way that is as usable as possible, which is why I think that the concepts should be separated. The encoding doesn't have to have a preferred set of primaries. It could just as easily be replaced with yet another type parameter: Rgb<Encoding, Primaries, Wp, T>, but I'm not that fond of it. The thing is that I don't want to blur the lines between linear and encoded colors too much. The encoded variants are only really interesting as input/output.

If the user has just a separate gamma, wouldn't it be easier to let users convert it into a known color directly or implement the entire RgbVariant trait? That way all other operations will work easily and will be a more explicit error.

Separating encoding and primaries makes this even easier. Just implement the encoding trait and don't care about primaries.

Do we need Hsl, LinearHsl, Hwb, LinearHwb, Hsv, LinearHsv as well?

Yes, eventually. They are tightly connected to RGB. Didn't the colormine tests assume that they converted to encoded RGB?

We need to rethink the conversions. Right now From and Into only handle different colors (same white point and rgb variant). Chromatic adaptation handles different whitepoints and uses From and Into for conversions.

I'm not sure this is entirely related to this particular topic, but yes, we need to decide the limits of those conversions.

Gamma + Primaries vs single RgbVariant - It's just an implementations detail. I prefer a single trait, I have only seen rgb color spaces with defined gama's used (like Srgb, adobe rgb etc)), but I don't have a strong opinion on it. We can start with the 2 separate traits.

Pixel Type - Having a separate Gamma Trait should take care of your concerns regarding the pixel types. Maybe add more methods on the pixels conversion, but those are implementation details. There is nothing else needed for the pixel type. Please correct me if I am wrong.

Hsl/Hwb/Hsv - Do you agree that these need a linear and gamma variant as well?

Luma - Does luma need to be color space aware too?

Gamma Encoded color -  If I understand you correctly,  you want to make gamma rgb a sort of secondary color, all operations are only in linear space. But the CSS spec assumes gamma Srgb and all the formula's for blend (from the w3.org link) I believe also assume gamma encoded values. I know its not recommeded but, it will be useful to follow  the standard. So I am proposing that we implement those for gamma types as well.

From / Into - I am proposing that we add into_linear_rgb, into_linear_hsl etc..  so that GammaRgb -> GammaHsl, GammaHwb -> Xyz etc work seamlessly. It will be easier this way than color.to_linear().to_xyz().

ColorSpace conversions - I think this be taken care of in the implementation of the from and into color traits for the linear and gamma types.

Color Enum - I am not sure how to handle this. Do we add all linear and gamma variant (Rgb, Hsl, Hwb etc)?

Again, there are more encoding methods than a gamma exponent. That's why I don't call it gamma, and that's also why I don't think adding a gamma() method to the encoding trait is such a great idea. sRGB is a common example of a non-exponential encoding, but there are more.

I prefer a single trait, I have only seen rgb color spaces with defined gama's used (like Srgb, adobe rgb etc)), but I don't have a strong opinion on it.

Here are some more: http://www.color.org/chardata/rgb/rgb_registry.xalter Some of them are linear and some of them are a bit more exotic.

Having a separate Gamma Trait should take care of your concerns regarding the pixel types. Maybe add more methods on the pixels conversion, but those are implementation details. There is nothing else needed for the pixel type. Please correct me if I am wrong.

I'm not 100% sure I follow. I would gladly remove the pixel types, but I have seen them used as input types to functions, so I think they may still be useful in those situations. They are otherwise not particularly useful and their conversion functionality can be replaced by functions, as demonstrated by the existence of Srgb::linear_to_pixel.

Hsl/Hwb/Hsv - Do you agree that these need a linear and gamma variant as well?
Luma - Does luma need to be color space aware too?

Yes, but they will just have to follow the same pattern as for RGB. We can ignore them for now. The idea is to convert like this: Hsl -> EncodedRgb -> LinearRgb -> whatever. Luma is gray, which is why it's also affected.

If I understand you correctly, you want to make gamma rgb a sort of secondary color, all operations are only in linear space. But the CSS spec assumes gamma Srgb and all the formula's for blend (from the w3.org link) I believe also assume gamma encoded values. I know its not recommeded but, it will be useful to follow the standard. So I am proposing that we implement those for gamma types as well.

Working with encoded colors will give the wrong result in most situations, except where the opposite is defined. The very description of Palette is "A Rust library for linear color calculations and conversion", and I would like to keep that focus. It's, by the way, very easy to ignore the the linearity restriction, by interpreting pixel values as if they were linear.

I am proposing that we add into_linear_rgb, into_linear_hsl etc.. so that GammaRgb -> GammaHsl, GammaHwb -> Xyz etc work seamlessly. It will be easier this way than color.to_linear().to_xyz()

Conversion to and from linear should only be necessary during input/output. A color doesn't have to be listed in the trait to implement it, meaning that color.to_xyz() is still possible.

Color Enum - I am not sure how to handle this. Do we add all linear and gamma variant (Rgb, Hsl, Hwb etc)?

I think only the linear versions are necessary, given that we only restrict the implementations of various operations to them. Color is lazy mode and could work just as well with only a few variants, but more are included to make it a bit less lossy.

Again, there are more encoding methods than a gamma exponent. That's why I don't call it gamma, and that's also why I don't think adding a gamma() method to the encoding trait is such a great idea. sRGB is a common example of a non-exponential encoding, but there are more.

I am not disagreeing with this. I expect the Rgb encoding to have 3 methods
trait RgbEncoding {
     ///The default gamma value for the rgb color space
    fn gamma() -> T;

    ///Converts linear rgb into the gamma encoded rgb
    fn encode(inp: Rgb<Wp, T>) -> Rgb<Wp, T>{
        let inv_gamma = 1.0 / Self::gamma();
        Rgb::with_wp(inp.red.powf(inv_gamma), inp.green.powf(inv_gamma), inp.blue.powf(inv_gamma) )
    }

    ///Converts gamma enocded rgb into the linear rgb
    fn decode(inp: Rgb<Wp, T>) -> Rgb<Wp, T> {
        let gamma = Self::gamma();
        Rgb::with_wp(inp.red.powf(gamma), inp.green.powf(gamma), inp.blue.powf(gamma) )
    }

}

So this is just a basic implementation which avoid repetition for common use cases like AdobeRgb and AppleRgb. Just set the gamma and you're done. For Srgb etc. gamma() is not used and encode and decode method need to be overridden. This is just to simplify the implementation for a common use case.

I'm not 100% sure I follow. I would gladly remove the pixel types,

Pixel type needs to be there. I was trying to understand the point you were making in the first comment regarding the pixel type. I wanted to know if you expect any major changes to the pixel type or will having the RgbEncoding trait cover most of the use case?

Conversion to and from linear should only be necessary during input/output. A color doesn't have to be listed in the trait to implement it, meaning that color.to_xyz() is still possible.

Yes we can implement from and into color for gamma rgb but it  will be different from the rest of the colors.
This works
let xyz = lin_rgb.into_xyz()
let xyz = gamma_rgb.into_xyz()

This won't work
let linear_rgb = xyz.into_rgb()
// will not work
let gamma_rgb = xyz.into_rgb()

//Needs to be
let gamma_rgb = xyz.into_rgb().encode()
//or if From is impelmented
let gamma_rgb = FromColor::from_xyz(xyz)

//This will be a lot easier to use
let gamma_rgb = xyz.into_gamma_rgb()

Adding into and from for the encoded colors makes sense from a usage point of view.

So this is just a basic implementation which avoid repetition for common use cases like AdobeRgb and AppleRgb. Just set the gamma and you're done. For Srgb etc. gamma() is not used and encode and decode method need to be overridden. This is just to simplify the implementation for a common use case.

But gamma must still return a value. There are more ways to achieve the same goal. Like this, for example:
pub trait ExponentialEncoding {
    pub fn gamma<T: Float>() -> T;
}

impl<E: ExponentialEncoding> RgbEnchoding for E {
    //...
}

pub struct Gamma18;

impl ExponentialEnchoding for Gamma18 {
    fn gamma<T: Float>() -> T {
        flt(1.8)
    }
}

type AppleRgb = Gamma18;

...modified for however the primaries are to be connected to the encoding.

Adding into and from for the encoded colors makes sense from a usage point of view.

I'm concerned about them becoming too monolithic, but I don't know... Maybe an Encode trait should be a thing. It may even be possible to blanket implement it for each output.

I was playing around with the idea of having separate traits for RgbStandard, Primaries and Encoding. Here's a small and simplified proof of concept: https://play.rust-lang.org/?gist=d6ce765ba1ee1b735ba5&version=stable
This allows RGB standards that functions as both primary sets and encodings, loose encodings and primary sets, and on-the-fly custom standards using a (Primaries, Encoding) tuple. A color can easily be decoded, using the associated primaries from the standard, but encoding it will still require an encoding to somehow be specified. That's hard to avoid but it should hopefully not be a huge deal.

I was just thinking something similar. Is it possible to take it one step further and include white point in it as well

impl<P: Primaries, E: Encoding, W: WhitePoint> ColorProfile for (P, E, W) {
    type Primaries = P;
    type Encoding = E;
   type WhitePoint= W;
}

We can then use this for all colors including the cie colors. For cie colors when the primaries/encodings are not needed, you can just use dummy values ( like SrgbEncidng, SrgbPrimaries  etc) which will not be used any way.
We can even define a WhitePoint<Dummy, Dummy, Wp> if it makes it any simpler for the cie colors.
I think it will make the library a lot simpler.
What's not clear to me is how it will play with the encoded colors.
Do we include the float trait in it as well?
One downside I see is that right now it is easier to see which trait is being used where. For example From and Into color use the Primaries Trait for Xyz <-> Rgb, but with the single trait it will be a bit harder to track the effects of a particular trait.

I was just thinking something similar. Is it possible to take it one step further and include white point in it as well

Absolutely!

We can then use this for all colors including the cie colors.

I'm not so sure about that, on the other hand... Any RGB specific infos totally irrelevant for the CIE types, unless they are converted to RGB, in which case the RGB type can provide them. I fear that they may become type checking noise, and behave badly because of Dummy != Srgb.

Do we include the float trait in it as well?

Yes, but not in the RgbStandard. I just didn't use it in this example, for simplicity. It shouldn't cause any trouble.

One downside I see is that right now it is easier to see which trait is being used where. For example From and Into color use the Primaries Trait for Xyz <-> Rgb, but with the single trait it will be a bit harder to track the effects of a particular trait.

Downside with my proof of concept or with the ubiquitous ColorProfile trait? I think my proof of concept, where only the necessary traits and types are used, is fine in this case.

Downside with my proof of concept or with the ubiquitous ColorProfile trait? I think my proof of concept, where only the necessary traits and types are used, is fine in this case.

I was talking about the ColorProfile trait.
The more I think about, the more I like the ColorProfile trait. I think for most users of the library this will be a lot simpler. You just choose a color profile and it keeps track through all conversions etc and will need fewer type annotations.
For advanced as you have said it is very customizable. Let's think on it a bit more.
For now I will change the Rgb colors to use the ColorProfile trait and lets see if it makes sense.

I'm still not convinced. Forcing the user to pick a particular set of primaries and encoding, even if it's irrelevant, doesn't feel right. It's also like trading one problem for another. What if the user uses more than one "profile"? This extra information will only be used when converting to and from RGB, so it's otherwise just baggage that will interfere with type checking. Xyz<Srgb> to Xyz<AdobeRgb>, for example, will be considered different types, even if they will look and behave exactly the same. The only difference is that they convert to different RGB standards, by default.

For now I will change the Rgb colors to use the ColorProfile trait and lets see if it makes sense.

Can we at least call it something else, and save ColorProfile for actual ICC color profiles?

So here is the plan for the Rgb colors
trait RgbColorSpace<T: Float> {
    type TransferFunction: TransferFunction<T>;
    type Primaries: Primaries<T>;
    type WhitePoint: WhitePoint<T>;
}

///Represents the tristimulus values for the Rgb primaries
pub trait TransferFunction<T: Float>
{
    fn to_gamma(T) -> T;
    fn to_linear(T) -> T;
}

pub trait Primaries<T: Float> {
    type W: WhitePoint<T>;

    fn red() -> Yxy<Self::W, T>;
    fn green() -> Yxy<Self::W, T>;
    fn blue() -> Yxy<Self::W, T>;
}

impl RgbColorSpace for SrgbColorSpace {
 type WhitePoint = D65;
type Primaries = SrgbPrimaries;
type TransferFunciton  =SrgbTransferFunciton
}

pub struct RgbLinear<R = SrgbColorSpace, T = f32>
    where T: Float,
        R: RgbSpace<T>,
{
    pub red: T,
    pub green: T,
    pub blue: T,
    pub rgb_apce: RgbColorSpace<T>
}

I was planning to make the the RgbColorSpace whitepoint same as the primaries white point, but then I realized that you might need something like RgbColorSpace<SrgbTransFn, SrgbPrimaries, D50> where the Srgb primaries are in D65 and need to be converted into D50 space for any conversion implementations.
Also, in the color profile, encoding means something else and the gamma correction function is called color component transfer function. So I changed it. Let me know if you prefer TransferFunction or Gamma or GammaCorrection. I think Gamma is the more common term.

First, as I have said multiple times now, "gamma" is the exponent in the f(x, gamma) = xgamma function. It's very specific, while a transfer function can be non-exponential. "Transfer function" is an ok term, but I would prefer something else than to_gamma for the previously mentioned reason. Maybe from_linear. Good enough for me. I'm also not 100% convinced regarding the RgbColorSpace name, but the only alternative I have at the moment is RgbStandard (not too bad IMO).
The traits can, in general, be simplified by moving the type parameters and associated types to their functions. That will avoid some conflict risks. Something like this (I'm also playing with other ideas):
trait RgbStandard {
    type TransferFunction: TransferFunction;
    type Primaries: Primaries;
    type WhitePoint: WhitePoint; //The WhitePoint trait shouldn't really need its Float parameter
}

pub trait TransferFunction {
    fn from_linear<T: Float>(T) -> T;
    fn to_linear<T: Float>(T) -> T;
}

///Represents the tristimulus values for the Rgb primaries
pub trait Primaries {
    //The primaries are the same, no matter the white point.
    fn red<W: WhitePoint, T: Float>() -> Yxy<W, T>;
    fn green<W: WhitePoint, T: Float>() -> Yxy<W, T>;
    fn blue<W: WhitePoint, T: Float>() -> Yxy<W, T>;
}

//We can make use of the fact that one type can be many things here,
//like in my proof of concept. Just put it in a module and there won't be
//any significant naming collisions.
impl RgbStandard for Srgb {
    type WhitePoint = D65;
    type Primaries = Srgb;
    type TransferFunciton = Srgb;
}

//I'm still not convinced that we should take the
//transfer function into account in linear RGB.
//I'm therefore checking how it looks without them,
//but a third alternative could be a trait for only primaries
//and white point.
pub struct RgbLinear<P = Srgb, W = D65, T = f32>
    where T: Float,
        P: Primaries,
        W: WhitePoint
{
    pub red: T,
    pub green: T,
    pub blue: T,
    pub rgb_space: PhantomData<(P, W)>,
}

I didn't think about it before, but the WhitePoint trait should not really be parametric over the float type. It should actually be just as generic as Primaries above, since its purpose is to represent a perception-independent and type-independent value.

First, as I have said multiple times now, "gamma" is the exponent in the f(x, gamma) = xgamma function
True. But gamma in color is typically associated with any non linear transformation, not necessarily just an exponent. Srgb is called gamma corrected even though it is strictly not gamma (has a linear transform for low values). While not semantically correct, gamma is a more familiar term and it will be easier for understand. I am ok with using TransferFunction.

I'll change RgbColorSpace to RgbStandard.
I don't see how this will work,
impl RgbStandard for Srgb {
    type WhitePoint = D65;
    type Primaries = Srgb;
    type TransferFunciton = Srgb;
}

Won't I have to do
import primaries::Srgb as SrgbPrimaries;
import transfer_fn::Srgb as SrgbFn;
impl RgbStandard for Srgb {
    type WhitePoint = D65;
    type Primaries = SrgbPrimaries;
    type TransferFunciton = SrgbFn;
}

If we have to rename them, I'd rather name them separately. I think it will avoid confusion.
I don't fully understand why
pub trait TransferFunction {
    fn from_linear<T: Float>(T) -> T;
    fn to_linear<T: Float>(T) -> T;
}

is simpler than having a bound on the trait (for whitepoit or primaries or any other trait). There will never be a case when from_linear will return f32 and into linear will return f64. Is that something that needs to be enforced (that they return the same type of value?)? Do we do this for FromColor and IntoColor as well.

I'm still not convinced that we should take the

Look at this use case. I have some color in Rgb, I convert it into hsl and do some operations on it. convert it back to rgb and return it. I am not using the whitepoint here even though I am forced to select a white point here. I think we should optimize for the most common case case here rather than choosing which characteristics are the important and which are not. That is part of the reason I was suggesting we make cie colors also use color space trait.
Thinking about this can we make it so that only the traits being used need to be provided. Ex

Only using Rgb <-> Hsl no traits
Using Rgb <-> Xyz conversion etc. Need to specify white point
Converting D50 to D65 within cie space (xyz , lab), White point needed
Converting D50 to D65 for rgb, White point and primaries needed.
Rgb to Linear - Only TransferFunction needed.

Note that I'm not angry, annoyed or anything like that, even if this turned into a bit of a rant. It's just that we seem to have slightly different views of what these type parameters are meant to do, and these views are currently colliding.

If we have to rename them, I'd rather name them separately. I think it will avoid confusion.

I thought they could be the same type. I don't see a reason why the type Srgb can't represent an an RGB standars, as set of primaries and a transfer function, at the same time.

I don't fully understand why [...] is simpler than having a bound on the trait (for whitepoit or primaries or any other trait).

The float types are not part of the transfer function type, so I would say that making the trait parametric over the float types would be restricting in the wrong way. A transfer function is what it is, no matter if it's used for f32 or f64. It's not like in colors, where they actually stores the values.

There will never be a case when from_linear will return f32 and into linear will return f64.

We don't know that. It may be improbable, but not impossible. The user is free to do whatever they want with them, including using them for multiple float types. fn do_something<T: TransferFunction>() -> ...{ can't easily be expressed if it has to have a type parameter. You would need multiple transfer functions, or maybe be able to do something like T: for<F: Float> TransferFunction<F>, but I don't know if that's possible. It's at least very obscure. You would otherwise have to enumerate the floats you need:  T: TransferFunction<f32> + TransferFunction<f64>.

Do we do this for FromColor and IntoColor as well.

I don't think so. That's also a different case, since it's more connected to the types that implements them.

I think we should optimize for the most common case case here

Let's take a look at what's the most common case. These are the projects here on GitHub where "palette" occurs in a .toml file. They are only 13, some of them are false positives and some of them are no longer active. Not much for statistics, but still.

One includes the library, but doesn't use it. Let's ignore that one, since we don't know the intent.
One uses it as a pixel type.
Four uses it for just HSL/HSV <-> RGB conversion. Two of them it for gradients as well, and one of them does some more extensive shading and mixing as well.
One uses it for Lab <-> Lch <-> Rgb conversion.

That's six useful samples, one without clear intent, and six false positives. I guess we should really optimize for HSL/HSV <-> RGB (66.66% of the cases) and maybe gradients (33.33% of the cases), based on those. None of that needs any information about white point and primaries. Only the transfer function, for "proper" (linear) gradients and shading. 33.33% of all of them makes use of the built in sRGB encoding. It could be 50% if one of them didn't use its own encoding system.
I don't think that's a helpful source for optimization decisions, at the moment.

rather than choosing which characteristics are the important and which are not.

I'm trying to model the definitions of the colors and the only reason why the white point is everywhere is that it's a part of every definition, except two. Filling them in seemed to be more good than bad.

RGB is defined by a set of parameters and a white point.
HSL, HSV, HWB, etc. are all different ways to express RGB, so they have the same basic definition.
XYZ is more or less atomic in the sense that the physical meaning of its parameters are fixed.
xyY is an alternative way of expressing XYZ, although a bit warped, but its physical meaning is fixed.
Luma is defined by the white point, since it's a scale from black to white.
Lab is defined by the white point, since it's normalized by it.
Lch is a different way to express Lab.

This is the minimal knowledge you need to express their physical properties. Extending XYZ and xyY, the only non-preceptive spaces, with the white point makes sure they are also preceptive at the same at the type level, saying that they are meant to be observed under the same conditions.
We could express a limited set of the same properties by removing all of the type parameters and using names such as RgbSrgbD65F32, since that's how the type checker interprets it. This is also why I'm so much against using every parameter everywhere, because we would end up with the equivalence of XyzSrgbD65Gamma22F32 and XyzAdobeRgbD65LinearF32. They are both physically and visually similar, but still completely different types, because even more intent is baked into them. They are different because they appear to originate from or appear be intended to be converted to different RGB models.
So, the question isn't exactly what's important or not, but how much intent we want to reflect with the types. We must also remember that this isn't optional. There is no way to opt out of expressing some intent if there is a type parameter for it, so it's always or never.
The primaries are only really relevant when converting to and from the RGB group, and the transfer function is only really relevant when converting between the linear and non-linear RGB groups.
Converting to and from linear RGB should normally only happen on two occasions in a program; during input and output, and it will probably not be much worse than Srgb::from_linear(color).to_pixel(). Sometimes even less, since the output function may take Srgb as input, which makes type inference kick in and allows things like blit(x, y, color.encode()).
Converting to and from something outside the RGB group will only happen during more advanced color computations and that's what I would expect happens most, in a more advanced application. It will hopefully still not be too bad, since such an application is hopefully divided into functions, where the input and output types should define the RGB space well enough for type inference.
The way I see it is that there isn't much impact we can have on these two situations without making other situations way worse. Extending the bundled intent of every color with RGB primaries and transfer functions will reduce Srgb::from_linear(color).to_pixel() to color.encode().to_pixel() and (in worst case) LinearRgb::<Srgb>::from(xyz) to (in best case) xyz.to_rgb(). The side effect would be that LinearRgb::<Srgb>from(liner_adobe_rgb.to_xyz()) would turn into something like LinearRgb::<Srgb>from(liner_adobe_rgb.to_xyz().interpret_as()) or worse.
Sure, most users would probably stick to only one RGB model, but I think they can survive having to specify it once or twice, if that would mean that almost every other scenario wouldn't get worse. The combination of carefully designed traits, a good selection of type aliases, and the usual type inference will probably be helpful enough.
That's the way I'm reasoning.

Converting to and from linear RGB should normally only happen on two occasions in a program; during input and output, and it will probably not be much worse than Srgb::from_linear(color).to_pixel(). Sometimes even less, since the output function may take Srgb as input, which makes type inference kick in and allows things like blit(x, y, color.encode()).

The two conversions might happen in different parts of the program and we are losing that information. Think of a pipeline that does multiple operations on a color and at the end with the linear color I have no way of encoding it into the original form without manually keeping track of it. Is this a common enough use case to need an explicit type?
Finally, any reason to choose the trait over a type alias
type Srgb = Rgb<SrgbTf, SrgbPrimaries, D65>

// vs

pub struct Srgb;

impl<T: Float> RgbColorSpace<T> for Srgb {
    type TransferFunction = SrgbTransferFunction;
    type Primaries = SrgbPrimaries;
    type WhitePoint = D65;
}

The two conversions might happen in different parts of the program and we are losing that information.

I think it's ok to lose some of that information, as long as the risk of any mix-ups and accidents is minimized. The transfer function is pretty insignificant during computations, since it doesn't affect how the color is perceived (except if it's misinterpreted, but that's always a risk), and it will be known again when the color is encoded. No big deal, IMO. The primaries are more critical, since they do have an effect on the colors, but I don't know if they are critical enough to be as ubiquitous as the white point.

Think of a pipeline that does multiple operations on a color and at the end with the linear color I have no way of encoding it into the original form without manually keeping track of it. Is this a common enough use case to need an explicit type?

I'm very tired and may have misunderstood everything, but I'll try to answer anyway. The user will know what they start out with and what they want to end up with. That should be enough for the type inference to fill in most of the blanks. Let's say you start out with Srgb and want to end up with Srgb. Then this tiny example is completely valid:
//`pix` is some pixel representation, like `[u8; 3]`.
//`Srgb<T = f32>` is an alias for `Rgb<rgb_standards::Srgb, T>`.
let color = Srgb::from_pixel(&pix).into_linear(); //We just lost the transfer function
let lab = color.into_lab(); //We just lost the primaries
let lab_shifted = lab.shift_hue(180.0.into());
//The compiler knows that this will become an `Srgb`,
//so it know the primaries from the future.
let new_color = lab_shifted.into_rgb();
//The transfer function will also become known here
pix = Srgb::from_linear(new_color).into_pixel();

Expanding this to a larger application should introduce enough input and return types to aid the type inference and keep it simple enough. Still, it's hard to be sure about anything without trying it for real.

Finally, any reason to choose the trait over a type alias

I'm not sure what you are asking, but I'll sleep on it.

I have an alternate proposal.
Currently this is an issue
// this works
let xyz: Xyz<D65> = Xyz::with_wp(...)
let rgb: LinearRgb<SrgbPrimaries, D65> = Xyz::with_wp(...)

// this does not
let xyz: Xyz<SrgbPrimaries, D65> = Xyz::with_wp(...)
let rgb: LinearRgb<TransferFunction, SrgbPrimaries, D65> = Xyz::with_wp(...)

So remove the trait bounds from all colors. Ex
Struct Xyz<C, T>{
    x: T,
   ...
   color: C // whatever you want to call this
}

Add the 3 basic traits
// the basic traits
trait WhitePoint
trait Primaries
trait TransferFunction

Add the composing traits
trait ColorSpace{
    type TransferFucntion,
    type Primaries,
    type WhitePoint
}
impl WhitePoint for ColorSpace
impl Primaries for ColorSpace
impl TransferFunction for ColorSpace

trait LinearColor{
    type Primaries,
    type WhitePoint
}
impl WhitePoint for LinearColor
impl Primaries for LinearColor

All the conversions and adaptation need to be modified to use the base traits
Example
FromColor for Hsl{
     from_xyz<C: Primaries + WhitePoint>(xyz: Xyz<C>) -> Hsl<C>{
          ....
     }

     // no trait bounds here. Should work for trait
     from_rgb<C>(rgb: Rgb<C,T>) -> Hsl<C>{
         ....
     }
}

So this will work now
let xyz_d50: Xyz<D65> = Xyz::with_color(...)

let xyz_linear: Xyz<LinearSrgb> = Xyz::with_color(...)

let xyz_color: Xyz<SrgbColorSpace> = Xyz::with_color(...)

// should work
xyz_d50.to_lab
// will not work. needs primaries trait and will be a compile error
xyz_d50.to_hsl()

// both should work
xyz_linear.to_lab()
xyz_linear.to_rgb()

//should work for gamma encoding
xyz_color.into_encoded()
xyz_color.to_lab()
xyz_color.to_rgb()

So you only opt in to the traits required and higher order traits can be passed through conversions
Issues:

Not sure if there are any issue in implementation, especially for traits like from and Into color and chromatic adaptation
Color enum - what should be the trait bound. Should it be limited be to LinearColor trait? Or the Color Space trait? For simplicity, limiting to ColorSpace might be a good idea.

Further optimizations:
Can the trait made optional i.e ````Xyz and Xyz<D50,T> or Xyz<Srgb,T>``` should work. I can't think of a way this will work. Maybe using a wrapper type? Might get too complicated

That's a very interesting idea, and it's actually a quite good compromise. The only problems I can see is that Xyz<D65> and Xyz<Srgb> won't be the same type, but converting is trivial, and (above all) that there may be conflicts between implementations for SomeType<C: WhitePoint> and SomeType<C: ColorSpace>. I don't know, though. We may be lucky.


[SPARK-16287][SQL] Implement str_to_map SQL function

What changes were proposed in this pull request?
This PR adds str_to_map SQL function in order to remove Hive fallback.
How was this patch tested?
Pass the Jenkins tests with newly added.

Test build #61525 has started for PR 13990 at commit 1f888ab.

Test build #61525 has finished for PR 13990 at commit 1f888ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61525/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61685 has started for PR 13990 at commit fa294bc.

cc: @cloud-fan @rxin

Test build #61685 has finished for PR 13990 at commit fa294bc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61685/
Test PASSed.

Test build #61690 has started for PR 13990 at commit 6b2390d.

Test build #61690 has finished for PR 13990 at commit 6b2390d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StringToMap(child: Expression, pairDelim: Expression, keyValueDelim: Expression)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61690/
Test PASSed.

Test build #61767 has started for PR 13990 at commit ca74f9a.

Test build #61767 has finished for PR 13990 at commit ca74f9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StringToMap(text: Expression, pairDelim: Expression, keyValueDelim: Expression)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61767/
Test PASSed.

Test build #61806 has started for PR 13990 at commit f7c03c5.

cc @dongjoon-hyun can you help review this

Test build #61815 has started for PR 13990 at commit d1573b6.

Test build #61806 has finished for PR 13990 at commit f7c03c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61806/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61819 has started for PR 13990 at commit 94c18ff.

Sure, @rxin .

Oh, @techaddict . Could you check Hive code again? The default parameter is different from the documentation. We had better follow the code instead of document.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java

I've done my first pass. Feel free to fix the PR.

Test build #61815 has finished for PR 13990 at commit d1573b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61815/
Test PASSed.

Test build #61819 has finished for PR 13990 at commit 94c18ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61819/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61844 has started for PR 13990 at commit 2725445.

Test build #61844 has finished for PR 13990 at commit 2725445.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61844/
Test PASSed.

Merged build finished. Test PASSed.

Thank you for fast updating, @techaddict !

cc @dongjoon-hyun or @petermaxlee want to take over the pull request and bring it to completion?

@rxin no need, I will update this today.

Test build #62090 has started for PR 13990 at commit cc96fea.

Test build #62090 has finished for PR 13990 at commit cc96fea.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62090/
Test FAILed.

Test build #62091 has started for PR 13990 at commit 0b4419f.

Test build #62091 has finished for PR 13990 at commit 0b4419f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62091/
Test PASSed.

Test build #62166 has started for PR 13990 at commit 1592ffc.

Test build #62166 has finished for PR 13990 at commit 1592ffc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test build #62174 has started for PR 13990 at commit 34db7bb.

Test build #62174 has finished for PR 13990 at commit 34db7bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62174/
Test PASSed.

Test build #62208 has started for PR 13990 at commit 34db9cf.

@cloud-fan all comments addressed.

Test build #62209 has started for PR 13990 at commit 4dff9b4.

Test build #62209 has finished for PR 13990 at commit 4dff9b4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class JavaStructuredNetworkWordCountWindowed
case class CallMethodViaReflection(children: Seq[Expression])
      TypeCheckFailure(s\"class $className not found\")
case class OptimizeMetadataOnlyQuery(
class TextSocketSource(host: String, port: Int, includeTimestamp: Boolean, sqlContext: SQLContext)
abstract class Filter
case class EqualTo(attribute: String, value: Any) extends Filter
case class EqualNullSafe(attribute: String, value: Any) extends Filter
case class GreaterThan(attribute: String, value: Any) extends Filter
case class GreaterThanOrEqual(attribute: String, value: Any) extends Filter
case class LessThan(attribute: String, value: Any) extends Filter
case class LessThanOrEqual(attribute: String, value: Any) extends Filter
case class IsNull(attribute: String) extends Filter
case class IsNotNull(attribute: String) extends Filter
case class And(left: Filter, right: Filter) extends Filter
case class Or(left: Filter, right: Filter) extends Filter
case class Not(child: Filter) extends Filter
case class StringStartsWith(attribute: String, value: String) extends Filter
case class StringEndsWith(attribute: String, value: String) extends Filter
case class StringContains(attribute: String, value: String) extends Filter

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62209/
Test FAILed.

Test build #62210 has started for PR 13990 at commit 0da97da.

Test build #62210 has finished for PR 13990 at commit 0da97da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62210/
Test PASSed.

Test build #62208 has finished for PR 13990 at commit 34db9cf.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62208/
Test PASSed.

Test build #62250 has started for PR 13990 at commit cbc8798.

Test build #62256 has started for PR 13990 at commit e701716.

Test build #62250 has finished for PR 13990 at commit cbc8798.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62250/
Test PASSed.

Test build #62257 has started for PR 13990 at commit 8172bd5.

Test build #62256 has finished for PR 13990 at commit e701716.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62256/
Test PASSed.

Test build #62257 has finished for PR 13990 at commit 8172bd5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62257/
Test PASSed.

@cloud-fan anything else, it good to merge ?

Sorry I was OOO last few days, LGTM except some minor comments, thanks for working on it!

@cloud-fan np, 👍

Test build #62471 has started for PR 13990 at commit 1e35779.

Test build #62471 has finished for PR 13990 at commit 1e35779.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62471/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan Comment addressed, test passed 👍

Test build #62681 has started for PR 13990 at commit 8aabd37.

Test build #62681 has finished for PR 13990 at commit 8aabd37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62681/
Test PASSed.

thanks, merging to master and 2.0!

https://github.com/apache/spark/pull/14315 fixed the odd compile error for this.
Is this really something we should be merging in branch 2.0 now? this looks like part of a new feature, and not even obviously something for 2.0.1.

@srowen please see https://issues.apache.org/jira/browse/SPARK-16275, there is an explaination why we wanna merge them into 2.0


Update scalafmt-core to 3.3.0

Updates org.scalameta:scalafmt-core from 3.0.8 to 3.3.0.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalameta", artifactId = "scalafmt-core" } ]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #144.


Source map not working in production

Hi,
I'm using the following sourcemap config: devtool: "source-map",
On production, I see the following files downloaded:

So source maps are being generated.
However, when an error occurs, I do not see the source map link:

It only points to my compiled code which is obfuscated and hard to understand.

https://github.com/mishoo/UglifyJS2/pull/738 may fix the issue

@geekyme Can you try against the current version? I think uglify has received some nice fixes.

No response, closing. Feel free to re-open with more details.


WIP: Remove legacy authentication code.

What's this PR do?
This pull request removes code for legacy login & registration, and a lot of the user hooks for syncing Phoenix changes into Northstar. This simplifies the codebase by getting rid of unused code & a ton of if-statements, and also will allow devs to login via Northstar QA when working locally. Nice.
How should this be reviewed?
To test, pull down this branch & run ds pull stage --db to get a fresh copy of the staging database.
⚠ You will see a ton of errors until you pull the staging database (after checking out this branch), since the Northstar module is now required and we all had it disabled. I'll make another announcement in #phoenix once this gets merged.
Any background context you want to provide?
🔥
Relevant tickets
Fixes #7281
Checklist

[ ] Documentation added for new features/changed endpoints.
[ ] Tested on staging.
[ ] Pinged a PM if this is a larger PR that would benefit from some additional testing love
[ ] Post a message in #phoenix if this includes something that causes a rebuild!

Re-created this PR from #7282 because of a weird StyleCI commit hook error.

I'm going to do some more testing on my local, but this should be ready for review! I'd probably recommend going commit-by-commit since the diff is pretty hefty otherwise.

I've removed the old methods for syncing data into Northstar, since otherwise it'll be easy to shoot ourselves in the foot and corrupt QA data from local instances. This should not affect anything in normal day-to-day use, since they're only referenced from old scripts.

🙇! Will address your comment & do some further cleanup in another PR.


On linux try a different malloc

Tests have shown that a mining node slowly will run out of memory. The test results showed that there was no memory-leak in the code, however.
The repeated calling of getblocktemplate (via the RPC interface) had the effect that malloc was called and the libc malloc implementation caused fragmentation. Which, in short, means that a large amount of memory stopped being available as a one-continues-block.
The suggestion is to make Bitcoin Classic work with a different malloc implementation. For instances JEMalloc.

This does not appear to still be an issue with the rewritten CreateNewBlock code added by morcos in 0.12, as that version of CNB does not do UTXO lookups in CNB any longer. It was the transaction verification and UTXO lookups that were causing memory allocation inside CNB, which in turn caused the memory fragmentation and increasing memory usage over time due to glibc's one-heap-per-thread design. I suggest closing this issue.

ok


Don't show error messages of Abide in the added reveal modal from 5.5.1 to 5.5.2 version

Hi Foundation
it seems errors in reflow function when I added a modal to a panel ,then submitting the panel. The error messages in the modal don't show. i see 5.5.1 doesn't have the function , it is ok
I have to use the old version in the case
Thanks

I think this related to https://github.com/zurb/foundation/issues/6740
If not, we can help more if you open a new issue with steps to reproduce and an code example. Thanks!
We'll close this favor of it. https://github.com/zurb/foundation/issues/6740


🚀 Feature: Utility API for VMware Cloud auth

🔖 Feature description
Let's make it easy for integrators to include VMware Cloud sign-in in their apps without writing any special client code.
🎤 Context
This requirement arose from discussion on #22392.
Suppose I set up my (old backend system) packages/backend/src/plugins/auth.ts like:
import { Router } from 'express';
import { createRouter } from '@backstage/plugin-auth-backend';
import { createOAuthProviderFactory } from '@backstage/plugin-auth-node';
import { vmwareCloudAuthenticator } from '@backstage/plugin-auth-backend-module-vmware-cloud-provider';
import { PluginEnvironment } from '../types';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return await createRouter({
    providerFactories: {
      vmwareCloudServices: createOAuthProviderFactory({
        authenticator: vmwareCloudAuthenticator,
        signInResolver: async (_, ctx) =>
          ctx.issueToken({
            claims: { sub: 'user:guest', ent: ['user:guest'] },
          }),
      }),
    // ...

and I set up my (classic frontend system) App.tsx a bit like:
import { createApp } from '@backstage/app-defaults';
import { vmwareCloudAuthApiRef } from '@backstage/core-plugin-api';

const app = createApp({
  components: {
    SignInPage: props => {
      return (
        <SignInPage
          {...props}
          provider={{
            id: 'vmware-cloud-auth-provider',
            title: 'VMware Cloud',
            message: 'Sign In using VMware Cloud',
            apiRef: vmwareCloudAuthApiRef,
          }}
          title="Select a sign-in method"
          align="center"
        />
      );
    },
  },
  // ...

and my app-config resembles:
app:
  baseUrl: http://<IP_ADDRESS>:3000
backend:
  baseUrl: http://<IP_ADDRESS>:7007
  cors:
    origin: [http://<IP_ADDRESS>:3000]
auth:
  session:
    secret: mysecret
  environment: development
  providers:
    vmwareCloudServices:
      development:
        clientId: ${VMWARE_CLOUD_APP_ID}
        organizationId: ${VMWARE_CLOUD_ORG_ID}

Then I'd like to be able to run the app with yarn dev, visit http://<IP_ADDRESS>:3000 in my browser, click 'Sign In', sign in with my VMware account, visit the User Settings page and see my full name and email.
✌ Possible Implementation
As suggested by @Rugvip, it should be enough to:

create API refs in @backstage/core-plugin-api and @backstage/frontend-plugin-api
create an implementation in @backstage/core-app-api
create an API factory in @backstage/app-defaults

👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
Yes I am willing to submit a PR!

of course this would be a great addition @jamieklassen! 🤩


Failed to polyfill fs/promises

Calling polyfillPath with fs/promises fails due to this line:
  const jspmPath = resolve(
    require.resolve(`@jspm/core/nodelibs/${module}`),
    "../../.."
  );

I'ld suggest maybe this workaround:
  const jspmPath = resolve(
    require.resolve(`@jspm/core/nodelibs/${module}`),
    "../../.." + (module.includes('/') ? "/.." : "")
  );

Happy to make a PR if we agree on a resolution.

If it passes the tests, feel free to!

@Aslemammad raised a PR, please approve it to run so we can see if test pass.


Classes inheriting from IState Should Be Sealed

Closing due to realizing that I may need them to not be sealed later, when playing around with none-core state machines.


add brew formula for pangolin

when auditting was getting the error
Failed to load cask: pangolin.rb
Cask 'pangolin' is unreadable: wrong constant name #<Class:0x000000012972eb68>

otherwise made changes based off the output from the audit

when auditting was getting the error
Failed to load cask: pangolin.rb
Cask 'pangolin' is unreadable: wrong constant name #<Class:0x000000012972eb68>

otherwise made changes based off the output from the audit

You're not maintaining a cask, (prebuilt package) so I think that's okay.


Incorrect output on example with sub-materials and morph targets

From @mrdoob in #52 —

Seems like the materials no longer export with the latest version of the exporter 😕
alligator.blend.zip

alligator_old.gltf.zip

alligator_new.gltf.zip

Materials are working in the new exporter and Blender 2.8 (after changing everything to Principled BSDF) but animation is failing.


Tests fail to diffrentiate nodes because of port casuing invalid hmac

When we run a bootstrap test and start multiple nodes within the same test binary, they connect to the bootstrap node, it returns them a list of addresses (all of them are localhosts on test), then they are all updated with the same ip:port and the next time we try to connect a node it has the bootstrap port, hence the handshake fails as we try to handshake with the wrong public key.
Possible solutions

Exchange the listening port and populate the dht with the right listen port for each node, (incoming connections don't disclose the listening port)
Decide on a constant service port and try to connect all nodes on this port. - build a network simulation to make this work on tests.

fixed by passing port in handshake.


Publish the emacs-lsp-booster on crates.io

This way it can be installed as cargo install emacs-lsp-booster.

Thanks, that's a good idea.  I'm not familiar with that though, I will look into that.


G915 not detected

Information

ratbagd --version : 0.16
Distribution: Arch
Kernel version (ex. uname -srmo): Linux arch 5.15.5-arch1-1

Device info (if applicable):
Logitech G915 (black, tactile)
Used in Wireless (wired doesn't work neither)

Describe the bug
Keyboard is not detected
To Reproduce

With systemd ratbagd service started and G915 connected wirelessly, run  sudo ratbagctl list
Get message : No devices available

Logs
Verbose, with mice unplugged for easier readings:
[lr@arch ~]$ sudo ratbagd --verbose=debug
Initializing libratbag
ratbag debug: New device: ITE Tech. Inc. ITE Device(8595)
ratbag debug: Using data directory '/usr/share/libratbag'
ratbag debug: No data file found for 048d:8297
ratbag debug: New device: Corsair Commander PRO
ratbag debug: Using data directory '/usr/share/libratbag'
ratbag debug: No data file found for 1b1c:0c10
ratbag debug: New device: Logitech USB Receiver
ratbag debug: Using data directory '/usr/share/libratbag'
ratbag debug: 046d:c541 is a Logitech receiver, not a device. Ignoring...
ratbag debug: New device: Logitech USB Receiver
ratbag debug: Using data directory '/usr/share/libratbag'
ratbag debug: 046d:c541 is a Logitech receiver, not a device. Ignoring...
ratbag debug: New device: Logitech USB Receiver
ratbag debug: Using data directory '/usr/share/libratbag'
ratbag debug: 046d:c541 is a Logitech receiver, not a device. Ignoring...

Additional context
I have been able to create a custom file in /usr/share/libratbag/logitech-g915-custom.device using ID 046d:c541, but I could only get the latest used color (full red) working, which is a win already, as the keyboard previously had LED off, but could probably be fixed by somebody with the required knowledge.

I also experienced this issue, and creating a custom config file as @UltimateByte mentioned also worked for me. My usecase are primarily keybinds, which work without an issue so far.


accept crypto currency (bitcoin) donations

as title states

Hey, this is the issue tracker for the git-scm.com website, not Git itself. If this issue is still relevant and you still want to discuss it, you should reach out to the community.

I think this isse is actually up to the Software Freedom Conservancy, as they handle all of the donations to the project. The complete list of options can be found at https://sfconservancy.org/donate/ (and you can donate there and earmark for the Git project). It's possible that the git-scm.com donate page needs some updating to make the options more obvious.


fix(deploy-package): remove comments in build file

:tada: This PR is included in version 1.2.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Updated FDO with new ServiceInfo Module framework

Added new ServieInfo classes Module and ModuleManager.
Updated Owner storage to reflect new SVI database
Updated samples to use new ServiceInfo framework
Removed old SDO service info.
Signed-off-by: Templeton, Randall<EMAIL_ADDRESS>

In multiple files, I am observing difference in sample voucher, devicecreds and rvinfo. I am assuming that is because the code has not been rebased on latest master.

We need atleast one unit test in place to perform a E2E sanity test for fdo_sys implementation. Currently there is none.


Restore the ability to stop when a specified string has been generated

This was removed in #366 to simplify the PR. It should be simple to implement it at the SequenceGenerator level.

I see how we could check whether all sequences contain the stop characters after last_state = next(states) in SequenceGenerator.__call__ but I can't find a way of ending the generation for a particular sequence if this one includes the stop characters (but not the others) without modifying the sequence_generator function. Do you have a solution in mind?

Can't we do this at the SequenceGenerator level?


release(prod): handle safe shutdown errors

Will wait to merge at least until testnet QA is complete. Want to get this in later today for tests that will take place at manufacturing.
Issue

Link: https://github.com/NebraLtd/product-management/issues/46
Summary: Correctly handle Balena supervisor shutdown errors.

How
Screenshots

References

hm-diag PR: https://github.com/NebraLtd/hm-diag/pull/373

Checklist

[ ] Tests added
[ ] Cleaned up commit history (rebase!)
[ ] Documentation added
[ ] Thought about variable and method names

@marvinmarnold Will get this tested asap


[ENHANCEMENTS] Various Fit Finish enhancements

Required Basic Info

Accelerator Version:  v1.3.1

Describe enhancements

Enable 'deletion protection' on all ASEA deployed ALB and NLB's

also ensure we can still remove them if customer removes from config file (i.e. does not cause SM failure)

Some log groups missing retention settings and subscriptions

example Ops account (4 of 32) missing settings:
/aws/lambda/ASEA-Operations-Phase5-CustomCentralLoggingSubscri-W8YILGMA7ROF
/aws/lambda/ASEA-Operations-Phase5-CustomS3PutBucketReplicatio-1CJZTP9WIDR2C
/aws/lambda/ASEA-Operations-Phase5-CustomS3PutBucketVersioning-16N94KMJSEBQO
/aws/lambda/ASEA-Operations-Phase5-CustomSSMIncreaseThroughput-I0QB3BI8PBG0

rsyslog NLB missing central logging
add an HTTPS policy to all ASEA deployed buckets including cdk buckets

test to make sure it doesn't break anything

    {
      "Action": "s3:*",
      "Effect": "Deny",
      "Resource": [
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET",
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
      ],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } },
      "Principal": "*"
    }

Apply back-off retry fixes:

backport Gilles Gitlab backoff-retry fix: in \src\lib\common\src\aws\backoff.ts

("fix(core): Add additional retry exception type for AWS Organizations")

also see email from Brian with another potential fix in Phase 4 Error: Associating Hosted Zones

specifically, add backoff-retry on "ConcurrentModifications" === e.code`
while use the email as general guidance, fix properly in the base codebase :-).

Enable 'deletion protection' on all ASEA deployed ALB and NLB's

Created a custom resource depends on ALB and enables Deletion protection also disable deletion
protection on Delete ALB

rsyslog NLB missing central logging

Enabling Access Logging after creation irrespective of Configuration


php7 debian random number row

Hi all,
Using sqlsrv 4.0.7 on debian 7 I get very strange result with very simple request while fetching a whole table (9301 rows)
<?php
$conn = sqlsrv_connect('SERV', ['Database' => 'Test','Uid' => 'sa','PWD' => 'pass']);

$sql = "SELECT * FROM TABLE";
$rows = sqlsrv_query($conn , $sql);
$total = 0;
while ($row = sqlsrv_fetch_array($rows))
    $total++;
echo $total;

result :

9301 fetched on windows
1011 on debian
447 id adding "order by name" on debian

count(*) return the correct value of 9301
Have no idea where the problem come from any help is welcome
Thanks in advance

@vinise What version of Debian and the driver are you using?

@vinise Any updates from your end?

Hello @meet-bhagdev
Linux version 3.2.0-4-amd64<EMAIL_ADDRESS>(gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.73-2+deb7u2
Microsoft ODBC Driver 13 for SQL Server

Hi @vinise ,
We are now officially supporting Debian Jessie. I tried your repro script in our test lab and runs fine. Looks like you are using older version of Debian, but you can still try the binaries here: https://github.com/Microsoft/msphpsql/releases/tag/v4.2.0-preview
To install the latest driver using PECL:
sudo pecl install sqlsrv-4.2.0preview
Let me know if it works for you.

Hi @ulvii ,
Thanks for this update 😃
I've install it and mande multiple test.
No problem with sql server 2016
But Sql server 2008 (with database compatibility leverl 2000) still the same problem :/
I know this is bit old but stuck with all application... migration in progress but take time

I just tried the script on Debian Jessie against SQL Server 2008 R2(SP 2), Compatibility level 80, but could not reproduce the issue.

@vinise What is your error message?

Hi @vinise ,
As I mentioned above, both the underlying driver MSODBCSQL and SQLSRV driver are not supported for Debian 7. I would try the script against different SQL Server instances, preferably on a clean VM. I will keep the issue open for a few more days, in case you have questions during the investigation.

hello 😃
@meet-bhagdev no i've no errors
@ulvii i'm gonna (as soon as i have a bit of time) reinstall a new VM with a debian 8 and retry.
If the problem persist i'll try with a new installation of mysql server
If i don't succeed i'll gave you access to the entire configuration 😃
Thanks a lot for your help

@vinise Do you still need help with this?

Hi,
I have the same issue on Ubuntu 16.04
I did the same test as @vinise, and i got 8 rows when i used a while loop from the result.
SELECT COUNT(*) gives me 1094
If i use sqlcmd from cli on my Linux server, everything works fine.
The following query returning correct data in sqlcmd: "SELECT * FROM G_CONTAC WHERE G_CONTAC.CONTID = 35089;"
If i do the same query in php7, i got nothing. No errors, nothing.
If i change the query to (CONTID = 35088), "SELECT * FROM G_CONTAC WHERE G_CONTAC.CONTID = 35088;", i get correct data from both sqlcmd and php7.
Windows Server:
Microsoft SQL Server Express (64-bit) (11.0.2100.60 )
Windows Server 2008 R2 Standard (Version 6.1.7601) 64bit
Linux Server:
lsb_release -a:
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

@sofagris Does this happen with the sqlsrv or pdo_sqlsrv driver? Which version of the php sqlsrv drivers are you using? Run php --ri sqlsrv and php --ri pdo_sqlsrv to find out. If it is not the latest version (5.1.2), please try the latest drivers from from here or by running sudo pecl install sqlsrv-5.1.2preview or sudo pecl install pdo_sqlsrv-5.1.2preview. Let us know if the problem persists. Thanks!

@david-puglielli
sqlsrv
sqlsrv support => enabled
ExtensionVer => 4.3.0
Directive => Local Value => Master Value
sqlsrv.WarningsReturnAsErrors => On => On
sqlsrv.LogSeverity => 0 => 0
sqlsrv.LogSubsystems => 0 => 0
sqlsrv.ClientBufferMaxKBSize => 10240 => 10240
pdo_sqlsrv
pdo_sqlsrv support => enabled
ExtensionVer => 4.3.0
Directive => Local Value => Master Value
pdo_sqlsrv.log_severity => 0 => 0
pdo_sqlsrv.client_buffer_max_kb_size => 10240 => 10240
I will update asap, and try.

@david-puglielli I have now updated both sqlsrv and pdo_sqlsrv, and  same result.
sqlsrv
sqlsrv support => enabled
ExtensionVer => 5.1.2-preview
pdo_sqlsrv
pdo_sqlsrv support => enabled
ExtensionVer => 5.1.2-preview
You can see the result here

BTW: I was using sqlsrv. I tested PDO today, and PDO is working fine.

Thank you @sofagris. We will investigate and keep you updated.

@sofagris I am unable to reproduce the behaviour you see. Is there any other information you can provide? Such as the following: Does this happen regularly, or only in the specific case you indicated above? Do you have a full repro script we can use? Does it seem to depend on the table characteristics of the table or what type of query? Do you get the correct result using sqlsrv_num_rows (you need a client side, static, or keyset cursor for sqlsrv_num_rows)?

@david-puglielli I have tested only with this database, as i only have this windows server. It seems to be the same result every time, and it's the same problem in all tables.
Some queries return 2-3 rows, and other return nothing. The queries returns the same result every time.
"SELECT * FROM G_CONTAC;" will allways return 8 rows.
This is the code i use:
`
$mssql_host = "<IP_ADDRESS>";
$connectionOptions = array(
"Database" => "Client00010001",
"Uid" => "user",
"PWD" => "password"
);
$conn = sqlsrv_connect($mssql_host,$connectionOptions);
$query = "SELECT * FROM G_CONTAC;";
$result = sqlsrv_query($conn, $query);
echo 'This is the result of "SELECT * FROM G_CONTAC ORDER BY NAME"';
echo "";
$total = 0;
while ($row = sqlsrv_fetch_array($result)){
$total++;
echo "$total: " . $row['NAME'] . "";
}
echo "";
echo "Total number of records : $total " ;
$query2 = "SELECT COUNT () AS NUM FROM G_CONTAC";
$result2 = sqlsrv_query($conn, $query2);
while ($row2 = sqlsrv_fetch_array($result2)){
echo "SELECT COUNT () AS NUM FROM G_CONTAC = "  . $row2['NUM'];
}
`
I will check sqlsrv_num_rows tomorrow.

Thank you @sofagris. I'm afraid this doesn't tell us much. Could you run the ODBC trace for this script and attach it here? Add "TraceOn"=>true and "TraceFile"=>"trace.log" to your $connectionOptions to generate a trace file.

@david-puglielli see  attached trace file.
trace.log

We have found a possible cause of the issue. The trace indicates that reading the output set proceeds as normal until the ninth row, at which point an error message is seen when trying to read from the fourth column.
[ODBC][1453][1512297635.656935][SQLGetData.c][237]
		Entry:
			Statement = 0x7f2b7de78480
			Column Number = 4
			Target Type = 1 SQL_CHAR
			Buffer Length = 51
			Target Value = 0x7f2b76c02380
			StrLen Or Ind = 0x7ffea21c1678
[ODBC][1453][1512297635.656982][SQLGetData.c][501]
		Exit:[SQL_SUCCESS_WITH_INFO]
			Buffer = [HUSEBY, RAGNA OG HÃ…KON                           ]
			Strlen Or Ind = 0x7ffea21c1678 -> 51
		DIAG [01004] [Microsoft][ODBC Driver 13 for SQL Server]String data, right truncation

It looks like this is a 50-character field, and the field data is overflowing the buffer and an error is thrown, which stops query execution. Could you examine the data at this location and verify if it is too long (including any whitespace at the end)?

Closing this issue because of inactivity. Feel free to reopen if necessary.


add a new network support, Optimism

https://community.optimism.io/docs/

@zhangzhongnan928 We can add it as a chains like other chains. But is there value, in the short term, if we figure out L1-L2 bridging and how it might be immediately useful for users?

@zhangzhongnan928 We can add it as a chains like other chains. But is there value, in the short term, if we figure out L1-L2 bridging and how it might be immediately useful for users?

Yes,

they are launching in June, it is the most important L2, many people will want to try, plus Uniswap SNX and many other DApp already on it.
AutographNFT will be deployed on it.


Ollama loads gemma2 27b with --ctx-size 16384

What is the issue?
Isn't gemma2 context size 8192?
When I access GEMMA2:27b-instruct-q8_0 with OpenAI api, it loads the model with --ctx-size 16384 according to log.
OS
macOS
GPU
Apple
CPU
Apple
Ollama version
0.1.48

It looks like parallel multiplies the total context size.


User wants to be notified once update to kernel 4.19.118 is available

Relevant PR: https://github.com/balena-os/balena-raspberrypi/pull/496

On rpi3 / balena fin

Assuming this can be closed now @AlidaOdendaal @floion ?

Yes, we are now on kernel 5.4


[OSPRH-6951] Auto-detect OSImage checksum name and type

The OpenStack Baremetal operator currently relies on hardcoded checksum filename and type to be present in the EDPM container image.  If possible, we should have the operator's agent look at the EDPM container image and detect these checksum details dynamically.
OSPRH-6951

/hold until after beta freeze

Example resultant OpenStackProvisionServer CR: https://review.rdoproject.org/zuul/build/4e0ed0de905344c4b9ce3ecd421a50d2/log/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/crs/openstackprovisionservers.baremetal.openstack.org/openstack-edpm-ipam-provisionserver.yaml

/unhold

looks good to me, but would be good if someone who is more into the baremetal deployment also have look, maybe @rabi

If possible, we should have the operator's agent look at the EDPM container image and detect these checksum details dynamically.

I've not yet understood the kind of use-cases this would address, as we would be building the OsContainerImage based on OpenStackVersion and would provide to the customers? checksum file name is based on the OSImage name which can be changed in the spec and the checksum type which we intentionally limited to sha256. I just want to ensure that we're not trying to over-engineer the whole thing for downstream packaging.

If possible, we should have the operator's agent look at the EDPM container image and detect these checksum details dynamically.

I've not yet understood the kind of use-cases this would address, as we would be building the OsContainerImage based on OpenStackVersion and would provide to the customers? checksum file name is based on the OSImage name which can be changed in the spec and the checksum type which we intentionally limited to sha256. I just want to ensure that we're not trying to over-engineer the whole thing for downstream packaging.

What we encountered were differences in the filename extension of the sha256 checksum image between the upstream version and the downstream version.  One was .sha256 while the other became .sha256sum and caused problems in CI , given that we had hardcoded this operator to look for .sha256 (so in .sha256sum naming-case, the file would not be found and dataplane deployment was blocked).  We also have to tell Metal3 to expect this type, and we had hardcoded it to always assume sha256.  With this new approach, as long as sha256, sha512 or md5 is present as a substring in the filename extension, we'll be able to detect the proper filename and pass the proper type to Metal3.

One was .sha256 while the other became .sha256sum and caused problems in CI

Yeah, I'm aware of that (I think that was a mistake on part of downstream packager to assume that they can have any checksum file extension). As we decide on how we build OsContainerImage , we can decide what the checksum file and the OSImage name/extension should be. Unless there is an use-case where we don't know what they could be, we can auto-detect. My intention is to not add complexity for things that would only address the downstream packaging mistake/assumption.

recheck
TASK [download_tools : Download kuttl url=[https://github.com/kudobuilder/kuttl/releases/download/v{{](https://github.com/kudobuilder/kuttl/releases/download/v%7B%7B) kuttl_version }}/kubectl-kuttl_{{ kuttl_version }}_linux_x86_64, dest={{ lookup('env', 'HOME') }}/bin/kubectl-kuttl, mode=0755, timeout=30] ***
    Tuesday 06 August 2024  06:22:52 -0400 (0:00:00.461)       0:00:09.961 ********
    [0;31mAn exception occurred during task execution. To see the full traceback, use -vvv. The error was: ConnectionResetError: [Errno 104] Connection reset by peer[

/cherry-pick 18.0.0-proposed


Can't use controllers and mouse at the same time.

Hi,
I am using the Controllable mod for Minecraft 1.18 and I'm using two controllers and a mouse to play split-screen with three players on one computer.
I enable the two controllers using the Controllable mod, but when I want to use the mouse the one controller gets interrupted from the mouse movement, and I can't get the controller to work without interference from the mouse.
Please help, Thanks.

Make sure "Pause on Lost Focus" is turned off. Press F3 + P while in game to do this.


Feed URL
https://betterdev.blog/rss.xml
More Info
No response

closed by https://github.com/murtuzaalisurti/rssed/commit/267ef2c4dae3b3c17e69ba10822edba6852e9518


diskless image creation fails when gpgcheck is enabled in repo configuration

Description
The diskless image creation fails when gpgcheck=1 in the repository configuration.
Info
OS: CentOS 8.2
Branch: master + stable-1.3
Reproducer
Repositories configuration:
# ansible-inventory --host management1 |grep -w repositories -A 6
    "repositories": [
        "bluebanquise",
        {
            "gpgcheck": true,
            "name": "os"
        }
    ],
# grep gpgcheck /etc/yum.repos.d/CentOS-{AppStream,Base}.repo
/etc/yum.repos.d/CentOS-AppStream.repo:gpgcheck = 1
/etc/yum.repos.d/CentOS-Base.repo:gpgcheck = 1

Run disklessset to create a livenet image:
# disklessset
BlueBanquise Diskless manager
 1 - List available kernels
 2 - Generate a new initramfs
 3 - Generate a new diskless image
 4 - Manage existing diskless images
-->: 3

Starting new image creation phase.
Many questions will be asked, a recap will be provided before starting procedure.

Select image type:
 1 - nfs
 2 - livenet
-->: 2
[INFO] Loading kernels from /var/www/html/preboot_execution_environment/diskless/kernels/

Select kernel:
 1 - vmlinuz-4.18.0-147.3.1.el8_1.x86_64
-->: 1
Please enter image name ?
-->: test1
Please enter clear root password of the new image: root
[INFO] Entering livenet dedicated part.
Please select livenet image generation profile:
 1 - Standard: core
 2 - Small: openssh, dnf and NetworkManager
 3 - Minimal: openssh only
 4 - Custom: core + selection of additional packages
-->: 1
Please choose image size:
(supported units: M=1024*1024, G=1024*1024*1024)
(Example: 5120M or 5G)
-->: 2G
Enter path to SSH public key (left empty to disable key injection):
-->: /root/.ssh/id_rsa.pub
Do you want to activate SELinux inside the image?
Enter yes or no: y
Do you want to create a new livenet image with the following parameters:
  Image name:           test1
  Kernel version:       vmlinuz-4.18.0-147.3.1.el8_1.x86_64
  Root password:        root
  Image profile:        1
  Image size:           2048M
  SSH pubkey:           /root/.ssh/id_rsa.pub
  Enable SELinux:       y
Confirm ? Enter yes or no: y
[INFO] Cleaning and creating image folders.
[INFO] Generating new ipxe boot file.
[INFO] Creating empty image file, format and mount it.
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 17.2339 s, 125 MB/s
meta-data=/var/tmp/diskless/workdir/test1/LiveOS/rootfs.img isize=512    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=524288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[INFO] Installing system into image.
Unable to detect release version (use '--releasever' to specify release version)

You have enabled checking of packages via GPG keys. This is a good thing.
However, you do not have any GPG public keys installed. You need to download
the keys for packages you wish to install and install them.
You can do that by running the command:
    rpm --import public.gpg.key

Alternatively you can specify the url to the key you would like to use
for a repository in the 'gpgkey' option in a repository section and DNF
will install it for you.

For more information contact your distribution or package provider.

Problem repository: [AppStream]
[...]
[INFO] Setting password into image.
Traceback (most recent call last):
  File "/usr/bin/disklessset", line 435, in <module>
    with open(os.path.join(installroot, 'etc/shadow'), "r+") as ff:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/diskless/workdir/test1/mnt/etc/shadow'

This is a configuration issue. Following the advice in the error message, we must specify the gpgkey in the inventory:
# cat inventory/group_vars/all/general_settings/repositories.yml
repositories:
  - bluebanquise
  - { name: 'os', gpgcheck: True, gpgkey: "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial" }

# ansible-inventory --host management1 |grep -w repositories -A 7
    "repositories": [
        "bluebanquise",
        {
            "gpgcheck": true,
            "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial",
            "name": "os"
        }
    ],

# ansible-playbook playbooks/managements.yml --tags repositories_client --limit management1 --diff
[...]
TASK [repositories_client : yum_repository █ Setting OS repositories] *************************************************
Tuesday 22 September 2020  10:29:40 -0400 (0:00:00.502)       0:00:04.044 *****
--- before: /etc/yum.repos.d/CentOS-Base.repo
+++ after: /etc/yum.repos.d/CentOS-Base.repo
@@ -2,5 +2,6 @@
 baseurl = http://<IP_ADDRESS>/repositories//centos/8/$basearch/os/BaseOS
 enabled = 1
 gpgcheck = 1
+gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
 name = BaseOS gen by Ansible

changed: [management1] => (item={'name': 'os', 'gpgcheck': True, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', 'file': 'CentOS-Base', 'repoid': 'BaseOS'})
--- before: /etc/yum.repos.d/CentOS-AppStream.repo
+++ after: /etc/yum.repos.d/CentOS-AppStream.repo
@@ -2,5 +2,6 @@
 baseurl = http://<IP_ADDRESS>/repositories//centos/8/$basearch/os/AppStream
 enabled = 1
 gpgcheck = 1
+gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
 name = AppStream gen by Ansible

changed: [management1] => (item={'name': 'os', 'gpgcheck': True, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', 'file': 'CentOS-AppStream', 'repoid': 'AppStream'})

With this setting, the image creation succeeds.
Might need to document it in the readme for reference.

Thanks a lot for this. I agree, this must be documented in the new tool readme.

Doc has been updated.


deis register bad request error

Hello guys,
I've installed deis on AWS using official docs and CF template. I have my own 7insyde.xyz domain on Route53 where I created wildcarf prefix *.7insyde.xyz and pointed as CNAME to deis loadbalancer.
The deis seems to be working fine:
[sporokh@BRONZINO]:~$ deisctl machines
MACHINE		IP		METADATA
1b15eb2d...	<IP_ADDRESS>	controlPlane=true,dataPlane=true,routerMesh=true
4bb141df...	<IP_ADDRESS>	controlPlane=true,dataPlane=true,routerMesh=true
e991a5a5...	<IP_ADDRESS>	controlPlane=true,dataPlane=true,routerMesh=true

and
[sporokh@BRONZINO]:~$ deisctl list
UNIT				MACHINE			LOAD	ACTIVE	SUB
deis-builder.service		1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-controller.service		4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-database.service		e991a5a5.../<IP_ADDRESS>	loaded	active	running
deis-logger.service		4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-logspout.service		1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-logspout.service		4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-logspout.service		e991a5a5.../<IP_ADDRESS>	loaded	active	running
deis-publisher.service		1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-publisher.service		4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-publisher.service		e991a5a5.../<IP_ADDRESS>	loaded	active	running
<EMAIL_ADDRESS>e991a5a5.../<IP_ADDRESS>	loaded	active	running
<EMAIL_ADDRESS>4bb141df.../<IP_ADDRESS>	loaded	active	running
<EMAIL_ADDRESS>1b15eb2d.../<IP_ADDRESS>	loaded	active	running
<EMAIL_ADDRESS>e991a5a5.../<IP_ADDRESS>	loaded	active	running
deis-store-daemon.service	1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-store-daemon.service	4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-store-daemon.service	e991a5a5.../<IP_ADDRESS>	loaded	active	running
<EMAIL_ADDRESS>1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-store-metadata.service	1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-store-metadata.service	4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-store-metadata.service	e991a5a5.../<IP_ADDRESS>	loaded	active	running
deis-store-monitor.service	1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-store-monitor.service	4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-store-monitor.service	e991a5a5.../<IP_ADDRESS>	loaded	active	running
deis-store-volume.service	1b15eb2d.../<IP_ADDRESS>	loaded	active	running
deis-store-volume.service	4bb141df.../<IP_ADDRESS>	loaded	active	running
deis-store-volume.service	e991a5a5.../<IP_ADDRESS>	loaded	active	running

I've set config platform set domain correctly pointing to my 7insyde.xyz
[sporokh@BRONZINO]:~$ deisctl config platform get domain
7insyde.xyz

But when I try to register i fall into this error:
[sporokh@BRONZINO]:~$ deis register deis.7insyde.xyz
Error: http://deis.7insyde.xyz/v1/ does not appear to be a valid Deis controller.
Make sure that the Controller URI is correct, the server is running and
your client version is correct.

If I try to open it on my browser I got the HTTP 400 Bad Request error and
[sporokh@BRONZINO]:~$ deisctl status router@1
●<EMAIL_ADDRESS>- deis-router
   Loaded: loaded<EMAIL_ADDRESS>linked-runtime; vendor preset: disabled)
   Active: active (running) since Mon 2016-06-06 14:51:25 UTC; 20min ago
  Process: 27015 ExecStartPre=/bin/sh -c docker inspect deis-router >/dev/null 2>&1 && docker rm -f deis-router || true (code=exited, status=0/SUCCESS)
  Process: 26950 ExecStartPre=/bin/sh -c IMAGE=`/run/deis/bin/get_image /deis/router` && docker history $IMAGE >/dev/null 2>&1 || flock -w 1200 /var/run/lock/alpine-pull docker pull $IMAGE (code=exited, status=0/SUCCESS)
 Main PID: 27065 (sh)
   CGroup<EMAIL_ADDRESS>           ├─27065 /bin/sh -c IMAGE=`/run/deis/bin/get_image /deis/router` && docker run --name deis-router --rm -p 80:80 -p 2222:2222 -p 443:443 -p 9090:9090 -e EXTERNAL_PORT=80 -e HOST=$COREOS_PRIVATE_IPV4 $IMAGE
           └─27091 docker run --name deis-router --rm -p 80:80 -p 2222:2222 -p 443:443 -p 9090:9090 -e EXTERNAL_PORT=80 -e HOST=<IP_ADDRESS> deis/router:v1.13.1

Jun 06 14:51:37 ip-10-21-2-219.ec2.internal sh[27065]: [INFO] - 2016/06/06 14:51:37 [notice] 45#0: signal process started
Jun 06 14:51:37 ip-10-21-2-219.ec2.internal sh[27065]: [INFO] - deis-router running...
 Spent like 3 hours and still no success with registration.
P.S Tried to stop/uninstall and install again with no success.

Did you try deisctl config router set proxyProtocol=1? On AWS the ELB uses tcp listeners which require PROXY protocol to be enabled, which we enable here but usually it fails out until deisctl has been installed.

@bacongobbler
Huge thanks to you. IT works! Saved me:)
Probably it should be present in the docs, cause its really unclear.

It's fairly clear in the script when you provision the cluster, but normally this would be enabled for you. PRs welcome for amending v1 documentation if you're up for it. :)

@bacongobbler I found out why I didn't receive this warning during provisioning. Nevermind, thanks for your help anyways. Much appreciate.


some applications ignore focused container

Some applications "ignore" or somehow manage to have the focused window change / clear before the tiling is attempted, thus causing the window to follow the default tiling rules for where it should be positioned, rather than tiling at the position of the previously focused container.
I suspect this is due to the changes made to properly support cmd+tab, function didActivateApplication() in kwm/workspace.mm

I have confirmed that his is indeed happening because of the code specified in didActivateApplication() in kwm/workspace.mm that allows for cmd+tab support.

playing with it for a moment it does appear to be fixed

It also fixed a focus issue when opening a link in a browser across displays! Awesome!