Generate native Unity sprites

Currently, a custom scriptable object is used to represent diced sprite asset. This leads to multiple complications and issues, like the need to use a custom renderer and the lack of some core native sprite features, like animation and masking support.
The main issue stopping us from switching to the native sprites is the lack of a way to set custom texture UVs to the Sprite objects. This is, alongside with the ability to set custom mesh geometry, is essential to reconstruct the original sprite from the diced atlas texture.
In case someone have an idea how to deal with the issue, or have any info on possible upcoming changes to the Unity’s sprite API, which will allow to set the UVs, please share it in this thread.

@Elringus i was wondering if this would help achieve what you are looking for.
https://www.patreon.com/posts/tips-how-to-chan-25620556

Yeah, it could be it, thanks for sharing!
Tried to run a quick test for this, but looks like I'm missing something: https://github.com/Elringus/SpriteDicing/commit/d03389acbd4a501c568b832f0ba968a69437cd1f
Unfortunately, I don't currently have time to properly work on this; in case anyone will figure how to make it work, please send a PR (or just a code snippet) and I'll make sure to properly re-write the stuff.

@Elringus i try your test and it worked like a charm for me. the only thing is that the sprite preview look like this.

`[SerializeField]private SpriteRenderer spriteRenderer;
public DicedSprite dicedSprite;
// Start is called before the first frame update
void Start()
{
    RenderSpriteMesh();
}

private void RenderSpriteMesh()
{
    if(spriteRenderer == null)
        spriteRenderer = GetComponent(typeof(SpriteRenderer)) as SpriteRenderer;

    if (dicedSprite == null) return;
    var spriteGenerated =
        Sprite.Create(dicedSprite.AtlasTexture, dicedSprite.EvaluateSpriteRect(), dicedSprite.Pivot, 100);
    spriteGenerated.name = name;
    spriteGenerated.SetVertexCount(dicedSprite.Vertices.Count);
    spriteGenerated.SetIndices(new NativeArray<ushort>(dicedSprite.TrianglesList.Select(t => (ushort) t).ToArray(),
        Allocator.Temp));
    spriteGenerated.SetVertexAttribute(VertexAttribute.Position,
        new NativeArray<Vector3>(dicedSprite.VerticesList.Select(v => new Vector3(v.x, v.y, 0)).ToArray(),
            Allocator.Temp));
    spriteGenerated.SetVertexAttribute(VertexAttribute.TexCoord0,
        new NativeArray<Vector2>(dicedSprite.UVsList.ToArray(), Allocator.Temp));
    if (spriteRenderer != null) spriteRenderer.sprite = spriteGenerated;
}

private void OnValidate()
{
    RenderSpriteMesh();
}`

Thanks for checking and all the tips! Will look into this more today.

I will also look into today as soon as I can I think I know now how to achieve the correct result. I'm changing the calculation for the size of the texture Atlas and also the calculation for the sprite.

i made a PR -> Unity sprite support #2 that will add correct native unity sprite support, i fix the calculation for the rect sized and the calculation for the size of the spriteAtlas.

Looks like everything is working, but there is one issue. When the sprite rect of a generated sprite is bigger than the atlas texture, Sprite constructor raises ArgumentException: Could not create sprite () from a () texture.: https://github.com/Elringus/SpriteDicing/blob/8b3073a886d4ad858214be24207a0671db9feecd/Assets/SpriteDicing/Editor/DicedSpriteAtlasEditor.cs#L527
It's possible to provide a fake sprite rect, which is smaller or equal to the atlas texture, but in this case some features would be broken (eg, culling, pivot, editor position and scale tools, etc).
I'll keep the unity-sprite branch until this is resolved.
Meanwhile, I've added asmdef and package.json files to the master branch, so it now can be used with Unity's package manager.

Looks like everything is working, but there is one issue. When the sprite rect of a generated sprite is bigger than the atlas texture, Sprite constructor raises ArgumentException: Could not create sprite () from a () texture.:
https://github.com/Elringus/SpriteDicing/blob/8b3073a886d4ad858214be24207a0671db9feecd/Assets/SpriteDicing/Editor/DicedSpriteAtlasEditor.cs#L527
It's possible to provide a fake sprite rect, which is smaller or equal to the atlas texture, but in this case some features would be broken (eg, culling, pivot, editor position and scale tools, etc).
I'll keep the unity-sprite branch until this is resolved.
Meanwhile, I've added asmdef and package.json files to the master branch, so it now can be used with Unity's package manager.

that's why i change a little the size limit adding a minimum size for width and height on DicedSpriteAtlasEditor.cs.
I add this private variables to use.

then on the method DiceSourceTextures i add the code to get the biggest width and height based on the texture2Ds used.

On CreateAtlasTextures method i change a lot so here is the code and the pictures of the changes. this change apply minimum width or height size to the xLimit and yLimit but i still don't know how to increase the textureSize atlas if it's smaller than the minimum width and height.

private bool CreateAtlasTextures (Dictionary<string, List<DicedUnit>> dicedUnits)
        {
            DisplayProgressBar("Processing diced textures...", .5f);

            // Delete any previously generated atlas textures.
            for (int i = atlasTexturesProperty.arraySize - 1; i >= 0; i--)
            {
                var unusedTexture = atlasTexturesProperty.GetArrayElementAtIndex(i).objectReferenceValue;
                AssetDatabase.DeleteAsset(AssetDatabase.GetAssetPath(unusedTexture));
                DestroyImmediate(unusedTexture, true);
            }
            atlasTexturesProperty.arraySize = 0;

            var atlasCount = 0;
            var unitSize = diceUnitSizeProperty.intValue;
            var paddingSize = paddingProperty.intValue;
            var paddedUnitSize = unitSize + paddingSize * 2;
            var forceSquare = this.forceSquareProperty.boolValue;
            var atlasSizeLimit = this.atlasSizeLimitProperty.intValue;
            var unitsPerAtlasLimit = Mathf.Pow(atlasSizeLimit / paddedUnitSize, 2);

            if (atlasSizeLimit < minHeight || atlasSizeLimit < minWidth)
            {
                atlasSizeLimit = 8196;
            }

            // Group name->units to name->hash->units map.
            var unitsToPackMap = dicedUnits.Select(nameToUnits => new KeyValuePair<string, Dictionary<int, List<DicedUnit>>>(nameToUnits.Key, nameToUnits.Value
                .GroupBy(units => units.ColorsHashCode).ToDictionary(hashToUnitsGroup => hashToUnitsGroup.Key, hashToUnitsGroup => hashToUnitsGroup.ToList())))
                .ToDictionary(nameToHashToUnits => nameToHashToUnits.Key, nameToHashToUnits => nameToHashToUnits.Value);

            // Pack units with distinct (inside atlas group) colors to the atlas textures.
            // Insure sprites integrity (units belonging to one sprite should be in a common atlas) and atlas size limit (distinct units per atlas count).
            while (unitsToPackMap.Count > 0)
            {
                atlasCount++;

                var atlasTexture = Utilities.CreateTexture(atlasSizeLimit, name: $"{target.name} {atlasCount:000}");
                var hashToUV = new Dictionary<int, Rect>(); // Colors hash to UV rects map of the packed diced units in the current atlas.
                var yToLastXMap = new Dictionary<int, int>(); // Y position of a units row in the current atlas to the x position of the last unit in this row.
                var xLimit = Mathf.NextPowerOfTwo(paddedUnitSize); // Maximum allowed width of the current atlas. Increases by the power of two in the process.
                var packedUnits = new List<DicedUnit>(); // List of the units packed to the current atlas.

                // Find units that can be packed to the current atlas (respecting atlas size limit and remaining free space).
                Func<KeyValuePair<string, Dictionary<int, List<DicedUnit>>>> findSuitableUnitsToPack = () => {
                    return unitsToPackMap.FirstOrDefault(nameToHashToUnits => {
                        var unitsToPackCount = nameToHashToUnits.Value.Count(hashToUnits => !hashToUV.ContainsKey(hashToUnits.Key));
                        return hashToUV.Keys.Count + unitsToPackCount <= unitsPerAtlasLimit;
                    });
                };

                var suitableUnits = findSuitableUnitsToPack();
                if (suitableUnits.Key == null) // None of the source textures fit atlas limit.
                {
                    Debug.LogError("SpriteDicing: Unable to fit input textures to the atlas. Consider increasing atlas size limit.");
                    return false;
                }

                while (suitableUnits.Key != null)
                {
                    var packingProgress = 1f - (unitsToPackMap.Count / (float)dicedUnits.Count);
                    DisplayProgressBar("Packing diced textures...", .5f + .5f * packingProgress);

                    // Iterate suitable for packing units grouped by their color hashes.
                    foreach (var hashToUnits in suitableUnits.Value)
                    {
                        if (hashToUV.ContainsKey(hashToUnits.Key))
                        {
                            // We've already packed unit with the same colors to this atlas; assign it's UVs to the others in the group.
                            hashToUnits.Value.ForEach(unit => unit.QuadUVs = hashToUV[hashToUnits.Key]);
                            continue;
                        }

                        int posX, posY; // Position of the new unit on the atlas texture.
                        // Find row positions that have enough room for more units until next power of two.
                        var suitableYToLastXEnumerable = yToLastXMap.Where(yToLastX => xLimit - yToLastX.Value >= paddedUnitSize * 2);
                        if (suitableYToLastXEnumerable.Count() == 0) // When no suitable rows found.
                        {
                            // Handle corner case when we just started.
                            if (yToLastXMap.Count == 0)
                            {
                                yToLastXMap.Add(0, 0);
                                posX = 0;
                                posY = 0;
                            }
                            // Determine whether we need to add a new row or increase x limit.
                            else if (xLimit > yToLastXMap.Last().Key)
                            {
                                var newRowYPos = yToLastXMap.Last().Key + paddedUnitSize;
                                yToLastXMap.Add(newRowYPos, 0);
                                posX = 0;
                                posY = newRowYPos;
                            }
                            else
                            {
                                xLimit = Mathf.NextPowerOfTwo(xLimit + 1);
                                posX = yToLastXMap.First().Value + paddedUnitSize;
                                posY = 0;
                                yToLastXMap[0] = posX;
                            }
                        }
                        else // When suitable rows found.
                        {
                            // Find one with the least number of elements and use it.
                            var suitableYToLastX = suitableYToLastXEnumerable.OrderBy(yToLastX => yToLastX.Value).First();
                            posX = suitableYToLastX.Value + paddedUnitSize;
                            posY = suitableYToLastX.Key;
                            yToLastXMap[posY] = posX;
                        }

                        // Write colors of the unit to the current atlas texture.
                        var colorsToPack = hashToUnits.Value.First().PaddedColors;
                        atlasTexture.SetPixels(posX, posY, paddedUnitSize, paddedUnitSize, colorsToPack);
                        // Evaluate and assign UVs of the unit to the other units in the group.
                        var unitUVRect = new Rect(posX, posY, paddedUnitSize, paddedUnitSize).Crop(-paddingSize).Scale(1f / atlasSizeLimit);
                        hashToUnits.Value.ForEach(unit => unit.QuadUVs = unitUVRect);
                        hashToUV.Add(hashToUnits.Key, unitUVRect);
                    }

                    packedUnits.AddRange(suitableUnits.Value.SelectMany(u => u.Value));
                    unitsToPackMap.Remove(suitableUnits.Key);
                    suitableUnits = findSuitableUnitsToPack();
                }

                // Crop unused atlas texture space.
                if (xLimit < minWidth) xLimit = minWidth;
                var yLimit = yToLastXMap.Last().Key + paddedUnitSize;
                if (yLimit < minHeight) yLimit = minHeight;
                var needToCrop = xLimit < atlasSizeLimit || (!forceSquare && yLimit < atlasSizeLimit);
                if (needToCrop)
                {
                    var croppedWidth = xLimit;
                    var croppedHeight = forceSquare ? croppedWidth : yToLastXMap.Last().Key + paddedUnitSize;
                    var croppedPixels = atlasTexture.GetPixels(0, 0, croppedWidth, croppedHeight);
                    atlasTexture = Utilities.CreateTexture(croppedWidth, croppedHeight, name: atlasTexture.name);
                    atlasTexture.SetPixels(croppedPixels);

                    // Correct UV rects after crop.
                    packedUnits.ForEach(unit => unit.QuadUVs = unit.QuadUVs
                        .Scale(new Vector2(atlasSizeLimit / (float)croppedWidth, atlasSizeLimit / (float)croppedHeight)));
                }

                // Save atlas texture.
                atlasTexture.alphaIsTransparency = true;
                atlasTexture.Apply();
                var savedTexture = atlasTexture.SaveAsPng(AssetDatabase.GetAssetPath(target));
                atlasTexturesProperty.arraySize = Mathf.Max(atlasTexturesProperty.arraySize, atlasCount);
                atlasTexturesProperty.GetArrayElementAtIndex(atlasCount - 1).objectReferenceValue = savedTexture;
                packedUnits.ForEach(unit => unit.AtlasTexture = savedTexture);
            }

            return true;
        }

If I've got it right, you're forcing the atlas to be larger just for the sake of preventing those error when creating sprites? Don't think that's a good idea, as the main purpose of this solution it to compress the textures as much as possible. Guess we'd have to either find some other solution or wait for Unity to allow using sprite rects without limitations.

@Elringus that's exactly what I did. The other way around is create a sprite with size smaller but then the uvs and mesh must be small as well so it can be inside the generate small rect. And although is a big texture map because it is on base 2 we still get improved size base on the number of sprites used to create the Atlas. Or we can create a empty total white texture2D with the biggest size and use it to "trick" the sprite create method. But that means have a white with no alpha sprite just for a workaround the sprite
Create limitation

looking at the UnityCSReference  Repository i find on the Sprites.bindings.cs this method.
// Workaround for Overloads as described in
        [FreeFunction("SpritesBindings::CreateSpriteWithoutTextureScripting")]
        internal extern static Sprite CreateSpriteWithoutTextureScripting(Rect rect, Vector2 pivot, float pixelsToUnits, Texture2D texture);

so i think is possible to use reflection to expose it and use to create the sprites without the texture needed.

Thanks for the info, I guess we're one step closer now :)
https://github.com/Elringus/SpriteDicing/blob/2b819aab3014f3ef09f9fedbe6a9952cb5677cf2/Assets/SpriteDicing/Editor/DicedSpriteAtlasEditor.cs#L530
Sprite with the correct rect is now created, but it's rendered with a huge offset. Not sure why this is happening, maybe it's still somehow connected with the rect being larger than the texture.

the rectsize if correct and the sprite uv, triangles and all is the correct size. the only problem is that it render from the center of rect size to >> and up só the only problem is the position where the sprite start render. or the rect size init position to up

Thanks for the info, I guess we're one step closer now :)
https://github.com/Elringus/SpriteDicing/blob/2b819aab3014f3ef09f9fedbe6a9952cb5677cf2/Assets/SpriteDicing/Editor/DicedSpriteAtlasEditor.cs#L530
Sprite with the correct rect is now created, but it's rendered with a huge offset. Not sure why this is happening, maybe it's still somehow connected with the rect being larger than the texture.

the problem was the pivot. i don't think the pivot calculation works like the other sprite.create. i just changed to Vector2.zero and the pivot is put on the center and work like a charm

As I've mentioned in the PR, it doesn't work. I've also noticed that the extent of the offset depends on the diced unit size.

As I've mentioned in the PR, it doesn't work. I've also noticed that the extent of the offset depends on the diced unit size.

can you give me more sprites to test it? i want to make more tests in different sprites.

Sure, here is the one from the screenshot with an offset: https://user-images.githubusercontent.com/2056864/61733209-f0950900-ad87-11e9-963f-6323d871d346.png

@Elringus do you know a way to reposition the mesh/uv/vert position so it can be render like 50% to the left or up? If you know how let me know I'm stuck trying to understand how repositioning.

Just changing the vertices position should be enough for a local position offset, eg: https://github.com/Elringus/SpriteDicing/blob/2b819aab3014f3ef09f9fedbe6a9952cb5677cf2/Assets/SpriteDicing/Editor/DicedSpriteAtlasEditor.cs#L497

i'm trying to learn how to calculate uvs an others things i looking into the SVG packge from unity they have a sprite generator using uv there.
void GenerateSprite(VertexHelper vh)
    {
        var spriteSize = new Vector2(sprite.rect.width, sprite.rect.height);

        // Covert sprite pivot into normalized space.
        var spritePivot = sprite.pivot / spriteSize;
        var rectPivot = rectTransform.pivot;
        Rect r = GetPixelAdjustedRect();
        var drawingSize = new Vector2(r.width, r.height);
        var spriteBoundSize = sprite.bounds.size;

        // Calculate the drawing offset based on the difference between the two pivots.
        var drawOffset = (rectPivot - spritePivot) * drawingSize;

        bool hasColorAttribute = sprite.HasVertexAttribute(VertexAttribute.Color);
        if (hasColorAttribute)
            s_SpriteColor = sprite.GetVertexAttribute<Color32>(VertexAttribute.Color);

        bool hasTextCord2Attribute = sprite.HasVertexAttribute(VertexAttribute.TexCoord2);
        if (hasTextCord2Attribute)
            s_TextCord2 = sprite.GetVertexAttribute<Vector2>(VertexAttribute.TexCoord2);

        var color32 = color;
        vh.Clear();

        Vector2[] vertices = sprite.vertices;
        Vector2[] uvs = sprite.uv;
        for (int i = 0; i < vertices.Length; ++i)
        {
            vh.AddVert(new Vector3((vertices[i].x / spriteBoundSize.x) * drawingSize.x - drawOffset.x, (vertices[i].y / spriteBoundSize.y) * drawingSize.y - drawOffset.y),
                hasColorAttribute ? color32 * s_SpriteColor[i] : color32, uvs[i]);

            // VertexHelper access to uv2 isn't great work around the API for now. Copy current vert out and then back with the proper uv2 if we have it.
            if (hasTextCord2Attribute)
            {
                vh.PopulateUIVertex(ref s_TempVertex, vh.currentVertCount - 1);
                s_TempVertex.uv2 = s_TextCord2[i];
                vh.SetUIVertex(s_TempVertex, vh.currentVertCount - 1);
            }
        }

        UInt16[] triangles = sprite.triangles;
        for (int i = 0; i < triangles.Length; i += 3)
        {
            vh.AddTriangle(triangles[i + 0], triangles[i + 1], triangles[i + 2]);
        }
    }

right now i'm just able to generate sprite on the right size with pivot always on center here is the CreateSprite method.
private Sprite CreateSprite (string name, Texture2D atlasTexture, List<DicedUnit> dicedUnits)
        {
            var vertices = new List<Vector2>();
            var uvs = new List<Vector2>();
            var triangles = new List<ushort>();

            #region Functions

            void AddDicedUnit (DicedUnit dicedUnit) => AddQuad(dicedUnit.QuadVerts.min, dicedUnit.QuadVerts.max, dicedUnit.QuadUVs.min, dicedUnit.QuadUVs.max);

            void AddQuad (Vector2 posMin, Vector2 posMax, Vector2 uvMin, Vector2 uvMax)
            {
                var startIndex = vertices.Count;

                AddVertice(new Vector2(posMin.x, posMin.y), new Vector2(uvMin.x, uvMin.y));
                AddVertice(new Vector2(posMin.x, posMax.y), new Vector2(uvMin.x, uvMax.y));
                AddVertice(new Vector2(posMax.x, posMax.y), new Vector2(uvMax.x, uvMax.y));
                AddVertice(new Vector2(posMax.x, posMin.y), new Vector2(uvMax.x, uvMin.y));

                AddTriangle(startIndex, startIndex + 1, startIndex + 2);
                AddTriangle(startIndex + 2, startIndex + 3, startIndex);
            }

            void AddVertice (Vector2 position, Vector2 uv)
            {
                vertices.Add(position);
                uvs.Add(uv);
            }

            void AddTriangle (int idx0, int idx1, int idx2)
            {
                triangles.Add((ushort)idx0);
                triangles.Add((ushort)idx1);
                triangles.Add((ushort)idx2);
            }

            // Reposition the vertices so that they start at the local origin (0, 0).
            Vector2 TrimVertices (Rect rect)
            {
                if (rect.min.x > 0 || rect.min.y > 0)
                    for (int i = 0; i < vertices.Count; i++)
                        vertices[i] -= rect.min;

                var pivotX = rect.min.x / rect.size.x;
                var pivotY = rect.min.y / rect.size.y;
                return new Vector2(-pivotX, -pivotY);
            }

            // Evaluate sprite rectangle using vertex data.
            Rect EvaluateSpriteRect ()
            {
                var minVertPos = new Vector2(vertices.Min(v => v.x), vertices.Min(v => v.y));
                var maxVertPos = new Vector2(vertices.Max(v => v.x), vertices.Max(v => v.y));
                var spriteSizeX = Mathf.Abs(maxVertPos.x - minVertPos.x);
                var spriteSizeY = Mathf.Abs(maxVertPos.y - minVertPos.y);
                var spriteSize = new Vector2(spriteSizeX, spriteSizeY);
                return new Rect(minVertPos, spriteSize);
            }

            Vector2 GetSpriteSize()
            {
                var minVertPos = new Vector2(vertices.Min(v => v.x), vertices.Min(v => v.y));
                var maxVertPos = new Vector2(vertices.Max(v => v.x), vertices.Max(v => v.y));
                var spriteSizeX = Mathf.Abs(maxVertPos.x - minVertPos.x);
                var spriteSizeY = Mathf.Abs(maxVertPos.y - minVertPos.y);
                return new Vector2(spriteSizeX, spriteSizeY);
            }

            #endregion

            foreach (var dicedUnit in dicedUnits)
                AddDicedUnit(dicedUnit);

            var ppu = pixelsPerUnitProperty.floatValue;
            var spriteRect = EvaluateSpriteRect().Scale(1);
            var originalPivot = TrimVertices(spriteRect);
            var pivot = keepOriginalPivotProperty.boolValue ? originalPivot : defaultPivotProperty.vector2Value;

            //new sprite pivot
            var keepSize = spriteRect.size;
            var drawOffset = Vector2.zero;
            drawOffset.x = (spriteRect.size.x * pivot.x);
            drawOffset.y = (spriteRect.size.y * pivot.y);

            // Public sprite ctor won't allow using a rect that is larger than the atlas texture.
            var sprite = typeof(Sprite).GetMethod("CreateSpriteWithoutTextureScripting", BindingFlags.NonPublic | BindingFlags.Static)
                ?.Invoke(null, new object[] { spriteRect, pivot, 1, atlasTexture }) as Sprite;
            sprite.name = name;
            sprite.SetVertexCount(vertices.Count);
            sprite.SetIndices(new NativeArray<ushort>(triangles.ToArray(), Allocator.Temp));
            sprite.SetVertexAttribute(VertexAttribute.Position, new NativeArray<Vector3>(vertices.Select(v => new Vector3((v.x/spriteRect.width) * keepSize.x - drawOffset.x, (v.y/spriteRect.height) * keepSize.y - drawOffset.y)).ToArray(), Allocator.Temp));
            sprite.SetVertexAttribute(VertexAttribute.TexCoord0, new NativeArray<Vector2>(uvs.ToArray(), Allocator.Temp));

            return sprite;
        }


Consistent syntax for *.proto

How should the file extension *.proto be shown in tests? There is inconsistent usage between italics - *.proto - and code - *.proto.
Which is correct?

Code without an asterisk, as `.proto` for .proto. The asterisk wasn't thought to be meaningful; and although the MS Style Manual doesn't call for it, we code-fence them to prevent localization.

I agree with @guardrex , Code without an asterisk, as `.proto` for .proto.
@guardrex  BTW, Jason Card is no longer our Loc rep.

Thx ... do we have a replacement for Jason to ping in case a future question arises?

This should stay open until all the .proto references are updated. It should just be a matter of a couple of find/replaces to make them consistent.

Do you want me to knock it out in the morning?

hi @guardrex
@khoiph1 (Khoi Pham) is the current localization owner you can ping then needed.

Thanks @JasonCard ... and for all of your help here over the last few years. It was always great working with you on loc issues.


Navigation scrolls per page, not per item

Panning / sliding the carousel works fine, but clicking next or previous arrows scrolls per page, not per item.
:scroll-per-page="false" doesn't help

Hey @nataliawojtkowska, this is an open issue - you can follow @ #134! If you have any suggestions feel free to comment there or open a PR :~)


KS test gives incorrect test_statistic

First: thank you for creating and sharing this software.
Describe the bug
When performing a KS test on extremes selected with POT and a generalized Pareto distribution, the location parameter is not given to scipy.stats.kstest. As a result, test_statistic is incorrect and the result of the KS test may be incorrect.
To Reproduce
$ wget https://github.com/georgebv/pyextremes-notebooks/raw/master/data/battery_wl.csv

import pyextremes as px
import pandas as pd

data = (
    pd
    .read_csv("battery_wl.csv", index_col=0, parse_dates=True)
    .squeeze('columns')
    .sort_index(ascending=True)
    .astype(float)
    .dropna()
)
data = data.loc[pd.to_datetime("1925"):]
data = data - (data.index.array - pd.to_datetime("1992")) / pd.to_timedelta("365.2425D") * 2.87e-3

model = px.EVA(data=data)

model.get_extremes(
    method="POT",
    extremes_type="high",
    threshold = 1.4,
    r = '12H'
)

model.fit_model()

model.test_ks()

output:
                     Kolmogorov-Smirnov Test
------------------------------------------------------------------
Null hypothesis: data follows genpareto distribution
Alternative hypothesis: data doesn't follow genpareto distribution

Test statistic: D = 0.9906864591234011
p-value: 3.829434059334826e-179

Significance level: 0.05
Critical value: 0.14274059053041283
Critical region: reject null-hypothesis if D > 0.14274059053041283

Test result: data doesn't follow genpareto distribution
------------------------------------------------------------------

Note: Test statistic returns D = 0.9906864591234011
Expected behavior
Compute test statics directly with scipy.stats.kstest:
from scipy import stats
import functools

# old test statistic
gpcdf = functools.partial(stats.genpareto.cdf,**model.model.fit_parameters) # fixed parameters not included
print(stats.kstest(model.extremes,gpcdf).statistic)
# 0.9906864591234011

# correct test statistic
gpcdf2 = functools.partial(
    stats.genpareto.cdf,
    **{'loc' if k=='floc' else k:v for k,v in model.distribution.fixed_parameters.items()}, # fixed parameters included
    **model.model.fit_parameters
)
print(stats.kstest(model.extremes,gpcdf2).statistic)
# 0.09295193640733063

System:
Python 3.10.4
  Operating System: CentOS Linux 7 (Core)
      Architecture: x86-64

@eastjames good catch. Based on my review, the test itself is implemented correctly but fit_parameters passed to it in the EVA class are indeed incorrect (must be both fixed and free (aka fit) parameters). We need to provide a merged dict of fixed and free parameters in the EVA.test_ks method. Would you like to submit a pull request with this fix? I would appreciate a contribution, especially since you found the error and solution already.

@georgebv apologies for my delay. Yes -- if it is not immediately urgent, I would be glad to submit a pull request with a fix in the next few weeks.

@eastjames absolutely, no rush, contributions are welcome any time. Please ping me here whenever you prepare the PR

@eastjames I know that you likely moved on, but reminding in case you are still interesting in contributing to the project. If not, no worries - let me know and I will close the issue.


Upgrade to expat-2.6.2

The old release is no longer hosted upstream so the build fails.

Thanks, Alex!
Perhaps I should figure out some way to get these from GitHub, because
these fossies.org links have been unreliable. That's the second time this
month it's happened. The problem with the official download URL has been
that the libexpat devs use weird tag names that don't match the actual
version number, e.g. R_2_6_2 instead of 2.6.2 or v2.6.2.


Update DWN SDK

Updating the DWN SDK to the latest version.
*** Needs another release cut and version bump from dwn-sdk-js due to needed exported types, as well as a change to the data size threshold.
This Includes:

encodedData stored within the MessageStore below a certain threshold
MessageStore interface now accepts multiple Filters to create OR conditions between filters.
'MessageStore' interface accepts sorting and pagination inputs
If sorting by datePublished, add the property published set to true within the filter.

Relevent dwn-sdk-js PRs:
https://github.com/TBD54566975/dwn-sdk-js/pull/478
https://github.com/TBD54566975/dwn-sdk-js/pull/456

Codecov Report

Merging #7 (dbcf195) into main (857cceb) will increase coverage by 2.97%.
The diff coverage is 96.49%.

@@            Coverage Diff             @@
##             main       #7      +/-   ##
==========================================
+ Coverage   77.04%   80.02%   +2.97%
==========================================
  Files          10       10
  Lines         806      956     +150
  Branches      120      140      +20
==========================================
+ Hits          621      765     +144
- Misses        185      191       +6

@amika-sq version was bumped on sdk, ready for final review.


mle: fix on darwin

Description of changes

fix on darwin, see https://github.com/NixOS/nixpkgs/pull/80131#issuecomment-899122362
install man page

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 169200 run on x86_64-darwin 1

1 package built:

mle