Instead of librosa.stft, what can i use in audio.py?

Guys, i am trying to implement this project on a embedded device. Because of some limitations on the embedded device, i can not install librosa. That's why, i am trying to replace stft and istft methods which are utilized inside audio.py. Do you have any suggestions? @Kyubyong

Tensorflow has the tf.signal.stft function in Tensorflow 1.15 and Tensorflow 2.X. Have you tried those?


locking with package and git dependencies gets confused

With one of the latest updates rebar3 gets confused when the same dependency is included at top level via a pkg but in a dependency via a git.
It seems what happens is that when he .lock file is first created the top level pkg takes precedence but then when the next command is run some of the lower level git takes priority and fails with:
===> sh info:
	cwd: "/Users/heinz/Projects/fifo/core/howl"
	cmd: git --git-dir="/Users/heinz/Projects/fifo/core/howl/_build/default/lib/jsxd/.git" rev-parse --verify HEAD

===> 	opts: [{use_stdout,false},
                       {debug_abort_on_error,
                           [76,111,99,107,105,110,103,32,111,102,32,103,105,
                            116,32,100,101,112,101,110,100,101,110,99,121,32,
                            102,97,105,108,101,100,32,105,110,32,
                            "/Users/heinz/Projects/fifo/core/howl/_build/default/lib/jsxd"]}]

===> Port Cmd: git --git-dir="/Users/heinz/Projects/fifo/core/howl/_build/default/lib/jsxd/.git" rev-parse --verify HEAD
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> sh(git --git-dir="/Users/heinz/Projects/fifo/core/howl/_build/default/lib/jsxd/.git" rev-parse --verify HEAD)
failed with return code 128 and the following output:
fatal: Not a git repository: '/Users/heinz/Projects/fifo/core/howl/_build/default/lib/jsxd/.git'

===> Locking of git dependency failed in /Users/heinz/Projects/fifo/core/howl/_build/default/lib/jsxd

a repository showing this can be found here: https://github.com/project-fifo/howl/tree/rebar3-lock-error

The first package winning is likely a bug because source always takes precedence over packages.

Even when src is in a dependency and not the same level? I think that would be dangerous.

let me explain dangerous, if any dependency of a dependency of a dependency happens to include  a {branch, "master"} it would overwrite a versioned package and all hell could brake loose and the only way to mitigate it would be moving from a package to git.
That seems to make packages rather useless unless everyone everywhere uses them.

That's been the documented behaviour as long as the tool has existed. The idea is that if there's a behaviour you hate in packages dependency chains, you now have a way to override it at any level and have your source dep take over the whole chain.
I believe we're trying to operate with the assumption that eventually, people will use more and more packages, not that we're gonna be stuck in the crappy github purgatory forever.

Yea, you can always use overrides to get rid of the source dep.
But this is a bug. I'll look into it.

I agree with the goal but this seems to force people into the different behavior.
say we have app A, that depends on B, that depends on C. A also depends on on C, since A is written by a modern guy A uses packages and picks a version that suites them.
A -> B (git tagged) -> C (git master)
`-> C (pkg 0.1.0)

This would end up with the deps
B (git tag xxx)
C (git master)

so the modern author of A is forced to also include A as a git dependency so instead of promoting package in A now A is 'infected' by git.

but then you'd need to include the entire overrides all of the deps of the library would't it? I mean it's possible but in larger projects it'd probably be a huge PITA.
I don't see what advantage there is that a sub (sub sub sub sub sub sub) dependency could override the top level rebar config it sounds just wrong to me.
In the above example updating B would potentially breaking A by indirection which feels so wrong to me.

Yes, but that's how we went for it anyway. It's easy to patch a source dep to fix it and change things. You can't easily fork a package and then crush it with a new version without renaming it and just forking it permanently and publicly. Therefore source taking precedence is a reasonable default.


Add serialization support for various Crypto types

Also adds a serialization test for the Block.

There seem to be weird MacOS failures on GithubCI, something to do with missing symbols. And I just got a relevant e-mail from Github saying they are upgrading to MacOS Catalina, but we can still use Mojave. I wonder if they screwed something up. :D

And I just got a related e-mail from Github saying they are upgrading to MacOS Catalina, but we can still use Mojave. I wonder if they screwed something up. :D

No they didn't, but it means we can't use < 2.087.1 because of this.

No they didn't

Actually scratch that. It shouldn't fail on 10.14.

At this point... Should we just remove the restriction ?

Restriction on what?

The serialization requiring to implement serialize instead of just tupleofing stuff.

Fixed missing public attributes. Once it's green, you can merge it @TrustHenry.


Update toolchain and manifest to Rust 1.70.0

I noticed this error when opening the repo:

So updated Cargo.toml and rust-toolchain in the project root from 1.69.0 » 1.70.0
This repo looks really amazing, btw, bravo!

Thank you for bringing this up! We have workarounds for building with 1.69 and use them in CI, but missed to highlight them to users.
We cannot yet bump the MSRV as some users of this crate still need to use Rust 1.69.0. To prevent issues like this, I’ve opened #131. After it lands, user with a toolchain >= 1.70.0 should not hit this issue anymore, as the override is removed from rust-toolchain (ref).


Extension proposal: multiscale arrays v0.1

As a first draft of support for the multi scale use-case (https://github.com/zarr-developers/zarr-specs/issues/23), this issue proposes an intermediate nomenclature for describing groups of zarr arrays which are scaled down versions of one another, e.g.:
example/
├── 0    # Full-sized array
├── 1    # Scaled down 0, e.g. 0.5; for images, in the X&Y dimensions
├── 2    # Scaled down 1, ...
├── 3    # Scaled down 2, ...
└── 4    # Etc.

This layout was independently developed in a number of implementations, including:

https://github.com/napari/napari
https://github.com/glencoesoftware/bioformats2raw
Yours here

but without consistent metadata to make those arrays discoverable.
This issue proposes adding metadata at the group layer of the form:
{
  “multiscale”: {
    “version” : “0.1”,
    “series” : [
      {
        “datasets” : [“0”, “1”, “2”, “3”, "4"]
      }
     // See the detailed example below for optional metadata
  }
}

where multiple series of datasets can be present in a single group. The goals of this metadata include:

fostering a common vocabulary between existing implementations
enabling other implementations to reliably detect multiscale arrays
testing this extension for limitations against multiple use cases
permitting the upgrade of v0.1 arrays to future versions of this or other extension

Process
An RFC process for Zarr does not yet exist. Additionally, the v3 spec is a work-in-progress. However, since the implementations listed above as well as others are already being developed, I'd propose that if a consensus can be reached here, this issue should be turned into an .rst file similar to those in the v3 branches (e.g. filters) and used as a temporary spec for defining arrays with the understanding that this a prototype intended to be amended and brought into the general extension mechanism as it develops.
I'd welcome any suggestions/feedback, but especially around:

Better terms for "multiscale" and "series"
The most useful enum values
Is this already too complicated? (Limit to one series per group?) or on the flip side:
Are there existing use cases that aren't supported? (Note: I'm aware of some examples like BDV's N5 format but I'd suggest they are higher-level than just "multiscale arrays".)

Deadline for a first round of comments: March 15, 2020
Detailed example
Color key (according to https://www.ietf.org/rfc/rfc2119.txt):
- MUST     : If these values are not present, the multiscale series will not be detected.
! SHOULD   : Missing values may cause issues in future versions.
+ MAY      : Optional values which can be readily omitted.
# UNPARSED : When updating between versions, no transformation will be performed on these values.

Color-coded example:
-{
-  “multiscale”: {
!    “version” : “0.1”,
-    “series” : [
-      {
-        “datasets” : [“0”, “1”, “2”, “3”], (A)
!        “metadata” : [
!          “scales”: [0.5, 0.5, 1, 1, 1], (B)
!          “type”: “gaussian” (C)
+          “method”:
#               “skimage.transform.pyramid_gaussian”,
+          “version”:
#                “0.16.1”,
+          “arguments”: {
#            “multichannel”: true (D)
+          }
+        ]
-      },
+      {
+        // Other series can be in the same group
+      }
-    ]
-  }
-}

Notes:

(A) These paths could potentially point to datasets in other groups via “../foo/0” in the future. For now, the identifiers MUST be local to the annotated group.
(B) I.e. the first two dimensions were halved in size.
(C) These values SHOULD (MUST?) come from the enumeration.
(D) An example taken from https://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.pyramid_reduce

Type enumeration:

gaussian, e.g. skimage.transform.pyramid_gaussian
laplacian,  e.g. skimage.transform.pyramid_laplacian
reduce,  e.g. skimage.transform.pyramid_laplacian
pick, e.g. SimpleImageScaler's “top-left” strategy
Suggestions welcome

Sample code
import zarr
import numpy as np
from skimage import data
from skimage.transform import pyramid_gaussian, pyramid_laplacian
from tempfile import TemporaryDirectory

base = np.tile(data.astronaut(), (2, 2, 1))

gaussian = list(
    pyramid_gaussian(base, downscale=2, max_layer=4, multichannel=True)
)

laplacian = list(
    pyramid_laplacian(base, downscale=2, max_layer=4, multichannel=True)
)

with TemporaryDirectory(suffix='.zarr') as fout:

    store = zarr.DirectoryStore(fout)
    grp = zarr.group(store)
    print(fout)

    grp.create_dataset("base", data=base)

    series_G = []
    for g, dataset in enumerate(gaussian):
        if g == 0:
            name = "base"
        else:
            name = "G%s" % g
            grp.create_dataset(name, data=gaussian[g])
        series_G.append(name)

    series_L = []
    for l, dataset in enumerate(laplacian):
        if l == 0:
            name = "base"
        else:
            name = "L%s" % l
            grp.create_dataset(name, data=laplacian[l])
        series_L.append(name)

    grp.attrs["multiscale"] = {
        "version": "0.1",
        "series": [
            {"dataset": series_G,
             "metadata": {"type": "gaussian"}},
            {"dataset": series_L,
             "metadata": {"type": "laplacian"}}
        ]
    }
    # .zarr disappears after this block closes

which results in a .zattrs file of the form:
{
    "multiscale": {
        "series": [
            {
                "dataset": [
                    "base",
                    "G1",
                    "G2",
                    "G3",
                    "G4"
                ],
                "metadata": {
                    "type": "gaussian"
                }
            },
            {
                "dataset": [
                    "base",
                    "L1",
                    "L2",
                    "L3",
                    "L4"
                ],
                "metadata": {
                    "type": "laplacian"
                }
            }
        ],
        "version": "0.1"
    }
}

and the following on-disk layout:
/var/folders/z5/txc_jj6x5l5cm81r56ck1n9c0000gn/T/tmp77n1ga3r.zarr
├── G1
│   ├── 0.0.0
...
│   └── 3.1.1
├── G2
│   ├── 0.0.0
│   ├── 0.1.0
│   ├── 1.0.0
│   └── 1.1.0
├── G3
│   ├── 0.0.0
│   └── 1.0.0
├── G4
│   └── 0.0.0
├── L1
│   ├── 0.0.0
...
│   └── 3.1.1
├── L2
│   ├── 0.0.0
│   ├── 0.1.0
│   ├── 1.0.0
│   └── 1.1.0
├── L3
│   ├── 0.0.0
│   └── 1.0.0
├── L4
│   └── 0.0.0
└── base
    ├── 0.0.0
...
    └── 1.1.1

9 directories, 54 files

Thanks to @ryan-williams, @jakirkham, @freeman-lab, @petebankhead, @jni, @sofroniewn, @chris-allan, and anyone else who's GitHub account I've forgotten for the preliminary discussions.

I'm really happy see this. We also used a similar layout for storing pyramids initially to that proposed above and it's fantastic to see this formalized.
I'm curious about the decision to store the base array in the same group as the downsampled levels. We initially did the same, but then moved towards a structure separating the two:
└── example/
    ├── .zgroup
    ├── base
    │   ├── .zarray
    │   ├── .zattrs
    │   ├── 0.0.0
    │   └── ...etc
    └── sub-resolutions/
        ├── .zgroup
        ├── .zattrs
        ├── 01/
        │   ├── .zarray
        │   ├── 0.0.0
        │   └── ...etc
        └── 02/
            ├── .zarray
            ├── 0.0.0
            └── ...etc

as a more general "image" format in zarr. One could expect to find a "base" array and then check for the "sub-resolutions" group to determine if it is a pyramid or not. We thought this structure would allow for other types of data (e.g. segmentation) to be store along side the base array. Again, thanks for the work here in formalizing this!

Thanks, @manzt. Let's see if there are more votes for the deeper representation. It's certainly also what I was originally thinking about in #23. The downside is that one likely needs metadata on all the datasets pointing up and down the hierarchy in order to support detection of the sequence from any scale. It's the other major design layout I can think of. (If anyone has more, those would be very welcome.)

@joshmoore amazing to see this kick off. A couple short comments

If looking for alternate names I'd consider multiresolution, but multiscale definitely works for me. We have been using pyramid at in napari but are thinking of changing (see here https://github.com/napari/napari/issues/1019#issuecomment-595325260 and we can try and go with whatever the majority likes).

One thing that has come up for me and a list of "scales" is that when you have large volumetric timeseries, where you might create a pyramid for each timepoint, some of the axes are unscaled, so you really need to look at the shapes of the arrays to do the right thing. I see that the field is optional but I wonder how much is gained from it (I'm also not opposed though)

Multiple series per group is probably good flexibility to have, say if you have two independent multiscale datasets you want to put in the same group, it lets the group abstraction remain separate from the multiscale details.

The concept of having base + subresolutions like @manzt proposes is intriguing to me too. Ultimately for visualization purposes I want something like a single list of arrays so I guess I find that representation little simpler, but I can construct that from the later representation if I know the data is multiscale and maybe it is nice to keep that a little separate. I will think on it more, curious what others say.

Glad to see the discussion here. Some thoughts:

Philosophically, I'd like to suggest a few constraints (both of which are satisfied by @joshmoore's proposal, but not by a lot of other existing multiscale image schemas): First, individual images should be portable -- wherever possible, images should not have metadata/attributes that indicates their role in a multiscale representation, so that they can be copied somewhere else and viewed on their own without losing context. Second, no magic dataset names like s0, s1, etc. The use of the list of datasets in @joshmoore's group attributes solves this problem.

Personally I'm not a fan of putting the base image at a different level of the hierarchy, since most software i've seen assumes that all the different scale levels will all be elements in the same collection. @manzt you suggest that you adopted this structure in order to facilitate checking for a multiscale representation, but I think this is a job for group metadata, not hierarchy.

For simplicity, I would propose a restriction of one multiscale representation per group. Groups are cheap; if you to represent 2 multiscale images, then make 2 groups.

A multiscale image is a collection of images. Accordingly, the "multiscaleness" should be a group attribute that lists the images in the collection, which is how @joshmoore does it in the draft prosposal. I would add some dataset-specific information to the group attributes: software that consumes multiscale images needs to know about how the spatial properties of each image, and on cloud storage it can be cumbersome to query each image individually; so for convenience this image metadata could also be in the group attributes that describe the multiscale representation. I think explicitly listing the transform attributes of each image is safer than just listing "scales", as long as the transform attributes of each image are small.

Here's example metadata that implements this concept. The specifics of the "transform attributes" don't really matter -- this could be an affine transform, or something fancier. But I think the basic idea of putting the spatial information of each dataset in the group attributes is solid.
// group attributes
{
  “multiscale”: {
    “version” : “0.1”,
        “datasets” : [“0” : {transform attributes of 0},
                      “1” : {transform attributes of 1},
                      “2” : {transform attributes of 2},
                      “3” : {transform attributes of 3},
                      "4" : {transform attributes of 4}]
    }
    // optional stuff
  }
}

// example transform attributes of dataset 0

"transform" : {
    "offset" : {"X" : 0, "Y" : 0, "Z" : 0},
    "scale" : {"X" : 1, "Y" : 1, "Z" : 1},
    "units" : {"X" : "nm", "Y" : "nm", "Z" : "nm"}
}

// example spatial attributes of dataset 1

"transform" : {
    "offset" : {"X" : .5, "Y" : .5, "Z" : 0},
    "scale" : {"X" : 2, "Y" : 2, "Z" : 1},
    "units" : {"X" : "nm", "Y" : "nm", "Z" : "nm"}
}

For posterity, I've written about this issue (as it pertains to the data our group works with) here

@sofroniewn The concept of having base + subresolutions like @manzt proposes is intriguing to me too. Ultimately for visualization purposes I want something like a single list of arrays so I guess I find that representation little simpler, but I can construct that from the later representation if I know the data is multiscale and maybe it is nice to keep that a little separate. I will think on it more, curious what others say.

I generally have the same feelings. I'm for the simplicity of the current proposal, and I wonder if my suggestion adds an extra layer of complexity unnecessarily.

@d-v-b For simplicity, I would propose a restriction of one multiscale representation per group. Groups are cheap; if you to represent 2 multiscale images, then make 2 groups.

Wouldn't this require copying the base image into a separate group? Perhaps I'm misunderstanding.

Wouldn't this require copying the base image into a separate group? Perhaps I'm misunderstanding.

The base image would be in the same group with the downscaled versions. So on the file system, it would look like this:
└── example/
    ├── .zgroup
    ├── base
    │   ├── .zarray
    │   ├── .zattrs
    │   ├── 0.0.0
    │   └── ...etc
    ├── base_downscaled
    │   ├── .zarray
    │   ├── .zattrs
    │   ├── 0.0.0
    │   └── ...etc
    ...etc

Apologies, I thought you were suggesting that separate groups should be created for different sampling of the same base image (e.g. gaussian and laplacian).

@manzt
this is actually my mistake -- I was not thinking at all about the use case where the same base image is used for multiple pyramids, and I agree that copying data is not ideal. I will remove / amend the "one multiscale representation per group" part of my proposal above.

I would add some dataset-specific information to the group attributes: software that consumes multiscale images needs to know about how the spatial properties of each image, and on cloud storage it can be cumbersome to query each image individually;

Adding to the practical importance here: the spatial position of the first pixel is shifted in subresolutions, and the physical spacing between pixels changes also. This must be accounted for during visualization or analysis when other datasets, e.g. other images or segmentations, come into play. If this metadata is readily and independently available for every subresolution, i.e. scale factors do not need to be fetched and computations made, each subresolution image can be used independently, effortlessly, and without computational overhead.
One option is to build on the model implied by storing images in the Xarray project data structures, which has Zarr support. This enables storing metadata such as the position of the first pixel, the spacing between pixels, and identification of the array dimensions, e.g., x, y, t, so that data can be used and passed through processing pipelines and visualization tools. This is helpful because it enables distributed computing via Dask and machine learning [2] via the scikit-learn API. Xarray has broad community adoption, and it is gaining more traction lately. Of course, a model that is compatible with Xarray does not require Xarray to use the data. On the other hand, Xarray coords have more flexibility than what is required for pixels sampled on a uniform rectilinear grid, and this adds a little complexity to the layout.
Generated from this example, here is what it looks like:
.
├── level_1.zarr
│   ├── rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6
│   │   ├── 0.0.0
│   │   ├── 0.0.1
....
│   │   ├── 9.9.9
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── x
│   │   ├── 0
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── y
│   │   ├── 0
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── z
│   │   ├── 0
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── .zattrs
│   ├── .zgroup
│   └── .zmetadata
├── level_2.zarr
│   ├── rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6
│   │   ├── 0.0.0
│   │   ├── 0.0.1

│   │   ├── 8.9.9
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── x
│   │   ├── 0
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── y
│   │   ├── 0
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── z
│   │   ├── 0
│   │   ├── .zarray
│   │   └── .zattrs
│   ├── .zattrs
│   ├── .zgroup
│   └── .zmetadata
....
├── rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6
│   ├── 0.0.0
│   ├── 0.0.1
...
│   ├── 9.9.9
│   ├── .zarray
│   └── .zattrs
├── x
│   ├── 0
│   ├── .zarray
│   └── .zattrs
├── y
│   ├── 0
│   ├── .zarray
│   └── .zattrs
├── z
│   ├── 0
│   ├── .zarray
│   └── .zattrs
├── .zattrs
├── .zgroup
└── .zmetadata

34 directories, 62359 files

This is the layout generated by xarray.DataSet.to_zarr. It does not mean that Xarray has to be used to read and write. But, it would mean that Zarr images would be extremely easy to use via xarray. In this case, .zmetadata is generated on each subresolution so it can be used entirely independently. Due to how Xarray/Zarr handles coords, x, y, are one dimensional arrays. This results in every resolution having its own group.
The metadata looks like this:
{
    "metadata": {
        ".zattrs": {
            "_MULTISCALE_LEVELS": [
                "",
                "level_1.zarr",
                "level_2.zarr",
                "level_3.zarr",
                "level_4.zarr",
                "level_5.zarr",
                "level_6.zarr"
            ],
            "_SPATIAL_IMAGE": "rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6"
        },
        ".zgroup": {
            "zarr_format": 2
        },
        "level_1.zarr/.zattrs": {},
        "level_1.zarr/.zgroup": {
            "zarr_format": 2
        },
        "level_1.zarr/rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zarray": {
            "chunks": [
                64,
                64,
                64
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "zstd",
                "id": "blosc",
                "shuffle": 0
            },
            "dtype": "|u1",
            "fill_value": null,
            "filters": null,
            "order": "C",
            "shape": [
                1080,
                1280,
                1280
            ],
            "zarr_format": 2
        },
        "level_1.zarr/rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z",
                "y",
                "x"
            ],
            "direction": [
                [
                    1.0,
                    0.0,
                    0.0
                ],
                [
                    0.0,
                    1.0,
                    0.0
                ],
                [
                    0.0,
                    0.0,
                    1.0
                ]
            ],
            "units": "\u03bcm"
        },
        "level_1.zarr/x/.zarray": {
            "chunks": [
                1280
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                1280
            ],
            "zarr_format": 2
        },
        "level_1.zarr/x/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "x"
            ]
        },
        "level_1.zarr/y/.zarray": {
            "chunks": [
                1280
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                1280
            ],
            "zarr_format": 2
        },
        "level_1.zarr/y/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "y"
            ]
        },
        "level_1.zarr/z/.zarray": {
            "chunks": [
                1080
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                1080
            ],
            "zarr_format": 2
        },
        "level_1.zarr/z/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z"
            ]
        },
        "level_2.zarr/.zattrs": {},
        "level_2.zarr/.zgroup": {
            "zarr_format": 2
        },
        "level_2.zarr/rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zarray": {
            "chunks": [
                64,
                64,
                64
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "zstd",
                "id": "blosc",
                "shuffle": 0
            },
            "dtype": "|u1",
            "fill_value": null,
            "filters": null,
            "order": "C",
            "shape": [
                540,
                640,
                640
            ],
            "zarr_format": 2
        },
        "level_2.zarr/rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z",
                "y",
                "x"
            ],
            "direction": [
                [
                    1.0,
                    0.0,
                    0.0
                ],
                [
                    0.0,
                    1.0,
                    0.0
                ],
                [
                    0.0,
                    0.0,
                    1.0
                ]
            ],
            "units": "\u03bcm"
        },
        "level_2.zarr/x/.zarray": {
            "chunks": [
                640
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                640
            ],
            "zarr_format": 2
        },
        "level_2.zarr/x/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "x"
            ]
        },
        "level_2.zarr/y/.zarray": {
            "chunks": [
                640
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                640
            ],
            "zarr_format": 2
        },
        "level_2.zarr/y/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "y"
            ]
        },
        "level_2.zarr/z/.zarray": {
            "chunks": [
                540
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                540
            ],
            "zarr_format": 2
        },
        "level_2.zarr/z/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z"
            ]
        },
        "level_3.zarr/.zattrs": {},
        "level_3.zarr/.zgroup": {
            "zarr_format": 2
        },
        "level_3.zarr/rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zarray": {
            "chunks": [
                64,
                64,
                64
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "zstd",
                "id": "blosc",
                "shuffle": 0
            },
            "dtype": "|u1",
            "fill_value": null,
            "filters": null,
            "order": "C",
            "shape": [
                270,
                320,
                320
            ],
            "zarr_format": 2
        },
        "level_3.zarr/rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z",
                "y",
                "x"
            ],
            "direction": [
                [
                    1.0,
                    0.0,
                    0.0
                ],
                [
                    0.0,
                    1.0,
                    0.0
                ],
                [
                    0.0,
                    0.0,
                    1.0
                ]
            ],
            "units": "\u03bcm"
        },
  [...]
        "rec20160318_191511_232p3_2cm_cont__4097im_1500ms_ML17keV_6/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z",
                "y",
                "x"
            ],
            "direction": [
                [
                    1.0,
                    0.0,
                    0.0
                ],
                [
                    0.0,
                    1.0,
                    0.0
                ],
                [
                    0.0,
                    0.0,
                    1.0
                ]
            ],
            "units": "\u03bcm"
        },
        "x/.zarray": {
            "chunks": [
                2560
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                2560
            ],
            "zarr_format": 2
        },
        "x/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "x"
            ]
        },
        "y/.zarray": {
            "chunks": [
                2560
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                2560
            ],
            "zarr_format": 2
        },
        "y/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "y"
            ]
        },
        "z/.zarray": {
            "chunks": [
                2160
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                2160
            ],
            "zarr_format": 2
        },
        "z/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "z"
            ]
        }
    },
    "zarr_consolidated_format": 1
}

Here _MULTISCALE_LEVELS prevents the need to hardcode the identifiers as suggested by @d-v-b @manzt , but it could be renamed to multiscale, etc. _ARRAY_DIMENSIONS is the key that Xarray uses in Zarr files to identify the dims.
This example is generated with itk, but it could also just as easily be generated with scikit-image, or dask-image via [1] (work in progress) or pyimagej.

Thanks for the link to that example @thewtex! Conforming with xarray.DataSet.to_zarr where possible seems reasonable to me too.
@constantinpape, @bogovicj, @axtimwalde might also be interested in weighing in.

👍 to flat vs hierarchical representation. Also 👍 to "multiscale".
I also like the constraint that the sub-datasets should be openable as zarr arrays by themselves. I think @thewtex's example satisfies this. Having said this, @thewtex, the xarray model looks too complex to me compared to @joshmoore's proposed spec. It would be great if it could be stripped down to its bare essentials. I agree that it's nice to have the pixel start coordinate handy, but it can also be computed after the fact, so it should be optional I think.
Last thing, which may be out of scope, but might not be: for visualisation, it is sometimes convenient to have the same array with different chunk sizes, e.g. orthogonal planes to all axes for a 3D image. I wonder if the same data/metadata layout standard can be used in these situations.
Oh and @joshmoore

anyone else who's GitHub account I've forgotten for the preliminary discussions

whose. Regret pinging me yet? =P

Great to see so much discussion on this proposal. I didn't have time to read through all of it yet, will try to catch up on the weekend.
Fyi, there is a pyramid storage format for n5 used by BigDataViewer and paintera already and I have used this format for large volume representations as well:
https://github.com/bigdataviewer/bigdataviewer-core/blob/master/BDV N5 format.md

Great to see this moving on!
In our projects xcube and xcube-viewer image pyramids look like so:
example.levels/
├── 0.zarr    # Full-sized array
├── 1.zarr    # Level-0 X&Y dimensions divided by 2^1
├── 2.zarr    # Level-0 X&Y dimensions divided by 2^2
├── 3.zarr    # Level-0 X&Y dimensions divided by 2^3
└── 4.zarr    # Etc.

As @joshmoore mentioned, also this goes without special metadata, because

To make pyramids discoverable, we simple use the file extension .levels.
Resolution decreases by factor 2^Level.
The number of levels is obvious from the entries in the .levels folder.
The level zero, can also be named 0.lnk. In this case it contains the path the original data rather then a copy of the "pyramidized" original dataset.

(See also the xcube level CLI tool that implements this.)
We are looking forward to adopt our code to any commonly agreed-on Zarr "standard".

All-
Here's a quick summary from my side of discussions up to this point. Please send corrections/additions as you see fit. ~Josh
Apparent agreement
Name
The name "multiscale" seems to be generally acceptable (https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595332383, https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595505162)
Multiple series
Support for multiple series per groups seems to be generally acceptable (e.g. https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595332383).
Special names
There are a few explicit votes for no special dataset names (e.g. https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595359246), but under "New ideas" there was one mention of group naming schemes.
Less clear
Layout
One primary decision point seems to be whether to use a deep or a flat layout:

Deep comments include https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595269311: "allows storing other types of data (segmentations) beside the base".
Flat comments typicall revolve around simplicity: https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595371288, https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595332383, https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595505162

Here I'd add that if flat is generally accepted as being the simplest approach for getting started, later revisions can always move to something more sophisticated. However, I'm pretty sure at that point we would want metadata not just at a single group level but either on multiple groups or all related datasets (or both).
Scaling information
Another key issue seems to be the scaling information. There are a range of ways that have been shown:

The fairly simple “scales”: [0.5, 0.5, 1, 1, 1], representation in the current revision (2) of this issue.
A transform submap with the keys "offset", "scale", and "units" (https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595359246).
An xarray representation with "direction" and "units" attributes (https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595432323).
The COSEM proposal from https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595359246 with gridSpacing and 'origin' metadata of the form {"gridSpacing": [r_x, r_y, r_z], "origin": [o_x, o_y, o_z]}
The BDV N5 format from https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595662150 with downsamplingFactors at two possible locations (see "Either/or" below).

@sofroniewn even asked if they are even useful as they stand (https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595332383).
To be honest, I punted on this issue knowing that it would be harder to find consensus on it. To my mind, this could even be a second though related extension proposal. My reasoning for that is that it can also be used to represent the relationship between non-multiscale arrays, along the lines of @jni's "multiple chunk sizes" question below, and in the case of BDV, the relationship between the individual timepoints, etc.
My first question then would be: to what extent can the current multiscale proposal be of value without the spatial/scale/transform information?
New ideas
Explicit "name" key
@d-v-b's New proposed COSEM style from https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595359246 uses this format:
        {"multiscale": [{"name": "base",  ...}, {"name" : "L1", ...}]}

Though this would prevent directly consuming the list (e.g. datasets = multiscale["series"][0]["datasets"]), it might provide a nice balance of extensibility, especially depending on the results of the coordinates/scales/transforms discussion.
Group naming
@forman showed an example from xcube in https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-596449313 in which group names were used rather than metadata to detect levels:
example.levels/

Links
@forman also showed in https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-596449313 one solution for linking: "The level zero, can also be named 0.lnk. In this case it contains the path the original data rather then a copy of the 'pyramidized' original dataset." This would likely need to be a pre-requisite proposal for this one if we were to follow that route. cc: @alimanfoo
Either/or logic
In @d-v-b's COSEM writeup from https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595359246, there is an example of either/or logic, where could would need to check in more than one location for a given piece of metadata:
 -     ├── (required) s1 (optional, unless "scales" is not a group level attribute): {"downsamplingFactors": [a, b, c]})

Multiple chunk sizes
@jni pondered in https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-595505162: "for visualisation, it is sometimes convenient to have the same array with different chunk sizes, e.g. orthogonal planes to all axes for a 3D image. I wonder if the same data/metadata layout standard can be used in these situations."

For the record, I'd currently err on the side of:

sticking with a flat "multiscale" object
without links or either/or logic
and without any special names,
while likely moving to the more flexible [{"name": "base"}] format
and saving coordinates for a follow-on proposal.

(whew) But opinions, as always, are very welcome.

Further CCs:  @saalfeldlab @axtimwalde @tpietzsch

My first question then would be: to what extent can the current multiscale proposal be of value without the spatial/scale/transform information?

I think there's value in the current effort, insofar as standardizing spatial metadata is a separable issue.
For a multiscale image spec, I would propose abstracting over the specific implementation of spatial metadata, e.g. by stipulating that the group multiscale attribute must contain the same spatial metadata as the collection of array attributes. This assumes as little as possible about the details of the spatial metadata; (but a key assumption I'm making is that duplicating this metadata is not prohibitive)

For the record, I'd currently err on the side of:
* sticking with a flat "multiscale" object
* without links or either/or logic
* and without any special names,
* while likely moving to the more flexible [{"name": "base"}] format
* and saving coordinates for a follow-on proposal.

These all look good to me!

@joshmoore outstanding summary! Thanks for leading this endeavor.

My first question then would be: to what extent can the current multiscale proposal be of value without the spatial/scale/transform information?

To correctly analyze or visualize the data as a multiscale image pyramid, then some spatial/scale/transform information is required.
To:

Compare with image subregions
Handle anisotropically sampled volumes
Compare with segmentations stored as meshes whose node positions are in "world space" or were generated from a derived volume sampled on a different pixel sampling grid.
Use model-based annotations defined in "world space"
Effectively utilize image registration

Spacing / scale and offset / origin and/or transforms are required. Without them, these use cases are either complex and error prone (requiring provenance and computation related to source pixel grids), or not possible. This is why the majority of scientific imaging file formats have at least spacing / scale and offset / origin in some form.
That said, the specs could still be split into two to keep things moving along.

Thanks so much to everyone who is putting detailed thought into this complex issue. Since the discussion has mostly focused on the bioimaging side of things, I'll try to add the xarray & geospatial perspective.

The main precedent for "multiscale arrays" in geospatial comes from the GeoTIFF / COG format. In geospatial lingo, they are called "overviews". GDAL has good documentation on this.
There has been some discussion in xarray about supporting overview (see https://github.com/pydata/xarray/issues/3269), but it is not currently part of our data model, which is derived from the common data model and tied closely to netCDF.
However, xarray does have some a very convenient utility for generating overviews: the coarsen method.
For climate model data, generating overviews is not trivial because the cell geometry can be non-euclidean. You need to know an area weighting factor to apply when coarse graining. It's not clear to me from the discussion above whether zarr needs to know how to actually generate these overviews, or of that is up to a third-party library.
Nevertheless, given the proliferation of high resolution weather and climate models, the ability to store overviews would be quite valuable, particularly for interactive visualization. For broader adoption, this concept would need to make its way into the NetCDF standard itself.
The bigger cells / pixels get, the more important become coordinates and cell bounds. It seems like this conversation is closely tied to the question of how to represent coordinates in zarr. As noted by @thewtex, we have already established some de-facto standards about how to do this in order to plug zarr into xarray. So these discussions need to happen in parallel.

Great discussion. These are my $0.02. Largely, I agree with @joshmoore's summary in https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-596687408. Being able to open each scale level as an individual data set and not part of a pyramid is probably the most important feature and should be part of any standard the comes out of this. With this in mind, the spatial meta data (gridSpacing and origin) would need to be stored in the attributes of the individual datasets. This means either

duplication of the spatial meta data, or
no spatial meta data of the individual datasets in the multiscale group.

This also does not consider other spatial meta data like rotations. As far as I know, this is a relevant use case for @tpietzsch. If such (arbitrary) transforms should not be considered in the standard, then the question arises of how to combine this with the gridSpacing and origin. In such a scenario, I would probably set the origin to zero with appropriate shifts in downscaled levels as needed, and have the actual offset after the rotation in a global transform. But then again, each scale dataset could not be loaded individually with the correct scaling, rotation, and offset, without explicit knowledge of the pyramid.
Other than that, here are a few comments:

I think the bare minimum spatial information is going to be the gridSpacing and origin for each scale level. I do not have a strong opinion about nomenclature. In Paintera, it is resolution and offset, but I am ok with anything reasonable.
If scales are defined, they should be fully specified for all of the spatial dimensions, i.e. for 3D or  3D+channel, it would be [[sx, sy, sz], ...].I like having the scales attribute but the scales can be inferred from gridSpacing, so it is redundant information.
I prefer the format that @d-v-b proposed that stores an array of dictionaries for the datasets, e.g.

[{"name": "s0", "meta1": ...}, {"name": "s1", "meta1": ...}]

over storing multiple arrays like
{"datasets": ["s0", "s1", ...], "meta1": [...]}

I do like the idea of having multiple multi-scale groups within a group and specifying scale levels at arbitrary paths (relative to the group). I had not thought of that before but it sounds very intriguing. On caveat here is that it may get out of control and result in very chaotic dataset hierarchies but that would be the responsibility of the user. I am not aware of any good restriction, yet. Considering this extension, maybe using "path" as a key instead of "name" in @d-v-b's proposal may be more descriptive and appropriate.

I think that a common standard would be a great thing to have and help interaction between the wealth of tools that we are looking at. Paintera does not have a great standard and should update its format if a reasonable standard comes out of this (while maintaining backwards compatibility).
Disclaimer: I will start a position outside academia soon and will not be involved in developing tools in this realm after that. My comment should be regarded as food for thought and to raise concerns that may not have been considered yet. Ultimately, I will not be involved in the decision making of any specifics of this standard.
cc @igorpisarev

Apologies, all, for letting this slip into April. Hopefully everyone's managing these times well enough despite the burden of long spec threads.
I've updated the description to include the new {"name": ...} syntax and added a new deadline of April 15th for further responses.
A few points on the more recent comments:

In https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-599782137, @hanslovsky suggested "path" rather than "name". I'm on board and will make the change if there are no vetoes, but in the documentation for the metadata (when it appears) we will need to specify whether or not super- and sub-paths are allowed (i.e. ".." and "/").

Then the general topic of the spatial metdata. Both https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-596719361 and https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-596701736 give a :thumbsup: to splitting it out into a separate proposal. It sounds like https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-599782137 is proposing to have origin and gridSpacing (and not scale) on the datasets themselves rather than the group. If there were agreement on that, I'd omit scale from this proposal and hold off for the next. @d-v-b may be the main opponent of that where in https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-599782137 there's a clear call for duplicating the metadata when/if possible. My major concern with duplication would be keeping the two representations consistent.

As an aside on the geospatial front, https://gis.stackexchange.com/a/255847 helped me understand the GeoTIFF overviews. I don't see anything contradictory.

@rabernat brings up in https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-598456145 non-Euclidean geometries which were also discussed in a recent Zarr call. I'm all for saving that for the follow-up discussion, since it's likely going to be a big one. I'd tend to err on the side of having that external, though perhaps storing (non-standardized?) provenance metadata if possible.

Otherwise, it sounds like the newer comments are generally onboard with the current proposal, but let me know if I've dropped anyone's concerns.

I like path much more than name. +1 to that.

My major concern with duplication would be keeping the two representations consistent.

This is a valid concern. Personally I don't like duplicating spatial metadata in the group -- my original conception a long time ago was for the group multiscale metadata to simply list the names/paths to the datasets that comprise the pyramid, with no additional information. But I was reminded by @axtimwalde that accessing metadata from multiple files on cloud stores can be bothersome, and this led to the idea of consolidating the array metadata at the group level. Maybe this can be addressed via the consolidated metadata functionality that has already been added to zarr: https://zarr.readthedocs.io/en/latest/tutorial.html#consolidating-metadata.
For a spec, a way to resolve this could be to specify that, for dataset entry in the group multiscale metadata, a path field is required but additional fields per dataset are optional. In this regime, programs that attempt to parse the multiscale group may look for consolidated metadata in the group attributes, but they should have a fallback routine that involves parsing the individual attributes of the datasets.

What would we do if cloud storage wouldn't have high latency?  I am similarly worried about the consolidated meta-data hack because we may store a lot of meta-data and parsing very long JSON texts isn't particularly fast either, it also doesn't scale very well.

NB: Updated description to use "path".

https://github.com/zarr-developers/zarr-specs/issues/50#issuecomment-607947712

I had never considered a level of consolidation between none and everything, e.g. all arrays (but not groups) within a group are cached within the group metadata. It's an interesting idea, but discussing it here seems dangerous.
If we assume that consolidation is out-of-scope for this issue, I think the only question remaining is if we want optional spatial metadata at the group level, where the array metadata would take precedence. Here, I'd likely also vote for being conservative and not doing that at this point, though we could add it in the future (more easily than we could remove it).
If all agree, I'll add hopefully one last update to remove all mention of "scale" and then start collecting all the spatial ideas that we've tabled in this issue into a new one.

Description updated removing use of "scale" and clarifying a few items like the ordering of the datasets which have come up recently during conversations on image.sc, twitter, etc. Thanks again to everyone for the feedback.

This issue has been migrated to image.sc after the 2020-05-06 community discussion and will be closed. Authors are still encouraged to make use of the specification in their own libraries. As the v3 extension mechanism matures, the specification will be updated and registered as appropriate. Many thanks to everyone who has participated to date. Further feedback and request changes are welcome either on this repository or on image.sc.


Use Module Augmentation

This is to keep updated with the changes from Purify without having to reinvent the wheel. This will be populated with issues that need to be closed for this changed to be marked completed and published.

[x] Migrate all ADTs to use purify ADTs
[x] Expose new pointfree methods
[x] Ensure all tests are passing
[x] Add new tests where necessary
[x] close #41 as resolved
[x] Drop use of ts-toolbelt

Merged #45 ❤


NoMethodError (undefined method `update_subscription' for #Stripe::Customer:0x00007fda314b0f88

After successfully upgrading the plan. I receive the "update_subscription" method error. Does anyone know why this is happenining?

This answered the question for me:
https://stackoverflow.com/questions/64199340/nomethoderror-undefined-method-update-subscription-for-stripecustomer

The new checkout and subscription system for Stripe - has totally changed the system in a way that brought an end to the lifecycle of Koudoku -- in its current state.


Store error messages as part of the deployment

When a deployment fails, we should store error messages along-side the status inside the deployment

Do you want to store the deployment at all if the expansion fails? I was just surprised by:
$ dm deploy templates/spark/v1/spark.yaml
2015/11/19 16:45:55 cannot deploy configuration named spark.yaml: status code: 400 status: 400 Bad Request : cannot expand template named spark.yaml (template expansion: expandybird response:
error expanding template: error expanding template spark.yaml: Traceback (most recent call last):
  File "./expansion/expansion.py", line 372, in <module>
    main()
  File "./expansion/expansion.py", line 369, in main
    print Expand(template, imports, env=env, validate_schema=validate_schema)
  File "./expansion/expansion.py", line 57, in Expand
    raise ExpansionError('config', str(e))
__main__.ExpansionError: unexpected '}' Resource: config
):
imports:
- path: spark.jinja

resources:
- name: spark
  type: spark.jinja

zml@zml:~/deployment-manager$ dm deploy templates/spark/v1/spark.yaml
2015/11/19 16:49:12 cannot deploy configuration named spark.yaml: status code: 400 status: 400 Bad Request : Deployment spark.yaml already exists

Yeah, so the thinking was that the flow would be deploy, if it fails, you'd look at the errors and update. However, if it seems like it's more natural to fail fast and not even create a deployment, I can change the semantics. Unless somebody complains loudly about this, I can make the change.

That change will be hard to keep around once we move to an asynchronous API.
On Mon, Nov 23, 2015 at 11:46 AM, vaikas-google<EMAIL_ADDRESS>wrote:

Yeah, so the thinking was that the flow would be deploy, if it fails,
you'd look at the errors and update. However, if it seems like it's more
natural to fail fast and not even create a deployment, I can change the
semantics. Unless somebody complains loudly about this, I can make the
change.
—
Reply to this email directly or view it on GitHub
https://github.com/kubernetes/deployment-manager/issues/79#issuecomment-159042302
.


serCustom method not run outside database

Version: orientdb-community-2.1.9
bug description: when using the graph database. ie schema changes are automatically run outside the current transaction. But in the setCustom method (both on vertex as well as property) that is not the case. it is throwing exception saying you cant make schema changes during a transaction.
The rest of the graph schema changes like creating class, adding property does not have a problem as it is automatically run outside a transaction.
To recreate:
OrientGraphFactory factory = new OrientGraphFactory("plocal:"+, , ).setupPool(1, 1);
graph=factory.getTx();
OrientVertexType typedef=graph.getVertexType(param.File.name());
prop = def.createProperty("attribute name", OType.LINK, OType.STRING);
prop.setCustom("someparam","param value");

Workaround:
prop=graph.executeOutsideTx(new OCallable<OProperty, OrientBaseGraph>() {
     @Override
     public OProperty call(final OrientBaseGraph g) {
          OProperty prop=def.getProperty("attrname");
          return prop.setCustom("someparam","param value");
     }
}, "");

Awesome, that was quick!

Hi @smallik
the fix is easy, but it does not only involve setCustom(), I could reproduce the same problem with setMax(), setMin() and all the other constraints
I'm working to a fix
Thanks
Luigi

Hi @smallik
I pushed a fix on 2.1.x branch, together with some test cases.
The fix will be released next week with 2.1.11
I'm closing this issue, please reopen in case you see any problems
Thanks
Luigi


Fixes test being dependent on AAS order

Signed-off-by: Frank Schnicke<EMAIL_ADDRESS>

approved


Add explicit "translating" bit to translations

This would allow leaders to explicitly state that they are not translating some tasks. Also allow staff to do the same on leaders' behalf as they are able to freeze leaders' translations already.
Depends on #25

Can we rebase this?

Can we rebase this?

Rebased.

Rebased.

Added screenshots to comment #1.

Added screenshots to comment #1.


Upload Error

The video I tried to upload is successfully uploading but onComplete event i need to send email after successful uploading but unfortunately I am getting following error. Because of this error I can't get uploaded video url etc
PATCH https://api.vimeo.com/videos/190425955 403 (Forbidden)

Well it was my mistake..I have not given upload access to my vimeo client ..

Hi @JavedBoqo, hai have the same problema but i don't understand why. I already use a token with all scopes but nothing. Always the same error. Can you help me to understand?  Thanks


gitlab fixes

properly detect apiURL for watcher when it's not set in the cr Repository setting.
Fix reporting comment which hasn't  been working for a while 😢

Fixes #989
Related to #990 (still need e2e)
Signed-off-by: Chmouel Boudjnah<EMAIL_ADDRESS>
Changes

Submitter Checklist

[ ] ♽  Run make test lint before submitting a PR (ie: with pre-commit, no need to waste CPU cycle on CI
[ ] 📖 If you are adding a user facing feature or make a change of the behavior, please verify that you have documented it
[ ] 🧪 100% coverage is not a target but most of the time we would rather have a unit test if you make a code change.
[ ] 🎁 If that's something that is possible to do please ensure to check if we can add a e2e test.
[ ] 🔎 If there is a flakiness in the CI tests then don't necessary ignore it, better get the flakyness fixed before merging or if that's not possible there is a good reason to bypass it. (token rate limitation may be a good reason to skip).

not sure what's goign on with github e2e 🤷🏻


Shoulder offset corrections ready to merge...

Hello Mpstark,
first of all thanks for this great add-on. I really love it and in fact I loved it so much that I made some improvements for myself that I think have now reached a level of maturity allowing them to be merged into the real project. Here is what I did:
As you may have noticed, the value of test_cameraOverShoulder has a different effect on the camera's position -- relative to the player's center -- based on what model (race, gender, mount, vehicle) is currently active. I added the CorrectShoulderOffset() function that determines the current model and outputs a factor by which the current test_cameraOverShoulder value needs to be multiplied such that the camera stays exactly where it is. I have not even started to enter these factors for all models, because I do not have all mounts. But this is something we could do in the future and also ask people of the community to make out these factors for us. I would create a youtube-video explaining how to do this. But for now, you could already have a look at my code and tell me what you think.
Another thing I added, is the GetShoulderOffsetZoomFactor() function. This is for gradually reducing the Camera Shoulder Offset as you zoom-in (mouse wheel) on your character. Otherwise a greater shoulder offset may be awkward while walking through narrow corridors. With this option enabled you just have to zoom in a little more on your character while walking through such corridors and it will be more pleasant as the shoulder offset is reduced.
For both these features I added the checkboxes "Correct Shoulder Offset" and "Shoulder Offset Zoom-In" to the Default Camera Settings of DynamicCam so people can enable and disable them.
I think the code as it is now is already a great improvement for all people loving shoulder offset like myself. And maybe even some more will start to like it now.
Would be great to hear back from you!

I have moved my work fixing the shoulder offset to an addon of its own:
https://github.com/LudiusMaximus/CameraOverShoulderFix
It works as a stand-alone but also integrates into DynamicCam when both are installed.
I will publish it soon when I have time to record a tutorial video.


Issue with soundness of HLIL control flow structuring

HLIL can produce unsound control flow structuring in some conditions.
Consider this MLIL code:
orig_state_exec.zip
In MLIL everything looks correct:

Consider the case when 'i = 0the path meansiis assigned to5and then ultimately goes to instruction21`
Now in HLIL:

The control flow is a series of if statements rather than if-else statements. In the case of i == 0 it meets the first condition and sets i = 5 and then can also satisfy the second condition too incorrectly setting var_20 = 1
Special Thanks to: Zao Yang and Stefan Nagy for their research in Decompiler Fuzzing for reporting this issue.

This is exposing a fundamental issue in control flow restructuring. We have desire to revisit the control flow restructuring algorithm to significantly reduce the number of conditions that get out of hand and become unreadable, which is also an issue with the current algorithm. This bug will probably have to be fixed alongside a refactor of the control flow restructuring itself.

A user reported a similar case:

BN version: 4.0.4958 Windows
Compiler: MSVC 2023 x64

Compile the following C code using cl /c /O2 test.c and disassemble the resulting binary in Binary Ninja:
bool test(int *input)
{
    bool result;
    if(input[0] == 1) {
        result = true;
        input[0] = 2;
        if(input[1] == 1)
            goto other;
    } else {
other:
        result = false;
    }
    return result;
}

This results in the following High Level IL output:
 int64_t test(int32_t* arg1) {
1    int64_t rax;
2    if (*arg1 == 1)
3        rax = 1;
4        *arg1 = 2;
5    if (*arg1 != 1 || (*arg1 == 1 && arg1[1] == 1))
6        rax.b = 0;
7    return rax;
 }

However, this output is not actually correct: if the function's input is { 1, 0 }, at line 4 in the HLIL, arg1[0] will be set to 2. Then, at line 5, the condition in the if statement evaluates to true (since arg1[0] now evaluates to 2), which in turn leads to line 6 being executed, and the whole function returning false. This differs from the actual behavior of the program (which should be returning true).
In contrast, the Hex-Rays Decompiler generates the following output, which is correctly in correspondence to the actual semantics of the program:
char __fastcall test(int *a1)
{
  char result; // al

  if ( *a1 != 1 || (result = 1, *a1 = 2, a1[1] == 1) )
    result = 0;
  return result;
}

I see the fundamental issue is the body of the if statement is modifying the variables used in the condition, so it alters the result

Fixed in 4.1.5547


Migrate to senzing/stream-producer

Is your feature request related to a problem? Please describe.
senzing/mock-data-generator has been deprecated. Use  senzing/stream-producer instead.
See https://github.com/Senzing/stream-producer/issues/3

Reference: https://github.com/Senzing/mock-data-generator/issues/44


In Mapbox Studio, when adding new layer, can't browse to network drives on Windows

When looking for source files to add as new layers, the browse window only allows you to go to the C root, not to other drives.

Accessing other drives is in the works and should be available with the next release.
In the meantime you could try to mount that drive into a folder on C:, (Technet: Assign a mount point folder path to a drive)

Upcoming release will allow to browse to other drives.


Where is rookPositions() defined?

I was porting some of the LDS distributions to use them in a Java project. The NRooks() method references a rookPositions() method however it doesn't seem to be included in the source code?

Hey there!
Is this the code you are talking about?
src\families_2d\samples\lds\LDS.cpp line 111:
`void _2d::Samples::LDS::NRooks(std::vector& values, size_t numValues, std::mt19937& rng)
{
// make N rooks that can't see each other on x or y axis
std::vector<size_t> rookPositions(numValues, 0);
for (size_t i = 0; i < numValues; ++i)
rookPositions[i] = i;
std::shuffle(rookPositions.begin(), rookPositions.end(), rng);
float offset = 1.0f / float(numValues * 2);

// convert them to sampling locations
values.resize(numValues);
for (size_t i = 0; i < numValues; ++i)
{
    values[i][0] = offset + float(rookPositions[i]) / float(numValues);
    values[i][1] = offset + float(i) / float(numValues);
}

}`
rookPositions is the std::vector<size_t> defined at the top of the function, and is made by making an array with values 0, 1, 2, ... numValues and then shuffling them.

Oh right, it's a vector initialisation🤦.

ha, no problem. I apologize for this being vapor ware. If someone followed through on this kind of thing, it could be quite useful.
Real cool you are porting some to a java project. What does your program do?

I'll be adding a point distribution class to this geometry project. The idea is point sets (having different characteristics) can serve as input into other methods -- delaunay triangulation and the like.


WFLY-4305: exit immediately on non-zero status

Rebase of #7137

Windows Build 969 is now running using a merge of<PHONE_NUMBER>3a80f36499fa4893a76295f0d8eecb

Linux with security manager Build 845 is now running using a merge of<PHONE_NUMBER>3a80f36499fa4893a76295f0d8eecb

Linux Build 5860 is now running using a merge of<PHONE_NUMBER>3a80f36499fa4893a76295f0d8eecb

Linux with security manager Build 845 outcome was SUCCESS using a merge of<PHONE_NUMBER>3a80f36499fa4893a76295f0d8eecb
Summary: Tests passed: 839, ignored: 316 Build time: 0:07:00

Windows Build 969 outcome was SUCCESS using a merge of<PHONE_NUMBER>3a80f36499fa4893a76295f0d8eecb
Summary: Tests passed: 2924, ignored: 312 Build time: 0:49:20

Linux Build 5860 outcome was SUCCESS using a merge of<PHONE_NUMBER>3a80f36499fa4893a76295f0d8eecb
Summary: Tests passed: 2924, ignored: 312 Build time: 0:50:51


Overriding of Normative reference boilerplate doesn't work

[bibliography]
== Normative references

The following contract document is referred to in the text in such a
way that some or all of its content constitutes requirements of this
document.

Still shows this:

The [.boilerplate] override of the default text in terms sections has not been extended to normative references.

Cannot use the same strategy as for terms & definitions boilerplate: cannot insert a hanging clause before references, since the references will end up within that hanging clause, and the bibliography status of the parent clause will not inherit into the bibliography list.
Instead, will need to make this a boilerplate Note, which is stripped of its note wrapper in postprocessing.

Realised as:
[bibliography]
== Normative references

[NOTE,type=boilerplate]
--
The following contract document is referred to in the text in such a
way that some or all of its content constitutes requirements of this
document.
--


Gamepads don't work on Android (at least on Pixel4)

On Pixel4, an unexpected device uinput-fpc is recognized as a gamepad besides a connected gamepad. I'm not sure what it is, but we should filter this out.
CC @SolarLune

It looks like this is a fingerprint censor? https://github.com/libgdx/libgdx/issues/5596

Hi @SolarLune, does the latest Ebiten 36209465fc5114d9e019ae09b6a0bf25a7ea3f21 on master or 1a2b308ae33914f93a7d9c5e67a7ef498a6ae3ad on 2.0 work?

Friendly ping @SolarLune


Fix registation image to svg

#519  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @thisisgoldman and the rest of your teammates on  Graphite


ascending/descending keywords not suggested for partially written ascending/descending keywords

Description:
Consider
    string[][] usages = from var [_, usage] in data.entries()
        order by usage.employeeId a<cursor>

At cursor, it doesn't suggest ascending keyword.
Steps to reproduce:
See description
Affected Versions:
2201.0.3 RC2

Related to #35774

The parser has added the missing select keyword as a recovery strategy. Therefore the effective context for the completions become FunctionBodyBlockNodeContext. Therefore the direction keywords are not suggested at the cursor position.


Running custom datasets

I'm trying to train the fcn8s model on a dataset but get the following runtime error:
RuntimeError: shape '[1, 573, 447, 1]' is invalid for input of size 768393
Solution: I used mask images with the RGB format. Converted it to grayscale with range [0,11] (12 classes)

Hi, I come across this problem on CamVid dataset. Would you please be more specific about how to solve this problem. Thank you.

这是来自QQ邮箱的自动回复邮件。嗨，这里是吴颖的信箱。谢谢你的来信，我将会尽快给你回复。


slider with links doesn't work on mobile browsers if the slider is not wide enough

I've use lightslider with images and links in each li element:

james craven

if lightslider does not contain enough images to fill the slider (arrows shown) the link href does not work.
The problem is only present when using the slider on mobile device using a mobile browser like Opera.
I believe the problem sits in line 839:
changing the return value to true make things work as expected:
$slide.on('touchend', function () {
                        if (w < elSize) {
                            if (w !== 0) {
                                return true;
                            }
                        }
                        var distance;
                        if (settings.vertical === true) {
                            distance = endCoords.pageY - startCoords.pageY;
                        } else {
                            distance = endCoords.pageX - startCoords.pageX;
                        }
                        $this.touchEnd(distance);
                    });

@Leftyx
Thank you so much

@Leftyx
I can't thank you enough for this one!
note to self: always google your issues before trying to fix them


Is rq.schedule() not working?

Is rq.schedule(...) not implemented or not working? I try as in doc, but nothing happen. But rq.queue(...) working fine.
@rq.job
def add(x, y):
    return x + y

# queue job at a certain datetime (UTC!) from other function.
add.schedule(datetime(2016, 12, 31, 23, 59, 59), 1, 2)

@hveda Are you running the scheduler to actually put the job into the queue at the scheduled time?
Use the scheduler CLI command, which is a wrapper for the rqscheduler command from rq-scheduler.

Yes, so i just use rqscheduler command to do that?
Am i doing it right?
My worker docker-compose.yml something like:
  worker
    build: .
    command: python -m flask rq worker scheduler
    environment:
      - FLASK_APP=serve.py

My init file:
    rq.init_app(app)
    scheduler = rq.get_scheduler(interval=10)
    scheduler.run()

I solve the problem as you said but i must run scheduler container itself. Additional docker config i use:
scheduler:
    build: .
    command: python -m flask rq scheduler
    environment:
      - FLASK_APP=serve.py
    volumes:
      - .:/var/www/scheduler

And access from function:
@rq.job
def add(x, y):
    return x + y
# Queue job scheduled
scheduler = rq.get_scheduler(interval=10)
schedule(datetime(2016, 12, 31, 23, 59, 59), add, 1,  2)


next/head title should support translations

Bug report
Describe the bug
When  contains a react element, title is correctly shown in SSR code, but error is thrown on client. This would be very useful for some l10n components, which just return translated string.
import Head from 'next/head';
const Hello = () => "Hello";
...
<Head>
    <title><Hello/></title>
</Head>

Client throws:
Uncaught (in promise) TypeError: children.join is not a function
    at HeadManager.updateTitle (head-manager.js:68)
    at HeadManager.doUpdateHead (head-manager.js:55)
    at head-manager.js:41

Expected behavior
I would expect the client to show the same as ssr.
Discussion
Could we add here head-manager.js#L45 something like third option - to render a react element? I am not sure, if context,etc is ready.
System information

Version of Next.js: 7.0.2

<title> already supports translations if you integrate with next-18next
Ex:
import React from "react";
import Head from "next/head";

import { i18n, withNamespaces } from "../i18n";

const Homepage = ({ t }) => (
  <React.Fragment>
    <Head>
      <title>{t("title")}</title>
    </Head>
    <button
      onClick={() => i18n.changeLanguage(i18n.language === "en" ? "de" : "en")}
    >
      {t("change-locale")}
    </button>
  </React.Fragment>
);

export default withNamespaces("common")(Homepage);

In other words, rather than using a component, use a function that will return a string.

Thanks @ztanner, great answer 👍
The reason you can't use components is that we render the exact children of <Head> into the document's <head> tag.

@timneutkens , how should we act in case we are using <Trans> component, provided by i18next?
Plain Title Text goes here.
Will this force to use locales files like `common.json`?


Updated Readme

Fixed typo in Readme:
Changed "before hand" to "beforehand"

Closing as CLA has not been signed.


Please fix typo (encounted) in templates

Version
What package version of the SDK are you using. master
Describe the bug
The string encounted appears in the template, should be encountered.
To Reproduce
Steps to reproduce the behavior:

Search the repo for the string encounted.
https://github.com/microsoft/botbuilder-python/search?q=encounted&unscoped_q=encounted
Error: I currently get three hits, all in the templates.

Expected behavior
We should eliminate typos in user-facing code, especially in templates.
Additional context
re: BotBuilder-Samples PR 2025
[bug]

Corrected in Samples staging branch:
https://github.com/microsoft/BotBuilder-Samples/tree/trboehrer-move-samples


Investigate why orbits are behaving unexpectedly

Tried a toy orbital simulation and was unable to get a body moving in a circular or near-circular orbit.

Noticed that orbits are not as expected, having more than one perigee/apogee per orbit.
Noticed that in the first computed step of a scenario an error is introduced when the initial velocity vector doesn't take into account that the end of the first step is not on the tangent to the starting position of the first step.
Noticed that orbiting bodies are gradually accelerating to higher minimum and maximum velocities each orbit.

Definitely need to change to have much finer granularity calculation steps (multiple calculation steps per render step).

Orbit scenario I had been testing was not working as expected because acceleration limits were always being applied every frame in every scenario. Turned them off for the orbit scenario and got expected orbits.
Still need to need to change to have much finer granularity calculation steps (multiple calculation steps per render step).

Calculation precision added in Issue #3 implementation has made orbit in toy orbit scenario smooth and much more stable at 1000 calculations/frame.


build: Make the build more friendly.

Added a new "make help" target which shows:

overview.
available targets.
variables that users may wish to override.

Also, display a summary for a normal build showing the final destination
for the runtime binary and the configuration file.
Signed-off-by: James O. D. Hunt<EMAIL_ADDRESS>

Coverage decreased (-7.8%) to 33.744% when pulling fa554ea206e535cf56d2fa9d8935f3d4db869479 on jodh-intel:make-build-more-friendly into cba99227eab84f772bdd34ffaba931765c401ace on clearcontainers:master.

Coverage remained the same at 41.516% when pulling ad9a7f3c068f29766b3276b82c2d3adcddb18fb4 on jodh-intel:make-build-more-friendly into cba99227eab84f772bdd34ffaba931765c401ace on clearcontainers:master.

Coverage remained the same at 41.516% when pulling 02009811f0f1a5a58451c992f85112ae9822e1bd on jodh-intel:make-build-more-friendly into cba99227eab84f772bdd34ffaba931765c401ace on clearcontainers:master.

LGTM

lgtm


Can now use python2 or python3 interchangeably.

Guys, I would like to thank you. I, like many other devs, needed this and will make extensive use. Cheers!


refactor: Remove iNetwork functions From Toolkit

Removed iNetwork Functions from Toolkit:

docs
code
tests

Added test/functional/iNetworkFunctional.js
Part of #161

$ npm run test-integration

> ./node_modules/mocha/bin/mocha --timeout 3s test/functional/*.js

  CommandCall Functional Tests
    CL command tests
      ✓ calls CL command using idb transport (290ms)
      ✓ calls CL command using rest transport (197ms)
      ✓ calls CL command using ssh transport (851ms)
      ✓ calls CL command using odbc transport (227ms)
    SH command tests
      ✓ calls PASE shell command using idb transport (1372ms)
      ✓ calls PASE shell command using rest transport (1369ms)
      ✓ calls PASE shell command using ssh transport (1884ms)
      ✓ calls PASE shell command using odbc transport (1422ms)
    QSH command tests
      ✓ calls QSH command using idb transport (1765ms)
      ✓ calls QSH command using rest transport (1669ms)
      ✓ calls QSH command using ssh transport (2085ms)
      ✓ calls QSH command using odbc transport (1622ms)

  ProgramCall Functional Tests
    Test ProgramCall()
      ✓ calls QWCRSVAL program checks if it ran successfully using idb transport (72ms)
      ✓ calls QWCRSVAL program checks if it ran successfully using rest transport (67ms)
      ✓ calls QWCRSVAL program checks if it ran successfully using ssh transport (653ms)
      ✓ calls QWCRSVAL program checks if it ran successfully using odbc transport (82ms)
    Test ProgramCall()
      ✓ calls QWCRSVAL program and returns arbitrarily named parameter using idb transport (78ms)
      ✓ calls QWCRSVAL program and returns arbitrarily named parameter using rest transport (74ms)
      ✓ calls QWCRSVAL program and returns arbitrarily named parameter using ssh transport (579ms)
      ✓ calls QWCRSVAL program and returns arbitrarily named parameter using odbc transport (79ms)
    Test ProgramCall()
      - Should be successful with addReturn arbitrary attribute specified using using idb transport
      - Should be successful with addReturn arbitrary attribute specified using using rest transport
      - Should be successful with addReturn arbitrary attribute specified using using ssh transport
      - Should be successful with addReturn arbitrary attribute specified using using odbc transport

  Toolkit Functional Tests
    DataQueue Functional Tests
      sendToDataQueue
        ✓ sends data to specified DQ using idb transport (78ms)
        ✓ sends data to specified DQ using rest transport (62ms)
        ✓ sends data to specified DQ using ssh transport (605ms)
        ✓ sends data to specified DQ using odbc transport (79ms)
      receiveFromDataQueue
        ✓ receives data from specfied DQ using idb transport (76ms)
        ✓ receives data from specfied DQ using rest transport (75ms)
        ✓ receives data from specfied DQ using ssh transport (634ms)
        ✓ receives data from specfied DQ using odbc transport (80ms)
      clearDataQueue
        ✓ clears the specifed DQ using idb transport (69ms)
        ✓ clears the specifed DQ using rest transport (82ms)
        ✓ clears the specifed DQ using ssh transport (593ms)
        ✓ clears the specifed DQ using odbc transport (78ms)
    retrUsrAuth
      ✓ returns uses's authority for an object using idb tranport (88ms)
      ✓ returns uses's authority for an object using rest tranport (86ms)
      ✓ returns uses's authority for an object using ssh tranport (621ms)
      ✓ returns uses's authority for an object using odbc tranport (93ms)
    rtrCmdInfo
      ✓ returns command info using idb transport (73ms)
      ✓ returns command info using rest transport (83ms)
      ✓ returns command info using ssh transport (586ms)
      ✓ returns command info using odbc transport (86ms)
    retrPgmInfo
      ✓ returns program info using idb transport (79ms)
      ✓ returns program info using rest transport (84ms)
      ✓ returns program info using ssh transport (692ms)
      ✓ returns program info using odbc transport (91ms)
    retrSrvPgmInfo
      ✓ returns service program info using idb transport (74ms)
      ✓ returns service program info using rest transport (76ms)
      ✓ returns service program info using ssh transport (588ms)
      ✓ returns service program info using odbc transport (85ms)
    retrUserInfo
      ✓ returns specified user profile info using idb transport (78ms)
      ✓ returns specified user profile info using rest transport (71ms)
      ✓ returns specified user profile info using ssh transport (634ms)
      ✓ returns specified user profile info using odbc transport (92ms)
    retrUsrAuthToObj
      ✓ retrieves info for users who are authorized to an object using idb transpsort (68ms)
      ✓ retrieves info for users who are authorized to an object using rest transpsort (79ms)
      ✓ retrieves info for users who are authorized to an object using ssh transpsort (622ms)
      ✓ retrieves info for users who are authorized to an object using odbc transpsort (93ms)
    addToLibraryList
      ✓ appends lib to user's lib list using idb transport (77ms)
      ✓ appends lib to user's lib list using rest transport (63ms)
      ✓ appends lib to user's lib list using ssh transport (677ms)
      ✓ appends lib to user's lib list using odbc transport (81ms)
    getPTFInfo
      ✓ returns info for specified ptf using idb transport (80ms)
      ✓ returns info for specified ptf using rest transport (76ms)
      ✓ returns info for specified ptf using ssh transport (661ms)
      ✓ returns info for specified ptf using odbc transport (97ms)
    getProductInfo
      ✓ returns info for specified product using idb transport (71ms)
      ✓ returns info for specified product using rest transport (68ms)
      ✓ returns info for specified product using ssh transport (622ms)
      ✓ returns info for specified product using odbc transport (95ms)
    getInstalledProducts
      ✓ returns info for installed products using idb transport (828ms)
      ✓ returns info for installed products using rest transport (832ms)
      ✓ returns info for installed products using ssh transport (1477ms)
      ✓ returns info for installed products using odbc transport (826ms)
    UserSpace Functional Tests
      createUserSpace
        ✓ creates a user space using idb transport (74ms)
        ✓ creates a user space using rest transport (156ms)
        ✓ creates a user space using ssh transport (676ms)
        ✓ creates a user space using odbc transport (118ms)
      setUserSpaceData
        ✓ sets data within the user space using idb transport (66ms)
        ✓ sets data within the user space using rest transport (73ms)
        ✓ sets data within the user space using ssh transport (640ms)
        ✓ sets data within the user space using odbc transport (82ms)
      getUserSpaceData
        ✓ returns specified length of data using idb transport (75ms)
        ✓ returns specified length of data using rest transport (63ms)
        ✓ returns specified length of data using ssh transport (636ms)
        ✓ returns specified length of data using odbc transport (83ms)
      deleteUserSpace
        ✓ removes a user space using idb transport (70ms)
        ✓ removes a user space using rest transport (71ms)
        ✓ removes a user space using ssh transport (636ms)
        ✓ removes a user space using odbc transport (81ms)
    getSysValue
      ✓ returns the value of system variable using idb transport (78ms)
      ✓ returns the value of system variable using rest transport (70ms)
      ✓ returns the value of system variable using ssh transport (599ms)
      ✓ returns the value of system variable using odbc transport (81ms)
    getSysStatus
      ✓ returns basic system status information about the signed-on users and batch jobs using idb transport (70ms)
      ✓ returns basic system status information about the signed-on users and batch jobs using rest transport (68ms)
      ✓ returns basic system status information about the signed-on users and batch jobs using ssh transport (669ms)
      ✓ returns basic system status information about the signed-on users and batch jobs using odbc transport (106ms)
    getSysStatusExt
      ✓ returns more detailed system status info using idb transport (1166ms)
      ✓ returns more detailed system status info using rest transport (1127ms)
      ✓ returns more detailed system status info using ssh transport (1725ms)
      ✓ returns more detailed system status info using odbc transport (1175ms)
    getJobStatus
      ✓ returns status of specified job using idb transport (65ms)
      ✓ returns status of specified job using rest transport (72ms)
      ✓ returns status of specified job using ssh transport (598ms)
      ✓ returns status of specified job using odbc transport (85ms)
    getJobInfo
      ✓ returns info on specfed job using idb transport (72ms)
      ✓ returns info on specfed job using rest transport (69ms)
      ✓ returns info on specfed job using ssh transport (615ms)
      ✓ returns info on specfed job using odbc transport (85ms)
    getDataArea
      ✓ returns contents of a data area using idb transport (65ms)
      ✓ returns contents of a data area using rest transport (91ms)
      ✓ returns contents of a data area using ssh transport (758ms)
      ✓ returns contents of a data area using odbc transport (85ms)

  iNetwork Functional Tests
    getTCPIPAttr
iNetwork deprecated As of v1.0, 'iNetwork Class' is deprecated. Please use 'Toolkit Class' instead. test/functional/iNetworkFunctional.js:54:21
      ✓ retrieves TCP/IP Attributes using idb transport (139ms)
      ✓ retrieves TCP/IP Attributes using rest transport (126ms)
      ✓ retrieves TCP/IP Attributes using ssh transport (708ms)
      ✓ retrieves TCP/IP Attributes using odbc transport (175ms)
    getNetInterfaceData
iNetwork deprecated As of v1.0, 'iNetwork Class' is deprecated. Please use 'Toolkit Class' instead. test/functional/iNetworkFunctional.js:100:21
      ✓ retrieves IPv4 network interface info using idb transport (160ms)
      ✓ retrieves IPv4 network interface info using rest transport (145ms)
      ✓ retrieves IPv4 network interface info using ssh transport (808ms)
      ✓ retrieves IPv4 network interface info using odbc transport (109ms)

  iSql Functional Tests
    prepare & execute
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:53:21
      ✓ prepares & executes stored procedure then fetch results using idb transport (480ms)
      ✓ prepares & executes stored procedure then fetch results using rest transport (961ms)
      ✓ prepares & executes stored procedure then fetch results using ssh transport (1364ms)
      ✓ prepares & executes stored procedure then fetch results using odbc transport (273ms)
    addQuery & fetch
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:89:21
      ✓ runs a query and fetches results using idb transport (530ms)
      ✓ runs a query and fetches results using rest transport (403ms)
      ✓ runs a query and fetches results using ssh transport (920ms)
      ✓ runs a query and fetches results using odbc transport (317ms)
    added test to ensure issue #11 was resolved
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:117:21
      ✓ should parse SQL result set empty data tags correctly using idb transport (239ms)
      ✓ should parse SQL result set empty data tags correctly using rest transport (209ms)
      ✓ should parse SQL result set empty data tags correctly using ssh transport (840ms)
      ✓ should parse SQL result set empty data tags correctly using odbc transport (237ms)
    tables
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:146:21
      ✓ returns meta data for specified table using idb transport (241ms)
      ✓ returns meta data for specified table using rest transport (479ms)
      ✓ returns meta data for specified table using ssh transport (848ms)
      ✓ returns meta data for specified table using odbc transport (221ms)
    tablePriv
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:173:21
      ✓ returns privilege data for a table using idb transport (292ms)
      ✓ returns privilege data for a table using rest transport (242ms)
      ✓ returns privilege data for a table using ssh transport (738ms)
      ✓ returns privilege data for a table using odbc transport (242ms)
    columns
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:202:21
      ✓ returns meta data for a column using idb transport (280ms)
      ✓ returns meta data for a column using rest transport (299ms)
      ✓ returns meta data for a column using ssh transport (933ms)
      ✓ returns meta data for a column using odbc transport (278ms)
    columnPriv
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:242:21
      ✓ returns privilege data for a column using idb transport (265ms)
      ✓ returns privilege data for a column using rest transport (242ms)
      ✓ returns privilege data for a column using ssh transport (735ms)
      ✓ returns privilege data for a column using odbc transport (250ms)
    procedures
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:273:21
      ✓ returns meta data on for a procedure using idb transport (412ms)
      ✓ returns meta data on for a procedure using rest transport (253ms)
      ✓ returns meta data on for a procedure using ssh transport (732ms)
      ✓ returns meta data on for a procedure using odbc transport (223ms)
    pColumns
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:303:21
      ✓ returns meta data for procedure column using idb transport (440ms)
      ✓ returns meta data for procedure column using rest transport (272ms)
      ✓ returns meta data for procedure column using ssh transport (738ms)
      ✓ returns meta data for procedure column using odbc transport (299ms)
    primaryKeys
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:344:21
      ✓ returns meta data for a primary key using idb transport (233ms)
      ✓ returns meta data for a primary key using rest transport (239ms)
      ✓ returns meta data for a primary key using ssh transport (724ms)
      ✓ returns meta data for a primary key using odbc transport (238ms)
    foreignKeys
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:372:21
      ✓ returns meta data for a foreign key using idb transport (346ms)
      ✓ returns meta data for a foreign key using rest transport (226ms)
      ✓ returns meta data for a foreign key using ssh transport (730ms)
      ✓ returns meta data for a foreign key using odbc transport (246ms)
    statistics
iSql deprecated As of v1.0, class 'iSql' is deprecated. Use odbc, idb-connector, or idb-pconnector npm package instead. test/functional/iSqlFunctional.js:409:21
      ✓ returns stats info for table using idb transport (587ms)
      ✓ returns stats info for table using rest transport (595ms)
      ✓ returns stats info for table using ssh transport (908ms)
      ✓ returns stats info for table using odbc transport (391ms)
    special
      - returns meta data for special columns using idb transport
      - returns meta data for special columns using rest transport
      - returns meta data for special columns using ssh transport
      - returns meta data for special columns using odbc transport
    rowCount
      - returns the number of rows affected by statement using idb transport
      - returns the number of rows affected by statement using rest transport
      - returns the number of rows affected by statement using ssh transport
      - returns the number of rows affected by statement using odbc transport

  168 passing (1m)
  12 pending

@kadler Resolved conflicts this is now ready for review.


Move statistic related code into a helper

@disforw  As you suggested before, I've moved all the stand-alone functions into statistics helper file

This is excellent! Looks great