COS / More alerts

Steps to reproduce
Follow tutorial for COS integration
Expected behavior
A few alerts I would like to see

Machine is down
Machine is up but service is down
Cluster is not writable
Cluster will not be writable if I lose one more node
Query latency goes over X ms
Number of connections is close to max connections limit

And any similar related to degradation of to prevent the system stops working as expected
Actual behavior
No errors in the alerts, just two alerts

Versions
Juju 3.5.4
data-integrator                    active      1  data-integrator           latest/stable   41  no
grafana-agent                      active      3  grafana-agent             latest/edge    299  no       tracing: off
mongodb                            active      3  mongodb                   6/stable       199  no
self-signed-certificates           active      1  self-signed-certificates  latest/stable  155  no

Hi @gustavosr98 I have just opened a PR(#521) for these desired alerts. I've added most of them. The ones I did not add in that PR are:

Machine is down
Machine is up but service is down
and
Query latency goes over X ms

The first two I did not add since they are already there (screenshots 1+2) the third I do not believe is possible since I don't think grafana supports alerts based on user provided input (i.e. X), if you have a specific latency that you have in mind let me know and I can implement that for you ASAP
Please note that #1 will be further improved on o11y end since they are currently undergoing work for it


cmake: include dependencies headers

Update cmake script to add jemalloc and compression libraries' include directories. This is to fix missing headers in build, e.g: https://travis-ci.org/pingcap/rust-rocksdb/jobs/529604990
After the fix rust-rocksdb build will pass: https://travis-ci.org/yiwu-arbug/rust-rocksdb/jobs/529906506

CI failed

Let me fix CI with another PR.

pew! build is finally green.


Hosts are gone when restarting Atom

Hi guys,
When I restart Atom, I have to add hosts again to remote-edit package every time. Can you help me with this?
macOS 10.12.5 beta, Atom 1.16.0
Br,
Zsolt

It happens to me too, did you get the solution?

confirmed ..
remoteEdit.json becomes corrupted... reproducible when refreshing (Ctrl+shift+F5) Atom while having remote file(s) open
Atom 1.18 x64
Ubuntu 17.04
kernel: 4.10.0-26-generic
remote-edit v1.90


Make whitenoise an optional dependency

When using JinjaX without the JS/CSS parts, and thus without the middleware, one does not really need the whitenoise dependency.
It would be nice if the extra dependency was made optional by using "extras". Users wanting to use it would then install jinjax[whitenoise] or jinjax[middleware] depending on how the set of extra dependencies is named.

Agree! Fixed in the latest release


Fix typo ☺

@abydal : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good; thanks @abydal.


nixos/resilio: fix group name in documentation

Motivation for this change
The group name is rslsync, not resilio.
https://github.com/NixOS/nixpkgs/blob/20998873e8924b7a4f9ec6c367475912ced0cf0a/nixos/modules/services/networking/resilio.nix#L239-L245
Can we use a variable like users.users.rslsync.group in the doc? Would it make sense?
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

/marvin opt-in

/status needs_review

Thanks!


Issue with Mkdocs external hyperlink

Example URL(s)

https://developers.cloudflare.com/pages/framework-guides/deploy-an-mkdocs-site/

Actual Behavior
Top link goes to https://mkdocs.org/ which gives SSL error
Expected Behavior
Link should go to: https://www.mkdocs.org/
Additional information
No response

Thanks for raising, updated the link in the mentioned PR.


MasterWindow.Run wait for events?

Hi,
It looks like it runs at 30 fps regardless of what's going on, but if it used WaitEvents it could allow saving CPU cycles and battery. PostEmptyEvent could be used to break out of the WaitEvents for exit or updates.
This makes UI updates somewhat more difficult, requiring an explicit update call to invalidate the current display and force update, but this is fairly common in GUIs. Or at least it seems like a tradeoff the application might want to make.

@millerlogic Thanks for your advice. I'm wrapping more widgets right now. After that I will try to implement this.

@AllenDang There is a PR in imgui main repo  https://github.com/ocornut/imgui/pull/2749

@Nicify Thanks for the information!
I'm currently working on change the backend from nuklear to imgui.
Because the auto generated nuklear binding has random crash issue on MacOS, and since cgo cannot provide necessary information it's nearly impossible to fix at this point.

@Nicify I implemented it in imgui version, and it works like magic!!!!
The cpu usage is decreased from 10% to 0.4%. Amazing!!!!

@AllenDang  Wow, great job!


[PIO-167] Remove temporal maintenance tool from community projects

Further update following #472.
First of all, Great thanks for @Wei-1's documentation work.
But I don't want to list my maintenance scripts for the official templates in the community page because we need a essntial solution to make the official templates maintenance easy and these scripts are very temporary. I don't want to spread them officially. This pull request is just remove it from the community powered projects page.
I appreciate your understanding and cooperation.

CC: @dszeto @shimamoto

Thanks! Merging.

@dszeto Thanks!

LGTM!

@shimamoto Thanks!!


Mobile - One-Time Page - Nav Rendering Issues + 2 Column Layout

On the one time page on mobile the hamburger nav is not displaying.
https://main--shredit--stericycle.aem.live/en-us/secure-shredding-services/one-off-shredding-service

You have to side scroll the page to see it and then it shows a white bar running down the side with no content

Additonallly. The main content section is not rendering properly. The form should stack not be a two column on mobile

I am thinking the two may be related.

This will be fixed via #508
https://issue486--shredit--stericycle.aem.live/en-us/secure-shredding-services/one-off-shredding-service

This issue is now fixed

@ravise5 still having this issue

Is the cache cleared?
Seems to work at my side

@ravise5 looks like it was mainly cache issues.
There is still one issue with the 2 column on smaller tablet layout. See below. It's not spanning the content the full width and adding a white bar up the side.


ci: fix docs build on jenkins

This reverts commit 84445e10ab20d9aaf9ea55147c50c2fed68bff3f.
This re-enables windows documentation builds.
The jenkins nodes are now fixed at python 3.9, for running sphinx/ docs target
Please squash merge, the commit message is not clean (will commit ammend later)

OK, I dropped the whole revert-of-a-previous-commit from this branch. It now concentrates on pinning the python version to 3.9 on windows jenkins hosts.


The code is not working

Hello,
The code is not working using the WhatsApp web, would you check it?

Hi, sorry for the delay in responding.
I hadn't seen the bug. I will analyze what happened.

@hAbd0u,
Please follow these steps:
1 )Open whatsapp web (https://web.whatsapp.com/)
2) Open the chat;
3) Open the devTools
4) Paste the code into the console.
let b = setInterval(() => {
    let hora = new Date;
    let a = document.getElementsByClassName("_2YPr_ i0jNr selectable-text copyable-text");
    if (a.length == 0) {
        console.log("off");
    } else if (a[0].title == "online") {
        console.log(`online ${hora.getHours()}:${hora.getMinutes()}`);
    }
}, 3000)
b;

Attention, I'm using google chrome.
Test and let me know if it worked.
Inappropriate use of this tool is your responsibility;

Hello @brenovambaster,
I believe it is still not working even the tested browser is Chrome 106.0.52.


Populate domains drop down with what's been ingested in datahub

Resolves https://github.com/ministryofjustice/find-moj-data/issues/385
Our hardcoded domain list has drifed from the CaDeT domain model. We now have a way to ingest domains from CaDeT, so we can dynamically populate our domain model in the service.
On dev, this now looks like this (there are some old domains hanging around still)

I've sorted these alphabetically and removed the subdomains drop down for now. This is based on there being no subdomains available to select any more, regardless of which domain you choose.
Also noticed a small bug where domains weren't being displayed for Chart results, so fixed this in the process.

Because of how our search is structured, only domains for which there are one of the datatypes we have defined are being pulled through. Is this the intended behaviour? eg electronic_monitoring https://datahub-catalogue-dev.apps.live.cloud-platform.service.justice.gov.uk/domain/urn:li:domain:electronic_monitoring/Entities?is_lineage_mode=false


Create main.yml

hi , I propose CI for the Project
@szabgab

Thanks for the contribution @Eliyahou !


"local resource missing" for symbolic links

There seems to be a problem with following symbolic links.
To reproduce the problem:
Install html5validator, because I have not removed dependences on it.
git clone https://github.com/randoop/randoop.git
cd randoop
./gradlew manual
./gradlew htmlSanityCheck

The ouptut of ./gradlew htmlSanityCheck includes:
> Task :installGitHooks UP-TO-DATE

> Task :htmlSanityCheck
********* HTML Sanity Checker findings report *********
created on 03. May 2019, 08:28 by version 1.1.3
checking took 16468 msecs.

Summary for all pages:
======================
checked 750 items on 5 pages,
found 1 issue, 99% successful.
--------------------------------------------------
Summary for file index.html

page path  : /homes/gws/mernst/research/testing/randoop-branch-master/src/docs/index.html
page title : Randoop: Automatic unit test generation for Java
page size  : 4340 bytes

...

--------------------------------------------------
Results for Missing Local Resources Check
6 anchor tag href attribute checked,
1 missing local resources found.

local resource "api/" missing
--------------------------------------------------

...

Unless I am mistaken, the local resource "api/" does exist, but it is a symbolic link.
I am able to click on the link from randoop/src/docs/index.html and view the API documentation.

I'm not sure if I'll find the time to investigate that ... afaik I rely on standard Java file operations here.

Would you mind setting up a small sample project or even test case for #273 and #274, @mernst?
Reproducing this error is a lot of work based on the given randoop project (have to work through the project setup and install a lot of stuff as pre-requisite).
Perhaps you could even check on your side whether the problem still exists?

The problem persists, using version 1.1.6.
The prerequisites are discussed at https://randoop.github.io/randoop/manual/dev.html#prerequisites (which is linked from https://randoop.github.io/randoop/manual/index.html#getting_randoop), so you can install them all at once.  Is there a particular one that you are having trouble with?  Then I will know what parts of the project to prune away.

Thanks for your fast reply, @mernst.
Randoop looks like an interesting project, maybe I should have a deeper look into this, but this is a different story.
However, it's a question of time. I have to work through your documentation which mentions a Dockerfile to meet prerequisites. But I do not understand whether I can just build your project by just using Docker? Neither do I have the time to reverse engineer the Dockerfiles for several Ubuntu versions (which one should I use)? Besides, I am working on macOS, so which tools would I need to build locally? How do they pollute my system (e.g., pip install)?
As you are an OSS maintainer yourself, you maybe have a good understanding how time consuming all of this is, in particular if it goes beyond just executing one script. We cannot afford the time to first fully understand the installation of your project to finally produce an (HTML) artifact which shows a problem in our project.
So please attach a (minimal) HTML example or ZIP/Tar including a symlink which shows your problem(s) in #273 and #274.


Examples reference '/Users/acabrera/dev/data'

Do a search for the key word /Users/acabrera/dev/data and you'll see many examples use this path

On a sidenote, I now know where @cabreraalex keeps all his data. Honestly I should probably use this strategy for sharing across projects.

hehe it has saved many a headache but yes should change that


Syncing assignments

From the FAQ:

I have set a preference to open a website in a particular container. Will that also be synced?

No. This data is not available to any extensions by the API.

https://github.com/mozilla/multi-account-containers/pull/1095 was merged (and released in 6.0), would this be enough to add syncing of assignments?
(Irrelevant) I also mentioned this plugin in https://github.com/mozilla/multi-account-containers/issues/339.

Hi,
Thanks for informing about the API to get assignment for a given URL.
For this extension to sync the assignments, I would need the API to perform the following

Given a container ID, provide the list of all assigned URLs. (this is required for backup)
Given a container ID and URL, update or remove its assignment. (this is required for restore)

The getAssignment API takes an URL as the input and provides the container ID.
Since it does not satisfy the requirements for this extension, this API is not enough to add syncing of assignments.
I also saw that it is possible to add the required functionality for multi-account-containers. Hence, I have created a pull request https://github.com/mozilla/multi-account-containers/pull/1197 implementing the required API changes.


Gist button: problem grabbing notebook JSON code

Great extension. But there seems to be a problem in grabbing the JSON code that it posts to Gist, such that nbviewer throws a 500 error when trying to open it. When I copy the raw JSON code myself into the Gist web interface it works fine.
Here's an example of a notebook posted using the button:
https://gist.github.com/bf9692b833727299c1f9
And here's the same notebook posted manually:
https://gist.github.com/d1a2d7075c5e13f9e840

I think this is because the extension only works with IPython ≥ 3. I think it's a tiny fix for 2, though. I'll look into it.

Should be fixed now.

Works great, thanks!


Update tag and push plugin

CLOUDBLD-9030

Maintainers will complete the following section

[x] Commit messages are descriptive enough
[x] Code coverage from testing does not decrease and new code is covered
[x] Python type annotations added to new code
[n/a] JSON/YAML configuration changes are updated in the relevant schema
[n/a] Changes to metadata also update the documentation for the metadata
[n/a] Pull request has a link to an osbs-docs PR for user documentation updates
[n/a] New feature can be disabled from a configuration file

Just notice there is a typo "export_image_sequence" in commit message.


ProgramWithUnnecessaryThingsRemoved is contrary to the spec

Migrated from https://bugs.webkit.org/show_bug.cgi?id=190301:
At 2018-10-05T02:07:43Z<EMAIL_ADDRESS>wrote:
We need to report errors in things, even if they aren't referenced by any functions.

https://github.com/gpuweb/WHLSL/issues/238


csl.sty fails with other styles

I tried example.tex with other .csl styles, but without luck. For example, if I replace plainwith Zeitschrift für Fantastikforschung (Deutsch) which I wrote, the compilation produces all kinds of errors and completely borked pdf (.log attached)
example.log
.

It is because this .csl style requires locales-de-DE.xml which is not installed to TEXMF. You can copy one from citation-style-language/locales to the working directory. After that the example works for me. I'm planning to pack all the locale files in citation-style-language/locales together when publishing to CTAN and there will be no need to perform the copying.
Currently the module_error function from LaTeX2e kernel (ltluatex.dtx) is used to raise an error. This function always prints the stack traceback, which may be very long and makes it difficult to read. I'll figure out a way to give cleaner error messages.

With locales-de-DE.xml it works. Thanks!


Enhance the  social icons in footer in Home Page

Describe the feature
Hey @anuragverma108
Currently, in the footer section of home page, the icons are squared and I would like to make them look like original icon which enhance the visual appearance.
please add the gssoc and the required level labels and I would like to work on  it under GSSOC'24. Thank you!!
Add ScreenShots
Before making the changes:

After making the changes:

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Assigned!
Make sure to attach before & after screenshots in your PR and include the issue number.
Also, please star the repo before working on it.
I am the project mentor so you can always reach out to me for any help!
Happy open-source contributing!


Use SPORK_6_NEW_SIGS to switch from signing string messages to hashes

Actually, let's fix all sigs at once while we are at it. Using signature-specific hash (GetSignatureHash()) should help avoiding issues on p2p/caching level (which uses GetHash()) while being able to cover all required fields at the same time. These changes are grouped into separate commits on a per-message basis.
a721459 also fixes way too aggressive policy for new format of mnp signatures introduced in previous PR which could lead to excessive banning.
3715146 fixes docs by adding missing mnp fields and adjusting SPORK_6_NEW_SIGS description to match changes proposed here.

Regarding the GetSignatureHash implementations. What about just serializing a copy of *this with vchSig being cleared before? This way you make sure that you are on the safe side when future changes are done to the serialized objects, e.g. new fields being added and smart version detection is in place.

Good idea about using serialization 👍
Could be smth like
if (!(s.GetType() & SER_GETHASH))
    READWRITE(vchSig);

in SerializationOp and then the rest of it should work just like we need.
I'd still keep it explicit here though and leave this enhancement for another PR.

Sounds good. Btw, there's ::SerializeHash(...) for exactly this purpose.

Ahh, I knew there was smth! :) Will re-work 97a9e93

@codablock 97a9e93 -> 9af4e3392
Turned out that not everything is replaceable this way though :/ We'll have to wait for migration to 70209 to complete before we can drop s.size() == 0 conditions in CMasternodePing and re-use SerializeHash there and for CMasternodeBroadcast (due to lastPing).

utACK

FYI: Slightly tested on the client side, everything seems to be working now. Thanks for reviews! 👍


Alternative for reducing array to more than 1 element along axes of reduction.

The engine is modified to delay calculating chunk size till the end for implementing operations that give variable block shape (e.g. np.unique). The strategy of block merging is moved into a class.
For a 2D array arr, np.argsort(arr, axis=1) returns an ind array, where each column of ind is indexing corresponding column of arr. I call this an index along axis.  The function for applying index that takes more than 1 element along an axis to an array is mostly correct, except I didn't think keepdims should affect the result. The old code can only handle indexing 1 element along an axis (works for np.argmin(arr, axis=1), but not for argtopk with axis != None and k > 1). This means reduction works, but arg_reduction doesn't work yet. I am still trying to fix this.

reduction.py
# -*- coding: utf-8 -*-
"""Tree Reduction
This module implements tree reduction of a dask array by operations (e.g.
np.sum) and arg operations (e.g. np.argmin).

The limitations of this module are:

- The intermediate results are arrays of join-block (tuple of blocks).
- All blocks in each joint-block array have the same number of dimensions.
- The reduction is along one or all of the axes.
- The final chunk shape(s) and dtype(s) are known.

The steps of a tree reduction are:

1. Create a joint-block array (JBA) by either: A. Map a prep_func onto value (
and optionally offset) for all blocks of an input dask array. Or, B. Convert
several dask arrays with the same numblocks to a joint-block array. And then
map a prep_func over the joint-block array.
2. Concatenate the joint-blocks by merge_func according to the Strategy.
3. Map func with keepdims=True onto current joint-blocks.
4. Repeat steps 2 and 3 until StopCondition is satisfied.
5. Map func with keepdims=False onto current joint-blocks.
6. Convert the join-blocks to dask array(s) by a fixer function,
which calculates the final chunk(s) shape(s) from the input dask array and
axis of reduction.

This module offers two interfaces: reduction and simple_reduction.

1. reduction requires prep_func, func, fixer, and final dtype.
2. simple_reduction only requires func, and final dtype. The simple_fixer
assumes the final result is a single dask array with a chunk shape of 1 along
each axis of reduction. This doesn't work for top-k, which reduces the dask
array to k elements along each axis of reduction. The func should only
accept/return one (instead of multiple) numpy array.
"""
from __future__ import absolute_import, division, print_function

from functools import partial, reduce, wraps
from itertools import product, repeat
from math import ceil, log
from operator import mul, add

import numpy as np
from toolz import partition_all, accumulate

from . import jointblockarray as jba
from dask.array import chunk, Array
from dask.array.core import handle_out
from dask.compatibility import builtins
from dask.base import tokenize
from dask.context import _globals
import dask.sharedict as sharedict

# -- Helpers -------------------------------------------------------------------
def _axis_to_axes(axis, ndim):
    """Convert from numpy axis to axes
    See Also
    --------
        Parameters : for description of parameters and returns.
    """
    if isinstance(axis, int):
        axes = (axis,)
    elif axis is None:
        axes = range(ndim)
    else:
        axes = axis
    # normalize the axes
    norm_func = np.core.multiarray.normalize_axis_index
    return tuple(norm_func(ax, ndim) for ax in axes)

# -- Block Reduction -----------------------------------------------------------
def _normalize_split_every(split_every, numblocks, axes):
    """Normalize split_every to dictionary representation.
    See Also
    --------
        Parameters : for description of parameters and returns.
    """
    se = split_every or _globals.get('split_every', 4)
    if isinstance(se, dict):
        se = dict((k, se.get(k, 2)) for k in axes)
    elif isinstance(se, int):
        n = builtins.max(int(se ** (1.0 / (len(axes) or 1))), 2)
        se = dict.fromkeys(axes, n)
    else:
        se = dict((k, v) for (k, v) in enumerate(numblocks) if k in axes)
    return se

class Parameters(object):
    """Parameters for tree reduction of a Dask array.
    Parameters
    ----------
        x : Array
            Input dask array for the tree reduction.
        axis : int, or None
            Axis of reduction in numpy's representation. If None, reduction
            is along all axes.
        split_every : int, dict, or optional
            If not provided, merge every n joint-blocks over each axis. If
            context specifies 'split_every', n is the value; otherwise n is 4.
            If dict, merge every <value> joint-blocks along axis <key> for each
            key-value pair.
            If int, merge every <split_every> joint-blocks over each axis.
            For example, {0: 4} means to reduce every 4 joint-blocks to 1
            joint-block along the 0th dimension and don't reduce along other
            dimensions.
    Attributes
    ----------
        axes : tuple
            Tuple of non-negative integers. Reduction is along axes
            represented by these integers.
    """
    def __init__(self, x, axis, split_every=None):
        if not isinstance(x, Array):
            raise ValueError('x should be a Dask array.')
        self.x = x  # Store input dask array for fixer.
        self.axis = axis
        self.axes = _axis_to_axes(axis, x.ndim)
        self._split_every = split_every

    def split_every(self, numblocks=None):
        """Normalize split_every to dict representation.
        Parameters
        ----------
            numblocks : tuple, optional
                Number of joint-blocks along each dimension. Default to
                numblocks of input Dask array (self.x).
        """
        numblocks = numblocks or self.x.numblocks
        return _normalize_split_every(self._split_every, numblocks, self.axes)

class Strategy(object):
    """Strategy for reduction over an array of joint-blocks.
    Parameters
    ----------
        parameters : Parameters
    """

    def __init__(self, parameters):
        self._parameters = parameters

    def make_matcher(self, numblocks):
        """Make block matcher for an array of joint-blocks
        Parameters
        ----------
            numblocks : tuple
                Number of joint-blocks along each dimension.
        """
        se = self._parameters.split_every(numblocks)
        fused_ids = self._strategy(numblocks, se)
        return Matcher(fused_ids)

    def _strategy(self, numblocks, split_every):
        """Modify this to implement alternative strategy.
        Returns
        -------
            fused_ids : list
                See Matcher for description of fused_ids
        """
        return _default_strategy(numblocks, split_every)

def _default_strategy(numblocks, split_every):
    """Find fused_ids
    See Also
    --------
        Strategy : for description of parameters
        Matcher : for description of returns.
    """
    # fused_ids = []
    # for i, n in enumerate(numblocks):
    #     # Along the i^{th} dimension, we should merge every every_m blocks.
    #     every_m = split_every.get(i, 1)
    #     # Split range(n) to tuples of at most every_m elements.
    #     a = partition_all(every_m, range(n))
    #     fused_ids.append(a)
    fused_ids = [list(partition_all(split_every.get(i, 1), range(n)))
                 for (i, n) in enumerate(numblocks)]
    return fused_ids

class Matcher(object):
    """Joint-block matcher for joint-block reduction
    Parameters
    ----------
        fused_ids : list, None
            Defines merge of input blocks along each dimension.
            For example, [[(0, 1, 2)], [(0,), (1,), (2,), (3,), (4,)]] means
            along dimension 0, reduce the 0th-2nd block to one tuple (
            broadcast this operation over other dimensions), and leaves
            dimension 1 intact.
    Attributes
    ----------
        new_numblocks : tuple, None
            Output joint-block array from the joint-block reduction.
        axes : tuple
            Tuple of non-negative integers representing a set of axes.
    """

    def __init__(self, fused_ids):
        self.fused_ids = fused_ids
        self.new_numblocks = _sub_len(fused_ids)
        self.extra_names = (fused_ids,)

    def __eq__(self, other):
        return isinstance(other, Matcher) and (
            self.fused_ids == other.fused_ids)

    def idgen(self):
        """Match input and output joint-blocks by indices.
        Iterate over each output joint-block and gives joint-block indices of
        the input and output joint-blocks.
        Yields
        ------
            in_ids : list
                List of indices of input joint-blocks.
            fused_shape : tuple
                Numpy's shape refers to arrangement of elements. Similarly,
                concatenating the input joint-blocks by the arrangement
                represented by fused_shape gives the output joint-block.
            out_id : tuple
                Indices of the output joint-block.
        """
        return _matcher_idgen(self.fused_ids, self.new_numblocks)

def _sub_len(in_tuple):
    """Tuple of len of each element of in_tuple"""
    return tuple(len(t) for t in in_tuple)

def _matcher_idgen(fused_ids, new_numblocks):
    """Index matcher for input and output joint-blocks for block reduction.
    See Also
    --------
        Matcher : See the idgen method of Matcher for description of
        parameters and yields.
    """
    # ids of output-joint blocks.
    out_block_ids = product(*map(range, new_numblocks))
    for p, out_id in zip(product(*fused_ids), out_block_ids):
        # p = ((0, 1, 2), (0,)) means input block with block indices
        # (0, 0), (1, 0), and (2, 0) are reduced to the output block.
        yield list(product(*p)), _sub_len(p), out_id

def blockreduction(func, x, matcher, merge_func=jba.jointblock_concatenate,
                   name=None):
    name = name or 'blockreduction-' + tokenize(
        func, merge_func, x.name, *matcher.extra_names)
    return _blockreduction(func, merge_func, x, matcher.idgen(),
                           matcher.new_numblocks, name)

def _blockreduction(func, merge_func, x, idgen, new_numblocks, name):
    dsk = {}
    for in_ids, numblocks, out_id in idgen:
        # in_keys must be a list instead of a tuple. Otherwise, dask would
        # interpret in_keys[0] as a function receiving in_keys[1:] as arguments.
        in_keys = [(x.name,) + id for id in in_ids]
        out_key = (name,) + out_id
        merged_key = (merge_func, in_keys, numblocks)
        dsk[out_key] = (func, merged_key)

    overall_dsk = sharedict.merge(x.dask, (name, dsk))

    return jba.JointBlockArray(overall_dsk, name, new_numblocks)

class StopCondition(object):
    def __init__(self, parameters):
        p = parameters
        self.cycle_id = -1
        self.n_cycles = _find_tree_depth(p.x.numblocks, p.split_every()) - 1

    def __repr__(self):
        return "StopCondition<cycle_id: %d, n_cycles: %d>" % (
            self.cycle_id, self.n_cycles)

    def check(self):
        self.cycle_id += 1
        return self.cycle_id < self.n_cycles

def _find_tree_depth(numblocks, split_every):
    if numblocks == ():
        return 0  # For reduction over 0-dimensional array (a number).

    tree_depth = 1
    for i, n in enumerate(numblocks):
        if i in split_every and split_every[i] != 1:
            depth_now = ceil(log(n, split_every[i]))
            tree_depth = int(builtins.max(depth_now, tree_depth))
    return tree_depth

# -- Tree Reduction ------------------------------------------------------------
def argmap_jointblock(func, x, name=None):
    name = name or tokenize(func, x.name, 'argmap')
    return jba.map_jointblock(func, x, extra_gen=_offset_gen(x), name=name)

def _offset_gen(x):
    # cumsum for each tuple in x.chunks
    cumsum_chunks = (accumulate(add, bd[:-1], 0) for bd in x.chunks)
    offsets_gen = product(*cumsum_chunks)
    overall_shape = x.shape
    return offsets_gen, repeat(overall_shape)

def _tree_reduction2(interim_func, final_func, jb_arr, strategy, stopcondition,
                     merge_func=jba.jointblock_concatenate):
    if jb_arr.numblocks == ():
        return jb_arr  # For 0-dimensional array (a number).

    new_jb_arr = jb_arr
    matcher = strategy.make_matcher(new_jb_arr.numblocks)
    while stopcondition.check():
        new_jb_arr = blockreduction(interim_func, new_jb_arr, matcher,
                                    merge_func=merge_func)
        matcher = strategy.make_matcher(new_jb_arr.numblocks)
    return blockreduction(final_func, new_jb_arr, matcher,
                          merge_func=merge_func)

def _tree_reduction(func, fixer, jb_arr, parameter, strategies=None,
                    merge_func=None, out=None, dtype=None, extra_kwargs=None):
    """Tree reduction over a JointBlockArray
    Parameters
    ----------
        func : callable
            Signature is func(merged_block). Returns an output block.
        jb_arr : jba.JointBlockArray
        axis : int, tuple, optijb_arrjb_arronal
            Reduction is along this numpy representation of axes
        strategy : class
            Strategy for reduction
    """
    merge_func = merge_func or jba.jointblock_concatenate
    extra_kwargs = extra_kwargs or {}
    strategy, stop_condition = strategies or (Strategy, StopCondition)
    p = parameter
    strat = strategy(p)
    stop_cond = stop_condition(p)
    interim_f = partial(func, axis=p.axis, keepdims=True, **extra_kwargs)
    final_f = partial(func, axis=p.axis, keepdims=False, **extra_kwargs)
    new_jb_arr = _tree_reduction2(interim_f, final_f, jb_arr, strat, stop_cond,
                                  merge_func=merge_func)
    return handle_out(out, fixer(p, new_jb_arr, dtype=dtype))

def _apply_prep(prep_func, x, axis, applyer=jba.map_jointblock,
                split_every=None, extra_kwargs=None):
    if prep_func is None:
        jb_arr = jba.from_dask_arrays([x])
    else:
        extra_kwargs = extra_kwargs or {}
        prep_f = partial(prep_func, axis=axis, **extra_kwargs)
        jb_arr = applyer(prep_f, x)
    p = Parameters(x, axis, split_every=split_every)
    return p, jb_arr

def reduction(prep_func, func, fixer, x, axis=None, split_every=None,
              merge_func=None, strategies=None, out=None, dtype=None,
              extra_kwargs=None):
    p, jb_arr = _apply_prep(prep_func, x, axis, applyer=jba.map_jointblock,
                            split_every=split_every, extra_kwargs=extra_kwargs)
    return _tree_reduction(func, fixer, jb_arr, p, strategies=strategies,
                           merge_func=merge_func, out=out, dtype=dtype,
                           extra_kwargs=extra_kwargs)

def arg_reduction(prep_func, func, fixer, x, axis=None, split_every=None,
                  merge_func=None, strategies=None, out=None, dtype=None,
                  extra_kwargs=None):
    applyer = argmap_jointblock
    p, jb_arr = _apply_prep(prep_func, x, axis, applyer=applyer,
                            split_every=split_every, extra_kwargs=extra_kwargs)
    return _tree_reduction(func, fixer, jb_arr, p, strategies=strategies,
                           merge_func=merge_func, out=out, dtype=dtype,
                           extra_kwargs=extra_kwargs)

# -- Indexer -------------------------------------------------------------------
def shift_index(local_index, local_shape, offset, overall_shape, axis):
    if axis is None:
        return shift_flat_index(local_index, local_shape, offset, overall_shape)
    else:
        return shift_index_along_axis(local_index, offset, axis)

def apply_index(x, index, axis):
    if axis is None:
        return x.ravel()[index]
    else:
        return index_along_axis(x, index, axis)

def shift_flat_index(local_index, block_shape, offset, overall_shape):
    local_multi_index = np.unravel_index(local_index.ravel(), block_shape)
    global_multi_index = tuple(l + o for l, o in zip(local_multi_index, offset))
    return np.ravel_multi_index(global_multi_index, overall_shape)

def shift_index_along_axis(local_index, offset, axis):
    return local_index + offset[axis]

def index_along_axis(arr, ind, axis):
    """Apply an index along axis to an array.
    Parameters
    ----------
        arr : np.ndarray
        ind : np.ndarray
            Indexing of an array along an axis. This cannot be an int.
            For a 2D array arr and axis=1, this means: from each column of
            arr, select elements along the column by the indices at the
            corresponding column of ind. In other words, the ith column of
            the result is arr[ind[:, i], i].
        axis : int
    Returns
    -------
        result : np.ndarray
    Example
    -------
        >>> import numpy as np
        >>> arr = np.flip(np.arange(8).reshape(2, 4), axis=1)  # test array
        >>> arr
        array([[3, 2, 1, 0],
               [7, 6, 5, 4]])
        >>> ind = np.argsort(arr, axis=1)  # indexing along axis 1
        >>> result = index_along_axis(arr, ind, axis=1)
        >>> result  # show the flat indexing
        array([[0, 1, 2, 3],
               [4, 5, 6, 7]])
        >>> answer = np.sort(arr, axis=1)
        >>> np.all(result == answer)
        True
    """
    # Does not check if axis is legal
    shape = arr.shape
    before = reduce(mul, shape[:axis], 1)
    at = shape[axis]
    after = arr.size // at // before
    a = arr.reshape(before, at, after)
    idx = [
        np.arange(before).reshape(before, 1, 1),
        ind.reshape(before, -1, after),
        np.arange(after).reshape(1, 1, after),
    ]
    return a[idx].reshape(ind.shape)

# -- Keepdims ------------------------------------------------------------------
def _keep_dimension(x, axes, ndim):
    if ndim == x.ndim:
        return x

    r_slice = []
    for each_axis in range(ndim):
        if each_axis in axes:
            r_slice.append(np.newaxis)
        else:
            r_slice.append(slice(None))  # get everything over this axis

    return x[r_slice]

def keep_dimensions(jointblock, axis, ndim, keepdims=True):
    if keepdims:
        # Assumes all blocks in the input jointblock have the same number of
        # dimensions. This means the axes of reduction should be same for
        # these blocks.
        axes = _axis_to_axes(axis, ndim)
        return tuple(_keep_dimension(x, axes, ndim) for x in jointblock)
    else:
        return jointblock

# -- Simple reduction interface ------------------------------------------------
def simple_reduction(func, x, axis=None, split_every=None, out=None,
                     extra_kwargs=None, dtype=None):
    extra_kwargs = extra_kwargs or {}

    def prep_func(x2, **kwargs2):
        return func(x2, keepdims=True, **kwargs2),  # tuple

    def reduce_func(jointblock, **kwargs2):
        return func(jointblock[0], **kwargs2),  # tuple

    return reduction(prep_func, reduce_func, simple_fixer, x, axis=axis,
                     split_every=split_every, out=out, dtype=dtype,
                     extra_kwargs=extra_kwargs)

def simple_arg_reduction(argfunc, x, axis=None, split_every=None,
                         out=None, extra_kwargs=None, dtype=None):
    extra_kwargs = extra_kwargs or {}

    def prep_func(x2, offset, overall_shape, **kwargs2):
        axis2 = kwargs2.get('axis', None)
        local_arg = argfunc(x2, keepdims=True, **kwargs2)
        val = apply_index(x2, local_arg, axis2)
        global_arg = shift_index(local_arg, x2.shape, offset, overall_shape,
                                 axis2)
        return global_arg, val

    def reduce_func(jointblock, **kwargs2):
        global_arg, val = jointblock
        keepdims2 = kwargs2.get('keepdims', True)
        axis2 = kwargs2.get('axis', None)
        local_arg = argfunc(val, keepdims=keepdims2)
        new_global_arg = apply_index(global_arg, local_arg, axis2)
        new_val = apply_index(val, local_arg, axis2)
        return new_global_arg, new_val

    return arg_reduction(prep_func, reduce_func, simple_fixer, x, axis=axis,
                         split_every=split_every, out=out, dtype=dtype,
                         extra_kwargs=extra_kwargs)

def simple_fixer(p, jb_arr, dtype=None, index=0):
    fixer_dtype = dtype or p.x.dtype
    if p.axis is None:
        chunks = ()
    else:
        chunks = tuple(c for i, c in enumerate(p.x.chunks) if i != p.axis)
    return jb_arr.extract(index, chunks, dtype=fixer_dtype, drop_axes=p.axes)

# -- Routines ------------------------------------------------------------------
@wraps(chunk.sum)
def sum(a, axis=None, dtype=None, split_every=None, out=None):
    dt = dtype or np.empty((1,), dtype=a.dtype).sum().dtype
    extra_kwargs = {'dtype': dt}
    return simple_reduction(chunk.sum, a, axis=axis, dtype=dt,
                            split_every=split_every, out=out,
                            extra_kwargs=extra_kwargs)

@wraps(chunk.argmin)
def argmin(a, axis=None, split_every=None, out=None):
    dt = np.argmin([1]).dtype
    return simple_arg_reduction(chunk.argmin, a, axis=axis, dtype=dt,
                                split_every=split_every, out=out)

def argmin_prep(x, offset, overall_shape, axis=None):
    local_arg = np.argmin(x, axis)
    val = apply_index(x, local_arg, axis)
    global_arg = shift_index(local_arg, x.shape, offset, overall_shape, axis)
    return val, global_arg

def argmin_reduce(jointblock, axis, keepdims=True):
    val = jointblock[0]
    global_arg = jointblock[1]

    local_arg = np.argmin(val, axis)

    new_global_arg = apply_index(global_arg, local_arg, axis)
    new_val = apply_index(val, local_arg, axis)
    new_jointblock = (new_val, new_global_arg)

    return keep_dimensions(new_jointblock, jointblock[0].ndim, keepdims)

def argmin_fix(x, newx, axis):
    chunks = list(x.chunks)
    chunks[axis] = 1
    dtype = x.dtype
    return newx.extract(1, chunks, dtype)

jointblockarray.py
import dask.sharedict as sharedict
from dask.base import tokenize
import dask.array as da
from operator import getitem
from dask import get
from pprint import pformat
from itertools import product
import numpy as np

class JointBlockArray(object):
    """Array of tuples of blocks.
    Require all blocks have the same number of dimensions.
    """

    def __init__(self, dsk, name, numblocks):
        self.dask = dsk
        self.name = name
        self.numblocks = numblocks
        self.ndim = len(numblocks)
        self._base_arrays = None

    def extract(self, index, chunks, dtype, drop_axes=None, name=None):
        if self._base_arrays is None:
            return self._extract(index, chunks, dtype, drop_axes=drop_axes,
                                 name=name)
        else:
            return self._base_arrays[index]

    def __repr__(self):
        return "JointBlockArray<numblocks=%s, name=%s>" % (
            self.numblocks, self.name)

    def nd_block_index(self):
        return product(*map(range, self.numblocks))

    def format(self, compute=True):
        op = get if compute else getitem
        nested_list = [(idx, op(self.dask, (self.name,) + idx))
                       for idx in self.nd_block_index()]
        if self.numblocks != ():
            nested_list = da.core.reshapelist(self.numblocks, nested_list)
        rep = self.__repr__()
        return '%s\n%s' % (rep, pformat(nested_list))

    def _extract(self, index, chunks, dtype, drop_axes=None, name=None):
        """Extract the n^{th} dask array within a JointArray
        Parameters
        ----------
            index : int
                index to the dask array within the JointArray
        """
        dsk = {}
        name = name or tokenize(name, index)

        in_id_gen = self.nd_block_index()
        if drop_axes is None:
            for midx in in_id_gen:
                key_to_tuple = (self.name,) + midx
                out_key = (name,) + midx
                dsk[out_key] = (getitem, key_to_tuple, index)
        else:
            reduced_numblocks = _reduce_numblocks(self.numblocks, drop_axes)
            out_id_gen = _ndindex(reduced_numblocks)
            gen = zip(in_id_gen, out_id_gen)
            print("drop axes", drop_axes)
            for midx, ridx in gen:
                key_to_tuple = (self.name,) + midx
                out_key = (name,) + ridx
                print("midx, ridx", midx, ridx)
                dsk[out_key] = (getitem, key_to_tuple, index)

        overall_dsk = sharedict.merge((name, dsk), self.dask)
        return da.Array(overall_dsk, name, chunks, dtype)

def _reduce_numblocks(numblocks, drop_axes=None):
    if np.isscalar(drop_axes):
        drop_axes = (drop_axes,)
    if len(drop_axes) == len(numblocks):
        return ()
    if any(numblocks[i] > 1 for i in drop_axes):
        raise ValueError("Can't drop an axis with more than 1 block.")
    return tuple(v for i, v in enumerate(numblocks) if i not in drop_axes)

def jointblock_concatenate(jointblocks, numblocks):
    """ Concatenate a list of tuple of blocks along axes
    Assumes all tuples contains the same number of blocks, and all blocks
    have the same number of dimensions.
    Parameters
    ----------
        block_tuples : list, tuple
            A list/tuple of tuple of blocks
        numblocks : tuple
            A tuple of integers that describes the block
    Returns
    -------

    See Also
    --------
        _concatenate2 : currently this function uses _concatenate2
    """
    jb_length = len(jointblocks[0])

    if any(len(t) != jb_length for t in jointblocks):
        raise ValueError("The jointblocks do not contain the same number of "
                         "blocks")

    result = []
    for i in range(jb_length):
        extracted = [jb[i] for jb in jointblocks]
        # should check if all arrays in extracted have the same number of
        # dimensions here. TODO
        lol = da.core.reshapelist(numblocks, extracted)
        # da.core.concatenate3 doesn't work here.
        # For example, it can't work with:
        # [array([[0, 1],
        #         [4, 5],
        #         [8, 9]]), array([[2, 3],
        #                          [6, 7],
        #                          [10, 11]])]
        axes = tuple(range(len(numblocks)))
        r = da.core._concatenate2(lol,axes)
        result.append(r)
    return result

# -- Additional Constructors ---------------------------------------------------
def from_dask_arrays(dask_arrays, name=None):
    """Create a TupleArray from a list of dask arrays.
    Requires the dask arrays to have the same numblocks
    Returns
    -------
        ja : JointBlockArray
    """
    # Assert all dask arrays have the same numblocks
    numblocks0 = dask_arrays[0].numblocks
    if any(d.numblocks != numblocks0 for d in dask_arrays):
        raise ValueError(
            'Sorry. Creating tuple array from dask arrays with unequal '
            'numblocks has not been implemented. The numblocks are : %s' % [
                d.numblocks for d in dask_arrays])

    name = name or tokenize(*(dask_arrays + [name]))
    base_names = [d.name for d in dask_arrays]  # names of input dask arrays.

    dsk = {}
    # Iterate over multi-indices of all corresponding blocks
    for idx in _ndindex(numblocks0):
        out_key = (name,) + idx
        in_keys = tuple((in_name,) + idx for in_name in base_names)
        dsk[out_key] = (_make_tuple,) + in_keys

    overall_dsk = sharedict.merge((name, dsk), *(d.dask for d in dask_arrays))
    result = JointBlockArray(overall_dsk, name, numblocks0)
    result._base_arrays = dask_arrays

    return result

def map_jointblock(func, x, extra_gen=None, name=None):
    in_name = x.name
    name = name or tokenize(func, in_name)
    dsk = {}
    idx_gen = _ndindex(x.numblocks)

    if extra_gen is None:
        for idx in idx_gen:
            in_key = (in_name,) + idx
            out_key = (name,) + idx
            dsk[out_key] = (func, in_key)
    else:
        for bundle in zip(idx_gen, *extra_gen):
            in_key = (in_name,) + bundle[0]
            out_key = (name,) + bundle[0]
            dsk[out_key] = (func, in_key) + bundle[1:]
    overall_dsk = sharedict.merge((name, dsk), x.dask)
    return JointBlockArray(overall_dsk, name, x.numblocks)

# -- Helper --------------------------------------------------------------------
def _make_tuple(*args):
    return args

def _ndindex(shape):
    return product(*map(range, shape))

tests/test_reduction.py
from __future__ import absolute_import, division, print_function
from dask.compatibility import builtins

import pytest

pytest.importorskip('numpy')

import dask.array as da
from dask.base import tokenize

import numpy as np

from .. import reductions as module
from .. import jointblockarray as jba

from dask.array.utils import assert_eq as _assert_eq, same_keys
from math import ceil

from functools import partial, reduce
from operator import mul

# -- Helpers -------------------------------------------------------------------
def assert_eq(a, b):
    _assert_eq(a, b, equal_nan=True)

def ndarange(*shape, **kwargs):
    dtype = kwargs.get('dtype', np.intp)
    shuffle = kwargs.get('shuffle', False)
    prod = reduce(mul, shape, np.intp(1))
    arr = np.arange(prod, dtype=dtype)
    if shuffle:
        np.random.shuffle(arr)
    return arr.reshape(shape)

# -- Tests ---------------------------------------------------------------------
def test__axis_to_axes():
    cases = [
        [(0,), 2, (0,)],
        [(1,), 2, (1,)],
        [(-1,), 2, (1,)],
        [(0, 1,), 2, (0, 1,)],
        [None, 2, (0, 1,)],
    ]

    for axis, ndim, answer in cases:
        result = module._axis_to_axes(axis, ndim)
        assert (result == answer)

    fail_cases = [
        [(-2,), 2],
        [(2,), 2],
        [(-1, 2,), 2],
    ]

    for axis, ndim in fail_cases:
        try:
            module._axis_to_axes(axis, ndim)
        except np.core._internal.AxisError:
            pass

def test__normalize_split_every():
    cases = [
        [None, (2, 3), (0, 1), {0: 2, 1: 2}],
        [16, (4, 4), (0, 1), {0: 4, 1: 4}],
        [{0: 4, 1: 8}, (2, 3), (0,), {0: 4}]
    ]

    for split_every, numblocks, axes, answer in cases:
        result = module._normalize_split_every(split_every, numblocks, axes)
        assert result == answer

def test_Parameters():
    arr = ndarange(2, 3)
    x = da.from_array(arr, 2)
    axis = 1
    split_every = 4
    p = module.Parameters(x, axis, split_every)
    assert p.axis == axis
    assert_eq(x, p.x)
    assert same_keys(x, p.x)
    assert p.axes == module._axis_to_axes(axis, x.ndim)
    assert p._split_every == split_every

    answer = module._normalize_split_every(split_every, x.numblocks, p.axes)
    assert p.split_every() == answer

    for numblocks in [(2, 3), (14, 14), (3, 5, 7)]:
        answer = module._normalize_split_every(split_every, numblocks, p.axes)
        assert p.split_every(numblocks) == answer

def test__default_strategy():
    cases = [
        [(1, 2), {0: 4, 1: 4}, [[(0,)], [(0, 1)]]],
        [(4, 2), {0: 2, 1: 2}, [[(0, 1), (2, 3)], [(0, 1)]]]
    ]
    for numblocks, split_every, answer in cases:
        assert answer == module._default_strategy(numblocks, split_every)

def test__sub_len():
    cases = [
        [[[(0,)], [(0, 1)]], (1, 1)],
        [[[(0, 1), (2, 3)], [(0, 1)]], (2, 1)]
    ]
    for fused_ids, answer in cases:
        assert answer == module._sub_len(fused_ids)

def matcher_idgen_case_generator():
    # case 0:
    #     +-+-+      +---+
    #     |a|b|  ->  | c |
    #     +-+-+      +---+
    #
    #   in_ids are [[(0, 0) for a, (0, 1) for b]]
    #   numblocks are [(1, 2)]
    #   out_id are [(0, 0)]
    #
    #
    # case 1:
    #     +-+-+      +---+
    #     |a|b|      |   |
    #     +-+-+      | i |
    #     |c|d|      |   |
    #     +-+-+  ->  +---+
    #     |e|f|      | j |
    #     +-+-+      +---+
    #
    #   in_ids are [[a, b, c, d], [e, f]], which is:
    #          [[(0, 0), (0, 1), (1, 0), (1, 1)],
    #           [(2, 0), (2, 1)]]
    #   numblocks are: [(2, 2), (1, 2)]
    #   out_id are [(0, 0), (1, 0)] for [i, j]
    cases = [  # fused_ids, new_numblocks, axes of reduction
        [[[(0,)], [(0, 1)]], (1, 1), (1,)],
        [[[(0, 1), (2,)], [(0, 1)]], (2, 1), (0, 1)]
    ]

    answers = [  # in_ids_list, numblocks_list, out_id_list
        [
            [[(0, 0), (0, 1)]],
            [(1, 2)],
            [(0, 0)]
        ],
        [
            [[(0, 0), (0, 1), (1, 0), (1, 1)],
             [(2, 0), (2, 1)]],
            [(2, 2), (1, 2)],
            [(0, 0), (1, 0)]
        ]
    ]
    # Yields (fused_ids, new_numblocks, axes), answer
    return zip(cases, answers)

def matcher_idgen_verify(idgen, answer):
    in_ids_list = []
    numblocks_list = []
    out_id_list = []
    for a, b, c in idgen:
        in_ids_list.append(a)
        numblocks_list.append(b)
        out_id_list.append(c)
    result = [in_ids_list, numblocks_list, out_id_list]

    assert result == answer

def test__matcher_idgen():
    gen = matcher_idgen_case_generator()
    for (fused_ids, new_numblocks, axes), answer in gen:
        idgen = module._matcher_idgen(fused_ids, new_numblocks)
        matcher_idgen_verify(idgen, answer)

def test_Strategy():
    # Test make_matcher method of Strategy
    # The two cases described in matcher_idgen_case_generator
    x0 = da.from_array(ndarange(2, 2), ((2,), (1, 1)))
    x1 = da.from_array(ndarange(3, 2), ((1, 1, 1), (1, 1,)))
    cases = [
        [x0, 1, 2, [[(0,)], [(0, 1)]]],
        [x1, (0, 1), {0: 2, 1: 2}, [[(0, 1), (2,)], [(0, 1)]]]
    ]
    for x, axis, split_every, expected_fused_ids in cases:
        split_every = 4
        p = module.Parameters(x, axis, split_every)
        s = module.Strategy(p)
        result_matcher = s.make_matcher(x.numblocks)
        assert result_matcher.fused_ids == expected_fused_ids

def test_Matcher():
    gen = matcher_idgen_case_generator()
    for (fused_ids, new_numblocks, axes), answer in gen:
        s = module.Matcher(fused_ids)
        # Assert that attributes are correct
        assert s.fused_ids == fused_ids
        # Assert that idgen method is correct
        idgen = s.idgen()
        matcher_idgen_verify(idgen, answer)
        # Assert that extra names conserve
        s2 = module.Matcher(fused_ids)
        assert s.extra_names == s2.extra_names
        # Assert that equality works
        assert s == s2

def test__blockreduction():
    def func(t):
        return t[0] + 1, t[1] + 2

    # Create input JointBlockArray
    a = ndarange(2, 3)
    chunks = ((1, 1), (1, 1, 1))
    darr = da.from_array(a, chunks)
    da_list = [darr, darr.copy()]
    in_jb_arr = jba.from_dask_arrays(da_list)

    # Create idgen and new_numblocks
    fused_ids = [[(0,), (1,)], [(0, 1), (2,)]]
    new_numblocks = module._sub_len(fused_ids)
    idgen = module._matcher_idgen(fused_ids, new_numblocks)

    # Apply block reduction
    merge_func = jba.jointblock_concatenate

    name = tokenize(in_jb_arr.name, func, merge_func, fused_ids)

    out_jb_arr = module._blockreduction(func, merge_func, in_jb_arr,
                                        idgen, new_numblocks, name)

    # Assert key conserves
    idgen2 = module._matcher_idgen(fused_ids, new_numblocks)
    out_jb_arr2 = module._blockreduction(func, merge_func, in_jb_arr,
                                         idgen2, new_numblocks, name)
    assert same_keys(out_jb_arr, out_jb_arr2)

    # Check if dask arrays extracted from the output JoitBlockArray are as
    # expected
    chunks = ((1, 1), (2, 1))

    for i in range(2):
        result = out_jb_arr.extract(i, chunks, np.int)
        answer = darr + i + 1
        assert_eq(result, answer)

        # Assert key conserves
        result2 = out_jb_arr2.extract(i, chunks, np.int)
        assert same_keys(result, result2)

def test_blockreduction():
    def func(t):
        return t[0] + 1, t[1] + 2

    def blockreduce(jb_arr, fused_ids):
        matcher = module.Matcher(fused_ids)
        return module.blockreduction(func, jb_arr, matcher)

    def check_value(result, chunks, darr, shift_func):
        for i in range(2):
            r = result.extract(i, chunks, darr.dtype)
            ans = darr + shift_func(i)  # chunks of ans and r are different.
            assert_eq(r, ans)

    def check_keys(result1, result2, new_chunks, darr):
        assert same_keys(result1, result2)
        for i in range(2):
            r1 = result1.extract(i, new_chunks, darr.dtype)
            r2 = result2.extract(i, new_chunks, darr.dtype)
            assert_eq(r1, r2)
            assert same_keys(r1, r2)

    def check(jb_arr, fused_ids, darr, new_chunks, shift_func):
        result = blockreduce(jb_arr, fused_ids)
        check_value(result, new_chunks, darr, shift_func)
        result1 = blockreduce(jb_arr, fused_ids)
        check_keys(result, result1, new_chunks, darr)
        return result, result1

    def check_main():
        # Create input JointBlockArray
        arr = ndarange(3, 2)
        darr = da.from_array(arr, 1)
        jb_arr = jba.from_dask_arrays([darr, darr.copy()])

        # -- Check call once --
        fused_ids0 = [[(0, 1), (2,)], [(0, 1)]]
        chunks0 = ((2, 1), (2,))
        ans0 = lambda i: i + 1
        result0 = check(jb_arr, fused_ids0, darr, chunks0, ans0)

        # -- Check call twice --
        fused_ids1 = [[(0, 1)], [(0,)]]
        chunks1 = ((3,), (2,))
        ans1 = lambda i: i * 2 + 2
        check(result0[0], fused_ids1, darr, chunks1, ans1)

        # Assert key conserves
        check_keys(result0[0], result0[1], chunks0, darr)

    check_main()

def test__find_tree_depth():
    # Test with power of base
    base = 2
    n = 10
    answers = np.arange(n, dtype=int) + 1
    powers = base ** answers
    numblocks = tuple(powers.tolist())

    split_every = {}
    for i in range(n):
        split_every[i] = base
        tree_depth = module._find_tree_depth(numblocks, split_every)
        assert tree_depth == answers[i]
        assert isinstance(tree_depth, int)

    # Test with cases
    cases = [
        [(6,), {0: 2}, 3],
        [(6, 3), {0: 2, 1: 2}, 3],
        [(), {0: 2, 1: 2}, 0]  # For 0-dimensional array (a single number).
    ]
    for numblocks, split_every, answer in cases:
        result = module._find_tree_depth(numblocks, split_every)
        assert result == answer

def test_StopCondition():
    base = 3
    for delta in [-1, 0, 1]:
        for depth in range(0, 5):
            n_leave_nodes = base ** depth + delta
            if n_leave_nodes < 1:
                n_leave_nodes = 1
            arr = np.arange(n_leave_nodes, dtype=int)
            x = da.from_array(arr, 1)
            p = module.Parameters(x, None, split_every=base)
            stop_cond = module.StopCondition(p)

            width_now = n_leave_nodes
            skip_loop = True
            while stop_cond.check():
                skip_loop = False
                width_now = ceil(width_now * 1.0 / base)  # perform reduction
                assert width_now > 1
            width_now = ceil(width_now * 1.0 / base)  # last reduction
            assert width_now == 1
            if n_leave_nodes < base:
                assert skip_loop

def test__offset_gen():
    # Offset is the coordinate within the overall dask array and for the 0th
    # element of each block. For a dask array darr with blocks of single
    # elements and containing (0, 1, 2, ...), darr[i^{th} offset] == i.
    shapes = [(), (5,), (2, 3), (8, 3), (4, 10, 3), (9, 7, 11), (2, 3, 5, 7)]
    for shape in shapes:
        # Create input dask array
        arr = ndarange(*shape)
        darr = da.from_array(arr, 1)

        # Create offset generator
        offset_gen, overall_shape_gen = module._offset_gen(darr)

        # Assert the generators yield correct values.
        gen = enumerate(zip(offset_gen, overall_shape_gen))
        for i, (offset, overall_shape) in gen:
            assert arr[offset] == i
            assert overall_shape == arr.shape

def test_argmap_jointblock():
    def f(arr, offset, overall_shape):
        a = arr + 1
        # Add extra dimension to make all blocks have two dimensions.
        b = np.array(offset)[np.newaxis, :]
        c = np.array(overall_shape)[np.newaxis, :]
        return a, b, c

    # -- Check individual cases --
    shape = (4, 5)
    arr = ndarange(*shape)
    darr = da.from_array(arr, chunks=2)
    result = module.argmap_jointblock(f, darr)

    # Check result
    chunks12 = ((1, 1), (2, 2, 2))
    ans1 = np.array([[0, 0, 0, 2, 0, 4], [2, 0, 2, 2, 2, 4]])
    ans2 = np.array([list(shape) * 3] * 2)
    answers = [(darr.chunks, darr + 1), (chunks12, ans1), (chunks12, ans2)]
    for i, (chunks, ans) in enumerate(answers):
        r = result.extract(i, chunks, np.intp)
        r2 = result.extract(i, chunks, np.intp)
        assert same_keys(r, r2)  # Assert keys are conserved
        assert_eq(r, ans)  # Assert values are correct

# -- Reduction -----------------------------------------------------------------
def keepdim2d(val, axis, keepdims=True):
    if keepdims:
        if axis == 0:
            shape = (1, len(val))
        elif axis == 1:
            shape = (len(val), 1)
        elif axis is None:
            shape = (1, 1)
        else:
            shape = val.shape
        val = val.reshape(shape)
    return val

def min2d_prep(x, axis=None):
    val = np.min(x, axis=axis)
    return keepdim2d(val, axis, keepdims=True),  # return a tuple

def min2d_func(jointblock, axis=None, keepdims=True):
    old_val = jointblock[0]
    val = np.min(old_val, axis=axis)
    val = keepdim2d(val, axis, keepdims)
    return val,  # return a tuple

def min2d_fixer(p, jb_arr, dtype=None):
    axis = p.axis
    if axis is None:
        chunks = (())
    else:
        orig_chunks = p.x.chunks
        chunks = tuple(c for i, c in enumerate(orig_chunks) if i != axis)
    return jb_arr.extract(0, chunks, p.x.dtype, drop_axes=p.axes)

def test__tree_reduction2():
    arr = ndarange(13, 17)
    x = da.from_array(arr, 3)
    jb_arr = jba.from_dask_arrays([x])

    for axis in [None, 0, 1]:
        interim_func = partial(min2d_func, axis=axis, keepdims=True)
        final_func = partial(min2d_func, axis=axis, keepdims=False)

        p = module.Parameters(x, axis)
        strat = module.Strategy(p)
        stop_cond = module.StopCondition(p)

        reduced = module._tree_reduction2(interim_func, final_func, jb_arr,
                                          strat, stop_cond)

        result = min2d_fixer(p, reduced)
        answer = np.amin(arr, axis=axis)
        assert_eq(result, answer)

def test__tree_reduction():
    arr = ndarange(13, 17)
    x = da.from_array(arr, 3)
    jb_arr = jba.from_dask_arrays([x])

    for axis in [None, 0, 1]:
        p = module.Parameters(x, axis)
        func = min2d_func
        fixer = min2d_fixer
        result = module._tree_reduction(func, fixer, jb_arr, p)
        answer = np.amin(arr, axis=axis)
        assert_eq(result, answer)
        # Check if the out keyword works.
        out = da.zeros(answer.shape, dtype=answer.dtype, chunks=result.chunks)
        module._tree_reduction(func, fixer, jb_arr, p, out=out)
        assert_eq(out, result)
        assert same_keys(out, result)

@pytest.mark.parametrize('fixer', [min2d_fixer, module.simple_fixer])
def test_reduction(fixer):
    arr = ndarange(13, 17)
    x = da.from_array(arr, 3)
    for axis in [None, 0, 1]:
        result = module.reduction(min2d_prep, min2d_func, fixer, x, axis=axis)
        answer = np.amin(arr, axis=axis)
        assert_eq(result, answer)
        # Check if the out keyword works.
        out = da.zeros(answer.shape, dtype=answer.dtype,
                       chunks=result.chunks)
        module.reduction(min2d_prep, min2d_func, fixer, x, axis=axis, out=out)
        assert_eq(out, result)
        assert same_keys(out, result)

# -- Test Indexers -------------------------------------------------------------
def test_index_along_axis():
    cases = [
        [np.arange(6), 0, np.arange(3), np.arange(3)],
        [ndarange(3, 4), 1, np.zeros(3, dtype=int),
         np.array([0, 4, 8], dtype=int)],
        [ndarange(3, 4), 1, np.array([[0, 1]] * 3),
         np.array([[0, 1], [4, 5], [8, 9]], dtype=int)]
    ]

    for arr, axis, arg_along, answer in cases:
        result = module.index_along_axis(arr, arg_along, axis)
        assert_eq(result, answer)

    for ndim in range(2, 4):
        shape = np.random.choice(10, size=ndim) + 1
        arr = ndarange(*shape, shuffle=True)

        for axis in range(ndim):
            answer = np.amin(arr, axis=axis)
            arg_along = np.argmin(arr, axis=axis)
            result = module.index_along_axis(arr, arg_along, axis)
            assert_eq(result, answer)

def test_shift_flat_index():
    def check(shape, offset, block_shape, local_flat_id):
        local_id = local_flat_id.ravel()  # ensure it is 1D array
        arr = ndarange(*shape)  # overall array.
        block_sel = [slice(a, a + b) for a, b in zip(offset, block_shape)]
        block = arr[block_sel]
        answer = block.ravel()[local_id]
        global_flat_id = module.shift_flat_index(local_id, block_shape,
                                                 offset, shape)

        result = arr.ravel()[global_flat_id]
        assert_eq(result, answer)

    def check_main():
        cases = [
            [(10, 8), (2, 3), (3, 4), [np.array(2)]],
            [(10, 11, 12), (2, 3, 4), (3, 5, 2),
             [np.array(2), np.array([1, 2, 3])]]
        ]
        for shape, offset, block_shape, local_flat_ids in cases:
            local_flat_id2 = np.arange(np.product(block_shape), dtype=np.intp)
            for lid in local_flat_ids + [local_flat_id2]:
                check(shape, offset, block_shape, lid)

    check_main()

def test_shift_index_along_axis():
    shape = (12, 20)
    arr = ndarange(*shape, shuffle=True)
    blocks2 = np.array_split(arr, 3, axis=0)
    blocks = [np.array_split(b, 4, axis=1) for b in blocks2]
    numblocks = len(blocks2), len(blocks[0])

    axis = 0
    for i in range(numblocks[0]):
        args = []
        vals = []
        for j in range(numblocks[1]):
            block_now = blocks[i][j]
            offset = i * 4, j * 5
            idx = np.argmin(block_now, axis=axis)
            r = module.shift_index_along_axis(idx, offset, axis)
            vals.append(np.amin(block_now, axis=axis))
            args.append(r)
        args = da.core.concatenate3(args)
        result = da.core.concatenate3(vals)
        answer = module.index_along_axis(arr, args, axis)
        assert_eq(result, answer)

    axis = 1
    for j in range(numblocks[1]):
        args = []
        vals = []
        for i in range(numblocks[0]):
            block_now = blocks[i][j]
            offset = i * 4, j * 5
            idx = np.argmin(block_now, axis=axis)
            r = module.shift_index_along_axis(idx, offset, axis)
            vals.append(np.amin(block_now, axis=axis))
            args.append(r)
        args = da.core.concatenate3(args)
        result = da.core.concatenate3(vals)
        answer = module.index_along_axis(arr, args, axis)
        assert_eq(result, answer)

# -- Arg_reduction -------------------------------------------------------------
def argmin2d_prep(arr, offset, overall_shape, axis=None, keepdims=True):
    val = arr
    loc_arg = np.argmin(val, axis=axis)
    global_arg = module.shift_index(loc_arg, val.shape, offset, overall_shape,
                                    axis)
    new_val = np.amin(val, axis=axis)
    return keepdim2d(new_val, axis), keepdim2d(global_arg, axis)

def argmin2d_func(jointblock, axis=None, keepdims=True):
    val = jointblock[0]
    global_arg = jointblock[1]
    new_loc_arg = np.argmin(val, axis=axis)
    new_global_arg = module.apply_index(global_arg, new_loc_arg, axis)
    new_val = module.apply_index(val, new_loc_arg, axis)
    return keepdim2d(new_val, axis, keepdims), keepdim2d(new_global_arg,
                                                         axis, keepdims)

def argmin2d_fixer(p, jb_arr, dtype=None):
    axis = p.axis
    if axis is None:
        chunks = (())
    else:
        orig_chunks = p.x.chunks
        chunks = tuple(c for i, c in enumerate(orig_chunks) if i != axis)
    return jb_arr.extract(1, chunks, p.x.dtype, drop_axes=p.axes)

def test_arg_reduction():
    arr = ndarange(5, 7)
    x = da.from_array(arr, 3)
    for axis in [None, 0, 1]:
        result = module.arg_reduction(argmin2d_prep, argmin2d_func,
                                      argmin2d_fixer, x, axis=axis)
        answer = np.argmin(arr, axis=axis)
        assert_eq(result, answer)

        # Check if the out keyword works.
        out = da.zeros(answer.shape, dtype=answer.dtype, chunks=result.chunks)
        module.arg_reduction(argmin2d_prep, argmin2d_func, argmin2d_fixer, x,
                             axis=axis, out=out)
        assert_eq(out, result)
        assert same_keys(out, result)

# -- Test Routines -------------------------------------------------------------
def reduction_check_data(dtype=np.intp, shape_limit=8, n_cases_per_ndim=3,
                         ndim_limits=(0, 3)):
    """Produce arrays for testing reduction routines.
    Parameters
    ----------
        dtype : np.dtype, str, or optional
            Data type of number array.
        shape_limit : int, optional
            maximal possible size for each dimensions.
        n_cases_per_ndim : int, optional
            produce how many cases for each number of dimensions
        ndim_limits : tuple, optional
            limits of number of dimensions. (0, 3) means ndim can be 0, 1, or 2.
    Yields
    ------
        x : da.Array
        arr : np.ndarray
        axis : int or None
    """
    ndim_range = range(*ndim_limits)
    cases = [ (1, 5), (6, 3)]
    shape_limit = 8  # maximal possible size for all dimensions.
    for i in ndim_range:
        for j in range(n_cases_per_ndim):
            cases += [np.random.choice(shape_limit, size=i) + 2]
    cases += [()]
    for shape in cases:
        ndim = len(shape)
        arr = ndarange(*shape, shuffle=True, dtype=dtype).astype(dtype)
        for chunks in range(2, shape_limit + 1):
            for axis in tuple(range(ndim)) + (None,):
                x = da.from_array(arr, chunks)
                yield x, arr, axis

@pytest.mark.slow
@pytest.mark.parametrize('dtype', ['f4', 'i4'])
def test_sum(dtype):
    for x, arr, axis in reduction_check_data(dtype):
        result = module.sum(x, axis=axis)
        answer = np.sum(arr, axis=axis)
        assert_eq(result, answer)

@pytest.mark.slow
@pytest.mark.parametrize('dtype', ['f4'])
def test_argmin(dtype):
    import dask
    dask.set_options(get=dask.get)
    for x, arr, axis in reduction_check_data(dtype):
        # arr = np.array([[4., 2., 1., 3., 0.]])
        # axis = None
        # chunks = ((1,), (1, 1, 1, 1, 1))
        # x = da.from_array(arr, chunks)
        print("------ shape: %s, axis: %s, chunks: %s ------" % (arr.shape,
                                                                 axis,
                                                                 x.chunks))
        print("arr\n", arr)
        result = module.argmin(x, axis=axis)
        answer = np.argmin(arr, axis=axis)
        from dask.dot import dot_graph
        dot_graph(result.dask, filename='%s.png' % axis)
        print("result chunks:\n", result.chunks)
        print("result dtype:\n", result.dtype)
        print("result blocks:\n")
        print("    ########")
        for idx in np.ndindex(result.numblocks):
            print("idx", idx)
            print("key:", (result.name,) + idx)
            print(dask.get(result.dask, (result.name,) + idx))
        print("    ########")
        print("answer:\n", answer)

        #result.compute()
        #print("result:\n", )
        break
        # assert_eq(result, answer)

tests/test_jointblockarray.py
from __future__ import absolute_import, division, print_function

import pytest

pytest.importorskip('numpy')

import dask.array as da
import numpy as np

from .. import jointblockarray as module
from dask.array.utils import assert_eq as _assert_eq, same_keys
from . import test_reductions as tr

def assert_eq(a, b):
    _assert_eq(a, b, equal_nan=True)

def test_jointblock_concatenate():
    a = tr.ndarange(2, 3)
    b = np.array([[0]])
    c = [(a + i, b + i) for i in range(4)]
    d = tuple(
        np.vstack([np.hstack([ar + 0, ar + 1]),
                   np.hstack([ar + 2, ar + 3])]) for ar in [a, b]
    )

    e = [(np.array([[i]]),) for i in range(8)]
    f = (np.arange(8).reshape(4, 2),)

    g0 = tr.ndarange(3, 2)
    g = [(g0,), (g0 + 2,)]

    cases = [
        [c, (2, 3), d],
        [e, (4, 2), f],
        [g, (1, 2), (np.hstack([g0, g0 + 2]),)],
        [tuple(c), (2, 3), d],  # check if tuple works.
        [tuple(e), (4, 2), f],
    ]

    for jointblocks, numblocks, answer in cases:
        result = module.jointblock_concatenate(jointblocks, numblocks)
        for r, a in zip(result, answer):
            assert_eq(r, a)

def test_from_dask_arrays():
    aa = [da.from_array(np.arange(3), 2), da.from_array(np.arange(4), 2)]
    jb_arr = module.from_dask_arrays(aa)

    for f in [jb_arr._extract, jb_arr.extract]:
        for i in range(2):
            a = aa[i]
            ax = f(i, a.chunks, a.dtype)
            assert_eq(ax, a)
            # Assert name conserves
            ax2 = f(i, a.chunks, a.dtype)
            assert same_keys(ax, ax2)

def test_map_jointblock():
    # -- Without extra_gen --
    def f(x):
        return x + 1,  # returns a tuple

    darr = da.from_array(np.arange(3), 2)
    jb_arr = module.map_jointblock(f, darr)
    result = jb_arr.extract(0, darr.chunks, darr.dtype)
    answer = 1 + darr
    assert_eq(result, answer)

    # Assert that name conserves
    jb_arr2 = module.map_jointblock(f, darr)
    result2 = jb_arr2.extract(0, darr.chunks, darr.dtype)
    assert same_keys(jb_arr, jb_arr2)
    assert same_keys(result, result2)

    # -- With extra_gen --
    def f2(x, y):
        return x + y,  # returns a tuple

    extra_gen = [1, 2]

    jb_arr = module.map_jointblock(f2, darr, extra_gen=[extra_gen])
    result = jb_arr.extract(0, darr.chunks, darr.dtype)
    answer = da.from_array(np.array([1, 2, 4]), 2)
    assert_eq(result, answer)
    result2 = jb_arr.extract(0, darr.chunks, darr.dtype)
    assert same_keys(result, result2)

    # -- Use JointBlockArray as input --
    def f3(x):
        return x[0] + 1, x[1] + 2

    # Notice that all dask arrays in the list should have the same numblocks.
    # This implies the dask arrays have the same number of dimensions.
    dask_arrays = [da.from_array(np.arange(6).reshape(2, 3), 2),
                   da.from_array(np.arange(4).reshape(2, 2), ((2,), (1, 1)))]
    jb_arr_original = module.from_dask_arrays(dask_arrays)

    jb_arr = module.map_jointblock(f3, jb_arr_original)

    for i in range(2):
        darr = dask_arrays[i]
        result = jb_arr.extract(i, darr.chunks, darr.dtype)
        answer = (darr + i + 1)
        assert_eq(result, answer)
        result2 = jb_arr.extract(i, darr.chunks, darr.dtype)
        assert same_keys(result, result2)

    # Assert key is conserved
    jb_arr2 = module.map_jointblock(f3, jb_arr_original)
    assert same_keys(jb_arr, jb_arr2)

def test__reduce_numblocks():
    cases = [
        [(2, 1, 3), (1,), (2, 3)],
        [(1, 2, 3), (0,), (2, 3)],
        [(3, 2, 1), (2,), (3, 2)],
        [(1,), (0,), ()],
        [(), (), ()],
        [(2, 3, 4), (0, 1, 2), ()],
    ]
    for numblocks, axes, answer in cases:
        result = module._reduce_numblocks(numblocks, drop_axes=axes)
        assert result == answer

bug 1: shift_flat_index doesn't preserve the shape of original index.
Make name for task based on an op_name.
Grouped all prints to a temporary diagnostic module.

Here is the repository:
https://github.com/rzu512/dask-array-reductions
I implemented sum, argmin, topk, and argtopk operations with the tree reduction engine.

Neat! Would you be interested in PRing that stuff over?

Before that I think the onus is on us to provide high-level feedback of if
we want to accept a change like this.  In general supporting reductions
across multiple inputs is certainly valuable.  The number of new concepts
added in this work concerns me though.
On Tue, Dec 12, 2017 at 11:17 PM, jakirkham<EMAIL_ADDRESS>wrote:

Neat! Would you be interested in PRing that stuff over?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2953#issuecomment-351285490, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszHkDKiFpO-v-QNB7DQFLnYuDPWjlks5s_14GgaJpZM4Qz59H
.

Probably can merge StopCondition and Strategy because they have to match each other.
Made Strategy class because strategy for merging blocks might be a bit better if it considers chunks of the original dask array. I haven't thought about how to do that yet.

For most operations (such as min and argmin), the current strategy does most work in 0th tree reduction step. Subsequent steps might merge many blocks of only 1 element.
The current implementation of topk uses another strategy: topk on blocks -> merge all chunks -> topk again. This would work if the merge all chunks step doesn't use too much memory. And the memory usage depends on chunks of the original dask array.

The 'Matcher' class is for possible future implementation for operations over dask array with unequal but broacastable numblocks. Not sure if someone will want that though.

Alternatively, a function like np.nditer for the keys to blocks of a dask array would make implementing operation over dask array with broadcastabe numblocks easier. A function can replace the Matcher

The engine uses two parts of the JointBlockArray module: mapping a function over multiple arrays of blocks, and converting an array of blocks to a Dask array with drop_axes.

When the shape of intermediate blocks are unpredictable (e.g. np.unique), this avoids calculating chunks for intermediate array of blocks. The current implementation makes a new Dask array in every tree reduction step, which requires calculating chunks.
The current implementation uses a structured numpy array instead of tuple of blocks. The structured array approach requires copying multiple arrays into a structured array.
Tried to get rid of it by doing everything directly in a Dask graph: https://github.com/dask/dask/issues/2899
Needs an extra getitem step to extract Dask arrays from array of tuple of blocks.
Maybe can replace this class with two functions. But that would needs an argument to keep track of the list of name of each array of blocks.
Using tuple of blocks instead of multiple Dask array reduces number of tasks. The reduction operation have to take multiple blocks and return multiple blocks. Using multiple Dask arrays would require extra getitem steps to extract individual block from the return of the reduction operation.

Had to change the jointblock_concatenate function. The current implementation uses _concatenate2, which can't merge 1D arrays with unequal lengths over multiple axes. I was happy that I made that function. If that function doesn't need any more change, probably can merge it back. But testing a small function is easier than testing a large one.
Made the chunk_indexer module because both the engine and chunk2.argtopk uses the take_along_axis function. Not sure if that is a good idea... I haven't read slice module of dask array and don't know if things in the chunk_indexer module have been implemented.

chunk_indexer._offset_gen generates offset of each block in a dask array. This is more like a method or function over a dask array. Probably the slice module uses something similar, but I am not sure.

The Parameter class holds things that are used in several places.

The original implementation converts from axis to axes and then back to axis in some places. So I just put both axes and axis into the Parameter class. The actual numpy operations use axis. The concatenation of blocks and removal of axes uses axes.
Not sure if this is the right place to check if the axis is legal.

These are the motivations I remember.

Replace Matcher class with fused_ids, which is a tuple. This makes future implementation for reduction over arrays with broadcastable numblocks more difficult.

Building technology to solve potential future problems is generally avoided within the project.  Certainly one should design code that won't get in the way of future decisions, but adding new technology has very real maintenance costs that affect the project immediately.  Everyone maintaining this code will need to learn how it works.  The more classes and concepts we add makes this much much harder.  If you look throughout the Dask codebase there are only a few classes and a small number of concepts in each of the collections.  This has helped the project to grow quickly without a large community of maintainers.

Does Dask already have a class for array of blocks? I don't know how dask.array implement np.unique, which gives blocks with unpredictable shape.

We just use dask.array with np.nan values for unknown chunks.  See http://dask.pydata.org/en/latest/array-creation.html#unknown-chunks

Ok. I will try to simplify the code and reduce the number of classes in my spare time.


termios is linux only

Can't run on windows, since termios is only available on linux. Didn't try WSL, might work on that.

Yeah sorry mate! I only have Linux and unfortunately I have no way to test on windows easily, so i'm afraid I can't really help you there my apologies!

Hey @IDKaccountFlagged
Here's the windows implementation,
https://github.com/hafeezhmha/Automated-AI-Web-Researcher-Ollama/tree/main
check it out and let me know if you find any issue!


Accessibility

Let's add label that can be detected by screen reader. Label shouldn't be visible but if user has screen reader on, it should detect label behind avatar...
Here are places we might need label...

We can wait to rn upgrade. in 0.60 version, they added some accessibility features


Docs: Suggested .vscode/launch.json requires installing vite

Clear and concise description of the problem
Going through the VS Code debugging guide in https://vitest.dev/guide/debugging.html#vscode, when I run F5 on a repo that doesn't have vite listed as a dependency:
joshgoldberg ~/repos/template-typescript-node-package $  cd /Users/josh/repos/template-typescript-node-package ; /usr/bin/env 'NODE_OPTIONS=--require "/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/ms-vscode.js-debug/src/bootloader
.bundle.js"     --inspect-publish-uid=http' 'VSCODE_INSPECTOR_OPTIONS={"inspectorIpc":"/var/folders/sr/j__6_9gn0dv557z0xf3p7y8h0000gn/T/node-cdp.67533-14.sock","deferredMode":false,"waitForDebugger":"","execPath":"/Users/josh/.nvm/versions/node/v18.7.0/bi
n/node","onlyEntrypoint":false,"autoAttachMode":"always","fileCallback":"/var/folders/sr/j__6_9gn0dv557z0xf3p7y8h0000gn/T/node-debug-callback-00402a5d1b51117e"}' /Users/josh/.nvm/versions/node/v18.7.0/bin/node ./node_modules/vitest/vitest.mjs run src/gree
t.test.ts
Debugger attached.
 MISSING DEP  Can not find dependency 'vite'

? Do you want to install vite? › (y/N)

After accepting with y (or running pnpm install vite manually), the debug command works.
Reproduction:

Check out https://github.com/JoshuaKGoldberg/template-typescript-node-package/tree/c122ea1aa2e70ed5ead091b345f2b0a424dcd2bb
pnpm install
Open src/greet.test.ts
Launch the Debug Current Test File command

Suggested solution
I don't know why this is happening enough to suggest a solution 😞 sorry. ./vitest.mjs is the bin entry in node_modules/vitest/package.json.
Alternative
No response
Additional context
Note that pnpm test in that repository works without adding vitest as a dependency.
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

We use isPackageExists from local-pkg library. I guess it doesn't recognize that vite is installed with Vitest 🤔


Suppress display of thermostat's current temperature if data is not available

Make sure you are running the latest version of Home Assistant before reporting an issue.
You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:
Home Assistant release (hass --version):
0.30.2
Python release (python3 --version):
Python 3.5.2
Component/platform:
climate
Description of problem:
I have ZWave thermostats (Danfoss 014G0013) that don't report the current room temperature, i.e.
have "current_temperature": null set permanently.
In the UI the thermostats show up with their current target temperature and a (useless) "Currently: °C" line below each of them.
Expected:
In case the current_temperature is null, the "Currently: °C" line should not appear in the UI view.

Same thing for Qubino devices witch have a slot for connecting a temperature sensor; if that sensor is not connected they still report -999 degrees. Probably we should exclude (temperature) sensors which either report null values or values under -50 or over 50 celsius degrees since anyway their not certified for that range or it means u cant live in those conditions

Has this been resolved?

Nope. But last time I checked (it's been a while though...) the "Currently:" part was hardcoded into the HTML source files, so one would need to remove it from there and have all thermostat types generate the line dynamically (or omit it).
Sebastian


General Issue: derive_var_chg is calculating CHG for baseline records

Background Information
when we are using derive_var_chg function to calculate CHG variable, CHG is calculated for baseline records as well and populating ZERO. Generally CHG is calculated only for post baseline records right. please let me know your thoughts.

Definition of Done
No response

@YaminiBollu , thanks for creating this issue.
derive_var_chg() derives CHG for all observations where BASE and AVAL are not NA. We did not restrict the derivation to post-baseline records because:

Often CHG is not derived for baseline records but sometimes it is.
There is no standard way of identifying post-baseline records.

If you need to restrict the CHG derivation to post-baseline records, you can use restrict_derivation() in combination with derive_var_chg() (see last example for restrict_derivation()).

Hi @YaminiBollu.  I believe @bundfussr's answer is sufficient.  Let us know if you need any other help.  Also, feel free to post in Slack channel if you don't want to make an issue.

Thanks Stefan.. Yes Ben I am good.


SST V3 monorepo template won't run

When I visit this page and follow the steps, I get to the final step, which is to run:
npx sst dev

I initially see:
⠏  Updating dependencies...

But I eventually get:
✕  Could not install dependencies

If I remove node_modules and the .sst folder and run everything from scratch:
npm i

and:
npx sst dev

The error is the same as above.
Do you know what could be happening here and how I can get this working?
UPDATES
System info:
Computer: MacBook Air
Chip: Apple M2
Memory: 24GB
MacOS: Sonoma 14.6.1
Running npx sst install --verbose --print-logs gives, as the last line:
time=2024-09-02T19:00:22.974+03:00 level=ERROR msg="exited with error" err="failed to run bun install "

UPDATE:
This fixes it.


Override arch optimizations

Is there any support or any planned support for being able to override the -march=native flag?  I am currently trying to support a continuous integration and delivery pipeline which begins by checking out application source code to a virtual machine in the cloud, building executables, saving the executables to a docker image, and then deploying the docker image to another cloud environment.
Unfortunately, since -march=native is hard coded in a few different places [1] [2] [3], it means that when my cloud CI provider encounters internal scaling issues due to rapid growth of their platform, they may upgrade their underlying virtual machines with fancier hardware.  In my case, they upgraded to machines with Intel Xeon processors which supported AVX-512 instructions sets.  This led to binaries being compiled with fancy optimizations which later crashed when deployed to our vanilla x86-64 hardware.
For the time being, we are using a fork with those -march=native flags set to -march=x86-64, but it would be nice if those settings could be overridable somehow.  Any thoughts? Could perhaps the configurator in dune be of any use in this situation?
[1]
https://github.com/owlbarn/eigen/blob/master/eigen/dune#L29
[2]
https://github.com/owlbarn/owl/blob/master/src/owl/config/configure.ml#L81
[3]
https://github.com/owlbarn/owl/blob/master/src/aeos/config/configure.ml#L15

We can read environment variables, it would not be hard to allow to override thse by reading them out of OWL_CFLAGS when set. That would work cleanly both with opam and manually compiling.
@ryanrhymes what is your opinion in this respect?

This would probably also provide a fix for https://github.com/owlbarn/owl/issues/354

env variables is a good solution, we can provide default settings, so I agree.

Ok, than this is an easy fix. I will send a PR here and to eigen after #396 is merged

@rdavison with the latest eigen and the current master branch of owl, you should be able to override all the C and CPP flags by setting the right env variables (refer to owl and eigen's readmes for the names and the default values). Does this solve your issue?

@rdavison with he released eigen and owl you can customise it, see
https://github.com/owlbarn/owl/blob/master/docker/Dockerfile.ubuntu and
https://github.com/owlbarn/owl/blob/master/docker/Dockerfile.ubuntu.arm as examples of use. Let me know if this solves your issues

@mseri: Hey, thanks a lot for the quick response!  I wanted to actually test things out before commenting, and I think I've finally gotten everything to work great.  I'm actually using esy as the package manager and was momentarily blocked by it.  I did review the Dockerfiles as they were an excellent source of inspiration, so thank you for that! I'm happy to share the configuration that finally seems to be working for me.  Mostly leaving this here in case other people may need to stumble upon it in the future.
{
  "dependencies": {
    "@opam/owl": "0.5.0"
  },
  "resolutions": {
    "@opam/conf-openblas": {
      "source": "link:./conf-openblas/opam",
      "override": {
        "exportedEnv": {
          "PKG_CONFIG_PATH": {
            "scope": "global",
            "val": "/opt/OpenBLAS/lib/pkgconfig:$PKG_CONFIG_PATH"
          },
          "LD_LIBRARY_PATH": {
            "scope": "global",
            "val": "/opt/OpenBLAS/lib:$LD_LIBRARY_PATH"
          }
        }
      }
    },
    "@opam/eigen": {
      "source": "git+ssh://git@github.com:owlbarn/eigen.git:eigen.opam#85cbb31",
      "override": {
        "buildEnvOverride": {
          "EIGENCPP_OPTFLAGS": "-Ofast -march=x86-64 -mfpmath=sse -funroll-loops -ffast-math",
          "EIGEN_FLAGS": "-O3 -Ofast -march=x86-64 -mfpmath=sse -funroll-loops -ffast-math"
        }
      }
    },
    "@opam/owl": {
      "source": "git+ssh://git@github.com:owlbarn/owl.git:owl.opam#8819213",
      "override": {
        "buildEnvOverride": {
          "OWL_CFLAGS": "-g -O3 -Ofast -march=x86-64 -mfpmath=sse -funroll-loops -ffast-math -DSFMT_MEXP=19937 -msse2 -fno-strict-aliasing -Wno-tautological-constant-out-of-range-compare",
          "OWL_AEOS_CFLAGS": "-g -O3 -Ofast -march=x86-64 -funroll-loops -ffast-math -DSFMT_MEXP=19937 -fno-strict-aliasing"
        }
      }
    }
  }
}

Thank you very much!

I am going to close the issue, but link to it from the README


feat:responsive navbar

Responsive Navbar for assignment of #127

Description of Feature
Responsive Navbar has been developed for Curious Connect as a component  <Navbar/> . Previously, no such feature had existed in this project, making navigation extremely difficult.
Features incorporated

Highlighting of selected tabs
Navigation across defined routes
Responsive design
Hamburger menu for smaller devices

External libraries used

React Icons - npm install react-icons

Can you make full width

Can you make header full width

Can you make header full width

done

Can you make full width

done

changes made. please mention any changes if required

@thecuriousteam
Requested changes have been made in Navbar component. Please review them and add on any other changes if required

Can you make header full width

done

fixed design issue

@Adhil523 your image is not visible

We are merging your code @Adhil523


clarify separation of high-level and low-level API

The DemeGraph class provides a high-level API for constructing a demographic model. The methods on this class ensure consistency within the demographic model. They also allow omission of detail, when there is a single unambiguous interpretation (or a very sensible default). We use these methods to construct a demographic model when loading a yaml file.
In contrast, we also have the low-level "building block" classes in demes/demes.py, such as Deme, Epoch, Migration, etc. that are used by the DemeGraph methods. We gaurantee correctness only when they are constructed via the DemeGraph. It's possible to instantiate these outside of the context of a DemeGraph, and so these classes do not themselves provide any gaurantees about correctness.  But it is sometimes convenient to use these classes directly, in part because they don't complain when you omit information required by a strictly-correct DemeGraph. E.g. when converting from msprime demographies, we iterate through the msprime.DemographyDebugger.epochs and we can instantiate some of these objects without complete information, and add more information as it becomes available. Developers might find other uses. General users (i.e. model implementers) should not be using this low-level API.
Probably this low/high separation is mostly a matter of documention. But there might currently be some edge cases where we blur the semantics (maybe in the Deme class, which has an add_epoch() method?). We should identify these and fix the API(s).

I think it's mainly a documentation thing. It's probably a case where "documentation-driven-development" would help - as you try to explain the distinctions, ways to clarify and make things simpler could occur.

No longer relevant after #208.