High Level Expressions

Summary
We should make space for high level query optimization.  There are a couple of ways to do this.  This issue includes motivation, a description of two approaches, and some thoughts on trade-offs.
Motivation
There are a variety of situations where we might want to rewrite a user's code.
Dataframes

Column projection, as in dd.read_parquet(...)["x"] -> dd.read_parquet(..., columns=["x"])
Predicate pushdown (same as above)
High level expression fusion (what we do today with blockwise)
Pushing length calls down through elementwise calls
Pushing filters earlier in a computation
...

Arrays

Automatic rechunking at the beginning of a computation based on the end of the computation
Slicing

History
Today there is no real place where we capture a user's intent or their lines of code.  We immediately create a task graph for the requested operation, dump it into a mapping, and create a new dask.dataframe.DataFrame or dask.array.Array instance.  That instance has no knowledge of what created it, or what created the other input dataframes on which it depends.
This was a good choice early on.  It made it easy for us to quickly implement lots of complex operations without thinking about a class hierarchy for them.  This choice followed on from the choices of Blaze, where we started with high level expressions, but got a bit stuck because they constrained our thinking (and no one really cares about high level query optimization for a system that they don't use.
However today we have maybe reached a point where our "keep everything low-level and simple" strategy has hit a limit, and now we're curious about how best to bolt on a high level expression system.  Doing this smoothly given the current system is hard.  I see two ways out.
High Level Graph layers
We do have a record of what operations came before us in the high level graph layers.  Currently the API of layers is very generic.  They must be a mapping that adheres to the Dask graph spec, and they must be serializable in a certain way.  There are some consistent subclasses, like blockwise, which enable high level optimizations which have proven useful.
There isn't really much structure here though, and as a result it's hard to do interesting optimizations.  For example it would be nice if we could change a layer at the very bottom of the graph, and then replay all of the operations on that input over again to see how they would change.  High level layers today don't have enough shared structure that we know how to do this.
I like High Level Graph Layers because it gives us a space to hijack and add in all sorts of complex machinery, but without affecting the user-facing Dataframe class.  We would have to add a lot more structure here though, and we'll always be working around the collection class, which is a drawback.
Collection subclasses
I'm going to focus on an alternative that is a bit more radical.  We could also have every user call generate a DataFrame subclass.  There would still be a DataFrame instance that took in a generic graph/divisions/meta, but that would be mostly for backwards compatibility.  Instead most dataframe operations would produce subclasses that had a well-defined common structure, as well as more custom attributes for their specific operation.  Let's look at a couple of examples.
# API calls just create instances.  All logic happens there.
def read_parquet(file, columns, filters):
    return ReadParquet(file, columns, filters)

class ReadParquet(DataFrame):
    args = ["file", "column", "filters"]  # List of arguments to use when reconstructing
    inputs = []  # List of arguments that are DataFrame objects

    def __init__(self, file, columns, filters):
        self.file  = file
        self.columns = columns
        self.filters = filters

        self.divisions, self.meta = # do a bit of work on metadata

    def _generate_dask_layer(self) -> dict:
        ...

class ColumnProjection(DataFrame):
    args = ["dataframe", "columns"]
    inputs = ["dataframe"]

    def __init__(self, dataframe, columns):
        self.dataframe = dataframe
        self.columns = columns
        self._meta = self.dataframe._meta[columns]

    def _generate_dask_layer(self) -> dict:
        ...

class Add(DataFrame):
    args = ["left", "right"]

    def __init__(self, left, right):
        self.left = left
        self.right = right
        self.inputs = []
        if is_dask_collection(left):
            self.inputs.append("left")
        if is_dask_collection(right):
            self.inputs.append("right")

        self._meta = ...
        self._divisions = ...

    def _generate_dask_layer(self) -> dict:
        ...

As folks familiar with SymPy will recall, having attributes like args/inputs around makes it possible to re-generate a DataFrame automatically.  So if we do something like the following:
df = dd.read_parquet(...)
z = df.x + df.y

Then this turns into an expression tree like the following:
Add(
    ColumnProjection(
        ReadParquet(..., columns=None),
        "x",
    ),
    ColumnProjection(
        ReadParquet(..., columns=None),
        "y",
    ),
)

We can then traverse this tree (which is easy because we have a list of all attributes that are dask collections in the inputs attribute) and apply optimizaitons (which is easy because we can easily reconstruct layers because we have the args attribute).
For example the ColumnProjection class may have an optimization method like the following:
class ColumnProjection(DataFrame):
    ... # continued from above

    def _optimize(self) -> DataFrame:
        if isinstance(self.dataframe, ReadParquet):
            args = {arg: getattr(self.dataframe, arg) for arg in self.dataframe.args}
            args["columns"] = self.columns
            return ReadParquet(**args)._optimize()

        # here is another optimization, just to show variety
        if isinstance(self.datafarme, ColumnProjection):  # like df[["x", "y"]["x"]
            return ColumnProjection(self.dataframe.dataframe, self.columns)._optimize()

        # no known optimizations, optimize all inputs and then reconstruct (this would live in a superclass)
        args = []
        for arg in self.args:
            if arg in self.inputs:
                arg = getattr(self, arg)._optimize()
            else:
                arg = getattr(self, arg)
            args.append(arg)

        return type(self)(*args)

This is just one way of doing a traversal, using a method on a class.  We can do fancier things.  Mostly what I wanted to show here was that because we have args/inputs and class types it's fairly easy to encode optimizations and rewrite things.
What's notable here is that we aren't generating the graph, or even any semblance of the graph ahead of time.  At any point where we run code that requires something like _meta or _divisions from an input we stop all opportunities to change the graph under that stage.  This is OK for our internal code.  We can defer graph generation I think.
I think that the main advantage to this approach is that we can easily reconstruct expressions given newly modified inputs.  I think that this is fundamentally what is lacking with our current HLG layer approach.
However, this would also be a significant deviation from how Dask works today.  It's likely that this would affect all downstream projects that subclass Dask arrays/dataframes today (RAPIDS, Pint, yt, ...).  I think that that's probably ok.  Those groups will, I think, understand.
Personal Thoughts
I've historically been against doing collection subclasses (the second approach), but after thinking about this for a while I think I'm now more in favor.  It seems like maybe we've arrived at the time when the benefits to doing this outweigh the complexity costs.  I think that this is motivated also by the amount of complexity that we're encountering walking down the HLG path.
However, I do think that this will be hard for us to implement in an incremental way.  If we want to go down this path we probably need to think about an approach that lets the current DataFrame API persist for backwards compatibility with downstream projects (they wouldn't get any benefits of this system though) and a way where we can move over collection subclasses incrementally.
cc @rjzamora @jcrist

Thank you for taking the time to think through this problem and write this up.  I 100% agree that it will be a challenge to get what we really want out of the Layer-based approach.  As you stated: “Today there is no real place where we capture a user's intent or their lines of code. We immediately create a task graph for the requested operation, dump it into a mapping, and create a new dask.dataframe.DataFrame or dask.array.Array instance”.  As you know, this limitation has been driving me a bit crazy, because it feels like the “least-disruptive” solution is still very disruptive (likely requiring us to add a lot more complex machinery and structure to the existing HLG Layers).
For the reasons above, I am certainly open to the idea of making radical changes to the way Dask-collections operate.  To me, the only important consideration that you haven’t really discussed is: How should collection subclasses interact with HighLevelGraph Layers (if at all)? The general purpose of the HLG is currently two-fold: (1) To provide a “high-level” optimization mechanism, and (2) to avoid communicating a fully-materialized graph over the wire during distributed execution.  It seems like collection-subclasses can address both of these requirements.  However, we learned over the past year that enabling materialization on the scheduler can significantly limit the kind of logic that we can include in the corresponding Layer class.  Therefore, it seems reasonable that we will want ReadParquet._generate_dask_layer to produce a Layer object on the client (rather than a dict).
Since we will probably still need to deal with all the HLG materialization headaches with the new collection-subclass approach, I do want to establish that the Layer-focused approach would probably require about the same level of effort to achieve the same functionality. I say this, because we could still attach the same inputs and args information to the Layer class and regenerate a collection by traversing a tree in a very similar way. With that said, I do like the idea of separating the “state” of the collection from the HLG.  Also, I think I like the idea of using collection subclasses on the client, while continuing to use the HLG representation for communicating the graph to the scheduler.

If we had a good high level expression system on the collections side, what do you think the right design around layers would be?  Some of this stuff becomes redundant.

Right - exactly. It feels like the HLG becomes redundant unless we preserve a stripped-down version for serialization/materialization purposes only.  However, in the end the most-maintainable structure would probably remove HLGs altogether and simply require the same “clean” graph-materialization logic we already require in Layers.

The scheduler could grow a def update_collection_graph method or something similar that was a peer to the update_graph_hlg method.  I think that it's reasonable to think broadly here for a while.

I really like the idea of using collections this way.  Historically have people proposed other approaches to this issue that were considered too high-level at the time?

People have certainly said things like "it would be great if Dask could do predicate pushdown", but I'm not sure that there has ever been a fully thought out proposal on expressions in Dask.  The efforts I've played with / proposed in the past have been for layers surrounding Dask.  That adds more complexity though.

I think that it's reasonable to think broadly here for a while.

I totally agree - I am honestly very attracted to the idea of converting the collections themselves into something closer to symbolic expressions, and I like that you are thinking “radical.” In the previous two comments I was just making an honest attempt to play “devil’s advocate.”  I want to make sure we are acknowledging the fact that we will need to address similar materialization/serialization challenges to the ones we have faced with Layers.  Also, since we have already faced this for Layers, it is worth considering that the HLG class structure could also be redesigned as high-level expressions in a similar way.
Of course, I do expect that the long-term user API would be much more elegant with the collection approach.  It is actually somewhat similar to what NVTabular is currently doing (the “high-level” column-group dependencies and computations are defined separately from Dask).

I made the simplest possible system that I could think of in https://github.com/dask/dask/pull/7948.  It supports elementwise unary operations, binary operations, and  column selection.  It is also a valid dask collection and produces high level graphs.
I don't like the way we use args/inputs, but these aren't important until we get optimizations around.  Other folks can probably suggest something smoother here.

So I think that the approach taken in that PR is somewhat smooth, in that it doesn't ask us to give up on HLGs as the communication mechanism to the scheduler.  We can shift things over smoothly without doing a massive disruptive change.  Then once we're ready we can change how we send things over.
This doesn't fix any of the dask layer serialization stuff, but it also doesn't break it any further :)
In the meantime we get to solve things like your parquet metadata length problem.

We can shift things over smoothly without doing a massive disruptive change

Nice!

In the meantime we get to solve things like your parquet metadata length problem.

Woohoo!

I looked briefly at what it would take to implement reduction/apply_concat_apply and in general it seems like most of our methods are of the form ...

Scrape some metadata from the input dataframes (meta, divisions, ...) and make new meta and divisions
Do a ton of complex work to make an intesting graph layer
Wrap that into a HighLevelGraph
Wrap that into a new dataframe object

Really, we want to peel off layers 3 and 4, and just copy over step 1 into __init__ and step 2 into _generate_dask_layer.  The main difference of the design is that these stages to all of our methods are broken up, and so we can easily redo 1 many times as we perform optimizations and such, and only do 2-4 when we're ready to compute.

I think an expression system for dask arrays (and/or dataframes) sounds great!
One question is how closely tied this should be to Dask itself. I can see advantages (especially backwards compatibility) to using subclasses of existing dask data structures, but on the other hand I can see an abstract expression system for arrays potentially being useful for much more than only Dask.

I agree that a general abstract expression system for arrays would be
valuable.  If such a system existed then we would support its use.  I see a
path of modest cost to add this to Dask array/dataframe/bag and so barring
the existence of such a full solution that's probably what I would do.  I
do not personally intend to implement a full independent symbolic system
(at least not without a lot of support).
On Thu, Jul 29, 2021 at 3:09 PM Stephan Hoyer @.***>
wrote:

I think an expression system for dask arrays (and/or dataframes) sounds
great!
One question is how closely tied this should be to Dask itself. I can see
advantages (especially backwards compatibility) to using subclasses of
existing dask data structures, but on the other hand I can see an abstract
expression system for arrays potentially being useful for much more than
only Dask.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/7933#issuecomment-889492905, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTBAUT22FNCM5JK5WDDT2HGP3ANCNFSM5A6YCJQA
.

In https://github.com/dask/dask/pull/7948 I tried adding this sort of system directly into the existing class.  It's not great, but also not terrible.  It does allow for smooth changes and does allow us to reuse a lot of our code (I moved over the elemwise and aca functions, which account for a great deal of our methods).

OK, so taking a step back here we have a few classes with a few different requirements and considerations
Collections
Documented here
These require methods like __dask_graph__, __dask_keys__, and have methods like __dask_optimize__. In general these carry and manipulate dask graphs, which are mappings of individual tasks.
Subclasses of these typically also include attributes that show elementwise structure, namely _meta, as well as collection-wide structure, namely

Bag: npartitions
Dataframe: divisions
Array: chunks

Expr
These contain enough information to quickly recreate themselves (args) as well as information to traverse down to dependencies (dependencies).  Using these, as well as a class hierarchy, they could implement their own optimization methods.  It is important that little work is done on initialization, because we reserve the right to re-create these things frequently.
Can we do everything at once?
Sure!  But we need to be a little bit careful.
In principle we could take the existing dask collection functions, turn them into classes which in the constructor do all of the following:

store the args/dependencies on the object (required by expr)
store the meta, divisions/chunks, and graph on the object (required by collections)

Step 2 here is what is already written, so adding this into the constructor would be the easiest path, but this can also be slow.  Instead we want to move the slow parts of 2 to some other function (currently named _generate_dask_layer) but we want to keep the structural parts of 2 (meta, divisions/chunks) in the constructor like the following:

On initialization: Create args/dependencies for expr and meta/divisions for collection
On demand later on: Create graph

The challenge here is that some of our code creates the divisions and the graph and the meta at the same time.  We need to tease apart this creation.  I suspect that this is going to be possible almost all of the time, and in the other times we can always just create and store the graph in the constructor (slow, but not the end of the world).  Really, we just need to go through and tease apart all of our methods.  I think that calling out these necessary pieces of metadata is useful.

For args/dependencies I've gone through a few different options of representation.
As a reminder we need these for the following reasons:

args: we need to be able to recreate an object with different inputs.  Something like the following should always work
new = type(obj)(*obj.args)
assert new == obj

dependencies: we need to be able to see the other dask expressions on which this one was based so that we can traverse down this tree when optimizing

How do we store these things?
I first started with the names of attributes, like the following:
class BinOp:
    args = ("operator", "left", "right")
    dependencies = ("left", "right")

    def __init__(self, operator, left, right):
        ...

(we should traverse down left and right, but don't need to traverse down the operator, which is unlikely to be a dask expression.
However this is challenging because there are a lot of getattr calls, and often the dependencies at least aren't known ahead of time.  For example in the case of x + 1, where expr.right is not a dask collection.
So in the more recent implementation when I change the dd._Frame class directly I use attributes and properties
class Binop:
    def __init__(self, operator, left, right):
        ...

    @property
    def args(self):
        return self.operator, self.left, self.right

    @property
    def dependencies(self):
        out = []
        if is_dask_collection(self.left):
            out.append(self.left)
        if is_dask_colleciton(self.right):
            out.append(self.right)
        return out

This seemed to work better.
I also found that using only *args was limiting, because in many cases we encourage the use of **kwargs.  I wonder if instead we want the invariant new = type(obj)(*obj.args) to instead be new = type(obj)(**obj.parameters), where we use a dictionary with named parameters instead.  This seems more predictable.

In dask dataframe I ran into problems with subclassing.  Currently we have four classes, DataFrame, Series, Index, and Scalar.  (More if you include RAPIDS and GeoPandas).  I found that I had to make classes like ElemwiseDataFrame, ElemwiseSeries, and so on.  These were very simple definitions
class ElemwiseDataFrame(ElemwiseBase, DataFrame):
    pass

But still it was a code smell that pointed to some of the bad behaviors of object inheritance.  I suspect that this approach would put us in even more pain when it came to downstream projects like RAPIDS/GeoPandas, and also maybe Pint.
A common approach in this case is to switch from inheritance to encapsulation.  I don't yet have a clean design for that.

@quasiben @rjzamora and I spoke this morning (along with some other folks).  It looks like @rjzamora has time and interest to look into executing something like this starting around one-to-two weeks from now.  I would like for folks here to come to some consensus on a path forward before then.

This has been fun to read (along with https://github.com/dask/dask/pull/7948/). I'm really interested in helping in the effort. w.r.t. consensus on a path forward: do you mean having an answer to the handful of proposed questions that you've laid out in the issue so far? Like property vs. attribute args/dependencies, *args vs. **parameters, and inheritance vs encapsulation? Or something more broad, like incremental implementation vs all at once? Or all of the above!

consensus on a path forward: do you mean having an answer to the handful of proposed questions that you've laid out in the issue so far

Mostly I'm looking to hear from people who have different experiences than me and who can anticipate issues that I haven't.  My expectation is that this will mostly come from some of the other grizzled maintainers :)

I'm really interested in helping in the effort.

I'm glad to hear this.  My sense is that when we first start there will lots of false starts and redesigns that will be frustrating for anyone who is not familiar with and comfortable with tearing apart Dask's design.  (this is what I'm going through now).  However, I think that once we find a path that seems to be working then there will be a lot of work that I think many people could probably participate in in parallel more effectively.
In particular, I expect @rjzamora to focus his work on Dask Dataframe (due to the RAPIDS connection) but probably not invest as much effort in Dask array.  Doing the rewrite process within Dask array would probably be a good month-long project for someone if they were interested.

I agree that a general abstract expression system for arrays would be valuable.

That sounds a bit like what metadsl was intended to support? Unfortunately development has stalled there but think the concept has a lot of merit so I'm very interested to see this proposal evolve...

In particular, I expect @rjzamora to focus his work on Dask Dataframe (due to the RAPIDS connection) but probably not invest as much effort in Dask array.

Yea I've sensed that this is the case

Doing the rewrite process within Dask array would probably be a good month-long project for someone if they were interested.

Happy to target this. Will keep an eye on the path planning

I think that this design would be incompatible with dask.graph_manipulation and any other (third party?) code that perform collection-agnostic graph alterations. The way that module works is:

fetch the HLG from the collection with __dask_graph__()
add its own layers on top
rebuild the collection using __dask_postpersist__(..., rename={old_collection_name: new_collection_name}))

The module has no clue about which specific collections it's working with.
Without HLG, it would be forced to completely materialise the graph client side - or am I missing something?

I gave encapsulation some thought today.  I didn't actually do any work on it though, so these thoughts may be completely silly once we hit actual code.
There is an open question of where all of the necessary metadata sits.  This metadata includes task graph construction, meta object, divisions, expression tree, and the dataframe/series/index API type.
The challenge of subclassing is that we want the API to depend strongly on the dataframe/series/index API type, so that needs to have a strong influence on the class.  Adding the expression tree into that typing structure creates two different dimensions and so an cartesian product of subclasses.
So we leave the top-level class to the API type (dataframe/series/index) and have that contain an expression
class FrameExpr:
    ...

class DataFrame:
    expr: FrameExpr
    ...

While all of the graph, metadata, divisions, etc need to be accessible from the DataFrame/Series/Index classes, it's not clear where the logic for these things should live.  Should they live on the DataFrame/Series/Index classes (I'm going to start calling these user API classes) or should they live on the expression classes (Elemwise/IO/ReadParquet/Reduction/...)?  This is an open quesiton.
My gut reaction right now though is that everything should go on the FrameExpr.  This leaves the user API classes to be quite  simple
class DataFrame:
    def __init__(self, expr):
        self.expr = expr

    @property
    def _meta(self):
        return self.expr._meta

    ... # and so on

This ends up being a bit less clean from an implementation perspective.  We need to change all calls to functions like new_dd_object, which is unpleasant.
Backwards compatibility
Or not?
We should have a fallback, an expression that contains a task graph/meta/divisions without provenance.
class LeafFrame(FrameExpr):
    def __init__(self, dsk, name, meta, divisions):
        ...
    @property
    def args(self):
        return ()

    @property
    def dependencies(self):
        return ()

And then current API points that create dask dataframes, like new_dd_object will have a legacy mode
def new_dd_object(dsk=None, name=None, meta=None, divisions=None, expr=None):
    if expr is None:
        expr = LeafFrame(dsk, name, meta, divisions)
    return get_parallel_type(expr._meta)(expr)

I don't think that calling dd.DataFrame is particularly common any more, but we can probably do the same thing there.
Adding actual operations
Functions like elemwise will move their logic into classes, as before, and then they'll create a new_dd_object with that expression
def elemwise(*args, **kwargs):
    expr = Elemwise(*args, **kwargs)
    return new_dd_object(expr)

Mostly I'm looking to hear from people who have different experiences than me and who can anticipate issues that I haven't. My expectation is that this will mostly come from some of the other grizzled maintainers :)

Here are some thoughts from a SymPy maintainer. You'll have to pardon my ignorance of the dask internals. If something I say here doesn't make sense, please let me know.

My first thought when I saw this idea is that the expression system would be separate from the classes. However, Matt makes a good point that this may require keeping the separate things in sync too much. You considered making the collections classes themselves the expression tree classes, and having collections link to separate expression tree classes, but what about the reverse, expression tree classes that contain the collections? Anyway, it's not clear to me yet which is the best option. I honestly don't feel I have enough knowledge of the dask internals to say.

My biggest question here is whether or not these collection objects you plan to make into expression trees are immutable. If they are, you should be good to go. If they aren't, you may run into a lot of headaches. The issue being that if you mutate an expression that may require updating any tree that contains it, but expression tree DAGs point down the tree not up (expressions don't know which other expressions contain them).

Regarding using type(*obj)(**obj.parameters) instead of type(obj)(*obj.args), that sounds good. The only limitation of **parameters is that to support an actual *args signature you'd to also support passing them in as a parameter. But that's should be easily doable with def func(*args, **, _args=()) or something. You also wouldn't be able to use Python 3.8+ positional-only args.

I will say that one thing I have always found nice about the SymPy expression system that I would recommend emulating is that every expression tree can be uniformly walked using .args. Compare this to the Python ast module, which only has attributes, and a list of the attributes. The resulting system is annoying to work with by hand, because you have to know the exact type of each node in order to walk down it. You are effectively forced to use the visitor pattern to examine and manipulate AST objects. But the visitor pattern is only one way to manipulate expressions. It works well when you have different code for each expression tree type, or when you want the visiting operations to be extensible. But it is also quite verbose, as it requires you to write a class for even the simplest manipulation. There are other ways, which are more terse and simpler to reason about. For example, you could just use recursive functions that walk through the tree. Another way, which I know Matt is fond of (because he wrote it), is a functional API like that found in sympy.strategies, where you have a lot of expression manipulation functions that compose nicely, allowing you to write code like bottom_up(f, expr) (see the sympy.simplify.fu module for a large example of code written in this style). Yet another way is based on pattern matching, e.g., using something like MatchPy (which is indicendally the route I would recommend for doing expression optimizations). The point isn't that one way is better than any other. All methods have their valid uses. The point is that the simpler expression tree design doesn't lock you into only one way of thinking about manipulation.
SymPy has the additional restriction that every element of .args should itself be a SymPy class, which has both advantages and disadvantages (I know Matt and I have disagreed about that in the past). The main disadvantage of not making this requirement is that you always have to be on guard for non-expression tree classes when walking an expression tree. It especially hinders the second type of tree walking I mentioned above (manual recursive functions). On the other hand, the main disadvantage of having the requirement is that you have to have classes to wrap basic collection types like tuples or str. And for instance the BinOp class in https://github.com/dask/dask/pull/7948 wouldn't be possible in SymPy. Each operation has to have a separate class (not that that's necessarily a bad thing; it's not clear to me from a pattern matching perspective whether BinOp would be a good idea). Matt's idea of separating args and dependencies might be a clean way of dealing with this issue, assuming I understood correctly the purpose of that distinction (although I would personally change dependencies to just deps or something else that is shorter to type).
I do think you can make the design here simpler than SymPy's in many ways. SymPy uses expr.func(*expr.args) instead of type(expr)(*expr.args) to allow for expressions' head object to not necessarily be its class. But for a simple expression tree design, this is unnecessary complexity. As another point, you should avoid the temptation to do automatic simplification on the expressions at instantiation time, instead preferring to represent exactly what the user input. All simplification, even "trivial" simplifications, should be done separately as optimizations (although any kind of type checking that would lead to an error can be done in the constructors). In SymPy, basic simplifications like x + x -> 2*x happen, but what this means internally is that Add(x, x) has to return Mul(2, x), so the structure requires the use of __new__. Automatic simplification is also a bane for many other reasons. See https://github.com/sympy/sympy/wiki/Automatic-Simplification for a list of some of the problems it can lead to.

Thank you for the comments @asmeurer .  Some responses!

So far I like expressions being in collections.  This keeps the user API the same as it is today.
I think that mutability will be ok.  Expressions can be immutable, but operations on a collection can create and swap out a new expression.  We do this today with Dask graphs.  It is also what Theano did back in the day.
"every expression tree can be uniformly walked using .args" Yup.  Totally agreed.  This is definitely a part of SymPy that I would like for us to steal.
Regarding traversal techniques I think that we can leave this for a future stage.  Just getting expressions, and doing a dead simple optimization routine (maybe just with recursive method calls) seems like a good first step.  We can do lots of things after we have that base.
I like the args/deps separation.  I think that it is a pragmatic choice.  It's been a while, but I think that we might have done this in Blaze?  (my memory is hazy.)  If so then I think it might have worked out well?
I agree with avoiding automatic simplification.  I'd like to avoid the use of __new__ (I suspect that you will agree)

I think that mutability will be ok. Expressions can be immutable, but operations on a collection can create and swap out a new expression. We do this today with Dask graphs. It is also what Theano did back in the day.

It sounds like you are saying that the design of collections containing corresponding expressions makes it so that collections can be mutable with expressions immutable. If collections are indeed mutable, I agree this is the best design.

Regarding traversal techniques I think that we can leave this for a future stage. Just getting expressions, and doing a dead simple optimization routine (maybe just with recursive method calls) seems like a good first step. We can do lots of things after we have that base.

This sounds good. It might be worth getting some basic familiarity with MatchPy to make sure you don't design your way out of the possibility of using it. But other than that, my point is that as long as you have a simple expression tree structure, different kinds of traversal techniques are straightforward to implement.

I think that it is a pragmatic choice. It's been a while, but I think that we might have done this in Blaze? (my memory is hazy.) If so then I think it might have worked out well?

I never worked with Blaze. I'll have to take a look at how it was designed.

This sounds good. It might be worth getting some basic familiarity with MatchPy to make sure you don't design your way out of the possibility of using it. But other than that, my point is that as long as you have a simple expression tree structure, different kinds of traversal techniques are straightforward to implement.

We'll have to make sure that someone who works on this is familiar with MatchPy ;)

I never worked with Blaze. I'll have to take a look at how it was designed.

I wouldn't look too closely :)

Hopefully this isn't too OT, but I wanted to make Dask folks aware that we have an adjacent effort/discussion under way in Apache Arrow to develop a language-independent serialized expression format connecting user SDKs (e.g. things like Ibis or even dask.dataframe potentially) with Arrow-based/Arrow-compatible computing engines:
https://lists.apache.org/thread.html/r90c1e6ba1c7b27a960df3f27dc2c1aeb542f07bb7af72c17138667f0%40<dev.arrow.apache.org>
The idea is to provide a common ground consisting a standard lower-level operators rather than each engine having their own slightly idiosyncratic front end (e.g. using a certain SQL dialect, or something else) — there would be the presumption of using Arrow's schemas / type metadata to describe input/output types of operations.
In the same way that Dask uses pandas as a dataframe computing engine, it would be interesting to see if this High Level Expression effort could lead to a world of pluggable single-node engines for Dask to orchestrate (e.g. using one of the various Arrow-based engines — Rust, C++ — in development or even an small-footprint embeddable analytic SQL engine like DuckDB — which can now input/output Arrow easily). One might imagine that dask.dataframe in the future could not use pandas at all and permit use of alternative (or even multiple) backends to execute various tasks (and if useful, Arrow could be used as a serialization middleware).

Thanks for jumping in @wesm .  That seems like an important topic to me.  I've made a dedicated issue for it at https://github.com/dask/dask/issues/8074
For the purposes of this conversation I think that Dask will need a high level expression system that is separate from Arrow's compute engine expression serialization format.  I don't think that this is what you were suggesting, but I doubt that Arrow would find the things that Dask needs from such an expression system in-scope.

Quick logistical update.  The plan here was for Rick or I to take a first pass at implementing the plan above before Aaron came back from vacation.  However, Rick was sidelined by Parquet issues and I was sidelined by shuffling ones.  As a result we're a little bit behind schedule.
@asmeurer I don't know if you are comfortable enough to try this blind (I imagine that you'll run into a couple of false starts, which is fine by me) of if you want to wait for someone else to take the first step.

How long do you think it would take to do whatever it was you were planning on doing? I have other projects I can work on in the interim, but I'd say I'd probably like to get started no later than next week, so if it would take longer than that, I should probably just try to jump in.

I'm fully supportive of you just jumping in and was hoping that this is
what you were going to propose.  I like this just because it gets us moving
faster.  It does increase your frustration load I think though.  I suggest
that you take on the dev burden here and that you and I check in frequently
this week if that works for you.  I'll follow-up off-list as well.
On Mon, Aug 23, 2021 at 11:59 AM Aaron Meurer @.***>
wrote:

How long do you think it would take to do whatever it was you were
planning on doing? I have other projects I can work on in the interim, but
I'd say I'd probably like to get started no later than next week, so if it
would take longer than that, I should probably just try to jump in.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/7933#issuecomment-903948711, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTHTZMYGIFEMNQ6DIHDT6J457ANCNFSM5A6YCJQA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.

Just to reiterate a starting place I suggest starting with https://github.com/dask/dask/issues/7933#issuecomment-892087327 . This might proceed in a few steps:
Create DataFrameExpr class, support backwards compatibility
We would make a DataFrameExpr and a DataFrameLeafExpr class (better names welcome).
We would then modifying the existing DataFrame/Series/_Frame classes to consume mostly just that expr (as mentioned in the comment above) and change the new_dd_object function to support the old API, but create LeafExprs internally and then hand them to the appropriate API class (dataframe, series, index, ...).
Create non-trivial expressions
Now that everything works but doesn't provide any benefit, we might want to try changing over some of the more base functions.  The biggest one here is elemwise, which is used very often throughout the codebase.
After this would probably be apply_concat_apply, what we use for reductions.
After this would probably be some IO thing, like read_parquet.
Circle back
At this point we've probably learned a lot and we'll want to revisit the design

I think that Dask will need a high level expression system that is separate from Arrow's compute engine expression serialization format. I don't think that this is what you were suggesting

Indeed it wasn't what I was suggesting, so wanted to reiterate that I see these are parallel/complementary efforts, but where each can learn from each other (e.g. we want to make sure that the Arrow serialized expressions can support a lot of the concepts that a dask.dataframe-like interface would produce)

What is the best way to compare two meta objects to see if they are equal?

What is the best way to compare two meta objects to see if they are equal?

You could probably start with assert_eq for now.  We could probably work out a more-streamlined check if one doesn't already exist.

I pushed up my very initial thoughts at https://github.com/dask/dask/pull/8084. I'm probably going to play around with this some more tomorrow, but if you have any initial thoughts on what I am doing before then, I'd love to hear them.

Ok, I've finally had a day to poke deeply at this. I don't have any real code
to show for it (lots of scrapped implementations), but I've come up with a few
design issues/questions we'll want to tackle.
1. Structure of the expression datastructure
There are a few ways to structure an expression datastructure.
Direct references
All of the discussion above represents it as a DAG of Expr node objects, each
of which may contain zero or more direct references to child nodes (as well as
any extra metadata needed).
For example:
from collections import namedtuple

Add = namedtuple("Add", "name, x, y")
Const = namedtuple("Const", "name, value")

a = Const("a", 1)
b = Const("b", 2)
c = Add("c", a, b)
d = Add("d", b, c)

This is nice because any recursive dependency of a node must be a recursive
child reference of that node. This means that if a rewrite pass rearranges a
few nodes in the DAG (or drops a node due to an optimization), the total
expression remains self consistent since the state is always localized
(there aren't extra references we'd have to keep up-to-date/cull).
The downside is that since computations are a DAG not a tree, there can be
multiple references to the same node. Any graph traversal (optimization pass,
low-level-graph generation, ...) would need to be memoized to avoid doing
repeated work. Additionally, any rewrite of a subexpression in the graph would
have to traverse the whole expression to find and substitute out the
subexpression.
Indirect references
A different way of writing the same expression system would be to "flatten" the
structure and use indirect references. Nodes no longer contain direct
references to other nodes, instead they hold a reference to the node names. For
example:
from collections import namedtuple

Add = namedtuple("Add", "x, y")
Const = namedtuple("Const", "value")
ExprDag = namedtuple("ExprDag", "exprs")

ExprDag(
    dict(
        a=Const(1),
        b=Const(2),
        c=Add("a", "b"),
        d=Add("b", "c"),
    )
)

This representation means that rewriting a single subexpression requires no
traversal, since all references are done indirectly through the name->expr
dictionary.
This also makes it easier to merge the expressions for multiple objects during
a compute call. For example, say we call:
dask.compute(c, d)

on the expressions above. With the direct-reference representation we'd need to
create a new node (e.g. TupleExpr(c, d)) to merge the expression dags
together, then optimize the whole structure. With the indirect references, this
is just creating a new ExprDag object with the .exprs dicts merged.
The downside here is that any rewrite that drops a node would
result in dead nodes still remaining in the structure (a pass to cull these
would be required).

The second (indirect references) representation is what we do for the existing
HighLevelGraph objects, except the expressions are called "layers".
After thinking about the pros/cons of each approach, and the types of
operations we'd want to do on the expression system, I'm mildly leaning towards
the second approach.  I think it will be easier to work with for the operations we want
to do (cc @asmeurer, you may have thoughts on this).
But if we go with that approach, why not just modify the existing HighLevelGraph layers
to contain the extra metadata we need (since the overall datastructure is basically the same)?
2. How do HighLevelGraphs fit into this bright new future?
We already tried an expression system with the HighLevelGraph/Layer
objects. These were created to do two things:

Reduce I/O when sending large graphs from the client -> scheduler, since we
could send some layers symbolically rather than fully materializing the graph
on the client side.

Whatever approach we use should be able to handle all the optimizations that
the existing HighLevelGraph system can handle (since if you squint, they're
basically the same but with more metadata attached), so there's no reason IMO
to keep optimizations at that level. So the question is, do we still use
HighLevelGraph layers to reduce IO to the scheduler, or do we directly send
our expression objects instead?
I'm going to argue that for the purpose of reducing I/O, we probably need fewer
types of "Layer" to send between the client and scheduler than we'd need types
of expressions to perform optimizations. This may be clearer with an example.
For example, say for dask.bag we want to rewrite .count() operations before
.map() operations, since bag.map(func1).map(func2).count() is equivalent to
bag.count(). To do this, we'd need a way to distinguish .count() nodes from
other expressions in the tree (perhaps with a BagCount expression, idk).
However, the scheduler doesn't actually care about this information, the
scheduler only cares about the low-level graph. To the scheduler, symbolic
layers are only a cheaper way to communicate the graph structure. So the number
of graph layer representations the scheduler would need to understand is much
lower than the number of expressions the client would need for optimizations.
Separating these representations would make it easier to have mismatches in
Dask versions between the client and scheduler. It also makes it easier for
external libraries to create their own expression nodes, since the expression
classes would only have to exist client side (provided they could be lowered
into one of the representations the scheduler understands).
3. A suggested path forward
After poking at this for a bit, I think this is less of an engineering design
question, and more of an engineering process question. How can we convert an
existing large ungainly system like the existing HighLevelGraph
representation into something that looks kinda similar but contains more info,
without breaking everything or preventing biweekly releases?
I see two paths forward:

Iterate on the existing HighLevelGraph system to add additional layers/info
to layers to support the optimizations we want to do. In the process we'd
likely want to fix-up many of the warts that this system has (for example, I
find the serilization mechanism here quite confusing). This is likely to be
painful as the existing code paths are large, but may be easier to do
iteratively. Not sure.

Create a brand new expression system that looks like the HighLevelGraph
representation from scratch, but does it "right". This representation would
lower to a few existing Layer classes (e.g. Blockwise) for serialization
alone, but we'd want all optimizations to be handled by the new expression
system. Once that's in place, we'd then start to cull back the now extraneous
layers, and cleanup the HighLevelGraph/Layer objects. This has all the
joys and pains of writing new code.

Neither process sounds that enjoyable, since there's a lot of existing code
that will have to be modified.
I'm going to punt on answering the path-forward question and ask those that
have been more involved in the previous HighLevelGraph work to chime in here
with some thoughts (cc @gjoseph92, @rjzamora, @jrbourbeau,
apologies if there are others I should ping, not sure who all was involved).
Either way, I think the compute pipeline we end up with is:

User calls dask.compute(...)
The expressions for all objects are merged into some larger expression so
common sub-expressions can be shared
Optimization passes are applied to this expression, resulting in a new
simpler/more efficient expression.
This expression is then lowered into a serializable graph representation to
send to the scheduler. Ideally no optimizations are performed at this level.
The scheduler receives these layers, and materializes any that were sent as
symbolic representations.

Thank you for the writeup @jcrist .  I've only skimmed it so far, and plan to go through it more thoroughly tomorrow.  However, some quick notes:
I'd like to add a third approach to how to handle dependencies

Use a tree, but be careful about duplicated expressions
Use a named layer approach
(new!) Use a .dependencies/children attribute as in the current plan, but also add a .dependents/parents attribute that goes the other direction.  This would be a Weakset to avoid a reference mess.  Optimizations would then look both at children and parents when deciding what to do

I'm not pushing for one over the other, just adding another option for consideration.
I like the idea of reusing/building on HLGs if we can "do them right".

I think my homework now is to take a closer look at HighLevelGraph, so I can't really say much here until I do that. It does seem to me from what I've done so far that 1) we don't really want expressions to be tied to closely to the container classes and 2) the expression system is very similar to the graph, just with more higher level information. So it might make sense to merge them. OTOH, the expressions being higher level could in fact mean we don't want them to be the same, since that would make it harder to separate high level and low level details. So I think the question boils down to just what it is the high level graph is doing—and isn't doing—compared to the low-level graph.
I will say this about your direct vs. indirect references approaches. The indirect references approach makes walking the expression tree a bit more complicated, since you always have to dereference node names. This is basically the same approach used by the dask graph itself, so you might have a better sense for how complicated this gets than I would. I'm actually curious how the general graph code in dask handles repeated references. Is the logic for this complicated? Does using named references simplify avoiding repeated work on duplicated noes? Is there some part of dask I should look at to get a better idea of this myself?
I would reiterate my comment https://github.com/dask/dask/issues/7933#issuecomment-895552688 that having a uniform way of walking the expression tree should not be underrated, especially for things like optimization passes which amount to tree rewriting. I'm still quite partial to the SymPy .args approach and would lobby pretty hard for it.
I expect some of your downsides to direct references can be mitigated without too much difficulty. For instance, one could create a flag dag graph from a direct reference graph to hold alongside the expression. This would basically amount to memoization, but storing the info on repeated nodes on the top-level expression rather than on the function. Or another option could be to use reverse references as @mrocklin suggested.

Question: The documentation at https://docs.dask.org/en/latest/high-level-graphs.html says "For example, in the case above we may want to automatically rewrite our code to filter our datasets before adding 100:" Is this optimization implemented somewhere? I saw optimizations.py but I couldn't tell if any of the functions in it actually do this. This is obviously the sort of optimization we want to do. If it doesn't exist yet, I it would be useful to think about what it would look like.

(new!) Use a .dependencies/children attribute as in the current plan, but also add a .dependents/parents attribute that goes the other direction. This would be a Weakset to avoid a reference mess. Optimizations would then look both at children and parents when deciding what to do.

Since this information is easy to compute later (and once computed could be maintained as part of the optimization passes, as we do now), I'm skeptical of the benefits of maintaining bidirectional pointers. Usually you want to find nodes that have only one dependency/dependent (for e.g. fusion) or all dependents of a certain node. Having an external mapping computed during the optimization pass that tracks this info doesn't seem too onerous.

I'm actually curious how the general graph code in dask handles repeated references. Is the logic for this complicated? Does using named references simplify avoiding repeated work on duplicated noes?

During most optimization passes we compute the dependencies of all tasks in bulk at the start of the pass (as a mapping from task name to dependencies). If needed, this mapping can be inverted to find dependents. Some passes are able to update the dependencies structure dynamically and forward this on to subsequent passes to avoid repeated work. It's a bit of a bookkeeping mess sometimes, but it's also worked well enough so far.

I'd really like to hear from some of the people that worked on the HighLevelGraph stuff. Rebumping:

I'm going to punt on answering the path-forward question and ask those that
have been more involved in the previous HighLevelGraph work to chime in here
with some thoughts (cc @gjoseph92, @rjzamora, @jrbourbeau,
apologies if there are others I should ping, not sure who all was involved).

Question: if during an optimization pass we change the inputs to a call such that they would have hashed differently, do we regenerate the layer name? Or do we leave it as is? For example:
df1 = dd.read_parquest("data/*.parquet")  # This currently hashes based on only the file paths
res1 = df1.x + df1.y  # with column projection, we can rewrite the above to only read `x`, `y`

df2 = dd.read_parquet("data/*.parquet", columns=["x", "y"])
res2 = df2.x + df2.y

Here we have res1 and res2, which should have the same result. The first relies on column projection to automatically cull the loaded columns. The second specifies it manually. Do these share structure in the low level graph? If we hash during expression generation (and thus "freeze" the node names), even after the rewrite pass that makes them equivalent they'll have different names. BUT making names dynamic would mean that some nodes might have a different name everytime .compute() is called (e.g. different expressions computed on df1 may result in different column subselections needed each time).
Right now I'm thinking that we don't handle this, and names are frozen at creation time, based only on the parameters passed by the user. This would prevent some optimizations (e.g. we couldn't treat res1 and res2 as the same), but I'd argue that those cases are likely rare and contrived. It seems easier to make names frozen on creation time.

I think the complexity here is thinking about how we handle duplicate nodes in a sane way. In other words, we really do have to think of expressions as a DAG, not just a tree. This will probably require some adaptation on my part. I'm used to SymPy. In SymPy, if you have an expression like (x + 1)**2 + (x + 1)**3, the x + 1 is just repeated twice in the expression tree. The literal Add class may or may not be de-duplicated in memory due to caching/memoization, but that's an implementation detail which ultimately doesn't matter because 1) expressions are immutable, and 2) == always compares expression by exact symbolic equality. But in dask, if you have df = read_csv('f.csv') + 1; df[df.name == 'Alice'], you don't want to represent that as (read_csv('f.csv') + 1)[(read_csv('f.csv') + 1).name == 'Alice']. Functionally they are the same, but computationally they aren't.
Hence dask is built with graphs rather than expressions. The graph lets you explicitly compute read_csv('f.csv') exactly once and reuse the result. Ultimately what we are building here is an optimizing compiler.
My first question about this is whether we can always assume a computation is functionally sane like this. Obviously if you start adding mutations in the mix computations can't be reordered so simply, but does dask even allow that sort of thing in the first place? The docs say that side-effects are not recommended, but to what degree can we assume there aren't any?

My first question about this is whether we can always assume a computation is functionally sane like this. Obviously if you start adding mutations in the mix computations can't be reordered so simply, but does dask even allow that sort of thing in the first place? The docs say that side-effects are not recommended, but to what degree can we assume there aren't any?

We let users write some things using the high-level api that look like mutation. But no low-level task (the thing that is actually in the low-level graph) should ever mutate one of their inputs. So while you can do:
df = make_some_dask_dataframe()
df["a new column"] = df.x + df.y

The actual pandas objects during execution are never mutated. Instead a copy is made, a new column added, and the result returned.

Here we have res1 and res2, which should have the same result. The first relies on column projection to automatically cull the loaded columns. The second specifies it manually. Do these share structure in the low level graph? If we hash during expression generation (and thus "freeze" the node names), even after the rewrite pass that makes them equivalent they'll have different names. BUT making names dynamic would mean that some nodes might have a different name everytime .compute() is called (e.g. different expressions computed on df1 may result in different column subselections needed each time).

I'm probably missing some context here. Why would they have different hashes? Does the hash include some sort of timestamp that makes the exact same expression unique?
I guess I'm also not really understanding why this matters. Do you foresee this being a problem during optimization passes, perhaps related to the "duplicate computation" problem I talked about in my previous comment? It seems to me that if these collapse, it would only be by coincidence. What if it instead optimized it to df2 = dd.read_parquet("data/*.parquet", columns=["y", "x"])? Then they would be equivalent but would not look the same. Effectively, node de-duplication would need to be its own optimization pass, which either applies some canonicalization to the nodes so they tend to look the same if they are the same, or does some sort of looser equality. In general, such things would necessarily be heuristic and there would always be cases that it would miss (the general problem would almost certainly be Turing complete).
However, if hashes really do contain more than just the exact input, then I think that means we shouldn't use it for equality (as Jim suggested to me on a recent call), because you can have situations where two expression should be exactly equal simply because they were copied in the tree. Just take the (read_csv('f.csv') + 1)[(read_csv('f.csv') + 1).name == 'Alice'] example I talked about above). If read_csv('f.csv') + 1 isn't guaranteed to hash to the same thing both times, we would need an __eq__ method based on something other than hash that compares them equally. To be sure, I don't think is a particularly hard problem if this is the case. It's just something I wanted to mention since it relates to this.

Currently all dask collection objects (e.g. dd.DataFrame) have a name that usually becomes part of the keys for that operation in the low-level graph (e.g. (df.name, 0) would be the key for the 0th partition in a dask dataframe). This name is usually computed by intelligent hashing of the arguments to the method that produced that object (the tokenize function is what handles this). For example, for random_split the name is composed of a hash of the input frame, frac and random_state: https://github.com/dask/dask/blob/9f0b00f039a7775dccccffe59d6802468508d9f7/dask/dataframe/core.py#L1047-L1049. I say we "intelligently" hash, since operations are free to canonicalize their arguments in any way before calling tokenize to handle things like order-independent operations, etc... This system has served us well, and provided a very simple way of pulling out common subexpressions to avoid duplicated work.
When moving to an expression system, we won't be eagerly generating the low-level-graph anymore, which lets us  do more interesting rewrite passes. However, we'll still need to make df.name available as part of the API for any custom algorithms that won't be representable as part of the expression system.
If we eagerly generate the name before applying optimization passes, then two nodes that would be the same post-optimization won't hash the same pre-optimization. The column projection example I gave above is one case of this.

If we eagerly generate the name before applying optimization passes, then two nodes that would be the same post-optimization may not hash the same pre-optimization.

Isn't that expected? In your example, they aren't the same. res3 = df1.x + df1.y + df1.z would work but res3 = df2.x + df2.y + df2.z would not. Requiring df1 and df2 to know this requires global knowledge of the entire graph, which complicates things.

I say we "intelligently" hash, since operations are free to canonicalize their arguments in any way before calling tokenize to handle things like order-independent operations, etc... This system has served us well, and provided a very simple way of pulling out common subexpressions to avoid duplicated work.

This worries me, regarding using hash for equality. Canonicalization at construction should be avoided. You can do anything that you would be done with canonicalization as an extra pass on the expression, but the problem is that you can't undo canonicalization. And it becomes hard to remove once it is there. I should note that very things like kwargs being undered would happen anyway if you used frozenset, so that's not canonicalization in my view. Neither is type checking since a failed type check just leads to an error anyway. I also talked about this in my earlier comment.
It does sound though like two equivalent expressions would hash the same, so at least for that hashing would be usable for equality.

Edit: actually, I think we need to regenerate the token for cases like column projection, since failure to do so would break our hashing guarantees, allow semantically different tasks to still have identical keys. This would be bad. So instead we may need a way to replace static keys in custom layers with new names? Or make df.name always a uuid, and insert an alias during expr -> graph lowering to connect the staticly provided uuid with the dynamically generated hash.

It sounds to me like you conflating two separate issues, the hash as an identifier and the hash as equality. To me this suggests that they should indeed be separated.

I think we're talking past each other, partly because our tower of abstractions here is complicated and partly because I'm doing a bad job of explaining things. I think this would be easier to clarify when we meet later this week.

I agree, although it would be helpful if I had something concrete to work on in the interim, if you can suggest anything. For now, the only "work" I feel I can do on this is trying to become more familiar with the current Dask internals, and trying to think about how this should be implemented based on my (likely confused) understanding of things.

That's fair. I'm actually free now for a bit if you want to meet to go through the current dask internals.

Thanks for highlighting this thread in the Dask Heartbeat post. I wanted to mention that we're working on a similar-ish (more high level?) effort in https://github.com/ibis-project/ibis/, which now has a (very alpha) dask backend. Following the developments here with interest.

@gerrymanoim any references you can provide for similar work in ibis would be helpful to see how you are tackling the same problems. My understanding was that ibis already has an expression system, so I'd be interested to hear what works and what doesn't work in the ibis design.

Hey all,
I am reading through this thread to try to get up to speed with the current status of this work.
I have also been exploring some different ways of handling expression graphs in Python (previously in metadsl and now in a different soon-to-be open-sourced library) and am curious about exploring what approaches might work for Dask.
I know that currently in Dask there is a concept of a High Level Graph. This thread seems to suggest adding an alternative or additional system.
So I am wondering if anyone would be interested in summarizing:

What are some concrete examples of optimizations Dask would like to support but cannot due to its existing optimization strategies? (A parquest column loading example has been mentioned above a few times. Is this the main driver?)
Are there examples of optimizations that the current High Level Graph implements that should also be possible if a new system was developed that replaced it? Is the way that they are currently implemented with the High Level Graph not ideal for some reason?

Personally, I find it much easier to iterate on a solution if the constraints are well defined in terms of how it should perform, since the space of implementing expression systems is so large.
It's also exciting to hear that Ibis is considering restructuring its expressions as well! I think it would unlock a lot of potential in the ecosystem if libraries like Ibis and Dask could start sharing some of this machinery!

Thanks for continuing this discussion @saulshanabrook - I apologize in advance for the following information dump that is unlikely to answer any of your actual questions (since my immediate thoughts focus more on the high-level requirements of Dask rather than the requirements of the specific expression system that we ultimately need)…
I think you are correct that a clear list of required features and optimizations would be useful.  The column-projection example has been used in several places, because it a relatively simple optimization to understand.  I haven’t spent much time thinking through the other collection APIs, but I suspect that the primary optimizations we need in Dask-Dataframe are:

partition culling - Available for both dictionaries and HLGs. Fast and easy with HLGs.
task fusion - Available for both dictionaries and HLGs. Slow for large dictionaries. Faster for HLGs, but support is currently limited to adjacent Blockwise-based HLG layers.
column projection - Only available in HLGs. Support currently limited to getitem operations directly following a DataframeIOLayer.
predicate pushdown - Not yet available in Dask. Basic support is certainly possible with HLGs, but the distinct “partition-planning” stage used in read_parquet/read_orc makes this more complex than column projection.

Furthermore, my personal understanding of the “high-level” requirements for the overall HLG and/or HLE system is:

We want to avoid materializing a full dictionary-based graph when creating or operating on a collection

This is already available with the existing HLG system, and is what has been the focus of that design (in my opinion).

We want to generate an optimized graph at compute/persist time

This is already available with the existing HLG system for optimizations (1)-(3), but the code is already a mess without general support for (3) or basic support for (4).

From my perspective, the High-Level Expression conversation was motivated by a consensus that the HLG system (alone) is turning out to be an unnecessarily messy and painful way to achieve the desired high-level goals.  I think that we all agree that it should be possible to get what we need out of HLGs, but experience has clearly illuminated some HLG pain points:

”Proper” serialization is complicated

The need to avoid importing pandas or numpy on the scheduler has made HLG-Layer implementations surprisingly complex. We have also been challenged by the need to avoid unpicking data on the scheduler.

It is difficult to use Information within the graph to optimize the graph

Since the HighLevelGraph API was effectively implemented as a “drop-in” alternative for python dictionaries, the HLG does not track basic DataFrame-collection information (schema, partition count, etc…), and is eagerly generated before compute/persist is called.  This means that Dask cannot generate an optimal HLG from scratch, but must use the limited information stored within the “un-optimized” graph to produce an “optimized” graph. Optimizations (1)-(3) currently do this by modifying the original HLG.

In my opinion, it is this second pain point which makes a high-level expression system attractive. Adding a high-level expression object to the Dask collection seems unlikely to help us with serialization.  However, such a system should make optimization much-less “hacky.”  This is because we could avoid eager construction of a graph, and instead use this expression object to produce an optimized HLG at compute/persist time (or when an operation without HLE support is used).

Thank you @rjzamora for the very in-depth reply! The high-level requirements you listed are getting at exactly my original question.  Before we can determine the requirements of the expression system I think it's helpful to outline, as you have started, what are its intended use cases and the shortcomings of current solutions to those.
I want to take some time to digest the specific examples and pain points you listed above.
I was thinking a possible next step could be to come up with complete examples for each and pointers to the existing code which implements the optimizations, so we are all on the same page on the current "hackiness" and its repercussions for limitations in optimizing concrete use cases.

I've been looking at HLEs again recently, so let me share at a very high level how I'd like to tackle it in a way that helps us experiment and learn.  I'm going to be somewhat vague and hand-wavy, because there is so much good information and details above that don't need repeating, and because I don't have a detailed, low level plan that I want to advocate for as the way forward (I actually have several in mind, including adapting prior efforts).
In principle, I believe some form (or perhaps many forms) of HLEs is achievable.  But, I expect any approach to be experimental for a while, and we're likely to want to iterate.  As others have pointed out, some of the difficulties in implementing HLEs in Dask are that the problem, design, solution spaces are high-dimensional, and that Dask has a large, complicated API surface area to cover, which may require some invasive changes.  I looked into HLEs briefly a few years ago, and I think it is daunting to try to experiment with it directly in the Dask codebase (but this may just be me).
My first goal is to create a sort of playground that makes it easier to experiment.  I envision this to be new Dask collections that mimic Array, DataFrame, and Bag collections (Delayed and user-defined collections can follow, but I'm not thinking about them yet).  By adding a shim layer, we can easily capture the function arguments, and call the methods on the original Dask collections when necessary.  Hopefully this will give us much of what we need, but I'm certain there will be shortcomings.  So, I'm also making it easy to modify the code of the original function.  There may be a little magic *cough cough*, but the goal is to be convenient, flexible, and maintainable-ish (i.e., it should complain loudly when code it changes is changed).  I plan to package this effort into a, uh, Python package.  Maybe multiple packages for different experiments.  This should make it easier for other people to try it out, and to experiment themselves.  If invasive changes become too extensive or if maintenance becomes too much a burden, I expect it would be straightforward to convert such a package to a Dask PR.  Adding a layer around Dask isn't a new idea--mrocklin above said "The efforts I've played with / proposed in the past have been for layers surrounding Dask. That adds more complexity though"--but I think the complexity is worth it short term, and such a layer may be worth it long term.  It may also make it easier to adapt to use for other packages, not just Dask.
Using MatchPy seems like a great idea.  I think we should.  There are many things we could match against and behaviors we could expose to the user.
My near term goal is to get a playground package that performs HLE pattern matching and rewriting for some DaskSQL patterns, including more robust predicate pushdown.  This should be like bread and butter for MatchPy.  From there, I think there will be many directions we could go or new experiments to try.
As a former boss of mine used to say, "for developers, doing is learning." There is a lot for me to learn (which is to say I have whole bunch of questions and concerns), and I'll get there by doing (hopefully--or I may crash and burn!).  I'll provide an update once I have a working playground.  I would also like to share other experiments I'd like to try.  Again, my main goal here is to learn, and to make it easier for other people to experiment and learn as well.  There's a lot of interest--and potential--in HLEs, so I say let's push forward however we can.
If anybody has a violent reaction to the way I'm thinking here, please reply (non-violently ;) ).

Hey @eriknw,
This sounds like an exciting plan!
I wanted to mention that I started with a similar intent a few years ago to make a library that makes it possible to make shim layers like this, called metadsl. It actually originally used MatchPy under the hood to do the replacement, but over time I built up an alternative matcher and UX and ended up replacing it entirely. The main improvement it provides, in terms of UX, over constructing MatchPy expression directly, is the ability to describe your nodes as Python function and classes and build them up by calling them. This also lets you use type annotations to describe the relationship of the node types of the args and return values.
I went looking through the history and found this earlier version which creates MatchPy opreations dynamically from Python functions: https://github.com/metadsl/metadsl/blob/bee9ddbb3b5d31c7a5f42a0c4ea42f19ad45168e/uarray/machinery.py#L154-L173
I had originally designed it for another use case, but I will have more bandwidth soon and would be excited to chat with you about what I have learned from there and to see if I could help in anyway, if you would be interested!

Hi @eriknw, happy to have someone else thinking about this issue. I looked at this a couple months ago and came to the following high-level conclusions (apologies for the unorganized nature of this comment):

Any change made will have to be done incrementally, which leads to organizational issues. Even if we know where we want to end up (we don't), we'll also have to figure out a set of incremental changes we can make where each step is valid and doesn't break the world. This is hard for something as low level as the collections api.
There are a bunch of complications that make it easier to have the expression tree (or whatever we call it) as an attribute on the user facing class, rather than the top-level class itself be an expression. By this I mean, the user would still only see an e.g. dask.array.Array, but an attribute on that class would contain the expression nodes/layers/whatever describing the computation to get to that point.
If we're storing an expression tree on the class itself, why not just make high-level graphs better? They're already an expression system (just one that's hard to work with), and group pieces of the shared expression state together. It's easier to work incrementally from what we have already than to start a new place where state will live.

As such, I think we should work on improving the existing system by:

Cleaning up how high-level-graphs generate low level graphs. Right now the code path for this is convoluted and coupled to the serialization logic (to avoid unpickling on the scheduler). @rjzamora has a PR up right now attempting to clean up some of this, but it'll be an iterative effort to make this simpler.
Expanding what's stored on a Layer in the graph. Right now we don't store enough metadata on each layer to perform high level operations, but we can change that! We might move _meta and divisions to live on the layer for example.
Deprecating and eventually removing the Mapping interface for things in the .dask attribute. Originally .dask was a real dict, then it was a SharedDict, then it was a HighLevelGraph. We're sufficiently divorced from the actual data structure looking like a mapping, and keeping the mapping interface is sometimes confusing and sometimes complicated. We'll still want to support layers in the graph composed of low-level dask dict representations, but at this point I think the high-level object on .dask shouldn't need to look like a Mapping anymore.

Using MatchPy seems like a great idea. I think we should. There are many things we could match against and behaviors we could expose to the user.

IMO, the optimizations we want to write currently are few in number and custom enough that writing them by hand doesn't seem that onerous. Our main issue is not how to express the optimizations, it's more how do we store the expression tree and metadata in a way that lets us rewrite things before computing them:

Efficiently
Without breaking low level layers in the graph that might depend on fixed key names in upstream & downstream layers
Without accidentally duplicating computations (there's a bunch of nuance to this, see e.g. https://github.com/dask/dask/issues/7933#issuecomment-918424353).

If after we solve all these and we end up in a situation where handwriting the optimizations is hard, then I'd call that a success (and maybe a time to adopt such a system). But we're so far from that point right now that I don't think it's worth thinking about.
Anyway, if you ever want to hop on a call to talk further, please let me know. Due to how much of our internals we've exposed as a public interface here, changes here have a ton of nuance and may seem fine for one operation and become a footgun for another. This is a complicated problem.

Hi all, I have been reading this issue with great interest. Slice pushdown in dask would be a pivotal feature for our application.
I (with my colleagues from Nelen & Schuurmans) developed dask-geomodeling a few years ago to address this issue specifically for the pushdown of geospatial requests. Using that package we store expression trees in a database and from that create task graphs on-the-fly after a user requested a subset through a web application. After we developed this, other projects like dask-geopandas came up and we realized that we are missing out on a lot of features because we took an approach that basically reimplements every single operation. I am currently looking for ways to make more close use of dask directly. One must-have for us is this expression system.
So I am really excited reading the ideas here and the plans in https://github.com/dask/dask/issues/9159.
It may be useful to share some experiences we had running (or own style of) high level optimization in production for a couple of years.
(de)serialization of expressions
At dask-geomodeling we use the same data structure for serializing layers as for serializing the tasks. Instead of functions, you use classes (types). The result of evaluating a layer graph is a collection instance ("view" in terms of dask-geomodeling). See https://dask-geomodeling.readthedocs.io/en/latest/views.html#view-serialization for an example.
I am no expert of ideal datastructures for optimization approaches. But the advantage here is that you can re-use code for things like constructing and optimization. Materializing the collection is like "computing". I tried to quickly create a mock implementation for this approach, but in the end I circled back and realized that dask.delayed does exactly this job:
import dask
import dask.dataframe as dd

ReadParquet = dask.delayed(dd.read_parquet)
src = ReadParquet(path)

def collection_optimize(dsk, keys):
    """ Your high-level optimizations go here """
    return dsk

with dask.config.set(delayed_optimize=collection_optimize):
    collection = src.compute()

collection.compute()  # computes the result

In my example above you can directly start solving the optimization problem (or use existing RewriteRule classes).
The "delayed" collection could be top-level or carried along as an attribute the the collections.
While I realize that this would be something "next to" HLG instead of an extension of HLGs (as described in #9159), I do think this approach is interesting for its simplicity. There are no new concepts introduced. Also I feel that HLGs are violating the "Single Responsibilty Principle" too much (but this is just a hunch that needs exploration).
There are cons here: one being that the Delayed() has to clue of the parameters and return type of the wrapped function. So there is no way for parameter parsing/validation or listing methods. You could do src.non_existing_method(foo="bar"). Also, there is no way for accessing metadata (like columns) for optimization rules.
I'd like to explore this option, but I'd like to know first what is your position on this approach. Maybe such an approach was already discarded, in that case sorry for the noise.

Thanks for sharing @caspervdw !
If I undertstand correctly, you are pointing out that much of the logic in Delayed can be reused to design a simple expression system for dask collections. This is a cool idea!  It does seem worth exploring - I'll try to think more about the possible opportunities and blockers.


[Guide] Issue in retrieval list from the DB in Input form

When does the issue occur?
When I have generate dropDownList or radioButtonList from DB at time return same key and value like
return [
    2 => 2,
    3 => 3
]

because column method is return first column in select statement. when i change position in select statement like set label field first at time return my expected results like.
What was the expected result?
return [
    2 => 'My label/title-1',
    3 => 'My label/title-2'
]

Q
A

Is bugfix?
no

New feature?
no

Breaks BC?
no

Tests pass?
yes

@samdark Ok have updated issue descriptions

Still don't get how to reproduce the issue...

@samdark Please refer this link
http://www.yiiframework.com/doc-2.0/yii-db-query.html#column()-detail
http://www.yiiframework.com/doc-2.0/yii-db-command.html#queryColumn()-detail

your issue description is quite cryptic. Can you try to explain it simple in one line?

Ah, I think I've got it. Will check in detail later.

Thank you!

how would this method be able to index by ID if ID is not selected?


Add equalsInAnyOrder Matcher for NSArray

There are a bunch of Gists and blogs on this, but I find myself doing beSupersetOf and hasCountOf together all the time to get the same effect. Figured I would submit this back in.
Converts the NSArrays to NSSets and compares them so that order doesn't matter.
First PR to Expecta so feedback welcome!!

I like this idea, but I feel like the current implementation is lossy:
expect(@[@1,@1,@2]).to.equalInAnyOrder(@[@1,@2]);

Would end up passing here, but they are not the same arrays, in a different order. Perhaps the implementation could order them both by description, then loop through one array and check if the other arrays item is the same?

Also, CHANGELOG please :)

Perhaps the implementation could order them both by description, then loop through one array and check if the other arrays item is the same?

NSCountedSet should do the trick here

All in all, I'm not sure I think this is worth adding to expecta. It seems that if you frequently need to compare two arrays independent of order, using an NSArray is the actual problem.

@robb Thanks for doing so much else! :sparkles:
I end up using this a ton for times where I want to ensure the same exact keys in an NSDictionary and allKeys returns an NSArray.
I 100% see your point, if you don't care about ordering stop using an NSArray. Up to you really :)

Yeah, I think I'm onboard with the having as an extension, it could fit quite well in Expecta-Comparison or it could be it's own independent thing.


Documentation translation - how and where to pull

Hi! We got polish translation of Mautic documentation, and we would like to contribute this way. How and where can we push this? Contributing notes are not covering this. It would be nice if someone can change language @ https://docs.mautic.org

Hi there, great to hear that there is interest
In translating the documentation into Polish!
We are using Grav as our platform, and it supports multilingual natively.
We would require all the pages to be translated fully before we go about adding a new translation. We would also need a team of people committed to translating, reviewing, and editing
on an ongoing basis.
What I would suggest as a starting point is joining the Education Team Slack channel (Get an invite at mautic.org/slack), and asking everybody who would be willing to contribute to the documentation in Polish to join as well. Then we can spin up a small working group to focus on this.
If you already have the documentation translated, it would be super helpful if you could share this, for example in a Google doc or several Google Docs, so that we can review.  Please make sure that any translation is current, as since we re-platformed the structure has been changed.
Excited to work on this with you, look forward to chatting more in Slack!

Was postponed to be worked at in the Slack channel, thus I close this until we get a new spin and need to re-open. :)


Add more tests

We need to add tests for the negative cases as well as the positive cases.

Implemented by #4


[ASCII-2415] Add merged CPU profiles for each agent process and use PGO

What does this PR do?
Same as https://github.com/DataDog/datadog-agent/pull/30313 but using merged profiles from prod.
Motivation
Describe how to test/QA your changes
Possible Drawbacks / Trade-offs
Additional Notes

Regression Detector Results
Run ID: f9ba7d3a-26ae-469b-845e-effef8ec6509 Metrics dashboard Target profiles
Baseline: 7.58.0
Comparison: 7-58-1-beta-pgo-prod-profiles-merged-py3
Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

Significant changes in experiment optimization goals
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

perf
experiment
goal
Δ mean %
Δ mean % CI
trials
links

❌
tcp_syslog_to_blackhole
ingress throughput
-9.08
[-9.10, -9.06]
1
Logs

Fine details of change detection per experiment

perf
experiment
goal
Δ mean %
Δ mean % CI
trials
links

➖
idle_all_features
memory utilization
+0.60
[+0.56, +0.64]
1
Logs bounds checks dashboard

➖
quality_gate_idle_all_features
memory utilization
+0.45
[+0.41, +0.49]
1
Logs bounds checks dashboard

➖
idle
memory utilization
+0.22
[+0.20, +0.25]
1
Logs bounds checks dashboard

➖
basic_py_check
% cpu utilization
+0.13
[-1.06, +1.33]
1
Logs

➖
file_to_blackhole_1000ms_latency
egress throughput
+0.09
[-0.13, +0.30]
1
Logs

➖
file_to_blackhole_0ms_latency
egress throughput
+0.02
[-0.13, +0.17]
1
Logs

➖
file_to_blackhole_500ms_latency
egress throughput
+0.02
[-0.09, +0.13]
1
Logs

➖
uds_dogstatsd_to_api
ingress throughput
+0.00
[-0.04, +0.05]
1
Logs

➖
tcp_dd_logs_filter_exclude
ingress throughput
+0.00
[-0.00, +0.00]
1
Logs

➖
quality_gate_idle
memory utilization
-0.01
[-0.04, +0.01]
1
Logs bounds checks dashboard

➖
file_to_blackhole_300ms_latency
egress throughput
-0.04
[-0.13, +0.04]
1
Logs

➖
file_to_blackhole_100ms_latency
egress throughput
-0.11
[-0.21, -0.00]
1
Logs

➖
otel_to_otel_logs
ingress throughput
-0.63
[-0.99, -0.27]
1
Logs

➖
file_tree
memory utilization
-0.64
[-0.69, -0.59]
1
Logs

➖
pycheck_lots_of_tags
% cpu utilization
-1.39
[-2.48, -0.30]
1
Logs

➖
uds_dogstatsd_to_api_cpu
% cpu utilization
-4.06
[-4.39, -3.74]
1
Logs

❌
tcp_syslog_to_blackhole
ingress throughput
-9.08
[-9.10, -9.06]
1
Logs

Bounds Checks

perf
experiment
bounds_check_name
replicates_passed

❌
idle
memory_usage
20/50

❌
quality_gate_idle
memory_usage
27/50

✅
file_to_blackhole_0ms_latency
memory_usage
50/50

✅
file_to_blackhole_1000ms_latency
memory_usage
50/50

✅
file_to_blackhole_100ms_latency
memory_usage
50/50

✅
file_to_blackhole_300ms_latency
memory_usage
50/50

✅
file_to_blackhole_500ms_latency
memory_usage
50/50

✅
idle_all_features
memory_usage
50/50

✅
quality_gate_idle_all_features
memory_usage
50/50

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

Its configuration does not mark it "erratic".


Confused about 1.07-Exercise-ConfigureATask

So, this may be a dumb question, but I'm confused about the 1.07 exercise for configuring a task. In the quiz on the Udacity web site, the answer for the "group" question was "udacity". Unless I'm missing something, according to the build.gradle, the group should be "Other tasks", correct?
Here are the instructions:

Create an empty task definition and name your task "hello".
Give your task a description.
Add an action to your task which prints "Hello World!"

This does not mention specifying the "group" name for the task. While the solutions.gradle does, it appears to be referencing a different build.gradle (group = 'udacity' // 2. Add task to "udacity" group -- in the build.gradle, #2 is actually "give it a description").
If the build.gradle is followed, shouldn't the resulting "group" be "Other tasks"?
Sorry - just trying to understand everything and confused as to why I got the wrong answer :)

Hey Kyle,
Sorry for taking so long to get back to you. You're totally right, that was confusing. The exercise doesn't mention anything about the task group. The quiz is specifically asking you to get the tasks from the solution.gradle file. I changed the comments in the solution file so the numbering matches the instructions, and added an additional comment explaining that the group setting is something extra that we didn't ask you to do.
Thanks again!


Quick start documentation is broken

The documentation is broken at http://frees.io/docs/
Taking the example in the quickstart guide:
package meeh

import freestyle.free._
import freestyle.free.implicits._

@free trait Validation {
  def minSize(s: String, n: Int): FS[Boolean]
  def hasNumber(s: String): FS[Boolean]
}

@free trait Interaction {
  def tell(msg: String): FS[Unit]
  def ask(prompt: String): FS[String]
}

@module trait Application {
  val validation: Validation
  val interaction: Interaction

  import cats.implicits._

  def program: FS.Seq[Unit] =
    for {
      userInput <- interaction.ask("Give me something with at least 3 chars and a number on it")
      valid     <- (validation.minSize(userInput, 3), validation.hasNumber(userInput)).mapN(_ && _).freeS
      _         <- if (valid)
        interaction.tell("awesomesauce!")
      else
        interaction.tell(s"$userInput is not valid")
    } yield ()
}

object ApplicationHandler {
  import cats.effect.IO

  implicit val validationHandler = new Validation.Handler[IO] {
    override def minSize(s: String, n: Int): IO[Boolean] = IO{s.size >= n}
    override def hasNumber(s: String): IO[Boolean] = IO{s.exists(c => "0123456789".contains(c))}
  }

  implicit val interactionHandler = new Interaction.Handler[IO] {
    override def tell(s: String): IO[Unit] = IO{println(s)}
    override def ask(s: String): IO[String] = IO { println(s); "This could have been user input 1" }
  }
}

object Main {
  import cats.implicits._
  import cats.effect.IO
  import ApplicationHandler._

  Application.instance.program.interpret[IO].unsafeRunSync
}

then sbt/scalac will complain that it cannot find interpret for program. It took me quite a while to find out that the type annotation for programm is the reason. If it is rewritten as def program = for {... everything works as expected. I do not provide a PR since it would be interesting to know why.

scalacOptions += "-Ypartial-unification" is the missing part!


JsonCsrf for GET image.

Upon upgrading from eb7e4c9a176d to 1.5.3, our Sinatra app is getting attack prevented by Rack::Protection::JsonCsrf .
Not sure how to fix this (besides set :protection, :except => [:json_csrf]).
The resource in question returns an image, and does not take params. Any advice would be appreciated. Thank you.

any updates on this?

Sorry for the delay, I will take a look into these issues after sinatra-contrib is passing on sinatra master.

@georgeu2000 or @chrisallick I'm not sure if I know how to repro this. Looking at the code, it seems the request must have application/json content-type in order to trigger the protection of JsonCsrf.
I wonder if it's getting caught else-where.. or if a CSP would help alleviate the situation.

Closing this after copying the ticket upstream, but feel free to discuss which ever.


Doesn't show in menu on Mac

I installed the plugin but doesn't show items in menu, I try install agin but doesn't work.
I use eclipse on Mac, does work on eclipse on Mac?

It is working fine for me on Mac, other than the shortcut for interactive cleanup is the same as screenshot.

It is working fine for me on Mac, other than the shortcut for interactive cleanup is the same as screenshot.

Here doesn't show on menu

Hi Renivaldo,
meanwhile, I asked a colleague who knows better about Mac than me, but he confirmed that from the screenshots, everything looks fine, so it's hard to understand why the menu isn't showing.
Maybe the error log contains some hints? (menu "Help / Collect Support Information…") If you'd like us to take a look, you could send them to "abap-cleaner at sap dot com" (but no guarantees I can really help…)
Kind regards, Jörg-Michael

Hi the log shows nothin.

Hi Renivaldo,
hm, sort of running out of ideas here what could have gone wrong. Maybe you could try on a fresh Eclipse + ADT installation?
Kind regards,
Jörg-Michael


Controller displays an empty temp chart in Web UI

You may notice that controller now displays an empty temp chart in Web UI.  Previously, it showed a histogram of operating mode (over the last 24h).

Note that the component functions normally otherwise.

This is due to a change that occurred around version 0.77 of HA.  From then on, climate entities must have a current_temperature.
I disagree that the controller should have a current_temperature (I don't think outside temp counts), and the idea of a target_temperature makes no sense at all.  So what's the answer?
a) revert back to situation where climate entities are not required to have a temperature, or
b) implement the controller as a non-climate entity?
FWIW, some controllers can have an outside weather sensor, and in all other cases, the controller uses a web api for outside temp.  Also, a controller can be configured as a thermometer for a zone, in which case the zone has the temperature, not the controller.
There are problems with both options (for example, the controller has away mode, the zones do not), and I haven't decided how to square the circle as yet - so watch this space!
As it stand, the UI is disappointing, but the system still behaves as expected.  See here for more information:
https://github.com/home-assistant/architecture/issues/22

In issue #13 (now closed), @megapearl asked:

This is not an issue, but a feature request.
Is it possible to display the outside temperature in homeassistant via your component?
My evohome controller has a sensor outside which displays the outside temperature on the controller (in the upper right corner of the evotouch)
It would be great if you can implement it in the future.
Best Regards,
Donald.

The answer I gave was:

The honeywell api, as currently documented, does not expose this information.  Either for locations with a weather sensor (like yours), or for those who don't, but still get the 'outside' temperature displayed in the UI.

So, all I could do is poll another web api for that data.  That is, it wouldn't always be the same reading your weather sensor has - it could be off by a couple of degrees.
I could poll the same web api as does the EvoTouch (I forget which).  Why should l though? HA does a good enough job of that already.
I can certainly show a temperature.  In fact, I will probably end up showing a temperature in the HA component (see issue #12). I am not happy about it, though...
Balloob suggested I use the average of all the reported temps in the house!

@megapearl said:

So you can’t pull the outside temperature out of the controller, local and remote.

Correct

As I remembered correctly, the Domoticz guys did something like that using a honeywell HGI80,

Unlike most evohome users, I have a HGI80, but it is out of scope for this platform/component.

then you don’t need Honeywell’s bad webapi.

It is neither bad, nor good. Just is. :)

I was reading through my old notes - it seems my controller polls api.openweathermap.org

IMO, the controller should be a climate object, but a hub.  This work is to be addressed later on, awaiting some architectural issues in HA to be addressed.
Closed (for now).


Grid view redraws if next/prev buttons pressed

If the next/previous buttons are pressed in the grid view, all images are reloaded.
Expected behavior would probably be to select the first (or last) image.

Duplicate of #44 (more or less); sorry about that.