fix: adapt to new Task spec in dask, now used in blockwise

Adapts one bit of dask_awkward code that makes a graph to use a task object instead.
c.f.:
https://github.com/dask/dask/pull/11568

I asked on the dask PR whether there are specific migration instructions associated with their change.

@martindurant rewrite_layer_chains needs to be adjusted, along with various checking that's happening during the tests. The latter is all localized in one function for the most part so hopefully a reasonable fix.
There must be some way to just pop out a good old dictionary from the Task class.

We will likely need to put a dask >= 2024.12.0 requirement on the next release, and a dask < 2024.12.0 for python < 3.10.

If I'm understanding this correctly a Task allows to .substitute itself by another one? This could be useful for graph cloning and then replacing the IO layer - or am I misunderstanding this?

Task allows to .substitute itself

Some docs on that method would be very useful.

appease mypy's insatiable lust for perfect correctness

:)

rewrite_layer_chains needs to be adjusted, along with various checking that's happening during the tests

Is this something I'll need to get on?

@martindurant seems like it - but it also looks like it'll get significantly more easy to read using the new fuse/substitute interfaces in Task.

rewrite_layer_chains needs to be adjusted, along with various checking that's happening during the tests

Is this something I'll need to get on?

I'd be interested to get familiar with this new Task class - I think it would be beneficial to know what substitute allows to do.
If you want I can try to look into updating rewrite_layer_chains aswell.

probably a good first introduction to update rewrite_layer_chains +1 to that.

I can try to look into updating rewrite_layer_chains

sure, go for it! If it's hard to understand, ask any time.

This is off-topic but I am confused about what rewrite_layer_chains is supposed to do. I haven't dug into the code but the doc string reads as if this was a reimplementation of optimize_blockwise. We're already fusing adjacent blockwise layers internally so I'm curious what this is doing that we are not already by default.

fwiw the one code change you'll need is
diff --git a/src/dask_awkward/layers/layers.py b/src/dask_awkward/layers/layers.py
index 9244144..e89401f 100644
--- a/src/dask_awkward/layers/layers.py
+++ b/src/dask_awkward/layers/layers.py
@@ -8,6 +8,7 @@ from dask.blockwise import Blockwise, BlockwiseDepDict, blockwise_token
 from dask.highlevelgraph import MaterializedLayer
 from dask.layers import DataFrameTreeReduction
 from typing_extensions import TypeAlias
+from dask._task_spec import Task, TaskRef

 from dask_awkward.utils import LazyInputsDict

@@ -163,7 +164,7 @@ class AwkwardInputLayer(AwkwardBlockwiseLayer):
         super().__init__(
             output=self.name,
             output_indices="i",
-            dsk={name: (self.io_func, blockwise_token(0))},
+            task=Task(name, self.io_func, TaskRef(blockwise_token(0))),
             indices=[(io_arg_map, "i")],
             numblocks={},
             annotations=None,

If I disable the rewrite_layer_chains  optimization, most of the test suite passes with this change. The test failures relate to specific assertions on the dsk attribute of the blockwise layer (which has been removed in favor of the Task class) so that needs a bit more thorough review.
With all of this, please note the underscore in the _task_spec module. It's not unlikely that we'll end up refactoring things around that class. The high level API, e.g. the above diff, should stay the same but if you intend to dig deeper this may become brittle down the line.
Regarding a migration guide, the challenging part of this PR will be to migrate the rewrite_layer_chains optimization. I don't think this will just be a syntactic rewrite as the above diff but requires some more thought. The code looks simpler than the dask internal blockwise optimize so I assume this shouldn't be too much trouble.

I assume you are referring to https://github.com/dask/dask/issues/9795? I believe there is a misunderstanding of some kind, either on your or on my part.
Dask has a couple of different optimizer steps (that we're actually currently overhauling partially)

blockwise fusion / optimization
Low level task fusion

The blockwise fusion takes consecutive Blockwise layers and is merging them into a single Blockwise layer if certain requirements are fulfilled. The graph underneath that is not materialized at this stage.
After this step, the graph is materialized (i.e. then you'll have millions of tasks) and we are calling "low level task fusion" on that graph that is looking at chains of individual tasks. We actually removed the old low level fusion code with something simpler (and faster) recently.
It feels like rewrite_layer_chains and our blockwise fusion is doing the same. Or at least, it is supposed to do the same.
(I know you pinged me repeatedly on https://github.com/dask/dask/issues/9795 but I haven't had the time to follow along back then so I'm catching up right now)

Some performance comparisons regarding rewrite_layer_chains and optimize_blockwise:
# takes from https://github.com/dask/dask/issues/9795#issuecomment-1372537369

import sys
import dask.array as da

sys.setrecursionlimit(10000)

def make_graph(num_layers):
    array = da.zeros(1000, chunks=(100,))
    for i in range(num_layers):
        array = array + 1
    return array

arr = make_graph(1_000)

# -------------
# dask.__version__ == [2024.12.0](https://github.com/dask/dask/releases/tag/2024.12.0)
from dask.blockwise import optimize_blockwise

%timeit optimize_blockwise(arr.dask, arr.__dask_keys__())
# > 291 ms ± 2.16 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# -------------
# dask.__version__ == [2024.11.2](https://github.com/dask/dask/releases/tag/2024.11.2)
from dask.blockwise import optimize_blockwise

%timeit optimize_blockwise(arr.dask, arr.__dask_keys__())
# > 838 ms ± 16.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# -------------
# dask.__version__ == [2024.11.2](https://github.com/dask/dask/releases/tag/2024.11.2)
# dask-awkward.__version__ == '2024.9.1.dev9+g1d4d4e9'
from dask_awkward.lib.optimize import rewrite_layer_chains

%timeit rewrite_layer_chains(arr.dask, arr.__dask_keys__())
# > 388 μs ± 1.84 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

So it looks like there's a speedup already in optimize_blockwise by upgrading dask to 2024.12.0, but rewrite_layer_chains is still significantly faster.

Oh wait, something is off with rewrite_layer_chains, I guess I'm using it wrongly? Looking at the output HLG no fusion happened in the case of rewrite_layer_chains?
print(optimize_blockwise(arr.dask, arr.__dask_keys__()))
# HighLevelGraph with 1 layers.
# <dask.highlevelgraph.HighLevelGraph object at 0x10958f6a0>
# 0. add-cf3b8e445fcb2608bb3016be1ec4b299

print(rewrite_layer_chains(arr.dask, arr.__dask_keys__()))
# HighLevelGraph with 1002 layers.
# <dask.highlevelgraph.HighLevelGraph object at 0x1090bbd00>
# 0. zeros_like-f0d60d8a5800b8c89bf04b383f380015
# 1. add-41ea24f1508f8670fba2857f55292924
# 2. add-043382e00b1563572167b7cd00b6c4fa
# ...
# 999. add-93272e7661ed72d8cd3475c2834e3c84
# 1000. add-cf3b8e445fcb2608bb3016be1ec4b299

(Maybe you should ignore the benchmark for rewrite_layer_chains for now until I use it/make it work correctly...)

Ok, so TLDR is that if our blockwise optimizer was faster and less complex, you wouldn't have a need for your own (and possibly your own is a drop in replacement for ours). That's good enough for me now.
Regarding your migration, I suppose you can copy a couple of sections from https://github.com/dask/dask/pull/11568
I suspect your optimizer is migrated with something like
--- a/src/dask_awkward/lib/optimize.py
+++ b/src/dask_awkward/lib/optimize.py
@@ -5,7 +5,7 @@ import logging
 import warnings
 from collections.abc import Iterable, Mapping, Sequence
 from typing import TYPE_CHECKING, Any, cast, no_type_check
-
+from dask._task_spec import Task
 import dask.config
 from awkward.typetracer import touch_data
 from dask.blockwise import Blockwise, fuse_roots, optimize_blockwise
@@ -340,23 +340,19 @@ def rewrite_layer_chains(dsk: HighLevelGraph, keys: Sequence[Key]) -> HighLevelG
         deps[outkey] = deps[chain[0]]
         [deps.pop(ch) for ch in chain[:-1]]

-        subgraph = layer0.dsk.copy()  # mypy: ignore
+        all_tasks = [layer0.task]
         indices = list(layer0.indices)
-        parent = chain[0]

         outlayer.io_deps = layer0.io_deps  # mypy: ignore
         for chain_member in chain[1:]:
             layer = dsk.layers[chain_member]
             for k in layer.io_deps:  # mypy: ignore
                 outlayer.io_deps[k] = layer.io_deps[k]
-            func, *args = layer.dsk[chain_member]  # mypy: ignore
-            args2 = _recursive_replace(args, layer, parent, indices)
-            subgraph[chain_member] = (func,) + tuple(args2)
-            parent = chain_member
+            all_tasks.append(layer.task.substitute(...))
         outlayer.numblocks = {
             i[0]: (numblocks,) for i in indices if i[1] is not None
         }  # mypy: ignore
-        outlayer.dsk = subgraph  # mypy: ignore
+        outlayer.task = Task.fuse(*all_tasks)

with the appropriate input to substitute. Task.fuse will raise if the fusion is invalid (i.e. if the subgraph that's spanned by the task are not reducing to a single key)
Note: substitute is currently not very fast. For ~7k layers I hope this is fast enough but it's something I want to look into in the near to mid term future. Generally, I expect that the graph construction cost will go up with the Task class but some foo on the scheduler and during submission should go down and make up for the increased cost of construction. If you end up seeing something differently please let me know

@pfackeldey : rewrite chains only works on dask-awkward layers.

@pfackeldey : rewrite chains only works on dask-awkward layers.

Yes, I realized this aswell, I've updated my comment with the benchmarks accordingly. :+1:

@fjetter Given that there was some low hanging fruit in terms of improving computational complexity of the optimizer, there is probably more. Likely worth a look.
I also had some reasonable success using numba to optimize hotspot special cases.

@fjetter while we can capture a small amount of your attention, and since it's somewhat pertinent as we are discussing graph optimization. We see the following behavior, calculating a single object of interest (in this case a histogram) tends to optimize very well (the bottom squares are some large constants, the middle node is processing, the last is filling histograms in a tree reduce):

(the boxes at the bottom are dask.delayed objects which are constant correction data and could be inlined but there's no out-of-the-box function for it for HLGs)
However, when we start plotting multiple histograms, in this case one histogram is of an input value to the quantity in the other, the optimizations seem to stop where data gets reused *:

In particular, here you can see "met.phi" gets sent to multiple nodes for various uses and that seems to more or less ~stop a great degree of optimization. Likewise the control region definition logic (the blobs of nodes right before the end) seem to not be collapsed as before. All in all, lose the graph layer fusion that really helps with creating tractable, scalable optimized task graphs. What would be a good strategy for making an optimizer that would be able to still contract the graph to the minimum number of nodes while still allowing data re-use?
* here's the un-optimized task graph for your amusement:
kelci_opt.pdf

The pictures are unfortunately not properly loading for me so I can't inspect them closely. However, it is a known problem that shared dependencies are hard to deal with.
It's a common fallacy to think that the reuse is beneficial or even necessary. Sometimes breaking this sharing is actually beneficial for the computation because you can pipeline computations and release memory much more quickly. That's particularly true if you are computing in a memory constrained environment. If the shared dependency is very expensive to compute (and/or small), it should be cached and shared. If it is cheap to compute, just doing it many times is good. That is intuitive for most people. Data loading/production is the part that's tripping people. It is typically relatively expensive so people want to do it as little as possible. However, while it is not cheap, it is often also very large so caching is expensive (having a larger graph is also a type of cost, especially for your computation scale).
I don't know what's blocking your optimization. Shared dependencies sometimes inherently block optimization. Sometimes it's also just a bug. I don't know and can't investigate without a reproducer

Ok, sorry that didn't render correctly. I'll see if I can get a concise reproducer with few dependencies.

Also, thanks for your description of generally what we'd like out of optimization. It confirms my own understanding of the trade offs and goals.

I managed to update rewrite_layer_chains successfully as far as I could test it, however I wasn't able to make it work with .substitute() here; maybe @fjetter could give me some hints/tipps how to update it correctly?
I'll look into the output mocking next, which needs to be updated aswell.

I'll look into the output mocking

This is an "optimization v1" feature. I wonder is v2 needs any work at all - but since we've not been pushing on that, maybe not worth checking right now.

Nice, down to one failure :-)

I think now there's only one test left that needs to be fixed: tests/test_core.py::test_partitionwise_op_with_delayed.
The problem here is that the new dask treats the key of the delayed as a DataNode which is passed down to the actual awkward operation. This results in something like: ak.Array([....]) * DataNode(...).key where DataNode(...).key is something like a_delayed_array-da5d2abd-ca30-4227-9243-dd7b7d9ba5df.
If someone has an idea how to solve this, I'd appreciate some tipps as I'm not really understanding how this happens...

Seems like it may be a hidden performance issue now turned bug when mixing dak.Array with dask.delayed.Delayed. If I start inspecting the arguments to that operator.mul call via a higher level function, we see in the typetracer only graph computation that we have a materialized array representing the thing that is delayed.
def mult_func(a, b):
    breakpoint()
    return operator.mul(a, b)

def test_partitionwise_op_with_delayed():
    array = ak.Array([[1, 2, 3], [4], [5, 6, 7], [8]])
    dak_array = dak.from_awkward(array, npartitions=2)
    result = map_partitions(
        mult_func,
        dak_array,
        a_delayed_array(),
        meta=dak_array._meta,
        output_divisions=1,
    )

    ...

dask==2024.10.0:
> /Users/ddavis/software/repos/dask-awkward/tests/test_core.py(929)mult_func()
-> breakpoint()
(Pdb) a
a = <Array-typetracer [...] type='## * var * int64'>
b = <Array [2, 4] type='2 * int64'>      # <--- the delayed thing!

dask==2024.12.0:
> /Users/ddavis/software/repos/dask-awkward/tests/test_core.py(929)mult_func()
-> breakpoint()
(Pdb) a
a = <Array-typetracer [...] type='## * var * int64'>
b = 'a_delayed_array-69f75d8d-9f27-454b-83f7-cbda699c7f12'

The fact that the older version of dask ends up yielding a concrete awkward array in the typetracer only graph is a red flag. I suspect that the compute that occurs in the optimization step is doing a "real" compute. If that delayed were expensive then this would be a bad optimization scenario. Seems like the issue may be more serious than whatever evolution has happened in Dask, and perhaps new Dask has exposed the problem. In the past mocks were introduced for some special cases. Perhaps this is another place where a mock is needed to represent the Delayed layer.

@douglasdavis - thanks for taking a look, super useful.
My worry about this is when the delayed object is a function, then it gets rather complicated.
I would argue, however, a concrete awkward array in this case is expected since Delayed doesn't know anything about typetracers and the computation of the graph for data-touching is a "real compute". So this really all checks out.
I think we even had a conversation about this since keeping the optimization time down is critical for coffea.
So I guess we're not understanding how to embed things in the graph correctly? Or something fundamental changed in delayed?

Ah! if I turn all optimization off it still yields the task key in the function rather than the value (which is should eventually).
So I would say something else is up.
@douglasdavis @pfackeldey

I'm still in the process of understanding when it should materialize, because in principle we need it already during the typetracing pass to trace the function with arrays. But then: what's the point of having a delayed that yields a concrete Array anyway if it has to evaluate already in the typetracing pass?
At that point one can directly pass the materialized array to map_partitions aswell?

Lindsey Gray @.***> writes:

@douglasdavis - thanks for taking a look, super useful.
My worry about this is when the delayed object is a function, then it gets rather complicated.

Yes of course. The idea is certainly to be able to do "anything"
represented by a Delayed object

I would argue, however, a concrete awkward array in this case is
expected since Delayed doesn't know anything about typetracers and the
computation of the graph for data-touching is a "real compute". So
this really all checks out.

I agree a concrete array makes sense for the test of interest here. But
like you said above, we want to have maximal flexibility with Delayed--
and I don't think we want to be computing that "anything" if it's some
expensive function.

I think we even had a conversation about this since keeping the
optimization time down is critical for coffea, but it's been quite a
long time.
So I guess we're not understanding how to embed things in the graph
correctly? Or something fundamental changed in delayed?

Possibly a two part problem? The simpler one being that something
fundamental changed in dask.delayed (without digging into this more..
I'm assuming Dask's evolution to using dedicated Tasks rather than
tuples maybe led to some internal changes there?). I think it's possible
that change has uncovered an issue in dask-awkward w.r.t. performing
dask-awkward's optimizations for graphs that combine Delayed with
dak.Array. It appears the optimization works when the result of the
delayed object is a concrete awkward Array (as proven by the originally
passing test). The optimization graph is actually getting that
concrete array to materialize during its execution, and the typetracer
array (from the dak.Array) that exists during the compute of
optimization-time-only graph is perfectly fine doing binary operations
with that concrete array (e.g. a * b is totally fine when a has
typetracer backend while b has numpy backend).
Unfortunately its impossible to know exactly what kind of object will
result from an operation between an arbitrary Delayed and a dak.Array,
and the typetracer pass is completely dependent on being able to know
what the result would be. It seems like we somehow need to have metadata
attached to a Delayed

delayed < - > dask-awkward interactions really only happen in HEP when someone is passing a big correction into the task graph where we use delayed to wrap the function.
So I'd argue it should be materialized because in such a case you'd need to pass the typetracer to the function in any case.
I think this is just an essential limitation of "going across the fence" between dak and specifically delayed since you don't know what you're getting when you materialize it. Otherwise it significantly complicates the delayed interface of other packages.
Though maybe it is a wrong choice to do so?

Let's try to first get the status quo re-established since that was at least reasonable behavior. If we can do that we'll have more understanding of what's going wrong, as well.

Lindsey Gray @.***> writes:

delayed < - > dask-awkward interactions really only happen in HEP when
someone is passing a big correction into the task graph where we use
delayed to wrap the function.
So I'd argue it should be materialized because in such a case you'd
need to pass the typetracer to the function in any case.

Yeah the more I think about it the more it makes sense to have whatever
is represented by the Delayed go ahead and "exist" when combined with
the typetracer during dak's optimization pass. The (potentially
terrible) side effect here is that the Delayed thing gets computed twice
(once for optimization pass, and again for the real compute).

I think this is just an essential limitation of "going across the
fence" between dak and specifically delayed since you don't know what
you're getting when you materialize it. Otherwise it significantly
complicates the delayed interface of other packages.
Though maybe it is a wrong choice to do so?

Maybe "questionable" instead of "wrong" :P. certainly useful feature,
but a complex to get it right for all cases. When I first starting
working on Delayed <--> dak stuff, I took inspiration from dask.array
and dask.dataframe; for going between the different collections one
needed to go ahead and at least run the optimizations for the "original"
collection before shoving the (now optimized graph) into the "new"
collection (all the to_delayed functions have an optimize_graph argument
that defaults to true). So optimizations have always been something to
explicitly worry about in this area. Conversions are one thing (I'd say
simpler), but combined operations is another. I wonder if perhaps the
new expression stuff solves that complexity. All that said I agree
getting the status quo working again is absolutely highest priority.

Started to poke around the new Task stuff and found
https://github.com/dask/dask/blob/02185bedf2e6aba70ac2d3987c4f739998da835e/dask/blockwise.py#L388-L404
Looks like
https://github.com/dask-contrib/dask-awkward/blob/1d4d4e9afd3c7c68debc762297b2ff75b6103828/src/dask_awkward/lib/core.py#L1930-L1931
needs to be wrapped in dask._task_spec.Alias. I blindly tried it:
diff --git a/src/dask_awkward/lib/core.py b/src/dask_awkward/lib/core.py
index d7c1a4e..4c8cd45 100644
--- a/src/dask_awkward/lib/core.py
+++ b/src/dask_awkward/lib/core.py
@@ -1910,6 +1910,7 @@ def partitionwise_layer(
         The Dask HighLevelGraph Blockwise layer.

     """
+    from dask._task_spec import Alias
     pairs: list[Any] = []
     numblocks: dict[str, tuple[int, ...]] = {}
     for arg in args:
@@ -1928,7 +1929,7 @@ def partitionwise_layer(
             pairs.extend([arg.name, "i"])
             numblocks[arg.name] = (1,)
         elif isinstance(arg, Delayed):
-            pairs.extend([arg.key, None])
+            pairs.extend([Alias(arg.key), None])
         elif is_dask_collection(arg):
             raise DaskAwkwardNotImplemented(
                 "Use of Array with other Dask collections is currently unsupported."

and got a different error with default compute (e.g. a.compute()), but now the no-optimize case works (a.compute(optimize_graph=False)), so that's some positive progress.
Need to look more into how the Alias impacts the typetracer graph pass
In [1]: import dask_awkward as dak
   ...: from dask.delayed import delayed
   ...: import awkward as ak
   ...:
   ...:

In [2]: x = ak.Array([[1,2,3], [4]])

In [3]: a = dak.from_awkward(x, npartitions=2)

In [4]: @delayed
   ...: def one():
   ...:     return 1
   ...:

In [5]: import operator

In [6]: b = a.map_partitions(operator.mul, one(), meta=a._meta)

In [7]: b.compute(optimize_graph=False)
Out[7]: <Array [[1, 2, 3], [4]] type='2 * var * int64'>

In [8]: b.compute()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 1
----> 1 b.compute()

File ~/software/repos/dask-awkward/.venv/lib/python3.13/site-packages/dask/base.py:372, in DaskMethodsMixin.compute(self, **kwargs)
    348 def compute(self, **kwargs):
    349     """Compute this dask collection
    350
    351     This turns a lazy Dask collection into its in-memory equivalent.
   (...)
    370     dask.compute
    371     """
--> 372     (result,) = compute(self, traverse=False, **kwargs)
    373     return result

File ~/software/repos/dask-awkward/.venv/lib/python3.13/site-packages/dask/base.py:660, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    657     postcomputes.append(x.__dask_postcompute__())
    659 with shorten_traceback():
--> 660     results = schedule(dsk, keys, **kwargs)
    662 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/software/repos/dask-awkward/.venv/lib/python3.13/site-packages/dask/local.py:455, in get_async(submit, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, chunksize, **kwargs)
    452     rerun_exceptions_locally = config.get("rerun_exceptions_locally", False)
    454 if state["waiting"] and not state["ready"]:
--> 455     raise ValueError("Found no accessible jobs in dask")
    457 def fire_tasks(chunksize):
    458     """Fire off a task to the thread pool"""

ValueError: Found no accessible jobs in dask

@pfackeldey @douglasdavis @fjetter
After some further poking around it is specifically running the optimize_blockwise pass that causes the graph to throw ValueError: Found no accessible jobs in dask.
If I merely comment out https://github.com/dask-contrib/dask-awkward/blob/main/src/dask_awkward/lib/optimize.py#L52 (i.e. optimize_blockwise!!)
HLG with optimize.py#52 commented out:
un-optimized: HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x115839310>
 0. a_delayed_array-07e0822d-9a8c-4164-a4b5-26570955e35e
 1. from-awkward-85ae75dd547aa970cf45ae978772e6e4
 2. <dask-awkward.lib.core.ArgsKwargsPackedFunction ob-92230ff6d385deae6aca0cffcabdad9f

optimized:    HighLevelGraph with 1 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x117b7d430>
 0. <dask-awkward.lib.core.ArgsKwargsPackedFunction ob-92230ff6d385deae6aca0cffcabdad9f

with L52 restored:
un-optimized: HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x1089cc980>
 0. a_delayed_array-0f23670a-dae0-49de-b285-3732450abc70
 1. from-awkward-db6b93afdbe5c3bf69e753378b7fd590
 2. <dask-awkward.lib.core.ArgsKwargsPackedFunction ob-25fddb97bdfc2ebcd68404195ebb0836

optimized:    HighLevelGraph with 1 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x108fe31d0>
 0. <dask-awkward.lib.core.ArgsKwargsPackedFunction ob-25fddb97bdfc2ebcd68404195ebb0836

The test passes and at the highest level appears to be producing the same HLG structure, I will dig into it a bit more to try to understand what's packed up differently.

With optimize_blockwise commented out the resulting materialized layer contains this:
[(('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-6c779eab894c25ed7b3f751615b94fe3',
   0),
  <Task ('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-6c779eab894c25ed7b3f751615b94fe3', 0) <dask_awkward.lib.core.ArgsKwargsPackedFunction ob(...)>),
 (('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-6c779eab894c25ed7b3f751615b94fe3',
   1),
  <Task ('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-6c779eab894c25ed7b3f751615b94fe3', 1) <dask_awkward.lib.core.ArgsKwargsPackedFunction ob(...)>),
 (('from-awkward-d54829cedb4c32f715e92986dfca221f', 0),
  <Task ('from-awkward-d54829cedb4c32f715e92986dfca221f', 0) <dask_awkward.lib.io.io.FromAwkwardFn object at 0x(...)>),
 ('a_delayed_array-1abece2e-5d98-44eb-abcb-8ea21563a7ac',
  (<function __main__.a_delayed_array()>,)),
 (('from-awkward-d54829cedb4c32f715e92986dfca221f', 1),
  <Task ('from-awkward-d54829cedb4c32f715e92986dfca221f', 1) <dask_awkward.lib.io.io.FromAwkwardFn object at 0x(...)>)]

where with optimize_blockwise the resulting layer contains:
[(('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-1beb66991b79fa2f49330ef901027c85',
   0),
  <Task ('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-1beb66991b79fa2f49330ef901027c85', 0) _execute_subgraph(...)>),
 (('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-1beb66991b79fa2f49330ef901027c85',
   1),
  <Task ('<dask-awkward.lib.core.ArgsKwargsPackedFunction ob-1beb66991b79fa2f49330ef901027c85', 1) _execute_subgraph(...)>)]

if I then dig into the Task and look at its dependencies I get the set of keys:
frozenset({'a_delayed_array-ffab6114-4add-4b7c-9004-fd3770ba9771'})

and a_delayed_array is definitely not in the graph!
So, how to we keep the delayed node in the graph?

With dask 2024.11.x the results are as follows:
no optimize_blockwise:
HighLevelGraph with 1 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x10af66ba0>
 0. repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05

and that one layer contains:
[(('repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05',
   1),
  ('from-awkward-repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05',
   1)),
 (('repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05',
   0),
  ('from-awkward-repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05',
   0)),
 (('from-awkward-repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05',
   1),
  (subgraph_callable-fc246caf28f6aec6243873f1810cbd77,
   (subgraph_callable-5f18606dd7d6041a38bfe013d4e97716, (2, 4)),
   'a_delayed_array-cc51865c-c9b1-4111-8e9f-98f655e1ced9')),
 (('from-awkward-repacked-instrumented-mul at 0x1093cc540-535da6829bbcce92a42b85b9be259a05',
   0),
  (subgraph_callable-fc246caf28f6aec6243873f1810cbd77,
   (subgraph_callable-5f18606dd7d6041a38bfe013d4e97716, (0, 2)),
   'a_delayed_array-cc51865c-c9b1-4111-8e9f-98f655e1ced9')),
 ('a_delayed_array-cc51865c-c9b1-4111-8e9f-98f655e1ced9',
  (<function __main__.a_delayed_array()>,))]

It executes as expected.
with optimize_blockwise turned on the resulting graph is:
HighLevelGraph with 2 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x1053fc4d0>
 0. a_delayed_array-aa8cfacb-8634-4dc5-a65d-4a19fd4bd6e2
 1. repacked-instrumented-mul at 0x10412e520-7ed96731e732a0de63e1eff62ca2e023

and the last layer contains:
[(subgraph_callable-19a20e3065e8a2b3047767df7a8e80f3,
  'a_delayed_array-aa8cfacb-8634-4dc5-a65d-4a19fd4bd6e2',
  (0, 2)),
 (subgraph_callable-19a20e3065e8a2b3047767df7a8e80f3,
  'a_delayed_array-aa8cfacb-8634-4dc5-a65d-4a19fd4bd6e2',
  (2, 4))]

So the old optimize_blockwise is de-inlining the delayed array into its own node and the new one is completely removing it? Though I think the errant behavior may be in rewrite_layer_chains which shouldn't be touching the Delayed layer in the first place?
@martindurant thoughts?

Ah, my bad - it's after the cull step:
after fuse_roots HighLevelGraph with 2 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x10ee260f0>
 0. a_delayed_array-099cec65-ea25-4b10-8183-48aa676a54b3
 1. repacked-instrumented-mul at 0x10cb6d3a0-070fec221c3aceaa77bd0b44557dd84c

after cull HighLevelGraph with 1 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x10cb098e0>
 0. repacked-instrumented-mul at 0x10cb6d3a0-070fec221c3aceaa77bd0b44557dd84c

So somehow we're not retaining the information in the graph that marks the delayed layer as necessary.

Indeed, if I skip the cull step everything computes as expected.

if I skip the cull step everything computes as expected

So the graph is right, the dependency must be there, else it wouldn't compute - but cull is making some other assumption that we don't meet.

Indeed, something has definitely changed:

Even though the blockwise_optimized layers claim the Delayed as a dependency (!!) cull drops the Delayed.

I have found the source of the difference.
After https://github.com/dask/dask/pull/11568 the delayed array no longer shows up as a constant dependency in the task graph coming from this loop:
https://github.com/dask/dask/blob/main/dask/blockwise.py#L674-L684
I checked it's not another kind of dep, even in old dask the delayed-wrapped array is only a constant.
If I add code to correctly deal with Aliases to https://github.com/dask/dask/blob/main/dask/blockwise.py#L674-L684, i.e.:
...
            if isinstance(arg, Alias):
                arg = arg.target.key
...

Then everything works as expected again.

@fjetter is this skipping of Aliases as constant dependencies of blockwise graphs desired or expected?

I also find that if I use TaskRef instead of Alias for our Delayed object everything executes correctly without issue.
Understanding of what is correct would be appreciated. For the time being I will change the dask_awkward code to use a TaskRef as opposed to an Alias to see what else breaks in our tests.

OK - only failing tests are over in uproot where we'll have to patch things up to deal with Tasks there as well!
@jpivarski

However, we should settle these correct usage issues and get bugfixes in the right places if they are necessary.
We then wait on further input and guidance from @fjetter as to correct/expected usage.

It's also a bit weird that the GraphNode base class's .ref() function returns an Alias. Hmm.

@fjetter when you have time, we would appreciate your commentary so that we can resolve this.

@fjetter just a ping

@fjetter are you available to discuss this? Thanks!

@fjetter Just for reference, the source of confusion about Alias vs. TaskRef w.r.t. constant delayed objects comes from here: https://github.com/dask/dask/blob/main/dask/blockwise.py#L388-L404
I realize it's a sharp edge but since you said TaskRef is correct here, it would be good to correct the docs for posterity.

@martindurant @pfackeldey I would say we merge as-is, and deal with the backwards compat stuff later. We should get uproot and coffea passing again with some priority...

uproot was an easy fix https://github.com/scikit-hep/uproot5/pull/1352


io/htp_proxy_test and io/secure_builtin_roots_test are timing out

The two standalone tests
io/htp_proxy_test
io/secure_builtin_roots_test
are timing out and need investigation.

I noticed today that these tests are just disabled since Feb 2, 2016. Did something happen in this area? We have random timeouts on io/*http* tests - they might be connected. Can somebody revisit this issue?

/cc @kasperl

This issue has been last updated 304 days ago. Please consider investigating whether it is still meaningful and if not close it. If the issue is still meaningful please find a relevant owner. Thank you.
Note: as comment above indicates this tests have been disabled. We should not have disabled tests.

This issue has not been updated for another year. Tests are still disabled. Investigating whether these are still issues.
These tests are also disabled by other status file entries (for other configurations. Sometimes even without issue numbers (I'll list the commits):
io/http_proxy_test is also disabled because of issue #27638, and by commits https://github.com/dart-lang/sdk/commit/bd2a8e1d60f (migrate most of the tests in standalone to be strong mode) and https://github.com/dart-lang/sdk/commit/848d3f4218a (Normalize standalone_2).
$ git grep -n 'io/http_proxy_test'
pkg/status_file/test/data/standalone_2_vm.status:68:io/http_proxy_test: Skip # Issue 27638
tests/standalone_2/standalone_2.status:100:io/http_proxy_test: Skip # These tests have started timing out and issue 25649 has been filed to investigate, skipping these tests temporarily to get the bots to be green again.
tests/standalone_2/standalone_2.status:134:io/http_proxy_test: Skip # Flaky.
tests/standalone_2/standalone_2_kernel.status:114:io/http_proxy_test: Skip # Flaky.
tests/standalone_2/standalone_2_vm.status:33:io/http_proxy_test: Skip # Issue 27638

Disabled by commits https://github.com/dart-lang/sdk/commit/dc35b04a189 (Update status file for different status on bots vs local) and https://github.com/dart-lang/sdk/commit/31d3d2c6b3a (Follow up fixes to make bots green), and because of issues #32137 and #32138:
$ git grep -n 'io/secure_builtin_roots_test'
tests/standalone_2/standalone_2.status:101:io/secure_builtin_roots_test: Skip # These tests have started timing out and issue 25649 has been filed to investigate, skipping these tests temporarily to get the bots to be green again.
tests/standalone_2/standalone_2_kernel.status:124:io/secure_builtin_roots_test: Timeout, Pass # Times out on bots.
tests/standalone_2/standalone_2_kernel.status:165:io/secure_builtin_roots_test: Pass, Timeout
tests/standalone_2/standalone_2_kernel.status:177:io/secure_builtin_roots_test: Skip # Issues 32137 and 32138.

The tests run fine on mac:
$ tools/test.py -m release -v standalone_2/io/http_proxy_test
Test configuration:
    custom configuration(architecture: x64, compiler: dartk, mode: release, runtime: vm, system: mac, vm-options: [], timeout: null, preview-dart-2)
Suites tested: standalone_2
Running "vm" command: DART_CONFIGURATION=ReleaseX64 xcodebuild/ReleaseX64/dart --sync-async --ignore-unrecognized-flags --packages=/Users/me/dart-sdk/sdk/.packages /Users/me/dart-sdk/sdk/tests/standalone_2/io/http_proxy_test.dart
Done dartk-vm release_x64 standalone_2/io/http_proxy_test: pass

$ tools/test.py -m release -v standalone_2/io/secure_builtin_roots_test
Test configuration:
    custom configuration(architecture: x64, compiler: dartk, mode: release, runtime: vm, system: mac, vm-options: [], timeout: null, preview-dart-2)
Suites tested: standalone_2
Running "vm" command: DART_CONFIGURATION=ReleaseX64 xcodebuild/ReleaseX64/dart --sync-async --ignore-unrecognized-flags --packages=/Users/dacoharkes/dart-sdk/sdk/.packages /Users/dacoharkes/dart-sdk/sdk/tests/standalone_2/io/secure_builtin_roots_test.dart
Done dartk-vm release_x64 standalone_2/io/secure_builtin_roots_test: pass

The status file was updated at some point that one of these tests works on mac: https://github.com/dart-lang/sdk/commit/caede1d2ef7.
The status files line with this issue number only disables it on linux:
[ $system == linux && ($runtime == flutter || $runtime == vm) ]

I'll try re-enabling these tests on linux to see whether they still fail.

Tests have not timed out last two weeks, closing issue.


Record refactor

Refactors code to support the record refactor.
Requires the following additional PRs:

biothings/api-respone-transform.js/pull/29
biothings/bte_trapi_query_graph_handler/pull/101

Noting that all currently failing tests are failing for known reasons related to automated github testing and should all pass on local.


DOC: fix Read the Docs builds

The problem was that they migrated to a new yaml version, plus that notifications on build failures in the RTD UI was not enabled for anyone.
Closes gh-522

@rgommers I don't have a way to check that this does the right thing, thus please merge it when you are happy.

I think that using readthedocs does not buy us anything and having the docs built on another platform is more trouble, thus I would be happy to migrate to Github Pages for hosting the documentation.

I think that using readthedocs does not buy us anything and having the docs built on another platform is more trouble, thus I would be happy to migrate to Github Pages for hosting the documentation.

Let's discuss on gh-522? Happy to consider it of course.

I don't have a way to check that this does the right thing, thus please merge it when you are happy

We used to have a CI job for this, but it's no longer running. Not sure why, probably the webhook needs an update. I'll wait with that until we resolve gh-522.

This passes says the RTD interface, so I'll merge it.


Add beep sounds when time is up

Hello, I'm a BJMUNer and I noticed that the Dais need to knock the microphone when there's 1/3 of time (30 seconds for 120 seconds speech) left or time's up. And many of times they forgot that.
How about adding a new feature, for example, add a <audio></audio>, to prevent that from happening?
Thank you.

Quick patch:
let audioBeep = $('audio');
audioBeep.src = 'path/to/beep.wav';
audioBeep.css('display','none');
timer.on('timesup', () => {
    audioBeep.appendTo($(document));
    audioBeep.get(0).autoplay = true;
});

大概就是这么个意思，手机上打字不太方便2333

Thanks for your report. This has been a long-awaited feature, so I'll try to make it happen as soon as I have some free time. As I have been busy with my study, this project have been inactive for a long time, and I'm sorry for the inconvenience.
Regarding that all timers/lists are designed to work in parallel, and the projector window is separate from the controller window, I assume the timer/list that is being projected can ring a bell.

iNotify seems good and light-weight (no dependencies, ~4KB)


Tag Values: Record as text not general

We are seeing tag values that have an 'e' in them being translated by Excel into numbers.
e.g. Tagname: GitCommitID
TagValue: 4182e245
Excel shows the value as 4.182E+245 and makes the cell very wide/won't display the value.
If all tag values were prepended by a ' this would cause them to be read as Text and display correctly?

Hello,
That's a good point, I believe a fix for this would be possible, but I'll have to update all the modules (which might take some time).
Regarding the idea of adding ' : I believe it might work, but I would prefer to format the tag columns as text.


Feature Suggestion: Improvements to Resuming Batches

Is your feature request related to a problem? Please describe
It's unclear if resuming a batch download has remembered what has previously been downloaded, or if it's retrying what has failed.
Describe the solution you'd like
Keep all results previously available in the batch download window. Retain the completed status of those previously downloaded. Remove the error state of failed downloads so they can be retried. Show a Missing/Removed status for any items that are no longer available.
Change the "Downloading pages" message in this case to "Downloading pages and comparing to previous results"
Alternative solutions
Change the "Downloading pages" message in this case to "Downloading pages and comparing to previous results. Completed downloads will be omitted. Failed downloads will be retried."
Also, does "Follow downloaded images" mean to remember previously downloaded images? And is that within the batch only, or across all past batches? It would be good to make this a little clearer. Also, "Remember downloaded images" might be more appropriate, as "Follow" implies checking server status for updates, like following a tag, unless that's what it means?

Also, when an incomplete batch resumes, does it resume from a page position, or does it load all pages, load all files and ignore ones it has downloaded previously? In the first situation, some items can be missed entirely if the search includes an order by a factor that can change, like favorite


Missing Dependencies

I have downloaded the repo, ran the following command, and received this output. Where do I find the Marshaller or do I need to create this directory somewhere?
java -cp target/marshalsec-0.0.3-SNAPSHOT-all.jar marshalsec.jndi.LDAPRefServer "http://x.x.x.x:8888/#Exploit"
zsh: no such file or directory: Marshaller

cd into main/java/marshalsec/jndi and rename the class in your command marshalsec.jndi.LDAPRefServer  to jndi.LDAPRefServer


RoBERTa-large: RTE results in GLUE VS SuperGLUE

Hi,
It's not clear to me why the reported results for RTE with roberta.large here on the README are so different for GLUE (86.6) and SuperGLUE (89.5).
I might be missing something.
Is it because of the variance on 5 runs (median of 5 runs)?
As far as I understand, it's the same task, with the same data (modulo tsv vs jsonl).
Victor

Hey @VictorSanh,
So for GLUE table, the reported results are without initializing from MNLI model to keep it consistent with how XLNet (SoTA on GLUE when we publish RoBERTa).
For SuperGLUE, they are with initializing from MNLI model to keep them consustent with Bert++ (SoTA on SG that time).


remove alert data queue and bugfix cpu surge

remove alert data queue and fix cpu surge
add AlarmSender.alarm use note

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #5168 (bb45dea) into master (31241b7) will decrease coverage by 0.10%.
The diff coverage is 30.00%.

@@             Coverage Diff              @@
##             master    #5168      +/-   ##
============================================
- Coverage     60.86%   60.77%   -0.10%
+ Complexity     8469     8457      -12
============================================
  Files          1242     1242
  Lines         37433    37428       -5
  Branches       3532     3526       -6
============================================
- Hits          22785    22747      -38
- Misses        12692    12717      +25
- Partials       1956     1964       +8

Files Changed
Coverage Δ

...g/apache/shenyu/plugin/base/alert/AlarmSender.java
0.00% <ø> (ø)

...nyu/plugin/base/alert/AlarmThreadPoolExecutor.java
0.00% <0.00%> (ø)

...u/admin/service/impl/AlertDispatchServiceImpl.java
19.40% <31.57%> (-7.36%)
:arrow_down:

... and 13 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[Static Analysis] Integration of SonarCloud for CII BP Badge process

Cross reference from @tdrozdovsky: https://github.com/lf-edge/edge-home-orchestration-go/pull/251#discussion_r567477862

@tdrozdovsky @MoonkiHong How about adding a .sonarcloud.properties file for configuration?
It would be great if we exclude unnecessary files related test, vendor, and so on.
Please refer the edgex-go case below.

edgex-go properties file
edgex-go sonarcloud

@t25kim This is a great idea!


upgrade AWS Java SDK dependency

jump from 1.9.31 to 1.10.32
update joda-time as well
fix overload ambiguity in dynamodb wrapper
(AWS SDK 1.10.5 added new overloads in
AmazonDynamoDBAsync interface that caused
overload ambiguity. This was fixed by being
explicit in the type params for wrapAsyncMethod.)

fixes #37

@glidester, would you mind letting me know whether this resolves your issues with AWS SDK 1.10.x or if you encounter any other issues.

Yes this looks good from my end.
Thanks for resolving this!

Released in v0.8.0


Fix issue with do_delayed_calls_at_end when using multiple threads

Description :sparkles:
update_now_or_later results in error when multiple threads are used, as threadlocal variable is not initialized
Issues :bug: HL-317
Testing :alembic:
Screenshots :camera_flash:
Additional notes :spiral_notepad:

Codecov Report

Merging #448 (6b2f174) into develop (18c8ad5) will decrease coverage by 1.16%.
The diff coverage is 90.90%.

@@             Coverage Diff             @@
##           develop     #448      +/-   ##
===========================================
- Coverage    95.58%   94.42%   -1.17%
===========================================
  Files          201      152      -49
  Lines         7631     5932    -1699
===========================================
- Hits          7294     5601    -1693
+ Misses         337      331       -6

Impacted Files
Coverage Δ

backend/benefit/common/delay_call.py
84.84% <90.90%> (+1.51%)
:arrow_up:

backend/shared/shared/audit_log/viewsets.py
46.96% <0.00%> (-43.94%)
:arrow_down:

backend/shared/shared/azure_adfs/views.py
41.17% <0.00%> (-41.18%)
:arrow_down:

backend/shared/shared/audit_log/audit_logging.py
76.59% <0.00%> (-21.28%)
:arrow_down:

backend/shared/shared/common/tests/conftest.py
87.87% <0.00%> (-12.13%)
:arrow_down:

backend/shared/shared/oidc/auth.py
81.48% <0.00%> (-1.86%)
:arrow_down:

...end/kesaseteli/companies/tests/test_company_api.py

...pplications/jobs/monthly/clean_old_applications.py

...plications/migrations/0009_add_application_user.py

...es/migrations/0005_remove_company_eauth_profile.py

... and 45 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 18c8ad5...6b2f174. Read the comment docs.


Unexpected keyword argument 'scale'

Running the example, I get this error:
File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 702, in __call__
    noise_pred = self.unet(
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\diffusers\models\unet_2d_condition.py", line 1029, in forward
    sample = upsample_block(
  File "C:\Users\45239\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: register_free_upblock2d.<locals>.up_forward.<locals>.forward() got an unexpected keyword argument 'scale'

I'm on the latest alpha of Diffusers and Win 11.

In the latest version, the upblock2d has an argument scale, https://github.com/huggingface/diffusers/blob/v0.21.2-patch/src/diffusers/models/unet_2d_blocks.py line 2292.  I use the version of 0.19.3. You can modify the free_lunch_utils.py like below
`def register_free_upblock2d(model, b1=1.2, b2=1.4, s1=0.9, s2=0.2):
def up_forward(self):
def forward(hidden_states, res_hidden_states_tuple, temb=None, upsample_size=None, scale: float = 1.0):
for resnet in self.resnets:
# pop res hidden states
#print(f"in free upblock2d, hidden states shape: {hidden_states.shape}")
res_hidden_states = res_hidden_states_tuple[-1]
res_hidden_states_tuple = res_hidden_states_tuple[:-1]
            # --------------- FreeU code -----------------------
            # Only operate on the first two stages
            if hidden_states.shape[1] == 1280:
                hidden_states[:,:640] = hidden_states[:,:640] * self.b1
                res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s1)
            if hidden_states.shape[1] == 640:
                hidden_states[:,:320] = hidden_states[:,:320] * self.b2
                res_hidden_states = Fourier_filter(res_hidden_states, threshold=1, scale=self.s2)
            # ---------------------------------------------------------

            hidden_states = torch.cat([hidden_states, res_hidden_states], dim=1)

            if self.training and self.gradient_checkpointing:

                def create_custom_forward(module):
                    def custom_forward(*inputs):
                        return module(*inputs)

                    return custom_forward

                if is_torch_version(">=", "1.11.0"):
                    hidden_states = torch.utils.checkpoint.checkpoint(
                        create_custom_forward(resnet), hidden_states, temb, use_reentrant=False
                    )
                else:
                    hidden_states = torch.utils.checkpoint.checkpoint(
                        create_custom_forward(resnet), hidden_states, temb
                    )
            else:
                hidden_states = resnet(hidden_states, temb, scale=scale)

        if self.upsamplers is not None:
            for upsampler in self.upsamplers:
                hidden_states = upsampler(hidden_states, upsample_size, scale=scale)

        return hidden_states

    return forward

for i, upsample_block in enumerate(model.unet.up_blocks):
    if isinstance_str(upsample_block, "UpBlock2D"):
        upsample_block.forward = up_forward(upsample_block)
        setattr(upsample_block, 'b1', b1)
        setattr(upsample_block, 'b2', b2)
        setattr(upsample_block, 's1', s1)
        setattr(upsample_block, 's2', s2)`

I just created a new branch for diffusers-v0.21.2 support that you can try. I haven't tested it though.

I just created a new branch for diffusers-v0.21.2 support that you can try. I haven't tested it though.

confirm working but the image result still bad https://colab.research.google.com/drive/1A4zo-D-3U-c5lLdk-dc3TMc_4wJp5RET?usp=sharing
how to use with sdxl?

Your suggested change here, made it work. Thank you! I've implemented your Free Lunch solution in Pallaidium, so soon the users will be able to test it there. Thank you!

Tweaking the values a bit, looking at this: https://wandb.ai/nasirk24/UNET-FreeU-SDXL/reports/FreeU-SDXL-Optimal-Parameters--Vmlldzo1NDg4NTUw?accessToken=6745kr9rjd6e9yjevkr9bpd2lm6dpn6j00428gz5l60jrhl3gj4gubrz4aepupda
This was the original values:

Using:  register_free_upblock2d(pipe, b1=0.6, b2=0.4, s1=1.1, s2=1.2)

e values a bit, looking at t

It looks like you got the values of b and s reversed.
should be
s1 = 0.6
s2 = 0.4
b1 = 1.1
b2 = 1.2

Ah, that made a difference. Now using a refiner is not necessary at all:

guys i still didn't understand why the result getting worse with freeu?
generator = torch.Generator(device="cuda").manual_seed(46579129)

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt,generator=generator,).images[0]

original image without freeu block registration

image with freeu block registration enable

@lyn-rgb thank you for your contribution! It also works on 0.18.2 Diffusers version


Can we whitelist certain projects?

There are certain projects where there is a strong belief that it will not violate any laws or regulations.  Examples of this might be things like the C standard library.  If there are projects within ARL that fall into this category, is it possible to whitelist them so that there is no need for OPSEC review on minor releases?  Are there rules or regulations preventing this?  If not, are there any process adjustments that would need to be made to make this possible?

A permanent pass-through and future bypass of OPSEC review would be essential for many of the projects we work on. The very nature of seeing OPSEC approval requires some legal weigh in on the payload, which would make development in the open significantly cumbersome.
Passing through a manuscript, which is human readable, can take days. Passing through a codebase, which is machine parse-able and not lawyer readable becomes a trust proxy type event that necessitates the internal legal council has both a relationship with, and bandwidth for, interacting with people who are at a significantly lower level of detail than what is typical for counsel interactions.

This wouldn't be a complete drop of OPSEC; it is just for the minor reviews.  That said, under ARL's policy there is only ONE person that has to read the complete code; the level-1 OPSEC officer.  Everyone else reads the 1-2 page abstract as described here.  If this is not clear in the policy, then that is a bug and I need to fix it.  If you read through the whole document, what does it imply to you, that everyone needs to read the code, or only the level-1 OPSEC officer?
ARL's plan is that all developers will become level-1 OPSEC officers, and then this turns into a pair-programming exercise.  If we get approval for the whitelisting part, then the pair-programming part will go away.

I like this recommendation from a license/patent review perspective. I am not a fan of this recommendation from an OPSEC perspective.
Issues:

Who creates the whitelist?
Is the person that puts glibc on the whitelist qualified to evaluate glibc?
Is the level of review of glibc consistent with the amount of trust we put into it?
Is the level of review of incremental changes to glibc consistent with the amount of trust we put into it?
The whitelist will represent opportunities for the enemy to introduce malware that will have reduced level of review -- will the list be public?

Proposed solution:
(Maybe this is exactly what you were thinking but in different words.)
To reduce duplication of efforts, ARL will maintain a "preapproved list" of non-ARL projects. ARL OPSEC officers will perform identical review of this software as is performed for ARL public release (or maybe there is a stronger policy for internal, non-public projects). OPSEC officers may add their approval of a non-ARL project to this list by submitting to (some type of higher-level OPSEC officer):

Positive identification of the project, version and files they are reviewing (hash)
Statement of the level of involvement of their review

Did they consider the project as a whole or only the incremental changes?
Did they rely on statements by e.g. Google Zero or assurances made by other parties as to the security of the project?

Statement of how the officer believes that the project is being used in other projects (impact analysis)
Recommended classification level for their review (considering if they used any non-public techniques to perform their evaluation)
Review outcome and caveats

Duplicate entries in the preapproved list are allowed and encouraged. This especially applies in different classification levels.
When an ARL OPSEC officer is evaluating a project for public release (and this policy may also apply for internal project OPSEC review) then that OPSEC officer may consider entries from the preapproved list.
Discussion
The information added to the preapproved list gives the OPSEC officer the information they need to become comfortable with, and trust, the other OPSEC officer's review of that project.
The list itself is sensitive and should probably not be published.

@fulldecent actually, the purpose is to whitelist certain ARL initiated and controlled projects.  For example, BRL-CAD is a very long running ARL Open Source project.  It is already whitelisted, but this was done on an individual, special-case basis.  There are likely to be a large number of similar projects that require only a reduced level of oversight.  I want to know if there is a way of creating a process whereby project owners can request that their projects be whitelisted.  I'm not sure if there is a way under current laws, regs, and rules though.

The use case I was chiming in about was more for things that we have an ultimate intent to do continued development out in the open. If all contributions have the intent to be benign/open/academic in nature, can there be a set of internal checklists for contributors so that their work can happen seamlessly without a cumbersome OPSEC process.
If I'm working on a project where we are trying to optimize something where the payload of the optimization should not be shared due to OPSEC concerns, but the implementation of the optimization strategy can be decoupled from the domain specific sensitive information... I'd like to move that work out in the open.
We also just have a bunch of benign projects that could easily pass a sniff test for sensitivity and be pushed in the open for continued development, but it would be cumbersome to work on them if we'd have to seek OPSEC approval for each commit.

Sorry, I never chimed back in for @ckaran about the level-1 OPSEC officer "hack". That actually would be aligned with what I was lamenting for above. It's a formalized transferal of trust to let the developer themselves review and release the work that they've been recognized to have some dominion over.
It'd be minimally cumbersome to facilitate individual enablement. I'd be in favor of a model like this.

@storrgie OK, just so I'm clear, this means you're comfortable with the way minor reviews are setup right now, right?  The way I see this happening is that all developers become level 1 OPSEC officers, and then they pair off so they can check each other's work.  This should help catch bugs, and generally improve the quality of code as well, so it's not just for OPSEC.
If you're comfortable with all this, then we can close this issue.


Also generate arm Docker containers on release

Hi! And many thanks for this project!
I use matterircd on arm (v7) servers through Docker.
Until now, I built the image on the host after each matterircd update.
As I just notice you already built containers for arm64 hosts, could you also built Docker images for arm?

Thanks, new releases will now also add arm containers


Add: Macedonian keyboard layout

There is missing macedonian keyboard layout I want to add it.
Can you help?

There is no such thing as Macedonian Language, only Macedonian dialect of Bulgarian language! This layout should not be allowed!

If wikipedia doesn't lie, there is a Macedonian language.

Any update on this?


Set sphinx gallery default thumbnail?

the default sphinx gallery thumbnail for examples without a plot is faded out giving the impression that something went wrong or something is missing. But it's valid for an example not to create any matplotlib plots, for example example 0 getting started.
shall we consider replacing the default thumbnail with something else, like the skrub logo?

Yes! I was thinking exactly this looking at the example gallery with the latest merge. Thanks heaps, merging!


chore(tests): separate storyshots into individual tests

Summary

Switch out snapshotWithOptions() for multiSnapshotWithOptions()

"Like snapshotWithOptions, but generate a separate snapshot file for each stories file rather than a single monolithic file (as is the convention in Jest). This makes it dramatically easier to review changes."

Also improves test time by ~30%! Should also speed up our Travis CI builds a bit.

Change List (commits, features, bugs, etc)

Switch out function
Update snapshots, deletes the old monolithic file, creates new separate files.

Acceptance Test (how to verify the PR)

Does this update miss any previous components that were included in the monolithic file?

I've resolved the conflicts with master from the new dashboard components, and generated the new separated snapshot for the added components.
@davidicus @scottdickerson This is ready for review and merge

:tada: This PR is included in version 2.5.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


websockets are blocking ajax requests

When I am trying to access session values by setting the session again from session_Id, my Ajax requests are blocked..!

Sessions in Ratchet are read only. You should not be setting the session id.

It is fixed, is there any way to deploy without using supervisor bcs i am getting an error in installing supervisor.

It is fixed, is there any way to deploy without using supervisor bcs i am getting an error in installing supervisor.

supervisor is just a suggestion. You can use any process manager such as the system ones that come on Linux distributions. Alternatively you can just run the script manually.

supervisor is just a suggestion. You can use any process manager such as the system ones that come on Linux distributions. Alternatively you can just run the script manually.


Update README.md to add Shinkai Desktop

Adding Shinkai Desktop to the list of Apps using Ollama (opensource), free and a two click install! no docker required

hi @jmorganc any feedback so we hopefully can be added to the ollama app list? we are fully opensource: https://github.com/dcSpark/shinkai-apps and I think it's the best UI / UX for a normal user to go from zero to fully locally using an LLM

hi @jmorganca any feedback so we hopefully can be added to the ollama app list? we are fully opensource: https://github.com/dcSpark/shinkai-apps and I think it's the best UI / UX for a normal user to go from zero to fully locally using an LLM


make permit price consistent throughout

Fixes #218.
Pages affected:

/pre-payment
/permit-price
/enter-card
/confirm-card-payment

@sanjaypoyzer
When you go through this and come to the confirmations screen I think it skips the renewal notification step...
https://drive.google.com/open?id=0B5wEPxglOrqoNmJucFJUMWJ3V28

@kaydale That's because you've only just merged that into master, and not merged master or that branch into this one yet

@sanjaypoyzer I've merged it now but its still broken...

@kaydale that screenshot doesn't work


Embedded font refresh

For the next release, it would be nice to update the following embedded fonts to their latest versions:

Andika
Gentium Plus
Scheherazade New

We can also add some other embedded fonts, to help round out Adapt It Mobile's character set support.

New fonts for release 1.12.0:

Annapurna
Charis
Doulos


[BUG] Unpairing function is failed with src/app/ReadClient.cpp:682: CHIP Error 0x00000032: Timeout

Reproduction steps
I have used python controller to done a stress test on my RPI sample dut (all-clusters-app) on Pair-Unpair operation for 10000 iteration. The iteration 5688 is passed without any error but the 5689 is failed to Unpair the dut during the OnReadAttribute step for the currentFabricIndex attribute with the below traceback error and the next iteration 5690 is also passed without any issues.

Traceback (most recent call last):
File "/Users/chaitanyajandhyala/code/python/matter-qa/src/matter_qa/library/helper_libs/matter_testing_support.py", line 1187, in unpair_dut
controller.UnpairDevice(node_id)
File "/Users/chaitanyajandhyala/code/python/matter-qa/.venv/lib/python3.9/site-packages/chip/ChipDeviceCtrl.py", line 452, in UnpairDevice
return self._ChipStack.CallAsync(
File "/Users/chaitanyajandhyala/code/python/matter-qa/.venv/lib/python3.9/site-packages/chip/native/init.py", line 87, in raise_on_error
raise self.to_exception()
chip.exceptions.ChipStackError: src/app/ReadClient.cpp:682: CHIP Error 0x00000032: Timeout

Bug prevalence
Whenever i do this
GitHub hash of the SDK that was being used
3718e996b56f36a553c1b4571f77728396590776
Platform
python, raspi
Platform Version(s)
RPI-4, 8GB RAM
Anything else?
Example of Failed iteartion details:
The iteartion number 5688 is passed below file for the same
TH log: log_2024-04-27_04-43-23.log
DUT log:  Dut_log_5688_2024-04-27T04_43_33_846904.log
The iteration number 5689 is failed with this error below the logfile
TH log : log_2024-04-27_04-43-44.log
DUT log: Dut_log_5689_2024-04-27T04_48_18_251279.log
The iteartion number 5690 is passed below file for the same
TH log: log_2024-04-27_04-48-28.log
Dut log: Dut_log_5690_2024-04-27T04_48_38_665934.log
For the detailed summary please reference this json: summary.json
Script reference: https://github.com/CHIP-Specifications/matter-qa/blob/main/src/matter_qa/scripts/reliability_scripts/TC_Pair.py

closing the ticket for now, following-up with a new ticket..


Clean up imports

remove some unused module imports

thanks for that! I was missing the acceleration does not make sense since its mounted fixed on the wall.


mediapipe android build in win10 failed

I'm trying to build mediapepe android demo but failed:
(base) D:\doc.bazel\hello\mediapipe>bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server after 10 seconds ...
WARNING: API level 29 specified by android_ndk_repository 'androidndk' is not available. Using latest known API level 28Analyzing: target //meAnINFO: Analyzed target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu (127 packages loaded, 11185 targets configured).
INFO: Found 1 target...
Slow read: a 18345904-byte read from C:/users/administrator/_bazel_administrator/6xe6qcvh/external/android_opencv/sdk/native/libs/arm64-v8a/libopencv_java4.so took 5014 ms.
Slow read: a 57251970-byte read from C:/users/administrator/_bazel_administrator/6xe6qcvh/external/remotejdk11_win/lib/src.zip took 8592 ms.
INFO: From Executing genrule @com_github_glog_glog//:logging_h:
select bazel-out/android-arm64-v8a-opt/bin/external/com_github_glog_glog/src/glog/logging.h.arm to be the glog logging.h file.
ERROR: C:/users/administrator/_bazel_administrator/6xe6qcvh/external/com_github_gflags_gflags/BUILD:12:1: C++ compilation of rule '@com_github_gflags_gflags//:gflags' failed (Exit 2)
c:\users\administrator_bazel_administrator\6xe6qcvh\execroot\mediapipe\external\com_github_gflags_gflags\src\util.h(255): error C3861: 'GetTempPathA': identifier not found
Slow read: a 11877669-byte read from C:/users/administrator/_bazel_administrator/6xe6qcvh/external/android_tools/all_android_tools_deploy.jar took 5173 ms.
Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 713.572s, Critical Path: 57.74s
INFO: 26 processes: 26 local.
FAILED: Build did NOT complete successfully
(base) D:\doc.bazel\hello\mediapipe>

Similar to https://github.com/google/mediapipe/issues/91. Unfortunately, MediaPipe doesn't support Windows. If you are only interested in Windows, we support Windows Subsystem for Linux. You can find how to set up MediaPipe in WSL here.

I have figured it out in Virtual Box, Thank you!


Storm-like Full Windowing Support

Testing for windowing level examples

[x] Tumbling Count
[x] Tumbling Duration
[x] Sliding Count
[x] Sliding Duration

Initial Test cases are added and tested. Storm-like native window support is added.


Add a logic to remove stale clusters

When an istio service with an associated endpoint is removed, Pilot seem to
have kept the stale cluster(s) in StreamAggregatedResources, part of the problem
was the initial check was only looking for the length equality of the clusters
from service discovery and the ones that Envoy knows about. This change performs
a deep equality check on both cluster sets. If both sets are equal no
changes made, otherwise we refresh clusters that envoy knows about with
the new set.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Nino-K
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
Assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Hi @Nino-K. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Nino-K, rshriram
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [rshriram]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Nino-K: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests.sh
b6329fd883db5a77fcc0e0602713347a177cd003
link
/test e2e-bookInfo

prow/e2e-mixer-no_auth.sh
b6329fd883db5a77fcc0e0602713347a177cd003
link
/test e2e-mixer-no_auth

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@Nino-K: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests.sh
b6329fd883db5a77fcc0e0602713347a177cd003
link
/test e2e-bookInfo

prow/e2e-mixer-no_auth.sh
b6329fd883db5a77fcc0e0602713347a177cd003
link
/test e2e-mixer-no_auth

prow/e2e-bookInfoTests-envoyv2-v1alpha3.sh
b6329fd883db5a77fcc0e0602713347a177cd003
link
/test e2e-bookInfo

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


I cannot get this working

Hi there
I really would like to get this working but I am struggling. I have followed all the instructions but it still does not work for me.
I cannot see where it has put the gpt.py file. Where should it be?

I managed to sort it all out except getting it to respond to \gpt

Hi @DrSiriusUK,
To make \\gpt work, you need to follow these steps:
First, double click "Snippet"

Second, set the "Keyword" gpt and "Prefix" \\

Btw, could you please share the issue that arises when implementing this workflow? This will allow me to update the README.md file for other users.
Thanks!


If a residue-sampling checkpoint exists, don't add it to the list of residues to be sampled

Pull Request Checklist

[x] Will your PR merge into the dev branch?
Exceptions will be made for urgent bugfixes.
[x] Have you forked from dev?
If not, please rebase your PR onto the most recent dev tip.
[x] Does your PR title succinctly describe the changes?
Explain to a new user by completing the sentence: 'This PR will: ...'
[x] Fill out the template below.

Description of the Change
In a QFitProtein job, we throw all rotameric residues onto a queue, and fork PoolWorker jobs to residue sampling.
At the start of these jobs, we check to see if a residue-sampling checkpoint (multiconformer_residue.pdb) exists.
The issue with this approach is that the map around the residue has to be extracted, pickled, sent to the PoolWorker, unpickled before we check whether this residue has been already sampled. If it has, we do ~1.5s worth of work for nothing!
This PR changes this behaviour.
We now check to see if a residue-sampling checkpoint (multiconformer_residue.pdb) exists before we add the residue to the queue of residues to be sampled.
Will fix #191.
The approach taken also adds some convenience functions to the _Residue, _Ligand and CovalentLigand objects that are used in logging and constructing filepath names.

Release Notes

Performance improvement: if a residue checkpoint exists, don't add it to the list of residues to be sampled (fixes #191)

I think these changes should be propagated to qfit_residue or qfit_ligand.

I think these changes should be propagated to qfit_residue or qfit_ligand.

For qfit_ligand, I agree.
For qfit_residue --- I think that it'd be better (and maybe faster) to enable selecting a single residue when running qfit_protein, and then just remove qfit_residue.


Expose padding property in SuperTextField

We need to add/expose padding property to SuperTextField.
It can be helpful in some cases. Please see the example below.
SuperDesktopTextField already has padding but it's not available through SuperTextField public API.
With padding
https://user-images.githubusercontent.com/4456735/170072322-d1633053-4f2d-43e8-a840-913aa1b974b3.mp4
Without padding
https://user-images.githubusercontent.com/4456735/170071896-8e029ae9-e5de-4640-a1c0-2d24ab7b999c.mp4

FYI @miguelcmedeiros

@matthew-carroll can you assign this to me please


Fix/resource path

Include the resource path for the SVG id, this ensures the browser loader respects public path from webpack.
Closes #11

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


add Install WKHTMLTOPDF in php-fpm container

I completed the 3 steps below:

[ X ] I've read the Contribution Guide.
[ X ] I've updated the documentation. (refer to this for how to do so).
[ X ] I enjoyed my time contributing and making developer's life easier :)

sorry same issue merged from #2469


Failed negotiate version

Hey guys,
I'm trying to run the beancounter tool but receive a Failed negotiate version error:
go run ./main.go compute-balance --type multisig --block-height 1438791 --debug
paste public key:
tpubDBrCAXucLxvjC9n9nZGGcYS8pk4X1N97YJmUgdDSwG2p36gbSqeRuytHYCHe2dHxLsV2EchX9ePaFdRwp7cNLrSpnr3PsoPLUQqbvLBDWv
Result is:
2020/10/19 19:27:58 connecting to testnet.hsmiths.com
2020/10/19 19:27:59 <IP_ADDRESS>:53012 <- {"id":1,"method":"server.features","params":[]}
2020/10/19 19:27:59 <IP_ADDRESS>:53012 -> {"jsonrpc": "2.0", "result": {"hosts": {"testnet.hsmiths.com": {"ssl_port": 53012, "wss_port": 53014}}, "pruning": null, "server_version": "ElectrumX 1.15.0", "protocol_min": "1.4", "protocol_max": "1.4.2", "genesis_hash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943", "hash_function": "sha256", "services": ["ssl://testnet.hsmiths.com:53012", "wss://testnet.hsmiths.com:53014"]}, "id": 1}
2020/10/19 19:27:59 <IP_ADDRESS>:53012 <- {"id":2,"method":"server.version","params":["beancounter","1.2"]}
2020/10/19 19:28:00 <IP_ADDRESS>:53012 -> {"jsonrpc": "2.0", "error": {"code": 1, "message": "unsupported protocol version: 1.2"}, "id": 2}
2020/10/19 19:28:00 error on recv from <IP_ADDRESS>:53012: server error (1: unsupported protocol version: 1.2)
panic: Failed negotiate version

goroutine 1 [running]:
github.com/square/beancounter/utils.PanicOnError(...)
	/Users/ilia/Sandbox/beancounter/utils/utils.go:13
main.doComputeBalance()
	/Users/ilia/Sandbox/beancounter/main.go:229 +0xbe5
main.main()
	/Users/ilia/Sandbox/beancounter/main.go:71 +0x1ca
exit status 2

Do you plan to support the v1.4 protocol?

Is this is a maintained project?


Star/Unstar endpoint return object

Feature Request
Desired behaviour
Star/unstar endpoints should return something non-void to indicate success or failure.
┆Issue is synchronized with this JIRA Story
┆Issue Number: DOCK-134

usual caveat, should be idempotent

@NatalieEO maybe a 500 as well if something really does go wrong server side vs client side.
I defer to Charles and Gary if it would actually be useful/more efficient

If it's already returning different status codes, then it's fine by me but would like the UI to better handle failure.


Everything seems to work but no subtitles added

Using model -> [ small ]
MarkIn: 90000
Rendering Audio for Transcription...
Progress:  100 %
Audio Rendering Complete!
Transcribing Audio...

Transcribe:   0%|          | 0/23.93 [00:00<?, ?sec/s]

Detected language: english

Transcribe:   0%|          | 0/23.93 [00:01<?, ?sec/s]
Transcribe: 100%|##########| 23.93/23.93 [00:12<00:00,  1.89sec/s]
Transcribe: 100%|##########| 23.93/23.93 [00:12<00:00,  1.89sec/s]
Saved: C:\Users\beniz\AppData\Roaming\Blackmagic Design\DaVinci Resolve\Support\Fusion\Scripts\Utility\audio.srt
Transcription Complete!
Subtitles saved to -> [ C:\Users\USER\AppData\Roaming\Blackmagic Design\DaVinci Resolve\Support\Fusion\Scripts\Utility\audio.srt ]
Adjusting timestamps by 0:00:00
Found 25 subtitles in SRT file
Text+ selected as template
Adding template subtitles...
Modifying subtitle text content...
Subtitles added to timeline!

BUT, there are NO subtitles in my timeline and there is also no Subtitle Content in the AutoSubs-Console.

Did you ever find a way round this issue?

Did you ever find a way round this issue?

Nope, but haven't tried anything since then.


Properly fetch and assign reward account value when fetching wallets

Context

Glue code between #903 and #902
Decision

Effectively implement the reward account reporting by fetching the value from the network layer when fetching or listing a wallet.

Acceptance Criteria

The 'balance.reward' of the wallet resource must reflect the actual value for that reward account as known by the node.

Development Plan

PR

Number
Base

#?
master

QA

I don't see why this would be completely blocked by #899 so I'm starting it.

lgtm.


Effects: Multi-Tap support for Presets to update Echo Tempo

So according to the manual, and seemingly in the official app, repeatedly tapping on a effect preset button will adjust the echo tempo value.
From some basic testing and investigation, the app seems to create an array of size 4 which is initially filled with the current value. A BPM value is calculated between the last two presses of the button (based on I would assume time) which replaces an entry in the array and a new BPM is calculated as an average.
Find a way to smartly implement this :D

Also, if I may add:

When the tempo calculation process starts,  stop announcing the preset name on button press to SRS; as they might confuse users who want to sync the tempo with the music they're hearing.
When the tempo calculation process stops, and after the timer passes the 1 second mark and gets canceled, announce the current tempo to SRS, and go back to the default behavior, which is announcing the preset's name on button click.

Thank you!


Very large bundle size

I'm probably doing something wrong with my webpack config but I noticed when I added style-components it added like 200kb to my bundled output.  Seems very odd...

     Asset       Size  Chunks                    Chunk Names
 bundle.js     290 kB       0  [emitted]  [big]  main
styles.css    7.92 kB       0  [emitted]         main
index.html  424 bytes          [emitted]
   [0] ./~/preact/dist/preact.js 20.5 kB {0} [built]
   [1] ./src/styles/index.js 259 bytes {0} [built]
   [2] ./~/styled-components/dist/styled-components.es.js 251 kB {0} [built]
   [4] ./src/styles/global.css 41 bytes {0} [built]
   [5] ./src/styles/normalize.css 41 bytes {0} [built]
   [7] ./src/main.js 370 bytes {0} [built]

Is there something special I have to do to make it so it's not so large??

No, that unfortunately seems correct. v2 will have a much smaller bundle size, as we've worked on this issue a lot.
Note that this is the unminified, uncompressed size. Even in v1 it will go down to 73kb, and 20.9kb gzip'd.
In v2 you're looking at 107kb raw, 33kB minified, 12kb gzip'd right now.
We've also added experimental preprocessing to v2's babel plugin, which removes the need for a css parser, which brings down the bundle size even more.
I hope that clears things up :)

Excellent, just wanted to make sure I wasn't missing something obvious!  Thanks @philpl 👍

I don't know if this is the right place, just trying to follow the contribution guide.

Look for old & closed issues (replying might be ok if they're not too old or have no conclusion)

I really like using styled-components, but the size makes me wonder if I should look for a smaller alternative (like emotion), so I'd like to work on making styled-components smaller and stop asking
that question 😄
Note: 14k isn't crazy big for the feature set it gives, but smaller is better? (I do a bunch of performance work on slow networks so this probably bugs me more than others)
I know @philpl was working on a pre-processing mechanism for 3.x which could strip out some code from the runtime. Happy to contribute where I can.

@siddhartkp react-emotion is also 9kB min+gzip, so not a massive difference anymore.
If you want to work on this a bit that'd be awesome, a big win would probably be to remove all our singleton functions from the modules? (most files export a function to simulate fake dependency injection, but that's proven useless, so we can just get rid of the function exports and just make them normal modules)


Conformer CTC converted with nemo2riva 2.13.1 deployed on Riva 2.13.1 fails to load

I have a conformer CTC model built with the NeMo framework (https://github.com/NVIDIA/NeMo), which can be normally converted and deployed with Riva 2.11.0. However, if I convert the same NeMo file to Riva 2.13.1, and deploy, Riva (Triton server) fails to start with the error
UNAVAILABLE: Internal: onnx runtime error 1: Load model from /data/models/streaming/1/model.onnx failed :/workspace/onnxruntime/onnxruntime/core/graph/model.cc:146 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, constonnxruntime::ModelOptions&) Unsupported model IR version: 9, max supported IR version: 8

I have tried building with --onnx_opset=15, and --onnx_opset=17, like it was mentioned in https://github.com/NVIDIA/NeMo/discussions/7278, but nothing helps.

The same issue exists with Riva 2.14.0, even when deploying a model built with the latest Nemo (1.22.0). It seems to be caused by an incompatibility between the onnx library used by Nemo (1.14.0) and the Riva Triton server's one.
A workaround is to downgrade the onnx library to 1.13.0 prior to nemo2riva conversion, or build a TRT engine -- avoid using the parameter --nn.use_onnx_runtime when building the speech recognition Riva pipeline (i.e. conversion from .riva to .rmir with riva-build).


adding isAggregator flag

This commit adds an optional isAggregator flag to beaconResponseMeta which will allow especially aggregators to avoid duplications & loops.

... and this is a non-breaking addition so no delay necessary.

Would it be better to put it in the beaconInformationalResponseMeta?
I mean that if the endpoint is an aggregator all its beaconResponseMeta are also isAggregator?
Other thing is to distinguish resultsets, but in this case it should be per resultset (ResultsetInstance)?
Dmitry

@redmitry Good point; added it to beaconInformationalResponseMeta, too https://github.com/ga4gh-beacon/beacon-v2/pull/53/commits/4a6efe828a6d2bc70ea6999bfc5bc060fab753c3
But it should also stay in the normal responses since otherwise it could only be accessed w/ multiple passes.
Resultsets: The idea is that all resultSets are treated equally; i.e. an aggregator will concatenate the ResultsetInstance objects from different Beacons in its resultSets. With that we don't introduce another layer or type of response. However, it would be helpful here to have some dedicated source parameter which indicates the originating beacon.

it would be helpful here to have some dedicated source parameter which indicates the originating beacon.
Agree. Ideally it should contain the beacon identifier that originated the result set.
Something like "source": beaconId

@redmitry Like this https://github.com/ga4gh-beacon/beacon-v2/pull/53/commits/3f25a4501d6dad321927009acdbd32261b13457f ?

@redmitry Like this 3f25a45 ?

Yes. just.

"Optional identifier of the beacon beacon this resultSet was collected from, especially for use in beacon aggregators."
"beacon beacon" :-)


Updated sequencer collector filters to accept wider range of data

Current Status
The SequencerCollector scans now in a single-line-change hunk basis, instead of on a single-line-change commit basis. This allows us to collect as many as possible single-line-changes inside the same commit.
Todo in this PR

[x] Tests for the PatchFilter class, since the functionality was extended, tests need to be modified.

Signed-off-by: Javier Ron<EMAIL_ADDRESS>

I believe this one is ready.
Next, I will be storing the diffs in the other repo we created.

ready to merge?

Yes, I believe so. :thinking:


Dataset ( train/test split) of your pretrained cifar-10 model

Thank you for your work and for releasing the model weights
However, I obtained different results using your method on the official CIFAR-10 diffusion model ddpm-cifar10-google with the cifar10 official train/test split.
Could you please release the training/testing dataset corresponding to your pretrained ddpm model?

We have already released the splits https://github.com/jinhaoduan/SecMI/tree/main/mia_evals/member_splits

We didn’t use the official train/test splits. As we mentioned in our paper (Section 5.1), we randomly select 50% of the official training split as the member set and use the rest half as the nonmember set.

Oh, I see. Thanks!


Prediction Model Not Working -Using Own Inputs in a KAN y equation keeps churning out same number no matter what inputs are used

Hi, firstly thank you all very much for all the work done on this so far. It is really cutting edge and a massive discovery so well done.
I'm trying to use the KAN method as predictor of the compressive strength of concrete. I have all the inputs and all the compressive strengths in a CSV however after I've trained the model is seems the y value keeps the same regardless of the inputs, they're all around 45.......
This is the code I've used, is anything glaring wrong?:
from kan import *
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sympy as sp
model = KAN(width=[5,10,1], grid=500, k=3, seed=0) #KAN model size, 5 inputs, 10 neurons in hidden layer, 1 output
file_path = r"C:\Users\GA\OneDrive - University of Strathclyde\Masters\Dissertation\Concrete Data Shtuff.csv" #imports Concrete Data File
Function to read CSV and extract columns within a given range, converting items to numbers
def inputs(file_path,):
# Read the CSV file
df = pd.read_csv(file_path)
# Extract columns 3-7 (index 2-6) within the specified range
extracted_data = df.iloc[1:49, 2:7].values

# Convert all items in the extracted array to numbers
numeric_data = np.array(extracted_data).astype(float)

return numeric_data

def outputs(file_path, start_row, end_row):
# Read the CSV file
df = pd.read_csv(file_path)
# Extract column 8 (index 7) within the specified range
training_outputs = df.iloc[start_row:end_row, 7].values.astype(float)

return training_outputs

df = pd.read_csv(file_path)
S_number=int(input("Please insert which sample you'd like to test for: "))
WCM_ratio=df.iloc[S_number, 2]
C_Content=df.iloc[S_number, 3]
Sand_Content=df.iloc[S_number, 4]
CA_Content=df.iloc[S_number, 5]
Cement_Content=df.iloc[S_number, 6]
day28_Compressive_Strength=df.iloc[S_number,7]
training_inputs = inputs(file_path)
training_outputs = outputs(file_path, 1, 49)
x_train = training_inputs.astype(np.float32)
y_train = training_outputs.astype(np.float32)
x_test = training_inputs.astype(np.float32)
y_test = training_outputs.astype(np.float32)
dataset = {}
dataset['train_input'] = torch.from_numpy(x_train)
dataset['test_input'] = torch.from_numpy(x_test)
dataset['train_label'] = torch.from_numpy(y_train)
dataset['test_label'] = torch.from_numpy(y_test)
model(dataset['train_input'])
results = model.fit(dataset, opt="LBFGS", steps=50, lamb=0.01, lamb_entropy=10)
model.plot(beta=1000)
train_rmse = []
test_rmse = []
for i in range(5):
train_rmse.append(results['train_loss'][-1].item())
test_rmse.append(results['test_loss'][-1].item())
lib = ['x','x^2','x^3','x^4','exp','log','sqrt','tanh','sin','tan','abs']
model.auto_symbolic(lib=lib)
formula = model.symbolic_formula()[0][0]
print(formula)
Dictionary of variable values
variable_values = {
'x_1': WCM_ratio,
'x_2': C_Content,
'x_3': Sand_Content,
'x_4': CA_Content,
'x_5': Cement_Content
}
#new model hopefully retrained
model(dataset['train_input'])
model = model.prune()
results = model.fit(dataset, opt="LBFGS", steps=50, lamb=0.01, lamb_entropy=10)
model.plot(beta=1000)
formula = model.symbolic_formula()[0][0]
print(formula)
formula= str(formula)
Substitute variables in the formula
for var, val in variable_values.items():
formula = formula.replace(var, str(val))
Evaluate the formula
result = eval(formula)
print("The formula after substitution:", formula)
print(f"Predicted Compressive Strength for Sample {S_number} is:", result)
print(f"Actual Compressive Strength for sample {S_number} is: {day28_Compressive_Strength}")
print(train_rmse)
print(test_rmse)
[output] Predicted Compressive Strength for Sample 4 is: 45.7450918077127
Actual Compressive Strength for sample 4 is: 47.16
My regression and train_loss/test_loss is quite high:
| train_loss: 5.38e+00 | test_loss: 5.38e+00 | reg: 8.40e+01 | : 100%|█| 50/50
Might it have something to do with that?
Many thanks in advance for your help

Solution found! The training_outputs had the wrong transform this resolves it
training_outputs = torch.tensor(extracted_data, dtype=torch.float32).unsqueeze(1)


adding adonis framework in Examples

a small Example about how to run nextjs as subapp with adonisjs.

Please run yarn lint --fix / npm run lint -- --fix to fix the code style. Then I'll merge.

I will close it for now.
it's not able to work since adonisjs uses 'use' instead of 'require'.


[Search] Support Azure Search

Closing this issue as basic search functionality has been implemented using Azure Search. All that remains are:

Additional search parameters like skip, take, prerelease, and semVerLevel. See https://github.com/loic-sharma/BaGet/issues/49
Improve rankings. See https://github.com/loic-sharma/BaGet/issues/23

Closing this epic.