Filesystem persistence and using dask as a make/snakemake alternative

I brought this issue/feature request on an issue on at the cachey repo and  @mrocklin  suggested that I open up a new issue here.
It would be useful to add a persistent file-system caching mechanism to dask, to complement its in-memory caching ability. While the main advantage of the in-memory caching is to speed up computations by saving redundant ouputs, a file-system based caching is more about maintaining persistence across sessions, and potential saving interesting intermediate outputs. This is particularly important when developing computational pipelines, because there are frequently bugs in new code which cause crashes. And it is also useful in maintaining persistence between ipython or jupyter notebook sessions. For example, suppose I am developing a workflow in a typical exploratory fashion where A is some dask object:
B = correct_but_expensive_function(A)
C = sketchy_under_development_function(B)
C.to_hdf5("done.h5")

The correct_but_expensive_function could successfully finish after 5 minutes work, but the sketchy_under_development_function might use up all the memory on my linux desktop and force me to quit the process, if I am lucky, or hard-reset the computer, if I am unlucky.
There are many tools out there which are used for automatic scientific workflows by having the users specify some sort of dependency graph between files. These tools are especially popular in bioinformatics, but are broadly applicable in many disciplines. Some popular tools for this include make, luigi, Snakemake, nextflow, and many others. I like these tools a lot, but it can be tedious breaking up a lengthy python script into pieces and manually specifing the dependency graph by hand. This effort seems especially redudant because dask already builds a computational graph behind the scenes.
It would be nice to manually specify nodes of a given dask graph that should be cached and be automatically reloaded from disk or recomputed depending on certain conditions. In theory this could provide a very convenient Make-like replacement which does not require manually specifying file names and allows one to use python data structures. Moreover, some simple wrappers around dask.delayed could be written that allow users to run external command line utilities on the intermediate outputs.
Possible syntax
Some possible syntax for this could be something like this:
cache = FileSystemCache(...)
c = cache.saved(a + b)
d = c**10

The conditions for reloading or rerunning the a+b computation should depend on the type of cache object. Some nice caching reloading/ recomputing conditions could be things like

file modication time (like make)
argument memoization

Another possible syntax, which is slightly more verbose, could be a dict-like cache object:
cache['c'] = a + b
d = cache['c']**10

Automatic memoization of functions using some heuristic like Cachey does could be useful, but I am personally okay with manually indicating which steps should be saved to disk.

I suspect that you could implement this today with a scheduler plugin and an on-disk mutable mapping like shelve or chest.
The plugin would look at a dask graph and its on-disk mutable mapping and replace any keys in the dask graph with the values from the mutable mapping (or functions that obtain those values). It would then cull (dask.optimize.cull) the graph to remove excess nodes that no longer need to be there.
This is what the opportunistic caching plugin does already (dask.diagnostics.Cache) except that it also includes heuristics for what to store and, by default, it uses an in-memory dict (though this can be changed).
Is this enough information to get started on @nbren12 ?

Thanks for the suggestions @mrocklin. I am not very familiar with the dask internals, but I am willing to make a stab at it, since this would be very useful to me. I will take some time to understand how the opportunistic caching plugin works.
Ultimately, it would be also very useful to let the user specify the exact file-type (and possibly path) that they want the data to be stored on disk as. This would then enable including calls to command line tools as part of the graph, like in the following image.

Is this something that the could be handled using delayed?

Delayed doesn't know anything about storing things on disk.  I suspect that
you could make a slightly more complex decorator that wrapped around
delayed and did what you wanted though.
On Fri, Mar 24, 2017 at 1:23 PM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

Thanks for the suggestions @mrocklin https://github.com/mrocklin. I am
not very familiar with the dask internals, but I am willing to make a stab
at it, since this would be very useful to me. I will take some time to
understand how the opportunistic caching plugin works.
Ultimately, it would be also very useful to let the user specify the exact
file-type (and possibly path) that they want the data to be stored on disk
as. This would then enable including calls to command line tools as part of
the graph, like in the following image.
https://camo.githubusercontent.com/45d2cf1cfda0090581b235db57f9a5a961e33d0f/687474703a2f2f692e696d6775722e636f6d2f3634544d6241632e706e67
Is this something that the could be handled using delayed?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-289087629, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszNr8NC4tY5EDGSqpSHKaDJhg9hh0ks5ro_wLgaJpZM4MnVKL
.

Thanks.
Re: marking and caching individual nodes, I implemented a simple Callback with a save method that takes a dask.array object and adds the dask keys corresponding to its chunks to internal list of objects to be saved. It get's called like this
cache = Saved()

np.random.seed(seed=0)
r= rand(100,100)

a = da.from_array(r, (50,50))
b = a**2
c = cache.save(b + a) # just returns a + b, while storing its keys
d = c**10

with cache:
    d.compute()

Unfortunately, it seems like dask.optimize is removing the nodes corresponding to c behind the scenes. Is there some easy way to prevent this?

Okay. I think I can make cache.save modify the dask graph of a+b to add some opaque function call like lambda x: x, which dask.optimize won't be able to inline. Does that make sense?

Actually, that doesn't work because optimize.fuse still removes the nodes I want to store to disk.

You can avoid optimization by adding the keyword optimize_graph=False to your compute call.
Docstring:
Compute several dask collections at once.

Parameters
----------
args : object
    Any number of objects. If it is a dask object, it's computed and the
    result is returned. By default, python builtin collections are also
    traversed to look for dask objects (for more information see the
    ``traverse`` keyword). Non-dask arguments are passed through unchanged.
traverse : bool, optional
    By default dask traverses builtin python collections looking for dask
    objects passed to ``compute``. For large collections this can be
    expensive. If none of the arguments contain any dask objects, set
    ``traverse=False`` to avoid doing this traversal.
get : callable, optional
    A scheduler ``get`` function to use. If not provided, the default is
    to check the global settings first, and then fall back to defaults for
    the collections.
optimize_graph : bool, optional
    If True [default], the optimizations for each collection are applied
    before computation. Otherwise the graph is run as is. This can be
    useful for debugging.
kwargs
    Extra keywords to forward to the scheduler ``get`` function.

Examples
--------
>>> import dask.array as da
>>> a = da.arange(10, chunks=2).sum()
>>> b = da.arange(10, chunks=2).mean()
>>> compute(a, b)
(45, 4.5)

By default, dask objects inside python collections will also be computed:

>>> compute({'a': a, 'b': b, 'c': 1})  # doctest: +SKIP
({'a': 45, 'b': 4.5, 'c': 1},)

Thanks. I did see that command line option, and have been working on a
prototype. I can't decide if it is better to implement persistence on the
low-level dask graph level, or to use the dask.array's  store method.
Ideally, I would like to be able to be able to do things like change the
number of chunks of a dask array without having to recompute the whole
graph.
On Thu, Mar 30, 2017 at 8:33 AM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

You can avoid optimization by adding the keyword optimize_graph=False to
your compute call.
Docstring:
Compute several dask collections at once.
Parameters
args : object
Any number of objects. If it is a dask object, it's computed and the
result is returned. By default, python builtin collections are also
traversed to look for dask objects (for more information see the
traverse keyword). Non-dask arguments are passed through unchanged.
traverse : bool, optional
By default dask traverses builtin python collections looking for dask
objects passed to compute. For large collections this can be
expensive. If none of the arguments contain any dask objects, set
traverse=False to avoid doing this traversal.
get : callable, optional
A scheduler get function to use. If not provided, the default is
to check the global settings first, and then fall back to defaults for
the collections.
optimize_graph : bool, optional
If True [default], the optimizations for each collection are applied
before computation. Otherwise the graph is run as is. This can be
useful for debugging.
kwargs
Extra keywords to forward to the scheduler get function.
Examples

import dask.array as da
a = da.arange(10, chunks=2).sum()
b = da.arange(10, chunks=2).mean()
compute(a, b)
(45, 4.5)

By default, dask objects inside python collections will also be computed:

compute({'a': a, 'b': b, 'c': 1})  # doctest: +SKIP
({'a': 45, 'b': 4.5, 'c': 1},)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-290396804, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABUokgMnFKk1GQURKEL-KE48rx5vXm1Xks5rq6EtgaJpZM4MnVKL
.

Thanks. I did see that command line option, and have been working on a
prototype. I can't decide if it is better to implement persistence on the
low-level dask graph level, or to use the dask.array's  store method.
Ideally, I would like to be able to be able to do things like change the
number of chunks of a dask array without having to recompute the whole
graph.

@mrocklin I have been thinking some more about how to integrate native dask operations with external command line utilities. Stringing together several external calls using dask.delayed should be pretty straightforward, but reading the output of this data back into dask is not straightforward. To do this, I think there needs to be some way to store/read a dask array from a delayed object (like a filename). For example, if f is a function returning a filename, it would be nice to be able to load data like this da.from_array(delayed(f)(*args), ...). Do you think this would be difficult to implement?
With this functionality, it would be possible to implement a really slick interface for combining command line tools with dask.array operations. I have something like the following in mind
# start with dask arrays a and b

@shell(chunks_shape='preserve')
def f(input, output):
    subprocess.check_call(["external_command", input, output])
    return output

# some python operations...
c = a+b

# The @shell decorator automatically saves c to disk, runs f, and loads output.
# to do this it would need some sort of delayed vesion
d = f(c)

See from_delayed
http://dask.pydata.org/en/latest/array-api.html#dask.array.from_delayed
On Thu, Apr 6, 2017 at 6:29 PM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

@mrocklin https://github.com/mrocklin I have been thinking some more
about how to integrate native dask operations with external command line
utilities. Stringing together several external calls using dask.delayed
should be pretty straightforward, but reading the output of this data back
into dask is not straightforward. To do this, I think there needs to be
some way to store/read a dask array from a delayed object (like a
filename). For example, if f is a function returning a filename, it would
be nice to be able to load data like this da.from_array(delayed(f)(*args),
...). Do you think this would be difficult to implement?
With this functionality, it would be possible to implement a really slick
interface for combining command line tools with dask.array operations. I
have something like the following in mind
start with dask arrays a and b
@shell(chunks_shape='preserve')
def f(input, output):
subprocess.check_call(["external_command", input, output])
return output
some python operations...
c = a+b
The @shell decorator automatically saves c to disk, runs f, and loads output.
to do this it would need some sort of delayed vesion
d = f(c)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292343120, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszBJxh6E9KA_M28Tjt9-P6cvkQ8Btks5rtWcsgaJpZM4MnVKL
.

Thank you!
Unfortunately, the docs say that "the dask array will consist of a single chunk", which won't scale for large datasets, but I guess I could concatenate the individually chunks manually to construct the full array..
What about a delayed store to a delayed object (like an hdf5 file which has yet to be created)? I could imagine this need arising if all the workers cannot see the same filesystem, and need to be able to dump a dask array to disk for processing with some external utiliity.

Yeah, you need to construct many delayed tasks that produce many numpy
arrays, call da.from_delayed and then call da.concatenate or da.stack.
Here is an example:
http://matthewrocklin.com/blog/work/2017/01/17/dask-images
On Thu, Apr 6, 2017 at 7:18 PM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

Thank you!
Unfortunately, the docs say that "the dask array will consist of a single
chunk", which won't scale for large datasets, but I guess I could
concatenate the individually chunks manually to construct the full array..
What about a delayed store to a delayed object (like an hdf5 file which
has yet to be created)? I could imagine this need arising if all the
workers cannot see the same filesystem, and need to be able to dump a dask
array to disk for processing with some external utiliity.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292358152, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszIRUSx3BMqXuZ2jbR_woaP_XCVNdks5rtXLigaJpZM4MnVKL
.

Ok that seems like a reasonable approach for loading arrays. Hopefully it
wouldn't incur too many performance issues for large arrays. What about
storing to a delayed  task which, say, creates an hdf file and returns a
variable where the data will be stored.
-Noah
On Thu, Apr 6, 2017 at 7:21 PM Matthew Rocklin<EMAIL_ADDRESS>wrote:
Yeah, you need to construct many delayed tasks that produce many numpy
arrays, call da.from_delayed and then call da.concatenate or da.stack.
Here is an example:
http://matthewrocklin.com/blog/work/2017/01/17/dask-images
On Thu, Apr 6, 2017 at 7:18 PM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

Thank you!
Unfortunately, the docs say that "the dask array will consist of a single
chunk", which won't scale for large datasets, but I guess I could
concatenate the individually chunks manually to construct the full array..
What about a delayed store to a delayed object (like an hdf5 file which
has yet to be created)? I could imagine this need arising if all the
workers cannot see the same filesystem, and need to be able to dump a dask
array to disk for processing with some external utiliity.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292358152, or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AASszIRUSx3BMqXuZ2jbR_woaP_XCVNdks5rtXLigaJpZM4MnVKL
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292358818, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABUoksDy6QOOniyzAIhLrFURUUHybVyzks5rtXN1gaJpZM4MnVKL
.

ok. I made a small modification to dask.array.insert_to_ooc which allows storing a dask array to a delayed task, I also added a simple test. Would you be open to a PR?

Have you seen da.store?
On Thu, Apr 6, 2017 at 9:21 PM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

ok. I made a small modification to dask.array.insert_to_ooc which allows
storing a dask array to a delayed task, I also added a simple test. Would
you be open to a PR?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292388898, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszNEdhnZryS243nGVIzpVaND1eqN7ks5rtY-wgaJpZM4MnVKL
.

Indeed I have, but it doesn't work when the output array is a delayed value, so I had to modify insert_to_ooc to make it work.

The use case I am thinking of is this
@delayed
def create_on_disk_temporary():
     f = h5py.File(...)
     v = f.create_dataset(...)
     return v

delayed_store =x.store(create_on_disk_temporary(), compute=False)

Currently, the store method is not fully lazy because the output array needs to be created when the graph is constructed. My modification allows the output dataset to be created when the graph is executed.

Yes, supporting delayed values in the .store method seems like a good idea
to me.
On Thu, Apr 6, 2017 at 10:28 PM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

The use case I am thinking of is this
@delayed
def create_on_disk_temporary():
f = h5py.File(...)
v = f.create_dataset(...)
return v
delayed_store =x.store(create_on_disk_temporary(), compute=False)
Currently, the store method is not fully lazy because the output array
needs to be created when the graph is constructed. My modification allows
the output dataset to be created when the graph is executed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292405359, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszJYPiM27Dc1qQEkz5F2Zw7mPh1qXks5rtZ9LgaJpZM4MnVKL
.

I thought some more about it, and it might be better for the delayed to return a generator which yields the storage array, and then performs any necessary cleanup like closing files, etc. This is a bit trickier to implement though.

Generators are a no-go.  Results of tasks should be concrete.
On Fri, Apr 7, 2017 at 11:41 AM, Noah D Brenowitz<EMAIL_ADDRESS>wrote:

I thought some more about it, and it might be better for the delayed to
return a generator which yields the storage array, and then performs any
necessary cleanup like closing files, etc. This is a bit trickier to
implement though.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/2119#issuecomment-292572128, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszLZnt81gS2cKhIcrZprJt7bZH88qks5rtlk3gaJpZM4MnVKL
.

Ok. my current code should work then. I guess it would be possible for users to keep track of the opened files in some sort of queue, and close them if necessary.

Hello,
I jump in the discussion maybe a little bit late, but I want to share a question/proposition.
I am quite a beginner in dask so take it as a very naive proposition and I could understand that It is not relevant at all!
Should it make sens to have a persistency mechanism for Delayedcollections using an option similar to dask_key_name?
I imagine something like:
delayed(f)(dask_serializer=serializer, *args, **kwargs)

where serializer is an object with a method serializer.dump(key, value).
If the object serializer is also able to give the state of the data and to load it (with methods like serializer.load(key), serializer.is_computed(key)), it could permits to reload the data instead to compute them again.
But, maybe the drawback of that is that it necessits to introduce a notion of serializer, which should be not desired in the core a dask.
I played with that in this file on my github repo. This is done with a little different spirit than Delayed but it might be adapted for core Delayed object.

+1 I would love to see this functionality in dask.
The below is a brain dump; I hope it doesn't drag on too much (skip to the end if you don't want context). @mrocklin following up on https://github.com/mrocklin/streamz/issues/14, when building a data pipeline in the financial modelling / trading space, my typical workflow is something like

Create a simple DAG with some largely static input data (say stock prices)
Develop some computed data (calculate stock returns or some other metric)
Run some evaluation metric
Repeat

I've used Luigi pretty extensively in the past, and also joblibs memory.cache, and my thoughts are:
Luigi
Pros:

Pretty stable
Linking dependencies is very clear and disk storage can be intuitive (custom defined filenames and lots of custom plugins for persistence)
Error handling aggregation is very good, easy to see systematic bugs across a grid of parameters (can easily see if we are missing data for a particular reporting date for example)
Cons:
The scheduler isn't great and it isn't really designed to run sub 1 min operations.
Changes to the data in an upstream node do not trigger the re-computation of downstream nodes
Lots of boilerplate for simple operations

joblib.cache
Pros:

Speed
intelligent hashing of function & inputs. I believe that joblib actually hashes the whole function and any changes to the function code invalidates the cache.

Cons:

No easy way to specify dependencies / a pipeline
Filenames are a hash rather than something intuitive - which probably can't really be avoided but makes debugging slightly more complex.

I think I am would ideally like to have the following for my specific pipeline needs:

Reasonable fast scheduler (diagnostics a bonus)
DAG capabilities
Persistence to disk
"Flow-on" computation; if  a node cache is invalidated, child nodes are also invalidated.
Aggregated exception handling / visibility

So finally, my actual questions:

how to add a disk based cache into dask. I believe this is answered above - some sort of joblib.memory scheduler plugin.
How best to go about adding in some more scheduler visualisation re task status, particularly errors.
How would I go about passing information down the DAG if a parent nodes cache is invalidated?

how to add a disk based cache into dask. I believe this is answered above - some sort of joblib.memory scheduler plugin.

What is your objective for a disk-based cache?  Are you worried about failure?  Or do you want to permanently store every intermediate value?

How best to go about adding in some more scheduler visualisation re task status, particularly errors.

You might want to take a look at the current visualization code in the dask.distributed scheduler here: https://github.com/dask/distributed/tree/master/distributed/bokeh

How would I go about passing information down the DAG if a parent nodes cache is invalidated?

That probably depends on how you decide to build your DAG system (which as I understand it is separate from Dask's task graph).
I hope that whatever you build I get to see eventually :)  It would be interesting to see your situation more concretely

What is your objective for a disk-based cache? Are you worried about failure? Or do you want to permanently store every intermediate value?
Permanently store immediate values. Typically this data is all time series, so this would actually be intermediate Dataframes. We have multiple complex DAGs, so the ability to just rerun all graphs, and intelligently fill in gaps, or invalidate changes (changes to operations/functions could come from end users) would be amazing. We typically run a DAG, then fit a model and evaluate, so the ability to make some changes (perhaps to multiple operations/functions) and rerun the entire DAG for model evaluation is very powerful.

That probably depends on how you decide to build your DAG system (which as I understand it is separate from Dask's task graph).
In the past it has been separate but I would much prefer it to be a dask task graph. Assuming it is in dask, how would I go about invalidating children?

FWIW, since I opened this issue I have been using snakemake lately for this stuff, and for each step of the workflow, I may or may not use dask. While it seems redundant to use two separate DAG systems--- an inner one for computations using dask, and an outer dependency graph between input/output files using snakemake---it might be a lot of work to implement some the conveniences of snakemake (e.g. job submission, re-runnable workflows, make style modification time based rerunning) using dask. @mrocklin Do think these kinds of things are within the scope of this project? Or should dask only be used within the individual steps of a luigi/make/snakemake workflow?

@nbren12 what has been your experience with snakemake thus far? What do you see as the downside? The appeal of dask IMO is the low task overhead and the intelligent scheduler. These two things are somewhat important, which is why I am even considering trying to fit dask to my problem rather than just using Luigi (or snakemake etc).
It may be that this work is outside of the scope of dask/distributed and the answer lies somewhere between snakemake and dask but it is not clear to me yet.

Wanted to share an Array Cache that can be used with Dask Distributed. Much like has been discussed here, the interest in this cache would be to store intermediate values that have long term value while allowing for their retrieval. As there is a lot of value in being able to name the intermediate value placed in the cache, found the MutableMapping interface to be a nice fit. Have found come up with a simple solution shown in this Gist.

Is this issue still being considered? I think it's priceless.
As mentioned multiple times in the docs, dask with a distributed scheduler is a great for running workloads also on a single machine. Persisting on disk can also be very useful when RAM is limited but computation time is long + you want transparent caching to disk without having to explicitly write out a parquet file or such.

Graphchain might be of interest to this thread. It is a caching optimiser for dask graphs. Some of its features:

Caches dask computations to any PyFilesystem FS URL, including for example the OS filesystem, to Memory, and S3.
Cache keys are based on a chain of hashes (hence the name graphchain) so that we can identify a cached result almost immediately. This is different from a joblib.Memory-style approach of hashing a computation's inputs (which can be very expensive when these inputs are large numpy arrays or pandas DataFrames).
A result is only cached if it is expected that it will save time compared to just computing the computation (which depends on the latency and bandwidth characteristics of the cache's PyFilesystem).

It can be used as a dask graph optimiser (e.g., with dask.config.set(delayed_optimize=graphchain.optimize):), or with the built-in get convenience function (i.e., graphchain.get(dsk, keys, location='s3://mybucket/__graphchain_cache__')).
There's two years worth of comments here, so I'm not sure if it fits all the use cases described, but if you have any feature requests we'd be happy to take a look!

This looks a little stale, but including a note here since I spent all weekend trying to do similar things!
As long as your distributed nodes have access to a shared filesystem, you can use joblib.memory cached functions directly with dask. The cached functions gets pickled/sent as normal, and the workers apply the normal memory logic when calling the function. For example:
mybag = db.from_sequence(long_list)
results = mybag.map(expensive_function)

could be cached at the individual function call level with
mybag = db.from_sequence(long_list)
memory = Memory('./cachedir')
results = mybag.map(memory.cache(expensive_function))

and as long as all of the workers see cachedir in their working directory it works out. This scaled very nicely (>10k tasks, >200 workers) since the cache logic was distributed and solved most of the problems I was facing.
I was able to get graphchain to work for simpler cases, but for map operations on partitions it looked like

The resulting caches were specific to individual partitions, so if you change partitions or datasets it might have to recompute and
The graphchain.optimize logic got very time consuming for large numbers of partitions (>1000 tasks)

Hope this helps someone, and this might be related to https://github.com/dask/distributed/issues/3811

It might be worth filing an issue with the graphchain team about the things it struggled with (if you haven't already).

After some thought, I now think that dask and a larger workflow scheduler occupy different niches. Yes tools like Dask and airflow both represent tasks as a DAG, but the similarity ends there. Tools like airflow schedule and observe individual long-running tasks, whereas I think dask is focused on coordinating a swarm of tiny tasks.
That said, the ability to move data to disk for global shuffles is a really nice feature of spark.

I think that makes Prefect particularly interesting as it uses dask for the scheduling of larger workflows.
That said, there's additional engineering overhead in understanding and using Prefect or larger schedulers, but joblib+dask is pretty easy to explain/use!

Agreed. joblib + dask will work really well for non-distributed computations.

It seems like this discussion has reached a reasonable conclusion. Please feel free to open a documentation pull request if there is a place where you think this conversation can be distilled. I am closing this issue now.


fixed the version table

It now reflects the latest available & supported versions

@PoulChapman : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"


DOC: Build and upload 1.18 reference documentation

One of the last steps in the release is to upload documentation to numpy.org/doc via the https://github.com/numpy/doc repo. This is new to 1.18.
It would be nice if someone else would work through the steps to make sure the directions are clear,  but I can do it if desired.

Sorry, wrong repo :(


Cancel Fly mode

Hello
Is there a way to undo/cancel a fly in fly mode?
auto-pairs's fly mode has an <M-b> mapping to back-insert the character skipped by fly mode where it was typed, effectively undoing the fly like if it wasn't enabled.
Can this be done with this plugin?
Thanks

It can be done.
Will add it in a future release.

Done, you can now do ua.setup({extensions={fly={undomap='<M-b>'}}}).

Unfortunately, this isn't working for me, I'm getting this error when trying to undo a fly:
.../lua/ultimate-autopair/extension/fly.lua:73: attempt to index field 'save' (a nil value)
stack traceback:
        ...t/ultimate-autopair-nvim/lua/ultimate-autopair/debug.lua:77: in function '__index'
        ...te-autopair-nvim/lua/ultimate-autopair/extension/fly.lua:73: in function <...te-autopair-nvim/lua/ultimate-autopair/extension/fly.lua:72>
        [C]: in function 'xpcall'
        ...t/ultimate-autopair-nvim/lua/ultimate-autopair/debug.lua:97: in function 'run'
        ...rt/ultimate-autopair-nvim/lua/ultimate-autopair/core.lua:96: in function <...rt/ultimate-autopair-nvim/lua/ultimate-autopair/core.lua:87>

Can you give an example?

Sure. On my config:
require('ultimate-autopair').setup({
  extensions = {
    fly = { undomap='<M-b>' },
  },
})

Then, editing e.g. a .nix file:
pkg = |  <- type {
pkg = {|}  <- type }
pkg = {}|  <- press <M-b> to undo fly
pkg = {}|}  <- expected this, but get the traceback above

Fixed.

Thank you very much! :pray:


Mouse Scroll Wheel Input

I just got around to this, you can only detect mouse wheel using Input.is_action_just_released so there probably should be an option to allow/disallow mouse wheel inputs

I'll look into it.


/action_runner/LocProject.json pointing to wrong spot

https://github.com/microsoft/PowerToys/blob/master/src/action_runner/LocProject.json seems to be wrong. I think "runner" should be "action_runner" instead, please follow up with Dev to make a correction.
{
    "Projects": [
        {
            "LanguageSet": "Azure_Languages",
            "LocItems": [
                {
                    "SourceFile": "src\\runner\\Resources.resx",
                    "CopyOption": "LangIDOnName",
                    "OutputPath": "src\\runner"
                }
            ]
        }
    ]
}

**


library homepage link fixed

Codecov Report

Merging #20 (82904b6) into master (1cd65ec) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master       #20   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           17        17
  Lines         1388      1388
  Branches       240       240
=========================================
  Hits          1388      1388

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1cd65ec...82904b6. Read the comment docs.


Upgrade from V 0.7.0 to 4.5 does not work

After spending the time to upgrade from Botkit 0.7.0 to 4.5 my app no longer works. It seems to connect without error, but does not come online or respond to anything.  Version 0.7.0 works fine with the same slack app, token and signing secret.
Am I missing something simple?
code:
    const slackAdapterOptions: SlackAdapterOptions = {
            botToken: auth.slackToken,
            clientSigningSecret: auth.slackSigningSecret,
            redirectUri: '' // why is this required?
        }) {

      const adapter = new SlackAdapter(slackAdapterOptions);
      const controller = new Botkit({
        adapter,
        webserver_middlewares: [],
        webhook_uri: '/api/messages',
      });

      controller.hears('hello','message', async(bot: BotWorker, message: BotkitMessage) => {
       console.log("word up!")
       await bot.reply(message,'I heard you say hello.');
      });
     controller.spawn();

Are you set up to use the events API?

That could be.  Let me read up - Thanks.

Going to close this. We ended up forking 0.7.4 and updating the security vulnerabilities ourselves, because it was less effort and the old code actually works. I'm sorry to see what Microsoft has done.

@jfelten, could you comment more on how you updated security vulnerabilities. I'm trying to upgrade my codebase (used botkit-starter-web to get started, at version 0.0.1, which depended on botkit 0.6.12, while remaining open source; also using watson-middleware-watson 1.8.2)

you can update dependencies by running npm audit -fix


Patient ID is not displayed when route is refreshed

Hi all
I have a query, when viewing an image directly from the list of studies the image loads me well with all the patient data, but when doing the same example with an iframe it does not bring me the patient's ID, someone knows what this should .
Thank.
without iframe

with iframe

I correct is not with the iframe, this happens when reloading the page with the exam

I am unable to reproduce this with the LIVE OHIF Viewer demo: https://viewer.ohif.org/viewer/<IP_ADDRESS>.4.1.25403.345050719074.3824.20170126085406.1

Are you able to provide a minimal reproduction?
What version of the viewer are you using?
Have any modifications been made / is it a fork?

Hello,
Thanks for answering, I am using the version using Viewers-fix-issue-183, because for some reason it is not found in GITHUB but it is the most stable version I have tested, the others have given me problems with the proxy wado and different types of errors in the visualization of the image.
I also test the LIVE OHIF Viewer and the error that you mentioned does not occur.

@balth1590, if you create or link me to the issues that are preventing you from upgrading, I may be able to assist with those. However, it sounds like this specific issue is resolved.
It may be worth noting that we have a large release coming soon in the react branch.


Replace “software crafting” with “software as a craft”

Community already using it: @rachelcarmena, @dirv...
Happy to see more and more people ditching non-inclusive terms ❤
Consistency will hopefully help more and more to follow ;)

Extensively thought about this. At the end… decided not to replace it, just using them alternatively (as well as “craft”, “as a craft”, “crafters”, etc.).
I think that illustrates the point of the name not being the important thing (inclusivity is).


[Bug] DockerCompose安装没有表结构

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
研究了三天，问题很多
1,按以下文档安装，基于已有的mysql, 那个.env文件下载不存在。使用了h2中的.env配置的。然后报异常提示表不存在,现在主要看这个问题。
https://streampark.apache.org/zh-CN/docs/user-guide/docker-deployment
2,h2方式安装后 flink任务启动不了。一直在init阶段然后就报了资源不足。但是我独立安装的flink就能瞬间启动。
3,安装后里面默认的演示示例无法运行。
StreamPark Version
2.0
Java Version
No response
Flink Version
1.16
Scala Version of Flink
1.12
Error Exception
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-02-01 11:01:18 | ERROR | main | org.springframework.boot.SpringApplication:824] Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'webMvcObjectMapperConfigurer' defined in class path resource [springfox/documentation/spring/web/SpringfoxWebMvcConfiguration.class]: BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'authorizationAttributeSourceAdvisor' defined in class path resource [org/apache/streampark/console/system/authentication/ShiroConfig.class]: Unsatisfied dependency expressed through method 'authorizationAttributeSourceAdvisor' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityManager' defined in class path resource [org/apache/streampark/console/system/authentication/ShiroConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.shiro.mgt.SecurityManager]: Factory method 'securityManager' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'shiroRealm': Unsatisfied dependency expressed through field 'userService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userServiceImpl': Unsatisfied dependency expressed through field 'memberService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'memberServiceImpl': Unsatisfied dependency expressed through field 'teamService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'teamServiceImpl': Unsatisfied dependency expressed through field 'applicationService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'applicationServiceImpl': Unsatisfied dependency expressed through field 'k8SFlinkTrackMonitor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'registerFlinkK8sWatcher' defined in class path resource [org/apache/streampark/console/core/task/FlinkK8sWatcherWrapper.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.streampark.flink.kubernetes.FlinkK8sWatcher]: Factory method 'registerFlinkK8sWatcher' threw exception; nested exception is org.springframework.jdbc.BadSqlGrammarException:
### Error querying database.  Cause: java.sql.SQLSyntaxErrorException: Table 'streampark.t_flink_app' doesn't exist
### The error may exist in org/apache/streampark/console/core/mapper/ApplicationMapper.java (best guess)

Screenshots

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

说英文， 不说英文要被劝退的


Dotty effect system

Hi,
this plugin is really interesting, and it solves many things I've been trying to do for a long time.
But I wonder if you have talked with the dotty team about it, because they are thinking about implementing an effect system based on implicits (with capabilities to produce effects).
As the functionality of both things overlaps in some areas, do you think your idea could be used in the dotty effect system to get the best of both?
Just to be clear, I have no idea about the implementation details of the effect system, I just know what Martin Odersky explains in his presentations.

Thanks! Sure, the Dotty team is aware of this, and I believe they will end up with something similar once their design is more fleshed out.


content-api: list products python example

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

BingAds/shopping-content/code-example-get-list-product-offers.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Remove "Preview" tag in VS Code extension

I removed the preview field in cd1d762d3deebb5d0684612df01c151699834e93, but that didn't seem to fix it. Maybe the default is true, and it should be there, assigned to false?

Seems like this may have just been a UI bug in VSCode. Now, the "Preview" badge is gone.


Hydra not triggering Manald in Simulator

Hydra not triggering Manald in Simulator

I don't know the proc coefficients for hydra attacks, will have to investigate first.

Hi Riv,  I assume you've seen this by now, but just in case...
https://us.battle.net/forums/en/d3/topic/20753695205
Couple of the more important bullet points:

Each head has 15% chance to proc Paralysis, and therefore MH. (Yes, it's the same chance regardless of which head cast.)
Hydra is treated as a speed coefficient = 1 spell. This means when your hydra head procs paralysis it will proc manald heal exactly one time.

It has to be lightning hydra but if I'm understanding it correctly... Every lightning damage attack has a flat 15% chance to proc paralysis and when paralysis is proc'd, so is manald heal:

Hydra is treated as a speed coefficient = 1 spell. This means when your hydra head procs paralysis it will proc manald heal exactly one time. It does not have a "proc window" mechanism.

 So 15% per zap, seems easy enough. Does it also say that the 255% multiplier from the skill is included somehow?

Yes (looks like you found it) and attack speed:

Attack Speed is factored in as a damage component for Hydra MH procs. Higher AS will increase your hydra MH proc damage. Hydra MH proc damage scales 1:1 with any AS increase or decrease.

I added the mechanic, though I ignored the last section with the list of stats that don't affect proc damage, since it is a bit too complicated to separate these atm.


Rework content scale to provide more options, add integer scaling

I need help to finish this feature. If you want to see this for 4.0, please lend a hand :slightly_smiling_face:
cc @yukitty, @starry-abyss

Add content stretch modes Fractional (default), Integer and Hybrid.

All modes with with any stretch mode and aspect.
Fractional behaves as Godot currently does when scaling viewports. Pixel art will only look good when the window is scaled at multiples of the original size.
Integer enforces an integer scale for the final display (rounded down from the automatically computed fractional scale). Black bars may be displayed on all sides when using the viewport stretch mode. This provides crisp pixel art appearance.
(Not implemented yet): Hybrid enforces an integer scale on the internal rendering resolution, but the viewport will be stretched down to the final display with linear filtering. This provides good pixel art appearance, though not as crisp as the Integer mode.

Tweak project setting for stretch scale to allow values as low as 0.5 (values below 1 are valid, and can be useful for UI testing purposes).

Testing project: test_stretch_master.zip
TODO

[ ] Make UI anchors work properly when using integer scaling mode.
[ ] Implement hybrid scaling mode. (Not critical for 4.0 – this can be left for a future PR.)

This closes https://github.com/godotengine/godot-proposals/issues/1666.

I've just attached a minimal hybrid scaling project for Godot 3 implemented in nodes: https://github.com/godotengine/godot-proposals/issues/1666#issuecomment-1195768620
Hope it helps!

Maybe I don't fully understand how to use this, but I don't see what I expect to see with integer scaling. When I resize the window, the new integer scale isn't automatically chosen. Instead, there is a smooth fractional scaling.
Ideally I'd like to have 2 approaches:

the default one with fixed game size, bars on sides and automatic scale calculation to fit all the scene with max possible zoom while staying integer.
Example 1: if game size is 400x300 and window size is 700x600, then scale is 1, because for a scale of 2 one needs 800 or more window pixels horizontally. So there would be 150 pixels bars on each side around the game scene.
Example 2: if game size is 400x300 and window size is 1500x1000, then scale is 3, with 150 pixels bars on the left and right, and 50 pixels bars on top and bottom.
a special case for some games: imagine a FPS or RTS style 3D game, where the amount of what you can see depend on the aspect ratio of the window (there are no bars on sides). Then replace graphics with 2D pixel art and add integer scaling on top. So instead of bars you get more area of view. And to balance scale factor depending on resolution and aspect ratio, I expect a way to turn off the main scale calculator from approach 1, to calculate the desired integer ratio in the script on window resize event and then be able to set it in the engine.

Is there a reason why the content scale feature is implemented on Window instead of Viewport?
I have a project that uses viewports to contain multiple game instances in the same window. It would be nice to be able to scale the canvas items (i.e. GUI) in each viewport instead of globally on the window.


[aur-repo] Ensure aur-repo always gets the pacman config

[aur-build] Ensure we propegate pacman.conf to aur-repo when finding
the repository we build against.

[aur-build] Renamed p_conf to pacman_conf for internal consistency

[aur-chroot] Ensure pacman.conf is propegated to aur-repo when we find
the build repo.

Please rebase \o/
PS. Documentation for --pacman-conf would also be nice.

@Foxboron


Update get-networkcontrollerpublicipaddress.md

Added description and example

@AnirbanPaul,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


https not working for swashbuckle.core in owin host

I have the hosted the owin as SSL enabled. Without SSL i can able to see the swagger ui documentation. what is the procedure to get the Swagger UI with SSL enabled.
The following code i have added in the startup.cs of owinhost.
config
.EnableSwagger(c =>
{
c.SingleApiVersion("v1", "Desigo");
c.Schemes(new[] { "http", "https" });
c.IncludeXmlComments(GetXmlCommentsPath());
}
)
.EnableSwaggerUi();
Kidly provide your suggestion

Hmmm, SSL should just work out-of-the-box.
As a matter of interest, what happens if you remove the c.Schemes line above. If schemes are not explicitly provided in the swagger description, the UI will assume the same scheme that it's currently being served through. So, to access SSL protected methods, you would need to view the swagger-ui via the https scheme.
Also, could you elaborate on the error scenario? Do you see an error, nothing, a JavaScript error?

Nothing is displayed in console when I am trying to browse in https, my owin is hosted successfully in console application with https enabled.
And also I have another doubt,
Is it possible to inject swagger-ui.js in swashbuckle.core?
Kindly provide your suggestion

Nothing is displayed in console when I am trying to browse in https, my owin is hosted successfully in console application with https enabled.
And also I have another doubt,
Is it possible to inject swagger-ui.js in swashbuckle.core?
Kindly provide your suggestion

That looks more like a web server issue than an application issue. Have you confirmed you can access a regular page/API endpoint in your website (i.e. not the swagger ones) via the scheme, host and port in your browser above?

Yes.. It is working perfectly for the regular endpoint. Only swagger is not working. Kindly suggests what am i missing in configuration.

Hi, It is not working when i am hosting(OWIN) the web api in https. This query is not interlinked with #295. I am looking for your suggestions. Please see my query which i mentioned above

Doesn't work here either.
I go to a https:// url for my Web API, but what is generated in index:
window.swashbuckleConfig = {
        rootUrl: 'http://my.api.is:80/MyAPI',

Tried to change it with configuration code:
c.RootUrl(req => req.RequestUri.Scheme + "://" + req.RequestUri.Authority + System.Web.VirtualPathUtility.ToAbsolute("~"));

Didn't work. Still generates http:// so I always get:
Mixed Content: The page at 'https://my.api.is/MyAPI/swagger/ui/index' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://my.api.is/MyAPI/swagger/docs/v1'. This request has been blocked; the content must be served over HTTPS.

+1

:+1:

try this way:
c.RootUrl(req => GetRootUrlFromAppConfig(req));
private static string GetRootUrlFromAppConfig(System.Net.Http.HttpRequestMessage req)
{
string scheme = GetHeaderValue(req, "X-Forwarded-Proto") ?? req.RequestUri.Scheme;
string host = GetHeaderValue(req, "X-Forwarded-Host") ?? req.RequestUri.Host;
string port = GetHeaderValue(req, "X-Forwarded-Port") ?? req.RequestUri.Port.ToString(CultureInfo.InvariantCulture);
        var httpConfiguration = req.GetConfiguration();
        var virtualPathRoot = httpConfiguration.VirtualPathRoot.TrimEnd('/');

        if (port.Equals("80") || port.Equals("443"))
        {
            return string.Format("{0}://{1}{3}", scheme, host, port, virtualPathRoot);
        }
        else
        {
            return string.Format("{0}://{1}:{2}{3}", scheme, host, port, virtualPathRoot);
        }

    }

    private static string GetHeaderValue(HttpRequestMessage request, string headerName)
    {
        IEnumerable<string> list;
        return request.Headers.TryGetValues(headerName, out list) ? list.FirstOrDefault() : null;
    }

Based on logic this should work, but it forces HTTP in the window.swashbuckleConfig

I tried to use a custom template for Index, that forced swashbuckleConfig.rootUrl to use https URI like such:
      var rootUrlFromServer = '%(RootUrl)';
      if (window.location.protocol === "https:") {
          rootUrlFromServer = rootUrlFromServer.replace("http:", "https:");
      }

      window.swashbuckleConfig = {
        rootUrl: rootUrlFromServer,
        discoveryPaths: arrayFrom('%(DiscoveryPaths)'),
        booleanValues: arrayFrom('%(BooleanValues)'),
     ...
     ...

And I am still getting http only calls when trying to test out endpoints in my Web API.
Mixed Content: The page at 'https://api.corporate.com/swagger/ui/index#!/..........' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://api.corporate.com/api/resource'. This request has been blocked; the content must be served over HTTPS.

Any word on this?

There's two places where the custom RootUrl lambda (or default implementation) is being used:

When serving "index.html", it's used to generate the discovery URL that get's consumed by the swagger-ui JS
When responding to the discovery URL, it's used to generate the API "schemes", "host" and "basePath" properties that get returned in the Swagger JSON document. This is then used to generated the AJAX request whenever you click a "Try it out" button in the UI

So, the custom "index.html" approach used above only addresses the first case and not the second. I think the root cause is elsewhere.
Swashbuckle uses the result of the RootUrl lambda to populate these properties directly with one exception - you can override the schemes properties through the Schemes config option. Before reading on, you might want to double check you're not using this.
If not, then there must be something about you're environment that is causing the following code to return  "http" instead of "https" for the current request:
req => req.RequestUri.Scheme

I would suggest this is the root cause of your problem. One possibility that comes to mind ... if your hosting your application in a proxy environment where SSL actually gets terminated at the proxy server, you would see this behavior.
Can you confirm any of this?

We are acctually running in a proxy environment, offloading SSL with Netscaler (and communicating HTTP with inside services).
So, yes, that is probably the root cause.

No Luck.. It is not working for https. The configuration is as follows,
config
.EnableSwagger(c =>
{
c.SingleApiVersion("v1", "Desigo Access Management API");
c.Schemes(new[] { "http", "https" });
c.RootUrl(req => req.RequestUri.Scheme);
c.ResolveConflictingActions(apiDescriptions => apiDescriptions.FirstOrDefault());
c.IncludeXmlComments(XmlCommentsPath());
}
)
.EnableSwaggerUi();

Issue is fixed..
Changed the configuration as
config
.EnableSwagger(c =>
{
c.SingleApiVersion("v1", "Desigo Access Management API");
c.ResolveConflictingActions(apiDescriptions => apiDescriptions.FirstOrDefault());
c.IncludeXmlComments(XmlCommentsPath());
}
)
.EnableSwaggerUi();
and modified the index.html as Mr.Haukurk suggested
Thanks a lot for everyone

Is there any solution for those who are running in a proxy environment? To force HTTPS?

I'm having the exact same problem as @gaui and @haukurk in that the SSL terminates at the netscaler. Could the root url lambda implementation be changed to pick up only the scheme on the client-side as opposed to determining the scheme on the server. If that's not practical, then why not add a force scheme option?
It's unfortunate but the resolution is a hack that doesn't work for the SSL proxy problem.

I have the same problem as @gaui and @haukurk and resolved it by forcing HTTPS for swagger:
config
	.EnableSwagger("docs/{apiVersion}",
	c =>
	{
	  ...
	  c.RootUrl(ResolveBasePath);
	  ...
	})
	.EnableSwaggerUi();

private static string ResolveBasePath(HttpRequestMessage message)
{
	var virtualPathRoot = message.GetRequestContext().VirtualPathRoot;

	var schemeAndHost = "https://" + message.RequestUri.Host;
	return new Uri(new Uri(schemeAndHost, UriKind.Absolute), virtualPathRoot).AbsoluteUri;
}

Of course this does not address the use case where you might want to test both HTTP and HTTPS schemes at runtime, but if you are forcing HTTPS for your services (which you should be) then this is an acceptable solution.

For those running in a proxy environment, as long as you have the following headers it should create the correct base path:

X-Forwarded-Proto
X-Forwarded-Host
X-Forwarded-Port

We fixed the issue by updating the headers our proxy server was passing.

Solution with X-Forwarded-* worked perfectly without any changes in code. Thanks @willdrob !


A few core API updates

Added User Agent
added jsonlite::flatten
changed default on queryAPI to teams just in case of a rogue call
changed verbose to debug on queryAPI to allow for verbose to be
used to toggle on/off messages

Some ideas for next version:

add a tiny sleep/pause between api calls
create nhl, nba, nfl functions that use and set the sport, league and use the main stattleship function-- less parameters to set

Good to merge! :shipit:


Character preview in lobby doesn't show unless you click save

Description

Reproduction

after clicking save

Screenshots

Additional context

dupe of #13067

Fixed in https://github.com/space-wizards/RobustToolbox/pull/4065


local video

new VideoPlayerController.asset(
      "assets/videos/test.mp4",

local videos crash...

Add more details like crash report flutter doctor etc.

For some reason assets do not load on for me when used in the chewie_example. But only on start. If I add them to one of the buttons it does work as expected.

videos don't crash on the emulator
but then, when trying on a real cell phone
downloading from the Play Store
then the whole App crashes...


Checkstyle

Fixes gh-832

Codecov Report

Merging #833 into master will increase coverage by 0.39%.
The diff coverage is 66.63%.

@@             Coverage Diff              @@
##             master     #833      +/-   ##
============================================
+ Coverage     69.65%   70.05%   +0.39%
- Complexity      994      996       +2
============================================
  Files           136      136
  Lines          3994     4064      +70
  Branches        287      291       +4
============================================
+ Hits           2782     2847      +65
- Misses         1059     1065       +6
+ Partials        153      152       -1

Impacted Files
Coverage Δ
Complexity Δ

...handler/predicate/MethodRoutePredicateFactory.java
100% <ø> (ø)
5 <0> (ø)
:arrow_down:

...cloud/gateway/sample/GatewaySampleApplication.java
90.41% <ø> (-1.14%)
26 <0> (ø)

...teway/config/PropertiesRouteDefinitionLocator.java
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

...mework/cloud/gateway/support/TimeoutException.java
100% <ø> (ø)
3 <0> (ø)
:arrow_down:

...mework/cloud/gateway/event/PredicateArgsEvent.java
100% <ø> (ø)
3 <0> (ø)
:arrow_down:

...RequestHeaderToRequestUriGatewayFilterFactory.java
100% <ø> (ø)
4 <0> (ø)
:arrow_down:

...ork/cloud/gateway/filter/GatewayMetricsFilter.java
88.57% <ø> (ø)
9 <0> (ø)
:arrow_down:

...rk/cloud/gateway/filter/ratelimit/RateLimiter.java
40% <ø> (ø)
0 <0> (ø)
:arrow_down:

...eway/mvc/config/ProxyExchangeArgumentResolver.java
84.21% <ø> (ø)
6 <0> (ø)
:arrow_down:

.../filter/factory/SetStatusGatewayFilterFactory.java
91.66% <ø> (ø)
5 <0> (ø)
:arrow_down:

... and 133 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb95038...66153ac. Read the comment docs.


Feature request: max amount option when choosing new channel amount

I put a few millibits into the wallet intending to put it all (less fees) into a channel, but there is no easy way to put the max amount into a channel.
Something like your "Empty wallet" option when sending an on chain transaction is what I'm looking for, but for opening a channel.

+1
it's a frequent use case:

send some coins to eclair
put all of those into LN channel

right now it's almost impossible to avoid dust output,
having a MAX option that spends all funds - miner fee (and creates a single output) would be very desirable. LBW has such option and it works great.


my account was deactivated and I don't know why

I tried to contact to the hackclub team via<EMAIL_ADDRESS>and I haven't received any response, if I violated any rule please tell me.
I can't access the slack workspace for arcade club, I try to access it via the invitation link and it pops up the following message:

the text says (in Spanish) It seems like your account in your hackblub workspace has been deactivated.
kind regards.
Samuel Mateo Rodriguez.

I'm working on alpinecade, an embedded distro made for low powered machines, I decided to switch from alpine linux to artix cause I can't figure out some things about the previous one, but maybe I will make a fork for alpine in the future.
You can check out my repository at: https://github.com/Sam-coder183/AlpineCade

What email did you use to register for slack? I will look into it

<EMAIL_ADDRESS>

Hi, my account in slack is yet to be reactivated.
I want to keep participating in the arcade contest for motivating me to release my first alpha of the distro that I'm working on (I've decided to move from alpine to devuan because is more user friendly and based on debian and I can use an alternative init system like runit cause I hate systemd) and then I will try to make it multiplatform with automated syncs using google firestone for making a rapid prototype and then find a way so people can self-host their own game sync servers.
I will be pleased to continue to participate in your next hackatons cause In my country (Spain) the maker culture is not as prominent as in other countries like the US and having a community that I can rely on and share my proyects would be awesome for me.
thanks for considering my issue and I will keep track on your new hackatons for making my software and arduino proyects with this new cool community that I've discovered this summer, but for making that I assume that I have to have access to your slack workplace.
kind regards.
Samuel Mateo Rodriguez.
📍 Huesca, Spain.


Write down documentation

Have screenshots on how to bid / withdraw, etc ...

Documentation on how to send a bid: https://docs.google.com/document/d/16CuSnoQF67EHijxOMpYmdSzfhnOHBsavoRoCTUZgorc/edit#
Documentation on how to withdraw: https://docs.google.com/document/d/13hqn-76bJa_dIqFXfBSuEfzvOYpCttF8USPCN9qiXyk/edit#
Doc is under review by Kristoffer


fix power round set

Set power round as current round minus one

https://github.com/Giveth/giveth-dapps-v2/issues/762


can't get transaction history from the database

hello,
Latest Transactions section doesn't show any information,
balance shows correct amount,
postgres@ubuntu:~$ psql -d transaction_db
psql: FATAL:  database "transaction_db" does not exist
i think i missed something when i was setting up a psql user, any thought on that?
thanks.

You're not doing anything wrong. I haven't pushed the transaction logging code yet. It should be up soon

oh , ok , thanks. i have added Wallet address QR Code too, im going to add QR code scan and Payment ID Generation too.

I think Separate Send, Receive, Transactions, ... Tabs are also Great.


Organisation fn read success testcase

Organisation fn read success testcase

@bvinayakumar incorporated all the comments. Please review it once


Create learn_physic_model.py

Based on the arXiv paper: Relational inductive biases, deep learning, and graph networks.
Tutorial: This code proposes to deal with the problem of learning a physical model using
graphs and the torch geometric library. This work is based on the work done by
the GraphNets authors. The idea is to see how to use a pytorch based library
to achieve the same result as with the GraphNets library. Some simplifications
have been made compared to the approach proposed in the code using GraphNets.

Thanks! This looks cool. I will have a look in the next week since I'll be on vacation until then.

One mistake in the code ;)
Don't hesitate if you think I have to change something.


Readme

Update README file.

Hi @Sparktan24  , Your project is complete! There is nothing else to say other than... it's time to merge it :shipit: Congratulations! 🎉

Highlights

You have added a descriptive Readme file. Good Job. ✔

Optional suggestions
Every comment with the [OPTIONAL] prefix won't stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better. Some of them were simply missed by the previous reviewer and addressing them will really improve your application.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear. Please, remember to tag me in your question so I can receive the notification.
As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.


add public products to ProductLookupView queryset

Could you please add 1 test demonstrating that currently this functionality is broken, and you have fixed it?

Could you please add 1 test demonstrating that currently this functionality is broken, and you have fixed it?

100%, just gotta take some time, very busy now

@xplsek03  Can you please explain the reason for adding each variant individually to the recommended products dropdown list? Currently we just show the parent or standalone products. Although for parent products we are redirected to a separate page and there we have to select the variant to add to our basket. I guess you may want to avoid such behavior and display just a specific variant in the recommendations so that it can be added without additional steps.

@xplsek03 Can you please explain the reason for adding each variant individually to the recommended products dropdown list? Currently we just show the parent or standalone products. Although for parent products we are redirected to a separate page and there we have to select the variant to add to our basket. I guess you may want to avoid such behavior and display just a specific variant in the recommendations so that it can be added without additional steps.

Overflown with another projects. Will have to get to oscar again for next proj.


custom tracks not showing up in the server creator on latest version.

Hey y'all, love what you guys do and have been really hyped to see the update for BeatTogether. I'm having a issue with the mod not showing up in mod settings and when launching a private server with the BeatTogether setting in multiplayer selected I can't see any custom tracks. My friend is having the same issue and we don't know what could be causing it. we installed the mod properly in the plugins folder in the beatsaber directory from steam and it does show up in the multiplayer menu. would love to see if anyone has a fix or knows if I've just messed up my install some how. Additionally I had done a whole fresh reinstall of beatsaber just to see if that would fix anything but it didn't. again really would appreciate any help someone could give me.

For customs to appear you must be connecting to our servers, and have MultiplayerExtensions for PC.
Refer to our custom song support instructions in the readme.


Keep Spec.Networks and Spec.TaskTemplate.Networks consistent

- What I did
While looking into docker issue #28247 to support network update, noticed several potential issues in the current service spec:

When a service is created, the Networks in ServiceSpec are always resolved into Network ID. This guarantees the spec for network. However, tt also means when user is doing a docker service inspect, the Spec returned is different from the original Spec as network might
be changed from foo to d928....
This is quite problematic because if user wants to update the network, there is no match for network-add/network-rm. Instead, user will have to perform a network resolve on the client side, then find out which network to remove and which network to add.

Currently there are two places for Networks. One is in Spec.TaskTemplate.Networks and another is in Spec.Networks. As was specified in the code comment,  Spec.Networks was deprecated and may be removed in the future. However, Spec.Networks is still used in case Spec.TaskTemplate.Networks is not present.
Furthermore, in the current implementation when networks are resolved, only the Spec.TaskTemplate.Networks is updated. Therefore, a docker service inspect actually returns the spec with different Networks specifications. The value in Spec.Networks is the name while Spec.TaskTemplate.Networks is the network ID.

- How I did it
This fix tries to address the problem by:

Add a Name field in NetworkAttachmentConfig to keep a copy of the original network name. Target is still the network ID and is still the one being used. But users will get the original network names they provide with docker service inspect.
In docker daemon, when network names are resolved into network IDs, both Spec.TaskTemplate.Networks and Spec.Networks are updated so that they are always in sync.

This fix will help the implementation of #28247.
- How to verify it
An integration test has been added.
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)

A swarmkit PR has been opened:
~~https://github.com/docker/swarmkit/pull/1813~~
Signed-off-by: Yong Tang<EMAIL_ADDRESS>

/cc @stevvooe @aaronlehmann @mavenugo
@yongtang tests are red somehow

Thanks @vdemeester. Now all tests passed except for the re-vendoring of swarmkit. The swarmkit PR is https://github.com/docker/swarmkit/pull/1813
/cc @stevvooe @aaronlehmann @mavenugo

ping  @stevvooe @aaronlehmann @mavenugo PTAL

Let's discuss in the swarmkit PR: docker/swarmkit#1813

Based on the comment from https://github.com/docker/swarmkit/pull/1813#issuecomment-270813849, it is preferable to resolve the network ID at the client side.
As a result, much of the original PR may not be needed (will also close https://github.com/docker/swarmkit/pull/1813 shortly).
However, there is still one issue left. In the current daemon code, when resolving the network ID from network name, only spec.TaskTemplate.Networks is updated to have the network ID. The spec.Networks would not change.
This PR has been updated to address the inconsistency between spec.TaskTemplate.Networks and spec.Networks.
/cc @stevvooe @aaronlehmann

I think the inconsistency is intentional.
If I remember correctly, Networks was moved to TaskTemplate. To keep backward compatibility, s.Networks is used if s.TaskTemplate.Networks isn't populated.
Since s.TaskTemplate.Networks is always preferred, I don't think there's a need to keep s.Networks up to date.
I believe the Docker CLI will never create a service that has both s.Networks and s.TaskTemplate.Networks (old versions will use s.Networks, new versions will use s.TaskTemplate.Networks), but maybe it's a good idea to clear s.Networks when updating s.TaskTemplate.Networks just to make sure both are not set at the same time.

@yongtang Unfortunately, the field documentation was removed, but looking at https://github.com/docker/swarmkit/blob/master/api/specs.proto#L75, we can see the field on ServiceSpec is deprecated.
The only thing that should change is to have a PR to update the documentation on that field to direct the user to the TaskSpec.
There is a comment at the start of this function, indicating the correct behavior. Updating this will cause odd things to happen.

@yongtang @aaronlehmann @stevvooe what's up here?

I think we can close this.

@aaronlehmann @stevvooe @LK4D4 Sorry forgot to close this PR. Let me close it.


Add run criterias to StateStages

What problem does this solve or what need does it fill?
This feature would allow limiting the frames per second of a game using StateStages.
What solution would you like?
The way this can already be done for SystemStages would be a very nice to have on StateStages (e.g. with_run_criteria).
What alternative(s) have you considered?
I don't know of any alternatives appart from doing this manually with timers on each system, but would be interested in better options :)

You can put a run criteria in the SystemStage of a StateStage
`my_state_stage.update_stage(MyState::MySubState, |it: &mut SystemStage| it.with_run_criteria(my_run_criterion));`

You can put a run criteria in the SystemStage of a StateStage
`my_state_stage.update_stage(MyState::MySubState, |it: &mut SystemStage| it.with_run_criteria(my_run_criterion));`

I am not getting that to compile. Current code:
impl Plugin for GamePlugin {     fn build(&self, app: &mut AppBuilder) {         let mut my_state_stage = StateStage::<AppState>::default();         my_state_stage.update_stage(AppState::InGame, |it: &mut SystemStage| {             it.with_run_criteria(FixedTimestep::steps_per_second(60.))         });         app             .add_stage_after(stage::UPDATE, STAGE, my_state_stage)     } }
Maybe I am just missing some rust knowledge here... The return type of the closure seems wrong (it expects &mut SystemStage), but if I return &mut in the closure that also fails to compile with cannot return reference to temporary value.

I am not getting that to compile. Current code:
impl Plugin for GamePlugin {     fn build(&self, app: &mut AppBuilder) {         let mut my_state_stage = StateStage::<AppState>::default();         my_state_stage.update_stage(AppState::InGame, |it: &mut SystemStage| {             it.with_run_criteria(FixedTimestep::steps_per_second(60.))         });         app             .add_stage_after(stage::UPDATE, STAGE, my_state_stage)     } }
Maybe I am just missing some rust knowledge here... The return type of the closure seems wrong (it expects &mut SystemStage), but if I return &mut in the closure that also fails to compile with cannot return reference to temporary value.

Ok, so this issue can probably be repurposed for that issue, with a title and contents:
SystemStage has no add_run_criteria
https://github.com/bevyengine/bevy/blob/0a39c81be6b5712c24a75a7c215dbe254097adff/crates/bevy_ecs/src/schedule/stage.rs#L67-L71
is the method used to add a system to a SystemStage, but there's no equivalent method when you only have a mutable reference.
The naïve implementation works fine:
    pub fn add_run_criteria<S: System<In = (), Out = ShouldRun>>(
        &mut self,
        system: S,
    ) -> &mut Self {
        self.run_criteria = Some(Box::new(system));
        self.run_criteria_initialized = false;
        self
    }

Ok, so this issue can probably be repurposed for that issue, with a title and contents:
SystemStage has no add_run_criteria
https://github.com/bevyengine/bevy/blob/0a39c81be6b5712c24a75a7c215dbe254097adff/crates/bevy_ecs/src/schedule/stage.rs#L67-L71
is the method used to add a system to a SystemStage, but there's no equivalent method when you only have a mutable reference.
The naïve implementation works fine:
    pub fn add_run_criteria<S: System<In = (), Out = ShouldRun>>(
        &mut self,
        system: S,
    ) -> &mut Self {
        self.run_criteria = Some(Box::new(system));
        self.run_criteria_initialized = false;
        self
    }