Full support for multiindex in dataframes

Dask can load a dataframe from a pytables hdf5 file, and pytables already supports a hierarchy tables.  Why not simulate a multiindex (like in pandas) by loading all tables from an hdf5 file into one dask dataframe with nested column indices?

I encourage you to prototype this, perhaps with dask.delayed.  http://dask.readthedocs.io/en/latest/delayed-collections.html

I was originally thinking of doing this as a dict that wraps a bunch of dask.dataframes, but as you recommended I'm trying this with dask.delayed.  I am using pandas to read/write the hdf data rather than pytables using these functions:
import pandas as pd

def custom_load(key):
    return pd.read_hdf('test.hdf', key)

def custom_save(df, key):
    df.to_hdf('test.hdf', key)

Unfortunately, when I try to use them to build a dask.dataframe I get a TypeError exception:
import dask.dataframe as dd
from dask.delayed import delayed

dfs = [delayed(custom_load)(key) for key in ['msft','aapl']]
df = dd.from_delayed(dfs)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-19-59138932b0db> in <module>()
----> 1 df = dd.from_delayed(dfs)

C:\Python34\lib\site-packages\dask\dataframe\io.py in from_delayed(dfs, metadata, divisions, columns, prefix)
    670         return Series(merge(dsk, dsk2), name, metadata, divisions)
    671     else:
--> 672         return DataFrame(merge(dsk, dsk2), name, metadata, divisions)
    673
    674

C:\Python34\lib\site-packages\dask\dataframe\core.py in __new__(cls, dask, name, columns, divisions)
   1322         result._name = name
   1323
-> 1324         result._pd, result._known_dtype = cls._build_pd(columns)
   1325         result.divisions = tuple(divisions)
   1326         return result

C:\Python34\lib\site-packages\dask\dataframe\core.py in _build_pd(cls, metadata)
    201         else:
    202             if np.isscalar(metadata) or metadata is None:
--> 203                 _pd = cls._partition_type([], name=metadata)
    204             else:
    205                 _pd = cls._partition_type(columns=metadata)

TypeError: __init__() got an unexpected keyword argument 'name'

I don't have much experience with dask.delayed so I'm not sure what the problem is.  For reference, this is how I built test.hdf:
from pandas_datareader import data as web

df1 = web.get_data_yahoo('msft', '2000-01-01', '2016-01-01')
df2 = web.get_data_yahoo('aapl', '2000-01-01', '2016-01-01')

df1.to_hdf('test.hdf', 'msft', format='table', complevel=9, complib='blosc')
df2.to_hdf('test.hdf', 'aapl', format='table', complevel=9, complib='blosc')

Can you do me a favor and try this from git master?
On Mon, Aug 22, 2016 at 8:05 PM, dirkbike<EMAIL_ADDRESS>wrote:

I was originally thinking of doing this as a dict that wraps a bunch of
dask.dataframes, but as you recommended I'm trying this with dask.delayed.
I am using pandas to read/write the hdf data rather than pytables using
these functions:
import pandas as pd
def custom_load(key):
return pd.read_hdf('test.hdf', key)
def custom_save(df, key):
df.to_hdf('test.hdf', key)
Unfortunately, when I try to use them to build a dask.dataframe I get a
TypeError exception:
import dask.dataframe as dd
from dask.delayed import delayed
dfs = [delayed(custom_load)(key) for key in ['msft','aapl']]
df = dd.from_delayed(dfs)
TypeError                                 Traceback (most recent call last)
 in ()
----> 1 df = dd.from_delayed(dfs)
C:\Python34\lib\site-packages\dask\dataframe\io.py in from_delayed(dfs, metadata, divisions, columns, prefix)
670         return Series(merge(dsk, dsk2), name, metadata, divisions)
671     else:
--> 672         return DataFrame(merge(dsk, dsk2), name, metadata, divisions)
673
674
C:\Python34\lib\site-packages\dask\dataframe\core.py in new(cls, dask, name, columns, divisions)
1322         result._name = name
1323
-> 1324         result._pd, result._known_dtype = cls._build_pd(columns)
1325         result.divisions = tuple(divisions)
1326         return result
C:\Python34\lib\site-packages\dask\dataframe\core.py in _build_pd(cls, metadata)
201         else:
202             if np.isscalar(metadata) or metadata is None:
--> 203                 _pd = cls._partition_type([], name=metadata)
204             else:
205                 _pd = cls._partition_type(columns=metadata)
TypeError: init() got an unexpected keyword argument 'name'
I don't have much experience with dask.delayed so I'm not sure what the
problem is. For reference, this is how I built test.hdf:
from pandas_datareader import data as web
df1 = web.get_data_yahoo('msft', '2000-01-01', '2016-01-01')
df2 = web.get_data_yahoo('aapl', '2000-01-01', '2016-01-01')
df1.to_hdf('test.hdf', 'msft', format='table', complevel=9, complib='blosc')
df2.to_hdf('test.hdf', 'aapl', format='table', complevel=9, complib='blosc')
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1493#issuecomment-241588548, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszLSaZvyokPKWr5qE1Kdj6Dnnq2Tkks5qijlVgaJpZM4JpFvx
.

That worked, thanks.  So, what happened is that all of the 'aapl' data was concatenated to the end of 'msft' data in one large dataframe.  However, in this case it would be more desirable to have a top-level index that uses key from custom_load in a way similar to this:
df1 = pd.read_hdf('test.hdf', 'msft')
df2 = pd.read_hdf('test.hdf', 'aapl')
df = pd.concat([df1, df2], keys=['msft','aapl'], axis=1)
df['msft'].head() # first key
                Open     High       Low     Close    Volume  Adj Close
Date
2000-01-03  117.3750  118.625  112.0000  116.5625  53228400  39.840438
2000-01-04  113.5625  117.125  112.2500  112.6250  54119000  38.494621
2000-01-05  111.1250  116.375  109.3750  113.8125  64059600  38.900502
2000-01-06  112.1875  113.875  108.3750  110.0000  54976600  37.597410
2000-01-07  108.6250  112.250  107.3125  111.4375  62013600  38.088740

df['aapl'].head() # second key
                  Open        High         Low       Close     Volume  Adj Close
Date
2000-01-03  104.874997  112.499998  101.687501  111.937502  133949200   3.660058
2000-01-04  108.250001  110.625002  101.187503  102.500003  128094400   3.351477
2000-01-05  103.749998  110.562497  103.000001  103.999997  194580400   3.400523
2000-01-06  106.124999  106.999999   94.999998   94.999998  191993200   3.106247
2000-01-07   96.499999  101.000002   95.500003   99.500001  115183600   3.253385

Then perhaps you're right that your dict-of-dataframes idea would suit better

Just curious, but why can't the dask.dataframe object support these kinds of keys internally?  I think that is all that's necessary to simulate a pandas multiindex (at least across columns).  Would modifying the _Frame class be the best place for this?

Eventually yes, it would be nice for DataFrame to support multiindices.  It's non-trivial to change all functions within dask.dataframe to support this.  I budget this task at somewhere between a week and a month of developer time, though I am often pessimistic in things like this.
Have you read through the design documentation of dask.dataframe? http://dask.readthedocs.io/en/latest/dataframe-partitions.html

So if I'm understanding correctly, it seems that the best way to support multiindex would be to map them to multiple dimensions of partitions since the multiindex itself provides a natural place to create a partition.  My example above would only add a second dimension to the partitions (partitions would span time index and then first-level column keys).  It would be a lot easier to maintain these partitions outside of dask by managing multiple dask.dataframe objects, but you would lose the ability to slice across multiple of these partitions and won't guarantee the data stays aligned with the index.  I agree that this is non-trivial, since it looks like it would require changing how all of the blocked algorithms are handled and might add undesired overhead.

Yes, that seems like a reasonable synopsis.  We would choose some depth of the multi-index along with to partition.  For example we might partition along the second or third step of the multi-index.  Partitions would then hold a list of tuples of values rather than a list of single values.  Many of the operations can probably be changed in bulk, by changing some of the heavier functions, like elemwise and reduction, but I would expect groupbys, joins, etc. to take a fair amount of finesse.  I don't yet see a way to do this incrementally.

This may be a bit of a stretch, but maybe it's worth considering more abstract partitioning. I got some inspiration from this paper A Hierarchical Aggregation Framework for Efficient Multilevel Visual Exploration and Analysis that breaks data into hierarchical chunks of smaller and smaller index-slices to make data exploration faster.  Partitioning the data would be an expensive operation, but can be done as data is collected.  You would need at least one hierarchy for the index and possibly one for the columns (column groupings can come from an existing storage hierarchy or can be made dynamically using groupbys).  The index hierarchy would define the partitions similar to dask's current structure except using multiple levels (i.e. days are grouped into months, which are grouped into years, etc.).  Columns could use a pseudo-index to map to the main index (i.e. a range or years, months, or specific days) to keep the data dense (no filler NaNs) and allow calculations to quickly skip regions with no data.  Index and column groupings would be exposed to the end user via indexing and slicing methods and would provide natural partition boundaries for applied computation.  A column hierarchy also provides an organized structure for caching intermediate computation results.

Sounds very cool.  I encourage you to explore that further.

I started a prototype using basic python structures (dicts, and subclasses of lists) and realized that data columns either need to use a sequence of index labels to identify each element (because of the hierarchical index), or the columns can map to a flat representation of the hierarchical index (using a pseudo-index).  I couldn't think of another way to do this, and mapping each element individually with labels would be very wasteful.  The problem with using a pseudo-index is that when data is appended to the data set, the pseudo-index needs to be recalculated. I'm starting to re-think the use of hierarchies at all.  Relational databases can represent hierarchical structures by referencing keys between tables of data, and joining tables on a specific column already aligns tables to each other.  Perhaps it's better to treat every chunk of data (of n-columns) as a regular 2D dataframe, and use a relational representation to tie all of the dataframes together.  Each dataframe would have its own independent index, avoiding the pseudo-index problem, and only when chunks are joined would the index need to be adjusted.  The end user could still reference a specific subset of data using slices and labels, but chunks of data would be dynamically joined (or split if necessary) behind the scenes.  I'm going to try and prototype something using sqlite and pandas with some more stock data and see how that might work.

Any update on this issue?

I ended up using a regular SQL database to track chunks of data and assembling them as necessary into Pandas DataFrames.

Does dask support reading multi-level indices yet?  I'm particularly interested in reading a table written to parquet with a multi-level column index, and I'm getting the following traceback when I try to do this:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<timed exec> in <module>()

/ssd/lsstsw/stack3_20171021/python/miniconda3-4.3.21/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in read_parquet(path, columns, filters, categories, index, storage_options, engine)
    797
    798     return read(fs, fs_token, paths, columns=columns, filters=filters,
--> 799                 categories=categories, index=index)
    800
    801

/ssd/lsstsw/stack3_20171021/python/miniconda3-4.3.21/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in _read_pyarrow(fs, fs_token, paths, columns, filters, categories, index)
    557     dtypes = {storage_name_mapping.get(k, k): v for k, v in dtypes.items()}
    558
--> 559     meta = _meta_from_dtypes(all_columns, dtypes, index_names, column_index_names)
    560
    561     if out_type == Series:

/ssd/lsstsw/stack3_20171021/python/miniconda3-4.3.21/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in _meta_from_dtypes(to_read_columns, file_dtypes, index_cols, column_index_names)
    140         df.columns.name = column_index_names[0]
    141     else:
--> 142         df.columns.names = column_index_names
    143     return df
    144

~/.local/lib/python3.6/site-packages/pandas/core/indexes/base.py in _set_names(self, values, level)
   1116         if len(values) != 1:
   1117             raise ValueError('Length of new names must be 1, got %d' %
-> 1118                              len(values))
   1119         self.name = values[0]
   1120

ValueError: Length of new names must be 1, got 3

If multi-level indices are generally supported, but not in read_parquet, then perhaps this should be a new issue?

I definitely agree @vss888 if this is something that you'd like to contribute that would be very welcome!

This is maybe a simple but not ideal hack with a less that idea resolution.  If there were a way to do element-wise concatenation on the two indexes you could create a unique multi-index value (sort of).
The issue that I am running into is that i can't figure out how to do element-wise concat on two dask arrays.  Any way to do the following?
index = np.array([x1 + x2 +x3 for x1,x2,x3 in zip(index_id1.astype('str'),
                                           repeat('-', len(index_id1) ) ,
                                           index_id2.astype('str'))])

Hi there. I'm trying to get a handle on what all might be involved in supporting development on this. It sounds like a few options were previously explored, but the method discussed by @dirkbike and @mrocklin above is the preferred path forward, although the main blocker to that is the amount of work and inability to implement such a change incrementally. @mrocklin do you have a ballpark on the number of functions in the DataFrame API that would be affected by this? I see that it's a complex issue, but I'd like to at least look into supporting this or breaking it down and finding some people to help chip away at it.

I don't personally have a ballpark estimate, no.  Others might though.
On Fri, Jun 28, 2019 at 11:33 PM Marissa<EMAIL_ADDRESS>wrote:

Hi there. I'm trying to get a handle on what all might be involved in
supporting development on this. It sounds like a few options were
previously explored, but the method discussed by @dirkbike
https://github.com/dirkbike and @mrocklin https://github.com/mrocklin
above is the preferred path forward, although the main blocker to that is
the amount of work and inability to implement such a change incrementally.
@mrocklin https://github.com/mrocklin do you have a ballpark on the
number of functions in the DataFrame API that would be affected by this? I
see that it's a complex issue, but I'd like to at least look into
supporting this or breaking it down and finding some people to help chip
away at it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1493?email_source=notifications&email_token=AACKZTFSSIGDF4PLUZDU3XTP42GVPA5CNFSM4CNELPY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3J45I#issuecomment-506895989,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTFAJLRMCFUKX5NDOITP42GVPANCNFSM4CNELPYQ
.

@TomAugspurger, do you have any thoughts on this one?

Still open, still worth doing.
On Thu, Aug 8, 2019 at 10:31 AM jakirkham<EMAIL_ADDRESS>wrote:

@TomAugspurger https://github.com/TomAugspurger, do you have any
thoughts on this one?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1493?email_source=notifications&email_token=AAKAOIU3Y7QFNSAWW2YUW2LQDQ34TA5CNFSM4CNELPY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3374FQ#issuecomment-519568918,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKAOIVFR476BN5MDFKVICTQDQ34TANCNFSM4CNELPYQ
.

In the interim, does anyone have a workaround?
@jvivian-atreca have you found any?

Small status update: I worked on this a bit. We need to find a good (approximate) quantile algorithm to discover the partitions. Everything we have right now works with 1D arrays. I think the primary difficulty is that the algorithm needs to work with heterogenous data if we want to avoid casting to object dtype.
Alternatively, we can do just that: cast to object dtype and treat MutliIndex as a big list of tuples. With some slight modifications, that becomes 1-D enough for the purposes of quantiling and discovering partitions. Casting to object dtype is expensive though, so I'd like to avoid that if possible.
A secondary issue is that pandas' indexing API isn't well specified with a MultiIndex. Some behaviors like df.loc['A', 'B'] are ambiguous with a MultiIndex, since it's unclear whether that's specifying a slice into a 2-level MultiIndex, or a depth-1 slice on the index and a column selection. But that's a limitation we'll just have to document and accept.

@rjzamora and the RAPIDS folks have been playing with multi-column indexing I think
One of the nice things about approximate quantiles is that they're decently robust to sampling.  I imagine that if we collected 100 samples per partition, and then turned those into objects/tuples that we would still provide a decent solution most of the time.  cc @eriknw

To determine the partitions, I would have few reservations about creating a 1-D array of dtype object of tuples and this using the existing partition_quantiles method.  I realize this isn't the most performant (speed or memory), but I bet it would be "good enough" for a while, and partition_quantiles is written to work on any dtype.  This operation should be compared to what happens next: shuffling the data into new partitions, which is much more costly.

That's good to hear. I'll dust off my branch that started down this path and push something up, hopefully later this week.

Related code suggestion in https://github.com/dask/dask/issues/6074

@TomAugspurger - I support any effort to support multiIndex in dask.dataframe (including multi-column sort_values/set_index operations).
Just a note (for your information) - In dask_cudf we currently support a multi-column sort_values.  This method uses the same logic as upstream dask to calculate quantiles.  Hoever, it works on multiple columns, because we have a DataFrame-level quantiles method, as well as a DataFrame-level searchsorted method (neither of which is available in pandas, if I understand/remember correctly).  Our general approach is to follow the same logic that dask already uses, but to allow divisions to be passed in or calculated/represented as a cudf DataFrame.

@rjzamora

Just a note (for your information) - In dask_cudf we currently support a multi-column sort_values. ...

Does this solve https://github.com/dask/dask/issues/958#issuecomment-573121520 as well? If so, how can it be used in dask?
Kind knowledge request: groupby supports multiple columns, why is it so different for set_index?

Does this solve #958 (comment) as well? If so, how can it be used in dask?

Yes, but only for for cudf backed dask DataFrames (using dask_cudf).
Unfortunately, the implementation in dask_cudf cannot do much to inform necessary changes in dask.dataframe itself, because it is more-or-less using the same algorithm as dask.dataframe.  The fact that cudf has a DataFrame-level searchsorted method (while pandas only has a Series-level method) means that the existing logic used for a single "key" column can now be used for a list of "key" columns (the new divisions just need to be represented as a DataFrame, rather than a list/tuple). The existing quantile logic in dask.dataframe can also be re-used to calculate new divisions, because cudf has a DataFrame-level quantiles method (while pandas will only return separate quantiles for each column/Series).
I guess the overall take-away from the dask_cudf work is that sort_values would become easy if pandas were to offer DataFrame-level searchsorted and quantile(s) functionality.

Kind knowledge request: groupby supports multiple columns, why is it so different for set_index?

If I recall correctly, global ordering is still a challenge/issue for groupby aggregations with a multiple-partition output.  You can certainly leverage groupby machinery to implement a global sort, but this is only easy for split_out=1.  If you want your groupby aggregation to map onto multiple output partitions, I believe you will still need to perform a global shuffle.  I have not looked into this approach, so it may be easier than I am currently thinking.

I guess the overall take-away from the dask_cudf work is that sort_values
would become easy if pandas were to offer DataFrame-level searchsorted and
quantile(s) functionality.

Nice.  I really appreciate NVIDIA folks building functionality such that
this occurred.  This feels really clean to me.
On Thu, Apr 9, 2020 at 8:52 AM Richard (Rick) Zamora <
<EMAIL_ADDRESS>wrote:

Does this solve #958 https://github.com/dask/dask/issues/958 (comment)
as well? If so, how can it be used in dask?
Yes, but only for for cudf backed dask DataFrames (using dask_cudf).
Unfortunately, the implementation in dask_cudf cannot do much to inform
necessary changes in dask.dataframe itself, because it is more-or-less
using the same algorithm as dask.dataframe. The fact that cudf has a
DataFrame-level searchsorted method (while pandas only has a Series-level
method) means that the existing logic used for a single "key" column can
now be used for a list of "key" columns (the new divisions just need to
be represented as a DataFrame, rather than a list/tuple). The existing
quantile logic in dask.dataframe can also be re-used to calculate new
divisions, because cudf has a DataFrame-level quantiles method (while
pandas will only return separate quantiles for each column/Series).
I guess the overall take-away from the dask_cudf work is that sort_values
would become easy if pandas were to offer DataFrame-level searchsorted
and quantile(s) functionality.
Kind knowledge request: groupby supports multiple columns, why is it so
different for set_index?
If I recall correctly, global ordering is still a challenge/issue for
groupby aggregations with a multiple-partition output. You can certainly
leverage groupby machinery to implement a global sort, but this is only
easy for split_out=1. If you want your groupby aggregation to map onto
multiple output partitions, I believe you will still need to perform a
global shuffle. I have not looked into this approach, so it may be easier
than I am currently thinking.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1493#issuecomment-611603727, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTHNWJKPE7A4EZO64LTRLXVLJANCNFSM4CNELPYQ
.

pandas has a DataFrame.quantile. Does it match the behavior of cudf?  Does it handle non-numeric columns, and would it need to to use it for partitioning? I'll look into opening an issue on pandas for DataFrame.searchsorted.

pandas has a DataFrame.quantile. Does it match the behavior of cudf? Does it handle non-numeric columns, and would it need to to use it for partitioning? I'll look into opening an issue on pandas for DataFrame.searchsorted.

Right - If I understand correctly, the pandas quantile method is the same as the cudf quantile method.  It will return separate quantiles for each column in a DataFrame.  The cudf quantiles method (not the "s"), effectively treats each row as a tuple of values, and returns a single quantile series.  It can indeed handle non-numeric columns.
A method like quantiles may not be necessary for partitioning, but it makes it pretty easy to calculate new divisions using the same logic already used in dask.

Is there any progress on this?
It's a very big problem that ddf does not allow to use multiindex.

Not from me recently. My branch is at https://github.com/TomAugspurger/dask/tree/multiindex. Let me know if you're interested in taking it up @Hoeze.
There's still some architectural things to work out (partitioning the MultiIndex as a 1-D array of tuples vs. a distinct array per level) and a decent number of edge cases.

Thanks @TomAugspurger, I'll try to have a look.

Would a suitable workaround be just concatenating the string versions of the multiple indices into one column, then making that column the index?
For example:
ddf['%s_%s_%s' %('id1', 'id2', 'id3')] = ddf['id1'].astype(str) + ddf['id2'].astype(str) + ddf['id3'].astype(str)
ddf = ddf.set_index('%s_%s_%s' %('id1', 'id2', 'id3'))

Would a suitable workaround be just concatenating the string versions of the multiple indices into one column, then making that column the index?
For example:
ddf['%s_%s_%s' %('id1', 'id2', 'id3')] = ddf['id1'].astype(str) + ddf['id2'].astype(str) + ddf['id3'].astype(str)
ddf = ddf.set_index('%s_%s_%s' %('id1', 'id2', 'id3'))

That's what I've done to get around this in the past.

How do we do this when doing read_sql_table please
ddf['%s_%s_%s' %('id1', 'id2', 'id3')] = ddf['id1'].astype(str) + ddf['id2'].astype(str) + ddf['id3'].astype(str)
ddf = ddf.set_index('%s_%s_%s' %('id1', 'id2', 'id3'))

daskDF = ddf.read_sql_table('test_table', sqluri, index_col=['column1','column2'])

@TheXu How do we do this when doing read_sql_table please
ddf['%s_%s_%s' %('id1', 'id2', 'id3')] = ddf['id1'].astype(str) + ddf['id2'].astype(str) + ddf['id3'].astype(str)
ddf = ddf.set_index('%s_%s_%s' %('id1', 'id2', 'id3'))

daskDF = ddf.read_sql_table('test_table', sqluri, index_col=['column1','column2'])

Since dask doesn't support MultiIndex yet, you might have to concatenate column1 and column2 within SQL (naming it 'column1_and_column2' for example),  then
daskDF = ddf.read_sql_table('test_table', sqluri, index_col=['column1_and_column2'])

@TheXu  Thank you. Can we construct this in the dask ? something like. If yes, would appreciate if you can share syntax?
 Dindex_col = ddf['id1'].astype(str) + ddf['id2'].astype(str) + ddf['id3'].astype(str)
daskDF = ddf.read_sql_table('test_table', sqluri, index_col=Dindex_col)

I ran into this problem earlier in the week when, for the first time, converting a package to use Dask instead of Pandas. It was a serious bummer since the performance benefits I was starting to see were dramatic. If I have more time in the future, I would be more than happy to do the implementation if one has already been designed.

@0x00b1 see https://github.com/dask/dask/issues/1493#issuecomment-704969040 / https://github.com/TomAugspurger/dask/tree/multiindex.
I don't recall where that branch is at, but the main choice is how to represent the data. The two choices are a list of arrays, or as an ndarray of tuples. pandas does a bit of both, but tries to avoid "materializing the tuples" as long as possible. I think Dask has to take the tuple approach. Things like partitionquantiles seems pretty hard to do on a list of arrays, but is relatively easy on an ndarray of tuples (our existing code mostly works).

@TomAugspurger What about representing the MultiIndex as a sparse CSD array?
This would also allow for fast partitioning by slicing the most outer dimension of the CSD array.
Being more fancy, we could also have multi-dimensional partitioning along multiple dimensions of the array.
This would be especially useful as a secondary index.

I'm not sure, would any advantages to using a sparse CSD array apply equally well to a pandas MultiIndex?

Being more fancy, we could also have multi-dimensional partitioning along multiple dimensions of the array.
This would be especially useful as a secondary index.

I think I looked into this briefly, but struggled with the requirement that DataFrames are ordered so (AFAICT) the partitioning strategy needs to include information from every level of the MultiIndex.

My company is interested in paying someone to implement this functionality.  Would significantly simplify our infrastructure and be a great contribution to the community.  Please reach out to me if you're interested or would like to suggest someone who can do it. @TomAugspurger @Hoeze @jsignell

^ @mrocklin , I don't know if Coiled does paid feature requests or if there is any capacity. Perhaps you have a lead for this.

Hey @terramars Nice to virtually meet you :) Coiled would certainly like to try and help out! Especially since it would help out the community as well as make you and your teammates lives easier.
Are you free sometime next week to talk over the specifics? Here's a link to my calendar where you can find a time that works best for you.

Thanks for reaching out, definitely would love to chat next week. After I
wrote this I actually realized a bigger problem for us is the inability for
dask to shuffle large data frames, which seems like an ongoing issue
although at least it's being worked on. We can see what makes sense and if
there's some way to start an engagement.
On Thu, Aug 4, 2022, 5:39 PM 0-ren @.***> wrote:

Hey @terramars https://github.com/terramars Nice to virtually meet you
:) Coiled would certainly like to try and help out! Especially since it
would help out the community as well as make you and your teammates lives
easier.
Are you free sometime next week to talk over the specifics? Here's a link
to my calendar https://calendly.com/d/dtn-w6c-tr9/talk-about-coiled-o_p
where you can find a time that works best for you.
—
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1493#issuecomment-1205795969, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABHA64IYPXDJDJE4ICAUARTVXQ2BXANCNFSM4CNELPYQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Sounds great! Excited to help you and hopefully the community too! :)

Hello, I'm also interested in this feature, but my use-case is perhaps simpler, and there might be a way to implement this in an acceptable way that's very simple.
In my case, partitioning and chunking only need to be performed on the first level of the index.
Could a simple implementation involve Dask just keeping track of the first level of the index as a simple index, and delegating everything else to Pandas? This would restrict partitioning and chunking to the first level, but I think in a lot of cases that might be good enough.

I’ll upvote this. In my work, many of our datasets have multiple “natural index” columns that we frequently join on together or filter/group by. Or sometimes there isn’t a single column that has high enough cardinality to effectively distribute a DF across many partitions. The result is that we can’t get the optimizations that dask has when operating with the index. This can also be a stumbling block in learning to transition from pandas code.
I’ll throw a thought out there and maybe one of the dask devs can say whether it’s totally off base: pandas now has pretty solid support for using Arrow-backed dtypes, including Structs. Would having the dask DF index be technically a single column, but consisting of a struct with multiple child arrays (kind of like the single MultiIndex object) help mitigate some of the technical difficulties that adding multiple-column indexes ran into?


[WIP] Neovim Python & Synchronous +PlugInstall

Buffer updates now managed by __main__ loop with buf_q. Entirely for neovim support.
Synchronous neovim install temporarily provided by python installer.

Known issues:

No ctrl-c/interrupt support on nvim.
Graphical bug: neovim/python-client#103
Travis Python 3 build segfaults.

@junegunn Was interested in this to respond to #104 . This PR is work in progress still need to fix segfault. Open to comments.
Open Questions:

Readme changes? Should it be noted in the front page like: ATTENTION NEOVIM USERS for sync install etc...
Should nvim +PlugInstall require the neovim python lib. That is to say, don't do any installation from starting to prevent asynchronous install entirely or fallback to neovim's installer?

Thank you. My suggestion is that we don't explicitly say anything about this change since it's a temporary workaround to a known issue. A subset of Neovim users who have python support enabled will benefit from this change, while the others will still find that vim-plug doesn't work like Vundle but they'll notice that there is an open issue. It's clearly a better situation than now anyway.
What is tricky is that there's no guarantee that has ('vim_starting') means that the user is expecting synchronous installation. One can just start Neovim with +PlugUpdate to make sure that the plugins are up-to-date. So I don't think we should disallow installation or warn the users as we can't be sure of their intentions.

@junegunn When you say don't explicitly say anything do you mean also remove my warning? Or I'll leave it?
If you ask me, most users of +PlugInstall/Update will want synchronous behaviour because most use it as part of script/docker etc...

New problem, during update with nvim ultisnips stops update?
Error detected while processing function UltiSnips#CursorMoved..provider#python#Call:
line   12:
Invalid channel "1"

When you say don't explicitly say anything do you mean also remove my warning? Or I'll leave it?

I meant removing warning as well, but either is fine. Your call.

If you ask me, most users of +PlugInstall/Update will want synchronous behaviour because most use it as part of script/docker etc...

True, but we have to assume that there can be exceptional use cases.

Hmmm, found new small bug. On neovim only, update gets interrupted by ultisnips:
Error detected while processing function UltiSnips#CursorMoved..provider#python#Call:
line   12:
Invalid channel "1"

@junegunn Think I've gotten all the problems. Currently no warning printed to users, simply uses python installer silently.
Limitations still existing:

No ctrl-c/interrupt support on nvim.
Graphical bug: neovim/python-client#103

I commented on the code. Just some minor nitpicking. Other than those, looks great. You can squash the commits so I can merge. Thanks!

@junegunn Changes made, overall a nice improvement. Hope nvim users notice.

@starcraftman Looks like travis build is getting delayed. I'll merge when it's done :)

Merged, many thanks as always!


Why is the keystore not on the required object

When I console log the required object it shows this:
{"txutils":{},"encryption":{},"signing":{},"upgrade":{}}
After looking at the index.js file I would expect there to be a keystore object as well:
module.exports = {
  txutils: require('./lib/txutils.js'),
  encryption: require('./lib/encryption.js'),
  signing: require('./lib/signing.js'),
  keystore: require('./lib/keystore.js'),
  upgrade: require('./lib/upgrade.js'),
};

Seems as though browserify didnt like something in the file.  Will have to figure out what...

Looks like the exported KeyStore var is null in keystore.js.  Any idea why it would be null?

I am using browserify version 14.4.0 on a mac.  This is the command I use to generate the browserifyed file:
browserify main.js -o build.js -d
This is my package.json file
{
  "name": "name",
  "version": "0.0.1",
  "main": "main.js",
  "dependencies": {
    "eth-lightwallet": "^2.5.6"
  },
  "devDependencies": {}
}

This is my main.js file:
var lightWallet = require('eth-lightwallet');
global.doStuffs = function() {
  return JSON.stringify(lightWallet);
}

This is the return value of the doStuffs function:
{"txutils":{},"encryption":{},"signing":{},"upgrade":{}}

Ive been stuck on this for a couple of days now and could really use some input.  Any help is much appreciated.

I think I figured it out.  The stringify method seems to be ignoring the keystore object but its actually there.


[Question] Fiber yield behavior

I am looking for a way to 'yield' from the currently executing fiber in the sense that the fiber would be sent to the back of the worker Work::fibers queue (as opposed to Work::waiting) and would otherwise resume once its turn comes, without requiring any notification, event signaling or timeout.
I have been looking at the Scheduler::Fiber API and the various wait() overloads and cannot see a way to achieve that.
So my questions are:

is there a way to achieve that that I overlooked?
if there isn't a way, is there a fundamental/strong reason why the API doesn't include that, i.e. could it be added as a contribution?

I have just thought of a workaround: calling
.wait(std::chrono::system_clock::now());
would achieve something similar to what I want, using 'now time' to force a reschedule without actually waiting.
However it's still not semantically very clean and might be inefficient, so I am open to better suggestions.

Hi @Guillaume227,
Marl doesn't really make any guarantees about order of execution - scheduled tasks that are unblocked can execute in any order. Pushing an unblocked task back on the work queue could immediately resume that task, without invoking any others, in which case you've made no forward progress with other tasks.
I'm lacking context to give advice here. Is there a reason why you can't suspend the task and use a signal to resume the task later?
Cheers,
Ben

I came upon marl as I was looking for a way to simulate some single-core embedded thread scheduling.
The embedded implementation uses round robin scheduling (with some optional priority specification) for threads.
My initial attempt using std::thread and std::condition_variable failed because the OS just doesn't give you enough control over when threads are going to yield/run. I then looked into coroutines (I am using c++20) but was also defeated due to the inability to co_await from nested functions.
So my scenario is specifically looking at the single thread / single worker case.
In simulation mode, I swap every embedded thread instance for a marl Fiber. Those threads can be either signal-based or they run in an infinite loop with a sleep statement at the end of every iteration.
Signal and sleep statements map nicely to marl's signal and wait API. I.e. I am happy for those threads to run without interruption until the next sleep/wait_signal statement (In the simulation code, I inject a call to suspend() in every sleep call from the embedded side).
On top of that I have a special control fiber that gets called once all other fibers are blocked. It is responsible for advancing the simulation clock (I swapped std::chrono::system_clock for a custom clock that give me control of the time), and for breaking out of the simulation after some time.
That's the fiber that I wanted to just yield from once it has advanced the clock, without a definite wait time.
Regarding order of execution, my understanding was that I could rely on some FIFO behavior in the fibers queue. Some initial testing with just two simulated fibers (in addition to the control fiber) seemed to work, but as you say there is no guarantee so that might just be luck.
I think I am getting close to something that fits my use case, I might have to maintain some overlay in my fork for the custom clock bit and a way to specify Fiber priorities (i.e. an integer such that the Fiber with the highest prio will be picked at every suspend step out of all the fibers ready to be resumed). Maybe adding some optional sorting func that suspend would call on fibers to pick the next one to run.
For the clock I am not sure how to customize the Scheduler class cleanly without  making it a template which is a mess because of the logic in the .cpp file.
template<typename ClockType>
class Scheduler { ... /*yuck, everything has to go in the h file now*/ };

Thanks for the reply @ben-clayton. I am having a good time looking at your work, it's quite neat, well thought-out, elegant!

Hi @Guillaume227 ,
Thank you for the additional context.
If you need explicit FIFO behaviours,  I wonder if marl::Ticket might be helpful.
Assuming you have a way to access a shared marl::Ticket::Queue, you could have each of the fiber tasks synchronize themselves with a queue to 'round-robin' with something like:
#include "marl/defer.h"
#include "marl/scheduler.h"
#include "marl/ticket.h"

int main() {
  // Scheduler with no threads, just fibers
  marl::Scheduler scheduler(marl::Scheduler::Config{});
  scheduler.bind();
  defer(scheduler.unbind());  // unbind before destructing the scheduler.

   Each task holds a loop that'll spin until the
  // work is done.
  for (int fiber_id = 0; fiber_id < 5; fiber_id++) {
    // Grab a ticket outside of marl::schedule, so that initial orders are
    // deterministic.
    marl::Ticket initial_ticket = queue->take();
    // schedule a task for the fiber of execution
    marl::schedule([=] {
      marl::Ticket ticket = initial_ticket;
      for (int i = 0; i < 3; i++) {  // Or loop until some other signal is set.
        ticket.wait();               // wait until this fiber is called
        printf("Fiber %d's turn!\n", fiber_id);
        // before we call the next ticket, grab another one at the end of the
        // queue.
        marl::Ticket next_ticket = queue->take();
        // signal we're done. The next queued fiber will now be unblocked.
        ticket.done();
        // next loop around, we'll want to block on the new ticket.
        ticket = next_ticket;
      }
    });
  }

  // All done.
  return 0;
}

Running this should give you the deterministic output:
Fiber 0's turn!
Fiber 1's turn!
Fiber 2's turn!
Fiber 3's turn!
Fiber 4's turn!
Fiber 0's turn!
Fiber 1's turn!
Fiber 2's turn!
Fiber 3's turn!
Fiber 4's turn!
Fiber 0's turn!
Fiber 1's turn!
Fiber 2's turn!
Fiber 3's turn!
Fiber 4's turn!

I'm not too sure how well this would fit with your existing changes, or the control fiber, but might give you some more ideas on how you could tackle this with the existing marl primitives.

For the clock I am not sure how to customize the Scheduler class cleanly without making it a template which is a mess because of the logic in the .cpp file.

Understood. The std clocks are not friendly for swapping, without heavy use of templates. Sorry, I don't have any suggestions here.

Thanks for the reply @ben-clayton. I am having a good time looking at your work, it's quite neat, well thought-out, elegant!

Very kind of you. Thank you!


[Feature] Maximum price

Since there is a minimum price feature, it would make sense to include a maximum price feature as well, in my opinion.
This has very valid usage scenarios on say, a prison server or really anything else with a delicate economy.

Yes, that should be possible 👌


Feat(eos_cli_config_gen): add router IGMP host proxy configuration support

Change Summary
add host proxy support under router IGMP
Related Issue(s)
N/A
Component(s) name
arista.avd.eos_cli_config_gen
Proposed changes
proposed data model
router_igmp:
  host_proxy:
    vrf:
      - name: default
        match_mroute: all
      - name: BLUE
        match_mroute: iif

How to test
see the molecule test included
Checklist
User Checklist

N/A

Repository Checklist

[ x ] My code has been rebased from devel before I start
[ x ] I have read the CONTRIBUTING document.
[ x ] My change requires a change to the documentation and documentation have been updated accordingly.
[ x ] I have updated molecule CI testing accordingly. (check the box if not applicable)

For me this is unrecognised command-
s1-leaf1(config-router-igmp)#host-proxy ?
% Unrecognized command

s1-leaf1(config-router-igmp)#host-proxy match mroute all
% Invalid input

For me this is unrecognised command-
s1-leaf1(config-router-igmp)#host-proxy ?
% Unrecognized command

s1-leaf1(config-router-igmp)#host-proxy match mroute all
% Invalid input

what version of EOS are you running?

For me this is unrecognised command-
s1-leaf1(config-router-igmp)#host-proxy ?
% Unrecognized command

s1-leaf1(config-router-igmp)#host-proxy match mroute all
% Invalid input

what version of EOS are you running?

its 4.30.2F-33092737.4302F


On macOS, sed is a bit more strict about syntax and flags

See https://github.com/jhipster/jhipster/pull/50#issuecomment-373915882

Ugh had to amend the commit, it worked but also left artifacts like pom.xml-e (it interpreted the -e as extension)

Well, that is one of the more annoying exercises I've done in a long time...
Turns out that it is basically not possible to find a command syntax for sed -i that works on gnu-sed (linux) as well as bsd-sed (macos) :-(
In the end I just removed the -i and redirect to a new file, and then mv it back into place.

thanks, LGTM


Terminal toggle size to content width vertical scroll bar is in wrong position

Repro:

ls
Size terminal narrow so lines wrap
Press alt+z

5. shift+wheel down

Related: https://github.com/microsoft/vscode/issues/133962


State "fronius.0.inverter.1.Power_DC_String_2" has no existing object, this might lead to an error in future versions

v1.1.3
State "fronius.0.inverter.1.Power_DC_String_2" has no existing object, this might lead to an error in future versions

with version v.1.1.6 there are some more:
`

fronius.0
2022-02-16 09:19:54.985
warn
State "fronius.0.inverter.1.UAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:54.983
warn
State "fronius.0.inverter.1.UAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:54.982
warn
State "fronius.0.inverter.1.UAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:54.982
warn
State "fronius.0.inverter.1.IAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:54.981
warn
State "fronius.0.inverter.1.IAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:54.939
warn
State "fronius.0.inverter.1.IAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:26.662
warn
State "fronius.0.inverter.1.UAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:26.662
warn
State "fronius.0.inverter.1.UAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:26.661
warn
State "fronius.0.inverter.1.UAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:26.660
warn
State "fronius.0.inverter.1.IAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:26.658
warn
State "fronius.0.inverter.1.IAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:19:26.657
warn
State "fronius.0.inverter.1.IAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:55.991
warn
State "fronius.0.inverter.1.UAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:55.990
warn
State "fronius.0.inverter.1.UAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:55.990
warn
State "fronius.0.inverter.1.UAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:55.990
warn
State "fronius.0.inverter.1.IAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:55.989
warn
State "fronius.0.inverter.1.IAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:55.988
warn
State "fronius.0.inverter.1.IAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:27.681
warn
State "fronius.0.inverter.1.UAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:27.681
warn
State "fronius.0.inverter.1.UAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:27.680
warn
State "fronius.0.inverter.1.UAC_L1" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:27.679
warn
State "fronius.0.inverter.1.IAC_L3" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:27.679
warn
State "fronius.0.inverter.1.IAC_L2" has no existing object, this might lead to an error in future versions

fronius.0
2022-02-16 09:18:27.678
warn
State "fronius.0.inverter.1.IAC_L1" has no existing object, this might lead to an error in future versions

`

Are the messages "self healing"? A restart should definitely fix it... Anyhow the target is to fix it mainly in version 2.0

no there is no self healing and i did  some restarts but isn't better.
so, you plan a version 2.0??

Hallo nkleber (norbert), weißt du schon, wann Version 2 erscheinen soll (Stable)? Ich bekomme auch diese Fehler...

fronius.0
2022-05-19 11:08:25.553
warn
State "fronius.0.inverter.1.Power_DC_String_2" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.552
warn
State "fronius.0.inverter.1.Power_DC_String_2" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.552
warn
State "fronius.0.inverter.1.Voltage_DC_String_2" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.551
warn
State "fronius.0.inverter.1.Power_DC_String_1" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.551
warn
State "fronius.0.inverter.1.Voltage_DC_String_1" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.550
warn
State "fronius.0.inverter.1.Temperature_Powerstage" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.549
warn
State "fronius.0.inverter.1.Current_DC_String_2" has no existing object, this might lead to an error in future versions

fronius.0
2022-05-19 11:08:25.548
warn
State "fronius.0.inverter.1.Current_DC_String_1" has no existing object, this might lead to an error in future versions

Leider bekomme ich keinen Support vom Repository maintainer. Somit ist das Thema leider on Hold.  Aber bitte gebt acht bzgl. der neuen Datenstruktur!...
https://github.com/nkleber78/ioBroker.fronius/tarball/DynamicParsing

Hab die Version mal drübergebügelt. Er verbindet sich garnicht mehr mit dem Dienst

Probleme sollten alle in der V2.0.0 gelöst sein. Jedenfalls bei mir weder im Test noch im Live system reproduzierbar


Allow long values in submit_histogram_bucket

We're currently enforcing an integer, which obviously fails with bigger values.
Question: should it be long long?

This should wait for https://github.com/DataDog/datadog-agent/pull/4887, it doesn't make sense to merge this before the bucket interpolation works better

(also, needs a release note)

Moving this to 7.19 since we're waiting for #4887
Should we dot the same for all other method (gauge, rate, ...) ?

moving this back to 7.18.0 actually, since we'll actually want to merge the present PR as soon as #4932 is merged


Asp.Net WebApi example has a performance issue

Hi,
I'm trying to write an Asp.Net WebApi controller following your example, but I noticed a performance issue in the code that will make the upload take a long time to finish.
The problem is in this portion of code:
[NonAction]
private bool AllChunksAreHere(ResumableConfiguration configuration)
{
	for (int chunkNumber = 1; chunkNumber <= configuration.Chunks; chunkNumber++)
		if (!ChunkIsHere(chunkNumber, configuration.Identifier)) return false;
	return true;
}

For a large file (let's say 4096 MB) you'll have 4000+ chunks and for each chunk uploaded there will be a check that all chunks are uploaded ... so that's a 4000 chunks * 4000 checks. It's no problem for smaller files, but a huge performance issue for large files.
I've changed the body of the function a little and now it looks like:
   if (configuration.ChunkNumber < configuration.TotalChunks)
      return false;
   return true;

where ChunkNumber is resumableChunkNumber and TotalChunks is resumableTotalChunks, both passed to the upload method of the controller.
Regards,
Daniel Nutu

Hi Daniel
While it's true that it's "inefficient" to check all chunks every time, we cannot guarantee the order of chunks in resumable.js - it's quite likely that the "end chunk" is not the last to get uploaded. A better solution is probably to have a server-side cache/counter adds each chunk as it is completed.
Say a file with 4 chunks (1, 2, 3, 4) is sent. Because of glitchy internet (or perhaps a switch from 3G to WiFi), chunks are sent 1, 2, 3, 4 but are received 2, 1, 4, 3. Suddenly chunk 3 is the "final" chunk, but the server will assume that all chunks have been sent when chunk 4 is received and an incomplete file will be assembled.
Creating an efficient server-side check is beyond the scope of this library, but that should be the focus instead of assuming the chunk order.

Thank you for the clarification!


[Solved] prettier conflicts with beautify

I was using beautify before using prettier and it seemed to work well for a while.
However, immediately after that, I realized that storing jsx in js would always cause breaking.
I did not find a cause for it, but it turned out to be conflict between prettier and beautify.
Expected

Actual

The best way is to remove the beautify of duplicate personality.
But if you want to use it at the same time in other code like html, css, else, add one to your user settings.
"beautify.ignore": [
    "**/*.js",
    "**/*.jsx",
    "**/*.ts",
    "**/*.tsx"
 ]

If you use the VSCode and the Prettier, and you use .js to React.
This helps you.
"files.associations": {
        "*.js": "javascriptreact"
    },


_config.yml | phone number formatting

need dashes

Inserted telephone dashes, descriptions and email code into each program config file


Add: Streaming to doSwap() examples

Add a new section for streaming swaps, Clear examples and comments

@Thorian1te I will do this


Add a flag to make the connection automatically emit the source object.

Mainly used to improve the connection dialog.
It seems that the previous connection dialog only supports constant bindings. Though duplicating a node in the editor will also duplicate the existing connections, but we still need to adjust some parameters, and it's not very efficient to frequent operations in a pop-up dialog.
For existing connection-related parameters, these can be divided into three categories, listed in the order of use: the parameters of the signal, the binds arguments in the connection, the bound arguments of the callable.
It is probably well known that the binding parameters in the connection dialog are the bound arguments of the callable.
In the current situation, it seems difficult for the receiver to know the information of the signal sender when only use the connection dialog.
In this commit, add a flag bit ConnectFlags.CONNECT_SOURCE_AUTOBIND to make the connection automatically emit the source object.
If this flag bit is enabled, the source object will be emitted at the end of the emitted parameters, right before the bound arguments of the callable. at first, no matter what the other parameters are.
Hopefully this will make the connection dialog a bit more usable.
~~https://user-images.githubusercontent.com/30386067/162772481-c046b302-178d-41ba-ae8b-145c67ba3a39.mp4~~
https://user-images.githubusercontent.com/30386067/162824296-ea974077-20fe-4a4e-8a06-037341798362.mp4

I somewhat like this new flag, but the fact that it requires so much text to be explained properly concerns me and makes me question its simplicity.

"Auto-bind the source object during emission. Mainly used to improve the usability of the connection dialog. Connection-related parameters can be divided into three categories, listed in the order of use: the parameters of the signal, the [code]binds[/code] arguments in the connection, the bound arguments of the callable. If this flag is enabled, the order of the new parameter in the list is right before the bound arguments of the callable (that is, the last parameter to emit. By the way, the binding parameters in the connection dialog are the bound arguments of the callable.)."

The current feeling is that signal and connection binds know the parameters of the original callable, so their parameters are combined to be the parameters of the original callable; while callable.unbind hides the parameters of the original callable that are not needed, and callable.bind will append some.
Now it seems to be expected to append a source object after the unbind/bind processing is completed.
IMHO, right now, it actually feels a bit confusing about Callable, even without adding this new parameter:

The result of Callable.bind and Callable.unbind is that we can't find the original Callable, only the Callable after these operations, so it seems that bind/unbind behaves like +=/resize.
bind/unbind behaves more like an insert than an append. Because the calling order and the argument list order of the new callable are reversed. c.bind(A).bind(B) => func c(B, A).

On the other hand, it might make sense if we thought that the callable (the Receiver Method) was trying to receive these parameters. This is how I understand it.:
First, its parameter list size is 0, use bind/unbind to adjust the parameter list, receives (insert) parameters for singal, receives (insert) parameters for connection binds, and now also receives (insert) the source object. It becomes what we see.

Heyo, I already removed the callable binds in #63595 so feel free to rebase your PR. I think for merging, as was discussed, the source object should be appended and not prepended. Other than that it looks good!

Now use a new commit to solve the callable.unbind problem: by changing the order of bind.
The source object will be appended, right before the callable binds, after the original signal arguments, after the callable unbind (if it exists). This will ensure that unbind still only work on the original signal arguments.

Okay, don't really agree, but following your advice. Let's wait for the feedback from the users.

2. bind/unbind behaves more like an insert than an append. Because the calling order and the argument list order of the new callable are reversed. c.bind(A).bind(B) => func c(B, A).

For reference:  #62891.

I modified the usage condition of this option in connection dialog.

Using unbind in the connection dialog will automatically disable this option, in the same way that "Extra Call Argument" is disabled.
It is up to the user when used in scripts.


End of August workshop?

We have the Hatcher Library Gallery Lab reserved on Wednesday, August 31 and Thursday, September 1 for a potential workshop (3 weeks from today!). Does anyone feel that they have the capacity to plan and teach then? Is it enough time to advertise? Potential audiences could be WISE or the library, or we could run a general Data Carpentry workshop.
I moved it to the top of our planning spreadsheet. I'm happy to help out with planning and some teaching...

@alixk I just signed up.  I can only be a helper since I have not been certified to teach Data Carpentry.  I can help in other capacities if needed just let me know what you would like me to do.

I can only be a helper too - I just don't have the bandwidth to do more. Sorry.

If needed, I can teach or help.

@marcsze you can teach without certification as long as there is a certified instructor who is there in person and also teaches a module.  In January, I ran a workshop as the lead instructor and had 3 first time instructors teach the other 3 modules and I think it worked well!
Unfortunately, I will be out of town for this workshop but I'd be happy to help set up the website.

Also to get certified as a DC instructor requires a desire and sitting in on an inboarding conference call. If you reach out to Tracy Teal<EMAIL_ADDRESS>and explain the situation, I’m sure she’d help you get certified.
Pat
On Aug 12, 2016, at 11:53 AM, Marian L. Schmidt<EMAIL_ADDRESS>wrote:

@marcsze https://github.com/marcsze you can teach without certification as long as there is a certified instructor who is there in person and also teaches a module. In January, I ran a workshop as the lead instructor and had 3 first time instructors teach the other 3 modules and I think it worked well!
Unfortunately, I will be out of town for this workshop but I'd be happy to help set up the website.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/UMSWC/Administration/issues/23#issuecomment-239484466, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNKBNLM1qEimrVk4_KsSnJjYG43U8_oks5qfJbvgaJpZM4JhkQp.

Thanks all. One one hand, it's coming up really soon, so we would need to send out announcements ASAP and firm up our plans. On the other hand, we haven't had a workshop in a while, so we could just go ahead and do an open campus workshop and see who shows up...
Do we want to try out Data Carpentry? The only lesson I'm comfortable teaching at the moment is Open Refine, and the Unix shell, if we chose to do that. If you want to teach--and this could be an excellent opportunity for first timers, since it may be a pretty small workshop!--maybe add in to the spreadsheet which DC lessons you can teach.

Ok in that case I can do either (teach or help) whichever is needed.  I'd be okay with spreadsheets and the visualization in R sections.

Can we add components of software carpentry to the data carpentry program?  I like the idea of having an open workshop since it is coming up really soon.  I'm guessing we still need a website up and running even if it is an open workshop?

@marcsze sure--what were you thinking? I always like including the shell lesson, because I think that material is important for everyone to learn :)
Yeah, we definitely need a website soon... @marschmi did you volunteer to do this or am I imagining it? Do we also need to contact Maneesha for the Eventbrite? I can look at this more tomorrow.

I was just coming here to note that I had volunteered to help with organizing and have not done anything to help with this yet! I can help with this. I think the two main things are:

setting up a website, and
contacting Maneesha about the Eventbrite.
I can do both of those.

It seems like the dates are set and the room is reserved (thank you, Alix!)
Alix and Marc are signed up as people who could be instructors, and Arthur and Sonal are signed up as potential helpers. Are there other folks who are interested & available? This is the next most important thing to get worked out, as we'll need to tell potential attendees about what we'll cover, and that will depend on the instructors.
I'm putting a link to the host checklist here so I can find it easily again in the future:
http://software-carpentry.org/checklists/host/

@alixk If it is a two day course what do you think of
Day 1: spreadsheets, openrefine, shell
Day 2: Git and data analysis & visualizations in R
So the git and shell would be software carpentry and the rest would be data carpentry.  I've never taught before so I'm not sure how this looks with respect to flow and time.  What do you think?

Awesome, thanks @duffymeg!
@marcsze That schedule sounds great to me. How about using the Ecology lessons from DC?
As far as instructors:
Day 1: spreadsheets (Marc), openrefine (Alix), shell (Arthur or Marian? Or Marc? I'm happy to teach this as well)
Day 2: Git and data analysis (Marian?) & visualizations in R (Marc)
And for helpers we have @singhal and @arthur-e. What does everyone think?

@alixk I'm okay with using the Ecology lessons from DC.  I'll just have to brush up on the material covered.
I'm okay with spreadsheets although I would probably prefer to teach Git and switch with @marschmi on the R visualizations part if that is okay with you and her.

I made the DC workshop link with the eventbrite ready to go (in its rough draft form at the moment) you can see it here:
https://umswc.github.io/2016-08-31-umswc/
Typically data carpentry workshops are not taught with the shell and git lessons.  Instead, DC workshops usually include SQL.   However, if we decide to keep the git lesson - I'd be happy to be the instructor.
We will need to e-mail<EMAIL_ADDRESS>ASAP once we have the workshop details.

Whether we teach SQL, Shell, or Git in addition OpenRefine and R, I think the important thing is to make it coherent. That is, how does, say, the UNIX shell, contribute to the workflow seen with OpenRefine and R?
What is the merit of teaching X in a DataCarpentry lesson plan?

UNIX shell can be used to quickly perform QA/QC and some data cleaning, especially on large datasets (e.g., with head, tail, wc, and redirection).
SQL reinforces advanced R concepts in transforming relational data that are taught with plyr and dplyr; at the same time, orienting students with databases encourages them to get durability et al. (ACID compliance) for free.
SQL builds on familiar tabular data, perhaps a "stepping stone" between spreadsheet-thinking and OpenRefine and data frames in R.
Git/ Mercurial prevents my peers from telling me they just lost a week of work because their OS or hard disk failed.
Anytime R and the UNIX Shel are taught, Git/ Mercurial is a natural compliment because of everything we know that is important about version control.
If Git is taught, UNIX shell lesson supports the command line experience (probably essential they are taught together, with the UNIX shell first).

I like @alixk 's idea of doing spreadsheets and OpenRefine in one (morning) session; they are both short enough for that. If we assume some basic R experience as a prerequisite, we can compress the advanced R stuff into one session, leaving two sessions open for...

UNIX shell + Version control?
UNIX shell + SQL?
UNIX shell + Some other advanced R topic?
Version control + Some other advanced R topic?

I would be okay with either

UNIX shell + Version control
UNIX Shell + SQL

Mostly since I think that either of these two combinations are really important as a foundation in reproducible data analysis.

I lean toward doing version control, for the reasons @arthur-e outlined. However, I'm not comfortable teaching either SQL or version control, so it's really up to whichever someone wants to teach!

Let's stick to Git then. There is a portion in the R data carpentry section that uses SQL so if we skip that portion we should have time for other components.

During the weekly instructor discussions on the SWC/DC mentorship committee, we do not suggest combining lessons between DC and SWC as they were made for separate purposes.  The DC lessons are meant to be domain specific (e.g. genomics/ecology/geospatial data) while SWC workshops are more geared towards general programming skills.  I am worried that if we make this workshop open to anyone and then start the workshop off with domain specific data within the spreadsheets and open refine lessons that participants may lose interest.
As we currently do not have a specific audience and it is last minute, I think it would be best to do a typical SWC workshop.  In the future when we have a domain specific audience (e.g. the SNRE/EEB workshop or human genetics) a DC workshop might be best.
Therefore, I propose the following SWC workshop schedule:

Day 1 morning: UNIX Shell
Day 1 afternoon: Introduction to R
Day 2 morning: Version Control with Git
Day 2 afternoon: Advanced/Visualizations in R

What do people think?

Truthfully, I'm good with either or and am just excited for the opportunity to teach.  I would defer to @marschmi on what can and can't be taught under the SWC and DC banner since I'm really new and am unfamiliar with proper protocol.

@marschmi I think that makes sense. Making the example problems and data relevant to participants is always a bit of a challenge, so I wouldn't want to create even more of a barrier for ourselves in teaching. I'd really like for us to host a DC workshop soon, but since this is very last minute, I agree that SWC is best!
What do people want to teach? I defer to @marcsze and @singhal since you guys have yet to teach. I'm most comfortable with shell, although I could maaaybe do Intro to R... I'll wait to see what everyone else wants to teach.

I'm okay with all four of these topic areas, however, since @alixk is more comfortable with UNIX shell and intro to R I can do either the Git or Advanced/Visualizations in R.

To be honest, right now I only have the capacity to teach one lesson, preferably the shell lesson. However, I am more than happy to be a helper and do more of the hosting work if there is a newer instructor who is most comfortable teaching that lesson.
Other than that, are we ready to go? @marschmi the website still mentions SQL as the 4th lesson--do we need to change that to Advanced R? Then we can start sending out announcements.
Is there funding for snacks this time around?

Since everyone is relatively busy at this point in time, how does the following look to everyone?
Day 1 morning: UNIX Shell ( @alixk )
Day 1 afternoon: Introduction to R ( me )
Day 2 morning: Version Control with Git ( @Ry4an or myself )
Day 2 afternoon: Advanced/Visualizations in R ( @marschmi )
Helpers @arthur-e and @singhal . We can then rotate on and off as helpers as need be throughout the two days.  I noticed that @Ry4an put down that he wanted to be an instructor so hopefully he sees this.  Everyone only has to worry about one lesson then.

I did sign up on the sheet, but then remembered I'm out for the previous three days and decided to opt out and didn't send the email -- but on third thought, I'll take the slot if everyone's okay with me skipping out at noon after my section.  I've got a 12:30 meeting downtown I ought not miss, but would like to teach the git portion in the morning.

I am still happy to help with administrative/organizational stuff, but right now it's not clear to me what I can do to help! If you see something, let me know.

@duffymeg do we have a website up?  I think I know someone who might want to take part.

@marcsze This was set up by @marschmi, but needs editing:
https://umswc.github.io/2016-08-31-umswc/

@duffymeg what about snack funding? I can do the ordering and pick ups from Panera, but if you know Pat's shortcode...

@duffymeg Just updated the website with the proper instructors list and syllabus.

Snacks?! You want to feed these people? :) Try 100201 - I think @marschmi was able to use it for the WISE workshop a while back

Thanks @pschloss!
@arthur-e FYI, I'm still seeing Managing Data with SQL in the syllabus on the site.

@alixk You're right! It seems the workshop template has recently been updated. Anyway, I made up some stuff about the advanced R lesson. @marschmi will want to eventually fill in that part of the syllabus.

This looks great, all. Excited to help!

I can assist as teacher or helper. I'm not yet certified but am comfortable with basic Shell and Python lessons.

I apologize for being MIA as I have been working out my schedule.  Unfortunately, it looks like I will no longer be in town to help out with the workshop on Wednesday afternoon and Thursday.  I will be around until ~10 am in the morning on Wednesday and could help buy coffee, prepare the room for the workshop and help with intros.
I apologize again for cancelling and hope that there is a kind soul out there who can pick up my teaching spot for the second R lesson on data visualization in R (we should remove the word advanced).  Would someone please be able to cover me?
Has anyone emailed the admin for software carpentry to say that we will be hosting this workshop?

Yikes, we are running out of time. I could fill in for the advanced R lesson, but it would indeed be an advanced R lesson (à la Data Carpentry) and not a soft intro to visualization (don't have that material ready). Alternatively, I could teach the SWC SQL lesson if we're really in a pinch.

@arthur-e I think that sounds like a great plan if you're willing to do that. Thank you!

I updated the workshop website. Where are we advertising this workshop? Is there an e-mail template I could use to announce it on a listserve? Currently, it seems there are 20 slots available.

@alixk Since you're opening on Day 1, are you planning on an equity and inclusion statement at the beginning? Here's an example if you're looking for material.

Thank you so much Arthur!  I really, really appreciate it and am bummed that I won't be able to help out in the end.  As you all put together the final details of this workshop, please do let me know if there is any way that I can help.
I wanted to mention that there are now 2 people on the waitlist!  Congratulations on selling out everyone!

@arthur-e  will you be going over knitr or best practices with R during your lesson?  I want to make sure I am teaching things that will be complimentary to your more advanced usage lesson.

@marcsze Here is the lesson plan for what I'm teaching. I'm thinking I would start at no. 3, "Project management with RStudio..." but if you want to cover that and knitr I would move on to the next topic.

Project management with RStudio
Starting with Data Structures

Reading in a CSV as a Data Frame
Factors
Tabulation
Cross-Tabulation

Data Frames
Sequences and Indexing
Subsetting and Aggregating Data

Subsetting Data Frames
aggregate()
Function Application
The plyr Package

Analyzing data with dplyr

select() and filter()
Pipes
Mutating Data
Split-Apply-Combine with dplyr

Cleaning Data with tidyr
Visualization in R with ggplot2
Writing for performance in R

Pre-allocation
Vectorization

It's a lot of material so we'll see how far we get. I'd like to take a look at the pre-workshop survey results to get a sense of where everyone is. I'll be interested to see the noon-hour feedback you get on Day 1 from your R Intro.

@arthur-e  Okay, I was planning to go over RStudio and how to read and write data into R and then move into the Analyzing patient data, creating functions, and analyzing multiple data sets lessons.  I also plan to cover (if there is time) the making choices lesson and Loops in R.  With this I think it will set up your whole plyr, tidyr, and dplyr lessons up nicely.
Within the lessons are also some base plotting functions so that will also give a little bit of a tie in to any ggplot2 stuff you want to cover
Let me know if you agree with this plan of action.

@marcsze That all sounds good!

Oops, sorry to be MIA--I wasn't getting notifications for some reason. @arthur-e Sure, I can do an equity/inclusion statement. Do we want to start off with the stereotype threat exercise as well? I think in the other workshop I taught, we did this after lunch on the first day, but starting off with it sounds good to me, too.

Also, @marcsze @arthur-e @Ry4an @marschmi, let me know if you want to meet for coffee or something tomorrow afternoon/evening to go over things. Totally optional, but it can be nice to touch base in person!

It'll have to  be in the evening for me since I won't be back in Ann Arbor until after 3pm

I'm cutting it even closer.  I won't be back until Wednesday afternoon, so I won't be able to join at all. I'm reviewing my lesson and practicing, but I'll be mostly unavailable before my morning.

I have GSI training all day tomorrow but am available at 4 PM.

Actually, on second thought, I think I might be spending tomorrow evening practicing and going over stuff. So if no one objects, perhaps we can just all convene Wednesday morning before the workshop? Sorry for the change of plans.
Does anyone know how to access the Eventbrite information? We'll need to email the participants ASAP about the pre-survey, etc. I can pick up the coffee and snacks about 8ish on Wednesday, and we can meet in Hatcher before 9 (Hatcher opens at 8am).

Hi @alixk, @marcsze, @Ry4an, and @arthur-e,
I have sent along the eventbrite information to you regarding the attendees and the waitlist.
Unfortunately, I do not think we will have access to the pre and post surveys if we haven't e-mail the software carpentry admin, Maneesha. In the workshops that I have hosted in the past, Maneesha has shared these with the lead instructor via e-mail.  Hopefully we can have better communication with her in the future.


Hub Center - Asean Cyber University e-Learning Center

Educational model of schools or educational centers that sponsor children with disabilities to develop and learn normally. The project is associated with local production and business practices; increasing the mobilization of financial resources and conditions of facilities and equipment, ensuring the organization of teaching/educational activities in schools/centers in the direction of practical and theoretical learning. associated with practice and practice, ensuring basic education at kindergarten, primary and lower secondary levels, serving vocational education and channeling effectively..Propaganda and mobilization to raise awareness about the ASEAN Community and ASEAN cooperation in general and in the field of education and training in particular. Translated, edited and put into use for reference in high schools.

😍😍


Add Admin Reset Password Capability

Administrators should be able to reset users' passwords

You should be able to do that from the fabmanager cli.

OK - administrators should be able to reset users' passwords in Superset's UI

Shouldn't users be able to reset that on their own?

If a user forgets their password, an admin should be able to reset it via the UI

Would be great to be able to change user's passwords via the UI. Admins can create users with new passwords via the UI and it would make sense to allow creating a new password as well.

The best place for these features would be in Flask App Builder itself, we regularly update our dependencies to use the latest FAB.


[C++][Parquet] Add a test for skipping repeated fields

The existing test only tests non-repeated fields. Adding a test for repeated fields to make it clear that it is skipping values and not records.
Reporter: fatemah / @fatemehp
Assignee: fatemah](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=panahi) / @fatemehp
PRs and other links:

GitHub Pull Request #14364
GitHub Pull Request #14366

Note: This issue was originally created as PARQUET-2179. Please see the migration documentation for further details.

Antoine Pitrou / @pitrou:
Issue resolved by pull request 14366
https://github.com/apache/arrow/pull/14366


Field properties

Could you, please help me to fill in field properties correctly?
Intersect entity is created instead of OOB N:N relationship and I am not sure what should I write in the fields: Relationship name and Relationship entity.
Thanks a lot

Could you, please help me to fill in field properties correctly? Intersect entity is created instead of OOB N:N relationship and I am not sure what should I write in the fields: Relationship name and Relationship entity.
Thanks a lot

Did you resolve this?


Feature: Support deletion of an environment

Summary
We would like to delete an environment with all the apps contained in it, with the click of a button.
Proposal

We should be able to delete an environment with a single click.
It should be available in Global Configs only to superadmins.
Once clicked on Delete, it should delete all the CD pipelines for that particular environment, thereby deleting all the apps in the corresponding namespace.

AB#417

Duplicate of #2499


duckdb filesystem custom secrets

Description

Use custom secrets location for fs tests
Warn the user if they use the default location when creating persistent secrets
Prevent persistent secrets for in memory database versions
Add some secrets tests

Note:
I tried implementing changing the secrets location in create_authentication, but somehow I could not get it right. I think this solution with warning the user should be good for now.

@rudolfix done!


Register provider via policy For PaloAltoNetworks.

Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

/azp run java - paloaltonetworks - ci

API change check
API changes are not detected in this pull request.

/azp run java - paloaltonetworks - mgmt - tests


Fix RemovedInDjango40Warning via ugettext_lazy

This PR fixes a warning issued by this package when calling ugettext_lazy, which has been aliased to gettext_lazy since Django 2.0. Details: https://github.com/django/django/blob/c1442e1192057a3bf14aecbaa1b713eee139eaff/django/utils/translation/init.py#L139-L149
This name (ugettext_lazy) is considered deprecated, and my understanding is that this project does not support any django version prior to the alias existing. The only django versions that would be expected to fail on the new import in this PR are strictly < 2.0, possibly even earlier.
I did not test this PR, but given that it's a 1 character change I hope you don't mind. I'm not aware of your designated PR process: if you would like me to do additional work for this change, please let me know!

ugettext_lazy is just an alias for gettext_lazy after 2.2.
Relevant code from django source:
gettext_lazy = ugettext_lazy = lazy(gettext, str)

I would really like to see this merged.  :)


Dimensionality reduction

Unsure how to fix these errors

appending the line
umap-learn==0.4.6

to the requirements.txt file should fix it


Feat/fix empty

fixes https://github.com/Weakky/ra-data-opencrud/issues/26
this does a disconnect if you e.g. select the empty value in a SelectInput within a ReferenceField.
its based on the previous pr: https://github.com/Weakky/ra-data-opencrud/pull/35

@Weakky any chance to look over it and release it? Or can you delegate?

@Weakky really no chance to merge this and release? No one at prisma interested at this?


How to add FAB buttons programatically?

Hi,
I need to add or remove the FAB buttons programmatically, so I've added the menu in the XML file and then i tried to add the FloatingActionButtons programmatically with calling addView method on the menu.
The problem is that the menu does't show the new fab buttons because the children have created on onMeasure. I tried to call the requestLayout() method on the menu but nothing changed.
How can i handle it ?

I'll add that in the next release.

Have you added in 1.4.0?

Not yet, it was mostly bug fix release.

+1 that is big problem also for me

CircularFloatingButton of oguzbilgener do it if you want... but your library is better :dancer:


Continuous authorization

I think you should review how the authorization is done. It keeps redirecting me to the GitHub authorization page even after I just got authorized. I tried on two devices. As a result, I cant go past the entry activity. Did this happen to you too?

Should be fixed by this commit: https://github.com/pusher/android-slack-clone/commit/58f48e909db18e497decc18364af0bc103d5d550

Surprisingly, it didn't. I noticed there were multiple calls made to the onNewIntent function. That should be as a result of the launchMode used.

Did you try uninstalling the app and revoking access to the app on github (only if there is one present on your account)?

@KingIdee Ok, it should be working again (not very elegantly though)

Still not working on my end. Where you able to reproduce the error? I uninstalled the app, revoked Chatkit's access and I'm still experiencing it. Let me try to fix it at my end and make a PR

Is the authentication smooth at your end?

I'm still facing this issue, on my end.


[MOBILE][WHISPER] Can't search for chats

Same issue as other text input fields that should filter a list. This time for searching through your chats in whisper. Only filters the list if you click out of the text input or hit 'enter'

see #982 for solution


add TryGetValue method

We use TryGetValue quite a bit when using a dictionary like a cache.

Pst ... StringKeyDictionaryTests :)

i added a unit test; not sure how to run it.  none of the GDX unit tests show up in my unity?

Usually, they show up in the Test Runner under the GDX.Tests category, however, I've seen it glitch out before where you need to change from PlayMode to EditMode a few times, or even directly select the packages folder.

You nailed it though.

Our normal workflow runs a complete suite of tests against commits against the dev branch and updates the status of the commit.

This does make me think that I should do something about PRs, at least some quick checks.


workflows.argoproj.io/creator label gets overwrite by Argo CLI and UI

Summary
What happened/what you expected to happen?
Before we start, around 2/3 of issues can be fixed by one of the following:

Have you double-checked your configuration? Maybe 30% of issues are wrong configuration.
Have you tested to see if it is fixed in the latest version? Maybe 20% of issues are fixed by this.
Have you tried using the PNS executer instead of Docker? Maybe 50% of artifact related issues are fixed by this.

Diagnostics
👀 Yes! We need all of your diagnostics, please make sure you add it all, otherwise we'll go around in circles asking you for it:
What Kubernetes provider are you using?
What version of Argo Workflows are you running?
What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
Did this work in a previous version? I.e. is it a regression?
Are you pasting thousands of log lines? That's too much information.
# Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:
kubectl get wf -o yaml ${workflow}

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

workflows.argoproj.io/creator consider as argo system label. user should not use it. If users really wants to use it, they create a workflow using kubectl or k8s api.


Fix sourceVersion for 4 element versions (like <IP_ADDRESS>)

It turns out my previous PR #49 was incomplete: it fixed building something, but it also broke building the ssl module for <IP_ADDRESS>+.
This time I've added a test that checks if the ssl module is available. So far the test only runs for Python 2.7.18.*, other versions still have problems with building this module.

Seems that arm is failing, can you take a look?

Failures on aarch64-linux are:

3.3: 3.3.1-3.3.6
3.4: 3.4.0, 3.4.1

For 3.4: It looks like Python 3.4.x only added support for AArch64 in 3.4.2:
"bpo-22176: Update the ctypes module’s libffi to v3.1. This release adds support for the Linux AArch64 and POWERPC ELF ABIv2 little endian architectures." (https://docs.python.org/3.4/whatsnew/changelog.html#id34)
Would it make sense to exclude building 3.4.0 and 3.4.1 on aarch64?
I still haven't looked at 3.3.

I'm out for now, you can find today's findings below. I'll probably try something next week, but I would like to talk to you about directions (should I write an email?).
I did a bisect for 3.4.1 over nixpkgs for this problem ("KeyError: 'AARCH64'"), and it suggests that 97625d0307db3db59b09f4122717cdaed7adc926 is the first bad commit. The commit is titled "cpython: prune patches and configure flags", so it sounds like a plausible suspect.
For the record: I've used one Google Cloud's T2A instance with 8 vCPUs as a builder. It works well. They have a generous free tier, but it's restricted to 90 days from registration.


Update to v0.14.0-rc3

Backlinking to purescript/purescript#3942

I think this fails because http-methods doesn’t depend on master versions of its dependencies.


ignore onnxruntime-gpu for macOS and for arm64 CPUs

clone of this PR: https://github.com/cubiq/PuLID_ComfyUI/pull/72
I hope this little pull request will be useful too.

@cubiq can we merge this as well?
2 days are pass and there were no new reports in Pulid repo about failing installs.
thank you


Query changes on Sql Lab do not reflect on Slices that use them

How to reproduce the bug
If a query on SQLlab  is used to create a slice by:
Sql Lab -> Write Query -> Save -> Explore -> Create Slice -> Saved
Lets say the query used is: SELECT name from table;
And the slice is called Details_Chart
Work great to this point.
Now, if we want to modify the query to SELECT name, address from table;
Sql Lab -> Saved Queries -> Edit Query -> Update the Query -> Save (UPDATE)
If we go slice to Details_Chart it will only show name as available column and not name & address
However, If we access Edit Dataset via the Explore Slice editor, it shows the updated SQL query ie  SELECT name, address from table;
Expected results
It would be great if the update to the Sql Lab query pushed down to entities that use it. Much like how the Legacy editor already does.
Alternatives
For now I use the following methods to solve this:
#1

Go to Sources -> Tables -> Remove the SQL LAB VIEW Filter -> Hit refresh
Navigate to the table you want to edit and click on edit record
Update the SQL query under Detail -> SQL
Then click on the Column tab and add new column
Hit save.
Navigate to the chart and you will find the column available.

#2

Go to SQL Lab -> Saved queries -> Navigate to the query you want to edit -> Click on Pop tab link
Make modifications to the SQL query -> Run the query
Hit the Save button, then select the update option with an appropriate name.
The click on Explore.
Rebuild the chart

Environment
(please complete the following information):

superset version: 0.37 to 0.38
python version: 3.7

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[Y] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[Y] I have reproduced the issue with at least the latest released version of superset.
[Y] I have checked the issue tracker for the same issue and I haven't found one similar.

Closing this since the issue is on an unsupported version of Superset. If this issue still exists in supported versions (1.5.x, 2.0.x) then please let me know! We can either re-open this issue or open a new one. Thank you for the report nonetheless!


Http4sImplicits breaks with http4s version >= v0.21.0-RC1

as of 1/21 http4s v0.21.0-RC1 Media supertype of Message was added and the EntityDecoder trait was updated (commit here).  This broke Http4sImplicits generation here.
I am happy to create a pull request if I get some guidance on how to generate different versions of Http4sImplicits based on the version of http4s the consumer is using.

Hey @aherrmann13, sorry for the delay in response.
Keeping source compatibility with http4s has been a challenge, if you take a look at COMPATIBILITY#http4s you can see a basic compatibility matrix.
I think the most straightforward thing to do would be to make the change in the file you linked, run runtimeScalaSuite to confirm that it can compile and that the tests pass.
If you notice that none of the RoundTrip tests fail, it'd be particularly kind of you to write a test that exercises the new functionality.

Hi we`ve encountered this issue as well. What is current status of this issue please? Guardrail is now incompatible with http4s 0.21.0+. I am honestly little bit confused as in compatibility section is clearly stated that 0.21.0 is compatible. Thanks

It looks like 0.21.x included multiple changes, #448 must have been adding support for a milestone or RC release. I think #544 finishes the job, and at that point 0.21.x would be fully supported. Sorry for the confusion, let me push that through.

sorry for the late response, working for me now - thanks!

Great news! Thanks for confirming!


current role

Hello, I want to ask if there a ready method to get direct to the current role?

Hi, I'm not pretty sure what you mean, but if you want to get the role of the current user you can get it from your user's instance like this...
user = User.first
user.role => the user's role.

# If you are using some authentication gem like devise you can get it like this in your controllers.
current_user.role

I hope it can help you.

thanks, I already did as you said.
I just thought may be after installing voltex, I can have the current role by just typing a keyword or method name like current_role!
never mind :)


Invalid commits when folder/file names passed

When I tried to get all commits made to the folder app1, it is returning invalid commits
const options = {
	'name-only': true,
	'_': 'app1'
};

I think there is an issue in the parser.
{ commit: { long: '\napp1/index.js\n', short: undefined },
09:30:37     tree: { long: undefined, short: undefined },
09:30:37     author: { name: undefined, email: undefined, date: Invalid Date },
09:30:37     committer: { name: undefined, email: undefined, date: Invalid Date },
09:30:37     subject: undefined,
09:30:37     body: undefined } ]

Hi, there's a test for this, please try using an argument separator (--):
https://github.com/bendrucker/git-log-parser/blob/master/test/index.js#L10-L12
If that doesn't work I'm happy to take a closer look.


EGL support for GLMakie on Wayland systems?

GLFW already supports EGL.
julia> scatter(1:4) Error showing value of type Scene: ERROR: GLFWError (VERSION_UNAVAILABLE): GLX: Failed to create context: GLXBadFBConfig
nvidia proprietary drivers don't seem to work with glx on Wayland... although it works with EGL...

This is a bug with the driver or GLFW. Try searching in the issues here or in GLFW to find it.


Proposal: Add CommandBarStyle and FlyoutPresenterStyle to CommandBarFlyout

Proposal: Add CommandBarStyle to CommandBarFlyout
Summary
Fixes #4301 in a robust way and allow per-control customization
Rationale

Currently, to customize any properties of command bar flyout that are not covered by light weight styling, users have to resort to overriding the default style. This is typically done by inserting a style targetting CommandBarFlyoutCommandBar in App.xaml inside XamlControlsResources. While it works, the customization is app-wise and users are not able to customize command bar flyout on a per-control basis. The properties would allow users to customize the CommandBar directly, thereby also allow users to customize the CommandBarOverflowPresenterStyle if necessary.
Fixes #4301 by allowing users to customize the MaxWidth of CommandBar directly, or MaxHeight of CommandBarOverflowPresenter.
There are prior arts. Other flyouts already have such properties. For example, MenuFlyout has MenuFlyoutPresenterStyle, and Flyout has FlyoutPresenterStyle. By adding CommandBarStyle directly, we get both the ability to customize the command bar and the overflow presenter.

Scope

Capability
Priority

Add CommandBarStyle to CommandBarFlyout
Must

Important Notes
While the implementation of the templates and the name of CommandBarFlyout clearly imply a coupling with command bar, some might argue that adding CommandBarStyle introduces unnecessary coupling with CommandBar. If this is a concern, adding CommandBarFlyoutPresenterStyle instead of CommandBarStyle is acceptable, even though assigning the style require writing out the TargetType, which would leak the implementation details anyways. Overall, I don't think this is better than simply exposing CommandBarStyle.

@ranjeshj fyi


[Suggestion] Align text option for LinkButton

Godot version:
3.0
IIRC, Button and Label classes have this feature too. Would be awesome for the LinkButton as well

Can put it as a child of CenterContainer >_<, my bad.


cuda.cuDeviceGetCount(ct.byref(nGpus) does not update nGpus.values

I ran cuda.cuDeviceGetCount(ct.byref(nGpus), it returns 34, but the nGpus.value remains 0.
>>> nGpus = ct.c_int()
>>> cuda.cuDeviceGetCount(ct.byref(nGpus))
34
>>> nGpus.value
0

Upgrading via pip install -U bitsandbytes should resolve this issue. I will close this issue. Please reopen if upgrading does not resolve the error.