Advanced indexing

This PR adds support for indexing Zarr arrays with Boolean or integer arrays. Resolves #78.

Some examples of usage, and some performance benchmarking, are in this notebook.
Regarding the API, I have added support for orthogonal indexing (a.k.a. outer indexing) via __getitem__ and __setitem__. N.B., if there is more than one bool or int array used in the indexing selection, then this differs from numpy fancy indexing. There are at least two possible options here:
(1) Keep this as-is, i.e., implement orthogonal indexing via __getitem__ and __setitem__. Pros: API and code simplicity. Cons: different behaviour from numpy for some operations.
(2) Keep behaviour of __getitem__ and __setitem__ consistent with numpy fancy indexing by disallowing more than one indexing array. Add a different accessor that supports full orthogonal indexing (e.g., called 'iloc' following pandas naming, or 'oindex' following naming proposed for a new numpy orthogonal indexing accessor). Pros: consistency with numpy; leaves open possibility for implementing more complete fancy indexing support in future (although I don't think I'll ever have the brain-power to do that). Cons: more complex API and code.
Regarding performance, results from some simple benchmarks look quite promising. Performance obviously depends on how many items are being selected, i.e., how dense or sparse the selection is. For relatively dense selections (~50% of items), indexing with a boolean array within a factor of 2 of speed for same operation on a plain numpy array, which seems decent given that zarr has to do the extra work of managing and decompressing chunks. For relatively sparse selections (~0.01% of items) we are about 10 times slower than numpy, but almost all the time is being spent in Array._decode_chunk which is where decompression happens, so I think this proves the overhead from processing the array selection is minimal compared with the time required for decompressing chunks, even when using a very fast compressor (Blosc with LZ4, multithreaded).
I also did a quick performance comparison with h5py, which isn't really fair as h5py was using a slower compressor (gzip level 1), however FWIW, with the sparse boolean array zarr is ~10X faster than h5py, and with the dense boolean array h5py performance is pathological taking longer than 1 minute to complete, so zarr wins big there taking <1 second.
Comments on API and implementation very welcome. cc @shoyer, @mrocklin, @jakirkham, @francescalted.

Very cool to see this!

Keep behaviour of __getitem__ and __setitem__ consistent with numpy fancy indexing by disallowing more than one indexing array.

Watch out: NumPy considers even scalars to be indexing arrays:
In [15]: x = np.zeros((1, 2, 3))

In [16]: x[0, :, [0, 1, 2]].shape
Out[16]: (3, 2)

(This is my favorite NumPy indexing edge case.)
I don't really have a opinion here on (1) vs (2), as long as it is clearly documented and you don't try to do both outer/orthogonal and vectorized/broadcasting indexing in the same API. NetCDF4-Python only does outer indexing and that works fine for it. I would be just as happy to use a special .vindex[] indexer for vectorized indexing if anymore ever bothers to add it.

leaves open possibility for implementing more complete fancy indexing support in future (although I don't think I'll ever have the brain-power to do that)

This might actually be easier than you think. @mrocklin wrote a version of this for dask that might be a good reference point:
https://github.com/dask/dask/blob/7113a3c9bf335f2fe58989760af7b671d940e92f/dask/array/core.py#L3024

xref https://github.com/dask/dask/pull/439, https://github.com/dask/dask/pull/1776, https://github.com/dask/dask/pull/2573, https://github.com/numpy/numpy/pull/6256, https://github.com/h5py/h5py/issues/652

Good work!  Maybe I'm looking at the benchmarks incorrectly, but I only see zarr being 4x faster (not 10x) than h5py:
%time zc[ix_sparse_bool]
CPU times: user 472 ms, sys: 88 ms, total: 560 ms
Wall time: 262 ms

vs
%time hc[ix_sparse_bool]
CPU times: user 1.1 s, sys: 0 ns, total: 1.1 s
Wall time: 1.1 s

For what is worth, I think zarr might benefit with the forthcoming introduction of dictionaries support for zstd inside Blosc2.  The nice thing about dictionaries is that you can make your data blocks ridiculously small (apparently up to 1 KB), but still get good compression ratios and more importantly, very fast decompression speed.  This should reduce the latency quite a bit when you have to decompress a whole block for getting just 1 (or a few) values out of it.

On Tuesday, October 31, 2017, Stephan Hoyer<EMAIL_ADDRESS>wrote:

Very cool to see this!
Keep behaviour of getitem and setitem consistent with numpy fancy
indexing by disallowing more than one indexing array.
Watch out: NumPy considers even scalars to be indexing arrays:
In [15]: x = np.zeros((1, 2, 3))
In [16]: x[0, :, [0, 1, 2]].shape
Out[16]: (3, 2)
(This is my favorite NumPy indexing edge case.)

Ouch. OK, maybe option (2) should be: allow only slices and/or ints in
getitem/setitem; implement orthogonal indexing via .oindex[] in
this PR; implement point selection via .vindex[] in future PR.

I don't really have a opinion here on (1) vs (2), as long as it is clearly
documented and you don't try to do both outer/orthogonal and
vectorized/broadcasting indexing in the same API. NetCDF4-Python only does
outer indexing and that works fine for it. I would be just as happy to use
a special .vindex[] indexer for vectorized indexing if anymore ever
bothers to add it.

leaves open possibility for implementing more complete fancy indexing

support in future (although I don't think I'll ever have the brain-power to
do that)
This might actually be easier than you think. @mrocklin
https://github.com/mrocklin wrote a version of this for dask that might
be a good reference point:
https://github.com/dask/dask/blob/7113a3c9bf335f2fe58989760af7b6
71d940e92f/dask/array/core.py#L3024

Well, one of @mrocklin comments on the original PR was basically, "I pity
the next person has to look at this" :-) But I see you did some further
work on it in another PR so maybe @mrocklin was being pessimistic :-)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-340629170, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QiSyp4AOV0EUeVBXtUiDzEGpEVKSks5sxnHKgaJpZM4QMBwG
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
Big Data Institute Building
Old Road Campus
Roosevelt Drive
Oxford
OX3 7LF
United Kingdom
Phone: +44 (0)1865 743596
Email<EMAIL_ADDRESS>Web: http://a http://purl.org/net/alimanlimanfoo.github.io/
Twitter: https://twitter.com/alimanfoo

Good work! Maybe I'm looking at the benchmarks incorrectly, but I only see zarr being 4x faster (not 10x) than h5py

Sorry, yes, my mistake, 4X faster.

For what is worth, I think zarr might benefit with the forthcoming introduction of dictionaries support for zstd inside Blosc2. The nice thing about dictionaries is that you can make your data blocks ridiculously small (apparently up to 1 KB), but still get good compression ratios and more importantly, very fast decompression speed. This should reduce the latency quite a bit when you have to decompress a whole block for getting just 1 (or a few) values out of it.

Very interesting, thanks!

@mrocklin regarding API, how would/should this play with da.from_array(fancy=True/False)? If fancy=True, what does dask assume about the API?

I believe that setting fancy=True means that the underlying data store supports fancy indexing (which my understanding is that now zarr does) and so Dask.array should feel comfortable sending complex slicing arguments down to the underlying store.  I think that we had to implement this because h5py didn't support some things that numpy did.
It's has been a while since then though, so I may be misremembering things.  The relevant docstring is here
    fancy : bool, optional
        If ``x`` doesn't support fancy indexing (e.g. indexing with lists or
        arrays) then set to False. Default is True.

It sounds like you do support these things, so presumably people loading dask arrays from zarr arrays should set fancy=True, which is the default.

Thanks @mrocklin. Currently in this PR zarr does not implement fancy indexing same as numpy, but rather implements orthogonal indexing. So I was concerned dask may get unexpected results if fancy=True assumes numpy fancy indexing, depending on what indexes are passed through.
Actually just looking at @shoyer favourite edge case, it looks like dask __getitem__ behaviour does something different from numpy fancy indexing anyway, before even worrying about zarr interaction. E.g.:
In [17]: x = np.arange(6).reshape(1, 2, 3)

In [18]: d = da.from_array(x, chunks=(1, 2, 3))

In [19]: x[0, :, [0, 1, 2]]
Out[19]:
array([[0, 3],
       [1, 4],
       [2, 5]])

In [20]: d[0, :, [0, 1, 2]].compute()
Out[20]:
array([[0, 1, 2],
       [3, 4, 5]])

So I guess there are a couple of separate questions:
(a) It looks like dask.array __getitem__ behaviour currently does not match np.ndarray __getitem__ behaviour for some edge cases, should it?
(b) What exactly does fancy=True mean in terms of expected behaviour of __getitem__ on wrapped array? I.e., if dask.from_array is given fancy=True and the wrapped array implements orthogonal indexing via __getitem__, could dask ever pass through a combination of indexes that would produce different results for numpy fancy indexing versus orthogonal indexing (e.g., two 1d integer arrays, or @shoyer edge case)?

cc @benjeffery

I think I prefer option (2): allow only slices and/or ints in __getitem__/__setitem__; implement orthogonal indexing via .oindex[] in this PR; maybe implement point selection via .vindex[] in future PR. Seems like a safer thing to do, less potential for confusion and bugs. When using zarr with dask, can stick with fancy=False for now, and down the line figure out if worth finding a way that dask could make use of .oindex[] and/or .vindex[] if available on the wrapped array.

Just to note, the way we currently disambiguate vectorized/orthogonal indexing internally in xarray is that we use dedicated classes to store each type of indexer:
https://github.com/pydata/xarray/blob/17956ea5de2cf5029992e8f83460fcc878e3d024/xarray/core/indexing.py#L280-L303
This way, indexing can go through the same code paths but still dispatch to appropriate backend specific methods (e.g., dask vs numpy vs netCDF4 vs zarr).

Actually just looking at @shoyer favourite edge case, it looks like dask getitem behaviour does something different from numpy fancy indexing anyway, before even worrying about zarr interaction. E.g.:

Yes, I think that this came up when we were hammering out slicing.  I think that it was intentionally decided to deviate from NumPy's behavior.  I wouldn't be surprised if @shoyer was the one to make this call actually.  My memory here is a bit hazy.

I don't recall agreeing intentionally deviating from numpy for dask. But I
did notice this recently and hadn't gotten around to filing a bug yet. Dask
is at least in good company here: h5py also gets this wrong.
On Wed, Nov 1, 2017 at 6:21 AM Matthew Rocklin<EMAIL_ADDRESS>wrote:

Actually just looking at @shoyer https://github.com/shoyer favourite
edge case, it looks like dask getitem behaviour does something
different from numpy fancy indexing anyway, before even worrying about zarr
interaction. E.g.:
Yes, I think that this came up when we were hammering out slicing. I think
that it was intentionally decided to deviate from NumPy's behavior. I
wouldn't be surprised if @shoyer https://github.com/shoyer was the one
to make this call actually. My memory here is a bit hazy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-341104126, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1lMfX_zsaVNOHKUIHiAHoiZjw9hmks5syHBBgaJpZM4QMBwG
.

OK, my mistake.  I must be mis-remembering things.
On Wed, Nov 1, 2017 at 11:22 AM, Stephan Hoyer<EMAIL_ADDRESS>wrote:

I don't recall agreeing intentionally deviating from numpy for dask. But I
did notice this recently and hadn't gotten around to filing a bug yet. Dask
is at least in good company here: h5py also gets this wrong.
On Wed, Nov 1, 2017 at 6:21 AM Matthew Rocklin<EMAIL_ADDRESS>wrote:

Actually just looking at @shoyer https://github.com/shoyer favourite
edge case, it looks like dask getitem behaviour does something
different from numpy fancy indexing anyway, before even worrying about
zarr
interaction. E.g.:
Yes, I think that this came up when we were hammering out slicing. I
think
that it was intentionally decided to deviate from NumPy's behavior. I
wouldn't be surprised if @shoyer https://github.com/shoyer was the one
to make this call actually. My memory here is a bit hazy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-341104126, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABKS1lMfX_
zsaVNOHKUIHiAHoiZjw9hmks5syHBBgaJpZM4QMBwG>

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-341138751, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszJw4Pm3V44IVtmCAhcgEotjVFn3Gks5syIy2gaJpZM4QMBwG
.

Hm, this is tricky.
For my part, there are three indexing use cases which I think are worth
considering.
The first is orthogonal indexing within any combination of int, slice,
ellipsis, 1d int array or 1d bool array. That's the one I need most often,
and is currently implemented in this PR via getitem.
The second is point selection with any combination of int and 1d int array
but no slice or ellipsis. The indexers can then be broadcast to provide
fully specified coordinate arrays and the output is always 1d. I think this
is essentially what is implemented in dask via .vindex and in xarray via
.isel_points(), and I think I can see how this could be done efficiently in
zarr.
The third is point selection with a single boolean array, where the boolean
indexer has the same shape as the array being indexed (generalization of
x[x > 0]). This maps to point selection via integer arrays by doing
np.nonzero() on the boolean indexer array.
Any other type of point selection/vectorized indexing (e.g., including
slices, int arrays with >1d, mixture of int and bool arrays, ...) I have
never needed and have trouble with understanding, mainly because dimensions
can get moved around in a way that I don't fully understand.
In this PR I think I'll change to limit getitem to basic indexing with
int, slice and ellipsis only. I may also expose this functionality via a
couple of methods, something like get_basic_selection(selection, out=None)
and set_basic_selection(selection, value), where selection is a fully
specified tuple of ints and/or slices (i.e., no ellipsis, all dims are
indexed). This is partly for clarity, but also get_basic_selection() could
accept an "out" param, which would allow the selected data to be loaded
directly into an array given by the user, which could be numpy or another
zarr array.
I think I'll then expose orthogonal indexing via .oindex[] as has been
proposed for numpy. I may also implement this via methods, e.g.,
get_orthogonal_selection(selection, out=None) and
set_orthogonal_selection(selection, value), again for clarity and
flexibility.
I will probably leave point selection for future work. However, I think I
would start by implementing methods separately targeting the second and
third use cases above. E.g., get_point_selection_int(selection, out=None),
where "selection" is fully specified tuple of ints and/or 1d int arrays,
and get_point_selection_bool(selection, out=None) where "selection" is
single bool array with same shape as indexed array. This would at least
simplify implementation and make it a bit clearer what subset of point
selection indexing is being implemented. Both of these could then be used
to provide implementations for a subset of the proposed functionality for
.vindex[] in numpy, if that goes forwards.
On Wed, Nov 1, 2017 at 3:23 PM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

OK, my mistake. I must be mis-remembering things.
On Wed, Nov 1, 2017 at 11:22 AM, Stephan Hoyer<EMAIL_ADDRESS>wrote:

I don't recall agreeing intentionally deviating from numpy for dask. But
I
did notice this recently and hadn't gotten around to filing a bug yet.
Dask
is at least in good company here: h5py also gets this wrong.
On Wed, Nov 1, 2017 at 6:21 AM Matthew Rocklin<EMAIL_ADDRESS>wrote:

Actually just looking at @shoyer https://github.com/shoyer favourite
edge case, it looks like dask getitem behaviour does something
different from numpy fancy indexing anyway, before even worrying about
zarr
interaction. E.g.:
Yes, I think that this came up when we were hammering out slicing. I
think
that it was intentionally decided to deviate from NumPy's behavior. I
wouldn't be surprised if @shoyer https://github.com/shoyer was the
one
to make this call actually. My memory here is a bit hazy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-341104126,
or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABKS1lMfX_
zsaVNOHKUIHiAHoiZjw9hmks5syHBBgaJpZM4QMBwG>

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-341138751, or
mute
the thread
<https://github.com/notifications/unsubscribe-auth/
AASszJw4Pm3V44IVtmCAhcgEotjVFn3Gks5syIy2gaJpZM4QMBwG>

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/pull/172#issuecomment-341139089, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8Qo-QXsC3I2Z-B9BdKSyodMfyK4Rzks5syIztgaJpZM4QMBwG
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
Big Data Institute Building
Old Road Campus
Roosevelt Drive
Oxford
OX3 7LF
United Kingdom
Phone: +44 (0)1865 743596
Email<EMAIL_ADDRESS>Web: http://a http://purl.org/net/alimanlimanfoo.github.io/
Twitter: https://twitter.com/alimanfoo

A better name for "get_point_selection_bool" could be "get_mask_selection", and "get_point_selection_int" could be better named "get_coordinate_selection".

I've pushed some new work on this, here's a synopsis.
Vectorized (inner) indexing
I've added support for vectorized indexing using coordinate arrays (a.k.a. point selection), actually wasn't too hard to do. This functionality is available via get/set_coordinate_selection() method and also for convenience via vindex[].
Vectorized indexing using a Boolean mask array is also supported via get/set_mask_selection() method and vindex[] - this just calls np.nonzero internally to construct coordinate arrays and the rest is done via coordinate selection.
More complicated vectorized indexing scenarios, e.g., mixing coordinate or mask arrays with slices, are currently not supported.
The indexing coordinates do not have to be sorted in any particular order. Zarr shuffles the coordinates so they are grouped by their corresponding chunk, so that each chunk is processed once only.
Orthogonal (outer) indexing
Orthogonal indexing is supported via get/set_orthogonal_selection() and for convenience via .oindex[]. Any mix of int, slice with step >= 1, 1D int array and 1D bool array is supported.
Integer arrays do not need to be sorted. Zarr shuffles the index values so they are grouped by their corresponding chunk, so that each chunk is processed once only.
Slice with step > 1
Slice with step > 1 is now supported in __getitem__ for 1D arrays and in .oindex[] for multi-dimensional arrays. Internally the slice is converted to an int array via np.arange then processed via the orthogonal selection machinery.
Open questions
What functionality should be available via __getitem__?
For 1D arrays there is no ambiguity on how to process advanced selections, i.e., there is no difference between vectorized versus orthogonal indexing. So for convenience, if a Zarr array is 1D, currently __getitem__ supports any of int, slice with step >= 1, 1D int array, 1D bool array.
For multi-dimensional arrays things are more complex, and so currently I restrict __getitem__ to only handle basic selections, i.e., any combination of int and slice with step == 1. To do advanced indexing the user must use one of .oindex[] or .vindex[] or the corresponding selection methods.
If anyone feels this isn't a good way to go, happy to discuss.
Benchmarks
Performance seems reasonable in all cases, can't see any obvious ways to improve. More examples and some benchmarking data are in this notebook.

Thank you @shoyer for the hugely useful feedback. Here's a summary of latest pushes:
Support has been added for coordinate indexing with multi-dimensional arrays.
For arrays with a structured dtype, all get/set_selection...() methods now support a fields argument to allow selecting data for specific fields (xref #112). I've also tentatively implemented h5py-style support for fields within __getitem__, although that deviates from the numpy API so not 100% sure it's a good idea.
I've also simplified __getitem__ so it only supports basic selections (mix of int and contiguous slice). I think that's a decent position for now, it would be possible to add support via __getitem__ for more advanced scenarios but it's not straightforward to work through all the different options and how they should be dispatched to appropriate selection methods.
Examples and benchmarks notebook has been updated for the above changes.

Just to mention I've reworked the implementation of slices with step > 1, these are now supported via __getitem__ as well as orthogonal selection methods, and are approx 10X faster and use less memory (no longer implemented via np.arange). The downside is that slices with step < 0 are not supported, but I think that's a reasonable compromise. Updated benchmarks here.
Test coverage is also back up, and I think I'm done with the main implementation work, so will work on docs and improving error messages before merging.

Nice job.  After having a look at your benchmarks, I see that _chunk_getitem is usually the most consuming function (cumtime wise) in your profiles, so I am wondering if that could be improved somehow.  I see that your chunksizes are typically between 256 KB to 1 MB, but the benchmark page does not show the blocksize per every chunk, which is the important parameter when you try to get a handful of values out of a chunk (only a block or a few need to be decompressed).  You could get such blocksize parameter by using the blosc_get_blocksize() call, and you can explicitly set it using  blosc_set_blocksize() (if you don't call it, an automatic blocksize is used).  You may want to add support to these functions in zarr and try a smaller blocksize to see how it would affect your current figures.
Also, I see that np.argsort() shows up sometimes the first in time usage.  I am wondering if you could make use of a handy keysort that I did many years ago. keysort() takes two arrays as arguments, sorting in-place the first one and also the second, but following the order of the first, in one shot.  This requires less temporaries and hence it is quite more efficient than an np.argsort followed by an indexing operation.  It has been in production in PyTables for years, so it should be safe enough.

Nice job. After having a look at your benchmarks, I see that
_chunk_getitem is usually the most consuming function (cumtime wise) in
your profiles, so I am wondering if that could be improved somehow. I see
that your chunksizes are typically between 256 KB to 1 MB, but the
benchmark page does not show the blocksize per every chunk, which is the
important parameter when you try to get a handful of values out of a chunk
(only a block or a few need to be decompressed). You could get such
blocksize parameter by using the blosc_get_blocksize() call, and you can
explicitly set it using blosc_set_blocksize() (if you don't call it, an
automatic blocksize is used). You may want to add support to these
functions in zarr and try a smaller blocksize to see how it would affect
your current figures.

Thanks Francesc. In fact the blocksize parameter is exposed for the
numcodecs.Blosc compressor, so this could be tuned. But the main thing for
the indexing work is to know that the compressor is the limiting factor.
The Blosc compressor is already extremely fast, and although could probably
be tuned even further, it is already very useful to know that my indexing
implementation is not getting in the way performance-wise, at least for
most of the indexing operations.
(Btw when I was running the benchmarks yesterday I could actually hear my
computer audibly fizzing, only Blosc can make it do that :-)

Also, I see that np.argsort() shows up sometimes the first in time usage.
I am wondering if you could make use of a handy keysort
https://github.com/PyTables/PyTables/blob/6782047b9223897fd59ff4967d71d7fdfb474f16/tables/indexesextension.pyx#L147
that I did many years ago. keysort() takes two arrays as arguments,
sorting in-place the first one and also the second, but following the order
of the first, in one shot. This requires less temporaries and hence it is
quite more efficient than an np.argsort followed by an indexing
operation. It has been in production in PyTables for years, so it should be
safe enough.

Thank you, actually I did do some trawling the internet to see if there was
any way to accelerate the argsort and saw a mention that you had done
something like this in pytables. I think it would be great to explore this,
I noticed that sorting an array in place takes less than half the time of
an argsort followed by indexing when just using numpy. I'm a bit hesitant
to do include cython code in zarr as currently zarr is pure Python, and
there are some benefits to not having any cython code to build. What would
be cool is if the pytables keysort implementation was available as a
standalone package, then zarr could depend on it. But I know you're
super-busy so don't want to ask you to do any more :-)

(Btw when I was running the benchmarks yesterday I could actually hear my
computer audibly fizzing, only Blosc can make it do that :-)

Ha ha, this probably has to do with the SIMD support in blosc shuffle/bitshuffle, which makes CPUs consume quite more energy.  Add multithreading to the equation and yeah, I can imagine you could fry something on top of your CPU while you are at it :)

OK, I think I am done here. There is a new tutorial section on advanced indexing. Error messages have been improved. I'll let the dust settle for a few days.

Just to follow up on @FrancescAlted suggestion to use keysort, it looks like it could accelerate indexing with unsorted indices by up to a factor of 2, e.g.:
In [1]: import numpy as np

In [2]: from tables.indexesextension import keysort

In [3]: n = 100000000

In [4]: ix = np.random.choice(n, size=n//5, replace=True)

In [5]: %time s = np.argsort(ix); ix_sorted = ix[s]
CPU times: user 4.18 s, sys: 36 ms, total: 4.22 s
Wall time: 4.22 s

In [6]: %time s = np.arange(ix.shape[0]); keysort(ix, s)
CPU times: user 2 s, sys: 12 ms, total: 2.01 s
Wall time: 2.01 s

I think I will leave this for a future version as pytables is too heavy a dependency, and I'd like to keep zarr pure Python for the moment at least, but it would be very nice to use keysort somehow in the future.

After rebasing, I hit this unicode weirdness on Windows:
>>> import numpy as np
>>> v = np.array('xxx', dtype='U3')[()]
>>> v
'xxx'
>>> a = np.empty(10, dtype='U3')
>>> a[:] = v
>>> a[0] == v
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-32-le' codec can't decode bytes in position 0-3: code point not in range(0x110000)

I think this is related to default locale of cp1252 on my Windows VM, but I don't really understand what's happening. In any case I've pushed a simple workaround.

Alright, merging.


Fix code scanning alert - CVE-2022-27664

Tracking issue for:

[ ] https://github.com/arlonproj/arlon/security/code-scanning/14

Upgrading golang used in the Docker image for controller from 1.19 to 1.19.1 will fix this.

This has been fixed in v0.9 branch and main. A new docker image has been pushed for tag v0.9.9


Constant string in GetAttribute causing a failure in Unit Test

Non-Constants
[Get("/ajax/origin")]
Task<CitiesRoot> GetOriginCities(); // Test passed

Constants
[Get(Constants.Origin)] // `Constants.Origin` == "/ajax/origin"
Task<CitiesRoot> GetOriginCities(); // Test failed

Perhaps in other attributes too, I haven't try with those attributes.

Constants are not supported for values in Refit's attributes. See #263.


Should we add example as document solution?

It would be fabulous  if go-zero add an example folder as document solution  like what tensorflow did.
Here's reason why:
1.more convinient to access documents, keep docs updated, tested more frequently
2. more convinient to access code rather than take much time trying to understand document, especially for developer using english.
3. much more convinent for user making contribution to go-zero
tensorflow example ref,: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples

1.more convinient to access documents, keep docs updated, tested more frequently
in tensorflow, code example be the most frequently visited code.
that counts

https://github.com/zeromicro/zero-examples


python312Packages.ttn-client: 1.0.0 -> 1.1.0

Diff: https://github.com/angelnu/thethingsnetwork_python_client/compare/refs/tags/v1.0.0...v1.1.0
Changelog: https://github.com/angelnu/thethingsnetwork_python_client/releases/tag/v1.1.0
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 325197 run on x86_64-linux 1

5 packages built:

home-assistant-component-tests.thethingsnetwork
python311Packages.ttn-client
python311Packages.ttn-client.dist
python312Packages.ttn-client
python312Packages.ttn-client.dist


Misplaced Commas

Use this prototype to find images that you can use and remix, across several open archives. Give us your feedback, and help us design a front door to the commons.
"Use this prototype to find images that you can use and remix across several open archives. Give us your feedback to help us design a front door to the commons."

This text was changed.


K8SPS-314 - Fix chaos mesh cleanup

CHANGE DESCRIPTION
Problem:
Chaos mesh objects have finalizers and if you try to delete crd with these objects not yet deleted it might fail and cause next test to fail to re-initialize chaos mesh.
Solution:
Empty finalizer in chaos mesh objects before deleting the object and crd.
CHECKLIST
Jira

[ ] Is the Jira ticket created and referenced properly?
[ ] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
[ ] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

[ ] Is an E2E test/test case added for the new feature/change?
[ ] Are unit tests added where appropriate?

Config/Logging/Testability

[ ] Are all needed new/changed options added to default YAML files?
[ ] Did we add proper logging messages for operator actions?
[ ] Did we ensure compatibility with the previous version or cluster upgrade process?
[ ] Does the change support oldest and newest supported PS version?
[ ] Does the change support oldest and newest supported Kubernetes version?

Test name
Status

async-ignore-annotations
passed

auto-config
passed

config
passed

config-router
passed

demand-backup
failure

gr-demand-backup
passed

gr-demand-backup-haproxy
passed

gr-haproxy
passed

gr-ignore-annotations
passed

gr-init-deploy
passed

gr-one-pod
passed

gr-recreate
passed

gr-scaling
passed

gr-self-healing
failure

gr-tls-cert-manager
passed

gr-users
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

recreate
passed

scaling
passed

service-per-pod
passed

sidecars
passed

smart-update
passed

tls-cert-manager
passed

users
passed

version-service
passed

We run 30 out of 30

commit: https://github.com/percona/percona-server-mysql-operator/pull/564/commits/8186b039b25be5e557f9e3eb2d12af896c7e1898
image: perconalab/percona-server-mysql-operator:PR-564-8186b03

Test name
Status

async-ignore-annotations
passed

auto-config
passed

config
passed

config-router
passed

demand-backup
failure

gr-demand-backup
passed

gr-demand-backup-haproxy
passed

gr-haproxy
passed

gr-ignore-annotations
passed

gr-init-deploy
passed

gr-one-pod
passed

gr-recreate
passed

gr-scaling
passed

gr-self-healing
failure

gr-tls-cert-manager
passed

gr-users
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

recreate
failure

scaling
passed

service-per-pod
passed

sidecars
passed

smart-update
passed

tls-cert-manager
passed

users
passed

version-service
passed

We run 30 out of 30

commit: https://github.com/percona/percona-server-mysql-operator/pull/564/commits/d4153fb1391ff53376b03f08101b043729f42c21
image: perconalab/percona-server-mysql-operator:PR-564-d4153fb

Test name
Status

async-ignore-annotations
passed

auto-config
passed

config
passed

config-router
passed

demand-backup
failure

gr-demand-backup
passed

gr-demand-backup-haproxy
passed

gr-haproxy
passed

gr-ignore-annotations
passed

gr-init-deploy
passed

gr-one-pod
passed

gr-recreate
passed

gr-scaling
passed

gr-self-healing
passed

gr-tls-cert-manager
passed

gr-users
passed

haproxy
passed

init-deploy
passed

limits
passed

monitoring
passed

one-pod
passed

operator-self-healing
passed

recreate
passed

scaling
passed

service-per-pod
passed

sidecars
passed

smart-update
passed

tls-cert-manager
passed

users
passed

version-service
passed

We run 30 out of 30

commit: https://github.com/percona/percona-server-mysql-operator/pull/564/commits/d4153fb1391ff53376b03f08101b043729f42c21
image: perconalab/percona-server-mysql-operator:PR-564-d4153fb


CSV file not being uploaded in the Suppliers tab

Is your feature request related to a problem? Please describe.
The CSV file does not get uploaded after choosing the file
Describe the solution you'd like
The CSV file needs to get uploaded after being selected
Additional context

This issue was fixed when the new CSV file was used.


✨ [experimental] Probe support for fuzzing check

Probe support for fuzzing check

Addressed the mahority of comments. (I also made the probes for Update-Dependency-Tool public)

My main concern is the same as the other PR, around how this works when RawResults isn't set.

friendly ping

Regarding the multiple warns for something we only want one of: e.g. one dependency update tool, one fuzzer: Should we make any changes to LogFindings that supports something like?:
if acceptAny:
  if any pass:
    log passing tools
  else:
    log failures

We can start with that, works for me. The alternative is to let each checks/evaluation/*.go create the logs. This way we each check can encode its own logic. We can still provide a set of functions for common "policies", like the acceptAny.
Wdut?

Regarding the multiple warns for something we only want one of: e.g. one dependency update tool, one fuzzer: Should we make any changes to LogFindings that supports something like?:
if acceptAny:
  if any pass:
    log passing tools
  else:
    log failures

We can start with that, works for me. The alternative is to let each checks/evaluation/*.go create the logs. This way we each check can encode its own logic. We can still provide a set of functions for common "policies", like the acceptAny.
Wdut?

Seems fine to me.

Regarding the multiple warns for something we only want one of: e.g. one dependency update tool, one fuzzer: Should we make any changes to LogFindings that supports something like?:
if acceptAny:
  if any pass:
    log passing tools
  else:
    log failures

We can start with that, works for me. The alternative is to let each checks/evaluation/*.go create the logs. This way we each check can encode its own logic. We can still provide a set of functions for common "policies", like the acceptAny.
Wdut?

Leaving the decision to the evaluation package is fine to me. Essentially, each check can filter the findings in the way it cares about, and then pass to the existing LogFindings.

Great. I'll clean this up for all checks at once in the next PR, tracked in https://github.com/ossf/scorecard/issues/3406


Error messages = Invalid DTS, Invalid timestamps

Hello! This project looks awesome but unfortunately isn't working for me on windows or linux for a video i'm trying to download.
The microsoft streams video gives the error message:
Error code 0x20400003
and then the terminal just keeps giving off errors like this:
[mp4 @ 0000017ceca76780] Invalid DTS: 885060 PTS: 881820 in output stream 0:0, replacing by guess [mpegts @ 0000017cec6e7000] Invalid timestamps stream=0, pts=7367850, dts=7371090, size=6535 [mpegts @ 0000017cec6e7000] Invalid timestamps stream=0, pts=7371090, dts=7373790, size=4845 [hls @ 0000017cec59c880] Invalid timestamps stream=0, pts=7367850, dts=7371090, size=6535 [mp4 @ 0000017ceca76780] Invalid DTS: 891090 PTS: 887850 in output stream 0:0, replacing by guess [hls @ 0000017cec59c880] Invalid timestamps stream=0, pts=7371090, dts=7373790, size=4845 [mp4 @ 0000017ceca76780] Invalid DTS: 893790 PTS: 891090 in output stream 0:0, replacing by guess

Is that a recent ffmpeg version you are using?

Hi, thank you for your quick reply! Yes it is, I installed all the prereqs yesteday when I attempted to download the video.
One other thing is, when the chromium window opens up the video doesn't play (despite working normally from chromium any other time) and instead gives an error message saying:
"There’s a problem with the browser.
Please try a different browser to play this video."
with the Error code 0x20400003
I don't know if this is normal?

It seems to be related to the chromium window that it opens up. The chromium installed on my machine seems to play the videos just fine but the chromium window that destreamer opens seems to be a different version that wont play the video (even in a new tab)

Update: despite the error messages constantly appearing it still seems to download (after a couple of hours) but doesn't download the sound with it

My ffmpeg skill and knowledge is pretty basic, you would probably have more success addressing this via a ffmpeg specific forum or StackOverflow. Nothing specific to the project itself (destreamer) since all we do here is pass a key URL with a cookie down to Youtube-dl which then calls into ffmpeg.

Closing for now. Feel free to reopen if this turns out to be a broader issue not isolated to one or two videos.

Hey just wanted to say the new version of destreamer has fixed my issue and it now downloads with sound! Thank you!


Update parser value set types

See https://github.com/p4lang/p4-spec/issues/868

This seems like a reasonable generalization to me.  I suspect that taking advantage of this generality in the language might require similar generalizations in the P4Runtime API and/or TDI, but I don't see a problem if the language spec "run ahead" of those other things.

@jafingerhut and @jnfoster will investigate and propose a list of types allowed, consistent with extant implementations, P4Runtime, and as general as possible.


RequestParam Array as parameter in OpenAPI 3.0

I'm using the latest release of
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-boot-starter</artifactId>
            <version>3.0.0</version>
        </dependency>

What kind of issue is this?
Bug Report
Repository: https://github.com/ilueckel/springfox-bug-repo
Description
Using an array-like RequestParam like
  @GetMapping("test.json")
  public void test(@ApiParam(allowMultiple = true) @RequestParam Collection<Long> ids) {  }

fails to represent the array type when using DocumentationType.OAS_30. When you use DocumentationType.SWAGGER2 instead, the parameter is mapped as array.
The Repository linked above creates to Dockets that show this behavior.
Screenshots

Array Parameter with OAS_30

Array Parameter with SWAGGER_2

I also encountered this problem

Me too

Me too

Same issue here. For DocumentationType.SWAGGER_2 works fine, but for DocumentationType.OAS_30 it does not.

Same issue here. For DocumentationType.SWAGGER_2 works fine, but for DocumentationType.OAS_30 it does not.

Same here.

Same here. And if you set @ApiParam, the documentation of this method is not generated for me.

Same here.

+1

To make this happen, put the request params inside an object as below :
  @GetMapping(value = "/filter")
   public ResponseEntity filterValues(RequestFilter requestFilter) {

public class RequestFilter {
    private List<String> filters;
    private List<Long> ids;
}

The rendered swagger operation is :

+1


Ae file commands

review @thieman

Going to deal with your comments later.

Finished first round of comments. I think the design needs to be reworked. Also could you split out the refactoring of existing functionality into a separate PR, merge that, then base this on that so this PR is easier to reason about?


Sitemap links to localized urls

This is a starting point for sitemaps with links to localized pages, like describe in issue #34 .
We have it in production for some weeks, mainly for sites with static pages from Rainlabs Pages plugin.
I've just tested it with some CMS pages and it seems to work after some fixes and produces the following URL items:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xhtml="http://www.w3.org/1999/xhtml" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
    <url>
        <loc>https://example.com/de/impressum</loc>
        <lastmod>2022-09-20T14:16:10+00:00</lastmod>
        <changefreq>always</changefreq>
        <priority>0.1</priority>
        <xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/impressum" />
        <xhtml:link rel="alternate" hreflang="fr" href="https://example.com/fr/impressum" />
    </url>
    <url>
        <loc>https://example.com/de</loc>
        <lastmod>2022-10-14T11:28:16+00:00</lastmod>
        <changefreq>always</changefreq>
        <priority>1.0</priority>
        <xhtml:link rel="alternate" hreflang="de" href="https://example.com/de" />
        <xhtml:link rel="alternate" hreflang="fr" href="https://example.com/fr" />
    </url>
    <url>
        <loc>https://example.com/de/informationen</loc>
        <lastmod>2022-10-14T09:04:03+00:00</lastmod>
        <changefreq>always</changefreq>
        <priority>0.9</priority>
        <xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/informationen" />
        <xhtml:link rel="alternate" hreflang="fr" href="https://example.com/fr/informations-generales" />
    </url>
    <url>
        <loc>https://example.com/de/programm</loc>
        <lastmod>2022-10-11T06:36:08+00:00</lastmod>
        <changefreq>always</changefreq>
        <priority>0.9</priority>
        <xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/programm" />
        <xhtml:link rel="alternate" hreflang="fr" href="https://example.com/fr/programme" />
    </url>
</urlset>

Some caveats:

I did not test it with any kind of models. If anyone is willing to provide some sample data, I'll gladly take a stab at it.
I've mainly used from sites with static pages, not many CMS pages where included
I only used it with OctoberCMS V2 so far. I've tested the plugin on a V3 docker version, but not extensivly. It seems to work and produces the expected results, but it certainly needs some more testing.

I hope you'll find it useful as a starting point and I'm happy to work further on it if anyone has any feedback or input.

Hi @marcogrueter,
Thanks for your contribution. We're close to release version 5 of SEO Storm which will support multilangual sitemaps. It'll use sitemap index to provide sitemaps for different locales and your pages will have to have hreflangs defined in their meta tags.
Hope you'll like it but if it won't help I'm happy to talk through your ideas so let me know.
Cheers,
Tomasz


Package config works on dev, but failed when build

Hi, I used SystemJS for module loading and SystemJS Builder for bundling, everything works well when in development, but when bundling, I must add these lines to make it work.
I expect https://github.com/antonybudianto/angular2-starter/blob/master/src/systemjs.conf.js#L30 should already do the job since it alone work when module loading.
Error I get if I didn't add those lines
error { Error on fetch for src/tmp/app/app.component at file:///Users/antony/Documents/Code/Web/angular2-starter/src/tmp/app/app.component
	Loading src/tmp/app/main.js
	Error: ENOENT: no such file or directory, open '/Users/antony/Documents/Code/Web/angular2-starter/src/tmp/app/app.component'

Maybe it's path related problem. or is it expected to have two path mapping like that? (one for normal loading, one for building), if I change src/app to app, then it works correctly without those additional lines.
Any help will be appreciated.

Finally get it working by tricking routes on browsersync, now the systemjs.config.js look like
    var map = {
        'app': 'src/tmp/app',
        'test': 'src/tmp/test',
        'rxjs': 'n:rxjs',
        '@angular': 'n:@angular',
        'lodash': 'n:lodash'
    };

my app structure:
src
__app (typescript files)
__tmp (compiled typescript files)
____app (same structure again with app)
__index.html
__systemjs.conf.js

is it a correct config for the structure?

You can use paths configuration (instead of map) to alter the bundle naming. Paths basically act like alternatives to the baseURL. This is something that will be documented better in future.


Clicking a kata link in the katas navigation does not load kata

Clicking one of the kata links in the footer navigation updates the location hash in the browser address bar but doesn't actually load the kata. Is this a feature? Shouldn't the app listen to the hashchange event?
On a side note: Could you elaborate on why the kata gets removed from the location.hash on startup? See startup.js line 16

+1 for this one.

oh man ... this is actually so kaputt (the reasoning why hash etc.) :) I have an update which will use standard queries, see here https://github.com/tddbin/tddbin-frontend/tree/use-query-param-refactoring

reviews to https://github.com/tddbin/tddbin-frontend/pull/16 welcome

fixed by 1bc0b92a95342839511e42bdeeaf0631561a7c70


[10.x] Add prependReportable and prependRenderable methods to Handler

The prependReportable method allows us to add reportable exceptions to the beginning of the reportable exceptions list. This can be useful when we want certain exceptions to be reported before others.
The prependRenderable method allows us to add renderable exceptions to the beginning of the renderable exceptions list. This can be useful when we want certain exceptions to be rendered before others.
These methods enhance the flexibility of exception handling in our application, allowing us to prioritize how exceptions are reported and rendered. For example, in the service provider, when we want to make sure that it is given the highest priority:
$this->app->make(ExceptionHandler::class)->prependRenderable(function (\Exception $e) {
});

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.


1-click upgrade prestashop v1.7

Subject: Problem with 1-Click Update - fatal error admin
http:///index.php?tab=AdminSelfUpgrade&token=
Fatal error: Call to undefined method AdminSelfUpgrade::displayErrors() in /home/comptoirr/www/admin7416/functions.php on line 278
Hello, After installing the module "1-Click Upgrade", as soon as I click on the button "Configure", the page in link is in failure (see above). For information, the PHP version was in 5.4, it was changed in 5.6, I do not know if that's what made the bug? Thank you for giving me the procedure to follow in order to finish the update of the admin prestashop in V 1.7.
Thanks !

Hi there,
You issue has already been fixed with the version 4.8.0 of the module, released today. Please see https://github.com/PrestaShop/autoupgrade/issues/294 where the issue has been reported for the first time.

Thank you for telling me how to solve the problem. I can not find my answer

lign 278 : 		echo $adminObj->displayErrors();

Upgrade your module to v4.8.0.

Bonjour,
C¹est celle que j¹avais téléchargé déjà, d¹où vient donc le problème ?
Merci
De :  Thomas N<EMAIL_ADDRESS>Répondre à :  PrestaShop/autoupgrade
<EMAIL_ADDRESS>Date :  lundi 13 mai 2019 à 09:34
À :  PrestaShop/autoupgrade<EMAIL_ADDRESS>Cc :  Laetitia DEVASSINE<EMAIL_ADDRESS>Author
<EMAIL_ADDRESS>Objet :  Re: [PrestaShop/autoupgrade] 1-click upgrade prestashop v1.7 (#313)
Upgrade your module to v4.8.0.

You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<https://github.com/PrestaShop/autoupgrade/issues/313#issuecomment-491709067

, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMBHMHIMJA2JE7IG637HG7TPV
EKYVANCNFSM4HMDY3KQ> .
[ { "@context": "http://schema.org", "@type": "EmailMessage",
"potentialAction": { "@type": "ViewAction", "target":
"https://github.com/PrestaShop/autoupgrade/issues/313#issuecomment-491709067
", "url":
"https://github.com/PrestaShop/autoupgrade/issues/313#issuecomment-491709067
", "name": "View Issue" }, "description": "View this Issue on GitHub",
"publisher": { "@type": "Organization", "name": "GitHub", "url":
"https://github.com" } } ]


Godot not detecting GPU

Installation info

Godot: Godot Engine v3.2.alpha.custom_build.a1fcac640
Install type: via AUR, using yay
Distro and version: Arch Linux (kernel 4.19.73-1-lts)
Desktop Environment: i3-gaps (4.17.1-1)
GPU: GeForce GTX 1080 (driver: 435.21)

Issue description
Similar to #25451, Godot refused to utilise my GPU. I've tried playing around with other GPU intensive work loads to identify if the issue was from the drivers, however, I've not been able to reproduce the same issue outside of Godot.
Up to now I've build the basic package of Godot from yay. To further test the bug I've attempted to run other versions of the engine:

downloading directly godotengine.org
testing godot-git package in the AUR

Additional info
Output from godot --verbose:
Godot Engine v3.2.alpha.custom_build.a1fcac640 - https://godotengine.org
XInput: Refreshing devices.
XInput: No touch devices found.
Detecting GPUs, set DRI_PRIME in the environment to override GPU detection logic.
Only one GPU found, using default.
XcursorGetTheme could not get cursor theme
Failed loading custom cursor: X_cursor
Using GLES3 video driver
OpenGL ES 3.0 Renderer: llvmpipe (LLVM 8.0, 256 bits)
PulseAudio: detected 2 channels
PulseAudio: audio buffer frames: 512 calculated latency: 11ms

CORE API HASH: -2720579051840101034
EDITOR API HASH:<PHONE_NUMBER>351750136
Loading resource: res://.config/godot//editor_settings-3.tres
EditorSettings: Load OK!
EditorSettings: Save OK!
Scanning projects at: /home/rahov/Documents/godot
Found 5 projects.
EditorSettings: Save OK!
Loaded builtin certs
EditorSettings: Save OK!```

I've been able to identify the issue, after downgrading the lts kernel from 4.19.73 to 4.19.71 Godot has successfully identified the GPU.
My assumptions were based on the mesa drivers causing issues, but it would seem I was wrong.
I'll close the issue now and leave it open for other people to find in case they come across the same problem, but if someone is capable of providing some insight, please feel free to add a comment on the topic.


code generation models

aimed at a 3090,4090,t4 level

https://huggingface.co/TheBloke/OpenHermes-2.5-neural-chat-7B-v3-1-7B-GGUF

open hermes 7b function calling
https://www.reddit.com/r/LocalLLaMA/comments/185fksv/forget_openai_function_calling_openhermes_outlines/

https://github.com/arthurwolf/llmi/blob/main/README.md

With a system limited machine (2017 i5 iMac Cpu only) I am getting very pleasing results with:
Openhermes2-mistral (7B 4bit K_M quant) for general chat, desktop assistant, and some coding assistance - Ollama backend with my own front end U/I and llama-index libraries implementation. Haven’t tried 2.5 but may.
Synatra 7B mistral fine tune (4bit K_M quant) seems to produce longer responses and spicier with same system prompt (same use case as above)
Deepseek-coder 6.7B (4bit quant) as a coding assistant alternative to GPT-3.5 - just trying out in last week or so and building the personalized coding assistant front end u/I for fun

I was using WizardCoder 13b Q4 with very good results. So, give it a shot, see how it compares to DeepSeek Coder 6.7b, which I now run in Q8 with again, very good results.

phind

LoneStriker_OpenHermes-2-Mistral-7B-8.0bpw-h6-exl2 - my generic goto

LoneStriker_airoboros-l2-70b-3.1-2.4bpw-h6-exl2 - this one (and the whole family) is great for creative and precise tasks. If they don't work I jump to wizardlm or vicuna.

oobabooga_CodeBooga-34B-v0.1-EXL2-4.250b and phind-codellama-34b-v2.Q4_K_M.gguf are great for coding. I haven't decided which one is better yet.

Llama2-70B for generating the plan and than using CodeLlama-34B for coding, or LLama-13B for executing the instructions from LLama-2-70B
Currently in the process of exploring what other models to add once LLama2-70B generates the plan for what needs to get done

latimar/Phind-Codellama-34B-v2-megacode-exl2 4.8BPW. As far as I know the best instruct programming model?

setting up the 34 b phind coder model
Phind-Codellama-34B
20GB size
GGUF (made for lama)
AWQ
GPTQ
The default groupsize for a GPTQ is 1024, so a model "without groupsize" is actually using a much higher value.
As perlamer then explains, the lower the groupsize, the better the accuracy but also the higher the VRAM requirements.
128 appears to be the optimal value, but it will use more VRAM than a model created with the default of 1024.
The reason that 30B models are being offered at both 128 and 1024 groupsize is that a 128 model is likely to exceed 24GB VRAM on longer responses, which means it can't be reliably run on a 3090, 4090, 3090Ti, A5000, etc. Whereas a model with 1024 can be.
vllm for serving
https://github.com/vllm-project/vllm
awq is slightly higher quality then gptq but its very slightly. Gguf and exl2 tho reach similar quality to awq.
the highest 4 bit quality i believe is with spqr
xwincoder 33b is new and subjectively anecdotally I think i'm happier with it than phind and deepseek
I think deepseek is a top contender in the battle to put random characters after pretty good output
One really powerful thing people do is train a ton of soft prompts and store them in a vectordb under an embedding of the user request that the soft prompt is trained on
I have a function that finds the last code block and merges the dict there with the workflow that the state machine is operating over
INSTLD: the simplest package management can solve package problems
oogabooga or llama are model installation environments

i use viimium and have a script running which monitors my clipboard for code
i just monitor ~/Downloads/ for text files with inotify/python


Map missing PSR3 levels to supported levels

An alternative to https://github.com/rollbar/rollbar-php/pull/187

This look great. I'm going to use this PR as base for for #140

Yeah this looks good to me too.

Do you know when it will be released?

@vdechenaux as soon as 1.2.0 is wrapped up, so probably next couple of days


problems building on x86 on windows

Hello.
I am encountering problems while building ort for x86 on windows.
```error[E0277]: unsafe extern "stdcall" fn(*const OrtCustomOp, usize) -> OrtCustomOpInputOutputCharacteristic doesn't implement Debug
--> C:\Users\beqap.cargo\registry\src\github.com-1ecc6299db9ec823\ort-1.15.1\src\sys.rs:2965:2
|
2952 | #[derive(Debug, Copy, Clone)]
|          ----- in this derive macro expansion
...
2965 |     pub GetInputCharacteristic: ::std::option::Option<_system!(unsafe fn(op: *const OrtCustomOp, index: size_t) -> OrtCustomOpInputOutputCharacteristic...
|     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unsafe extern "stdcall" fn(*const OrtCustomOp, usize) -> OrtCustomOpInputOutputCharacteristic cannot be formatted using {:?} because it doesn't implement Debug
|
= help: the trait Debug is not implemented for unsafe extern "stdcall" fn(*const OrtCustomOp, usize) -> OrtCustomOpInputOutputCharacteristic
= note: this error originates in the derive macro Debug (in Nightly builds, run with -Z macro-backtrace for more info)

Hmm, I can't reproduce this build error with rustc v1.71 building for --target i686-pc-windows-msvc. What is your rustc version, and which target are you compiling for?

I am compiling for i686-pc-windows-msvc and using rustc v1.69
On 7/19/23, Carson M. @.***> wrote:

Hmm, I can't reproduce this build error with rustc v1.71 building for
--target i686-pc-windows-msvc. What is your rustc version, and which
target are you compiling for?
--
Reply to this email directly or view it on GitHub:
https://github.com/pykeio/ort/issues/65#issuecomment-1642467992
You are receiving this because you authored the thread.
Message ID: @.***>

--
with best regards Beqa Gozalishvili
Tell: +995593454005
Email: @.***
Web: https://gozaltech.org
Skype: beqabeqa473
Telegram: https://t.me/gozaltech
facebook: https://facebook.com/gozaltech
twitter: https://twitter.com/beqabeqa473
Instagram: https://instagram.com/beqa.gozalishvili

Updating rustc resolved a problem, thank you


Fix SQLAlchemy readme with updated link to Ingres dialect repo

This SQLAlchemy index.rst (readme) has a reference to the Ingres dialect which points to https://github.com/clach04/ingres_sa_dialect and needs to be updated to instead point to https://github.com/ActianCorp/sqlalchemy-ingres

Opened discussion 11013

This has been taken care of via SQLAlchemy PR 11014.


Adoption of type-hints

The adoption of type-hints is becoming popular in the python community. Are there plans to add type hints within mujoco?

See #1292 for a temporary solution.

Going to close as a duplicate of #1292.


Fix HKSPI in the top diagram

HKSPI isn't connected to the wishbone bus.
Fix arrow as HKSPI is a slave.

@shalan :  You might not be aware of it, but I did remove the housekeeping SPI from the wishbone bus.  I am trying to anticipate future development where there will be on-board voltage regulators, and I want the housekeeping SPI to operate independently of the processor as it does on Raven/Ravenna/Raptor.  However, I thought that it would be helpful if the connection of the housekeeping SPI to the processor was not simply a "back door" with read-only access to the register contents.  So I put an SPI master on the wishbone bus, with a configuration bit that connects it directly to the housekeeping SPI, for both read and write access.  As a consequence, this makes the block diagram wrong, since the housekeeping SPI does not connect to the wishbone bus, but there is an SPI master on the bus that needs to be shown.


Site can't connect to the database

What happened (or feature request):
Ran ddev start on a project.
Navigated to local site
Received error- ddev web container: 500 or 501: Internal Server Error or Unimplemented
How to reproduce it (as minimally and precisely as possible):

Vanilla Drupal 8 install with a handful of contrib modules

Anything else do we need to know:
The site was working before so I'm not sure why the issue now.
I've checked the settings.php file and it matches the db creds from ddev describe
user: db
db: db
pw: db
host: db
port: 3306
Here's the full print out of the errors
2017/06/29 21:35:06 [error] 18#18: *3 FastCGI sent in stderr: "PHP message: PDOException: SQLSTATE[HY000] [2002] Connection timed out in /var/www/html/docroot/core/lib/Drupal/Component/DependencyInjection/PhpArrayContainer.php on line 79 #0 /var/www/html/docroot/core/lib/Drupal/Core/Database/Driver/mysql/Connection.php(149): PDO->__construct('mysql:host=db;p...', 'db', 'db', Array)
#1 /var/www/html/docroot/core/lib/Drupal/Core/Database/Database.php(376): Drupal\Core\Database\Driver\mysql\Connection::open(Array)
#2 /var/www/html/docroot/core/lib/Drupal/Core/Database/Database.php(166): Drupal\Core\Database\Database::openConnection('default', 'default')
#3 [internal function]: Drupal\Core\Database\Database::getConnection('default')
#4 /var/www/html/docroot/core/lib/Drupal/Component/DependencyInjection/PhpArrayContainer.php(79): call_user_func_array('Drupal\Core\Dat...', Array)
#5 /var/www/html/docroot/core/lib/Drupal/Component/DependencyInjection/Container.php(177): Drupal\Component\DependencyInjection\PhpArrayContainer->createService(Array, 'database')
#6 /var/www/ht" while reading response header from upstream, client: <IP_ADDRESS>, server: _, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.0-fpm.sock:", host: "drupal.ddev.local"
<IP_ADDRESS> - - [29/Jun/2017:21:35:06 +0000] "GET / HTTP/1.1" 500 535 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:54.0) Gecko/20100101 Firefox/54.0"

Hey @cedewey. Since this is a new site, can you confirm that there was no settings.php in sites/default directory before you ran ddev start? This is something we may need to do a  better job of tracking/cleaning up.

Yes, that's right. I downloaded Drupal from drupal.org and then ran through the install process as normal.

@cedewey can you provide the contents of your .ddev/config.yaml file?

Going to consider this closed at this point given that this spun into a convo in Slack that I believed got this to a completion point. Thanks, @cedewey!


Import cycle not allowed for a trivial source

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?
1.6

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/yassine/Dev/godev"
GORACE=""
GOROOT="/home/yassine/Dev/go"
GOTOOLDIR="/home/yassine/Dev/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?
compiling this code :

package main
import "fmt"
func main() {
fmt.Printf("hello, world\n")
}

What did you expect to see?
Binary code

What did you see instead?
this error :
import cycle not allowed
package github.com/golang/example/hello
imports fmt
imports errors
imports runtime
imports runtime/internal/atomic
imports runtime

NB : whene I install the version 1.5.3 its work fine

I think something is messed up in your Go installation or environment and you have a mix of Go versions installed atop each other. I suggest you bring this to the forum for help.


Not responding and logout without announcement.

Last login: Mon Sep 14 06:34:44 on ttys000
kapp1@kapp1 ~ % /Applications/Google\ Chrome.app/Contents/mac/Google\ Chrome ; exit;
[0914/063600.029151:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x7ffeefa21000, 0x2000): (os/kern) invalid address (1)
[0914/063600.161198:WARNING:crash_report_exception_handler.cc(240)] UniversalExceptionRaise: (os/kern) failure (5)
zsh: segmentation fault  /Applications/Google\ Chrome.app/Contents/mac/Google\ Chrome
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]

It seems this issue is referencing Chrome for MacOS. This library only works with Android browser. https://bugs.chromium.org/ is likely the right place for this bug. Closing it as it's outside the scope of this repo.

aha sorry and thank you 👍🏻


An Issues for triplet-loss in open-reid

hi,first thanks for this toolkit for re-id. There is a problem when I run the triplet-loss for market-1501,
python examples/resnet.py -d market1501 -b 64 -j 2 --loss triplet --logs-dir logs/resnet-triplet-test2 --num-instances 4 --optimizer adam
I only get 2.6% top1 accuracy. I find, finally the difference of hard-negative and hard-positive Almost goes to zero, that's not what we want. Did I have some operational errors? Or the code need to be modified?

@hh23333 It seems that you are using the old version open-reid. Please update to the latest version and run again.

@zydou Thank you very much for your help. The problem has been solved when I upgrade the latest version Open-reid and pytorch.
@Cysu Yes, it has been solved when I upgrade, and lr=0.01 is too large. In addition, I have modified lr to 0.0003 when run the old version, it also has this problem. When i upgrade the pytorch to version 0.2.0, and modifid the old code:
dist_an.append(dist[i][1-mask[i]].min()) to dist_an.append(dist[i][mask[i] == 0].min()) like latest version does, the problem solved.
thanks a lot for your code again.

@hh23333 Great to hear that! Happy experimenting with open-reid.


Docker load should return the list of images imported from the tar file

When docker loading a tar file with image(s), the (current v1.22) API should return the list of images it loaded.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
For more information about reporting issues, see CONTRIBUTING.md.
You don't have to include this information if this is a feature request
(This is an automated, informational response)

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

I've started initial work on implementing this. Will push a WIP branch & open a PR shortly.

I've successfully modified the load functions within the server to keep track of the imported images, but I'm unsure how to change the response format to include them.
The current wire response from docker load looks like the following:
&{0xc820652780 map[Content-Type:[application/json] Server:[Docker/1.11.0-dev (linux)] Date:[Fri, 01 Apr 2016 18:55:29 GMT] Content-Length:[1733]] 200}
{"status":"Loading layer","progressDetail":{"current":512,"total":2560},"progress":"[==========\u003e                                        ]    512 B/2.56 kB","id":"b652ec3a27e7"}
{"status":"Loading layer","progressDetail":{"current":1472,"total":2560},"progress":"[============================\u003e                      ] 1.472 kB/2.56 kB","id":"b652ec3a27e7"}
{"status":"Loading layer","progressDetail":{"current":1536,"total":2560},"progress":"[==============================\u003e                    ] 1.536 kB/2.56 kB","id":"b652ec3a27e7"}
{"status":"Loading layer","progressDetail":{"current":2048,"total":2560},"progress":"[========================================\u003e          ] 2.048 kB/2.56 kB","id":"b652ec3a27e7"}
{"status":"Loading layer","progressDetail":{"current":2560,"total":2560},"progress":"[==================================================\u003e]  2.56 kB/2.56 kB","id":"b652ec3a27e7"}
{"status":"Loading layer","progressDetail":{"current":2560,"total":2560},"progress":"[==================================================\u003e]  2.56 kB/2.56 kB","id":"b652ec3a27e7"}
{"status":"Loading layer","progressDetail":{"current":512,"total":1024},"progress":"[=========================\u003e                         ]    512 B/1.024 kB","id":"5f70bf18a086"}
{"status":"Loading layer","progressDetail":{"current":1024,"total":1024},"progress":"[==================================================\u003e] 1.024 kB/1.024 kB","id":"5f70bf18a086"}
{"status":"Loading layer","progressDetail":{"current":1024,"total":1024},"progress":"[==================================================\u003e] 1.024 kB/1.024 kB","id":"5f70bf18a086"}

@thaJeztah / @programmerq / others: what should the response format be? As far as I can tell, the current response output is ignored by all clients; can it be changed to just return a simple JSON object? Or should an additional JSON response line be added containing the details at the end. If so, what should that look like?
Thanks!

PR: https://github.com/docker/docker/pull/21754


feature: connected chains

Summary

Please summarize your changes, describing what they are and why they were made.

...
Changelog
- Please write a list of changes

Testing
Platforms

Please select any platforms where your changes have been tested.

Desktop

[ ] MacOS
[x] Linux
[ ] Windows

Mobile

[ ] iOS
[ ] Android

Instructions

Please describe the specific instructions, configurations, and/or test cases necessary to test and verify that your changes work as intended.

...
Checklist

Please tick the following boxes that are relevant to your changes.

[ ] I have followed the contribution guidelines for this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have added or modified tests that prove my changes work as intended
[ ] I have verified that new and existing unit tests pass locally with my changes
[ ] I have verified that my latest changes pass CI workflows for testing and linting
[ ] I have made corresponding changes to the documentation

Minor requested changes plus...
We don't currently have dark mode designs for this yet, but not sure if the card is suppose to be outlined. Feels a bit too much IMO (would be good to also ask @nicole-obrien):

We don't have dark mode design yet so I wouldn't let that hold up this pr


docs: update the README

:tada: This PR is included in version 1.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Initial work on AWS Lambda Client

Coverage increased (+1.0%) to 10.891% when pulling 0ba92ecfea28eebaf52de753e8b63f8ae923e14f on ewindisch:client into d1c59952e5bb952999ad0e0d814a991989ef044e on iopipe:master.


Improve error message in case of missing stable rustfmt

The build requires rust stable and rustfmt.
The rustfmt might not be installed in stable (e.g. rustup minimal profile), which causes following error message:
#13 400.7   error: toolchain 'stable-x86_64-unknown-linux-gnu' is not installed
#13 400.7   Error: Command '"rustup" "run" "stable" "rustfmt" "/opt/rust-esp32-std-hello/target/xtensa-esp32-espidf/debug/build/esp-idf-sys-e27296357ce0af82/out/bindings.rs"' failed with exit code Some(1).

Please, add command for the user to remedy the problem. The command should be:
rustup component add rustfmt --toolchain stable

Note: It's important to explicitly mention toolchain. Reason: when the user has default toolchain set to nightly and we omit the parameter --toolchain, then the installation will go to nightly.

Should be fixed with 4b72cf103dbff3bc8350763134a2dc260ec9a023


README.md doesn't specify the exact import and install commands

add the following somewhere:
Install: python3 -m pip install rpi-inky-layout
Import: from rpi_inky_layout import Layout, Rotation

While I'm on this, I'll add test scripts.
Currently, they're just filling the background black and displaying nothing. Soon get there!

While I'm on this, I'll add test scripts.
Currently, they're just filling the background black and displaying nothing. Soon get there!


Porting to newer versions

Your Idea
Make the bot work on newer versions like 1.20.1
Discord username (if you want updates)
philipou
Developer Notes (leave this blank)
Difficulty: __
Estimated Time Required: __
Once development starts.

Your Idea
Make the bot work on newer versions like 1.20.1
Discord username (if you want updates)
philipou, cheesyguy110
Developer Notes (leave this blank)
Difficulty: __ Estimated Time Required: __ Once development starts.

Edit: #359 seems to be closed. Alto main will likely stay on 1.18.2 until I can update it. I don't plan on doing this soon though. See Marvion's fork.


Fix broken link of Install Docker Engine - Enterprise on Windows Servers in Ko docs

Fix the broken link of Install Docker Engine - Enterprise on Windows Servers in Ko documentaion.
This PR is in response to issue #30101 and #29764.

Thank you @sandipanpanda ~!
/approve


Support bytes chunksize in dd.from_pandas

[x] Tests added / passed
[x] Passes black dask / flake8 dask

Start implementing https://github.com/dask/dask/issues/4555

Hows this going @henriqueribeiro? Anything we can do to help?

Sorry @TomAugspurger, I have been a bit busy. Later today I will take care of it

No rush! Just checking in.
On Mon, Aug 12, 2019 at 9:54 AM henriqueribeiro<EMAIL_ADDRESS>wrote:

Sorry @TomAugspurger https://github.com/TomAugspurger, I have been a
bit busy. Later today I will take care of it
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/5225?email_source=notifications&email_token=AAKAOIQQAYURX267B5Q67M3QEF2TFA5CNFSM4IJMXDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CZHTA#issuecomment-520459212,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKAOITULMH7LJGRPLOVT7LQEF2TFANCNFSM4IJMXDBQ
.

Lifting the conversion about what key to use for the config in https://github.com/dask/dask/pull/5225/files/89929d31a5e2e8fa80e170ab3acbc1af78d30930#diff-626a65eca39bf2787ec720b757a6803e out here.
Do we think that users will want to set these independently? I suspect the common case is to want to use the same value for both, so I'd prefer to just reuse the array.chunk_size config option here.

To be honest, I don't have a general overview if there are any other configurations that can be stored on the dataframe key. If this is the only one, I think it would be better to reuse the array

If this is the only one, I think it would be better to reuse the array

Let's do that. You'll need to update https://github.com/dask/dask/pull/5225/files#diff-626a65eca39bf2787ec720b757a6803eR199

We could also elevate array.chunksize to chunksize, and set up an alias.

@mrocklin by alias you mean the same value for two different keys on the config?

Yeah. The config option would now be documented as chunksize, and doing dask.config.get('array.chunksize') would return dask.config.get('chunksize').
I'm not sure exactly how migrating existing user configs would work. I'm perfectly fine with just reusing array.chunksize for this. I think it's justified by the fact that a series / dataframe can be thought of as a container for arrays.

Agree. I will update to use the array.chunksize instead

Here is an example of using aliases within our config system:
https://github.com/dask/distributed/blob/b83edbef74a1718d62e51a9cee0379b7617048e1/distributed/config.py#L22-L53
On Tue, Aug 13, 2019 at 8:35 AM henriqueribeiro<EMAIL_ADDRESS>wrote:

Agree. I will update to use the array.chunksize instead
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/5225?email_source=notifications&email_token=AACKZTBXE7XGWMQGNUCUBJLQEKTB5A5CNFSM4IJMXDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4FQM5Q#issuecomment-520816246,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTB2GITMCX5FOHPZEALQEKTB5ANCNFSM4IJMXDBQ
.

Unlike what happens in the distributed example, the array.chunk-size value is only set when dask.array module is loaded.

Maybe we should change that
On Tue, Aug 13, 2019 at 10:22 AM henriqueribeiro<EMAIL_ADDRESS>wrote:

Unlike what happens in the distributed example, the array.chunk-size
value is only set when dask.array module is loaded.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/5225?email_source=notifications&email_token=AACKZTBJAWZXDNDZ75PMYWTQEK7T3A5CNFSM4IJMXDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4F2FSA#issuecomment-520856264,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTFEGVYACJR5ZBUWN6DQEK7T3ANCNFSM4IJMXDBQ
.

I agree. Can I add it to the dask.yaml? Should it be a different PR?

@mrocklin, any thoughts on @henriqueribeiro's question?

Sorry about the delay @henriqueribeiro. Are you able to update for Matt's comments?

Sorry, I was on holidays. I will update it soon!

Checking in here @henriqueribeiro, have you had a chance revisit this? (no rush).

@TomAugspurger I had a few problems but I will take care of this soon.

@TomAugspurger I can't understand why CI is failing. Can you please help me here?

Results for windows:
partition_size : 180
mem_usage : 1716
npartitions : 10

For Linux:
partition_size : 180
mem_usage : 1804
npartitions : 11

So mem_usage is different...

Any hint @TomAugspurger?

No, I haven't had a chance to look.
On Fri, Nov 8, 2019 at 3:25 PM henriqueribeiro<EMAIL_ADDRESS>wrote:

Any hint @TomAugspurger https://github.com/TomAugspurger?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/5225?email_source=notifications&email_token=AAKAOITET7CKWNKL4366EUTQSXKLZA5CNFSM4IJMXDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDTMO2Y#issuecomment-551995243,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKAOIT22I2LGO7CIDZSOGTQSXKLZANCNFSM4IJMXDBQ
.

Hey @henriqueribeiro, this has been sitting for a while now so I'm going to close it out, but we'd love to have this in if you have interest in picking it back up.  Just let us know.  Thanks!

Hey @gforsyth! Sure, I will have a look into it! :)

Does this functionality exist with the current release? If not can this be reopened and merged?

@ba05 No, this doesn't exist yet.

If not can this be reopened and merged?

Not directly -- Looks like there were things to smooth out before this was closed. Moreover, if re-opened, it'll also need more work because the code might've changed in the past ~2 years. If @henriqueribeiro is still interested, they can pick it back up. :)
In general, I'm not sure when this will be prioritized. Would you be interested in working on this?
I'd also encourage you to move this discussion to the relevant issue: #4555


$return http binding returns body buffers as JSON

I am trying to serve an image from a function app running on node. At the moment we are doing this successfully using the context.res binding, however we are migrating our function apps to use the $return output name/binding (as is the recommendation in the docs for functions with a single output).
However, there seems to be inconsistent behaviour with named output bindings and the $return output binding. It appears that the isRaw flag is being ignored or not acted on in the same way when using the $return binding compared to a named binding.
Investigative information
Please provide the following:

Timestamp: 10/21/2019, 2:57:25.913 PM
Function App version (1.0 or 2.0): 2.0
Function App name: -
Function name(s) (as appropriate): -
Invocation ID: f40d7cbb-c5d1-435f-ae4c-058d4f9fe647
Region: West Europe

Repro steps
Create a function like so:
function.json:
{
  "disabled": false,
  "tracing": {
    "consoleLevel": "verbose"
  },
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "route": "image",
      "methods": ["GET"],
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
  ]
}

index.js:
module.exports = async () => {
    return {
        status: 200,
        headers: { 'content-type': 'image/jpeg' },
        body: Buffer.from([255, 216, 255, ...]), // this is the opening byte sequence for a jpg
        isRaw: true,
    };
}

Expected behavior
The response should be sent to the browser without being mangled (the isRaw flag is set to true) and the Buffer should be output as binary
$ curl [function host]/image
Date: Mon, 21 Oct 2019 15:22:45 GMT
Content-Type: image/jpeg
Content-Length: 38

���

Actual behavior
The response is a json-encoded Buffer and sent to the browser (with ; charset=utf-8 appended to the content type)
$ curl [function host]/image
Date: Mon, 21 Oct 2019 15:22:45 GMT
Content-Type: image/jpeg; charset=utf-8
Content-Length: 38

{"type":"Buffer","data":[255,216,255]}

Known workarounds
Use a named output binding:
function.json:
{
  "disabled": false,
  "tracing": {
    "consoleLevel": "verbose"
  },
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "route": "image",
      "methods": ["GET"],
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ]
}

index.js:
module.exports = async (context) => {
    context.res = {
        status: 200,
        headers: { 'content-type': 'image/jpeg' },
        body: Buffer.from([255, 216, 255, ...]), // this is the opening byte sequence for a jpg
        isRaw: true,
    };
}

Related information
This is a duplicate of https://github.com/Azure/azure-functions-host/issues/3892 but hopefully provides some greater detail

Moving this issue to nodejs worker. 'isRaw flag should be honored with $return as well

@pragnagopa is there an GitHub issue you can share for that?

@dhensby  - this is is the issue tracking the fix

This is the root cause - https://github.com/Azure/azure-functions-nodejs-worker/issues/228
Known bug, saving for v3 for the breaking fix :\


API not found

Summary
I can't find the following two APIs on windows crate:

MFGetAttributeSize
MFSetAttributeSize

Crate version: 0.51.0
Crate manifest
No response
Crate code
No response

And this:

MFGetAttributeUINT32
MFGetAttributeUINT64

If you look in mfapi.h, then you can see that MFGetAttributeSize is an inline function - a call to MFGetAttribute2UINT32asUINT64, which is another inline function:
inline
void
Unpack2UINT32AsUINT64(UINT64 unPacked, _Out_ UINT32* punHigh, _Out_ UINT32* punLow)
{
    *punHigh = HI32(unPacked);
    *punLow = LO32(unPacked);
}

inline
HRESULT
MFGetAttribute2UINT32asUINT64(
    IMFAttributes*  pAttributes,
    REFGUID         guidKey,
    _Out_ UINT32*   punHigh32,
    _Out_ UINT32*   punLow32
    )
{
    UINT64 unPacked;
    HRESULT hr = S_OK;

    hr = pAttributes->GetUINT64(guidKey, &unPacked);
    if (FAILED(hr)) {
        return hr;
    }
    Unpack2UINT32AsUINT64(unPacked, punHigh32, punLow32);

    return hr;
}

This crate can't create bindings for these functions. You have to implement them yourself.

That's it, thank you very much.

See Where's my favorite macro from the Windows SDK? for more information.


A couple of additional suggestions!

Once again, thanks for this. It's great.
Further, would it be possible to:

Define your own character name in the settings, and the HTML output colours that name differently. This way it differentiates your lines as opposed to other actors.
Run a different set of settings based on server you connect to. -- Not sure how this might be done if the input log doesn't differentiate. Possible to do based on nwmain.exe/IP address of server/name of direct connect shortcut?
The advantage of this is that you might have different filter lines for different servers, or other options.

Cheers!

For #2, actually, I just realised (after watching your video) I can use multiple .exe's in different folders and use the launcher instead of the direct connect shortcuts I have which might solve this request.

I'll add #1 to the list - it's possible.

Go ahead and try that please! (0.6.3 now has it in the export options)

Thanks. Applies character colour correctly (tested one character). However this has now removed other text colours, i.e. Tells, Whispers.https://i.gyazo.com/d8562af18fe2851c8dae9924aaa80024.png

Sorry about that, please try 0.6.4.

Colours look good. One thing with the unique case for servers.
Ravenloft works OK:

Arelith missing whispers because of their language system or delimiters(?):

Cheers.

Good catch. Should be fixed now in v0.6.6!

I'll raise a new issue if I see it. Thanks!


Removed @types/dotenv dependency

Removed unnecessary dependency
warning<EMAIL_ADDRESS>This is a stub types definition. dotenv provides its own type definitions, so you do not need this installed.

Nice, thanks for the contribution 👍


Add the command graph_stats and improve VLE messaging for load (#1750)

Added the command graph_stats. This command will force a reload of the specified graph. This is done to generate current statistics and dump any warning messages about the graph generated during a the load.
Cleaned up some messaging from the VLE load process for duplicate vertices and edges.
Added regression tests.

Looks good to me.


Migration from solid.community

Inform users coming from solid.community and suggested updates to documents elsewhere:

Users should change/reset passwords on solidcommunity.net ( via https://solidcommunity.net/account/password/reset as of this writing )
Remove solid.community from trustedApp list. Or consider replacing them with solidcommunity.net (see also https://solidproject.org/use-solid/apps#apps-inclusion-and-exclusion-criteria )
Remove solid.community from pod/IdP lists

Find the browser bookmarks which include solid.community and update them
Find any stored passwords which are for solid.community and update them

find user emails and send an email to all users


File upload with multipart/form-data not handled properly

Description
I noticed weird behavior when uploading a file through multipart/form-data field: if by mistake client uploads URL instead of file bytes, server crashes, thus returning 500 error. Since it was a malformed request, it should return 400.
Expected behaviour
Return 400 bad request error.
Actual behaviour
Returns 500 error.
Steps to reproduce
yaml config
/endpoint/:
    post:
      operationId: app.execute
      parameters:

        - in: formData
          name: image
          description: image to be processed
          type: file
          required: false

request
curl -X POST   http://<IP_ADDRESS>:8000/endpoint/  -F image=https://upload.wikimedia.org/wikipedia/en/thumb/7/7d/Lenna_%28test_image%29.png/220px-Lenna_%28test_image%29.png
response
500
server stack trace
KeyError: 'file'
	at make_type (/app/env/lib/python3.6/site-packages/connexion/decorators/parameter.py:56)
	at get_val_from_param (/app/env/lib/python3.6/site-packages/connexion/decorators/parameter.py:67)
	at wrapper (/app/env/lib/python3.6/site-packages/connexion/decorators/parameter.py:181)
	at wrapper (/app/env/lib/python3.6/site-packages/connexion/decorators/decorator.py:44)
	at wrapper (/app/env/lib/python3.6/site-packages/connexion/decorators/validation.py:300)
	at wrapper (/app/env/lib/python3.6/site-packages/connexion/decorators/uri_parsing.py:117)
	at wrapper (/app/env/lib/python3.6/site-packages/connexion/decorators/decorator.py:73)
	at dispatch_request (/app/env/lib/python3.6/site-packages/flask/app.py:1799)
	at full_dispatch_request (/app/env/lib/python3.6/site-packages/flask/app.py:1813)
	at reraise (/app/env/lib/python3.6/site-packages/flask/_compat.py:35)
	at handle_user_exception (/app/env/lib/python3.6/site-packages/flask/app.py:1718)
	at wrapped_function (/app/env/lib/python3.6/site-packages/flask_cors/extension.py:161)
	at full_dispatch_request (/app/env/lib/python3.6/site-packages/flask/app.py:1815)
	at wsgi_app (/app/env/lib/python3.6/site-packages/flask/app.py:2292)

Additional info:
It is likely caused by uploaded files being passed in form data, instead of requests.files: https://github.com/zalando/connexion/blob/master/connexion/decorators/parameter.py#L168-L190.
I managed to get a workaround this issue by setting the type of image field to string, and later handling the validity of file bytes.
Output of the commands:

python --version Python 3.5.2
pip show connexion | grep "^Version\:" 1.4.2

Hey, I just wanted to say thank for filing this ticket. I haven't been able to get around to it, but I would definitely review a PR if you want to make one.

Is there any update on this? Does the bug still on the latest version?

Can no longer reproduce


Fix tests

Hey @liiight,
Your changes look good to me!
View build log
TravisBuddy Request Identifier: a1857100-8c9a-11e9-94f5-3d756601374f


Fix minor typos

Fix minor typos

Thanks a lot for your pull request. Your attention to detail is greatly appreciated!


bug: Expected variable data for statement not found.

Bug Report
Cairo version:
af083f2ee0e5cc041d59728b4bc996afb40dfe1b

Current behavior:
Running cairo-run on the following code fails with error
Error: Failed setting up runner.

Caused by:
    #54: Expected variable data for statement not found.

Expected behavior:
The code should run successfully.

Steps to reproduce:
Run the following code where you pass a snapshot of an array to the array implementation of the clone trait.

Related code:
use array::ArrayTrait;
use array::ArrayTCloneImpl;
use clone::Clone;

fn main() {
    let arr0 = ArrayTrait::<felt252>::new();
    let arr_clone = (@arr0).clone();
}

Other information:
I tried to reproduce the bug myself without the libraries
use array::ArrayTrait;

fn main() {
    let arr0 = ArrayTrait::<felt252>::new();
    let arr_clone = clone_array(@arr0);
}

// Don't change these functions! They are used to print the output.
fn clone_array(arr: @Array<felt252>) -> Array<felt252> {
    let mut new_arr = ArrayTrait::new();
    clone_array_(arr, ref new_arr);
    new_arr
}

fn clone_array_(src: @Array<felt252>, ref dst: Array<felt252>) {
    match gas::withdraw_gas() {
        Option::Some(_) => {},
        Option::None(_) => {
            let mut data = ArrayTrait::new();
            data.append('Out of gas');
            panic(data);
        },
    }
    if src.len() == dst.len() {
        return ();
    }
    let value = *src.at(dst.len());
    dst.append(value);
    clone_array_(src, ref dst);
}

The error is linked to the gas management part. It works fine without it.

I can't reproduce this bug. Does it still occur?

I can't reproduce this bug. Does it still occur?

Yes, both with the exact same code snippets.
I just created a file named temp.cairo, pasted this, and ran cairo-test -p temp.cairo

On main?

yes 🤔
I ran cargo clean and rebuilt from source in release mode just before

cairo-run doesnt run with gas by default, run with "--available-gas=2000000". We should probably have a better error message.
Consdier using cairo-test.


Override changeset requirements

The issue

CI requires changeset on any PR including renovate

Maybe this should be overridden via a label or something. Every push or PR may not meant to trigger a patch.
WDYT?
@dcroote @andreogle

I think CI should probably not require a changeset (?) It can occasionally be useful if there are dependencies that cause breaking changes though I guess.
And maybe for other PRs that don't require a version bump, we can just require an empty changeset? I kinda like that it's opt-out, rather than opt-in, so we don't forget

I think this could be solved in two parts and still keep the changeset requirement (it can be easy to forget otherwise):

Bypass changesets for renovate using the same approach as Airnode
Ourselves run pnpm changeset --empty and commit the resulting empty file when we don't need to trigger a patch

I have opened a PR to demonstrate the solution @dcroote suggested.


exclude nodes , weird behavior

hey,
super great script  !!! but, i think i have found a weird behavior with exclude_nodes , with this configuration, it says "no route", but with the second version using exclude_channels behavior is correct.
"fee_limit_ppm"= 500
"amount"= 10000
"min_amount"= 1000
"probe_steps"= 5
"pfrom"= 20
"pto"= 80
"stat"= "stats.csv"
 "exclude_nodes"= [
     "033d8656219478701227199cbd6f670335c8d408a92ae88b962c49d4dc0e83e025", # bfx-lnd0
     "02cc125c2770dd293f56b50841c8d847276efd00634d48e947c9b83125315d8f", # bfx-lnd1
     "021c97a90a411ff2b10dc2a8e32de2f29d2fa49d41bfbb52bd416e460db0747d0d", # loop
     "024bfaf0cabe7f874fd33ebf7c6f4e5385971fc504ef3f492432e9e3ec77e1b5cf" # deezy
]
"to" = [
<PHONE_NUMBER>12229888
]
"from" = [

]
lost_profit = true
allow_unbalance_to = true
allow_unbalance_from = true

not working,
but this is working :
"fee_limit_ppm"= 500
"amount"= 10000
"min_amount"= 1000
"probe_steps"= 5
"pfrom"= 20
"pto"= 80
"stat"= "stats.csv"
"exclude_channels"= [
<PHONE_NUMBER>66611713, #bfx0
<PHONE_NUMBER>38146304, # bfx0
<PHONE_NUMBER>36823809, #bfx1
<PHONE_NUMBER>45056000,  #deezy
<PHONE_NUMBER>15519745 #deezy
]
"to" = [
<PHONE_NUMBER>12229888
]
"from" = [

]
lost_profit = true
allow_unbalance_to = true
allow_unbalance_from = true

Looks like you're missing 2 characters in bfx-lnd1 pubkey, it's shorter than the rest. I think this situation isn't handled correctly, will take a look later.

Should be fixed in 465b61829cd05bcf907459c2fb6754d539f1cc8e, now you'll get an error on startup regarding that invalid node id.


Prophet on Vertex Prediction Pipeline Error: deadline_exceeded

Expected Behavior
I'm following this documentation for the Prophet on Vertex Pipelines notebook. I expected the prediction pipeline to complete successfully after running the batch prediction job.
Actual Behavior
The prediction pipeline fails at the model batch prediction step with a deadline_exceeded error. The job consistently fails after running for around 40 minutes, regardless of the dataset size used. Additionally, I am unable to identify the exact source of the issue because the batch prediction logs do not provide any detailed information.

Steps to Reproduce the Problem

Follow the steps in the Prophet on Vertex Pipelines notebook to set up and run the training pipeline.
Proceed to the prediction pipeline step, using any dataset.
Observe the pipeline fail at the model-batch-predict stage after running for approximately 40 minutes with a deadline_exceeded error.

Specifications

Version: google-cloud-aiplatform version '1.59.0', Python version '3.12.2'
Platform: Google Cloud Vertex AI, OS: Microsoft Windows 11 version '10.0.22631'

Thanks for reporting this issue. This may be a memory issue. Please try to reduct the input size or use a higher memory VM for the prediction job.


Understanding Impact of frontend developer and how to motivate them with sense of accomplishment

Hi Tanay,
I read your article - "web dev is a win", and it highlights pretty well what a web developer does as a whole. Recently, I am facing trouble explaining to new web developers the impact that web dev makes in the company product. We  are part of a product team building a web app for cloud computing,however one intern asked me after working for an year that since we have a  B2B product, and its tech-heavy, the major logic part, orchestration of the features and the design of the logic in the feature is kept in the hands of a backend developer. According to him, since in frontend the tools are already laid out by senior folks, he feels, he is not creating much impact, creating up the UI seeing the VDs and using existing components and services at his disposal. Rather, he feels the backend developer is involved with handling a large amount of data, writing the algo for the feature, so now he wants to make a switch for the same reason. So my question is how do we motivate or retain developers in the frontend role giving them more opportunities that give them a sense of accomplishment of creating an impact for the customer.
I will be grateful, if you can answer this, or dedicate a blog post for the same, on how in B2B, web developers can create a meaningful impact related to revenue generation.
Thanks

Ha there been an answer to this? If yes I would like to know

Unfortunately, I don't have a link for the article now. Secondly, there is no one answer to this, it depends on how the team is showing respect to the frontend work and appreciating the efforts put forth by the team. Also, in between the individual needs to be shown value he adds to the product/software they are building by quantifying it with something like revenue, hours saved, etc. Appreciation is the key. Third, full stack dev should be encouraged more rather than just frontend dev, with engineers in the team should be shuffled in between sprints. Lastly, it is hard to find motivated frontend only engineers, initially you can get few young folks to work on it, however over a period of time, the motivation fades away, if there is no innovation.