trouble loading netcdf4 files with xarray on s3

I'm working on allowing direct access to netcdf4/hdf5 file-like objects (https://github.com/pydata/xarray/pull/2782). This seems to be working fine with gcsfs, but not s3fs (versions 0.2 from conda-forge). Here is a gist with the relevant code and error traceback:
https://gist.github.com/scottyhq/304a3c4b4e198776b8d82fb3a9f300e3
and an abbreviated traceback here:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/Documents/GitHub/xarray/xarray/backends/file_manager.py in acquire(self, needs_lock)
    166             try:
--> 167                 file = self._cache[self._key]
    168             except KeyError:

~/Documents/GitHub/xarray/xarray/backends/lru_cache.py in __getitem__(self, key)
     40         with self._lock:
---> 41             value = self._cache[key]
     42             self._cache.move_to_end(key)

KeyError: [<function _open_h5netcdf_group at 0x11d8b0ae8>, (<S3File grfn-content-prod/S1-GUNW-A-R-137-tops-20181129_20181123-020010-43220N_41518N-PP-e2c7-v2_0_0.nc>,), 'r', (('group', '/science/grids/data'),)]

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/test_env/lib/python3.6/site-packages/s3fs/core.py in readinto(self, b)
   1498         data = self.read()
-> 1499         b[:len(data)] = data
   1500         return len(data)

~/miniconda3/envs/test_env/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-darwin.so in View.MemoryView.memoryview.__setitem__()

~/miniconda3/envs/test_env/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-darwin.so in View.MemoryView.memoryview.setitem_slice_assignment()

~/miniconda3/envs/test_env/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-darwin.so in View.MemoryView.memoryview_copy_contents()

~/miniconda3/envs/test_env/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-darwin.so in View.MemoryView._err_extents()

ValueError: got differing extents in dimension 0 (got 8 and 59941567)

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/test_env/lib/python3.6/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

any guidance as to what might be going on here would be appreciated!

That is quite a traceback!
I am surprised that gcsfs worked, rather than that s3fs did not - hdf5 is a C-level reader that likes to have a local real file to read from. I have heard that they have tried to allow for python file-like objects, but I don't know how that's implemented - apparently something is subtly different between the two file implementation classes.

Same problem for me, I can't read a netCDF on S3 using s3fs with h5netcdf:
>>> s3 = s3fs.S3FileSystem(key=os.environ['AWS_DS_AGENT_KEY_ID'],
                    secret=os.environ['AWS_DS_AGENT_ACCESS_KEY'])
>>> fileobj = s3.open(s3_fp)
>>> nc = h5netcdf.File(fileobj,'r', invalid_netcdf=True)

Traceback

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in readinto(self, b)
   1498         data = self.read()
-> 1499         b[:len(data)] = data
   1500         return len(data)

~/miniconda3/envs/s3test/lib/python3.7/site-packages/h5py/h5fd.cpython-37m-x86_64-linux-gnu.so in View.MemoryView.memoryview.__setitem__()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/h5py/h5fd.cpython-37m-x86_64-linux-gnu.so in View.MemoryView.memoryview.setitem_slice_assign_scalar()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/h5py/h5fd.cpython-37m-x86_64-linux-gnu.so in View.MemoryView._memoryviewslice.assign_item_from_object()

TypeError: an integer is required

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
<ipython-input-7-49848b8cfbdc> in <module>
----> 1 nc = h5netcdf.File(fileobj,'r', invalid_netcdf=True)

~/miniconda3/envs/s3test/lib/python3.7/site-packages/h5netcdf/core.py in __init__(self, path, mode, invalid_netcdf, **kwargs)
    603                 else:
    604                     self._preexisting_file = mode in {'r', 'r+', 'a'}
--> 605                     self._h5file = h5py.File(path, mode, **kwargs)
    606         except Exception:
    607             self._closed = True

~/miniconda3/envs/s3test/lib/python3.7/site-packages/h5py/_hl/files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)
    392                 fid = make_fid(name, mode, userblock_size,
    393                                fapl, fcpl=make_fcpl(track_order=track_order),
--> 394                                swmr=swmr)
    395
    396             if swmr_support:

~/miniconda3/envs/s3test/lib/python3.7/site-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    168         if swmr and swmr_support:
    169             flags |= h5f.ACC_SWMR_READ
--> 170         fid = h5f.open(name, flags, fapl=fapl)
    171     elif mode == 'r+':
    172         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

h5py/defs.pyx in h5py.defs.H5Fopen()

h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

~/miniconda3/envs/s3test/lib/python3.7/site-packages/s3fs/core.py in seek(self, loc, whence)
   1234             from start of file, current location or end of file, resp.
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')
   1238         if whence == 0:

If you can post the file somewhere public, I can try to find out what's going on.

I am keen to see a way to do this without a fuse mount - here is an open file:
import xarray as xr
import s3fs
fs = s3fs.S3FileSystem(anon=True)
s3path = 'imos-data/IMOS/SRS/OC/gridded/aqua/P1D/2010/05/A.P1D.20100507T000000Z.aust.ipar.nc'

fobj = fs.open(s3path)
ds = xr.open_dataset(fobj,engine='h5netcdf')

Produces roughly the same stack trace

I am getting
      5
      6 fobj = fs.open(s3path)
----> 7 ds = xr.open_dataset(fobj,engine='h5netcdf')

~/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs)
    345     else:
    346         if engine is not None and engine != 'scipy':
--> 347             raise ValueError('can only read file-like objects with '
    348                              "default engine or engine='scipy'")
    349         # assume filename_or_obj is a file-like object

ValueError: can only read file-like objects with default engine or engine='scipy'

however, the problem for you seems to be here:
-> 1499         b[:len(data)] = data

The message implies that that the data being inserted is the wrong size; it would be good to debug at that point to see what the buffer b and data contain.

Yes - I think there were some recent updates that changed this behaviour to
allow h5netcdf with file-like objects
I was using these versions from pip to get the previously mentioned error:
h5netcdf-0.7.1
h5py-2.9.0
pytz-2018.9
xarray-0.12.0
On Wed, Apr 3, 2019 at 9:36 AM Martin Durant<EMAIL_ADDRESS>wrote:

I am getting
  5
  6 fobj = fs.open(s3path)

----> 7 ds = xr.open_dataset(fobj,engine='h5netcdf')
~/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs)
345     else:
346         if engine is not None and engine != 'scipy':
--> 347             raise ValueError('can only read file-like objects with '
348                              "default engine or engine='scipy'")
349         # assume filename_or_obj is a file-like object
ValueError: can only read file-like objects with default engine or engine='scipy'
however, the problem for you seems to be here:
-> 1499         b[:len(data)] = data
The message implies that that the data being inserted is the wrong size;
it would be good to debug at that point to see what the buffer b and data
contain.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/s3fs/issues/168#issuecomment-479289616, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AM3bQKm461Vg3QIEDlOcEDeoWk8xFVfOks5vdAWwgaJpZM4bTe4Y
.

You are correct - if I check back through the stack trace I am getting the error here:
/opt/conda/lib/python3.6/site-packages/s3fs/core.py in readinto(self, b)
   1498         data = self.read()
-> 1499         b[:len(data)] = data
   1500         return len(data)

/opt/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview.__setitem__()

/opt/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview.setitem_slice_assignment()

/opt/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview_copy_contents()

/opt/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView._err_extents()

ValueError: got differing extents in dimension 0 (got 8 and 24247921)

Not entirely clear to me how all these libraries tie together but it seems that
h5py:files.py 170        fid = h5f.open(name, flags, fapl=fapl)
is calling out to the hdf5 c-library function "H5Fopen", which is expecting a string filepath, where as name at this point is an s3fs.S3File object. Somehow passing the "name" parameter is invoking
s3fs:core.py
1497        def readinto(self, b):
                    data = self.read()
                    b[:len(data)] = data
                    return len(data)

where b is a <MemoryView of 'array' at 0x1b458a73be0> but I cant workout where b is instantiated - it only has a length of 8, where as the binary data read from s3 is much larger - hence the exception.
However even if that succeeded I dont know how this would work anyway, given the c library is expecting a string file path, rather than a binary memory view?

I am surprised that gcsfs worked, rather than that s3fs did not - hdf5 is a C-level reader that likes to have a local real file to read from.

If I get time I will take a look at why the gcsfs is working

OK, so something "new" :)
I would suspect that the memoryview has a complex type other than bytes, and s3fs is trying to fill the buffer with bytes (although it doesn't appear to be an exact multiple). readinto is very rarely used anywhere, surprised to see it, but I suppose the memory must have been allocated in C-land.
btw: the difficulties with hdf are the main reason for interest in libraries like zarr (or zarr as a backend for netcdf), which is known to work well with s3fs/gcsfs/etc. It may or may not be a viable alternative for you.

Thanks Martin - testing out a 'simple' (albeit slow) way converting of
converting netCDF that is already present in cloud storage into Zarr
format, with out formally 'mirroring' the files locally somewhere.
Noting that if performance is crucial and you have the AWS budget, then
setting up AWS FSx is the obvious way to go (i.e.
https://jiaweizhuang.github.io/blog/fsx-experiments/)
On Wed, Apr 3, 2019 at 10:32 PM Martin Durant<EMAIL_ADDRESS>wrote:

OK, so something "new" :)
I would suspect that the memoryview has a complex type other than bytes,
and s3fs is trying to fill the buffer with bytes (although it doesn't
appear to be an exact multiple). readinto is very rarely used anywhere,
surprised to see it, but I suppose the memory must have been allocated in
C-land.
btw: the difficulties with hdf are the main reason for interest in
libraries like zarr (or zarr as a backend for netcdf), which is known to
work well with s3fs/gcsfs/etc. It may or may not be a viable alternative
for you.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/s3fs/issues/168#issuecomment-479515416, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AM3bQDEM5VDX0QSbBQ8-XDMwLV1dDS0yks5vdLttgaJpZM4bTe4Y
.

@martindurant, I'm also hitting this as well.   Was hoping to follow up on @rabernat's suggestion to include this in our testing of different options for accessing NetCDF4/HDF5 on s3 (in addition to Zarr and HSDS).
I got the same error you did:
ValueError: can only read file-like objects with default engine or engine='scipy'

when I forgot to install h5netcdf and h5py into my environment.

@pbranson , did you manage to learn anything about this issue?

Hm, with everything updated (h5netcdf , h5py, s3fs), the first invocation just worked:
In [5]: fs = s3fs.S3FileSystem(anon=True)
   ...: s3path = 'imosdata/IMOS/SRS/OC/gridded/aqua/P1D/2010/05/A.P1D.20100507T000000Z.aust.ipar.nc'
   ...:
   ...: fobj = fs.open(s3path)
In [6]: nc = h5netcdf.File(fobj, 'r', invalid_netcdf=True)
In [7]: nc
Out[7]:
<h5netcdf.File 'A.P1D.20100507T000000Z.aust.ipar.nc>' (mode r)>
Dimensions:
    latitude: 7001
    longitude: 10001
    time: 1
Groups:
Variables:
    time: ('time',) float64
    latitude: ('latitude',) float64
    longitude: ('longitude',) float64
    ipar: ('time', 'latitude', 'longitude') float32
Attributes:
    history: b'File initialised at 2015-12-17T19:03:50.793738\nInitialised var ipar at 2015-12-17T19:04:36.563452\nAdd Granule A20100507_0230.20150923161152.L2OC_BASE.ipar.nc at 2015-12-17T19:04:38.498914\nAdd Granule A20100507_0235.20150923151200.L2OC_BASE.ipar.nc at 2015-12-17T19:04:38.975299\nAdd Granule A20100507_0240.20150923134822.L2OC_BASE.ipar.nc at 2015-12-17T19:04:39.483551\nAdd Granule A20100507_0245.20150923143121.L2OC_BASE.ipar.nc at 2015-12-17T19:04:39.793043\nAdd Granule A20100507_0405.20150923141146.L2OC_BASE.ipar.nc at 2015-12-17T19:04:40.401902\nAdd Granule A20100507_0410.20150923162326.L2OC_BASE.ipar.nc at 2015-12-17T19:04:40.977119\nAdd Granule A20100507_0415.20150923133857.L2OC_BASE.ipar.nc at 2015-12-17T19:04:41.430398\nAdd Granule A20100507_0420.20150923150036.L2OC_BASE.ipar.nc at 2015-12-17T19:04:41.923474\nAdd Granule A20100507_0540.20150923152402.L2OC_BASE.ipar.nc at 2015-12-17T19:04:42.336277\nAdd Granule A20100507_0545.20150923154421.L2OC_BASE.ipar.nc at 2015-12-17T19:04:43.116328\nAdd Granule A20100507_0550.20150923140042.L2OC_BASE.ipar.nc at 2015-12-17T19:04:43.709527\nAdd Granule A20100507_0555.20150923155628.L2OC_BASE.ipar.nc at 2015-12-17T19:04:44.321537\nAdd Granule A20100507_0600.20150923165701.L2OC_BASE.ipar.nc at 2015-12-17T19:04:44.871419\nAdd Granule A20100507_0720.20150923142308.L2OC_BASE.ipar.nc at 2015-12-17T19:04:45.394833\nAdd Granule A20100507_0725.20150923132636.L2OC_BASE.ipar.nc at 2015-12-17T19:04:46.131246\nAdd Granule A20100507_0730.20150923163350.L2OC_BASE.ipar.nc at 2015-12-17T19:04:46.614609\nAdd Granule A20100507_0735.20150923153102.L2OC_BASE.ipar.nc at 2015-12-17T19:04:47.083167\nAdd Granule A20100507_0740.20150923144622.L2OC_BASE.ipar.nc at 2015-12-17T19:04:47.608014'
    Conventions: b'CF-1.6'
    source_path: b'imos-srs/archive/oc/aqua/1d/v201508/2010/05/A20100507.L2OC_BASE.aust.ipar.ncimos-srs/archive/oc/aqua/1d/v201508/2010/05/A20100507.L2OC_BASE.aust.ipar.ncimos-srs/archive/oc/aqua/1d/v201508/2010/05/A20100507.L2OC_BASE.aust.ipar.nc'

but via xarray it does not.

@martindurant - how are you invoking xarray?

fobj = fs.open(s3path)
ds = xr.open_dataset(fobj, engine='h5netcdf')

That's weird, because it does work with gcsfs.
What xarray error are you getting?

ValueError: can only read file-like objects with default engine or engine='scipy'
(also a few comments higher up the thread)

Could we point this discussion to a public file instead? That would make debugging easier for me. I don't have any credentials to try the file in question.
When I try with the ERA5 public data, I can't even open it with h5py
fs = s3fs.S3FileSystem(anon=True)
s3path = 'era5-pds/2008/01/data/air_temperature_at_2_metres.nc'
file_obj = fs.open(s3path)
h5 = h5py.File(file_obj, 'r')

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

/srv/conda/lib/python3.6/site-packages/s3fs/core.py in readinto(self, b)
   1498         data = self.read()
-> 1499         b[:len(data)] = data
   1500         return len(data)

/srv/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview.__setitem__()

/srv/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview.setitem_slice_assignment()

/srv/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView.memoryview_copy_contents()

/srv/conda/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so in View.MemoryView._err_extents()

ValueError: got differing extents in dimension 0 (got 8 and<PHONE_NUMBER>)

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
h5py/h5fd.pyx in h5py.h5fd.H5FD_fileobj_read()

/srv/conda/lib/python3.6/site-packages/s3fs/core.py in seek(self, loc, whence)
   1235         """
-> 1236         if not self.readable():
   1237             raise ValueError('Seek only available in read mode')

SystemError: PyEval_EvalFrameEx returned a result with an error set

OK, solved it - and it seems this only happens for some specific files!
The reason it works with gcsfs is that it simply doesn't have a readinto method (but it should!), so it seems h5py falls back to read

Sorry that it took do long for me to dig this out!

I was excited to try this out, but the simple try below is not working for some reason:
import xarray as xr
import s3fs
import h5netcdf

print(xr.__version__)
print(s3fs.__version__)
print(h5netcdf.__version__)

fs = s3fs.S3FileSystem(anon=True)
fileObj = fs.open('esip-pangeo/pangeo/adcirc/adcirc_01.nc')
print(fileObj.info())

produces:
0.12.1
0.2.1
0.7.1
{'ETag': '"79ca97f44f5fed750f6dea35a16f6ac9-4986"', 'Key': 'esip-pangeo/pangeo/adcirc/adcirc_01.nc', 'LastModified': datetime.datetime(2019, 4, 12, 17, 46, 44, tzinfo=tzutc()), 'Size':<PHONE_NUMBER>4, 'StorageClass': 'STANDARD', 'VersionId': None}

but then this causes the kernel to die:
ds = xr.open_dataset(fileObj, engine='h5netcdf', chunks={'time':10, 'node':141973})

@martindurant , any ideas?

@rabernat , you verified this working for some other .nc files, correct?
A dead kernel suggests an exception in the C library, which would be very hard to diagnose.
Running
h5py.File(fileObj, 'r')

has not caused an error for me yet, but it seems to be downloading everything and filling up memory (I know the file is extremely big), so it's possible that the metadata is laid out in a particularly unfriendly way. That still doesn't explain your crash. Perhaps would be better with default_fill_cache=False for the fs.
I am looking into implementing https://github.com/dask/s3fs/pull/177/ across all filesystems in fsspec, which would be just the thing for a case like this.

@martindurant , yes!
fs = s3fs.S3FileSystem(anon=True, default_fill_cache=False)
fileObj = fs.open('esip-pangeo/pangeo/adcirc/adcirc_01.nc')
ds = xr.open_dataset(fileObj, engine='h5netcdf', chunks={'time':10, 'node':141973})

works within a few seconds!

Good, but also annoying! Making options that tend to work for most people most of the time is hard...

(I suppose this is why you want to encode all the options required for smooth working of a particular dataset into a catalog...)

None of that traceback appears to be in s3fs - are you sure it loads OK from local? If yes, then finding the problem will be tricky, as apparently, any exception is being hidden.

I'm curious how these caches behave with dask / distributed. Are the cache contents serialized, or is the cache cleared before pickling the file object?

Are the cache contents serialized, or is the cache cleared before pickling the file object?

The files are not sent around at all. What you actually send is an OpenFile object ( https://github.com/dask/dask/blob/master/dask/bytes/core.py#L143 ), which only creates the S3FileSystem object in a with block - so caches do not survive tasks.

@rsignell-usgs , how did you comment from the future? :)

do you think this a s3fs, xarray, h5netcdf or h5py issue

I would exclude xarray here. What happens within h5py when it calls s3 is a bit of a mystery - perhaps more logging in s3fs would help, set logger "s3fs.core" to DEBUG and you'll get some.

@martindurant , I can read the file locally with xarray using the netcdf4 engine, but not with the h5netcdf engine.   I can also read the file locally using h5py, so I guess that make it a h5netcdf issue.
Thanks!

I also really confused about how I managed to post a comment 5 hours from now.  🙄

Are the cache contents serialized, or is the cache cleared before pickling the file object?

PS: the file-system is serialised in this process, including directory listings. This is good or bad - you avoid potentially slow lookups when opening the file, but the instance is bigger. I notice that gcsfs does not preserve the listings cache. gcsfs came later and  is, in some ways, better designed (hence my attempt to consolidate such things into fsspec).

If I take a slice from a netcdf opened with s3fs+h5netcdf is it doing some
form of byte range request or essentially downloading the entire file into
a memory cache and then slicing?
In which case we should always chunk on a file basis when using this method?
On Wed., 8 May 2019, 11:19 pm Martin Durant<EMAIL_ADDRESS>wrote:

Are the cache contents serialized, or is the cache cleared before pickling
the file object?
PS: the file-system is serialised in this process, including directory
listings. This is good or bad - you avoid potentially slow lookups when
opening the file, but the instance is bigger. I notice that gcsfs does not
preserve the listings cache. gcsfs came later and is, in some ways, better
designed (hence my attempt to consolidate such things into fsspec).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/s3fs/issues/168#issuecomment-490529693, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADG5WQCGSUQI7DCVYQS4TZ3PULVQ5ANCNFSM4G2N5YMA
.

I don't know the internals of h5netcdf, but i would hope it's a range. You could time reading a whole array versus reading a single value; but it will not be linear, due to fixed costs of each connection  and metadata lookups. For a slice, it would depend on exact layout and chunking. You may want to turn on s3fs debug logging.
On May 12, 2019 8:35:48 PM EDT, Paul Branson<EMAIL_ADDRESS>wrote:

If I take a slice from a netcdf opened with s3fs+h5netcdf is it doing
some>
form of byte range request or essentially downloading the entire file
into>
a memory cache and then slicing?>

In which case we should always chunk on a file basis when using this
method?>

On Wed., 8 May 2019, 11:19 pm Martin Durant,
<EMAIL_ADDRESS>wrote:>

Are the cache contents serialized, or is the cache cleared before
pickling>
the file object?>

PS: the file-system is serialised in this process, including
directory>
listings. This is good or bad - you avoid potentially slow lookups
when>
opening the file, but the instance is bigger. I notice that gcsfs
does not>
preserve the listings cache. gcsfs came later and is, in some ways,
better>
designed (hence my attempt to consolidate such things into fsspec).>

—>
You are receiving this because you were mentioned.>
Reply to this email directly, view it on GitHub>
https://github.com/dask/s3fs/issues/168#issuecomment-490529693, or
mute>
the thread>

https://github.com/notifications/unsubscribe-auth/ADG5WQCGSUQI7DCVYQS4TZ3PULVQ5ANCNFSM4G2N5YMA>

.>

-- >
You are receiving this because you were mentioned.>
Reply to this email directly or view it on GitHub:>
https://github.com/dask/s3fs/issues/168#issuecomment-491642673

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.


docs: update example in getting started

Randomly noticed when blindly copy-pasting example there is an extra ). Also added mf.dispose() to make example more complete.

@pi0 do you happen to have more complete example how to get started with this?
I mean things like:

Do I need create Worker project first and install to miniflare package to there?
Which file that example should be put.
Any other needed configuration files needed to get started.

I ended up to see just a lot of errors which does not make any sense.


Improve the blueprint graph a bit

Thanks a lot! Somehow I still haven't figured out the right workflow for all of this. I write the LaTeX, I write the Lean statements, I glue the Lean to the LaTeX, I wish it were a little easier, I sometimes only do half a job. Thanks a lot for doing the other half.


Links open on page instead of new tab

Please have links open in a new tab instead of on the same page. Having someone try and do informative readings or extended processes and then find their way back to the documentation is painful. For example, if I am reading this page and click the Azure portal link to create a new resource, it opens in the same tab and once I'm done creating my resource I have to find this page again to remember what I was going to do next.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: af826396-0492-1f0f-1fd0-09eaf166847c
Version Independent ID: 4d2bb00d-5c47-680d-dde5-452a955f7e16
Content: Quickstart: Personalizer client library for Python - Azure Cognitive Services
Content Source: articles/cognitive-services/personalizer/python-quickstart-commandline-feedback-loop.md
Service: cognitive-services
Sub-service: personalizer
GitHub Login: @diberry
Microsoft Alias: diberry

Same here; spotted this on other pages too and it's really annoying.

@bbentley19  @GiantCrocodile  Thank you guys for the feedback, I have assigned it to @Powerhelmsman for reviewing.

@Powerhelmsman Could you please check this issue? Thanks.

This is default behavior for the docs.microsoft.com platform. You can right click and open in a new page or window but this is a platform issue not an issue specific to this doc set. It would be better if you closed this issue and opened a new issue https://github.com/MicrosoftDocs/feedback/issues
#please-close


Update example.yml

updated artifact and checkout to v4

FYI, actions used in example.yml were updated in 6b67656 (build: update example to v4, 2023-12-24).


Brave: switching webui to window.ipfs

HTTP Client AND HTTP Server in embedded js-ipfs running in the context of the same background page introduces some overhead.

[ ] see if exposing API over window.ipfs (https://github.com/ipfs-shipyard/ipfs-companion/issues/514) is faster than HTTP API

Closing due to https://github.com/ipfs-shipyard/ipfs-companion/issues/589#issuecomment-712356448 and webui precache being implemented a while ago.


[VOTE] osmdroid v6.0.0 release

osmdroid has had a ton of new committers and contributors lately, Thank you all so much for your hard work! I feel that we are near ready for a release. It's been a long time since 5.6.5 and we are well past due. That said, osmdroid has never really had an governance processes (that I know of), I figured we should start probably start with v6.0.0's release. It stands now at 88 closed issues for this release.
If you're new to open source governance, it's worth the time to read some of ASF's guidelines for releases, specifically decision making. Normally projects require a majority of the committers to have a +1 vote with no -1 votes.
This is used for lots of purposes in ASF but most commonly it is used for voting on a release. In the past, most of the conversation regarding cutting a new osmdroid release has been through email but there's probably a better way. We currently stand at about 10 individuals with commit privileges. Opening this up to the community makes sense as people do fun and creative things with maps and we don't always have test cases for everything (nor do we really know how it is used).
The best we have for group communication right now is google groups, which hardly anyone uses, so the next best option is to just use a github issue. Since v6.0.0 has significant changes, I'd like to open this one up for the broader community to test and/or provide feedback before we cut the final artifacts. As of this writing, osmdroid has 129 people watching this project and over 1200 stars. That's a 129 people that will get an email regarding this vote. If you don't make your voice heard than there's not much we can do about it.
That said, it would be nice if most recent committers vote an opinion and/or test results. Snapshots are up to date on central. According to github, here's some of the recent contributors
@Maradox  @monsieurtanuki  @MKergall  @secaw  @pasniak  @cbalster  @miguel-porto  @shiroyagi
@neilboyd  @k3b  @kurtzmarc @jmarks213
So, when you have time, please comment below with a +1, 0, or -1 with explanation of what the issue is. Since everyone is probably super busy and working on open source is probably not a high priority for many, we'll keep the vote open for 2 weeks, ending on February 16, 2018 00:00 hrs UTC.
There is currently 1 pull request open which may resolve a bug with loading map tiles related to in memory caches. It's a rather large change set and although it may solve some issues, I have some minor concerns that there is a possibility that it may introduce unforeseen issues.
That's said, I'll happily give this a +1 from me.

+1 lots of good work, works well for me

+1 All seems to work well, I already have latest snapshots in production. An incredible amount of improvements and new features have been introduced! Special thanks to @spyhunter99 for your work and presence. Cheers.

All for it! +1

+1 !
and 100% in line with @secaw comment.

+1 (with a little hope that #776 is fixed - possibly by @secaw and #915)

+1 In a few sparse tests 6.0 worked great for me.

+-0 I am currently busy with other job. My app works stable with the old 5.x version

Yep, +1 as there have been great improvements in the cache management (we can finally go offline for days without the panic of losing our tiles!) and in the UX in general (very smooth zoom and pan), but I think #908 should be incorporated, cause it really seems to fix the hang-on-detach bug, which is a very inconvenient bug!
BTW thank you for including me in the vote!

+1 I tested 6.0.0 via the OpenStreetMapViewer as well as via my own application and I didn't find any bigger problems, besides the following issues (which are minor):

https://github.com/osmdroid/osmdroid/issues/929
https://github.com/osmdroid/osmdroid/issues/928 - Maybe not that important for 6.0.0
https://github.com/osmdroid/osmdroid/issues/933 - PR pending
https://github.com/osmdroid/osmdroid/issues/926 - PR merged

+1 I was using the map on and offline for a week or so, and haven't observed tiling hangs.

thanks for the feedback everyone. I'll work as soon as I can to get the result cut. As always, please continue to report issues

done


[6.0] Make it possible to have full classnames in route

After @freekmurze's talk at laraconUs, it inspired me to wanting to cleanup our routes, but as we have over 483 routes I was toying with the idea to make both worlds work.
This results in a fully backwards compatible route declaration. I have checked that any() and resources() work as expected.
Benefits

FQDN in routes declaration
Make it easier to navigate around
Helps IDE to find usages for refactoring, where before it couldn't

Result
Devs can add normal routes with an action:
Route::get('/user', 'UserController@index');
Or like @freekmurze suggested to use the FQDN where the controller class uses __invoke
Route::get('/user', App\Http\Controllers\UserController::class);
This can be cheated today by adding a global namespace indicator, but doesn't look good.
Route::get('/user', '\\' . App\Http\Controllers\UserController::class);
But how?
By checking if $action is a FQDN, note that this could fail if someone users the namespace App in their controllers folder.
I'm PR:ing this to master as we're getting close to 6.0 and there is no rush to get it into 5.8.
Failing unittest
getAppNamespace doesn't seem to work in console environment. Tested with src/Illuminate/Console/DetectsApplicationNamespace.php trait as well, but it does the same as app()->getNamespace().
Think I need some pointers on this one to be able to fully test changes.

But you can also use [App\Http\Controllers\UserController::class, 'index'] which gives more benefits from IDE than any point provided. For example if you click on index in UserController, it show rotue usage and location, and if you click 'index' in route, it goes directly to the method (at least PhpStorm does).

But you can already use [App\Http\Controllers\UserController::class, 'index'] which gives more benefits from IDE than any point provided. For example if you click on index in UserController, it show route usage and location, and if you click 'index' in route, it goes directly to the method (at least PhpStorm does).

Yeah I'm sold, this is exactly what should have been used. Think it's better to add a PR for documentation as I can't seem to find it on the routing page?

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.


Perf: Static delegates

Now we've got c#9 i've audited all lambda expressions in the library and changed some of them over to using static delegates which allows the compiler to cache the delegate on first use.
I also adjusted some other callsites to use state parameters to reduce the size of the closure slightly or allow onFailure, or onCancellation delegates to be static.
I tried to apply a consistent style to all the lambda usage layout spacing named params etc. It feels better to me overall but let me know if you want a different style.

I think you are confusing anonymous methods and expression trees both of which can be created using lambda expressions.
Anonymous methods are compiled by c# compiler once at compilation time, while expression trees can be compiled to a delegate at runtime.
I've created small example to demonstrate how anonymous methods are compiled:

You can also investigated that using windbg tool which shows the method table information for a given type-object.
Static anonymous function has been introduced to restrict the access to instance and local variables if it is not intended so by the design.
However, one of the main benefits of anonymous delegates and local functions is the possibility to capture the variables and instance fields of a method so you don't have to pass them as parameters.
And this is my point here: you are increasing the code base to remove the use of the feature which has been introduced to decrease the codebase:

Captures contribute to the size and number of hidden classes generated by the compiler. Passing using existing state parameter placement decreases or obviates the need for captures at all. Using static lambdas allows the compiler to cache the delegate again avoiding per-call allocations.
I do know what I'm doing here. Try running a memory profile of an async test case with and without the PR and see what difference it makes.

Captures contribute to the size and number of hidden classes generated by the compiler. Passing using existing state parameter placement decreases or obviates the need for captures at all. Using static lambdas allows the compiler to cache the delegate again avoiding per-call allocations.

If you are talking about delegate object itself then does it make sense to do optimization in this code? Because probably even the average connection string is several times bigger than a delegate and captured state together.

I do know what I'm doing here. Try running a memory profile of an async test case with and without the PR and see what difference it makes.

Could you please share the result of your experiment?
(because from your PR the only metric seen at the moment has negative impact)

If you are talking about delegate object itself then does it make sense to do optimization in this code? Because probably even the average connection string is several times bigger than a delegate and captured state together.

Each connection string is allocated once and re-used. The paths through these methods allocate on each call. If you call them once you'll see a very minor improvement, if you call them thousands or millions of times that adds up considerably. It's a question of scale.
Results from running the https://github.com/aspnet/DataAccessPerformance which is an implementation of the TE Fortunes benchmark.

Test
Type
Threads
Througput
StdDev
branch

ado-sqlclient+async+16
async
16
26411
1040
perf-staticdelegates

ado-sqlclient+async+16
async
16
26165
1101
main

So a 0.9% throughput increase from some very minor changes.
These changes barely impact the size of the code in comparison to the overall size of the codebase. In general the size of the code is far less relevant than the correctness and speed. If your main concern is code size at the driver level I thing your priorities are wrong.

@cmeyertons there are a number of changes in bulk copy here and they may improve the performance enough to outweigh the problem in https://github.com/dotnet/SqlClient/pull/1048 if you want to give them a try.

SqlBulkCopy.WriteToServerInternalRestAsync : Closes over 4 parameters including this. We can't avoid the allocation and a 4 tuple is less readable than the closure. I haven't profiled SqlBulkCopy scenarios so I don't have information that makes me think that changing this is beneficial at the moment.
GetParameterEncryptionDataReaderAsync: The result is cached so it isn't per-call. I haven't started profiling always encrypted scenarios yet so I don't know if this is hot, the cache means it should not be.
OnError: only 2 parameters so it could be tupled easily and fairly clearly but the allocation still can't be avoided. We're about to throw an exception which is going to allocation the exception, it's message, error collection, error collection contents, stacktrace. Attempting to save memory at this point only hurts code clarity it does nothing to benefit the user or us.

In general the more parameters are captured by a closure the less likely it is that it is worth converting. Tuples hurt code clarity and we can deal with some of that for improved performance of users but there is a limit somewhere and more than 3 parameters is where I currently feel readability suffers more than users benefit. This can in change (and may have done since I've been working on perf) with new information like profiling results and discussions.
The changes made to convert to static delegates in this PR were not driven by profile results so they should retain clarity as much as possible while improving performance. I think I would only reduce clarity further if I had a very strong argument that it would improve performance.
Let me know what you think and whether I've gone too far in reducing clarity. There may be cases where I've converted to tuples in the past that it might be better to use closures for clarity.

/azp run

@Wraith2 Can you trigger a build by making a small change in your PR and make the tests to run once more? I have tried from our side and somethings are missing...

Fixed up a couple of anonymous typed lambdas.

LGTM, I am waiting for the Enclave tests to finish and see if  the failure is fixed after I cleaned up the databases.

I'd like to get https://github.com/dotnet/SqlClient/pull/934 https://github.com/dotnet/SqlClient/pull/925 and https://github.com/dotnet/SqlClient/pull/902 finished to unblock other work if possible.


Tooltip label is too dark in v10

See screenshot. "carbon-components": "9.84.12".
bx--tooltip__label color is too dark. The Carbon website and Storybook examples do not use this class.
I'm assuming this class name has not been deprecated since it's in the CSS. Either the color didn't get updated with the v10 tooltips went dark? Or the wrong color token is getting used?

Seeing the following in the-carbon-components.netlify.com:
.bx--tooltip--definition__bottom p, .bx--tooltip--definition__top p {
  ...
  color: #ffffff;
}

So I think the problem has been fixed.

Oh, this was invalid use of bx--tooltip__label. I shouldn't have been using it inside bx--tooltip. Not a bug.


libolm issues on regular Debian Buster 10

Hello,
Many thanks for this bot.
Unfortunalty, it can’t work on regular Debian Buster 10 fresh install even with backport repo.
Conficts around libolm2, libolm3 and python-libolm can’t’ be easily resolved.
Switching to docker build allow to run, after minimal change, linked to *.cfg to *.yaml in dockerfile ...
Best,

@nunux13  Sorry to hear that.
libolm2, libolm3 and python-libolm are a bit out of my reach ;)
Maybe the most appropriate thing would be to mention that on https://github.com/poljar/matrix-nio
I strongly suggest you mention your concerns and issues (e.g. copy paste this issue) as an issue on
https://github.com/poljar/matrix-nio  Matrix-nio is the underlying base for the libolm stuff. They might be able to
help you out and improve in this area.
Regarding the Docker container: You said you made a minimal change.
May I ask what? Do I need to update/adjust one of the Docker config files in my repo?
Which file? Which change? Feel free to share, or even better to make a PR so we include your feedback in the code.
Thanks again @nunux13  for sharing and pointing out these issues. Greatly appreciated!

The issue here is that in buster-backports the development package (libolm-dev) has the same name as the package from buster itself. So "apt install libolm-dev" will install the package from libolm2 (buster), which is not compatible. The easiest solution is to install that package manually: download from https://packages.debian.org/buster-backports/amd64/libolm-dev/download and then "dpkg -i ".
The only downside to this approach is that any future updates related to libolm3 in buster-backports will not lead to an updating of libolm-dev via "apt upgrade"; instead the new version must than be downloaded and installed.

PS: don't install python3-olm from buster (apt/dpkg) - install it from pip ("pip3 install python-olm")

I also was confounded by this.  What I did (Raspberry Pi 2B 1.1, armv7l) was to (sic):
First add deb http://ftp.debian.org/debian buster-backports main contrib to '/etc/apt/sources.list', then in bash sudo apt-get update.  This didn't work as I had to add the Debian public key(s).
That was: sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <key> - for two keys.  The Ubuntu keyserver worked but the Debian keyring didn't.  I hear it's a depreciated method but it worked nonetheless.  I tried rearranging my resolv.conf entries, etc as well.
Then I used the command sudo apt install -t buster-backports libolm-dev, which removed libolm2 (previously tried, and available in the raspberrypi.org/raspbian/ buster main, repo) and replaced it with libolm3.
As @juergenpabel has mentioned, there's likely to be issues in the upgrading of these packages, or perhaps their compatibility, but I'm not knowledgeable enough to comment on that.

Thank you @juergenpabel
Your comments are very helpful. Thank you for taking the time to writing them down here to help everybody in the community! Appreciated. Also, as far as I can tell all your comments are factual and correct.
Just to add another sample point: I did a test installation on Fedora-33.
pip3 install --user python-olm
Collecting python-olm
  Downloading python-olm-3.1.3.tar.gz (27 kB)
Requirement already satisfied: cffi>=1.0.0 in /usr/lib64/python3.9/site-packages (from python-olm) (1.14.1)
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 5.2 MB/s
Requirement already satisfied: pycparser in /usr/lib/python3.9/site-packages (from cffi>=1.0.0->python-olm) (2.20)
Requirement already satisfied: ply==3.11 in /usr/lib/python3.9/site-packages (from pycparser->cffi>=1.0.0->python-olm) (3.11)
Using legacy 'setup.py install' for python-olm, since package 'wheel' is not installed.
Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
Installing collected packages: future, python-olm
    Running setup.py install for future ... done
    Running setup.py install for python-olm ... done
Successfully installed future-0.18.2 python-olm-3.1.3

As you can see it installed version 3 of pythom-olm (which is correct and also required, v2 will NOT work).

Thanks @QuailQuark for chiming in and telling us about your use case. This is again slightly different as @QuailQuark is using a 32-bit OS (Raspberry Pi 2). Right?
To have it auto-upgradable and if you want to test, @QuailQuark you could do the following:

uninstall libolm with apt/dpkg
install with pip3 install --user python-olm or pip3 install python-olm
I expect that everything remains working, and olm3 will be automatically updated when you update your pip packages.

If you try this suggestion, please report back if this was successful for you, thanks @QuailQuark

Sorry for the pause @8go, yes, that's what I've installed this project on.
Sure, I copied your instructions and I've found python-olm 3.1.3 is installed and pip reports it as 'Requirement already satisfied...'.
The bot continues to work, and all seems fine.  Thanks!

Hi @8go, just to add insight, the bot service failed at some point.
Reinstalling libolm3 from the debian apt repo has resolved this issue.  So that was sudo apt install -t buster-backports libolm3.
Python-olm-3.1.3 is also installed when executing pip3 install "matrix-nio[e2e]".
It seems both are required for this project to run.
If there are thoughts on this I'd like to hear them - but this is a libolm issue!
Thanks

Thanks @QuailQuark for reporting back and sharing your results.
This will help other people.
I think this is specific to Debian 10. Most platforms will work fine with just the Python 'python-olm` installation.
If there are still problems with libolm, do as @QuailQuark and additionally add libolm3 via apt.
Closing issue. Thanks to all contributers: @nunux13 @juergenpabel @QuailQuark


Pivot Columns v2

For one instance, you have a user that subscribes to products and product can have many subscribers.
type Product struct{
   gorm.Model
   User []User `gorm:"many2many:user_product;"`
}

type User struct{
   gorm.Model
   Product []Product `gorm:"many2many:user_product;"`
}

What if I wanted that the table with the user_product name has another columns other than the id of the user and product which is typically like this.
USER_PRODUCT TABLE

user_id
product_id
other_columns

1
2
something

2
3
something_else

The other_columns is what I mean by pivot column. Is there a way that I can access this column from the gorm library?

refer: https://github.com/jinzhu/gorm/issues/719#issuecomment-168485989


Implement a FuzzReader which could return error at any time

Replaced by https://github.com/datafuselabs/opendal/issues/1270


Add NooBaa to support matrix

Per https://github.com/heptio/velero/pull/1334#issuecomment-479190807

Thanks for this, @cwilkers. The other thing I'd say is that if you want to post the sample YAML and/or documentation for Velero with NooBaa, we're happy to have you do that in an external repo under your username/org, and add a link to that in the support matrix -- we just can't commit to maintaining these things ourselves going forward, so we're not ready to include them in the core repo. Hope that helps!

Looks good, feel free to follow up with another PR pointing to a separate repo if you'd like!


Emmet wrap with abbreviation dialog should go away when it loses focus

#8207

VSCode Version: Code - Insiders 1.3.0-insider (d0c2b891903d73e83dcdc7786398ebfa764fc4ee, 2016-06-28T05:01:27.563Z)
OS Version: Linux x64 4.4.0-24-generic, Ubuntu 16.04

Steps to reproduce:

Trigger Emmet: Wrap with Abbreviation command
Focus editor

This gets particularly bad when you try to use the command palette now:

The same applied top the decode data image uri command.

As I understand it, the inputBox (quickOpenService.input) has no automatic closing when you change focus? I found this opportunity only for Pick.

I've also checked other uses of IQuickOpenService.input, e.g. git commit and they expose the same behaviour, e.g., the quick box remains open also when it loses focus.
Assigning to @bpasero for further investigation.

@bpasero, it seems to me that this is «hard code»:
this.doPick(TPromise.as([{ label: currentPick }]), {
	ignoreFocusLost: true,	// <------------------------- this
	autoFocus: { autoFocusFirstEntry: true },

I created a branch in my fork. Maybe this will help you. Link to commit. This option is not described in the documentation.

Moving to @jrieken who added this in https://github.com/Microsoft/vscode/commit/7a626a94ea1b0b9c6b5f43f1b0038cccb9da0fef


test: removed TestRunAdvancedExampleUpgrade upgrade test

Description
removed TestRunAdvancedExampleUpgrade upgrade test
Release required?

[x] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

Closing in favor of https://github.com/terraform-ibm-modules/terraform-ibm-icd-redis/pull/413


Send reply in event of a premature exit (via throw) from fold.

Allow a folder to throw its accumulator to send it to the fold client, e.g. hinted/repair/ownership handoff.

Not building 1.4 any more - presume this P/R can die.

Did this get merged into 2.x or does it have relevance for 2.x?

https://github.com/basho/riak_kv/pull/869
borshop  merged 1 commit into develop from bugfix/handoff-fold-abort-early-2.0 on 29 May 2014
Then looking at the file in 2.0 branch, the changes are present.
https://github.com/basho/riak_kv/blob/2.0/src/riak_kv_worker.erl#L50

Kill it with fire then ;-)

Can one of the admins verify this patch?


Add terminalGroup to tasks to allow running them in split panes

This is an implementation of https://github.com/Microsoft/vscode/issues/47265, which is currently assigned to @dbaeumer. It adds a new field terminalGroup to the presentation option of the .vscode/tasks.json file. When supplied, all tasks of the same terminalGroup open as split panes in the same tab. Tasks without terminalGroup are not allowed to reuse terminals created by tasks which have a terminalGroup.
I haven't written tests yet, but can add them if you think the general direction of this PR is good.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: cmfcmf sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Fix contacts on safari and iOS

Fixes contacts alignment on safari and iOS
This solution was tested on iPhone 5 (Safari and Chrome), iPad2 (Safari and Chrome), HTC One (Android 5.1, system web view, chrome and firefox), Firefox, Safari and Chrome.
Old:

New:

:+1:


--lint_out: protoc-gen-lint: Plugin failed with status code 3.

I do have some errors listed but final file is not generated for some reason?

Hey @pawmart, That's actually the intended behavior. The error code (3 in your case) is the number of issues found. An exit code of 0 means it didn't find anything wrong with your protobuf. It doesn't generate any files, in only prints the errors to stderr.
What kind of file would you expect to be generated?
I'll close this one but please to comment about the file, we can spin it off into a new issue :)


Suggestions for possible improvements

Hi Franklin,
Thank you for this collection of indicators!
Here are some suggestions which you may decide to apply:

Design each indicator as a simple function
This kind of design should be considered because of multiple reasons:

Imagine having an infinite stream of data, this will allow to save memory and maybe even increase performance.
Having simple functions allows composition
Having simple functions allows easy testing and rewriting to different language.
Please let me give you quick example of simple SMA in Java that covers all three points above:

public static class SMA implements Function<BigDecimal, BigDecimal> {
   private final var buffer = new LinkedList<>()
   private final int length
   private var sum = BigDecimal.ZERO

   SMA(int length) {
     this.length = length
   }

   @Override
   BigDecimal apply(BigDecimal currentValue) {
        if (buffer.size() == length) {
            sum -= buffer.removeFirst()
         }
        sum += currentValue
        buffer.addLast(currentValue)
        // at the end calculated SMA should be rounded to SOURCE scale
        return roundValue(sum / buffer.size(), currentValue)
  }
}

So it is basically a function, that accepts decimal value and produces new one. It has some minimal internal state like a self buffer etc. Usage is quite simple, it can be applied now to any datasource, streaming or not. RSI from SMA will look like this:
var datasource = candlesticks.map(c -> c.close)
     .map(new SMA(3).then(new RSI(14)))

I think you may improve your code in this way if you agree.
2. Rounding of values is a problem now. You used 4 decimals by default in any calculation, but I think this is not correct.
Whenever you know the scale you should use it. For example any moving average should use scale of original sourced value. RSI should/may use just 2 decimals. SMA from RSI should have the same 2 decimals for calculated value.
3. I tried to use some cryptocurrency candlestick data and many indicators failed to generate any useful data, I even couldnt find simple signal for RSI with StrongSell/StrongBuy values, which is weird.
The most important part is point with number 1, I do not have much experience with C# to help you with that. Please let me know what do you think.

@eugene-kamenev Thanks for the great feedback! I do understand the use case for this functionality as far as moving to functions goes but my immediate thoughts were how would this work with streaming for example. If you were streaming stock data to a list then unless I'm mistaken, you would have issues with your example with not being able to access previous values and if you were streaming then you would have to calculate from the beginning of the list each time due to formulas like sma where you need to get the last X values for example. And of course there is an issue with the super complicated formulas that I have written like Ehlers stuff or some that are recursive in nature so I'm not totally sure how these use cases would look like. That being said, I do like the idea and I'm very open to increasing performance and decreasing memory usage.
I think you are totally correct about the rounding issue and clearly I will need to fix the rounding issues so that it is adaptive based on the source data that it is calculating based on. I'm open to ideas for how to do this concept but I will come up with some ideas for how to better do this in the meantime.

@ooples I would really like to help but, I just started learning C# code with your project. I am mainly Java developer, but I was able to rewrite some of your code to review the data it generates from most of indicators. I have some reflection based C# code to get all indicators list and generate the data, please let me know if it can help.
Now going back to functions approach. As I said above function can have a state, this means you can cache values, have some buffers defined, to collect history values you need, to produce next value. If you will look at SMA example above, it has a buffer of previous values to calculate current value. It is up to stream consumer whenever it is decided to cache values or do something with data. Function only results in a value, and we only expect new values of the source to come each time. All you have to do is to remove all loops you have to calculate indicators, extract each one into a function. Combine functions when needed. Then you should be able to  subscribe to some price datastream and produce indicator values in realtime.
In Java we have a BigDecimal class which has all information about scale/precision of the value. So I basically round the value to original scale. Something similar should be available in C#. I am not sure if you should pay attention to each operation or only round resulting value.

@eugene-kamenev Would it be too much to ask you to write some test code in Java with a slightly more complicated calculation like Weighted Moving Average for example and compare it to a java version of what I'm already doing to see if this will improve memory usage and speed? I'm open to making these changes in a C# version but would need to see proof that it would improve the performance enough to justify the many hours it would take me to make these changes. As far as the rounding goes, I'm going to fix this issue in the next version once I figure out a good way to handle it.

@ooples Just a quick rewrite of your version, hopefully logic is the same.
It still uses a loop for a buffer:
public static class WeightedMovingAverage implements Function<BigDecimal, BigDecimal> {
        private final LinkedList<Double> buffer = new LinkedList<>();
        private final int length;

        public WeightedMovingAverage(int length) {
            this.length = length;
        }

        @Override
        public BigDecimal apply(BigDecimal v) {
            if (buffer.size() == length) {
                buffer.removeLast();
            }
            buffer.addFirst(v.doubleValue());
            double sum = 0;
            double weightedSum = 0;
            int j = 0;
            for (var bufferValue : buffer) {
                double weight = length - j;
                sum += bufferValue * weight;
                weightedSum += weight;
                j++;
            }
            BigDecimal wma = weightedSum != 0 ? new BigDecimal(sum / weightedSum) : BigDecimal.ZERO;
            return roundValue(wma, v.scale(), RoundingMode.FLOOR);
        }
    }

Thank you @ooples

@eugene-kamenev The only difference I can see is that you are ordering by dates in descending order and I'm ordering by dates in ascending order. Let me know when you have a java version of how I do my code so we can see which version uses less resources and runs faster when you can do a side by side

+1 on the suggestion to decouple all the indicators from the StockData class and design them as stand-alone functions.
This way, they can then be used with any arbitary series of data.
Right now, it feels a little un-intuitive to do everything through the StockData class.
(I've just started exploring this library, so please excuse me (and correct me) if I've made some wrong assumptions here.)

@ooples, for certain indicators, I can understand that require specific data points, but if we look at basic indicators such moving averages, there could be a need to calculate MA of literally any data point, for e.g. "Average Price", "Total Traded Value" etc. And maybe even MA of RSI, who knows!
So, I think it would make sense to expose a way to do this with any arbitrary user supplied data points.

@1nfected I understand what you are saying and I have contingency plans using the StockData class to take care of all scenarios you have listed which is why I'm using it for this project. My library can calculate indicators of indicators of indicators using any combo of inputs values and output values using my StockData class. You can use my code to do Ma of Rsi or whatever combo you want


Idea: consider merging this repo into depthai-core?

This xlink fork seems only used at the moment in the depthai-core library.
Is this library used anywhere else? If not, it might be simpler to include this xlink code in a subfolder in the depthai-core repository.
Notable benefits:

Easier building without usage of hunter
No seperate versioning of xlink and depthai-core, so no configuration management of the compatibility between the two

Downsides:

xlink cannot be used stand-alone (is this an actual use case at the moment?)

This is an idea I had when browsing through the code-base, but maybe I'm missing something? Is there any reference to the original xlink library? I was not able to find it.

Its only used by depthai-core as of right now. The fork is from: https://github.com/openvinotoolkit/openvino/tree/master/src/plugins/intel_myriad/third_party/XLink
As of right now, the usage is equivalent to other core dependencies, and I think is adequate as the code isn't as relevant to core. (I guess if we'd work more on it, we could merge it in)

Thanks for the upstream link, that's useful!
Main advantage of merging xlink into the depthai-core repo would be packaging effort. If depthai-core is to be packaged as deb/rpm/pacman/emerge/bitbake, then each recipe will probably not use hunter, and will have to face this extra repository. Since xlink is only used in depthai-core, there is no real benefit in having it as a seperate reposiroty.
One additional benefit of merging xlink is being able to browse code by opening the depthai-core in vs-code, if xlink is there, the headers are found, and code can be navigated easily. How is this done at this moment?
I hope I can convince you to merge xlink into depthai-core, as I created a bitbake recipes for depthai-core, and its dependencies, and it is in my interest to reduce the amount of packages / recipes I have to create :).

Has the status changed? I am looking to connect to the OAK cameras via C#.  I don't really care to use depthai-core unless I need to. If this repo is current and provides everything needed to talk with the cameras standalone, that's what I'm after.  So, specific questions.

Is this repo everything I need to establish comms with the cameras, stream video feeds?

beyond this repo, is there anything in depthai-core I'm likely to want as well?  I do not care about doing any AI for VPU-level processing.  I just want to get the video / data feeds.  Perhaps core does something additional like encode the data?  Or higher level abstractions pertaining to the device object model?

@aggiehorns the status have not changed yet. No, the core is the main requirement, XLink is only the "protocol" / communication layer / library that core relies upon.
We might merge XLink into Core one day, if we C++ify it also, but only when we iron out some issues that it has at the moment (or we'll submodule it).

Gotcha, and I now see it's pure C.  There's nothing about that to prevent including it in the same core repo though.  Is there something I'm missing?  To build core I just need to pull this repo as well and manually copy the files into the core directory structure so they can be included....?

@aggiehorns yes, not an issue per se, just that if we'd merge codebases, I'd bring it under dai namespace, etc...
You may do this already, clone wherever/submodule/etcc and then do cmake [build/dir] -D XLINK_LOCAL=[path/to/xlink]

Rewrite to C++ sounds like a good idea, and a good first step in this direction is in my opinion to merge the xlink repository as-is into this repository. This makes this transition simpler as well.


Put react-scripts back into dependencies

Note: this PR reverts a part of the changes from https://github.com/WonderTix/WonderTix/pull/617
Description
Due to the build process that we currently use, react-scripts must be labeled as a dependency within client/package.json. Currently the build process runs a npm install --omit=dev so that the build process can be quick. This means that react-scripts must be considered a normal dependency to run npm build when a production binary is created.
Fortunately, it looks like all of the "vulnerabilities" that npm audit displays for react-scripts are false alarms, and when npm build is run, it is doubtful that the package would even be included since it isn't used anywhere in the react code -- just in the build process.
In short, these vulnerabilities shouldn't be a problem, and we'll just have to ignore the "vulnerabilities" output stemming specifically from react-scripts. Output from other packages shouldn't be ignored though.
closes #536 again

Why does the client need react-scripts to build in some cases but not others? If no part of the client is using that package then why can't it be removed outright or the deploy process adjusted so that it can be removed?

Why does the client need react-scripts to build in some cases but not others? If no part of the client is using that package then why can't it be removed outright or the deploy process adjusted so that it can be removed?

With our build process, react-scripts is always necessary to create a client build. npm run build is essentially just a shortcut for react-scripts build. It looks like the following in the client/package.json:
"build": "DISABLE_ESLINT_PLUGIN=true react-scripts build",

react-scripts is essentially a build tool, so it's not necessary once the client is built, but is always necessary to build the client (at least because we're using it as our build tool).
There is probably more than one way to build a react-app, but that would require a bit of a dive to swap to something different.

react-scripts is essentially a build tool, so it's not necessary once the client is built, but is always necessary to build the client (at least because we're using it as our build tool).

Ah makes sense👍


unsound initialization logic with static values

This code prints <null>:
class Foo {
    shared static Foo instance = Foo();
    static value constant = "Gavin";
    shared String name = constant;
    shared new () {}
}

print(Foo.instance.name);

That's clearly unsound.
One solution to this is would be to:

require that constants, that is, static values defined with =, be defined before static functions and static values defined with =>, and
prohibit calls to constructors from the initializer of constants.

Not sure if maybe there's some other less heavy-handed solution.

Note: in principle, these restrictions need not be applied to late constants.

prohibit calls to constructors from the initializer of constants

That would be very sad.
This seems to be backend specific–it prints "Gavin" on JS & Dart. On the JVM, a static initializer is used to set constant.
I think it would be very natural to have the rules be the same as the code below, and make backend changes as necessary:
Foo instance = Foo();
String constant = "Gavin";

class Foo {
    shared String name = constant;
    shared new () {}
}

Well, this is one of the classic java puzzles, but translated to Ceylon.

require that constants, that is, static values defined with =, be defined before static functions and static values defined with =>, and

May this be the staticequivalent for restrictions in an initialization section? Shows regularity.

prohibit calls to constructors from the initializer of constants.

Yup. Like in java, we should disallow calling any non-static method from a static context. And this includes constructors.
John, I guess in Dart & JS it is working because = is more like => than in Java.
So nameis resolved later, when constantis already initialized. Just guessing.
If I understand your proposal correctly, you are proposing solving static attributes on first place, and then non-static ones.
I always understood java statics as something that is resolved the moment the class (as opposite to the instance) is initialized. So yes, it makes sense for me all statics are solved on first place.

Yup. Like in java, we should disallow calling any non-static method from a static context. And this includes constructors.

Nope, we definitely shouldn't do that, since that would prevent one of the main uses of static: writing factory methods.

Mmm... factory methods tend to be static, so it usage will be  static->static or non_static->static.
What a I am proposing to disallow is static->non_static calls.
Right, some factory methods can be non-static. Allowing those case is pretty more complicated, as imply class hierarchy initialization order. I go deeper on this when I got a computer.
Anyway, while writing an example for my proposal, I reach the following situation:
class A() {
   static value va = B.vb;
}
class B() {
   static value vb = A.va;
}

Those are clearly undefined (so should be disallowed), but are ok with the static initialization check rules I've seen so far.
Are we currently accepting this?

@someth2say just to point out, the circular assignment is perfectly legal for top-levels, but with a runtime exception, and I fail to see a significant distinction between statics and top-levels.

I've also always thought of value constructors as (lazily initialized) values of the class's containing scope, which in the example below would be top-levels, and that in theory they should follow the same initialization logic as top-levels (i.e. throw instead of null):
class Foo {
    shared Bar b;
    shared new instance {
        b = Bar.instance;
    }
}
class Bar {
    shared Foo f;
    shared new instance {
        f = Foo.instance;
    }
}
shared void run() {
    print(Bar.instance.f.b); // null!
}

Mmm... factory methods tend to be static, so it usage will be  static->static or non_static->static.
What a I am proposing to disallow is static->non_static calls.

Static factory methods have to be able to get an instance from somewhere, by calling a constructor of the class.

@gavinking You are right, I was thinking only about using the factories, not about the factory code itself.
I should stop writing comments after family dinner...
I think we can relax 2 a bit:

prohibit calls to constructors before any static value declaration . In other words, prohibit calls to constructors but at the end of the static initialize section.

This way we confirm class (not instance) is properly initialized before creating any instance.
My only concern is what about static late values.
It's a bit surprising this is actually working fine:
class Foo {
    static late String constant;
    shared static Foo s_instance = Foo();
    constant = "Gavin";
    shared String name = constant;
    shared new () {}
}
shared void run(){
    print(Foo.s_instance.name); // prints "Gavin"
}

So seems not something to care about right now.
On another topic: @jvasileff, I'm really surprised we allow things like
Integer topA = topB;
Integer topB = topA;

But that's not the worst. I updated my cross-dependency example, and found the weirdest:
class A {
    shared static Object va = B.vb;
    shared new () {}
}
class B {
    shared static Object vb = A.va;
    shared new () {}
}
shared void run2(){
    print(A.va); // Prints '<null>'
}

I would have preferred the InitializationError!
Even if replace Objectby Integer, it prints "0", so evidently static values just keep uninitialized.

I would have preferred the InitializationError!
Even if replace Objectby Integer, it prints "0", so evidently static values just keep uninitialized.

Yes, that's why I think all of this is a backend issue, with static members not being initialized in the same manner as top-levels.
On the Java backend, you can also, in a sense, circumvent top-level initialization checks by-way of a static member. Note that the top-levels may not even be in the same file.
shared void run() {
    printAll { A.s, toplevel1, toplevel2 }; // null null null
}

shared String toplevel1 = toplevel2;
shared class A {
    shared static String s = toplevel1;
    shared new () {}
}
shared String toplevel2 = A.s;

Note that this came up again in the discussion on #7294. There is an asymmetry between value constructors and callable constructors, in that only callable constructors can be used from a static method or value initializer. That's good, because it lets you write static factory methods, but it's also bad because it's the source of this unsoundness.


Users not being displayed on user admin panel

A user is being returned without a name from the data base which is causing none of the users to be displayed.

Solved. It was caused while changing a user's email address. There should probably be a way to do it without directly editing the database, but these are so rare that it's not worth the time at this point.


move Button css to docs

move .Button css to /docs folder so that it’s not pulled in by the app
gem.

@paddyduke shifted Button into /docs. could we also remove all the .btn classes here perhaps? everything up to line 419 in the docs/_buttons.scss file


[CON-99] Things I want as I use Continue

[ ] Give the model a page of documentation. Either could be a part of the explicit context selection, or a slash command
[X] Insert at cursor button
[ ] Look over the changes I just made in this range: were there any accidental regressions
[ ] Use @ to require user to specify when they are highlighting new ranges, otherwise cmd+f highlights a range
[ ] I want not to have GPT automatically take actions without me knowing beforehand. It's just springing them out on me. Even if I can reverse them, I don't immediately feel that trust and it's too large a barrier before I learn to trust. Would love links in the chat to take actions. Then can just click on them, makes it easier to read
[ ] Instead of just copy button on code snippets, an "insert intelligently" button that will place the code where it is needed even if not a clean insertion.
[ ] Often I want to see the documentation. Would be nice to have links provided, have the model do the searching for me.
[ ] Autocomplete in the textarea
[ ] I wrote some random note about why we use version 1.67.0 of @types/vscode and this is rarely ever going to be seen by someone, so want Continue to "just know about it".
[X] Our font is slightly hard to read and small. Slightly.
[X] Stop button instead of X
[ ] Write my commit messages for me (/commit)
[ ] When it goes wrong, I want to be able to just restart from scratch with something like a restart command
[ ] Maybe the best way to use functions is to have suggested actions that can be clicked at the bottom of chat output. Like "create tsconfig.json" or "run npm i react"
[ ] I want the code explanations to be much better; they are not super useful rn--I don't find myself really using them. probably gpt-3.5-turbo can read diffs and summarize. This is the way

From SyncLinear.com | CON-99

+1 for "insert intelligently", like the autocomplete/suggest done by Codeium or Copilot


How to view trace messages from the driver?

Hi, I'm currently developing a firmware that uses CDC-NCM over USB. On Linux and macOS, the device works perfectly. It enumerates without any problem and ICMP ping messages go both ways.
For Windows, I first had to add the "MS OS 2.0 descriptors" to automatically load this driver. Even then, the Windows Device Manager reports that the "device failed to start" with an error code of 0xC0000483 (which according to http://errorco.de/win32/ntstatus-h/status_device_hardware_error/0xc0000483/ means "STATUS_DEVICE_HARDWARE_ERROR"). In the source code of this driver, I can find many references to that error message, all of them in "device.c" and mostly related to the USB descriptor tables.
I am running Windows 10, 64 bit.
Since the Device Manager only displays the error code and not the associated "message", I investigated a bit on how I could access them. This brought me to "TraceView", so I installed VS 2019 and the WDK SDK 10. When starting TraceView, I've added a new report source and added this driver's Control GUID (from https://github.com/microsoft/NCM-Driver-for-Windows/blob/release_2004/inc/trace.h#L25). TraceView reports that the trace is running but when I plug my USB device, the "events counter" stays at 0.
Is there anything else I have to do to get the driver's trace messages?

Update: I can now see events coming in in TraceView. The problem was that I had "flags" still at 0x0. With 0xFFFFFFFF, events are coming in. However, it only shows GUIDs and cannot decode them, presumably because I'm missing the TMH files.
If I compile the source code myself, I assume the resulting TMH file won't help me much since the GUIDs will be differrent, right?
Thus, are the TMH files of the UsbNcm shipped with Windows available somewhere or could you please share them? An XML based Manifest seems to be an alternative as well ...

Hey, NCM driver uses WPP for tracing.  You can compile the source code yourself and load that NCM driver you built for the device in question, so this way TraceView should be able to decode the traces from your private NCM driver.


Update custom-domain.md IP Address FAQ

@kengaderdus - can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

@kengaderdus - quick nudge.
NOTE: the Validation suggestion here:  will need to be addressed before we can merge this. I made a change that might fix it.

@kengaderdus,
Can you review this old PR and determine whether it needs to be closed or merged?

#sign-off

Merging now.


E2e testing

Adding protractor and getting it to run on circleci

This also is an upgrade from node 4 to 6.10.1


default and initial of a property should be per instance and not per class

Opening up an issue to discuss why default and initial values of a property are set per class and not per instance.
@theengineear @klebba

@codeStryke I'm not sure I follow, you would have to propose an alternative approach. How would it work to configure these per instance at the class definition of the element?
You can override the class level definitions on your instances like this:
...
// property definition in class
{
  myNeatProperty: {
    type: String,
    initial: 1,
  },
},

// html
<my-cool-element my-neat-property="4"></my-cool-element>

// javascript
const element = document.querySelector('my-cool-element');
element.myNeatProperty = 4;

I should have clarified, when the default or initial value is set via a function, then, it should be per instance and not at the class level.

Ah ok, understood; that sounds like a bug. Thanks for reporting!

Man, I'm really losing it today — forgot to close this from my PR... This is closed by #81.


Automate updating translation template files

At the moment, updating kolibri-gnome.pot with new strings is a manual process, which makes it very easy to forget. Instead, consider adding a pre-commit hook or a github workflow which does this automatically.

For reference…
To update pot files, you need to have a terminal open inside the build directory generated by meson, for example using New Build Terminal in GNOME Builder, and then run this command:
ninja kolibri-gnome-pot

… and this command to merge that change with all the PO files (although that might not be necessary depending on how we talk to Transifex?):
ninja kolibri-gnome-update-po


Not generating warnings for empty directories

I have a bids dataset with empty anat directories. It passed bids validator test without reporting for these empty anat directories or other empty directories.

We may only be able to address this issue for the node use case. When you select a directory in chrome it returns a list of all the files within. Sub-directories can then be determined because each file has a relative path within the selected directory. In the case of empty sub-directories, no file is returned that includes their path and as far as I can tell there's no way to get information about their presence.
Were you using the web validator or the node version?
@chrisfilo let me know if you have any thoughts about whether it's worth addressing this in node and not the browser. And are empty directories allowed by the standard? I didn't see anything explicit about it at a glance.

The standard is not clear on empty directories, but they are usually a sign
of something going wrong (data that accidentally are missing).
I would say it's worth adding a warning, but not high priority.
On Mon, Oct 3, 2016 at 4:56 PM, constellates<EMAIL_ADDRESS>wrote:

We may only be able to address this issue for the node use case. When you
select a directory in chrome it returns a list of all the files within.
Sub-directories can then be determined because each file has a relative
path within the selected directory. In the case of empty sub-directories,
no file is returned that includes their path and as far as I can tell
there's no way to get information about their presence.
Were you using the web validator or the node version?
@chrisfilo https://github.com/chrisfilo let me know if you have any
thoughts about whether it's worth addressing this in node and not the
browser. And are empty directories allowed by the standard? I didn't see
anything explicit about it at a glance.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/INCF/bids-validator/issues/205#issuecomment-251260479,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp8lEQipHGpfKJ1a35ReJYARNrWniks5qwZY4gaJpZM4KMyHq
.


Full localization support

Great library, but not everything I want is localized, so I had multi-language views. I think we should update these localized strings
add .localized to:
"Close"
"Hey, listen!"
"We need a couple things\r\nbefore you get started."
"Permission for (permission) was denied."
"Please enable access to (permission) in the Settings app"
"OK"
"Show me"
"(permission) is currently disabled."
"Please enable access to (permission) in Settings"

Thanks, I missed some messages and also most of the new ones weren't localized


Project Maintainer Needed

If anyone is Interested to maintain this project properly, I will give the collaborator access to that person. I will be busy in my college for next few months, I don't want the contribution made in this project to be wasted. If anyone wants to get into open source through maintaining this project properly, I am more than happy to welcome.🙏🏼
Goal should be to make ZeroOctave-Javascript-Projects  a project for beginners where they can contribute small projects and improve by practicing basic web development skills.
Why you should maintain?

Get familiar with project maintainer role and learn to review codes.
Participate in open-source programs like GirlScript Summer of Code, DWoC and many more as maintainer or project Admin.
Collaborate with fellow contributors.

Expectation from you!

Don't Change Project directory structure.
Only accept project with HTML, CSS and JS (Not Js Frameworks)
Maintain repository and review codes properly.
Don't accept PRs of similar project instead ask contributors to improve existing ones.
Website and README should be updated properly.

Eligibility

You must be familiar with Web Development technologies required for this repository.
Working as contributor in the past for this repository will be good consideration.
You should have knowledge or any prior experience of code reviews on Github.

Note: Beginners can also join if they feel they can get the work done.

Yes i am ready to maintain this projects

What is the update. @Astrodevil
Will you add me as a collaborator?

What is the update. @Astrodevil
Will you add me as a collaborator?

Yes,for sure. Are you comfortable with the repo? I will also help you in managing sometimes.
Will also add 1,2 collaborators soon, as I can find any interested folks.

Yes i am ready to maintain this projects

Go through the repo once, tell me if you can manage it properly, so that I can give access to 2,3 folks as collaborator to make things easier.

@Anmol-Baranwal Are you ready to maintain? Please confirm so that I can give you collaborator access.

Yes, please give the collaborator access at the midnight. I want to see the repo once again for the clarifications.

Yes we can update README and ask new contributors to update this and cards.json properly for each new project.
And for discord yes I have one discord channel but inactive, I will add you and others there.

Join here: https://discord.gg/TH7hWD4Hqz @Anmol-Baranwal

Yes I am very much interested and to maintain this project

👋 Hello, I'm interested in managing the project, I just go through The source and I think am comfortable with it

@Nafiu29 @Emerald1996 I would like you to review some PRs properly for this repo. So that I can get an overview of your maintainer skill.

I'm also interested

I'll be happy to take up this opportunity

You can join us a mentor through GSSoC program. Apply here: https://gssoc.girlscript.tech/


Patch retrieve_work method than a subclass can support all sidekiq fetch classes

Patch retrieve_work method of the user chosen fetch class or the default BasicFetch, so it supports different sidekiq fetch classes

Let's submit a patch for sidekiq-rate-limiter once we're happy this is all working properly.


Lazy.js optimization

Removed dublicated functions and fixed any types.

@GiedriusGrabauskas Thank you for your contribution.
But please fix the some error on Travis CI: https://travis-ci.org/borisyankov/DefinitelyTyped/builds/86807236#L193