HDF error when trying to write Dataset read with rasterio to NetCDF

I'm getting an HDF error when trying to write a Dataset read from GeoTiff (rasterio backend) to NetCDF. See reproducible example below:
import urllib.request
import tempfile
import os

import xarray as xr

path = tempfile.gettempdir()
url = 'https://earthexplorer.usgs.gov/browse/gisready/landsat_8/LC08_L1TP_026047_20180110_20180119_01_T1.zip'
filename = os.path.join(path, url.split('/')[-1])
nc_name = os.path.join(path, 'landsat_rgb.nc')

# Download file if not exist (11 Mb)
if not os.path.isfile(filename):
    urllib.request.urlretrieve(url, filename)

# Read rgb file using rasterio backend
rgb_name = '/'.join(['/vsizip', filename,
                     os.path.basename(filename).split('.')[-2] + '.tif'])
ds = xr.open_rasterio(rgb_name)
ds = ds.to_dataset('band').rename({1:'blue', 2:'green', 3:'red'})
print(ds)

# <xarray.Dataset>
# Dimensions:  (x: 7611, y: 7761)
# Coordinates:
#   * y        (y) float64 2.193e+06 2.193e+06 2.193e+06 ... 1.961e+06 1.960e+06
#   * x        (x) float64 3.732e+05 3.732e+05 3.733e+05 ... 6.015e+05 6.015e+05
# Data variables:
#     blue     (y, x) uint8 ...
#     red      (y, x) uint8 ...
#     green    (y, x) uint8 ...
 I just tried this using xarray master and didn't get an error. So, you could try that and see if that works. Since our last release, we've had a fairly significant refactor of the IO backends in xarray.

Hi @jhamman, just tried what you suggest. Apparently it's a scipy vs netCDF4 thing; it works with scipy but not with netCDF4 on both master HEAD and the latest stable release.

Interesting. I'm not seeing any difference when using scipy or netcdf4 or h5netcdf.

yes, that's interesting... are you using a different OS? Let me know if there's something else I can help with.

Exactly what error message do you see?

@shoyer, below the full traceback; I installed everything with pip
mktmpenv -p python3
pip install xarray numpy netcdf4
pip install rasterio

Traceback (most recent call last):
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/api.py", line 724, in to_netcdf
    unlimited_dims=unlimited_dims, compute=compute)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/core/dataset.py", line 1179, in dump_to_store
    unlimited_dims=unlimited_dims)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/common.py", line 374, in store
    unlimited_dims=unlimited_dims)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 406, in set_variables
    super(NetCDF4DataStore, self).set_variables(*args, **kwargs)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/common.py", line 413, in set_variables
    self.writer.add(source, target)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/common.py", line 272, in add
    target[...] = source
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 48, in __setitem__
    data[key] = value
  File "netCDF4/_netCDF4.pyx", line 4648, in netCDF4._netCDF4.Variable.__setitem__
  File "netCDF4/_netCDF4.pyx", line 4913, in netCDF4._netCDF4.Variable._put
  File "netCDF4/_netCDF4.pyx", line 1754, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rasterio_reprex.py", line 41, in <module>
    ds.to_netcdf(nc_name)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/core/dataset.py", line 1254, in to_netcdf
    compute=compute)
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/api.py", line 729, in to_netcdf
    store.close()
  File "/home/loic/.virtualenvs/tmp-3caa6b25124e2f31/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 474, in close
    ds.close()
  File "netCDF4/_netCDF4.pyx", line 2276, in netCDF4._netCDF4.Dataset.close
  File "netCDF4/_netCDF4.pyx", line 2260, in netCDF4._netCDF4.Dataset._close
  File "netCDF4/_netCDF4.pyx", line 1754, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

Is this the same error you get with xarray/master?

I can reproduce this on my machine - also linux. This is going to be hard to track down though.

@jhamman, yes, same error message when installing from master

I have similar problem, when importing rasterio in the same script (not even using it for anything). This fails with HDF error:

import xarray as xa
import numpy as np
#import netCDF4
import rasterio

ds = xa.Dataset()
ds['z'] = (('y', 'x'), np.zeros((100, 100), np.float32))
print(ds)
ds.to_netcdf('test.nc')
ds.close()

with xa.open_dataset('test.nc') as ds:
    print(ds)

If I import netCDF4 before rasterio it works fine (uncomment line 3). This is probably an issue with rasterio somehow.
I installed everything with pip:
$ pip install Cython
$ pip install netCDF4 xarray rasterio numpy

From pip freeze:

affine==2.2.1
attrs==18.2.0
cftime==1.0.3
Click==7.0
click-plugins==1.0.4
cligj==0.5.0
Cython==0.29.1
netCDF4==1.4.2
numpy==1.15.4
pandas==0.23.4
pyparsing==2.3.0
python-dateutil==2.7.5
pytz==2018.7
rasterio==1.0.11
six==1.11.0
snuggs==1.4.2
xarray==0.11.0

It seems that this is not a problem with xarray but only with rasterio and netCDF4. Also this fails:
import rasterio
import netCDF4

with netCDF4.Dataset('test.nc', mode='w') as ds:
    ds.createDimension('x')
    ds.createVariable('foo', float, dimensions=('x'))
    print(ds)

Commenting out import rasterio removes the HDF error. I’ll report this to rasterio.

This looks like a binary incompatibility issue with wheels for rasterio and netCDF4 on PyPI.
Good alternatives (for now) would be building from source or using conda-forge.

Hi there,
I just wanted to let you know, that I do get the same error, when working with the following script. It does not use xarray or rasterio. So those will most likely not be the problem.
class Gif2NetCDF():
    def __init__(self, netcdf_name, netcdf_destfolder, gif_folder,
                 gif_filter_pattern=None, detailled_conversion=True):
        """

        Parameters
        ----------
        netcdf_name : str
            specifying the name of the netcdf file to write.

        netcdf_destfolder : str
            specifying the folder where the netcdf file is supposed to be created

        giffolder : str
            specifying the folder that contains the gif files, that are supposed to be written as an netcdf

        gif_filter_pattern : str
            specifying a re pattern to filter the all files in directory so you end up with the gif files you want.
            The here specified string will be passed to re.match which will be checked for each file in the provided giffolder

        Examples
        --------

        ## defining variables
        import netCDF4
        netcdf_name = "2016-07.nc"
        netcdf_destfolder = "./example/radar"
        giffolder = "./example/radar/2016-07/"

        ## create Instance
        cdf = Gif2NetCDF(netcdf_name, netcdf_destfolder, giffolder,gif_filter_pattern)
        ## write all (filtered) gifs in folder to netcdf file
        cdf.writeCDF()
        """

        ## creating global vars

        self.netcdf_name = netcdf_name
        self.netcdf_destfolder = netcdf_destfolder
        self.giffolder = gif_folder
        self.refilterpattern = gif_filter_pattern
        self.detailled_conversion = detailled_conversion

        self.netcdfFP = os.path.join(netcdf_destfolder, netcdf_name)

        # preparing the coordinate vectors
        self._lat_range = np.arange(479500, -160500, step=-1000)  # north
        self._long_range = np.arange(255500, 965500, step=1000)  # east

        # preparing time origin
        self.time_origin = dt.datetime.strptime("1970-01-01 00:00:00", "%Y-%m-%d %H:%M:%S")

        self.raincodes = np.array([....]) # this array is quite large, so I left it out... Which unfortunately means the code does not run... If anybody needs it please let me know

        return

    def list_gifs(self):
        self._gifs = os.listdir(self.giffolder)
        return self._gifs

    def filter_gifs(self):
        self._gifs = [file for file in self._gifs if re.match(self.refilterpattern, file)]
        return self._gifs

    def addDimensions(self):

        # adds dimensions to empty netcdf file
        self.latD = self.netcdfFile.createDimension('lat', self._lat_range.shape[0])  # north-south
        self.lonD = self.netcdfFile.createDimension('lon', self._long_range.shape[0])  # east-west
        self.timeD = self.netcdfFile.createDimension('time', None)

        return

    def addVariables(self):

        ## creating variables
        self.latV = self.netcdfFile.createVariable("chy", np.float32, ("lat",), complevel=9, zlib=True)  # north-south
        self.lonV = self.netcdfFile.createVariable("chx", np.float32, ("lon",), complevel=9, zlib=True)  # east-west
        self.timeV = self.netcdfFile.createVariable("time", np.float64, ("time",), complevel=9, zlib=True)

        self.rainV = self.netcdfFile.createVariable("rain", np.float32, ("time", "lat", "lon"), complevel=9, zlib=True,
                                                    fill_value=-100)

        ## adding units
        self.latV.units = "meters"
        self.lonV.units = "meters"

        self.timeV.units = "seconds"
        self.timeV.calender = "standard"

        self.rainV.units = "millimeter/hour"

        ## adding longname
        self.latV.long_name = "swiss northing CH1903"
        self.lonV.long_name = "swiss easting CH1903"
        self.timeV.long_name = "seconds since 1970-01-01 00:00:00"

        self.rainV.long_name = "precipitation intensity forecast"

        return

    def addDescription(self):

        self.netcdfFile.description = """..."""

        self.netcdfFile.history = """Created: {}""".format(dt.datetime.now().strftime("%Y-%m-%d %H:%M"))
        self.netcdfFile.source = '...'
        return

    def _write_static_dims(self):
        self.latV[:] = self._lat_range
        self.lonV[:] = self._long_range
        return

    def _write_time(self, file, datetime=None):

        if datetime is None:
            datestr = re.findall("\.([0-9]+)\.gif", file)[0]
            date = dt.datetime.strptime(datestr, "%Y%m%d%H%M")
        else:
            date = datetime

        seconds = (date - self.time_origin).total_seconds()
        current_size = self.timeV.size
        self.timeV[current_size] = seconds

        return

    def gif2array(self, file):

        xpix = 0
        ypix = 76
        n_pixel_x = 710 + xpix
        n_pixel_y = 640 + ypix

        gif = np.array(Image.open(file))[ypix:n_pixel_y, xpix:n_pixel_x].astype("float64")
        for idx, raincode in enumerate(self.raincodes):
            gif[gif == idx] = raincode[3]
        return gif

    def _write_rain(self, file):
        array = self.gif2array(os.path.join(self.giffolder, file))
        idx = self.rainV.shape[0] - 1
        self.rainV[idx, :, :] = array
        return

    def writeCDF(self):
        self.netcdfFile = Dataset(self.netcdfFP, 'w', format='NETCDF4_CLASSIC', )
        try:
            giflist = self.list_gifs()
            if self.refilterpattern is not None:
                fgiflist = self.filter_gifs()

            self.addDimensions()
            self.addVariables()
            self.addDescription()
            self._write_static_dims()

            for file in tqdm(self._gifs):
                self._write_time(file)
                self._write_rain(file)

        except Exception:
            self.netcdfFile.close()
            raise

        self.netcdfFile.close()

        return

Error
Traceback (most recent call last):
.
.
.
  File "C:\Users\foerstch\AppData\Local\Programs\Python\Python37\lib\site-packages\archiving\radar.py", line 358, in _write_rain
    idx = self.rainV.shape[0] - 1
  File "netCDF4\_netCDF4.pyx", line 4031, in netCDF4._netCDF4.Variable.shape.__get__
  File "netCDF4\_netCDF4.pyx", line 3369, in netCDF4._netCDF4.Dimension.__len__
  File "netCDF4\_netCDF4.pyx", line 1857, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

Does anybody have some advice on how to fix this?
Thanks a bunch!
Christian

HDF error usually means a corrupt file. Does ncdump -h work on your file?
I've found that sometimes ncdump -h will succeed but there'll still be some corrupt data which will result in an error when you try to load variable values

Well the error occurs while writing the file. So if the reason is, that the file is corrupt then necdf4 is corrupting it... right? The problem is rather, that it does not even finish writing the file.
FYI: The error does not occur at the same file every time.
Have a good weekend!! ;)

Similar error when using xr.open_rasterio, a workaround seems to be to change the order in which my datasets are opened. Example:
import xarray as xr
ds = xr.open_dataset('/data/someFile.nc') # netcdf
m = xr.open_rasterio('/data/otherFile.tif') # geotif
# Everything is happy

import xarray as xr
m = xr.open_rasterio('/data/otherFile.tif') # geotif
ds = xr.open_dataset('/data/someFile.nc') # netcdf
# Results in the following error

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/file_manager.py", line 186, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/lru_cache.py", line 42, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/data/someFile.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/api.py", line 420, in open_dataset
    filename_or_obj, group=group, lock=lock, **backend_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/netCDF4_.py", line 335, in open
    autoclose=autoclose)
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/netCDF4_.py", line 293, in __init__
    self.format = self.ds.data_model
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/netCDF4_.py", line 344, in ds
    return self._acquire()
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/netCDF4_.py", line 338, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/file_manager.py", line 174, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/usr/local/lib/python3.6/dist-packages/xarray/backends/file_manager.py", line 192, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1855, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/data/someFile.nc'

>>> xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6 (default, Sep 12 2018, 18:26:19)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
python-bits: 64
OS: Linux
OS-release: 4.15.0-58-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.2
libnetcdf: 4.6.3

xarray: 0.12.3
pandas: 0.25.1
numpy: 1.13.3
scipy: 1.3.1
netCDF4: <IP_ADDRESS>
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: <IP_ADDRESS>
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.0.26
cfgrib: None
iris: None
bottleneck: None
dask: 2.3.0
distributed: 2.3.2
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: None
numbagg: None
setuptools: 39.0.1
pip: 9.0.1
conda: None
pytest: None
IPython: None
sphinx: None

And I have the HDF5_USE_FILE_LOCKING environment variable set to FALSE.

Just a heads up, I can't reproduce this with latest packages from conda-forge for neither for xr.open_rasterio nor xr.open_dataset with engine=rasterio. Maybe the issue got resolved upstream.


feat(controller): add deis certs:transfer endpoint

From @sstarcher on September 28, 2016 16:19
Certificates are owned by the individual who creates them.  If this person leaves the only option is to delete the cert and re-add the cert.
Originally requested from Deis v1 - https://github.com/deis/deis/issues/4576
Similar to app:transfer - https://github.com/deis/deis/pull/4193
Copied from original issue: deis/workflow#526

From @bacongobbler on September 28, 2016 16:36
For now most administrators have been handling this inside the container using https://deis.com/docs/workflow/troubleshooting/kubectl/:
kubectl --namespace=deis exec -it deis-controller-h6lk6 python manage.py shell
>>> from api.models import Certificate
>>> from django.contrib.auth.models import User
>>> newowner = User.objects.get(username='newowner')
>>> c = Certificate.objects.get(name='mycertname')
>>> c.owner = newowner
>>> c.save()

But I agree that having some form of god-mode visibility and transfer-ability for all resources (including certificates) through the API would be useful.


More info on hosts outstanding

Often times when running the mimikatz module on a subnet, I'll have several stragglers, e.g.:
MIMIKATZ                                       [*] Waiting on 4 host(s)
MIMIKATZ                                       [*] Waiting on 4 host(s)

These stragglers never seem to finish, and there doesn't seem to be a straightforward way to get more information on them (hostname/IP, etc.) without digging through and correlating log entries.
Not a big deal, but it would be useful to have this info echoed back with the "Waiting" message or other such means such as an exit summary.

good feedback, will see what I can do.
Cheers

In my case I do have powershell installed on the system I am trying to exploit but still I see the same error message. Waiting on 1 host.  Any suggestion on what could be the reason and how to fix it?

Probably flaged by defender, we can't do much since we cannot know if the remote script is blocked by AV or just running. Make sure your enumeration on the target is complete before using a Module.


 I'm really sorry about this.
It was because the NPM client linked to a Github hash but old hashes were being overwritten by our CI pipeline. We didn't discover this until people like you reported it. Thank you again!
We have now removed any github dependencies from the package pinned the versions semantically.
Please try the latest version as this will remain stable (I promise!) from now on.
Jack

cc @KrisTemmerman @LorenzoOpWerk

Hey dude, thanks for fixing !


Add "hello world" example to README

The README should have a hello world example like: https://github.com/stubailo/graphql-tools-hello-world/blob/master/graphql-tools.js

Very low hanging fruit here if someone wants to earn some contributor accolades! :wink:

Um, the README has an example. Or am I missing something?
Scott


feat: Create DIDComm JWE with multiple recipients

Problem: We want to create a JWE for multiple recipients (ie.: different DID docs), by means of including these as recipients as well. Currently the JWE's recipients are only coming from the beneficiary's DID doc (keyAgreements).
Proposal: Extending the DIDCommMessage with a bcc field (~blind carbon copy) which is an array of additional beneficiary DIDs. Our proposed code resolves these into DIDdocs, and adds their keyAgreements into the recipients as well.
We don't care about sending DIDComm messages to the bcc recipients right now, but anyone can extend this logic in the future if so desired.

Testing: Proper unit test(s) can be added if necessary. But here's a POC test script for our use case:
import { IIdentifier } from "../../veramo/packages/core/build";
import { agent } from "./veramo/setup";

async function main() {
  const identifiers = await agent.didManagerFind();
  jwe(...identifiers);
}
async function jwe(
  originator?: IIdentifier,
  beneficiary1?: IIdentifier,
  beneficiary2?: IIdentifier
) {
  originator ||= await agent.didManagerCreate({
    kms: "local",
    provider: "did:key",
  });
  beneficiary1 ||= await agent.didManagerCreate({
    kms: "local",
    provider: "did:key",
  });
  beneficiary2 ||= await agent.didManagerCreate({
    kms: "local",
    provider: "did:key",
  });
  // these beneficiaries currently "live" in the same local database, but they can live anywhere apart.

  const message = {
    type: "test",
    from: originator.did,
    to: originator.did,
    bcc: [beneficiary1.did, beneficiary2.did],
    id: "test",
    body: { hello: "world" },
  };

  // create JWE
  const emsg = await agent.packDIDCommMessage({
    packing: "authcrypt",
    message,
  });
  console.dir(emsg, { depth: 10 });

  // decrypt JWE
  const msg = await agent.unpackDIDCommMessage({ message: emsg.message });
  console.dir(msg, { depth: 10 });
  // decryption is possible as long as we know the full `did:key` one of any of the recipients.
}

main().catch((e) => console.error(e));

Thank you very much for the contribution!
I'd like to propose a slight change that would allow us to maintain closer compatibility with the didcomm spec.
Instead of embedding the bcc property in the message object, pass it in as one of the options to packDidcommMessage.
This way, the DIDCommMessage type remains similar to the spec, and there is no ambiguity about embedding the extra recipients
IPackDIDCommMessageArgs can be modified to accept an optional options object with an optional bcc property.
The rest of the code would remain the same, I believe, except for loading the list of extra recipients.
It would also be nice to see the new code covered by tests :)

That's even better! Thank you, I'll include tests next as well.

@mirceanis in the new commit:

refactored bcc to go under new options field
unit tests (using did:key providers because these did not result in errors when resolving them after deletion)
small syntax improvements

Please add it as an optional to IPackDIDCommMessageArgs.
I'm not sure why this point is not getting across.

Please add it as an optional to IPackDIDCommMessageArgs. I'm not sure why this point is not getting across.

I wasn't done :) I'll rebase the commits in a bit

Okay should be good now? :D


[book] add chapter on breaking changes

As far as I can tell, rust-lang/rfcs#1105 are the de-facto guidelines that crate maintainers should use to guide their use of semver for this crates.
I have found it confusing in the past that these guidelines don't exist in any documentation outside that RFC, and discussion in rust-lang/rust#68004 led be to believe that it's worth it to add this to some official documentation.
See https://github.com/rust-lang/reference/pull/738 for more details
@ehuss I only added the chapter on breaking changes, not the

one on version unification and versioning guidelines,
I can add that as well, but I would need guidance on that. I think once that exists, that this should be a subchapter.
I also think I have edited this (originally derived from the rfc) to make sense as a part of the cargo guide, but of course that's up to review, and I'm willing to change it!

@ehuss I'm willing to help out on the missing pieces! I've noticed that other books in the rust ecosystem has TODO's, will that suffice here, so that extensions can be added in later pr's?

I think writing them down in a tracking issue on this repo would be fine.

@guswynn Was wondering if you're still interested in working on this. I might have a little extra time, so I was thinking of moving it forward.

I may be able to find time to work on in this weekend, can I put it on hold
for 2 more days, after that you can take it?
On Fri, May 15, 2020, 7:29 AM Eric Huss<EMAIL_ADDRESS>wrote:

@guswynn https://github.com/guswynn Was wondering if you're still
interested in working on this. I might have a little extra time, so I was
thinking of moving it forward.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/cargo/pull/7831#issuecomment-629267019, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJHND2FAG3TQBFIUWRC5ALRRVGUZANCNFSM4KLTTIHA
.

Sounds good!

I'm going to close, as this is now incorporated in the book: https://doc.rust-lang.org/nightly/cargo/reference/semver.html


[WIP] Make sure libsass headers are read first

In the case libsass is installed system-wide
(e.g. the headers are in /usr/local/include) and
node requires reading headers from the same path
for the unbundled shared libraries like libuv,
make sure -I/usr/local/include will not take
precedence over our local bundles libsass headers.

@saper I just flipped this to Draft since it had "WIP" in the title. If it's good to go, feel free to flip it back


8.0.1.GA Support

Hello. Good morning / afternoon.
I am trying to compile the module with 8.0.1.GA, but I am getting the following:
Part of the error message:
[TRACE] : [xcode-sim] In file included from /Users/tailored/NappDrawer/ios/Classes/DkNappDrawerDrawerProxy.m:11:
[TRACE] : [xcode-sim] /Users/tailored/NappDrawer/ios/Classes/DkNappDrawerDrawer.h:14:9: fatal error: 'TiUIiOSNavWindowProxy.h' file not found
[TRACE] : [xcode-sim] #import "TiUIiOSNavWindowProxy.h"
[TRACE] : [xcode-sim]         ^~~~~~~~~~~~~~~~~~~~~~~~~
[TRACE] : [xcode-sim] 1 error generated.

Full log:
trace.txt
Any feedback it is appreciated.
Thanks, and best,
Antonio.

It's not TiUIiOSNavWindowProxy.h  anymore.
Moved from Ti.UI.iOS.NavigationWindow to Ti.UI.NavigationWindow in order to make it cross platform and deprecated Ti.UI.iOS.NavigationWindow
so the module has to use TiUINavigationWindowProxy.h  I think. Feel free to make changes an pull request them

Hello @caspahouzer . Good afternoon / evening.
Thank you for the input.
I tried that replacing in all parts the TiUIiOSNavWindowProxy by TiUINavigationWindowProxy, but the compiler still sending the following:
[TRACE] [xcode-sim] In file included from /Users/tailored/NappDrawer/ios/Classes/DkNappDrawerDrawerProxy.m:11:
[TRACE] [xcode-sim] /Users/tailored/NappDrawer/ios/Classes/DkNappDrawerDrawer.h:21:3: error: unknown type name 'TiUINavigationWindowProxy'
[TRACE] [xcode-sim]   TiUINavigationWindowProxy *navProxy;
[TRACE] [xcode-sim]   ^
[TRACE] [xcode-sim] 1 error generated.

Full log:
trace.txt
Any feedback it is appreciated.
Thanks, and best,
Antonio.

the problem is that navigationWindow is not exposed to TitaniumKit see this: https://jira.appcelerator.org/browse/AC-6346?filter=-2

Hello @Brianggalvez . Good afternoon.
Did you find any workaround to solve the issue?
I read the ticket, but there is no update...
Thanks, and best,
Antonio.

Hello, faced this issue when tried to rebuild the module.
My workaround is to copy TiUINavigationWindowProxy.h to the module project.
dk.napp.drawer-iphone-2.1.2.zip


OAuth2 Performance improvements to refresh grant flow

We need to include following two indexes to default identity db scripts shipped with product. Those will add considerable improvement to the refresh token generation time  when there are large number of entries in the tables.
CREATE INDEX IDX_RT ON IDN_OAUTH2_ACCESS_TOKEN(REFRESH_TOKEN);
CREATE INDEX IDX_TID ON IDN_OAUTH2_AUTHORIZATION_CODE(TOKEN_ID);

The following two indexes are already available, hence closing this issue
CREATE INDEX IDX_AC_TID ON IDN_OAUTH2_AUTHORIZATION_CODE(TOKEN_ID);
CREATE INDEX IDX_AT_RT ON IDN_OAUTH2_ACCESS_TOKEN(REFRESH_TOKEN);


user_groups: Add users to members preview list without add button.

This makes the behaviour of this screen similar to the create channel screen where the user does not need to click Add button to add users to members preview. This follows most of the logic from that flow for soft removal, syncing between members preview list and the input pills, etc. The current user will have a pill in this case unlike channel creation since a user can create a group without them in it.
Most changes are similar to and built on top of #30610.

Screenshots and screen captures:

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

I also wonder if we should do a pass at sharing more code b/w add subscribers for channel and add members for groups. But then again, it was convenient to have the un-shared code when only trying to change the behaviour for channel creation flow last time.

@sahil839 your review would be much appreciated, since you reviewed the channel creation PR too!

Works for me in light manual testing!

Merged, thanks @shubham-padia and @sahil839! We'll need to be thoughtful about these patterns with the subgroups feature, possibly in a follow-up to the main PR there.


Refactor callbacks using metasync.cb

@tshemsedinov I don't think you should request reviews for many people simultaneously for trivial changes since if you do, you imply that the PR is required to be reviewed by all of them.

Ok

Landed in 57e5575


Correct process errors if storage is unavailable

For example after removing all storage (by accident for example) addons from heroku I get such weird exception:

I think it's not actual now.


Support additional YAML files

Changes proposed

The main change of this PR is setting gatsby-transformer-yaml's option to support additional YAML files.

If the current processed file is sidebar.yml, expose as  sidebarSidebarItems node and keep the current behavior, no breaking change if other people are about to update a newer version with this merged PR.

Add dataPath configuration and set a default value to data folder's name.
Add using-yaml-files.md of how to use YAML files in the project.
Rename imported SEO component to Seo to fix Pascal case naming warning.

My concern

If a user update gatsby-theme-docs-core to version 3.1.0 (minor), they will get a data folder automatically.
However if one does not commit a data folder to a repository because it is an empty folder, this causes gatsby-source-filesystem throw an error
I think after this PR get merged, we need to update @rocketseat/gatsby-theme-docs to use<EMAIL_ADDRESS> However, I get confused because I am very new with changeset tool and don't know will it be patch or minor for a version of @rocketseat/gatsby-theme-docs project. Therefore, I leave this for you.
If you need me to do an addition PR to update @rocketseat/gatsby-theme-docs's version, please let me know if it will be patch or minor because I don't know if you want to keep number versions of these these two projects synced.
We should consider to not specify typeName of sidebar.yml file and use a default typeName of gatsby-transformer-yaml.
This is because we can remove our custom typeName function and simplify our code. However, this causes a breaking change and is not backward compatibility for existing projects that use @rocketseat.

Fixes #120
Additional context

N/A

@jpedroschmitz Is it possible for you to review this PR, I kind of need this feature. Thank you so much.

Hey @aaronamm, sorry for taking so long to review it. A lot of stuff is going on right now.
I haven't worked with Gatsby in a long time, so I'm not sure about the code itself.
About the data folder, maybe we should rename it to yamlFiles or something like that. I think data is confusing.
--
The changeset will manage all the dependencies updates around the project! And minor is fine =D

@jpedroschmitz Thanks for your code review and sorry for late reply. I have changed "data" folder to "yamlFiles" already.
However, I am very new for changeset tool. I made a new commit  as a refactor commit without a new changeset file.
I tested this change and it works without any issues. If everything is okay, please could you merge this PR.
Thanks.

@jpedroschmitz  Thank you so much for merging this PR and help me improve the code.
However, is it possible for you to answer my question a last comment.
Thanks again.

I still couldn't publish it because my NPM token is no longer valid :/
Since I changed companies, I'm no longer in Rocketseat NPM organization.
I'll try to get a new one or rename the package.

@jpedroschmitz  Thank you for you supporting. I appreciate your help.

Hey, I was able to get a new token! The release was published!
Thanks for your patience ❤

@jpedroschmitz  Thank you so much.


Question about National Jukebox Collection samples and their inclusion in Citizen DJ

Hi. I'm a fan of Citizen DJ, and I had a question about it that prompted me to make a GitHub account just to ask this one question. There's a lot of really cool samples to explore and use in music projects of various genres. The acoustical era recordings from the Variety Stage and Inventing Entertainment collections for example are full of vintage lo-fi character and are practically tailor-made for lo-fi music.
Which is why I was excited for 2022, as at the bottom of the Citizen DJ homepage, there is a teaser for new samples arriving in 2022. Clicking on "Learn more" shows an explanation of what is coming in 2022, namely over 10000 sound recordings from a collection called the "National Jukebox Collection" that will become public domain due to copyright expiration. The thing is though, as of writing this (January 4th, 2022), those samples are in the public domain, but the page is still written like this hasn't happened yet, as the page asks you to "Check back here on January 1, 2022!"
I figured that maybe the 1922 and earlier recordings would be available for download on the Library of Congress website, as they weren't available before presumably due to them still being copyrighted, but the Jukebox samples that are from 1922 and earlier aren't available for download either, even though they are now in the public domain. To be clear, I used the link provided on the Citizen DJ webpage that automatically filters out recordings from after 1922 to make sure I wasn't trying to download recordings from 1923 onwards.
I decided to check the Citizen DJ GitHub to see what was going on, and on the thread titled "Search for a particular spoken word or phrase" (I'm not using a link because I'm not sure if GitHub will permit me posting links), Brian stated that Citizen DJ is currently frozen due to him departing from the project in October 2020, I assume this is why Citizen DJ wasn't updated in 2022 with the National Jukebox samples. If it's okay for me to ask, I'd love some clarification as to the status of Citizen DJ and whether the National Jukebox samples will be added to Citizen DJ.

Hello! Apologies for the delay in response.  Due to some technical hurdles, we could not release the Jukebox recordings right on January 1st, but they have since been released and you can now view and use them here:
https://citizen-dj.labs.loc.gov/public-domain-2022/
Thank you and hope you enjoy the new samples!

Hello! Apologies for the delay in response. Due to some technical hurdles, we could not release the Jukebox recordings right on January 1st, but they have since been released and you can now view and use them here:
https://citizen-dj.labs.loc.gov/public-domain-2022/
Thank you and hope you enjoy the new samples!

Hey, better late than never, I honestly wasn't expecting a response considering how inactive this forum was (the last thread excluding mine is from over a year ago). I just read the post you linked and I'm honestly excited to try this stuff out. Thank you so much Brian.


Dedupe CSAF Vulnerability IDs

CSAF 2.0 will fail validation if the same vulnerability ID is listed more than once per vulnerability. IDs are generated programmatically by parsing the list of references. When multiple references are listed which point to the same vulnerability, a duplicate entry is added.
Example:
https://github.com/advisories/GHSA-v845-jxx5-vc9f
https://github.com/urllib3/urllib3/security/advisories/GHSA-v845-jxx5-vc9f
Both references may be included but we must only add a single entry in the IDs section.
Similarly, since commit and issue IDs are not unique, we should extract the repository name and include it in the system name.

How do other vendors handle this scenario? Any examples we can find from providers such as redhat?

I'm afraid I don't know how vendors handle this - according to the generator section, Red Hat's CSAFs are produced using the Red Hat SDEngine, but I don't know any details.
The CSAF 2.0 schema is clear that it must be unique entries and there isn't any reason to store the exact same thing twice for the same vulnerability anyway. It doesn't have to be unique across the whole document, just in the ids attached per vulnerability. There's really only two ways to deal with this issue - either prevent duplicate IDs from being generated in the first place from the reference list, or remove them afterwards. The latter is quicker.

Does this tool help? Perhaps the product tree is used to link to all products that are related to the given vulnerability?
https://secvisogram.github.io/?tab=EDITOR

@prabhu This is unrelated to the product tree. The entries duplicated are in vulnerabilities, and their only fields are system_name and text - for the system that created the id, and the id itself, respectively.

@prabhu Here's an overview of the part of the document I am referring to from oasis.


May i get some help?

Im totally new to programming, i would like to ask if its working after ive finish the 'Build instructions' or i have to do something else? Would like to learn how to make it work. Thx

Btw, this is totally different with https://github.com/jussihi/PUBG-map-hack OR its a method changing firebase to nodejs? Im curious haha

when i put http://<IP_ADDRESS>:7890/ in chrome i can see a user connected, current users: [1] but there is no map or anything just a blank page so what should i do now?

you should check the post data by hack-control app

may i ask what is hack-control app?do you mean the kernel driver and userapp by jussihi?

i follow the guild of 'Build instructions' and seems like its success because the cmd shows how many user collecting, but what should i do next?

Start up both of PUBG-map-hack and the map,  map will draw the data send from PUBG-map-hack

opps i realise the problem is that i dont know how to build the kernel driver and usermode app:( could you tell me how to? ive start a new KMDF and put everything in kernel driver in it but i fail to build it :(

some error like this
[Building 'mapDriver' with toolset 'WindowsKernelModeDriver10.0' and the 'Universal' target platform.
Stamping x64\Release\mapDriver.inf [Version] section with DriverVer=11/06/2017,<IP_ADDRESS>5
ApiValidation : error : ApiComparer had an XmlException. Make sure XML file has correct format.
ApiValidation : error : Check that XML files are formatted correctly.]

if i build the application in test sign ill say file not found xxxx.sys, but if i set it to sign mode off the error ive paste will appear :( where is the problem :(


fix: allow failure of segment retrieval

Current dependencies on/for this PR:

main

PR #129

PR #131

PR #132

PR #136   👈

PR #130

This comment was auto-generated by Graphite.

Merge Activity

Sep 27, 9:57 AM: @pevisscher started a stack merge that includes this pull request via Graphite.


Replace O2ims_ with o2ims- in URL prefix

@danielerez @irinamihai please be aware of this fix, you will need to use the new URLs in your testing.


driver: fix special args passing to tcl and python

Fixes #4669. To test the python side this needs the changes from #4671 but even with that I get a segfault with valgrind ./yosys -y foo starting with this:
Invalid read of size 8
   at 0x4D033A3: PyList_New.constprop.0 (in /nix/store/sxr2igfkwhxbagri49b8krmcqz168sim-python3-3.11.8/lib/libpython3.11.so.1.0)
   by 0x4CEB408: PyImport_Import (in /nix/store/sxr2igfkwhxbagri49b8krmcqz168sim-python3-3.11.8/lib/libpython3.11.so.1.0)
   by 0x4CEB6DF: PyImport_ImportModule (in /nix/store/sxr2igfkwhxbagri49b8krmcqz168sim-python3-3.11.8/lib/libpython3.11.so.1.0)
   by 0xDA234D: Yosys::init_share_dirname() (kernel/yosys.cc:1017)
   by 0xDA0DCF: Yosys::yosys_setup() (kernel/yosys.cc:550)
   by 0xCC1FCC: main (kernel/driver.cc:497)
 Address 0x10 is not stack'd, malloc'd or (recently) free'd

@donn any ideas?

I broke this by accident in #4643 by the looks of it, oops. Because init_share_dirname depends on Python now when ENABLE_PYTHON is defined, Python needs to be initialized first.
This patch should fix it.
diff --git a/kernel/yosys.cc b/kernel/yosys.cc
index 374b07d06..774c7f37d 100644
--- a/kernel/yosys.cc
+++ b/kernel/yosys.cc
@@ -547,12 +547,6 @@ void yosys_setup()
 	if(already_setup)
 		return;
 	already_setup = true;
-	init_share_dirname();
-	init_abc_executable_name();
-
-#define X(_id) RTLIL::ID::_id = "\\" # _id;
-#include "kernel/constids.inc"
-#undef X

 #ifdef WITH_PYTHON
 	// With Python 3.12, calling PyImport_AppendInittab on an already
@@ -566,6 +560,13 @@ void yosys_setup()
 	}
 #endif

+	init_share_dirname();
+	init_abc_executable_name();
+
+#define X(_id) RTLIL::ID::_id = "\\" # _id;
+#include "kernel/constids.inc"
+#undef X
+
 	Pass::init_register();
 	yosys_design = new RTLIL::Design;
 	yosys_celltypes.setup();
diff --git a/setup.py b/setup.py
index b3a6a9280..b39b579e0 100644
--- a/setup.py
+++ b/setup.py
@@ -76,7 +76,7 @@ class libyosys_so_ext(Extension):
         # yosys-abc
         yosys_abc_target = os.path.join(pyosys_path, "yosys-abc")
         shutil.copy("yosys-abc", yosys_abc_target)
-        bext.spawn(["strip", "-S", "yosys-abc"])
+        bext.spawn(["strip", "-S", yosys_abc_target])

         # share directory
         share_target = os.path.join(pyosys_path, "share")

I'll go ahead and test that wheels are still okay.

Wheels tested. Super, super sorry for the inconvenience.

Don't worry about it, I'm just glad it's a quick fix


bug(mat-form-field): PLACEHOLDER IS EMPTY BY DEFAULT

Please look at the
Full Explanation here
This is either a bug or the documentation should be changed.
Here it states:
"If you want to create a legacy form field with a placeholder but no label, you will need to specify an empty label to prevent the placeholder from being promoted."
Together with this code:
<mat-form-field>
  <mat-label></mat-label>
  <input placeholder="Just a placeholder">
</mat-form-field>

However the <mat-label></mat-label> prevents the placeholder to be displayed.
Current Solution: Remove the mat-label
<mat-form-field>
  <input placeholder="Just a placeholder">
</mat-form-field>

As far as I can tell the docs mentioned here don't exist anymore. Closing.


Make internal training video on how to find a team client in the CRM

Describe the issue
We will be onboarding more people and having some CRM training material will save us time.
Additional context

Fair amount of learning debt in this ticket
This is internal facing, so nothing for the demo (the CRM is private)
The video is intended to be "clicking around and doing stuff"
There could be a PPT slide or two mixed in if Text is needed

Start demo by opening an example ticket like https://app.zenhub.com/workspaces/cloud-pathfinder-5e4dbb426c3c6af8dcbf06a7/issues/bcgov/cloud-pathfinder/1804
then going over to the CRM, search for the entry in 1804 above, and show the fields, what they mean, then click edit, and make a new update to the "Latest Update" field.  Also demonstrate how to add a person to one of the email fields (open a new mail in outlook, autocomplete their email and name, copy paste it into the CRM in the right field, then add a semi-colon after, hit save).  Then show how to copy paste from the CRM contact field into an email, explaining that's why we use the format of email and semi-colon separator.
Definition of done

Video about how a new team member can find an entry in the CRM to help them with their ticket in ZenHub.
Meet with Julian and get trained on the below items
Figure out with the team's help how to make a video in Windows (what format, how to edit and stitch together etc)
Try to capture the clicks, or a doc accompanying with clicks.  QuickTime on Mac yes, but on Windows?
Narrated training video demonstrating filters, how to find a particular entry, the different data views, how to edit
Have a section explaining gotchas (known bugs, little problems that could surprise a new user)

Might need to create a storyboard to outline and visualize what we plan to show.

If using synthesia for AI avatar, there will be monthly fee 30 Dollars for personal account. For corporate account, need to book a demo.
Now using the free software Clipchamp. It makes Text to Speech video for free.

Had a review session with Omo. Went through the DOD. Need to add a part for demonstrating filters. Wait for Julian back for adding a section explaining gotchas (known bugs, little problems that could surprise a new user).

Added the part for demonstrating filters. Wait for Julian back for adding a section explaining gotchas (known bugs, little problems that could surprise a new user).


Allow several attempts to specify a remote cluster

How to reproduce: Connect to a remote cluster in the GUI with the wrong URL (cluster not running in this case).
What should happen:

Error message
Try again

What happens:

Error message on the console, not in the GUI
Input box still visible, but "connect" button faded out
Not possible to try again

Edit: Reloading the browser window works as a work-around.

What are the prerequisite to solve this issue ?

What are the prerequisite to solve this issue ?

In this case, a prerequisite to fixing the bug is building the client from source, as there most likely need to be changes in the TypeScript client code. You may need to know the basics of some libraries used in the client, for example react and possibly redux. You may need to understand the very basics of backend and front end web dev, for debugging and understanding what happens when the user clicks the connect button etc.
Were you able to reproduce the bug?

Were you able to reproduce the bug?

Yeah, I did reproduce.
I'm not familiar with web dev. But I would love to solve this bug, I think knowing web dev would be helpful for GSOC project.

We have a small section on debugging the web GUI in our documentation, which has some pointers to indispensable debugging tools (browser's devtools, react and redux plugins for these, ...), which should help you trace what is going on.
As a hint on how to start with this, I would suggest to try to understand what exactly, on client and server, step by step, is happening in the "happy case", that is, in case the user presses the "Connect" button and either connects to an existing dask cluster or successfully creates a local cluster (the two cases are in general very similar). Let me know if you get stuck or need help to understand any specific part!

Thanks :+1:

I tried to figure out how "happy case" works. The client makes a connection request and server returns status ok. Then it makes action connected. I didn't understand how "browser" button showed after that.
Also in the "bad case" server returns 500: Internal Server Error.

I tried to figure out how "happy case" works. The client makes a connection request and server returns status ok.

:+1: - as evidenced by the request in the "network" tab of the browser's devtools. Where in the code does this request happen? How do we go from "user presses button" to "network request is made"? This may require some "detective work", but one possible approach is to have a look at the "initiator" column of the network tab. I'm not sure it is visible by default, but you can enable it by right clicking on the header:

Then it makes action connected.

Where in the code is that action created?

I didn't understand how "browser" button showed after that.

The browse button is rendered as part of the DatasetList component, which is conditionally shown by the ChannelStatus component. If you have a look at App.tsx, you can see this relationship, and in ChannelStatus.tsx you can see the condition.

Also in the "bad case" server returns 500: Internal Server Error.

Indeed. Which is not handled in the client, which then makes the UI unusable.

when the button is clicked, action "connect" is dispatched.Then it makes a request to the server for cluster connection. Here it receives the response from the server.

Where in the code is that action created?

In the "happy case" when the status of the received response is "ok".

Nice! Looks like you found out where the request is done - that's also where the error handling should be done. Also have a look at the connectToCluster function itself - maybe part of the error handling should also live there. Quite possibly instead of a 500 response, the server should also respond with a better error message... but the client should also not crash if it gets a 500 when connecting.
Do you want to give it a try, or do you need further discussion?

I will give a try first

I tried out to figure out what's happening on the server. But it felt complicated to me. How can I send a different message when the cluster connection isn't successful.

I tried out to figure out what's happening on the server. But it felt complicated to me.

It may look complicated at a first glance, but most of the code is not really relevant for fixing this issue. Try to go through it, line by line, and keep track at which point the cluster connection is established. Understanding everything after that is not needed for this issue. Also, keep in mind that this issue only deals with the non-local cluster.

How can I send a different message when the cluster connection isn't successful.

First, you have to find out what happens if the connection is not successful. Is an exception thrown, or does some function return another value in case of an error? Then, you can add a message for the error case and send that if the error condition is met.

I tried to go through line by line. This is the function responsible for making cluster connection. A partial function is created first.I didn't understand these statements :
executor = AsyncAdapter(wrapped=sync_executor)
await self.state.executor_state.set_executor(executor, request_data)

For just testing, I put this in a try block. Then there was no error, but now the server is sending response as the cluster connected.
I would like to send a different response when cluster is not connected. Then client could show some error message to user and reset the button.

A partial function is created first

Yeah, this is used here because sync_to_async only takes a function, not a function and arguments.

I didn't understand these statements :
executor = AsyncAdapter(wrapped=sync_executor)
await self.state.executor_state.set_executor(executor, request_data)

For understanding them, you need a bit of context how the web API works. The Web API is completely asynchronous - which means that handling of multiple requests can be done concurrently (maybe best described as "interleaved" with another?). In detail, every time you read await, you can imagine that the awaited operation is started, and another request/response that is currently in flight has the chance to continue running. Once the awaited operation is done, execution can continue here.
That also means if you need to do something that will take a long time, you need to put it into a thread. Now, because our DaskJobExecutor is synchronous, it would not directly work inside the Web API: it would block the main loop for however long it takes to connect, or execute a job, or basically do anything. So we wrap it into its own thread, using this AsyncAdapter.
But! This is not really important for understanding what's going on here - these lines are only executed once the connection to the cluster is successful.

I would like to send a different response when cluster is not connected. Then client could show some error message to user and reset the button.

Sounds like a good plan! Now, with exceptions, you always have to think about where you should catch the exception, and where you should allow it to "bubble up" (let me know if this is unclear, I can describe it in more detail).
If you put your exception handling into the executor, you would have to know in the executor how to handle this problem, which is not the case here: you can only send your message from the ConnectHandler! So that's where the error handling needs to happen.

Fixed by #699


Codefix for CollectionShouldOnlyHaveUniqueItemsByComparer

This fixes #3 .

since this uses FA 5.0 it's test didn't run...

@jnyrup your changes have been merged, thanks for your contribution 👍


Add your marker to the HacktoberMap! (final round)

If this is your first pull request, please read this article: How To Create a Pull Request on GitHub
Step 1
Fork the repo by pressing the "Fork" button on the top right side of the
repository and clone your forked repository.
Step 2
In your forked repository, edit hacktobermap.js and add your name, location,
latitude and longitude to the bottom of the file using this format:
  {
    name: "Your name",
    location: "Your location",
    latitude: 12.34,
    longitude: 56.78
  },

You can get the coordinates of your location from https://latlong.net.
Warning: Do NOT put your exact coordinates for your own safety!
Step 3
Commit your changes and submit a pull request! After your pull request gets accepted, you will be able to see your very own marker on the HacktoberMap!
Step 4
Please star the repo if you like this fun little Hacktoberfest project ❤

PR #180

hacktoberfest is closed.  Thank you for helping to involve people in the community.


fix(module:*): patch all components with setDisabledState

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/NG-ZORRO/ng-zorro-antd/blob/master/CONTRIBUTING.md#commit
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Application (the showcase website) / infrastructure changes
[ ] Other... Please describe:

What is the current behavior?

Issue Number: #134
What is the new behavior?
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information
Patch all components with setDisabledState.
Work in process:

[x] input
[x] input-number
[x] select
[x] slider
[x] switch
[ ] rate
[ ] datepicker
[ ] timepicker
[ ] checkbox-group
[ ] radio-group
[ ] cascader

Coverage increased (+0.002%) to 40.35% when pulling a6849525164255243f826338ad9302c357c63a61 on deart1mer:master into 29a5cbfcdf1dd8935448f385bec18d2b97a85bca on NG-ZORRO:master.

Coverage increased (+0.002%) to 40.35% when pulling 0632fe167f0044970cf1338fc38cf16f37449fbb on deart1mer:master into 29a5cbfcdf1dd8935448f385bec18d2b97a85bca on NG-ZORRO:master.

@vthinkxie
Hi. I'm trying to help to patch all components with setDisabledState, but I found an unnormal behavior in some of the disabled components. I can not make a decision to fix it or not.
I found that the popup(dropdown) of components like select, datepicker, timepicker and so on won't be disabled and closed while they have been opened.
This is not just about setDisabledState, which is related to the disabled behavior of the component itself.
Is this an unnormal behavior? If is, should I fix it when I fix the setDisabledState, or just follow the original behavior and open another issue to fix it?
For example:
Steps:

Open select component's dropdown.
Select component disable by a timer.

Expect:
Select component close dropdown or disable the selection from user.
Current:
Select component become disabled status but dropdown selection still work for user.

@deart1mer
Thanks for your pr, the dropdown should be closed if the component is disabled.
Please open another issue and fix it if you wish.

@vthinkxie Yeah. I would like to do that.

Coverage increased (+0.002%) to 40.35% when pulling f268686a121dc037a8177cec052ccdc3465d7f93 on deart1mer:master into 29a5cbfcdf1dd8935448f385bec18d2b97a85bca on NG-ZORRO:master.

Coverage decreased (-0.0008%) to 40.347% when pulling f268686a121dc037a8177cec052ccdc3465d7f93 on deart1mer:master into 29a5cbfcdf1dd8935448f385bec18d2b97a85bca on NG-ZORRO:master.

Coverage increased (+0.001%) to 40.35% when pulling e8cd9c52e246f6fe19450c14bd4747206f27f4fc on deart1mer:master into 29a5cbfcdf1dd8935448f385bec18d2b97a85bca on NG-ZORRO:master.

Coverage decreased (-0.02%) to 40.327% when pulling ebbc8adab1e413f5e0665a4002baaf5c40148b19 on deart1mer:master into 29a5cbfcdf1dd8935448f385bec18d2b97a85bca on NG-ZORRO:master.

@vthinkxie Hi, I have patched all components with setDisabledState. Should I fix #190 in a new pr?

@deart1mer yes, we will merge this after review, thanks for your effort.


Not showing validation errors

Laravel Version: 8.40.0
PHP Version: 7.4.16
Laravel-admin: 1.8.11

Description:
Validation works, but there are no visible errors. After form submitted it redirects back without updating model.
Steps To Reproduce:
$form->number('max_answers', __('fields.max_answers'))
->default(1)
->min(1)
->max(100)
->rules('exclude_if:is_multiple,off|numeric|min:2')
;

@trikey did you solved this problem. kindly share the answer.


Added DSC Debug and Analytic logs for capture

See lines 343ff

This change is

@GoateePFE,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Thius change looks good to me. Can you please update the Unreleased section https://github.com/PowerShell/xDscDiagnostics/blob/dev/README.md#unreleased with an entry of this change?

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Progress Bar for Download

Hi, can you add progress bar for download. So that people can view the progres and estimated extra time when downloading files from the remote server. Thank you.

Duplicate https://github.com/microsoft/vscode/issues/111580


[ADD] Add new rebus

Summary of Changes

Added one new rebus!

Pull Request Test Coverage Report for Build 169

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.789%

Totals

Change from base Build 164:
0.0%

Covered Lines:
97

Relevant Lines:
107

💛  - Coveralls


tRNA N-acetyltransferase activity incorrectly named

tRNA N-acetyltransferase activity
is defined
Catalysis of the reaction: a cytidine in tRNA + acetyl-CoA + ATP + H2O = ADP + an N4-acetylcytidine in tRNA + CoA + H+ + phosphate. [PMID:15037780]
so it is describing a more specific activity
please rename
N4-acetylcytidine transferase activity
exact synonym
cytidine N4-acetyltransferase
(other bases or the backbone can be acetylated)
(it isn't clear whether all annotations  will have used the definition, and we will  need a new term for
tRNA N-acetyltransferase activity, so we might want to broaden the definition of the existing term and reannotate?)

Also
rRNA cytidine N-acetyltransferase activity
-> rRNA cytidine N4-acetyltransferase activity
although all might be N4 (I have no idea), best to be precise


ons_available_datasets not found

The ons_available_datasets function doesn't work.

If you are using a local package you need to run devtools::load_all() - when the package is on CRAN this won't be necessary

I built it and installed the package using install.packages so I though I wouldn't need to use load_all. That gets me one step closer but I still get an error saying that it can't find function select.
library(THFOpenDataPipeline)
devtools::load_all('redacted_path/Open_data_pipelines')
#> Loading THFOpenDataPipeline
datasets <- ons_datasets_setup()
ons_available_datasets(datasets)
#> Error in select(., id): could not find function "select"

Created on 2020-06-09 by the reprex package (v0.3.0)
When I run check I get this as one of the warnings
'ons_available_datasets: no visible binding for global variable ‘id’' which I think might be linked to this problem.


Remove the old textdb-dataflow module

Most of the operators are now migrated to the new textdb-exp module from the old textdb-dataflow module. In order to avoid maintaining both the old version and the new version. We need to remove the textdb-dataflow branch step by step.
Current modules that depends on the dataflow module:
textql: it relies on some old Bean classes from web and some old predicates.
web: the old API heavily relies on the old dataflow module. The old set of Bean classes rely on the old Builders, which are not needed by the new architecture.
perftest: the perftest module still relies on the old dataflow operators
gui: The new GUI is pretty stable now, so we can get rid of the old GUI.

The old dataflow module and the old gui have been completely deleted. The code has been migrated to new exp module


Question: Why do you use broad macs2 peak calling for all modalities?

Hello from Spain!
Thank you very much for this wonderful resource. I am currently in the midst of implementing your nanoscope pipeline, and was just wondering, why is it that you perform your peak calling with the same parameters for all 3 of your modalities (H3K27ac, H3K27me3, ATAC), rather than specify narrow peak calling, for example, for your H3K27ac and ATAC modalities.
Just for reference, i'll leave the code that I am commenting on below (from nanoscope/workflow/Snakefile_preprocess.smk):
'macs2 callpeak -t {input} -g {params.macs_genome} -f BAMPE -n {wildcards.modality} --outdir {params.macs_outdir} --llocal 100000 --keep-dup 1 --broad-cutoff 0.1 --max-gap 1000 --broad 2>&1 '
Thank you very much in advance!

Hi Andrea,  Great question. There is no particular reason for this peak calling and it really depends on the modality what peak caller is the best. In my experience the current macs2 command works well both for H3K27ac and H3K27me3. Narrow peaks were a bit too restrictive for histone modifications in my opinion and gave some false negatives.
Of course for ATAC-seq probably the default cellranger peak caller would work great. This is not implemented at the moment, but since nanoscope includes regular cellranger run, the cellranger peaks are part of the output and can be used in downstream analysis.
I hope this is helpful :)
Marek


Merge 1.47.0 to master

~Currently WIP, given that v1.46.0 hasn't merged yet, and this includes that PR too.~

If you haven't experienced it yet, the VSCode merge tool was very helpful with this (until I accidentally clicked the wrong button half way through test-fs.c and had to finish that one manually). It has an smart merge feature that can do a merge at the word level granularity, instead of lines, which meant it figured out most of the resolutions and I just needed to accept them.

ASAN expected to fail without the workaround (https://github.com/libuv/libuv/pull/4200#pullrequestreview-1731847813), but I try to avoid porting too many commits when it is easier to just make a release and merge all of them at once.

bump?


Dockerfile: update to latest opam-repository

Because of https://github.com/ocaml/opam-repository/commit/5e99237bb46017d6c14f1109a3f5b5e1aa249885

/cc @talex5

CircleCI cache is busted, but DatakitCI is happy, so merging.


Navigate downtown Raleigh during Citycamp blog post.

Published but let me know if things are clear, any other transit concerns.
http://citycampnc.org/2017/09/25/navigating-around-downtown-raleigh-for-citycamp-nc/


The library(folder) section only has the Desktop folder and the Downloads folder

Explain the bug
The library(folder) section only has the Desktop folder and the Downloads folder.
To reproduce

Open Files
See the error

Expected behavior
I would like to show all of the following folders:

Desktop
Downloads
Documents
Pictures
Videos
Music

Logfile
debug.txt
debug_fulltrust.txt
Screenshots

System Information

OS Version: [Windows 10 Home 20H2 19042.928]
App version: [e.g. v<IP_ADDRESS>]

Additional comment
Does this problem happen again after restarting the app?

Yes, it does.

This should be resolved in v2.3.11


Fix TestPipelineLevelFinally_OneDAGTaskFailed test to avoid false negative error

Changes
For TestPipelineLevelFinally_OneDAGTaskFailed_InvalidTaskResult_Failure test list of skipped tasks is checked. In case if order of actual skipped tasks is different than expected, test may fail despite equality between actual and expected skipped tasks.
    pipelinefinally_test.go:288: Expected four skipped tasks, dag task with condition failure dagtask3, dag task with when expression,two final tasks with missing result reference finaltaskconsumingdagtask1 and finaltaskconsumingdagtask4 in SkippedTasks. Diff: (-want, +got):   []v1beta1.SkippedTask{
          	{Name: "dagtask3"},
          	{Name: "dagtask4", WhenExpressions: {{Input: "foo", Operator: "notin", Values: {"foo"}}}},
          	{
        - 		Name:            "finaltaskconsumingdagtask4",
        + 		Name:            "finaltaskconsumingdagtask1",
          		WhenExpressions: nil,
          	},
          	{
        - 		Name:            "finaltaskconsumingdagtask1",
        + 		Name:            "finaltaskconsumingdagtask4",
          		WhenExpressions: nil,
          	},
          }
...
--- FAIL: TestPipelineLevelFinally_OneDAGTaskFailed_InvalidTaskResult_Failure (29.51s)

Sort of actual skipped tasks based on their names will fix this problem.
/kind failing-test
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[ ] Includes tests (if functionality changed/added)
[ ] Includes docs (if user facing)
[x] Commit messages follow commit message best practices
[x] Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.
Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task to build and release this image.

Reviewer Notes
If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.
Release Notes
NONE

/test check-pr-has-kind-label

/test check-pr-has-kind-label


Add validation for go1.16

It was released 16-Feb-2021.  We should test it too.  (currently go_8 through go_15 are actively tested by our integration test suite.)

I updated docker-compose.yml and docker-compose.override.yml, and validated that go 16 works without problems.
I was most worried about https://github.com/golang/go/issues/40724 which plans to switch to a register-based calling convention, but it appears to have been deferred to 1.17 at this point.  (I confirmed this by temporarily adding GOEXPERIMENT=regabi to the go builds.  This appeared to have no impact.)

Unrelated to go, test changes that I added recently under https://github.com/criblio/appscope/pull/136 made it so our java test containers don't build.  Namely, this line RUN apt-get -o Acquire::Check-Valid-Until=false update && apt-get install -y bsdmainutils in test/testContainers/java/Dockerfile doesn't work in general because not all of the containers apt-get based.  I'm addressing the test container build issues here too.  After these changes, we run package-manager-appropriate commands to download hexdump in each container (well each one but java6).  And, we changed the "Testing that payload files don't contain tls" test to just skip the testing if hexdump is not available in the container.  We're using this in the java6 container, which would be difficult to update now that the apt-get repos for the container have been moved into an archive.


Ionic2:BarcodeScanner.encode error

** code
BarcodeScanner.encode(BarcodeScanner.Encode.TEXT_TYPE,'http://www.nytimes.com').then(() => {
    console.log('suc');
}, (err) => {
    console.log(err);
});

** Error
Exception calling native with command :: BarcodeScanner :: encode ::exception=TypeError: success is not a function

This is an ionic native issue, please open an issue there with your problem and close here

@ksphp yeah! please can you reopen the issue in Ionic Native

Thanks！


.container within .main-raised causes horizontal scroll due to .main-raise l/r margin

For the Bootstrap 3 version of material-kit:
.main-raised needs customized .container sizes due to .main-raised adding 30 px of margin left/right, but .container having set, hard widths at various media queries. Screenshot:

The solution is a set of .container sizes per media query inside .main-raised that are 60px (30px left margin, 30px right margin) smaller.

Here is the solution in SCSS:
$main-raised-margin: 30px;
.main-raised {
  .container {
    @include respond-to('sm') {
      width: $container-sm - $main-raised-margin * 2;
    }
    @include respond-to('md') {
      width: $container-md - $main-raised-margin * 2;
    }
    @include respond-to('lg') {
      width: $container-lg - $main-raised-margin * 2;
    }
  }
}

The respond-to mixin is this:
$breakpoints: (
  'sm': $screen-sm-min,
  'md': $screen-md-min,
  'lg': $screen-lg-min
);
@mixin respond-to($name) {
  // If the key exists in the map
  @if map-has-key($breakpoints, $name) {
    // Prints a media query based on the value
    @media screen and (min-width: #{inspect(map-get($breakpoints, $name))}) {
      @content;
    }
  }

    // If the key doesn't exist in the map
  @else {
    @warn "Unfortunately, no value could be retrieved from `#{$name}`. "
        + "Please make sure it is defined in `$breakpoints` map.";
  }
}

(From https://www.sitepoint.com/managing-responsive-breakpoints-sass/ and using breakpoint vars available from _variables_bootstrap.scss.)

@illepic thank you for pointing out. We've checked and you're right, the page is acting weird when you are close to the breaking point of the container. We will check the solution that you proposed and add it in the official archive.
Keep in touch,
Alex


Helper functions for SDFG scheduling

[x] Scope-aware topological sort traversal of SDFG states
[x] Some fixes for state-order traversal
[x] Properties denote their owner class (for properties of subclasses)
[x] Find whether a view is a slice (with squeeze/unsqueeze) and the correspondence

@acalotoiu I... don't know how without the next PR.

Accepting this now, on the basis that future PRs will add the deferred tests required before the new features are used in other contexts.


change np.int to int since np.int is deprecated

AttributeError: module 'numpy' has no attribute 'int'
np.int is deprecated in numpy 1.24 (https://numpy.org/devdocs/release/1.20.0-notes.html#using-the-aliases-of-builtin-types-like-np-int-is-deprecated)

Thank you for your contribution!


Doubt on calculation of smooth_l1 loss

In this line, according to the coment the code is supposed to filter out the samples marked as "ignore" state. What it does, instead, is extracting only the positive samples.
Shouldn't the line be: indices = tf.where(tf.keras.backend.not_equal(anchor_state, -1)) (or an equivalent version using >= 0)?
Unless I'm missing something, the network seems to be training only on positive samples for the regression branch.

Ah, it seems I did miss something. The targets for negative anchors are all zeros (and indeed for bounding box regression makes sense only for positive samples), got confused by the comment.


Test Error AND Train code

There are many superfluous varieties in train.py.
I don't readlly understand why the validation has a prediction part.
Furthermore, the test code can't  load the model from the training's model result.
At last, I don't know how to estimate the model in Jaccard,IoU,Dice and so on . I see I should use the Matlab in the reademe.md , but I  don't know how to use.
Can any one give me some advice in solving the test and estimate trouble?


TraceX - data / file upload for additional information / pictures is needed

Description
Ensure that additional information can be entered for Quality Alerts and Investigations
Impact

Functional demand
Usability feature

Additional information

[x] I'm willing to contribute to this feature

I think for this the standard regarding quality notification needs to be adapted.

Closed by NSC because no progress. Please reopen and assign an upcoming release if needed.


Redis strategy crashes because shop data isn't being returned

Issue originally created in shopify-node-app repo (https://github.com/Shopify/shopify-node-app/issues/55)

Let's wait for this PR to go in first https://github.com/Shopify/shopify-express/pull/7

If you rebase we should be good :)

one of us could rebase this if they don't by the end of the week I suppose


Add additional logging around search_controller's ES query building

Problem

Currently we only log the query itself, but not the process that led to the query being constructed. We are seeing queries get sent with sizes of up to 1800 but have no way of telling why 1800 was chosen for the size.
Description

Add logging to the _get_query_slice method. Log at each branch and be sure to log the variables that lead to each branch as well. The idea is to log as much as is necessary to understand how the method is working in production to produce the from and size results that are being sent to ES.
In addition, log some basic search facts like the term and the serializer data.
Be sure to introduce a "trace" variable so that we can easily follow a particular search request. This can just be a uuidv4 generated at the top of the search method and passed around and added to the logs.
Implementation

[ ] 🙋 I would be interested in implementing this feature.

@krysal @zackkrida @obulat Do y'all think this issue is still necessary? The original conceit of it is no longer valid: we pretty accurately understand why big queries get sent to Elasticsearch: dead links.
The methods mentioned in the issue description have all been well documented and unit tested at this point as well, so I think we understand them. If there are other motivations for adding process logging to the query building then we should update the issue description to accurately reflect the new motivations. Otherwise, we could close the issue. At the very least it doesn't seem like it should be high priority anymore :thinking:


Manage client errors

When Twilio or Sendgrid throws an exception, it is not stored either in Message entity
Thus, it's quite hard to debug.
Todo: when the exception is catched, set a special property in Message when coming back the Sender. Render this error message (if any) on the communication status.

Will be closed in today's release


possible way to expose devshell.toml's path?

Issue
I wanted to add a feature to the git.hooks module to allow specifying a path to a hook. I got all the logic wired in and working, but the issue is that I have to specify a full path :disappointed:
nix doesn't seem to have the ability to convert a string with a relative path to a full path, and since toml doesn't have a native notion of a path, I'm kinda stuck. builtins.toPath is depricated and doesn't really do anything.
The only work around I can think of is if we define a variable somewhere containing the current devshell.toml's path. Then I can specify a path relative to the location of the devshell.toml to the git.hook source file. I'm sure this will come in handy for similar reasons in other modules.
I guess the alternative would be to leave it as a nix only option, but setting from toml would be ideal.
Context
my devshell.toml is not at the root of the project, otherwise I probably would have thought to use DEVSHELL_ROOT

Do you want the file to be added to the /nix/store or just point to its location in the repo?
Where is that path being used?

Whatever you believe would be the best solution, if indeed we find a good solution
Here is a draft pr so you can see the implementation. #96
The directory containting the devshell.toml. The pre-commit file I would like to specify is also there. The nix specific parts are in the default.nix.

A couple of days ago, I also had a thought on relative-to-toml file paths in the context of wanting to load per-repo devshell modules (without resorting to .nix)

The module has a _file attribute that exposes its location. Ideally, it would be possible to set the type of attribute to types.path and have it be resolved if it starts with ./. @infinisil do you know if that would be possible?
Otherwise, we will have to revert to something less ideal like a post-processing step. Or install config.resolveRoot = builtins.dirOf file when the user uses importTOML


perf(modeling): remove use of slow Math.hypot

It turns out that Math.hypot is much slower than Math.sqrt, as pointed out by @briansturgill in #1095
I liked using hypot for increased accuracy, and nicer to read. But since it is slower, I think it should go away.
I had to remove the tests for accuracy on very small, and very large numbers (1e200+).
All Submissions:

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[x] Does your submission pass tests?

@platypii sorry. I'm not going for this without a good reason. For places where performance matters then the larger function can opt for using sqrt(). I think we should open an issue with gl matrix, and ask why that library uses hypot()

@z3dev that's fair. I'll see if I can measure the speed difference in real models.

Changing from hypot to sqrt makes a significant 30%+ speed up on unions. I think it's worth considering the change.

hypot
no hypot
improvement

disjoint (x=4)
480ms
480ms
0%

touching (x=2)
8500ms
5800ms
31.8%

overlap (x=1)
8900ms
5850ms
34.3%

inside (x=0)
7300ms
4800ms
34.2%

Test script
const jscad = require('@jscad/modeling')
const { sphere } = jscad.primitives
const { union } = jscad.booleans

const main = () => {
  const segments = 100
  const x = 1
  let s1 = sphere({ segments })
  let s2 = sphere({ segments, center: [x, 0, 0] })
  const startTime = performance.now()
  const out = union(s1, s2)
  console.log(`Union time ${performance.now() - startTime} ms`)
  return out
}

module.exports = { main }

that is massive, definitely worth changing

Changing from hypot to sqrt makes a significant 30%+ speed up on unions. I think it's worth considering the change.
hypot	sqrt	improvement
disjoint (x=4)	480ms	480ms	0%
touching (x=2)	8500ms	5800ms	31.8%
overlap (x=1)	8900ms	5850ms	34.3%
inside (x=0)	7300ms	4800ms	34.2%
Test script

I'm not saying that the replacement shouldn't be done. I'm saying that we should be selective about the replacement. Booleans may benefit but we don't want to create other issues.
We should do some research, and ask the gl matrix geeks. And try to understand the math completely.

https://github.com/toji/gl-matrix/issues/446

FYI, here's the js-core implementation of hypot()
https://github.com/zloirock/core-js/blob/master/packages/core-js/modules/es.math.hypot.js

The core-js polyfill implementation is somehow even slower than Math.hypot despite using sqrt. :vomiting_face:

environment
hypot
sqrt
core-js

chrome
235ms
12ms
328ms

firefox
64ms
12ms
160ms

safari
390ms
74ms
1830ms

node
248ms
11ms
370ms


Fix jwt plugin doc

Summary

Fix applicable algorithms on secret parameter of jwt plugin
Update available algorithms for jwt plugin
Fix response code when JWT has invalid signature

Reason

Make doc up-to-date.
Testing

Relevant commits in kong are included in commit message for reference

@bungle can you verify that this change is correct?

@pallxk can you please resolve the conflicts in this PR? You might need to update your fork first.
Once that's done, I can merge and publish.

I had some time and rebased. Thanks for the contribution @pallxk, and sorry it took a while to review 🥳


[Batch] Implement Netlify Best Practices

I talked to @krider2010 today 👋 about a few things that I'd love input for from Netlify. Also I don't know what I don't know so if anyone things our setup is overly complicated, I'm more than happy to change it.
Our deployment setup is as follows:

I disabled preview deploys because the URLs are variable and both GitHub Apps and Slack Apps to have a configured URL to which they send their respective webhook requests. So I figured the simplest would be is to only enable branch deploys for staging, as it results in a predictable URL: https://staging--github-app-slack-demo.netlify.app/
The two handlers for the GitHub and Slack webhook requests are implemented as Netlify Functions, specifically

https://github.com/gr2m/github-app-slack-demo/blob/main/netlify/functions/github-webhooks/
https://github.com/gr2m/github-app-slack-demo/blob/main/netlify/functions/slack-events/

That being said, it would be possible to make preview deploys with variable deploy URLs possible.

Both GitHub and Slack allows to programmatically update the webhook URLs. So we could register a pool of both GitHub Apps and Slack Apps for testing, and implement some kind of logic that takes one of each pool for each pull request that has a preview deployment.
Slack Apps can even be created and deleted programmatically, we might not even need a pool here.

I'd love to hear what people think we should do here as a best practices for this kind of apps. The deployment through staging works, but

I miss the comments by https://github.com/apps/netlify like this one: https://github.com/gr2m/github-app-slack-demo/pull/9#issuecomment-2175740626. For branch deployments, there are not comments or deployments in GitHub's UI that would indicate that anything was deployed.
Deploying to staging is tedious. First, a staging branch based off latest main needs to be created and pushed to the GitHub repository. Then changes can be made. Once merged, the staging branch gets deleted and will need to be re-created when needed again.

Another thing I'd like to do is to add a "deploy to Netlify" button to the GitHub README.md file, and make the flow of deploying an own instance with their own GitHub App / Slack App as seamless as possible. I'm not sure what the best strategy would be, but it might make sense to implement a "setup" state of the app that would be served at the / root path if no GitHub App and/or Slack App is registered. This Setup UI can walk the user through the necessary steps and e.g. show all environment variables in an .env file format which could be directly copied and pasted into the respective Netlify site configuration, after which the app could be re-deployed. If Netlify has APIs for it that we could implement with an OAuth flow, that would be even better, and probably quite fun to build.
Tasks

[ ] Implement Netlify deployment best practice for GitHub/Slack Apps
[ ] Document deployment and reasoning behind it
[ ] Add "Deploy to Netlify" button and necessary implementation to make an initial setup as frictionless as possible

@krider2010 here is what I see at https://app.netlify.com/sites/github-app-slack-demo/configuration/notifications#deploy-notifications

I don't think that turning the Deploy Previews settings back on will have the desired result. I assume it will add the comments to pull requsets from staging -> main, but it will also create preview deploys for any other pull request against either staging or main based on what it says in the form

Hey @gr2m! Here are some brief thoughts — I've tinkered with a similar approach (using Netlify to support Slack apps).
Programmatic URL workflows
When I've needed to register a URL remotely, I typically stick with the branch deploys as you've mentioned. However, I don't disable deploy previews. But if there's a need to test a one-off condition, I might just add that specific URL temporarily to the Slack app (as an example).
That's been enough for me. Aside from some workflow annoyance, is there a need to be more dynamic in the app URLs? Or is it just to avoid the staging branch workflow?
Using a predictable branch

I miss the comments by https://github.com/apps/netlify like this one: https://github.com/gr2m/github-app-slack-demo/pull/9#issuecomment-2175740626. For branch deployments, there are not comments or deployments in GitHub's UI that would indicate that anything was deployed.

Not sure I'm following this one. The trouble here is that if you use the staging workflow, you deploy by pushing directly to staging, so you don't get PR comments? Is there more to this?

Deploying to staging is tedious. First, a staging branch based off latest main needs to be created and pushed to the GitHub repository. Then changes can be made. Once merged, the staging branch gets deleted and will need to be re-created when needed again.

I hear you on this — I see how it can take you out of your typical way of working. I'll wait to see your response to my previous question before suggesting anything here.
But, in the meantime, if you protect the staging branch, it shouldn't be automatically deleted when merging into main, correct? (I haven't tested this in awhile.)
Deploy to Netlify button
As you're working through this, one thing that may help is using templates. You can include a set of environment variables that someone would have to fill out when creating a new Netlify site.
Would that solve your challenges in a new site setup? Or are there still OAuth things that you wouldn't be able to solve with environment variables?

Hi Sean, thank you so much for getting back to me!

I don't disable deploy previews. But if there's a need to test a one-off condition, I might just add that specific URL temporarily to the Slack app (as an example).

I find it confusing to get the notifications for deploys that are not usable because my staging app(s) are not hooked up to it. Ideally, I would like the notification or at least a deploy event, but on on pull requests from staging to master.
But the more I think about it, the more I like the idea of just using the standard preview deploys, and making sure they are usable with hooked up slack and github apps using automations on my side. I feel this could be a big rabbit whole that I don't have time for right now, but I've run into this before with past GitHub Apps I've built. I could imagine a custom automation like this

Register a pool of GitHub apps for staging, e.g. 10. Store all credentials and repository secrets.
Add a Slack workspace configuration access token to repository secrets.
Use a Repository variable to store JSON in order to keep state
When a new pull request is created, create a new slack app with the webhook URL pointing to the new deployment (based on branch) and update the webhook URL for one of the idle GitHub Apps to point to the new deployment as well. Update the state accordingly in the repository variable.
Now the deployment can be fully used for testing.
When the pull request is merged or closed, delete the slack up, reset the GitHub App, and update the state in the repository variable.

If done well, this could be a great setup for building and collaborating on GitHub and/or Slack apps hosted on GitHub.
I think I would implement this as two GitHub Actions, one for GitHub Apps, one for Slack. That way they would be useful independently of each other.
What do you think?

But, in the meantime, if you protect the staging branch, it shouldn't be automatically deleted when merging into main, correct? (I haven't tested this in awhile.)

Yes, a branch can be protected from being deleted. However, if staging is not deleted and re-created it becomes stale. We doing a new staging deployment, I want it to be based on the latest version of the repository's default branch.

As you're working through this, one thing that may help is using templates. You can include a set of environment variables that someone would have to fill out when creating a new Netlify site.

Thank you, that's a great start!
I also saw that there are APIs to set environment variables, so we could create a wizzard-like setup experience in future, if anyone would like to build it: https://open-api.netlify.com/#tag/environmentVariables. We can start out by asking users to create and pass in a Netlify personal access token, and if we want to improve on that, register a Netlify App in future?

Interesting! That could definitely work, although I agree it sounds like a big lift. Not sure there's a great workaround for what you're trying to accomplish.

We can start out by asking users to create and pass in a Netlify personal access token, and if we want to improve on that, register a Netlify App in future?

Great idea – definitely a possibility.

Sounds like you have a theoretical path forward, just need to find the time to build it out?

Sounds like you have a theoretical path forward, just need to find the time to build it out?

Yes. Not sure when that'd be right now. But if anyone from @netlify would be interested in collaborating I can prioritize it.

Sounds great, @gr2m! We'll be chatting internally over the next week or so (have some folks OOO right now) and will get back to you shortly.

Checking back in, @gr2m — we don't have the engineers to put on this right now. But we can certainly consult as you make progress. Ping me once you get some time to start digging in!


MRG, ENH: Allow setting tqdm backend

Closes #8096
@mmagnuski want to give it a try?

Great, thanks, I'll take a look tomorrow! Sorry I didn't follow follow through with the issue.

@hoechenberger Good, I'll test it today!

Friendly poke @mmagnuski 🙃

I'll just go ahead and merge this :) @mmagnuski If you have any suggestions, we can address them in a followup PR.

Thanks, @larsoner!

Oh, sorry, @hoechenberger, forgot about this, and had poor internet connection till now.

@mmagnuski No worries, I can relate 😇


feat(ui): show power source + battery level

This PR adds a power column to the nodes table showing the power source (mains-powered, battery) using an icon and the battery level for battery-powered devices.
Fixes #1638
Variant 1 (commit 3ae9d71, using already included icons):

Variant 2 (commit 8c5100a, using svg-icons):

Introducing valueFn for table property definitions
Since the battery level is not available as a direct node property the property definition for the nodes table had to be extended to show complex nested values by defining a value function using valueFn:
nodesProps: {
  ...
  power: { type: 'number', label: 'Power', valueFn: node => this.getBatteryLevel(node) },
  ...
}

Using valueFn the nodes table is now capable of displaying arbitrary node information directly on the table.
NOTE: This is a work-around to limitations in Vue Data Table which is described here: https://github.com/vuetifyjs/vuetify/issues/7066
Known issues:

[x] Sorting by battery level does not work. May be solved using custom sorting (see https://stackoverflow.com/questions/52678905/how-to-use-vutifys-custom-sort).
[ ] Grouping by battery level does not work. The implementation of genGroupedRows() may have to be adjusted to respect valueFn to extract the battery value instead of directly accessing it using the property name: https://github.com/vuetifyjs/vuetify/blob/4cc80ca7bea963be875b3ab0f9ba03a8b21c4d92/packages/vuetify/src/components/VDataTable/VDataTable.ts#L316

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

42 of 60 (70.0%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 29.957%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/components/nodes-table/nodes-table.js
0
18
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
3660

Relevant Lines:
12857

💛  - Coveralls

@robertsLando, I'd need some advice on the approach I've taken to deal with the battery level as well as the use of additional icons.
Approach to support the battery level
To display the battery level on the table I see two following approaches:

Extend the table to be able to dynamically find the required nested information (e.g. battery level)
Extend the node object to provide the battery level as top level property

I've implemented the 1st approach since I wanted the table to display as much information as possible without having to change the general node object structure. It works but for displaying and filtering but not yet for sorting and grouping.
I may solve these issues but I'm unsure if this is really the way to go, since extracting the battery level requires to iterate a lot to find the respective command class that holds the value which may have a negative impact to the overall performance of the table.
Before I continue to solve the remaining issues I'd like to get your opinion if it's not better to provide the battery directly as property of the node object.
Using icons from materialdesignicons.com
Another feedback I'd like to have is for the use of additional icons, since I find the available ones a bit limited.
I used Material Design Icons which recommends this way to include additional icons:
https://dev.materialdesignicons.com/getting-started/vuejs
This way I was able to show the battery level directly in the icon.
If I remember correctly we already had a discussion on increased file size because of additional icons I observed the size of the dist directory and it seems to have no negative impact. In fact it seems to take less space than using the built-in icons but I'm not sure, if that is really the case - an independent look would be helpful.
It seems that only the used icons are included into the final distribution due to tree shaking.

Hi @ahochsteger , thanks for your PR! I love to keep the code as simple as possible, if it was me I would have choose to set the battery level as a node property instead. It is also used on mesh graph so it makes sense to have it there. Just remember to add the code to the value update in order to keep it updated

Next steps:

Fix sorting and grouping by using custom sorting functions (see https://stackoverflow.com/questions/52678905/how-to-use-vutifys-custom-sort)

@zwave-js-bot fix lint

@zwave-js-bot fix lint

Battery level PR
@robertsLando the PR is ready to be reviewed from my point of view.
I'll give you a quick summary of what changes after your last review ...
The approach using a function to extract the battery level dynamically worked but had the limitation that VDataTable does not provide a way to override the standard behavior for grouping, thus grouping was broken for this column.
We could easily disable grouping for such columns but I wanted to have it working for the battery level.
Therefore I tried the other approach to map the battery level directly to a node property.
It turned out to be much easier than I thought and solved it by:
Map node properties (e.g. batteryLevel) to a ValueID (e.g. commandClass 128, property 'level') in ZwaveClient.ts:
// Define mapping of node properties to ValueIDs
const mappedNodeProps: Record<string, ValueID> = {
	batteryLevel: {
		commandClass: CommandClasses.Battery,
		property: 'level',
	},
}

The functionality is encapsulated in the function _mapNodeValuesToProps() that is triggered on node ready and value changed events.
The node property definitions for the table can now be extended using the following functions to be more dynamic:

valueFn: A function to extract the column value from a given node (as it already was). Note though, that grouping may have to be disabled for such columns, since it is not working in VDataTable (until https://github.com/vuetifyjs/vuetify/issues/7066 is solved).
formatFn: A function to format the value for displaying (e.g. add '%' to the battery level on grouping)
sortFn: A function to define a custom sorting for the column (used for batteryLevel to sort mains-powered devices like 100% battery).

Finally the Zwave graph has been refactored to directly use node.batteryLevel istead of iterating over the ZWave values.
It would be great if you could give me your final feedback to finalize the PR :-).


Forecast branch

Build PROPHET models to predict store performance, category performancce and overall performance
Deployed models in FastAPI endpoint

Model Working Correctly

Great work!


Reordering Chains is not permanent

Describe the bug
After reordering chains (e.g move THORChain to the top, on top of BTC), once we move to another screen and back, it reverts to default ordering.
Expected behavior
Ordered chains to remain.
Smartphone (please complete the following information):

Device: Both iPhone 11 Pro & 14
OS: 17.4.1
Version 24

@Rockindash would it be possible to also implement this request https://github.com/vultisig/vultisig-ios/issues/449 ?
As than we could remove the re-arrange button on the right on main to be concruent with figma design.

Instead of allow user to manually order the tokens, let's do this

Order by token balance , descending