Write a guest blog post

Write a guest post for my personal blog.
Any topic will work, but the general theme I try to follow is "bitesized thoughts about the web and the people who use it." Some types of post that I've written (and you could write) include:

Sharing a side project
Explaining the solution to a technical issue
Sharing a personal realization related to doing creative or technical work
Sharing an inspirational quote or passage from something I read or listened to, and a brief thought about it.

It's a jekyll blog. Posts are written in markdown and put in the _posts folder. See other posts for examples. The build is over a minute long.
For what it's worth the blog currently averages about 150 visitors a day, and posts go out to about 100 or so subscribers via RSS and email.
Update: Feel free to submit a post, even if other people have already submitted one. I'm planning on publishing several, and I'll leave this issue open until I'm done accepting them.

I got a some great blog submissions from @paniya, @scriptonist, @thibautvs, and @PrajapatiAnkit, which is probably enough for now. Thanks all! I'll merge and publish these over the next month or so.

Hi,
Glad you like it. Can you send me the link when you published it. So I can
share it.
Praneeth Perera
Undergraduate in University of Colombo School of Computing
IEEEXtreme Country Lead
Member of  ACM
Member of ICASL
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Mobile : +94 77 85 89 445
On Mon, Oct 30, 2017 at 12:07 AM, Bryan Braun<EMAIL_ADDRESS>wrote:

Closed #16 https://github.com/bryanbraun/bryanbraun.github.io/issues/16.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bryanbraun/bryanbraun.github.io/issues/16#event-1315650652,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APTl6QOB9rLGw1uO6bLQ34A-nFiyctWnks5sxMXRgaJpZM4QJJ_z
.

@paniya, yeah, I should be able to do that. 😄


[Secrets Store CSI Driver] Amazon Secrets Manager

See: https://github.com/aws/containers-roadmap/issues/895
https://aws.amazon.com/fr/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/

/reopen

/reopen

/reopen

/reopen


VFS-785 - credentials conflicting when root and proxy hosts are same name but different ports

VFS-785 e.g. if URI for root is https://github.com:8443/test.txt and proxy is github.com:3128 then credentials may get overridden in credential provider map

@satish-csi Needs unit tests.


Update NSData+ImageContentType.m

AVFileTypeHEIC is defined in AVFoundation via iOS 11 and A9 above CPU(iPhone 7 above models). For Example, iPhone SE +iOS11,the photo is not ".heic"
New Pull Request Checklist

[x] I have read and understood the CONTRIBUTING guide

[x] I have read the Documentation

[x] I have searched for a similar pull request in the project and found none

[x] I have updated this branch with the latest master to avoid conflicts (via merge from master or rebase)

[x] I have added the required tests to prove the fix/feature I am adding

[x] I have updated the documentation (if necessary)

[x] I have run the tests and they pass

[x] I have run the lint and it passes (pod lib lint)

This merge request fixes / reffers to the following issues: ...
Pull Request Description
...

AVFileTypeHEIC is defined in AVFoundation via iOS 11 and A9 above CPU(iPhone 7 above models)

AVFileTypeHEIC is defined in AVFoundation via iOS 11 and A9 above CPU(iPhone 7 above models)

Codecov Report

Merging #2091 into 5.x will increase coverage by 0.14%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              5.x    #2091      +/-   ##
==========================================
+ Coverage   78.45%   78.59%   +0.14%
==========================================
  Files          36       36
  Lines        3420     3420
==========================================
+ Hits         2683     2688       +5
+ Misses        737      732       -5

Impacted Files
Coverage Δ

SDWebImage/NSData+ImageContentType.m
53.22% <ø> (ø)
:arrow_up:

SDWebImage/SDWebImageManager.m
77.31% <0%> (-1.04%)
:arrow_down:

SDWebImage/SDImageCache.m
65.8% <0%> (+0.51%)
:arrow_up:

SDWebImage/SDWebImageDownloaderOperation.m
89.79% <0%> (+0.68%)
:arrow_up:

SDWebImage/SDWebImageWebPCoder.m
85.98% <0%> (+1.13%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd17cb8...f7f7cb1. Read the comment docs.

HEIF and HEIC is not the same, it a proper subset. The comments there just a tip and the value for that is the same as what AVFileTypeHEIC  contant value.
If you mean to add AVFileTypeHEIF. We should also add one new SDImageFormat SDImageFormatHEIF to represent all the heif format.

After that real device test, this comments is totally right. iPhone 6 with iOS 11 also define this constant as well. There is no what and A9 above CPU(iPhone 7 above models). For this one, it just means that if you try to encode a CGImage with these two kUTType, CGImageSouceCreate will return nil. But this two constants is already defined.
AVFileTypeHEIC: "public.heic"
AVFileTypeHEIF: "public.heif"


188364196 Fix Text Crash

PT Story: https://www.pivotaltracker.com/story/show/188364196
In some circumstances, StoryQ was still sending a slate object {} to a text tile. In modern slate, { text: "" } is the blank object, and an empty {} causes the library to crash.

@tealefristoe I'm going ahead and merging this so I can merge in another change to master into this feature branch.


Exception transformer

Include a simple function that runs a computation, and if it throws uses a provided function to throw a different exception.
transformException :: (Exception e, Exception e', MonadExcept m) => m a -> (e -> e') -> m a would be the relevant type signature I think.

I said I'd do this next week but actually it's already done and will be in the next version of my pr.
Notably I've restricted the exception type to BotException and added a constant version which just replaces all exceptions with some constant exception.


Concurrent Database Transaction worries

Hi there,
I've been testing Falcon since a few days and am pretty much in awe what has been accomplished here.
However, I've stumbled upon this comment on Hackernews regarding the per-thread nature of the pg gem and the issues with concurrent transactions.
So basically, my question is if this has already been addressed in some way or is this still an issue? What would we need to do to fix this? Write a new adapter using the db-postgres gem? Been looking at the examples in the readme and it doesn't mention concurrent transaction, only concurrent queries. So right now I'm hoping I can get a bit of advice how to proceed. Would it be wise to switch back to puma while concurrent transactions are still being worked on?

Use the latest version of Rails with per-fiber concurrency.
https://blog.saeloun.com/2022/02/23/rails-fiber-safe-connection-pools/

Oh wow, you guys rock so much


lifeline made my docker build fail starting about 2 weeks ago because of scipy

this is the error message:
`Collecting scipy>=1.2.0
Downloading scipy-1.5.2.tar.gz (25.4 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpl_7txen4
cwd: /tmp/pip-install-nkm0jqsk/scipy
Complete output (137 lines):
lapack_opt_info:
lapack_mkl_info:
customize UnixCCompiler
libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
NOT AVAILABLE
openblas_lapack_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

openblas_clapack_info:
customize UnixCCompiler
customize UnixCCompiler
  libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries tatlas,tatlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries tatlas,tatlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries tatlas,tatlas not found in /usr/lib/
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
customize UnixCCompiler
  libraries satlas,satlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries satlas,satlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries satlas,satlas not found in /usr/lib/
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/local/lib
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in /usr/lib
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in /usr/lib/
customize UnixCCompiler
  libraries lapack_atlas not found in /usr/lib/
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

lapack_info:
customize UnixCCompiler
  libraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
  NOT AVAILABLE

lapack_src_info:
  NOT AVAILABLE

  NOT AVAILABLE

setup.py:462: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-sgdinmto'), proceeding with generating Cython sources and expanding templates
  ' '.join(sys.argv[1:])))
Running from SciPy source directory.
/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/numpy/distutils/system_info.py:624: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()
/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/numpy/distutils/system_info.py:624: UserWarning:
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  self.calc_info()
/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/numpy/distutils/system_info.py:624: UserWarning:
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  self.calc_info()
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 133, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 267, in run_setup
    self).run_setup(setup_script=setup_script)
  File "/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 158, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 583, in <module>
    setup_package()
  File "setup.py", line 579, in setup_package
    setup(**metadata)
  File "/tmp/pip-build-env-5r_xbiq2/overlay/lib/python3.6/site-packages/numpy/distutils/core.py", line 135, in setup
    config = configuration()
  File "setup.py", line 477, in configuration
    raise NotFoundError(msg)
numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
----------------------------------------

ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpl_7txen4 Check the logs for full command output.`
when executing pip install -r requirements.txt and the requirements.txt file contains lifelines==0.25.0, I tried with 0.25.2 and I get the same error. If I comment lifeline out everything builds smoothly. Would it be possible to lock spicy to a stable version?

Hi @grugna, hm, I'm not sure what is causing this. Are previous version of lifelines causing problems too? (ex: v0.24.x?)

Would it be possible to lock spicy to a stable version?


Incrementally-populated Zarr Arrays

I believe there is a common usage pattern of Zarr that should be documented and more widely known – populating Zarr arrays incrementally and on-demand, chunk-by-chunk.
We have built internal tooling around this that resembles the chunk manifests (https://github.com/zarr-developers/zarr-specs/issues/154, https://github.com/zarr-developers/zarr-specs/issues/287, virtualizarr#chunk-manifests) and VirtualiZarr (https://github.com/TomNicholas/VirtualiZarr/issues/33, virtualizarr#manifestarray).
This issue explains the usage pattern and opens the question of how our tooling could be open-sourced.
Incrementally populating Zarr arrays
A core Zarr usecase for us is to initialize an empty Zarr array with a pre-determined grid and then fill it with data as needed.
For a great overview of this usecase, see Earthmover's recent webinar and the relevant code repository. Note that in the webinar, the entire grid is populated. In our case, parts of the grid are populated on-demand.
  Code example
This code defines an empty Zarr array given an example schema. For a fuller example, see create_dataset_schema in Earthmover's repository.

from odc.geo.geobox import GeoBox
from odc.geo.xr import xr_zeros
import pandas as pd

def define_schema(varname, geobox, time, band):
    schema = (
        # Chunks -1 to have a Dask-backed array
        # But without creating a massive computational graph
        xr_zeros(geobox, chunks=-1)
        .expand_dims(
            time=time,
            band=band,
        )
        .to_dataset(name=varname)
    )

    schema['band'] = schema['band'].astype(str)
    encoding = {varname: {"chunks": (1, 1, 256, 256)}}

    schema.to_zarr(
        'schema.zarr',
        encoding=encoding,
        compute=False,  # Do not initialize any data: no chunks written
        zarr_version=3,
        mode='w'
    )

geobox = GeoBox.from_bbox(
   (-2_000_000, -5_000_000,
   2_250_000, -1_000_000),
   "epsg:3577", resolution=1000)

define_schema(
    "data",
    geobox,
    time=pd.date_range("2020-01-01", periods=12, freq='MS'),
    band=["B1", "B2", "B3"]
)

This pattern is powerful, as it enables us only to compute and ingest what's needed in a query-or-compute fashion. Without going too far from the scope of this issue, it enables us to cascade the computation/ingestion of data in a chunk-by-chunk manner illustrated in the following diagram.

Note: The details of this example are made up for the purposes of this issue

Chunk-level MetaArray
When working with sparsely populated arrays, the natural question arises: which chunks are populated?
To this end, we employ an object we call the chunk-level meta-array (MetaArray for short). This is a second array where each pixel corresponds to a chunk in the original array. The value of the pixel is

1 if the corresponding chunk is populated
0 if the corresponding chunk is empty

Note: Chunk initialization is not the only information one can keep in such MetaArray. For example, the MetaArray could keep track of min & max of each chunk, enabling predicate pushdown, or aggregate statistics for providing approximate answers on the chunk level.

  Toy code example
Note: This is only a very rough sketch of the resulting object to illustrate the point.
import xarray as xr

def create_metaarray(schema):
    """Create meta-array from a schema.

    A very rough sketch. Not the actual implementation.
    """
    _, _, cy, cx = schema.data.encoding['chunks']

    # Create spatial pixels with coordinates equal to the centre of the chunk
    xx = schema['x'].values
    xdiff = xx[cx] - xx[0]
    x_meta = xx[::cx] + xdiff/2

    yy = schema['y'].values
    ydiff = yy[cy] - yy[0]
    y_meta = yy[::cy] + ydiff/2

    # Initialize an empty meta-array
    meta = xr.DataArray(
        data=0,
        dims=schema.dims,
        coords=dict(time=schema.time, band=schema.band, x=x_meta, y=y_meta),
    )

    meta.to_zarr('schema-meta.zarr', zarr_version=3, mode='w')

schema = xr.open_zarr('schema.zarr', zarr_version=3)
create_metaarray(schema)

# When writing to the Zarr array, also populate the metaarray.

Relationship to Chunk manifest & VirtualiZarr
Increasingly, the work in the Zarr world resembles the data structure we have built.

The chunk manifest keeps a data structure containing information about each chunk (https://github.com/zarr-developers/zarr-specs/issues/154, https://github.com/zarr-developers/zarr-specs/issues/287).
VirtualiZarr, the key piece of the upstreaming of kerchunk, will likely store this information in Numpy arrays (https://github.com/TomNicholas/VirtualiZarr/issues/33) and has even considered xarray (https://github.com/TomNicholas/VirtualiZarr/issues/33#issuecomment-2000414759, https://github.com/TomNicholas/VirtualiZarr/pull/107), while also tackling the question of initialized chunks (https://github.com/TomNicholas/VirtualiZarr/issues/33#issuecomment-2000529283, https://github.com/TomNicholas/VirtualiZarr/issues/32, https://github.com/TomNicholas/VirtualiZarr/issues/16).
Finally, ZEP 5 (https://github.com/zarr-developers/zarr-specs/pull/205) plans to propose a way to store chunk-level statistics, one of the potential use-cases of MetaArrays.

The similarities in these proposals and implementations make me think there is a path where the MetaArrays could become an open-source part of the Zarr ecosystem, potentially even enabling further use-cases such as chunk-level aggregate statistics outlined in ZEP 5.
The question is what the exact path forward would be.
Potential path forward
Here is my best guess at what the path forward could be.
There are essentially two layers to MetaArrays

Information on the chunk level in the index space, stored on disk. This could be taken over by the chunk manifest.
Convenient in-memory representation & an API to query this information in the label space. This would remain the domain of MetaArrays.

Note: Here I'm using the "index" and "label" space terminology from Xarray

1. Information on the chunk level
In the future, 1. could become part of Zarr. The chunk manifest would only need to represent the information on whether a chunk is initialized to contain the full information needed for construction of the MetaArray.
Currently, MetaArrays are stored as a separate Zarr array and updated using zarr. In the future

They could be read from whichever format chunk manifest is stored in (of which Zarr itself is one option)
Their updating & consistency would then be completely taken care of by the storage transformer for the chunk manifest

2. API to query this information
To be easily queryable, the chunk-level information must be exposed back in the coordinates of the original dataset and provide an API for its querying.
Xarray is our choice for this API. For example, asking the question "do we have data for this geometry" simply becomes a rioxarray clip operation and xarray reduction.
A similar conclusion was reached by VirtualiZarr, where the use of xarray e.g. exposes a familiar API for combining of multiple datasets.
MetaArrays could become another package building on top of Zarr, similar to VirtualiZarr. However, if the chunk manifest already contains information for the construction of MetaArrays and VirtualiZarr uses this information to construct Xarray(-like) datasets, a greater integration could be possible. For example in the form of common functionality to read the chunk manifest into xarray format.

Note: We have further APIs that utilize Pydantic and GeoPandas for further processing of chunk information. For simplicity, I am focusing only on the Xarray representation here

Questions
I would appreciate suggestions from the Zarr community on where to best focus the efforts.
Here is a set of concrete open questions and discussion points:

Does the chunk manifest storage transformer cover the data needed by MetaArrays? Could the chunk manifest also represent information about chunk initialization? Or should this information be another storage transformer that shares implementation with the chunk manifest?
What is the potential relationship to VirtualiZarr? Could they share functionality, or could a single package cover both uses? Perhaps there's a common functionality for going from the manifest into an xarray object?
Have I missed any potential difficulties here?
Is there a different path to the one that I outlined?

A secondary set of questions is around the usefulness of this – would the community find the MetaArrays useful? Be it for storing chunk initialization information, or aggregate statistics. Are people perhaps already using variants of this functionality? Creating this functionality was necessary for us – and thus likely for others, too!
Thank you for any suggestions and answers!

@DahnJ this is super interesting!

When working with sparsely populated arrays, the natural question arises: which chunks are populated?

I actually recently had a separate use case where I built a really big Zarr store chunk-by chunk using a big parallel dask operation, and at the end a couple of tasks had failed. The result had some un-initialised chunks, but finding out which ones are not initialized is trickier than it should be. It did not actually occur to me that this problem is closely related to the manifests I built in VirtualiZarr, but you're completely right that it is!
It also reminds me of this little tool that CarbonPlan made for quickly checking how much of a Zarr store had actually been initialized  (@andersy005 wrote it I think). I was already going to suggest upstreaming that.

What is the potential relationship to VirtualiZarr? Could they share functionality, or could a single package cover both uses?

I would imagine a lot of this can be upstreamed from VirtualiZarr into zarr-python at least. Reminds me also of @d-v-b 's issue https://github.com/zarr-developers/VirtualiZarr/issues/71, which also asks whether all Zarr arrays should have an intermediate chunk manifest layer.
In the meantime we could mess around with adding methods to VirtualiZarr's ManifestArray e.g. .chunks_initialized(self) -> np.ndarray, which would return your numpy array of 1s and 0s, with the same shape as the chunk grid.
FYI non-initialized chunks in VirtualiZarr are currently represented by the numpy array containing the chunk manifests' paths having entries of empty strings.

Perhaps there's a common functionality for going from the manifest into an xarray object?

Remember that you need more than just the Chunk Manifest itself to get to the xarray representation - you also need the array name, the dim names, and the dtype.

Finally, ZEP 5 (https://github.com/zarr-developers/zarr-specs/pull/205) plans to propose a way to store chunk-level statistics, one of the potential use-cases of MetaArrays.

@rabernat suggested something very similar on https://github.com/zarr-developers/VirtualiZarr/issues/33#issuecomment-2000617606

I'm not entirely clear how you are proposing to use the chunk manifest format
it is mostly a read-only format
The chunk manifest, operating at the key-value store level, also doesn't very naturally offer a way to store any richer information than just whether a chunk is present or not.

My initial understanding of this issue (@DahnJ please correct me if I'm wrong), was that it's more concerned with giving users access to useful in-memory representations about the state of the chunk grid. It's less about actually changing the chunk manifest format on-disk. Perhaps it would have made more sense to raise it on the zarr-python repo instead?
But then this

Their updating & consistency would then be completely taken care of by the storage transformer for the chunk manifest

makes it sound much more like the whole arraylake database-of-chunks idea...

Thank you for the quick feedback @TomNicholas and @jbms
Chunk manifest
I think that the crux here is what the ambitions for the chunk manifest are.

Read only format: If it's a "mostly read-only format" that gets created with the Zarr array and does not support updates, then I agree that it's not a suitable solution for an incrementally populated dataset.
Storage transformer: If it's implemented as a storage transformer that supports updating the chunk manifest upon writing into the Zarr array, then I think it's sufficient.
Concurrent writes: The question of write contention is an important one. For our purposes here, I would find it sufficient to assume no concurrent writes. Should the users need concurrent writes (we do), they can rely on solutions such as Arraylake's database-of-chunks (which we are users of). There's also a whole interesting discussion around alternative storage formats for the manifest – e.g. can we use a secondary Zarr array for this? That's along the lines of what we're currently doing. However, short of that, the no-concurrent-writes chunk manifest and MetaArrays still seem like a useful construct to pursue.

I would therefore say that as long as chunk manifest supports changes made to the zarr array, it's a good candidate to use for MetaArrays. In such a case, the issue becomes only about providing a useful in-memory representation, exactly as @TomNicholas notes.
Information necessary for MetaArrays

Remember that you need more than just the Chunk Manifest itself to get to the xarray representation - you also need the array name, the dim names, and the dtype.

Yes, you're right. For the MetaArrays, we also require the coordinate arrays. That way the user can query populated chunks in the label space – answering questions such as "do we have 2022 data for Ecuador".

chunk manifest, operating at the key-value store level, also doesn't very naturally offer a way to store any richer information than just whether a chunk is present or not.

Right, storing such information would be an extension of the manifest's planned capabilities. We can thus focus on the primary goal here – chunk initialization information, which the chunk manifest, as planned, does represent.
Other answers

binary mask indicating which individual elements have been written, in case writes can be unaligned to chunk boundaries

This is a good point. We restrict our writes to entire chunks precisely for that reason. That allows us to then only store information at the chunk level.

One issue that can come up with zarr arrays is the fact that there can be ambiguity between a chunk that was never written, and a chunk that was written but happens to be equal to the fill value at all positions. In some implementations, chunks that are equal to the fill value are not stored.

Thanks for noting that, the behaviour around write_empty_chunks is definitely something to be aware of and have clarity around when dealing with chunk initialization information.

This is a very cool idea and it does address one of the challenges people inevitably encounter when they start scaling zarr arrays. But I also agree with two points @jbms made:

The issue with the chunk manifest format is that, being a single json file either stored separately or embedded in the array metadata file, it is mostly a read-only format. If writes are supported at all, you are likely to run into contention if trying to concurrently write from multiple machines, unless you have some very specialized JSON storage system designed to support concurrent writes.

To elaborate, the basic premise of a conventionally chunked (i.e, unsharded) array is that concurrent chunk write operations are independent. A chunk manifest / index adds a synchronization step, which makes concurrent write operations dependent. This introduces new complexity (e.g., write contention) and new failure modes (e.g., writing a chunk succeeds but updating the manifest fails). New complexity is fine (and necessary in many cases) but it's important to map it out.

The chunk manifest, operating at the key-value store level, also doesn't very naturally offer a way to store any richer information than just whether a chunk is present or not.

I would rephrase this as "why not use a database / tabular format here?" Of course if we stick with vanilla zarr there are only two options, JSON or an array, and the array option looks more attractive than JSON. But if you want a rich, dynamic representation of a chunk mutation, then you probably want to store a record with multiple data types for each chunk (e.g, a creation time stamp, a modification timestamp, a size, some summary stats of the values in the chunk, etc), and in that case a Zarr array is going to be a very poor fit IMO. So I would be curious to see an analysis of how adding a non-zarr ingredient to the mix could solve this problem, weighed against the complexity that adds.

I will also throw out a random thought that occurred to me: zarr doesn't have a good way of structurally expressing a dependency relationship between arrays. But this could be solved if arrays could contain other arrays. If we support nested arrays, then it would be very natural to store a meta array as a sub-array of the array it describes. I'm not suggesting we do this, just pointing out that it solves a particular problem.

Thanks @dvb for providing helpful additional detail. You're right – this complexity absolutely needs to be mapped out. For the purposes of this issue, as long as the chunk manifest allows for updates, then I think it's a sufficient solution to power the MetaArrays (e.g. as a feature in VirtualiZarr).
I will wait for the community to converge on a concrete proposal for the chunk manifest – and potentially join in the conversation. Hopefully this issue serves as additional motivation for chunk manifests to go beyond just a read-only format.
Chunk Manifest update
Discussing the details of the chunk manifest is not in the scope of this issue. However, to help the manifest discussion with the additional context from this issue, here's a summary of what's been said so far and some of my thoughts.
Failure mode: Inconsistent chunks & manifest
As noted, updating both chunks and the manifest brings in a potential failure where chunks are successfully written, but the manifest update fails. This could cause an inconsistent state where the manifest contains the wrong data (e.g. wrong chunk length).
Possible solution: One method would be to provide a layer of indirection and store chunks under a different key than just their indexes (e.g. use a hash value, or append a version number) and then those chunks are only ever truly "written" if they've been updated in the manifest. There might also be a feature to "vacuum" orphaned chunks during failed writes. This is quite related to ideas that have been explored previously:

https://github.com/zarr-developers/zarr-specs/issues/154
https://github.com/zarr-developers/zarr-specs/issues/82
https://github.com/zarr-developers/zarr-specs/issues/76
https://github.com/zarr-developers/zarr-specs/issues/14

Write contention
Multiple processes could attempt to write into the chunk manifest at once.
Possible solution: The simplest solution here is an exclusive lock. More optimistic concurrency control may be possible, but I haven't thought that through.
Data format requirements
Some of the larger datasets out there have 10s of millions of chunks (example).  However, typically only a small fraction of those are populated, also in the order of 10s of millions.
The manifest must store at least three pieces of information. Due to regularities in the data and the right encoding and compression, this might compress quite well – but it's still a significant amount of data.
The data format should also support transactional operations (e.g. updating the key for a chunk). Alternatively, an append-only log might be considered (similar to event sourcing).

Special case of incremental arrays: Fetching from Zarr
I want to record a special case of populating arrays incrementally that is potentially relevant here. This was brought up by @dstansby in Zulip
In our general case, we're populating the arrays with data from non-Zarr sources, potentially produced through a complex processing pipeline. The MetaArrays themselves do not know how the data is populated, nor should they.
However, the user might need to simply fetch regions of existing zarr array incrementally. In that case, the MetaArrays could potentially provide a convenient API not only to discover unpopulated chunks, but to also aid in fetching them. This would provide functionality similar to what @dstansby described:
for chunk in chunks:
    if exists_on_hard_drive(chunk):
        pass
    download(chunk)

Perhaps you can clarify what you are trying to accomplish exactly, independent of any particular solution?
In https://github.com/zarr-developers/zarr-specs/issues/300#issuecomment-2204735276 you mention lazily-writing an array as individual chunks are requested.  This seems to be something that is easily supported by the existing zarr v2 and v3 formats without chunk manifests: you can simply query the underlying storage to determine if the key corresponding to the chunk is present.  If it is, you can read it.  Otherwise, you can compute and store it.
If you want to construct an in-memory "MetaArray" indicating which chunks are present, this can be done by performing a list operation on the underlying kvstore in order to obtain the list of all chunks.  List operations are very fast when using a chunk manifest since no additional I/O is required after loading the manifest, but I expect in most cases other underlying kvstores, like the local filesystem or even S3, would still provide adequate performance, as long as you can construct the in-memory MetaArray once and then do a bunch of queries.
If you want to construct a MetaArray corresponding to just a subregion of the full array, that can also be done by performing one or more "range-restricted" or "prefix-restricted" list operations on the underlying kvstore.  However, if the query does not happen to align well with the "chunk_key_encoding" of the array (and the current chunk key encoding options in zarr v3 are generally problematic for this type of use), it could require a large number of queries.  If you want to make this work better, you could propose some improved chunk_key_encodings for zarr v3.  Note that this type of querying is used in tensorstore for https://google.github.io/tensorstore/python/api/tensorstore.TensorStore.storage_statistics.html#
In https://github.com/zarr-developers/zarr-specs/issues/300#issuecomment-2204734027 you mention wanting write support when using a chunk manifest, but it is not clear what form of writing you mean.  Under the current chunk manifest proposal, zarr implementations would treat the manifest as read only, because the chunk manifest does not indicate where new keys are written.  For existing keys, in principle you could attempt to overwrite the same location in place, but that only works if the size of the chunk stays the same (effectively meaning no compression), but that is not necessarily the actually desired behavior.  You could also have some form of "writing by reference", where you "write" to the chunk manifest by linking in additional keys referring to existing data elsewhere.  But that might more naturally be handled entirely separate from the zarr implementation, since it would require specialized APIs.
It isn't really clear to me what benefits you hope to gain from using a chunk manifest.

Thanks @jbms, these are great questions and points. This is precisely why I wrote the issue in the first place 🙂

It isn't really clear to me what benefits you hope to gain from using a chunk manifest.

The primary motivation has been that this could entirely remove the need for code that updates the MetaArrays. Since the manifest and the MetaArrays store similar information, it would remove the need to keep two parallel implementations.
Constructing MetaArrays from object store list operations
You're asking a great question: why even store the chunk initialization information, when it's right there in the object store in the form of stored chunk objects?
This was actually our original method. We would initialize the xr.Dataset MetaArray on-the-fly from the output of list_prefix. However, at some point, with a growing number of populated chunks, the list_prefix method in Zarr became prohibitively slow. We saw two options at that point

Allow for range-restricted list operations and keep initializing MetaArrays on-the-fly.
Start storing the MetaArrays. This requires updating the data structure, but provides fast queries – querying millions of chunks requires only fetching millions of bool values.

We've explored these options for our storage (which is an on-prem object store, indexed by Arraylake) and, at that time, chose option #2 as the scalable solution.
However, this choice was in the context of that particular storage and may not apply in general. This issue is about opening up the functionality to the wider Zarr community and thus it's worth exploring #1 again.
It would be very interesting to see what the performance is in listing in the order of millions of chunks in e.g. S3 and, if necessary, whether an improved chunk_key_encodings could provide satisfactory performance.
It's also worth considering how well this works with sharding, virtual zarr and other potential (planned) features.

In #300 (comment) you mention lazily-writing an array as individual chunks are requested

If this is well-supported by Zarr already, then please ignore that comment. I only recorded it as a special case of incremental arrays that I saw in the Zulip discussion. It's not relevant to our use cases.


fix minor typo in troubleshooting github pages

this is a minor typo in troubleshooting section for github pages
someone might have run gh-pages with non-existing parameter 'again'

Thanks!


Logs directorys not being created if missing.

check_logs only checks if script log is missing and then installs it. If symlink or console log is missing it will so nothing

This was completed


Set bf16 AMP fallback to fp16 AMP

Issue #, if available:
Description of changes:
Hi 👋 We noticed a fallback to 32 bit floats on older T4 GPUs (g4dn family) for presets='multilingual'. This PR sets that fallback to fp16 AMP instead of falling back to 32.
This way, we still have fast training on the older machines. Or was there some reason to fall back to 32?
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. ✅

@zhiqiangdon taking the liberty to ping you from the blame :)


Raise error for disk size on MachineGroup instantiation

In this PR, I implement an error logging for when the  disk_size set by the user in the MachineGroup is smaller than 20 GB.

We discussed the disk size and it makes more sense that the user defines the disk space he wants to add to the default. The default disk size, should be defined in the backend. I will keep this open until the change happens on inductiva-web-api.

@cristianacarpinteiro : what is the problem we are trying to solve here? Could you please describe the situation that made you decide that this is a priority? I honestly don't see what we are trying to achieve here. Also: if you wanted to put a Maximum Limit, then I could understand that we are trying to protect the user (including us) of overspending, but I honestly don't see why we need this change. Can you pls clarify?

@cristianacarpinteiro : what is the problem we are trying to solve here? Could you please describe the situation that made you decide that this is a priority? I honestly don't see what we are trying to achieve here. Also: if you wanted to put a Maximum Limit, then I could understand that we are trying to protect the user (including us) of overspending, but I honestly don't see why we need this change. Can you pls clarify?

I was testing the functions implemented by @begaiym-k. When I put a disk_size_gb less than 20 GB there is an error on the bakend that is not explicit or informative, so I wanted to raise the error to prevent the user from putting a disk size smaller than that value. But then we discussed and it would be more intuitive to the user to put the disk size he wants to run the simulations (additional to the default 20GB, the 20GB are there to be able to fit all the docker images, that's why there is the lower limit, a smaller disk size wouldn't fit the images). The default size of the disk would be defined on the backend and the user would just define what he wanted more. So the user can define a disk size of 2GB and on the backend we would do a disk with 22 GB. You see?  It wasn't a question of priority, it was just an easy fix to make the error more explicit, but we decided to alter it on the backend and change the disk_size definition here. But I agree that a cap limit would also make sense.

@cristianacarpinteiro : what is the problem we are trying to solve here? Could you please describe the situation that made you decide that this is a priority? I honestly don't see what we are trying to achieve here. Also: if you wanted to put a Maximum Limit, then I could understand that we are trying to protect the user (including us) of overspending, but I honestly don't see why we need this change. Can you pls clarify?

I was testing the functions implemented by @begaiym-k. When I put a disk_size_gb less than 20 GB there is an error on the bakend that is not explicit or informative, so I wanted to raise the error to prevent the user from putting a disk size smaller than that value. But then we discussed and it would be more intuitive to the user to put the disk size he wants to run the simulations (additional to the default 20GB, the 20GB are there to be able to fit all the docker images, that's why there is the lower limit, a smaller disk size wouldn't fit the images). The default size of the disk would be defined on the backend and the user would just define what he wanted more. So the user can define a disk size of 2GB and on the backend we would do a disk with 22 GB. You see? It wasn't a question of priority, it was just an easy fix to make the error more explicit, but we decided to alter it on the backend and change the disk_size definition here. But I agree that a cap limit would also make sense.

OK! But then let's think carefully about this. Does a "delta" type of parameter make sense? Why don't we just allow the user to set the disk.  So, your initial solution of just informing what happened is the correct one, and then we should let the user correct it.
Having a parameter that is a plus change on a default base parameter is very confusing. We don;t have that pattern anywhere else. Users can SET values. We don't allow them to INCREMENT a value. That is not the pattern we are using.
Let's chat about this.

I did some more investigation on this and the exception that is raised on the client side is:

ApiException(status=500, reason='Internal Server Error', api_response=ApiResponseWithoutDeserialization(response=<urllib3.response.HTTPResponse object at 0x7feb4d15f310>, body=<inductiva.client.schemas.Unset object at 0x7feb48416b90>, headers=<inductiva.client.schemas.Unset object at 0x7feb48416b90>))

While  the GCP error that is happening on the backend is:

raise core_exceptions.from_http_response(response)
2023-08-09T10:55:12.955818845Z google.api_core.exceptions.BadRequest: 400 POST https://compute.googleapis.com/compute/v1/projects/inductiva-api/global/instanceTemplates: Invalid value for field 'resource.properties.disks[0].initializeParams.diskSizeGb': '10000000'. Must be less than or equal to 65536

This is much more informative and should be passed to the user. To this information be passed to the client side, we need to fix it on the backend.

I did some more investigation on this and the exception that is raised on the client side is:

ApiException(status=500, reason='Internal Server Error', api_response=ApiResponseWithoutDeserialization(response=<urllib3.response.HTTPResponse object at 0x7feb4d15f310>, body=<inductiva.client.schemas.Unset object at 0x7feb48416b90>, headers=<inductiva.client.schemas.Unset object at 0x7feb48416b90>))

While the GCP error that is happening on the backend is:

raise core_exceptions.from_http_response(response)
2023-08-09T10:55:12.955818845Z google.api_core.exceptions.BadRequest: 400 POST https://compute.googleapis.com/compute/v1/projects/inductiva-api/global/instanceTemplates: Invalid value for field 'resource.properties.disks[0].initializeParams.diskSizeGb': '10000000'. Must be less than or equal to 65536

This is much more informative and should be passed to the user. To this information be passed to the client side, we need to fix it on the backend.

OK! Let's try to pass this down to the client. It may be interesting to think if we want to do our own analysis of this sort of exceptions later.
Quick note: from the exception it is possible to see that users could potentially request 64TB of disk.
How do we make sure that NEVER happens? @cristianacarpinteiro : do you mind checking how much that would actually cost?
@begaiym-k: this is where we certainly want to have an upper limit configured on the server side. It could even be a const for now. What about 30Gb for now?

Since this is solved in the backend, I will close this PR for now.


”Bârsău de Sus”, ”Bârsău de Jos” are listed in the drop-down menu

Description:
When user writes ”Bârsău de Sus”, ”Bârsău de Jos” the locations appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Bârsău de Sus"

Expected results
The location appears in the drop-down menu as "Sat Bârsău de Sus, SATU MARE (Bârsău)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Bârsău de Jos"

Expected results
The location appears in the drop-down menu as "Sat Bârsău de Jos, SATU MARE (Bârsău)".


[WIP] Partial results display prototype, refactor warning / error handling

Query truncation prototype that includes potential datasource spec changes, fetch updates, and an additional actions property on TimeSeriesQuery results.
Here is the datasource I'm using to test these changes:
{
  "kind": "GlobalDatasource",
  "metadata": { "name": "PrometheusM3Proxy" },
  "spec": {
    "default": false,
    "plugin": {
      "kind": "PrometheusDatasource",
      "spec": {
        "headers": {
          "m3-limit-max-returned-datapoints": "50",
          "m3-limit-max-returned-series": "1"
        },
        "warnings_header": "m3-returned-data-limited",
        "proxy": {
          "kind": "HTTPProxy",
          "spec": {
            "url": "http://localhost:8000/data/metrics"
          }
        }
      }
    }
  }
}

Approach has changed to use onCreate as shown here, closing this and will open the real PR early next week


[WIP] 734 layout changes on resource specific tab

These are layout changes for the resource specific tab for each resource type. This changes make efficient use of the space provided and accommodates for the removal of the max height that restrained the tab. Also includes minor style changes for consistency.

Can one of the admins verify this patch?

@pkdash to list who can review which resource templates

@mohamedmorsy review changes to model instance template changes

@Castronova please review template/js code changes for model program

@gantian127 please review code changes for netcdf and raster resource types

@mauriel133 Could you please merge the new modifications in the develop branch? As I would like to see how these new modifications in this branch are going to affect the new implementation of the Model Instance and SWAT Model Instance resources.

@mauriel133 I have made a comment for issue

@mauriel133 What is the current status for this PR?

I'm applying some changes after feedback from other programmers.

@mauriel133 OK. I am planning to release 1.6.3 this week if possible, so please let me know when it is ready to be merged and whether anyone else needs to review the layout change before merging.

I have finished applying changes now. I think all resources look fine under resource specific tab of the landing page.
@mohamedmorsy, I merged the changes in the develop branch and applied the new style, so those can be tested now.

@mauriel133 @gantian127 @mohamedmorsy @zhiyuli I deployed this to park.hydroshare.org and created one raster resource and tested layout spacing for resource specific tab and it does not look correct to me. Please test them out for your own resource types and let @mauriel133 know if anything does not look correct. @mauriel133 after you apply further changes, let me know and I can apply new changes to park for continuous testing by others until they give thumbs up. Then I can merge this to develop branch.

@mauriel133 @hyi netCDF resource layout is working as expected. Thanks.

@gantian127 How about raster resource layout?

On raster resources, the "Coordinate Refetence System" label is repeated.

@mauriel133 @hyi
Do you mean these two labels are repeated? I think the second means the "coordinate reference system unit" for which it may put the "unit" on the same line

Moreover,  I found out that sometimes under editing mode, the fields which need to be editable for the spatial reference and cell information are not able to be edited.

@mauriel133 I made a public raster resource with multiple bands in the website and you can test with that file. The fields for the spatial reference and cell information are not editable which is not working correctly.
Another suggestion is for Band information. It may be better to list them starting with "band_1". If this suggestion is complicated to implement, you can just leave it as it is. Thanks

Noticed. I will look into these issues.

@gantian127, I cannot reproduce the layout error with multiple bands. Can you make sure that it wasn't due to outdated cached files?
If it keeps happening tell me which browser and version you are using.

@mauriel133 I think the layout under view mode is working correctly now for band information. Maybe it just happened on my computer yesterday. Thanks

@mauriel133 @gantian127 @zhiyuli @mohamedmorsy I have redeployed the latest change of this branch to park.hydroshare.org, so you can take a look as needed. @mauriel133 You have to rebase this branch with latest develop to resolve conflicts since it cannot be merged automatically now. Let me know when it is ready to merge.

Done.

@mauriel133 I found out that the "variable" forms of netcdf resource under editing mode is missing. Please check this see if you find the same error. Thanks.

@mauriel133 for raster resource. Under editing mode, the band information "save changes" button doesn't take action and the edited information can't be saved.
Under view mode, is it possible to make the " coordinate reference system unit" on the same line, so that user won't get confused to think that the terms are repeated. Thanks.

@mauriel133  https://github.com/mauriel133 I found that in the Model
Instance specific metadata under "Executedby" term, the version of the
Model Program used to execute this instance is available in the edit mode
while it is not available in the resource landing page:
[image: Inline image 1]
[image: Inline image 2]
On Thu, Dec 10, 2015 at 3:01 PM, gantian127<EMAIL_ADDRESS>wrote:

@mauriel133 https://github.com/mauriel133 I found out that the
"variable" forms of netcdf resource under editing mode is missing. Please
check this see if you find the same error. Thanks.
—
Reply to this email directly or view it on GitHub
https://github.com/hydroshare/hydroshare/pull/740#issuecomment-163734515
.

--
Mohamed M. Morsy
Graduate Research Assistant
Department of Civil & Environmental Engineering
Thornton Hall D107, University of Virginia
351 McCormick Road, P.O. Box 400742
Charlottesville, VA 22904-4742
Phone:<PHONE_NUMBER>
E-mail<EMAIL_ADDRESS>

@mauriel133 For the SWAT Model Instance, it has the same old style of the specific metadata landing page (the style did not updated to be look like the one for the Model Instance resource). Also it has the same problem I stated before for the Model Program Version used for execution ("Not Provided"):

In the editing mode for the specific metadata landing page, there is a wide white gap between "Model Program used for execution" and "Model Method" terms:

There is a big wide white gap between "Model Method" and "Model Parameter" terms:

I cannot set a specific width for " coordinate reference system unit" because it would conflict with the responsive layout for mobile devices.
@mohamedmorsy, I'll look into those issues. Thanks.

@mauriel133 that will fine if the width can't be set. I will test the update after it is deployed. Thanks for working on these issues.

@mauriel133 @gantian127 @zhiyuli @mohamedmorsy The latest code in the branch has just been deployed on park for your further testing. Please raise issues or give +1 for your resource types so that we know when it is ready to merge this.

+1 for netcdf and raster resource

@mauriel133  @hyi  +1 for Geog. Feature Res. Type

@mohamedmorsy @mauriel133 could you check your model-related resource types on park and see whether things look good now? Will merge this in for AGU if you give a +1.

@hyi I have two issues related to Model Instance and SWAT Model Instance resources and I am waiting for @mauriel133 to take a look on that.

@mohamedmorsy OK. @mauriel133 will not have time until he finishes getting my resources page ready for AGU. @mauriel133 you don't have to get this in for pre-deployment on beta today, and since this branch only involves front-end style changes, this can be easily pulled into AGU release on Monday before final www deployment.

@mohamedmorsy
I have applied the style to SWAT Model Instance resources. Please look at it and let me know if you find any issues.
The issue with the Version field seems to be caused by something else. I'm using the exact same conditional checks to render it, and the issue is also present in the current develop branch as deployed on beta prior to this work.
This is how it is currently implemented:

{% if executed_by.model_program_fk.metadata.program.software_version.strip %}
{{ executed_by.model_program_fk.metadata.program.software_version }}
{% else %}
Not Provided
{% endif %}
Perhaps adding ".strip" at the end of the line would make it show up. Could you try that and see if it works, otherwise another issue should be opened for this so that this PR can move on with the rest of the work.

@mauriel133 @hyi  I pulled the branch on my local computer and tested it. I found the bug related to the version field. We got this bug because we merged the new version of the model instances before the new version of the model program. So in the new version of the model program there is no field called software_version as it becomes "modelVersion". I did the fix and pushed it to this branch as it is a minor fix. Also I changed the name of the "Resource URL" field under the "ExecutedBy" to "Resource URI"
@mauriel133 I tested everything for the three model resorces, everything looks great and nice except for the SWAT Model Instance landing page. As you see in the following screenshot, the "Resource URI" field under "ExecutedBy" is interfering with the "ModelInput" fields. Also it would be great if we can put "Resource URI" in the same line. Also it would be great if we can put the name of the fields of the "ModelInput" metadata term on the same line.

@mohamedmorsy, I will move the Model Input section to the bottom and have everything in one column in order to avoid shortage of space in the field names.
Thanks for the feedback and for taking care of the bug.

You can test the changes now.

@mauriel133 Now, everything looks great! Thanks for your hard work on that!
@hyi +1 for Model resources


ImportCsvSheet and ImportXlsSheet are very different implmentations from the public API.

The public API docs say that these resources should go to sheets\import instead the C# sdk actually requires a folderId.
This seems to be an enhancement to make sure you import a CSV/XSLX into the right place.
Need to find out who need to update the API docs or just go fix the C# example ourselves.
https://smartsheet.redoc.ly/tag/import#operation/import-sheet-into-sheets-folder

@timothodge I don't really understand what this is talking about
There are three versions of import csv (confusing)
https://api.smartsheet.com/2.0/folders/{folderId}/sheets/import
ImportCsvSheet(string file, string? sheetName = null, int? headerRowIndex = null, int? primaryColumnIndex = null);
https://api.smartsheet.com/2.0/sheets/import <- no folderid
SheetResources.ImportCsvSheet(string file, string? sheetName, int? headerRowIndex, int? primaryColumnIndex)
https://api.smartsheet.com/2.0/workspaces/{workspaceId}/sheets/import
WorkspaceSheetResources.ImportCsvSheet(long workspaceId, string file, string? sheetName = null, int? headerRowIndex = null, int? primaryColumnIndex = null)


Del @Deprecated AOT

Issue description
Remove deprecated code for Micronaut Framework 5.
Document the breaking changes.

This module has no deprecated code to remove


Reorganize currying

Fixes #17

I tried running some tests (in the benchmark folder).
Believe it or not, this doesn't make any difference performance wise.
It still an improvement of readability and style though

Interesting. So for test on very large iterations (1mil items) I saw virtually not difference, I think because the by that time the function gets hot even if it was unoptimized at the beginning of the run. The key is that it's a different function each time you call map.
I designed a test case to maximize the effects of my changes by calling map repeatedly to do smaller iterations each time.
function mapVsBuiltinMap() {
  var a = Array.from(range(10))
  measureSpeed(function () {
    for (let i = 0; i < 1000; i++) {
      a = map(x => x + 1, a);
    }
    a = Array.from(a);
  }, { samples: 100, discard: 10 }, function (err, ms) {
    if (err) return console.log('Error!')
    console.log('************ map array ************')
    console.log(ms.toFixed(2))
  })

  var arr = Array.from(range(10))

  measureSpeed(function () {
    for (let i = 0; i < 1000; i++) {
      arr = arr.map(x => x + 1);
    }
  }, { samples: 100, discard: 10 }, function (err, ms) {
    if (err) return console.log('Error!')
    console.log('************ array.map ************')
    console.log(ms.toFixed(2))
  })
}

Results:
Without new changes:
************ map array ************
2.34
************ array.map ************
0.05
With new changes:
************ map array ************
0.72
************ array.map ************
0.05
My take away is that for pure speed you just can't compete with Array.map.


Add accessibility report docs and fix storybook

Summary | Résumé

Similarly to how EN (covid alert) has their "Accessibility Report" in their repo, I've added our version by slightly modifying their text
Also, storybook was broken (components like Label uses translations) as it couldn't resolve i18n but found a fix to resolve via alias

Note: I'll update Storybook via gh-pages after merging this PR.

@fitore-cds anything needs to be translated from your modifications from EN?

@mcman12 I copied over their FR translation and pinged you so that you're aware of what I'm doing hehe. I'll push it live and we can update the text later, if needed.


License?

What license is this under? If we wanted to include this in intermezzOS, it'd need to be open source, and by default, this is "all rights reserved".
I'm assuming since it's a tiny module you just forgot about this :smile:

Yeah, I just forgot it :). I think I'll choose the standard MIT/apache combination.
Great to see you working on intermezzOS again!


fix breakage in FM tests

The unit test is using a different path of plist file than the production code causing the test failed as it is checking the path of production code instead of unit test path. This fixes #7797

Thanks! Looks like there's still a failure in the spm tests.
Also, in case you missed it, there's a 2nd problem described in #7797 about header specifications causing the prerelease nightly test to fail.

@paulb777 weirdly messaging is one of the test in travis that is green. It doesn't look like it's relavant.


try-runtime-cli docs and tests

At least one try-runtime-cli command passes parameters in the incorrect order (https://substrate.stackexchange.com/q/7902/3991).
Need to audit all the examples in the docs and fix any issues, and write some CLI tests to make sure they don't break in the future.
We should also write some CLI tests. See subtrate CLI and I believe this repo is using similar testing patterns https://github.com/paritytech/staking-miner-v2/tree/main/tests for an examples on how to write the tests.
Docs

[ ] Audit doc examples

Tests

[x] follow-chain https://github.com/paritytech/substrate/pull/13794
[ ] on-runtime-upgrade
[ ] execute-block
[ ] fast-forward
[ ] create-snapshot

There are more example node tests here, just as an idea of how to do it in Substrate https://github.com/paritytech/substrate/blob/8a9f48bcf0c9f92949082535d77c12166522bb2f/bin/node/cli/tests/

@liamaharon Just to let you know I am looking to rewrite the example part of the try-runtime-cli docs, specifically for the try-state feature. I believe that some of the examples and explanations could be better.

@liamaharon If you haven't already, I would like to create a PR that covers execute-block and `create-snapshot with tests.

I'm not currently working on these, go ahead @Szegoo

Moved to https://github.com/paritytech/try-runtime-cli/issues/10


docs: temporal update to indicate persistent storage

Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
This is a temporal update to show how to enable persistent storage in HF Spaces, while the option is not available in the configuration page.
Closes #4299
Type of change
(Remember to title the PR according to the type of change)

[ x] Documentation update

How Has This Been Tested
(Please describe the tests that you ran to verify your changes.)

[ x] sphinx-autobuild (read Developer Documentation for more details)

Checklist

[ ] I added relevant documentation
[ ] I followed the style guidelines of this project
[ ] I did a self-review of my code
[ ] I made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I filled out the contributor form (see text above)
[ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

@davidberenstein1957 It should be fixed now


No Intelligence in Live Share client.

Steps to Reproduce
1.Open python application project
2.Connect host and client with Live share
3.Typing code in .py file in client
import decimal
decimal.DT
Expected behavior
IntelliSense should work well in client and host.
VS Environment

fixed using this build:
Build of Visual Studio: 16.6.0 Preview 3.0 [29923.199.master]
Build of Python Package: 16.5.20042.1

thanks for verifying


wso2iot 3.1.0 update2 uses cpu Memory to 100%

Wso2-3.1.0 update 2 is getting cpu usage to 100%.This happens when only running the wso2 service.
java JAVA_OPTS="-Xms1024m -Xmx2048m".

Can you please share the machine details, os, version, ram, memory, cpu. ?

Are you using WSO2 IoT Server with iOS device type, installed?
We have identified a possible cause for this related to notnoop library.
This is fixed with the latest update releases. Please let us know if you encounter this issue once again in the future.


Prevent crash when aborting connection before Frame is created (#738).

Check that _frame is not null
Queue work item via LoggingThreadPool to isolate future potential exceptions and prevent new crashes

#738
cc @halter73 @benaadams @yuwaMSFT

:shipit:

:shipit:


add some scripts for replacing and restoring max tokens

implement replacing and restoring max tokens for Windows

感谢！


Oracle cdc doc

Thanks @molsionmo for the contribution, I left a little comment

Note: flink-sql-connector-oracle-cdc-XXX-SNAPSHOT version is the code corresponding to the development branch. Users need to download the source code and compile the corresponding jar. Users should use the released version, such as flink-sql-connector-oracle-cdc-2.2.1.jar, the released version will be available in the Maven central warehouse.
We have released version 2.3. Can we update it here ?

The oracle connector implements incremental snapshots. In the snapshot reading phase, do we rely on checkpoint? If so, can we add sample code?

At present, the oracle connector can only read the source through a single thread ?

Thank you for reminding of the problem. The connector have been support incremental snapshot in multiple concurrency . please review again , thanks.

cc @leonardBang


exit status 0xdeaddead

What version of Go are you using (go version)?

$ go version
go version go1.18.4 windows/amd64

Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Bala\AppData\Local\go-build
set GOENV=C:\Users\Bala\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\Bala\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Bala\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=D:\installed programs\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\installed programs\go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18.4
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\Bala\AppData\Local\Temp\go-build3334445441=/tmp/go-build -gno-record-gcc-switches

What did you do?

What did you expect to see?
What did you see instead?

I got the error running the simple hello world program in terminal. The error was "exit status 0xdeaddead". Tell me if any one know this issue.

Could you provide the source code of your program, the exact command(s) you run, and the full error message? Also, what version  of Windows are you using? Thanks.


ENH: integrate: add array API-support

Towards gh-18867
This issue tracks progress toward the addition of array-API support to scipy.integrate functions. The functions listed below look ready for conversion.

[ ] trapezoid
[ ] simpson
[ ] romb
[ ] cumulative_trapezoid
[ ] cumulative_simpson

If I were to review, I'd prefer to see one PR per function in the order listed above with only one PR open at a time, learning from each as we go. (If others are willing to review without these conditions, go for it!)
I'd like to work on:

[ ] qmc_quad

These would be pretty substantial undertakings; there would need a pretty dedicated team of author(s)/reviewer(s). I'd suggest holding off on them until some of the lower-hanging fruit is picked. @nmayorov would you be interested in being part of this?

[ ] solve_ivp
[ ] solve_bvp

Of course, adaptive quadrature given a callable rather than points is also a very high priority. We'll see where gh-20252 is at the end of the summer.
I think we could convert these functions, but I'm not sure whether doing so is a high priority. LMK your thoughts.

[ ] newton_cotes
[ ] fixed_quad (IIUC, automatic differentiation would be easier with something non-adaptive like this. I'm assuming that gh-20252 will keep this in mind.)

I don't think these will be converted, per se.

quad_vec - either the outcome of gh-20252 or _tanhsinh would be a suitable array-API compatible replacement
quad - Fortran, and same as above
odeint(and related) - Fortran
dblquad, tplquad, nquad - only relevant if quad were converted

Just to throw an idea out there, solve_ivp, whilst very solid and battle tested, is now a little behind other offerings (DifferentialEquations.jl) so if it is going to be a substantial effort to rewrite it would be interesting to investigate possible alternatives. For example, perhaps considering the feasibility of a slimed down array API compatible version of https://docs.kidger.site/diffrax/

perhaps considering the feasibility of a slimed down array API compatible version of docs.kidger.site/diffrax

x-ref https://github.com/scipy/scipy/pull/20085#issuecomment-2029349921


Warning for ignored namespace in cross-provider context is too noisy

Do you want to request a feature or report a bug?

Bug

What did you do?
Define an IngressRoute with KubernetesCRD provider as follows:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: ingress
spec:
  entryPoints:
  - traefik
  routes:
  - kind: Rule
    match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
    services:
    - kind: TraefikService
      name: api@internal

What did you expect to see?
Relatively quiet logs with level for standard operation with --log.level=WARN
What did you see instead?
A constant stream of warnings about the namespace "ingress" being ignored.
Output of traefik version: (What version of Traefik are you using?)

Version:      2.1.1
Codename:     cantal
Go version:   go1.13.5
Built:        2019-12-12T19:01:37Z
OS/Arch:      linux/amd64
pod "traefik" deleted

What is your environment & configuration (arguments, toml, provider, platform, ...)?
    - --entryPoints.traefik.address=:9000
    - --entryPoints.web.address=:8000
    - --entryPoints.websecure.address=:8443
    - --api.dashboard=true
    - --ping=true
    - --providers.kubernetescrd
    - --log.level=WARN
    - --accesslog=true
    - --global.checknewversion=true
    - --global.sendanonymoususage=true
    - --providers.kubernetesingress

There is a section in the code somewhere that suppresses the namespace if it is default.   A current workaround is to explicitly specify a namespace of default for cross-provider references.
IMO this is incorrect behavior. It should suppress the messages if a namespace is not provided, or possibly if the namespace matches that of where the resource is defined.
Another thought to consider is reducing the level, warning is pretty high for something benign

Closed by #6087.


Grafana Dashboard URL ?

Hi,
I've just installed you addon that is really cool !
I had to fix 2 little things like adding pluggins to remove some errors and to get dashboard fully fonctional :
plugins:
  - natel-discrete-panel
  - pr0ps-trackmap-panel
  - grafana-piechart-panel
  - natel-plotly-panel
custom_plugins:
  - name: panodata-map-panel
    url: >-
      https://github.com/panodata/panodata-map-panel/releases/download/0.16.0/panodata-map-panel-0.16.0.zip
    unsigned: true

But one thing I haven't found :
Which URL should I set in Grafana Dashboard field in Teslamate settings to get dashboard link directly from Teslamate ?

Many thanks for your help
Regards,

I got the TeslaMate folder in grafana import dashboards but there is no data

Hi,
The URL is generated dynamically by Home Assistant supervisor. If the addon restarts the url will change so I don't recommend using this.

I didn't change anything, there is no URL after installation.
What's the good practice for using this addon ?
Do I have to go to Grafana panel et search for dashboard ? Can't I use links from Teslamate ?
Is there any dashboard to had or something ?
Many thanks,

Hi, I was also unable to find any usefull information about the grafana Dashboards URL.
Did you ever get this fixed?

Hi,
No, I don't have any news about this issue for the moment.
I took URL from one of my dashboards in Grafana and cut it just before "/d/..." to put it in Teslamate.
It seems to do the job but not everywhere, it's a good work around while waiting for official answer.

Does not work for me, but thanks anyway. :)

On your computer, go to Home Assistant, click on Grafana section, Toggle Grafana menu ("burger icon") and click on "Dashboards"
Here you'll see your Teslamate dashboards, right click on any one (for exemple "Drives") and copy link.
You'll have something like this :
https://yourhomeassistanturl:8123/api/hassio_ingress/xxxxxxxxxx/d/xxxxxxxxxx/drives
You'll just have to cut it before "/d/" like this :
https://yourhomeassistanturl:8123/api/hassio_ingress/xxxxxxxxxx
Then you paste it in Dashboard URL section of Teslamate
You'll see a Dashboards section appear in Teslamate and links will work but only in computer, not in mobile app

Hi Thanks for that. I think I understood what you meant in the first place, but it does not work for me somehow.

For completeness the URL from the grafana dashboard was:
https://xxxxxxxxxxxxxx.org:65199/api/hassio_ingress/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxTg/d/WopVO_mgz

And in the teslamate URL I entered:
https://xxxxxxxxxxxxxx.org:65199/api/hassio_ingress/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxTg

My bad, the training / was missing!

If I add "/" it's automatically removed for me, that's strange
But if it works for you, that's great :)

hello
I have the same problem, did you solve this?


How to Adjusting root size while creating container

using below commands we are able create containers and after creating container we are able override root storage but while launching new container how to give/override root size for the container
sudo lxc launch ubuntu:20.04 test
sudo lxc config device override test root size=20GB
->sudo lxc exec test bash
root@test:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
disk/containers/test    20G 1011M   19G   6% /
none                             492K  4.0K  488K   1% /dev
udev                              63G     0   63G   0% /dev/tty
tmpfs                            100K     0  100K   0% /dev/lxd
tmpfs                            100K     0  100K   0% /dev/.lxd-mounts
tmpfs                             63G     0   63G   0% /dev/shm
tmpfs                             13G  184K   13G   1% /run
tmpfs                            5.0M     0  5.0M   0% /run/lock
tmpfs                             63G     0   63G   0% /sys/fs/cgroup

how to give/override root size for the container

What do you mean by root size? If you are trying to change the output of df -h to not show 63G for tmpfs, it won't work for containers because they share the system's resources.
Perhaps VMs are better suited for what you are trying to achieve.

Actually we are using zfs pool to one of hard disk (3.0 TB)..
We are creating container using launch command.
in every container it is showing 2.9T.
root@Test-Storage:~# df -h
Filesystem                          Size  Used Avail Use% Mounted on
disk/containers/Test-Storage  2.9T   29G  2.9T   1% /
none                                492K  4.0K  488K   1% /dev
udev                                 79G     0   79G   0% /dev/tty
tmpfs                               100K     0  100K   0% /dev/lxd
tmpfs                               100K     0  100K   0% /dev/.lxd-mounts
tmpfs                                79G     0   79G   0% /dev/shm
tmpfs                                16G  196K   16G   1% /run
tmpfs                               5.0M     0  5.0M   0% /run/lock
tmpfs                                79G     0   79G   0% /sys/fs/cgroup
We want to assign storage size for each container differently.. For that used override like sudo lxc config device override test root size=20GB.At that time it showing like below
df -h
disk/containers/test 20G 1011M 19G 6% /
we want to assign storage size of container when we are launching new container.is their any command

we want to assign storage size of container when we are launching new container.is their any command

You cannot assign the storage size for each container individually using lxc launch. You have to do the following for each containers:
lxc init ubuntu:20.04 test
lxc config device override test root size=20GB # or: lxc config device set test root size=20GB
lxc start test

Thank you..

@Kumar6295 for non-bug issues please can you post over at https://discuss.linuxcontainers.org/ thanks :)


Linux symlinked data dir broken after updating to 2.45.1

Describe the bug
On Linux, I have my Ledger Live data dir symlinked to a different location. It worked properly until updating from 2.43.1 to 2.45.1. As of 2.45.1, as soon as I attempt to login to Ledger Live, it nukes the symlink, replacing it with a file, which causes Ledger Live to fail to launch.  Detailed repro steps below.
To Reproduce
Steps to reproduce the behavior:

On Linux, ln -s "/some/other/location/" "~/.config/Ledger Live"
Download Ledger Live 2.43.1 AppImage (or any earlier version).
Open LedgerLive & setup/use as usual. It works as expected, with the data residing in /some/other/location.
Now download & launch Ledger Live 2.45.1 AppImage.
When you enter your password, it says "The password entered is incorrect" (it isn't), and immediately nukes the symlink: ~/config/Ledger Live is replaced with a textfile containing some json

(at this point you can, of course, delete the new ~/config/Ledger Live textfile, recreate the symlink, and resume using previous versions of Ledger Live. But 2.45.1 will not function.)
Expected behavior
Launching Ledger Live 2.45.1 AppImage should work properly with its data symlinked to another location, as did all prior versions
Screenshots
n/a
Desktop (please complete the following information):

OS: KDE Neon 5.24

Also experiencing this on Ubuntu 20.04.
Ledger devs, is there an alternative way to point Ledger to a different data dir until this regression is fixed?

Still broken as of 2.46.
And 2.46.2.
Not really sure why Ledger can't even be bothered to respond, let alone do something to address this regression...

Still broken in 2.47.0

@gre @valpinkman are you aware of this? thank you vm

Hi and sorry for the delay in answering on this Github issue.
We now only use Github issues for developers topics (typically on topics that concerns our SDK, libraries,..). For questions, feature requests or Ledger Live issues, please go through our Customer Support to raise this voice and have it more prioritised: https://support.ledger.com/
Thanks

Doesn't seem like this should be closed considering it has not been properly fixed.
Also, if there's no publicly-visible venue to report issues, how can others possibly chime in? Support tickets are only one-on-one and are completely opaque - users have no visibility into how many other users are suffering from an issue, they provide no way to discuss issues amongst each other, etc.
Please reopen until this bug is actually fixed.

2.48.0 is still broken.
I repeat: where is the new place for providing public visibility on issues? Simply closing & hiding issues from users, thus preventing any sort of transparency into how prevalent an issue is & preventing any possibility of public discussion, is clearly not a feasible approach to open-source software development.

This is extremely discouraging behavior from a company that purportedly provides software for products sovereignty/transparency. The whole mantra of crypto is "don't trust, verify." You now seem to be shutting down any method for public accountability of software behavior by hiding issues behind an opaque, invisible ticket system. I see no rationale for this when github provides a perfectly capable, industry-standard, transparent & open way to discuss & collaborate on issues.  Worse, when I bring this up, you simply ignore.
Is this the direction Ledger is going? Is this the end of Ledger as the most open & secure crypto wallet provider?

@gre

@metal450 "don't trust, verify" only applies to the Nano because it has a secure display. Ledger Live runs on unsecure platforms i.e. laptops and mobiles and therefore this principle doesn't apply. What You See Is What You Sign is only valid for the Nano, not Ledger Live.
Our hardware wallet is indeed the most secure but Ledger Live is just a companion app, it will never be more than that, don't expect too much.

@adrienlacombe-ledger Regarding your explanation of the meaning of "don't trust, verify," yes, I understand that Ledger's security refers to keeping private keys isolated. Per my previous statements, I was very obviously referring to the transparency/openness of development, not security of assets. "Don't trust, verify" is applied to many different concepts in crypto, as I'm certain you're aware.
In this case, moving from Github's public/transparent issue system (which can be followed, discussed, & upvoted by anyone) to a system where nobody can see what issues are opened, their status, how prevalent they are, etc...is similar to having an open/decentralized/trustless voting regime, & moving it to a closed/centralized/opaque one. Previously all users could see all activity on a ticket. Now we cannot. We just have to trust whatever Ledger's support e-mail tells us, one-on-one.
Anyway, it seems you forgot to respond to:

"if there's no publicly-visible venue to report issues, how can others possibly chime in? Support tickets are only one-on-one and are completely opaque - users have no visibility into how many other users are suffering from an issue, they provide no way to discuss issues amongst each other, etc."

"where is the new place for providing public visibility on issues? Simply closing & hiding issues from users, thus preventing any sort of transparency into how prevalent an issue is & preventing any possibility of public discussion, is clearly not a feasible approach to open-source software development."

@gre @adrienlacombe-ledger

Hello?
2.49 this is STILL broken.

@gre @adrienlacombe-ledger hello?

Still broken in 2.49.2

Hello and sorry for the delay,
I've been digging into the problem and it appears that this relates to Electron, it is very unclear why it would have worked before but:

i've been able to reproduce the problem with production build 2.49.2 on Ubuntu 22.04. That builds is based on Electron 15.
Electron 15 being outdated, I'm currently working on upgrading to Electron 21: https://github.com/LedgerHQ/ledger-live/pull/1463 . I can no longer reproduce the problem with a custom build made with Electron 21: I can alias a folder and the symlink is preserved. So there is a high chance this is going to be fixed by this upgrade, but I would need to make sure we re-evaluate this when we do our quality assurance. I will internally attach this ticket to it so we don't forget to verify it.

I am not exactly sure when this work will land in a release but hopefully in a few weeks 🤞

Not stale. Just still not solved.

Here is a build that should fix it, in case you want to help confirming this too :
https://download.live.ledger.com/ledger-live-desktop-2.52.0-electron22.0-linux-x86_64.AppImage
this is a custom build that I build from https://github.com/LedgerHQ/ledger-live/pull/2049 ( not an upcoming release ) and we are still in non-regression tests & validation phase of this PR.

Confirmed!!
This build restores proper functionality, per the last fully-working version, 2.39.2.  Both the symlink breakage is fixed, and also the inaccurate balance breakage is fixed.
Can't wait for this to be officially released. It's been a long 8-month regression.

I just landed on develop. It is planned for next LLD release (2.52)


Update MNTP PVC to handle Umbraco special properties

Take the logic from the Umbraco core MNTP PVC around special properties and add it into the SVC version. This allows the expected behaviour to continue to function addressing #21

Hey @NikRimington,
Thanks for taking the time to contribute to SuperValueConverters! It certainly makes sense that a change like this would be in the package.
Having looked at the code, I'm wondering if a more generic solution could apply? Perhaps the SuperValueConvertersBase class could expose a virtual property to set these values, and handle the logic in there. This would avoid the overridden converters needing to know explicitly what the Core converter returns (e.g. line 50 in your PR) and would instead just defer to calling the Core converter.
Let me know if you think that would solve your issue, and I can get it applied + released today.
Cheers,
Callum

Hey @NikRimington,
I have released v2.2 (for V8) and v3.1 (for V9) just now with this feature included. You will see the SuperValueConverterBase class includes a new overridable IgnoreProperties property.
In a future breaking release I am going to move this property onto the IPickerSettings object.
Hopefully this solution works for you!
Cheers,
Callum


Additional cleaning

Codecov Report
Attention: Patch coverage is 0% with 20 lines in your changes are missing coverage. Please review.

Project coverage is 5.18%. Comparing base (0d61661) to head (b28e259).

Files
Patch %
Lines

...StatsCore/Distributions/MannWhitneyDistribution.cs
0.00%
9 Missing :warning:

...cs/StatsCore/Distributions/WilcoxonDistribution.cs
0.00%
9 Missing :warning:

...ailfish/Execution/ClassExecutionSummaryCompiler.cs
0.00%
2 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #166       +/-   ##
==========================================
- Coverage   66.05%   5.18%   -60.88%
==========================================
  Files         309     232       -77
  Lines        8192    6755     -1437
  Branches      955     838      -117
==========================================
- Hits         5411     350     -5061
- Misses       2478    6391     +3913
+ Partials      303      14      -289

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Failed to analyze multiport memory with RW:1 R:1

I am having problems compiling a multiport SRAM with 1 read-write port and 1 read-only port. I am using the following config:
word_size = 2
num_words = 16

num_rw_ports = 1
num_r_ports = 1
num_w_ports = 0

tech_name = "freepdk45"
nominal_corners_only = False
process_corners = ["TT"]
supply_voltages = [1.0]
temperatures = [25]

route_supplies = False
check_lvsdrc = False

analytical_delay = False

purge_temp = False

output_path = "."
output_name = "sram_{0}_{1}_{2}".format(word_size, num_words, tech_name)

The problem is that the function parse_spice_list() is failing to parse the delay_sen1 measurement from the tmp/timing.lis file, i.e the output file from ngspice, when analysing the delays for read port 1. This is the error message in timing.lis:
Error: measure  delay_sen1  (TARG) : out of interval
.meas tran delay_sen1 trig v(clk1) val=0.5 fall=1 td=27.5n targ v(xsram_2_16_freepdk45.s_en0) val=0.5 rise=1 td=27.5n failed!

Unfortunately, the error was not caught earlier and parse_spice_list() continuously returns the string "Failed" rather than a floating-point number. This then causes the try_period() function to repeatedly fail, which in turn, forces find_min_period_one_port() into an infinite loop until a timeout occurs here.
Do you know how I can get around the problem with ngspice? Thanks!

Hi Andres,
That error means that the delay_sen1 signal is not transitioning properly during the simulation time.
Depending on where you are in characterization, this is called twice.
In our characterizer, it will double the period until it finds a feasible time or it will time-out after so many times (9). This finds the "feasible" period.
After this, it tries to find the min period with a bisection search using the above feasible period and another infeasible period. This should converge naturally, but it has a limit of 25 times "just in case". In practice, that shouldn't happen.
Does your unit test 21_ngspice_delay_test.py pass? This is essentially the same as that config but it has word_size=1 and a single port.
Matt

Hi Matt,
Thanks for the quick reply. I ran the unit test you suggested using python3 21_ngspice_delay_test.py -n -t freepdk45 -v and got a pass. However, this tests memories with one port and I can generate those without any problems. I can also generate memories with 1 read-only and 1 write-only port, but I get the error I described above when generating memories with 1 read-write port and 1 read-only port.

After this, it tries to find the min period with a bisection search using the above feasible period and another infeasible period. This should converge naturally, but it has a limit of 25 times "just in case". In practice, that shouldn't happen.

I think this is the bit that is failing for me. I do not know how it works in detail, but I think the read ports are characterized one-by-one. It seems that the tool can characterize port 0 without problems, but ngspice fails when trying to run for port 1.
I noticed that ngspice does not return a failure code when run in batch mode despite finding errors. Therefore, OpenRAM thinks it worked and tries to use some regular expressions to parse the output. This parsing always fails and the convergence never happens, so OpenRAM times out after 25 times.
Please let me know what else I can do to help. I tried looking more closely at the code, but I am no expert...
Thanks,
Andres

@hznichol can you take a look at this?

@hznichol fixed this in 119bd946

Thanks for the help! I just ran the test and it appears to work!


Feature request: RFC 3576 / DM / CoA support

Can you help to add the "Dynamic Authorization Extensions" support in the captive portal?
It will allow Radius server send DM/CoA message to "OPNsense" and change a user session.
Disconnect Messages (DM)
Change-of-Authorization Messages (CoA)

We have limited time available, but if you send in code we will certainly try to look at it and give feedback. Please use the new captive portal system from development when extending authentication support (it uses the new pluggable auth system).

timeout.


register in npm?

can you register this in npm so I can do npm install angular-typeahead? we are trying to move everything from bower to npm and so far this is the only dependancy not on npm

I am also moving towards webpack and babel, so it would be awesome if we could npm install.

Done.