Set python-snappy as optional dependency to work with Python 3.11 on pip install

Is your feature request related to a problem? Please describe.
Trying to install spatialpandas in a Python 3.11 environment currently fails due to a hard dependency on python-snappy which doesn't have wheels for Python 3.11 (see https://github.com/andrix/python-snappy/issues/124).
mamba create --name temp python=3.11
mamba activate temp
python -m pip install spatialpandas==0.4.7

produces this traceback
Collecting spatialpandas==0.4.7
  Using cached spatialpandas-0.4.7-py2.py3-none-any.whl (120 kB)
Collecting dask (from spatialpandas==0.4.7)
  Using cached dask-2023.5.0-py3-none-any.whl (1.2 MB)
Collecting fsspec (from spatialpandas==0.4.7)
  Using cached fsspec-2023.5.0-py3-none-any.whl (160 kB)
Collecting numba (from spatialpandas==0.4.7)
  Downloading numba-0.57.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 22.8 MB/s eta 0:00:00
Collecting pandas (from spatialpandas==0.4.7)
  Downloading pandas-2.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 19.9 MB/s eta 0:00:00
Collecting param (from spatialpandas==0.4.7)
  Using cached param-1.13.0-py2.py3-none-any.whl (87 kB)
Collecting pyarrow>=1.0 (from spatialpandas==0.4.7)
  Downloading pyarrow-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.9/38.9 MB 3.2 MB/s eta 0:00:00
Collecting python-snappy (from spatialpandas==0.4.7)
  Downloading python-snappy-0.6.1.tar.gz (24 kB)
  Preparing metadata (setup.py) ... done
Collecting retrying (from spatialpandas==0.4.7)
  Using cached retrying-1.3.4-py3-none-any.whl (11 kB)
Collecting numpy>=1.16.6 (from pyarrow>=1.0->spatialpandas==0.4.7)
  Downloading numpy-1.24.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 7.6 MB/s eta 0:00:00
Collecting click>=8.0 (from dask->spatialpandas==0.4.7)
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting cloudpickle>=1.5.0 (from dask->spatialpandas==0.4.7)
  Using cached cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Collecting packaging>=20.0 (from dask->spatialpandas==0.4.7)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting partd>=1.2.0 (from dask->spatialpandas==0.4.7)
  Using cached partd-1.4.0-py3-none-any.whl (18 kB)
Collecting pyyaml>=5.3.1 (from dask->spatialpandas==0.4.7)
  Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.9/757.9 kB 3.3 MB/s eta 0:00:00
Collecting toolz>=0.10.0 (from dask->spatialpandas==0.4.7)
  Using cached toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting importlib-metadata>=4.13.0 (from dask->spatialpandas==0.4.7)
  Using cached importlib_metadata-6.6.0-py3-none-any.whl (22 kB)
Collecting llvmlite<0.41,>=0.40.0dev0 (from numba->spatialpandas==0.4.7)
  Downloading llvmlite-0.40.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.1/42.1 MB 9.5 MB/s eta 0:00:00
Collecting python-dateutil>=2.8.2 (from pandas->spatialpandas==0.4.7)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1 (from pandas->spatialpandas==0.4.7)
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting tzdata>=2022.1 (from pandas->spatialpandas==0.4.7)
  Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting six>=1.7.0 (from retrying->spatialpandas==0.4.7)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->dask->spatialpandas==0.4.7)
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
Collecting locket (from partd>=1.2.0->dask->spatialpandas==0.4.7)
  Using cached locket-1.0.0-py2.py3-none-any.whl (4.4 kB)
Building wheels for collected packages: python-snappy
  Building wheel for python-snappy (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      /home/user/mambaforge/envs/temp/lib/python3.11/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'cffi_modules'
        warnings.warn(msg)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/__main__.py -> build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/snappy.py -> build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/hadoop_snappy.py -> build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/snappy_cffi_builder.py -> build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/snappy_cffi.py -> build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/snappy_formats.py -> build/lib.linux-x86_64-cpython-311/snappy
      copying src/snappy/__init__.py -> build/lib.linux-x86_64-cpython-311/snappy
      running build_ext
      building 'snappy._snappy' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/src
      creating build/temp.linux-x86_64-cpython-311/src/snappy
      gcc -pthread -B /home/user/mambaforge/envs/temp/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/user/mambaforge/envs/temp/include -fPIC -O2 -isystem /home/user/mambaforge/envs/temp/include -fPIC -I/home/user/mambaforge/envs/temp/include/python3.11 -c src/snappy/crc32c.c -o build/temp.linux-x86_64-cpython-311/src/snappy/crc32c.o
      gcc -pthread -B /home/user/mambaforge/envs/temp/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/user/mambaforge/envs/temp/include -fPIC -O2 -isystem /home/user/mambaforge/envs/temp/include -fPIC -I/home/user/mambaforge/envs/temp/include/python3.11 -c src/snappy/snappymodule.cc -o build/temp.linux-x86_64-cpython-311/src/snappy/snappymodule.o
      src/snappy/snappymodule.cc:33:10: fatal error: snappy-c.h: No such file or directory
         33 | #include <snappy-c.h>
            |          ^~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for python-snappy
  Running setup.py clean for python-snappy
Failed to build python-snappy
ERROR: Could not build wheels for python-snappy, which is required to install pyproject.toml-based projects

Describe the solution you'd like
A clear and concise description of what you want to happen.
Convert python-snappy from a required to an optional dependency in the setup.py file:
https://github.com/holoviz/spatialpandas/blob/bc3e52cfb2ef84c411ad0aff34a683ee0955ce66/setup.py#L31-L40
Looking at the codebase, I only see snappy mentioned for the parquet I/O in two places:
https://github.com/holoviz/spatialpandas/blob/bc3e52cfb2ef84c411ad0aff34a683ee0955ce66/spatialpandas/dask.py#L208
https://github.com/holoviz/spatialpandas/blob/bc3e52cfb2ef84c411ad0aff34a683ee0955ce66/spatialpandas/io/parquet.py#L90-L181
So for operations that don't use parquet, it should not be necessary to use python-snappy. Note that pandas does support other compression methods like gzip as mentioned at https://pandas.pydata.org/pandas-docs/version/2.0/reference/api/pandas.DataFrame.to_parquet.html, though snappy compression is currently the default.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Ideally, python-snappy would release Python 3.11 compatible wheels at https://github.com/andrix/python-snappy/issues/124, but the last commit on that repo was 17 Mar 2022, so not likely to happen anytime soon.
Additional context
Add any other context or screenshots about the feature request here.
I noticed that there was a PR checking for Python 3.11 compatibility at https://github.com/holoviz/spatialpandas/pull/113, but in that case, python-snappy was installed from conda-forge (that does support Python 3.11 https://anaconda.org/conda-forge/python-snappy/files?version=0.6.1) rather than PyPI.
For historical context, snappy was added as a required dependency in 498e7fcd0f73132ffde472ada1fa22f3037205df/#60.
Happy to open a PR to make python-snappy optional if the above sounds good!

Happy to open a PR to make python-snappy optional if the above sounds good!

Hi @weiji14. Yes please!
It looks like from https://github.com/andrix/python-snappy/issues/124 that the recommendation is to replace use of python-snappy with cramjam instead. But that is long-term, at the moment I'd be happy if our CI passes without python-snappy.

It looks like from andrix/python-snappy#124 that the recommendation is to replace use of python-snappy with cramjam instead. But that is long-term, at the moment I'd be happy if our CI passes without python-snappy.

Cool, started a PR at #117. We could look into cramjam separately, it looks like a promising replacement built on Rust!


ability to filter assets based on multiple nested data values in GetAllAssets

Is your feature request related to a problem? Please describe.
In the v1beta1/assets api, we can filter based on query fields. Even though multiple query_fields can be supplied in the request, only one query value is taken.
For example -
with this API -
v1beta1/assets?types=table&q_fields=data.landscape.properties.entity,data.landscape.properties.attribute&q=mkns

It is possible to build a filter query like this ✅  -
WHERE type IN ('table')
AND (data->'landscape'->'properties'->>'entity' ILIKE '%mkns%'
OR data->'landscape'->'properties'->>'attribute' ILIKE '%mkns%')

But if we want to build a query like below, it is currently not possible. ❌
WHERE type IN ('table')
AND (data->'landscape'->'properties'->>'entity'= '%abcd%'
OR data->'landscape'->'properties'->>'entity' = '%mkns%')

Describe the solution you'd like
Make the Filter.Data field type to be map[string][]string, and in this block, handle it accordingly.
Describe alternatives you've considered
One of the alternative is to let the caller of this API, call it multiple times with different filters - but that would defeat the purpose of sorting, filtering and page-offsets in this API.
Additional context
Add any other context or screenshots about the feature request here.

Search API has the ability to pass many values for the same data field, but it does a search on ES and it's not filtering per se. Also, search API doesn't support pagination or offset.
GetAllAssets has ability to filter but it's just that it filters in an opinionated way, which is not apparent through the API spec. This change will help in reducing that and increasing filter use-cases.

Search API has the ability to pass many values for the same data field, but it does a search on ES and it's not filtering per se. Also, search API doesn't support pagination or offset.
GetAllAssets has ability to filter but it's just that it filters in an opinionated way, which is not apparent through the API spec. This change will help in reducing that and increasing filter use-cases.

I see, I can see the use case is clear, my only concern is more to do we want to focus more on GetAllAssets for this feature or should it be more on Search API.
Other than that I really don't have a strong opinion, wdyt @ravisuhag @sudo-suhas ?

I think it will be ok to have this on this API unless it affects the performances.

My concern was w.r.t implementation where each value in the map can be CSV. This can be a problem if any of the individual values of a field already contain a comma. I would instead prefer we add a new field GetAllTypesRequest.DataV2  of type map[string][]string. We can deprecate the existing field without breaking backward compatibility by translating Data to DataV2 if it is present in the request.


run cattle node agent in a non-privileged mode

rancher/rancher:2.0.x
Currently Rancher requires both the Kubelet and Kube-API server to be launched with --allow-privileged=true
In some circumstances users will be running clusters with those settings set to false. Rancher should degrade gracefully if privileged mode is not possible. Or, alternatively, we should find an alternate approach to using privileged containers for the node agent.

+1 for this as it is affecting customers

+1
We got a critical security issue because of that.
what is the exact reason that the agent has privileged: true?
https://github.com/rancher/rancher/blob/bad6400ab85b0ac2de6cf7698c389b7761fc54d2/pkg/systemtemplate/template.go#L196
is it possible to patch the agent to remove the privileged after the deployment? or maybe to get a flag while deploying the rancher server (using helm) so it will deploy the cattle agent without privileged ?


Programmatic Dismissal

Is it possible to have the Loaf dismiss programmatically? I.e. it appears with the message “Connecting ...” and it dismisses on a completion callback?

Nevermind, I see it in the docs, sorry for the noise.
Loaf.dismiss(sender: self) // Where self is the Loaf's presenter


Textile markup link

Closes https://www.pivotaltracker.com/story/show/90682766

The link under 'Citation' to the Textile details should look the same as the one under 'Notes on Taxomony'.  Personally I prefer making "Textile markup system" a link.

Yes, I agree with Nathan.  This could be dealt with in the translation, not the code, I think.  Just make the appropriate part of :form_names_citation_textilize_note a link.  Or does this really need anything??  The link is shown clearly just a short way down the page already!  I vote -- no offence intended, Ray! -- to just delete this PR.

Replaced with PR that just adds link to en.txt instead.


addes new object info

Summary

Checklist

[x] If code changes were made then they have been tested

[x] I have updated the documentation to reflect the changes
[ ] I have formatted the code properly by running black .

[ ] This PR fixes an issue
[ ] This PR adds something new (e.g. new method or parameters)
[ ] This PR is a breaking change (e.g. methods or parameters removed/renamed)
[ ] This PR is not a code change (e.g. documentation, README, ...)

tested it too

from disnake.ext import commands
import disnake
import typing
from disnake.ext.commands.core import command
bot = commands.Bot(command_prefix = "e*", intents=disnake.Intents.all())
@bot.event
async def on_ready():
print(f"Bot is ready")
@bot.command(brief="gives information on snowflakes")
async def snowflake_info(ctx, *, snowflake: typing.Optional[disnake.Object] = None):
if not snowflake:
  await ctx.send("you either returned nothing or an invalid snowflake now going to the current time for information.")

generated_time = await commands.ObjectConverter().convert(ctx, argument=f"{int((disnake.utils.utcnow()).timestamp() * 1000 -<PHONE_NUMBER>000) << 22 | 0x3fffff}")
#yes the later bit uses generate_snowflake's older version

snowflake = snowflake or generated_time

embed = disnake.Embed(title="❄ SnowFlake Info:", color=5793266)

embed.add_field(name="Created At:",
                value=f"{disnakeutils.format_dt(snowflake.created_at, style = 'd')}\n{disnake.utils.format_dt(snowflake.created_at, style = 'T')}")

embed.add_field(name="Worker ID:",
                value=f"{snowflake.worker_id}")

embed.add_field(name="Process ID:",
                value=f"{snowflake.process_id}")

embed.add_field(name="Increment:", value=f"{snowflake.increment_id}")

embed.set_footer(text=f"Snowflake ID: {snowflake.id}")

await ctx.send(embed=embed)

TOKEN = ""
bot.run(TOKEN)

Closing as the quality of this pr is unmanageable.


Add support for Java 16 and higher

This upgrades the ASM and ByteBuddy (via Mockito) dependencies so they are compatible with Java 16 and higher. To test, I built this library with the three current (as of 2 October 2021) LTS Java versions (8, 11, and 17) as well as Java 16. In addition, I tested by running the ClientOfMutabilityDetector tests with Java 16 and 17 while pointing to a local version of MutabilityDetector built with Java 8.
I also updated the build matrix for GitHub Actions to use only the current LTS versions. While doing so, I upgraded to version 2 of the Action and specified zulu as the distribution for consistency with previous builds. I can revert these changes or use temurin if necessary.
Resolves: #181

@l0s excellent! I am sorry I missed this. I'll get this merged and cut a release soon.

@l0s your contribution is much appreciated. I'll likely publicise this release (twitter, blog) so let me know if you have a preference in how you are mentioned (GitHub profile, twitter, not mentioned at all, etc).

Thanks for the offer @Grundlefleck ! I'm @l0s on Twitter as well. On other platforms, a link to my GitHub profile would be great! Thanks!


Enable setting the LivePreview server address in Brackets preferences.

Add functionality to configure the LiveServer address in Brackets preferences file. This enables setting like this to let you access the LiveServer across a local network:
"staticserver.address": "<IP_ADDRESS>"

Addresses #6424

@tony2nite You need to sign Brackets CLA.
@busykai Would you mind taking a look, when you get a chance?

@nethip I've signed-up. Twice actually. Let me know if it's not registering correctly.

@tony2nite The CLA checks seem to be have passed. The errors now are JS linting errors. Please fix the linting errors and update this PR. Thanks!

Hi @nethip. Fixed the linting errors and updated. Looks green to me.

Thanks @tony2nite . Let us wait to hear from @busykai.

oh, right. this has been a long-standing issue. thanks @tony2nite for bringing this up!
it has been discussed more recently in #10289. i actually created a branch which implemented the necessary changes, it's here: kai/ld-remote-connectivity. in that branch i'm just switching the implementation to INADDR_ANY, non-configurably (actually that's about the only piece that is missing in that branch). i'm wondering if we could reconcile the two. it seems that i've created a PR #10517 at some point, but it's gone now and github does not let me to create another one from the same branch. anyways, here is the comparison.
things i dislike in this PR:

address should not be passed as is from configuration to listen. there are really two options here: you either listen to INADDR_ANY (<IP_ADDRESS>) or to localhost. a more elaborate third choice should be expressed in terms of the physical/virtual interfaces to listen to, but it's not necessary.
i doubt this is functional in the first place since the url-to-document translation wouldn't work for the remote URLs. listening to INADDR_ANY is not enough.
multi-browser LivePreview implementation is not addressed.

it seems to me that the best course of action would be to take kai/ld-remote-connectivity as the base and make it configurable (with only one boolean switch indicating if listen to all interfaces or loopback only, defaulting to loopback). i will be able to finish this up and create a pull request in a couple of weeks or so. on the other hand, if @tony2nite is willing to take the work done in kai/ld-remote-connectivity and finish it up, that would be cool as well!

Bump!

What is the status on this?


Remote read performance improvement in distributed training of millions of files

Summary
Use Nvidia Data loading library DALI https://developer.nvidia.com/dali for distributed data loading. Nvidia  Data Loading Library DALI provides built in data loaders and data iterators. DALI supports overlapping training and data pre-processing to reduce latency and training time. It can be used as a benchmarking to test the data loading speed from Alluxio storage in AI workloads.
Workloads

Launch AWS EKS cluster

EKS cluster, 1 master and 6 workers, r5.8xlarge (32 vCPU, 256GB memory)
512 gp2 SSD for worker nodes

Launch Alluxio cluster

All the workers are SSD only with Worker embedded Fuse.
Root UFS is a s3 bucket with all imagenet original data, a little modification use the script https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh
1000 folders * 1300 imgs/folder * 105KB on average each file (range from 30k to 200k), total 140GB data

Use Arena https://github.com/kubeflow/arena to launch the data loading script in nodes running AlluxioWorker

Arena is a command-line interface for the data scientists to run and monitor the machine learning training jobs and check their results in an easy way

Data loading script
Our benchmarking code is modified from Nvidia DALI example script given by the DALI tutorial ImageNet Training in PyTorch. The original script supports reading from the imagenet original dataset, do data loading, data preprocessing, and data iteration with DALI, train the model with pyTorch resnet models.
Our modifications include

Remove the training logic from the script, only include the DALI data loader logics.
Change DALI to use CPU only instead of using GPU or  MIXED devices.
Support multi-process data loading in each node

Our script can:

Support running DALI data loader from multiple nodes, multiple data loading threads in each node
Record the images loaded number and data loading time of each node and calculate the image/s/node.

Launch arena job
arena --loglevel info submit pytorch \
--name=test-job \
--gpus=0 \
--workers=4 \
--cpu 4 \
--memory 32G \
--selector alluxio-master=false \
--image=nvcr.io/nvidia/pytorch:21.05-py3 \
--data-dir=/alluxio/ \
--sync-mode=git \
--sync-source=https://github.com/LuQQiu/DALI.git \
"python /root/code/DALI/docs/examples/use_cases/pytorch/resnet50/main.py \
--process 4 \
--batch-size 256 \
--print-freq 1000 \
/alluxio/alluxio-mountpoint/alluxio-fuse/dali"

$  arena list test-job                                                    ok
NAME      STATUS     TRAINER     AGE  NODE
test-job  SUCCEEDED  PYTORCHJOB  33m  N/A
$ kubectl get pods
NAME                READY   STATUS      RESTARTS   AGE
test-job-master-0   0/1     Completed   0          34m
test-job-worker-0   0/1     Completed   0          34m
$ kubectl logs test-job-master-0
$ kubectl logs test-job-worker-0

Experiment 1
Question: why cluster read throughput is that slow, only 2400 img/s
Observation:
cluster settings

CPU/memory
Alluxio configuration -> master/worker/fuse config
ML workloads, ML read pattern
Alluxio Fuse
Underlying FUSE/libfuse/jnifuse impl
Alluxio Fuse impl
Alluxio client code
Alluxio client metadata ops
Alluxio client data ops

Hypothesis:
CPU resources are not enough
Memory resources are not enough
Some of the Alluxio configs is missing
The ML read pattern is special, not like traditional workloads. We didn’t deal with the special pattern well
Performance is bounded by underlying modules: Fuse + libfuse + jnifuse
Performance is bounded by Alluxio Fuse + Alluxio client/server
Which steps are most cost-effective?
Which steps help rule out the biggest possibilities and help narrow down the question?
Try Hypothesis 1 & 2
Try to enlarge the CPU/Memory -> throughput doesn’t change

Experiment 2
Question: Narrow down the question: Performance is bounded by
Underlying Fuse + libfuse + jnifuse or Alluxio Fuse/client/master/worker
Hypothesis: Fuse + libfuse + jnifuse + directly reading from local filesystem can generate much better throughput than 1500 img/s. Alluxio impl has the potential to achieve much better performance.
Experiment:
StackFs = Fuse + libfuse + jnifuse + directly reading from local filesystem 6000 img/s
AlluxioFuse = Fuse + libfuse + jnifuse + AlluxioFuse with 6 replica on local machine 1600 img/s
Conclusion: Alluxio impl has the potential to achieve much better performance

Experiment 3
Question: How can Alluxio with 6 replica read performance increase from 1600 img/s to a value that close to 6000 img/s
Observation: compare the two env and understand the diff. Understand the Alluxio read path and see where could be the potential bottleneck. Use flamegraph/visualvm to see potential bottlenecks.
Hypothesis

Metadata operation: Metadata in StackFs is local while AlluxioFuse needs to fetch metadata from master
Data operation: RPC between Fuse and Worker. RPC code itself or RPC resources contention

Experiment

Enable client side metadata cache
Enlarge client pool size
Use hacky changes to remove RPC between Fuse and  local Worker

Result: increase from 1500 img/s to 4700 img/s ← which we are satisfied
Actions:

add worker storage java opts to client java opts since Fuse need to know the worker storage location
Add metadata cache to cover the whole test

ALLUXIO_FUSE_JAVA_OPTS: >-
  -Dalluxio.user.metadata.cache.enabled=true -Dalluxio.user.metadata.cache.expiration.time=2h -Dalluxio.user.metadata.cache.max.size=10000000

Enlarge the client pool size

-Dalluxio.user.block.worker.client.pool.max=4096

Implement Fuse on Worker
https://docs.google.com/document/d/1hdI5LPAPN-NPEYhcxdw47yGyP4GdyzJRf98oawJx0rA/edit?usp=sharing

Experiment 4
Question: Why Fuse on worker implementation does help improve the overall cluster read performance? How can we improve cluster read performance?
Hypothesis: Remote read is the performance bottleneck.
Experiment: Add a bunch of metrics to see where the time spend.
Result: Most of the time spend in remote read. Fuse process local read 576 MB/s, Fuse remote read 16.4 MB/s. More than 30X diff.
Conclusion: Remote read is the performance bottleneck.

Experiment 5
Question: Why remote read is the performance bottleneck? How to improve remote read performance?
Hypothesis:
gRPC itself is the performance bottleneck
Remote read slowness is expected, since it includes transferring data through network. Network IO is the performance bottleneck
Experiment:
Implement Fuse single node read stressbench test to compare Fuse process local read vs gRPC local read vs gRPC remote read
Result: the gap between the three read way (<2X)  is much smaller than what observed in MS env (>30X)

Read type/Thread number
1
2
4
8

Process Local
61MB/s
128MB/s
258MB/s
247MB/s

Grpc Local
54.9MB/s
109MB/s
213MB/s
250MB/s

Grpc Remote
43.8MB/s
91MB/s
186MB/s
251MB/s

Experiment 6
Question 1: Why remote read is the performance bottleneck? How to improve remote read performance?
Question 2: What are the differences between DALI env and Stressbench env that could result in the diff?
The hypothesis from observation:

CPU resources. DALI env worker only has 1 CPU (configured by kubectl yaml scripts）which stressbench env worker can use up to 16 CPU
ML read pattern. Stressbench open()-read()-close(), only 4 files are open() at the same time while ML env more than 1500 files are opened at the same time.

Experiment

Enlarge CPU resources
Close gRPC stream immediately after reading all the data

Results: Both hypotheses are performance bottlenecks. With the combined effort, we increase performance from 2k img/s to more than 13k img/s.

Enlarge CPU
CPU resources needed for training

Node CPU
Data Loading Process
Data Loading CPU
Arena Memory (GB)
Alluxio Worker CPU
Alluxio Worker Memory (GB)
Throughput

32
4
4
32
3
16
4723 img/s

32
8
10
32
4
16
9425 img/s

32
12
12
32
6
16
12449 img/s

32
16
20
32
10
32
13725 img/s

Close grpc stream immediately after reading all the data belongs to a file https://github.com/Alluxio/alluxio/issues/14020
Small improvement: Avoid unneeded async cache RPCs by setting alluxio.user.file.readtype.default=NO_CACHE. Async cache will be issued when reading data from remote worker, but because this is a single tier storage and enough space, async cache manager will not do anything

Can we provide the detail Alluxio configurations for this experiments? We do want to reproduce the results in our test bed.

@Binyang2014 That's the alluxio configuration i used
mlk8s.zip
Recommend using the docker image  with changes in https://github.com/Alluxio/alluxio/pull/14017


Add a change-type to updateData in Media.js

When we update data for plugins, there is no change-type. In order to act on these changes with editorState.getLastChangeType(), I propose to add a type called plugin-update-data. We can also make this type configurable and pass it in as a parameter.

Hey @lightninglu10!
Looks like it would be useful to add the change type when calling updateData.
I'm just not sure about the naming, looks like DraftJS has an enum of available change types, even if it accepts custom types, it probably won't be useful because these types are used internally.
So I believe we should use change-block-data instead of plugin-update-data.
What do you think?
Also, could you please add tests for this change?

@ricobl made the changes you requested and now passing all checks


Want to get tweet of one date only?(Specific Date)

I was trying to fetch tweets of one specific date. I am trying this code but every time it return no output(no tweet).
searchParameter.Since = new DateTime(2016, 06, 06, 00, 00, 01);
searchParameter.Until = new DateTime(2016, 06, 06, 23, 59, 59);

Unfortunately twitter does not provide this type of precision and you will not be able to search for only one day.
What I would suggest is that you do the following :
var date = new DateTime(2016, 06, 13);

var tweets = Search.SearchTweets(new TweetSearchParameters("hello")
{
    Since = date.AddDays(-1),
    Until = date,
    SearchType = SearchResultType.Recent
}).Where(x => x.CreatedAt.DayOfYear == date.DayOfYear).ToArray();

k thanks sir.

Don't forget to star tweetinvi github repository ;)

Done Sir :)

hello i need the  tweets that tweeted last day with a keyword. But this code turns me a null . can you help me please ?


local.settings.json values bug while reading in Program.cs

As you can see in the screenshot, I try to read values from local.settings.json. The debugger shows the correct value, but in console.writeline() it's empty. This would be necessary for configuration of dbContext for EF Core.

Also reading from Environment Variables like shown here, doesn't work with local debugging:
https://github.com/Azure/azure-functions-dotnet-worker/blob/main/samples/EntityFramework/Program.cs
Here is a sample for reproducing this issue:
SettingsBugSample.zip

I just found my failure... In the Console.WriteLine must be a '+' instead of ','. Did too much Typescript the past weeks... Now it is working


Unit Tests to avoid regression in existing demo programs

Proposing this issue as an open discussion.
I'd like to know your opinion about adding "unit" tests for the demo programs. That would imply:

to store the XLSX files generated by the demo programs in the Web Repository (W3MI objects)
to extract the code from the demonstration programs into new classes which can be called from the demo programs and from a test program
to create a test program whose ABAP Unit test checks that the file generated by each demo program

In the future, we could also use this feature to make sure that both "Writer" and "Huge Writer" generate the same result (currently we have many features which don't work in the Huge Writer), and the same for "Reader" and "Huge Reader". This
program can be used as a starting point for discussion on Writer/Huge Writer, it runs 2 demos online and displays the diff, which works as explained in #891, and it also contains the test classes for them.

I like the general idea and the few doubts I have will probably be swept aside once I see the test program.
Some questions I have:

how fine-grained will the checks have to be? Some changes (moving an option within a xml node) are harmless, whereas other are potentially disruptive.
how do we pick the reference files? Do we arbitrarily choose the currently generated versions or do you have more specific criteria to match?

Thanks for looking into it. I have changed my mind about ABAP Unit, because what I propose is not about unit testing. So I switched to a simple program which is to be run on demand and displays the differences of actual versus expected.
I have created this draft #902 so that you can run the demo program ZDEMO_EXCEL_CHECKER. Note that you must install https://github.com/sandraros/zip-diff if you want to compare the ZIP at file level. If not, it just shows that there is a difference. By using a separate project, I can improve zip-diff any time, any fix will not pollute abap2xlsx.

Any change would be shown, except for fields which change at each generation, and for cells containing sy-datum and sy-uzeit.
I have created the reference files in SMW0, copying them from files generated by old versions of ZDEMO_EXCEL## programs, except those which fill the Excel file with current date and time, which are now forced to a fixed date and time when running through ZDEMO_EXCEL_CHECKER, so that the comparison can be done. These programs are: ZDEMO_EXCEL_COMMENTS, ZDEMO_EXCEL1, ZTEST_EXCEL_IMAGE_HEADER, ZDEMO_EXCEL24.

It's just a preview, all questions and comments welcome!

This being a draft, I propose that I deal with it last, after the other PR in the queue have been reviewed. In the meantime, perhaps #891 could be closed and zip-diff included in the project documentation somewhere (Contributing?)

Well, I kept the PR as draft because I think we are more in the process of discussion, and I did the PR so that to clarify my proposal. When we are clear about the goal and what the program should do exactly, we can complete the PR and pass it "final".
I would suggest to do #894 first because it will permit to make sure easily that there are no regressions with all new fixes.
#891 should not be closed, it's a different program. I proposed 2 different programs in #891 and #894, both use zip-diff, and both bring functionality which doesn't exist in zip-diff:

#891: display screen with Excel workbook to the left and show differences in the XLSX file to the right after manual changes in Excel workbook and button pressed
#894: show list of most abap2xlsx demo programs and indicate if there are regressions compared to MIME Objects stored in Web Repository
zip-diff: display differences between 2 ZIP-like files (ZIP, XLSX, DOCX, etc.)

If possible, I'd prefer to update the Contributing part after we are clear about the program.

#891 should not be closed, it's a different program.

I ran zip-diff by itself once, saw that there was one option giving a display similar to the program snippet in #891 and thought that was all; my bad, I did not try the other option, now that I have used it with ZDEMO_EXCEL_CHECKER it all makes sense.

I proposed 2 different programs in #891 and #894, both use zip-diff

However the program in 891 works even without zip-diff: did you mean that both display a diff or am I perhaps using an older version?

I would suggest to do #894 first because it will permit to make sure easily that there are no regressions with all new fixes.

You mean #902 right? From a not-so-quick look at the code it looks like a good solution to the problem, without overburdening the rest of the code or the repository. I will start reviewing this in the coming days, then. Also, ok with updating the docs later on.

Sorry, I created zip-diff while proposing #894, and I didn't rework #891 yet, to use zip-diff.
#902 if you prefer 😉. As I said, it's worth discussing first the issue (#894), the draft PR can be used to support the discussion.
Thanks! 👍

Comments in no particular order:

we'd probably need a small section in the contributing guidelines on the new way to write demo reports (interface and so on)
I'm not sure I understand exactly why GET_NEXT_GENERATOR is empty except for ZDEMO_EXCEL15: once it gets there it runs through the rest of the rows in the table and the loop in the external program is bypassed
I need to dig deeper into the workings of the generator class and maybe force spurious errors to see how it looks then
Will try again in the coming days.

Moar comments:

why the full class path for ty_zip_structure ? It's used only in the same classe, zcl_excel_generator.
generator: if I am not mistaken, the loop in ZDEMO_EXCEL_CHECKER points to all the demos you can run as a user and the iteration done via GET_NEXT_GENERATOR serves to feed individual test cases for demos that might run with more than one file.
forced an error and saw how it behaves, very nice.

All in all, I'd say the PR could be un-drafted.

I answer the whole 6 points here:

Documentation created here but not sure how to integrate it in a more general "coding" documentation which should wrap the variable naming standards and how demo programs work.
GET_NEXT_GENERATOR is a quick and dirty solution for ZDEMO_EXCEL15 and for all future programs which generate more than one Excel file. Didn't think a lot about it, any suggestion welcome.
OK
zcl_excel_demo_generator=>ty_zip_structure is generated by ADT refactoring tool (maybe related to my old ABAP 7.52). I manually fixed it.
ZDEMO_EXCEL_CHECKER should test only demo programs which produce files of type Excel (not CSV), and only the ones which produce the same data all the time (so not ZDEMO_EXCEL25, and those which write the current time were modified to write a fixed time when ran by ZDEMO_EXCEL_CHECKER).
Thanks
The PR is un-drafted
Currently, I have kept one Excel file in the Web repository which is different from what is actually produced (ZDEMO_EXCEL1), just for the tests, we'll need to commit the latest Excel file.

One quick comment with respect to n.4 above; I'm not sure it was worth fixing, now that I know how it was created I'll take this into account for future reviews, thanks for the explanation.


Demangled C++ names are hard to read

All of the C++ STL containers are templates with default arguments that expand at compile time.  For example:
template < class T, class Compare = less<T>, class Alloc = allocator<T> > class set;
template < class charT, class traits = char_traits<charT>, class Alloc = allocator<charT> > class basic_string;

Which in source would be used as, for example, std::set<std::string> but at compile-time expand to
std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >

Which is hard to read.
BinaryNinja should parse and, in the case of default arguments, simplify these.

Fixed in 2.1.2481-dev (see the new "Simplify Templates" binary view setting, as well as new API's in demangle.py).


Issue LoxoneIntercom

Hi,
I have an Issue with the Plugin and the IntercomV2
Homebridge is restarting always because of an Charakteristik issue.
The Plugin is running on Version 1.4
[10/8/2023, 4:10:59 PM] [homebridge-loxone-proxy] Connected to Miniserver
TypeError: Cannot read properties of undefined (reading 'name')
    at IntercomV2.configureMotionSensor (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/IntercomV2.ts:28:42)
    at /homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/IntercomV2.ts:16:12
    at /homebridge/node_modules/homebridge-loxone-proxy/src/loxone/LoxoneHandler.ts:181:52
    at Array.forEach (<anonymous>)
    at LoxoneHandler.registerListenerForUUID (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/LoxoneHandler.ts:181:32)
    at IntercomV2.configureCamera (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/IntercomV2.ts:10:33)
    at IntercomV2.configureServices (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/Intercom.ts:20:10)
    at IntercomV2.setupAccessory (/homebridge/node_modules/homebridge-loxone-proxy/src/LoxoneAccessory.ts:45:10)
    at new LoxoneAccessory (/homebridge/node_modules/homebridge-loxone-proxy/src/LoxoneAccessory.ts:31:10)
    at new Intercom (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/Intercom.ts:9:1)

Would you be able to install the latest beta version? (1.4.0.beta.2)
This will give you some additional debug output. Could you post these debug messages?

For sure there you go.
[10/10/2023, 3:20:58 PM] [homebridge-loxone-proxy] Connected to Miniserver
!!! BEGIN DEBUG !!!
targetUuidPrefix: 17803381
intercomMotionUuid:
undefined
!!! END DEBUG !!!
[10/10/2023, 3:20:59 PM] [homebridge-loxone-proxy] This plugin generated a warning from the characteristic 'Configured Name': Characteristic not in required or optional characteristic section for service Switch. Adding anyway.. See https://homebridge.io/w/JtMGR for more info.
TypeError: Cannot read properties of undefined (reading 'name')
    at IntercomV2.configureMotionSensor (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/IntercomV2.ts:36:42)
    at /homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/IntercomV2.ts:16:12
    at /homebridge/node_modules/homebridge-loxone-proxy/src/loxone/LoxoneHandler.ts:181:52
    at Array.forEach (<anonymous>)
    at LoxoneHandler.registerListenerForUUID (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/LoxoneHandler.ts:181:32)
    at IntercomV2.configureCamera (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/IntercomV2.ts:10:33)
    at IntercomV2.configureServices (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/Intercom.ts:20:10)
    at IntercomV2.setupAccessory (/homebridge/node_modules/homebridge-loxone-proxy/src/LoxoneAccessory.ts:45:10)
    at new LoxoneAccessory (/homebridge/node_modules/homebridge-loxone-proxy/src/LoxoneAccessory.ts:31:10)
    at new Intercom (/homebridge/node_modules/homebridge-loxone-proxy/src/loxone/items/Intercom.ts:9:1)

I also checked my json from Loxone if the device UUID is matching.
The json and the Debug output with "targetUuidPrefix" is correct.

The latest release should fix your issue. There is a MotionSensor item available on these V2 intercoms. If you want to use it with this plugin you need to enable the "use in userinterface" option on this item in LoxoneConfig.
The latest release will just ignore it if it is not enabled.


Sbt upgrade

Summary of changes

Upgraded sbt to next major version (v0.13.x -> v1.x)
Support GPG v2 for release signing

Looks good, but tests are failing...

It would be so nice to merge this PR.

All these make sense, but we still have the OOM error. Maybe reverting back to -J-Xmx5500M in .sbtopts would help? @marcovzla, what do you think?

6GB RAM is a tiny amount for this type of application.

I agree it would be nice to merge this PR. Upgrading sbt would simplify things for me.
We haven't done so because Travis runs out of memory when executing the unit tests.
See https://travis-ci.org/clulab/processors/jobs/499513609#L2673
We wanted to figure out a way to decrease the memory consumption so that we can still use Travis, but no one is working on it as I know.

Seems like this PR is simple and straightforward. It works. If Travis is getting in the way, why not stop using it? Otherwise this project remains frozen in time, which would be a shame.

@marcovzla: does this PR pass if you set -J-Xmx5500M?

It doesn't. I can take a look again next week.

@marcovzla, @myedibleenso, @mslinn: This was merged in master through a different branch. Our unit tests have been switched from Travis to Jenkins, where we have more RAM.


Update s3 to 2.15.69

Updates software.amazon.awssdk:s3 from 2.15.61 to 2.15.69.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk", artifactId = "s3" } ]

labels: library-update, semver-patch

Superseded by #303.

Superseded by #303.


feat: percy

:tada: This PR is included in version 1.3.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release
v1.3.0

Your semantic-release bot :package::rocket:


ERROR - Failed to load skill: skill-iot-control.mycroftai

Attempting to use this skill and seeing the following error:
Traceback (most recent call last):
  File "/opt/mycroft/mycroft/skills/core.py", line 131, in load_skill
    imp.PY_SOURCE))
  File "/usr/lib/python3.5/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.5/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 693, in _load
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 661, in exec_module
  File "<frozen importlib._bootstrap_external>", line 767, in get_code
  File "<frozen importlib._bootstrap_external>", line 727, in source_to_code
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/opt/mycroft/skills/skill-iot-control.mycroftai/__init__.py", line 63
    self.speech_requests: DefaultDict[str, List[SpeechRequest]] = defaultdict(list)
                        ^
SyntaxError: invalid syntax

Python 3.5.2.
Mycroft 19.2.13
I am seeing a similar error with the Home Assistant skill (which is why I need this skill). Is this a Python version issue?

I managed to fix this by upgrading Python to 3.7.4


Fix persistent sessions scale down with envoy

Without it envoy will drop the draining endpoints and scale down will break.
Istiod does generate the DRAINING endpoints - and proxyless gRPC can use them, but in
envoy they don't show up in the EDS. This per-cluster flag allows them to be held - they will
not show up as DRAINING but as UNHEALTHY ( called 'coarse state' in envoy, quite confusing ).
Please provide a description of this PR:

Would this setting cause new connection established to the draining ep?

@hzxuzhonghu The effect is restricted to stateful session filter - and the purpose is indeed to continue sending requests to draining endpoint.

New connections will not go to draining EP - only if they have explicit cookie/header.
I'm checking if proxyless gRPC needs the same thing, will update after I get an answer.
Kuat - will add the other states, you are right.

@costinm: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
bb4e827d2a1d96cacf0180d41cab2113c7513aaf
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@costinm: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
bb4e827d2a1d96cacf0180d41cab2113c7513aaf
link
true
/test lint

gencheck_istio
bb4e827d2a1d96cacf0180d41cab2113c7513aaf
link
true
/test gencheck

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

New connections will not go to draining EP - only if they have explicit cookie/header.

@kyessenov @costinm I understand for the existing connection with draining EP, client can still send requests over it even without this patch.
The default behavior of unallowing new connection established to draining EP looks reasonable, and this is silimar to https://cloud.google.com/load-balancing/docs/enabling-connection-draining

/retest

Not sure I understand the comment "before DRAINING was available and supported" - but from testing with envoy
without this PR the DRAINING endpoints in EDS response don't show up in config dump and seem to be dropped.
With this change, they are kept and show up as UNHEALTHY.
Can someone approve - it doesn't make things worse - and we can refine in new PRs. Would be nice to backport this
so persistent sessions can work properly.

In response to a cherrypick label: new pull request created: #44652

In response to a cherrypick label: new pull request created: #44653


Enhance diagnostics plugins

Objective

Permits to change default diagnostic plugins max_history_length
Correctly set the default ema_smoothing_factor based on the max_history_length argument in Diagnostic::new

Solution

Add a max_history_length field to EntityCountDiagnosticsPlugin, FrameTimeDiagnosticsPlugin and SystemInformationDiagnosticsPlugin
Update the Diagnostic::new function to correctly set the ema_smoothing_factor based on the max_history_length argument.

Changelog
Changed

EntityCountDiagnosticsPlugin, FrameTimeDiagnosticsPlugin and SystemInformationDiagnosticsPlugin now take a max_history_length field, default is 20
The Diagnostic::new function now sets the ema_smoothing_factor based on the max_history_length argument

Fixed

Fixes an issue where SystemInformationDiagnosticsPlugin doesn't work if it's the only diagnostics plugin in the app. (unintended fix)

Migration Guide
If you were previously using one of the following plugins: EntityCountDiagnosticsPlugin, FrameTimeDiagnosticsPlugin or SystemInformationDiagnosticsPlugin.
They now have a max_history_length field.
Lazy update example
Before
use bevy::prelude::*;

fn main() {
    App::new()
        .add_plugins((
            DefaultPlugins,
            bevy::diagnostic::FrameTimeDiagnosticsPlugin,
            bevy::diagnostic::EntityCountDiagnosticsPlugin,
            bevy::diagnostic::SystemInformationDiagnosticsPlugin
        ))
        .run();
}

After
use bevy::prelude::*;

fn main() {
    App::new()
        .add_plugins((
            DefaultPlugins,
            bevy::diagnostic::FrameTimeDiagnosticsPlugin::default(),
            bevy::diagnostic::EntityCountDiagnosticsPlugin::default(),
            bevy::diagnostic::SystemInformationDiagnosticsPlugin::default()
        ))
        .run();
}

I've resolved the conflicts, but I'm not entirely confident about my changes related to the with_max_capacity function in Diagnostic. I don't really like the fact that we need to shrink/grow the VecDeque when we want a different max_history_length


Save button does nothing

Hi, I'm using the latest version as an add-on on 2024.7.3 of Home Assistant (HassOS). The issue I am facing is the save button is not working.
Is there any further info I can provide you to further trouble shoot?

Hi, yes, addon logs and browser console logs

Here you go:
js.log
Add-on logs:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
Starting Fusion...
ADDON: true
INGRESS_PORT: 8099
EXPOSED_PORT:
HASS_PORT: 8123

Thanks, in js.log i see
CodeConfig.CxOcBhK1.js:1 Y {name: 'YAMLException', reason: 'can not read a block mapping entry; a multiline key may not be an implicit key', mark: {…}, message: 'can not read a block mapping entry; a multiline ke… 12 | ffff\n 13 | \n 14 | theme: godis\n-----------^', stack: 'YAMLException: can not read a block mapping entry;…p/immutable/chunks/CodeConfig.CxOcBhK1.js:1:5418)'}

I think you're trying to save invalid yaml in code editor

Yeah I pasted the element incorrectly but the last attempt had valid yml.
On 22 Jul 2024 at 19:19 +0200, Mattias Persson @.***>, wrote:

Thanks, in js.log i see
CodeConfig.CxOcBhK1.js:1 Y {name: 'YAMLException', reason: 'can not read a block mapping entry; a multiline key may not be an implicit key', mark: {…}, message: 'can not read a block mapping entry; a multiline ke… 12 | ffff\n 13 | \n 14 | theme: godis\n-----------^', stack: 'YAMLException: can not read a block mapping entry;…p/immutable/chunks/CodeConfig.CxOcBhK1.js:1:5418)'}
I think you're trying to save invalid yaml in code editor
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Can you post your yaml like this
```yaml
theme: godis
```

Here it is. i've just installed it and am trying to get to grips with how it worked so added an entity to the yaml. There are no config errors reported in the editor.
views:
  - id:<PHONE_NUMBER>085
    type: configure
    name: Overview
  - name: Climate
    id:<PHONE_NUMBER>67
    sections:
      - type: horizontal-stack
        sections: []
        id:<PHONE_NUMBER>261
  - type: entity
    entity: binary_sensor.hue_secure_contact_sensor_1_opening
    icon: mdi:door
    name: Entrance
sidebar: []
theme: godis

Yeah, that's valid yaml but wrong config. There's no type entity and they need to be in a view. There's currently no validation, so if you're using code editor there's a chance it breaks. Use the ui and inspect the code later.

Thanks. Is there a way to add elements via the UI only? Or, can I edit the files in another editor and upload them? Do you have a sample file by any chance?

Is there a way to add elements via the UI only?

Yes, https://www.youtube.com/watch?v=D8mWruSuPOM

Or, can I edit the files in another editor and upload them?

Yeah, with ssh addon

Do you have a sample file by any chance?

https://github.com/matt8707/ha-fusion/blob/main/data/dashboard.yaml

Ok got it. Regarding the UI, how do I add an element like a sensor. Is it an object? If so, this is always greyed out.

Nvm figured it out. Thank you for the prompt replies.


Add sharedbackend blocking behaviour

the current backend makes use of tokio's block in place per db request: https://github.com/foundry-rs/foundry-fork-db/blob/fd94eb5876da5afab266156c71654c49a304e338/src/backend.rs#L637-L644
this is inefficient, especially if the value is already cached, but convenient if used inside tokio.
we can make this configurable via some enum that determines if it should use block_in_place or not.

I want to contribute to the project, can I get assigned to this issue?


[Feature Request] Provision for insertion of custom derived type in GetDerivedTypes in PolymorphicJsonConverter

Issue:
Currently during deserialization of polymorphic classes, PolymorphicJsonConverter looks for derived types in SDK level assembly only. This hinders conversion to other derived classes implemented by customer outside SDK assembly.
Requirement:
In one of our cases, customers have liberty to create their own derived classes from a generic class (part of SDK assembly) with discriminator field and send request to service. This will work fine during serialization phase but during deserialization object of generic class will be returned which cannot be type casted to derived class.
// assembly: microsoft.azure.management.recoveryservices.backup

// Discriminator field petType
public partial class Pet
{
  public string Name {get; set;}
  public string Color {get; set;}
}

public partial class Cat : Pet
{
  public bool IsCrookshanks {get; set;}
}

// In RecoveryServicesBackupClient.cs
DeserializationSettings.Converters.Add(new PolymorphicDeserializeJsonConverter<Pet>("petType"));

//==================================
// In client code outside sdk assembly

public partial class Mouse : Pet
{
  public bool IsPeterPettigrew {get; set;}
}

Here deserialization of {"name": "Tom", "petType": "Cat", "color": "grey", "isCrookshanks": false} of will return concrete object of type Cat but deserialization of {"name": "Jerry", "petType": "Mouse", "color": "grey", "isPeterPettigrew": false} will return object of type Pet which cannot be type casted to mouse.
Here on service side we are not interested in any Pet other than Cat but we are providing clients to add their own Pets if required.
Possible Solutions:

Currently autorest is looking for derived classes in the assembly of baseType. https://github.com/Azure/azure-sdk-for-net/blob/psSdkJson6/src/SdkCommon/ClientRuntime/ClientRuntime/Serialization/PolymorphicJsonConverter.cs#L33
This can be changed to look into all loaded assemblies in current app domain via AppDomain.CurrentDomain.GetAssemblies()
A static method GetCustomKnownTypes can be implemented in all models which will return a list of known types and list can be filled by client code outside sdk assembly. Then GetCustomKnownTypes can be invoked during GetDerivedTypes method using reflections and then deserialization can be handled normally.

@DheerendraRathor just so that I understand what you are proposing.
Are you saying, get list of all loaded assemblies, regardless of managed/native and then load each assembly and check for types that are in list?
Or is there a way to filter out assemblies before start fishing for custom types?

Hi. We're closing this issue as no response or updates have been provided in a timely. If you have more details and are encountering this issue please add a new reply and re-open the issue.


SyntaxError: Unexpected token, expected } for inline CSS block

I have a logo generated by Adobe Illustrator's SVG export plugin (supplied by our designers). I had transformed it to a React component by hand first, until I found this package. It doesn't seem to handle a style block though; I think the parser tries to interpret it as a react template construction.
Very minimal example, probably not valid SVG but it reproduces the parse error:
<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 21.0.2, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
<svg version="1.1" id="Layer_1">
    <style type="text/css">
      .st0{fill:#34107D;}
    </style>
</svg>

Error output:
$ svgr test.svg

(node:10071) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): SyntaxError: Unexpected token, expected } (5:14)
  3 | const Test = (props) => <svg id="Layer_1" {...props}>
  4 |   <style>
> 5 |     .st0{fill:#34107d}
    |              ^
  6 |   </style>
  7 | </svg>
  8 |
(node:10071) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

@fwielstra ah style is not supported, we could easily fix it by adding a h2x transform. Feel free to add a PR!

Hmm, looking into this a bit further, it might be better to disallow styles completely; react doesn't do CSS isolation (yet?), so two svg's with a style tag can overwrite each other's styles if they match in name (which for Adobe Illustrator's exporter they do apparently).

@fwielstra yes probably better to remove them completely.

It can be easily done by adding a h2x plugin, similar to removeComments, difference will be that we remove a whole tag "style".

Fixed in v0.5.0


Support where filters in classifications

Background
Weaviate allows a set three different where filters in classifications, but the client currently does not take any arguments.
Goals

[ ] .withSourceWhere() is added to the classification builder
[ ] .withTrainingSetWhere() is added to the classification builder
[ ] .withTargetWhere() is added to the classification builder

will not do this - classification is rarely used


Hari2

What does this PR do?
(Provide a description of what this PR does.)
Test Plan
(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your changes work.)
Related PRs and Issues
(If this PR is related to any other PR or resolves any issue or related to any issue link all related PR and issues here.)
Have you read the Contributing Guidelines on issues?
(Write your answer here.)

This basic JavaScript logic code which will ensure to check the password strength of passwords entered my the users in to any website!!


Framework definition also in other modules not just core-app

Framework definitions (stored in frameworks.csv file in migrations/framework), which go hand in hand with docker images, should be self-contained within the modules they are used in rather than only being defined in the core_app module.

Implemented in 6ea44be19396277bcae88547ffbf69a8980efaed


feat: explain a query and output a Graphviz representation of the query plan

In the case of a large complex query, a visualized representation can be clearer to diagnose problems, e.g. easier to know which two upstream are joined together if they are separated far away in the output of explain <SQL>.
https://github.com/risingwavelabs/risingwave/blob/main/src/frontend/src/scheduler/distributed/query.rs#L293-L294 output a task-level graphviz representation for debugging usage.
We can enable a query plan representation via the command explain (graphviz) <SQL>.

Related: https://github.com/risingwavelabs/risingwave/issues/14168

And also https://github.com/risingwavelabs/risingwave/issues/17218

Opening a tracking issue for these improvements to explain.

Currently we support structured output, so the major work is complete. We can support graphviz as well, via the DOT format.
This can be done in the following way:

Use serde to convert the IR into petgraph::Graph. https://docs.rs/petgraph/latest/petgraph/. Just enable serde-1 feature flag.
From petgraph::Graph, this can be converted to Dot format: https://docs.rs/petgraph/latest/petgraph/dot/struct.Dot.html.

Hi @kwannoel , I wound like to attempt on this issue - are we aiming to support logical plan and physical plan as dot format?

Hi @kwannoel , I wound like to attempt on this issue - are we aiming to support logical plan and physical plan as dot format?

Yes, via FORMAT DOT.

Use serde to convert the IR into petgraph::Graph. https://docs.rs/petgraph/latest/petgraph/. Just enable serde-1 feature flag.

looking at the code do we need to implement Deserialize for PrettySerde struct?

Yes, either implement Deserialize, or if you need more control over the way PrettySerde gets converted to petgraph::Graph, skip serde and manually convert it.

send a draft - one more q which syntax is better explain (format graphviz) or explain (graphviz)?

send a draft - one more q which syntax is better explain (format graphviz) or explain (graphviz)?

Missed this, replied here: https://github.com/risingwavelabs/risingwave/pull/19446#discussion_r1847798131. Thanks for your contribution!


Added link of Contribute Installation Guide Video for Ubuntu Linux User

Hi I made a Hyper™ contribute installation guide video for Ubuntu Linux User
Here is a link of video: https://www.youtube.com/watch?v=X_x3Genh78I
If you feel that video is unnecessary, feel free to just ignore this
Thanks.

hm...build failed? even I onley edited readme.md file?
Can anyone tell me why does that happened?


Input form example's source code doesn't work

Reproduction
Steps to reproduce:

Download the source code from https://stackblitz.com/angular/maxldbagbxe?file=app%2Finput-overview-example.ts
npm install, ng build, and ng serve --open
Open browser to observe that the HTML cannot generate the same style as displayed in the original example

Expected Behavior
The page shows the same style as the original example on https://material.angular.io/components/input/overview
Actual Behavior

This is what you actually see.
Environment

Angular: 8.0.3
CDK/Material: 8.0.1
Browser(s): Chrome
Operating System (e.g. Windows, macOS, Ubuntu): macOS

Looks like your app is missing a theme. See https://material.angular.io/guide/theming


Suggest stating specific whl package versions

[Enter feedback here]
I suggest you specify explicitly what package versions are required for Azure Synapse pools - such as Linux, 64 bit, etc.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4575464f-6077-3404-06ab-229c6d35670b
Version Independent ID: aaee2971-425a-2c1a-118a-503ca34fdc6d
Content: Manage Python libraries for Apache Spark - Azure Synapse Analytics
Content Source: articles/synapse-analytics/spark/apache-spark-manage-python-packages.md
Service: synapse-analytics
Sub-service: spark
GitHub Login: @midesa
Microsoft Alias: midesa

Thanks for reporting this. I have assigned the issue to the content author to investigate further and update the document as appropriate.

#reassign: @shuaijunye

@lon-tierney We have the following 3 docs that list the runtime information and the built-in libraries. Is that information sufficient? If not, can you please clarify the scenario you are trying to solve.
https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-32-runtime
https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-3-runtime
https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-24-runtime

@SnehaGunda while that information is helpful, we found that adding external WHL files often fail with an error stating that the package is unsupported on the platform. I believe (but have no proof) that is due to the packages being built for specific runtimes, such as Linux 32 bit or Linux 64 bit. It would be helpful on the referenced pages above if we listed the underlying platform that the Spark services run on, such as Linux 64 bit AMD or other as appropriate.

It’s on Linux 64 and maybe this doc can help: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-troubleshoot-library-errors#validate-wheel-files:~:text=Validate%20wheel%20files to help.

Yay! New docs! That likely would be helpful to the user who is manually managing packages (though I haven’t validated it works). If it’s not linked from the page in question here, please do so.
Get Outlook for iOShttps://aka.ms/o0ukef

From: Sneha Gunda @.>
Sent: Thursday, November 17, 2022 4:43:59 PM
To: MicrosoftDocs/azure-docs @.>
Cc: Lon Tierney @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Suggest stating specific whl package versions (Issue #93242)
It’s on Linux 64 and maybe this doc can help: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-troubleshoot-library-errors#validate-wheel-files:~:text=Validate wheel fileshttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fsynapse-analytics%2Fspark%2Fapache-spark-troubleshoot-library-errors%23validate-wheel-files%3A~%3Atext%3DValidate%2520wheel%2520files&data=05|01|Lon.Tierney%40microsoft.com|cf1f1a48dab048d4db6e08dac8fdfb74|72f988bf86f141af91ab2d7cd011db47|1|0|638043290423618151|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=idrAD7PVSNmTraqL31IClxqoARiu1mb6SUVoNLpVZDc%3D&reserved=0 to help.
—
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F93242%23issuecomment-1319403084&data=05|01|Lon.Tierney%40microsoft.com|cf1f1a48dab048d4db6e08dac8fdfb74|72f988bf86f141af91ab2d7cd011db47|1|0|638043290423618151|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=XVYZPuvIPDAvwSIy5h%2BmN5yNtzHxffOqqA6L75Z28fs%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAWKFP3JX3LEL2GQLK6T7FFTWI3GM7ANCNFSM5WQ4MOAQ&data=05|01|Lon.Tierney%40microsoft.com|cf1f1a48dab048d4db6e08dac8fdfb74|72f988bf86f141af91ab2d7cd011db47|1|0|638043290423618151|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=PGF16EkyVYIX5plae6L6n94A2%2Bd67cks4Na8EVfyIBg%3D&reserved=0.
You are receiving this because you were mentioned.Message ID: @.***>

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close


Add option to display optgroup caption for selected item

Rather than use the showSubtext option, add a new option showOptgroupCaption that will include the caption with the selected option.
This simplifies and adds clarity when an option is included in multiple groups.
For example:
Category 1
- Item 1
- Item 2
- Other
Category 2
- Item 3
- Item 4
- Other
Displaying other as the selected text is ambiguous.
Possible solution is to use the same sub text display to show the optgroup caption:
*Other* Category 2

For a workaround, see http://silviomoreto.github.io/bootstrap-select/#title

You can also use the title attribute as an alternative to display when the option is selected

So for Item 1, Category 1, you could add title="Item 1 (Category 1)"

Thanks.  I was able to get this working.
selectedTextFormat must be set to 'static'
Then setting the title and rendering worked properly:
$('.selectpicker')
   .selectpicker({ title: 'new title' })
   .selectpicker('render');


Unnecessary empty value on each mapping fields

Now UI generates mapping definition with empty value on all mapping field, no matter if it's constant or not. Runtime doesn't care, but it's confusing when a human looks at the mapping definition. We should remove value from all non-constant mapping fields. Although Constant is an exception, we might also remove constant value from mapping field due to #2706.
    "mappings" : {
      "mapping" : [ {
        "jsonType" : "io.atlasmap.v2.Mapping",
        "inputField" : [ {
          "jsonType" : "io.atlasmap.v2.ConstantField",
          "value" : "",
          "docId" : "DOC.Constants.760088",
          "path" : "/testv",
          "fieldType" : "STRING"
        } ],
        "outputField" : [ {
          "jsonType" : "io.atlasmap.json.v2.JsonField",
          "value" : "",
          "docId" : "JSONSchemaSource-50a01b0d-e834-4eec-9ad7-d71ddb117328",
          "path" : "/primitives/stringPrimitive",
          "fieldType" : "STRING",
          "name" : "stringPrimitive"
        } ],
        "id" : "mapping.167313"
      }, {
        "jsonType" : "io.atlasmap.v2.Mapping",
        "inputField" : [ {
          "jsonType" : "io.atlasmap.json.v2.JsonField",
          "value" : "",
          "docId" : "JSONSchemaSource-4a377e0b-7d77-4280-94be-30e2899dd57c",
          "path" : "/primitives/booleanPrimitive",
          "fieldType" : "BOOLEAN",
          "name" : "booleanPrimitive"
        } ],
        "outputField" : [ {
          "jsonType" : "io.atlasmap.json.v2.JsonField",
          "value" : "",
          "docId" : "JSONSchemaSource-50a01b0d-e834-4eec-9ad7-d71ddb117328",
          "path" : "/primitives/booleanPrimitive",
          "fieldType" : "BOOLEAN",
          "name" : "booleanPrimitive"
        } ],
        "id" : "mapping.544771"
      } ]
    },

Actually the runtime does care when preview mode is on.  The UI must pass an empty string (not null) which occurs when the user first enables preview mode (otherwise we get an unexpected null error from RT).  Also - React will issue a render error if value is referenced and is null or undefined.  So - the latest tweek to the UI serializer for #2706 seems to get as close as we can get.  When a user first establishes a mapping the value field is not defined.  If the user enables preview mode then an empty string is provided until they enter something.  If they toggle the preview mode button then they still retain the value.  I don't want to set the value field to undefined or null since it may trigger a React error (on render or on d&d).  We have minimal testing coverage (working on that...)

I mean runtime doesn't care if mapping field has value, it just ignores. The preview request is reusing same model, but it's a different object which does have field value. We might want to separate preview request model objects from mapping definition at design level if it's confusing.

correct - except for constants.  I think it's pulled apart ok now.

Corected via https://github.com/atlasmap/atlasmap/pull/2709


PostgreSQL xact lock in context manager produces warning

Hi! When I use following code, it produces warnings:
# `db` is my database connection
with create_sadlock(db, "my_key3", xact=True):
    pass

/my/env/python3.11/site-packages/sqlalchemy_dlock/lock/postgresql.py:182: RuntimeWarning: PostgreSQL transaction level advisory locks are held until the current transaction ends; there is no provision for manual release.

This is caused by __exit__ method of the lock calling close, which calls release. However, there's no release for transaction scoped (xact) PostgreSQL advisory lock. It could be fixed by overriding the close method in PostgresqlSadLock and checking for self._xact value.

commit 81b85d8 fixed the problem


docs: integrations reference updates

Added missed references; missed provider pages.

@eyurtsev Could you, please, check the Vercel error message? I don't have an access for it.

@ccurme Can you review, please?

Thank you for your help with this!


What is the "main.py" for this repo?

Hi Geoffroi,
Thanks for open-sourcing this code.
I am reproducing the results in your paper. However, I find this repo does not contain an entrance, like a main fun for the ray-tracing.
The "simulate aberrations.py" does not use the other ray-tracing functions.
Do I need to write my own main function to utilize the modules you present in the repo?
Thank you
Guangyuan

Hi Guangyuan,
Thank you for your interest. In the simulate_aberrations.py script, the ray-tracing operations take place when the RaytracedOptics model is called (OpticsSimulator.call -> RaytracedOptics.sample_optics_model -> RaytracedOptics.do_ray_tracing).
If you want to use ray tracing for a different purpose than the image simulation model proposed in this code, you will need to make some adjustments.
Hope that helps!
Geoffroi

Hi Guangyuan,
Thank you for your interest. In the simulate_aberrations.py script, the ray-tracing operations take place when the RaytracedOptics model is called (OpticsSimulator.call -> RaytracedOptics.sample_optics_model -> RaytracedOptics.do_ray_tracing).
If you want to use ray tracing for a different purpose than the image simulation model proposed in this code, you will need to make some adjustments.
Hope that helps! Geoffroi

Great! Thanks for guiding me. That helps!
Best,
Guangyuan


Memory leaks from boinc_zip()

Leaks from boinc_zip().
Referred from #20
Many leaks from the use of boinc_zip, albeit small ones.
If I add a 'return 1' directly after the very first call to boinc_zip() for unzipping the app zipfile then I get the first leak:
=================================================================
==1552==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 255 byte(s) in 1 object(s) allocated from:
    #0 0x7fd6a6b5ea06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x7fd6a698c6ee in boinc_zip(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*) (/home/glenn/github/boinc-install/lib/libboinc_zip.so.7+0x116ee)

SUMMARY: AddressSanitizer: 255 byte(s) leaked in 1 allocation(s).

255 bytes suggests a possible pointer being lost? Tried a couple of things but was unable to get around this leak in the controller code.
Am not sure if the problem is with the way the control code is calling the function boinc_zip() or with boinc_zip itself.

Have checked the control code and can see no obvious error in how the boinc_zip() is called. Am not going to chase possible errors in the boinc client code.


Client Side Image Enhancement

Client-side image enhancements that can be used to adjust the contrast within grayscale images.
previous tools "had a range slider that controlled dynamic range to cut out high-end or low-end of pixels.  Had 2 sliders to control min, max, and center of range."
<FeComponentTransfer> filter for SVG should help with this using a type='linear with a slope and intercept to adjust the low white and high black points.  With this linear slope you should be able to condense the the range of the white/black pixels.
This component could help: https://github.com/Kitware/UPennContrast/blob/master/src/components/ImageViewer.vue
This was designed for more complicated higher bit imagery but I believe the basics are in there for the remapping of the low and high points of an image.
See discussion notes: https://www.notion.so/Meeting-Notes-286fa21b125142e098c08006464f9780#178add01f7c74bb8ba7ed9d565b1b8da

initial testing of the system and need to decide on scoping of the issue.
https://user-images.githubusercontent.com/61746913/158190746-27a17a66-cd6a-464c-b9ea-70e0c938b84e.mp4

When talking about a "center of range" is this a third point that allows sliding left/right of the other two sliders (I.E. you drag the middle point left two pixels so both the left and right endpoints slide 2 pixels to the left)?
Do we want to support client side histogram generation so there is a visualization to use with these sliders (or would we consider this out-of-scope or now)?
Do we want to support sliders for each channel on RGB imagery or only support brightness (all channels)?

initial testing of the system and need to decide on scoping of the issue.
testContrast.mp4

When talking about a "center of range" is this a third point that allows sliding left/right of the other two sliders (I.E. you drag the middle point left two pixels so both the left and right endpoints slide 2 pixels to the left)?

No need for this, unless someone asks for it specifically. My intuition is that this operation of moving the entire range is not usually useful; in the vast majority of cases, people will want to manipulate each end of the range in isolation in order to investigate how each affects the image.

Do we want to support client side histogram generation so there is a visualization to use with these sliders (or would we consider this out-of-scope or now)?

Out-of-scope for a first attempt.

Do we want to support sliders for each channel on RGB imagery or only support brightness (all channels)?

Don't worry about RGB. The right way to handle color images is likely not to include three separate sliders, so let's wait until there's a need to deal with color images.

This looks awesome.  Agree with Roni's comments.
There will be conflicts with #1188 because I had to change how the sidebar context worked to enable web to register a sidebar component not in dive-common.   It should be minor to resolve, but just FYI.

Additional quick question.  I'm guessing this should be stored in the meta data for the dataset like the typeStyling?
Something we want persisted on the dataset between different users and set by the owner of the dataset?

I know we talked about this on Tuesday, but just for the record: we decided that a first pass could avoid persisting this setting altogether, and in a future iteration we can gather requirements to see what users actually need of persistence.


Problems panel vs problems status

Type: Bug

I have one file open. The problems side panel reports 22 info-level errors from the cSpell extension. But the problems status bar also reports 4 warnings. Why this inconsistency?
VS Code version: Code 1.70.1 (6d9b74a70ca9c7733b29f0456fd8195364076dda, 2022-08-10T06:08:33.642Z)
OS version: Windows_NT x64 10.0.22000
Modes:

System Info

Item
Value

CPUs
AMD Ryzen 7 3750H with Radeon Vega Mobile Gfx   (8 x 2296)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
15.45GB (5.34GB free)

Process Argv
--crash-reporter-id f4d11365-b8df-4076-a7d2-5647e124f85c

Screen Reader
no

VM
0%

Extensions (51)

Extension
Author (truncated)
Version

overtype
ada
0.2.0

Bookmarks
ale
13.3.1

rtf
ale
2.5.1

icons-carbon
ant
0.2.5

vscode-open-file-folder
auc
0.0.4

insert-unicode
bru
0.13.3

markdown-toc
Cha
1.5.6

reverse-search
Dan
2.0.2

vscode-markdownlint
Dav
0.47.0

vscode-dotnet-adapter
der
1.4.3

xml
Dot
2.5.1

vsc-material-theme-icons
equ
2.3.1

prettier-vscode
esb
9.5.0

font-switcher
eva
4.1.0

macros
ged
1.2.1

todo-tree
Gru
0.0.215

vscode-test-explorer
hbe
2.21.1

docomment
k--
0.1.31

markdown-link-updater
mat
2.2.0

markdown-shortcuts
mdi
0.12.0

render-crlf
med
1.6.0

fluent-icons
mig
0.0.18

ecdc
mit
1.8.0

HTMLHint
mka
0.10.0

csharp
ms-
1.25.0

vscode-dotnet-runtime
ms-
1.5.0

powershell
ms-
2022.7.2

test-adapter-converter
ms-
0.1.6

fix-json
oli
0.1.2

material-icon-theme
PKi
4.19.0

material-product-icons
PKi
1.4.0

word-counter
plu
0.0.1

toggle
reb
0.0.2

vscode-sort-json
ric
1.20.0

block-travel
sas
0.0.2

MDTools
sea
1.0.1

vscode-table-formatter
shu
1.2.1

vscode-fileutils
sle
3.5.0

vscode-autohotkey
sle
0.2.2

character-count
ste
0.1.1

code-spell-checker
str
2.5.4

code-spell-checker-french
str
0.2.2

code-spell-checker-german
str
2.0.4

code-spell-checker-greek
str
0.1.13

code-spell-checker-italian
str
0.1.13

code-spell-checker-spanish
str
2.1.2

html-preview-vscode
tht
0.2.5

duplicate-finder
tle
0.0.6

vscode-w3cvalidation
Umo
2.9.1

vscode-icons
vsc
11.15.0

markdown-all-in-one
yzh
3.4.3

(1 theme extensions excluded)

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263cf:30335440
vscoreces:30445986
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411cf:30542925
vsaa593cf:30376535
pythonvs932:30410667
wslgetstarted:30449410
cppdebug:30492333
vscaac:30438847
pylanb8912:30545647
vsclangdf:30486550
c4g48928:30535728
d7fjc500:30543455
dsvsc012cf:30540253
azure-dev_surveyone:30545435

I think this is an extension specific issue, the extension you are using code-spell-checker, isn't built in. So I think this issue may be made there.

I don't think so. Note that both the panel and the status report 22 info problems. Those are the spelling errors. But only the status bar reports 4 warnings. I have no idea what those are. Even with no files open I see warnings in the status bar and none in the panel.

Does anything happen if you click on the filter button at the right-hand end of the filter field of the Problems view?

I have everything enabled. And I hit "Clear filters" nothing changes.

Not sure what "Clear filters" you are mentioning, but this button indicates a filter is enabled.

Clear filters: you only see it when the active filters are hiding some reports:

The reason the filter is enabled is because I have "Show active file only" checked:

However, turning that off, which is what "Clear filters" does, will not change the issue.
And now that I'm playing around with this more I think the issue is that problems are not updating reliably. For example, I opened editors on about 30 files. In the panel only 5 of them are showing up with 12 warnings and 373 infos. Matches the status bar counts.
Now if I activate one of the 25 other files, that file suddenly appears in the problem panel too, and the problems in that file are added to the totals in both panel and status bar.
Now if close all editors, I still see 3 files listed in problems and the counts in the panel and status bar match.
So, from having 30 files open to having 0 files open:

If it was only counting active files (which I disabled for this experiment) nothing should have changed. (In fact, with no filters, shouldn't it show all files in the project?)
If it was only showing recently edited files (e.g. in the current session), shouldn't the problems panel and status bar remain unchanged?
If the active file filter was enabled and no files are open shouldn't there be no files in panel and zero counts in panel and status bar?

Problems are contributed by extensions, and extensions are responsible for removing their contributed problems when they should no longer be there.
Please run the Start Extension Bisect command, then follow its notification instructions, at each step repeating your tests. This procedure might identify an extension that is malfunctioning with regard to how it handles Diagnostics (which is what the API calls the entities).

Start Extension Bisect command

I was having a similar issue and I was able to identify the offending extension with the process you recommended. Thank you!

Clearing the filters shall show all problems including warnings. Is it possible to provide a sample repro using which I can reproduce this?

Closing this issue as there is not enough information provided. Please reopen it with requested info. Thanks.


Bring docs agda version in sync with .cabal file

Docs say you need 2.6.3, cabal file says you need 2.6.4.
https://github.com/agda/agda2hs/blob/b269164e15da03b74cf43b51c522f4f052b4af80/agda2hs.cabal#L49-L50
(the code above is from v1.2 tag, but github refuses to render it if it's not a permalink)

Thanks!


Halving the recording sampling rate doesn't seem to double the max length

Both these hex files record sound until the memory is full, one configured at 5.5K and the other at 11K, but the both only record less than 5 seconds of audio.
5500.hex.zip
11000.hex.zip
It's possible I'm not configuring the pipeline correctly, is this code correct?
#include "MicroBit.h"
#include "StreamRecording.h"

MicroBit uBit;

static const int SAMPLE_RATE = 11000;

static void onButtonLogo(MicroBitEvent) {
    static SplitterChannel *splitterChannel = uBit.audio.splitter->createChannel();
    splitterChannel->requestSampleRate(SAMPLE_RATE);
    static StreamRecording *recording = new StreamRecording(*splitterChannel);
    static MixerChannel *channel = uBit.audio.mixer.addChannel(*recording, SAMPLE_RATE);

    MicroBitAudio::requestActivation();
    channel->setVolume(75.0);
    uBit.audio.mixer.setVolume(1023);

    uBit.display.clear();
    uBit.audio.levelSPL->setUnit(LEVEL_DETECTOR_SPL_8BIT);

    recording->record();
    bool showR = true;
    while (uBit.logo.isPressed() && recording->isRecording()) {
        if (showR)
            uBit.display.print("R");
        else
            uBit.display.clear();
        uBit.sleep(150);
    }
    recording->stop();

    recording->play();
    while (recording->isPlaying()) {
        uBit.sleep(20);
    }
    recording->erase();
}

int main() {
    uBit.init();

    uBit.messageBus.listen(MICROBIT_ID_LOGO, MICROBIT_BUTTON_EVT_DOWN, onButtonLogo);

    while (true) {
        uBit.sleep(100);
    }
}

The code here is correct, but the current implementation of StreamRecording simply collects upto REC_MAX_BUFFERS in an array, rather than dynamically tracking blocks until it hits a specific memory limit.
I'll be updating this shortly so that rather than a buffer limit, we use a soft memory limit.

As a note to myself:

[ ] REC_MAX_BUFFERS should be replaced with a suitable CODAL_* config var
[ ] Add event for reaching the soft limit
[ ] Add default'd constructor arg to allow users to specify a custom limit per instance up to CODAL_* .. max

Ah, thanks John! I mistakenly thought it allocated buffers until there wasn't more memory available.

I've tested this with the latest commit 02a85fa86a00ac204cbd2a4caacd5a964e1cf26e, and reducing the sampling rate in the example doesn't increase the recording time. Going from 11_000 to 1_100 decreases the sound quality but still has around 5 seconds of audio.

I wasn't using "dev": true in the codal.json file, so I wasn't using the latest commits from codal-core.
With the latest I can confirm this now works 🎉


Require JSX closing tag to be aligned with the opening tag (react/jsx-closing-tag-location)

This rule checks all JSX multiline elements with children (non-self-closing) and verifies the location of the closing tag. The expectation is that the closing tag is aligned with the opening tag on its own line.
https://github.com/yannickcr/eslint-plugin-react/blob/master/docs/rules/jsx-closing-tag-location.md
The following patterns are considered warnings:
<Hello>
  marklar
  </Hello>

<Hello>
  marklar</Hello>

The following are not considered warnings:
<Hello>
  marklar
</Hello>

<Hello>marklar</Hello>

I know this is not controversial so I am going to ship it in standard 14.

@feross Doesn't this collide with indent/jsx-indent rule?

@MatanBobi I don't believe so. Can you explain how?

@feross I think that @MatanBobi means using non-self-closing tag with return:
  render () {
    return

      Closing tag below where should be?

  }

With this rule, this style raises an error (not aligned to open tag) and if I try to --fix it:
  render () {
    return

      Closing tag below where should be?

  }

which raises react/jsx-indent-problem and is ugly.
The only way to fix it is putting all in one line:
  render () {
    return
Closing tag below where should be?

  }

but is not really sustainable.

@joke2k The intended style is to add parens anytime you have a multiline JSX statement.
  render () {
    return (

        Closing tag below where should be?

    )
  }

I'm not sure why the autofixer isn't doing that for you. Looks like we need to play with the JSX rules some more to get what we want.

Re-opening so we can investigate fixing this for standard 15. Help wanted!

@joke2k You got exactly what I meant, thanks for the wide explanation.

@joke2k The intended style is to add parens anytime you have a multiline JSX statement.

This doesn't seem to match the jsx-closing-tag-location documentation. As you acknowledge in your first post, it says:

The following are not considered warnings:
<Hello>
  marklar
</Hello>

No parentheses are shown.

I believe this is fixed in standard 16.


Large genetic association browser region ?

From https://qa.targetvalidation.org/evidence/ENSG00000142192/EFO_0000249 for gene APP ENSG00000142192 18.12 production says it is at chr21:26,362,203−2,582,234 but 19.02 qa says chr21:117,628,648−11,026,049. https://www.ensembl.org/Homo_sapiens/Gene/Summary?db=core;g=ENSG00000142192;r=21:25880550-26171128 says chr21:25,880,550-26,171,128
This might have been an transient error as I can currently not reproduce it.
In theory, it might have come from the SNP2GENE mapping process which will be replaced soon with genetics platform.

Seems to no longer be an issue, closing.


An admin should be able to delete a user

Coverage increased (+0.5%) to 84.921% when pulling 7eb8a572d5f129fc3ebce296cc3eee1ab6a19cd1 on feature/delete-users into e5385ed874ef6be75acbf98ecdc853664b73ba01 on master.


Created modules: orienteer-notifications, orienteer-twilio

Created modules:

orienteer-notifications
orienteer-twilio

Close this PR. I added required changes. Will create new PR


Zod to JSONSchema conversion doesn't use openai custom logic for openai

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
OpenAI's new structured output support requires compiling zod to JSONSchema, which is something that LangChain already knows how to do. However -- OpenAI's code client uses a specialized, forked version of zod-to-json-schema internally to do this conversion in order to improve the compatibility of the generated JSON schemas with what is supported server side in their API. Specifically, they don't use relative references, which LangChain does, because they aren't supported server side. OpenAI's client also handles nullability differently.
You can see their custom converter here: https://github.com/openai/openai-node/blob/cc13af9fa7e76e774d3132bba2427bb0176bd622/src/helpers/zod.ts#L11-L19
and some commits that added the custom logic:  https://github.com/openai/openai-node/commit/fc22483173b521b0b194264e6d975c3517c06216 and https://github.com/openai/openai-node/commit/fc22483173b521b0b194264e6d975c3517c06216
As it stands now, LangChain's conversation of tool or structured output schemas doesn't use this fancy logic, and breaks in many more cases. If you re-use a chunk of schema within a bigger, outer schema, LangChain triggers an OpenAI 400 error, and the OpenAI client doesn't.
It'd be great if LangChain could be a client of OpenAI's zodResponseFormat or zodFunction helpers instead of doing its own conversion, or something like that.
Error Message and Stack Trace (if applicable)
No response
Description
I'm trying to use LangChain to a call a tool with the same inner bit of Zod schema repeated more than once, and it fails with OpenAI's new strict: true parameter when OpenAI's client doesn't.
System Info
@langchain/core 0.2.21
openai 4.55.3
node v22.2.0
pnpm v8.12.0

Hey, we're implementing this in #6438

Great! That looks like it addresses the response format for top-level structured output, but there's the same problem for function calling too for the function's input schema. Is that on y'alls radar as well?

Ah interested, I wasn't aware they made other changes to how they handle schemas for normal tool calling. This is now on my radar and I'll get it in soon (likely sometime next week). Thank you!

Can we keep this open until tool calling supports the feature as well? For me at least that's the important one because it allows selection among multiple tools!

ping @bracesproul sorry to bother but I do think there's a missing bit for proper support for strict OpenAI functions using their zod helpers for their special handling!

@airhorns what is the missing feature you're referring to?

OpenAI uses a special zod-to-json-schema converter for both top-level model structured outputs as well as tool calling. AFAICT, Langchain now uses OpenAI's zodResponseFormat to do this conversion for top level .withStructuredOutput calls, but it doesn't use OpenAI's zodFunction for tool calls. That breaks for various tools in strict mode because langchain's internal zod-to-json-schema behaves differently and emits JSON schemas that aren't compatible with OpenAI.

@airhorns we are using OpenAIs zodFunction util to convert tools passed in if they contain a Zod schema. See the _convertToOpenAITool function which is called from the _convertChatOpenAIToolTypeToOpenAITool function.
Could you provide me with some code to reproduce the error you're referencing?

Ah, sorry I missed that, I only saw the other PR! Never mind me then, thanks!


Add record count and timestamp to Avro compaction output files

Added logic to MRCompactorAvroKeyDedupJobRunner to rename the outputted files to include a timestamp and record count.

@zliu41 Can you review?
Tested on Azkaban for daily and hourly compaction

Please add license headers to new files.

Headers added, name updated.

Whoops - meant to cast to an int first. Changed to use Random.nextInt()

LGTM.


EffectGameObject broken offset

PulseHSVMode should have BYTE pad[3]; after it

Note to self: double check offsets

false alarm however, still important to keep in mind


About convert to vmd file

Thank you very much for sharing your wonderful project.  I am doing fine until the last step.  It gives me this error message: "usage: pos2vmd_multi.py [-h] [-t TARGET] [-b BONE] [-v VERBOSE] [-u UPRIGHT]                        [-c CENTERXY] [-z CENTERZ] [-x XANGLE]                        [-d DDECIMATION] [-m MDECIMATION] [-i IDECIMATION]                        [-a ALIGNMENT] [-k LEGIK] [-e HEELPOS]pos2vmd_multi.py: error: unrecognized arguments: -s 1". If I remove the s parameter in the last line of the bat file, it will execute and give me a vmd file but the result is not good.  I was wondering if you could point out what possibly going wrong with my passing the s factor.  Thanks!

最后一步的bat里一般是默认赋值的，我对mmd的配置研究不多，不过产生vmd的这一步确实是可以自行做不少调整的。

Are you guys still maintaining the scripts? This error still exists.

Are you guys still maintaining the scripts? This error still exists.

In the 3DToVmd.bat file, you can try deleting the '-s %SMOOTH_TIMES%' at the last line of script. The error should be solved.

Thank you very much for sharing your wonderful project. I am doing fine until the last step. It gives me this error message: "usage: pos2vmd_multi.py [-h] [-t TARGET] [-b BONE] [-v VERBOSE] [-u UPRIGHT] [-c CENTERXY] [-z CENTERZ] [-x XANGLE] [-d DDECIMATION] [-m MDECIMATION] [-i IDECIMATION] [-a ALIGNMENT] [-k LEGIK] [-e HEELPOS]pos2vmd_multi.py: error: unrecognized arguments: -s 1". If I remove the s parameter in the last line of the bat file, it will execute and give me a vmd file but the result is not good. I was wondering if you could point out what possibly going wrong with my passing the s factor. Thanks!

rename the csv file to english or number, then you are good to go