module/sched: Sched domain flag changes

That will fail if for any reason another instance is used

Seeing how flags is rebuilt, there's no reason for that to happen.

"SD_ASYM_CPUCAPACITY" in domain.flags_str

The reason for having those flags defined in a class dict is to get autocompletion (I only really know 2 of those flags), and you get the description if you're unsure as to what the flag is doing. To me that makes more sense to check for inclusion using those pre-defined objects rather than pure strings.

If for example you serialize then deserialize an this domain object, this will not work anymore:
>>> SchedDomain.SD_FLAGS["SD_ASYM_CPUCAPACITY"] in domain.flags

This is quite dangerous as it would fail silently. The string approach fixes that without having to introduce real singletons. If real singletons are needed, it seems a better idea to rely on the enum34 module that should provide a correct implementation of it.

Hmm so I don't know how enum34 would work for both Python2 & Python3.
Otherwise I could change the objectified domain.flags to be a dict(SchedDomainFlag), so we can test by keystring and still have access to a full-fledged object.

There is also an enum implementation in WA (which could be moved into devlib) that could be use for this:
https://github.com/ARM-software/workload-automation/blob/master/tests/test_utils.py#L98
This will work on Python 2 and 3.

@setrofim I'll have a look, thanks for the pointer.

Looking at WA's enums, it looks like we would have to somehow implement a reduce_ex method for it to make it possible to deserialize instances of the classes Enum created on the fly. Alternatively, it is possible to make the classes created with enum() available under a stable name, registered as the classe's __name__/__qualname__ and __module__, like what collections.namedtuple provide.
The singleton issue would be addressed by using equality comparison, since it WA's level implements __eq__ and __hash__.

@douglas-raillard-arm I went the enum34 way

@douglas-raillard-arm  Are you happy with the changes to the PR?

LGTM


Questions about OWL-ViT TFLite conversion

Hello!
I'm interested in trying a TFLite version of OWL-ViT for zero-shot object detection.
I've tried the Google Colab notebook for TFlite conversion but I'm facing two problems.
Problem 1
The "end-to-end" tflite file is about 1.5GB, while the pb file is only 60MB.
Why is the TFlite model much larger than the Tensorflow one? I used to think that tflite models are lighter than their Tensorflow version.
I tried adding the option converter.optimizations = [tf.lite.Optimize.DEFAULT] during conversion, but the resulting model file was still about 1GB.
Problem 2
Three weeks ago, I successfully converted "embed_images", "predict_boxes" and "predict_logits" modules to TFLite by setting MAX_NUM_QUERIES to 1 (when using the default MAX_NUM_QUERIES = 100, my session kept crashing due to a lack of memory).
However, today my session crashed when trying to convert the "embed_images" module regardless of the MAX_NUM_QUERIES value.
Is Google Colab Pro required to successfully convert OWL-ViT to TFLite?
I think the two problems are related at some point, but I'm not sure about what to investigate.
I don't think it's related to the problems above, but  I appended !sed -i '/    "tf-models-official",/d' ./setup.py after cloning scenic. Without this command, pip keeps trying to solve package dependencies when installing this repository.
Does someone have a hint about what's going wrong?
Let me know if you need more details.
Jordan

Hi Jordan,
Here are some thoughts on the issues you mention:
Problem 1: It looks to me like the tflite file includes the model parameters, but the pb file does not. The parameters are on the order of 1GB depending on the architecture. This will be hard to get around without reducing the architecture size. Quantization of model parameters might be a way to save space but we haven't tried that.
Problem 2: I have noticed that the public CPU runtime has pretty limited RAM. Perhaps try with the GPU or TPU runtimes, which may have more RAM. You can also run a Jupyter Notebook kernel locally on your machine and connect Colab to that (instructions), which may help if your machine has enough RAM.
tf-models-official: We're aware of the dependency conflicts and will remove tf-models-official from setup.py soon. As you say, this is probably not related to the other issues.
Hope this helps!

Hi Matthias,
Thank you very much for your support!
Nothing is decided yet, but there is a chance that my team investigates post-training quantization of OWL-ViT.
I tried converting OWL-ViT to TFLite on GPU and TPU runtimes but the sessions crashed too.
I'll give it a try on a local machine by following the instructions you provided.
For the moment, I think this issue can be closed.
Jordan

Sounds good. Please let me know how the quantization goes!


Google Play Services download corrupt

Been trying to get Google Play Services installed but the download reaches ~40% of the 105mb file, the download bar skips to 100% and attempts to install the package. Failure message is: "app is corrupt, cannot be installed." Have tried reinstalling Apps apk, uninstalling Play Store and Google Services Framework to no avail.

The error message is correct. You already have an app with a conflicting app id installed. You cannot install another variant of an app with a different signing key or a lower version even in another profile. The signing key is pinned globally and the version needs to be equal or greater. You likely have a newer version of Play services installed already. In the release notes for the release bundling our app repository client, you can see that we blocked the Play Store from updating Play services and the Play Store to prevent users ending up with a newer version than what's in our repository. The issue is primarily due to them taking literally weeks to roll out new versions.
This isn't an issue with GrapheneOS or the app repository client. It's nothing specific to them or to Google Play. It's part of what the package manager on Android enforces for apps in general.

Please ask for support in our main Matrix room if you need help resolving this beyond that.


Fix neuron cores auto-scaling

When trying to use 2 workers with for each one neuron core on inf2.xlarge with tensor_parallel_degree = 1, I can see the same core (nc0) is required by the 2 workers with:
Set NEURON_RT_VISIBLE_CORES=0
Causing an error as the same core is requested by the second worker.
The provided code should fix that.

I didn't see the issue you describe in nightly image.
Can you provide a serving.properties that can reproduce your issue?

Example serving.properties
engine = Python
option.entryPoint=djl_python.transformers-neuronx
option.entryPoint=djl_python.stable_diffusion_inf2
option.use_stable_diffusion = True
option.model_id=s3://sagemaker-example-files-prod-us-east-1/models/hf-large-model-djl-ds/neuron/stablediffusion21/
option.tensor_parallel_degree = 1
option.dtype=bf16
option.enable_venv=true
minWorkers=2
maxWorkers=2
env=NEURON_RT_NUM_CORES=1
Log extract:
for the 2 workers:
[INFO ] Connection - Set NEURON_RT_VISIBLE_CORES=0-0
Exception:
ERROR   NRT:nrt_allocate_neuron_cores               NeuronCore(s) not available - Requested:nc0-nc0 Available:0

@fafriat
You cannot use 2 workers for inf2 when using Python engine. This the limitation of the neuron driver.

this documentation seems to say the contrary:
https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-runtime/nrt-configurable-parameters.html#shared-weights-neuron-rt-multi-instance-shared-weights
inf2.xlarge has 2 cores nc0 and nc1 that may be allocated to different process using NEURON_RT_NUM_CORES property : A value from 1 to Max NeuronCore in the system.

@fafriat
DJLServing will automatically load the model onto 2 cores.
If you specific 2 workers, it actually trying to launch 4 process.

If you only want to load on on core, you should do the following:
load_on_devices=nc0

load_on_devices=nc0

ok but how to get 2 workers, one loaded nc0 the other one loaded on nc1 ?

load_on_devices=nc0;nc1

@frankfliu Adding load_on_devices=nc0;nc1 does not change anything as I get for both workers:
Set NEURON_RT_VISIBLE_CORES=0-0
which works for the first worker loading on nc0 but fails for the second one trying to load the model on the same core nc0 instead of nc1 !
This is exactly why the provided pull request is necessary.
You can try anything but while the NEURON_RT_VISIBLE_CORES is not depending on the workerId you cannot get auto-scaling  for neuron cores.
Remarks that the provided code was similar of the initial code for handling autoscaling except the fact that instead of passing the deviceId we need to pass the workerId:
https://github.com/deepjavalibrary/djl-serving/commit/5a8fed73525ef554e68f811b268d6de6613942f5
Please consider merging this pull request as without this fix, there is no autoscaling for neuron cores with DJL Serving which is blocking our project.

@fafriat Can you share your full log?
I don't have this issue with latest DJLServing. Here is my serving.properties:
engine=Python
option.model_id=s3://djl-llm/stable-diffusion-2-1-base-compiled/
option.tensor_parallel_degree=2
option.use_stable_diffusion=true

Docker run command:
docker run -it --rm --network=host -v $PWD/source:/workspace     --device /dev/neuron1:/dev/neuron0     --device /dev/neuron3:/dev/neuron1     --device /dev/neuron4:/dev/neuron2     --device /dev/neuron5:/dev/neuron3     deepjavalibrary/djl-serving:pytorch-inf2-nightly bash

And logs:
INFO  ModelServer Initializing model: /workspace/models/inf2
INFO  ModelInfo Apply per model settings:
	job_queue_size: 1000
	batch_size: 1
	max_batch_delay: 100
	max_idle_time: 60
	load_on_devices: *
	engine: Python
	option.entryPoint: null
	option.tensor_parallel_degree: 2
	option.use_stable_diffusion: true
	option.model_id: s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo Download completed! Files saved to /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  Platform Found matching platform from: jar:file:/usr/local/djl-serving-0.24.0-SNAPSHOT/lib/python-0.24.0-SNAPSHOT.jar!/native/lib/python.properties
INFO  ModelManager Loading model on Python:[**nc0, nc2, nc4, nc6**]
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  WorkerPool loading model inf2 (PENDING) on nc(0) ...
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  ModelInfo Available CPU memory: 373965 MB, required: 0 MB, reserved: 500 MB
INFO  **ModelInfo Loading model inf2 on nc(0)**
INFO  WorkerPool scale up 1 workers (1 - 1)
INFO  PyProcess Start process: 19000 - retry: 0
INFO  Connection **Set NEURON_RT_VISIBLE_CORES=0-1**
INFO  PyProcess W-790-inf2-stdout: 790 - djl_python_engine started with args: ['--sock-type', 'unix', '--sock-name', '/tmp/djl_sock.19000', '--model-dir', '/workspace/models/inf2', '--entry-point', 'djl_python.transformers_neuronx', '--device-id', '0']
INFO  PyProcess W-790-inf2-stdout: Letting libtpu.so load fail during _XLAC import. libtpu.so will be loaded from `libtpu` Python package when the ComputationClient is created.
INFO  PyProcess W-790-inf2-stdout: Python engine started.
INFO  PyProcess W-790-inf2-stdout: Loading pre-compiled model
INFO  PyProcess W-790-inf2-stdout: Dynamic batching is disabled for torch_neuronx.DataParallel
INFO  PyProcess Model [inf2] initialized.
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  WorkerPool loading model inf2 (READY) on **nc(2)** ...
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  ModelInfo Available CPU memory: 370365 MB, required: 0 MB, reserved: 500 MB
INFO  ModelInfo Loading model inf2 on nc(2)
INFO  WorkerPool scale up 1 workers (1 - 1)
INFO  PyProcess Start process: 19001 - retry: 0
INFO  Connection Set **NEURON_RT_VISIBLE_CORES=2-3**
INFO  PyProcess W-802-inf2-stdout: 802 - djl_python_engine started with args: ['--sock-type', 'unix', '--sock-name', '/tmp/djl_sock.19001', '--model-dir', '/workspace/models/inf2', '--entry-point', 'djl_python.transformers_neuronx', '--device-id', '2']
INFO  PyProcess W-802-inf2-stdout: Letting libtpu.so load fail during _XLAC import. libtpu.so will be loaded from `libtpu` Python package when the ComputationClient is created.
INFO  PyProcess W-802-inf2-stdout: Python engine started.
INFO  PyProcess W-802-inf2-stdout: Loading pre-compiled model
INFO  PyProcess W-802-inf2-stdout: Dynamic batching is disabled for torch_neuronx.DataParallel
INFO  PyProcess Model [inf2] initialized.
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  WorkerPool loading model inf2 (READY) on nc(4) ...
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  ModelInfo Available CPU memory: 366816 MB, required: 0 MB, reserved: 500 MB
INFO  ModelInfo Loading model inf2 on nc(4)
INFO  WorkerPool scale up 1 workers (1 - 1)
INFO  PyProcess Start process: 19002 - retry: 0
INFO  Connection Set NEURON_RT_VISIBLE_CORES=4-5
INFO  PyProcess W-813-inf2-stdout: 813 - djl_python_engine started with args: ['--sock-type', 'unix', '--sock-name', '/tmp/djl_sock.19002', '--model-dir', '/workspace/models/inf2', '--entry-point', 'djl_python.transformers_neuronx', '--device-id', '4']
INFO  PyProcess W-813-inf2-stdout: Letting libtpu.so load fail during _XLAC import. libtpu.so will be loaded from `libtpu` Python package when the ComputationClient is created.
INFO  PyProcess W-813-inf2-stdout: Python engine started.
INFO  PyProcess W-813-inf2-stdout: Loading pre-compiled model
INFO  PyProcess W-813-inf2-stdout: Dynamic batching is disabled for torch_neuronx.DataParallel
INFO  PyProcess Model [inf2] initialized.
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  WorkerPool loading model inf2 (READY) on nc(6) ...
INFO  ModelInfo S3 url found, start downloading from s3://djl-llm/stable-diffusion-2-1-base-compiled/
INFO  ModelInfo artifacts has been downloaded already: /tmp/.djl.ai/download/a406971cce6d796dceb1a3f6958c9900464be86b
INFO  ModelInfo Available CPU memory: 363320 MB, required: 0 MB, reserved: 500 MB
INFO  ModelInfo Loading model inf2 on nc(6)
INFO  WorkerPool scale up 1 workers (1 - 1)
INFO  PyProcess Start process: 19003 - retry: 0
INFO  Connection Set NEURON_RT_VISIBLE_CORES=6-7
INFO  PyProcess W-829-inf2-stdout: 829 - djl_python_engine started with args: ['--sock-type', 'unix', '--sock-name', '/tmp/djl_sock.19003', '--model-dir', '/workspace/models/inf2', '--entry-point', 'djl_python.transformers_neuronx', '--device-id', '6']
INFO  PyProcess W-829-inf2-stdout: Letting libtpu.so load fail during _XLAC import. libtpu.so will be loaded from `libtpu` Python package when the ComputationClient is created.
INFO  PyProcess W-829-inf2-stdout: Python engine started.
INFO  PyProcess W-829-inf2-stdout: Loading pre-compiled model
INFO  PyProcess W-829-inf2-stdout: Dynamic batching is disabled for torch_neuronx.DataParallel
INFO  PyProcess Model [inf2] initialized.
INFO  ModelServer Initialize BOTH server with: EpollServerSocketChannel.
INFO  ModelServer BOTH API bind to: http://<IP_ADDRESS>:8080

In my case, the difference I see is that I would like to get 1 neuron core per worker not 2.
So this is my complete serving.properties:
engine = Python
option.entryPoint=djl_python.transformers-neuronx
option.entryPoint=djl_python.stable_diffusion_inf2
option.use_stable_diffusion = True
option.model_id=s3://imagineai-models/sd21_512_b1_bf16_djl_neuronx
option.tensor_parallel_degree = 1
load_on_devices=nc0;nc1
option.dtype=bf16
option.enable_venv=true
minWorkers=2
maxWorkers=2
env=NEURON_RT_NUM_CORES=1
There are the logs :
ERROR StatusConsoleListener Unable to locate appender "server_metric" for logger config "server_metric"
ERROR StatusConsoleListener Unable to locate appender "model_metric" for logger config "model_metric"
[INFO ] ModelServer - Starting model server ...
[INFO ] Ec2Utils - DJL will collect telemetry to help us better understand our users? needs, diagnose issues, and deliver additional features. If you would like to learn more or opt-out please go to: https://docs.djl.ai/docs/telemetry.html for more information.
[INFO ] ModelServer - Starting djl-serving: 0.23.0 ...
[INFO ] ModelServer -
Model server home: /opt/djl
Current directory: /opt/djl
Temp directory: /tmp
Command line: -Dlog4j.configurationFile=/usr/local/djl-serving-0.23.0/conf/log4j2-plain.xml -Xmx1g -Xms1g -Xss2m -XX:+ExitOnOutOfMemoryError -XX:-UseContainerSupport
Number of CPUs: 4
Number of Neuron cores: 2
Max heap size: 1024
Config file: /opt/djl/conf/config.properties
Inference address: http://<IP_ADDRESS>:8080
Management address: http://<IP_ADDRESS>:8080
Default job_queue_size: 1000
Default batch_size: 1
Default max_batch_delay: 100
Default max_idle_time: 60
Model Store: /opt/ml/model
Initial Models: ALL
Netty threads: 0
Maximum Request Size: 67108864
Environment variables:
#011DJL_CACHE_DIR: /tmp/.djl.ai
#011SAGEMAKER_SAFE_PORT_RANGE: 22000-22999
[INFO ] FolderScanPluginManager - scanning for plugins...
[INFO ] FolderScanPluginManager - scanning in plug-in folder :/opt/djl/plugins
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: console/jar:file:/opt/djl/plugins/management-console-0.23.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: static-file-plugin/jar:file:/opt/djl/plugins/static-file-plugin-0.23.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: kserve/jar:file:/opt/djl/plugins/kserve-0.23.0.jar!/META-INF/plugin.definition
[INFO ] PropertyFilePluginMetaDataReader - Plugin found: cache-engines/jar:file:/opt/djl/plugins/cache-0.23.0.jar!/META-INF/plugin.definition
[INFO ] FolderScanPluginManager - Loading plugin: {console/jar:file:/opt/djl/plugins/management-console-0.23.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin console changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {static-file-plugin/jar:file:/opt/djl/plugins/static-file-plugin-0.23.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin static-file-plugin changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {cache-engines/jar:file:/opt/djl/plugins/cache-0.23.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin cache-engines changed state to INITIALIZED
[INFO ] FolderScanPluginManager - Loading plugin: {kserve/jar:file:/opt/djl/plugins/kserve-0.23.0.jar!/META-INF/plugin.definition}
[INFO ] PluginMetaData - plugin kserve changed state to INITIALIZED
[INFO ] PluginMetaData - plugin console changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin static-file-plugin changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin cache-engines changed state to ACTIVE reason: plugin ready
[INFO ] PluginMetaData - plugin kserve changed state to ACTIVE reason: plugin ready
[INFO ] FolderScanPluginManager - 4 plug-ins found and loaded.
[INFO ] ModelServer - Found model code_sd_full_bf16=file:/opt/ml/model/code_sd_full_bf16/
[INFO ] ModelServer - Initializing model: code_sd_full_bf16=file:/opt/ml/model/code_sd_full_bf16/
[INFO ] ModelInfo - Apply per model settings:
#011job_queue_size: 1000
#011batch_size: 1
#011max_batch_delay: 100
#011max_idle_time: 60
#011load_on_devices: nc0;nc1
#011engine: Python
#011option.entryPoint: null
#011option.dtype: bf16
#011maxWorkers: 2
#011option.use_stable_diffusion: True
#011option.tensor_parallel_degree: 1
#011option.enable_venv: true
#011minWorkers: 2
#011env: NEURON_RT_NUM_CORES=1
#011option.model_id: s3://imagineai-models/sd21_512_b1_bf16_djl_neuronx
[INFO ] ModelInfo - S3 url found, start downloading from s3://imagineai-models/sd21_512_b1_bf16_djl_neuronx
[INFO ] ModelInfo - Download completed! Files saved to /tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f
[INFO ] Platform - Found matching platform from: jar:file:/usr/local/djl-serving-0.23.0/lib/python-0.23.0.jar!/native/lib/python.properties
[INFO ] PyEnv - Extracting /djl_python_engine.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/init.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/arg_parser.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/deepspeed.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/encode_decode.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/fastertransformer.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/huggingface.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/inputs.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/np_util.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/outputs.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/pair_list.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/rolling_batch/init.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/rolling_batch/lmi_dist_rolling_batch.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/rolling_batch/rolling_batch.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/rolling_batch/scheduler_rolling_batch.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/rolling_batch/vllm_rolling_batch.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/init.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/batch.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/lm_block.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/search_config.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/seq_batch_scheduler.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/seq_batcher.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/seq_batcher_impl.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/step_generation.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/scheduler/utils.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/service_loader.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/stable-diffusion.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/stable_diffusion_inf2.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/streaming_utils.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/test_model.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/tests/init.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/tests/efficiency_benchmark.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/tests/test_input_output.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/tests/test_scheduler.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/tests/test_scheduler_bloom.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/transformers-neuronx.py to cache ...
[INFO ] PyEnv - Extracting /djl_python/ts_service_loader.py to cache ...
[INFO ] ModelManager - Loading model on Python:[nc0, nc1]
[INFO ] ModelInfo - S3 url found, start downloading from s3://imagineai-models/sd21_512_b1_bf16_djl_neuronx
[INFO ] ModelInfo - artifacts has been downloaded already: /tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f
[INFO ] WorkerPool - loading model code_sd_full_bf16 on nc(0) ...
[INFO ] ModelInfo - S3 url found, start downloading from s3://imagineai-models/sd21_512_b1_bf16_djl_neuronx
[INFO ] ModelInfo - artifacts has been downloaded already: /tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f
[INFO ] ModelInfo - Available CPU memory: 14106 MB, required: 0 MB, reserved: 500 MB
[INFO ] ModelInfo - Loading model code_sd_full_bf16 on nc(0)
[INFO ] PyEnv - Python virtual environment created successfully at /tmp/.djl.ai/venv/f7577a9b352089e296d4471037cb79d9dfa5aeaa!
[INFO ] WorkerPool - scale up 2 workers (2 - 2)
[INFO ] PyProcess - Start process: 18999 - retry: 0
[INFO ] Connection - Set NEURON_RT_VISIBLE_CORES=0-0
[INFO ] PyProcess - 115 - djl_python_engine started with args: ['--sock-type', 'unix', '--sock-name', '/tmp/djl_sock.18999', '--model-dir', '/opt/ml/model/code_sd_full_bf16', '--entry-point', 'model.py', '--device-id', '0']
[INFO ] PyProcess - Letting libtpu.so load fail during _XLAC import. libtpu.so will be loaded from libtpu Python package when the ComputationClient is created.
[WARN ] PyProcess - The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling transformers.utils.move_cache().
[WARN ] PyProcess -
[WARN ] PyProcess - 0it [00:00, ?it/s]
[INFO ] PyProcess - Python engine started.
[INFO ] PyProcess - Loading model in /tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f
[INFO ] PyProcess - Loading model in tensor_parallel_degree= 1
[INFO ] PyProcess - Loading model:COMPILED_model_loc=/tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f/compiled_model_full_bf16
[INFO ] PyProcess - Loading model: base model DTYPE={'torch_dtype': torch.bfloat16}:
[INFO ] PyProcess - Loading model: base model loaded
[INFO ] PyProcess - Loading model: unet:created
[INFO ] PyProcess - Loading model in device_ids=[0]::
[INFO ] PyProcess - Dynamic batching is disabled for torch_neuronx.DataParallel
[INFO ] PyProcess - Loading model: unet:wrap:created
[INFO ] PyProcess - Loading model: encoders:created
[INFO ] PyProcess - Loading model: encoders:created
[INFO ] PyProcess - Loading model: decoders:created
[INFO ] PyProcess - Loading model: vae:post_quant:created
[INFO ] PyProcess - Loading model: All Loaded Model:success
[INFO ] PyProcess - Model [code_sd_full_bf16] initialized.
[INFO ] PyProcess - Start process: 19000 - retry: 0
[INFO ] Connection - Set NEURON_RT_VISIBLE_CORES=0-0
[INFO ] PyProcess - 136 - djl_python_engine started with args: ['--sock-type', 'unix', '--sock-name', '/tmp/djl_sock.19000', '--model-dir', '/opt/ml/model/code_sd_full_bf16', '--entry-point', 'model.py', '--device-id', '0']
[INFO ] PyProcess - Letting libtpu.so load fail during _XLAC import. libtpu.so will be loaded from libtpu Python package when the ComputationClient is created.
[INFO ] PyProcess - Python engine started.
[INFO ] PyProcess - Loading model in /tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f
[INFO ] PyProcess - Loading model in tensor_parallel_degree= 1
[INFO ] PyProcess - Loading model:COMPILED_model_loc=/tmp/.djl.ai/download/46131b49d091ece202873e274367735ed6e8c36f/compiled_model_full_bf16
[INFO ] PyProcess - Loading model: base model DTYPE={'torch_dtype': torch.bfloat16}:
[INFO ] PyProcess - Loading model: base model loaded
[INFO ] PyProcess - Loading model: unet:created
[INFO ] PyProcess - Loading model in device_ids=[0]::
[INFO ] PyProcess - Dynamic batching is disabled for torch_neuronx.DataParallel
[WARN ] PyProcess - 2023-Sep-21 13:44:12.0465   136:136   ERROR   NRT:nrt_allocate_neuron_cores               NeuronCore(s) not available - Requested:nc0-nc0 Available:0
[INFO ] PyProcess - error:in handle():: traceback=Traceback (most recent call last):
[INFO ] PyProcess -   File "/usr/local/lib/python3.8/dist-packages/torch_neuronx/experimental/placement.py", line 64, in _get_neuron_visible_core_count
[INFO ] PyProcess -     nc_count = runtime.get_visible_nc_count()
[INFO ] PyProcess - RuntimeError: The PyTorch Neuron Runtime could not be initialized. Neuron Driver issues are logged
[INFO ] PyProcess - to your system logs. See the Neuron Runtime's troubleshooting guide for help on this
[INFO ] PyProcess - topic: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/
[INFO ] PyProcess -
[INFO ] PyProcess - The above exception was the direct cause of the following exception:
[INFO ] PyProcess -
[INFO ] PyProcess - Traceback (most recent call last):
[INFO ] PyProcess -   File "/opt/ml/model/code_sd_full_bf16/model.py", line 198, in handle
[INFO ] PyProcess -     model = load_model(inputs.get_properties())
[INFO ] PyProcess -   File "/opt/ml/model/code_sd_full_bf16/model.py", line 153, in load_model
[INFO ] PyProcess -     pipe.unet.unetwrap = torch_neuronx.DataParallel(
[INFO ] PyProcess -   File "/usr/local/lib/python3.8/dist-packages/torch_neuronx/xla_impl/data_parallel.py", line 196, in init
[INFO ] PyProcess -     self.loaded_modules = self._load_modules(self.module)
[INFO ] PyProcess -   File "/usr/local/lib/python3.8/dist-packages/torch_neuronx/xla_impl/data_parallel.py", line 237, in _load_modules
[INFO ] PyProcess -     torch_neuronx.experimental.placement.set_neuron_cores(module, self.device_ids[0], len(self.device_ids))
[INFO ] PyProcess -   File "/usr/local/lib/python3.8/dist-packages/torch_neuronx/experimental/placement.py", line 134, in set_neuron_cores
[INFO ] PyProcess -     _validate_nc_count(nc_count)
[INFO ] PyProcess -   File "/usr/local/lib/python3.8/dist-packages/torch_neuronx/experimental/placement.py", line 94, in _validate_nc_count
[INFO ] PyProcess -     visible_core_count = _get_neuron_visible_core_count()
[INFO ] PyProcess -   File "/usr/local/lib/python3.8/dist-packages/torch_neuronx/experimental/placement.py", line 66, in _get_neuron_visible_core_count
[INFO ] PyProcess -     raise RuntimeError("Neuron runtime cannot be initialized; cannot determine the number of available NeuronCores") from e
[INFO ] PyProcess - RuntimeError: Neuron runtime cannot be initialized; cannot determine the number of available NeuronCores
[INFO ] PyProcess - :
[INFO ] PyProcess - Stop process: -1:0
[INFO ] PyProcess - Python engine process died
[INFO ] PyProcess - Traceback (most recent call last):
[INFO ] PyProcess -   File "/tmp/.djl.ai/python/0.23.0/djl_python_engine.py", line 160, in main
[INFO ] PyProcess -     engine.run_server()
[INFO ] PyProcess -   File "/tmp/.djl.ai/python/0.23.0/djl_python_engine.py", line 109, in run_server
[INFO ] PyProcess -     inputs.read(cl_socket)
[INFO ] PyProcess -   File "/tmp/.djl.ai/python/0.23.0/djl_python/inputs.py", line 217, in read
[INFO ] PyProcess -     prop_size = retrieve_short(conn)
[INFO ] PyProcess -   File "/tmp/.djl.ai/python/0.23.0/djl_python/inputs.py", line 59, in retrieve_short
[INFO ] PyProcess -     data = retrieve_buffer(conn, 2)
[INFO ] PyProcess -   File "/tmp/.djl.ai/python/0.23.0/djl_python/inputs.py", line 35, in retrieve_buffer
[INFO ] PyProcess -     raise ValueError("Connection disconnected")
[INFO ] PyProcess - ValueError: Connection disconnected
[INFO ] PyProcess - 136 - Python process finished
[INFO ] PyProcess - Stop process: -1:1
[ERROR] ModelServer - Failed register workflow
java.util.concurrent.CompletionException: ai.djl.engine.EngineException: Failed to initialize model: prediction failure
#011at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
#011at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) [?:?]
#011at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702) [?:?]
#011at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) [?:?]
#011at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
#011at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
#011at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
#011at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
#011at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?]
Caused by: ai.djl.engine.EngineException: Failed to initialize model: prediction failure
#011at ai.djl.python.engine.PyProcess.predict(PyProcess.java:99) ~[python-0.23.0.jar:?]
#011at ai.djl.python.engine.PyProcess.startPythonProcess(PyProcess.java:156) ~[python-0.23.0.jar:?]
#011at ai.djl.python.engine.PyModel.newPredictor(PyModel.java:258) ~[python-0.23.0.jar:?]
#011at ai.djl.repository.zoo.ZooModel.newPredictor(ZooModel.java:155) ~[api-0.23.0.jar:?]
#011at ai.djl.Model.newPredictor(Model.java:260) ~[api-0.23.0.jar:?]
#011at ai.djl.repository.zoo.ZooModel.newPredictor(ZooModel.java:138) ~[api-0.23.0.jar:?]
#011at ai.djl.serving.wlm.WorkerThread.(WorkerThread.java:69) ~[wlm-0.23.0.jar:?]
#011at ai.djl.serving.wlm.WorkerThread$Builder.build(WorkerThread.java:298) ~[wlm-0.23.0.jar:?]
#011at ai.djl.serving.wlm.WorkerGroup.addThreads(WorkerGroup.java:103) ~[wlm-0.23.0.jar:?]
#011at ai.djl.serving.wlm.WorkerPool.doScaleWorker(WorkerPool.java:249) ~[wlm-0.23.0.jar:?]
#011at ai.djl.serving.wlm.WorkerPool.initWorkers(WorkerPool.java:205) ~[wlm-0.23.0.jar:?]
#011at ai.djl.serving.models.ModelManager.initWorkers(ModelManager.java:214) ~[serving-0.23.0.jar:?]
#011at ai.djl.serving.models.ModelManager.lambda$registerWorkflow$2(ModelManager.java:132) ~[serving-0.23.0.jar:?]
#011at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
#011... 6 more
[INFO ] PyProcess - ReaderThread(0) stopped - W-19000-code_sd_full_bf-stderr
[INFO ] PyProcess - ReaderThread(0) stopped - W-19000-code_sd_full_bf-stdout
[WARN ] PyProcess - Unexpected restartCount: 0, expect: 2
[WARN ] PyProcess - Unexpected restartCount: 0, expect: 2
[INFO ] ModelServer - Stopping model server.
[INFO ] ModelServer - Initialize BOTH server with: EpollServerSocketChannel.
[INFO ] ModelServer - BOTH API bind to: http://<IP_ADDRESS>:8080
[INFO ] ModelServer - Stopping model server.
[INFO ] ModelServer - BOTH listener stopped.
[INFO ] ModelServer - Model server stopped.

@fafriat
Please use the following serving.properties:
option.use_stable_diffusion=true
option.model_id=s3://imagineai-models/sd21_512_b1_bf16_djl_neuronx
option.tensor_parallel_degree=1
option.dtype=bf16

And please try nightly image, we have a few updates related neuron, and I have tested on nightly image.

@frankfliu
I would be grateful if you can share your logs with:
option.tensor_parallel_degree = 1
using your nightly image and 2 workers on inf2.xlarge or N workers where N is the number of neuron cores on your instance.

@fafriat
As I mentioned earlier. Neuron must use maxWorkers=1.
DJLServing will automatically load N copies of model on N cores

@frankfliu Why DJL Serving is imposing this ???? It is waste of resource.
I have successfully tested using 2 Python processes in parallel on inf2.xlarge for inference, each one associated to only one neuron core !
As explained above, the Neuron documentation is stating it is possible and I effectively tested it!

@fafriat
Currently implementation with maxWorkers=1 can achieve what you want to. Why you think it waste the resource?

@frankfliu You are right ! With maxWorkers = 1 it is allocating 2 processes and each one associated with one neuron core.
(On my case, inference is not working but that may be another problem not related to this issue.)
My problem now is that I cannot explain why it is working.
From what I read from the documentation and my common sense, it is surprising to see 2 workers when I specified maxWorkers = 1 !!!
Do you have any documentation on this behavior ? the name maxWorkers seems misleading...

@fafriat
maxWorkers is per device/partition (neuron core) setting.
Consider multithread model (Java/C++ engine):

In CPU case, you load model once and you can spin off N workers (threads)
In GPU case, you have to model model on each GPU, and spin off N workers (threads) for each GPU

So total number of workers = maxWorkers * number of devices


Auto-bump mapping API version

If a subgraph has defined a mapping API version that is behind the version of graph-ts being used: auto-bump the mapping API version in the subgraph manifest to the compatible version.
Perform the bump of the mapping API version during build.

It would be good if this was done in all of codegen, build and deploy.
I think a good design would be a sequence of migrations, each with a predicate function and a migrate function (and perhaps a skip predicate function?). At minimum each of these functions could receive:

the subgraph filename (because that doesn't have to be subgraph.yaml)
the working directory (so it can access package.json)
the loaded subgraph manifest (maybe undefined if loading failed, to allow migrations to fix manifests)

We could then apply each migration in turn and try to reload the subgraph manifest after every migration. If any of the migrations throw an error, we abort.
It seems reasonable to me to run this in its own CLI spinner.


Search

This PR adds search and discovery to the client.
Tests will be added in a patch or a separate PR in order to not hold up the review.
@morebrownies I added a default image (the same one used in the address bar) for the on-page search if the logo the search provider gives fails to load, which is happening currently with the default search provider.
I also added error modals if the search provider call fails. If the call is to a different provider than the current default (ie: you pasted in a http address), the error message will give you the option to try the same search term with your default provider.
If the error occurs with the user's default provider, they are given the option to open the settings modal and change it. On doing so, the search view will reload and try to use the corrected url.
You can test this by putting a bogus url into the address bar, and by changing the search provider in settings/general to something incorrect.
Closes #203

Should the search page be functional @jjeffryes? I'm not seeing any listings.

@morebrownies it looks like you don't have an up to date copy of the PR? That, or something is breaking during the page load. Do you see any errors in the console?
I just tried it on a new machine, and it is working normally. You should see this when clicking discover:

When there's no results, the previous arrow shouldn't be active:

The issue Mike saw was he had run this branch at least once before when the url for the default was different, and that was saved to his settings.
I've updated the code to account for that, I'll push the fix after reviews are in.

I've added a state for no results found, and fixed the issue with the previous page button being active if no results are found. I'll push that code after reviews are completed so I don't disrupt anything in progress.

Ok thanks @jjeffryes I'm now seeing results.
None of the listings seem to load when I click them though and the filtering/sorting also doesn't seem to work.

Also, we probably shouldn't display the suggestions label if there aren't any suggestions

In settings, I have my currency set to USD, yet the search results display in BTC.

Re: suggestions, those are just stubbed in, I don't think they exist yet in the search server code.
I can comment them out for now.

Also @jjeffryes is it possible to display the user's avatar? Right now it appears you're double outputting the listing image.

@morebrownies good catch on the currency, there was an issue with the server returning 'btc' instead of 'BTC'. I have that fixed, will push it when Rob's review is done.
The avatar being a listing image is a server issue, Tyler is working on it.

Tyler's working on suggestions, when he has them available I'll code them in. That might be a later PR.

Ok cool I found a listing that loads @jjeffryes. I noticed when I click the listing, the overlay does not display the Store name / view store link. We should definitely get that outputting otherwise it's not clear what store you're looking at.
Current

Expected

The spacing on the icons in the filters looks correct on both my Mac and my Windows machine, can you think of a reason it would be different on yours, Rob?

In Windows the flag for Germany renders as just "DE" and the USA as "US". I'm not sure how much we can do about that, it shows up that way in the json when I look at the same text in a browser. The spacing is the same issue, the icons and text are whatever the search provider sends us.

Any filters shown are what the server returns, right now I believe it's returning a default set with a blank search request. The client shouldn't intervene in any way in the filters, whatever the server sends should be assumed to be correct, since there's no way to know what filters a search provider will want to send.

All of the issues noticed in the review should be corrected now. Let me know if I missed anything.

Ok testing now

Store name not displaying @jjeffryes

@morebrownies that's correct. The search provider doesn't send the store name currently, until it does it won't display (I mentioned that in one of the 300 comments above. ;) )

ah, ok definitely missed that. my bad

Hey @jjeffryes sorry found one more small detail. On a user's store, when no search results are found, we display the following:

In search/discover, we display:

It would be nice if those could match up.

As noted above, there are a few places the search server isn't working as expected, however I've reviewed the related code in the client and everything appears okay, so when I get the backend issues shored up it should be all good to go.


Broken x_forwarded_for in access logs

Issue
If a request contains "Connection: close" then x_forwarded_for is not logged correctly.
Context
cf-release 254
Steps to Reproduce
Run the following command for an existent route:
curl -H "Host: xfftest.domain.net" -H "X-Forwarded-For: <IP_ADDRESS>" -H "X-Forwarded-For: <IP_ADDRESS>" -H "Connection: close" localhost:8081

And check application logs.
Expected result
x_forwarded_for should have all IPs joined with commas
2017-10-02T19:16:18.78+0900 [RTR/0]      OUT xfftest.domain.net - [2017-10-02T10:16:18.778+0000] "GET / HTTP/1.1" 200 0 61 "-" "curl/7.35.0" "<IP_ADDRESS>:16372" "<IP_ADDRESS>:61419" x_forwarded_for:"<IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>" x_forwarded_proto:"http" vcap_request_id:"87f77716-b0c6-4597-5c8e-d54f58bf7e00" response_time:0.002689487 app_id:"283c0bf9-1fdd-4b48-94b4-be6bf772c76f" app_index:"0"

Current result
x_forwarded_proto contains only IPs from the first header X-Forwarded-For
2017-10-02T19:16:43.16+0900 [RTR/0]      OUT xfftest.domain.net - [2017-10-02T10:16:43.160+0000] "GET / HTTP/1.1" 200 0 61 "-" "curl/7.35.0" "<IP_ADDRESS>:17014" "<IP_ADDRESS>:61419" x_forwarded_for:"<IP_ADDRESS>" x_forwarded_proto:"http" vcap_request_id:"d17061e4-3922-4846-6dc7-ae02e6e2fc10" response_time:0.003125079 app_id:"283c0bf9-1fdd-4b48-94b4-be6bf772c76f" app_index:"0"

Multiple message-header fields with the same field-name MAY be present in a message if and only if the entire field-value for that header field is defined as a comma-separated list [i.e., #(values)]. It MUST be possible to combine the multiple header fields into one "field-name: field-value" pair, without changing the semantics of the message, by appending each subsequent field-value to the first, each separated by a comma. The order in which header fields with the same field-name are received is therefore significant to the interpretation of the combined field value, and thus a proxy MUST NOT change the order of these field values when a message is forwarded.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

Prioritized for eng review. @giner are you able to send us a PR?

Hi @shalako, if I'm getting it right the issue is here https://github.com/golang/go/commit/3c0f69a52101c69b5a8288195fa74c7ecfa2fa43. X-Forwarded-For brakes if the initial request contains any of hop by  hop headers.

We have started upgrading gorouter to 1.9. The issue does appear to be fixed.

After upgrading to golang 1.9, it appears the issue is resolved.
on gorouter
$ curl -H "Host: dora-scoen.superman.cf-app.com" -H "X-Forwarded-For: <IP_ADDRESS>" -H "X-Forwarded-For: <IP_ADDRESS>" -H "Connection: close" localhost:80
Hi, I'm Dora

from workstation
$ cf logs dora-scoen
Retrieving logs for app dora-scoen in org scoen / space test as scoen...

   2017-10-19T21:24:53.88-0700 [RTR/0] OUT dora-scoen.superman.cf-app.com - [2017-10-20T04:24:53.875+0000] "GET / HTTP/1.1" 200 0 13 "-" "curl/7.35.0" "<IP_ADDRESS>:43174" "<IP_ADDRESS>:61002" x_forwarded_for:"<IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>" x_forwarded_proto:"http" vcap_request_id:"cd7edee6-886b-4290-51ba-cd03035ecdaa" response_time:0.005739814 app_id:"27a49ecb-b63c-4315-9159-da7b69ba9016" app_index:"0" x_b3_traceid:"2a50038a77c7c5ae" x_b3_spanid:"2a50038a77c7c5ae" x_b3_parentspanid:"-"
   2017-10-19T21:24:53.88-0700 [RTR/0] OUT

Fixed in 0.165.0


Typographical baselines

Code https://github.com/michaeltaranto/basekick
Video https://www.youtube.com/watch?v=TGHbkTGVqoU
Related:

http://compass-style.org/reference/compass/typography/vertical_rhythm/
http://type-scale.com/
http://foundation.zurb.com/sites/docs/typography-base.html
https://github.com/hiulit/Sassy-Gridlover (demo)
http://codepen.io/getflourish/full/vXqewy/

Thanks for the feature request @stereobooster. This is one we originally had on our list for V1 but the way it needs to work with global rules gave us pause. As we think about V2, we wouldn't be opposed to revisit this. Any thoughts on what a mixin for this would look like?

This is not final vision, trying to brainstorm this.
I suppose there is should be "theme" object, like state in redux. For vertical rhythm we need base font size and scale factor (at least).
Basic vertical rhythm
const theme = {
  base_font_size: "16px",
  scale_factor: 1.68
}
font(theme, 0) //paragraph
font(theme, 1) //min heading, for example H3
font(theme, 3) //max heading, for example H1

Gutenberg
Gutenberg uses base_font_size to fit images in grid.
Adaptive Modular Scale
Adaptive Modular Scale will need following theme object
const theme = {
min: {
  width: 320,
  base_font_size: "16px",
  contrast: 1.2 // the same as scale factor, I suppose
},
max: {
  width: 960,
  base_font_size: "18px",
  contrast: 1.6 // the same as scale factor, I suppose
}}

This reminds me about breakpoints, which probably also should be present in theme object.
Breakpoints
const theme = {
breakpoints: {
  min: 320,
  max: 960
},
base_font_size: "16px", // values at root level are defaults or min breakpoint (mobile first)
contrast: 1.2,
breakpoint_max: {
  base_font_size: "18px",
  contrast: 1.6 // the same as scale factor, I suppose
}}

Basekick
theme = {
  typeSizeModifier: 1,
  typeRowSpan: 1,
  descenderHeightScale: 1,
  baseFontSize: 1,
  gridRowHeight: 1,
  lineHeightOverride: 1
}

Some of Basekick settings are font family dependent. So we can have database with those values based on font family: typographicBaselineFor("Roboto") (in separate repo).
Bonus points
We also can implement verticalGrid(theme), which will generate styles for grid overlay.
Having this functionality as JavaScript function (instead of css/sass/less) it is easy to implement preview page, like Adaptive Modular Scale or font-style-matcher.

This is something similar https://github.com/kyleshevlin/shevyjs

Also, being able to set global fluid typography using the popular SASS mixins (https://css-tricks.com/snippets/css/fluid-typography/) would be super helpful!

@loganpowell as of now though. We are still not sold on adding a module to polished that relies on global values/styles. It breaks the paradigm of a component being shippable on it's own. While we understand that is a reality of how things work, it is harder to design for and properly communicate to a user of the library.

https://github.com/rafaelrinaldi/responsive-styles
const breaks = [48, 64, 80]
const r = (props, values) => responsiveStyles(props, values, breaks)

const styles = StyleSheet.create({
  // A la functional CSS
  padding: r('padding', [8, 24, 48]),
  fontSize: r('fontSize', [16, 24, 36]),

  // Combine multiple definitions into a single class name
  colors: combineSameKeys(
    r('color', ['#FFF', '#005782', '#820005', '#16160B']),
    r('backgroundColor', ['#FF0066', '#27D88E', '#FFF5C3', '#E1E1E1'])
  )
})

between (https://polished.js.org/docs/#between) and fluidRange ( can now be used to scale font-size or any other property using linear interpolation. Either of these should be able to be combined with theming to accomplish most of this functionality now with v2.

@bhough came across this when wanting to the mixin in this article https://css-tricks.com/snippets/css/fluid-typography/ and use it in styled-components.
I've tried both between and fluidrange, seem to need different values but both work fine. Was just wondering which one is recommended over the other?

@AWR14 it depends on your surrounding code. The only real difference between the two are the media query constraints. In most cases, you will want fluidRange as it includes the "stops" at either end of the range. between returns just the linear interpolation for the value for cases in which you are handling the media query separately.

@bhough Thanks, i'm going with fluidRange based on your answer. I'm new to styled components and moving over from SCSS, is this the recommended way of adding SCSS Mixin style functions?

@AWR14 generally speaking, yes we created these "officially" to ease the transition from SCSS to styled-components.

@bhough Thanks. does it only work with px units? i've set my base font to 10px {font-size: 62.5%;) but gives weird results when using fluidRange
${fluidRange(      {        prop: 'font-size',        fromSize: '2rem',        toSize: '10rem',      }    )};

@AWR14 What results are you seeing?

@bhough it appears to not be as smooth when using rem, seems to jump in size more.
`${fluidRange(
     {
       prop: 'font-size',
       fromSize: '40px',
       toSize: '100px',
     },
     '300px',
     '1200px',
   )};`

${fluidRange(
     {
       prop: 'font-size',
       fromSize: '40px',
       toSize: '100px',
     },
     '300px',
     '1200px',
   )};

i am doing this to set the default to 10px
html {
    font-size: 62.5%;
}


Add tests to ensure Update CLI works

There was a request to ensure that the updated command works when files are moved (see https://github.com/gtronset/beets-filetote/pull/124#issuecomment-1840293273). This PR adds tests to ensure that does occur.
By default, media files are automatically renamed based on their new metadata. The work done in https://github.com/gtronset/beets-filetote/pull/124 enables any CLI command that results in moving files to also enable Filetote to move artifacts and extra files.

@arogl were you able to test the update command to see if Filetote did or did not correctly handle artifact files? From what I can tell it should. I added a test to check; it appears to work as I'd expected.

I found the issue after I updated some meta data manually and then ran a beet update "albumartist:Metallica" the music files were moved but the lrc files that were moved during the import were left in the old folder as the year was updated

@arogl Would you mind sharing your config so I can replicate the scenario?

Config is large
directory: \\?\I:\Music
import:
  copy: no
  move: yes
  write: yes
  detail: yes
  resume: ask
  quiet_fallback: skip
  timid: no
  log: \\?\I:\new_Music\beetslog.txt
  languages: en

ignore: .AppleDouble ._* *~ .DS_Store *.sfv *.srr *.nfo *.m3u *.ini *.log

art_filename: cover

pluginpath: C:\Users\arogl\AppData\Roaming\beets\beetsplug\

plugins:
  - abcalc
  - arttools
  - audit
  - badfiles
  - bandcamp
  - barcode
  - bpmanalyser
  - chroma
  - deezer
  - discogs
  - duplicates
  - edit
  - embedart
  - fetchart
  - filetote
  - fromfilename
  - importadded
  - info
  - inline
  - lastgenre
  - mbsync
  - missing
  - popularity
  - replaygain
  - scrub
  - spotify
  - similarity
  - unimported
  - zero

# - autofix
# - absubmit
# - acousticbrainz
#  - beatport
# - check
# - convert
# - copyartifacts
# - dates
# - extrafiles
# - follow
# - ftintitle
# - gracenote
# - lyrics
# - mosaic
# - moveall
# - oldestdate
# - series
# - wikiPlug
# - wlg
# - yearfixer

replace:
  #    '[\\/]': ''
  '[\x00-\x1f]': ""
  '^\.\.\.': "…"
  '^\.\.': "‥"
  '^\.': "․"
  '\.\.\.': "…"
  '\.\.\.$': "…"
  '\.\.$': "‥"
  '\.\.': "‥"
  '\.$': "․"
  '\s+$': ""
  '^\s+': ""
  '‐': "-"
  ":": "∶"
  "\/": "⁄"
  '\*': "★"
  '\|': "ǀ"
  '\u003F': "？"
  "’": "'"
  '"': "″"
  '\\': "⧵"
  "<": "‹"
  ">": "›"
  "<<": "«"
  ">>": "»"
#  "&": "＆"
  "Florence And The Machine": "Florence + The Machine"
  "Marina & The Diamonds": "MARINA"
  "Marina and The Diamonds": "MARINA"
  "Lauren Spencer-Smith": "Lauren Spencer Smith"
  "Lølø": "LØLØ"
  "“Weird Al” Yankovic": "Weird Al Yankovic"
  "Kaylen's Rain": "Kaylens Rain"
  "Derek & Clive": "Derek and Clive"

threaded: yes
#threaded: no

va_name: "Various Artists"
id3v23: no

time_format: "%Y-%m-%d %H:%M:%S"
format_raw_length: no

sort_album: albumartist+ year+ album+
sort_item: artist+ year+ album+ disc+ track+
sort_case_insensitive: yes

ui:
    terminal_width: 200
    length_diff_thresh: 10.0
    color: yes
#    colors:
#        text_success: ['bold', 'green']
#        text_warning: ['bold', 'yellow']
#        text_error: ['bold', 'red']
#        text_highlight: ['bold', 'red']
#        text_highlight_minor: ['white']
#        action_default: ['bold', 'cyan']
#        action: ['bold', 'cyan']
#        # New Colors
#        text: ['normal']
#        text_faint: ['faint']
#        import_path: ['bold', 'blue']
#        import_path_items: ['bold', 'blue']
#        added:   ['green']
#        removed: ['red']
#        changed: ['yellow']
#        added_highlight:   ['bold', 'green']
#        removed_highlight: ['bold', 'red']
#        changed_highlight: ['bold', 'yellow']
#        text_diff_added:   ['bold', 'red']
#        text_diff_removed: ['bold', 'red']
#        text_diff_changed: ['bold', 'red']
#        action_description: ['white']
    import:
        indentation:
            match_header: 2
            match_details: 2
            match_tracklist: 5
        layout: column

format_item: "\u001b[92m$artist\u001b[39m - \u001b[31m$album\u001b[39m - \u001b[93m$title\u001b[39m"
format_album: "\u001b[92m$albumartist\u001b[39m - \u001b[93m($year)\u001b[39m - \u001b[31m$album\u001b[39m"

paths:
  default: %upper{%asciify{%left{$first_artist,1}}}/$first_artist/($year) $album%aunique{}/$album - $track - $artist - $title
  singleton: Singles/$artist - $title
  albumtype:soundtrack: Soundtracks/($year) $album%aunique{}/$album - $track - $artist - $title
  albumartist:\“Weird\ Al\” Yankovic: W/Weird Al Yankovic/($year) $album%aunique{}/$album - $track - $artist - $title
  albumartist:various\ artists: Various Artists/($year) $album%aunique{}/$album - $track - $artist - $title
  albumartist:various: Various Artists/($year) $album%aunique{}/$album - $track - $artist - $title
  comp: Various Artists/($year) $album%aunique{}/$album - $track - $artist - $title
  ext:.pdf: %upper{%asciify{%left{$first_artist,1}}}/$first_artist/($year) $album%aunique{}/$album - 00 - $artist - $title
  paired_ext:.lrc: $albumpath/$artist - $track - $title
  paired_ext:.txt: $albumpath/$artist - $track - $title
#  paired_ext:.lrc: %upper{%asciify{%left{$first_artist,1}}}/$first_artist/($year) $album%aunique{}/$medianame_old

#types:
#  date: date

item_fields:
  first_artist: |
    import re
    artistlist = ["Beth Hart & Joe Bonamassa", "Bob Marley & The Wailers", "David Guetta & Cathy Guetta", "Elton John & Billy Joel", "H & Claire", "Hootie & the Blowfish", "Jakle & Hide"]
    artistlist += ["Joan Jett and the Blackhearts", "Jess & Matt", "Jon and Vangelis", "Lily & Madeleine", "Rihanna, Kanye West & Paul McCartney", "Simon & Garfunkel", "She & Him"]
    artistlist += ["The Mamas & the Papas", "Maddie & Tae", "Aly & AJ", "Presley & Taylor", "Tones and I", "Jon and Vangelis"]
    return re.split(r'(?i)(,\s)|(;\s)|(\s&\s)|(\sfeat)|(\sft\.)|(\sft\ )|(\ X\ )',albumartist,1)[0:1][0] if albumartist not in artistlist else albumartist
#  date: |
#    import datetime
#    return datetime.datetime(year or 1, month or 1, day or 1).timestamp()
#  item_data_source: |
#    for item in items:
#        if item.get('data_source'):
#            return f'{item.data_source}'

#album_fields:
#  date: |
#    import datetime
#    return datetime.datetime(year or 1, month or 1, day or 1).timestamp()

zero:
  fields: comments lyrics # ENCODEDBY LYRICIST PUBLISHER UNSYNCEDLYRICS
  comments:
    [
      EAC,
      LAME,
      from.+collection,
      "ripped by",
      ExactAudioCopy,
      "Exact Audio Copy",
      "www.tr7music.me",
      "WWW.ITUNESCLUB.COM",
      "itunesvn.com",
      "http://mzmusiczone.blogspot.com/",
      "www.t.me/pmedia_music",
      "PMEDIA NETWORK",
    ]
  lyrics: [XXX, "© ¤ @"]
  update_database: true

scrub:
  auto: yes

replaygain:
  overwrite: no
  auto: yes
  backend: ffmpeg

embedart:
  auto: yes

fetchart:
  auto: yes
  cautious: yes
  cover_names: cover front art album folder
  google_search: yes
  minwidth: 0
  enforce_ratio: no
  sources: filesystem coverart itunes albumart amazon fanarttv wikipedia google
  fanarttv_key: f3cea7bb050be2d0f704bfc8738c24d9
  google_API_key: [REDACTED]
  google_engine_ID:<PHONE_NUMBER>23014548361:lndtuqkycfu

acoustid:
  apikey: [REDACTED]

#convert:
#    ffmpeg: c:\agrbin\bin\ffmpeg.exe
#    embed: true
#    format: mp3
#    max_bitrate: 192
#    dest: %upper{%left{$first_artist,1}}/$artist/(%if{$original_year,$original_year,$year}) $album%aunique{}/$album - $track - $artist - $title
#    threads: 4
#    paths:
#        default: %upper{%left{$artist,1}}/$first_artist/(%if{$original_year,$original_year,$year}) $album%aunique{}/$album - $track - $artist - $title

copyartifacts:
  extensions: .pdf .txt .lrc .jpg .jpeg .png
  print_ignored: yes

check:
  import: yes
  write-check: yes
  write-update: yes
  convert-update: yes
  threads: 8
  tools:
    mp3val:
      cmd: "mp3val {}"
      formats: MP3
      error: '^WARNING: .* \(offset 0x[0-9a-f]+\): (.*)$'
      fix: "mp3val -f -nb {}"
    flac:
      cmd: "flac --test {}"
      formats: flac

duplicates:
  album: no
  checksum: ffmpeg -i {file} -f crc -
  copy: none
  count: no
  delete: no
  format: $album - $track - $artist - $title
  full: no
  strict: no
  move: none
  keys: [ffmpeg]
  #    keys: [mb_trackid, mb_albumid]
  path: no
#    tag: no

match:
  ignore_video_tracks: no
  distance_weights:
    barcode: 1.0
#    ignored_media: ['Data CD', 'DVD', 'DVD-Video', 'Blu-ray', 'HD-DVD',
#                    'VCD', 'SVCD', 'UMD', 'VHS']

importadded:
  preserve_mtimes: yes

lyrics:
  fallback: ""
  synced: yes
  sources: lrclib musixmatch genius # lyricwiki google
  google_API_key: [REDACTED]
  google_engine_ID:<PHONE_NUMBER>23014548361:lndtuqkycfu

ftintitle:
  auto: yes
  drop: no
  format: feat. {0}

#follow:
#    email<EMAIL_ADDRESS>#    password: [REDACTED]
#    userid: [REDACTED]
#    auto: yes

wlg:
  auto: yes
  force: no
  count: 4
  separator: ", "
  whitelist: beets

bandcamp:
  art: true
  lyrics: true

badfiles:
  commands:
    flac: flac --test --warnings-as-errors --silent

acousticbrainz:
  auto: yes

lastgenre:
  auto: yes
  count: 3
  canonical: yes

musicbrainz:
  host: musicbrainz.org
  ratelimit: 1
  ratelimit_interval: 1.0
  searchlimit: 20
  genre: yes
  user: arogl
  password: [REDACTED]
  extra_tags: [year, catalognum, country, media, label]

absubmit:
  auto: yes
  extractor: D:\win_prog\acousticbrainz-gui\streaming_extractor_music.exe
#    extractor: ~/bin/streaming_extractor_music
dates:
  prefix: "="

extrafiles:
  patterns:
    pdf:
      - "*.pdf"
    artworkdir:
      - "[sS]cans/"
      - "[aA]rtwork/"
      - "[cC]over/"
      - "[cC]overs/"
      - "[sS]can/"
    lyrics:
      - "*.lrc"
      - "*.txt"
    images:
      - "*.png"
      - "*.jpeg"
      - "*.jpg"
      - "*.gif"
  paths:
    artworkdir: $albumpath/Artwork

bpmanalyser:
  auto: yes
  dry-run: no
  write: yes
  threads: 8
  force: no
  quiet: no

discogs:
  source_weight: 0.0

deezer:
  source_weight: 0.2

autofix:
  max_exec_time: 0
  tasks:
    missing_file_checker:
      enabled: yes
    year_fixer:
      enabled: yes
    audio_conversion:
      enabled: no
    tag_cleaner:
      enabled: no
    ab_data_fetcher:
      enabled: yes
    xtractor:
      enabled: no
    genre_finder:
      enabled: yes

oldestdate:
  auto: yes
  ignore_track_id: yes
  filter_on_import: yes
  prompt_missing_work_id: yes
  force: yes
  overwrite_date: no
  filter_recordings: yes
  approach: "releases"

yearfixer:
  auto: yes

filetote:
  pairing:
    enabled: true
    pairing_only: true
    extensions:
      .lrc
      .txt
  extensions: .pdf
  patterns:
    artworkdir:
          - "[sS]cans/"
          - "[aA]rtwork/"

I think it may have to do with the paired file, during normal import the files are paired, but during the update the files are no longer paired

@arogl You're actually spot on that this is due to pairing, in particular the combination of "paired only" and how the originally paired files are renamed during the original import. Here's what I believe is occurring:
The config is set to only grab artifacts and extra files that are paired (from pairing being enabled with pairing_only being set):
filetote:
  pairing:
    enabled: true
    pairing_only: true
    extensions:
      .lrc
      .txt

This grabs the paired files as expected in the original import and renames them using the below path settings:
paths:
  default: %upper{%asciify{%left{$first_artist,1}}}/$first_artist/($year) $album%aunique{}/$album - $track - $artist - $title
  ...
  paired_ext:.lrc: $albumpath/$artist - $track - $title
  paired_ext:.txt: $albumpath/$artist - $track - $title

Interestingly, this renaming causes the files to no longer be paired, since the non-media files have a different naming structure that no longer matches the media ($album - $track - $artist - $title for media vs $artist - $track - $title for non-media/lrc).
The result of this is when the update is run, Filetote looks for only paired files but doesn't find any matches since the names are now different, and thus does not move the non-media files in those folders since they are no longer paired.
Do you have a specific use case for changing the name of the paired files to not match? If so, I'd love to dig in and try to accommodate that functionality (I have a rough idea of how I could do that). Otherwise, I believe the above naming changes will effectively unpair the non-media files and likely should be changed to paired_ext:.lrc: $albumpath/$medianame_new for future imports.

Files are paired when I import, but I change/keep them as $albumartist - $track - $title on import and how my internal lyrics provider grabs lyrics and saves them if none exist.
If I can use them as $medianame_new I can change my lyrics provider to match the default track format.

@arogl can you give that a go, and if you run into any problems or questions create a new issue so that we can discuss?
update did work before this PR, though I did expose a bug unrelated to not moving. This is now corrected.


Show/hide directions button doesn't appear on new itineraries

If you create a new itinerary from scratch the show/hide directions button doesnt appear after adding places until you refresh the page

Was this fixed?


[UI] use vector or Retina icons for Wi-Fi and cellular bars

I could try making them in Sketch, if you don't mind having a Sketch file checked into the repo, @twostraws.

I tried to address this in #55
let me know if those icons look off, I tried to just use some pdf exported SF Symbols

Fixed in #55


move to github actions

Signed-off-by: Martin Schurz<EMAIL_ADDRESS>

I verified this in my own fork, since actions only run, when at least one action is present in master.
check there: https://github.com/schurzi/puppet-os-hardening/actions/runs/552346826 (now it runs in just under 5 minutes ;) )


Implement spawn-based parallel reduce

Codecov Report

Merging #25 into master will decrease coverage by 3.34%.
The diff coverage is 25%.

@@            Coverage Diff             @@
##           master      #25      +/-   ##
==========================================
- Coverage    85.7%   82.35%   -3.35%
==========================================
  Files          15       15
  Lines        1056     1060       +4
==========================================
- Hits          905      873      -32
- Misses        151      187      +36

Impacted Files
Coverage Δ

src/processes.jl
82.75% <25%> (-11.22%)
:arrow_down:

src/evals.jl
50% <0%> (-50%)
:arrow_down:

src/simd.jl
72.72% <0%> (-9.1%)
:arrow_down:

src/core.jl
61.82% <0%> (-4.66%)
:arrow_down:

src/library.jl
89.12% <0%> (-1.21%)
:arrow_down:

src/show.jl
95.38% <0%> (-0.04%)
:arrow_down:

src/comprehensions.jl
90% <0%> (+13.07%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 655b194...a86db7f. Read the comment docs.


part segmentation visualization script

Hi，
Thank you very much for sharing the code on DGCNN. I have completed the training on part segmentation and want to visualize it. I read your suggestions on #8 and #26. But I only exported points (x, y, z) and cannot assign the color (R,G,B ) by the label . Can you share your visualization  script? thank you very much!
I'm sorry to trouble you, and I'm sorry to ask you such a simple question, I am a beginner in point cloud work, I hope you understand, thank you very much.

I'm sorry that I didn't write a visualization script.
Please tell me if you don't know the label of each point or don't know how to connect colors with labels.

I don't know how to connect colors with labels.

You can init a color list with 50 colors, where each color corresponds to a label.
See #10 for more details.

Thank you very much!

Thank you very much!

I don't know how to connect colors with labels.

Thank you very much!

Can you share your visualization script? thank you very much!

Thank you very much!

Can you share your visualization script? thank you very much!

The visualization code is added in this repo.


MSSQL. Translation of DateTime.AddDays(double)

I think, your translation of DateTime.AddDays to SQL [for MSSQL] is incorrect.
I not tested, just looked to code:
https://github.com/aspnet/EntityFrameworkCore/blob/master/src/EFCore.SqlServer/Query/ExpressionTranslators/Internal/SqlServerDateAddTranslator.cs
MSSQL (2016)
Query "select DATEADD(DAY, 1.5, CONVERT(datetime,'01.01.2000'))" returns "2000-01-02 00:00:00.000"
DateTime.AddDays(Double) Method
Remarks
This method does not change the value of this DateTime. Instead, it returns a new DateTime whose value is the result of this operation.
The fractional part of value is the fractional part of a day. For example, 4.5 is equivalent to 4 days, 12 hours, 0 minutes, 0 seconds, 0 milliseconds, and 0 ticks.

Duplicate of #11794

@smitpatel
Ok, you knew about this problem.
But why you not added note about it in source code?
This small comment could prevent a copying of this issue into other providers :)
FirebirdSql.EntityFrameworkCore.Firebird.BLA-BLA-BLA.FbDateAddTranslator


[HOLD Onyx Cache] Android - Loading chat items is extremely slow

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Expected Result:
Chat messages are displayed just as the user opens the conversation
Actual Result:
Chat messages take a lot to load and are displayed in parts, first it's blank, then the users and time and after the messages.
Action Performed:

Open the android app
Navigate to a conversation with several messages

Workaround:
Yes, user has to wait for the messages to load.
Platform:
Where is this issue occurring?
Web
iOS
Android ✔
Desktop App
Mobile Web
Version Number: 1.0.32-0
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
I was unable to reproduce this with the beta build from the playstore
https://user-images.githubusercontent.com/44479856/116468184-8bb0f180-a83e-11eb-827e-5558576d71f2.mp4
Expensify/Expensify Issue URL:
View all open jobs on Upwork

From @quinthar https://expensify.slack.com/archives/C01GTK53T8Q/p1619487604117700

ISSUE: I think something has changed in how we populate chats after switching to them. I find they start out blank, then I will see the time and dates for recent chats, then I will see the time and dates back populate for a while, and then I will see the chats individually populate one by one from oldest to most recent. It takes a long time and it's very disconcerting.

Triggered auto assignment to @michaelhaxhiu (AutoAssignerTriage), see https://stackoverflow.com/c/expensify/questions/4749 for more details.

I'm very occupied with Digital Card the past day, and plan to be tomorrow as well. Can I please re-assign this one to free up space?

Triggered auto assignment to @bfitzexpensify (AutoAssignerTriage), see https://stackoverflow.com/c/expensify/questions/4749 for more details.

Rajat reproduced here

Triggered auto assignment to @joelbettner (Engineering), see https://stackoverflow.com/c/expensify/questions/4319 for more details.

This issue is still in a holding pattern.

I believe this is no more required.

Hey @isagoico are you able to confirm if this is resolved and the issue can be closed?

Yep! This is not reproducible anymore in android 🎉


[build-script] Don't disable modules when ASan is enabled

In the past, we needed to disable building swift with -fmodules because
the module caching logic ignored sanitizer flags (rdar://28356072).
This is fixed now, so let's remove the workaround.

@swift-ci smoke test and merge


[New Project] Combind tree-grid-directive & angular-ui-tree

Demo: http://thienhung1989.github.io/angular-tree-table/

This product is the combination and optimization of 2: https://github.com/angular-ui-tree/angular-ui-tree (Drag2Drop) and https://github.com/khan4019/tree-grid-directive (Display Tree2Table).

*Stil using jQuery. I fix it soon.
Thanks @JimLiu and all.

@thienhung1989 looks nice! :-) Is this a personal experiment or are you working on a project which needs this?

That's really cool. I have a project that might actually benefit from something like this.

Thanks @Voles & @enkodellc, I working on this project. Used to manage languages (English, Vietnam, ... ), permission,... and see also other details.


feat: add unallocated resource into mid resource

implement mid-tier overcommitment in https://koordinator.sh/docs/designs/node-prediction/#mid-tier-overcommitment
Allocatable[Mid]' := min(Reclaimable[Mid], NodeAllocatable * thresholdRatio) + Unallocated[Mid] Unallocated[Mid] = max(NodeAllocatable - Allocated[Prod], 0)
Ⅰ. Describe what this PR does

Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how to verify it
Ⅳ. Special notes for reviews
V. Checklist

[x] I have written necessary docs and comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in make test

maybe Allocatable[Mid]' := min(Reclaimable[Mid] + Unallocated[Mid], NodeAllocatable * thresholdRatio) is better than Allocatable[Mid]' := min(Reclaimable[Mid], NodeAllocatable * thresholdRatio) + Unallocated[Mid]

#1762

/remove-lifecycle stale

/reopen

/close


Loading global variable uses incorrect address (zero)

Discovered in get_inputs test.
Loading __stack_pointer is incorrect(address is zero) in MASM.
Wasm:
...
  (func $entrypoint (;3;) (type 3) (param i32)
    (local i32 i32 i32)
    global.get $__stack_pointer
    ...
  )
  ...
  (global $__stack_pointer (;0;) (mut i32) i32.const 1048576)

HIR:
...
    (module #abi_transform_tx_kernel_get_inputs_4
        ;; Constants
        (const (id 0) 0x00100000)

        ;; Global Variables
        (global (export #__stack_pointer) (id 0) (type i32) (const 0))

        ;; Functions
        (func (export #entrypoint) (param i32)
            (block 0 (param v0 i32)
                (let (v1 i32) (const.i32 0))
                (let (v2 i32) (global.load i32 (global.symbol #__stack_pointer)))
                ...

MASM:
...
export.entrypoint
    mem_load.0x00000000
...

So, one thing to be aware of: the stack pointer address should actually be zero (LLVM places it at the start of linear memory when compiling to Wasm).
Because there are no data segments, the global table is being placed at the start of the linear memory (address zero), and stack_pointer appears to be the first global in the table.
The actual bug though, I think, is that we're not reserving the first 64k page for the stack. LLVM does this when compiling to memory, but it doesn't look like that is coming through when translating to the IR (i.e. I'd expect a data segment for the stack, but we may have to always assume the first 64k is reserved.

Oh, you're right! The memory address can and should be zero. I just messed up the global var value with its address.
So the problem I'm experiencing (assert_i32 failing) is that at this zero address 1048576 value is expected to be stored (initial global variable value, see Wasm above), which is not the case. The loaded value is zero, so the next 0 - 16 ops produces the unexpected value (instead of  1048576 - 16 = 1048560).
You mentioned that you are working on rodata init, so you might already know about this issue.


Rename withDefault to getOrElse

We recently introduced withDefault https://github.com/optics-dev/Monocle/pull/886, a combinator that transform an Optic[A, Option[B]] into a Optic[A, B] if you pass a default value of type B.
@kenbot proposed to renamed it getOrElse since it offers a similar behaviour than getOrElse on the Option class.
What do you think?

I think I agree, it feels natural in scala and withDefault could have other interpretations depending on the context.

My only concern is that Optional and Prism have a method getOption which fetch the targeted data while getOrElse is an optic combinator. Would it be confusing? Here are the signatures:
val optEmail: Optional[User, Option[Email]] = ...

optEmail.getOption(user)
// res: Option[Option[Email]]

optEmail.getOrElse(defaultEmail)
// res: Optional[User, Email]

My gut feeling is that it doesn't make much sense to call getOption on an Optional/Prism targeting an Option because you get back an Option[Option[To]].


Caches are receiving mangled metrics

So I have this cluster that I'm playing with and everything is talking to each other... more or less. I have two relays, three caches, one 'solo' graphite web, one grafana, etc. Everything is on 0.9.15.
So I have haggar sending metrics to a relay but when they hit the caches... I get something like this:
2017-01-06_15:16:01.55760 06/01/2017 15:16:01 :: [listener] invalid line (Uhaggar.agent.26.metrics.7533GA@G@aq
                                                                                Uhaggar.agent.26.metrics.7539GA@G@\q
                                                                                Uhaggar.agenUhaggar.agent.26.metrics.7552GA@G@qUhaggar.agent.26.metrics.7557GA@G@qUhaggar.agent.26.metrics.7563GA@G@b@qUhaggar.agent.26.metrics.7573GA<EMAIL_ADDRESS>
...

2017-01-06_15:16:01.60592 06/01/2017 15:16:01 :: [listener] MetricLineReceiver connection with <IP_ADDRESS>:48674 lost: Connection to the other side was lostin a non-clean fashion.

I couldn't find anything like this through googling and I'm not sure what I'm doing wrong.

Oh, shoot. This is the web project. Uh, should I file this in carbon?

Created issue here: https://github.com/graphite-project/carbon/issues/618


Setup new devicelab rack space

Please setup a new rack as per available equipment.
Also, note how many available network connections are available from our existing switches, that we can use in the new rack.

Will check and work on this.

There are 8 available drops.
Ordering a network switch to try and expand into the new racks, risk is shared bandwidth with existing switch.
Also ordered network cables.

Ordered GS348 switch.

We were able to secure 4 trays for the new rack, still waiting on lab team to acquire additional trays for us.


project panel collapse all entries from outside context

Check for existing issues

[X] Completed

Describe the feature
Allow collapse all entries to be used outside of the project panel context.
Currently to use the project panel: collapse all entries you need to focus on the project panel first. Often I find that I want to collapse and then focus the project panel.
So currently you either have to randomly click on a file first or close and open the project panel with the kb shortcut.
I know this is not a major problem, just putting it out there.
If applicable, add mockups / screenshots to help present your vision of the feature
No response

This is certainly something I find painful myself. There is an alternative that makes things slightly better. If you learn the keybinding to focus the project panel: cmd-shift-e, you can run that first, before running the project panel: collapse all entries action.  This isn't better than having the command avaiable to you globally, but at least you don't have to open and close a random file when trying to focus the project panel.


Revise harbor build version

Coverage decreased (-0.03%) to 60.683% when pulling fdf3e4d91905c9177cb67be54d52a70574586251 on wy65701436:refine-build-num into 8aa63b661c0e46c71792f7f9e2e040b18cb4e2bf on vmware:master.


Fail to maintain context between Subqueries

The following code fails:
var cypher = """
	CALL {
		CREATE (m:Movie)
		RETURN m
	}
	CALL {
		WITH m
		RETURN m { .title } AS movies
	}
	RETURN movies""";
var cfg = Configuration.newConfig()
	.withPrettyPrint(true)
	.withGeneratedNames(true)
	.build();
var renderer = Renderer.getRenderer(cfg);
var parseOptions = Options.newOptions().createSortedMaps(true).build();
var normalized = renderer.render(CypherParser.parse(cypher, parseOptions));
assertThat(normalized).isEqualTo("""
	CALL {
	  CREATE (v0:Movie)
	  RETURN v0
	}
	CALL {
	  WITH v0
	  RETURN v0 {
		.title
	  } AS v1
	}
	RETURN v1""");

The generated Query is:
CALL {
    CREATE (v0:Movie)
    RETURN v0
  }
  CALL {
    WITH v1
    RETURN v1 {
      .title
    } AS v2
  }
  RETURN v2

the v1 in the second subquery should be v0

As usual, thanks a lot for the report. For whatever reason I missed the notification and for what's worse, pushed the release button to early.


Issue#1936 annotation

Made lots of changes! It now assigns the glow class to the spans and splits them if needed. Made changes to method calls and added a migration file to support inclusion of column offsets. Also fixed an issue where the mouse listeners weren't removed if the line was no longer glowing
The glow method is pretty long right now but I didn't really see many opportunities to break it into more methods, open to suggestions.
Also I assume there will be some formatting issues, hopefully hound/travis let me know so I can make the changes ASAP.

I think the tests need to be updated. Changes look cool - I'll take a closer look later this week.

@spring-ryanc This is looking quite impressive. My main thought on splitting up glow is to break up each conditional block into its own method. I don't think many parameters would need to be passed into each helper, though I could be wrong.

Hi David, I've made the style changes you suggested. But I'm not sure which test need to be updated. Could you provide a little more information about that?

You can see from the Travis status: https://travis-ci.org/MarkUsProject/Markus/builds/49055684

@spring-ryanc Looks good, although I think Hound would like you to change the formatting of that blueprints.rb file. :)

@spring-ryanc I merged it, but looks like multi-line annotations aren't working now. Also, I tried double-clicking a word and adding an annotation, and the annotation was created, but added to the wrong location.

Odd, those both worked locally, must be an issue with my git/commit. I'll figure out when I'm back home in a couple days. Thanks
On 2015-02-15, at 12:54 PM, David Liu<EMAIL_ADDRESS>wrote:

@spring-ryanc I merged it, but looks like multi-line annotations aren't working now. Also, I tried double-clicking a word and adding an annotation, and the annotation was created, but added to the wrong location.
—
Reply to this email directly or view it on GitHub.


Configurable keyring store directory

Closes: #1541
Description

This PR adds a configuration allowing the user to specify which directory should be used for the keyring store.

PR author checklist:

[x] Added changelog entry, using unclog.
[x] Added tests: integration (for Hermes) or unit/mock tests (for modules).
[x] Linked to GitHub issue.
[ ] Updated code comments and documentation (e.g., docs/).
[ ] Tagged one reviewer who will be the one responsible for shepherding this PR.

Reviewer checklist:

[ ] Reviewed Files changed in the GitHub PR explorer.
[ ] Manually tested (in case integration/unit/mock tests are absent).

Pushed a fix to ensure the default folder is under the $HOME directory and otherwise treat the given path as absolute: 0fc4087


FEATURE: api interface

I have checked the following:

[X] I've searched existing issues and found nothing related to my issue.

Describe the feature you want to add
Is it possible to add a field for fast api body specification? in this specification it is possible to input like a coded interface and the input body in json is for this specification linted.
Mockups or Images of the feature
const interface theExample {
name: String;
anyNumber: integer;
}

{
"name" : "a name",
"anyNumber" : 5
}

Or alternatively JSON-Schema? Since we are in a NodeJS Ecosystem, this is probably easier to implement.
Maybe an option to convert fastapi specs afterwards?

Especially developers interfaces, because they writing code not json.

It is indeed a good proposal, and something like Apidog seems to be supported, which would be much more convenient.


Fix defect for matches when input is null

Expected Behavior
Actual Behavior
jDMN Version where this issue was observed
Steps to reproduce the behavior

Fixed in 8.2.1


Monitor if a recent sent message changes the Epoch Subgraph State appropriately

From a monitoring point of view, it would be interesting if the Block Oracle could validate that a sent message changed the Epoch Subgraph State in the expected ways.
If the state doesn't change accordingly, the Block Oracle could emit an alert to the maintainers of all involved components.

@tilacog revisiting this.  I'm not sure we're still interested in duplicating subgraph logic.

It's not strictly necessary, but maybe we could consider renaming this to:
Detect if the state has changed between epochs
This could be used as an alarm, given that the subgraph state is expected to change between epochs.
From a code perspective, this would be easy to implement as the Oracle keeps the previous state around, so it would be just diffing the new state against the old one.
wdyt?

I'll add a test for this!

Fixed by https://github.com/graphprotocol/block-oracle/pull/173.


Add font-family to label inside an icon fixes #4204

I would recommend wrapping the icon with a position relative container, I don't think the library should recommend nesting other components inside an icon.


cupyx.scipy.sparse.linalg.spsolve give incorrect result if RHS is non-contiguous

I think spsolve needs to enforce that the right hand side vector, b is contiguous. I came across an incorrect result when running a program where the second input to spsolve was specified as B[:, 0] which is non-contiguous when B is C-contiguous. If instead, I made a copy (B[:, 0].copy()) the result was as expected. Based on this, I suspect spsolve needs a contiguous input, but does not check to verify it.
Conditions
OS                           : Linux-5.4.0-65-generic-x86_64-with-glibc2.10
CuPy Version                 : 9.0.0b2
NumPy Version                : 1.19.5
SciPy Version                : 1.6.0
Cython Build Version         : 0.29.21
Cython Runtime Version       : 0.29.21
CUDA Root                    : /usr/local/cuda
CUDA Build Version           : 11000
CUDA Driver Version          : 11020
CUDA Runtime Version         : 11000
cuBLAS Version               : 11200
cuFFT Version                : 10201
cuRAND Version               : 10201
cuSOLVER Version             : (10, 6, 0)
cuSPARSE Version             : 11101
NVRTC Version                : (11, 0)
Thrust Version               : 100909
CUB Build Version            : 100909
Jitify Build Version         : 60e9e72
cuDNN Build Version          : None
cuDNN Version                : None
NCCL Build Version           : 2804
NCCL Runtime Version         : 2804
cuTENSOR Version             : 10202
Device 0 Name                : GeForce GTX 1080 Ti
Device 0 Compute Capability  : 61
Device 1 Name                : GeForce GTX 1070 Ti
Device 1 Compute Capability  : 61

Thanks, and sorry for the very late response. It should definitely handle the non-contiguous RHS properly. I'm going to post a fix soon.


Stringify store before sending it to devtools

Redux devtools uses structural cloning algorithm which
does not like some objects like errors. By running  the sore
through JSON.stringify we make sure the store is not going to fail
when Devtools is running it through structural cloning algorithm.

Actually, this needs to parse the string again so the store is a JavaScript object. That will allow Devtools to set its custom properties on it.
But even if I do that, it will de-ref the store properties and will break the time travel features.
I'm looking for a way to sanitize the store from things that are not serializable without modifying it too much...

So the error I got when I started using this module was that my store was failing to pass through postMessage call that Devtools uses to communicate with the extenuation.
I tried strigifying my store using JSON and got it working but  some features like time travel was not working.
Apparently MobX @action methods are not actual functions and structural cloning algorithm does not complain about them. I had a few methods in my stores that were not @actions. converting them to @actions fixed the issue for me.
But there is an issue with stores that have methods on them.
@mweststrate, doe your MobX State Tree structure work with this module?

@mohsen1, thanks for investigating it!
It should be a regression introduced by the latest changes in Redux DevTools extension. Before we had a shouldStringify parameter you could pass to force stringifying instead of using structural cloning algorithm. However, the latter wasn't as efficient as it should be. So, in the latest, we're always stringifying.
The extension uses jsan.stringify, which does exactly what you suggested here, with the difference that if JSON.stringify fails, it is still able to serialize the object.
So, when we're sending a message, before posting it should stringify the action and payload objects.
I'll look into it tomorrow and publish a patch there. If you could find the problem before that it would be more than welcome.

@mohsen1, I cannot reproduce the issue. You can try it with our todomvc example. Make sure to use the latest version of the extension.

You're right @zalmoxisus the new version of extension works great.


remove msmexplorer tests

re: https://github.com/msmexplorer/msmexplorer/issues/68

Not sure why the Travis build isn't being triggered

@jchodera: removing the import tests works. not sure how you feel about this

Huh. Sure, this sounds like a reasonable workaround for now. Thanks!


ELIFECYCLE error after repeated windowManager.getWindows() usage.

Bug Description
After calling windowManager.getWindows() repeatedly over a long period of time, RAM fills up very slowly and causes the application to crash when it reaches the limit.
The application consistently opens with an expected 5% of RAM being used, but then the percentage gradually increases until the application runs out of RAM and crashes. The entire process usually takes about an hour.
Considering the small number of packages that I am directly using, the stack trace I am getting leads me to believe that the problem is either with this package or with how I am using it.
Running with npm start --verbose.
Condensed To Possible Keywords
unsafe-perm in lifecycle true

CWD: C:\Users\NAME\Documents\GitHub\Webtop

Failed to exec start script

Exit status<PHONE_NUMBER>

at EventEmitter.<anonymous> (C:\Users\NAME\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\index.js:332:16)

at EventEmitter.emit (events.js:223:5)
at ChildProcess.<anonymous> (C:\Users\NAME\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)

at ChildProcess.emit (events.js:223:5)
at maybeClose (internal/child_process.js:1021:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
npm verb pkgid<EMAIL_ADDRESS>
node v12.14.1
npm  v6.13.7

npm timing npm Completed in 3163250ms

¯\_(ツ)_/¯Click to open the full error log.

npm verb lifecycle<EMAIL_ADDRESS>unsafe-perm in lifecycle true
npm verb lifecycle<EMAIL_ADDRESS>PATH: C:\Users\NAME\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;C:\Users\NAME\Documents\GitHub\Webtop\node_modules\.bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Razer Chroma SDK\bin;C:\Program Files\Razer Chroma SDK\bin;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Users\NAME\AppData\Local\Microsoft\WindowsApps;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Progr;C:\ProgramData\chocolatey\bin;C:\Program Files\Git\cmd;C:\ProgramData\chocolatey\lib\mpv.install\tools;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\Yarn\bin\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.4.0\;C:\tools\cuda;C:\Users\NAME\AppData\Local\Programs\Python\Python37\Scripts\;C:\Users\NAME\AppData\Local\Programs\Python\Python37\;C:\Users\NAME\Anaconda3\pkgs\openssl-1.1.1d-he774522_2\Library\bin;C:\Users\NAME\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\NAME\AppData\Local\atom\bin;C:\Users\NAME\AppData\Local\Programs\Fiddler;C:\Users\NAME\AppData\Local\Yarn\bin;C:\Users\NAME\AppData\Roaming\npm;C:\Users\NAME\AppData\Local\GitHubDesktop\bin
npm verb lifecycle<EMAIL_ADDRESS>CWD: C:\Users\NAME\Documents\GitHub\Webtop
npm info lifecycle<EMAIL_ADDRESS>Failed to exec start script
npm verb stack Error<EMAIL_ADDRESS>start: `electron .`
npm verb stack Exit status<PHONE_NUMBER>
npm verb stack     at EventEmitter.<anonymous> (C:\Users\NAME\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\index.js:332:16)
npm verb stack     at EventEmitter.emit (events.js:223:5)
npm verb stack     at ChildProcess.<anonymous> (C:\Users\NAME\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
npm verb stack     at ChildProcess.emit (events.js:223:5)
npm verb stack     at maybeClose (internal/child_process.js:1021:16)
npm verb stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
npm verb pkgid<EMAIL_ADDRESS>npm verb cwd C:\Users\NAME\Documents\GitHub\Webtop
npm verb Windows_NT 10.0.18362
npm verb argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\NAME\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "start" "--verbose"
npm verb node v12.14.1
npm verb npm  v6.13.7
npm ERR! code ELIFECYCLE
npm ERR! errno<PHONE_NUMBER>
npm ERR<EMAIL_ADDRESS>start: `electron .`
npm ERR! Exit status<PHONE_NUMBER>
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm verb exit [<PHONE_NUMBER>, true ]
npm timing npm Completed in 3163250ms

npm ERR!      C:\Users\NAME\AppData\Roaming\npm-cache\_logs\2020-03-06T21_24_24_445Z-debug.log

To Reproduce
Follow the instructions here to install and run Webtop.
Screenshots

Details

Operating System: Windows 10 (Windows_NT 10.0.18362)
node-window-manager version: 2.0.1
Last known working node-window-manager version: None

Additional Context
I'm using a very slightly modified version of node-window-manager located here in my repo that implements the fix for this issue. It is meant to be a temporary fix until the release is published to NPM.
The higher CPU usage is expected. However, I have 32GB of RAM. And only 25% RAM usage of 32GB is 8GB, which is 100% RAM usage on most standard systems. Yeah, very high.

After further digging, the memory leak issue could also be on Electron's end with desktopCapturer.
https://github.com/electron/electron/issues/21555

@KyzaGitHub Can it be reproduced using Node.js?

I tried using only node-window-manager and only desktopCapturer separately for a long period of time. I could only reproduce it with desktopCapturer.
The problem is definitely with Electron.


feat(icons): add icons-angular

Closes #61
First pass at an icons-angular implementation.

Unless rollup decides to use up all the memory again (locally the build process uses ~1.2GB of ram!) I think we're good for another review!

Thanks again for doing this @cal-smith! 🎉 Went ahead and added your name as one of the CODEOWNERS for icons-angular, let me know if there is anyone else I should add! Also, if you'd rather not have your name listed let me know and I can definitely remove it 😄