Very slow quantized tflite model

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
TensorFlow installed from (source or binary): binary
TensorFlow version (or github SHA if from source): 2.2.0

Command used to run the converter or code if you’re using the Python API
If possible, please share a link to Colab/Jupyter/any notebook.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.representative_dataset = representative_dataset_gen
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tf_lite_model = converter.convert()

with open("model.tflite", "wb") as f:
    f.write(tf_lite_model)

The output from the converter invocation
2020-06-05 10:53:29.063149: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-06-05 10:53:29.063233: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-06-05 10:53:29.080730: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: graph_to_optimize
2020-06-05 10:53:29.080748: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799]   function_optimizer: function_optimizer did nothing. time = 0.006ms.
2020-06-05 10:53:29.080752: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799]   function_optimizer: function_optimizer did nothing. time = 0ms.
2020-06-05 10:53:32.284115: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-06-05 10:53:32.284242: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-06-05 10:53:33.407982: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:797] Optimization results for grappler item: graph_to_optimize
2020-06-05 10:53:33.408011: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799]   constant_folding: Graph size after: 1092 nodes (-568), 1139 edges (-568), time = 474.12ms.
2020-06-05 10:53:33.408016: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:799]   constant_folding: Graph size after: 1092 nodes (0), 1139 edges (0), time = 213.886ms.

Also, please include a link to the saved model or GraphDef
https://drive.google.com/file/d/1imjVvw8IqQ6tvQRYaKJi_ynxQUHBBSH_/view?usp=sharing

Failure details
Before conversion, running standard keras model on CPU took ~300ms per frame. After conversion it takes ~55s.
Eventually I want to deploy the model on Coral Dev Board. Currently after compiling it for edge TPU inference takes ~4s using Coral.
Is it normal that it's so slow? I expect it to be at least not slower than before conversion.
Any other info / logs
Logs from edge tpu compiler:
Edge TPU Compiler version 2.1.302470888
Input: model.tflite
Output: model_edgetpu.tflite

Operator                       Count      Status

ADD                            1          More than one subgraph is not supported
ADD                            71         Mapped to Edge TPU
MAX_POOL_2D                    1          Mapped to Edge TPU
PAD                            35         Mapped to Edge TPU
MUL                            35         Mapped to Edge TPU
CONCATENATION                  1          More than one subgraph is not supported
QUANTIZE                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
QUANTIZE                       3          Mapped to Edge TPU
CONV_2D                        115        Mapped to Edge TPU
CONV_2D                        4          More than one subgraph is not supported
DEQUANTIZE                     1          Operation is working on an unsupported data type
RESIZE_BILINEAR                2          Operation is otherwise supported, but not mapped due to some unspecified limitation
RESIZE_BILINEAR                6          Mapped to Edge TPU
SOFTMAX                        1          Max 16000 elements supported

I'm having the exact same issue:
MobilenetV2 which I trained and quantized with the same settings as above, runs about 1.7FPS
MobilenetV2 from here https://www.tensorflow.org/lite/guide/hosted_models, runs about 7FPS

wonder what model is it?
can you use benchmark tool (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark)
to get the detailed profiling?
thanks

It's semenatic segmentation FPN with ResNet101 backbone (trained using https://github.com/qubvel/segmentation_models).
Output from the benchmark tool:
STARTING!
Min num runs: [50]
Min runs duration (seconds): [1]
Max runs duration (seconds): [150]
Inter-run delay (seconds): [-1]
Num threads: [8]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [model.tflite]
Input layers: []
Input shapes: []
Input value ranges: []
Input layer values files: []
Allow fp16 : [0]
Require full delegation : [0]
Enable op profiling: [0]
Max profiling buffer entries: [1024]
CSV File to export profiling data to: []
Max number of delegated partitions : [0]
Use gpu : [0]
Use xnnpack : [0]
Loaded model model.tflite
The input model file size (MB): 47.4346
Initialized session in 48.019ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=15908626

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=9 first=16856867 curr=16865810 min=16767506 max=17015249 avg=1.68505e+07 std=69772

Average inference timings in us: Warmup: 1.59086e+07, Init: 48019, Inference: 1.68505e+07
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Peak memory footprint (MB): init=11.707 overall=181.352

Hi Chao, can you help verify if there's regression?
thanks

It's semenatic segmentation FPN with ResNet101 backbone (trained using https://github.com/qubvel/segmentation_models).
Output from the benchmark tool:
STARTING!
Min num runs: [50]
Min runs duration (seconds): [1]
Max runs duration (seconds): [150]
Inter-run delay (seconds): [-1]
Num threads: [8]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [model.tflite]
Input layers: []
Input shapes: []
Input value ranges: []
Input layer values files: []
Allow fp16 : [0]
Require full delegation : [0]
Enable op profiling: [0]
Max profiling buffer entries: [1024]
CSV File to export profiling data to: []
Max number of delegated partitions : [0]
Use gpu : [0]
Use xnnpack : [0]
Loaded model model.tflite
The input model file size (MB): 47.4346
Initialized session in 48.019ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=15908626

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=9 first=16856867 curr=16865810 min=16767506 max=17015249 avg=1.68505e+07 std=69772

Average inference timings in us: Warmup: 1.59086e+07, Init: 48019, Inference: 1.68505e+07
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Peak memory footprint (MB): init=11.707 overall=181.352

On what HW architecture was this benchmarking performed? Coral board? And what's the compiling option did you use to compile the binary?

On the PC with i7-8650U CPU. All options default.

On the PC with i7-8650U CPU. All options default, I followed instructions from readme file.

I see. I added T.J who could give more insights here. He is much more familiar with the x86_64 optimization in the underlying math library RUY fo quant models in TFLite.
Btw, if you are ok w/ float models on x86, then you could try the new xnnpack delegate (i.e. see https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/xnnpack#enable-xnnpack-via-bazel-build-flags-recommended) that will deliver much better performance on x86.
As for perf. on Coral dev board (esp. w/ the edgetpu inside the board), could you report this issue to the Coral repo(i.e. https://github.com/google-coral/edgetpu/issues)?

Here is a detailed report of my case, please note that I'm benchmarking the cpu-tflite models, the edge compiler output is only added for additional information.
CPU: Threadripper 1920x
###My Mobilenetv2:
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
base_model.trainable = True
model = tf.keras.Sequential([
base_model,
tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(units=2, activation='softmax')
])
model.build([None, 96, 512, 3])
...
#custom tf traning loop with gradienttape
...
model.save(f"./models/model_{epoch}.hdf5")

import tensorflow.compat.v1 as tf
def representative_data_gen():
dataset_list = tf.data.Dataset.list_files(df.Fn.values)
for i in range(df.shape[0]):
image = next(iter(dataset_list))
image = tf.io.read_file(image)
image = tf.io.decode_jpeg(image, channels=3)
image = tf.cast(image, tf.float32) / 255.0
image = tf.expand_dims(image, 0)
yield [image]
converter = tf.lite.TFLiteConverter.from_keras_model_file(chkp)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()
with open('mobilenet_v2_1.0_224_quant.tflite', 'wb') as f:
f.write(tflite_model)
kriszfekete@datascience01:~/jupyter/tensorflow$ bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model   --graph=mobilenet_v2_1.0_224_quant.tflite   --num_threads=1
STARTING!
Duplicate flags: num_threads
Min num runs: [50]
Min runs duration (seconds): [1]
Max runs duration (seconds): [150]
Inter-run delay (seconds): [-1]
Num threads: [1]
Use caching: [0]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [mobilenet_v2_1.0_224_quant.tflite]
Input layers: []
Input shapes: []
Input value ranges: []
Input layer values files: []
Allow fp16 : [0]
Require full delegation : [0]
Enable op profiling: [0]
Max profiling buffer entries: [1024]
CSV File to export profiling data to: []
Enable platform-wide tracing: [0]
#threads used for CPU inference: [1]
Max number of delegated partitions : [0]
Min nodes per partition : [0]
External delegate path : []
External delegate options : []
Use gpu : [0]
Use xnnpack : [0]
Loaded model mobilenet_v2_1.0_224_quant.tflite
The input model file size (MB): 3.22496
Initialized session in 1.063ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=630344
Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=621263 curr=631745 min=603622 max=655479 avg=623294 std=14346
Inference timings in us: Init: 1063, First inference: 630344, Warmup (avg): 630344, Inference (avg): 623294
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Peak memory footprint (MB): init=2.38672 overall=5.89453

#Pretrained net from: https://www.tensorflow.org/lite/guide/hosted_models, Mobilenet_V2_1.0_224_quant
bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model \

--graph=hub_mobilenet_v2_1.0_224_quant.tflite
--num_threads=1
STARTING!
Duplicate flags: num_threads
Min num runs: [50]
Min runs duration (seconds): [1]
Max runs duration (seconds): [150]
Inter-run delay (seconds): [-1]
Num threads: [1]
Use caching: [0]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [hub_mobilenet_v2_1.0_224_quant.tflite]
Input layers: []
Input shapes: []
Input value ranges: []
Input layer values files: []
Allow fp16 : [0]
Require full delegation : [0]
Enable op profiling: [0]
Max profiling buffer entries: [1024]
CSV File to export profiling data to: []
Enable platform-wide tracing: [0]
#threads used for CPU inference: [1]
Max number of delegated partitions : [0]
Min nodes per partition : [0]
External delegate path : []
External delegate options : []
Use gpu : [0]
Use xnnpack : [0]
Loaded model hub_mobilenet_v2_1.0_224_quant.tflite
The input model file size (MB): 3.57776
Initialized session in 0.772ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=4 first=166513 curr=159202 min=156239 max=166513 avg=159732 std=4064

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=158890 curr=162784 min=156544 max=167988 avg=161145 std=1948
Inference timings in us: Init: 772, First inference: 166513, Warmup (avg): 159732, Inference (avg): 161145
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Peak memory footprint (MB): init=1.55859 overall=9.98047
#######################################################
Edge-TPU compiler outputs:
My model:
edgetpu_compiler mobilenet_v2_1.0_224_quant.tflite -s
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 478 ms.
Input model: mobilenet_v2_1.0_224_quant.tflite
Input size: 3.08MiB
Output model: mobilenet_v2_1.0_224_quant_edgetpu.tflite
Output size: 3.07MiB
On-chip memory used for caching model parameters: 3.33MiB
On-chip memory remaining for caching model parameters: 4.39MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 73
Operation log: mobilenet_v2_1.0_224_quant_edgetpu.log
Operator                       Count      Status
PAD                            5          Mapped to Edge TPU
QUANTIZE                       2          Mapped to Edge TPU
CONV_2D                        36         Mapped to Edge TPU
DEPTHWISE_CONV_2D              17         Mapped to Edge TPU
MEAN                           1          Mapped to Edge TPU
SOFTMAX                        1          Mapped to Edge TPU
FULLY_CONNECTED                1          Mapped to Edge TPU
ADD                            10         Mapped to Edge TPU

MobileNet_V2 from hosted models:
edgetpu_compiler hub_mobilenet_v2_1.0_224_quant.tflite -s
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 378 ms.
Input model: hub_mobilenet_v2_1.0_224_quant.tflite
Input size: 3.41MiB
Output model: hub_mobilenet_v2_1.0_224_quant_edgetpu.tflite
Output size: 3.88MiB
On-chip memory used for caching model parameters: 3.75MiB
On-chip memory remaining for caching model parameters: 3.16MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 65
Operation log: hub_mobilenet_v2_1.0_224_quant_edgetpu.log
Operator                       Count      Status
CONV_2D                        36         Mapped to Edge TPU
DEPTHWISE_CONV_2D              17         Mapped to Edge TPU
RESHAPE                        1          Mapped to Edge TPU
ADD                            10         Mapped to Edge TPU
AVERAGE_POOL_2D                1          Mapped to Edge TPU

Here is a detailed report of my case, please note that I'm benchmarking the cpu-tflite models, the edge compiler output is only added for additional information.
CPU: Threadripper 1920x
###My Mobilenetv2:
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
base_model.trainable = True
model = tf.keras.Sequential([
base_model,
tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(units=2, activation='softmax')
])
model.build([None, 96, 512, 3])
...
#custom tf traning loop with gradienttape
...
model.save(f"./models/model_{epoch}.hdf5")
import tensorflow.compat.v1 as tf
def representative_data_gen():
dataset_list = tf.data.Dataset.list_files(df.Fn.values)
for i in range(df.shape[0]):
image = next(iter(dataset_list))
image = tf.io.read_file(image)
image = tf.io.decode_jpeg(image, channels=3)
image = tf.cast(image, tf.float32) / 255.0
image = tf.expand_dims(image, 0)
yield [image]
converter = tf.lite.TFLiteConverter.from_keras_model_file(chkp)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()
with open('mobilenet_v2_1.0_224_quant.tflite', 'wb') as f:
f.write(tflite_model)
kriszfekete@datascience01:~/jupyter/tensorflow$ bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model --graph=mobilenet_v2_1.0_224_quant.tflite --num_threads=1
STARTING!
Duplicate flags: num_threads
Min num runs: [50]
Min runs duration (seconds): [1]
Max runs duration (seconds): [150]
Inter-run delay (seconds): [-1]
Num threads: [1]
Use caching: [0]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [mobilenet_v2_1.0_224_quant.tflite]
Input layers: []
Input shapes: []
Input value ranges: []
Input layer values files: []
Allow fp16 : [0]
Require full delegation : [0]
Enable op profiling: [0]
Max profiling buffer entries: [1024]
CSV File to export profiling data to: []
Enable platform-wide tracing: [0]
#threads used for CPU inference: [1]
Max number of delegated partitions : [0]
Min nodes per partition : [0]
External delegate path : []
External delegate options : []
Use gpu : [0]
Use xnnpack : [0]
Loaded model mobilenet_v2_1.0_224_quant.tflite
The input model file size (MB): 3.22496
Initialized session in 1.063ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=630344
Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=621263 curr=631745 min=603622 max=655479 avg=623294 std=14346
Inference timings in us: Init: 1063, First inference: 630344, Warmup (avg): 630344, Inference (avg): 623294
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Peak memory footprint (MB): init=2.38672 overall=5.89453
#Pretrained net from: https://www.tensorflow.org/lite/guide/hosted_models, Mobilenet_V2_1.0_224_quant
bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model \

--graph=hub_mobilenet_v2_1.0_224_quant.tflite
--num_threads=1
STARTING!
Duplicate flags: num_threads
Min num runs: [50]
Min runs duration (seconds): [1]
Max runs duration (seconds): [150]
Inter-run delay (seconds): [-1]
Num threads: [1]
Use caching: [0]
Benchmark name: []
Output prefix: []
Min warmup runs: [1]
Min warmup runs duration (seconds): [0.5]
Graph: [hub_mobilenet_v2_1.0_224_quant.tflite]
Input layers: []
Input shapes: []
Input value ranges: []
Input layer values files: []
Allow fp16 : [0]
Require full delegation : [0]
Enable op profiling: [0]
Max profiling buffer entries: [1024]
CSV File to export profiling data to: []
Enable platform-wide tracing: [0]
#threads used for CPU inference: [1]
Max number of delegated partitions : [0]
Min nodes per partition : [0]
External delegate path : []
External delegate options : []
Use gpu : [0]
Use xnnpack : [0]
Loaded model hub_mobilenet_v2_1.0_224_quant.tflite
The input model file size (MB): 3.57776
Initialized session in 0.772ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=4 first=166513 curr=159202 min=156239 max=166513 avg=159732 std=4064

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=158890 curr=162784 min=156544 max=167988 avg=161145 std=1948
Inference timings in us: Init: 772, First inference: 166513, Warmup (avg): 159732, Inference (avg): 161145
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Peak memory footprint (MB): init=1.55859 overall=9.98047
#######################################################
Edge-TPU compiler outputs:
My model:
edgetpu_compiler mobilenet_v2_1.0_224_quant.tflite -s
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 478 ms.
Input model: mobilenet_v2_1.0_224_quant.tflite
Input size: 3.08MiB
Output model: mobilenet_v2_1.0_224_quant_edgetpu.tflite
Output size: 3.07MiB
On-chip memory used for caching model parameters: 3.33MiB
On-chip memory remaining for caching model parameters: 4.39MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 73
Operation log: mobilenet_v2_1.0_224_quant_edgetpu.log
Operator Count Status
PAD 5 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 36 Mapped to Edge TPU
DEPTHWISE_CONV_2D 17 Mapped to Edge TPU
MEAN 1 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
FULLY_CONNECTED 1 Mapped to Edge TPU
ADD 10 Mapped to Edge TPU
MobileNet_V2 from hosted models:
edgetpu_compiler hub_mobilenet_v2_1.0_224_quant.tflite -s
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 378 ms.
Input model: hub_mobilenet_v2_1.0_224_quant.tflite
Input size: 3.41MiB
Output model: hub_mobilenet_v2_1.0_224_quant_edgetpu.tflite
Output size: 3.88MiB
On-chip memory used for caching model parameters: 3.75MiB
On-chip memory remaining for caching model parameters: 3.16MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 65
Operation log: hub_mobilenet_v2_1.0_224_quant_edgetpu.log
Operator Count Status
CONV_2D 36 Mapped to Edge TPU
DEPTHWISE_CONV_2D 17 Mapped to Edge TPU
RESHAPE 1 Mapped to Edge TPU
ADD 10 Mapped to Edge TPU
AVERAGE_POOL_2D 1 Mapped to Edge TPU
UPDATE: My coral edgetpu (usb stick) arrived, this is even weirder, my model which ran about 5* as slow (on CPU) as the one from the hosted models, is actually faster on edgetpu.
Acked. This is possible as it's possible that the quant execution path on x86-64 in TFLite hasn't been that well-optimized for that on ARM CPUs, edgetpu etc.
My model had a latency about 3.7-3.8 ms
The hosted model: 3.8-3.9 ms

Hi,
OS Linux 18.04.04 LTS
TensorFLow nightly 2.3.0-dev20200601
CPU Intel Core i7-855OU
I have a very similar issue. A trained a ResNet-50 V2 model and it takes ~ 40ms per inference on my CPU with efficiency 64.15 %. I then converted first to tf lite and speed was ~ 89 ms efficiency identical. Then with dynamic range quantization and it took ~ 548 ms per inference efficiency dropped to 54.35%. On Monday I converted it to full integer quantization with uint8 input/output and it takes ~ 7 seconds on cpu and ~40 ms on Edge TPU. Worth mentioning that besides ridiculously slow speed on CPU the full integer quantized model predicts the same value on CPU and Edge TPU all the time. Therefore, slow and broken. Does anyone have a clue how to solve this? Here you can find a link to a google folder with original mode, conversion code, converted model, and the code I use to test the tf lite model. Many thanks!
https://drive.google.com/file/d/1hNc6xCLch1T9EEqahpiT6FzIDg3P423u/view?usp=sharing

Hi all,
The model speeds up after deploying on the edgetpu (compiling from cpu tflite to edgetpu tflite) for both cases so it is the expect behavior. Since the compiler can only delegates from a fully quantized cpu tflite model, it can't do much about the original graph. It seems very odd to me that tflite model is performing much worse than the original graph model though.
It's also good to mention that I've also observed similar behavior (tiny bit difference) when testing out a yolov4 model (note that unfortunately, only 1/962 ops were mapped to edgetpu so we don't see much speed up here):
On my x86_64 debian 10:
Original model: ~55 seconds on CPU
(non quantized) tflite modes: ~5 seconds
(fully quantized) tflite model: ~56 seconds
(edgetpu) tflite model: ~55 seconds

A quick look into the model with netron, I can see many quantized/dequantized ops that I suspect is what's causing the slowdown. Again, tflite models wasn't optimized for x86_64, so I suspect this the issue.
Now let's check this again on my dev board, everything is as expected:
On my dev board:
Original model: Unfortunately cannot run this on the dev board.
(non quantized) tflite modes: ~ 27 seconds
(fully quantized) tflite model: ~ 13 seconds
(edgetpu) tflite model: ~12 seconds

My suggestion for all is to run the tflite model on arm platform since that's what tflite models are optimized for. If you are benchmarking a tflite model against a CPU graph model is not ideal.
Hope these finding is helpful!

Hi all,
The model speeds up after deploying on the edgetpu (compiling from cpu tflite to edgetpu tflite) for both cases so it is the expect behavior. Since the compiler can only delegates from a fully quantized cpu tflite model, it can't do much about the original graph. It seems very odd to me that tflite model is performing much worse than the original graph model though.
It's also good to mention that I've also observed similar behavior (tiny bit difference) when testing out a yolov4 model (note that unfortunately, only 1/962 ops were mapped to edgetpu so we don't see much speed up here):
On my x86_64 debian 10:
Original model: ~55 seconds on CPU
(non quantized) tflite modes: ~5 seconds
(fully quantized) tflite model: ~56 seconds
(edgetpu) tflite model: ~55 seconds

A quick look into the model with netron, I can see many quantized/dequantized ops that I suspect is what's causing the slowdown. Again, tflite models wasn't optimized for x86_64, so I suspect this the issue.
Now let's check this again on my dev board, everything is as expected:
On my dev board:
Original model: Unfortunately cannot run this on the dev board.
(non quantized) tflite modes: ~ 27 seconds
(fully quantized) tflite model: ~ 13 seconds
(edgetpu) tflite model: ~12 seconds

My suggestion for all is to run the tflite model on arm platform since that's what tflite models are optimized for. If you are benchmarking a tflite model against a CPU graph model is not ideal.
Hope these finding is helpful!

TFLite on x86 CPUs may not have been fully optimized for quantized models. But w/ XNNPACK delegate, as mentioned in https://github.com/tensorflow/tensorflow/issues/40183#issuecomment-640436345, will deliver significant x86 performance improvements for float models.

I've checked the same model conversion but using much smaller MobileNetV2 classification model from tf.keras.applications. I can still observe increased processing time when running tflite model on x86 cpu(700ms using tflite vs 30ms using keras model), but as you said it's normal since it's optimized for ARM. After deploying to Coral Dev Board single frame processing time is ~7ms which is even faster then using CPU on my PC.
How can I check what makes my original segmentation model so slow? Is it simply too big or it contains some operations that cannot be mapped to Edge TPU?

TFLite has a couple of built-in profilers that are available wherever you can run tflite and look at terminal output. One is enabled by passing --define=ruy_profiler=true to bazel build, or equivalently in other buildsystems just add -DRUY_PROFILER to the compiler flags. If you build the benchmark binary with that, it will dump an ascii "treeview" in the terminal with % time spent in each node.

So can we conclude that tFLite post training optimizations are not for x86 cpu?
Is there any inference optimization for x86 CPUs?

Hi,
Apologies that this issue has gone stale. Some additional x86 optimizations have landed (for AVX, AVX2, and AVX512) and they will soon be the default on x86, but aren't yet. For this issue, it would be good to know if the poor performance persists for you on x86 CPU. Can you please do as follows:

Please build with:

bazel build -c opt --define=tflite_with_ruy=true -copt=-DRUY_PROFILER

Please run the benchmark_model tool with --enable_op_profiling=true

and then post the output to this issue. Also, please provide your exact build line for any executable you are running. Thanks!

Hi,
Apologies that this issue has gone stale. Some additional x86 optimizations have landed (for AVX, AVX2, and AVX512) and they will soon be the default on x86, but aren't yet. For this issue, it would be good to know if the poor performance persists for you on x86 CPU. Can you please do as follows:

Please build with:

bazel build -c opt --define=tflite_with_ruy=true -copt=-DRUY_PROFILER

Please run the benchmark_model tool with --enable_op_profiling=true

and then post the output to this issue. Also, please provide your exact build line for any executable you are running. Thanks!

any plan on make this default?

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The TFLite team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the TFLite space.
Thanks.


ftui.config.basedir in slideout.min.js path

slideout.min.js was expected in path /js absolute to html file

already fixed


chore: Initial impl for KZG10

This provides an initial impl for some of the methods needed in Deneb, however we compute the quotient commitment using the "conventional" strategy of commiting to the quotient polynomial in monomial form.

This is related to #162 -- one can use this branch to check that the difference in using monomial form is actually negligible while it results in much simpler code


Release 1.0.0rc1

PR for updating master with release 1.0.0rc1. See if all looks good?

so I think what would be best is to go back to the state at latest develop and run the notebook script again

I think it's fine to just run that here, and add a new commit, and then just make sure to merge this with squash and merge into release-1.0.0rc1.

so I think what would be best is to go back to the state at latest develop and run the notebook script again
I think it's fine to just run that here, and add a new commit, and then just make sure to merge this with squash and merge into release-1.0.0rc1.

I am not sure what I should commit and where?

Do you also want to do

Update Changelog section header and "Full Changelog" link to point to specific version tag instead of HEAD.

in this PR? otherwise looks good 👍

@kjun9 this is what you are talking about? So this already seem to be fixed? Am I missing something?
Change Log

[1.0.0rc1](https://github.com/stellargraph/stellargraph/tree/v1.0.0rc1)

[Full Changelog](https://github.com/stellargraph/stellargraph/compare/v0.11.0...v1.0.0rc1)

@kjun9 this is what you are talking about? So this already seem to be fixed? Am I missing something?
Change Log

[1.0.0rc1](https://github.com/stellargraph/stellargraph/tree/v1.0.0rc1)

[Full Changelog](https://github.com/stellargraph/stellargraph/compare/v0.11.0...v1.0.0rc1)

Yep but as Huon pointed out, it's already been done as part of the changelog update https://github.com/stellargraph/stellargraph/pull/1287 so no need to do anything here 👍


Switch to eslint flat config

Issue
Blog post: https://eslint.org/blog/2022/08/new-config-system-part-2/
Flat config is going to be the default in eslint v9.x
https://eslint.org/blog/2023/11/whats-coming-in-eslint-9.0.0/#flat-config-now-the-default-and-has-some-changes
Docs:

Flat config: https://eslint.org/docs/latest/use/configure/configuration-files-new
Migration to flat config: https://eslint.org/docs/latest/use/configure/migration-guide
typescript-eslint flat config support: https://typescript-eslint.io/blog/announcing-typescript-eslint-v7/#new-features---flat-config-support

Description
Switch to eslint flat config
Testing
CI

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This work can be resumed when upgrading eslint to v9

Blocked on https://github.com/import-js/eslint-plugin-import/issues/2556

Also blocked on https://github.com/microsoft/vscode-eslint/issues/1644

Migrating to biome instead in https://github.com/aws/aws-sdk-js-codemod/pull/886


Fix discriminative metric

I was trying to use the discriminative metric in tsgm following evaluation.ipynb. I found in most cases, the results would be very close to 0.5. Besides, there would be a warning about using softmax in one class classification and the training loss was abnormal.
I have checked the source codes of classification models in the zoo, and I find that all models utilize softmax activation before the output layer.
 m_output = layers.Dense(self._output_dim, activation="softmax")(x)

When setting the output_dim to 1 following evaluation.ipynb for classification models, the output would be all ones, in which the classification model would not work.
# use LSTM classification model from TSGM zoo.
model = tsgm.models.zoo["clf_cl_n"](
    seq_len=Xr.shape[1], feat_dim=Xr.shape[2], output_dim=1).model # set output_dim to 1 would not work for softmax
model.compile(
    tf.keras.optimizers.Adam(),
    tf.keras.losses.CategoricalCrossentropy(from_logits=False)
)

If directly set output_dim to 2, there would be a size error for metric calculation. I think this is due to the implementation of DiscriminativeMetric:
 	y_pred = (model.predict(X_test) > 0.5).astype(int)
    if metric is None:
        return sklearn.metrics.accuracy_score(y_test, y_pred)
    else:
        return metric(y_test, y_pred)

This version is suitable for binary classification with sigmoid activation. Therefore, I recommend modifying it so that it also works with softmax.
        # check the shape, 1D array or N-D arrary
        if len(pred.shape) == 1: # binary classification with sigmoid activation
            y_pred = (pred > 0.5).astype(int)
        else: # multiple classification with softmax activation
            y_pred = np.argmax(pred, axis=-1).astype(int)
        if metric is None:
            return sklearn.metrics.accuracy_score(y_test, y_pred)
        else:
            return metric(y_test, y_pred)

Besides, the definition of the classification model should be changed as follows:
# use LSTM classification model from TSGM zoo.
model = tsgm.models.zoo["clf_cl_n"](
    # set output_dim to 2 so that softmax can work properly
    seq_len=Xr.shape[1], feat_dim=Xr.shape[2], output_dim=2).model
model.compile(
    tf.keras.optimizers.Adam(),
    # SparseCategoricalCrossentropy for multiple classes
    tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)
)

In addition, function test_discriminative_metric() in test_metrics.py should be updated:
def test_discriminative_metric():
    ts = np.sin(np.arange(10)[:, None, None] + np.arange(6)[None, :, None])  # sin_sequence, [10, 6, 3]
    D1 = tsgm.dataset.Dataset(ts, y=None)

    diff_ts = np.sin(np.arange(10)[:, None, None] + np.arange(6)[None, :, None]) + 1000  # sin_sequence, [10, 6, 3]
    D2 = tsgm.dataset.Dataset(diff_ts, y=None)
    # output_dim =2 for softmax
    model = tsgm.models.zoo["clf_cl_n"](seq_len=ts.shape[1], feat_dim=ts.shape[2], output_dim=2).model
    model.compile(
        tf.keras.optimizers.Adam(),
        # SparseCategoricalCrossentropy for multiple classes
        tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)
    )
    discr_metric = tsgm.metrics.DiscriminativeMetric()
    # should be easy to be classified
    assert discr_metric(d_hist=D1, d_syn=D2, model=model, test_size=0.2, random_seed=42, n_epochs=5) == 1.0
    assert discr_metric(d_hist=D1, d_syn=D2, model=model, metric=sklearn.metrics.precision_score, test_size=0.2, random_seed=42, n_epochs=5) == 1.0


Ability to call an initial program per connection

In RDi it's possible to call an initial program at connection time, per connection. I use this to set my job's library list (which is somewhat dynamic) as well as to set the system library to English (as I work on a German machine) and to do some other set-up work for my development environment. Such a feature would be really useful in VSCode for i too.

Interesting idea, but likely not possible. Compiles are run in a stateless fashion in their own job, and uses the user library list as set in the extension:

set the system library to English

What command do you use to do this? Perhaps we could add some support to set the 'system library'?
I would recommend you check out connection profiles if you need to switch between environments/projects on a single connection: https://halcyon-tech.github.io/vscode-ibmi/#/?id=connection-profiles

Thanks for the reply Liam
CHGSYSLIBL LIB(QSYS2924) OPTION(*ADD)

...is used to change the system library.
How about the ability to define more than one command per action? We use Turnover which creates development libraries for each task, my initial program adds any task libraries to my library list automatically for my active tasks, so I always have them in the library list. I could manually maintain the current library list, but it's just nicer not to have to think about it. The connection profiles don't seem to work for me, I set one up but switching to it has no impact on the library that's used to compile, I will try to replicate and submit a bug report it if necessary. In any case, they're not quite what I'm after.
I also use...
 ADDENVVAR  ENVVAR(RPGINCDIR) VALUE('/home/tim/Source')

.. in the initial program to set a list of include paths that are sometimes needed.
If all else fails, I can probably get around it by wrappering the compile commands.
Thanks,
Tim.

@fathert I see you're appending to the system library list! Not something anyone has asked for before. Up until now, we've only allowed the user to manage the user library list.
I am open to the idea of adding the ability to manage the system library list, but I'd love to hear ideas on how you'd expect the user (you) to do that in VS Code.

How about the ability to define more than one command per action?

I do like this idea. Perhaps we could make it so the command box in the Actions is a multi-line and each line is a separate command. Likely needs a separate issue as this is a large change.
About environment variables for Actions: another good idea. There is actually a way to set connection variables (which can be used in Actions as variables), but maybe those connection variables should have a setting on there so they are also created as environment variables when the action is run. What do you think of that?

I guess relatively few people work on a machine where the primary language isn't the one in which they wish to work 😄 but it's the way you have to do it if you want to use a secondary installed language. In terms of a solution, I think multiple commands per action would solve all of them, I could just call my CL first to set-up the environment and then that would manage everything, or add multiple commands to set the system library, env vars, etc.. I'm not convinced that it really warrants a special place in VS Code to set it.

Okay, that makes sense to me too.
Do you want to close this issue and raise another for the multiple commands in an Action?

Yeah sure! Thanks again.


Improve functional test suite

Current functional test suite is very hit or miss - very likely needs to be redone altogether.
Would probably leave kafka for last due to slow initialization times (and thus difficult to quickly iterate).
Also need to figure out how we'll test messaging systems that can't be run locally (such as GCP PubSub, SQS, Azure).
EDIT: Now that we have relay mode and plumber is expected to run without issues long-term - we must have a reliable testing story.

We can try and use the following for testing locally
Azure - https://hub.docker.com/_/microsoft-azure-storage-azurite
GCP - https://github.com/marcelcorso/gcloud-pubsub-emulator
AWS - https://hub.docker.com/r/localstack/localstack

Decided to move these tests to the package level


Store introduction-preview and use srcset

Description
Optimize image loading with responsive <picture> and srcset for better performance and accessibility.

Im not sure if we really want to sure 3 different or even more images now. The "old" solution worked well and the images where still stored on github

My problem is, when I access the page, then the image takes very long to load. And it makes no sense to have a high-res image when the display size is that small and it makes even less sense on mobile devices.

Mhh i see, Never encountered that Problems myself but i can see that it can lead to uncomfortable image loading speeds. It also may worth looking at https://squoosh.app/ which can easily compress these images


Add test cases for diagnostic reporting in InteropValidator

Description:
There has been a bug in diagnostic reporting in Interop where the diagnostic are not attached to a module and hence they were not been captured during compile time. This resulted in issue #30679 and the fix has been done with #31525. Subsequent to the fix, we need to add test cases to validate the diagnostic reporting in InteropValidator.

We are closing this issue due to inactivity. If you need further assistance or have additional input, feel free to reopen it. Thank you for helping us maintain a relevant and focused issue list!


Upgrade Aspose libaries to a recent version

❗ please do not add sensitive information in issues, you can provide extra information via email using issue number as reference ❗
Describe the issue
We have observed in production that certain files cannot be converted by the current Aspose libaries (19.1) and causing the thread to freeze/hang, when upgrading the Aspose libaries locally to 23.1, the problem is solved and the file can succesfully be converted within acceptable times.
Reporter
Laurens

After some investigation on what is mentioned in 3293, it turned out to be #3825 and a PR has been made to solve that issue.
Our new issue causes the pipeline to hang/freeze, for investigation purposes we did not restart the affected node, the thread is currently still hanging... since yesterday...


roleArn

Missing roleArn in 2.View_Campaign_And_Interactions and personalize_sample_notebook.

@yifeim can you specify where this is needed in the View campaign notebook?

also it is definitely in the sample notebook as role_arn

Could not reproduce, closing now.


Metrics for pods or nodes?

The table of metrics shows only the count of pods under the /pods namespace. Are the CPU and memory metrics under /nodes available at a pod level? It doesn't seem like any of the screenshots actually clearly show a pod under a node (they all just show Other Processes), so I'm unclear as to what would be displayed for a pod.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 00bba50e-e07b-0a1c-5ac9-4721436f73e2
Version Independent ID: 2f8ee7a7-b75d-39d9-4b02-673a6cceff2b
Content: Kubernetes monitoring with Azure Monitor for containers - Azure Monitor
Content Source: articles/azure-monitor/insights/container-insights-analyze.md
Service: azure-monitor
GitHub Login: @MGoedtel
Microsoft Alias: magoedte

@dcbrown16 Thanks for the comment. We are actively investigating on the issue and will get back to you shortly with an update.

@dcbrown16 Yes.

Assigning @MGoedtel for visibility, review, to update required screenshot(s) in the document and to provide additional insights if any.


Limit WAL instances within WAL and enforce on hydration

Stored WAL instances are limited to 5 by calls from runner start loop.
If for whatever reason runner is interrupted repeatedly before purge is called or the WAL directory has more than 5 WAL files the current implementation reads them all on hydration.
Change the implementation such that this cannot happen by enforcing the maximum number of epochs within WAL itself, may it be applied as WAL rotates or during hydration.

Kicking this out of milestone 2 as it is non-critical: for this to happen there needs to be a string of errors and even then a single progress of instance would wipe the prior data.

IMO non-issue as any Purge call will clean up older instances. Closing.


Nerd fonts + Kitty

After updating my system (I'm on unstable) and doing the following change
# Before 25.05
fonts.packages = with pkgs; [
  (nerdfonts.override { fonts = [ "FiraCode" "DroidSansMono" ]; })
];

# After 25.05
fonts.packages = with pkgs; [
  nerd-fonts.fira-code
  nerd-fonts.droid-sans-mono
];

I get the following error when trying to open Kitty (which I styled with Stylix):
fast_data_types.FreeTypeError: Failed to load face from path: /etc/profiles/per-user/r3ddy/share/fonts/truetype/NerdFonts/JetBrainsMonoNerdFont-SemiBold.ttf with error: cannot open resource

I searched a little bit and found out that the Jetbrains font folder has been moved from /etc/profiles/per-user/r3ddy/share/fonts/truetype/NerdFonts/JetBrainsMonoNerdFont-SemiBold.ttf to /etc/profiles/per-user/r3ddy/share/fonts/truetype/NerdFonts/JetBrainsMono/JetBrainsMonoNerdFont-SemiBold.ttf

I had a similar error happen to me. I just ran fc-cache -r. Sadly it's imperative, so it has to be manually ran on every system; but only once.

Closing this issue, as it seems to be a duplicate of https://github.com/danth/stylix/issues/650. Feel free to re-open, if this is not the case.


Race condition in Pool

I think I found a bug in Pool which sometimes causes unexpected PoolTimedOut errors. Here is a minimal reproducible test case:
#[tokio::test(flavor = "multi_thread")]
async fn race_condition_reproduction_case() {
    let pool: sqlx::Pool<Sqlite> = sqlx::pool::PoolOptions::new()
        .max_connections(1)
        .connect_timeout(std::time::Duration::from_secs(1))
        .connect(":memory:")
        .await
        .unwrap();

    sqlx::query("CREATE TABLE foo (name TEXT, value TEXT)")
        .execute(&pool)
        .await
        .unwrap();

    sqlx::query("CREATE TABLE bar (name TEXT, value TEXT)")
        .execute(&pool)
        .await
        .unwrap();
}

The failure is not deterministic - the test might need to be run several times before the failure is observed.
Note: I decreased the connect timeout to make the test fail faster, but the failure still occurs even with the default timeout. Also, this failure is most likely when max_connections is 1, but I think as long as the number of subsequent queries is higher than max_connections then the failure can still occur.
What happens I think is that there is a race condition. The first query acquires a connection and the releases it. But the release happens in a spawned task, so the second query might call acquire first. It gets to this point before a context switch happens and the release of the first query finishes, waking a Waiter if there is one (there are none in this case). Then context gets switched back to the acquire and it runs the poll_fn which pushes a new waiter to the queue. But it never gets woken up because the waking up already happened and eventually the call timeouts.
A simple fix which seems to work for me is to check whether idle_conns is still empty before pushing the waiter:
                future::poll_fn(|cx| -> Poll<()> {
                    if !self.idle_conns.is_empty() {
                        return Poll::Ready(());
                    }

                    let waiter = waiter.get_or_insert_with(|| Waiter::push_new(cx, &self.waiters));

                    // ...
                }),

EDIT: after a bit more testing, this doesn't seem to be a sufficient fix. The only synchronization between idle_conns.push and idle_conns.is_empty is via the waiters, but in cases like I described there are no waiters yet, so depending on timing, the idle_conns.is_empty() call might still return true and the problem persists :frowning_face:
EDIT 2: seems that moving the idle_conns.is_empty() check after the let waiter = waiter.get_or_insert(... line does the trick. That seems to eliminate the last race condition - namely when the is_empty check runs first, then release runs and notifies the waiters and finally the Waiter::push_new is called.

I opened #1320 to just redo the pool internals using futures_intrusive::Semaphore like I should've done a while ago.


probe payment as sanity check

What
I added sanity check of ProbePayment.
If the result of ProbePayment is assumed to be a payment failure, log the reason.
Also, change MinSwapAmountMsat to be able to set by policy to test stuck channels in test.
Why
LN channels can get stuck in rare.
The onchain fees were wasted because they could not be detected.
By probing payment, sanity checks can detect them.

Asking for opinions @nepet.
I think removing the previous sanity check isn't ideal. The existing balance checking current sanity check can tell us for sure that a probe would fail. The cost isn't high to probe but it is not costless. The current sanity check prevents a tiny bit of database bloat in situations where it is guaranteed to fail.
Probing seems to be the correct check after the costless sanity check. That is worthwhile to do at this step as it that prevents on-chain fee wasting.

I restored sanity check of the SpendableMsat.
Because the maximum htlc was checked more strictly than the CLN's own limit by it.

On Liquid testnet I tried L-BTC swaps:
CLN swap-out initiator:

Can confirm the probe occurs (at least I assume this is the probe):

2023-11-29T14:23:34.399Z INFO    035ca2fe4793a5e789ce846062eb4834f573c060d9200ce77544a29b48a0aa5923-chan#4: htlc 143 failed from 0th node with code 0x400f (WIRE_INCORRECT_OR_UNKNOWN_P
AYMENT_DETAILS)

The swap will complete successfully:

         "state": "State_ClaimedPreimage",

Restored max HTLC sanity check works:

{
   "code": -1,
   "message": "exceeding spendable amount_msat: 1"
}

LND swap-out initiator:

Probe occurs twice?

2023-11-29 11:51:51.466 [WRN] CRTR: Attempt 26002 for payment cf0684ff0b2ef2b71b8e3aec9fb1b70452df4a2bc0a23694e4d7ccd9a78f034c failed: IncorrectOrUnknownPaymentDetails(amt=100000000000 mSAT, height=171455)@1
2023-11-29 11:51:51.479 [INF] CRTR: Payment cf0684ff0b2ef2b71b8e3aec9fb1b70452df4a2bc0a23694e4d7ccd9a78f034c failed: final_outcome=incorrect_payment_details, raw_err=IncorrectOrUnknownPaymentDetails(amt=100000000000 mSAT, height=171455)@1
2023-11-29 11:51:51.503 [WRN] RPCS: Unable to send payment: IncorrectOrUnknownPaymentDetails(amt=100000000000 mSAT, height=171455)@1
2023-11-29 11:51:52.398 [WRN] CRTR: Attempt 26003 for payment 8088cad49156ae9c94c177180b63f77b828a8a8abc3bcde6a1b7ddd83d5835e0 failed: IncorrectOrUnknownPaymentDetails(amt=100000000000 mSAT, height=171455)@1
2023-11-29 11:51:52.410 [INF] CRTR: Payment 8088cad49156ae9c94c177180b63f77b828a8a8abc3bcde6a1b7ddd83d5835e0 failed: final_outcome=incorrect_payment_details, raw_err=IncorrectOrUnknownPaymentDetails(amt=100000000000 mSAT, height=171455)@1
2023-11-29 11:51:52.434 [WRN] RPCS: Unable to send payment: IncorrectOrUnknownPaymentDetails(amt=100000000000 mSAT, height=171455)@1

Swap is able to finish:

2023/11/29 16:53:13 [INFO] [Swap:f59b1983e3b9e63fb1d16528574bdcb9aad6a32979e0909a7cb52aa8e9cc5f10] Swap claimed with preimage eabf819c4c651d68345a5efb8f73db5e5ec7ae20ffb4b0856081755c50d65cba

Max HTLC sanity check works:

2023/11/29 12:31:39 rpc error: code = Unknown desc = exceeding spendable amount_msat: 1000

I will add another comment with mainnet test results.

On mainnet the probing / checks in this PR seem to still allow impossible swaps to occur when using a stuck channel:
"state": "State_ClaimedCoop",

"cancel_message": "could not pay invoice: timeout, last err: 204:failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 6978sat)",

More info:
$ lightning-cli peerswap-swap-out <short chan id> 1015624 lbtc

      {
         "peer_id": "redacted",
         "connected": true,
         "state": "CHANNELD_NORMAL",
         "channel_id": "redacted",
         "short_channel_id": "redacted",
         "our_amount_msat": 157147510,
         "amount_msat":<PHONE_NUMBER>0,
         "funding_txid": "redacted",
         "funding_output": 0
      }

https://github.com/ElementsProject/peerswap/pull/260#issuecomment-1832667734
Thanks for the test.
I do not know the cause of the behavior change in test and main.
Will investigate.

I think it is possible that the stuck channel may have been caused by the payment of the fee invoice.
If you could share the results of the lightning-listpeerchannels and the amount of the fee invoice payment with me, I try to find out more.

OHHH I read the description wrong. This sounds correct. I would strongly prefer that @nepet reviews this though.

Corrected the parts that were reviewed.


Refactored to use ES6 modules and bundle with rollup

As proposed in issue #119
I tried to disturb the least amount of code and kept the general structure, but some restructuring was needed to keep the interface and to avoid assigning to imported values.
Locally all tests passed.

Also, I had to increase the test timeout because the some tests timed out on my PC

@vibornoff any feedback on this? do you have intentions of merging this?

@dignifiedquire , sorry, I'm out of spare time.

@vibornoff Thanks for the quick reply. No need to be sorry. I am going to do some experiments with this branch and see if I can get things into a good state on my fork for now.

Bump

@vibornoff can we help? What do you need to be able to merge this?

@alippai , open gitter.im please


named_args_spec: add tests for unreadable formulae/casks

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?
[x] Have you successfully run brew man locally and committed any changes?

Follow-up to https://github.com/Homebrew/brew/pull/10405
I was going to include this in a different PR, but I thought I should first open this one, where I'm only adding tests. Once this is merged, the intent behind the PR I plan open will hopefully be more clear

Review period will end on 2021-01-27 at 02:48:14 UTC.

Review period will end on 2021-01-27 at 02:48:14 UTC.

Review period ended.

Review period ended.


GNB_IP is set to null leading to UERANSIM UE crash while deploying

In the Values.yaml, there is no option to set the GNB_IP and this value is set to NULL and deployment of UE crashes with below error
GNB_IP:
Launching ue: nr-ue -c ue.yaml
UERANSIM v3.2.6
terminate called after throwing an instance of 'LibError'
what():  Bad Inet address: null Invalid argument
/entrypoint.sh: line 41:    15 Aborted                 (core dumped) nr-ue -c ue.yaml $@
As a workaround, if the GNB_IP is manually editted on the configMap and then restarting the POD makes it work, setting up the GNB_IP needs to be allowed in the values.yaml file, it is currently missing...
https://github.com/Gradiant/openverso-charts/blob/master/charts/ueransim-gnb/values.yaml
ues:
enabled: false
count: 1
initialMSISDN: '0000000001'
key: 465B5CE8B199B49FAA5F0A2EE238A6BC
op: E8ED289DEBA952E4283B54E88E6183CA
opType: OPC
apnList:
- type: 'IPv4'
apn: 'internet'
slice:
sst: 1
sd: "0x111111"
emergency: false
resources:
limits: {}
requests: {}
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
podLabels: {}
podAnnotations: {}
affinity: {}
nodeSelector: {}
tolerations: []

Here in the ue.yaml file within the resources (this file is fetched while the UE configMap is created), it has a reference to the $GNB_IP for the gnbSearchList but there is no way to config this leading to UE POD crash during deployment
IMSI number of the UE. IMSI = [MCC|MNC|MSISDN] (In total 15 or 16 digits)
supi: 'imsi-{{ .Values.mcc }}{{ .Values.mnc }}{{ .Values.ues.initialMSISDN }}'
mcc: '{{ .Values.mcc }}'
mnc: '{{ .Values.mnc }}'
Permanent subscription key
key: '{{ .Values.ues.key }}'
Operator code (OP or OPC) of the UE
op: '{{ .Values.ues.op }}'
This value specifies the OP type and it can be either 'OP' or 'OPC'
opType: '{{ .Values.ues.opType }}'
Authentication Management Field (AMF) value
amf: '8000'
IMEI number of the device. It is used if no SUPI is provided
imei: '356938035643803'
IMEISV number of the device. It is used if no SUPI and IMEI is provided
imeiSv: '4370816125816151'
List of gNB IP addresses for Radio Link Simulation
gnbSearchList:

${GNB_IP}

Got to add as below in the ues config of values.yaml
gnb:
hostname: ueransim-gnb
ues:
enabled: false
count: 1
initialMSISDN: '0000000001'
key: 465B5CE8B199B49FAA5F0A2EE238A6BC
op: E8ED289DEBA952E4283B54E88E6183CA
gnb:
hostname: ueransim-gnb
opType: OPC
Also, there is an appropriate change needed in the templates "ues-deployment.yaml" file to fetch the GNB hostname from the values.yaml file
  containers:
    - name: ues
      image: {{ template "ueransim.image" . }}
      imagePullPolicy: {{ .Values.image.pullPolicy }}
      securityContext:
        privileged: true
        capabilities:
          add: ["NET_ADMIN"]
      env:
        - name: GNB_HOSTNAME
          value: {{ .Values.ues.gnb.hostname }}

Sorry, but there is a missunderstanding here.
If you want to provide a GNB_IP or GNB_HOSTNAME to the ues, you have to use the ueransim-ues chart.
The expected behavior of ueransim-gnb with ues.enabled: true is the expected.
ues are automatically configured to reach the gnb through the name of its kubernetes service (GNB_HOSTNAME). GNB_IP must be null, and the entrypoint.sh of openverso/ueransim image resolve the gnb service IP on runtime (https://github.com/Gradiant/openverso-images/blob/c723e45d45cc2e4d19454b0414c0941b05af63d8/images/ueransim/entrypoint.sh#L34).

Hi Carlos,
Without providing the gnb.hostname in the values of the ueransim-gnb chart with ues.enabled: true, the UE POD crashes as it tries to fetch the GNB_IP and it is always blank, ue.yaml files in the resources folder makes a reference to gnbsearchlist is set to ${GNB_IP} and this goes in the k8s configMap as-is (there is no IP fetched whatsoever) and the POD gets killed with a core, there were 2 ways to get this working,

Edit the k8s configMap specific to ueransim-gnb-ue --> replace ${GNB_IP} in the pod annotation with actual gnb IP and restart the ueransim-gnb-ue POD
To make it an automated way, bring in the gnb.hostname under the ues container of the ueransim-gnb chart (values.yaml) and also make appropriate changes in the templates of ues-deployment of same chart

Please, take a look to the entrypoint.sh of openverso/ueransim image.
The configMap content must be ${GNB_IP}, since the entrypoint.sh use envsubst and modify it with the content of the GNB_IP env variable. The GNB_IP env variable of the container is also created at runtime by the entrypoint.sh resolving the hostname vaule of GNB_HOSTNAME.
The GNB_HOSTNAME env variable is configured in the ues-deployment.yaml template to be the kubernetes service name of the gnb.
If you want an advance configuration and manually set the GNB_HOSTNAME or the GNB_IP, please, run ueransim-gnb with ues.enabled: false and install ueransim-ues with the manual configuration you want.


KafkaJSProtocolError: Request parameters do not satisfy the configured policy

I am getting the error "KafkaJSProtocolError: Request parameters do not satisfy the configured policy" while creating a new topic. I can successfully connect with the admin client. Here is my code. I am trying to create the topic on confluent using kafkajs.
await admin.createTopics({
                validateOnly:false,
                topics: [{
                    topic: top.name,
                    numPartitions: top.parition,
                    configEntries:[{
                        name:'cleanup.policy',
                        value:'delete'
                    }]
                }],
            });

Error Trace
{"level":"ERROR","timestamp":"2020-01-08T14:06:30.270Z","logger":"kafkajs","message":"[Connection] Response CreateTopics(key: 19, version: 2)","broker":"b13-pkc-4nym6.us-east-1.aws.confluent.cloud:9092","clientId":"kafkajs","error":"Request parameters do not satisfy the configured policy","correlationId":2,"size":79}
What config I should pass when creating the topic? I am already passing the cleanup policy. Do I need to pass any other config parameters while creating the topic?

This is not an issue with KafkaJS, but with your Kafka cluster configuration. You are trying to create a topic that violates the configured policy, as the error states. You'll have to consult whoever manages your Kafka cluster for support.
Note that you misspelled top.partition, just in case it's the number of partitions that is invalid.


[FEATURE] - Custom error message / success message for actions

Summary
Right now, successMessage /  errorMessage is a string that we setup when we define an action. However there can be multiple reasons why an action fails. It would be great to be able to return a custom success / error message from the action method itself
Basic Example
For instance, trying to call a "sendEmail" method on a non paying user could fail with a message "This user is non paying, can't send email"
Drawbacks
No response
Unresolved questions
No response

Hi, thanks for your feedback
We've just added these features, you can now use the previous system to display messages, and a new one based on the return (MessageData) of the action function or thrown error during its execution. Dialog actions can now also display a message on their closing by using a MessageData object and passing it to the onClose function in the component.
The new v6.1.0 is available, feel free to give another feedback 🚀


Create Features Page

The new website needs a page to describe the features of Organic Maps, including summary and details, primary features and smaller secondary features.
Criteria

Create a page
Implement with design (link will be provided when design is complete)
Navigation integrates with the other pages
All design elements and standards of the website apply to this page

Can we reuse the text and screenshots for features in/from App Stores?

There is definitely content we can use from there to create initial designs, but after the design is complete, it's more likely that the app stores will need to be updated with the more current content.


Deprecation of React.createClass

Hi, after upgrading React I get this warning:
Warning: Formsy: React.createClass is deprecated and will be removed in version 16. Use plain JavaScript classes instead. If you're not yet ready to migrate, create-react-class is available on npm as a drop-in replacement.
Is there any plans to start using class Formsy extends Component?

@christianalfoni I'm seeing this issue, too:
Warning: DropdownTrigger: React.createClass is deprecated and will be removed in version 16. Use plain JavaScript classes instead. If you're not yet ready to migrate, create-react-class is available on npm as a drop-in replacement.

I came here to open a PR for this, but there are 21 pull requests currently open. I’m beginning to think it’s going to be a while.

I think this repo is dead. 😢

Anything we can do about it?

this is a better approach https://github.com/jaredpalmer/formik for forms

This issue has been moved to the new formsy-react organization: https://github.com/formsy-react/formsy-react/issues/2

This issue has been SOLVED in the new formsy organization with Formsy 1.0.1: https://github.com/formsy/formsy-react/issues/2


Separate Prototype Kit Docs from the Prototype Kit

What
Separate the Prototype Kit Docs from the the Prototype Kit
Why

Give users a better docs site with proper navigation and search
Simplifies the Prototype Kit codebase, making it easier to maintain and develop
Putting the docs in a more standard codebase all in Markdown makes them easier to maintain and develop

Done when

[x] Proof of concept of Prototype Kit docs based on the Design System
[x] Cleaned up version of separate Prototype Kit docs
[ ] Remove docs from Prototype Kit
[ ] Add page templates back into Prototype Kit
[ ] Investigate hosting working examples in the Prototype Kit
[ ] Get a new URL for the docs
[ ] Redirect from old docs to new
[ ] Create deploy process for new Prototype Kit Docs
[ ] Publish new docs
[ ] Release new Prototype Kit with docs removed

Hi @joelanman, are you able to share the proof of concept for the Prototype Kit docs? 👀

@davidhunter08 I'm having problems running it on heroku, so the easiest way for you to see it is to run it locally from cloning this repo: https://github.com/alphagov/govuk-prototype-kit-docs

closing this as we have a different approach

We might want to have a kickoff before we get back into this.
There are a couple of questions we should agree on:

Can we have a docs freeze while we're doing this work?
Where should the templates live? Do we need a separate repo?
How do we manage releases and versions? Do we need to find a way to match docs with kit version now, or can that wait?

I've removed the following tasks as being out of scope:

Investigate hosting working examples in the Prototype Kit
Get a new URL for the docs
Redirect from old docs to new

I agree that they would be good to do, but we can do them after we've split the docs out. I'd also like to look at hosting on PaaS, and using a service domain, but again I think those can wait.

@joelanman suggested we might need to publish the templates as a separate package, so that we can reduce duplication. I'd suggest that for now we keep the templates duplicated, in the same place as before, and then look at solutions.
For instance, pulling them into the docs app from the prototype kit release archive.
I'm open to other approaches, but I'm going to provisionally remove Add page templates back into Prototype Kit from the tasks for this epic.

I'm not fully against it, but I am concerned that leaving templates in /docs/views/templates, and it being the only folder left in that tree might be a confusing experience for users. We could move them to /templates and update documentation. We then have our recurring issue of what to do when documentation depends on Prototype Kit version.

I'm not fully against it, but I am concerned that leaving templates in /docs/views/templates, and it being the only folder left in that tree might be a confusing experience for users. We could move them to /templates and update documentation. We then have our recurring issue of what to do when documentation depends on Prototype Kit version.

Hmm, good point. I thought it would be less confusing though than moving the templates for users who are used to its currently location? What do you think.

This needs to go in the changelog, I will update that.


Why decorators instead of JSON keys?

What is the advantage of using text decorators over JSON keys?

decorators are designed to be used raw, and to be intentionally unintuitive for regular users who don't want advanced features, so they don't get confused by them and make too complex editors. and for advanced users who are willing to use the advanced features, they would be able to use them because they have already tried other intuitive ways to do the same thing and failed. so the decorators are the last option for them to try.
This may vary depending on the implementation of each application, but it can at least lead to this format.


Notify focused client on changed selection

This fixes a data.c bug. Currently, it does not send notifications to the focused client when a selection changes which results in a segmentation fault if the clients tries then to receive from the (no longer valid) data offer. Usually clients don't use the wayland protocol when they want to access their own data source but manage it internally which is why this did not introduce any real problems until now, but would e.g. when implementing Sircmpwn/sway#926 (since then the source is changed without that the client can know it).

This looks fine, but I'm not convinced it'll solve https://github.com/SirCmpwn/sway/issues/926. CLI clients (i.e. clients that don't open a window) aren't going to get focus.

I fixed the sway issue with a new ipc message (currently preparing the pr).

Cool, thanks!


Debug TtlUpdateManagerTest failure.

TtlUpdateManagerTest is failing but couldn't duplicate locally.

Since we couldn't duplicate the issue locally, used this PR to debug the TtlUpdateManagerTest failure.
Before TtlUpdateManagerTest run, the router.getOperationsCount was not zero either. Seems like it was caused by com.github.ambry.router.BatchOperationCallbackTrackerTest > testOperationCompleteAfterFinalBlob
JUnit tests can influence each other if we use mutable static variables.

Closing this debugging PR. Will be related PR after.


Update getting-started-with-windbg.md

made file-path to bold
made windbg.exe to bold

article link https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/getting-started-with-windbg
after adjustment, here is the output

@RAJU2529 - These edits look fine, thanks.


Feedback

Here's my first pass. There are some comments within brackets. And I also made some suggested changes directly in the text.

I think I'll just merge this and then address all the feedback in a future commit.


VIP Availability and DIP Availability there aren't

Both terms don't show in drop-down list. I see Frontend IP address or Backend Ip Address. In metrics drop-down list, there aren't any VIP Availability or DIP Availability references. Can you correct the doc, please?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fabcb1d6-d6df-23a8-6258-d8cc3600804c
Version Independent ID: 3a6cab2e-d26b-3a60-f9ef-ad85c5324206
Content: Azure Standard Load Balancer diagnostics
Content Source: articles/load-balancer/load-balancer-standard-diagnostics.md
Service: load-balancer
GitHub Login: @KumudD
Microsoft Alias: Kumud

@redaniel74, Thanks for the feedback! I have assigned the issue to the content author to review further and update the document as appropriate.

Those are called "Data Path Availability" and "Health Probe Status".

@redaniel74 Thanks for bringing this to our attention! I have updated the terminology and images.
#please-close


BModal header's close button not updating v-model value

This issue mentioned in https://github.com/cdmoro/bootstrap-vue-3/pull/658
He was correct in the second comment https://github.com/cdmoro/bootstrap-vue-3/pull/658#issuecomment-1245293371
The close event should be fired through vue's click handler, Events should never depend on Bootstrap's event callbacks in such matter since we can't fully control the events coming from Bootstrap
BEGIN_COMMIT_OVERRIDE
fix(BModal): fixed the modal's close button located in the header where the button click was not updating v-model value
END_COMMIT_OVERRIDE

Hey there
Just realized there still another case, where the binding is not updated: #674


Hiding conda environments

Is there a way to hide conda environments so they are not visible or discoverable via the notebook?
A common practice is to setup one or more base conda environment and then expose other environments.   It is desired to hide these base conda environments from users of the nb_conda_kernel extension.  Is this possible?

I'd like to see if there is something like this as well. I like having conda environments for my users picked up, but I create the kernel specs manually for site provided kernels and when I do that they're duplicated. I'm looking into this for the time being

Documentation says there is Regex env_filter:

Configuration
This package introduces two additional configuration options:
env_filter: Regex to filter environment path matching it. Default: None (i.e. no filter)

It should be possible to setup a negative Regex, but I don't know where this setting (env_filter) is supposed to be located and how to edit it, eh.
Something like this (selects text with strawberry or green, but rejects text with poison):
https://regex101.com/r/PKmouO/3/

There's a mini-guide here how to stop nb_conda_kernels spam https://github.com/jupyterhub/jupyterhub/issues/715#issuecomment-463756411, but it doesn't work for me.
I see double kernels of everything:

Tried setting filter like so:
c.NotebookApp.kernel_spec_manager_class = 'nb_conda_kernels.CondaKernelSpecManager'
c.CondaKernelSpecManager.env_filter = '(?:.*root.*|.*base.*)'

How do I either filter out [conda env:root] kernels from the list, or disable nb_conda_kernels entirely?
I build my image in Docker, nb_conda_kernels gets installed as a dependency and doing conda uninstall nb_conda_kernels causes conda do do hour+ long rebuild.

@Zireael So I found what's needed a bit ago, but never updated this issue. Anyways, for me the issue is solved by basically specifying the file system paths where the kernels are stored. Also, this is done in the $JUPYTER_PREFIX/etc/jupyter/jupyter_notebook_config.json file. Mine looks like the following:
{
  "NotebookApp": {
    "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
  },
  "CondaKernelSpecManager": {
    "env_filter": "/ncar/usr/jupyterhub/*|/glade/u/ssg/*"
   }
}

And that negated any kernels that I store under the directory trees of /ncar/usr/jupyterhub or /glade/u/ssg/ and my duplicates are now gone, but conda environments still are picked up for my users so they don't need to write kernel specs.

closing this one as stale; there is an env_filter but I wouldn't be surprised if it needs improving


Application Insights: Add Favorites and Web Test Locations APIs

Web Test Locations API:
Application Insights has web tests already defined in their REST API, but the locations for web tests were not discoverable for users at the time we wrote that spec. This PR brings the web test location query to our customers to make this information available.
Favorites API:
Application Insights components contain many reportable items and each of those items are highly configurable. We've given our users the ability to store their commonly used queries and reports as favorites within the Application Insights component. This PR exposes the management of these favorites to the REST API.
Misc:
Correct some minor testing failures with webTest and components APIs.
PR information

[x] The title of the PR is clear and informative.
[x] There are a small number of commits...
[x] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[x] If applicable, the PR references the bug/issue that it fixes.
[x] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[x] I have read the contribution guidelines.
[x] My spec meets the review criteria:

[x] The spec conforms to the Swagger 2.0 specification.
[x] The spec follows the guidelines described in the Swagger checklist document.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Did a commit to Azure/azure-sdk-for-python:
https://github.com/Azure/azure-sdk-for-python/commit/c184e9cc61828b665753770cdc9190346f5490fd

Did a commit to Azure/azure-sdk-for-go:
https://github.com/Azure/azure-sdk-for-go/commit/f05b00fd905c2be3ce125760c2eba627684fa9b5

@ravbhatnagar new APIs

Lets set up a call to understand this and see if would be correct to fix them considering these are existing.
Please set up a call and loop in folks from Azure API review board as well - @johanste  @devigned  @NiklasGustafsson. Also please bucket all the AI APIs that have similar issues and we can discuss them together. I think there is another PR for AI which have these issues.

@ravbhatnagar agreed, I would like to ensure that we get this resolved very soon too! I'll reach out via Outlook and try and get a meeting in place shortly.

Signing off from ARM side. Although there are compliance issues here, these APIs have been in prod for several years. Ongoing conversation with AI team on whether its possible to fix these in a new api-version.

@AutorestCI rebuild azure-sdk-for-python

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

Completed my updates. Could I get a review sign off from @johanste , @devigned , or @NiklasGustafsson ?
Note that I added 3 suppressions for noisy warnings from files not from my PR. (description/title missing and response schema - all warnings for redundant or false negative failures).
Thanks!

@olydis any other feedback you'd like to impart prior to merging?


Restart of node led to an inconsistent stance

After restarting the node while dbsync was rrunning i ran into this issue:
cardano-db-sync_1  | [db-sync-node:Info:164] [2021-09-02 18:43:27.56 UTC] File /var/lib/cdbsync/38620796-7ad5a7da06-286.lstate exists
postgres_1         | 2021-09-02 18:43:27.565 UTC [88] ERROR:  duplicate key value violates unique constraint "block_pkey"
postgres_1         | 2021-09-02 18:43:27.565 UTC [88] DETAIL:  Key (id)=(6183371) already exists.
postgres_1         | 2021-09-02 18:43:27.565 UTC [88] STATEMENT:  INSERT INTO "block"("hash","epoch_no","slot_no","epoch_slot_no","block_no","previous_id","slot_leader_id","size","time","tx_count","proto_major","proto_minor","vrf_key","op_cert","op_cert_counter") VALUES('\x0cdaf4109c3c80149098e943efbc94538094905cbe22f2a7bc6276cac582602a',287,38620939,139,6172664,6241587,5813210,46405,'2021-08-28 21:47:10Z',74,4,0,'vrf_vk1hrs5cxxznqmc6tj7tk9hhhl0fx50dewf0nzwjwegngs7yrlcw7dsrp0w7a','\x51bcf0ab2378ccd9318eef4c370f71d6aedc12eed44bec6124b4f876d20860cd',1) RETURNING "id"
cardano-db-sync_1  | [db-sync-node:Error:164] [2021-09-02 18:43:27.56 UTC] runDBThread: DbInsertException "Block" (SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"block_pkey\"", sqlErrorDetail = "Key (id)=(6183371) already exists.", sqlErrorHint = ""})
cardano-db-sync_1  | [db-sync-node:Error:167] [2021-09-02 18:43:27.56 UTC] recvMsgRollForward: AsyncCancelled
cardano-db-sync_1  | [db-sync-node:Error:159] [2021-09-02 18:43:27.56 UTC] ChainSyncWithBlocksPtcl: DbInsertException "Block" (SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"block_pkey\"", sqlErrorDetail = "Key (id)=(6183371) already exists.", sqlErrorHint = ""})
cardano-db-sync_1  | [db-sync-node.Mux:Info:156] [2021-09-02 18:43:27.56 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Dead
cardano-db-sync_1  | [db-sync-node.Mux:Notice:156] [2021-09-02 18:43:27.56 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Miniprotocol MiniProtocolNum 5 InitiatorDir terminated with exception DbInsertException "Block" (SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"block_pkey\"", sqlErrorDetail = "Key (id)=(6183371) already exists.", sqlErrorHint = ""})
cardano-db-sync_1  | [db-sync-node.Subscription:Error:155] [2021-09-02 18:43:27.56 UTC] Identity Application Exception: LocalAddress "/node-ipc/node.socket" DbInsertException "Block" (SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"block_pkey\"", sqlErrorDetail = "Key (id)=(6183371) already exists.", sqlErrorHint = ""})
cardano-db-sync_1  | [db-sync-node.ErrorPolicy:Error:6] [2021-09-02 18:43:27.56 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicyUnhandledApplicationException (DbInsertException "Block" (SqlError {sqlState = "23505", sqlExecStatus = FatalError, sqlErrorMsg = "duplicate key value violates unique constraint \"block_pkey\"", sqlErrorDetail = "Key (id)=(6183371) already exists.", sqlErrorHint = ""}))
cardano-db-sync_1  | [db-sync-node.Subscription:Notice:64] [2021-09-02 18:43:28.56 UTC] Identity Restarting Subscription after 16.485109515s desired valency 1 current valency 0
cardano-db-sync_1  | [db-sync-node.Subscription:Notice:64] [2021-09-02 18:43:28.56 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync_1  | [db-sync-node.Subscription:Notice:186] [2021-09-02 18:43:28.56 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync_1  | [db-sync-node.Subscription:Notice:186] [2021-09-02 18:43:28.56 UTC] Identity Connection Attempt End, destination LocalAddress "/node-ipc/node.socket" outcome: ConnectSuccessLast
cardano-db-sync_1  | [db-sync-node.Handshake:Info:186] [2021-09-02 18:43:28.56 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073),(NodeToClientV_4,TInt 764824073),(NodeToClientV_5,TInt 764824073),(NodeToClientV_6,TInt 764824073),(NodeToClientV_7,TInt 764824073),(NodeToClientV_8,TInt 764824073),(NodeToClientV_9,TInt 764824073)]))
cardano-db-sync_1  | [db-sync-node.Handshake:Info:186] [2021-09-02 18:43:28.56 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_9 (TInt 764824073))
cardano-db-sync_1  | [db-sync-node.Mux:Info:186] [2021-09-02 18:43:28.56 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Handshake Client end, duration 0.000200323s
cardano-db-sync_1  | [db-sync-node.Mux:Info:187] [2021-09-02 18:43:28.56 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Mature
cardano-db-sync_1  | [db-sync-node:Info:190] [2021-09-02 18:43:28.56 UTC] Starting chainSyncClient
cardano-db-sync_1  | [db-sync-node:Info:190] [2021-09-02 18:43:28.57 UTC] Cardano.Db tip is at slot 38620796, block 6172663
cardano-db-sync_1  | [db-sync-node:Info:195] [2021-09-02 18:43:28.57 UTC] Running DB thread
cardano-db-sync_1  | [db-sync-node:Info:195] [2021-09-02 18:43:28.58 UTC] Rolling back to slot 38620796, hash 7ad5a7da06eeb5a15ddac5b437afcab5d696d1561f2e7549fbe9a6a3adb762e1
cardano-db-sync_1  | [db-sync-node:Info:195] [2021-09-02 18:43:28.58 UTC] No blocks need to be deleted
cardano-db-sync_1  | [db-sync-node:Info:195] [2021-09-02 18:43:28.58 UTC] Found in memory ledger snapshot at slot 38620796, hash 7ad5a7da06eeb5a15ddac5b437afcab5d696d1561f2e7549fbe9a6a3adb762e1
cardano-db-sync_1  | [db-sync-node.Subscription:Notice:64] [2021-09-02 18:43:28.59 UTC] Identity Required subscriptions started

Restart of db-sync does not help

ok mybad i spun up a second db-sync and it connected to the same db and screwed up everything.. maybe there could be a locking mechanism in future?

@cardano-tools-nft Why was this re-openned? If you want to locking mechanism, please open that as a separate issue and close this.


Sort sites in Manifest (and rename two files for readability)

Note: should only be merged after #225 is pulled and I have fixed the conflicts due to it, I've decided that it would be best to keep this PR open in the meantime!

Please, resolve conflict and I'll merge it.

Done, everything should be good to go!  ;)


Toggling legend items with enter generates different events than with mouse clicks

Create a chart with multiple groups, the a11y-legend plugin, and with this test plugin:
{
  id: "test",
  afterDatasetUpdate: (chart, args, options) => {
    console.log("dataset updated. Mode: ", args.mode);
  }
}

Click a legend item for one of the groups. You'll see this console message:
dataset updated. Mode: hide

Tab to that legend item and press spacebar (or enter). You'll see this console message:
dataset updated. Mode: undefined

This is inconsistent behavior between the mouse events and the keyboard events.

Fixed: https://github.com/julianna-langston/chartjs-plugin-a11y-legend/pull/11


ambient: app pod's readinessProbe cannot work when ambient is installed via manifests

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

When ambient is installed via istioctl manifest generate, app pod's readinessProbe cannot work.
istioctl install works so it is a problem of istioctl manifest generate.
I know that istioctl manifest generate is not recommended but we need to use it - e.g. https://github.com/knative/serving/pull/14103

step to reproduce
1. Deploy Istio w/ ambient via istioctl manifest generate.
$ kubectl create ns istio-system
$ kubectl label namespace default istio.io/dataplane-mode=ambient

$ istioctl manifest generate --set profile=ambient  | kubectl apply -f -

$ sleep 3
$ kubectl -n istio-system wait --timeout=300s --for=condition=Available deploy --all

2. Deploy httpbin with readinessProbe

httpbin is from samples/httpbin/httpbin.yaml with readinessProbe

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kong/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
        readinessProbe:
          periodSeconds: 1
          httpGet:
            scheme: HTTP
            port: 80
EOF

3. readinessProbe never success
$ kubectl get pod
NAME                       READY   STATUS    RESTARTS   AGE
httpbin-849568d64c-8ljsn   0/1     Running   0          2m3s

$ kubectl describe pod httpbin-849568d64c-8ljsn
   ... snip ...
  Normal   Created    97s                 kubelet            Created container httpbin
  Normal   Started    97s                 kubelet            Started container httpbin
  Warning  Unhealthy  97s                 kubelet            Readiness probe failed: Get "http://<IP_ADDRESS>:80/": read tcp <IP_ADDRESS>:52974-><IP_ADDRESS>:80: read: connection reset by peer
  Warning  Unhealthy  77s (x20 over 96s)  kubelet            Readiness probe failed: Get "http://<IP_ADDRESS>:80/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

When namespace does not have istio.io/dataplane-mode=ambient, the readinessProbe works so ambient makes this failure.
Again, istioctl install works so istioctl manifest generate makes the failure.

Version
$ istioctl version
client version: 1.19-alpha.fd63c3e0ec2dbb38667466d68cc8a075d39ac7da
control plane version: 1.19-alpha.fd63c3e0ec2dbb38667466d68cc8a075d39ac7da
data plane version: 1.19-alpha.fd63c3e0ec2dbb38667466d68cc8a075d39ac7da (3 proxies)

### Additional Information

_No response_

### Affected product area

- [X] Ambient
- [ ] Docs
- [X] Installation
- [ ] Networking
- [ ] Performance and Scalability
- [ ] Extensions and Telemetry
- [ ] Security
- [ ] Test and Release
- [ ] User Experience
- [ ] Developer Infrastructure
- [ ] Upgrade
- [ ] Multi Cluster
- [ ] Virtual Machine
- [ ] Control Plane Revisions

I cannot reproduce this in my environment. Ambient traffic works fine with the istioctl manifest generate components. Including the bug report or some components' logs may be helpful.

Ambient traffic works fine with the istioctl manifest generate components

The issue is not normal traffic but probe traffic. And our CI and I can produce the issue when the pod is deployed in the order istio-cni-node -> ztunnel.

Including the bug report or some components' logs may be helpful.

What logs do you want to see, istio- cni or ztunnel? Ambient prints a lot of false postive error...