Not able to get TFlite Model Maker Object Detector to produce expected output

1. System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): WSL Linux Ubuntu 20.04 (although I've also tried on Linux Ubuntu 22.04)
TensorFlow installation (pip package or built from source): Pip package (t.
TensorFlow library (version, if pip package or github SHA, if built from source):
I built it with Conda and Python 3.9. Basically I install the environment in a Jupyter kernel and then use it to install the packages and eventually run the script.

# Install Miniconda
!apt install wget
!wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh
!chmod +x Miniconda3-py37_4.9.2-Linux-x86_64.sh
!bash ./Miniconda3-py37_4.9.2-Linux-x86_64.sh -b -f -p /usr/local

# Update Conda
!conda update -n base -c defaults conda -y

# #Create a Python 3.9 environment
!conda create --name py39_environment python=3.9 -y

# # Initialize shell for Conda
!conda init bash

# # Activate the environment and check Python version
!source activate py39_environment && python --version
!source /usr/local/etc/profile.d/conda.sh && conda activate py39_environment && pip install tflite_model_maker
!source /usr/local/etc/profile.d/conda.sh && conda activate py39_environment && pip install -q pycocotools
!source /usr/local/etc/profile.d/conda.sh && conda activate py39_environment && pip install opencv-python-headless
!source /usr/local/etc/profile.d/conda.sh && conda activate py39_environment && pip uninstall -y tensorflow && pip install -q tensorflow==2.8.0
!source /usr/local/etc/profile.d/conda.sh && conda activate py39_environment && pip install ipykernel
!source /usr/local/etc/profile.d/conda.sh && conda activate py39_environment && pip install --upgrade numba llvmlite

2. Code
Provide code to help us reproduce your issues using one of the following options:
train.py:
import numpy as np
import os

np.object = object
np.bool = bool
np.complex = complex

from tflite_model_maker.config import ExportFormat
from tflite_model_maker import model_spec
from tflite_model_maker import object_detector

import tensorflow as tf
assert tf.__version__.startswith('2')

tf.get_logger().setLevel('ERROR')
from absl import logging
logging.set_verbosity(logging.ERROR)

label_map = {1: 'Plastic', 2: 'Trash'}

train_images_dir = '/trashnet-training/pascal_voc/train/images'
train_annotations_dir = '/trashnet-training/pascal_voc/train/Annotations'
val_images_dir = '/trashnet-training/pascal_voc/valid/images/'
val_annotations_dir = '/trashnet-training/pascal_voc/valid/Annotations'
# test_images_dir =  '/trashnet-training/pascal_voc/train/Annotations'
# test_annotations_dir = '/trashnet-training/pascal_voc/train/Annotations'

train_data = object_detector.DataLoader.from_pascal_voc(
    train_images_dir, train_annotations_dir, label_map=label_map)

validation_data = object_detector.DataLoader.from_pascal_voc(
    val_images_dir, val_annotations_dir, label_map=label_map)

# test_data = object_detector.DataLoader.from_pascal_voc(
#     test_images_dir, test_annotations_dir, label_map=label_map)

spec = model_spec.get('efficientdet_lite0')

model = object_detector.create(train_data=train_data,
                               model_spec=spec,
                               validation_data=validation_data,
                               epochs=50,
                               batch_size=10,
                               train_whole_model=True)

model.evaluate(validation_data)

TFLITE_FILENAME = 'trained_model.tflite'
LABELS_FILENAME = 'model-labels.txt'

model.export(export_dir='.', tflite_filename=TFLITE_FILENAME, label_filename=LABELS_FILENAME,
             export_format=[ExportFormat.TFLITE, ExportFormat.LABEL])

test_model.py:
import tensorflow as tf
import numpy as np

interpreter = tf.lite.Interpreter(model_path="trained_model.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.invoke()

output_data = interpreter.get_tensor(output_details[0]['index'])
input_data = interpreter.get_tensor(input_details[0]['index'])
print(f"Input tensor: {input_data}")
print(f"Output tensor: {output_data}")

tf.lite.experimental.Analyzer.analyze(model_path="trained_model.tflite")

My dataset is in the Pascal VOC format with train and valid subfolders, each with their own images/ and Annotations/ folders. I've been able to train the model several times but whenever I put the resulting .tflite into the testing script, it doesn't product the shape it's supposed to. According to the TF documentation for the TFLite Object Detector export_tflite function here the output shape should be:
Four Outputs:
  detection_boxes: a float32 tensor of shape [1, num_boxes, 4] with box
    locations.
  detection_classes: a float32 tensor of shape [1, num_boxes] with class
    indices.
  detection_scores: a float32 tensor of shape [1, num_boxes] with class
    scores.
  num_boxes: a float32 tensor of size 1 containing the number of detected
    boxes.

3. Failure after conversion
The training seems to go fine, although somewhere along the way it warns me that:
2023-10-17 04:41:28.736735: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'resample_p7/PartitionedCall' has 1 outputs but the _output_shapes attribute specifies shapes for 3 outputs. Output shapes may be inaccurate.
Which is concerning. And I think part of the problem because when I run my testing script on the model the output tensors are all blank:
...
d_4;class_net/class-predict/bias11) shape:[1, 3, 3, 18], type:INT8
  T#583(Reshape) shape:[1, 14400, 2], type:INT8
  T#584(Reshape_1) shape:[1, 14400, 4], type:INT8
  T#585(Reshape_2) shape:[1, 3600, 2], type:INT8
  T#586(Reshape_3) shape:[1, 3600, 4], type:INT8
  T#587(Reshape_4) shape:[1, 900, 2], type:INT8
  T#588(Reshape_5) shape:[1, 900, 4], type:INT8
  T#589(Reshape_6) shape:[1, 225, 2], type:INT8
  T#590(Reshape_7) shape:[1, 225, 4], type:INT8
  T#591(Reshape_8) shape:[1, 81, 2], type:INT8
  T#592(concat) shape:[1, 19206, 2], type:INT8
  T#593(Sigmoid) shape:[1, 19206, 2], type:INT8
  T#594(Sigmoid1) shape:[1, 19206, 2], type:FLOAT32
  T#595(Reshape_9) shape:[1, 81, 4], type:INT8
  T#596(concat_1) shape:[1, 19206, 4], type:INT8
  T#597(tfl.dequantize) shape:[1, 19206, 4], type:FLOAT32
  T#598(StatefulPartitionedCall:3) shape:[], type:FLOAT32
  T#599(StatefulPartitionedCall:2) shape:[], type:FLOAT32
  T#600(StatefulPartitionedCall:1) shape:[], type:FLOAT32
  T#601(StatefulPartitionedCall:0) shape:[], type:FLOAT32

---------------------------------------------------------------
Your TFLite model has ‘1’ signature_def(s).

Signature#0 key: 'serving_default'
- Subgraph: Subgraph#0
- Inputs:
    'images' : T#0
- Outputs:
    'output_0' : T#601
    'output_1' : T#600
    'output_2' : T#599
    'output_3' : T#598

---------------------------------------------------------------
              Model size:    4444720 bytes
    Non-data buffer size:     799672 bytes (17.99 %)
  Total data buffer size:    3645048 bytes (82.01 %)
    (Zero value buffers):        256 bytes (00.01 %)

I've trained it several times with different formats but I just can't get the right output shapes, every time they're blank Could someone help me to understand why this is not working? Any guidance or insight would be greatly appreciated.

Hi @jansdhillon
The output shapes for the tflite model can be observed using interpreter.get_output_details(). The outputs might be ordered and need to be accessed accordingly.
Could you please share a toy tflite model in order to debug the issue?
Thanks.

@jansdhillon Were you able to solve this and could please post the solution to this issue? I am having the same problem at the moment

Hi @jansdhillon
The output shapes for the tflite model can be observed using interpreter.get_output_details(). The outputs might be ordered and need to be accessed accordingly.
Could you please share a toy tflite model in order to debug the issue?
Thanks.

I did run that, which you can see the output of in the last block of code. They all have a shape of [].

@jansdhillon Were you able to solve this and could please post the solution to this issue? I am having the same problem at the moment

Unfortunately, no I was not. I was trying to use this for a project but wasn't able to get it working in time so I had to pivot.

Alright thanks for responding I will try to reproduce! I have run model maker on two different datasets. With one the inference afer conversion works just fine. With the other (the current one) I got the same results as you. Maybe I can find out the reason

I was not able to reproduce it again. I am also using WSL with Ubuntu 22.04. I tested the same model as you with model maker and this time loading worked. However, when exporting the model to Tflite I also got this
2023-11-13 00:52:04.885225: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'resample_p7/PartitionedCall' has 1 outputs but the _output_shapes attribute specifies shapes for 3 outputs. Output shapes may be inaccurate.
But it did not matter. Loading the model and inferring it worked fine afterwards. I was able to access to output tensors like this:
` python
signature_fn = interpreter.get_signature_runner()
Feed the input image to the model
output = signature_fn(images=image)
Get all outputs from the model
count = int(np.squeeze(output['output_0']))
scores = np.squeeze(output['output_1'])
classes = np.squeeze(output['output_2'])
boxes = np.squeeze(output['output_3'])
`

Hi @jansdhillon
As mentioned @alexw92 , use the signature function to get the output details. You can add the following code to the interpreter. This information can be found in the documentation as well.
def detect_objects(interpreter, image, threshold):
  """Returns a list of detection results, each a dictionary of object info."""

  signature_fn = interpreter.get_signature_runner()

  # Feed the input image to the model
  output = signature_fn(images=image)

  # Get all outputs from the model
  count = int(np.squeeze(output['output_0']))
  scores = np.squeeze(output['output_1'])
  classes = np.squeeze(output['output_2'])
  boxes = np.squeeze(output['output_3'])

  results = []
  for i in range(count):
    if scores[i] >= threshold:
      result = {
        'bounding_box': boxes[i],
        'class_id': classes[i],
        'score': scores[i]
      }
      results.append(result)
  return results

Please let us know if any problem persists.
Thank You

I come to open this again.
Actually this tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'resample_p7/PartitionedCall' has 1 outputs but the _output_shapes attribute specifies shapes for 3 outputs. Output shapes may be inaccurate.
can be a problem for libraries. In python using the code @LakshmiKalaKadali  has mentioned we extract the values ourselves from the single output
  # Feed the input image to the model
  output = signature_fn(images=image)

  # Get all outputs from the model
  count = int(np.squeeze(output['output_0']))
  scores = np.squeeze(output['output_1'])
  classes = np.squeeze(output['output_2'])
  boxes = np.squeeze(output['output_3'])

However, when you work with libraries they except the specified output shape to match the output.
This leads to problems with the flutter tflite package for instance right here
This function extracts the interpreter result and it will throw a broken NullAssertion error here
outputTensors[i].copyTo(outputs[i]!);

because we code will iterate through 4 specified output when only one is specified. It Looks like this:
{0: {0: [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]], 1: [[[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]]], 2: [0.0], 3: [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]}}

As you can see the output of 4 elements containing the results we need for OD is wrapped again in an outer layer making it a single output.
This is a problem here and applying a workaround in the library package does not solve the problem here. The specified output shapes should match the output!

@alexw92 Hi, I'm not sure if you've deleted your comment but I got a notification about it so I figured I'd respond.
That was the same issue I was having. I should have been more clear about what I was actually trying to do, which was use the tflite model directly in my Flutter app with tflite_flutter. However, as you mentioned, it seems like the only way to get the actual output is with Python, which obviously does not work with Flutter directly.
Of course, you could use a Flask server but at that point what's the point of on-device inference, really. This is why I pivoted away from this approach.
But I do agree, it would be ideal if the TFLite Model Maker could create models that work outside of Python.

@jansdhillon Hi, yes sorry I did delete it. I thought I could reopen the issue but it does not work :D
Yes I fully aggree! I had to do manual fixes which are not appropriate in flutter tflite package to make this work. We should not be forced to do it like that. And since both Tensorflow, Flutter and also now the Flutter TFLite Package are under the hood of Google there should also be some interest of Google to make these work together I guess.


[React] Allow controllable activeIndex for Composite component

Composite is the greate component for build a Menu without the context of floating. Can we have a props like activeIndex to controll what CompositeItem is active or is there anyway to controll it ? Thank you for making the good library.

Focusing an item (e.g. compositeItem.focus()) will change the activeIndex internally.

Controlling the active index (via props like activeIndex and onNavigate) would be helpful in cases where the number of items may vary, or the active item may change unpredictably.
I created a calendar using the Composite component, and this is what happens when I select a day from the next month:

Selecting a day from a different month changes the currently displayed month, consequently altering the index of the selected day. However, the activeIndex managed internally by the Composite component doesn't update, leading to this little issue when navigating again with the arrow keys.
Furthermore, when the selected item is in the last row and the calendar switches to a month with fewer rows, the entire calendar becomes inaccessible via keyboard, as there is no active item at this point (i.e., no items with tabindex="0").
Having props to control the active item would help resolve these issues (using .focus() doesn't support all use cases because sometimes you may wish to change the activeIndex without focusing the item, such as when changing the month with the chevron buttons).
I think this would be a great feature. Love the library btw! :)

Yeah I agree it's a necessary feature and is easy to implement. As a workaround at the moment, you can focus the item, changing the active index internally, then focus the chevron button immediately after to keep it on the button.


Async runnables

I have a need to support async runnables, for example a runnable that waits for distributed work on workers to complete before moving on. In this case the runnable would stall the execution of the schedule until it resolves.

Merged PR.


Enable use of auto-tuning to pick default configuration

Currently all backends use the LoweringConfig attribute to drive the compilation. The fields in this attribute represent the tunable parameters while generating code for a dispatch region. Currently, the most important thing here is the tileSizes used for the different levels of tiling (based on the pipeline that is chosen).
This LoweringConfig can be set on the operation at MHLO level (see example here) to override the default heuristics that are currently encoded in the backends in an adhoc fashion. The defaults can be chosen after doing some search over tile sizes.
The steps involved in flushing this out are

[ ] Currently allowing control of the lowering from MHLO level is only tested on CPU. The other backends need to be tested as well.
[ ] Initial exploration is bound to throw up errors, these need to be addressed.
[ ] Set up a framework to search the space of possible tile sizes. This framework doesnt need to be part of IREE. The tile size also potentially depends on the problem size. So it is possible to end up with a decision tree that for a given problem size + architecture returns a "good" tile size to use. For now this "decision tree" can be hard-coded into IREE.
[ ] Eventually this decision tree will be auto-generated. There is separate work from other groups who are exploring ways to build the decision tree. The decision tree involves choosing between different expert compiler strategies as well as the values for the tunable parameters for the given strategy, based on the architecture and problem size. The approach in IREE is meant to emulate that/use that infrastructure when it becomes available.

With PR #7385 the tuning is now controlled using the attribute
#iree_codegen.compilation.info<
  #iree_codegen.lowering.config<tile_sizes = [[..], [...], ...], native_vector_size = [...]>,
  #iree_codegen.translation.info<"---pass-pipeline----", workload_per_wg = [...]>,
  workgroup_size = [...]>

These work in conjunction with the pass-pipelines used in the backends here. The fields in the #iree_codegen.compilation.info are configuration parameters that control the generated code in each pipeline. But a pipeline might not use all of the exposed parameters. For example, a pipeline that has a single level of tiling does not need the tile_sizes for the second level of tiling. At the same time, if a pipeline uses two-level tiling, only specifying one level can make the pipeline pick a default value for the second-level of tiling (could be 0 which defaults to no-tiling).
The issue that I haven't fully worked out is what level of verification is needed. There are two options

Before executing each pipeline, a verifier will check that the set configuration exactly matches what the pipeline expects. The impact of this is that based on the configuration, the compilation might fail. In other words, the search has to encode exactly which are legal points in the space which is very pipeline specific.
Each pipeline produces valid outputs for all configurations, but ignores some of the set values. It means the search space is much wider than it needs to be, but the search is completely agnostic to the pipeline details.

My intent so far has been towards the second approach. The first approach seems like a leak of implementation details to the search, and presumably a search algorithm can avoid traversing parts of the space that do not affect the performance.
This is the primary reason I havent added any verifiers per pipeline to check for validity of the configuration. I am just putting this reasoning down here to see if I can get an input on what path to take.

With PR #8446 and PR #8504 the initial setup of the having a path to use auto-tuning with IREE is setup. Overall

The ops at MHLO level can be annotated with the iree_codegen.compilation_info attribute exposes control to the auto-tuner to pick the pipelines to use and the parameters of these configurations
There is work in progress for adding verifiers for the configuration set on various pipelines. It was found to be the best way to handle the implicit contract between what configuration is valid for a pipeline, an dhow the auto-tuner picks the configuration. Now the verifier method exposed to auto-tuners (here)[https://github.com/google/iree/blob/5783b51d60b62b8312d2eed76052c169cd7137e1/iree/compiler/Codegen/Passes.h#L28] can be used to establish the contract.

The attribute based control mechanism is inherently limited. There is work in other areas of MLIR to build a dialect that can allow control over transformations applied, and their parameters. This will eventually supercede the attribute based approach here, but the attribute based approach is expected to be sufficient till then.


Fix Python 3 compatiblities

Thanks for contributing. There are much more absolute import stuffs need to be fixed for python 3, would you like to add them all?

Use from __future__ import absolute_import

Thank you for reviewing. I believe that I added all files needed to be fixed for Python 3. Could you check it out?


Fix typo in Signed Package Reference

/cc @ridomin

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/reference/Signed-Packages-Reference.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Clear disk space on CI.

Cargo's testsuite uses a considerable amount of disk space. On windows-gnu, the target directory can get over 12GB, and there is only 13GB free.  We're starting to run out of disk space, so this is a stop-gap that clears out the test data before running the smoke test which uses a fair bit of space itself.
We will probably need to think about addressing #9701 somehow, otherwise we will start running out of space as we add more tests. See the linked issues in https://github.com/rust-lang/cargo/pull/9701#issuecomment-881765517 for additional context.

@ehuss: no appropriate reviewer found, use r? to override

In my test, the new step only takes about 30s, which isn't too bad.

I wonder if the move to snapbox for testing will help or hurt this issue

@bors r+
This will work for now. If there are objections, or better plans, we can always have a new PR.

:pushpin: Commit 65cc01075e753533a5ba4459b588bb78abd83619 has been approved by Eh2406

:hourglass: Testing commit 65cc01075e753533a5ba4459b588bb78abd83619 with merge 03e22354d0d6d8b9186db431185e026b52a2bf4a...

:sunny: Test successful - checks-actions
Approved by: Eh2406
Pushing 03e22354d0d6d8b9186db431185e026b52a2bf4a to master...


Several static Libs/Binaries contain full path of (many) CC files for errors/asserts/warnings

Hi,
In the release mode the static Libs/binaries should not contain the filenames (for sure not the full path) of the CC files used in several error messages/warning/asserts.
I found the "FILE" usage in:
(version 0.6)
hwy\base.h
decode.cc
encode.cc
base\status.h
This is nice in Debug mode, but not in the Release mode ... or add a flag to CMake.

@gorza47 I don't understand why full paths are not ok in release mode, but relative paths and the debug messages (from the asserts that use __FILE__) are ok. If the issue is including unnecessary strings in the release binary bloating the file size, then use -DJXL_DEBUG_ON_ABORT=0 in the cflags to get rid of the strings (both messages and file paths). It is not always the case that "release mode" means size-optimized, it depends on the use case so we give you the option.
We could set -fmacro-prefix-map when compiling with gcc in cmake too. For some reason when I build locally from a build directory I don't see absolute paths, I see "../lib/jxl....", but anyway using fmacro-prefix-map when supported sounds reasonable to me.


🐛 BUG: Support for typescript inside

Describe the Bug
Hello! I'm currently experiencing the following issue. When using the astro language tools inside VS Code, I'm getting typescript errors like "Type assertion expressions can only be used in TypeScript files.ts(8016)".
Generally the typ checking is working flawlessly, however this odd error appears when I'm adding the define:vars property.
The lack for define:vars is mentioned inside this pull request: https://github.com/withastro/language-tools/pull/263
Is this behaviour still intentional? Are there any workarounds till this gets fixed?
This is working:

This is not working:

My tscconfig.json:
{
  "extends": "astro/tsconfigs/strict",
  "compilerOptions": {
    "jsx": "react-jsx",
    "jsxImportSource": "react"
  }
}

Extension (astro-build.astro-vscode) version: v2.6.2
VS Code version: 1.85.1 (Universal)
    "astro": "^4.0.2",
    "@astrojs/check": "^0.3.1",
    "@astrojs/react": "^3.0.9",
    "@astrojs/tailwind": "^5.0.3",
    "typescript": "^5.3.2"

Steps to Reproduce

Write typescript annotations inside a script tag in an astro codebase.
Add define:vars={{ title: "test " }}

After further reading I discovered, that is behaviour is intentional. Whenever someone defines the property define:vars on a script tag, it will inherit the is:inline directive as stated in the big yellow warning on this docs page: https://docs.astro.build/en/reference/directives-reference/#definevars
Inline script won't be bundled and won't be processed by typescript, sadly.

Would be nice if that could get added. I've had to remove the types so many times now because I at some point realize that I need define:vars.
At this point I'm considering to just not use typescript. But I would honestly prefer that typescript would be transpiled also when it inline.


X3D DEF/USE for IndexedFaceSet causing null pointer, crash

New master (v 4.0), IndexedFaceSet DEF/USE causing null pointer.
Was working in maint_v3.x
When parsing
Inside X3Dobject, SAX parser end command has gvrVertexBuffer null.
About line 3630 in the code
Test found in our Tests repo -> x3d -> Scripting_Per_Frame -> JavaScript_PerFrame_02_LaunchSphere.x3d
https://github.com/gearvrf/GearVRf-Tests/blob/master/x3d/Scripting_Per_Frame/JavaScript_PerFrame_02_LaunchSphere.x3d
Same issue with test case JavaScript_PerFrame_05_LightControls.x3d found in the same repo directory.

Fixed with Pull Request https://github.com/Samsung/GearVRf/pull/1527


bau: log session cookie size if it's large

We store a lot of stuff in the session cookie. This commits add code
to log a warning if it's getting too large.  Without going too deep
into Rails we can log an approximation of the size, at a slightly
lower threshold.
Perhaps we can move this into the ApplicationController; I don't mind
too much as long as we can have some sort of visibility of this
becoming an issue.
Pair: WillPa

# --- Caused by: ---
     # NoMethodError:
     #   undefined method `flatten' for #<ActionDispatch::Request::Session:0x00000006c932d0>
     #   ./app/models/session_validator/cookie_size_validator.rb:5:in `validate'


useWatchBlocks subscribes / unsubscribes on each new Block with WebSocket

Check existing issues

[X] I checked there isn't already an issue for the bug I encountered.

Describe the bug
When using useWatchBlocks with a WebSocket connection, the expected behavior is as follows:

Subscribe to new blocks
Receive (onBlock) callbacks without unsubscribing
Unsubscribe when the component using useWatchBlocks is unmounted / modified

The observed behavior is:

Subscribe to new heads
Receive a block, execute onBlock
Unsubscribe as a dependency seems to change in the implementation of useWatchBlocks, probably caused by  useEffect in the useWatchBlocks implementation
Re-Subscribe and repeat

It's important to be aware that subscribing / Unsubscribing can be an expensive call on certain API providers (e.g. 10 Compute Units on Alchemy), causing unnecessary spikes in API Usage.
Alchemy Compute Unit Costs
Link to Minimal Reproducible Example
No response
Steps To Reproduce

Configure Wagmi with a WebSocket RPC endpoint
Use the snippets below, which only slightly modify the useWatchBlocks code to get console log outputs
You will observe that the subscription is closed after each block, and a new subscription is established.

Code to use the modified hook:
  useWatchBlocks({
    emitOnBegin: true,
    chainId: Constants.CHAIN_ID,
    onBlock: (b) => {
      console.log(b)
    }
  })

The modified hook:
'use client'

import {
type Config,
type ResolvedRegister,
type WatchBlocksParameters,
watchBlocks,
} from '@wagmi/core'
import type { UnionCompute, UnionExactPartial } from '@wagmi/core/internal'
import { useEffect } from 'react'
import type { BlockTag } from 'viem'

// Import wagmi config and constants for test
import { config } from '../AppKit'

export type EnabledParameter = {
enabled?: boolean | undefined
}

export type ConfigParameter<config extends Config = Config> = {
config?: Config | config | undefined
}

export type UseWatchBlocksParameters<
includeTransactions extends boolean = false,
blockTag extends BlockTag = 'latest',
config extends Config = Config,
chainId extends
config['chains'][number]['id'] = config['chains'][number]['id'],
> = UnionCompute<
UnionExactPartial<
  WatchBlocksParameters<includeTransactions, blockTag, config, chainId>
> &
ConfigParameter<config> &
EnabledParameter
>

export type UseWatchBlocksReturnType = void

/** https://wagmi.sh/react/hooks/useWatchBlocks */
export function useWatchBlocks<
config extends Config = ResolvedRegister['config'],
chainId extends
config['chains'][number]['id'] = config['chains'][number]['id'],
includeTransactions extends boolean = false,
blockTag extends BlockTag = 'latest',
>(
parameters: UseWatchBlocksParameters<
  includeTransactions,
  blockTag,
  config,
  chainId
> = {} as any,
): UseWatchBlocksReturnType {
const { enabled = true, onBlock, config: _, ...rest } = parameters
const chainId = parameters.chainId

// TODO(react@19): cleanup
// biome-ignore lint/correctness/useExhaustiveDependencies: `rest` changes every render so only including properties in dependency array
useEffect(() => {
  if (!enabled) return
  if (!onBlock) return

  // Slightly modify for console logs
  console.log('WATCH')
  const unwatch = watchBlocks(config, {
    ...(rest as any),
    chainId,
    onBlock,
  })

  return () => {
    console.log('UNWATCH')
    unwatch()
  }
}, [
  chainId,
  config,
  enabled,
  onBlock,
  ///
  rest.blockTag,
  rest.emitMissed,
  rest.emitOnBegin,
  rest.includeTransactions,
  rest.onError,
  rest.poll,
  rest.pollingInterval,
  rest.syncConnectedChain,
])
}

What Wagmi package(s) are you using?
wagmi
Wagmi Package(s) Version(s)
2.12.17
Viem Version
2.21.21
TypeScript Version
5.6.0
Anything else?
No response

After investigating the issue further, it seems that onBlock, when passed like in the official example code:
import { useWatchBlocks } from 'wagmi'

function App() {
  useWatchBlocks({
    onBlock(block) {
      console.log('New block', block.number)
    },
  })
}

is causing the behavior. The following change seems to help:
const onBlock = useCallback((block) => {
  console.log('New block', block.number);
}, []);

useWatchBlocks({
  onBlock
})

However, I suggest a code-change where modifying the callback is not causing a subscribe / unsubscribe.

Just a quick draft on how this could be improved. Using a ref to store onBlock to prevent unnecessary triggers in the useEffect.
'use client'

import {
  type Config,
  type ResolvedRegister,
  type WatchBlocksParameters,
  watchBlocks,
} from '@wagmi/core'
import type { UnionCompute, UnionExactPartial } from '@wagmi/core/internal'
import { useEffect, useRef, useState } from 'react'
import type { BlockTag, OnBlockParameter } from 'viem'

// Import wagmi config and constants for test
import { config } from '../AppKit'

export type EnabledParameter = {
  enabled?: boolean | undefined
}

export type ConfigParameter<config extends Config = Config> = {
  config?: Config | config | undefined
}

export type UseWatchBlocksParameters<
  includeTransactions extends boolean = false,
  blockTag extends BlockTag = 'latest',
  config extends Config = Config,
  chainId extends
  config['chains'][number]['id'] = config['chains'][number]['id'],
> = UnionCompute<
  UnionExactPartial<
    WatchBlocksParameters<includeTransactions, blockTag, config, chainId>
  > &
  ConfigParameter<config> &
  EnabledParameter
>

export type UseWatchBlocksReturnType = void

/** https://wagmi.sh/react/hooks/useWatchBlocks */
export function useWatchBlocks<
  config extends Config = ResolvedRegister['config'],
  chainId extends
  config['chains'][number]['id'] = config['chains'][number]['id'],
  includeTransactions extends boolean = false,
  blockTag extends BlockTag = 'latest',
>(
  parameters: UseWatchBlocksParameters<
    includeTransactions,
    blockTag,
    config,
    chainId
  > = {} as any,
): UseWatchBlocksReturnType {
  const { enabled = true, onBlock, config: _, ...rest } = parameters
  const chainId = parameters.chainId

  // Use a ref to hold the onBlock callback
  const onBlockRef = useRef(onBlock)
  const [watchEnabled, setWatchEnabled] = useState(enabled && !!onBlock)

  // Update the ref if onBlock changes, but don't cause re-subscription
  useEffect(() => {
    console.log('UPDATE REF')
    onBlockRef.current = onBlock

    // Adjust watchEnabled when onBlock goes from undefined to defined or vice versa
    if (!!onBlock !== watchEnabled) {
      setWatchEnabled(!!onBlock)
    }
  }, [onBlock, watchEnabled])

  useEffect(() => {
    if (!watchEnabled) return

    console.log('WATCH')
    const unwatch = watchBlocks(config, {
      ...(rest as any),
      chainId,
      // Use the latest onBlock from the ref
      onBlock: (block, prevBlock) => {
        if (onBlockRef.current) {
          onBlockRef.current(block, prevBlock)
        }
      },
    })

    return () => {
      console.log('UNWATCH')
      unwatch()
    }
  }, [
    chainId,
    config,
    watchEnabled, // Dependency that changes based on whether we should be watching or not
    ///
    rest.blockTag,
    rest.emitMissed,
    rest.emitOnBegin,
    rest.includeTransactions,
    rest.onError,
    rest.poll,
    rest.pollingInterval,
    rest.syncConnectedChain,
  ])
}

I cant reproduce ur issue, I dont see there're unsubscribe action on new block. Could you provide a reproduce example?

@coderodigital unable to repro your issue:
'use client'

import { createConfig, WagmiProvider, webSocket } from "wagmi";
import { mainnet } from "wagmi/chains";
import { useWatchBlocks } from "./watchblock";

export const config = createConfig({
    chains: [mainnet],
    transports: {
      [mainnet.id]: webSocket('<WS_RPC_URL>'),
    },
  });

function Xyz() {
    useWatchBlocks({
        onBlock: (b) => {
          console.log('found new block >>>>', b.hash.slice(0, 10))
        }
      })

  return <></>
}

export default function WatchBlocks() {
  return (
    <WagmiProvider config={config}>
            <Xyz />
    </WagmiProvider>
  );
}

where useWatchBlocks is your adapted hook - can you share a repro?

Let me prepare an example repo (give me a day). To identify the bug, you must examine the network traffic on the open WebSocket connection. The code works as intended but could be more effective when using the WebSocket subscription (which only subscribes once). I will also share screenshots of the network traffic.


Using this to provision ArgoCD and ingress-nginx

[x] ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

I am trying to use this repository to easily (and repeatable) deploy an ArgoCD cluster with an ingress-nginx controller to create a public-facing load balancer. However, everything I've tried deploys a load balancer with an internal scheme. I'm obviously missing something.
Here's the code I'm trying to use:
module "eks_blueprints_addons" {
  source  = "aws-ia/eks-blueprints-addons/aws"
  version = "1.9.2" #ensure to update this to the latest/desired version

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider_arn

  eks_addons = {
    aws-ebs-csi-driver = {
      most_recent = true
    }
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
  }

  enable_argocd                       = true
  enable_ingress_nginx                = true
  enable_argo_events                  = true
  enable_argo_rollouts                = true
  enable_argo_workflows               = true
  enable_aws_load_balancer_controller = true
  /* enable_cluster_proportional_autoscaler = true */
  enable_karpenter             = true
  enable_kube_prometheus_stack = true
  enable_metrics_server        = true
  enable_external_dns          = true
  /* enable_cert_manager                    = true */
  /* cert_manager_route53_hosted_zone_arns  = ["arn:aws:route53:::hostedzone/XXXXXXXXXXXXX"] */

  enable_aws_for_fluentbit = true
  aws_for_fluentbit = {
    enable_containerinsights = true
    kubelet_monitoring       = true
    set = [{
      name  = "cloudWatchLogs.autoCreateGroup"
      value = true
      },
      {
        name  = "hostNetwork"
        value = true
      },
      {
        name  = "dnsPolicy"
        value = "ClusterFirstWithHostNet"
      }
    ]
  }

  argocd = {
    set = [
      /* {
        name  = "server.insecure"
        value = "true"
      }, */
      {
        name  = "server.ingress.enabled"
        value = "true"
      },
      {
        name  = "server.ingress.ingressClassName"
        value = "nginx"
      }
    ]

    set_sensitive = [
      {
        name  = "configs.secret.argocdServerAdminPassword"
        value = bcrypt_hash.argo.id
      }
    ]
  }

  ingress_nginx = {
    name          = "ingress-nginx"
    chart_version = "4.8.3"
    repository    = "https://kubernetes.github.io/ingress-nginx"
    namespace     = "ingress-nginx"
    values        = [templatefile("${path.module}/helm/nginx-ingress/values.yaml", {})]
  }

  tags = local.tags
}

Can you share your annotations for ingress-nginx?

I don't recall adding anything special. Should the annotations go in the code above?

@Vermyndax your trying to expose ArgoCD API Server for UI and CLI using an ingress object?
We have a pattern that shows how to do this with ingress object using aws-loadbalancer-controller here https://github.com/gitops-bridge-dev/gitops-bridge/tree/main/argocd/iac/terraform/examples/eks/argocd-ingress
Do you mind opening an issue requesting an example using nginx-ingress load balancer controller here https://github.com/gitops-bridge-dev/gitops-bridge/issues

Thank you! I actually switched to the AWS ALB controller and got this working. Your pattern unlocked what I needed!


inconsistent punctuation and capitalization

Hello, I have been using your Software For disability reasons because I Have difficulty typing This is way more accurate than naturally speaking The issue I wanted to report is that sometimes The whispering output has Punctuation and Capitalization, that's really strange. I Mean to say that it skips out on punctuation altogether sometimes I know this is not a fault of your software but of Something to do with the whisper technology itself What I was wondering is Is there a way you could add a feature where we have the option to use our open AI AI API key to Automatically run the transcribed text output through a Chat GPT instruction something that tells it to format the punctuation in a certain way or Potentially for fun just convert anything you say into whisper into pirate speak for instance It could be useful and entertaining Please note that I this is the first time I've ever posted anything on github So I don't know if the issues page is the appropriate place for this comment But thank you for reading it.A slight edit, it would be great if this post process with custom chat GPT instruction were an option that you could check or uncheck to save money with your API key.

I'm glad you're so open to feedback. I understand that Your schedule is busy I Can only see your profile picture in a small way, but it almost looks like you're at a graduation Again, I didn't zoom in but I'm seeing a graduation cap Let me zoom in. Oh I see that I was wrong. But I certainly do hope you have a graduation cap in your future seeing as you started school. Thank you for your work on the project. I have one or maybe one or two more suggestions that might be of note. And instead of bringing them up here, I'll start a different issue item to keep everything more organized.

There is also the ability to specify a prompt to the whisper model. I think that would be good to be able to customize as well. Adding a filter at the end using a custom ChatGPT prompt and model selection sounds like a neat feature as well!

Hello,
thank you very much for your wonderful work, it has made my life different, really, thank you ! <3
As said above, it would be wonderful to be able to add custom prompt.
Do you to integrate this feature ?
I would love to help you.
I am not a great developer, but now, with AI, I can do quite a lot of things.
Please, get in touch if you have the opportunity, to see what can be done.
All the best, thank you very much. :)

This is where the clipboard workflow is super-conducive on Windows.  I do the the following, incorporating Push-To-Talk:

Clear clipboard (I don't bother restoring it ... nothing is that important, but one could restore if wishes to)
Press CapsLock, which sends Whispering recording toggle ON
Hold CapsLock (through AutoHotkey "KeyWait")
Release CapsLock, which sends Whispering recording toggle OFF
Wait for Clipboard to fill with content (with AHK "ClipWait"))
As soon as Clipboard got content, pass content through a few RegEx statements to do post processing, which "fixes" punctuations (with caveats and some limitations), and translates verbatim "new paragraph" to actual newlines.  I don't currently force casing of lead character - but that could be easily toggled, especially if I find myself pausing and composing mid-sentence more often.
Paste processed output to where cursor is currently located.

So far above workflow is quite satisfactory.
However, I wouldn't mind an option, which I think is possible with offline Whisper instances, to force punctuation one way or another, so I can more consistently enunciate my own punctuations all the time - this is generally how folks dictate on Dragon.
The problem with Whisper's auto-punctuation is that when it works it works well.  But when you "pause and compose", it tends to falter.  Yes in theory one can "pipe" the output through a GPT to fix things.  But why bother if you don't have to?

@swimJim mainly: take a look here my friend: https://forums.knowbrainer.com/forum/third-party-command-utilities-vocola-unimacro-voicepower-python/3409-openai-s-transcription-api-beginner-level-interaction-and-early-user-experience#post3430
Using cURL and phiola, I have been able to write a Whispering equivalent - on Windows only of course, with AHK (v1 or v2, take your pick).
Because the "barriers to entry" on AHK, cURL, and phiola are so low, you could do it, too.  If you run into any issues, go to the friendly AHK forum, where I am sure you will more than likely get your questions answered.
Whispering if amazing because it is cross-platform.  But I don't need cross-platform myself.  AHK+cURL+phiola gives such flexibility and ease of usability on Windows, it is worth sharing with the enthusiasts here.


liburing.h looks incompatible with anything that may include /usr/include/sys/mount.h

I met this compiler error when I build libfuse together with liburing:
[6/49] Compiling C object 'lib/76b5a35<EMAIL_ADDRESS>FAILED: lib/76b5a35<EMAIL_ADDRESS>cc -Ilib/76b5a35@@fuse3@sha -Ilib -I../lib -Iinclude -I../include -I. -I.. -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -g -D_REENTRANT -DHAVE_CONFIG_H -Wall -Wextra -Wno-sign-compare -Wstrict-prototypes -Wmissing-declarations -Wwrite-strings -fno-strict-aliasing -Wno-unused-result -fPIC -pthread -DFUSE_USE_VERSION=35 '-DFUSERMOUNT_DIR="/usr/local/bin"' -MD -MQ 'lib/76b5a35<EMAIL_ADDRESS>-MF 'lib/76b5a35<EMAIL_ADDRESS>-o 'lib/76b5a35<EMAIL_ADDRESS>-c ../lib/mount.c
In file included from /usr/include/liburing/io_uring.h:11:0,
from /usr/include/liburing.h:15,
from ../include/fuse_i.h:8,
from ../lib/mount.c:12:
/usr/include/sys/mount.h:35:3: error: expected identifier before numeric constant
MS_RDONLY = 1,  /* Mount read-only.  */
^
In file included from ../lib/mount.c:28:0:
../lib/mount.c:127:13: error: ‘MS_RDONLY’ undeclared here (not in a function)
{"rw",     MS_RDONLY,     0},
^
../lib/mount.c:129:14: error: ‘MS_NOSUID’ undeclared here (not in a function)
{"suid",    MS_NOSUID,     0},
^
../lib/mount.c:131:14: error: ‘MS_NODEV’ undeclared here (not in a function)
{"dev",     MS_NODEV,     0},
^
../lib/mount.c:133:14: error: ‘MS_NOEXEC’ undeclared here (not in a function)
{"exec",    MS_NOEXEC,     0},
^
../lib/mount.c:135:14: error: ‘MS_SYNCHRONOUS’ undeclared here (not in a function)
{"async",   MS_SYNCHRONOUS, 0},
^
../lib/mount.c:137:14: error: ‘MS_NOATIME’ undeclared here (not in a function)
{"atime",   MS_NOATIME,     0},
^
../lib/mount.c:140:14: error: ‘MS_DIRSYNC’ undeclared here (not in a function)
{"dirsync", MS_DIRSYNC,     1},
^
../lib/mount.c: In function ‘get_mnt_flag_opts’:
../lib/mount.c:502:14: error: invalid operands to binary & (have ‘int’ and ‘const struct mount_flags *’)
if (!(flags & MS_RDONLY) && fuse_opt_add_opt(mnt_optsp, "rw") == -1)
^
../lib/mount.c: In function ‘parse_mount_opts’:
../lib/mount.c:522:24: error: invalid operands to binary | (have ‘const struct mount_flags *’ and ‘const struct mount_flags *’)
mo->flags = MS_NOSUID | MS_NODEV;
^
../lib/mount.c:522:12: warning: assignment makes integer from pointer without a cast [-Wint-conversion]
mo->flags = MS_NOSUID | MS_NODEV;
^
it looks like liburing includes linux/fs.h，which already defines such macros that conflicts with sys/mount.h?
part of quoted from linux/fs.h:
/*

These are the fs-independent mount-flags: up to 32 flags are supported
/
#define MS_RDONLY    1  / Mount read-only /
#define MS_NOSUID    2  / Ignore suid and sgid bits /
#define MS_NODEV     4  / Disallow access to device special files /
#define MS_NOEXEC    8  / Disallow program execution /
#define MS_SYNCHRONOUS  16  / Writes are synced at once /
#define MS_REMOUNT  32  / Alter flags of a mounted FS /
#define MS_MANDLOCK 64  / Allow mandatory locks on an FS /
#define MS_DIRSYNC  128 / Directory modifications are synchronous /
#define MS_NOATIME  1024    / Do not update access times. /
#define MS_NODIRATIME   2048    / Do not update directory access times /
#define MS_BIND     4096
#define MS_MOVE     8192
#define MS_REC      16384
#define MS_VERBOSE  32768   / War is peace. Verbosity is silence.
MS_VERBOSE is deprecated. /
#define MS_SILENT   32768
#define MS_POSIXACL (1<<16) / VFS does not apply the umask /
#define MS_UNBINDABLE   (1<<17) / change to unbindable /
#define MS_PRIVATE  (1<<18) / change to private /
#define MS_SLAVE    (1<<19) / change to slave /
#define MS_SHARED   (1<<20) / change to shared /
#define MS_RELATIME (1<<21) / Update atime relative to mtime/ctime. /
#define MS_KERNMOUNT    (1<<22) / this is a kern_mount call /
#define MS_I_VERSION    (1<<23) / Update inode I_version field /
#define MS_STRICTATIME  (1<<24) / Always perform atime updates /
#define MS_LAZYTIME (1<<25) / Update the on-disk [acm]times lazily */

quoted from sys/mount.h
enum
{
MS_RDONLY = 1,        /* Mount read-only.  /
#define MS_RDONLY   MS_RDONLY
MS_NOSUID = 2,        / Ignore suid and sgid bits.  /
#define MS_NOSUID   MS_NOSUID
MS_NODEV = 4,         / Disallow access to device special files.  /
#define MS_NODEV    MS_NODEV
MS_NOEXEC = 8,        / Disallow program execution.  /
#define MS_NOEXEC   MS_NOEXEC
MS_SYNCHRONOUS = 16,      / Writes are synced at once.  /
#define MS_SYNCHRONOUS  MS_SYNCHRONOUS
MS_REMOUNT = 32,      / Alter flags of a mounted FS.  /
#define MS_REMOUNT  MS_REMOUNT
MS_MANDLOCK = 64,     / Allow mandatory locks on an FS.  /
#define MS_MANDLOCK MS_MANDLOCK
MS_DIRSYNC = 128,     / Directory modifications are synchronous.  /
#define MS_DIRSYNC  MS_DIRSYNC
MS_NOATIME = 1024,        / Do not update access times.  /
#define MS_NOATIME  MS_NOATIME
MS_NODIRATIME = 2048,     / Do not update directory access times.  /
#define MS_NODIRATIME   MS_NODIRATIME
MS_BIND = 4096,       / Bind directory at different place.  */
#define MS_BIND     MS_BIND
MS_MOVE = 8192,
#define MS_MOVE     MS_MOVE
MS_REC = 16384,
#define MS_REC      MS_REC
MS_SILENT = 32768,

I removed fs.h from liburing.h and the build is OK
the only dependency of fs.h is
typedef int __bitwise __kernel_rwf_t;
and I did a dirty work to define it in liburing.h

another way is to adjust the header file include sequences, the mount.h must be included before liburing.h

This isn't a liburing issue, liburing.h doesn't use any mount related bits. This should be fixed in the system headers, or in the application.


Use no cookies version of youtube iframe

It would be great to use the youtube-nocookie.com of the embed because it would remove the need to get cookie-consent at all.
This plugin already solves part of the problem - there is no initial need to get cookie consent until the embed is clicked on. However, in the current version that uses the regular embed, cookies are added as soon as the youtube video is clicked. As I understand it, it is the responsibility of the container website to ask permissions to use cookies. However, if the youtube-nocookie.com version is used, no cookies are added, even when the video is started. And therefore there is no need to ask for cookie consent.
To test this, I embedded the normal iframe and watched my browser's Developer Tools storage window. Cookies we added when the video started.
Normal cookie-enabled iframe (similarly used by this plugin):

In comparison if I used youtube-nocookie.com version, no cookies were added even when the link was clicked:

As far as I can tell, the only thing that would need to change in this package is replacing youtube.com/embed with youtube-nocookie.com/embed/.
This was previously mentioned at #26 but there's more to this.

Thank you. I might have forgotten. Let me get this done.

Fixed in<EMAIL_ADDRESS>

Fantastic, thank you


get rid of unnecessary buffering during decode

babysteps for #57
sadly there's no impl<T:Iterator, E> Iterator for Result<T, E>, so I stole some code from FromIterator::from_iterator::<Result<...>>

r? @alexcrichton
(rust_highfive has picked a reviewer for you, use r? to override)

Hm could you take a look at some benchmarks for this as well? I think that calling .chars() on an I/O object is not performant compared to calling it on a str

benchmark:
    #[bench]
    fn bench_large_file(b: &mut Bencher) {
        use std::fs::File;
        use std::io::BufReader;
        let f = File::open("src/big_file.json").unwrap();
        let mut buf = BufReader::new(f);
        b.iter( || { let _ = Json::from_reader(&mut buf); });
    }

big_file.json consists of ~1000 lines of
{ "a": true, "b": null, "c":3.1415, "d": "Hello world", "e": [1,2,3]},
this pr:
test json::tests::bench_large_file      ... bench:      4515 ns/iter (+/- 273)

old:
test json::tests::bench_large_file      ... bench:     15356 ns/iter (+/- 711)

second benchmark:
    #[bench]
    fn bench_large_file_reopening(b: &mut Bencher) {
        use std::fs::File;
        use std::io::BufReader;
        b.iter( || {
        	let f = File::open("src/big_file.json").unwrap();
        	let mut buf = BufReader::new(f);
            let _ = Json::from_reader(&mut buf);
        });
    }

this pr:
test json::tests::bench_large_file_reopening ... bench:   3506554 ns/iter (+/- 54927)

old:
test json::tests::bench_large_file_reopening ... bench:   2793185 ns/iter (+/- 160453)

I don't even know what the first benchmark does... I'd have said after the first iteration all further iterations should be a nop...

There is a huge difference in the implementations of std::io::Chars and std::str::Chars.
So... to get the performance to the same level we need to use BufRead instead of Read and fill_buf instead of read. Should be simple enough to have std::io::Chars use a BufRead internally to the Read it was created with. Since ReadExt::chars consumes self, everything should be fine as long as noone calls it on a &mut (which apparently is fine: https://github.com/rust-lang/rust/issues/22802)

The first benchmark may be skewed because it decodes once and then never decodes again (e.g. you don't seek back to the beginning). The second benchmark is what I would expect in that using std::io::Chars is much slower.
Even if we use BufRead I think that using std::io::Chars will be slower, but it would be worth benchmarking.
Also, could this avoid checking in a large file? It looks like it should be easy to generate in-memory.


Call capl function does not work

Hello,
Thank you for taking an interest in implementing this, but unfortunately it doesn't work.
When calling function it returns KeyError, it cannot find the function in self.meas.user_capl_function_obj_dict.
I have passed the function name as argument during CANoe instance creation.
I'm using CANoe v15 with py_canoe v2.0.1 release.

this issue resolved in latest version.
please upgrade package to 2.0.2 version.
pip install py_canoe --upgrade
https://pypi.org/project/py_canoe/2.0.2/

it works now


Typo fixed

I think this is small typo. @SchrodingersGat could you look at it?

Good catch, thanks @rrakso


GUI - filter option

We should reimplement the search bar to enable filtering the projects by marking multiple tags at once. See an example

I think we could start by allowing for just two keywords to be selected at a time. Then, if that worked ok, we could think about enabling larger intersections.

Having now worked on this feature, I realized it's easier to change our code to allow an arbitrary number of keywords or language tags to be selected.

Closing as completed for now. This can be reopened later, or a new issue filed, for further development of the feature.


Windows Stack Overflow on EfficentNet Test _deepwalk

I get a windows stack overflow with

Python 3.9.0
python .\test\models\test_train.py TestTrain.test_efficientnet

With the following trace if I enable faulthandler (note the first error is just related to tqdm and can be ignored).
test\models\test_train.py .Windows fatal exception: stack overflow

Thread 0x00002074 (most recent call first):
  File "C:\Users\Patrick\AppData\Local\Programs\Python\Python39\lib\threading.py", line 316 in wait
  File "C:\Users\Patrick\AppData\Local\Programs\Python\Python39\lib\threading.py", line 574 in wait
  File "D:\COMMAI\tinygrad\venv\lib\site-packages\tqdm\_monitor.py", line 60 in run
  File "C:\Users\Patrick\AppData\Local\Programs\Python\Python39\lib\threading.py", line 950 in _bootstrap_inner
  File "C:\Users\Patrick\AppData\Local\Programs\Python\Python39\lib\threading.py", line 908 in _bootstrap

Current thread 0x00002238 (most recent call first):
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in <listcomp>
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in _deepwalk
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in <listcomp>
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in _deepwalk
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in <listcomp>
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in _deepwalk
  File "D:\COMMAI\tinygrad\tinygrad\tensor.py", line 159 in <listcomp>
...

If I change the deepwalk loop in tensor.py from this:
if node._ctx:
  [_deepwalk(i, visited, nodes) for i in node._ctx.parents if i not in visited]

To this:
for i in node._ctx.parents:
  if i not in visited:
    _deepwalk(i, visited, nodes)

Then all what I get is just the following warning and the test passes:
:\commai\tinygrad\tinygrad\ops.py:93: RuntimeWarning: overflow encountered in multiply
  else: ret = cls(cls.fxn_for_op[ast.op](*([x._buf for x in srcs] + ([ast.arg] if ast.arg else []))))
D:\COMMAI\tinygrad\venv\lib\site-packages\numpy\core\_methods.py:49: RuntimeWarning: overflow encountered in reduce
  return umr_sum(a, axis, dtype, out, keepdims, initial, where)

Not sure why that would make a difference (not that familiar with python inline loops). Any thoughts?

Hmm, maybe tail call recursion. Even though it's two lines, I'd merge the PR to change

This works:
for i in [k for k in node._ctx.parents if k not in visited]:
  _deepwalk(i, visited, nodes)

but this does not:
to_visit = [k for k in node._ctx.parents if k not in visited]
[_deepwalk(i, visited, nodes) for i in to_visit]


Security vulnerability in python dependencies

this dependencies need to be updated:

jupyterhub>=0.9.6

I already did it in the repository I'm using to test the installation process of the IJava kernel.
I'll update it here too.

Excellent; see branch #issue17_security

closed by https://github.com/linkedbuildingdata/SummerSchoolOfLDAC/commit/b7f1df6171021237de7b5eb91b071d55b9871dd8


default_actions should exclude actions not defined in the controller. closes #200

default_actions should exclude actions not defined in the controller. closes #200

+1

@pcreux, the multiple belongs_to support had some edge cases not covered that I discovered after a while when using my fork. However I'll send you a pull req with the "intelligent" default_actions, which is the thing the people seems to be +1ing.

Woo I see it's already implemented in 82bd4c1c101ca5860fefdfc39a153e4f2f66dd74, closing this then.


Clicking folder button next to image does nothing

Describe the bug
In the attached screenshot, pressing the Folder button for an image does nothing:

To Reproduce

Go to images
Find an image
Click the Folder button
Notice nothing happens

Expected behavior
Something should happen… maybe opening the image's files in Finder?
Diagnostic report (required)
OrbStack info:
Version: 0.16.1 (160100)
Commit: 4235b341267c814080ec0e117d1e680d8f8f8676 (v0.16.1)
System info:
macOS: 13.5.1 (22G90)
CPU: arm64, 10 cores
CPU model: Apple M1 Max
Full report: https://orbstack.dev/_admin/diag/orbstack-diagreport_2023-08-25T15-35-53.235127Z.zip

Screenshots and additional context (optional)
No response

What's the exact image tag? Feel free to change the words but the casing, symbols, etc. are important, e.g. ghcr.io/some/image:1234
Can you find the image in ~/OrbStack/docker/images?

I had exactly the same issue. I couldn't open some of the image in Finder by clicking on the folder icon and they didn't appear in ~/OrbStack/docker/images.
Seems resolved after update in v0.17.0.

@jean-humann Were they images you built locally by any chance?

@kdrag0n yes they were !

Fixed for the next version.

Released in v0.17.2.


deployment of crowdsec-lapi fails when upgrading from 0.13.0 to 0.14.0

I'm running on the helm chart version 0.13.0. When I try to deploy an upgrade to 0.14.0, the crowdsec-lapi pod isn't able to come up. The log file says:
no configuration for Central API (CAPI) (/etc/crowdsec/config.yaml)

I did no changes to my values.yaml so far:
# -- for raw logs format: json or cri (docker|containerd)
container_runtime: containerd

# Here you can specify your own custom configuration to be loaded in crowdsec agent or lapi
# Each config needs to be a multi-line using '|' in YAML specs
# for the agent those configs will be loaded : parsers, scenarios, postoverflows, simulation.yaml
# for the lapi those configs will be loaded : profiles.yaml, notifications, console.yaml
config:
  # -- to better understand how to write a postoverflow, you can take a look at (https://docs.crowdsec.net/docs/next/whitelist/create/#whitelist-in-postoverflows)
  postoverflows:
    s01-whitelist:
      mydyndns-whitelist.yaml: |
        name: my/dyndnswhitelist ## Must be unqiue
        description: "Whitelist events from my own dynamic IP"
        whitelist:
          reason: "My dynamic IP"
          expression:
              - evt.Overflow.Alert.Source.IP in LookupHost("test.net")

tls:
  enabled: true
  bouncer:
    reflector:
      namespaces: ["traefik"]

# lapi will deploy pod with crowdsec lapi and dashboard as deployment
lapi:
  # -- environment variables from crowdsecurity/crowdsec docker image
  env:
    # by default disable the agent because it only needs the local API.
    # - name: DISABLE_ONLINE_API
    #   value: "true"
    - name: ENROLL_KEY
      valueFrom:
        secretKeyRef:
          name: crowdsec-credentials
          key: enroll-key
    - name: ENROLL_INSTANCE_NAME
      value: "Homecluster"
    - name: ENROLL_TAGS
      value: "k3s linux home"

  resources:
    limits:
      memory: 350Mi
    requests:
      cpu: 15m
      memory: 350Mi

  # -- Enable persistent volumes
  persistentVolume:
    # -- Persistent volume for data folder. Stores e.g. registered bouncer api keys
    data:
      enabled: true
      accessModes:
        - ReadWriteOnce
      storageClassName: ceph-block
      size: 5Gi
    # -- Persistent volume for config folder. Stores e.g. online api credentials
    config:
      enabled: true
      accessModes:
        - ReadWriteOnce
      storageClassName: ceph-block
      size: 100Mi

  # -- Enable service monitoring (exposes "metrics" port "6060" for Prometheus)
  metrics:
    enabled: true
    # -- Creates a ServiceMonitor so Prometheus will monitor this service
    # -- Prometheus needs to be configured to watch on all namespaces for ServiceMonitors
    # -- See the documentation: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusioscrape
    # -- See also: https://github.com/prometheus-community/helm-charts/issues/106#issuecomment-700847774
    serviceMonitor:
      enabled: true
      additionalLabels: {}

# agent will deploy pod on every node as daemonSet to read wanted pods logs
agent:
  acquisition:
    # -- Specify each pod you want to process it logs (namespace, podName and program)
    # Traefik
    - namespace: "traefik"
      # -- to select pod logs to process
      podName: "traefik-*"
      # -- program name related to specific parser you will use (see https://hub.crowdsec.net/author/crowdsecurity/configurations/docker-logs)
      program: "traefik" #nginx
      # -- If set to true, will poll the files using os.Stat instead of using inotify
      poll_without_inotify: false
    # Authelia
    - namespace: "authelia"
      podName: "authelia-*"
      program: "authelia"

  resources:
    limits:
      memory: 350Mi
    requests:
      cpu: 15m
      memory: 350Mi

  # -- Enable persistent volumes
  persistentVolume:
    # -- Persistent volume for config folder. Stores local config (parsers, scenarios etc.)
    config:
      enabled: true
      accessModes:
        - ReadWriteMany
      storageClassName: ceph-filesystem
      existingClaim: ""
      size: 100Mi

  # -- environment variables from crowdsecurity/crowdsec docker image
  env:
    # by default we configure the docker-logs parser to be able to parse docker logs in k8s
    # by default we disable local API on the agent pod
    # - name: SCENARIOS
    #   value: "scenario/name otherScenario/name"
    # - name: PARSERS
    #   value: "parser/name otherParser/name"
    # - name: POSTOVERFLOWS
    #   value: "postoverflow/name otherPostoverflow/name"
    # - name: CONFIG_FILE
    #   value: "/etc/crowdsec/config.yaml"
    # - name: DSN
    #   value: "file:///var/log/toto.log"
    # - name: TYPE
    #   value: "Labels.type_for_time-machine_mode"
    # - name: TEST_MODE
    #   value: "false"
    # - name: TZ
    #   value: ""
    # - name: DISABLE_AGENT
    #   value: "false"
    # - name: DISABLE_ONLINE_API
    #   value: "false"
    # - name: LEVEL_TRACE
    #   value: "false"
    # - name: LEVEL_DEBUG
    #   value: "false"
    # - name: LEVEL_INFO
    #   value: "false"
    - name: PARSERS
      value: "crowdsecurity/cri-logs"
    - name: COLLECTIONS
      value: "crowdsecurity/traefik LePresidente/authelia"
    # When testing, allow bans on private networks
    # - name: DISABLE_PARSERS
    #   value: "crowdsecurity/whitelists"

  # -- Enable service monitoring (exposes "metrics" port "6060" for Prometheus)
  metrics:
    enabled: true
    # -- Creates a ServiceMonitor so Prometheus will monitor this service
    # -- Prometheus needs to be configured to watch on all namespaces for ServiceMonitors
    # -- See the documentation: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#prometheusioscrape
    # -- See also: https://github.com/prometheus-community/helm-charts/issues/106#issuecomment-700847774
    serviceMonitor:
      enabled: true
      additionalLabels: {}

The agent pods come up without trouble.
Any idea, where I could look further into? Or is it a bug?

/kind bug

Hello,
Can you provide the entire log output from the LAPI pod ?
Can you also share the content of /etc/crowdsec/config.yaml from the volume mounted in the LAPI pod ?
We have changed the way we handle CAPI credentials to not require a persistent volume anymore to persists them, and we had no issue upgrading clusters during the testing phase, but it seems we have missed a corner case.

I'm facing the same error.
LAPI pod log output
Populating configuration directory...
sending incremental file list
acquis.yaml
config.yaml
console.yaml
dev.yaml
local_api_credentials.yaml
online_api_credentials.yaml
simulation.yaml
user.yaml
acquis.d/
appsec-configs/
appsec-rules/
collections/
collections/linux.yaml -> /etc/crowdsec/hub/collections/crowdsecurity/linux.yaml
collections/sshd.yaml -> /etc/crowdsec/hub/collections/crowdsecurity/sshd.yaml
console/
console/context.yaml
contexts/
contexts/bf_base.yaml -> /etc/crowdsec/hub/contexts/crowdsecurity/bf_base.yaml
hub/
hub/.index.json
hub/collections/
hub/collections/crowdsecurity/
hub/collections/crowdsecurity/linux.yaml
hub/collections/crowdsecurity/sshd.yaml
hub/contexts/
hub/contexts/crowdsecurity/
hub/contexts/crowdsecurity/bf_base.yaml
hub/parsers/
hub/parsers/s00-raw/
hub/parsers/s00-raw/crowdsecurity/
hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml
hub/parsers/s01-parse/
hub/parsers/s01-parse/crowdsecurity/
hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml
hub/parsers/s02-enrich/
hub/parsers/s02-enrich/crowdsecurity/
hub/parsers/s02-enrich/crowdsecurity/dateparse-enrich.yaml
hub/parsers/s02-enrich/crowdsecurity/geoip-enrich.yaml
hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml
hub/scenarios/
hub/scenarios/crowdsecurity/
hub/scenarios/crowdsecurity/ssh-bf.yaml
hub/scenarios/crowdsecurity/ssh-cve-2024-6387.yaml
hub/scenarios/crowdsecurity/ssh-slow-bf.yaml
notifications/
notifications/email.yaml
notifications/sentinel.yaml
notifications/slack.yaml
notifications/splunk.yaml
parsers/
parsers/s00-raw/
parsers/s00-raw/syslog-logs.yaml -> /etc/crowdsec/hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml
parsers/s01-parse/
parsers/s01-parse/sshd-logs.yaml -> /etc/crowdsec/hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml
parsers/s02-enrich/
parsers/s02-enrich/dateparse-enrich.yaml -> /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/dateparse-enrich.yaml
parsers/s02-enrich/geoip-enrich.yaml -> /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/geoip-enrich.yaml
parsers/s02-enrich/whitelists.yaml -> /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml
patterns/
patterns/aws
patterns/bacula
patterns/bro
patterns/cowrie_honeypot
patterns/exim
patterns/firewalls
patterns/haproxy
patterns/java
patterns/junos
patterns/linux-syslog
patterns/mcollective
patterns/modsecurity
patterns/mongodb
patterns/mysql
patterns/nagios
patterns/nginx
patterns/paths
patterns/postgresql
patterns/rails
patterns/redis
patterns/ruby
patterns/smb
patterns/ssh
patterns/tcpdump
postoverflows/
scenarios/
scenarios/ssh-bf.yaml -> /etc/crowdsec/hub/scenarios/crowdsecurity/ssh-bf.yaml
scenarios/ssh-cve-2024-6387.yaml -> /etc/crowdsec/hub/scenarios/crowdsecurity/ssh-cve-2024-6387.yaml
scenarios/ssh-slow-bf.yaml -> /etc/crowdsec/hub/scenarios/crowdsecurity/ssh-slow-bf.yaml
sent 811,961 bytes  received 1,104 bytes  1,626,130.00 bytes/sec
total size is 810,010  speedup is 1.00
Error: no matches found
Generate local agent credentials
Machine 'crowdsec-lapi-76677cd6cc-8rm4q' successfully added to the local API.
API credentials written to '/etc/crowdsec/local_api_credentials.yaml'.
Check if lapi needs to register an additional agent
Machine 'zxqv2mgdxUWf8O5eVhIF0b8RXcgKBi5aCaF2CvQkcNr22UQe' successfully added to the local API.
API credentials written to '/dev/null'.
Agent registered to lapi
level=fatal msg="no configuration for Central API (CAPI) in '/etc/crowdsec/config.yaml'"

There's no /etc/crowdsec/config.yaml volume mounted in the LAPI pod as far as I can tell. Here's the pod YAML file
apiVersion: v1
kind: Pod
metadata:
  name: crowdsec-lapi-76677cd6cc-8rm4q
  namespace: crowdsec
spec:
  containers:
    - env:
        - name: LOCAL_API_URL
          value: http://localhost:8080
        - name: AGENT_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: agent-credentials
        - name: AGENT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: agent-credentials
        - name: DISABLE_AGENT
          value: 'true'
        - name: INSECURE_SKIP_VERIFY
          value: 'false'
        - name: CS_LAPI_SECRET
          valueFrom:
            secretKeyRef:
              key: csLapiSecret
              name: crowdsec-lapi-secrets
        - name: REGISTRATION_TOKEN
          valueFrom:
            secretKeyRef:
              key: registrationToken
              name: crowdsec-lapi-secrets
        - name: CUSTOM_HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: DISABLE_ONLINE_API
          value: 'false'
        - name: ENROLL_KEY
          valueFrom:
            secretKeyRef:
              key: crowdsec_enroll_key
              name: crowdsec-credentials
        - name: ENROLL_INSTANCE_NAME
          value: Talos_Cluster
        - name: ENROLL_TAGS
          value: homelab
        - name: DISABLE_AGENT
          value: 'true'
      image: crowdsecurity/crowdsec:v1.6.3
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /health
          port: lapi
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      name: crowdsec-lapi
      ports:
        - containerPort: 8080
          name: lapi
          protocol: TCP
        - containerPort: 6060
          name: metrics
          protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /health
          port: lapi
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      resources:
        limits:
          cpu: 500m
          memory: 500Mi
        requests:
          cpu: 500m
          memory: 500Mi
      securityContext:
        allowPrivilegeEscalation: false
        privileged: false
      startupProbe:
        failureThreshold: 30
        httpGet:
          path: /health
          port: lapi
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /docker_start.sh
          name: crowdsec-docker-start-script
          subPath: docker_start.sh
        - mountPath: /staging/etc/crowdsec/online_api_credentials.yaml
          name: crowdsec-capi-credentials-volume
          subPath: online_api_credentials.yaml
        - mountPath: /etc/crowdsec/profiles.yaml
          name: crowdsec-profiles-volume
          subPath: profiles.yaml
        - mountPath: /etc/crowdsec/config.yaml.local
          name: crowdsec-config-local-volume
          subPath: config.yaml.local
        - mountPath: /etc/crowdsec/notifications/http.yaml
          name: crowdsec-notifications-http
          subPath: http.yaml
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-f5qjg
          readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: talos-worker-03
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  volumes:
    - configMap:
        defaultMode: 420
        name: crowdsec-docker-start-script-configmap
      name: crowdsec-docker-start-script
    - name: crowdsec-capi-credentials-volume
      secret:
        defaultMode: 420
        secretName: crowdsec-capi-credentials
    - configMap:
        defaultMode: 420
        name: crowdsec-profiles
      name: crowdsec-profiles-volume
    - configMap:
        defaultMode: 420
        name: crowdsec-config-local
      name: crowdsec-config-local-volume
    - configMap:
        defaultMode: 420
        items:
          - key: http.yaml
            path: http.yaml
        name: crowdsec-notifications
      name: crowdsec-notifications-http
    - name: kube-api-access-f5qjg
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace

@srkoster can you try to set lapi.storeCAPICredentialsInSecret to true in your values ?
This should make crowdsec regenerate the credentials (if you had no persistent volume for the LAPI config before this release, crowdsec would always regenerate CAPI credentials on start, which is never what you want)

@blotus I already have set lapi.storeCAPICredentialsInSecret to true. Also, I have disabled all lapi persistence, have disabled tls and am using postgresql as my database.
I'll post my values.yaml later today for reference.

Hey, im having the same issue after updating the chart's version. I tried the proposed solution by disabling persistence and setting storeCAPICredentialsInSecret to true. Before that i also had problem with my custom profiles.yaml and notification's http.yaml config files. LAPI pod was complaining about not being able to copy them over cos they already exist, so i had to temporary comment them out in my helmrelease but then this issue came up. Here is my helmrelease file for reference.
kind: HelmRelease
metadata:
  name: crowdsec
spec:
  interval: 30m
  chart:
    spec:
      chart: crowdsec
      version: 0.14.0
      sourceRef:
        kind: HelmRepository
        name: crowdsec
        namespace: flux-system
  maxHistory: 3
  install:
    remediation:
      retries: 3
  upgrade:
    cleanupOnFail: true
    remediation:
      strategy: rollback
      retries: 3

  values:
    config:

      #profiles.yaml: |
        #name: default_ip_remediation
          #debug: true
          #filters:
          #- Alert.Remediation == true && Alert.GetScope() == "Ip"
          #decisions:
          #- type: ban
            #duration: 4h
          #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
          #notifications:
            #- pushover
          #on_success: break
          #---
          #name: default_range_remediation
          #debug: true
          #filters:
          #- Alert.Remediation == true && Alert.GetScope() == "Range"
          #decisions:
          #- type: ban
            #duration: 4h
          #duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)
          #notifications:
            #- pushover
          #on_success: break

      #notifications:
        #http.yaml: |
          #type: http
          #name: pushover

          #log_level: info

          #group_wait: "30s"
          #group_threshold: 10
          #max_retry: 5
          #timeout: "10s"

          #-------------------------
          # plugin-specific options

          # The following template receives a list of models.Alert objects
          # The output goes in the http request body
          #format: |
            #{
            #"token": $$CROWDSEC_PUSHOVER_TOKEN,
            #"user": $$PUSHOVER_USER_KEY,
            #"message": "{{`{{range . -}}{{$alert := . -}}{{range .Decisions -}}{{.Value}} will get {{.Type}} for the next {{.Duration}} for triggering {{.Scenario}}.\r\n https://www.shodan.io/host/{{.Value}}{{end -}}{{end -}}`}}",
            #"html": "1",
            #"title": "Scenario triggered on IDS/IPS !"
            #}
          #url: https://api.pushover.net/1/messages.json
          #method: POST
          #headers:
            #Content-Type: "application/json"

      config.yaml.local: |
        db_config:
          type:     postgresql
          user:     $${DB_USER}
          password: $${DB_PASSWORD}
          db_name:  crowdsec
          host:     postgres17-rw.database.svc.cluster.local
          port:     5432
          sslmode:  require

    container_runtime: containerd

    agent:
      wait_for_lapi:
        image:
          tag: "1.37.0@sha256:768e5c6f5cb6db0794eec98dc7a967f40631746c32232b78a3105fb946f3ab83"

      acquisition:
        - namespace: network
          podName: nginx-external-controller-*
          program: nginx

        - namespace: media
          podName: jellyseerr-*
          program: jellyseerr

        - namespace: vaultwarden
          podName: vaultwarden-*
          program: VAULTWARDEN

        - namespace: idp
          podName: keycloak-0
          program: keycloak

      env:
        - name: LOCAL_API_URL
          value: "https://crowdsec-api.${PUBLIC_DOMAIN}"

        - name: PARSERS
          value: "crowdsecurity/cri-logs crowdsecurity/whitelists crowdsecurity/geoip-enrich"

        - name: COLLECTIONS
          value: "crowdsecurity/modsecurity inherent-io/keycloak LePresidente/jellyseerr crowdsecurity/nginx Dominic-Wagner/vaultwarden crowdsecurity/base-http-scenarios crowdsecurity/http-cve crowdsecurity/http-dos crowdsecurity/whitelist-good-actors"

      metrics:
        enabled: true
        serviceMonitor:
          enabled: true

    lapi:
      deployAnnotations:
        reloader.stakater.com/auto: "true"

      extraInitContainers:
        - name: initdb
          image: "ghcr.io/onedr0p/postgres-init:16.5@sha256:53fa938cfe6d2dae23b120145362341950f1154bf9d0e7f2ec4e471d61624f79"
          imagePullPolicy: IfNotPresent

          envFrom:
            - secretRef:
                name: &secret crowdsec-secret

      persistentVolume:
        data:
          enabled: false

        config:
          enabled: false

      storeCAPICredentialsInSecret: true

      ingress:
        enabled: true

        annotations:
          nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
          external-dns.alpha.kubernetes.io/target: internal.${PUBLIC_DOMAIN}
          cert-manager.io/cluster-issuer: zerossl-prod

        ingressClassName: internal

        host: &host "crowdsec-api.${PUBLIC_DOMAIN}"

        tls:
          - hosts:
            - *host
            secretName: crowdsec-tls

      env:
        - name: CROWDSEC_PUSHOVER_TOKEN
          valueFrom:
            secretKeyRef:
              name: *secret
              key: CROWDSEC_PUSHOVER_TOKEN

        - name: PUSHOVER_USER_KEY
          valueFrom:
            secretKeyRef:
              name: *secret
              key: PUSHOVER_USER_KEY

        - name: DISABLE_ONLINE_API
          value: "false"

        - name: ENROLL_KEY
          valueFrom:
            secretKeyRef:
              name: *secret
              key: ENROLL_KEY

        - name: ENROLL_INSTANCE_NAME
          value: "cluster"

        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: *secret
              key: INIT_POSTGRES_PASS

        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: *secret
              key: INIT_POSTGRES_USER

        - name: BOUNCER_KEY_NGINX_EXTERNAL
          valueFrom:
            secretKeyRef:
              name: *secret
              key: BOUNCER_KEY_NGINX_EXTERNAL

      metrics:
        enabled: true
        serviceMonitor:
          enabled: true

Hello,
I've managed to reproduce the issue locally.
It's caused by the ENROLL_KEY env var in the LAPI pod configuration.
If you remove it, the pod should start fine. I'm starting to work on a fix.

Good to hear that you've found the issue. I can confirm that my pods can start when I remove the ENROLL_KEY env var in the LAPI pod configuration. However, removing it removes the capability to enroll the engine into the online console. Waiting for your patch and hope that you'll find a solution for the enrollment.

Hello,
I've just released version 0.14.1 which should fix the issue.
Please let me know if you still encounter any errors with it.

I can confirm that the fix worked. Thanks for your effort.


Request your work to be included in the awesome Zepp OS project

Hello @CuberQAQ
Please allow me to introduce myself, I am Myoung XUE, an engineer from Zepp OS Team.
We have an awesome project https://github.com/zepp-health/awesome-zeppos, as the name suggests, where we include many projects from the community that are relevant to Zepp OS.
We noticed that several of your projects are publicly available on Github and would like to ask your permission to include them in the official awesome zeppos repository.
Hopefully, we can get your approval and your project will help more developers in the community.

Ohhhhhhh, nice to meet you. But I am afraid I don't have enough time to complete this project because of my busy studies. Now the project has been suspended. I will continue it when I was free!!!


Failing tests

New issue with requirements syntax, likely due to new pip release 24.1 (specifically 24.1b1 dependency specifiers).

numpy2.0 has some 32 vs 64 changes, pin old version for now (#26 )


Envoy deprecation: use of enabled in CorsPolicy

We have ~ 3 months to address this deprecation:
Use of enabled in CorsPolicy, found in route.proto. Set the filter_enabled field instead.
https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md#version-1100-pending
cc @duderino

@andraxylia can I work on it this issue, I'm not well versed with code-base but want to give it a try and figure things out.


AnthropicChat fails when "until" is not provided explicitly in "generation_kwargs" or contains whitespace-only options

While adding a GLUE-style benchmark for the Belarusian language, I noticed unexpected behavior. Below are details and example based on the nq_open task. I can suggest a fix and implement it, but first decided to align on that by reporting a bug.
Description
The AnthropicChat is failing when the "until" parameter is not explicitly provided in the "generation_kwargs" or contains whitespace-only option. This leads to an API request failure due to an invalid stop sequence value.

"generation_kwargs" should be provided in the config.
By default, "until" is set to "\n\n" (code link).
When the payload for the Anthropic API is created (code link), "gen_kwargs" already has the "until" key in the dict. This prevents the fallback option ["\n\nHuman:"] from being used.
The Anthropic API requires a non-whitespace sequences for "stop_sequences", causing the request to fail when at least one is present.

Steps to reproduce

Code version is currently the latest version of the code from main (commit 9b052fdc)
Use the nq_open configuration with (contains whitespace-only option) or without the "until" section (set to "\n\n"):

task: nq_open
dataset_path: nq_open
output_type: generate_until
training_split: train
validation_split: validation
description: "Answer these questions:\n\n"
doc_to_text: "Q: {{question}}?\nA:"
doc_to_target: "{{answer}}" # TODO: should be multi-target
fewshot_delimiter: "\n"
generation_kwargs:
  do_sample: false
  temperature: 0.0
filter_list:
  - name: remove_whitespace
    filter:
      - function: remove_whitespace
      - function: take_first
target_delimiter: " "
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: true
    regexes_to_ignore:
    - "\\b(?:The |the |An |A |The |a |an )"
metadata:
  version: 4.0

Run the evaluation using this configuration.

ANTHROPIC_API_KEY="{your_api_key}" lm_eval \
  --model="anthropic-chat-completions" \
  --model_args="model=claude-3-5-sonnet-20240620" \
  --tasks="nq_open" \
  --apply_chat_template

The evaluation fails with the following error message:

API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...

Workaround
The bug can be easily fixed by excluding all whitespace-only "until" options and explicitly adding non-whitespace ones to override the default "\n\n" value.
Traceback
2024-10-20:17:04:34,449 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-20:17:04:37,942 INFO     [__main__.py:376] Selected Tasks: ['nq_open']
2024-10-20:17:04:37,945 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-20:17:04:37,945 INFO     [evaluator.py:201] Initializing anthropic-chat-completions model, with arguments: {'model': 'claude-3-5-sonnet-20240620'}
2024-10-20:17:04:37,945 INFO     [api_models.py:111] Using max length 2048 - 1
2024-10-20:17:04:37,945 INFO     [api_models.py:114] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-10-20:17:04:37,945 INFO     [api_models.py:124] Using tokenizer None
2024-10-20:17:04:37,945 WARNING  [anthropic_llms.py:287] Chat completions does not support batching. Defaulting to batch size 1.
2024-10-20:17:04:37,945 WARNING  [anthropic_llms.py:292] Using Anthropic Version: 2023-06-01. Confirm the current version here: https://docs.anthropic.com/en/api/versioning
2024-10-20:17:04:44,347 INFO     [task.py:415] Building contexts for nq_open on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3610/3610 [00:00<00:00, 4208.38it/s]
2024-10-20:17:04:45,237 INFO     [evaluator.py:489] Running generate_until requests
Requesting API:   0%|                                                                                | 0/3610 [00:00<?, ?it/s]2024-10-20:17:04:45,445 WARNING  [api_models.py:347] API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
2024-10-20:17:04:46,666 WARNING  [api_models.py:347] API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
2024-10-20:17:04:47,884 WARNING  [api_models.py:347] API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
Traceback (most recent call last):
  File "{path_to_code}/lm-evaluation-harness/.venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/evaluator.py", line 500, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/models/api_models.py", line 569, in generate_until
    outputs = retry(
              ^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_python}/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "{path_to_python}/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/models/api_models.py", line 350, in model_call
    response.raise_for_status()
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.anthropic.com/v1/messages
Requesting API:   0%|                                                                                | 0/3610 [00:02<?, ?it/s]

Hi! Thanks for reporting this, and your solution seems very reasonable. If you have the time, a PR will be greatly appreciated. Otherwise, I'll try implementing it. :)

@baberabb I'll do that and submit a PR


Remove ui-components dependency from application

This addresses point 3 in https://github.com/elyra-ai/elyra/issues/785#issuecomment-664719618
This moves the error handling code in requests.tsx that depends on ui-components into ui-components and replaces them with a Promise reject
Then it wraps all the calls to requests.tsx in try catch blocks that call the error handling that was moved to ui-components
This PR is built on top of https://github.com/elyra-ai/elyra/issues/785#issuecomment-664719618 and thus also includes all it's changes
Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

For reference this is what the dependency tree looks like after these changes:


One-time binding and load static files do not translate

Subject of the issue
One-time binding in filter and load static files do not translate before ui-router navigate somewhere
Your environment

angular-translate 2.10
angular.js 1.5.0
latest google chrome linux xubuntu

Steps to reproduce
This plunkr reproduce main issue: https://plnkr.co/edit/SJtSpPSlQddpfd3plDlx?p=preview
Just see one-time binding and many times binding how they work.
Expected behaviour
Translate should load static translation and apply it to even one-time bindings. Because one-time bindings it is an performance solution.
Actual behaviour
However we see nothing translation for block {{:'hello'|translate}}
But only before ui-router navigate somewhere and back

Your expectation is not right when using one-time binding.
If you want to use one-time-binding, the filter will be evaluated only one time (well, the filter's feature is naming it). Any later loadings cannot be resolved again.
That is why one-time-binding + asynchronous loading of translations cannot work by design. You cannot have both.
However, if you can ensure the filter will be called after the translations are already loaded, perhaps you will have a solution. But this requires a custom setup, perhaps with a combination of your router/resolving events.

Solution here https://atticuswhite.com/blog/angularjs-translate-once-angular-translate-one-time-binding/


Yordis/add interleave config to plugin

@yordis bumping this

@ScriptedAlchemy did you added it to another branch? I am sorry for the delay


PF feedback

"Don't generate Rules"
-- this would allow to add a selfdefined rule to the interface of your liking, usually WAN only.
At the moment, we get a floating rule for all interfaces, which isn't very optimal since i have a lot of vlan communication between interfaces.
Opnsense is the router for all networks
Adding as an issue so user can track the outcome

In the pfsense plugin, we create the ipv4/ipv6 rules (which are not editable nor shown in the interface) but they can be disabled. The user is free to add their own.

Is this enough if we do the same for opnsense? I understand an explicit selection of the interface would be preferable, we would need to see how this translates in both opnsense and pfsense

@Ramalama2


removed boost in AI as well

[ x ] The tests are passing

Summary
When testing Yuhanun PR in Roboteam Utils, I found the Keeper still uses boost. This fixes that too.
fixes Boost dependency
Comments for the reviewer
If there are comments for the reviewer, leave them here
Checklist for the reviewer
This acts as a guide to help you and/or the reviewer to analyze the code.
functionality

Is the code is properly tested?
Is the code working as expected
Is the code is understandable and easy to read
Are proper abstractions made? (no duplicate code, nice use of functions/objects)

style

Variables/objects/classes have proper names
Proper use of variables: nu unused vars, no magic numbers
No possible memory leaks. Mutexes properly used when multithreading? possible pointer issues?
Are smart pointers used?
unused variables?
magic numbers?
well documented?
No unused imports?
No leftover code / unused comments
Proper indentation
Proper use of namespaces

duplicate: https://github.com/RoboTeamTwente/roboteam_ai/pull/711