Question about changing model in system and real-time issue

Hi @juanmc2005 ! it's me again, sorry for bothering you...
I have several questions

I have already fine-tuned a segmentation model with other dataset (not AIM, VoxConverse) and store in ".ckpt" model. I sure that it can be loaded by using
ckpt_path = "path/where/store/ckpt/epoch=9_best.ckpt"
Model.from_pretrained(ckpt_path, strict=False)
i want to put replace this model in your system for some testing. But when i change here by segmentation: PipelineModel = Model.from_pretrained('path/where/store/ckpt/epoch=9_best.ckpt'). When using "python -m diart.stream", the system is still using "pyannote/segmentation". could you give me some hints?
Is there any useful augmentation method for training or fine-tuning? I know that can add back noise or something. Where should i add them?
this system can use for real-time right (streaming)? If yes, how to confirm "real-time"? like real-time factor or something else?

Thanks for your awesome project, it helps me a lot!!
Looking forward to your reply~~
Best Regards

Hi @Shoawen0213,

If you want to run your own pipeline, you should write a separate script and use diart's python API. Right now diart.stream does not support running custom pipelines (although this is something I'd like to add in the future).
I recently released version 0.3 where there's a new inference API. I would recommend you to start with the template from here and set the model in your PipelineConfig like this:

my_model = Model.from_pretrained('path/to/model.ckpt')
pipeline = OnlineSpeakerDiarization(PipelineConfig(segmentation=my_model))

As an example, pyannote/segmentation is trained with additive background noise and impulse responses. You should do this while training. I recommend you take a look at torch-audiomentations from the asteroid team.

Yes, this is the main goal of the library. In order for your system to be able to actually run in real time you need to make sure that the real time latency (the processing time of an audio chunk) is lower than the step. For example if you send new chunks to the system every 500ms (step=0.5) and your real time latency is 1s, then chunks will accumulate faster than you are able to make predictions.
In the new inference API that I mentioned above, there's also a Benchmark class that allows you to measure the real-time latency of your pipeline (when you set batch_size=1).
For more info on Benchmark see here and here.

I hope this answers your questions.
Don't hesitate to ask if the usage of the inference API is not clear.

Hi! @juanmc2005 , very much appreciate your reply!!
For question 1, i fix the problem and it seems that it can load my own segmentation model. I will try your recommendation.
In this part, I would like to ask for some advice. If i want to evaluate in my own dataset, I can fine-tune the segmentation model then put it into the pipeline(like mentioned above). In your opinion, do I need to fine-tune the embedding model ?
For question 2, i have seen torch-audiomentations before, but i'm not sure where to add it when doing training. could you give some hint or template in this situation?
For question 3, forgive my stupidity.
so, here means "send new chunks to the system every 500ms (step=0.5) " and here  means the real time latency?
I have read the paper several times in these several weeks. I'm still confused about these two meanings.
In my understanding, the rolling buffer is fix in "5s", right?
and the latency means to output the result every latency, which means, for example, if the latency is 3s, then the system will output the result every 3 seconds? or it means the chunk size?
Then i don;t know step means truly...
sorry for bothering...
I will try the Benchmark API after i clarify my confusion!!
Thank you for your guidance and support.
It is hard to find words to express my gratitude.

In your opinion, do I need to fine-tune the embedding model ?

I think this depends a lot on your specific application. If you have reasons to believe that the training domain of pyannote/embedding is considerably different of your target domain, then I'd suggest you fine-tune it as well. However, for most purposes pyannote/embedding should do fine.
Another reason you might want to train your own embedding model is if you can have a better one, although this might change the architecture and not necessarily be compatible with diart's default pipeline because the model needs to accept per-speaker weights. There was a similar discussion concerning speechbrain embeddings in #34.

For question 2, i have seen torch-audiomentations before, but i'm not sure where to add it when doing training. could you give some hint or template in this situation?

This is about training, so not related to diart. I would suggest you ask this question in the pyannote.audio repo.

so, here means "send new chunks to the system every 500ms (step=0.5)"

Yes.

and here means the real time latency?

No. The system makes a prediction at every step (500ms), and the size of the input chunk is fixed (5s). What changes with the latency argument is the timestamp of the prediction.
This might be confusing because there are 2 different latencies in the pipeline.
The one that you mention here is the system latency. It answers to the question "at a current recording time t, for what time in the recording is the system giving me an answer?".
If the system is given a conversation chunk that goes from 5s to 10s, then t=10. If the latency is 3s, the output of the system is from t - latency == 7s to t - latency + step == 7.5s. When it moves to the next chunk (from 5.5s to 10.5s), the output will be from 7.5s to 8s.
The real-time latency is something entirely different and it is not set by the user. It answers to the question "how much processing time does it take (on a given CPU/GPU) for the system to calculate the prediction of a given chunk?".
While the former is a hyper-parameter set by the user, the latter is a property.
The system latency allows you to be more/less tolerant with delays in the answer, it plays with the perceived responsiveness on purpose to let the system have more information and make better predictions.
The real-time latency is affected by the machine in which you're running the algorithm (CPU and/or GPU), the inference time of both segmentation and embedding models, the complexity of clustering, etc.

Hi! @juanmc2005 Thanks for your clear reply for question 1 and I have opened an issue on pyannote.audio for question 2.
Hope to have some answers or suggestion QAQ.
For this part ↓

If the system is given a conversation chunk that goes from 5s to 10s, then t=10. If the latency is 3s, the output of the system is from t - latency == 7s to t - latency + step == 7.5s. When it moves to the next chunk (from 5.5s to 10.5s), the output will be from 7.5s to 8s.

 Please correct me if i misunderstand anything!

From my understanding, there's an input chuck is fixed in 5s as input and it sent into the system every 0.5s, which means the system will get the duration of 5s per 0.5s ?
Then the system might contain a rolling buffer to store the input chunk (5s).
as shown in the figure, if the latency is 3s and the step is 0.5s (does this step the same as mentioned above?)
then can i realize that the system will output the diart result at 7s to 7.5s to represent the result of the 5s to 5.5s audio in the buffer? And if the latency is set to 5, then the system will output the diart result at 5s to 5.5s to represent the result of the 5s to 5.5s audio in the buffer?
Sorry for bothering and please forgive my stupidity.

From my understanding, there's an input chuck is fixed in 5s as input and it sent into the system every 0.5s, which means the system will get the duration of 5s per 0.5s ?

The system will receive 5s of audio every 500ms. Since it's a sliding window over the conversation (this is what we call a rolling buffer), chunk i will overlap the last 4.5s with the first 4.5s of chunk i + 1. This is depicted in Figure 4 of the paper.

does this step the same as mentioned above?

Yes there is only one notion of step.

then can i realize that the system will output the diart result at 7s to 7.5s to represent the result of the 5s to 5.5s audio in the buffer?

No, the timestamps of the input and the output are aligned, so an output for 7s-7.5s is a prediction for the same 7s-7.5s in the input audio.

And if the latency is set to 5, then the system will output the diart result at 5s to 5.5s to represent the result of the 5s to 5.5s audio in the buffer?

Yes, because the current time t is 10s, so in order to understand what region of the input the output corresponds to you have to do the following:
output_start_time = t (10s) - latency (5s) = 5s
output_end_time = output_start_time + step (0.5s) = 5.5s

The duration of the output is always the size of the step, this is completely independent from the latency.
Notice that if you try different latencies in the formula above, it's as if you were "moving" the focus of the system at a given chunk. This is why lower latencies are more "responsive": the output window is closer to t, whereas a higher latency reduces the responsiveness by having a bigger delay in the prediction, i.e. the output window is farther away from t.
However, the latency can't exceed 5s (the chunk size) because it's out of the scope of the current chunk, making it essentially the same as a 5s latency. And it can't be lower than 0.5s (the step) either because a single update already includes 0.5s of audio, so getting closer to t would only make the system miss audio regions.

Gosh! I finally understand this part of this awesome project!!!!
Thanks for your clear and kind reply! Also appreciate your patience!
I'm very very grateful for your help!!
One more small question, the paper indicated that the system has the same performance as the official offline baseline of
the DIHARD III challenge by setting the latency as 3s(25.5% vs 25.4%). But why not said that the system can beat the official offline baseline when setting the latency as 4s or 5s? Are there any considerations?

I'm glad it's clear now :)
At this point you should probably be able to understand quite a bit of diart's code, so if you ever think of some interesting feature or you find a bug, any contribution/pull request is more than welcome!

But why not said that the system can beat the official offline baseline when setting the latency as 4s or 5s? Are there any considerations?

That is also shown, although not specifically written. I guess the takeaway from that sentence is that we succeed in obtaining the same performance with much lower requirements: low system latency, very low real-time latency (165ms on CPU), online processing, and with continual improvement.

Thanks for your clear reply!!
Very grateful for your help!!
I will try to understand more about diart's code!
If I have other questions about code or the paper, I will beg for your help again QQ

AH!!! I figure out that i forget to ask a simple question.

same performance with much lower requirements: low system latency, very low real-time latency (165ms on CPU)

for here, I saw that the paper indicates 165ms on CPU or 50ms on GPU.
Does the diart code default using the GPU?

In version 0.3 it defaults to GPU yes, both in the python API and diart.stream/diart.inference.
Adding --cpu when you run the script forces CPU usage.

@juanmc2005 Got it!! thanks for your answer~~


sambamba and piped files

Hi,
I am running the pipeline on a server and it is crashing when the pipeline runs multiple alignments in parallel. We suspect sambamba is playing funny with the pipes/fifos and use of temps.
Have you have any experience with this?
Can you give some hints on where sambamba is storing temp files to guide us in here debugging this error.
thanks a lot in advance and have a great Xmas guys ... please do not assume I expect a answer before xmas :)

Closing this out as its pretty stale and we haven't been getting reports of issues like this recently. Please reopen if it is still a problem. Thanks!


Feature Request

Simple Shortcut Key.
seriously just a simple UI option to just open the program with a simple key stroke.

You create a shortcut on the desktop and then set the shortcut key. (GeForce Experience recording function needs to be closed)

I've seen some keyboards come with a dedicated calculator key to open the Microsoft calculator, but I've only seen this on laptop keyboards.


python3Packages.scapy: build with libpcap

Description of changes
This allows doing packet capture directly from scapy.
Fixes https://github.com/NixOS/nixpkgs/issues/186294.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

There's no build regressions, landing. (The few packages that nixpkgs-review says fail to build fail already on master.)


Support large scale jitter

Motivation
Support stronger baselines of Mask R-CNN that uses large-scale jitter with long training schedule.
Modification
Several configs are added.
As the baseline also requires deeper RPN head with two shared convolutional layers, RPNHead is modified without BC-breaking.
Relies on https://github.com/open-mmlab/mmcv/pull/1205
BC-breaking (Optional)
Handled to be compatible with the original case.

lint failed

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: ZwwWayne:x: wayne.zw<EMAIL_ADDRESS>seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi, can I check why is this feature closed? Has it been merged into master?


Settings of repo visible where i have no access

Component
web-ui
Describe the bug
i can see the settings of a repo that i don't have access to
e.g. https://ci.codeberg.org/steko/ci-test/settings
i can change settings there, but get an error

Failed to update the repository settings

i think i shouldn't see this or at least not be able to change the settings (inactive elements)
secrets and registry load forever, so i'm not able to see them, but it should also be more user friendly. like showing "you don't have access to this information."
when i have opened publicly visible repos while being logged in, they will be listed in my repo list an on the left side. it looks like i can disable them, but also get an error. it is an issue that they clutter my list and i can't remove them https://ci.codeberg.org/account/repos
maybe that will be fixed by https://github.com/woodpecker-ci/woodpecker/pull/245? @anbraten
System Info
{"source":"https://github.com/woodpecker-ci/woodpecker","version":"a07f120d"}

Additional context
No response
Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server.

Duplicate of #229
Will be fixed by #245


sensors: Specify sensor channel in DT

Introduction
Configuration parameters in Zephyr are normally set in DT. In particular consumers of a device usually refer to it using phandle-array type, which allows them to specify configuration parameters in addition to the phandle itself.
The goal of this proposal is to enable the specification of a sensor channel through DT.
Problem description
Some temperature sensors expose multiple temperature probes from a single device. The probe to read the temperature from is selected using the sensor channel. Usually driver specific, a.k.a private channels are used for this purpose. See PR with F75303 driver as example - #60833.
In addition to that Zephyr itself defines three different temperature channels:
        /** Device die temperature in degrees Celsius. */
        SENSOR_CHAN_DIE_TEMP,
        /** Ambient temperature in degrees Celsius. */
        SENSOR_CHAN_AMBIENT_TEMP,
        /** Gauge temperature  **/
        SENSOR_CHAN_GAUGE_TEMP,

Application that reads the temperature needs to specify which channel to use. Right now it has to be hardcoded in the source itself. This is not ideal, and definitely not scalable.
Proposed change
This proposal enables the specification of a sensor channel in DT. As mentioned above, it's aimed at consumer applications of temperature sensors, but it might prove to be useful in other types of sensors as well.
Detailed RFC
The change can be split into two parts:

Define #sensor-cells property in sensor-device.yaml binding. This way a sensor device node can be referenced using the phandle-array property type.
Define temperature related sensor channels in a separate header under dt-bindings directory. In the future additional channels might be added.

Proposed change (Detailed)
The Commit with the proposed change can be found here.
To make sure that the #sensor-cells property is inherited by all sensor devices bindings defined in sensor-device.yaml binding. Additionally it’s marked as optional to retain backward compatibility.
  "#sensor-cells":
    type: int
    required: false
    description: |
      Number of items to expect in a sensor specifier.
      By default it's only one attribute is expected - a sensor channel to use.

sensor-cells:
  - channel

Then each DT node of a sensor that wants to use this has to be modified to contain: #sensor-cells = <1>;
To expose all temperature related channels in DT their values have been redefined in a dedicated header under dt-bindings directory.
/* The values were copied over from drivers/sensor.h */
#define DT_SENSOR_CHAN_DIE_TEMP 12
#define DT_SENSOR_CHAN_AMBIENT_TEMP 13
#define DT_SENSOR_CHAN_GAUGE_TEMP 43

The redefinition is necessary for two reasons:

DT parsing logic doesn’t support enums, so extracting enum sensor_channel is not an option.
We can’t remove the enum itself and replace it with macros, as that would break the stable API.

In the consumer DT node the following can be added to reference the temperature sensors
sensors = <&temp_cpu DT_SENSOR_CHAN_DIE_TEMP>,
          <&soc_pct2075 DT_SENSOR_CHAN_AMBIENT_TEMP>,
          <&amb_pct2075 DT_SENSOR_CHAN_AMBIENT_TEMP>;

Dependencies
N/A
Concerns and Unresolved Questions
Admittedly duplication of sensor channel value definitions is not among the best coding practices. Any suggestion on how to implement this in a cleaner way is more than appreciated.
Alternatives

Remove the enum sensor_channel altogether and define sensor channels using macros. This would cause a stable API breakage, hence I decided not to go this way.

should we align with iio? https://github.com/devicetree-org/dt-schema/tree/main/dtschema/schemas/iio

should we align with iio? https://github.com/devicetree-org/dt-schema/tree/main/dtschema/schemas/iio

I'd say no myself. We don't have IIO, and IIO (I tried) turns out to be a poor fit, IMHO, for Zephyr.

I agree with this idea overall, I did something very similar with my pitch for the sensor subsystem (https://github.com/zephyrproject-rtos/zephyr/pull/56963/files#diff-3144f3f020602f03044b01fa08365e64c890a32f059002db005450463f6ab640R12) where I added the sensor types (very very similar to channels). I think having the channels in the devicetree and adding them to the sensor info struct is a great solution.

I think its generally a great idea, but as noted because some sensors provide multiple channels of the same type (the referenced PR #60833) it would be nicer if this could refer to not just the type but also the index of the channel.
Frankly that problem should be solved before this, or minimally this should always refer to channel index 0 to start until the problem is fixed in the sensor API.

I think its generally a great idea, but as noted because some sensors provide multiple channels of the same type (the referenced PR #60833) it would be nicer if this could refer to not just the type but also the index of the channel.

That's very interesting, are you aware of any previous proposal for adding an index parameter to the sensor API?

I think its generally a great idea, but as noted because some sensors provide multiple channels of the same type (the referenced PR #60833) it would be nicer if this could refer to not just the type but also the index of the channel.

That's very interesting, are you aware of any previous proposal for adding an index parameter to the sensor API?

Yes, in my two attempts at updating the API as well as the current iteration from @yperess that supports a decoder, where the channel type is returned as each channel is iterated over.
It's somewhat implicit in the current case though and being discussed further. But this was one of the many requirements laid out in #1387 by @microbuilder

should we align with iio? https://github.com/devicetree-org/dt-schema/tree/main/dtschema/schemas/iio

I'd say no myself. We don't have IIO, and IIO (I tried) turns out to be a poor fit, IMHO, for Zephyr.

Note that IIO scheme can be used in DT anyway, in the end, this is about describing the hardware.

should we align with iio? https://github.com/devicetree-org/dt-schema/tree/main/dtschema/schemas/iio

I'd say no myself. We don't have IIO, and IIO (I tried) turns out to be a poor fit, IMHO, for Zephyr.

Note that IIO scheme can be used in DT anyway, in the end, this is about describing the hardware.

That's fine but then how would io-channels map to sensor channel types which are completely different things? iio channels are described by a channel spec I believe? It's unclear what a io-channel in this scenario would be.
e.g. is it this? https://docs.kernel.org/driver-api/iio/core.html#c.iio_chan_spec and the io-channel in this case is the mapped channel id?
Zephyr sensors lack this channel descriptor and mapping idea entirely for better or worse

We use IIO and I agree with Tom, this issue is complex enough without adding another requirement to it. I would vote for us finding a clean way to migrate things forward in a way that fits Zephyr (i.e. backward compatible). I think that listing these "capabilities" in devicetree is great, but some projects have very specific application code and probably won't benefit from this. In a similar way, the sensor info struct is only added behind a Kconfig. That's the pattern we should model here. This is great for generic apps/libraries that want to iterate of expose sensors to another client, but it's not really needed for many other applications.

@semihalf-Duleba-Kornel you may want to open a draft or DNM PR for 2810e96, that way people can comment on the diff

Considering that there is no opposition to the approach I proposed here I'll just finish the code and make a normal PR.
I'll probably have it done EOD, or tomorrow.

I think its generally a great idea, but as noted because some sensors provide multiple channels of the same type (the referenced PR #60833) it would be nicer if this could refer to not just the type but also the index of the channel.
Frankly that problem should be solved before this, or minimally this new binding should always refer to channel index 0 to start until the problem is fixed in the sensor API.
So I'd change this slightly for future proofing to be
phandle channel_type channel_index

I'll update the bindings to be more flexible.
We can move part of the sensor-cells specification to binding of a specific sensor.
This way each device can specify whether it wants to have (channel, index), or just (channel) specified in the DT.
When it comes to adding "index" to the sensor API itself it might be a bit more complicated.
There are two ways to do it that I can think of:

Break the driver API and change the definitions of sensor_sample_fetch_t and sensor_channel_get_t to take an argument index. The existing get and fetch syscalls could be replaced with helpers that point to the new API with index set to 0.
Create two syscalls - fetch and get - that use the index argument.

Option 1. is much cleaner, but it breaks API. With option 2 we also bloat the API and increase the overall flash size.
Do we care about the breakage? It'll only affect out-of-tree sensor drivers, rather then the consumers/application that use the sensor API.

should we align with iio? https://github.com/devicetree-org/dt-schema/tree/main/dtschema/schemas/iio

I'd say no myself. We don't have IIO, and IIO (I tried) turns out to be a poor fit, IMHO, for Zephyr.

Note that IIO scheme can be used in DT anyway, in the end, this is about describing the hardware.

That's fine but then how would io-channels map to sensor channel types which are completely different things? iio channels are described by a channel spec I believe? It's unclear what a io-channel in this scenario would be.
e.g. is it this? https://docs.kernel.org/driver-api/iio/core.html#c.iio_chan_spec and the io-channel in this case is the mapped channel id?
Zephyr sensors lack this channel descriptor and mapping idea entirely for better or worse, there's no notion that sensor X has channel 0 which represents acceleration in the X axis in SI units, iio seems to have this sort of descriptor available and mapped to a numerical channel.

Apart from that we also might want to specify a pair of (channel, index).
For that reason alone iio-channel can't be used here.

@teburd Take a look at https://github.com/semihalf-Duleba-Kornel/zephyr/commits/sensor-index-dt.
It's based on #61271 and it adds support specifying the channel index in DT.
I made it to show that the #61271 can be easily expanded to support that.

cc @mbolivar-ampere @galak (Devicetree)

No objections from me assuming the sensor maintainers agree.

In the consumer DT node the following can be added to reference the temperature sensors
sensors = <&temp_cpu DT_SENSOR_CHAN_DIE_TEMP>,
         <&soc_pct2075 DT_SENSOR_CHAN_AMBIENT_TEMP>,
         <&amb_pct2075 DT_SENSOR_CHAN_AMBIENT_TEMP>;

What hardware would the consumer DT node represent? It seems like this approach would lead to nodes in the devicetree that have no relation to hardware.

In the consumer DT node the following can be added to reference the temperature sensors
sensors = <&temp_cpu DT_SENSOR_CHAN_DIE_TEMP>,
         <&soc_pct2075 DT_SENSOR_CHAN_AMBIENT_TEMP>,
         <&amb_pct2075 DT_SENSOR_CHAN_AMBIENT_TEMP>;

What hardware would the consumer DT node represent? It seems like this approach would lead to nodes in the devicetree that have no relation to hardware.

Is having a physical piece of hardware that corresponds to a DT node a hard requirement?
From what I can see there are some pseudo-devices already described in the Zephyr bindings e.g. input-longpress.
I'm currently working on implementing a thermal framework, something similar to Linux thermal zones.
Basically the node would describe a configuration for a subsystem that would periodically read temperature from sensor(s) and then perform an action, e.g adjust fan speed, toggle a PROCHOT pin etc.

Is having a physical piece of hardware that corresponds to a DT node a hard requirement? From what I can see there are some pseudo-devices already described in the Zephyr bindings e.g. input-longpress.

Yeah I think these sort of soft-devices are the right thing to do in these cases, before the input-longpress there was cdc-acm-uart, now we have the lvgl input nodes.
Besides being a good spot for a structured config and laying down a pattern for the config/data struct and multiple instances in the driver, you also get the phandle based dependency check for the initialization sequence by doing that.

Is having a physical piece of hardware that corresponds to a DT node a hard requirement? From what I can see there are some pseudo-devices already described in the Zephyr bindings e.g. input-longpress.

Yeah I think these sort of soft-devices are the right thing to do in these cases, before the input-longpress there was cdc-acm-uart, now we have the lvgl input nodes.

Virtual-devices, CDC ACM UART is a type of emulated UART device, in the application it can be replaced with a real controller.

Besides being a good spot for a structured config and laying down a pattern for the config/data struct and multiple instances in the driver, you also get the phandle based dependency check for the initialization sequence by doing that.

+1

Is having a physical piece of hardware that corresponds to a DT node a hard requirement?
From what I can see there are some pseudo-devices already described in the Zephyr bindings e.g. input-longpress.

In that case I think this should be zephyr,sensors:
https://docs.zephyrproject.org/latest/build/dts/bindings-upstream.html#the-zephyr-prefix

You must add this prefix to property names in the following cases: [...] Configuration values that are specific to a Zephyr device driver.

[...] Though devicetree in general is a hardware description and configuration language, it is Zephyr’s only mechanism for configuring driver behavior for an individual struct device. Therefore, as a compromise, we do allow some software configuration in Zephyr’s devicetree bindings, as long as they use this prefix to show that they are Zephyr specific.

In that case I think this should be zephyr,sensors:
https://docs.zephyrproject.org/latest/build/dts/bindings-upstream.html#the-zephyr-prefix

That's the plan :)
This is outside the scope of this RFC.
Right now what I'm proposing is to extend the sensor/DT APIs so that a sensor and its channel can be directly referenced in DT using the phandle-array type.


** (UndefinedFunctionError) undefined function: SomeModel.schema/1

In the following code, the function find_all_channels fails with the error:
** (exit) an exception was raised:
    ** (UndefinedFunctionError) undefined function: Application.__schema__/1
        (elixir) Application.__schema__(:source)
        (ecto) lib/ecto/repo/queryable.ex:91: Ecto.Repo.Queryable.execute/5

defmodule Slacker.Application do
  use Slacker.Web, :model
  alias Slacker.Application
  alias Slacker.Settings
  alias Slacker.Repo
  alias Slacker.Channel

  @primary_key {:id, :string, []}
  @foreign_key_type :string
  @derive {Phoenix.Param, key: :id}

  schema "applications" do
    field :user_id, :integer
    field :name, :string

    has_one :settings, Settings
    has_many :channels, Channel

    timestamps
  end

  @required_fields ~w( name )
  @optional_fields ~w( user_id )

  def changeset(model, params \\ :empty) do
    model
    |> cast(params, @required_fields, @optional_fields)
  end

  def find_all_channels(application_id) do
    Repo.all(
      from c in Channel,
      join: a in assoc(c, :application),
      where: c.application_id == ^application_id)
  end

That's because you are using Elixir's Application module instead of your Slacker.Application. My suggestion: call it something else to avoid confusion.

I had a issue in my Repo.insert , I defined as previously  in User model
`  schema "users" do
has_many(:authorize_tokens, AuthorizeToken)

timestamps()

end`
When I was calling  this function
`  changeset = User.changeset(%User{}, user_params)
case Repo.insert(changeset) do
{:ok, _user} ->
    conn
    |> put_flash(:info, "User created successfully.")
    |> redirect(to: user_path(conn, :index))
  {:error, changeset} ->
    render(conn, "new.html", changeset: changeset)
end`

It is producing following error
function AuthorizeToken.schema/1 is undefined (module AuthorizeToken is not available)
I only add one thing in my model
  alias BlogTest.AuthorizeToken
Now everything is working fine

I got bit by this in a weird and embarrassing way so I'll share:
This can occur if there is a naming conflict between modules. e.g. you have a schema Foo and another module also named Foo.
:bowtie:


[New Model]: Chameleon support

The model to consider.
https://huggingface.co/facebook/chameleon
(as of now, the models can be downloaded using the model form)
Chameleon is an interesting multimodal model architecture based on Llama 2. It adds image inputs and outputs to Llama 2 by tokenizing images using a VQ-VAE and adding the codebook to Llama's tokenizer vocabulary.
In principle, it supports text and images as input and output in arbitrary combination. However, the released models were finetuned to prevent image generation.
The closest model vllm already supports.
LlamaForCausalLM
What's your difficulty of supporting the model you want?
For text->text support, the implementation should actually be fairly easy. The model is based on Llama-2 with the following differences:

QK norm
reordering the norm similar to Swin Transformer (normalizing the outputs of the attention and ffn blocks instead of the inputs)

To enable image inputs, image tokenization using the provided VQ-VAE needs to be added.
Further info:

Original paper: https://arxiv.org/abs/2405.09818
Reference implementation: https://github.com/facebookresearch/chameleon

Apparently there is ongoing work by HF to get this architecture into transformers https://github.com/huggingface/transformers/issues/31505
Once that is implementation is made we can look at porting it over.

Chameleon PR ontransformers to track https://github.com/huggingface/transformers/pull/31534

I made a PR https://github.com/vllm-project/vllm/pull/5770 based off the transformers PR as an initial text-only support for this model and will wait for HuggingFace to release the weights to verify the implementation.
For the VQVAE - I plan to add it in next PR if that makes sense.

Looks like they landed it today!

@nopperl The two chameleon PRs have been merged and will be included in the upcoming 0.5.3 release, so please give it a try and let me know if you run into any issue!


SimpleBST

Implement all methods in SimpleBinarySearchTree class: put(), remove() and private methods removeRec() and putRec()

you have to:

add  size change  when put/remove
simplify remove()
(?) rename updateLeftSon to updatedLeftSon
fix typo (line 41)
(?) remove extra 'else's in RemoveRec


feat(rust): use proper url data type

Current behavior

Proposed changes

Use e a Url instead of a String for tenant_base_url
https://github.com/build-trust/ockam/issues/5346
Checks

[ ] All commits in this Pull Request are signed and Verified by Github.
[ ] All commits in this Pull Request follow the Ockam commit message convention.
[ ] I accept the Ockam Community Code of Conduct.
[ ] I have accepted the Ockam Contributor License Agreement by adding my Git/Github details in a row at the end of the CONTRIBUTORS.csv file in a separate pull request to the build-trust/ockam-contributors repository. The easiest way to do this is to edit the CONTRIBUTORS.csv file in the github web UI and create a PR, this will mark the commit as verified.

This looks like a trivial task, but I have very limited Rust knowledge.
So when I use Url instead of a String, I am getting these errors:
103 | #[derive(Encode, Decode, Serialize, Deserialize, Debug, Clone, Eq, PartialEq)]
    |          ^^^^^^ the trait `minicbor::Encode<Ctx>` is not implemented for `Url`
    |
    = help: the following other types implement trait `minicbor::Encode<C>`:
              <&T as minicbor::Encode<C>>
              <&mut T as minicbor::Encode<C>>
              <() as minicbor::Encode<C>>
              <(A, B) as minicbor::Encode<Ctx>>
              <(A, B, C) as minicbor::Encode<Ctx>>
              <(A, B, C, D) as minicbor::Encode<Ctx>>
              <(A, B, C, D, E) as minicbor::Encode<Ctx>>
              <(A, B, C, D, E, F) as minicbor::Encode<Ctx>>
            and 233 others

error[E0277]: the trait bound `Url: minicbor::Decode<'_, Ctx>` is not satisfied
   --> implementations/rust/ockam/ockam_api/src/cloud/project.rs:103:18
    |
103 | #[derive(Encode, Decode, Serialize, Deserialize, Debug, Clone, Eq, PartialEq)]
    |                  ^^^^^^ the trait `minicbor::Decode<'_, Ctx>` is not implemented for `Url`
    |
    = help: the following other types implement trait `minicbor::Decode<'b, C>`:
              <&'a ByteSlice as minicbor::Decode<'b, C>>
              <&'a std::path::Path as minicbor::Decode<'b, C>>
              <&'a str as minicbor::Decode<'b, C>>
              <() as minicbor::Decode<'b, C>>
              <(A, B) as minicbor::Decode<'b, Ctx>>
              <(A, B, C) as minicbor::Decode<'b, Ctx>>
              <(A, B, C, D) as minicbor::Decode<'b, Ctx>>
              <(A, B, C, D, E) as minicbor::Decode<'b, Ctx>>
            and 227 others

So I know that Encoder and Decoder are missing for Url, but not sure what is the idiomatic way to deal with this.
There is no way to call the derive macro on an external type like Url, and I was not able to find a similar example in the codebase. @etorreborre

@aleksandarskrbic I didn't think of this difficulty. The usual work-around is to use a newtype:
// in ockam_api::minicbor
use url::Url as RegularUrl

#[derive(Encode, Decode)
struct Url(RegularUrl);

// so that we can call url.into() to recover a regular Url
impl Into<RegularUrl> for Url {
    fn into(&self) -> RegularUrl {
        self.0.clone()
    }
}

// in ockam_api::cloud::project

// use the newtype instead of url::Url
use crate::minicbor::Url;

#[derive(Encode, Decode, Serialize, Deserialize, Debug, Clone, Eq, PartialEq)]
#[rustfmt::skip]
#[cbor(map)]
pub struct OktaConfig {
    #[cfg(feature = "tag")]
    #[serde(skip)]
    #[cbor(b(0))] pub tag: TypeTag<6434814>,

    #[cbor(b(1))] pub tenant_base_url: Url,
...
}