SEP proposal: Unified format for timeseries data

I would like to propose a unified format to represent timeseries data for use in SDyPy.
The main motivation is to simplify the interface across packages. In particular my proposal would target input-output packages to return the data in this common style (UTS - Unified Time Series)
Proposal
My proposal is to always work with a list of dictionaries
Structure
[
    {
         "name": "channel_id_1",        # Channel name
         "unit_str": "m/s²",                  # Unit string
         "data": np.Array([1,2,3,4]) ,   # 1D array of data
        "time" :
              {
                   "start_timestamp" : "2023/06/29 15:00:00+00:00", # Timestamp in absolute time
                   "fs": 125,                                                                   # Sampling frequency in Hz
              }
         "properties": {} # Additional dictionary for non-essential information, e.g. type of data-acquisition, SN
    },
    ...
]

Properties
Aside from the structure the UTS has to comply with some ground rules:

information on how to obtain the timevector should be contained in the file (either indirectly from Fs, or directly by providing a time vector" (Do we enforce equidistant sampling?)
channel names should be unique within the list
some properties are required, e.g. name, unit_str, properties (even when empty)
(?) enforce 1D signals (May conflict with some data types??)

Further actions
To create a separate repo to make the definition more formal and to provide a test for compliance with the UTS format. This can be used to have ease the integration of new io-packages

Handling time-series data is very important. This proposal is simple and sound to me. Could we maybe try to resolve the 1D/ xD issue? For example, how do we handle triaxial measurements? 3x1D data of 1x3D data. How do we handle image time-series which are 3D time-series? I am adding @klemengit, @tkosir, @raphaeltimbo, @dagghe to the communication.

@wweijtje proposes categorizations per channels. Each axis is stored under its own channel. This can resolved be adding extra information under "property" such as:
"Node" : %d,
"Axis" : %s #x, y, z, or 1, 2, 3

Looking at my own workflows I would indeed consider the three axis of a tri-axial sensor as three separate channels, so 3x1D. Typically the name is in an indication of the orientation. I personally don't see an immediate advantage of grouping the channels into a 1x3D array (aside from a small data-size reduction).
As @itomac mentions we can include more details about the exact position and orientation of the signal inside 'properties'. I would also propose a fixed set of keywords to specify these properties.  I would personally consider this a non-obligatory properties as far a wide array of cases the exact position of the signal is not necessary.
I personally would be tempted to extend this for all dimensions of data. But understand this might be pushing it for image based timeseries.

I agree with the format @wweijtje proposed and I also think that each direction can be stored as a separate channel.
As for the image-based time series, I believe it's best to store them in a 2D array (location is flattened and the shape is stored in the properties). Something similar is already working in LDAQ.
An alternative structure
I would also suggest an alternative structure (just so we consider all options). Maybe it would make sense to decouple the actual data from the meta data? All the time series would therefore be stored in a 2D array, and meta data would be linked to the appropriate channel in the array.
The data:
#                  Ch0,     Ch1, ...
Data = np.array([[0.749, 0.48, ...]
              [0.33, 0.535, ....]
              [..., ..., ..., ...]])

Meta data would be stored in a list of dicts:
Meta = [
    {'name': "ch0 name", 'fs': 51200, ...},
    {'name': "ch1 name",...}
]

Now, I immediately see an issue with different sampling frequencies and series' lengths. But the data could maybe be a list of 1D arrays to solve this issue.
Why I think this structure could also be a good choice? With this approach, the data can be more easily selected (without the need to iterate the entire data structure to extract the data) and the meta data can be easily converted to a pandas DataFrame, further simplifying the workflow (e.g. grouping the time series for FRF computation).
Don't know if this is a better approach, just wanted to add it to the discussion.

I agree that representing channels separately, packaged together with their metadata, makes a lot of sense in most situations, and probably ensures the highest level cross-comaptibility.
Perhaps we should nonetheless consider allowing the signals (the data field of the structure) to be 2D if specific conditions are met (e.g. matching sampling frequency and obviously length etc.).
I think we could cover a lot of cases where it makes sense to process such data together more efficiently, as @klemengit said, for example:

if they represent multiple repetitions of the same measurement,
for image-based displacements of different points in the same video (and similarly for other comparable methods, such as scanning lasers etc.),
for passing FRFs at multiple DOFs between different processing stages (in case we decide to also represent frequency-domain data this way,
...

If we go down this route we should perhaps identify all the allowed cases of this and decide on how to clearly and consistently mark them in the metadata, to avoid creating a situation where everyone is again using this format in their own, probably incompatible way. There would then also be an easy way to convert between the 1D and multy-D prepresentation of such data, that should be part of the core package.

I like the discussion. The time-series representation should be simple and human-readable.
e.g., @wweijtje proposal can be further simplified (having mandatory and optional items):
{
         "data": np.Array([1,2,3,4]) ,   # 1D array of data
         "fs": 125,                               # Sampling frequency in Hz
         "time": np.Array([1,3,4,7]),    # Sampling freq is required for equally spaced data, while `time` for non-equally
         "name": "channel_id_1",        # Optional: channel name
         "start_timestamp" : "2023/06/29 15:00:00+00:00", # Optional: Timestamp in absolute time
         "unit_str": "m/s²",                  # Optional: Unit string
         "unit_tex": "m/s$^2$",          # Optional: Unit latex string
         "whatever": whatever           # Optional
}

Having all items at the same level enables easy checking of whether a parameter is in the dictionary or not. We would have to define which are the required parameters
For later postprocessing, we could have a method to prepare a 2D representation for a list of such time-series (e.g. .to_array()). While the 2D representation would be prepared data checking could also be performed (e.g. same length). At this stage data could be separated from metadata.
How does this sound?

I agree with @jankoslavic that having all properties/data in the same level (not nesting dictionaries) makes sense.
I also think that, as @domengorjup said, in certain cases, the data could be 2D array (most importantly in case of a video). The shape of the data is easily checked. Alternatively, setting a property could be mandatory when the data is not 1D (e.g. data_shape).
@domengorjup, @jankoslavic; efficient methods for data condensation to numpy arrays are indeed very important.

Uncouppling data from metadata, as @klemengit suggested, allows direct access of data from storage, using numpy.memmap().

I think uncoupling data from metadata could create a lot of potential issues. After all, the main point of this format (as I see it) is to standardize and simplify data transfer between different packages, and to me passing two separate objects for a single timeseries doesn't seem to be the best way to do it.
I also don't think it's and issue to have a numpy.memmap be used instead of normal arrays inside of a dictionary, if needed to deal with very large data as @itomac suggested.

I think uncoupling data from metadata could create a lot of potential issues. After all, the main point of this format (as I see it) is to standardize and simplify data transfer between different packages, and to me passing two separate objects for a single timeseries doesn't seem to be the best way to do it.

Exactly! I'm personally indeed not a fan of splitting the data into two seperate objects.

Ok. great discussion!
I would suggest we agree on mandatory items which are: data and fs in case of equally spaced data and data and time in case of non-equally spaced data. All other items are optional, but I suggest that we prepare a list of optional items: e.g., unit_str, unit_tex, orientation_ks.
For 2D time-series can be assembled from 1D time-series, for example if fs and length are equal, the new index in this case correspond to a time-series, and parameters can become lists (or np.arrays):
data -> np.array[data,data,...],
unit_str -> [unit_str, unit_str,...];
scalar values shared between series are still scalar, e.g.:
fs ->'fs`.
I will try to invite some more people to participate in this discussion, so we have as many views as possible.
I suggest moving on with drafting a SEP5 on time-series data guidelines.
@wweijtje you initiated this issue, would you be willing to start drafting the SEP5? if yes, please start from SEP4.
I will try to get additional people involved so we have more views. This is a very important SEP!
Thank you all!

I made a draft proposal for the SEP005 and launched a pull request on it. I tried to group all discussions above but also tried to simplify it a bit (e.g. the discussion on multi-dimensional arrays I left out of it for now as I don't have a clear opinion on it, I however propose to not enforce 1D array).
I've also made a list of compulsory, optional and even prohibited fields. I personally think the inclusion of prohibited fields aids to avoid future confusion, e.g. caused by similar fields or typo's.  But this practice is up for discussion.

Thank you @wweijtje! As this is the first public SEP, we have to test the workflow as defined in SEP0; @domengorjup can you take this over please?

This is now partially implemented in SEP5. The work is still in progress.
https://github.com/sdypy/sdypy/blob/main/docs/seps/sep-0005.rst

@jankoslavic I've opened a pull request to change the type of the proposed SEP to "Standards", and added a link to this discussion to the proposed document.
Draft is the appropriate SEP-005 status while the discussion is still ongoing. This will be changed to Accepted once the discussion is concluded with a consensus (SEP-000). The SEP will become Final, once a reference implementation (e.g. a package to enforce and test compliance with the agreed format etc.) is complete.

Coincidentally, I wanted to discuss whether it would make sense to not limit this unified format to only timeseries data, but also use it for frequency-domain signals?
A distinction between the two could be easily made based on an agreed upon key in the signal dictionary. I think most of the other information we plan to store in these objects make sense in both cases.
Also, are there any objections to including the "2D time series" functionality in the SEP, for when we want to efficiently represent multi-channel signals with matching sampling parameters (fs, time etc.)?

Regarding the representation of signals in the frequency domain, they can be treated in the same way as time data when expressed as complex numbers (not as amplitude and phase). However, the spectrum can be represented in different ways. This can be specified with an additional information in the metadata: e.g. 'spectrum = {'single', 'double', 'shifted'}', or simply require the frequency axis?
2D time series are useful when dealing with a series of measurements to reduce redundancy in metadata and to allow ease of use when shareing data between packages.
There is a potential issue in the case of large data sets, which may be addressed, can also apply to single channel files. Should division of the data into several files be specified?

Hey all, in light of this I've implemented a first version of a package that serves to check the compliance with this SEP005, it is available here sdypy-sep005-compliance
I see it for two purposes :

future packages that want to comply with SEP005 can use the assert functions in it to check compliance.
sdypy-io could use it to validate any incoming data for compability with the format (e.g. it could in the future contain even smarter criteria (e.g. timevector is always increasing)..

Definitely open for discussion whether this is strictly necessary, but at the very least I'm going to use it for the tdms io package I'm developing : sdypy_io_tdms

Dear @wweijtje this looks great!
What about implementing testing similar to numpy?
See: https://github.com/numpy/numpy/tree/main/numpy/testing
or help here: https://numpy.org/doc/stable/reference/routines.testing.html#module-numpy.testing
If sdypy would have testing methods this would be very convenient. If we make a good example for SEP5 here, this can serve as a showcase for others. How does this sound to you?

With the ambition that lives for sdypy I can't imagine it doing it without testing. IMO you can't have collaborative code without testing.
For now I would like to move on with the independent SEP005 testing package (and putting it on pypi) so I can use it as a dependency for testing my other code.  But I don't mind it being integrated in sdypy at a later stage.
From my side I don't have the time or oversight where SDyPy is going to to define a fixed 'styleguide' / 'namingconvention' to have a uniform style of functions. But I'm willing to comply, the SEP005-testing package is open-source.

I've opened a new PR to push this SEP draft forward, according to our discussion.
In particular I don't think it's necessary to enforce that the timeseries is always a list of dictionaries. Is there a particular reason not to allow a single timeseries to be represented by just a single dictionary of the agreed structure?
In the PR I've also tried to clarify the case for "2D" arrays in the data field. I think this is not limited to just specific measurements such as cameras etc, but would be very beneficial whenever we have multiple channels with the same time-discretization, which we can efficiently process when they are represented as a single 2D "data" np.array.
For example, we can perform vectorized numpy operations (e.g. fft, filtering, convolution...) on all channels simultaneously, without having to first iterate over all the timeseries dicts in the list. I think this will become the main mode of operation for applications such as modal analysis, where we normally deal with multiple simultaneously measured channels.
This does not interfere with the possibility of representing a single timeseries per dictionary, and having lists of such dictionaries represent multiple channels, where this makes more sense. I've added an example of this "list of dictionaries" structure to the document.

Lovely a different opinion, to be honest I'm not a fan of what you are proposing as you are creating two routes to the same solution. Which implies that every stage you need to accomodate compability with both. My biggest concern is not ease of matrix manipulation, but rather keeping track on the origin and properties of each signal.  This is my key motivation to say 1 channel  = 1 dict.
The philosiphy of the original proposal comes from my own experience and preference to consider measurement channels as independent entities that can be readily merged on request.

In particular I don't think it's necessary to enforce that the timeseries is always a list of dictionaries. Is there a particular reason not to allow a single timeseries to be represented by just a single dictionary of the agreed structure?

In general I always prefer to work with fixed datatypes, which simplifies typehinting and all subsequent code. E.g. if I enforce it always is a list, I can enforce this type (and use simple typehinting) and I can always start my code from the assumption it is a list.
If you allow for it being a dictionary, you need a bit more advanced typehinting, and your first line is something to convert it to a list anyway (as a lot of the operations will assume individual signals are in a list).:
if type(timeseries)==dict():
    timeseries = list(timeseries)

In the PR I've also tried to clarify the case for "2D" arrays in the data field. I think this is not limited to just specific measurements such as cameras etc, but would be very beneficial whenever we have multiple channels with the same time-discretization, which we can efficiently process when they are represented as a single 2D "data" np.array.
For example, we can perform vectorized numpy operations (e.g. fft, filtering, convolution...) on all channels simultaneously, without having to first iterate over all the timeseries dicts in the list. I think this will become the main mode of operation for applications such as modal analysis, where we normally deal with multiple synchronously sampled channels.

The main idea behind 1 dict per channel is that each channel can carry it's own information. So it can have an individual name, but also practical information about the sensor (serial no., position, sensitivity,...) associated to this unique signal.  If you want to carry this information into a 2D array you need to work with another datastructure inside the main dictionary to accomodate.  It might get messy soon.
The comment that you often want to make data into 2D arrays for various operations is valid, but for me not a dealbreaker. Routines to quickly combine different 1D arrays into a single array are readily implemented.

I see where you're coming from. Nonetheless, I think that the requirement of a single channel per dictionary would be quite a divergence from the way most Python packages, focused on numerical computation, currently share data.
I can say that for most of our use cases for example, it would ultimately just introduce two additional steps each time we would be passing the data among packages (first split the arrays, each into its dictionary, pass that to another package / function, and the iterate over these dicts to get back to the same array structure for processing).
The way I look at it is: we are currently exchanging data as numpy arrays (each inventing our own way to specify metadata). The main value of this SEP to me is a standardization of the way we structure this data.

f you want to carry this information into a 2D array you need to work with another datastructure inside the main dictionary to accomodate. It might get messy soon. In addition you defacto end with a situation in which the order inside the 2D array is information, (i.e. which column is associated with which signal?)

I think this SEP should remove any ambiguity regarding this specific issue, and I think the solution if obvious if we consider the way Scipy is handling this - scipy.fft.fft would, for example, perform the time-frequency transformation along the last axis (axis=-1) by default. This would also make it easy to enforce the correct shapes of data and metadata such as name (for example, len(ts['name']) == len(ts['data']) checks for correct number of channels).
Regarding typehinting  ("list vs dict"), I think  that in the proposed structure, a dictionary represents a timeseries:

'One signal is One dict'

If a function / package expects multiple timeseries, of course it makes sense for it to expect a list, but I don't think this eutomatically means it should always be enforced. If we have / expect just a single timeseries, for example, I don't thinks it's ever necessary to convert it into a list at all.

I feel we are having two discussions simultaneous, one on whether we need to enforce it being a list or not and secondly what is the better overall structure. For me we can close the first one  I agree that enforcing it as a list is overkill, indeed if you pass one timeseries it should be possible to just give a dict.
The second discussion is more fundamental: i.e. 'One signal is one dict and multiple signals go into a  list of dicts with 1D-arrays' vs. 'Multiple channels are grouped into 2D arrays, with a metastructure to accomodate'.
In the end both can work. I have my preference, and my reasons to do so. But all really depends on how you are going to handle it inside ´sdypy´. Just to illustrate this is how I program it at the receiving end, starting from the 1 signal = 1dict idea. With these simple lines you quickly reach the 2D array, even e.g. from a slice of the original list of signals.
import numpy as np

class SEP005(list):
    def __getitem__(self, item):
        if isinstance(item, slice) or isinstance(item, int):
            return SEP005(list.__getitem__(self, item))
        if isinstance(item, str):
            return SEP005([s for s in self if item in s['name']])

    def to_numpy(self):
        return np.stack([s['data'] for s in self], axis=0)

signals = [
    {
        'name':'signal_top_1',
        'data': np.array([1,2,3,4,5])
    },{
        'name':'signal_top_2',
        'data': np.array([6,7,8,9,10])
    },{
        'name':'signal_bottom_3',
        'data': np.array([11,12,13,14,15])
    },{
        'name':'signal_diff_fs',
        'data': np.array([11,12,13]) # different sample frequency, to illustrate
    }
]

signals_sep005 = SEP005(signals)

print(signals_sep005[:3].to_numpy()) # Only the first three signals
print(signals_sep005['top'].to_numpy()) # Only the signals with 'top' in their name
print(signals_sep005.to_numpy()) # Will raise a relevant error that they can't be stacked due to shape differences.

Whatever you decide, I'll see to comply. We'll get there :)

I agree completely.
After some consideration I think I see the problem you point out regarding multi-channel data a bit better. If both ways (list of dicts as well as  2D data) are possible then all functions expecting multiple channel must always consider two different input data structures, and it's not obvious to the user which one they should actually be using, which is what we're trying to eliminate with this SEP.
I still think the above points are valid though, and see a lot of merit to being able to store multiple channels in a single timeseries object, if they are synchronized or even acquired using the same method. If nothing else, splitting such channels into their own dicts, repeating essentially the same metatdata for each of them, would often seem like an unnecessary step.
This is even more true for very large data. As an example, I'm working with some camera measurements right now, where the size of the time series on disk is ~5GB. For even larger datasets, having the option to use np.memmap objects instead of the 2D arrays to represent data would greatly boost the practicality of this format.
On the other hand, when there is a possibility of variable time steps, or any other reason for specifying metadata fundamentally differently for different channels, I think the option with a list of "1D" timeseries dicts is much more versatile, and the only practical format to use.
In the end, I think it's sensible to allow both. After all, both ways would need to adhere to the same SEP-005 standard, and they would both be the same timeseries object, essentially. This is then just about the shape of the data we're trying to represent using our standard, which is probably heavily application-specific anyway.
If we did allow both options, we could solve the ambiguity issue by having methods (like to_numpy() above) to easily convert between the two structures in our pacakges, as you propose. That way it doesn't actually matter which one the user chooses (although if one makes more sense than the other in a specific case, the docs should reflect that), and we can count on the data being compatible, or convert it in a single line of code when we want to work with one of them specifically.

This is an important discussion. How do we see the performance issue of such an implementation: to_numpy is rather fast, but as I understand it we are creating additional copies of the data.

@domengorjup I think we found our middle ground.  Leaving the door open for 2D arrays is ok for me, it is not a feature I need, but I get the usecase.
@domengorjup @jankoslavic in my use-cases these concerns about performance do not play, the largest files I handle are <100MB.

@domengorjup , @wweijtje this is an excellent discussion; it is great we have multiple opinions!
We need to support both options; if we find an implementation that can easily handle both, this would be a winner. @domengorjup can you try to further develop your PR?

To be honest I think the current proposal might accomodate @domengorjup's concern when we allow for 2D-arrays in data.
@domengorjup would you need still a list of e.g. names to go along the 2D array? Because if not, I would propose to say obligatory properties like 'name', 'fs', 'time' are limited to a single value (and not lists).  This would mitigate all my concerns and enforce that 2D-array are just a special type of channels where the columns have shared properties. Keep in mind that other keywords in the dict are free, so those can be used for e.g. spatial information about the array.

@jankoslavic, @wweijtje  I think my current proposal describes both options in the current form. Are there specific parts of  the document you think we should further clarify at this point?
Regarding the actual implementation of the conversions between formats, I agree with @wweijtje that they probably won't happen so often that we should be overly concerned with their efficiency at this stage. In cases where one form makes more sense than the other, I imagine this structure will be maintained throughout most of the workflow. And when there isn't a clear preference, this will probably mean that the amount of data is relatively small, and performance won't be an issue.
If we determine (and document) the main logic of the structure well enough now and implement a minimal working solution, I think we can easily work on its efficiency at every new release later.

To be honest I think the current proposal might accomodate @domengorjup's concern when we allow for 2D-arrays in data.
@domengorjup would you need still a list of e.g. names to go along the 2D array? Because if not, I would propose to say obligatory properties like 'name', 'fs', 'time' are limited to a single value (and not lists). This would mitigate all my concerns and enforce that 2D-array are just a special type of channels where the columns have shared properties. Keep in mind that other keywords in the dict are free, so those can be used for e.g. spatial information about the array.

I think this is a great point. In my proposal, fs and time would already need to be strictly single values (otherwise this would imply we have different timings for different channels, and it would make no sense to me to package them in a single dict).
I can see a use-case where channel names could be different for different rows of the 2D data array. An example of this is when we are using multi-axial accelerometers, They are all the same quantity, measured at the same location on the sample, have the same units, but it's important to distinguish them by channel name.
Perhaps we can have two separate fields, "name" and "channel name"? So the "name" is a mandatory scalar field, a single unique name of the timeseries object, and "channel name" is an optional (but documented in the standard) field that needs to be of the same length than "data", if this is a 2D array?
On a slightly different note, I agree with @wweijtje's point that it doesn't make sense to allow different quantities to be bundled into the same timeseries dicts.
Short examples of a typical measurement with different physical quantities that we would want to compute MIMO FRFs from would look like this:
Option 1: Single "channel type" allowed
  [
    # Force (excitation) data
    {
          "data": # array of shape (2, n), representing 2 input excitation force channels
          "time": # array of shape (n, )
          "fs": fs,
          "name": "Force",
          "unit_str": "N",
          "dof": np.array([0, 3]) # on example of custom metadata to represent the spatial information of our measurement
    },

    # Acceleration response data
    {
          "data": # array of shape (2, n), representing 2 response measurement channels
          "time": # array of shape (n, )
          "fs": fs,
          "name": "Acceleration",
          "unit_str": "g",
          "dof": np.array([0, 1])
    },

    # Velocity response data
    {
          "data": # array of shape (2, n), representing 2 response measurement channels
          "time": # array of shape (n, )
          "fs": fs,
          "name": "Velocity",
          "unit_str": "m/s",
          "dof": np.array([2, 3])
    },
  ]

Option 2: Multiple "channel types" allowed, as long as time discretization matches
    # All 6 channels in a single dict:
    #    - 2 force measurements (DOFs [0, 3]),
    #    - 2 acceleration measurements (DOFs [0, 1]),
    #    - 2 velocity measurements (DOFs [2, 3]).
    {
          "data": # array of shape (6, n)
          "time": # array of shape (n, )
          "fs": fs,
          "name": ["Force", "Force", "Acceleration", "Acceleration", "Velocity", "Velocity"],
          "unit_str": ["N", "N", "g", "g", "m/s", "m/s"],
          "dof": np.array([0, 3, 0, 1, 2, 3]) #
    }

So the first option is much clearer, and while the second one is a bit more concise, it would make things significantly more confusing.
When we measure different quantities now, we already group them by type anyway (for example our FRF methods already expect excitation and response signals as separate arguments). So option 2 is probably just pointlessly complicating everything.
Any thoughts on this from other participants in this discussion?

thanks @domengorjup, @wweijtje . I would continue with option 1. If you agree we would need to re-iterate the mandatory and optional arguments.

I've now added the changes, proposed above, to an updated version of this SEP in my open PR, so we have a basis to work from.

Option 1 for sure. As  I would always avoid having in the metadata lists of properties that match with columns of a 2D array.

I can see a use-case where channel names could be different for different rows of the 2D data array. An example of this is when we are using multi-axial accelerometers, They are all the same quantity, measured at the same location on the sample, have the same units, but it's important to distinguish them by channel name.

Well that particular example I've historically always considered them as three different channels. Because I often have to group data from different sensors but with a common orientation; e.g. continuing from my previous example.
signals = [
    {
        'name':'ACC1_X',
        'orientation': 'X',
        'position': {#Position1}
        'data': np.array([1,2,3,4,5])
    }, {
        'name':'ACC1_Y',
        'orientation': 'Y',
        'position': {#Position1}
        'data': np.array([1,2,3,6,5])
    },  {
        'name':'ACC2_X',
        'orientation': 'X',
        'position': {#Position2}
        'data': np.array([1,2,3,4,5])
    }, {
        'name':'ACC2_Y',
        'orientation': 'Y',
        'position': {#Position2}
        'data': np.array([4,5,6,6,7])
    },{
]

signals_sep005 = SEP005(signals)

print(signals_sep005['ACC1'].to_numpy()) # Only the data from ACC1
print(signals_sep005['X'].to_numpy()) # Only the data in the X direction

Of course there is some double metadata in my example, and yes you do need a way to formally join the sensors together (here this is achieved through a common element in their names). But those things happened naturally for me.  So to me there has never really been a gain to join data from a tri-axial accelerometer into a 2D array.

So to me there has never really been a gain to join data from a tri-axial accelerometer into a 2D array.

Nonetheless, even option 1 would inherently allow that. So the question is if we should anticipate that this might sometimes happen, and reserve a specific metadata field ("channel_name" in the current proposal, but we could find a better keyword) to do this kind of labeling in a standardized and predictable manner.

It looks to me like we are very close to a good working solution. We now have data which can be 1D or 2D array. The most common data-series will have data, fs, and some sort of description of the data i.e., the name.
We find an obvious solution for the name. To me, it would be obvious that name is scalar for 1D data and 1D for 2D data. Would this work in general? I would try to avoid any other "name" (e.g., channel_name, row_name).
How do you @wweijtje and @domengorjup or any other see it?

@wweijtje can you handle your use cases, if we continue with only one name which would describe the one timeseries in case of 1D data and a list of names in case of 2D data?

I've raised my concerns, but I'm not going to stop progress on this, as you correctly state my preferred way of working fits in the scheme so I'm happy. :-)

Hi @wweijtje! thank you for understanding. This is still an open SEP. I suggest continuing with this SEP and trying to find the best solution which will be easy and efficient to use.

Hi all and good work!
Very interesting discussion that is important for collaboration and that potentially adds tremendous value (monetary/time/scientific) to research and engineering in general. I have been interested in the topic of data structures and metadata for many years myself, please check out this note for some thoughts and practices on data+metadata.
I realize that I am late to the party, and that you are already deep in the design and implementation. You have already resolved many points of discussion, but I will give my two cents on some of them anyway.

Splitting data and metadata: Looking down the pipeline, it is certainly possible to work with both structures, i.e. data in two different objects vs metadata and data in a single object. The workflow and overall efficiency of the two alternatives are likely very similar. However, I think there is a much greater potential for making errors if the two are split, the programmer has to make the link instead of the program. This is also one of the fundamental innovations of an object over a basic array, it enables grouping functionality and properties with data. I am glad you landed on keeping them together :)
1D vs nD: I want data and dataseries to be 1D, not only because it will simplify implementation of methods and functions down the pipeline, but also because all sensors/observations are inherently 1D (with time). In my mind, a tri-axial accelerometer are three distinct sensors in one housing. The only benefit of modelling it as one sensor with three outputs, is that you save some storage space (and work?) on metadata, i.e. you don't have to specify the position of three 1D sensors, or specify 3 times "x-axis of Dytran 3085 S/N: 321", but instead you merge that metadata into one. Unfortunately, I have found that there are sensors which it makes a lot more sense to structure them as nD rather than force them as 1D. Cameras can also be viewed as a gathering of 5million sensors (pixels) in one housing, but for most/all use cases of the data, you then have to structure it into a 2D array before further processing. I am sure it is possible to make a structure that incorporates all device outputs as a collection of 1D time series (as suggested in earlier discussion), but ultimately I think the most useful solution is the one that you landed on, i.e. that it is not prohibited to have data as nD.
As for implementation, I have no preference. One could have a class hierachy with "BaseTimeSeries> TimeSeries1D = TimeSeries2D" instead of a single object, but I dont think it really simplifies anything. Using already established objects, such as ndarrays, lists and dictionaries, is a strategy that I use myself and that I find effective and efficient. At one point I was looking into subclassing a numpy ndarray to add metadata, such that v.position -> np.array([0, .1, 3.]), but that has over time been taken over by a simple dict structure.

Hi @Gunnstein! Thank you for your input! This is great! Together we are much stronger than individually!

The repository is now public, the pdf link should be available.

This is a great repo! Should we add optional arguments to the SEP5?
e.g.:
"position": np.array([0., 0., 3.]),
"orientation": np.array([1., 0. ,0.]),
"description": "description of the time series"
What do you think?
Once we have the data structure set, it is easy to prepare a method to dump it to hd5:)

Should we add optional arguments to the SEP5?

Why specify Optional fields and Prohibited fields?

Isn't it better to just focus on the compulsory fields, which ideally is the minimum set of metadata for maximum cross compatibility and stability?
If the data is bundled with fs (as per SEP5) why does it matter if it also contain Fs and/or sample_rate? I guess this may confuse users that are only given the data and are not familiar with SEP5, and it forces people to adopt this convention, but it will certainly break backward compatibility in some cases and likely put some people/groups of from adopting this SEP?

Also, in the case of the data field and multidimensional sensordata, it may be necessary to specify the dimension of time?
Camera data, with NxM pixelarray and K points in time, can be structured KxMxN or MxNxK.
For 1D data it is not nesessary to have a axis for time (as per numpy convention), but for nD data we should perhaps specify the first or last axis for time?
Some use the first axis to denote progression in time,

Thanks @Gunnstein .

I would preserve optional arguments: with this, we define the data's name and format (e.g. position instead of location, np.array instead of list); we are also already prepared for later processing: e.g. visualisation already knows the location and orientation of the measurement.

Prohibited fields maybe can go to SEP2 (nomenclature) also the fs and sample_rate issue?

I would stay with 2D data; even for camera (where additional information is required about the pixel location for a particular channel). @klemengit what do you think?

@Gunnstein I think the two of us are much on the same page.
Maybe just to clarify one thing, I'm fan of having prohibited fields in particular to avoid confusion. The initial idea was that you can have any additional field you would want to add. So there is no limited list of valid options. This allows for maximum flexibility, however this poses a risk to introduce possible confusion e.g. with the obligatory fields (e.g. case differences etc).
Having a list of optional keywords seems sensible as well. They align what people CAN add, but it is not strictly necessary. I consider it best practice to use those keywords over a random alternative.
For those who remember I initiated this idea because a need I felt in organizing my own code. So I already made two packages:

https://github.com/wweijtje/sdypy-sep005-compliance: introduces tests to assert compliance with SEP005.
https://github.com/wweijtje/sdypy_io_tdms: loads tdms files (as are generated from our hardware) to the SEP005 standard.

These were both made to fit my needs, I didn't validate every (edge) case, but they are open-source, so feel free to make additions. Both are available from pip as well.
Regards,
Wout

thanks @klemengit ! Supported parameters sound very good to me!

We have waited for a while to gather different views; I would suggest continuing with this issue / sep5. The next step I see is to integrate the idea @wweijtje presented here  https://github.com/wweijtje/sdypy-sep005-compliance/.
I suggest integrating sep5 testing into sdypy.testing. The assest_sep5 would be the first testing function we would have and would serve as a model for future implementations. How does this sound to you? @wweijtje would you be prepared to prepare a PR? I was talking with @klemengit and we can do it, but as you are the author, I would prefer if you at least start (you can submit in a new PR this file https://github.com/wweijtje/sdypy-sep005-compliance/blob/main/sdypy_sep005/sep005.py and we will take it from here).
I have already prepared the testing folder:
https://github.com/sdypy/sdypy/tree/main/sdypy/testing

Hey @jankoslavic good to hear this keeps progressing. I happy you want to integrate the SEP005 compliance tests into sdypy. But would it be an option to keep it as a separate package and make it a dependency to sdypy. I ask so for three distinct reasons:

A technical reason, I think there are several advantages of keeping this a standalone package with a clear and single objective. We plan to integrate SEP005 compliance testing in several of our packages and their associated CI/CD pipelines. Having it standalone and dedicated facilitates this integration and means we can work without having the heavier sdypy and its heavier dependencies (matplotlib, scipy, ...) . Speeding up our CI/CD pipelines. Just to give an idea, on days with heavy development activity SEP005-compliance testing already gets 300 downloads from pip, just from our team.
A philosophical/practical reason, I've learned to appreciate the concept of 'Seperation of concerns' where python packages serve a single purpose. This aids maintenance, CI/CD and discussions. My concern would be that if I were to want to make an update to   the compliance testing that I would need to get this pushed through the whole sdypy versioning and have to wait for the sdypy CI/CD pipeline, which given its scope will always have more inertia. E.g. I can now push an update to the code and have a new version on pip within minutes. In our team I steer development similarly. The smaller a package, the better.
An administrative reason, maybe the weakest reason but a trend we need to be aware of. Our university is slowly but surely asking that we attribute code developments to (research) projects. Having it a seperate repo on github makes this easy, while a contribution to a bigger package is less evident.

I don't think the choice of keeping it a separate package has any downsides, au contraire the reasons listed above IMO benefit us all. The package is also fully open source, so anyone can contribute. I'm also willing to transfer ownership/admin rights, whatever. I just would like to keep the ability to install this lightweight package directly from pip. :-)

Thank you @wweijtje .
I understand your need (dependency, control, speed); We try to replicate numpy.testing module, but for sdypy. I am also in favor of small packages; however, sdypy has the ambition to be the framework package for structural dynamics and in the long run should be stable, and backward incompatibility should be carefully handled. If we have a separate package for this sep, will we have similar packages for any future seps or functionalities? This will be hard to handle. Hoe do you @wweijtje see it? Any other opinions?


Allow schema customization?

first of all, i really want to appreciate your hard work on this project. this is amazing considering that the old list plugin is not working fluently.
i just wanted to ask you if you have plans to allow users to customize node spec for list nodes. it's because i give my own classes for all nodes and marks of my editor schema and flat-list seems to not support the feature yet.

Thanks for your kind words.
You can use customize the node spec by wrapping the createListSpec API. Here is an example:
import { createListSpec } from 'prosemirror-flat-list'

function createCustomListSpec() {
  const spec = createListSpec()
  return {
    ...spec,
    attrs: { ...spec.attrs, color: { "default": "red" } }
    toDOM: (node) => {
      let domArray = spec.toDOM(node)
      let domAttrs = domSpec[1]
      if (typeof domAttrs === 'object') {
        domAttrs['data-list-color'] = node.attrs.color
      }
    }
  }
}

Hope this helps.

thank you very much. i will try it ^_^

@ocavue just one more question. can i opt out of some kinds, when creating the spec?
i only want to apply the ordered, and bullet kind.
i know i can do that using your example but just wanted to know if there's an option!

There is no an option to override kind.

Closing this, as it seems no action is required.


Please invite me to the community

Name
Perfect0B0D
Additional Context
I would like to be part of awesome community.

Invitation sent to join the GitHub Organization. Welcome to Infraform 🎉Please check your mail to accept the invitation sent by GitHub! 🤖About usInfraform is a remarkable open-source solution tailored for serverless platforms, aimed at empowering creators to craft their infrastructure solutions. 💻 We're thrilled to welcome you into our vibrant community! ✨ We eagerly anticipate your valuable contributions! 🤓 Don't forget to explore our community on Github and connect with us on LinkedIn 💜 Feel free to spread the word about our community to like-minded individuals interested in development, DevOps, technology, and collaborative contributions. 👉💜👈


Algorithms are Obsolete.

The algorithms that you have used no more give the correct solutions for the Assignments!

Yes they are. I tried doing the questions on my own but was struggling a lot. Decided to look up for solutions to see how they work and found this. I have been trying to make this work but these are obsolete and do not run or even if they do, the solutions are wrong.

heres the code that works if anyone is still looking for the answers
import time
file_path = "clustering1.txt"
with open(file_path, "r") as file:
num_nodes = int(file.readline())
clusters = {x: [x] for x in range(1, num_nodes + 1)}
edges = []
for line in file:
node1, node2, cost = map(int, line.strip().split())
edges.append((node1, node2, cost))
sorted_edges = sorted(edges, key=lambda x: x[2])
k = 4
def get_key_by_value(d, k):
for leader, nodes in d.items():
if k in nodes:
return leader
def update_clusters(c, a, d):
c[a].extend(c[d])
del c[d]
iteration = 0
start_time = time.time()
for edge in sorted_edges:
node1, node2, cost = edge
key1 = get_key_by_value(clusters, node1)
key2 = get_key_by_value(clusters, node2)
if key1 == key2:
continue
elif len(clusters[key1]) >= len(clusters[key2]):
update_clusters(clusters, key1, key2)
else:
update_clusters(clusters, key2, key1)
if len(clusters) == k:
break
max_spacing = None
for edge in sorted_edges:
node1, node2, cost = edge
key1 = get_key_by_value(clusters, node1)
key2 = get_key_by_value(clusters, node2)
if key1 != key2:
max_spacing = cost
break
end_time = time.time()
execution_time = end_time - start_time
print("Execution time: {:.6f} seconds".format(execution_time))
print("Execution time: {:.6f} minutes".format(execution_time / 60))
print("Execution time: {:.6f} hours".format(execution_time / 3600))
print("Maximum spacing of a 4-clustering: ", max_spacing)

i was able to do it


all channels showing as offline

I just set this up, and using the default mqtt setup I see my fireboard appear on my mqtt broker, and upon initial connection I get the current temperature of the probes as the "state", however I also get "offline" on the availability topic. I get no temperature updates beyond the initial value, and in the integration all channels simply show "offline."
The log (verbose, although that doesn't seem to make a difference) is:
10:27:26] INFO: Preparing to start...
[10:27:26] INFO: Verbose logging enabled.
[fireboard2mqtt:cli] starting fireboard2mqtt
[fireboard2mqtt:cli] initializing mqtt client...
[fireboard2mqtt:cli] mqtt client ready.
[fireboard2mqtt:mqttclient] connecting to mqtt
[fireboard2mqtt:mqttclient] connected to mqtt broker mqtt://core-mosquitto:1883
[fireboard2mqtt:controller] initializing fireboard device GM9GJR6G7
/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:216
const speedPercentage = updateLog.speedPercentage;
^
TypeError: Cannot read properties of undefined (reading 'speedPercentage')
at Fireboard.getDrivePayload (/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:216:43)
at Fireboard.updateDrive (/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:153:31)
at Fireboard.start (/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:182:14)
at Controller.start (/opt/fireboard2mqtt/lib/Controller.js:38:23)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

I have the exact same issue. Running Home Assistant 2023.7 and the latest fireboard2mqtt version 2.0.1 from this repo. Any thoughts @gordlea?
[22:31:12] INFO: Preparing to start...
[fireboard2mqtt:cli] starting fireboard2mqtt
[fireboard2mqtt:cli] initializing mqtt client...
[fireboard2mqtt:cli] mqtt client ready.
[fireboard2mqtt:mqttclient] connecting to mqtt
[fireboard2mqtt:mqttclient] connected to mqtt broker mqtt://<IP_ADDRESS>:1883
[fireboard2mqtt:controller] initializing fireboard device GJKK6737J
/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:216
const speedPercentage = updateLog.speedPercentage;
^
TypeError: Cannot read properties of undefined (reading 'speedPercentage')
at Fireboard.getDrivePayload (/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:216:43)
at Fireboard.updateDrive (/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:153:31)
at Fireboard.start (/opt/fireboard2mqtt/lib/fireboard/Fireboard.js:182:14)
at Controller.start (/opt/fireboard2mqtt/lib/Controller.js:38:23)
at processTicksAndRejections (node:internal/process/task_queues:96:5)

https://github.com/gordlea/fireboard2mqtt/issues/8#issuecomment-1656749749

This should be fixed with gordlea/fireboard2mqtt#40


Add Babel user: directlyrics

Would love to be listed in your users list.

@ymschaap thanks for the PR! any chance you have a svg version of the logo (sized to 500x200 & optimized)?

Not right now. I would have to lookup the source, and create a true SVG.

@existentialism replaced it with the SVG

@ymschaap thanks!


feat: extend multi-tenancy for LinodeMachines

What type of PR is this?
/kind feature

What this PR does / why we need it:
Updates LinodeMachines to be able to supply an (optional) CredentialsRef field to override the credentials of their owner LinodeCluster.
This allows for a (potentially future) enhancement where clusters can provision Linode resources cross-account.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
This is being introduced now (rather than when the feature is determined to be absolutely needed) to provide a more stable API during development. However, we may wait to add this feature since:

The setting introduced is optional. Adding it later would have no effect on existing clusters post-addition.
We are allowed to break, drop support, etc. for alpha APIs without notice: https://kubernetes.io/docs/reference/using-api/#api-versioning

TODOs:

[X] squashed commits
[X] includes documentation
[ ] adds unit tests
[ ] adds or updates e2e tests

LGTM


feat: improve QR code error-handling

Description

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[ ] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #
Risk

Testing

Engineering

Operations

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #4958   👈

This comment was auto-generated by Graphite.


Creation of cas_view_only user role

Dependencies
Blocked by #2493 (not technically blocked, but will avoid conflicts and be easier to wrap our heads around)
Same with #2522
Description:
As a cas_view_only user, I should have view-only functionality as listed in this table. PR started here: https://github.com/bcgov/cas-registration/pull/2521/files#diff-ec9c317c0857a29d2d59d94f0a0bc81f91a3b6dd731246575bcbb206c9d41161. The BE permissions will probably need to be redone, but the dashboard and FE routing should be reusable.
Acceptance Criteria:

[ ] View-only

Development Checklist:

[x] migration to add the cas_view_only role (e.g. of adding cas_analyst: cas-registration/bc_obps/registration/migrations/0002_prod_data.py, or see how cas_director is added)
[x] in cas-registration/bciers/apps/registration/app/idir, create a cas_view_only folder. Copy/paste the structure of cas_admin into this folder (cas_view_only is allowed to view everything)
[x] add cas_view_only to the "access_roles" in the internal.json files to give cas_view_only users access to all dashboard tiles
~- [ ] add conditionality to the operator admin role approvals so that cas_view_only cannot manage first-time operator admin role approvals (only cas_director and cas_analyst can). Could fetch the role in the grid component, or pass the role_name (we know what it is from the file structure) as a prop in page.tsx~ approval feature isn't built yet; added this to #2059 instead
[x] can't assign boro or bcghg id (this should already be done as part of #2493
[x] Check and update the endpoint and service permissions for this new role to ensure the user has the correct access.
[ ] [ ] Meets the DOD

Definition of Ready (Note: If any of these points are not applicable, mark N/A)

[ ] User story is included
[ ] User role and type are identified
[ ] Acceptance criteria are included
[ ] Wireframes are included (if required)
[ ] Design / Solution is accepted by Product Owner
[ ] Dependencies are identified (technical, business, regulatory/policy)
[ ] Story has been estimated (under 13 pts)

·Definition of Done (Note: If any of these points are not applicable, mark N/A)

[ ] Acceptance criteria are tested by the CI pipeline
[ ] UI meets accessibility requirements
[ ] Configuration changes are documented, documentation and designs are updated
[ ] Passes code peer-review
[ ] Passes QA of Acceptance Criteria with verification in Dev and Test
[ ] Ticket is ready to be merged to main branch
[ ] Can be demoed in Sprint Review
[ ] Bugs or future work cards are identified and created
[ ] Reviewed and approved by Product Owner

Notes:

I recommend (with agreement from some other devs) that it would be better to name this role cas_read, cas_view_only, cas_read_only, or the like, for the sake of being more explicit in what that role does. cas_user is very generic.

Let's change it! I vote to change it to cas_view_only
@andrea-williams

assumed cas_view-only and navigated back and forth to confirm the role can view but not perform an action. accepting ticket.


Thumbnail system doesn't handle tiny source images very nicely

Follow-up to #192.
The thumbnail spec currently has 'small': {size: 250, quality: 85} as its smallest thumbnail dimension. If a source image is smaller than that, the smallest one is the only thumb generated.
That's problematic because "small" is specifically intended for display in contexts like grids or carousels, where resolution is less important. But if it's the smallest thumb, then it will be used in other contexts too, like the cover art container, commentary, and even the large image overlay!
I'm not exactly sure the best solution, but maybe the spec could indicate what purposes each potentially-available thumb is "suitable" for. For example, we could set high-jpeg-quality images to never be suitable for display in a grid; and low-jpeg-quality images to never be suitable as key art.
I don't think we'd actually need to define an additional "key art but really small" option (e.g. {size: 250, quality: 95}) because medium already suffices. We'd always generate a medium thumb because it's the smallest size suitable for one purpose; just like we'd always generate small because it's the smallest suitable for another purpose. And of course, medium limits dimensions to 400x400 - it doesn't scale up, so the quality:95 is all that takes effect.
I've labelled this as "bug (user-facing)" because it would impact small pixel art badly, but it's not particularly time-sensitive because the wiki doesn't actually have any such low-resolution track art.

See above issues referencing this one!
The "suitabilities" discussed will probably be named according to intrinsic properties of the thumbnail, rather than extrinsic "where does this fit best" conditions.

heavier-compression
lighter-compression
cropped-to-square
retains-aspect-ratio

My thought is, the code (whether client or content generation) would declare what suitabilities it's looking for, as well as a desired target dimensions. Then the thumbnail engine would respond with whichever thumbnail suits all of the stated requirements and is closest to the stated dimensions (possibly within a "good enough" threshold).
This would result in a generally cleaner system than the "cover" and "semilarge" dimensions proposed in #245, while providing the same underlying capabilities. (Selecting the correct dimensions for display on the client, preferably with only HTML, is still TBD - but this implementation would provide the capability to automatically identify which size is best suitable for retina display, just by doubling the provided target dimensions!)


More fixes for the German translation

I have corrected some more small issues in the German translation.

Merged, thx!


Lots of Improvements

cleaned up DriverBase
locked EosLib ver
fixed async MQTT bugs
refactored orchEOStrator and logging
implemented a basic shared queue between driver threads
made a test driver to demonstrate priority queue between read/command thread

looks good to me


Browser compatibility

My configuration is out of order: Uncaught Error: The node API for babel has been moved to babel-core.__

This issue seemed not related to Nuxt.js, you can refer: https://github.com/babel/babel-loader/issues/259


Not working with on("click", ... ) handlers

Hi,
it seems that the plugin is not working when I have the event handlers registered like this:
  $(document).on "click", ".do-confirm", ->
    console.log "hey"
    return

  $(".do-confirm").click ->
    console.log "hoe"
    return

  $(".do-confirm").popConfirm()

The "hoe" gets printed, but the "hey" does not.
Is there a reason why this is this way?
Cheers,
Gerwin
P.S.: I'd lover to use this plugin with meteorJS and they register the events like mentioned in the first example.

Hi @gerwinbrunner
I had the same problem so created the pull request below. It's not really a solution but it's a workaround that might get your further.
https://github.com/Ifnot/PopConfirm/pull/33


Timeline summary should contain CPU/GPU and memory usage

as of flutter.dev/go/engine-cpu-profiling, we collect the CPU and Memory usage. With work being done to collect GPU usage on iOS as well.
Relevant PRs:

CPU: https://github.com/flutter/engine/pull/18087
Memory: https://github.com/flutter/engine/pull/18516
GPU: https://github.com/flutter/engine/pull/18752

This issue is to track collecting and posting these as avg, 90th and 99th percentile metrics to the timeline_summary.json file.
cc: @liyuqian , @AlbertWang0116, @gaaclarke

It would be nice to also have them on Android!


[New] Some feature requirements

Right clicking a ladder with another one will place it in the next block .
Swing Through Grass
Note Blocks with heads attached to it can emit the respective mob's sound.
A Trowel like Quark

Addressed in 929b181848aace5e9f7a496bf224e2c733bd8649:

Done
This would require a large rewrite of the combat system, which isn't feasible at this time.
Done
Done


compress html if desired

If compress is passed as true at the time of express-handlebars instantiation, engine would return compressed markup
var exphbs = require('express-handlebars');
...
app.engine('html', exphbs({
    handlebars: handlebars,
    compress: true
}));

@sarbbottam what sort of savings does this provide over gzip on a real world test page with compress set to true over gzip of that same page without this option set?

I'm closing this because I don't believe it will have any meaningful impact on a page that's served with gzip at the HTTP level.


S3 comments and modifications to TIGRESS position

There was some confusion during the setting up for the upcoming TIGRESS experiment regarding the S3 position codes, this caused some delays while double checking detector mapping problems.
I've added some additional comments which should hopefully reduce the problem in the future.
I've also changed the way the TIGRESS position function works in order to add some extra debugging functionality (which turned out not to be needed ...)
I'm doubt there is any computational advantage to having the vectors built once rather than every time they are called as I have it done now, but it seemed neat at the time.

One small change: PostionVectors, CloverRadial, and CloverCross should all have a lower-case f in front (and you might want to correct the spelling of PostionVectors but that isn't important).

Fixed those.

Thanks, merged.


[Question] How to get the current buffer size from a buffer_state?

Hello, Is there any method that can be used to return the current buffer size? I overview the code, however, I do not find any methods like:
size = rb.size(rb_state)

It seems that I can indirectly get it from the field TrajectoryBufferState.current_index. Any suggestions?

Misc

[ ] Check for duplicate requests.

Indeed! If you have a buffer_state, you can simply access it as buffer_state.current_index :)

Hey @ZaberKo. Indeed! If you have a buffer_state, you can simply access it as buffer_state.current_index :)

Thanks for the quick reply. However, if I understand the code correctly, when the buffer reaches its maximum capacity, it will run like a ring buffer, and buffer_state.current_index cannot represent the buffer_size anymore. Besides, I am not sure that all types of buffers behave like this. Maybe we need to add a function for each type of buffer class to get their current batch_size, or explicitly add a size field to buffer_state?

That's a great point, @ZaberKo! The correct buffer size is the current index if we haven't filled the buffer, otherwise it's the max possible size. You can easily do this as:
from flashbax.utils import get_tree_shape_prefix
max_time_size = get_tree_shape_prefix(state.experience, 2)[1]
filled_buffer_size = size = state.current_index if not state.is_full else max_time_size

We could add this is a simple utility function, if it's desired. But it's easily accessible as the 2nd axis of the state's experience—we don't need a separate field.
What do you think, @EdanToledo?

@EdanToledo good catch on the batch size! Happy to PR this sometime today :)

Just that b_size = buffer_state.experience.shape[0] won't work, as buffer_state.experience will have fields (e.g. "acts" or whatever). Hence using the flashbax.utils.get_tree_shape_prefix instead.

So something like this,
def get_total_timesteps(state):
    b_size, t_size_max = get_tree_shape_prefix(state.experience, 2)
    t_size = t_size_max if state.is_full else state.current_index
    return b_size * t_size

Will PR and test ASAP 💯

Hey @ZaberKo! Does #34 solve your problem? 🙏🏻

So something like this,

def get_total_timesteps(state):

    b_size, t_size_max = get_tree_shape_prefix(state.experience, 2)

    t_size = t_size_max if state.is_full else state.current_index

    return b_size * t_size

Will PR and test ASAP 💯

Great work, does the control flow work in JIT, I can't remember if they updated jax to work with if like that. Otherwise just use a lax.cond please.

Good catch, @EdanToledo! Thanks. Fixed with lax.cond

Hey @ZaberKo! Does #34 solve your problem? :)

Yes. I will close this issue. And when will it be put on a new release?


Missing assets in Chapter 2

Holograms-211 - Chapter 2 - Navigation
Errata: 2. In the Holograms folder, find the ScrollFeedback asset.

But there's no ScrollFeedback asset in the provided Unity package

@cre8ivepark @keveleigh @davidkline-ms can you guys please take a look at this? thanks!

Which Unity package are you referring to? Only Holograms 210 provides a package, while Holograms 211 and 212 contain starting projects, which both contain slightly different contents than the 210 package.
The 211 starting project does contain ScrollFeedback in the Holograms folder: https://github.com/Microsoft/HolographicAcademy/blob/Holograms-211-Gesture/Starting/ModelExplorer/Assets/Holograms/ScrollFeedback.prefab


Fix/1538

Fixes #1538
By default no algos are allowed now.
Related to https://github.com/oceanprotocol/docs/pull/1032

mostly fixed over in compute PR in edd80eed7719f779a044e71639edbf97d3f04822 & ec34dc8f537b5c3e957661833f12567bf7d6403d.
But still missing that empty array is correctly read out in edit form


What do you specify in the LIST_OF_ANSWERS slot type?

What do you provide in the "LIST_OF_ANSWERS" Slot type definition?  I am just learning, so sorry if this is a basic question./

Click Add Slot and Mention the type as "LIST_OF_ANSWERS" and in the values section
1 | 2 | 3 | 4
some thing like above. depending on what values you would need. the values must be line separated ( | )

See the minecraftHelper sample.  It uses a custom slot type with a list of items.
https://github.com/amzn/alexa-skills-kit-js/blob/master/samples/minecraftHelper/speechAssets/customSlotTypes/LIST_OF_ITEMS


vc_runtimeminimum_x64 error when performing update

Microsoft PowerToys version
0.55.2
Running as admin

[X] Yes

Area(s) with issue?
Installer
Steps to reproduce
Hi
I recently attempted to update Powertoys through the app and I am receiving an error every time.
the installer runs and then it throws the following window up

Presumably the vc_runtimeminimum_x64.msi should already be included in the with the installation files instead of asking me to locate it?
✔ Expected Behavior
I was expecting the updater to run and have all of the files within it that were required to update the program
❌ Actual Behavior
The updater seems to be missing some files that it needs to perform the update. It then asks me to locate where these particular files are.
Other Software
No response

did you run some random cleaner tool?

/needinfo

No I have never run any tool like that. Is this a package you would expect to be on a windows machine as standard?
Cheers

Are you trying to install 0.55.2?  Or install .60.1 and getting that error?
/reportbug

PowerToysReport_2022-07-20-09-25-01.zip
My current version is 0.55.2. This error happens when I am trying to update to 0.60.1
Thanks

Does this still happen?
/needinfo

Hi Aaron
Nope, everything updates fine now
Thanks


Set the min and maximum

Hi,
When I try to set a min and max year, It doesn't work well...
So I want to get a first year selection of 1970 and then the maximum year must be the current year, how can I do this? Screen: http://prntscr.com/7tqtea
So that dat moest end with 2015 (for now) and begin with 1970. How can I do this?
Kindest regards,
Robin.

Ohw, also when I set the formatSubmit to formatSubmit: 'Y-m-d 00:00:00', It doesn't give the year with it? I really need to get this working...

http://amsul.ca/pickadate.js/date/#disable-dates-use-arrays
In examples given under this link for min and max date doesn't work.
Minimum is set to 20 March but show maximum upto 20 April
and max is set to 14 July, goes upto 14 August.

Yes, this is a bug. It allows the user to select a date up to one month LATER than the specified max date.

I'm wondering why this issue was closed, since the bug still exists. You can even see it in the demo on this website. The max date is specified as 2015-07-14, yet the control allows the user to select up to 2015-08-14.

Reopend since the bug still exists, however this bug was repored since 2015 and nothing has been done with it.

For "new Date()" in JavaScript month parameter is zero-based index, so for 2015-07-14 you have to set new Date(2015, 6, 14).

Exactly, and this is why this is not a bug.


[Question] any API can be used to delete images?

hi
i want to delete old images, for example,  1 month ago,
any API can be used to do this ?
thanks
@ywk253100

I assume you have figured out.


Port to ESP32 hardware

We would like to port the Python code to work with the ESP32. This thread will discuss the work associated with this port.

Yes, sirrrrr.
Now I think the first step should be to create our own "bleaker" package.

Here is something you can use:
https://github.com/2black0/MicroPython-ESP32-BLE
On Fri, Jun 28, 2024, 8:41 AM notrealdreamy @.***>
wrote:

Yes, sirrrrr.
Now I think the first step should be to create our own "bleaker" package.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2196820076,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37Q3XPMZHSO557FHDBDZJVKWNAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJWHAZDAMBXGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

We can get rid of the temp and humidity sensor code.I think its sending temp data to itself?

That's solved some of it.

👍
On Sun, Jun 30, 2024, 8:12 AM notrealdreamy @.***>
wrote:

That's solved some of it.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2198541984,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37X2JIKJ3SOG72EYGXLZJ7Y4JAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJYGU2DCOJYGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

from machine import Pin, Timer, SoftI2C
from time import sleep_ms
import ubluetooth
Dummy sensor class for emulation
class DummySensor:
def read_temperature(self, fahrenheit=False):
return 25.0 if not fahrenheit else 77.0
def read_humidity(self):
    return 50.0

Initialize sensor
sensor = DummySensor()
class BLE():
def init(self, name):
self.name = name
self.ble = ubluetooth.BLE()
self.ble.active(True)
    self.led = Pin(2, Pin.OUT)
    self.timer1 = Timer(0)
    self.timer2 = Timer(1)

    self.disconnected()
    self.ble.irq(self.ble_irq)
    self.register()
    self.advertiser()

def connected(self):
    self.timer1.deinit()
    self.timer2.deinit()

def disconnected(self):
    self.timer1.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(1))
    sleep_ms(200)
    self.timer2.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(0))

def ble_irq(self, event, data):
    if event == 1:
        '''Central connected'''
        self.connected()
        self.led(1)

    elif event == 2:
        '''Central disconnected'''
        self.advertiser()
        self.disconnected()

    elif event == 3:
        '''New message received'''
        buffer = self.ble.gatts_read(self.rx)
        message = buffer.decode('UTF-8').strip()
        print(message)
        if message == 'red_led':
            red_led.value(not red_led.value())
            print('red_led', red_led.value())
            self.send('red_led' + str(red_led.value()))
        if message == 'read_temp':
            print(sensor.read_temperature(True))
            self.send(str(sensor.read_temperature(True)))

def register(self):
    # Nordic UART Service (NUS)
    NUS_UUID = '6E400001-B5A3-F393-E0A9-E50E24DCCA9E'
    RX_UUID = '6E400002-B5A3-F393-E0A9-E50E24DCCA9E'
    TX_UUID = '6E400003-B5A3-F393-E0A9-E50E24DCCA9E'

    BLE_NUS = ubluetooth.UUID(NUS_UUID)
    BLE_RX = (ubluetooth.UUID(RX_UUID), ubluetooth.FLAG_WRITE)
    BLE_TX = (ubluetooth.UUID(TX_UUID), ubluetooth.FLAG_NOTIFY)

    BLE_UART = (BLE_NUS, (BLE_TX, BLE_RX,))
    SERVICES = (BLE_UART, )
    ((self.tx, self.rx,), ) = self.ble.gatts_register_services(SERVICES)

def send(self, data):
    self.ble.gatts_notify(0, self.tx, data + '\n')

def advertiser(self):
    name = bytes(self.name, 'utf-8')
    adv_data = bytearray(b'\x02\x01\x02') + bytearray((len(name) + 1, 0x09)) + name
    self.ble.gap_advertise(100, adv_data)

test
i2c = SoftI2C(scl=Pin(22), sda=Pin(21))
red_led = Pin(2, Pin.OUT)
ble = BLE("ESP32")

'''
from machine import Pin, Timer, SoftI2C
from time import sleep_ms
import ubluetooth
Dummy sensor class for emulation
class DummySensor:
def read_temperature(self, fahrenheit=False):
return 25.0 if not fahrenheit else 77.0
def read_humidity(self):
    return 50.0

Initialize sensor
sensor = DummySensor()
class BLE():
def init(self, name):
self.name = name
self.ble = ubluetooth.BLE()
self.ble.active(True)
    self.led = Pin(2, Pin.OUT)
    self.timer1 = Timer(0)
    self.timer2 = Timer(1)

    self.disconnected()
    self.ble.irq(self.ble_irq)
    self.register()
    self.advertiser()

def connected(self):
    self.timer1.deinit()
    self.timer2.deinit()

def disconnected(self):
    self.timer1.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(1))
    sleep_ms(200)
    self.timer2.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(0))

def ble_irq(self, event, data):
    if event == 1:
        '''Central connected'''
        self.connected()
        self.led(1)

    elif event == 2:
        '''Central disconnected'''
        self.advertiser()
        self.disconnected()

    elif event == 3:
        '''New message received'''
        buffer = self.ble.gatts_read(self.rx)
        message = buffer.decode('UTF-8').strip()
        print(message)
        if message == 'red_led':
            red_led.value(not red_led.value())
            print('red_led', red_led.value())
            self.send('red_led' + str(red_led.value()))
        if message == 'read_temp':
            print(sensor.read_temperature(True))
            self.send(str(sensor.read_temperature(True)))

def register(self):
    # Nordic UART Service (NUS)
    NUS_UUID = '6E400001-B5A3-F393-E0A9-E50E24DCCA9E'
    RX_UUID = '6E400002-B5A3-F393-E0A9-E50E24DCCA9E'
    TX_UUID = '6E400003-B5A3-F393-E0A9-E50E24DCCA9E'

    BLE_NUS = ubluetooth.UUID(NUS_UUID)
    BLE_RX = (ubluetooth.UUID(RX_UUID), ubluetooth.FLAG_WRITE)
    BLE_TX = (ubluetooth.UUID(TX_UUID), ubluetooth.FLAG_NOTIFY)

    BLE_UART = (BLE_NUS, (BLE_TX, BLE_RX,))
    SERVICES = (BLE_UART, )
    ((self.tx, self.rx,), ) = self.ble.gatts_register_services(SERVICES)

def send(self, data):
    self.ble.gatts_notify(0, self.tx, data + '\n')

def advertiser(self):
    name = bytes(self.name, 'utf-8')
    adv_data = bytearray(b'\x02\x01\x02') + bytearray((len(name) + 1, 0x09)) + name
    self.ble.gap_advertise(100, adv_data)

test
i2c = SoftI2C(scl=Pin(22), sda=Pin(21))
red_led = Pin(2, Pin.OUT)
ble = BLE("ESP32")
'''

`from machine import Pin, Timer, SoftI2C
from time import sleep_ms
import ubluetooth
Dummy sensor class for emulation
class DummySensor:
def read_temperature(self, fahrenheit=False):
return 25.0 if not fahrenheit else 77.0
def read_humidity(self):
    return 50.0

Initialize sensor
sensor = DummySensor()
class BLE():
def init(self, name):
self.name = name
self.ble = ubluetooth.BLE()
self.ble.active(True)
    self.led = Pin(2, Pin.OUT)
    self.timer1 = Timer(0)
    self.timer2 = Timer(1)

    self.disconnected()
    self.ble.irq(self.ble_irq)
    self.register()
    self.advertiser()

def connected(self):
    self.timer1.deinit()
    self.timer2.deinit()

def disconnected(self):
    self.timer1.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(1))
    sleep_ms(200)
    self.timer2.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(0))

def ble_irq(self, event, data):
    if event == 1:
        '''Central connected'''
        self.connected()
        self.led(1)

    elif event == 2:
        '''Central disconnected'''
        self.advertiser()
        self.disconnected()

    elif event == 3:
        '''New message received'''
        buffer = self.ble.gatts_read(self.rx)
        message = buffer.decode('UTF-8').strip()
        print(message)
        if message == 'red_led':
            red_led.value(not red_led.value())
            print('red_led', red_led.value())
            self.send('red_led' + str(red_led.value()))
        if message == 'read_temp':
            print(sensor.read_temperature(True))
            self.send(str(sensor.read_temperature(True)))

def register(self):
    # Nordic UART Service (NUS)
    NUS_UUID = '6E400001-B5A3-F393-E0A9-E50E24DCCA9E'
    RX_UUID = '6E400002-B5A3-F393-E0A9-E50E24DCCA9E'
    TX_UUID = '6E400003-B5A3-F393-E0A9-E50E24DCCA9E'

    BLE_NUS = ubluetooth.UUID(NUS_UUID)
    BLE_RX = (ubluetooth.UUID(RX_UUID), ubluetooth.FLAG_WRITE)
    BLE_TX = (ubluetooth.UUID(TX_UUID), ubluetooth.FLAG_NOTIFY)

    BLE_UART = (BLE_NUS, (BLE_TX, BLE_RX,))
    SERVICES = (BLE_UART, )
    ((self.tx, self.rx,), ) = self.ble.gatts_register_services(SERVICES)

def send(self, data):
    self.ble.gatts_notify(0, self.tx, data + '\n')

def advertiser(self):
    name = bytes(self.name, 'utf-8')
    adv_data = bytearray(b'\x02\x01\x02') + bytearray((len(name) + 1, 0x09)) + name
    self.ble.gap_advertise(100, adv_data)

test
i2c = SoftI2C(scl=Pin(22), sda=Pin(21))
red_led = Pin(2, Pin.OUT)
ble = BLE("ESP32")
`

from machine import Pin, Timer, SoftI2C
from time import sleep_ms
import ubluetooth

# Dummy sensor class for emulation
class DummySensor:
    def read_temperature(self, fahrenheit=False):
        return 25.0 if not fahrenheit else 77.0

    def read_humidity(self):
        return 50.0

# Initialize sensor
sensor = DummySensor()

class BLE():
    def __init__(self, name):
        self.name = name
        self.ble = ubluetooth.BLE()
        self.ble.active(True)

        self.led = Pin(2, Pin.OUT)
        self.timer1 = Timer(0)
        self.timer2 = Timer(1)

        self.disconnected()
        self.ble.irq(self.ble_irq)
        self.register()
        self.advertiser()

    def connected(self):
        self.timer1.deinit()
        self.timer2.deinit()

    def disconnected(self):
        self.timer1.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(1))
        sleep_ms(200)
        self.timer2.init(period=1000, mode=Timer.PERIODIC, callback=lambda t: self.led(0))

    def ble_irq(self, event, data):
        if event == 1:
            '''Central connected'''
            self.connected()
            self.led(1)

        elif event == 2:
            '''Central disconnected'''
            self.advertiser()
            self.disconnected()

        elif event == 3:
            '''New message received'''
            buffer = self.ble.gatts_read(self.rx)
            message = buffer.decode('UTF-8').strip()
            print(message)
            if message == 'red_led':
                red_led.value(not red_led.value())
                print('red_led', red_led.value())
                self.send('red_led' + str(red_led.value()))
            if message == 'read_temp':
                print(sensor.read_temperature(True))
                self.send(str(sensor.read_temperature(True)))

    def register(self):
        # Nordic UART Service (NUS)
        NUS_UUID = '6E400001-B5A3-F393-E0A9-E50E24DCCA9E'
        RX_UUID = '6E400002-B5A3-F393-E0A9-E50E24DCCA9E'
        TX_UUID = '6E400003-B5A3-F393-E0A9-E50E24DCCA9E'

        BLE_NUS = ubluetooth.UUID(NUS_UUID)
        BLE_RX = (ubluetooth.UUID(RX_UUID), ubluetooth.FLAG_WRITE)
        BLE_TX = (ubluetooth.UUID(TX_UUID), ubluetooth.FLAG_NOTIFY)

        BLE_UART = (BLE_NUS, (BLE_TX, BLE_RX,))
        SERVICES = (BLE_UART, )
        ((self.tx, self.rx,), ) = self.ble.gatts_register_services(SERVICES)

    def send(self, data):
        self.ble.gatts_notify(0, self.tx, data + '\n')

    def advertiser(self):
        name = bytes(self.name, 'utf-8')
        adv_data = bytearray(b'\x02\x01\x02') + bytearray((len(name) + 1, 0x09)) + name
        self.ble.gap_advertise(100, adv_data)

# test
i2c = SoftI2C(scl=Pin(22), sda=Pin(21))
red_led = Pin(2, Pin.OUT)
ble = BLE("ESP32")

ran without a bug now i will shred it down to what's useful for us.

Ok. For the Casio watches, there are 2 uuid's that are used, one for
reading data from the watch, and other for setting values on the watch. In
your example, they use different uuid's for another device, like LED
blinker. Look at my Python code, and similarly insulate the ble related
stuff into a few functions.
I think you can fork my project from GitHub, and make changes to your fork.
Also create a new branch on your side. Eventually, we can figure out how to
merge the code, or perhaps keep different branches for the esp and Linux
versions.
On Mon, Jul 1, 2024, 11:30 AM notrealdreamy @.***>
wrote:

ran without a error now i will shred it down to what's useful for us.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2200466585,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37RNH5JHOQWMY475YRDZKFY2VAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGQ3DMNJYGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

Also, do not try to integrate with my project yet. Write a small python
script to connect to the watch, and try to get the watch's name. This will
be a great milestone.
On Mon, Jul 1, 2024, 11:49 AM Ivo Zivkov @.***> wrote:

Ok. For the Casio watches, there are 2 uuid's that are used, one for
reading data from the watch, and other for setting values on the watch. In
your example, they use different uuid's for another device, like LED
blinker. Look at my Python code, and similarly insulate the ble related
stuff into a few functions.
I think you can fork my project from GitHub, and make changes to your
fork. Also create a new branch on your side. Eventually, we can figure out
how to merge the code, or perhaps keep different branches for the esp and
Linux versions.
On Mon, Jul 1, 2024, 11:30 AM notrealdreamy @.***>
wrote:

ran without a error now i will shred it down to what's useful for us.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2200466585,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37RNH5JHOQWMY475YRDZKFY2VAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBQGQ3DMNJYGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

Ok this is the part that you need to teach me how to send ble data to a gshock.

BTW, what is the mode of your watch?
On Tue, Jul 2, 2024 at 9:30 AM Ivo Zivkov @.***> wrote:

So, basically what you have to do is the following:

Connect to your watch.
Read and write to your watch.

Let's start with the connection. All code related to connection is in the
existing project in the file scanner.py.
There are two ways to connect:

Connect without knowing the MAC address of the
watch, something like EC:71:14:33:FD:51. In this case, we scan for BT
devices that match certain criteria, like the name starting with "casio".
This filters out other devices, like your BT headset. If such a device is
found, we save the name and the address in the config.ini file for the
next time we try to connect.
The second way to connect is if we know the MAC address. If we are
trying to connect the same watch, we can get the saved address from the
config.ini file.

So, you need to do the same thing, but using your BLE library, not
BleakScanner from the Linux bleak library.
I think this would be a good starting point.
A word about development tools: It would be good to have an IDE, like
Visual Code. This will allow you to set breakpoints and look at the data at
these breakpoints. This applies to the current Linux code, but it will help
you understand how the code works, and what you have to duplicate in your
project. That is why the Raspberry Pi is not an ideal environment for
development. I would start with installing Visual Code (
https://code.visualstudio.com/) on some old machine, preferably Linux,
and then step through the code to understand what is going on. There is a
way to run the code on the Pi and use Visual Code on another machine, like
the Mac, but this is a bit cumbersome.
So, spend some time learning how to use Visual Code if you are not
familiar with it. It will save you a lot of time in the future.
On Tue, Jul 2, 2024 at 7:42 AM notrealdreamy @.***>
wrote:

Ok this is the part that you need to teach me how to send ble data to a
gshock.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2202930041,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37Q2Z5X6SFHNG6NWDVTZKKG3JAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSHEZTAMBUGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

GA-B2100

So, basically what you have to do is the following: 1. Connect to your watch. 2. Read and write to your watch. Let's start with the connection. All code related to connection is in the existing project in the file scanner.py. There are two ways to connect: 1. Connect without knowing the MAC address of the watch, something like EC:71:14:33:FD:51. In this case, we scan for BT devices that match certain criteria, like the name starting with "casio". This filters out other devices, like your BT headset. If such a device is found, we save the name and the address in the config.ini file for the next time we try to connect. 2. The second way to connect is if we know the MAC address. If we are trying to connect the same watch, we can get the saved address from the config.ini file. So, you need to do the same thing, but using your BLE library, not BleakScanner from the Linux bleak library. I think this would be a good starting point. A word about development tools: It would be good to have an IDE, like Visual Code. This will allow you to set breakpoints and look at the data at these breakpoints. This applies to the current Linux code, but it will help you understand how the code works, and what you have to duplicate in your project. That is why the Raspberry Pi is not an ideal environment for development. I would start with installing Visual Code ( https://code.visualstudio.com/) on some old machine, preferably Linux, and then step through the code to understand what is going on. There is a way to run the code on the Pi and use Visual Code on another machine, like the Mac, but this is a bit cumbersome. So, spend some time learning how to use Visual Code if you are not familiar with it. It will save you a lot of time in the future.
…
On Tue, Jul 2, 2024 at 7:42 AM notrealdreamy @.> wrote: Ok this is the part that you need to teach me how to send ble data to a gshock. — Reply to this email directly, view it on GitHub <#12 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7M37Q2Z5X6SFHNG6NWDVTZKKG3JAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBSHEZTAMBUGE . You are receiving this because you authored the thread.Message ID: @.>

plan b,i bought a new esp32, flashed micropython and now i will be testing on it.

got esp32 in vs code

Great!
On Tue, Jul 2, 2024, 10:19 AM notrealdreamy @.***>
wrote:

Screenshot.2024-07-02.at.7.48.34.PM.png (view on web)
https://github.com/izivkov/GShockTimeServer/assets/88233084/9a5db592-2d91-4238-8cde-7b7b1347a786
got esp32 in vs code
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2203351101,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37XBNVZ7OMNVGWKDWPTZKKZGRAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBTGM2TCMJQGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

with the help of gpt
from bluetooth import BLE
import ubinascii

def scan_callback(event, data):
    if event == BLE.GAP_SCAN_RESULT:
        addr_type, addr, adv_type, rssi, adv_data = data
        addr_str = ubinascii.hexlify(addr).decode('utf-8')
        print(f"Device found: Address={addr_str}, RSSI={rssi}")

# Initialize BLE
ble = BLE()
ble.active(True)

# Start scanning for BLE devices
ble.gap_scan(10000, 30000, 30000, scan_callback)

Do you think it will work?

I see you are testing gab2100 with your python code.

Ok, this would be a good starting point. Scan for all BT devices in the
vicinity and display their address and name. Then we can filter for Casio
watch only.
On Tue, Jul 2, 2024, 11:49 PM notrealdreamy @.***>
wrote:

I see you are testing gab2100 with your python code.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2205034357,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37UD7BBL633QVRSVCYDZKNYFLAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVGAZTIMZVG4
.
You are receiving this because you commented.Message ID:
@.***>

You seem to have a good handle on tools and ChatGPT. Next, I would try to
really learn Python well. There are many tutorial courses on YouTube. The
best one I have seen is this 15-hour course from Harvard University:
https://www.youtube.com/watch?v=nLRL_NcnK-4&t=29323s. It is too long for
most people, but if you have the time, you will know Python well at the
end. Of course, there are many other shorter courses.
Good luck,
Ivo
On Wed, Jul 3, 2024 at 8:24 AM Ivo Zivkov @.***> wrote:

Ok, this would be a good starting point. Scan for all BT devices in the
vicinity and display their address and name. Then we can filter for Casio
watch only.
On Tue, Jul 2, 2024, 11:49 PM notrealdreamy @.***>
wrote:

I see you are testing gab2100 with your python code.
—
Reply to this email directly, view it on GitHub
https://github.com/izivkov/GShockTimeServer/issues/12#issuecomment-2205034357,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7M37UD7BBL633QVRSVCYDZKNYFLAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVGAZTIMZVG4
.
You are receiving this because you commented.Message ID:
@.***>

You seem to have a good handle on tools and ChatGPT. Next, I would try to really learn Python well. There are many tutorial courses on YouTube. The best one I have seen is this 15-hour course from Harvard University: https://www.youtube.com/watch?v=nLRL_NcnK-4&t=29323s. It is too long for most people, but if you have the time, you will know Python well at the end. Of course, there are many other shorter courses. Good luck, Ivo
…
On Wed, Jul 3, 2024 at 8:24 AM Ivo Zivkov @.> wrote: Ok, this would be a good starting point. Scan for all BT devices in the vicinity and display their address and name. Then we can filter for Casio watch only. On Tue, Jul 2, 2024, 11:49 PM notrealdreamy @.> wrote: > I see you are testing gab2100 with your python code. > > — > Reply to this email directly, view it on GitHub > <#12 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/AA7M37UD7BBL633QVRSVCYDZKNYFLAVCNFSM6AAAAABKBZ72MCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVGAZTIMZVG4 > . > You are receiving this because you commented.Message ID: > @.***> >

Thank you for sending this, I was also thinking that i was too dependent on these tools.I am gona watch the whole vid part by part.☺

I have removed one of the dependencies for the project, namely the "reactivex" library. I was using this library before, but it is no longer needed. This should make things easier for MicroPython since it does not support this library.
Also, for the BLE section, they recommend this higher-level library for MicroPython:
https://github.com/micropython/micropython-lib/tree/master/micropython/bluetooth/aioble
I am not familiar with it; maybe it is not useful, but we can look into it when you are ready.
Regards

By reading readme it seems useful.


[Parse] Disable support for multiline/extended escaping string literal in attribute message

Strings of diagnostics message processed in EncodedDiagnosticMessage aren't necessarily from parsed Swift source code. That means, they might not have quotes around it. Furthermore, memory around them might not be managed. The logic in Lexer::getEncodedStringSegment() used to cause
access violation.
For now, disable multiline string literal and extended escaping in string literal for attribute message position. Considering the message might be from Clang, we cannot simply enable this.
rdar://problem/44228891

@swift-ci Please ASAN test

@swift-ci Please smoke test

Hi @rintaro, fair enough reading of the end of the buffer was not clever here. I’m not fussed about raw strings in depreciation messages but multiline strings could be useful for longer messages.
How about:
  // Special case when being called from EncodedDiagnosticMessage(...)
  // This allows multiline strings to work as attribute messages.
  if (IndentToStrip == ~0u) {
    const char *Trailing = Bytes.end() - 1;
    while (Trailing > BytesPtr && (*Trailing == ' ' || *Trailing == '\t'))
      Trailing--;

    IndentToStrip = *Trailing == '\n' ? Bytes.end() - Trailing - 1 : 0;
  }

@johnno1962 Sounds good. Please post a new PR with test cases after I merge this change.

Followed up with https://github.com/apple/swift/pull/19219 which gets multiline messages working while respecting the bounds of the StringRef passed in.


Store NetworkPolicy in filesystem as fallback data source

In the previous implementation, traffic from/to a Pod may bypass NetworkPolicies applied to the Pod in a time window when the agent restarts because realizing NetworkPolicies and enabling forwarding are asynchronous.
This patch stores NetworkPolicy data in files when they are received, and makes antre-agent fallback to use the files as data source if it can't connect to antrea-controller on startup. This prevents security regression: a NetworkPolicy that has been realized on a Node will continue to work even if antrea-controller is not available after antrea-agent restarts.
The benchmark results of the storage's operations are as below:
BenchmarkFileStoreAddNetworkPolicy-40              70383             16102 ns/op             520 B/op          9 allocs/op
BenchmarkFileStoreAddAppliedToGroup-40             45382             25880 ns/op            3019 B/op          9 allocs/op
BenchmarkFileStoreAddAddressGroup-40                7400            180000 ns/op           49538 B/op          9 allocs/op
BenchmarkFileStoreReplaceAll-40                       10         127088004 ns/op        17815943 B/op      33099 allocs/op

The disk usage when storing 1k NetworkPolicies, AddressGroups, and AppliedToGroups created by BenchmarkFileStoreReplaceAll is as below:
16M     /var/run/antrea-test/file-store/address-groups
4.0M    /var/run/antrea-test/file-store/applied-to-groups
4.0M    /var/run/antrea-test/file-store/network-policies

Note that the patch doesn't synchronize NetworkPolicy initial sync and Pod flow installation, so NetworkPolicy breach could still happen on agent restart. I plan to address it via a separate PR to make each PR more focused.

I assume this is for new connections, as OVS kernel should be able to handle existing connections with cached state?

I assume this is for new connections, as OVS kernel should be able to handle existing connections with cached state?

Yes, without the PR, every time an agent restarts, before it connects to antrea-controller successfully, new connections would always be allowed regardless of what the policies define, even when the policy had been realized on this Node before.

/test-all


Smooth transition

Hey Crayon!
Great project you got. I am using it currently for an SPA like navigation in my Django App.
It all works really well (except for back/forth button for which I am still messing around with popstate and programmatic ajax calls - can't have it all I guess ;) unless I missed a solution here too?).
Anyway, I was wondering what your approach would be to make smooth slide out and slide in transitions.
I am currently using the ajax:before Event to slide out the content. When the content is loaded, the transition is automatically applied to slide in. Most of the time, the content is loaded before the transition finishes, sometimes the load takes longer and i'll have an empty page for a while. Do you have any suggestions? Can i somehow have more fine control?
Thanks, keep it up!

Hi Pj,
My first thought would be to use the View Transitions, that would be the most performant and easy to use, the only drawback is that it's not fully cross-browser (yet).
You could make a page transitions with some CSS like this:
@keyframes fade-in {
  from { opacity: 0; }
}

@keyframes fade-out {
  to { opacity: 0; }
}

@keyframes slide-from-right {
  from { transform: translateX(25%); }
}

@keyframes slide-to-left {
  to { transform: translateX(25%); }
}

/* define animations for the old and new content */
::view-transition-old(slide-fade) {
  animation: 200ms ease 150ms both fade-out, 200ms ease 150ms both slide-to-left;
}
::view-transition-new(slide-fade) {
  animation: 300ms ease 50ms both fade-in, 300ms ease 50ms both slide-from-right;
}

#page {
  view-transition-name: slide-fade;
}

and then add the transition modifier to x-merge:
<div id="page" x-merge.transition>
If that won't cut it, I'd probably fallback to using events like you're doing. I'd use ajax:before to animate the page away and then ajax:after to animate it back into view.
If that feels like too much custom code though, there's also https://swup.js.org it does page transitions really well.

I'm going to close this out as work related to this issue is now happening in #93 and #101.


fix: development script watch loop

Context & Description

The loop is now resolved. The dev task ends and remain in watch.
However there is another issue: when editing a file inside the apps/documentation folder the watch process doesn't recompile the frontend part. Everything work if I restart the dev task but I'm assuming this is not expected.
BTW thanks for the quick support!

That's weird. I wouldn't expect it to reload, since the documentation doesn't use the workspace library but the deployed one ("tuono": "0.12.0"). It should act exactly like a normal tuono app, so it should reload only when watched with tuono dev.