tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: cloud server
TensorFlow installed from (source or binary): source
TensorFlow version: 2.2.0
Python version: 3.7.7
Installed using virtualenv? pip? conda?: pip
Bazel version (if compiling from source): 2..0.0
GCC/Compiler version (if compiling from source): 7.5
CUDA/cuDNN version: CUDA 10.1 & cuDNN 7.6.5
GPU model and memory:
00:07.0 VGA compatible controller: NVIDIA Corporation Device 1eb8 (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 130e
Physical Slot: 7
Flags: bus master, fast devsel, latency 0, IRQ 37
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at fa000000 (64-bit, non-prefetchable) [size=32M]
I/O ports at c500 [size=128]
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

Describe the problem
Here is how i got the error
``Python 3.7.7 (default, May  7 2020, 21:25:33)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
2021-04-16 16:46:25.884063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-04-16 16:46:25.888707: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-16 16:46:25.889366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:07.0 name: GRID T4-4Q computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 3.97GiB deviceMemoryBandwidth: 298.08GiB/s
2021-04-16 16:46:25.889562: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-04-16 16:46:25.891366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-04-16 16:46:25.893293: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-04-16 16:46:25.893593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-04-16 16:46:25.895611: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-04-16 16:46:25.896780: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-04-16 16:46:25.901308: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-16 16:46:25.901433: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-16 16:46:25.902118: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-16 16:46:25.902701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-04-16 16:46:25.903031: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-04-16 16:46:25.910663: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency:<PHONE_NUMBER> Hz
2021-04-16 16:46:25.910940: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa728000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-16 16:46:25.910965: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-04-16 16:46:25.967935: W tensorflow/compiler/xla/service/platform_util.cc:210] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_UNKNOWN: unknown error
2021-04-16 16:46:25.968035: I tensorflow/compiler/jit/xla_gpu_device.cc:161] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA
2021-04-16 16:46:25.968258: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-16 16:46:25.968912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:07.0 name: GRID T4-4Q computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 3.97GiB deviceMemoryBandwidth: 298.08GiB/s
2021-04-16 16:46:25.968993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-04-16 16:46:25.969017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-04-16 16:46:25.969042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-04-16 16:46:25.969066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-04-16 16:46:25.969089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-04-16 16:46:25.969113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-04-16 16:46:25.969140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-04-16 16:46:25.969237: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-16 16:46:25.969898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-04-16 16:46:25.970481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-04-16 16:46:25.970546: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
File "", line 1, in
File "/home/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 262, in constant
allow_broadcast=True)
File "/home/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File "/home/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 95, in convert_to_eager_tensor
ctx.ensure_initialized()
File "/home/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/python/eager/context.py", line 515, in ensure_initialized
context_handle = pywrap_tfe.TFE_NewContext(opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable
Provide the exact sequence of commands / steps that you executed before running into the problem

Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Indeed, before this error, I had another error:
Could not load dynamic library 'libcublas.so.10'; dlerror: libcublasLt.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.1/lib64:
I didn't know why, but libcublas.so.10 and libcublasLt.so.10 were in /usr/local/cuda-10.2/targets/x86_64-linux/lib instead of /usr/local/cuda-10.1/lib64, so I copied them both to /usr/local/cuda-10.1/lib64 and it solved the problem. Would there be any connections between these two errors? If no, any help would be appreciated !

@Leiga
this seems Nvidia related error, please refer to these links and let us know if it helps:
Link, link1,link2,link3
Also please confirm if you have referred to these tested build configurations.
You may also try:
Restart the program with os.environ["CUDA_VISIBLE_DEVICES"] = '0'

@Leiga Let's try limiting gpu memory growth and see if it helps. Before that please make sure you close all python sessions and exit python interpreter, we want to make sure your gpu is not utilized any simulations.
# on the top of your script
import tensorflow as tf
gpus = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
# Paste rest of your code here

@Leiga Let's try limiting gpu memory growth and see if it helps. Before that please make sure you close all python sessions and exit python interpreter, we want to make sure your gpu is not utilized any simulations.
# on the top of your script
import tensorflow as tf
gpus = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)
# Paste rest of your code here

I have tried putting those on top of my test script and run it. Unfortunately, It produced the same error as mentioned above.

@Leiga
this seems Nvidia related error, please refer to these links and let us know if it helps:
Link, link1,link2,link3
Also please confirm if you have referred to these tested build configurations.
You may also try:
Restart the program with os.environ["CUDA_VISIBLE_DEVICES"] = '0'

I have tried using os.environ["CUDA_VISIBLE_DEVICES"] = '0'  but it did not solve the problem.
After browsing this link you recommended, I found that I had an issue with the Recommended Actions under Post-Installation Actions from Nvidia docs
I tried running /usr/bin/nvidia-persistenced --verbose
but it raised an unexpected error:
nvidia-persistenced failed to initialize. Check syslog for more details.
And I had a trouble running ./deviceQuery. Would those cause the problem?

Perhaps this thread can help you fix

nvidia-persistenced failed to initialize. Check syslog for more details.

@Leiga
Same error is reported and solved at Nvidia webiste shared below, please follow it and move this issue to closed status as it is not a tensorflow related issue, in case of any further questions related to same issue you may open a ticket with Nvidia.
https://forums.developer.nvidia.com/t/nvidia-persistenced-failed-to-initialize-check-syslog-for-more-details/74052

@Leiga
Same error is reported and solved at Nvidia webiste shared below, please follow it and move this issue to closed status as it is not a tensorflow related issue, in case of any further questions related to same issue you may open a ticket with Nvidia.
https://forums.developer.nvidia.com/t/nvidia-persistenced-failed-to-initialize-check-syslog-for-more-details/74052

Are you sure the error nvidia-persistenced failed to initialize. Check syslog for more details. is the main cause of tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable ? I have checked my nvidia-persistenced status with systemctl status nvidia-persistenced and it was active.


Synchronization

It was clear from the beginning that Isar should support some kind of synchronization capability. Until now I wanted to implement something similar to dexie's sync strategy which provides eventual consistency and requires the users to resolve conflicts themselves. It also can lead to some form of data loss in the event of a conflict.
But it would be really cool if we could design a system that guarantees strong eventual consistency with automatic and deterministic conflict resolution.
Over the past two days, I studied countless papers on the topic and there are multiple ways to achieve this goal. One very promising paper is A Conflict-Free Replicated JSON Datatype.
When the database only contains conflict-free types ("CRDT"s), it would not only be possible to implement synchronization to a server but also using peer to peer connections or a decentralized network, for example, a Bluetooth connection because there is no need for resolving conflicts in a centralized way.
I would love to hear your opinion, ideas, and if possible further writings on the topic.
Since using CRDTs implies very strict rules and detailed tracking of changes, this decision has a great impact on the future development of Isar Core.
Edit:
Naturally with every upside comes a downside. In this case the downside of implementing this proposal it that there will be a small performance impact event when synchronization is disabled. The reason is that we cannot enforce a strict schema any more because different peers may have different versions of the database. This requires using something like Flatbuffer's Flexbuffers to support dynamic schemas.

Also a very interesting approach is the Gun database.

Speaking as a (prospective) user, I think letting users handle sync conflicts is probably the best idea.
I'll read that paper that you posted and come back with further thoughts.

It's also worth considering the options for future development that CRDTs would provide.
Given that a large percentage of Hive's current users use it as a persistent cache, and that is also the suggested usage of LMDB and IndexedDB, interoperability with foreign databases is high on the list of priorities.
We could also write integrations (or have third-party integrations) into compatible databases. Mongo, Redis, and more support the CRDT style architecture.

@AKushWarrior Yes, I agree. Strong eventual consistency would allow us to provide "batteries included" integrations without user interference or manual conflict resolution.

On the other hand, this architecture may introduce problems if the database is not compatible with CRDTs. Given that this is kind of a weedy implementation detail that's not clearly evident to the end user, CRDT might hurt Isar's goal of being user and platform agnostic.

I have a HLC + CRDT implementation in dart here: https://github.com/cachapa/crdt
Though I suspect that you probably want to implement it in Rust, it might still be interesting as a reference.
Please note that the current implementation does not include node IDs. I'm working on that on branch 2.0.0 - they're relevant in order to perform deterministic conflict resolution.

this architecture may introduce problems if the database is not compatible with CRDT

I think we will have to implement the database integration anyway since there is no standardized protocol to distribute changes across the network. It should be possible to do that with almost any backend database.

I have a HLC + CRDT implementation in dart

Very cool! I'll definitely take a look :+1:

I think we will have to implement the database integration anyway since there is no standardized protocol to distribute changes across the network.

Unfortunately true.
On another note, we should look into rust implementations of CRDTs.

Sled has CDRT support. Might be useful to see it working in a database environment.
There's also a whole bunch of crates for working with CDRT classes.

Great talk on the topic.

Looks interesting. I'll watch it...

Finished the talk. I wonder if those papers in the references have pseudocode we can use?

The paper I linked before has a reference implementation and a port to rust.
But we will probably need to come up with a mix of existing solutions. The automerge algorithm for example keeps the entire change history which is not suitable for mobile devices with limited resources.

The first thing i want to say is to know what your objective is. You can do build very different systems with CRDT.
For example an oldie but an old one is Roshi ( https://github.com/soundcloud/roshi ), which uses CRDT Server side as a WAL ( Write Head Log).
A recent one based on IPFS is https://github.com/StreamSpace/ants-db.
See here: https://github.com/StreamSpace/ants-db/blob/master/go.mod#L10
It is attempting to create a WAL also. So that you can have many of these stores and they are synchonise and each be eventually consistent using CRDT.
Once you have a DB with a WAL, then you can do CRDT in general with HIVE for example.
The Mutations go into the DB WAL, then get transacted against your local DB to make a KV Store from which you can do queries and you can easily push the change event to the Flutter GUI.  You need this type of Kappa style architecture because a CRDT mutation can result in many buckets in your KV store mutating once you build real work apps.
Then you broadcast the data in your WAL out to all Peers ( or Server ), and visa Versa. The Visa Versa bit is often the crux. Often its better to have a relay Server that is a store and forward Public IP where the CRDT WAL mutations can be shared between client. This is because for Group chats or group work on a document you need a relay point. Signal for example does the same for Group chat. For One to One chat it uses Peer to Peer  to do One to One sync of the CRDT mutations.
SO from above you can maybe see that there  are many ways of building a CRDT based system.
Martin Kleppmann is the main person behind Automerge (https://github.com/automerge/automerge-rs). He is at Cambridge and has been workng on it for maybe 15 years.
The JS version that works well is called yjs. https://github.com/yjs/yjs
https://github.com/dmonad

He is in Berlin
The examples work very well and show off how powerful CRDT can be in the right scenarioes.

There is also Yorkie: https://github.com/yorkie-team/yorkie

golang and mongo DB with CRDT based synchronisation
It has a Server that acts as a relay point and a source of truth with a mongo DB
The client is written in JS. Its easy to make that work for Flutter by porting the actual logic which is here:  https://github.com/yorkie-team/yorkie/tree/master/pkg/document
It is however far better i feel to just cross compile that Document code to WASM and C ad then use FFI or Protocolbuffers to reuse it in flutter. Why ? Because you then have one code base that is your CRDT engine.

Then you have to decide a Transport if you want to do Client / Server Sync and Client / Clients sync.
For Client / Client most use WebRTC, partly because you also get the Stun and TURN aspects to find the Peers, and so you need some servers anyway.  See alot use https://github.com/pion/ion
I have been working on this and ended up going down the path of not using Dart. Why ?

CRDT is hard and the maths and logic is very complex and difficult to getright.
Your going to need the CRDT Logic is exist on Clients but as well as Servers.
You need network transport libs that dont really exist in the dart world.

SO, i have been working on taking golang code and cross compiling it to work with Flutter Web, Desktop and Mobile.
Many others also take this approach.
This is a great example that works well and is something you need anyway if you want to do Peer to Peer CRDT, because it gives you a local Identity system: https://github.com/jerson/flutter-openpgp

It compiles the golang to WASM for flutter Web
It compile the golang to c for flutter Mobile and Desktop.

For the Client and Server i use GRPC and GRPC WEB, and so does Yorkie. This works very well with Hive too since hive also has Protobuf Schema Evolution dues to the Fields having numbers. you need this for a real world app because you will have clients that are 1 or 2 versions behind the latest code form the Server or another client.
There is a CRDT implementation that uses Protobuf called Cloud State, where CRDT aspects are formalised into the Protocol buffers which is really the way to go IMHO.
https://cloudstate.io/docs/go/current/index.html
https://cloudstate.io/docs/go/current/gettingstarted.html
https://github.com/cloudstateio
DART: https://github.com/cloudstateio/dart-support
GOLANG: https://github.com/cloudstateio/go-support
Whats really going on is that they have a Event Sourcing ( https://github.com/cloudstateio/go-support/blob/master/cloudstate/eventsourced.go) kappa based Architecture, and the Server is CRDT away and so mutations can be building up on the Server for the clients to use. Its basically the same as Yorkie in this way.
Here is the Server Entry point: https://github.com/cloudstateio/cloudstate
However you need a big Server based on Akka and Java.
I think that can easily be replaced however with golang because what the Server does is actually not that complex.

Hey guys,
recently I was exploring how CRDT can be utilized to build a sync layer. So I did a university project written in Dart. The main purpose was to get some insides of how CRDT works etc and I still have a lot to learn :): If you are interested what I did here is the project:  https://github.com/Manuelbaun/sync_layer_crdt_playground
I used some form of delta state CRDT with HLC and basically used them as events when any updates occurred. Updates looked like this in pseudocode:
class Atom {
    ID, //  siteID, hybrid logical clock
    type, // like the table or object
    ObjectID,
    data, // any form of data usually a kv map
}

To emulate tables I used a "CRDT Map" (Grow only), where the key represents the column. Note that my implementation does not support to remove keys from a CRDT map.
I was watching the talk from James Long and that is who I designed my CRDT sync layer after. I used a Merkel trie to diff two nodes, and then send all events that happened after that time to the other nodes, so they sync.
What I did not look into, how garbage collection could work. Still on my list though. And there are a lot of things to do, to make a sync layer actually work well.

Great talk on the topic.

The talk from Martin Kleppmann was very informative, how automerge achieved that great compression for the text editing.
For another project I used yjs with WebRTC (because at the time it had the best performance) to make a simple P2P drawing game. While WebRTC is great in theory, it is not working everywhere. On my university campus, I couldn't make it run. So peers would not find each other. But it worked perfectly at home.

THis discussion is really good !!

Hey guys,
recently I was exploring how CRDT can be utilized to build a sync layer. So I did a university project written in Dart. The main purpose was to get some insides of how CRDT works etc and I still have a lot to learn :): If you are interested what I did here is the project: https://github.com/Manuelbaun/sync_layer_crdt_playground
I used some form of delta state CRDT with HLC and basically used them as events when any updates occurred. Updates looked like this in pseudocode:
class Atom {
    ID, //  siteID, hybrid logical clock
    type, // like the table or object
    ObjectID,
    data, // any form of data usually a kv map
}

To emulate tables I used a "CRDT Map" (Grow only), where the key represents the column. Note that my implementation does not support to remove keys from a CRDT map.
I was watching the talk from James Long and that is who I designed my CRDT sync layer after. I used a Merkel trie to diff two nodes, and then send all events that happened after that time to the other nodes, so they sync.
What I did not look into, how garbage collection could work. Still on my list though. And there are a lot of things to do, to make a sync layer actually work well.

Great talk on the topic.

The talk from Martin Kleppmann was very informative, how automerge achieved that great compression for the text editing.
For another project I used yjs with WebRTC (because at the time it had the best performance) to make a simple P2P drawing game. While WebRTC is great in theory, it is not working everywhere. On my university campus, I couldn't make it run. So peers would not find each other. But it worked perfectly at home.

This is normal. WebRTC will not work if either party is behind a Symmetric Router. See: https://superuser.com/questions/1525664/how-to-fix-symmetric-nat-router-router
Most Governments runs Cisco stuff and its Symmetric. i see this all the time.
The solution used by Google and everyone is to have a Relay Proxy for when one party is behind a Symmetric Router.
This a standard part of a WebRTC Setup and called a SFU ( Selective Forwarding Unit )
This works well: https://github.com/pion/ion-sfu
https://webrtcglossary.com/sfu/

@joe-getcouragenow Thanks for all the resources and topics to think about. I'll take me some time to explore all of them and properly respond.

Once you have a DB with a WAL, then you can do CRDT in general

Why does the database need a WAL? Isn't any kind of transaction mechanism sufficient?

You need this type of Kappa style architecture because a CRDT mutation can result in many buckets in your KV store mutating once you build real work apps

LMDB is incredibly fast and should be able to handle the log.

Often it is better to have a relay Server

Yes I agree but I still hope that we can create a system that can rely on pure p2p if necessary. These networks will very small for example a local WiFi or a Bluetooth mesh network.

I have been working on this and ended up going down the path of not using Dart

I agree but the problem with WASM is that is can be VERY slow when heavy interaction with JavaScript or in our case cross-compiled Dart is required. WASM, for example, cannot directly access IndexedDB. There are multiple proposals to solve this problem but it will take years until these solutions will be implemented in the majority of browsers.

Garbage collection being one of the big ones.

This is exactly the problem I'm currently trying to find a solution that works with mobile devices and their limited resources.

You do not have to use Dart FFI, and neither shoudl you because there are many things in Dart FFI that do not work so well.

How else could we communicate with the Rust backend?

FLutter Method Channel Binding

They are SLOOOW :(

@Manuelbaun Thanks, this looks very interesting. We'll probably still end up implementing it in Rust and JS respectively.

@joe-getcouragenow Thanks for all the resources and topics to think about. I'll take me some time to explore all of them and properly respond.

Once you have a DB with a WAL, then you can do CRDT in general

Why does the database need a WAL? Isn't any kind of transaction mechanism sufficient?

Well it depends what your building. In the README there is not actual definitive statement of the thing your building so i had to guess a bit.
SO sometimes, on the Server you want a WAL that then takes the Event Stream and materialises your KV Tables ready for non CRDT usrs to be able to do a RPC query on that. Its CDC ( Chaneg Data Capture) but in reverse. SO its essentially KAPPA.
On the Client in a P2P situation you want the same because somewhere with Event Streaming you want a place where all mutations come in before the CQRS code takes over to do something with that event.
This is what signal and Cloudstate and others do. Not everyone does this though. It really depends. I think that not doing it is asking for trouble. Look at any DB and the first thing is does is put all mutations into a WAL so that if the power goes, and then you restart the device the WAL is there and the DB can play back the last few writes into the DB.

You need this type of Kappa style architecture because a CRDT mutation can result in many buckets in your KV store mutating once you build real work apps

LMDB is incredibly fast and should be able to handle the log.

Often it is better to have a relay Server

Yes I agree but I still hope that we can create a system that can rely on pure p2p if necessary. These networks will very small for example a local WiFi or a Bluetooth mesh network. If internet connection is available, a server or cluster can be used.

Ah so what Berty are doing basically. https://github.com/berty/go-ipfs-log
They use Bluetooth and local WIFI. Telegram also does this.

I have been working on this and ended up going down the path of not using Dart

I agree but the problem with WASM is that is can be VERY slow when heavy interaction with JavaScript or in our case cross-compiled Dart is required. WASM, for example, cannot directly access IndexedDB.

For me WASM is screaming fast, when you not touching the DOM which i dont. You can use a Service worker an access the IndexDB.
See: https://developers.google.com/web/ilt/pwa/live-data-in-the-service-worker
Numerous examples on the web....

"The raw execution of an algorithm in WASM is almost always faster than in JavaScript. However, the cost of writing data into the WASM module’s memory can be so high that it removes the benefit of using WASM in the first place."

Can you let me knwo where you got that quote from ? Often its a matter of Context, and am curious of their context.

There are multiple proposals to solve this problem but it will take years until these solutions will be implemented in the majority of browsers. Especially with Apple taking forever to implement new WASM features in Safari.

YEAH Safari is the one holding it back a bit. Microsoft using Chrome as a god send, and Firefox is pretty close to Chrome on Supporting PWA and WASM. WASM does work with Safari on Desktop and IOS now.  I honestly have no tested it that much though.
https://caniuse.com/#feat=wasm

Garbage collection being one of the big ones.

This is exactly the problem I'm currently trying to find a solution that works with mobile devices and their limited resources.

You do not have to use Dart FFI, and neither shoudl you because there are many things in Dart FFI that do not work so well.

How else could we communicate with the Rust backend? I hope that most of the very bad problems like finalizers dart-lang/sdk#35770 and copy-free Strings dart-lang/sdk#39787 will be solved until Isar is released.

FLutter Method Channel Binding

They are SLOOOW :(

Yeah i agree Method Channel is slow. But when you do the tradeoffs you have a few things to consider:

How often are calls between the flutter and native layer happening ? Are you using Hive as a Cache of Materialsied View ? your could.
Developer Productivity
You will need to make network calls ( p2P and Client Server) and Protobufs will make that easy and fast.

https://github.com/google/note-maps/blob/main/note_maps/lib/mobileapi/repository.dart#L207
They have a 100% golang based embedded layer with a golang DB and Protobufs.
The use Protobufs as their transport FLutter <--> Embedded, Embedded <--> Server.
Now when you are later doing P2P connections your are going to want to be using Protobufs also i posit.
So the protobufs are used fo the 3 reasons protobufs are always used. Code Gen, Speed ( runtime and dev time) and Evolution ( not breaking against version).
FFI are faster and maybe you can use them with Protobufs ? Link a PB <--> FFI marshaller.
Hard to say as this is getting very specific. I bet someone has looked into it :)
https://github.com/ajrcarey/dart-rust-minimal-ffi-grpc-example

Can you let me knwo where you got that quote from ? Often its a matter of Context, and am curious of their context.

https://blog.sqreen.com/webassembly-performance/
The context is communication with JavaScript or any Browser API. WASM cannot even do network calls without sending the data through JavaScript afaik.

You will need to make network calls ( p2P and Client Server) and Protobufs will make that easy and fast

But network calls will be made by Rust anyway. It is not important whether Dart communicates via Dart > Rust or Dart > Java > Rust.

FFI are faster and maybe you can use them with Protobufs

Protobuf /  Cap'n Proto etc. can be used by Rust anyway.

@Manuelbaun Thanks, this looks very interesting. We'll probably still end up implementing it in Rust and JS respectively.

yeah, I also would not recommend to implement it in Dart. It was ok for a prototype. Before I started to implement it, I was thinking of doing it in Rust, since there are already some CRDTs implementations to use.
I regard to the garbage collection of crdt:
the author of yjs, Kevin Jahns, did implement a snapshot feature. I have not looked into it, how he did it, but you can see it on his website https://yjs.dev/ under demos in action. This might be a way to garbage collect, the log of mutations on the database.

I think something that we should consider is forking automerge and seeing how integrated/easy to remove the change history is. As far as I can tell, it seems to be the foremost technology in the field of CRDTs.
I have no comment on the architectural (rust ffi vs WASM) stuff yet, I need to read up a bit.

seeing how integrated/easy to remove the change history is

Unfortunately this is more of a conceptual problem then a coding problem. Automerge and most other CRDTs use the log to guarantee divergence in the case of network partition (e.g. a peer goes offline and commits concurrent changes)

Automerge and most other CRDTs use the log to guarantee divergence in the case of network partition (e.g. a peer goes offline and commits concurrent changes)

Maybe I'm missing something here. Why would we want to "guarantee divergence"? Isn't the whole point to reconverge when the peer goes back online?

Isn't the whole point to reconverge when the peer goes back online?

Sorry, avoid divergence, guarantee convergence :D

Sure, but there's no need for a permanent log. Just a log of every change since the last convergence event, right?

If your app only communicates with a server to sync, then I think it is ok to only persist the logs since the last sync, and after successful sync, you can delete those logs.
It gets a bit more tricky if you want to support a P2P setup. If several nodes, let's say A, B, C mutate data offline and then node A syncs with Node B. Node A and B are in the same state now. If they would now delete the logs of every change and Node C comes and want to merge.  Node C sends his log to A and B, and they can apply the changes from C and how would you sync now Node C? You can't get any logs from A or B since they deleted those logs already. The only way I can think of right now is to send the whole state, form either A or B to C, and basically override your local state. There are probably other ways to solve this too.
So the question is, when is it ok to delete the logs? If you can guarantee that all nodes in a system received the logs of changes, then those logs can be deleted.

So the question is, when is it ok to delete the logs? If you can guarantee that all nodes in a system received the logs of changes, then those logs can be deleted.

Maybe a good compromise would be to introduce a parameter Duration keepHistoryFor which could be either null for I define storage, 0 for immediate deletion after sync to server or something else for p2p. The developer probably knows the situation best and if there is a good explanation in the docs, it could work well...

As long as it's very clear that changes will be wiped after that duration, I agree.
It's important that the default is keep the changes indefinitely though, to avoid unpredictable behavior when syncing.

Okay, I think we have a (general) plan.
TODO:

Fork https://github.com/automerge/automerge/tree/performance and https://github.com/automerge/automerge-rs

Support change history wiping as discussed above

Provide hooks for sync using the forked libraries

We will have to do this in the Dart layer, not in the native/web layers
Automerge-rs has only reached FFI maturity, meaning that we can probably use Dart FFI to reach it but can't use it from the Rust layer.
Should still work AFAIK, because we can also FFI into automerge-performance using https://pub.dev/packages/js

Almost complete collection of research on CRDTs

Almost complete collection of research on CRDTs

Very cool. I'm reading through the papers now...

@leisim done reading through them. I haven't really been enlightened or anything, but I might be able to parameterize this issue better now. As I can see it, there is a few use cases:

Merge an Isar-Web db with an Isar-Native db
Merge an Isar-Web db with an Isar-Web db
Merge an Isar-Native db with an Isar-Native db
Merge an Isar-Web/Native db with arbitrary foreign db

The last one is by far the most code-heavy: we'll need to write drivers for any foreign db we want to sync with, or expose hooks for users to do it themselves. The other three will be a matter of porting a CRDT library to the platforms we're using, and editing to our needs.

@leisim thoughts here? I'd be happy to start writing a port of automerge, in either Dart or Rust.

Hi,
I read this topic,
And I have a syncing experience for the mobile app that I developed as Offline-First Mobile .
Server side db is PostgreSQL  and client side db is Realm-DB
Actually Realm support syncing itself by realm-object-server ,
But I would like to have PostgreSQL as back end on the server so I implement my approach for that.
Roles:

There are only two commands on the client for syncing data :  pull and push.

There are only two api on the server for syncing data /pull/ and /push/.

Models in client and server have only three extra fields:

DateTime lastUpdate;
Boolean synced;
Boolean deleted;

It always runs first pull and then push.
The pull always get records after lastUpdate.
If pull failed,  It stop and don't continue to push.
The pull only apply changes on the records that are currently synced in the client.
The push only push  out of synced records.
The push  always need to confirm before applying synced.
Client will delete a record while record is both of deleted and synced.
Conflict resolve on the server just by prefer new records by lastUpdate.

Thanks to everybody that is doing efforts for this new thrilling project.
@omidraha I've done a similar Sync Engine with completely personalized back-end and front-end. My solution is a little bit more comprehensive, but also more complicated and counterintuitive (because of the nature of the system), so I will continue the discussion with your solution.
I have a few questions though:

How does the backend know what to transmit on pull? Does the backend transmit the entire DB on pull and the client applies only the synced data? Would be an enormous waste of syncing time/bandwidth/client space/client activity/server activity.
Said that, what about historical data? Should the sync engine delete from client the cached data after a few times that is not queried (so both read/write)?
What about conflicts in the same row? Could be that multiple offline users change the same row in multiple different columns. So, when syncing, the changed columns of those multiple rows should be kept and merged together. Should the merge action be done in the server instead that on the client?

I know that the target is doing the synchronization in the client-side, but maybe is not possible having a reliable Syncronization without an adaptor in the back-end. Do AWS AppSync works for the need?
Just for knowledge, an analysis of Firestore synchronization could be useful.

I have a few questions though:

How does the backend know what to transmit on pull? Does the backend transmit the entire DB on pull and the client applies only the synced data? Would be an enormous waste of syncing time/bandwidth/client space/client activity/server activity.

We would record the changes to the database, and transmit the change history.
The change history, in our model, can optionally represent only the changes in the last (user-defined period of time). This helps limit the amount of data transferred over networks.

Said that, what about historical data? Should the sync engine delete from client the cached data after some time that is not queried (so both read/write)?

Not an architectural decision that should be made by a client side database. The user is free to implement this functionality if they wish.

What about conflicts in the same row? Could be that multiple offline users change the same row in multiple different columns. So, when syncing, the changed columns of those multiple rows should be kept and merged together. Should the merge action be done in the server instead that on the client?

This database will support direct P2P merging. There doesn't have to be a "central" database. For more, see operational transactions vs CRDTs. Every client will have an algorithm that can merge two conflicting rows in a symmetric fashion.

I know that the target is doing the synchronization in the client-side, but maybe is not possible having a reliable Syncronization without an adaptor in the back-end. Do AWS AppSync works for the need?

We plan on writing adaptors for various backends.

Just for knowledge, an analysis of Firestore synchronization could be useful.

Agreed!

We would record the changes to the database, and transmit the change history.
The change history, in our model, can optionally represent only the changes in the last (user-defined period of time). This helps limit the amount of data transferred over networks.

OK So, I know you plan to transfer only the changes but... I mean another thing... Let me be more clear:
What about the data that are unuseful for the clients? This still could be a problem with projects with very complicated business domains: Let's say that there are tables with 10+ millions of records, with thousands of changes every minute... would the sync engine still sync ALL the changes to keep eventual consistency? Or would the sync engine sync only the changes related to the client used data/queries instead (like Firestore does)?
If all the changes are synchronized, even if you only apply the changes on the used data/query the entire 10+ million record table's changes are still transferred over the network (with possibly a security issue when constantly receiving sensible data not related to the user, even if not used in the client-side), and processed for applying the changes with consequently battery drain, data usage, performance issues.

Not an architectural decision that should be made by a client-side database. The user is free to implement this functionality if they wish.

Sure... but I think this could still be a "native feature" of the DB (again, as Firestore), as in a offline-first mobile app storage usage is still one of the main concerns

This database will support direct P2P merging. There doesn't have to be a "central" database. For more, see operational transactions vs CRDTs. Every client will have an algorithm that can merge two conflicting rows in a symmetric fashion.

Wow! I didn't go deep in the topic, I will have a more detailed look soon

We plan on writing adaptors for various backends.

Great, maybe also something like an SDK to let someone to build his own solution with legacy back-end could be useful.
I've used Firestore as example too many times! 😄 But I think the team has done a great job optimizing synchronization especially for low-end devices
Thank you very much

@leisim thoughts here?

I'm still experimenting with different solutions... I have no definitive ideas yet.

In my opinion, it would be great if this feature be something plugin-able, so that the server side can be written for any language and database, for example for Python/Django with PostgreSQL, and we have plugins for each of them.

I completely agree.

How does the backend know what to transmit on pull? Does the backend transmit the entire DB on pull and the client applies only the synced data?

This could for example be solved using vector clocks so the client can tell the backend its last state and the backend can only send the changes.

Should the sync engine delete from client the cached data after some time that is not queried

I don't think this should be part of the implementation but rather be handled by the user. Maybe we can add the option for a TTL for entries?

Should the merge action be done in the server instead that on the client?

With strong eventual consistency, there should be no need for a server that resolves the conflicts. Conflicts will be resolved on a property level. An interesting question is how to guarantee inter-property consistency.

What about the data that are unuseful for the clients?

I suggest some kind of pub sub functionality. This would also allow easy access control.

@leisim I agree that pub-sub is a cool (and probably necessary) addition here. I'd suggest another term: selective sync.
My reasoning is that pub-sub has stream processing implications, which many people have tried (and failed) to implement efficiently on top of relational databases. We aren't really providing data streams to subscribers; instead, we are allowing CRDT merging on some attributes instead of the whole table. Thus, a client isn't really "subscribing" to anything, because the client still has to initiate one-off synchronization events.
Obviously, this is kind of semantic, but might be important for future documentation purposes.

I'd suggest another term: selective sync.

Could you please briefly explain what you mean by this? Are there resources on this topic?

I'd suggest another term: selective sync.

Could you please briefly explain what you mean by this? Are there resources on this topic?

I totally made the term up, as an alternative to pub-sub. In my mind, what this looks like is:

the user passes a where clause
The where clause is run on both the foreign and local database; the resultant of both are merged using CRDT.
The merge then is written over the correspondent rows in the local database (which have been temporarily earmarked by the original where clause)

This is a pretty simple approach, and will probably fail in some more complex cases (relationships). It's also inefficient if we have a highly columnar database (where individual rows contain a lot of data).
I don't know if there's literature on the topic; my argument was more semantic in that we should call whatever we do selective sync instead of pub-sub.

Hi this not a place for such a comment, but I guess you can delete this after reading as unrelated.  I am a medical teacher and newbie to programming started with flutter. Excited to see the solution to some of my problems
You are solving a genuine problem. So far I have not found a query efficient database in a flutter. Medical student's use-case is searching through a large textual database, even with embedded indexing sometimes it takes too long to search through the medical database.
And looks like you are also solving another problem of "connectivity", it is a big issue in our college. Slide sync esp. like sli.do don't work well, maybe because of dependency of the central server, P2P might be the solution I hope.
Also P2P sync, this helps much more opening new scenarios for peer gamification of study content
I think you are solving a genuine problem, Do you have a todo list page, so all actionable pieces are in one place and someone from outside can recognize the need at the point of time and try to be a part.
I wish I could contribute to this project, I will try my best, I hope with my non-technical background I can do something at least.

Hi this not a place for such a comment, but I guess you can delete this after reading as unrelated.  I am a medical teacher and newbie to programming started with flutter. Excited to see the solution to some of my problems
You are solving a genuine problem. So far I have not found a query efficient database in a flutter. Medical student's use-case is searching through a large textual database, even with embedded indexing sometimes it takes too long to search through the medical database.
And looks like you are also solving another problem of "connectivity", it is a big issue in our college. Slide sync esp. like sli.do don't work well, maybe because of dependency of the central server, P2P might be the solution I hope.
Also P2P sync, this helps much more opening new scenarios for peer gamification of study content
I think you are solving a genuine problem, Do you have a todo list page, so all actionable pieces are in one place and someone from outside can recognize the need at the point of time and try to be a part.
I wish I could contribute to this project, I will try my best, I hope with my non-technical background I can do something at least.

Thanks for the kind words! Until the underlying P2P architecture is solved, the database itself can't be written, so this project is on hold until a plan is decided regarding this issue. There is a TODO.md in this project, which you can look at.

I think having abstractions for synchronization would be wonderful. GRPC, GraphQL, Websockets, Swagger, and more.

@listepo I agree 100% but it's hard to find an API that's flexible enough and not too complicated. Do you have a suggestion?

@leisim I have a couple of ideas to think about, I hope there will be something to discuss

@leisim I would like to have relationships before going into sync detail.

@leisim what do you think about https://nozbe.github.io/WatermelonDB/Advanced/Sync.html ?

@listepo Sorry I must've missed your previous posts.

I have a couple of ideas to think about, I hope there will be something to discuss

Great let's definitely do that. I need input and ideas because I want to make sync as universal as possible.

I would like to have isar relationships before going into sync detail.

Done :) IsarLinks are implemented

what do you think about https://nozbe.github.io/WatermelonDB/Advanced/Sync.html ?

That looks very interesting and I like the fact that developers can implement the actual sync themselves. We could provide examples for common backends. The only issue I see here is that conflict resolution is hard.

@leisim we can detect conflicts and provide an opportunity to resolve them by the method resolveConflicts. Manual resolved results or current, incoming, abort changes like merge conflicts:)

@leisim what do you think?

@listepo Could you maybe share a small code sample how that would work?

@leisim yes, I will try to find time this weekend

@leisim
// Part of Isar sync
class Resolver<T> {
  List<T> current;
  List<T> incoming;
  void acceptCurrent() {
    // Force push changes to the backend
  }
  void acceptIncoming() {
    // Force push changes to the db
  }
  void accept(List<T> data) {
    // For manually resolving
    // Force push changes to the db and backend
  }

  void abort() {
    // Abort synchronization
  }
}

// Implemented by the user and called when conflicts detected
void resolveConflicts(Resolver resolver) {
  resolver.acceptIncoming();
  // or show ui for user
}

link with models
syncAdapter.addResolver([User, City], resolveConflicts);

Do you know what method Telegram uses for syncing data between clients and cloud?
Its knowledge may be useful.

https://jepsen.io/consistency/models/causal
https://doc.replicache.dev/design
Replicache handles it in a server-client environment. I know you're trying to achieve peer-to-peer bidirectional sync... But having a look to the implementation (and to the APIs) could bring new ideas to how to handle this complexity ;)

Hi,
I read this topic,
And I have a syncing experience for the mobile app that I developed as Offline-First Mobile .
Server side db is PostgreSQL and client side db is Realm-DB
Actually Realm support syncing itself by realm-object-server ,
But I would like to have PostgreSQL as back end on the server so I implement my approach for that.
Roles:

There are only two commands on the client for syncing data :  pull and push.
There are only two api on the server for syncing data /pull/ and /push/.
Models in client and server have only three extra fields that are inherited for facilitation:

abstract class SyncModel
   DateTime lastUpdate;
   Boolean synced;
   Boolean deleted;

class User extends SyncModel
   String uid
   String name
   String email
   String mobile

It always runs first pull and then push.
The pull always get records after lastUpdate.
If pull failed,  It stop and don't continue to push.
The pull only apply changes on the records that are currently synced in the client.
The push only push  out of synced records.
The push  always need to confirm before applying synced.
Client will delete a record while record is both of deleted and synced.
Conflict resolve on the server just by prefer new records by lastUpdate.
The pull and push are atomic and If one of them failed finally, transaction will rollback.

In my opinion, it would be great if this feature be something plugin-able, so that the server side can be written for any language and database, for example for Python/Django with PostgreSQL, and we have plugins for each of them, for example:
isar-flutter-sync
isar-django-sync

Funny, that's exactly what I plan. The DB will be propably isar, because it has date filter as I required, the sync will be done on websocket and the other clients can get real time the new data.
Is there anyone who would like to join? The backend is with django channels, with redis/ or with google pubsub. An experienced flutter developer for the frontend stuff would be welcomed, since I am a backend heavy developer, I am not sure I can implement with best practices the frontend code by myself.
The frontend (can be named as isar-flutter-sync) will based on the https://github.com/cachapa/crdt package, but it's not perfect for my needs, so need to rethink a bit.

Btw, the class inheritence would work as the above example?

@leisim Is there any workaround for class inheritance? Is there plan for creating BaseCollection decorator? Currently, an abstract class' fields is not appear in the generated file. The links could be a workaround maybe, but seems impractical

Wanted to add my comments, I like this initiative, few things I wish to see a way of authenticating data that lands to a user device something like https://docs.couchbase.com/sync-gateway/current/channels.html to be in mind in the first place, also I don't know but would be nice for sync to be open-source too!

Wanted to add my comments, I like this initiative, few things I wish to see a way of authenticating data that lands to a user device something like https://docs.couchbase.com/sync-gateway/current/channels.html to be in mind in the first place, also I don't know but would be nice for sync to be open-source too!

This like to say that specific date will be pulled by specific users, not all users etc...

https://www.youtube.com/watch?v=B5NULPSiOGw

Check this https://condensation.io/

Also is open source https://github.com/CondensationDS/Condensation

I like that it has Peer-to-peer synchronisation which I believe in my opinion adding to isar will add more value than the competitor.

There are multiple proposals to solve this problem but it will take years until these solutions will be implemented in the majority of browsers. Especially with Apple taking forever to implement new WASM features in Safari.

Just noticed that Isar fails to load when opening my Flutter Web site in Safari with
Can't find variable: BroadcastChannel
... before opening a new issue- is this expected and related to the lacking WASM features mentioned above?

Would like to suggest sync to have some sort of routing data something similar to https://docs.couchbase.com/sync-gateway/current/channels.html This is important because of the following reasons

Managing access to data to multiple users
Control who can access what
Enable users to access just the data they need
Minimize the amount of data synced to mobile devices

@leisim an update on this?

https://wiki.nikitavoloboev.xyz/distributed-systems/crdt
https://github.com/alangibson/awesome-crdt
More information about CRDT.

https://wiki.nikitavoloboev.xyz/distributed-systems/crdt
https://github.com/alangibson/awesome-crdt
More information about CRDT.

I am really waiting for this, reading stories like https://www.figma.com/blog/rust-in-production-at-figma/ make me believe that this will be dope

I like the sync mechanism of WatermelonDB (which is a React/React native database) : https://nozbe.github.io/WatermelonDB/Advanced/Sync.html
Here are the implementation details : https://nozbe.github.io/WatermelonDB/Implementation/SyncImpl.html#sync-procedure

@ojm-it I also like it very much. The big question is how we handle schema changes. Do you have an idea?
How should we handle remote changes for example that contain a field that the local schema does not have yet 🤔

I'm not sure but I think the remote schema changes should be fully handled by the backend. The client should just pass the local lastPulledAt timestamp and the local schemaVersion with each pull/push calls, and let the endpoint adapt the response based on those.

@ojm-it I'm not fully caught up with this entire thread, but the original comment talks about peer to peer connections being possible. What happens in this case?

From the "General design" section :

master/replica - server is the source of truth, client has a full copy and syncs back to server (no peer-to-peer syncs)

two phase sync: first pull remote changes to local app, then push local changes to server

client resolves conflicts

content-based, not time-based conflict resolution

conflicts are resolved using per-column client-wins strategy: in conflict, server version is taken except for any column that was changed locally since last sync.

local app tracks its changes using a _status (synced/created/updated/deleted) field and _changes field (which specifies columns changed since last sync)

server only tracks timestamps (or version numbers) of every record, not specific changes

sync is performed for the entire database at once, not per-collection

eventual consistency (client and server are consistent at the moment of successful pull if no local changes need to be pushed)

non-blocking: local database writes (but not reads) are only momentarily locked when writing data but user can safely make new changes throughout the process

@ojm-it I'm not fully caught up with this entire thread, but the original comment talks about peer to peer connections being possible. What happens in this case?

From the "General design" section :

master/replica - server is the source of truth, client has a full copy and syncs back to server (no peer-to-peer syncs)
two phase sync: first pull remote changes to local app, then push local changes to server
client resolves conflicts
content-based, not time-based conflict resolution
conflicts are resolved using per-column client-wins strategy: in conflict, server version is taken except for any column that was changed locally since last sync.
local app tracks its changes using a _status (synced/created/updated/deleted) field and _changes field (which specifies columns changed since last sync)
server only tracks timestamps (or version numbers) of every record, not specific changes
sync is performed for the entire database at once, not per-collection
eventual consistency (client and server are consistent at the moment of successful pull if no local changes need to be pushed)
non-blocking: local database writes (but not reads) are only momentarily locked when writing data but user can safely make new changes throughout the process

Peer to peer I would say is important if it is possible

@richard457 The WatermelonDB sync implementation relies on the fact that there is a single source of truth which is the server. This greatly simplifies things like conflict resolution, specially when different clients can have different versions of the database.
I have never used WatermelonDB before (I don't know React), but I digged into the implementation details of their syncing system, and it seems that Drift has everything needed to implement a similar system.
Peer-to-peer is a completely different sync mechanism with a different set of challenges, so I guess it could be implemented separately.

Drift

can you elaborate more on Drift?

I personally think even though Peer-to-peer is complex can give us an edge if it is something we start with. in this case we can bit both https://www.couchbase.com/ and https://objectbox.io/ so I don't see this feature as just a future I see it as something that will give this dabase a competitive advantage.

Drift

can you elaborate more on Drift?

Sorry I meant Isar

I personally think even though Peer-to-peer is complex can give us an edge if it is something we start with. in this case we can beat both https://www.couchbase.com/ and https://objectbox.io/ so I don't see this feature as just a future I see it as something that will give this database a competitive advantage.

Objectbox is closed, I didn't get access in a year, but maybe because that is not per user sync (didn't get back to me). P2P is not that important, as a backend sync imo and honestly not even sure the DB needs a feature like this. Sure it's good to have that, but for now, the inheritance feature would at least allow to come up plugins for Isar which does that. CRDT requires fields, which in this case is impossible to define into a base class.
For P2P and Backend solutions, the solution needs to be flexible, so everyone can implement in their favorite/already used llang or framework. So

Inheritance
Push and pull functions - to get data to push and save data from pull
Migration helper functions
Built-in CRDT support - if we have this, inheritance is not mandatory

would be a good start point, the community can help implement the backend side with Rest api/graphql/websocket syncs. And with some framework maybe the p2p sync is possible too. (You need backend to connect the 2 phone to each other anyway)

but for now, the inheritance feature would at least allow to come up plugins for Isar which does that

What do you mean by "inheritance feature"? Isar already supports inheritance of collections

Push and pull functions
Do you mean like watermelondb?

Migration helper functions
What exactly are you missing?

Built-in CRDT support
How do you envision this to work? What should Isar provide?

I didn't know it already support inheritance. It wasn't a year ago. Let me check.
If Isar aims to support sync, then these are the bricks first need to be implemented.
Yes, similar what watermelon or couchbase does.
By CRDT support, I mean, the object could have fields built-in, for time sync. And on flutter side, it could be updated respectively as it needs. If we save an object the save could update those fields as it should be by definition.
The fields I mean are: Object ID, HybridLogicalClock (Timestamp, device id, counter). But of course with inheritance we can define base class to have these fields, and can have save override, push, pull functions easily. All depends on the vision how built-in support you plan.
From the backend side, there are many way it can be done - one possible way is - Rest APIs with websocket support to listen changes real time. If you have plans to implement the sync in Isar, I can create a backend in Django, with socket support. The DB on the backend could store the objects as json in Postgre, and the above fields separately, to have the efficience filtering.

I didn't know it already support inheritance. It wasn't a year ago. Let me check.
If Isar aims to support sync, then these are the bricks first need to be implemented.
Yes, similar what watermelon or couchbase does.
By CRDT support, I mean, the object could have fields built-in, for time sync. And on flutter side, it could be updated respectively as it needs. If we save an object the save could update those fields as it should be by definition. The fields I mean are: Object ID, HybridLogicalClock (Timestamp, device id, counter). But of course with inheritance we can define base class to have these fields, and can have save override, push, pull functions easily. All depends on the vision how built-in support you plan.
From the backend side, there are many way it can be done - one possible way is - Rest APIs with websocket support to listen changes real time. If you have plans to implement the sync in Isar, I can create a backend in Django, with socket support. The DB on the backend could store the objects as json in Postgre, and the above fields separately, to have the efficience filtering.

Could this enable Peer-to-peer ?

https://wiki.nikitavoloboev.xyz/distributed-systems/crdt
https://github.com/alangibson/awesome-crdt
More information about CRDT.

This is really awesome!

Hi, I have a pretty low tech proposal.
It would be cool to be able to get the transaction log for certain collections.
This would allow to transmit the transactions as a json patch to a server which can then figure out which modifications should be made and which should be discarded.
This might result in some datalos, but makes it easy to transmit modification to the server.

Should we put together a draft document?

Should we put together a draft document? like a summary of what we have discussed here?

Hi so I've read the discussion and i m gonna be honest i didn't understand everything but a lot of it sounded for me like over engineering. I personally think that the way WatermelonDb manages it is completely fine and easy to implement. If you agree on that with me I work on an Offline Support for my Appwrite backend and it works but help would be much appreciated so if somebody is interested please join :)
https://github.com/ThalesVonMilet/offline_support_for_db

@ThalesVonMilet I think it is cool, visited the project could not see how to test your solution or see how to contribute.

Hi so I've read the discussion and i m gonna be honest i didn't understand everything but a lot of it sounded for me like over engineering. I personally think that the way WatermelonDb manages it is completely fine and easy to implement. If you agree on that with me I work on an Offline Support for my Appwrite backend and it works but help would be much appreciated so if somebody is interested please join :)
https://github.com/ThalesVonMilet/offline_support_for_db

I agree there is some very complex solutions here.
I think for most use case they are too complex. But I also think your proposal is to simple. If I understand correctly you are using a flag to detect if record was changed. But for example for my use case this is not enough. That's why I proposed to to log the transactions. That way you can either transmit the whole record or just the parts that changed. In your solution modifications on two clients would lead to a conflict even if they changed different parts of the record.

Hi so I've read the discussion and i m gonna be honest i didn't understand everything but a lot of it sounded for me like over engineering. I personally think that the way WatermelonDb manages it is completely fine and easy to implement. If you agree on that with me I work on an Offline Support for my Appwrite backend and it works but help would be much appreciated so if somebody is interested please join :)
https://github.com/ThalesVonMilet/offline_support_for_db

I agree there is some very complex solutions here.
I think for most use case they are too complex. But I also think your proposal is to simple. If I understand correctly you are using a flag to detect if record was changed. But for example for my use case this is not enough. That's why I proposed to to log the transactions. That way you can either transmit the whole record or just the parts that changed. In your solution modifications on two clients would lead to a conflict even if they changed different parts of the record.

Your absolutely right with the last part I wanted to solve that by adding a change log the other way would be to add for each field a extra field bit I think the change log is much better.

Is the intent to allow peer-to-peer decentralized databases without a central server? Here is my use-case ...
I am in a group called CERT (Community Emergency Response Team). There are nationwide groups (in the US) with no relationship to each other. I would like to develop an opensource project for any group to use ... but each group's members would only see data within their group.
I'm trying to find a way to have an open-source project without incurring database costs and without having users have to configure databases outside of the app.
Does it sound like this would be doable within the framework you are thinking about?
Thanks ... Jeff

Interesting Idea.
Depending on your needs have a look at object box. That might serve you better.
Or if you are not bound to flutter look at gundb or orbitdb maybe they fit better in a P2P.

https://github.com/ljwagerfield/crdt @leisim

This https://pocketbase.io/ can also help

If you aim for fully distributed applications there is holepunch for NodeJS. You can create distributed applications that seem to actually work. An application implemented via holepunch is Keet, a peer to peer video conferencing software.

Any thoughts how to track the transactions best? Is there a way to get before and after values in the watch functions?

so you know we are working on CRDT here https://github.com/cachapa/crdt for isar, your contribution is highly welcomed.

Well for my usecase crdts are to much. It would be best for me to just transmit transactions.

Well for my usecase crdts are to much. It would be best for me to just transmit transactions.

That is fine but the package is meant to support the community, so it is wise to widen the use case for the sake of community. not just one use case for one person.

@jonaswre
If your use case does not require CRDT, then your user is using one device only all the time, which doesn't seem a valid mobile user to me. Are you sure that crdt is not required? If so, you can use flutter_data or similar package to achieve your goal.
@richard457
I don't see any new branch in the linked crdt package. I have changed it last year to my needs so I am familiar with the concepts.
Also I would suggest to take a look how yjs handles this, because it can be used here. (there is a rust implementation of yjs, which could be useful as isar is using rust under the hood, but correct me if I am wrong)

If your use case does not require CRDT, then your user is using one device only all the time, which doesn't seem a valid mobile user to me. Are you sure that crdt is not required? If so, you can use flutter_data or similar package to achieve your goal.

Yes thats correct. Its acceptable to not being able to work on two records on diffrent devices.  I'm not sure if that make them less valid users but thats up for debate.
They still use flutter and need offline capabilities, fulltext search and all the others features isar offers when they are mobile.
So I guess its correct you can't get a transaction log?

@jonaswre If your use case does not require CRDT, then your user is using one device only all the time, which doesn't seem a valid mobile user to me. Are you sure that crdt is not required? If so, you can use flutter_data or similar package to achieve your goal.
@richard457 I don't see any new branch in the linked crdt package. I have changed it last year to my needs so I am familiar with the concepts.
Also I would suggest to take a look how yjs handles this, because it can be used here. (there is a rust implementation of yjs, which could be useful as isar is using rust under the hood, but correct me if I am wrong)

sorry wrong link, please find the repo here CRD-ISAR

@jonaswre If your use case does not require CRDT, then your user is using one device only all the time, which doesn't seem a valid mobile user to me. Are you sure that crdt is not required? If so, you can use flutter_data or similar package to achieve your goal.
@richard457 I don't see any new branch in the linked crdt package. I have changed it last year to my needs so I am familiar with the concepts.
Also I would suggest to take a look how yjs handles this, because it can be used here. (there is a rust implementation of yjs, which could be useful as isar is using rust under the hood, but correct me if I am wrong)

you are also right, we need CRDT for a reason for him please try this db for example http://pocketbase.io/

you are also right, we need CRDT for a reason for @jonaswre please try this db for example http://pocketbase.io/, but at least for the sake of community we need CRDT to support multiple devices going offline and online and still be able to have data in one place, and these devices can be mobiles,desktops etc..

I can't use a diffrent database then I currently have. I was asking for the transaction log.

As you know we are working on CRDT here https://github.com/davidlondono/isar_crdt for isar, your contribution is highly welcomed.

@richard457 might you have an example on how to handle this server-side?

As you know we are working on CRDT here https://github.com/davidlondono/isar_crdt for isar, your contribution is highly welcomed.

@richard457 might you have an example on how to handle this server-side?

The hive version has a websocket example, it might help you for the isar version

As you know we are working on CRDT here https://github.com/davidlondono/isar_crdt for isar, your contribution is highly welcomed.

@richard457 might you have an example on how to handle this server-side?

The idea is to make it compatible with any server for example http://pocketbase.io/

Making isar sync up with pocketbase would be amazing!!

I did something similar with drift (SQLite) and now am exploring ISAR.
I use pocketbase for a content CMS in my apps (ease of use for admin console) and firebase firestore for a user remote database.
But client side I usually run hive and SQLite through drift. This database interests me as it might provide some of the things I was creating manually.
https://github.com/rodydavis/pocketbase_drift

I would love to get an update on this topic. Will Isar add the feature to sync offline database to online? Developing mobile apps requires handling apps going offline because of connectivity issues depending on location.
Thanks you.


server/sse: return response in handle_post_message

Changes

Modified SseServerTransport.handle_post_message to return Response objects instead of sending them directly via ASGI

Motivation and Context

The current implementation directly sends ASGI responses, which breaks when used with Starlette because Starlette expects route handlers to return response objects. After the request is handled, Starlette tries to call the response object which currently fails because the handle_post_message method returns  None:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/artemm/Code/mcp_proxy/.venv/lib/python3.13/site-packages/starlette/routing.py", line 74, in app
    await response(scope, receive, send)
          ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

How Has This Been Tested?

Tested the change with this code running locally:
# server.py
import logging
from mcp.server import Server
from mcp.server.sse import SseServerTransport
import mcp.types as types
from starlette.applications import Starlette
from starlette.routing import Route

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("sse-server")

# Create SSE transport with endpoint
app = Server("example-server")
sse = SseServerTransport("/messages")

@app.list_tools()
async def handle_list_tools():
    return [
        types.Tool(
            name="hello",
            description="Says hello",
            inputSchema={},
        )
    ]

# Handler for SSE connections
async def handle_sse(request):
    async with sse.connect_sse(
        request.scope, request.receive, request._send
    ) as streams:
        await app.run(streams[0], streams[1], app.create_initialization_options())

# Handler for client messages
async def handle_messages(request):
    await sse.handle_post_message(request.scope, request.receive, request._send)

# Create Starlette app with routes
starlette_app = Starlette(
    debug=True,
    routes=[
        Route("/sse", endpoint=handle_sse),
        Route("/messages", endpoint=handle_messages, methods=["POST"]),
    ],
)

# Run with any ASGI server
import uvicorn

uvicorn.run(starlette_app, host="<IP_ADDRESS>", port=8000)

# client.py
import asyncio
from mcp.client.sse import sse_client
from mcp.client.session import ClientSession

async def get_tools():
    async with sse_client("http://localhost:8000/sse") as streams:
        async with ClientSession(streams[0], streams[1]) as session:
            await session.initialize()

            # List available tools
            return await session.list_tools()

async def main():
    tools = await get_tools()
    print(tools.tools)

if __name__ == "__main__":
    asyncio.run(main())

Breaking Changes

Technically, the change is breaking because it removes on of the unused parameter but it didn't work anyway.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation update

Checklist

[x] I have read the MCP Documentation
[x] My code follows the repository's style guidelines
[x] New and existing tests pass locally
[x] I have added appropriate error handling
[x] I have added or updated documentation as needed

Additional context

@jspahrsummers hello! could you please review this PR? i found a workaround but it's kinda ugly:
async def handle_messages(request):
    # Create a dummy response that we'll return to Starlette
    from starlette.responses import Response

    response = Response("", status_code=202)

    async def send_wrapper(message):
        # Skip sending response since we're handling it at the Starlette level
        if (
            message["type"] != "http.response.start"
            and message["type"] != "http.response.body"
        ):
            await request._send(message)

    await sse.handle_post_message(request.scope, request.receive, send_wrapper)
    return response

Closing this out since it seems we have a path forwrad that doesnt require a code change.

@dsp-ant will you update the documentation and code samples, then?

I've sent https://github.com/modelcontextprotocol/python-sdk/pull/121 to update


Cow: Watch emblixsolutions.com

Cow requests the watch of the watch_keyword emblixsolutions\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

emblixsolutions\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ

Approved by Jeff Schaller in Charcoal HQ


Not a SIA or ADM Spec event, compatibility

Hi guys, I've been trying to set up a SIA monitor on HA, but have problems.
The alarm is a DSC board with an ethernet module:
https://www.trikdis.com/alarm-communicators/ethernet-alarm-communicators/ethernet-communicator-e16/
Now, it says it supports TRK, TL150, DC-09_2007 and DC-09_2012 protocols.
When I set it to DC-09_2012 without encryption, the log is as follows:
Can somebody point me in any direction?
2023-08-30 22:05:31.097 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: K0029"NULL"0000R1L1#9413[]_23:05:28,08-30-2023. Line: K0029"NULL"0000R1L1#9413[]_23:05:28,08-30-2023
2023-08-30 22:05:31.203 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: 003D"ADM-CID"0001R1L1#9413[#9413|1700 99 999]_23:05:28,08-30-2023. Line: 003D"ADM-CID"0001R1L1#9413[#9413|1700 99 999]_23:05:28,08-30-2023
2023-08-30 22:05:33.007 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: 0n003D"ADM-CID"0002R1L1#9413[#9413|1602 99 999]_23:05:30,08-30-2023. Line: 0n003D"ADM-CID"0002R1L1#9413[#9413|1602 99 999]_23:05:30,08-30-2023
2023-08-30 22:05:33.759 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: N"003D"ADM-CID"0003R1L1#9413[#9413|1411 99 999]_23:05:30,08-30-2023. Line: N"003D"ADM-CID"0003R1L1#9413[#9413|1411 99 999]_23:05:30,08-30-2023
2023-08-30 22:06:03.898 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: @0029"NULL"0000R1L1#9413[]_23:05:58,08-30-2023. Line: @0029"NULL"0000R1L1#9413[]_23:05:58,08-30-2023
2023-08-30 22:06:10.863 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: 003D"ADM-CID"0004R1L1#9413[#9413|1401 01 040]_23:06:05,08-30-2023. Line: 003D"ADM-CID"0004R1L1#9413[#9413|1401 01 040]_23:06:05,08-30-2023
2023-08-30 22:06:41.001 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: 0029"NULL"0000R1L1#9413[]_23:06:36,08-30-2023. Line: 0029"NULL"0000R1L1#9413[]_23:06:36,08-30-2023
2023-08-30 22:07:11.203 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: E'0029"NULL"0000R1L1#9413[]_23:07:06,08-30-2023. Line: E'0029"NULL"0000R1L1#9413[]_23:07:06,08-30-2023
2023-08-30 22:07:41.305 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: "0029"NULL"0000R1L1#9413[]_23:07:36,08-30-2023. Line: "0029"NULL"0000R1L1#9413[]_23:07:36,08-30-2023
2023-08-30 22:08:11.507 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: Ah0029"NULL"0000R1L1#9413[]_23:08:06,08-30-2023. Line: Ah0029"NULL"0000R1L1#9413[]_23:08:06,08-30-2023
2023-08-30 22:08:17.327 WARNING (MainThread) [pysiaalarm.base_server] Last line could not be parsed succesfully. Error message: No matches found, event was not a SIA or ADM Spec event, line was: "003D"ADM-CID"0005R1L1#9413[#9413|1412 99 999]_23:08:12,08-30-2023. Line: "003D"ADM-CID"0005R1L1#9413[#9413|1412 99 999]_23:08:12,08-30-2023
2023-08-30 22:08:36.795 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:08:32+00:00
2023-08-30 22:08:36.895 WARNING (MainThread) [pysiaalarm.base_server] Code not found, replying with DUH to account: 9413
2023-08-30 22:08:37.700 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:08:33+00:00
2023-08-30 22:09:07.888 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:09:05+00:00
2023-08-30 22:09:37.991 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:09:35+00:00
2023-08-30 22:09:48.667 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:09:45+00:00
2023-08-30 22:10:18.793 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:10:16+00:00
2023-08-30 22:10:48.895 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:10:46+00:00
2023-08-30 22:11:18.999 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:11:16+00:00
2023-08-30 22:11:49.206 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:11:46+00:00
2023-08-30 22:12:19.401 WARNING (MainThread) [pysiaalarm.base_server] Event timestamp is no longer valid: 2023-08-30 23:12:16+00:00
2023-08-30 22:12:31.458 WARNING (MainThread) [pysiaalarm.base_server] Code not found, replying with DUH to account: 9413

Hi @ChronoRadion I do not have a ADM device so I could never test these, if you are able add the lines above to the test suite then we can start troubleshooting and making adjustments.

Since I have access to device logs on ethernet communicator, we could link HA logs with them and see which line does what.
Could you point me on how to add those lines to the test suite?

In this class all sorts of cases are present: https://github.com/eavanvalkenburg/pysiaalarm/blob/0df5af750412421e697a106aa5ac9dfec1727398/tests/test_sia_package_cases.py#L110

I'm a bit of a novice in python, so before playing with the test suite, here's some data:
It seems to me that I have a timestamp problem but have no idea how to mitigate that (I think it is a time zone issue)
TRIKDIS DC-09_2012, without encryption:
By https://www.securityindustry.org/wp-content/uploads/2017/10/dc09_r2021_20201027.pdf
the line should look like:
<0LLL><"id"><#acct>[|...data...][x…data…]

I've added the lines to the test suite, but when I try running tests\run_aio.py I get:
  File "tests\run_aio.py", line 9, in <module>
    from pysiaalarm.aio import SIAAccount, SIAClient, SIAEvent
ModuleNotFoundError: No module named 'pysiaalarm'

Can you please tell me how to run the test suite? Do I have to run it on HomeAssistant as a custom component? Or can I run it on a linux PC with python?
Thank you


json specification

Issue/Task Number:
157 https://github.com/omisego/ewallet/issues/157
Overview
ewallet provides a yaml api specification the requirements are that we also provide json file format specification.
Changes

JSON file serving

Implementation Details
Following yaml implementation.
Function callback in macro and catching exceptions.
Usage
java -jar bin/openapi-generator-cli.jar generate -i ewallet//apps/ewallet_api/priv/spec.yaml -g openapi -o output/
Impact
JSON specs can be added as a CI step (but since you're defining the yaml manually I don't think it's needed)

Note that I'm trying this PR out today. Will give feedbacks once done.

Ok. Rebased for your convenience

Looking good! Thanks for keep finding the solution 🙇
Some notes:

I got two errors when putting the generated JSON specs into https://editor.swagger.io:

Semantic error at components.schemas.UnpaginatedListSchema.properties.data
Schemas with 'type: array', require a sibling 'items: ' field
Jump to line 1100

Semantic error at components.schemas.PaginatedListSchema.properties.data
Schemas with 'type: array', require a sibling 'items: ' field
Jump to line 1116

One small test failure at the moment:

  1) test /errors endpoints returns the json spec when calling /errors.json (EWallet.Web.APIDocs.ControllerTest)
     test/ewallet/web/api_docs/controller_test.exs:82
     Assertion with =~ failed
     code:  assert conn.resp_body() =~ ~r"\\\"- token_not_found:\\\\n    code:"
     left:  "\"- invalid_access_secret_key:\\n    code: ..."
     right: ~r/\"- token_not_found:\\n    code:/
     stacktrace:
       test/ewallet/web/api_docs/controller_test.exs:88: (test)

@sirn Any opinions on having a new build dependency: openapi-generator-cli.jar? Since this is only needed during builds, we could install from Github release -> generate JSON -> remove installed binaries, all within the Dockerfile?

Ignore my feedback about the JSON spec errors. Looks like the YAML specs cause those errors as well (and even more). I'll deal with that as a separate PR.

#237 is merged. The online editor errors should be fixed now.
Please try rebase with the latest change from develop branch. :)

Looks good to go for me from implementation point of view.
@sirn Any comments on directly including the jar executable in the repo?

just to clarify why I included the jar in the repo. The idea was that we could test  yaml and json specs by generating a client lib. The client lib would then be used in acceptance tests. I’m positive robotframework allows us to do that.
But since we're not there yet, we can also decide to remove it.

Having the JSON file directly in the repo bothers me a bit, I was hoping we could just generate it on the fly and return it from the controller 🤔 We would need to re-generate it every time a change is made to one of the spec.yaml which might get forgotten along the way.

Invoking the jar on every request?
The other option is being a part of deployment procedure (https://hexdocs.pm/distillery/boot-hooks.html pre start hook?)

I like the idea of having it as part of the deployment procedure. Though I think maybe it's better to run during the Docker build, and so we have the JSON specs already ready with the Docker image?
This way should also be the easiest to implement, since it is similar to how admin panel's frontend assets are built today.
Also, for development environment, it would be nice to setup a worker that watches for file changes and invoke the generator, but this could be a separate PR.

If you make it part of Docker build, you're forcing the docker deployment model. Is that intended?

Ah sorry I wasn't too clear. We're moving to a distillery release being prepared by @sirn, but in the meanwhile the current build setup relies on Docker.
So if I understand correctly, using a distillery pre start hook now means this feature is dependent on when our distillery release is ready. Hence the suggestion to align with our current docker build process for this PR.
Thoughts? @sirn @T-Dnzt

In the upcoming weeks, we'll be switching our build to use distillery and produce Docker image from distillery artifacts. Right now it could be a step in CI or Docker. I'll convert it to boot hook in the  branch.
Distillery branch is currently waiting for #240 to finalize (need to test with staging data, in which I got distracted with few other stuff) so I hope to get it done by the beginning of next week.


Error needs version in mainfest, HA requirement

Got this error
2023-09-06 19:05:41.524 WARNING (SyncWorker_2) [homeassistant.loader] We found a custom integration grohe_sense which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-09-06 19:05:41.524 ERROR (SyncWorker_2) [homeassistant.loader] The custom integration 'grohe_sense' does not have a version key in the manifest file and was blocked from loading. See https://developers.home-assistant.io/blog/2021/01/29/custom-integration-changes#versions for more details
can this plz be imbedded

That makes sense that it stopped working for me as well.

Workaround is to go to manifest.json in /config/custom_components/grohe_sense
Add a line with "version": "1.0",
And restpart HA
hope it helps

@KlausLadegaard
Thank you for the suggestion.
It was already in there though (might have added it at some point, can't remember), so it doesn't seem to work for me at least.


issue raises when running vagrant up

Hi I was trying to run Standalone deployment. After running vagrant provisioning, I met some issues. Can you help me to check about that?

Vagrant version: Vagrant 2.2.5
Ansible version: ansible 2.8.3

Hi, I just updated the provisioning and resolved several bugs (including yours). Everything should be working now.


[assemble] java-archive ignores java.mainModule

Task List

[x] Steps to reproduce provided
[ ] Stacktrace (if present) provided
[ ] Example that reproduces the problem (link to git repository is ideal)
[x] Full description of the issue provided (see below)

Steps to Reproduce

git clone https://github.com/jreleaser/helloworld-java-bin.git
Modify project configuration to look like this

  java:
    groupId: org.jreleaser.examples
    artifactId: helloworld
    mainClass: org.jreleaser.examples.HelloWorld
    mainModule: org.jreleaser

jreleaser assemble

Expected Behaviour
Generated launcher should configure execution with the modulepath
Actual Behaviour
Classpath is used instead
Environment Information

Operating System: N/A
JReleaser Version: 1.5.0
JDK Version: N/A

cc: @bmarwell

🎉 This issue has been resolved in v1.5.1 (Release Notes)