WIP-CRUD on top of Event Sourcing

This is just a first step to investigate if the CRUD state model could be done on top of Event Sourcing. If it is the case it can be a good candidate for CRQS and also Projection. So it is about this issue #50 (CRUD) and #81 (CRQS).
For CRUD the User Function should not deal with events and only with the snapshot of the whole object graph. For this i just add code to deal with snapshot and event in the EventSourcedEntity and comment code related to events in the language support.

@ralphlaude I think you should have a look at how I did it for KVS, also built on top of EventSourcing: https://github.com/cloudstateio/cloudstate/pull/143/files

@viktorklang i will do

annotation for CrudEntity is added and the registration of such entity in cloudstate is now possible. I am looking forward to deal with the following questions:

how to deal with the compression of events because and it can be very big
how to externali

@viktorklang, @jroper,
i want to know if my proposal regarding how to deal with the CRUD entity state in the User Function is the right way to go.
Here is the content of the crud_entity.proto file:
syntax = "proto3";

import "google/protobuf/any.proto";

package cloudstate;

option java_package = "io.cloudstate.crud";

// The CRUD state.
message CrudEntityState {
  bytes state = 1;
}

// The CRUD change events.
message CrudEntityStateModification {
  bytes state = 1;
}

Here is the class which deals with the CrudEntity state and will be in the java support:
package io.cloudstate.samples.shoppingcart;

import com.example.shoppingcart.Shoppingcart;
import com.example.shoppingcart.crud.persistence.Domain;
import com.google.protobuf.Any;
import com.google.protobuf.Empty;
import com.google.protobuf.Message;
import io.cloudstate.crud.CrudState;
import io.cloudstate.javasupport.EntityId;
import io.cloudstate.javasupport.crud.CrudEntity;
import io.cloudstate.javasupport.eventsourced.CommandContext;
import io.cloudstate.javasupport.eventsourced.CommandHandler;
import io.cloudstate.javasupport.eventsourced.EventHandler;
import io.cloudstate.javasupport.eventsourced.Snapshot;
import io.cloudstate.javasupport.eventsourced.SnapshotHandler;

import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

// This class will be in the java support
public abstract class AbstractCrudEntityState<T extends Message> {

  // CrudState.CrudEntityState represents the persistent state when snapshot is done
  // CrudState.CrudEntityStateModification represents the persistent events
  // CrudState.CrudEntityState and CrudState.CrudEntityStateModification will likely be renamed

  // How to handle get rid of the type url?
  private String typeUrl;

  // How to deal with specific type here? I will try to use generics here but i am sure it will help.
  private T state;

  public final void state(T state) {
    // do some check here
    this.typeUrl = state.getDescriptorForType().getFullName();
    this.state = state;
  }

  public final T state() {
    // How to deal with reflection or grpc here?
    return state;
  }

  public final CrudState.CrudEntityState toEntityState() {
    return CrudState.CrudEntityState.newBuilder().setState(state.toByteString()).build();
  }

  public final CrudState.CrudEntityStateModification toEvent() {
    return CrudState.CrudEntityStateModification.newBuilder().setState(state.toByteString()).build();
  }

  // transform the state (domain state) in CrudState.CrudEntityStateModification
  public final CrudState.CrudEntityStateModification toEvent(T modifiedState) {
    return CrudState.CrudEntityStateModification.newBuilder().setState(modifiedState.toByteString()).build();
  }

  public final AbstractCrudEntityState<T> resetTo(CrudState.CrudEntityState entityState) {
    // How to get rid of the type url?
    this.state = (T) Any.newBuilder().setTypeUrl(typeUrl).setValue(entityState.getState()).build();
    return this;
  }

  public final AbstractCrudEntityState<T> applyEvent(CrudState.CrudEntityStateModification modification) {
    // How to get rid of the type url?
    this.state = (T) Any.newBuilder().setTypeUrl(typeUrl).setValue(modification.getState()).build();
    return this;
  }

  @CrudEntity
  public static final class ShoppingCart extends AbstractCrudEntityState<Domain.Cart> {
    private final String entityId;

    private Domain.Cart cart = Domain.Cart.newBuilder().build();

    public ShoppingCart(@EntityId String entityId) {
      this.entityId = entityId;
      this.state(cart);
    }

    @Snapshot
    public CrudState.CrudEntityState snapshot() {
      return toEntityState();
    }

    @SnapshotHandler
    public void handleSnapshot(CrudState.CrudEntityState state) {
      cart = resetTo(state).state();
    }

    @EventHandler
    public void handleEvent(CrudState.CrudEntityStateModification event) {
      cart = applyEvent(event).state();
    }

    @CommandHandler
    public Shoppingcart.Cart getCart() {
      Collection<Shoppingcart.LineItem> lineItems =
              cart.getItemsList().stream().map(this::convert).collect(Collectors.toList());

      return Shoppingcart.Cart.newBuilder().addAllItems(lineItems).build();
    }

    @CommandHandler
    public Empty addItem(Shoppingcart.AddLineItem item, CommandContext ctx) {
      if (item.getQuantity() <= 0) {
        ctx.fail("Cannot add negative quantity of to lineItem" + item.getProductId());
      }

      Domain.Cart modifiedCart = applyCommand(item); // create new modified state for shopping cart
      CrudState.CrudEntityStateModification modification = toEvent(modifiedCart); // create event sourced event of the state

      ctx.emit(modification);

      return Empty.getDefaultInstance();
    }

    private Domain.Cart applyCommand(Shoppingcart.AddLineItem item) {
      Domain.LineItem lineItem =
              Domain.LineItem.newBuilder()
                      .setProductId(item.getProductId())
                      .setName(item.getName())
                      .setQuantity(quantity(item))
                      .build();

      List<Domain.LineItem> items = cart.getItemsList().stream()
              .filter(i -> !i.getProductId().equals(item.getProductId()))
              .collect(Collectors.toList());

      return Domain.Cart.newBuilder()
              .addAllItems(items)
              .addItems(lineItem)
              .build();
    }

    private Shoppingcart.LineItem convert(Domain.LineItem item) {
      return Shoppingcart.LineItem.newBuilder()
              .setProductId(item.getProductId())
              .setName(item.getName())
              .setQuantity(item.getQuantity())
              .build();
    }

    private int quantity(Shoppingcart.AddLineItem item) {
      return cart.getItemsList().stream()
              .filter(lineItem -> lineItem.getProductId().equals(item.getProductId()))
              .findFirst()
              .map(lineItem -> lineItem.getQuantity() + item.getQuantity())
              .orElse(item.getQuantity());
    }
  }
}

@ralphlaude This isn't really CRUD (there's no delete), and it's pushing the definition of retrieve as well (since you can't arbitrarily retrieve any entity from a function, you're given an entity to work with), so I wouldn't call this CRUD. It's a key value, and we already have user function level support for key value storage.
What we'd like to do is support non eventsourced support for key value storage, so that things can be stored optimised (ie, without all the history). It could be accompanied by a log for CQRS, but for most use cases, that log need only be an index of when the values last changed, again stored optimised for the given store.

Hi @jroper,
thanks for the insights and i was thinking of CRUD just for an entity and not for the whole entity type. I will rethink this.

@jroper Loading any instance in the stateful function will completely mess up the entity_key routing, will it not? Or you are thinkin about storing multiple sub-aggregates under the same entity_key?

@viktorklang, @jroper,
CRUD will leverage key value store and will be built on top of event sourcing.
For building the change logs for CQRS one option could be to have CRUD chnage log containing last updated values with index and also all removed values.
I am not sure if it makes sense to save the change logs as additional part of the state (the change logs could grow and could be very big). I think it also stored optimised.
@viktorklang i was thinking to use an entity_key for the whole aggregate graph. I am not sure here but perhaps there are another options.

@viktorklang, @jroper,
This is an extented implementation i propose based on (https://github.com/cloudstateio/cloudstate/pull/143/files of @viktorklang).
What i did:

Introduce a new class io.cloudstate.javasupport.crud.KeyValue.ChangedMap for dealing with last changed (updates and remove) in the CRUD entity. This is useful for CQRS
I change the definition of the io.cloudstate.javasupport.crud.CrudEntity annotation and also the logic for CloudState.registerCrudEntity

What i would like to do:

Who should register the key value proto file descriptor io.cloudstate.keyvalue.KeyValue.getDescriptor()?
i would propose to do in CloudState.registerCrudEntity so it is implicit to the user
How to define persistent actor id for the CRUD entity? The CRUD entity is backed by a persistent actor and this actor should receive all request
for the CRUD entity. Unfortunately the id (entity_key) for this actor cannot be extract from the GRPC request paylod, the entity_key should be created upfront.
I would suggest to introduce a new entity type that can be passed to io.cloudstate.javasupport.impl.eventsourced.EventSourcedStatefulService and
further down to io.cloudstate.proxy.EntityUserFunctionTypeSupport which can decide how to create the entity_key of persitent actor.

What are the next steps:

better naming, better documentation and better code design
how to deal with the CRUD entity state in the io.cloudstate.javasupport.eventsourced.CommandContext to be able to ctx.setState like @viktorklang proposed?
how to deal with sub-aggregates regarding the key? Use the same key or not for saving sub-aggregates?

Any comments are welcome.

The progress is as follow:

the GRPC protocol is defined and implemented see below
the java support for CRUD is defined and implemented
the CRUD service can be registered as service in cloudstate
the CRUD entity is defined and implemented in the proxy
the CRUD service support snapshotting
there is a shopping cart example in the java sample that should be completed

The CRUD service in the java-support is right now initialized when a command come in. When the service starts there is not information upfront for initializing the CRUD service until a command is sent. I am wondering if there is another way to do the initialization upfront.
Each command has a type so we know which service operation to route the command to.
syntax = "proto3";

import "google/protobuf/descriptor.proto";

package cloudstate;

option java_package = "io.cloudstate";
option go_package = "github.com/cloudstateio/go-support/cloudstate/;cloudstate";

extend google.protobuf.FieldOptions {
    bool crud_command_type = 50005;
}

Here is the content of the crud.proto file which defines the CRUD service grpc protocol:
syntax = "proto3";

package cloudstate.crud;

// Any is used so that domain events defined according to the functions business domain can be embedded inside
// the protocol.
import "google/protobuf/any.proto";
import "google/protobuf/empty.proto";
import "cloudstate/entity.proto";

option java_package = "io.cloudstate.protocol";
option go_package = "cloudstate/protocol";

// The type of the command to be executed
enum CrudCommandType {
    UNKNOWN = 0;
    CREATE = 1;
    FETCH = 2;
    FETCHALL = 3;
    UPDATE = 4;
    DELETE = 5;
}

// The persisted state with the sequence number of the last snapshot
message CrudState {
    // The state payload
    google.protobuf.Any payload = 2;

    // The sequence number when the snapshot was taken.
    int64 snapshot_sequence = 1;
}

// Message for initiating the command execution
// which contains the command type to be able to identify the crud operation being called
message CrudEntityCommand {
    // The ID of the entity.
    string entity_id = 1;

    // The ID of a sub entity.
    string sub_entity_id = 2;

    // Command name
    string name = 3;

    // The command payload.
    google.protobuf.Any payload = 4;

    // The command type.
    CrudCommandType type = 5;
}

// The command to be executed which can be for any of the supported
// (create, fetch, save, delete, fetchAll) crud operations.
message CrudCommand {
    // The name of the service this crud entity is on.
    string service_name = 1;

    // The ID of the entity.
    string entity_id = 2;

    // The ID of a sub entity.
    string sub_entity_id = 3;

    // A command id.
    int64 id = 4;

    // Command name
    string name = 5;

    // The command payload.
    google.protobuf.Any payload = 6;

    // The persisted state to be conveyed between persistent entity and the user function.
    CrudState state = 7;
}

// A reply to a command.
message CrudReply {

    // The id of the command being replied to. Must match the input command.
    int64 command_id = 1;

    // The action to take
    ClientAction client_action = 2;

    // Any side effects to perform
    repeated SideEffect side_effects = 3;

    // An optional state to persist.
    google.protobuf.Any state = 4;

    // An optional snapshot to persist. It is assumed that this snapshot will have
    // the state of any events in the events field applied to it. It is illegal to
    // send a snapshot without sending any events.
    google.protobuf.Any snapshot = 5;
}

// A reply message type for the gRPC call.
message CrudReplyOut {
    oneof message {
        CrudReply reply = 1;
        Failure failure = 2;
    }
}

// The CRUD Entity service
// Provides read and write operations for managing the entity state.
// For a CRUD entity an event represents also the whole state of the entity.
// A read operation only transport the state from the entity without changing it.
// Typical read operations are fetch and fetchAll.
// A typical write operation might transform the state of the entity and crate a new one.
// Typical write operations are create, save, and delete.
// Each write operation may generate zero or one event which is then sent to the entity.
// The entity is expected to apply these event to its state.
service Crud {

    // Create a sub entity.
    rpc create(CrudCommand) returns (CrudReplyOut) {}

    // Fetch the state of a sub entity.
    rpc fetch(CrudCommand) returns (CrudReplyOut) {}

    // Save a updated sub entity.
    rpc save(CrudCommand) returns (CrudReplyOut) {}

    // Delete a sub entity.
    rpc delete(CrudCommand) returns (CrudReplyOut) {}

    // Fetch the state of the whole entity.
    rpc fetchAll(CrudCommand) returns (CrudReplyOut) {}
}

Converting this to draft PR while it's still WIP.

Provide a definition of the protocol based on the proposal of @jroper with snapshotting capabilities and implement it. the java support and the proxy entity are implemented.
I still have some issues regarding the facing:
@CrudEntity
public class ShoppingCartCrudEntity {

    @StateHandler
    public void handleState(Optional<Domain.Cart> cart) {
        ...
    }

    @CommandHandler
    public Shoppingcart.Cart getCart() {
        ...
    }

    ...
}

what are the next steps:

[ ] validating the protocol definition
[ ] validating the protocol implementation and the user facing interface
[ ] I want to use optional because the state is not always there. it not easy to provide a type safe behavior with the RelectionHelper and the io.cloudstate.javasupport.impl.crud.AnnotationBasedCrudSupport.StateHandlerInvoker.
the method handleState defines Optional<Domain.Cart> and it should be handle per reflection so that the method does not accept Optional<Domain.OthetEntity>. I would be glad to have some hints here.
[ ] write tests for the annoatation support and the entity
[ ] provide an sample in a dedicated project
[ ] extends the TCK

@jroper what do you think?

the implementation is done. please @viktorklang, @jroper , @pvlugter, @sleipnir, @marcellanz take a look and give some feedback.

the CRUD native support is added and only the in-memory CRUD support is integrated right now. the next is to support other native CRUD database like Postgres. But first of all we should agree on the way the native CRUD support is implemented.
What should be reviewed:

native CRUD support based on Slick
In memory CRUD support added
the protocol implementation based on native CRUD
remove the snapshot from the GRPC protocol
remove the sequence number from the GRPC protocol

please take a look a give some feedback :).
ping! @viktorklang @jroper @pvlugter @sleipnir @marcellanz

@viktorklang I would like to know if the API proposal for CRUD native support (JdbcStore) is fine and the way it is integrated in the CrudEntity. i was thinking about a possible way to have the same API for CRUD native support and KV native support. I am not sure it will work and it is worth investigating when we agree on the API for CRUD native support.

There are still open questions regarding subsequent invocations on the CommandContext (with methods getState, updateState and deleteState) during an command execution. What should be the behavior for those combinations of invocations?
ctx.deleteState()
ctx.updateState(...)

ctx.deleteState()
ctx.deleteState()

ctx.updateState(...)
ctx.deleteState()

ctx.deleteState()
ctx.getState()

ctx.updateState(...)
ctx.updateState(...)

more combinations

The following options are available:

the restrictive approach allows only one call of updateState or deleteState. Here we are not sure about the intention of the user. Every violation of the contract will produce a failure and the user knows what went wrong.
the non restrictive approach allows all the combination by collecting all calls and apply them.

Those are the only options i see now and i don't know what is the best one. i will argue to put this discussion on another track because we don't know yet what is a valuable option. we can still discuss it later on (i will write an issue for this) and hopefully we will have more options. For now i will argue to go with the first option.
We started this issue to do CRUD based on Event Sourcing. On the way we discovered or decided it is better to do it on top of native CRUD support. I will strongly argue to focus more on the native CRUD support because it is the last missing part. It is important to have a functional CRUD protocol we can iterate on and also improve.
What do you think @viktorklang?

From discussion on the contributors' call: use option 2, the non-restrictive approach (for language supports that have an imperative API). Apply the changes locally and report the final operation to the proxy. Should be semantically clear with get, update, and delete operations.

@pvlugter thanks for writing the approach here and for the inputs in the contributor call!
i think the non restrictive approach is the optimal one because the current state is in the CommandContext. the state can be freely manipulated in the CommandContext with every combination of operations or subsequent invocations. at the end of the command execution the CommandContext has the current state and that state can be passed to the entity by the proxy. no further caution is needed here!

The CRUD protocol implementation passes the state to the CommandContext and every change on that state with updateState or deleteState is maintained in the CommandContext. The changes are only applied to entity when the command execution is finished. Here there are still updateState-then-fail or deleteState-then-fail to deal with. In those cases i would assume the state is corrupted and restart the CRUD entity (meaning the state is reset) to deal with the next command. The restart behavior suggested and implemented here (https://github.com/cloudstateio/cloudstate/pull/439) can be adapted for the CRUD entity eventhough the CRUD entity is an pure Actor and not a PersistentActor. @pvlugter i would be interested in your opinon regarding the restarting behaviour.
The CRUD native support has been added using Slick and is already integrated in the proxy-jdbc and proxy-postgres. The dependencies for Slick has been added to the core. Right now it means proxy-jdbc and proxy-postgres support event sourcing and CRUD. In the core the native CRUD support is wrapped around a repository which is used by the CRUD entity to access the database.
There are still some work to be done for configuring the Slick-hikari connection pool, for writing the tests for the restart behavior, for writing the TCK tests and for integrating metrics.
Please take a look and give me feed back (specially on Slick and also the Actor behavior of the CRUD entity). I know you are all busy :).

I have a failure in the build (https://app.circleci.com/pipelines/github/cloudstateio/cloudstate/522/workflows/1508b0e5-db84-4e3e-ae29-1e46fc50ec5d/jobs/1375). I would like some support here. @pvlugter could you please take a look?

I have a failure in the build (https://app.circleci.com/pipelines/github/cloudstateio/cloudstate/522/workflows/1508b0e5-db84-4e3e-ae29-1e46fc50ec5d/jobs/1375). I would like some support here. @pvlugter could you please take a look?

For the TCK integration tests, you probably only need to add the CRUD protocol to the supported types here:
https://github.com/ralphlaude/cloudstate/blob/wip-crud-on-top-of-event-sourcing/tck/src/main/scala/io/cloudstate/tck/CloudStateTCK.scala#L109-L113
For the smoke tests, it doesn't seem to successfully bring up the postgres shopping cart. It also fails to print the logs (which we should fix) so there's not much information. I'd suggest running through the smoke test steps manually (see the scripts) to be able to see what's going on — if it is an actual issue and not just timing.

The CRUD protocol implementation passes the state to the CommandContext and every change on that state with updateState or deleteState is maintained in the CommandContext. The changes are only applied to entity when the command execution is finished. Here there are still updateState-then-fail or deleteState-then-fail to deal with. In those cases i would assume the state is corrupted and restart the CRUD entity (meaning the state is reset) to deal with the next command. The restart behavior suggested and implemented here (#439) can be adapted for the CRUD entity eventhough the CRUD entity is an pure Actor and not a PersistentActor. @pvlugter i would be interested in your opinon regarding the restarting behaviour.

Yes, could support restart on failure as well. Although I expect that it's also easier to support rollback for CRUD entities in the language supports — as it should be easy to save the state from before command processing, while for event sourced entities it can mean saving the last snapshot and all persisted events, or asking for new snapshots before command processing (and snapshots are expected but optional). So might not need the restart mechanism at all, but good to check with atomic failure tests and have it supported in some way. Great to see all the testing in this PR 👍

I've made sure the logs are correctly printed on failures in #443.
The smoke test is failing when loading the application config for postgres proxy:
application.conf: 50: Could not resolve substitution to a value: ${cloudstate.proxy.postgres.schema}

The setting can't be substituted when it's null (when the env var is not set). Needs a default.
I've pushed small fixes to this branch, for the config and adding as an expected protocol to the TCK, so we can get the tests green.

@pvlugter thanks a lot for the support :thumbsup:

Very good work @ralphlaude and @pvlugter

Hi @pvlugter,
Could you please support here?
I have added TCK tests for CRUD. I have some issue with the tests and I didn't find out the root cause yet.
running the tck test locally sbt -Dconfig.resource=application.conf tck/it:test
produces errors for the tests ManagedCloudStateTCK.when verifying app test: event-sourced shopping cart and ManagedCloudStateTCK.when verifying proxy test: HTTP API
here the error:
[service] [ERROR] [10/13/2020 13:04:02.329] [StatefulService-akka.actor.default-dispatcher-4] [EventSourcedImpl] Fail invoked for command [Process] for entity [entity:16]: expected failure
[service] [ERROR] [10/13/2020 13:04:02.368] [StatefulService-akka.actor.default-dispatcher-8] [EventSourcedImpl] Fail invoked for command [Process] for entity [entity:17]: expected failure
[service] [ERROR] [10/13/2020 13:04:02.408] [StatefulService-akka.actor.default-dispatcher-5] [EventSourcedImpl] Fail invoked for command [Process] for entity [entity:18]: failure 1
[service] [ERROR] [10/13/2020 13:04:02.444] [StatefulService-akka.actor.default-dispatcher-8] [EventSourcedImpl] Fail invoked for command [Process] for entity [entity:18]: failure 2
[service] [ERROR] [10/13/2020 13:04:02.510] [StatefulService-akka.actor.default-dispatcher-8] [EventSourcedImpl] Fail invoked for command [Process] for entity [entity:19]: expected failure
[service] [ERROR] [10/13/2020 13:04:02.775] [StatefulService-akka.actor.default-dispatcher-8] [akka.actor.ActorSystemImpl(StatefulService)] Error reported from sidecar: Warning: com.example.shoppingcart.ShoppingCart.GetCart: Expected reply type_url to be [type.googleapis.com/com.example.shoppingcart.Cart] but was [type.googleapis.com/com.example.crud.shoppingcart.Cart].
[info]   - must verify the HTTP API for ShoppingCart service *** FAILED ***
[info]     java.lang.AssertionError: assertion failed: expected class io.cloudstate.testkit.eventsourced.InterceptEventSourcedService$Connection, found class io.cloudstate.testkit.crud.InterceptCrudService$Connection (io.cloudstate.testkit.crud.InterceptCrudService$Connection@5c2b143b)
[info]     at scala.Predef$.assert(Predef.scala:282)
[info]     at akka.testkit.TestKitBase.expectMsgClass_internal(TestKit.scala:570)
[info]     at akka.testkit.TestKitBase.expectMsgType(TestKit.scala:541)
[info]     at akka.testkit.TestKitBase.expectMsgType$(TestKit.scala:540)
[info]     at akka.testkit.TestKit.expectMsgType(TestKit.scala:969)
[info]     at io.cloudstate.testkit.eventsourced.InterceptEventSourcedService.expectConnection(InterceptEventSourcedService.scala:34)
[info]     at io.cloudstate.testkit.InterceptService.expectEventSourcedConnection(InterceptService.scala:57)
[info]     at io.cloudstate.tck.ShoppingCartVerifier.verifyConnection(CloudStateTCK.scala:596)
[info]     at io.cloudstate.tck.CloudStateTCK.$anonfun$new$67(CloudStateTCK.scala:529)
[info]     at io.cloudstate.tck.CloudStateTCK.checkHttpRequest$1(CloudStateTCK.scala:522)
[info]     ...

Thanks

@ralphlaude, I haven't tried running it, but from the exception it looks like it's getting the CRUD shopping cart instead of the event sourced shopping cart. Do they have the same service name?
Also, no need to add an additional TCK test class, you should be able to add the CRUD TCK tests in with the other tests.

@pvlugter thanks for the hints :), it was my bad. I managed the issue and it is fine now.

Hi,
all regarding the functionality for the new ValueEntity is done.
It is about the implementation of the GRPC protocol, the JDBC native support and ValueEntity Actor.
Please take a look and give some feedback. It would be interesting to have some feedbacks on JDBC native support and also the Actor behavior of the ValueEntity.
What should be done are:

integrate tests for ValueEntity in circleci
remove duplicate codes

Ping @jroper, @pvlugter, @viktorklang, @sleipnir, @marcellanz

Thanks for keeping on pulling this feature across the finishing line, @ralphlaude!

I'm thinking of it in terms of the journey. So, discarding everything you know about Cloudstate. Cloudstate offers a way for you to manage entities in a serverless world. An Entity is state with an identifier, they are durable and strongly consistent. The word Entity doesn't need to describe this because it's a word that we have introduced and defined. So, Hello World in Cloudstate will demonstrate using an Entity. It doesn't need to be distinguished from anything because it is the first concept we introduce to people, it's the base thing.
Now, as we move along the Cloudstate journey, a user might need more power, they might want to be able to build views based on an event stream. So, they can use EventSourcedEntity. This is a specialisation of Entity, it's an entity that rather than just storing its state, it stores the events that led to the current state. It's definition is in contrast to Entity. Or, they might need entities that are highly scalable and available, and so they might consider ReplicatedEntity, this is an entity that gets replicated across the cluster using CRDTs.

@jroper @pvlugter @viktorklang @marcellanz Thinking in terms of the journey, I usually like to have at least a vague idea of what it is that I am learning and implementing. My first contact with any technology should cause as little cognitive stress as possible for me to be interested. For example when following the journey at some point I would run into EventSourced or Replicated entities (which would lead me to ask how it is replicated, but so far so good), and then the question would come to mind, and the first entity of the journey that I did I implement what it was about? Once I have a clear, concise and objective name, just looking at sample code would be enough for me to understand the basics, or guess about them before I even read anything.
I think everything is about what we want to say to the user, if we really want to hide the infrastructure ok, forget about EventSourced, Replicated and think in other terms because if we want to hide none of that matters, in another way if we want to demonstrate the benefit of using some technologies and techniques and show how much Cloudstate allows transparency in the development of these technologies so we have to make it clear what are the technologies, techniques, and behaviors that we use in an underlying way.
All that said, DurableEntity remains generic to me. It is assumed that any type of entity used would be durable, persistent, stored in some physical medium at the end of the day, and it is actually a surprise when we discover that CRDTS are not yet persistent.
Giving names is hard!

I still like using just Entity as the base concept, with EventSourcedEntity and ReplicatedEntity as specialisations, which replace the simple state storage with more sophisticated state models.
Unless we introduce an even simpler base Entity, which isn't backed by persistent storage, and where the state is just in-memory in the user function and will be gone when the entity passivates after receive timeout. There are use cases for this too, which effectively provides an in-memory cache with TTL. So if we had an Entity like this, equivalent to a sharded actor in Akka, then I would see that as the base thing, and then DurableEntity or PersistentEntity as a specialisation too.

It looks like you just defined a use case and a new entity. So I don't see how to leave what we're discussing without a name, be it DurableEntity, PersistentEntity, or MutableEntity (I vote for the latter)

The durability of replicated entities will hopefully some day be a configuration property of them... ie, there won't be DurableReplicatedEntities and ReplicatedEntities, nothing fundamentally changes about them, the model is still identical. I think the same might be true about regular entities, we could have entities that are only stored in memory, it wouldn't change the state model, protocol or programming model, just the implementation in the proxy.
The point here is that the durability has nothing to do with the abstract concepts exposed by the state model, it's an implementation detail, whether they were durable or not would not impact anything about the protocol or programming model. So I don't think it makes sense to include durable in the name. MutableEntity is weird to me... would you ever have an ImmutableEntity? If I think of entity, I think of something that is inherently mutable. Also, event sourced entities and replicated entities are both mutable too, so if the fact that its mutable is important enough to expose in the name we should also call it MutableEventSourcedEntity and MutableReplicatedEntity.
Let me phrase all this a different way... what if, in the first proof of concept in Cloudstate, we didn't support event sourcing or CRDTs, and we only had one entity type, called Entity, which was just a blob identified by a key that could be updated. Would the name Entity make sense in that context? I think it would, we've only got one type of entity, it's a very fundamental concept that is intuitively understandable I believe... but if it's not, there is a whole paper written about these entities, Pat Hellands Life Beyond Distributed Transactions, which was first published in 2007 and has actually been foundational in influencing the development of both Akka (I challenge you to find a talk given by Jonas Boner where he doesn't quote Pat Helland) and Cloudstate. That paper just calls them entities, and I think these entities are the simplest incarnation of the entities described in that paper. And I believe Cloudstate is essentially attempting to provide a serverless implementation of the concepts described in that paper.
Now, sometime later, we decided that these entities are not enough, and we should introduce new entity types... replicated and event sourced. Would we consider changing the name of entity then? I don't think so... I think event sourcing, and CRDTs, are both concepts that build on the concept of entity. They don't change it, an entity is still an entity, they add to it. And therefore, entity does not need to be differentiated from them, they need to be differentiated from entity in terms of what they add to it.

The durability of replicated entities will hopefully some day be a configuration property of them... ie, there won't be DurableReplicatedEntities and ReplicatedEntities, nothing fundamentally changes about them, the model is still identical. I think the same might be true about regular entities, we could have entities that are only stored in memory, it wouldn't change the state model, protocol or programming model, just the implementation in the proxy.

Yes, agree with this and was also going to mention it. I also see durability as being a configuration setting for replicated entities, and same for whether entity state is persistent or in-memory. We shouldn't have a new entity type for different configurations or backends, only for different models.

Agree that in the scenario of what if Cloudstate had started with a simple Entity and then progressed from there, that we would've called that Entity originally and then kept the name when specialised event sourced and replicated entities are added. I also think it works from a user's perspective to have the base concept of an Entity and then build on it with event sourcing and replication. It seems understandable and natural to me.

I agree with this story. I think it's consistent and having the most basic state model entity called Entity makes sense. It's also well known what properties an entity by itself is expected to fulfil. Having Cloudstate promoting these semantics makes it even stronger.
Regarding a distributed cache. I had similar thoughts having an entity immutable and configured to be limited in its lifetime which could be useful. Both properties, as said, can be orthogonal to their state model behaviours.

The durability of replicated entities will hopefully some day be a configuration property of them... ie, there won't be DurableReplicatedEntities and ReplicatedEntities, nothing fundamentally changes about them, the model is still identical. I think the same might be true about regular entities, we could have entities that are only stored in memory, it wouldn't change the state model, protocol or programming model, just the implementation in the proxy.

I agree and it is a really good point. All those capabilities (storage in memory, persistence storage, etc ...) don't change the state model. From the perspective of the user journey it also makes sense. I think most developers know what to expect from an Entity.
The project is evolving and it would make sense to have a more broadly discussion about Names in Cloudstate. We already have an issue (https://github.com/cloudstateio/cloudstate/issues/395) for that and we should continue there.

Regarding a distributed cache. I had similar thoughts having an entity immutable and configured to be limited in its lifetime which could be useful. Both properties, as said, can be orthogonal to their state model behaviours.

I also find this use case interesting. Similar to what Redis does

I agree that James' story is good, the defined Entity semantics makes sense, but I still think that we are not talking about a persistence mapping framework, we are talking about Serveless and I would like to know how the proxy is handling my "Entity" , this is clear with EventSource, it is more or less clear with CRDT or Replicated, but it is not clear with Entity, it still does not demonstrate the behavior of this basic Entity. But if everyone agreed with Entity then we stay with Entity.
As long as everything is well documented ok.

I think my main problem with Entity is that as long as the proposed API
requires the user to explicitly get and update or delete—that it is
apparent to the user that they have to do that—so they don’t expect the
platform to do it automagically.--
Cheers,
√

I think my main problem with Entity is that as long as the proposed API requires the user to explicitly get and update or delete—that it is apparent to the user that they have to do that—so they don’t expect the platform to do it automagically.-- Cheers, √

What do you propose as an API?

I need support here, I have integrated the new entity in circleci and the tests failed. I cannot understand the issue and I was not able to reproduce it locally by accessing postgres. Here is one of the failed tests (https://app.circleci.com/pipelines/github/cloudstateio/cloudstate/615/workflows/d08f21ba-5294-4ec0-8819-bc0fcb5813b6/jobs/2250).
@pvlugter could you please support me here? Thanks

The error in the failed test is:
InsertOrUpdate is not supported on a table without PK.
slick.SlickException: InsertOrUpdate is not supported on a table without PK.
	at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl$InsertOrUpdateAction.<init>(JdbcActionComponent.scala:560)
	at slick.jdbc.JdbcActionComponent$InsertActionComposerImpl.insertOrUpdate(JdbcActionComponent.scala:478)
	at io.cloudstate.proxy.valueentity.store.JdbcValueEntityQueries.insertOrUpdate(JdbcValueEntityQueries.scala:35)
	at io.cloudstate.proxy.valueentity.store.JdbcStoreImpl.update(JdbcStore.scala:80)
	at io.cloudstate.proxy.valueentity.store.JdbcStoreImpl.update(JdbcStore.scala:65)
	at io.cloudstate.proxy.valueentity.store.JdbcRepositoryImpl.update(JdbcRepository.scala:85)
	at io.cloudstate.proxy.valueentity.ValueEntity.io$cloudstate$proxy$valueentity$ValueEntity$$performAction(ValueEntity.scala:375)
	at io.cloudstate.proxy.valueentity.ValueEntity$$anonfun$io$cloudstate$proxy$valueentity$ValueEntity$$running$1.$anonfun$applyOrElse$1(ValueEntity.scala:311)
	at scala.Option.map(Option.scala:242)
	at io.cloudstate.proxy.valueentity.ValueEntity$$anonfun$io$cloudstate$proxy$valueentity$ValueEntity$$running$1.applyOrElse(ValueEntity.scala:310)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at io.cloudstate.proxy.valueentity.ValueEntity.aroundReceive(ValueEntity.scala:168)

So it's missing primary key. The logs show this for creating the value entity table:
2020-10-29 17:29:28.272 DEBUG io.cloudstate.proxy.jdbc.EnsureValueEntityTablesExistsActor - Creating table, executing: create table "value_entity_state" ("persistent_id" VARCHAR(255) NOT NULL,"entity_id" VARCHAR(255) NOT NULL,"state" BYTEA NOT NULL)

While the event sourced journal create table is:
2020-10-29 17:29:28.272 DEBUG io.cloudstate.proxy.jdbc.EnsureTablesExistsActor - Creating table, executing: create table "journal" ("ordering" BIGSERIAL NOT NULL,"deleted" BOOLEAN DEFAULT false NOT NULL,"persistence_id" VARCHAR(255) NOT NULL,"sequence_number" BIGINT NOT NULL,"message" BYTEA NOT NULL,"tags" VARCHAR(255)); alter table "journal" add constraint "journal_pk" primary key("persistence_id","sequence_number"); create unique index "journal_ordering_idx" on "journal" ("ordering")

I expect it should be reproducible locally, by running minikube and the scripts in the same way as the integration tests.

Thanks. I will try to reproduce it locally on minikube but I was not successful yet.

I was not able to reproduce it locally (Mac OSX Catalina) on minikube. I have this on minikube.
event sourcing
2020-10-30 11:35:01.282 DEBUG io.cloudstate.proxy.jdbc.EnsureTablesExistsActor - Creating table, executing: create table "journal" ("ordering" BIGSERIAL NOT NULL,"deleted" BOOLEAN DEFAULT false NOT NULL,"persistence_id" VARCHAR(255) NOT NULL,"sequence_number" BIGINT NOT NULL,"message" BYTEA NOT NULL,"tags" VARCHAR(255)); alter table "journal" add constraint "journal_pk" primary key("persistence_id","sequence_number"); create unique index "journal_ordering_idx" on "journal" ("ordering")

entity
2020-10-30 11:35:01.297 DEBUG io.cloudstate.proxy.jdbc.EnsureValueEntityTablesExistsActor - Creating table, executing: create table "value_entity_state" ("persistent_id" VARCHAR(255) NOT NULL,"entity_id" VARCHAR(255) NOT NULL,"state" BYTEA NOT NULL); alter table "value_entity_state" add constraint "value_entity_state_pk" primary key("persistent_id","entity_id")

I am not able to build a native-image locally. I don't have enough resource and it just take too long on my machine. I would to be able to investigate this my self, but unfortunately it is just not possible. I still need support here. @pvlugter, please take a look.

@ralphlaude, I've pushed some changes and tests are green now.
I merged in latest master, mainly for the updates to the integration script tests. In particular, the deployment scaling wasn't working, so that tests were sometimes running with terminating pods. And I've also applied these changes to the separate shopping cart test script for value entities.
For the native-image version not setting the primary key, this was missing some native-image configuration for the table class. As a reference, there's config for the journal table.
The in-memory smoke test was also failing, as the request to delete entities was incorrect. Fixed this.
And then I had tests timing out because they hit connection limits for postgres. For now, I've reduced the max pool size from 20 down to 5. Might be hitting the default 100 connections limit because there are both multiple nodes (test creates a 3 node cluster) and there are multiple connection pools (for both event sourced and value entities)? Haven't confirmed this, but something to look at.
Also, something else to look at and create a test for: if an entity is passivated, it fails when recreated:
ERROR akka.actor.OneForOneStrategy - Unexpected Value entity failure - Value entity unexpected failure: Unable to find descriptor for type:
fooA foo

java.lang.Exception: Unexpected Value entity failure - Value entity unexpected failure: Unable to find descriptor for type:
fooA foo

    at io.cloudstate.proxy.valueentity.ValueEntity.io$cloudstate$proxy$valueentity$ValueEntity$$crash(ValueEntity.scala:245)
    at io.cloudstate.proxy.valueentity.ValueEntity$$anonfun$io$cloudstate$proxy$valueentity$ValueEntity$$running$1.applyOrElse(ValueEntity.scala:333)
    at akka.actor.Actor.aroundReceive(Actor.scala:537)
    at akka.actor.Actor.aroundReceive$(Actor.scala:535)
    at io.cloudstate.proxy.valueentity.ValueEntity.aroundReceive(ValueEntity.scala:168)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
    at akka.actor.ActorCell.invoke(ActorCell.scala:547)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
    at akka.dispatch.Mailbox.run(Mailbox.scala:231)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

@pvlugter thanks for your support!
I will take a look at the connection pool for postgres and see what can be done here for the entity and the eventsourced entity. It would be a good idea to make the connection pool size configurable and have a specific configuration for the tests. It can also be done later.
Good catch, I will be checking the reactivation after passivation.

We had a lot of discussions for the naming :). Entity seems to be widely accepted.
ANNOUNCEMENT!!! Entity will be used as name unless they are still some concerns.
Ping! @pvlugter, @jroper, @viktorklang, @sleipnir, @marcellanz

ANNOUNCEMENT!!! Entity will be used as name unless they are still some concerns.

@ralphlaude No concerns from me.

We had a lot of discussions for the naming :). Entity seems to be widely accepted.
ANNOUNCEMENT!!! Entity will be used as name unless they are still some concerns.
Ping! @pvlugter, @jroper, @viktorklang, @sleipnir, @marcellanz

What most decide for me is fine

@ralphlaude Entity is more than OK from me.

@pvlugter the renaming is done, please take a look and give feeback.

So this initial work is basically all done now? Needs to be based on latest master.

@pvlugter. thanks for the feedback. Yes, the main work is done here. The only thing left is to make the value entity as default and adapt the the health-checks/readiness-checks in the accordingly in the proxy JDBC module. This means change the way the JDBC tables creation are done.

@pvlugter now the initial work is all done. This can be changed to a PR and reviewed again if needed.

@pvlugter thanks.
The main issue here now is how to make the creation of the store pluggable or dynamic. Using the config or passing the factory around sounds good. I need more insights here.

@ralphlaude, I've added some minimal changes to dynamically create stores, and moved the JDBC store under the JDBC proxy.

@pvlugter the end goal is achieved here. Please take also a look.

cool. @ralphlaude awesome work 🎉

It was a long and interesting journey with all the problems and insights on the way. Thanks @pvlugter, @jroper, @viktorklang, @sleipnir and  @marcellanz for the support, the discussions and the feedback. It was all fun :). CRUD was very CRAZY! :).
@pvlugter yes, let continue with the remaining work.

Wooohooo! Thanks @ralphlaude for your extreme perseverance! Awesomely done :)

We who thank you for your commitment. Now let's take advantage of what you have done and build lots of examples and friendly documentation for our users. Thanks for the work @ralphlaude !


Error during cockpit install causing RetroNAS to stop working properly

Details
Hi guys!

DNS: Temporary failure resolving 'deb.debian.org'
Addressing: the MY_IPS var is populated on the fly from ip -4 -br a

that's exactly the issue
but idk why this is happens only after install the cockpit :(
after every boot, i need to manually restart the network to get it work again

what Git Branch are you on, main? It should tell you on the retronas main menu.
i have no idea how truenas vm networking works, so if its at that level its not an issue with retronas.
other than that check for network startup failure messages

sudo dmesg
sudo systemctl status networking
ip a

if you are on testing branch because we are testing new features that may be in conflict

systemctl status NetworkManager

As far I know, I'm using the main branch.
I will check these messages tomorrow and post here 😀


[💡FEATURE REQUEST]: Split monolitic http plugin into the http/http2/fcgi plugins

At the moment, we have a big monolithic http plugin which consists of three servers:

HTTP server
HTTP2 server
FCGI server

We need to split these responsibilities into three separate plugins. In the first stage, all three plugins will have a single configuration (so no changes for the configuration from the user perspective). But http will show a warning that in the future http config section will be split into 3 separate sections.
Pros:

Easy to update a single plugin w/o affecting other bundled sub-plugins.
Easy to have different implementations of http, for example (fasthttp, stdhttp, fiber, and so on).
Good for the dependencies. If, for some reason, a user needs to have a dependency on the HTTP plugin, it only requests the http plugin, but not the whole bundle.

Cons:

Breaking change for the .rr.yaml [!]
http, https, fcgi shared some code from the bundle between those plugins. When we split the bundle, we have to duplicate those handlers (or might create a separate package within the pkg folder)

done


Support ScanCode 31.0.0

ScanCode version 31.0.0 changes the output data structure, which will require some adjustments in ORT:
https://github.com/nexB/scancode-toolkit/blob/v31.0.0rc2/CHANGELOG.rst#important-api-changes
The adjustments for these changes should be done transparently, and ORT's ScanCode implementation should still work with older ScanCode versions.

We should probably drop support for ScanCode < 3 before that to simplify the code. I started working on this in the drop-sc-before-3 branch.

We should probably drop support for ScanCode < 3 before that to simplify the code. I started working on this in the drop-sc-before-3 branch.

This has been done in https://github.com/oss-review-toolkit/ort/pull/5700.
And meanwhile ScanCode 31.1.0 is available.

ScanCode version 31.0.0 changes the output data structure, which will require some adjustments in ORT

Actually https://github.com/oss-review-toolkit/ort/pull/6105 shows that no changes on ORT side are needed, probably because we do not deserialize the full data model but only iterate over needed JSON nodes.

For some reason the tests passing in #6105 was bogus, the copyright result format actually did change slightly.


One time link generated for filling in an interactive form instead of chatting

Generate a one-time link (ie: long, random url param) for accessing a form to add a talk that way instead of having to get the chat syntax exactly correct.

Fixed in a7ceee0


jackett: 0.16.2152 -> 0.16.2236

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/Jackett/Jackett/releases.
meta.description for jackett is: "API Support for your favorite torrent trackers"
meta.homepage for jackett is: "https://github.com/Jackett/Jackett/"
meta.changelog for jackett is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 0.16.2236 with grep in /nix/store/nrskggw17y2g121wkxd7brjww6yfkhx9-jackett-0.16.2236

directory tree listing: https://gist.github.com/356f3aaafe3ed65a3e9daa3d518dfb4d

du listing: https://gist.github.com/da0ed443e8c6f6c1f46e9b3589c38119

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
jackett

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/nrskggw17y2g121wkxd7brjww6yfkhx9-jackett-0.16.2236 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A jackett https://github.com/r-ryantm/nixpkgs/archive/109bcd82b0d7ca81a2a36fa1aac1f6abe0cce694.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/nrskggw17y2g121wkxd7brjww6yfkhx9-jackett-0.16.2236
ls -la /nix/store/nrskggw17y2g121wkxd7brjww6yfkhx9-jackett-0.16.2236/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review 1

1 package built:

jackett

Maintainer pings
cc @edwtjo @nyanloutre @purcell for testing.

Result of nixpkgs-review pr 104996 run on x86_64-darwin 1

1 package built:

jackett


Storage tests improvement

Improved coverage of StandStorage and RecordStorage. These classes had sufficient coverage because of adjacent tests that included storage usage, but their implementations (e.g. gae-java storages) were not covered since they didn't have enough own tests.
Added FieldMasks#maskOf for fast field mask creation.

@armiol PTAL

Current coverage is 80.57% (diff: 100%)

Merging #211 into master will increase coverage by 0.08%

@@             master       #211   diff @@
==========================================
  Files           147        147
  Lines          5427       5436     +9
  Methods           0          0
  Messages          0          0
  Branches        597        599     +2
==========================================
+ Hits           4368       4380    +12
+ Misses          948        946     -2
+ Partials        111        110     -1

Powered by Codecov. Last update 9601e16...ba9cdb5

@armiol PTAL again


unclear how to delete DMs

[x] I've searched the Issues
[x] I've read the basic concepts
[x] I'm using the latest version

Deleting a message seems not to work
Im trying to delete all messages from a thread (basically the whole thread). When I run my code I get an error saying that there is no method deleteItem() on the thread, but in the sourcecode I found it here
// load threads
let directInbox;
let threads;
(async () => {
  directInbox = await ig.feed.directInbox(loggedInUserId);
  threads = await directInbox.items();
})();
console.log(threads[0].items.length); // => 10 (indicating that items exist)

// trying to delete all items of the first thread
threads[0].forEach(item => {
  console.log(item.item_id); // item_id is being logged
  threads[0].deleteItem(item.item_id);
});

ERROR MESSAGE
Thrown:
TypeError: threads[0].deleteItem is not a function

Am I doing something wrong? Thanks for any help, much appreciated!

You have to use the direct entity.
Assuming you're in an async context:
const [ firstThread ] = await ig.feed.directInbox().items();
// to delete everything you'd have to get the directThread-feed
await Promise.all(
  firstThread.items.map(({ item_id }) =>
    ig.entity.directThread(firstThread.thread_id).deleteItem(item_id)));

awesome thanks a lot!


Implement dependabot in all repositories

We should plan issues for the repositories to implement dependabot.

requirements*.txt
GitHub Actions

@rdimitrov, I guess this one can be closed, right?

How about we close this one with the one about the updating the Python dependencies automatically?

I have a draft PR on that but I have to test it 👍


proc-macro derive panicked on enum with discriminant

Slightly contrived MWE:
struct Baz {
    val: u8,
}

const VALUE: Baz = Baz { val: 1 };

#[repr(u8)]
#[derive(Deserialize)]
enum Foo {
    Bar = VALUE.val,
}

This fails to compile with
proc-macro derive panicked

help: message: called `Result::unwrap()` on an `Err` value: ParseError(Some("failed to parse derive input: failed to parse"))

Thanks, I released syn 0.14.3 with a fix. We limit what expressions are allowed in discriminants to save time in compiling the parser, but VALUE.val seems common enough and easy to parse that we can allow it.


Fixes #13769 Improved documentation for getting first unread messages

Based on the changes made here 7bf3312 , we now have:

Argument
Example
Required
Description

anchor
"first_unread"
Yes
Fetches first unread message when first_unread is given as value.          Other special values like 'newest' and 'oldest' are also supported for anchoring the query at the most recent or oldest messages.

Fixes #13769

@Rodriq can you address the CI failures/alerts and let me know when it's ready for a review?

So the CI failure is because we need to detail use_first_unread_anchor as intentionally_undocumented=True since it's now a supported but deprecated option.  git grep intentionally_undocumented will show you how to do that.
I suppose an alternative solution would be to still include it in the docs, but just change the text to make clear that it's deprecated.  Might be a better answer for now.

@timabbott Followed the latter and it helped it. Thanks!

@Rodriq sorry I lost track of this, and then I accidentally fixed this in b29213ef9096df6d62cfaa6f8b57ef532ed10a78 without having reviewed this.  At this point, it makes sense to close.


404 render error

Hello,
I am running into an issue where I get an error when it tries to render the 404. Is there still something I need to configure to get it working? Or is this a bug?
Steps to reproduce
hugo new site demosite
cd demosite/
mkdir -p themes/hugo-geekdoc/
curl -L https://github.com/thegeeklab/hugo-geekdoc/releases/latest/download/hugo-geekdoc.tar.gz | tar -xz -C themes/hugo-geekdoc/ --strip-components=1

Adding the following to hugo.toml.
baseURL = "http://localhost"
# languageCode = 'en-us'
title = "test"
theme = "hugo-geekdoc"

pluralizeListTitles = false

# Geekdoc required configuration
pygmentsUseClasses = true
pygmentsCodeFences = true
disablePathToLower = true

# Required if you want to render robots.txt template
enableRobotsTXT = true

hugo new content _index.md

hugo
Start building sites …
hugo v0.122.0-b9a03bd59d5f71a529acb3e33f995e0ef332b3aa+extended darwin/amd64 BuildDate=2024-01-26T15:54:24Z VendorInfo=gohugoio

Total in 80 ms
Error: error building site: render: render of "404" failed: "/Users/marco/Public/demosite/themes/hugo-geekdoc/layouts/404.html:18:9": execute of template failed: template: 404.html:18:9: executing "404.html" at <partial "site-header" (dict "Root" . "MenuEnabled" false)>: error calling partial: "/Users/marco/Public/demosite/themes/hugo-geekdoc/layouts/partials/site-header.html:16:60": execute of template failed: template: partials/site-header.html:16:60: executing "partials/site-header.html" at <.Root.Site.Home.Permalink>: error calling Permalink: runtime error: invalid memory address or nil pointer dereference

Hello, which hugo version do you use?

0.122. You can see it at the last command.

Thanks missed that. The issue is, that the empty home page created by hugo new content _index.md has draft = true set by default. You can either use hugo --buildDrafts or remove the draft flag from the home page.

Thank you for the update. I did not try to change or remove each of the lines in the header.
But what you wrote matches then with what I saw last night, when I created new content while I had enableGitInfo = true set in the config. I got a different, but the _index.md was created with a yaml header and the draft option was not there.
After setting enableGitInfo to false to get rid of the git error, the hugo run was fine.
Is it more of a hugo problem, or still something that can be fixed in the theme?

This is just an intentional error, I would say. In the theme templates, we use the Hugo built-in variable .Site.Home.Permalink. If no home page exists (e.g. because it is marked as draft and the build was created without draft pages), an error occurs. We can try to catch this error in the theme, but a project without a home page is very unlikely, and we would suppress an error silently that would help users to detect the missing home page.

I have not much experience with Hugo yet. Just stared using it and all files that were created had draft. Did not think about changing it yet.
Based on that, I guess one only runs into that problem at the very beginning, when trying to build with all the defaults.
Thank you very much again for the quick response.
Feel free to close the issue.

Fair point. I think we can at least improve the getting started documentation and explain why we use -D flag in step 5 and what's required for the theme to work.


Adjustable amount of reduction

Hello Jean-Marc! This is probably something that you already were planing but completely remove noise during low SNR portions of the sound doesn't always sounds good. Easily fixable in my plugin (https://github.com/lucianodato/speech-denoiser) but something you should consider for the library.
By the way I'm having troubles making it work in my plugin, does the library support 32 bit floats or do I have to cast them down to 16 bits???

@lucianodato
RNNoise currently has a fixed frame size. At first glance, your implementation doesn’t account for that. Maybe I overlooked something.

Such options would be great!

Thanks for the sharing! What have you done to the RNNoise to fix the low SNR noise reduction problem?

I can fix that problem introducing an adjustable reduction but in my plugin not in the library itself. But I can't get it to work. I believe it something related to the data I'm feeding it with but I'm not sure.

@lucianodato
Did you try the sample code? You can also give this a spin:
https://gist.github.com/JanX2/b4d283b0ab83986b16315cf2817ee50d

@JanX2 I'm mostly interesting in realtime usage of the library or at least building on top of the library for that use case. The example code works offline and also yours (by the way are you an audacity dev?).  I'm developing an lv2 plugin based on this library because is much better than what I previously developed myself for adaptive speech noise reduction.

@lucianodato
In such cases, I use a ring buffer to decouple the samples counts produced and consumed.

Hi! I have read your paper, and i believe it is a good idea. So i want use my noise to train in your way. But i met some strange problem. Can you please provide the python version、 keras version to me. And i think the dump_rnn.py is for model reading?

@liuanping Hi maybe that comment should be in another issue instead of this one.
@JanX2 I had to implement better buffering, yes. I'm using a delay block now and that's lazy I know.
Anyways I understand that this library is for webrtc and codecs but adding support for 32 bits floats and any arbitrary block size would be great (maybe I should file another issue for that).

This library isn’t for WebRTC AFAIK. @jmvalin can probably tell us more about what he want’s to use it for, but the blog post is probably the best source of information about RNNoise for now:
https://people.xiph.org/~jm/demo/rnnoise/

I wasn't really considering a floor, but it's something that would be easy. In the demo files, some people made me realized that the denoised files sometimes sound a little too "dry", so I just checked in a change to make the gain decay less abrupt. Let me know if that reduces the need for a floor on the attenuation.

Well there is a fork of rnnoise that already has this https://github.com/GregorR/rnnoise-nu. I'm using that now.

How do i use GregorR's? It says Model not found..

@jmvalin the denoised files for my use case are very choppy / dry. Any suggestions on where to adjust?
@lucianodato does GregorR's fork fixes this?


Update github action name for ruby

Description:
Update github action name for ruby -> actions/setup-ruby@v1 is deprecated

https://github.com/actions/setup-ruby
Please, migrate your workflows to the [ruby/setup-ruby](https://github.com/ruby/setup-ruby), which is being actively maintained by the official Ruby organization.

Changes in workflow concern ruby/setup-ruby@v1 which comes from verified creator and has proper form.

Hi @msobo1 - Its now updated to a fixed (1.126.0) latest stable version of ruby/setup-ruby.
Please let us know if we need to fix it to a precise version here - they seem to be releasing newer versions every few days - https://github.com/ruby/setup-ruby/tags and we might be missing newer updates if we fix the version here ?


LCD backlight stays on while charging / waiting for SOLAR

Describe the bug
The LCD keeps on while charging, or while waiting for sun in SOLAR mode. The backlight timer keeps stays 120 secs.
Upload your config
{"version":"v3.6.6","serialnr":7376,"mode":"SOLAR","mode_id":2,"car_connected":true,"wifi":{"status":"WL_CONNECTED","ssid":"SD-IoT","rssi":-79,"bssid":"***omitted**"},"evse":{"temp":31,"temp_max":65,"connected":true,"access":true,"mode":2,"loadbl":0,"pwm":1024,"solar_stop_timer":0,"state":"Charging Stopped - Waiting for Solar","state_id":9,"error":"None","error_id":0,"rfid":"Not Installed"},"settings":{"charge_current":130,"override_current":0,"current_min":6,"current_max":13,"current_main":25,"current_max_circuit":16,"current_max_sum_mains":600,"max_sum_mains_time":0,"solar_max_import":0,"solar_start_current":6,"solar_stop_time":5,"enable_C2":"Always Off","mains_meter":"API","starttime":0,"stoptime":0,"repeat":0},"mqtt":{"host":"***omitted**","port":1883,"topic_prefix":"SmartEVSE-7376","username":"wouter","password_set":true,"status":"Connected"},"home_battery":{"current":0,"last_update":0},"ev_meter":{"description":"Eastron3P","address":12,"import_active_power":0,"total_kwh":45.90000153,"charged_kwh":0.5,"currents":{"TOTAL":0,"L1":0,"L2":0,"L3":0},"import_active_energy":45.90000153,"export_active_energy":0},"mains_meter":{"import_active_energy":0,"export_active_energy":0},"phase_currents":{"TOTAL":43,"L1":2,"L2":3,"L3":38,"last_data_update":1721938616,"original_data":{"TOTAL":43,"L1":2,"L2":3,"L3":38}},"backlight":{"timer":120,"status":"ON"}}

Upload a debug log
N/A
To Reproduce
Put SmartEVSE in charging mode, or in SOLAR mode while waiting for sun
Backlight stays on forever
Expected behavior
Backlight time should count down from 120 to 0 and turn off.
Screenshots
N/A

Thanks for your report!
This is intended behaviour; when Waiting For Solar, the backlight is supposed to stay on, as a "warning" that no normal charging is going on.
I even flashed the original manufacturers firmware and that also behaves like this; it has always been like this.


Want to add bottom up button to website

Description
I Want to add bottom up button to website such that user can go to the top of the website without scrolling (Just Clicking is required).
It will users a better impression for the site.
bottom up button will appear when user cover web scrolling of 100vh
please assign this issue to me, under GSSOC2023
Screenshots

🥦 Browser
Google Chrome
Checklist ✅

[X] I checked and didn't find similar issue
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (blank for no)

Hi @Lobi29 , you just checked this :

But in #696 , this is already been worked on.
Please CHECK PROPERLY always.
Thanks.


la-go-live

Docs Build status updates of commit c028ac7:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/messaging-extensions/how-to/action-commands/define-action-command.md
:bulb:Suggestion
View
Details

msteams-platform/TOC.yml
:white_check_mark:Succeeded
View

msteams-platform/apps-in-teams-meetings/create-apps-for-teams-meetings.md
:white_check_mark:Succeeded
View

msteams-platform/apps-in-teams-meetings/teams-apps-in-meetings.md
:white_check_mark:Succeeded
View

msteams-platform/assets/images/ask-away-app.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/buttons.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/chiclets-and-pills.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/selection-controls.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/tabs/mobile-app-drawer.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/tabs/mobile-tab-grid.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/tabs/mobile-tab-list.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/tabs/mobile-tab-single-canvas.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/tabs/mobile-tab-with-bot.png
:white_check_mark:Succeeded
View

msteams-platform/assets/images/tabs/mobile-tab.png
:white_check_mark:Succeeded
View

msteams-platform/bots/how-to/add-power-virtual-agents-bot-to-teams.md
:white_check_mark:Succeeded
View

msteams-platform/build-your-first-app/build-first-app-overview.md
:white_check_mark:Succeeded
View

msteams-platform/concepts/deploy-and-publish/appsource/publish.md
:white_check_mark:Succeeded
View

msteams-platform/index.yml
:white_check_mark:Succeeded
View

msteams-platform/samples/app-templates.md
:white_check_mark:Succeeded
View

msteams-platform/tabs/design/tabs-mobile.md
:white_check_mark:Succeeded
View

msteams-platform/whats-new.md
:white_check_mark:Succeeded
View

msteams-platform/messaging-extensions/how-to/action-commands/define-action-command.md

Line 4, Column 14: [Suggestion-duplicate-descriptions] Attribute 'description' with value 'An overview of messaging extensions on the Microsoft Teams platform' is duplicated in 'messaging-extensions/how-to/action-commands/define-action-command.md(4,14)', 'messaging-extensions/what-are-messaging-extensions.md(4,14)'. NOTE: This Suggestion will become a Warning in late October, 2020.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


This pages inconsistently uses terms "toolset" and "VS version"

For example, in this paragraph:

All object files and libraries compiled using /GL and /LTCG must use exactly the same toolset for the compile and the final link. For example, code built by using /GL in the Visual Studio 2019 version 16.7 toolset can't be linked to code built by using /GL in the Visual Studio 2019 version 16.8 toolset. The compiler emits Fatal error C1047.

, it says Visual Studio 2019 version 16.7 toolset and Visual Studio 2019 version 16.8 toolset, which lists Visual Studio versions and not toolsets, which use their own numbering nomenclature - The toolset version is v140 for Visual Studio 2015, v141 for 2017, v142 for 2019, and v143 for 2022.
This page should clearly specify that even for the same toolset, such as v142, code compiled with compilers from different VS versions, such as 16.7 and 16.8, the linker will issue these errors.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: fcccbfe4-34d9-67cb-efbb-6c15ff35b38e
Version Independent ID: cc082cbf-a967-6709-f3ef-190e78cf84e9
Content: C++ binary compatibility 2015-2022
Content Source: docs/porting/binary-compat-2015-2017.md
Product: visual-cpp
Technology: cpp-language
GitHub Login: @corob-msft
Microsoft Alias: corob

Moved this issue to an internal work item tracking system because we are moving away from using GitHub issues to track C++ doc work.


Feature Request: assertThat(file).doesNotHaveContent("string")

Currently it is possible to conveniently check a file's content with hasContent, but not the opposite.
Also I'm regularly bumping into the absence of the possibility to directly analyse files as it is possible for strings, so to have something analogous to assertThat("String").contains & doesNotContain. Of course one can do assertThat(FileUtils.readFileToString(file)) but having syntactic sugar in assertJ to avoid this would be lovely as the brevity and clarity assertJ enables is one of it's main strengths IMHO.
These last two may be a separate feature request, however i wanted to gather general feedback about feasibility first.

Fair enough, so adding hasContentContaining and hasContentNotContaining would be enough for you?
Another possibility would be to add a content() method that would be get the content and return string assertions on it, that would avoid an explosion of hasContentXxx methods and provide all string assertions:
Ex:
assertThat(file).content()
                .contains("foo")
                .matches(".*d")
                .endsWith("bar")

I prefer the content() option as I'm sure once you have hasContentContaining you will request hasContentMachcing(regex) 😆

Nope, we would have to add content() to path to, would be better to create a different issue though

Hi @joel-costigliola and @Philzen, I am sorry for the late reply since I was really busy this week. Currently, I have completed the basic functions for this feature, but the tests are still. I wonder whether should I submit a draft PR or wait for the tests completed.

@wtd2 nothing criminal about doing a draft PR 😉

@wtd2 no problem at all, we all do this on our spare time and I happens quite frequently that I don't have enough time to.

Nope, we would have to add content() to path to, would be better to create a different issue though

@joel-costigliola Happy to do that, but i wanted to mention a suggestion that i've filed which would basically include that (#2205). Would love to hear what you think about it.

Hi @joel-costigliola and @Philzen, I am sorry for the late reply since I was really busy this week.

@wtd2 Pls don't beat yourself up over this – i'm very grateful for somebody taking this up! The world of assertJ has not missed it until now, and i'm sure it can wait a little longer.

@wtd2 are you still keen to finish up this one? it's ok if you don't have time, we can finish it (you'll get the credit)


feat: equation extraction

Describe the solution you'd like
The equations in the document are identified and extracted, probably in LaTeX
Additional context
Would be useful for scientific literature.

Hi @SimranJha2408 can you explain how this is now complete? I don't see anything in the release notes or any commits related to this. Is there anything documenting this feature?


Share to group PM by existing conversation, vs. selecting each participant from scratch

I have a few private group conversations on Zulip, to which I would like to share stuff from other apps (on Android). Currently every time I share (say an image) to Zulip, I have to individually find and select every member of that group in order for the image to get sent to the right place. If I forget someone then a new group conversation gets created, which is never what I want.
What I would like instead is that in the "Private Message" tab of the sharing dialog, I see a list of my current conversations so I can choose one. In the rare case where I want to share to a group that doesn't already exist, there can be a button to create a new one.

Related issue: #5057

Thanks @Nadrieril for the feedback! I agree this would be an improvement:

What I would like instead is that in the "Private Message" tab of the sharing dialog, I see a list of my current conversations so I can choose one. In the rare case where I want to share to a group that doesn't already exist, there can be a button to create a new one.

For anyone looking to take this issue on: there are existing data structures describing the user's recent PM conversations, so we can reuse those. See the existing PmConversationsScreen for an example.


Connecting to Failover Group does not list databases in Server Tree

Issue Type: Bug
When connecting to an Azure failover group, in the servers pane, when expanding the Databases folder in the tree, the "Failed to connect to server xxxxxx".
The databases are available and listed in the main window
Azure Data Studio version: azuredatastudio 1.4.5 (887f4e8985023602e599cf86bdb44df48bb28569, 2019-02-12T06:20:08.633Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7 CPU         930  @ 2.80GHz (8 x 2807)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: enabledvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Memory (System)
23.99GB (12.94GB free)

Process Argv
C:\Program Files\Azure Data Studio\azuredatastudio.exe

Screen Reader
no

VM
0%

Extensions: none

FYI, still occurring with 1.5.2

Hi @BroMarduk
Could you try setting "MultiSubnetFailover" property to "True"?
Related https://github.com/microsoft/azuredatastudio/issues/17901

Thanks @cheenamalhotra but the issue is not related to this as I CAN see the databases in the main pane window on the database tab, they just are not listed in the tree when expanding the Databases folder.
Seems to be related to the tree view only.


Coffee Maker - Functionality Limited

I have model C7CDAASP1 coffee maker.  I'm on the latest v0.5.0 and I've tried all other dev versions available on HACS.
The only functionality that is showing up is Sabbath Mode on/off switch.  I was hoping some of the app functionality would come through:

Brew Controls - Brew Strength, Brew Temp, Brew Size
Auto Brew - Set Schedule
Descaling - On/Off switch

I'll attach logs shortly.

GE Home Debug Log - 1.31.22.txt

Not sure if these logs are what you need.  Let me know if there is anything else I can do to help.

I'm having this issue as well. Not seeing any devices or entities related to the coffee maker

The coffee maker support isn't complete in HA.
However, if you pull down this repository and follow the directions in the readme (i.e. run the websocket_example.py found in the examples directory after creating the credentials file), you should be able to generate the logs that I'd need to confirm that the library can see the coffee maker appropriately and we can complete the HA implementation.
Hope this helps.

@simbaja can you help with my request?

@simbaja can you help with my request?

To which request are you referring?   The original request for coffee maker support?  If so, the logs provided don't really help because the version of the library that is being used in Home Assistant is older and doesn't support any of the coffee maker functionality.

I can't figure out how to update the code. That's my main issue right now. The files in the custom component folder in HA are completely different than these. I tried deleting and then uploading these manually but it didn't work.. As noted I couldn't find the repository on HACS

You can't upload this project directly into Home Assistant, this is a communication library for GE appliances, not a HA integration.  The integration should still be compatible with HACS, but there have been breaking changes in HA itself, by keeping HA fully up to date, it may make some integrations incompatible.  It's possible that may be happening, but you'd have to check your logs.  You also should be adding the ha_gehome repository, not the gehome repository into HACS.

Got it - sorry for all the basic questions. I know just enough to break things at this point.
Thanks,
Kale Sligh
VP, Marketing
o<PHONE_NUMBER>
m<PHONE_NUMBER>

From: simbaja @.>
Sent: Sunday, May 1, 2022 10:21:32 AM
To: simbaja/gehome @.>
Cc: kksligh @.>; Author @.>
Subject: Re: [simbaja/gehome] Coffee Maker - Functionality Limited (Issue #35)
You can't upload this project directly into Home Assistant, this is a communication library for GE appliances, not a HA integration. The integration should still be compatible with HACS, but there have been breaking changes in HA itself, by keeping HA fully up to date, it may make some integrations incompatible. It's possible that may be happening, but you'd have to check your logs. You also should be adding the ha_gehome repository, not the gehome repository into HACS.
—
Reply to this email directly, view it on GitHubhttps://github.com/simbaja/gehome/issues/35#issuecomment-1114264566, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AT3AQU5HGSS2CGZUDJEYNRLVH2OPZANCNFSM5NGRY36A.
  I used HA_GEHOME everything went easily!  Everything is working smoothly and the Coffee Marker errors are fixed.  Tons of functionality!