[SUPPORT] Need Assistance with Hudi Delta Streamer for Community Video

Hello All
firstly thank you very much for all help from community. i would want to mention i am new to delta streamer i have worked a lot with Glue jobs and i want to experiment with delta streamer so i can make videos and teach the community
i have setup complete pipeline from AWS Aurora Postgres  > DMS > S3 and i have EMR cluster 6.9 with Spark 3
Attaching links for sample parquet files and sample json how it looks like

Link to data files https://drive.google.com/drive/folders/1BwNEK649hErbsWcYLZhqCWnaXFX3mIsg?usp=share_link

Here is how i submit jobs
    spark-submit
    --master yarn
    --deploy-mode cluster
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer
    --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer /usr/lib/hudi/hudi-utilities-bundle.jar
    --table-type COPY_ON_WRITE
    --source-ordering-field replicadmstimestamp
    --source-class org.apache.hudi.utilities.sources.ParquetDFSSource
    --target-base-path s3://sql-server-dms-demo/hudi/public/sales
    --target-table invoice
    --payload-class org.apache.hudi.common.model.AWSDmsAvroPayload
    --hoodie-conf hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator
    --hoodie-conf hoodie.datasource.write.recordkey.field=invoiceid
    --hoodie-conf hoodie.deltastreamer.source.dfs.root=s3://sql-server-dms-demo/raw/public/sales/

Error i get
23/03/28 13:08:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/03/28 13:08:49 INFO DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at ip-172-32-147-4.ec2.internal/<IP_ADDRESS>:8032
23/03/28 13:08:50 INFO Configuration: resource-types.xml not found
23/03/28 13:08:50 INFO ResourceUtils: Unable to find 'resource-types.xml'.
23/03/28 13:08:50 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (6144 MB per container)
23/03/28 13:08:50 INFO Client: Will allocate AM container, with 2432 MB memory including 384 MB overhead
23/03/28 13:08:50 INFO Client: Setting up container launch context for our AM
23/03/28 13:08:50 INFO Client: Setting up the launch environment for our AM container
23/03/28 13:08:50 INFO Client: Preparing resources for our AM container
23/03/28 13:08:50 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
23/03/28 13:08:52 INFO Client: Uploading resource file:/mnt/tmp/spark-7f0fabb5-07de-43c3-8a26-a2325d5be63a/__spark_libs__363124573059127100.zip -> hdfs://ip-172-32-147-4.ec2.internal:8020/user/hadoop/.sparkStaging/application_1680007316515_0003/__spark_libs__363124573059127100.zip
23/03/28 13:08:53 INFO Client: Uploading resource file:/usr/lib/hudi/hudi-utilities-bundle_2.12-0.12.1-amzn-0.jar -> hdfs://ip-172-32-147-4.ec2.internal:8020/user/hadoop/.sparkStaging/application_1680007316515_0003/hudi-utilities-bundle_2.12-0.12.1-amzn-0.jar
23/03/28 13:08:53 INFO Client: Uploading resource file:/etc/spark/conf.dist/hive-site.xml -> hdfs://ip-172-32-147-4.ec2.internal:8020/user/hadoop/.sparkStaging/application_1680007316515_0003/hive-site.xml
23/03/28 13:08:54 INFO Client: Uploading resource file:/etc/hudi/conf.dist/hudi-defaults.conf -> hdfs://ip-172-32-147-4.ec2.internal:8020/user/hadoop/.sparkStaging/application_1680007316515_0003/hudi-defaults.conf
23/03/28 13:08:54 INFO Client: Uploading resource file:/mnt/tmp/spark-7f0fabb5-07de-43c3-8a26-a2325d5be63a/__spark_conf__2001263387666561545.zip -> hdfs://ip-172-32-147-4.ec2.internal:8020/user/hadoop/.sparkStaging/application_1680007316515_0003/__spark_conf__.zip
23/03/28 13:08:54 INFO SecurityManager: Changing view acls to: hadoop
23/03/28 13:08:54 INFO SecurityManager: Changing modify acls to: hadoop
23/03/28 13:08:54 INFO SecurityManager: Changing view acls groups to:
23/03/28 13:08:54 INFO SecurityManager: Changing modify acls groups to:
23/03/28 13:08:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
23/03/28 13:08:54 INFO Client: Submitting application application_1680007316515_0003 to ResourceManager
23/03/28 13:08:54 INFO YarnClientImpl: Submitted application application_1680007316515_0003
23/03/28 13:08:55 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:08:55 INFO Client:
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time:<PHONE_NUMBER>287
	 final status: UNDEFINED
	 tracking URL: http://ip-172-32-147-4.ec2.internal:20888/proxy/application_1680007316515_0003/
	 user: hadoop
23/03/28 13:08:56 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:08:57 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:08:58 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:08:59 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:00 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:01 INFO Client: Application report for application_1680007316515_0003 (state: RUNNING)
23/03/28 13:09:01 INFO Client:
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: ip-172-32-12-21.ec2.internal
	 ApplicationMaster RPC port: 34367
	 queue: default
	 start time:<PHONE_NUMBER>287
	 final status: UNDEFINED
	 tracking URL: http://ip-172-32-147-4.ec2.internal:20888/proxy/application_1680007316515_0003/
	 user: hadoop
23/03/28 13:09:02 INFO Client: Application report for application_1680007316515_0003 (state: RUNNING)
23/03/28 13:09:03 INFO Client: Application report for application_1680007316515_0003 (state: RUNNING)
23/03/28 13:09:04 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:04 INFO Client:
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: default
	 start time:<PHONE_NUMBER>287
	 final status: UNDEFINED
	 tracking URL: http://ip-172-32-147-4.ec2.internal:20888/proxy/application_1680007316515_0003/
	 user: hadoop
23/03/28 13:09:05 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:06 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:07 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:08 INFO Client: Application report for application_1680007316515_0003 (state: ACCEPTED)
23/03/28 13:09:09 INFO Client: Application report for application_1680007316515_0003 (state: RUNNING)
23/03/28 13:09:09 INFO Client:
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: ip-172-32-12-21.ec2.internal
	 ApplicationMaster RPC port: 42741
	 queue: default
	 start time:<PHONE_NUMBER>287
	 final status: UNDEFINED
	 tracking URL: http://ip-172-32-147-4.ec2.internal:20888/proxy/application_1680007316515_0003/
	 user: hadoop
23/03/28 13:09:10 INFO Client: Application report for application_1680007316515_0003 (state: RUNNING)
23/03/28 13:09:11 INFO Client: Application report for application_1680007316515_0003 (state: FINISHED)
23/03/28 13:09:11 INFO Client:
	 client token: N/A
	 diagnostics: User class threw exception: java.io.IOException: Could not load key generator class org.apache.hudi.keygen.SimpleKeyGenerator
	at org.apache.hudi.keygen.factory.HoodieSparkKeyGeneratorFactory.createKeyGenerator(HoodieSparkKeyGeneratorFactory.java:74)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.<init>(DeltaSync.java:235)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.<init>(HoodieDeltaStreamer.java:675)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:146)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:119)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:571)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.keygen.SimpleKeyGenerator
	at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91)
	at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:118)
	at org.apache.hudi.keygen.factory.HoodieSparkKeyGeneratorFactory.createKeyGenerator(HoodieSparkKeyGeneratorFactory.java:72)
	... 10 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89)
	... 12 more
Caused by: java.lang.IllegalArgumentException: Property hoodie.datasource.write.partitionpath.field not found
	at org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:67)
	at org.apache.hudi.common.config.TypedProperties.getString(TypedProperties.java:72)
	at org.apache.hudi.keygen.SimpleKeyGenerator.<init>(SimpleKeyGenerator.java:41)
	... 17 more

	 ApplicationMaster host: ip-172-32-12-21.ec2.internal
	 ApplicationMaster RPC port: 42741
	 queue: default
	 start time:<PHONE_NUMBER>287
	 final status: FAILED
	 tracking URL: http://ip-172-32-147-4.ec2.internal:20888/proxy/application_1680007316515_0003/
	 user: hadoop
23/03/28 13:09:11 ERROR Client: Application diagnostics message: User class threw exception: java.io.IOException: Could not load key generator class org.apache.hudi.keygen.SimpleKeyGenerator
	at org.apache.hudi.keygen.factory.HoodieSparkKeyGeneratorFactory.createKeyGenerator(HoodieSparkKeyGeneratorFactory.java:74)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.<init>(DeltaSync.java:235)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.<init>(HoodieDeltaStreamer.java:675)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:146)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.<init>(HoodieDeltaStreamer.java:119)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:571)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:742)
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.keygen.SimpleKeyGenerator
	at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91)
	at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:118)
	at org.apache.hudi.keygen.factory.HoodieSparkKeyGeneratorFactory.createKeyGenerator(HoodieSparkKeyGeneratorFactory.java:72)
	... 10 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89)
	... 12 more
Caused by: java.lang.IllegalArgumentException: Property hoodie.datasource.write.partitionpath.field not found
	at org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:67)
	at org.apache.hudi.common.config.TypedProperties.getString(TypedProperties.java:72)
	at org.apache.hudi.keygen.SimpleKeyGenerator.<init>(SimpleKeyGenerator.java:41)
	... 17 more

Exception in thread "main" org.apache.spark.SparkException: Application application_1680007316515_0003 finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1354)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1776)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1006)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1095)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1104)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/03/28 13:09:11 INFO ShutdownHookManager: Shutdown hook called
23/03/28 13:09:11 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-dbd4de25-f7b1-4875-b7fd-c599028ae4e0
23/03/28 13:09:11 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-7f0fabb5-07de-43c3-8a26-a2325d5be63a
Command exiting with ret '1'

Any advice | Feedback and pointing out what i am doing wrong would be great

References
https://cwiki.apache.org/confluence/display/HUDI/2020/01/20/Change+Capture+Using+AWS+Database+Migration+Service+and+Hudi
https://aws.amazon.com/blogs/big-data/apply-record-level-changes-from-relational-databases-to-amazon-s3-data-lake-using-apache-hudi-on-amazon-emr-and-aws-database-migration-service/
https://github.com/apache/hudi/issues/2406

@n3nash @bvaradar

@soumilshah1995 : This looks like the correct package is not added to classpath. Can you check if org.apache.hudi.keygen.SimpleKeyGenerator is present in the jar being passed.

Hi @soumilshah1995 The bottom exception complains that Property hoodie.datasource.write.partitionpath.field not found.  You need to specify the partition path field with --hoodie-conf hoodie.datasource.write.partitionpath.field=<partition_column>.

does this looks okay to you @bvaradar  | @yihua
  spark-submit
    --master yarn
    --deploy-mode cluster
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer
    --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension
    --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog
    --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer /usr/lib/hudi/hudi-utilities-bundle.jar
    --table-type COPY_ON_WRITE
    --source-ordering-field replicadmstimestamp
    --source-class org.apache.hudi.utilities.sources.ParquetDFSSource
    --target-base-path s3://sql-server-dms-demo/hudi/public/sales
    --target-table invoice
    --payload-class org.apache.hudi.common.model.AWSDmsAvroPayload
    --hoodie-conf hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator
    --hoodie-conf hoodie.datasource.write.recordkey.field=invoiceid
    --hoodie-conf hoodie.datasource.write.partitionpath.field=destinationstate
    --hoodie-conf hoodie.deltastreamer.source.dfs.root=s3://sql-server-dms-demo/raw/public/sales/

Please correct me if I'm doing something incorrectly. I've attached base files for your reference. I see hudi folders being created, but I don't see any base files (Parquet files) being created.  any idea why ?

@yihua @bvaradar
i just fired job

As I was describing, even after 10 minutes, there are still only two files on S3 and no generated base files. I do, however, see the meta data folder that was created by hudi.

Do you see any exceptions in logs? Also are you writing to this base path first time? Is your .hoodie folder empty? @soumilshah1995

@yihua @pratyakshsharma
First of all, I want to thank you for taking time out of your busy schedule to assist me with this matter.
Yes i cancelled the job since i am performing the lab from AWS Account to avoid charges the job was in running state from past 10 minutes and i could not see any Base Files. Yes i am writing to abse path for first time
if you all can tell me what am i missing or how can i resolve error that would be great please let me know if you need any further details happy to provide you. attaching screenshot of S3 which shows hudi folder was created but base file was not created

Links to parquet file can be found https://drive.google.com/drive/folders/1BwNEK649hErbsWcYLZhqCWnaXFX3mIsg?usp=share_link

Step 1

Step 2:

Even after running for 10 minutes i dont see base files

issue has been resolved  Daniel Ford was very kind to help on slack i will post detailed video  about delta streamer on Youtube Channel

  spark-submit \
    --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer  \
    --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension  \
    --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
    --conf spark.sql.hive.convertMetastoreParquet=false \
    --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory \
    --master yarn \
    --deploy-mode client \
    --deploy-mode cluster \
    --executor-memory 1g \
    --driver-memory 2g \
     /usr/lib/hudi/hudi-utilities-bundle.jar \
    --table-type COPY_ON_WRITE \
    --op UPSERT \
    --source-ordering-field replicadmstimestamp  \
    --source-class org.apache.hudi.utilities.sources.ParquetDFSSource \
    --target-base-path s3://sql-server-dms-demo/hudi/public/sales \
    --target-table invoice \
    --hoodie-conf hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator \
    --hoodie-conf hoodie.datasource.write.recordkey.field=invoiceid \
    --hoodie-conf hoodie.datasource.write.partitionpath.field=destinationstate \
    --hoodie-conf hoodie.deltastreamer.source.dfs.root=s3://sql-server-dms-demo/raw/public/sales \
    --hoodie-conf  hoodie.datasource.write.precombine.field=replicadmstimestamp

Do you mind sharing what was the issue? @soumilshah1995

umm sure i think i was missing some config

These are config that worked for me
  spark-submit \
    --class                 org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer  \
    --conf                  spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf                  spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension  \
    --conf                  spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
    --conf                  spark.sql.hive.convertMetastoreParquet=false \
    --conf                  spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory \
    --master                yarn \
    --deploy-mode           client \
    --deploy-mode           cluster \
    --executor-memory       1g \
     /usr/lib/hudi/hudi-utilities-bundle.jar \
    --table-type            COPY_ON_WRITE \
    --op                    UPSERT \
    --enable-sync \
    --source-ordering-field replicadmstimestamp  \
    --source-class          org.apache.hudi.utilities.sources.ParquetDFSSource \
    --target-base-path      s3://delta-streamer-demo-hudi/raw/public/sales \
    --target-table          invoice \
    --payload-class         org.apache.hudi.common.model.AWSDmsAvroPayload \
    --hoodie-conf           hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator \
    --hoodie-conf           hoodie.datasource.write.recordkey.field=invoiceid \
    --hoodie-conf           hoodie.datasource.write.partitionpath.field=destinationstate \
    --hoodie-conf           hoodie.deltastreamer.source.dfs.root=s3://delta-streamer-demo-hudi/raw/public/sales \
    --hoodie-conf           hoodie.datasource.write.precombine.field=replicadmstimestamp \
    --hoodie-conf           hoodie.database.name=hudidb_raw  \
    --hoodie-conf           hoodie.datasource.hive_sync.enable=true \
    --hoodie-conf           hoodie.datasource.hive_sync.database=hudidb_raw \
    --hoodie-conf           hoodie.datasource.hive_sync.table=tbl_invoices \
    --hoodie-conf           hoodie.datasource.hive_sync.partition_fields=destinationstate

@soumilshah1995 Just got to this again.  So it looks like the preCombine field config (hoodie.datasource.write.precombine.field) was missing in your original configs, causing the job failure.  If you have the driver logs, you should see an exception because of this.

You are right I was missing that )
On Thu, Mar 30, 2023 at 6:53 PM Y Ethan Guo @.***>
wrote:

@soumilshah1995 https://github.com/soumilshah1995 Just got to this
again. So it looks like the preCombine field config (
hoodie.datasource.write.precombine.field) was missing in your original
configs, causing the job failure. If you have the driver logs, you should
see an exception because of this.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/hudi/issues/8309#issuecomment-1491068636, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AJMF5PZ6GHJ2UQU2BZRJVOTW6YFGZANCNFSM6AAAAAAWKSFH24
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Thanking You,
Soumil Nitin Shah
B.E in Electronic
MS Electrical Engineering
MS  Computer Engineering
+1-646 204 5957

Project : Using Apache Hudi Deltastreamer and AWS DMS Hands on Labs

Video Tutorials

Part 1: Project Overview : https://www.youtube.com/watch?v=D9L0NLSqC1s
Part 2: Aurora Setup : https://youtu.be/HR5A6iGb4LE
Part 3: https://youtu.be/rnyj5gkIPKA
Part 4: https://youtu.be/J1xvPIcDIaQ

Steps
Step 1:  Create Aurora Source Database and update the seetings to enable CDC on Postgres

Read More : https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html
Create new Create parameter group as shows in video and make sure to edit these two setting as shown in video part 2

rds.logical_replication  1
wal_sender_timeout   300000

once done please apply these to database and reboot your Database
Step 2: Run Python file to create a table called sales in public schema in aurora and lets populate some data into table
Run python run.py

try:
    import os
    import logging

    from functools import wraps
    from abc import ABC, abstractmethod
    from enum import Enum
    from logging import StreamHandler

    import uuid
    from datetime import datetime, timezone
    from random import randint
    import datetime

    import sqlalchemy as db
    from faker import Faker
    import random
    import psycopg2
    import psycopg2.extras as extras
    from dotenv import load_dotenv

    load_dotenv(".env")
except Exception as e:
    raise Exception("Error: {} ".format(e))

class Logging:
    """
    This class is used for logging data to datadog an to the console.
    """

    def __init__(self, service_name, ddsource, logger_name="demoapp"):

        self.service_name = service_name
        self.ddsource = ddsource
        self.logger_name = logger_name

        format = "[%(asctime)s] %(name)s %(levelname)s %(message)s"
        self.logger = logging.getLogger(self.logger_name)
        formatter = logging.Formatter(format, )

        if logging.getLogger().hasHandlers():
            logging.getLogger().setLevel(logging.INFO)
        else:
            logging.basicConfig(level=logging.INFO)

global logger
logger = Logging(service_name="database-common-module", ddsource="database-common-module",
                 logger_name="database-common-module")

def error_handling_with_logging(argument=None):
    def real_decorator(function):
        @wraps(function)
        def wrapper(self, *args, **kwargs):
            function_name = function.__name__
            response = None
            try:
                if kwargs == {}:
                    response = function(self)
                else:
                    response = function(self, **kwargs)
            except Exception as e:
                response = {
                    "status": -1,
                    "error": {"message": str(e), "function_name": function.__name__},
                }
                logger.logger.info(response)
            return response

        return wrapper

    return real_decorator

class DatabaseInterface(ABC):
    @abstractmethod
    def get_data(self, query):
        """
        For given query fetch the data
        :param query: Str
        :return: Dict
        """

    def execute_non_query(self, query):
        """
        Inserts data into SQL Server
        :param query:  Str
        :return: Dict
        """

    def insert_many(self, query, data):
        """
        Insert Many items into database
        :param query: str
        :param data: tuple
        :return: Dict
        """

    def get_data_batch(self, batch_size=10, query=""):
        """
        Gets data into batches
        :param batch_size: INT
        :param query: STR
        :return: DICT
        """

    def get_table(self, table_name=""):
        """
        Gets the table from database
        :param table_name: STR
        :return: OBJECT
        """

class Settings(object):
    """settings class"""

    def __init__(
            self,
            port="",
            server="",
            username="",
            password="",
            timeout=100,
            database_name="",
            connection_string="",
            collection_name="",
            **kwargs,
    ):
        self.port = port
        self.server = server
        self.username = username
        self.password = password
        self.timeout = timeout
        self.database_name = database_name
        self.connection_string = connection_string
        self.collection_name = collection_name

class DatabaseAurora(DatabaseInterface):
    """Aurora database class"""

    def __init__(self, data_base_settings):
        self.data_base_settings = data_base_settings
        self.client = db.create_engine(
            "postgresql://{username}:{password}@{server}:{port}/{database}".format(
                username=self.data_base_settings.username,
                password=self.data_base_settings.password,
                server=self.data_base_settings.server,
                port=self.data_base_settings.port,
                database=self.data_base_settings.database_name
            )
        )
        self.metadata = db.MetaData()
        logger.logger.info("Auroradb connection established successfully.")

    @error_handling_with_logging()
    def get_data(self, query):
        self.query = query
        cursor = self.client.connect()
        response = cursor.execute(self.query)
        result = response.fetchall()
        columns = response.keys()._keys
        data = [dict(zip(columns, item)) for item in result]
        cursor.close()
        return {"statusCode": 200, "data": data}

    @error_handling_with_logging()
    def execute_non_query(self, query):
        self.query = query
        cursor = self.client.connect()
        cursor.execute(self.query)
        cursor.close()
        return {"statusCode": 200, "data": True}

    @error_handling_with_logging()
    def insert_many(self, query, data):
        self.query = query
        print(data)
        cursor = self.client.connect()
        cursor.execute(self.query, data)
        cursor.close()
        return {"statusCode": 200, "data": True}

    @error_handling_with_logging()
    def get_data_batch(self, batch_size=10, query=""):
        self.query = query
        cursor = self.client.connect()
        response = cursor.execute(self.query)
        columns = response.keys()._keys
        while True:
            result = response.fetchmany(batch_size)
            if not result:
                break
            else:
                items = [dict(zip(columns, data)) for data in result]
                yield items

    @error_handling_with_logging()
    def get_table(self, table_name=""):
        table = db.Table(table_name, self.metadata,
                         autoload=True,
                         autoload_with=self.client)

        return {"statusCode": 200, "table": table}

class DatabaseAuroraPycopg(DatabaseInterface):
    """Aurora database class"""

    def __init__(self, data_base_settings):
        self.data_base_settings = data_base_settings
        self.client = psycopg2.connect(
            host=self.data_base_settings.server,
            port=self.data_base_settings.port,
            database=self.data_base_settings.database_name,
            user=self.data_base_settings.username,
            password=self.data_base_settings.password,
        )

    @error_handling_with_logging()
    def get_data(self, query):
        self.query = query
        cursor = self.client.cursor()
        cursor.execute(self.query)
        result = cursor.fetchall()
        columns = [column[0] for column in cursor.description]
        data = [dict(zip(columns, item)) for item in result]
        cursor.close()
        _ = {"statusCode": 200, "data": data}

        return _

    @error_handling_with_logging()
    def execute(self, query, data):
        self.query = query
        cursor = self.client.cursor()
        cursor.execute(self.query, data)
        self.client.commit()
        cursor.close()
        return {"statusCode": 200, "data": True}

    @error_handling_with_logging()
    def get_data_batch(self, batch_size=10, query=""):
        self.query = query
        cursor = self.client.cursor()
        cursor.execute(self.query)
        columns = [column[0] for column in cursor.description]
        while True:
            result = cursor.fetchmany(batch_size)
            if not result:
                break
            else:
                items = [dict(zip(columns, data)) for data in result]
                yield items

    @error_handling_with_logging()
    def insert_many(self, query, data):
        self.query = query
        cursor = self.client.cursor()
        extras.execute_batch(cursor, self.query, data)
        self.client.commit()
        cursor.close()
        return {"statusCode": 200, "data": True}

class Connector(Enum):

    ON_AURORA_PYCOPG = DatabaseAurora(
        data_base_settings=Settings(
            port="5432",
            server="XXXXXXXXX",
            username="postgres",
            password="postgres",
            database_name="postgres",
        )
    )

def main():
    helper = Connector.ON_AURORA_PYCOPG.value
    import time

    states = ("AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "FL", "GA", "HI", "ID", "IL", "IN",
              "IA", "KS", "KY", "LA", "ME", "MD", "MA", "MI", "MN", "MS", "MO", "MT", "NE", "NV", "NH", "NJ",
              "NM", "NY", "NC", "ND", "OH", "OK", "OR", "PA", "RI", "SC", "SD", "TN", "TX", "UT", "VT", "VA",
              "WA", "WV", "WI", "WY")
    shipping_types = ("Free", "3-Day", "2-Day")

    product_categories = ("Garden", "Kitchen", "Office", "Household")
    referrals = ("Other", "Friend/Colleague", "Repeat Customer", "Online Ad")

    try:
        query = """
CREATE TABLE public.sales (
  invoiceid INTEGER,
  itemid INTEGER,
  category TEXT,
  price NUMERIC(10,2),
  quantity INTEGER,
  orderdate DATE,
  destinationstate TEXT,
  shippingtype TEXT,
  referral TEXT
);
        """
        helper.execute_non_query(query=query,)
        time.sleep(2)
    except Exception as e:
        print("Error",e)

    try:
        query = """
            ALTER TABLE execute_non_query.sales REPLICA IDENTITY  FULL
        """
        helper.execute(query=query)
        time.sleep(2)
    except Exception as e:
        pass

    for i in range(0, 100):

        item_id = random.randint(1, 100)
        state = states[random.randint(0, len(states) - 1)]
        shipping_type = shipping_types[random.randint(0, len(shipping_types) - 1)]
        product_category = product_categories[random.randint(0, len(product_categories) - 1)]
        quantity = random.randint(1, 4)
        referral = referrals[random.randint(0, len(referrals) - 1)]
        price = random.randint(1, 100)
        order_date = datetime.date(2016, random.randint(1, 12), random.randint(1, 28)).isoformat()
        invoiceid = random.randint(1, 20000)

        data_order = (invoiceid, item_id, product_category, price, quantity, order_date, state, shipping_type, referral)

        query = """INSERT INTO public.sales
                                            (
                                            invoiceid, itemid, category, price, quantity, orderdate, destinationstate,shippingtype, referral
                                            )
                                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"""

        helper.insert_many(query=query, data=data_order)

main()

Step 3:  Create a DMS Replication Instance as shown in video 3 and create S3 bucket and iAM roles refer video part 3
Create Source and target in DMS

When you create target add following settings

{
    "CsvRowDelimiter": "\\n",
    "CsvDelimiter": ",",
    "BucketFolder": "raw",
    "BucketName": "XXXXXXXXXXXXX",
    "CompressionType": "NONE",
    "DataFormat": "parquet",
    "EnableStatistics": true,
    "IncludeOpForFullLoad": true,
    "TimestampColumnName": "replicadmstimestamp",
    "DatePartitionEnabled": false
}

Note add this as well in Extra connection attribute

parquetVersion=PARQUET_2_0;

Step 4: create Task  add following settings
{
    "rules": [
        {
            "rule-type": "selection",
            "rule-id": "861743510",
            "rule-name": "861743510",
            "object-locator": {
                "schema-name": "public",
                "table-name": "sales"
            },
            "rule-action": "include",
            "filters": []
        }
    ]
}

Create EMR cluster and fire the delta streamer

Note you can download parquert files https://drive.google.com/drive/folders/1BwNEK649hErbsWcYLZhqCWnaXFX3mIsg?usp=share_link
these are sample data files generated from DMS you can directly copy into your S3 for learning purposes

  spark-submit \
    --class                 org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer  \
    --conf                  spark.serializer=org.apache.spark.serializer.KryoSerializer \
    --conf                  spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension  \
    --conf                  spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
    --conf                  spark.sql.hive.convertMetastoreParquet=false \
    --conf                  spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory \
    --master                yarn \
    --deploy-mode           client \
    --deploy-mode           cluster \
    --executor-memory       1g \
     /usr/lib/hudi/hudi-utilities-bundle.jar \
    --table-type            COPY_ON_WRITE \
    --op                    UPSERT \
    --enable-sync \
    --source-ordering-field replicadmstimestamp  \
    --source-class          org.apache.hudi.utilities.sources.ParquetDFSSource \
    --target-base-path      s3://delta-streamer-demo-hudi/raw/public/sales \
    --target-table          invoice \
    --payload-class         org.apache.hudi.common.model.AWSDmsAvroPayload \
    --hoodie-conf           hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator \
    --hoodie-conf           hoodie.datasource.write.recordkey.field=invoiceid \
    --hoodie-conf           hoodie.datasource.write.partitionpath.field=destinationstate \
    --hoodie-conf           hoodie.deltastreamer.source.dfs.root=s3://delta-streamer-demo-hudi/raw/public/sales \
    --hoodie-conf           hoodie.datasource.write.precombine.field=replicadmstimestamp \
    --hoodie-conf           hoodie.database.name=hudidb_raw  \
    --hoodie-conf           hoodie.datasource.hive_sync.enable=true \
    --hoodie-conf           hoodie.datasource.hive_sync.database=hudidb_raw \
    --hoodie-conf           hoodie.datasource.hive_sync.table=tbl_invoices \
    --hoodie-conf           hoodie.datasource.hive_sync.partition_fields=destinationstate


Improve the overall look of the app and interface

I want to work on this issue .Will you please give an idea or some suggestions what kind of changes will be helpful in this

hey! thanks for taking interest in this, I would prefer the app to have a better starting screen as right now it only shows camera and gallery but if it could show both of these options in a better pleasing manner.
Also, the text output that comes out of the OCR, it is in a haphazard way i.e. not in proper paragraphs or something and is missing alignment, it would be great if you could work on that as well.

Hi, if it's still open, then I want to contribute!

Hi, if it's still open, then I want to contribute!
Hi Pooja, yes the issue is still open, you can contribute.

okay, thanks! please assign it to me, I will start working.

Okay, done.


dae: 0.4.0 -> 0.5.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/daeuniverse/dae/releases.
meta.description for dae is: A Linux high-performance transparent proxy solution based on eBPF
meta.homepage for dae is: https://github.com/daeuniverse/dae
Updates performed

Golang update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.5.1 with grep in /nix/store/5n0lf4m62rqfbcipaxi1cdsqknn8z965-dae-0.5.1
found 0.5.1 in filename of file in /nix/store/5n0lf4m62rqfbcipaxi1cdsqknn8z965-dae-0.5.1

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
dae

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/5n0lf4m62rqfbcipaxi1cdsqknn8z965-dae-0.5.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A dae https://github.com/r-ryantm/nixpkgs/archive/86952dd4321360b5398a909f7bec59d573cfbd4c.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/86952dd4321360b5398a909f7bec59d573cfbd4c#dae

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/5n0lf4m62rqfbcipaxi1cdsqknn8z965-dae-0.5.1
ls -la /nix/store/5n0lf4m62rqfbcipaxi1cdsqknn8z965-dae-0.5.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

dae

Maintainer pings
cc @oluceps @pokon548 for testing.

I could confirm 0.5.1 resolved external dns issue: https://github.com/daeuniverse/dae/issues/412, using smartdns as local dns server.
Shall we update dae to 0.5.1? @oluceps

Superseded by #287239.


python3Packages.bottleneck: disable failing test

Description of changes
Fix build (https://hydra.nixos.org/build/173965833)

switch to pytestCheckHook
add pythonImportsCheck
disable on older Python releases

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 170107 run on x86_64-linux 1

2 packages built:

python310Packages.bottleneck
python39Packages.bottleneck


inform users why password is needed for key

The key phase for the vault setup is confusing because users don't understand why the have to enter a password.
Improvement:

[ ] Inform users about the key phase
[ ] Inform users about why they need to provide a password

shout out to ChatGPT for helping with the copy write

goal
old
new

providing a password for key
Enter a password to encrypt your private key. This password will be used to unlock your key when you want to use it.
Enter a passphrase to protect your private key. You'll use this passphrase to unlock and use your key. Keep it strong and secret!


Documentation, Wiki and FAQ

Description
A common set of FAQs  and Wiki needs to be set up.
General documentation is required for

What is a Module?
Module Dependencies?
What is a Module used for?
How to deploy a Module (with PS, CLI, AzOps, TF, ...)?
What is a Workload?
How have Modules to do with Azure Landing Zones?
How does this fit into CAF?
What collaboration platform is recommended (GitHub/ADO)?
What is IaC?
Azure Resource Manager Reference?
What are GitHub Actions and Workflows?
Are Azure Pipelines supported?
How to contribute a Module?
How to structure a Module (ARM Template) according to an ARM Template syntax?
How to structure a Module (Bicep Module) according to an ARM Template syntax?
What are Constructs?

Other

naming in general for parameters in modules to remain consistent naming structure within templates and files (input parameter as well as output parameters, camelcasing)

References
https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/file
https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/syntax
https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/plan/workloads#what-is-a-workload
https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/ready/landing-zone/
https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/extension-resource-types
https://docs.microsoft.com/en-us/devops/deliver/what-is-infrastructure-as-code
https://docs.microsoft.com/en-us/dotnet/architecture/cloud-native/infrastructure-as-code
https://docs.microsoft.com/en-us/azure/architecture/framework/devops/automation-infrastructure

++ Authoring / Design Principals

Reading the wiki. I think we are leaving wiggle room in the guidelines for different approaches to be followed, which can limit the standardization like automation. A good example is the web app that's in the guideline.
functionApp [module] is good to have as a module, but cannot be deployed in isolation, same as sites, where they need to hook into their parents, respectively. So, what is the principal that needs to be followed so that we are able to deploy each component in isolation. And how much is too much for the levels of child resources?

Traditionally we took care of any required dependencies in the 'Master templates' that link to the ResourceModules as linked templates.
E.g. One would link to the ResourceGroup deployment module, VNET,  etc. and use this template orchestration to deploy an entire estate.
Now, regarding the tests for the individual modules it's a different story. Here I see to options. Either:

Deploy a persistent set of resources to be referenced by all modules for testing (e.g. VNets, NSGs, LA, etc.) which works quite well as long as we're not talking about constly resources
Add some logic to each module that is capable to deploy its dependencies for testing and remove them after. While this approach provides the most isolution it's naturally more effort to create and maintain. However, one the plus site it would work also for scenarios for approach #1 would lack.

How does an idea of having 'settings.json' look like?
{
"dependencies": [
{
"type": "Microsoft.Network/networkSecurityGroups",
"resourceId": "x"
}
]
}

Or create a folder called "dependencies" and add parameter files for the dependency resource (I.e microsoft.network_networkSecurityGroups.parameters.json)
Just thinking out loud :)

The team covered a lot of content around the Modules hence closing for now.
Some points weren't met but could be potentially resumed in future.


"ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)"

Hi @DiegoStock12,
Thanks for open-source this project and it is very interesting to me.
I was able to deploy the cluster and use function_lenet.py from the example and upload mnist dataset myself to create dataset. However, I met the following error. See log for the details.
Is my dataset format incorrect? Should the shape be (60000, 28, 28) rather than (60000, 784)? Are you able to post the mnist dataset that you used in the repo?
I didn't deploy storagesvc properly,  and it is pending somehow.
fission            storagesvc-576745cff6-z4m9n                              0/1     Pending     0          3h4m

I got the following error when using "kubectl describe".  "0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. "  Not sure if the model is saved into the storage or not.   My current understanding is that the model is saved in Redis and I am not sure the impact of this storagesvc.
Any suggestion and help will be appreciated.
ning@cpu33:~/repo/kubeml$ kubectl logs job-6da5de79 -n kubeml
2022-11-03T16:30:32.448Z        INFO    train/job.go:119        Creating new basic train job
2022-11-03T16:30:32.448Z        INFO    trainJob-6da5de79       train/api.go:153        starting job API        {"JobID": "6da5de79"}
2022-11-03T16:30:33.604Z        DEBUG   trainJob-6da5de79       train/api.go:32 creating task and starting training...
2022-11-03T16:30:33.604Z        DEBUG   trainJob-6da5de79       train/api.go:58 Assigned new task to the job    {"task": {"request":{"model_type":"example","batch_size":128,"epochs":10,"dataset":"mnist","lr":0.01,"function_name":"example","options":{"default_parallelism":2,"static_parallelism":false,"validate_every":0,"k":-1,"goal_accuracy":100}},"job":{"id":"6da5de79","state":{"parallelism":2,"elapsed_time":0}}}}
2022-11-03T16:30:33.604Z        INFO    trainJob-6da5de79       train/job.go:158        Starting to serve train job
2022-11-03T16:30:33.604Z        INFO    trainJob-6da5de79       train/job.go:159        Initializing model
2022-11-03T16:30:33.604Z        DEBUG   trainJob-6da5de79       train/job.go:270        Calling init function
2022-11-03T16:30:33.604Z        INFO    trainJob-6da5de79       train/function.go:74    Invoking init function
2022-11-03T16:30:33.604Z        DEBUG   trainJob-6da5de79       train/function.go:65    Built url       {"url": "http://router.fission/example?K=-1&N=0&batchSize=128&epoch=0&funcId=0&jobId=6da5de79&lr=0.01&task=init"}
2022-11-03T16:30:34.145Z        DEBUG   trainJob-6da5de79       train/job.go:279        Received layers {"layers": ["conv1.weight", "conv1.bias", "conv2.weight", "conv2.bias", "fc1.weight", "fc1.bias", "fc2.weight", "fc2.bias", "fc3.weight", "fc3.bias"]}
2022-11-03T16:30:34.145Z        DEBUG   trainJob-6da5de79       train/job.go:280        Creating model
2022-11-03T16:30:34.145Z        DEBUG   trainJob-6da5de79.model model/model.go:78       Building the model      {"jobId": "6da5de79"}
2022-11-03T16:30:34.145Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "conv1.weight"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "conv1.bias"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "conv2.weight"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "conv2.bias"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "fc1.weight"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "fc1.bias"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "fc2.weight"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "fc2.bias"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "fc3.weight"}
2022-11-03T16:30:34.195Z        DEBUG   trainJob-6da5de79.model model/model.go:87       Creating new layer      {"layerName": "fc3.bias"}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "conv1.weight", "shape": [6,1,5,5]}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "conv2.weight", "shape": [16,6,5,5]}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "fc1.weight", "shape": [120,256]}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "fc1.bias", "shape": [120]}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "fc3.weight", "shape": [10,84]}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "fc3.bias", "shape": [10]}
2022-11-03T16:30:34.197Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "conv1.bias", "shape": [6]}
2022-11-03T16:30:34.198Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "conv2.bias", "shape": [16]}
2022-11-03T16:30:34.198Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "fc2.weight", "shape": [84,120]}
2022-11-03T16:30:34.198Z        INFO    trainJob-6da5de79.model model/model.go:125      Layer   {"name": "fc2.bias", "shape": [84]}
2022-11-03T16:30:34.198Z        INFO    trainJob-6da5de79       train/job.go:296        Started new epoch       {"epoch": 1}
2022-11-03T16:30:34.198Z        DEBUG   trainJob-6da5de79       train/function.go:112   Invoking function       {"id": 0}
2022-11-03T16:30:34.198Z        DEBUG   trainJob-6da5de79       train/function.go:65    Built url       {"url": "http://router.fission/example?K=-1&N=2&batchSize=128&epoch=1&funcId=0&jobId=6da5de79&lr=0.01&task=train"}
2022-11-03T16:30:34.198Z        DEBUG   trainJob-6da5de79       train/function.go:112   Invoking function       {"id": 1}
2022-11-03T16:30:34.198Z        DEBUG   trainJob-6da5de79       train/function.go:65    Built url       {"url": "http://router.fission/example?K=-1&N=2&batchSize=128&epoch=1&funcId=1&jobId=6da5de79&lr=0.01&task=train"}
2022-11-03T16:30:34.198Z        DEBUG   trainJob-6da5de79.model model/model.go:119      Wiped model state
2022-11-03T16:30:34.198Z        DEBUG   trainJob-6da5de79       train/job.go:375        Waiting for functions to finish...
2022-11-03T16:30:35.379Z        DEBUG   trainJob-6da5de79       train/function.go:203   function finished, checking errors      {"id": 1}
2022-11-03T16:30:35.379Z        DEBUG   trainJob-6da5de79       train/function.go:207   returning error...      {"error": "ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)"}
2022-11-03T16:30:35.379Z        DEBUG   trainJob-6da5de79.model model/model.go:251      Updating model layers   {"funcId": 1}
2022-11-03T16:30:35.380Z        ERROR   trainJob-6da5de79.model model/model.go:279      Could not build layer from database     {"error": "ERR tensor key is empty or in a different shard", "name": "conv1.weight", "funcId": 1}
2022-11-03T16:30:35.380Z        DEBUG   trainJob-6da5de79       train/function.go:186   adding 1 to the finished functions
2022-11-03T16:30:35.836Z        DEBUG   trainJob-6da5de79       train/function.go:203   function finished, checking errors      {"id": 0}
2022-11-03T16:30:35.836Z        DEBUG   trainJob-6da5de79       train/function.go:207   returning error...      {"error": "ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)"}
2022-11-03T16:30:35.836Z        DEBUG   trainJob-6da5de79.model model/model.go:251      Updating model layers   {"funcId": 0}
2022-11-03T16:30:35.836Z        ERROR   trainJob-6da5de79.model model/model.go:279      Could not build layer from database     {"error": "ERR tensor key is empty or in a different shard", "name": "conv1.weight", "funcId": 0}
2022-11-03T16:30:35.836Z        ERROR   trainJob-6da5de79       train/job.go:190        Error training model    {"error": "error invoking functions: all functions finished with an error: ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)", "errorVerbose": "ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)\nall functions finished with an error\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).checkFunctionErrors\n\t/app/pkg/train/util.go:152\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).invokeTrainFunctions\n\t/app/pkg/train/function.go:120\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).train\n\t/app/pkg/train/job.go:308\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).Train\n\t/app/pkg/train/job.go:188\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337\nerror invoking functions\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).train\n\t/app/pkg/train/job.go:310\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).Train\n\t/app/pkg/train/job.go:188\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"}
2022-11-03T16:30:35.836Z        DEBUG   trainJob-6da5de79       train/function.go:186   adding 1 to the finished functions
2022-11-03T16:30:35.836Z        DEBUG   trainJob-6da5de79       train/job.go:394        Merging models after iteration  {"finishCh": [1, 0]}
2022-11-03T16:30:35.836Z        DEBUG   trainJob-6da5de79.parallel-sgd  model/parallelSGD.go:28 Averaging       {"num": 2}
2022-11-03T16:30:35.836Z        INFO    trainJob-6da5de79.model model/model.go:136      Publishing model on the database
2022-11-03T16:30:35.837Z        DEBUG   trainJob-6da5de79       train/util.go:231       Deleting tensors...     {"names": ["6da5de79:fc1.weight", "6da5de79:conv2.bias", "6da5de79:conv2.weight", "6da5de79:fc3.bias", "6da5de79:fc2.weight", "6da5de79:conv1.weight", "6da5de79:fc1.bias", "6da5de79:fc2.bias", "6da5de79:conv1.bias", "6da5de79:fc3.weight"]}
2022-11-03T16:30:35.837Z        DEBUG   trainJob-6da5de79       train/util.go:243       Delete from the database        {"num tensors": 10}
2022-11-03T16:30:35.837Z        DEBUG   trainJob-6da5de79       train/job.go:168        closing job     {"error": "error invoking functions: all functions finished with an error: ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)", "errorVerbose": "ValueError('pic should be 2/3 dimensional. Got 1 dimensions.',)\nall functions finished with an error\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).checkFunctionErrors\n\t/app/pkg/train/util.go:152\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).invokeTrainFunctions\n\t/app/pkg/train/function.go:120\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).train\n\t/app/pkg/train/job.go:308\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).Train\n\t/app/pkg/train/job.go:188\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337\nerror invoking functions\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).train\n\t/app/pkg/train/job.go:310\ngithub.com/diegostock12/kubeml/ml/pkg/train.(*TrainJob).Train\n\t/app/pkg/train/job.go:188\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"}
2022-11-03T16:30:35.886Z        INFO    trainJob-6da5de79.model model/model.go:158      Model published in the DB
2022-11-03T16:30:35.886Z        DEBUG   trainJob-6da5de79       train/job.go:412        Merge and save took     {"time": 0.049861545}
2022-11-03T16:30:35.886Z        DEBUG   trainJob-6da5de79       train/job.go:415        finished funcs are      {"num": 2}
2022-11-03T16:30:35.886Z        DEBUG   trainJob-6da5de79       train/job.go:419        all functions finished, quiting...

Hi @nwangfw! First of all thanks for your interest in the project, and I hope you find it interesting!
I have been taking a look at what you mention, and seems like the error has nothing to do with the storageSvc you mention as this is a fission-specific service (everything kubeml-related is in the kubeml namespace) that is only used to save fission function code and similar artifacts when their size becomes too big. See reference here. Nevertheless, I am not sure of what the error could be with that, but if the function code is run then for sure the issue is not that one.
Regarding the error itself, it is actually a PyTorch error and not related to KubeML itself. Indeed as you mention since the python library used to train the network is based on PyTorch, the dataset formats are more or less thought to have the shapes that are mostly used in PyTorch. In this case for MNIST that would be each image to be 2D and of shape [28, 28].
You can either save the data in [28,28] format in the storage or add some custom logic to the KubeDataset you create to change the dimensions of the images from [784,] to [28,28].
I can try to find the exact dataset I used but it has been a while, in the meantime, I found the code I used to create those datasets:
from torchvision import datasets
train_data = datasets.MNIST('../data', train=True, download=True)
validation_data = datasets.MNIST('../data', train=False, download=True)

From there, you can save the data that is already stored for example for the train samples [60000, 28, 28] by using np.save on the data and targets member variables of the dataset. This will save the X and Ys of the train and validation data which you can then upload using kubeml dataset create.
Hope this helps you fix your problems!

Hi DiegoStock12
It is a very interesting topic and this is the only github repo that I found online discussing serverless deep learning training.
Many thanks for your help! Yes, it is the dataset format issue. I happened to use the dataset with shape of (60000, 784). Per your suggestions, I changed it to  (60000, 28, 28) and it works now!
Many thanks!!!!


Release stable version Azure.ResourceManager.ServiceBus SDK

Add a comment to track this review comment:
https://apiview.dev/Assemblies/Conversation/c3d2892f4efd4daabce6d2c87cf02534#496b08db5c9440948fa2cc4016848a23


[Backport 6.x] Add per item routing for bulk requests

Cherry-picked backports to ES6 client for item routing for bulk requests.

https://github.com/elastic/go-elasticsearch/pull/265
https://github.com/elastic/go-elasticsearch/pull/360
https://github.com/elastic/go-elasticsearch/pull/426

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

Hi @dtrejod
Thank you but the 6.x branch has reached EOL, I would advise you to upgrade and use the latest 7.17 version instead.


More than 72 SCSS $colors map entries causes strange CSS issues

Ionic version:  (check one with "x")
(For Ionic 1.x issues, please use https://github.com/ionic-team/ionic-v1)
[ ] 2.x
[X ] 3.x
[ ] 4.x
I'm submitting a ...  (check one with "x")
[X ] bug report
[ ] feature request
Current behavior:
More than 72 SCSS $colors map entries causes strange CSS issues; Overridden class not displaying correctly.
Expected behavior:
Expect to be able to add unlimited color keys to color map or at least a very large number
Steps to reproduce:
Results may be unpredictable due to strange and random nature of the issue. No error is given.

Create an ionic project
Override some styles in your app.scss (was not honoring .col{ padding: 0} in my project. Other classes rendered correctly)
Build, run and verify styles display as set
Add > 72 color entries (top level keys) to color map
Build, run and verify styles are wonky

Tried deleting an existing style and adding the same one initially attempted. This works. The bug also manifests even if the 73rd color is not even referenced.
Related code:

insert short code snippets here

Other information:

Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
cli packages: (/usr/local/lib/node_modules)

    @ionic/cli-utils  : 1.19.2
    ionic (Ionic CLI) : 3.20.0

global packages:

    cordova (Cordova CLI) : 8.0.0

local packages:

    @ionic/app-scripts : 3.1.6
    Cordova Platforms  : android 6.3.0 ios 4.5.4
    Ionic Framework    : ionic-angular 3.9.2

System:

    Android SDK Tools : 26.1.1
    ios-deploy        : 1.9.2
    Node              : v6.11.3
    npm               : 3.10.10
    OS                : macOS High Sierra
    Xcode             : Xcode 9.3 Build version 9E145

Environment Variables:

    ANDROID_HOME : /Users/josh/Library/Android/sdk

Misc:

    backend : legacy

Thanks for the issue! We do not recommend using that many colors in the colors map. Each color you add will generate about 1200 selectors in the CSS. This may be the issue you're running into: https://github.com/ionic-team/ionic-app-scripts/issues/506
I would recommend creating your own colors map if you are not using the colors to style Ionic components via the colors attribute.

Ha ha! Thanks Brandy. Yes that would be a lot of selectors, which may explain why the SASS compiler is so slow lately. A second map should work fine in this case.
BTW, it seems pretty intuitive to just add to the existing map for custom styling (not theming), and apparently that's an assumption others have made too. A warning in the docs would be nice, since the existing themes are pretty inadequate for anything but vanilla apps. I had no idea this would be generating a selector for every color use case in Ionic.

@snortblt Thanks for the suggestion! I created an issue for the site here: https://github.com/ionic-team/ionic-site/issues/1361
We have redone theming for v4, so I don't believe this will be an issue anymore but I need to confirm that. Here's a sneak peek at the new theming docs: https://github.com/ionic-team/ionic-docs/tree/master/src/docs-content/theming
I am going to close this since it is resolved. 🙂


Scoped CSS

There is an example of whole-app-scoped css in src/style, but what about module-scoped css ?

you should use the modules feature of webpack's css loader: https://github.com/webpack-contrib/css-loader#modules


Validate RSI Calculation

When running the RSI the resulting output doesnt align with any other library.
   close = [       172.789993, 172.550003, 168.880005, 167.300003, 164.320007, 160.070007, 162.740005, 164.850006, 165.119995, 163.199997, 166.56, 166.23,       163.17, 159.3, 157.44, 162.95, 158.52, 154.73, 150.62, 155.09, 159.59, 160.62, 163.98, 165.38, 168.82, 170.21, 174.07, 174.72, 175.6,       178.96, 177.77, 174.61, 174.31, 178.44, 175.06, 171.83, 172.14, 170.09, 165.75, 167.66, 170.4, 165.29, 165.07, 167.4, 167.23, 166.42,       161.79, 162.88, 156.8, 156.57     ];
 const rsi14 = await ta.rsi(close, 14);
result   [       27.74661147031628, 26.33885045671019, 41.220006984308256, 38.01200628563883, 37.18940713554575, 37.32902951513339, 40.214880069939134,       43.659586224586825, 44.673301499211625, 50.892860515600226, 48.58581016299136, 52.88869060896722, 58.155699721964815, 67.11471610660485,       70.60085836909873, 66.9525596354865, 78.20088300220749, 84.24494649227115, 86.69317834200062, 83.69565217391303, 83.48134991119004,       73.67213114754098, 62.9239380968061, 61.54371584699453, 52.27680172104699, 42.769130998702956, 38.9061959155417, 43.02775984506133,       35.35927293382561, 28.344246959775447, 34.38723276121648, 37.7895433487756, 37.162382037097274, 23.34293948126799, 28.956461644782294,       26.360490720352303, 25.449385052034103     ];
the same result from using python
df['rsi(14)'] = ta.momentum.rsi(df['close'], 14)
result from python
[53.11352    52.70502392 46.78009175 44.46268251 40.39784045 35.42388839
 40.38950504 44.05092927 44.52052663 41.83171379 47.77591931 47.26505326
 42.70500973 37.74520229 35.60486603 45.46954028 40.14482839 36.23527393
 32.53498553 39.7422035  45.99678593 47.34401732 51.58700049 53.27640362
 57.22611274 58.74372189 62.70132145 63.33907943 64.23079379 67.48297331
 65.22121366 59.51673278 58.98923713 63.75217812 57.83246836 52.78804561
 53.20987594 50.02667325 44.02232863 47.03519379 51.1011733  44.27480827
 44.00227457 47.67580796 47.43132054 46.21523428 39.91551047 41.9226311
 34.91615995 34.68003641]

the python result alights with trading view and other apps
is there something different here with the calculations or am I doing something wrong?

The tradingview version of the RSI is actually the WRSI indicator.
You should be fine when you use ta.wrsi
My results match up with tradingview.
Let me know if it's still not working.

I can confirm its working as expected...strange that everyone else calls it rsi...
Any chance to make a dart/flutter library :)
Thanks again for the good work.

Yes, I will keep a dart/flutter library in mind.
Don't expect it any time soon, however.


Credits?

https://github.com/Tomato6966/Musicium

Already check readme

wrong credits
99% of the bot is from me, change it!

Ok
On Sun, Sep 12, 2021, 7:23 PM Tomato6966 @.***> wrote:

wrong credits
99% of the bot is from me, change it!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/KabirJaipal/JUGNU-MUSIC/issues/2#issuecomment-917640612,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARSWFKV6THTS6FWUN3FZYCLUBSWELANCNFSM5D355TVQ
.

Ok


[ownership] Move ownership elimination on the stdlib passed lower aggregate instrs.

@swift-ci test

@swift-ci smoke benchmark

Performance: -O

Regression
OLD
NEW
DELTA
RATIO

FlattenListLoop
1630
2513
+54.2%
0.65x (?)

FlattenListFlatMap
5341
6504
+21.8%
0.82x (?)

Improvement
OLD
NEW
DELTA
RATIO

StringFromLongWholeSubstring
5
4
-20.0%
1.25x

Code size: -O
Performance: -Osize

Improvement
OLD
NEW
DELTA
RATIO

StringFromLongWholeSubstring
5
4
-20.0%
1.25x

Code size: -Osize
Performance: -Onone

Regression
OLD
NEW
DELTA
RATIO

UTF8Decode_InitFromCustom_contiguous_ascii
375
438
+16.8%
0.86x (?)

UTF8Decode_InitDecoding_ascii
339
388
+14.5%
0.87x (?)

UTF8Decode_InitFromCustom_contiguous_ascii_as_ascii
480
536
+11.7%
0.90x (?)

Code size: -swiftlibs

How to read the data
The tables contain differences in performance which are larger than 8% and
differences in code size which are larger than 1%.
If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.
Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB


Eliminates misuse of connection.exec API

Fixes #579.

Bonus: I believe this issue was causing nodeunit failures every once in awhile where the tests just never complete.


GEODE-4691: Gfsh command Start Locator shows inconsistent behavior wi…

…th auto-connecting with a Security Manager
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[ ] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to<EMAIL_ADDRESS>

FYI. pre-checkin is green.


Windows Tuners: Use the same polling for signal lock as for Linux

Maybe use the same polling option as for Linux

Hi @SS-FF,
What you propose is in fact the original implementation of TSDuck tuner on Windows. Since the first TSDuck version was developed on Linux, the Windows implementation was more or less inspired from the original Linux implementation. But, after some time, I realized that some new tuners (or more precisely their proprietary Windows drivers) were unable to correctly report the state of the tuner or some parts of it. This includes current tuning parameters, state of signal (locked, quality, strength). So, with these tuners, the tuning operation "failed" because no lock state was ever reported.
As a last resort, I implemented the signal check as the read of one packet. It was clear that some tuners could actually read packets while reporting "no signal lock", "strength 0%" or "quality 0%".
On the other hand, the same tuner could correctly report its current tuning parameters and signal state on Linux, with the open source drivers.
The example which particularly frustrated me was the following. On DVB-T (unlike DVB-S), you don't really need to know all tuning parameters (FEC, guard interval, etc). As long as you know the frequency, just tune and the tuner will adapt. Even better, at least on Linux, when you read back the tuning parameters, you get the actual parameters. This is very handy when you need to draw a map a DVB-T network characteristics (a task that was useful to me at the time). But, after porting the tuner part of TSDuck to Windows to move to various areas using my WIndows laptop, I soon realized that the same tuner was unable to report the actual parameters on Windows. It just returned the dummy / default parameters I gave on input. This was of course a driver issue because the same information was available on Linux. And that was the case for all tuners I tested on Windows. Note that this is the reason why I developed the ability for tsterinfo to "guess" the tuning parameters based on the bitrate. The first tuners I used on Windows were however able to report "signal lock". But, after testing some others, I realized that was not the case for all of them and that "signal locked" was, at best, not a trustable information on Windows.
Since this was a long time ago, we are tempted to say that the situation should be better now. This is unfortunately the opposite. The bad driver implementation has event contaminated the Linux area. Now, even some Linux drivers can't report the signal quality properly to such extent that I had to simply disable the feature on Linux (report "not implemented" status).
The lessons I learned is that it is dangerous to rely on the behaviour of one specific tuner to change the code, on Windows from the beginning and, quite unfortunately, more and more on Linux. The current code is the result of many changes based on my or other user's experience (which includes yours for some other parts).
So, I am sorry, but I cannot replace the "read one initial packet" by "read signal locked", it won't work on some tuners.

The reporting of actual tuning parameters and signal-locked state are two different things.
The signal-locked state seems to be an intrinsic feature because it is also used for the event mechanism with the ITuner::TriggerSignalEvents method.
Anyway, I'll report back once I got some feedback from our testing group.


Remove a piece of code which was duplicate.

https://github.com/way-cooler/way-cooler/blob/2c6e6f17b8e9cb547b913344f9ac9b512e06e555/awesome/src/main.rs#L205-L235
https://github.com/way-cooler/way-cooler/blob/2c6e6f17b8e9cb547b913344f9ac9b512e06e555/awesome/src/keygrabber.rs#L114-L134

Thanks!


Add Heart-Breaking ritual

add ritual to enchant a weapon with heart-breaking
Add first user player guide

I have to admit that I've missed to mention that I changed a core mechanic.
Old behavior
You can add up to 5 stacks of any item to the ritual bowl. Nevertheless, only the type is been checked and all items over 1 are destroyed.
All items are consumed during a ritual.
New behavior
You can add exactly 5 items ( 5 ItemStack instances with size 1 ), we call them ingredients. Also you can now apply a specialItem as soon as the fire starts. This item is passed to all applied effects so that they can apply side effects.
As the ingredient items are consumed by a ritual. The special item should not.

@raphaelzoellner Can you test and verify everything is working as expected?


Running ddshow automatically.

I am adding an option to ddlog CLI to start ddshow before initializing the dataflow graph.  This should greatly simplify the profiling workflow as the user only needs to type one command and doesn't need to worry about providing matching arguments (socket address, number of workers).  Problem is, there is no easy way to know that ddshow is listening. Starting the graph before ddshow has called bind() returns an error. I noticed that ddshow prints a message to stdout on startup ("Waiting for 2 Timely connections on <IP_ADDRESS>:51317..."), so I thought of to capturing its stdout and waiting for the message; however I checked the source and it does this before actually calling bind.
It's be great to have a more reliable startup protocol (any ideas?) or at least reorder things to print the message once ddshow is actually listening...

Hum, that sounds like it'd be a good cli switch, I'm not entirely sure what would actually be done though

PS. The new version of ddshow no longer prints anything on stdout, so I gave up on trying to implement a reliable status check and just wait for few hundred milliseconds in the hope that ddshow will be ready by then.


"'Microsoft.NET.Sdk' could not be found" when running VS Code from Git Bash

Environment data
dotnet --info output:
.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Users\nosikil\scoop\apps\dotnet-sdk\current\sdk\2.0.0\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

VS Code version: 1.15.0
C# Extension version: 1.12.1
Steps to reproduce
In Git Bash:
dotnet new console
code.cmd .

Note that Git, the .NET SDK and VS Code are installed with Scoop on my machine.
Expected  behavior
This is what I get when running VS Code from the console or PowerShell:
Starting OmniSharp server at 14.9.2017, 15:09:52
    Target: d:\temp

OmniSharp server started
    Path: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\OmniSharp.exe
    PID: 5580

OmniSharp: -s d:\temp --hostPID 4480 --stdio DotNet:enablePackageRestore=false --encoding utf-8 --loglevel information formattingOptions:useTabs=false formattingOptions:tabSize=4 formattingOptions:indentationSize=4
[info]: OmniSharp.Startup
        Omnisharp server running using Stdio at location 'd:\temp' on host 4480.
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Initializing in d:\temp
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Auto package restore: False
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Update workspace context
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Resolving projects references
[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        Omnisharp will use local MSBuild.
    MSBUILD_EXE_PATH: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild\MSBuild.exe
    MSBuildExtensionsPath: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild

[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        No solution files found in 'd:\temp'
[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        Loading project: d:\temp\temp.csproj

[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        Update project: temp
[info]: OmniSharp.Script.ScriptProjectSystem
        Detecting CSX files in 'd:\temp'.
[info]: OmniSharp.Script.ScriptProjectSystem
        Could not find any CSX files
[info]: OmniSharp.Services.WorkspaceHelper
        Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpWorkspaceOptionsProvider
[info]: OmniSharp.Startup
        Configuration finished.

Actual behavior
This is what I get with Git Bash:
Starting OmniSharp server at 14.9.2017, 15:06:24
    Target: d:\temp

OmniSharp server started
    Path: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\OmniSharp.exe
    PID: 12952

OmniSharp: -s d:\temp --hostPID 7308 --stdio DotNet:enablePackageRestore=false --encoding utf-8 --loglevel information formattingOptions:useTabs=false formattingOptions:tabSize=4 formattingOptions:indentationSize=4
[info]: OmniSharp.Startup
        Omnisharp server running using Stdio at location 'd:\temp' on host 7308.
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Initializing in d:\temp
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Auto package restore: False
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Update workspace context
[info]: OmniSharp.DotNet.DotNetProjectSystem
        Resolving projects references
[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        Omnisharp will use local MSBuild.
    MSBUILD_EXE_PATH: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild\MSBuild.exe
    MSBuildExtensionsPath: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild

[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        No solution files found in 'd:\temp'
[info]: OmniSharp.MSBuild.MSBuildProjectSystem
        Loading project: d:\temp\temp.csproj
[warn]: OmniSharp.MSBuild.MSBuildProjectSystem
        Failed to load project file 'd:\temp\temp.csproj'.
d:\temp\temp.csproj(1,1)
Microsoft.Build.Exceptions.InvalidProjectFileException: The SDK 'Microsoft.NET.Sdk' specified could not be found.  d:\temp\temp.csproj
   at Microsoft.Build.Shared.ProjectErrorUtilities.ThrowInvalidProject(String errorSubCategoryResourceName, IElementLocation elementLocation, String resourceName, Object[] args)
   at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImportsFromUnescapedImportExpressionConditioned(String directoryOfImportingFile, ProjectImportElement importElement, List`1& projects, Boolean throwOnFileNotExistsError)
   at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImports(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.Evaluate(ILoggingService loggingService, BuildEventContext buildEventContext)
   at Microsoft.Build.Evaluation.Project.Reevaluate(ILoggingService loggingServiceForEvaluation, ProjectLoadSettings loadSettings)
   at Microsoft.Build.Evaluation.Project.ReevaluateIfNecessary(ILoggingService loggingServiceForEvaluation, ProjectLoadSettings loadSettings)
   at Microsoft.Build.Evaluation.Project.Initialize(IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectLoadSettings loadSettings)
   at Microsoft.Build.Evaluation.Project..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectCollection projectCollection, ProjectLoadSettings loadSettings)
   at Microsoft.Build.Evaluation.ProjectCollection.LoadProject(String fileName, IDictionary`2 globalProperties, String toolsVersion)
   at OmniSharp.MSBuild.ProjectFile.ProjectFileInfo.LoadProject(String filePath, String solutionDirectory, String sdksPath, ILogger logger, MSBuildOptions options, ICollection`1 diagnostics, ImmutableArray`1& targetFrameworks)
   at OmniSharp.MSBuild.ProjectFile.ProjectFileInfo.Create(String filePath, String solutionDirectory, String sdksPath, ILogger logger, MSBuildOptions options, ICollection`1 diagnostics)
   at OmniSharp.MSBuild.MSBuildProjectSystem.LoadProject(String projectFilePath)

[info]: OmniSharp.Script.ScriptProjectSystem
        Detecting CSX files in 'd:\temp'.
[info]: OmniSharp.Script.ScriptProjectSystem
        Could not find any CSX files
[info]: OmniSharp.Services.WorkspaceHelper
        Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpWorkspaceOptionsProvider
[info]: OmniSharp.Startup
        Configuration finished.

After launching VS Code from Git Bash, could you try running dotnet --info from within VS Code's Integrated Terminal and pasting the output here?

It's the same as above:
.NET Command Line Tools (2.0.0)

Product Information:
 Version:            2.0.0
 Commit SHA-1 hash:  cdcd1928c9

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Users\nosikil\scoop\apps\dotnet-sdk\current\sdk\2.0.0\

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.0
  Build    : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

Thanks! I'll get WSL set up on my machine and take a look.

This has nothing to do with WSL, just plain Windows + Bash which comes with Git.
Can I make the output more verbose, to see what parameters MSBuild gets?

Thanks! I was thinking this would be related to another issue I was talking to somebody about with WSL. Glad I'm wrong. :smile:
Try setting the "omnisharp.loggingLevel": "debug" option in VS Code's preferences. That might provide a few clues in the log.

Thanks, I'll try that on Monday.

The MSBuild parameters are the same for both cases:
[dbug]: OmniSharp.MSBuild.MSBuildProjectSystem
        MSBuild environment:
Mode: Standalone
RunningTests: False
RunningInVisualStudio: False
MSBuildToolsDirectory32: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild
MSBuildToolsDirectory64: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild
MSBuildSDKsPath: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild\Sdks
CurrentMSBuildConfigurationFile: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild\MSBuild.exe.config
CurrentMSBuildExePath: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild\MSBuild.exe
CurrentMSBuildToolsDirectory: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild
VisualStudioInstallRootDirectory:
MSBuildExtensionsPath: C:\Users\nosikil\.vscode\extensions\ms-vscode.csharp-1.12.1\.omnisharp\msbuild

The only difference I see before I get the warning message with the exception is this:
[dbug]: OmniSharp.MSBuild.ProjectFile.ProjectFileInfo
        Using MSBuildSDKsPath: C:\Users\nosikil\scoop\apps\dotnet-sdk\current\sdk\2.0.0\Sdks

This message is not logged when I run VS Code from Bash.
To me this looks like MSBuild searches for the SDKs in some default directory, probably somewhere in .vscode\ms-vscode.csharp-1.12.1\.omnisharp\, when I run VS Code from Bash. I guess I need a debug build of OmniSharp.

The message should be logged when this call adds something to the globalProperties dictionary.

I think I tracked it down. This is very similar to dotnet/cli#3995 and dotnet/corefx#13146.
When I'm in Bash, I have the home and HOME environment variables. I think this breaks the interaction between OmniSharp and dotnet --info, which then probably returns some default infos. When I do unset home before running VS Code, everything works as expected. You can reproduce it when you do foo=1 FOO=1 code . in Bash.

Thanks for the lead! At the very least, I'll add some logging around this to see if we can make it clearer when this fails. It's a little tricky because it's not exactly an error case when the .NET Core SDK doesn't work. After all, we support .NET Framework, Unity and Xamarin projects. However, it should be easier to trace the issue.

Ok, it looks like I managed to solve it by running dotnet restore on the project whcih fixed the omnisharp server which is now starting. After this I deleted a .env file and re-started VSCode to update it, after which things started working.

Another data point...  I just upgraded dotnet using chocolatey (was 2.1.101 or something, now 2.1.403).  Then created a new project:
dotnet new console --name New.Project
dotnet new sln --name New.Project
dotnet sln add .\New.Project\New.Project.csproj
dotnet build
code .

Which results in this OUTPUT:
Starting OmniSharp server at 10/13/2018, 1:52:37 PM
    Target: c:\Users\lucas\git\new-project\New.Project.sln

OmniSharp server started.
    Path: C:\Users\lucas\.vscode\extensions\ms-vscode.csharp-1.16.2\.omnisharp\1.32.5\OmniSharp.exe
    PID: 9056

[info]: OmniSharp.Stdio.Host
        Starting OmniSharp on Windows 6.2.9200.0 (x64)
[info]: OmniSharp.Services.DotNetCliService
        DotNetPath set to dotnet
[info]: OmniSharp.MSBuild.Discovery.MSBuildLocator
        Located 2 MSBuild instance(s)
            1: Visual Studio Build Tools 2017 15.7.27703.2000 - "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin"
            2: StandAlone 15.0 - "C:\Users\lucas\.vscode\extensions\ms-vscode.csharp-1.16.2\.omnisharp\1.32.5\msbuild\15.0\Bin"
[info]: OmniSharp.MSBuild.Discovery.MSBuildLocator
        Registered MSBuild instance: Visual Studio Build Tools 2017 15.7.27703.2000 - "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin"
[info]: OmniSharp.Cake.CakeProjectSystem
        Detecting Cake files in 'c:\Users\lucas\git\new-project'.
[info]: OmniSharp.Cake.CakeProjectSystem
        Could not find any Cake files
[info]: OmniSharp.WorkspaceInitializer
        Project system 'OmniSharp.DotNet.DotNetProjectSystem' is disabled in the configuration.
[info]: OmniSharp.MSBuild.ProjectSystem
        Detecting projects in 'c:\Users\lucas\git\new-project\New.Project.sln'.
[info]: OmniSharp.MSBuild.ProjectManager
        Queue project update for 'c:\Users\lucas\git\new-project\New.Project\New.Project.csproj'
[info]: OmniSharp.Script.ScriptProjectSystem
        Detecting CSX files in 'c:\Users\lucas\git\new-project'.
[info]: OmniSharp.Script.ScriptProjectSystem
        Could not find any CSX files
[info]: OmniSharp.WorkspaceInitializer
        Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpWorkspaceOptionsProvider
[info]: OmniSharp.WorkspaceInitializer
        Configuration finished.
[info]: OmniSharp.Stdio.Host
        Omnisharp server running using Stdio at location 'c:\Users\lucas\git\new-project' on host 16340.
[info]: OmniSharp.MSBuild.ProjectManager
        Loading project: c:\Users\lucas\git\new-project\New.Project\New.Project.csproj
[warn]: OmniSharp.MSBuild.ProjectManager
        Failed to load project file 'c:\Users\lucas\git\new-project\New.Project\New.Project.csproj'.
c:\Users\lucas\git\new-project\New.Project\New.Project.csproj(1,1)
Microsoft.Build.Exceptions.InvalidProjectFileException: The SDK 'Microsoft.NET.Sdk' specified could not be found.  c:\Users\lucas\git\new-project\New.Project\New.Project.csproj
   at Microsoft.Build.Shared.ProjectErrorUtilities.ThrowInvalidProject(String errorSubCategoryResourceName, IElementLocation elementLocation, String resourceName, Object[] args)
   at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImportsFromUnescapedImportExpressionConditioned(String directoryOfImportingFile, ProjectImportElement importElement, List`1& projects, Boolean throwOnFileNotExistsError)
   at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImports(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.Evaluate(ILoggingService loggingService, BuildEventContext buildEventContext)
   at Microsoft.Build.Evaluation.Project.Reevaluate(ILoggingService loggingServiceForEvaluation, ProjectLoadSettings loadSettings)
   at Microsoft.Build.Evaluation.Project.ReevaluateIfNecessary(ILoggingService loggingServiceForEvaluation, ProjectLoadSettings loadSettings)
   at Microsoft.Build.Evaluation.Project.ReevaluateIfNecessary(EvaluationContext evaluationContext)
   at Microsoft.Build.Evaluation.Project.Initialize(IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectLoadSettings loadSettings, EvaluationContext evaluationContext)
   at Microsoft.Build.Evaluation.Project..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectCollection projectCollection, ProjectLoadSettings loadSettings, EvaluationContext evaluationContext)
   at Microsoft.Build.Evaluation.ProjectCollection.LoadProject(String fileName, IDictionary`2 globalProperties, String toolsVersion)
   at OmniSharp.MSBuild.ProjectLoader.EvaluateProjectFileCore(String filePath)
   at OmniSharp.MSBuild.ProjectLoader.BuildProject(String filePath)
   at OmniSharp.MSBuild.ProjectFile.ProjectFileInfo.Load(String filePath, ProjectLoader loader)
   at OmniSharp.MSBuild.ProjectManager.LoadOrReloadProject(String projectFilePath, Func`1 loader)

[fail]: OmniSharp.MSBuild.ProjectManager
        Attemped to update project that is not loaded: c:\Users\lucas\git\new-project\New.Project\New.Project.csproj

My dotnet --info output is:
PS C:\Users\lucas\git\new-project> dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.403
 Commit:    04e15494b6

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.403\

Host (useful for support):
  Version: 2.1.5
  Commit:  290303f510

.NET Core SDKs installed:
  2.1.105 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

My guess is that it has something to do with this:
[info]: OmniSharp.MSBuild.Discovery.MSBuildLocator
        Located 2 MSBuild instance(s)
            1: Visual Studio Build Tools 2017 15.7.27703.2000 - "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin"
            2: StandAlone 15.0 - "C:\Users\lucas\.vscode\extensions\ms-vscode.csharp-1.16.2\.omnisharp\1.32.5\msbuild\15.0\Bin"

Which I would imagine might have dotnet mentioned (as in dotnet msbuild).  My dotnet install is at:
PS C:\Users\lucas\git\new-project> get-command dotnet

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     dotnet.exe                                         2.1.269... C:\Program Files\dotnet\dotnet.exe

i installed .NET Core SDKs: 2.2.104 AND 3.0.100-preview-010184. (ubuntu 18)
and had same problem.
but after creating global.json in my project folder and defining 2.2.104 as my sdk, this problem solved.

Closing as a duplicate of #3541 which will track these issues


fix(confirmation): fixed re-rendering of custom screen

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

4 of 4 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.004%) to 84.507%

Totals

Change from base Build<PHONE_NUMBER>:
0.004%

Covered Lines:
9369

Relevant Lines:
10256

💛  - Coveralls


Missing device configuration: CLEVERIO Smart Smoke Alarm

Checklist

[X] It is not in the configuration DB

[X] It was not merged recently or has a pending PR

Which device is missing?
CLEVERIO Smart Smoke Alarm
Manufacturer ID
0x041C
Product Type
0x8002
Product ID
0x1000
Firmware Version
3.01
Is the device listed on the Z-Wave Alliance website?
https://products.z-wavealliance.org/products/3365
Do you have a manual?

@zwave-js-bot import config 3365 from zwa

We do have a config file for this:
https://devices.zwave-js.io/?jumpTo=0x041c:0x8002:0x1000:0.0


Model factory pattern: ModelProps type issues

I'm trying to migrate my entities store pattern into mobx-keystone with typescript and it's pretty hard for me since my ts isn't really good.
Here is my entities store factory:
https://t.co/dKnklXwgnF
Here is usage:
https://t.co/4kpAzmAxCZ
It receives just a collection store I defined here: https://t.co/M0JNTErYQL
The problem with that part. It creates a map with all my collection stores and passes it as model props:
https://t.co/uSqmius1zF
But seems like that part isn't working since ts doesn't resolve todos prop of my entities store which is one of my collection stores:
https://t.co/rImQry1YES
Maybe you have some suggestions on how to trick it?

I took a look, the problem seems to lie in the createEntitiesStore typing, where it downgrades the argument property to a generic ModelProps (which has no actual property names).
Have you considered using an ObjectMap (or something similar) inside maybe a custom model class as entity store?

Managed to get it working by doing something like that:
export function createEntitiesStore(
  collectionStores: ModelProps = {},
) {
  class EntitiesStore extends Model(collectionStores) {
    @modelAction merge(normalizedEntities: { [k: string]: Object }) {}
	normalizeMerge(item: any, schema: Schema) {}
  }

  return EntitiesStore;
}

@model('EntitiesStore')
export class EntitiesStore extends ExtendedModel(
  createEntitiesStore(),
  {
    todos: prop<ITodoCollectionStore>(
      () => new TodoCollectionStore({}),
    ),
  },
) {
  // you can add all the async actions here
}

How ModalProps works? If it does not have all the props you're passing to it, how it knows what did I pass then? If I'm passing all my collection stores using  createEntitiesStore({}), it doesn't work.

Ok, I've got it working, will leave here, maybe will be useful for somebody
// definition
export function EntitiesModel<
  TBaseProps extends {
    [k: string]: ModelProp<any, any>;
  }
>(collectionStores: TBaseProps) {
  class EntitiesStore extends Model(collectionStores as ModelProps) {
    // some additional actions/computed
  }

  return ExtendedModel(EntitiesStore, collectionStores);
}

// actual store:

@model('EntitiesStore')
export class EntitiesStore extends EntitiesModel({
  todos: prop<ITodoCollectionStore>(
    () => new TodoCollectionStore({}),
  ),
  groups: prop<IGroupCollectionStore>(
    () => new GroupCollectionStore({}),
  ),
}) {}

export type IEntityStore = InstanceType<typeof EntitiesStore>;

const store = new EntitiesStore({});
store.todos // exists

Hey, I'm really glad you got it working!
Just wondering, are you really using the IEntityStore type? You should be able to just use EntitiesStore directly since the (final) class doesn't come from a factory

Oh, and just pushed a commit to make those public, so they will be in the next version :)

@xaviergonz you're right. thanks, haven't changed it since I've used a different approach :)


New package: Licenses v0.1.0

Registering package: Licenses
Repository: https://github.com/cadojo/Licenses.jl
Created by: @cadojo
Version: v0.1.0
Commit: 320135bb647c35fcdcd581278d7bdd7a60e6ede9
Reviewed by: @cadojo
Reference: https://github.com/cadojo/Licenses.jl/commit/320135bb647c35fcdcd581278d7bdd7a60e6ede9#commitcomment-113266587
Description: Provide executable licenses for your executable documents!

I feel conspicuous registering a package with such a common name. See my Discourse Post on the topic. Please let me know if you have alternative names, or guidance.

[noblock]
I feel conspicuous registering a package with such a common name. See my Discourse Post on the topic. Please let me know if you have alternative names, or guidance.

How about SoftwareLicenses.jl or ContentLicenses.jl?

How about SoftwareLicenses.jl or ContentLicenses.jl?

Yeah we talked about SoftwareLicenses in the Discourse post, that title feels a bit misleading to me because this package also pulls licenses for other forms of content, like writing and visual media with Creative Commons licenses.
For ContentLicenses, I feel the word Content doesn't add much. But what about CommonLicenses — all of the licenses provided by this package are "standard" or "common", so a word like that adds something to the meaning of the package name imo.
I am totally okay with merging Licenses.jl, but I wanted to give maintainers and the general community the opportunity to veto Licenses.jl. I'm going to change the package name to CommonLicenses.jl. If there are no objections, could you please add [noblock] to your comment @DilumAluthge?

Huh, didn't realize a new pull request would be created after the package rename since it has the same UUID! This PR should be closed; see https://github.com/JuliaRegistries/General/pull/83584.