Binomial error or Poisson error

It says in the docs:
"The error_method does not specify how the histograms or functions were filled, but how the fraction should be interpreted statistically. It may be unspecified, leaving that interpretation unspecified. The normal method (sometimes called “Wald”) is a naive binomial interpretation, in which zero passing or zero failing values are taken to have zero uncertainty. The clopper_pearson method (sometimes called “exact”) is a common choice, though it fails in some statistical criteria. The computation and meaning of the methods are described in the references below."
If Aghast is only about exchanging data, is it necessary to specify the error_method? This is not part of the data, but part of the interpretation. You follow here the assumption that histogram counts should be modeled as binomial proportions, but in HEP that usually does not make sense. We use the Poisson model not only because it is easier, but because it is the better approximation of reality.
Example. You want to measure the differential cross-section of some process p+p -> X as a function of a kinematic variable, let's say eta. You count how often a particle X falls into an eta bin, call that \Delta N. The differential cross-section then is:
d\sigma / d\eta = 1/L \Delta N / \Delta \eta
where L is the luminosity. I use "=" here, although strictly speaking the equality requires the bin width to go to zero.
You care about the uncertainty of d\sigma / d\eta. You don't know the total sum of all \Delta N here, because you have not observed all X, only those in the acceptance of your detector, so you cannot compute the binomial proportional error. The measurement also doesn't depend on the fraction of events in each bin, but really on the number, which you divide by L and not sum over \Delta N. The Poisson is more correct here.
It is not exactly correct either, because the bins are not really independent for a physics reason: you observe K particles per event, and these K particles are not generated independently, they form jets and have some correlations. We usually neglect these correlations. They are not correctly represented by either Poisson or binomial proportion errors.

I think one of the options for this field is "no interpretation." It doesn't need to be filled.
Some of the data in these objects are necessary for the various histogram → histogram conversions it needs to perform. Some only set the context, like this one. A date analyst would strongly desire to attach such information, as it would simplify bookkeeping, and much of it may be information we haven't thought of: there are JSON-valued metadata fields for that. But some metadata are significant enough to have a special form, so that each user doesn't invent their own format.
Also, I'm not sure you're interpreting FractionBinning right: it's for "efficiency plots." The counts in each corresponding bin of the numerator and denominator are highly dependent, but neighboring bins are almost always independent. If neighboring bins are not independent, then all of the error methods listed here would be inapplicable.


[SUPPORT] Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables

Spark structured Streaming writes to Hudi and synchronizes Hive to create only read-optimized tables without creating real-time tables   , no errors happening
Environment Description

Hudi version :0.6.0

Spark version : 2.4.4

Hive version : 2.3.7

Hadoop version : 2.7.5

Storage (HDFS/S3/GCS..) : HDFS

Running on Docker? (yes/no) : no

code as follows:
batchDF.write.format("org.apache.hudi")
.option(DataSourceWriteOptions.TABLE_TYPE_OPT_KEY, "MERGE_ON_READ")
.option(DataSourceWriteOptions.OPERATION_OPT_KEY, "upsert")
.option(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP, "10")
.option("hoodie.datasource.compaction.async.enable", "true")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "rec_id")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "modified")
.option(DataSourceWriteOptions.HIVE_DATABASE_OPT_KEY, "ads")
.option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY, hiveTableName)
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "dt")
.option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, "dt")
.option(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING_OPT_KEY, "true")
.option(HoodieWriteConfig.TABLE_NAME, hiveTableName)
.option(HoodieIndexConfig.BLOOM_INDEX_UPDATE_PARTITION_PATH, "true")
.option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY, "true")
.option(DataSourceWriteOptions.HIVE_URL_OPT_KEY, "jdbc:hive2://<IP_ADDRESS>:10000")
.option(DataSourceWriteOptions.HIVE_USER_OPT_KEY, "")
.option(DataSourceWriteOptions.HIVE_PASS_OPT_KEY, "")
.option(DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY, classOf[MultiPartKeysValueExtractor].getName)
.option(HoodieIndexConfig.INDEX_TYPE_PROP, HoodieIndex.IndexType.GLOBAL_BLOOM.name())
.option("hoodie.insert.shuffle.parallelism", "10")
.option("hoodie.upsert.shuffle.parallelism", "10")
.mode("append")
.save("/data/mor/user")
only create user_ro ,  no user_rt

can you copy the contents of hoodie.properties of the dataset here ?

can you copy the contents of hoodie.properties of the dataset here ?

can you copy the contents of hoodie.properties of the dataset here ?
hoodie.properties as follows:
hoodie.zip

can you copy the contents of hoodie.properties of the dataset here ?
hoodie.properties as follows:
hoodie.zip

It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message
"Trying to sync hoodie table "

It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message
"Trying to sync hoodie table "

It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message
"Trying to sync hoodie table "

error as follows:
there is no sql about  creating _rt table , only _ro table
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 371
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 337
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 404
21/01/07 13:23:05 INFO BlockManagerInfo: Removed broadcast_16_piece0 on bigdatadev03:18850 in memory (size: 73.0 KB, free: 2.5 GB)
21/01/07 13:23:05 INFO BlockManagerInfo: Removed broadcast_16_piece0 on bigdatadev02:6815 in memory (size: 73.0 KB, free: 3.5 GB)
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 333
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 418
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 385
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 410
21/01/07 13:23:08 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:08 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:10 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
21/01/07 13:23:10 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
21/01/07 13:23:10 INFO ObjectStore: Initialized ObjectStore
21/01/07 13:23:11 INFO HiveMetaStore: Added admin role in metastore
21/01/07 13:23:11 INFO HiveMetaStore: Added public role in metastore
21/01/07 13:23:11 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_all_databases
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_all_databases
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*
21/01/07 13:23:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_functions: db=dw pat=*
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_functions: db=dw pat=*
21/01/07 13:23:11 INFO HiveSyncTool: Trying to sync hoodie table api_trade_ro with base path /data/stream/mor/api_trade of type MERGE_ON_READ
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_table : db=ads tbl=api_trade_ro
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_table : db=ads tbl=api_trade_ro
21/01/07 13:23:11 INFO HoodieHiveClient: Found the last compaction commit as Option{val=null}
21/01/07 13:23:11 INFO HoodieHiveClient: Found the last delta commit Option{val=[20210107132154__deltacommit__COMPLETED]}
21/01/07 13:23:12 INFO HoodieHiveClient: Reading schema from /data/stream/mor/api_trade/dt=2021-01/350a9a01-538c-4a7e-8c17-09d2cdc85073-0_0-20-85_20210107132154.parquet
21/01/07 13:23:12 INFO HiveSyncTool: Hive table api_trade_ro is not found. Creating it
21/01/07 13:23:12 INFO HoodieHiveClient: Creating table with CREATE EXTERNAL TABLE  IF NOT EXISTS ads.api_trade_ro( _hoodie_commit_time string, _hoodie_commit_seqno string, _hoodie_record_key string, _hoodie_partition_path string, _hoodie_file_name string, topic string, kafka_partition string, kafka_timestamp string, kafka_offset string, current_time string, kafka_key string, kafka_value string,  modified string, created string, batch_time string) PARTITIONED BY (dt string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION '/data/stream/mor/api_trade'
21/01/07 13:23:12 INFO HoodieHiveClient: Executing SQL CREATE EXTERNAL TABLE  IF NOT EXISTS ads.api_trade_ro( _hoodie_commit_time string, _hoodie_commit_seqno string, _hoodie_record_key string, _hoodie_partition_path string, _hoodie_file_name string, topic string, kafka_partition string, kafka_timestamp string, kafka_offset string, current_time string, kafka_key string, kafka_value string,  modified string, created string, batch_time string) PARTITIONED BY (dt string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION '/data/stream/mor/api_trade'
21/01/07 13:24:09 INFO HiveSyncTool: Schema sync complete. Syncing partitions for api_trade_ro
21/01/07 13:24:09 INFO HiveSyncTool: Last commit time synced was found to be null
21/01/07 13:24:09 INFO HoodieHiveClient: Last commit time synced is not known, listing all partitions in /data/stream/mor/api_trade,FS :DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-398057260_1, ugi=root (auth:SIMPLE)]]
21/01/07 13:24:09 INFO HiveSyncTool: Storage partitions scan complete. Found 1
21/01/07 13:24:09 INFO HiveMetaStore: 0: get_partitions : db=ads tbl=api_trade_ro
21/01/07 13:24:09 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_partitions : db=ads tbl=api_trade_ro
21/01/07 13:24:10 INFO HiveSyncTool: New Partitions [dt=2021-01]
21/01/07 13:24:10 INFO HoodieHiveClient: Adding partitions 1 to table api_trade_ro
21/01/07 13:24:10 INFO HoodieHiveClient: Executing SQL ALTER TABLE ads.api_trade_ro ADD IF NOT EXISTS   PARTITION (dt='2021-01') LOCATION '/data/stream/mor/api_trade/dt=2021-01'
21/01/07 13:24:33 INFO HiveSyncTool: Changed Partitions []
21/01/07 13:24:33 INFO HoodieHiveClient: No partitions to change for api_trade_ro
21/01/07 13:24:33 INFO HiveMetaStore: 0: get_table : db=ads tbl=api_trade_ro
21/01/07 13:24:33 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_table : db=ads tbl=api_trade_ro
21/01/07 13:24:33 ERROR HiveSyncTool: Got runtime exception when hive syncing
org.apache.hudi.hive.HoodieHiveSyncException: Failed to get update last commit time synced to<PHONE_NUMBER>2154
at org.apache.hudi.hive.HoodieHiveClient.updateLastCommitTimeSynced(HoodieHiveClient.java:658)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:128)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:91)
at org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:229)
at org.apache.hudi.HoodieSparkSqlWriter$.checkWriteStatus(HoodieSparkSqlWriter.scala:279)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:184)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:91)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at com.chin.dmp.stream.mor.ApiTradeStream$$anonfun$1.apply(ApiTradeStream.scala:196)
at com.chin.dmp.stream.mor.ApiTradeStream$$anonfun$1.apply(ApiTradeStream.scala:163)
at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: NoSuchObjectException(message:ads.api_trade_ro table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy41.get_table(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy42.getTable(Unknown Source)
at org.apache.hudi.hive.HoodieHiveClient.updateLastCommitTimeSynced(HoodieHiveClient.java:654)
... 48 more
21/01/07 13:24:33 INFO HiveMetaStore: 0: Shutting down the object store...
21/01/07 13:24:33 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=Shutting down the object store...
21/01/07 13:24:33 INFO HiveMetaStore: 0: Metastore shutdown complete.
21/01/07 13:24:33 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=Metastore shutdown complete.
21/01/07 13:24:33 INFO DefaultSource: Constructing hoodie (as parquet) data source with options :Map(hoodie.datasource.write.insert.drop.duplicates -> false, hoodie.datasource.hive_sync.database -> ads, hoodie.insert.shuffle.parallelism -> 10, path -> /data/stream/mor/api_trade, hoodie.datasource.write.precombine.field -> modified, hoodie.datasource.hive_sync.partition_fields -> dt, hoodie.datasource.write.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload, hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.MultiPartKeysValueExtractor, hoodie.datasource.write.streaming.retry.interval.ms -> 2000, hoodie.datasource.hive_sync.table -> api_trade, hoodie.index.type -> GLOBAL_BLOOM, hoodie.datasource.write.streaming.ignore.failed.batch -> true, hoodie.datasource.write.operation -> upsert, hoodie.datasource.hive_sync.enable -> true, hoodie.datasource.write.recordkey.field -> rec_id, hoodie.table.name -> api_trade, hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://<IP_ADDRESS>:10000, hoodie.datasource.write.table.type -> MERGE_ON_READ, hoodie.datasource.write.hive_style_partitioning -> true, hoodie.datasource.query.type -> snapshot, hoodie.bloom.index.update.partition.path -

It is indeed a MOR table.Can you check your driver logs. You might find some exceptions around registering _rt table. You can look for logs around the log message
"Trying to sync hoodie table "

error as follows:
there is no sql about  creating _rt table , only _ro table
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 371
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 337
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 404
21/01/07 13:23:05 INFO BlockManagerInfo: Removed broadcast_16_piece0 on bigdatadev03:18850 in memory (size: 73.0 KB, free: 2.5 GB)
21/01/07 13:23:05 INFO BlockManagerInfo: Removed broadcast_16_piece0 on bigdatadev02:6815 in memory (size: 73.0 KB, free: 3.5 GB)
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 333
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 418
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 385
21/01/07 13:23:05 INFO ContextCleaner: Cleaned accumulator 410
21/01/07 13:23:08 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:08 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:10 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
21/01/07 13:23:10 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
21/01/07 13:23:10 INFO ObjectStore: Initialized ObjectStore
21/01/07 13:23:11 INFO HiveMetaStore: Added admin role in metastore
21/01/07 13:23:11 INFO HiveMetaStore: Added public role in metastore
21/01/07 13:23:11 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_all_databases
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_all_databases
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*
21/01/07 13:23:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_functions: db=dw pat=*
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_functions: db=dw pat=*
21/01/07 13:23:11 INFO HiveSyncTool: Trying to sync hoodie table api_trade_ro with base path /data/stream/mor/api_trade of type MERGE_ON_READ
21/01/07 13:23:11 INFO HiveMetaStore: 0: get_table : db=ads tbl=api_trade_ro
21/01/07 13:23:11 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_table : db=ads tbl=api_trade_ro
21/01/07 13:23:11 INFO HoodieHiveClient: Found the last compaction commit as Option{val=null}
21/01/07 13:23:11 INFO HoodieHiveClient: Found the last delta commit Option{val=[20210107132154__deltacommit__COMPLETED]}
21/01/07 13:23:12 INFO HoodieHiveClient: Reading schema from /data/stream/mor/api_trade/dt=2021-01/350a9a01-538c-4a7e-8c17-09d2cdc85073-0_0-20-85_20210107132154.parquet
21/01/07 13:23:12 INFO HiveSyncTool: Hive table api_trade_ro is not found. Creating it
21/01/07 13:23:12 INFO HoodieHiveClient: Creating table with CREATE EXTERNAL TABLE  IF NOT EXISTS ads.api_trade_ro( _hoodie_commit_time string, _hoodie_commit_seqno string, _hoodie_record_key string, _hoodie_partition_path string, _hoodie_file_name string, topic string, kafka_partition string, kafka_timestamp string, kafka_offset string, current_time string, kafka_key string, kafka_value string,  modified string, created string, batch_time string) PARTITIONED BY (dt string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION '/data/stream/mor/api_trade'
21/01/07 13:23:12 INFO HoodieHiveClient: Executing SQL CREATE EXTERNAL TABLE  IF NOT EXISTS ads.api_trade_ro( _hoodie_commit_time string, _hoodie_commit_seqno string, _hoodie_record_key string, _hoodie_partition_path string, _hoodie_file_name string, topic string, kafka_partition string, kafka_timestamp string, kafka_offset string, current_time string, kafka_key string, kafka_value string,  modified string, created string, batch_time string) PARTITIONED BY (dt string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION '/data/stream/mor/api_trade'
21/01/07 13:24:09 INFO HiveSyncTool: Schema sync complete. Syncing partitions for api_trade_ro
21/01/07 13:24:09 INFO HiveSyncTool: Last commit time synced was found to be null
21/01/07 13:24:09 INFO HoodieHiveClient: Last commit time synced is not known, listing all partitions in /data/stream/mor/api_trade,FS :DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-398057260_1, ugi=root (auth:SIMPLE)]]
21/01/07 13:24:09 INFO HiveSyncTool: Storage partitions scan complete. Found 1
21/01/07 13:24:09 INFO HiveMetaStore: 0: get_partitions : db=ads tbl=api_trade_ro
21/01/07 13:24:09 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_partitions : db=ads tbl=api_trade_ro
21/01/07 13:24:10 INFO HiveSyncTool: New Partitions [dt=2021-01]
21/01/07 13:24:10 INFO HoodieHiveClient: Adding partitions 1 to table api_trade_ro
21/01/07 13:24:10 INFO HoodieHiveClient: Executing SQL ALTER TABLE ads.api_trade_ro ADD IF NOT EXISTS   PARTITION (dt='2021-01') LOCATION '/data/stream/mor/api_trade/dt=2021-01'
21/01/07 13:24:33 INFO HiveSyncTool: Changed Partitions []
21/01/07 13:24:33 INFO HoodieHiveClient: No partitions to change for api_trade_ro
21/01/07 13:24:33 INFO HiveMetaStore: 0: get_table : db=ads tbl=api_trade_ro
21/01/07 13:24:33 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=get_table : db=ads tbl=api_trade_ro
21/01/07 13:24:33 ERROR HiveSyncTool: Got runtime exception when hive syncing
org.apache.hudi.hive.HoodieHiveSyncException: Failed to get update last commit time synced to<PHONE_NUMBER>2154
at org.apache.hudi.hive.HoodieHiveClient.updateLastCommitTimeSynced(HoodieHiveClient.java:658)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:128)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:91)
at org.apache.hudi.HoodieSparkSqlWriter$.syncHive(HoodieSparkSqlWriter.scala:229)
at org.apache.hudi.HoodieSparkSqlWriter$.checkWriteStatus(HoodieSparkSqlWriter.scala:279)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:184)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:91)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at com.chin.dmp.stream.mor.ApiTradeStream$$anonfun$1.apply(ApiTradeStream.scala:196)
at com.chin.dmp.stream.mor.ApiTradeStream$$anonfun$1.apply(ApiTradeStream.scala:163)
at org.apache.spark.sql.execution.streaming.sources.ForeachBatchSink.addBatch(ForeachBatchSink.scala:35)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5$$anonfun$apply$17.apply(MicroBatchExecution.scala:537)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch$5.apply(MicroBatchExecution.scala:535)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$runBatch(MicroBatchExecution.scala:534)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:198)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:351)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:166)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:160)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:281)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: NoSuchObjectException(message:ads.api_trade_ro table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:1808)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1778)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy41.get_table(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1208)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
at com.sun.proxy.$Proxy42.getTable(Unknown Source)
at org.apache.hudi.hive.HoodieHiveClient.updateLastCommitTimeSynced(HoodieHiveClient.java:654)
... 48 more
21/01/07 13:24:33 INFO HiveMetaStore: 0: Shutting down the object store...
21/01/07 13:24:33 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=Shutting down the object store...
21/01/07 13:24:33 INFO HiveMetaStore: 0: Metastore shutdown complete.
21/01/07 13:24:33 INFO audit: ugi=root	ip=unknown-ip-addr	cmd=Metastore shutdown complete.
21/01/07 13:24:33 INFO DefaultSource: Constructing hoodie (as parquet) data source with options :Map(hoodie.datasource.write.insert.drop.duplicates -> false, hoodie.datasource.hive_sync.database -> ads, hoodie.insert.shuffle.parallelism -> 10, path -> /data/stream/mor/api_trade, hoodie.datasource.write.precombine.field -> modified, hoodie.datasource.hive_sync.partition_fields -> dt, hoodie.datasource.write.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload, hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.MultiPartKeysValueExtractor, hoodie.datasource.write.streaming.retry.interval.ms -> 2000, hoodie.datasource.hive_sync.table -> api_trade, hoodie.index.type -> GLOBAL_BLOOM, hoodie.datasource.write.streaming.ignore.failed.batch -> true, hoodie.datasource.write.operation -> upsert, hoodie.datasource.hive_sync.enable -> true, hoodie.datasource.write.recordkey.field -> rec_id, hoodie.table.name -> api_trade, hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://<IP_ADDRESS>:10000, hoodie.datasource.write.table.type -> MERGE_ON_READ, hoodie.datasource.write.hive_style_partitioning -> true, hoodie.datasource.query.type -> snapshot, hoodie.bloom.index.update.partition.path -

@wosow : The _rt table syncing happens after _ro table and I see an HiveMetaStore exception when updating commit time in the _ro table saying that the table does not exist. This is weird as in the few log messages above, I can see that the _ro table is registered. Somehow _ro table is not visible to HiveMetaStoreClient. I think it is likely that HiveServer and HiveMetastore are not setup correctly and there could be more than one HiveMetastore involded (one probably local ) instance.

@wosow : The _rt table syncing happens after _ro table and I see an HiveMetaStore exception when updating commit time in the _ro table saying that the table does not exist. This is weird as in the few log messages above, I can see that the _ro table is registered. Somehow _ro table is not visible to HiveMetaStoreClient. I think it is likely that HiveServer and HiveMetastore are not setup correctly and there could be more than one HiveMetastore involded (one probably local ) instance.

@wosow Were you able to resolve your issue ?

@wosow : also, few quick questions as we triage the issue.

Were you running older version of Hudi and encountered this after upgrade? in other words, older Hudi version you were able to run successfully and with 0.7.0 there is a bug.
Is this affecting your production? trying to gauge the severity.
Or you are trying out a POC ? and this is the first time trying out Hudi.

@wosow : also, few quick questions as we triage the issue.

Were you running older version of Hudi and encountered this after upgrade? in other words, older Hudi version you were able to run successfully and with 0.7.0 there is a bug.
Is this affecting your production? trying to gauge the severity.
Or you are trying out a POC ? and this is the first time trying out Hudi.

 There is no impact on the production environment, only the problem occurred in the test 0.6.0, and the test was not performed in the 0.7.0.

 In addition, I have another question. I use sqoop to import the data in mysql to HDFS, and then use Spark to read and write the Hudi table. The table type is MOR. If I want to use asynchronous compaction, what parameters need to be configured, asynchronous Is Compaction automatic? Need manual intervention? Or is it necessary to manually intervene Compaction after opening asynchronous Compaction? If it is necessary to manually execute Compaction manually on a regular basis, what parameters need to be configured for manual Compaction, and what are the commands for manual Compaction? Looking forward to your answer! ! !

@wosow Were you able to resolve your issue ?

no

@wosow looks like this is not a real issue in production. For your questions on async compaction, have you taken a look at this blog -> https://hudi.apache.org/blog/async-compaction-deployment-model/ ? If your questions are still unanswered after reading this blog, please ping here and we will answer them

@wosow Did you get a chance to read the blog ? Please let us know if this issue is still valid

Closing this ticket due to inactivity. @wosow Please feel free to re-open if you need more information.

I'm seeing the same error on 0.8.0. I am using this with AWS Glue (managed) ETL jobs and trying to sync the Hudi metadata to Glue Data Catalog. This happens the first time I run my job and it is trying to create the tables in the Glue Data Catalog. I suspect there is a permissions issue or there is schema evolution being detected that isn't supported with my setup.
I'm getting an additional error" java.lang.UnsupportedOperationException: Table rename is not supported"
I'm just trying a PoC but we are pretty hot on using this as it solves a lot of our problems nicely.
2021-07-30 10:06:08,704 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Error from Python:Traceback (most recent call last):
  File "/tmp/raw-to-staging.py", line 146, in <module>
    main()
  File "/tmp/raw-to-staging.py", line 137, in main
    .save("s3://myBucket/Staging/mySourceDB/mySchema/myTable/")
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 734, in save
    self._jwrite.save(path)
  File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/opt/amazon/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/opt/amazon/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o416.save.
: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing ChargeActive
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:122)
	at org.apache.hudi.HoodieSparkSqlWriter$.org$apache$hudi$HoodieSparkSqlWriter$$syncHive(HoodieSparkSqlWriter.scala:391)
	at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$metaSync$2.apply(HoodieSparkSqlWriter.scala:440)
	at org.apache.hudi.HoodieSparkSqlWriter$$anonfun$metaSync$2.apply(HoodieSparkSqlWriter.scala:436)
	at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
	at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:436)
	at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:497)
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:222)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to get update last commit time synced to<PHONE_NUMBER>0502
	at org.apache.hudi.hive.HoodieHiveClient.updateLastCommitTimeSynced(HoodieHiveClient.java:496)
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:168)
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:112)
	... 40 more
Caused by: java.lang.UnsupportedOperationException: Table rename is not supported
	at com.amazonaws.glue.catalog.metastore.GlueMetastoreClientDelegate.alterTable(GlueMetastoreClientDelegate.java:515)
	at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.alter_table(AWSCatalogMetastoreClient.java:400)
	at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.alter_table(AWSCatalogMetastoreClient.java:385)
	at org.apache.hudi.hive.HoodieHiveClient.updateLastCommitTimeSynced(HoodieHiveClient.java:494)
	... 42 more

@dude0001 : just to confirm, you are also facing the issue only w/ spark streaming is it?

@rmahindra123 : Do you mind taking a look at this. let's sync up sometime today.

@nsivabalan that is one difference in my duplication steps. I am currently not using a Spark Streaming job. I'm reading from our raw zone in S3 that contains parquet files containing change data capture events from transactional databases. I'm trying to upset our cleansed zone also in S3 so that it contains the latest version of each row. If I turn off sync, it works fine otherwise.

@dude0001 : Hey if you don't mind, can you create a new github issue. do not want to pollute this issue. since yours is not spark streaming. We can add a link to this issue calling it out as related.

oh, btw COW does not have two tables. only MOR has two tables.

@dude0001 : Did you open up a new github issue? usually table rename exception happens if your table name in hudi mismatches w/ that in hive. Is there any case sensitivity that could be an issue? If your table has capital letters, can you try all small letters.

I apologize, I did not open a new issue as I've been in meetings and working on production stories. I got back to my POC this morning, and prove your theory correct. Renaming my table all lower case resolved my issue! To your point, I'm not sure my rename exception is the same as the original issue that may or may not be related to streaming. Would you still like me to open an issue for the table rename exception? One change might be to add a warning in Hudi for this scenario. Or adding documentation somewhere to make syncing Hudi metadata to hive in AWS Glue Data Catalog a little less painful. I'm happy to do it, or just move on. Please let me know, and thank you again for the help!

thanks, we will add an faq shortly.


Could not get unknow property 'mergeResourcesProvider' when I run my app on android

I am getting this error when I run the app on Android

Where:
Script 'C:\reactnative\auth\node_modules\react-native\react.gradle' line: 95

What went wrong:
A problem occurred configuring project ':app'.

Could not get unknown property 'mergeResourcesProvider' for object of type com.android.build.gradle.internal.api.ApplicationVariantImpl.

##Environment
System:
OS: Windows 10
CPU: (4) x64 Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz
Memory: 6.96 GB / 15.91 GB
Binaries:
Yarn: 1.13.0 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
npm: 6.4.1 - C:\Program Files\nodejs\npm.CMD
IDEs:
Android Studio: Version  <IP_ADDRESS> AI-182.5<IP_ADDRESS>14842

I am also facing the same issue
React Native Environment Info:
System:
OS: Windows 10
CPU: (4) x64 Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz
Memory: 593.94 MB / 3.89 GB
Binaries:
Yarn: 1.12.3 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
npm: 6.4.1 - C:\Program Files\nodejs\npm.CMD
IDEs:
Android Studio: Version  <IP_ADDRESS> AI-181.55<IP_ADDRESS>56338

I am also facing the same issue
React Native Environment Info:
System:
OS: Windows 10
CPU: (4) x64 Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz
Memory: 593.94 MB / 3.89 GB
Binaries:
Yarn: 1.12.3 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
npm: 6.4.1 - C:\Program Files\nodejs\npm.CMD
IDEs:
Android Studio: Version <IP_ADDRESS> AI-181.55<IP_ADDRESS>56338

I just re-did the whole project, fortunately, I was just beginning to code the app.

I just re-did the whole project, fortunately, I was just beginning to code the app.
Although I would still like to find a solution

React Native Environment Info:
System:
OS: macOS 10.14.1
CPU: (4) x64 Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
Memory: 18.05 MB / 8.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 10.0.0 - /usr/local/bin/node
Yarn: 1.12.3 - /usr/local/bin/yarn
npm: 6.8.0 - /usr/local/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
IDEs:
Android Studio: 1.5 AI-141.2836215
Xcode: /undefined - /usr/bin/xcodebuild
npmPackages:
react: ^16.6.3 => 16.8.4
react-native: ~0.59.1 => 0.59.1
npmGlobalPackages:
react-native-cli: 2.0.1
same problem

I faced with the same problem and this workaround works for me.
https://github.com/wix/react-native-navigation/issues/4757#issuecomment-468133753

Please check out the workaround.

same thing happened to me today....

after editing it this happened to me
Build file 'D:\projects\feb\android\app\build.gradle' line: 1

What went wrong:
A problem occurred evaluating project ':app'.

Failed to apply plugin [id 'com.android.application']
Minimum supported Gradle version is 4.10.1. Current version is 4.4. If using the gradle wrapper, try editing the distributionUrl in D:\projects\feb\android\gradle\wrapper\gradle-wrapper.properties to gradle-4.10.1-all.zip

For me this issue was resolved just after I run my project from Android studio

the same problem when updating to v0.59.1, I could solve it by updating gradle and the compilation version to 28.
in /android/build.gradle
buildscript {
ext {
minSdkVersion = 21
compileSdkVersion = 28
targetSdkVersion = 26
supportLibVersion = "28.0.0"
playServicesVersion = "15.0.1"
}
repositories {
google()
jcenter()
}
dependencies {
classpath 'com.android.tools.build:gradle:3.3.2'
}
}
allprojects {
repositories {
mavenLocal()
google()
jcenter()
maven {
// All of React Native (JS, Obj-C sources, Android binaries) is installed from npm
url "$rootDir/../node_modules/react-native/android"
}
maven {
url 'https://maven.google.com'
}
}
configurations.all {
resolutionStrategy.force "com.android.support:support-v4:${rootProject.ext.supportLibVersion}"
}
}
in /android/gradle/wrapper/gradle-wrapper.properties
distributionUrl=https://services.gradle.org/distributions/gradle-4.10.2-all.zip

classpath 'com.android.tools.build:gradle:3.3.2'

Changing only dependencies solved for me, many thanks.
from
dependencies {
classpath 'com.android.tools.build:gradle:3.0.1'
}
to
dependencies {
classpath 'com.android.tools.build:gradle:3.3.2'
}

ERROR: Could not get unknown property 'mergeResourcesProvider' for object of type com.android.build.gradle.internal.api.ApplicationVariantImpl.


Add individual Sensirion SEN5x models

This is after Learn Guide feedback concerning the "SEN54/SEN55 adapter breakout".

@tyeth  Approved and merged!


Customizable layouts - fully functional

Related to #207
Since this PR we've been able to load GUI configs from the command line with --gui-config and default to ~/.ignition/gazebo/gui.config when no <gui> was specified on SDF.
But because of issue #207, we couldn't really reuse configuration files across worlds that have different names. Which meant that the 2 features above couldn't really be used.
This PR, together with https://github.com/ignitionrobotics/ign-gui/pull/104, allow us to remove custom topics and services from SDF files. This means we can rely on the default gui.config all the time!
More advantages:

This lowers the bar for using Gazebo-classic SDF worlds with Ignition, because it's one less thing for them to set.
3,069 deleted lines!

See the new tutorial for more info.
:warning: Delete your existing default file to see the effects of this PR: rm ~/.ignition/gazebo/gui.config

:point_down: that's what loading ign gazebo shapes.sdf looks like now.

Fix
The fix is based on setting the worldNames on the MainWindow so that any plugin can find them. This means that even pure ign-gui plugins like WorldControl and WorldStats can access world names if they want.
This is not a solution to #207, but it makes it easier to write GUI plugins that will work with whatever world is currently loaded.
New layout
I tweaked a bit the default gui.config file so it includes more plugins. I used a gray background for shapes and transform control because it needs to look ok on both light and dark themes.
Deleted lines
I went over all SDF files and:

Deleted the <topic>s and <service> from WorldControl and WorldStats
If the world was using the default camera pose -6 0 6 0 0.5 0 and wasn't loading any plugin that's not on gui.config, I deleted the entire <gui> block so the default is used.
For a couple of worlds, like rolling_shapes, I moved things so that they work well with the default camera angle.

reflect that another reason for removing this file might be to receive the updated version of the default configuration.

Thanks, @caguero . Updated the tutorial in 2cd4b10


Open Telemetry:  Steps to GA a feature

Traces

[ ]  Update isStable flag here
[ ] Make sure, by default, open telemetry attributes are getting emitted are same as mentioned in open telemetry docs https://opentelemetry.io/docs/specs/semconv/database/cosmosdb/
[ ] Make sure backward compatibility is there, there should be a way to get back appinsights supported attributes. Preferably, as mentioned in this PR.
[ ] Review observability documentation https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/docs/observability.md#net-sdk-observability-feature-design and https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/sdk-observability and make sure it talks about:
a) Versioning
b) Event Source and Activity Source Name
[ ] Make sure Samples are available in this repo.

Metrics

[ ] Make sure, by default, open telemetry metrics are getting emitted are same as mentioned in open telemetry docs https://opentelemetry.io/docs/specs/semconv/attributes-registry/db/
[ ] Review observability documentation https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/docs/observability.md#net-sdk-observability-feature-design and remove client telemetry feature, related information.
[ ] Make sure Samples are available in this repo.

This list looks good to me! You have versioning partially covered with the backwards compatibility item, but it might be good to call out versioning story explicitly as a work item as well.
Not a GA blocker, but we should also double check that the Azure Montior UI still works well with the new OTel traces and attributes. They had some special mapping of our database/ collection for example, and the OTel attribute names have changed.


Sprockets error with 0.14.3

After updating imba-source to 0.14.3 I get a Sprockets error when using Imba with Rails.
Sprockets::FileNotFound - couldn't find file 'imba' with type 'application/javascript':
  sprockets (3.4.0) lib/sprockets/directive_processor.rb:182:in `rescue in block in process_directives'
  sprockets (3.4.0) lib/sprockets/directive_processor.rb:179:in `block in process_directives'
  sprockets (3.4.0) lib/sprockets/directive_processor.rb:178:in `process_directives'
  sprockets (3.4.0) lib/sprockets/directive_processor.rb:83:in `_call'
  sprockets (3.4.0) lib/sprockets/directive_processor.rb:68:in `call'

I have submitted a pull request to imba-source now which will hopefully fix this.

Fixed in 0.14.4 :)


Fix compiling with vc14

See https://bugreports.qt.io/browse/PYSIDE-321. I did not get any response from upstream, so I've added the patch to this recipe.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Needs a build number bump as well.

...and a re-render too.

Sorry, I think must have screwed up the re-render. Any idea why circleci has such an error?

Could you please try rebasing on PR ( https://github.com/conda-forge/pyside-feedstock/pull/7 )?

This is great news!

yes, it seems the patch does not apply cleanly under all platforms because it's not the same line endings format as the others, CRLF, LF and stuff...

@jschueller, should we close this one?

See #10, I reworked your patch to have crlf endings BUT with some lf endifs at patch tags like the other patches (I had to remove gitattributes), that was tricky.
Thanks a lot!


Fix cleanup on import overquota error.

I found this problem while debugging a new connector: When imports failed due to overquota imported tables where not cleaned up because the schema was missing.

/cc @rafatower for awareness
Now, we probably have a good number of  cdb_importer.importer_XXXXX leftovers. I don't remember if we have any periodical process to clean up those, maybe we should consider it.


Maru might not allow for dynamic configuration

Don't shoot me if I'm wrong, but I think this is an issue with Maru. The context is rather simple: I modified the configuration using Application.put_env before starting my dependencies such as Maru, but for some reason Maru does not respect this new configuration, even though when I use Application.get_env it reports the new configuration just as you would expect.
I would love to fix it myself, but unfortunately I have no clue about what is going on, because the soruce suggests Maru does load the configuration after it is started. Is there any chance you could look into it or tell me that it doesn't have anything to do with Maru? That would be really great. For more information, see samvv/elixir-takeoff#1.

Whoops, turned out it was a simple mistake. I forgot to add the :persistent flag. Never mind me. ;)

I'm sorry I haven't get your point but it seems you have found your answer 🙂
feel free to leave me a message if anything I can help


API endpoint for client

Signed-off-by: luoji<EMAIL_ADDRESS>

Codecov Report

Merging #20 (cb8931f) into master (4689b82) will decrease coverage by 0.05%.
The diff coverage is 92.30%.

@@            Coverage Diff             @@
##           master      #20      +/-   ##
==========================================
- Coverage   95.92%   95.87%   -0.06%
==========================================
  Files          19       20       +1
  Lines         883      896      +13
==========================================
+ Hits          847      859      +12
- Misses         19       20       +1
  Partials       17       17

Impacted Files
Coverage Δ

client.go
93.85% <ø> (ø)

api.go
92.30% <92.30%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4689b82...cb8931f. Read the comment docs.


Fix crash on export from action when instance has custom PK

Problem
If an model has a custom pk (e.g. UUIDCategory) then an export via the instance page would cause a crash.
Solution
replaced reference to obj.id with 'obj.pk
Acceptance Criteria

Added test

coverage: 100.0%. remained the same
when pulling 9db64cb3318567e39af08badb7789e0f7a800b17 on matthewhegarty:fix-crash-on-action-export-custom-pk
into fe5e2ffdcf0f0fb608b488c2ae44d8e2b81e484b on django-import-export:main.


Create wing_modal_tutorial

Here is an documentation for the example adopted from the APDL Wing Model tutorial in the Ansys Documentation.
Description
Please provide a brief description of the changes made in this pull request.
Issue linked
Please mention the issue number or describe the problem this pull request addresses.
Checklist

[ ] I have visited and read Develop PyMAPDL.
[ ] I have tested my changes locally
[ ] I have added necessary documentation or updated existing documentation.
[ ] I have followed the PyMAPDL coding style guidelines.
[ ] I have added appropriate unit tests that also ensure the minimal coverage criteria.
[ ] I have reviewed my changes before submitting this pull request.
[ ] I have linked the issue or issues that are solved to the PR if any.
[ ] I have assigned this PR to myself.
[ ] I have marked this PR as draft if it is not ready to be reviewed yet.
[ ] I have made sure that the title of my PR follows Commit naming conventions (e.g. feat: adding new MAPDL command)

Hi @mcMunich
thank you for this. I shall have a look at this, this week.


Bump LiteDB from 5.0.14 to 5.0.15

New LiteDB version got released about 2 weeks ago.
https://github.com/mbdavid/LiteDB/releases/tag/v5.0.15
Also noticed that 4 unit tests are failing apparently, but isn't related to this version bump.

New LiteDB version got released about 2 weeks ago. https://github.com/mbdavid/LiteDB/releases/tag/v5.0.15
Also noticed that 4 unit tests are failing apparently, but isn't related to this version bump.

Thanks for your pull request. I cannot reproduce the problems you had with the unit tests. I noticed that the test project was using Dotnet Core 3.1 support for which ended in December. I have updated it to DotNet 7.0 in any case, but either way all of the unit tests passed for me (Visual Studio 2022 17.4.3, WIndows 10)

Odd, I ran the tests on my MacBook (Rider 2022.3.1, OS X 13.1 Ventura) and can somehow consistently reproduce the failing tests, will check whether I can reproduce it on my Windows machine one of the upcoming days though.

Just letting you know that I checked it on my Windows machine (Rider 2022.3.1, Windows 10) and all tests are indeed green there as well, guess it might be an OS-specific issue perhaps.


Vim9: crash when using list with a circular reference

Describe the bug
In Vim9 script, creating a list with a circular reference might make Vim crash (actually, I'm not sure that's what really causes the crash...).
To Reproduce
Run this shell command:
vim -Nu NONE -S <(cat <<'EOF'
    vim9script
    var l = []
    var d = {a: 0}
    l = [d]
    l[0].e = {b: l}
    w:name = {f: {c: d}}
    getwininfo()->filter((_, v) => true)
EOF
)

Vim crashes.
Expected behavior
Vim doesn't crash.
Environment

Vim version: 8.2 Included patches: 1-2618
OS: Ubuntu 16.04.7 LTS
Terminal: xterm(366)

I don't have logs for now.  It took a looong time to get this MWE.  Loading the core in gdb gives this:
Reading symbols from ./src/vim...
[New LWP 14200]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./src/vim -Nu NONE -S /proc/self/fd/11'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000066ef10 in typval2type_int (
    tv=<error reading variable: Cannot access memory at address 0x7ffe7f789fc8>,
    type_gap=<error reading variable: Cannot access memory at address 0x7ffe7f789fc0>) at vim9type.c:258
258     {
#1  0x000000000066f347 in typval2type (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#2  0x000000000066efe0 in typval2type_int (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:281
#3  0x000000000066f347 in typval2type (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#4  0x000000000066f0c2 in typval2type_int (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:300
#5  0x000000000066f347 in typval2type (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#6  0x000000000066f0df in typval2type_int (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:302
#7  0x000000000066f347 in typval2type (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#8  0x000000000066efe0 in typval2type_int (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:281
#9  0x000000000066f347 in typval2type (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#10 0x000000000066f0c2 in typval2type_int (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:300
#11 0x000000000066f347 in typval2type (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#12 0x000000000066f0df in typval2type_int (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:302
#13 0x000000000066f347 in typval2type (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#14 0x000000000066efe0 in typval2type_int (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:281
#15 0x000000000066f347 in typval2type (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#16 0x000000000066f0c2 in typval2type_int (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:300
#17 0x000000000066f347 in typval2type (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#18 0x000000000066f0df in typval2type_int (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:302
#19 0x000000000066f347 in typval2type (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#20 0x000000000066efe0 in typval2type_int (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:281
#21 0x000000000066f347 in typval2type (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#22 0x000000000066f0c2 in typval2type_int (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:300
#23 0x000000000066f347 in typval2type (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#24 0x000000000066f0df in typval2type_int (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:302
#25 0x000000000066f347 in typval2type (tv=0x1f275c0, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#26 0x000000000066efe0 in typval2type_int (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:281
#27 0x000000000066f347 in typval2type (tv=0x1f27640, type_gap=0x7ffe7ff828f0) at vim9type.c:377
#28 0x000000000066f0c2 in typval2type_int (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:300
#29 0x000000000066f347 in typval2type (tv=0x1f27660, type_gap=0x7ffe7ff828f0) at vim9type.c:377
...


Run in pararel on remote repositories

We have all remote repositories in modules.
We can add:

workspaces configuration (it will affect how modules are fetched which we probably do not want)
Use parrarel runner like https://www.npmjs.com/package/run-in-all

Second option seems more appropriate to run set of UI's together, however to do that we need to unify how we run projects in the dev. kas-ui, kafka-ui have different scripts to start in dev mode thus it is not possible to issue same script on those etc.
WDYT?
Alternatively we can have script that allow us to define dev profiles in package.json (list of projects that will be started in dev mode etc.

FYI @pmuir @riccardo-forina

Yeah, I don't think we are ready to align all the dependencies for the projects.
The issue is that start:dev doesn't work on kas-ui for running federated because of ports. What about just using https://github.com/mysticatea/npm-run-all as then you can specify each module separately.
Or add a new target to each project like run:federate that is just a link to start:dev?

All ideas sound awesome to me


Controlling Large Memory Usage on GPU

Hi there!
I am trying to train a basic S4D model on an A10G GPU (g5.24xlarge EC2 instance). I copied the model definition in state-spaces/examples.py and configured it with the following parameters:

input_dim: 7
output_dim: 2
model_dim: 64
dropout: 0.2
num_layers: 6
prenorm: false

On device, the model weights take up ~700 MiB of memory. I then pass in batches of dimension (B, L, D) = (256, 1801, 7) with dtype=np.float32, which brings the total memory on device to ~1 GiB.
Once I execute the forward pass with my model by calling model(batch) I see a large increase in the GPU memory. The total used memory goes from ~ 1 GiB all the way up to ~10 GiB. If I try to scale to a batch size of 512 I will typically hit CUDA out-of-memory errors.
Right now this is a fairly large blocker for me, because my corpus is on the order of TB in size, and iterating over all of it in an epoch in a timely manner requires setting larger batch sizes (e.g. 1024, 2048, or even 4096). I was wondering if you might have any idea as to why the memory increases so much? or know if there are ways I can try to decrease the memory utilization?
I'm currently using the pykeops cauchy kernel (I've had issues getting the python setup.py install to work in my docker image) and the default measure for S4D. I'm not sure if using the custom CUDA kernel or switching the measure might help with decreasing the memory footprint?
Any advice would be much appreciated!

The memory usage you're seeing is expected. When running an actual training step (forward + backward pass) the GPU memory is usually dominated by storing the activations. These activations have shape (batch size, sequence length, model dim) (times number of layers), which provides a rough heuristic of how much memory it needs. For example, the S4 LRA config is 6 layers by (16, 16384, 256) (product about 6.7e7 activations per layer) and fills up most of an A100's 40GB RAM. In order to fit a batch size of 4096, you're asking for 6 layers by (4096, 1800, 64) (product about 4.7e8), which is nearly 10x as much memory. This explains why you're running out of RAM at around 512 batch size.
As a simple test, you can remove all of the S4 layers of your model and have only feed-forward components and you'll still run out of memory at those batch sizes.
Ultimately it sounds like training on your corpus simply needs more computational resources.

Thanks for replying so quickly! That makes a lot of sense, I mostly just wanted to check if I was doing something wrong or if there were any easy wins around this, but I'm glad that it is the expected behavior.


Gather information about configuration from multiple URLs

Patch add functionality for gathering informatio about
configuration and server list from multiple URLs.
URLs prefix is defined via global variables.
Author Dubravko Penezic

Invalid PR.

Thanks for fast but not clear info , do you like or not pull request ? , if yes, simple information what is wrong will be very helpfull


Face Tracking using Meanshift and Camshift algorithm

Meanshift and Camshift algorithm can be very well implemented to track detected faces in a live video, we can also do a comparative study which algorithm is better.
I am a GSSoC participant and would like to work on this issue, kindly assign me the same @geekquad @kritikaparmar-programmer .

Hey @iamakkkhil, go ahead!
Focus more on the implementation of the algorithm rather than its application.

Is it still open..?
Can I take this up @geekquad ?


About data pre-processing acceleration for small model

Dear author,
Thanks for your time and sharing!
I'm trying to apply BYOL on my task, where samples are limited. Accordingly, a small model for feature extraction is designed.
Based on the experiment observation, data augmentation operations on CPU become the performance bottleneck, i.e., GPU is always waiting for data.
For accelerating the pipeline, I am making efforts on NVIDIA/DALI. However, the samples are of type numpy.ndarray, I have not found good ways to deal with it yet.
Another library, kornia, has caught my attention, too. Maybe it supports the data augmentation on GPU?
Could you please offer some suggestions? Thanks in advance!

Preprocessing libs have been moving forward since I created this repo, and to be honest, I haven't caught up with them.
To use DALI, you should first convert the raw data to some formats DALI can process. Another concern is that the augmentations used in BYOL are not fully covered in DALI. (I'm not sure if they have been added now).
Actually, I think torchvision now support passing tensor (instead of PIL) images as input and do the pre-processing using GPUs. Maybe it deserves a try.

Thank you for your enthusiastic response!
I have tried DALI during this time, and now the data pre-processing operation of DALI has increased a lot and can meet my needs.
The preprocessing of torchvision can be done as nn.Sequential, placed in the forward part of the network and trained using the GPU.
kornia is able to compute on GPU but it is slower than torchvision due to its differentiable nature. I think the differentiable nature is the double-edged sword of the kornia library.
Thank you for providing the elegant implementation and wish you all the best!


trufflehog: 3.54.0 -> 3.54.1

Diff: https://github.com/trufflesecurity/trufflehog/compare/refs/tags/v3.54.0...v3.54.1
Changelog: https://github.com/trufflesecurity/trufflehog/releases/tag/v3.54.1
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 252428 run on x86_64-linux 1

1 package built:

trufflehog


Missing code block language

Codecov Report

Merging #3962 (990c280) into master (fbbc8fc) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3962   +/-   ##
=======================================
  Coverage   94.87%   94.87%
=======================================
  Files         454      454
  Lines       49462    49462
=======================================
  Hits        46928    46928
  Misses       2534     2534

Flag
Coverage Δ

servertests
55.73% <ø> (ø)

unittests
94.74% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fbbc8fc...990c280. Read the comment docs.


Update to support libs, remove need for Jetifier

this repo has not been updated for a bout an year. I was trying to disable Jetifier on my project and this is the only project left.
so how are you guys doing? actually I could update this project myself and send pr, will you guys accept it?

Fixed in feature/sdk34 branch.


How to read the value of propTypes of my Button component

I need a function that would return the propTypes and the defaultProps of any React Component in order to display a table like this:

I have created the following function to read the propTypes and defaultProps

function getInfo(component) {
  console.log('propTypes', component.propTypes)
  console.log('defaultProps', component.defaultProps)
}

Reproduction
https://www.webpackbin.com/bins/-KfXIWlTGcgSo-GeNAFR
Expected && Result
To log the propTypes and defaultProps
How about in webpack 2?
I am trying to reproduce the same thing with components that look like this:

export class Button extends React.Component {
  static propTypes = {
    data: PropTypes.object.isRequired,
    onSubmitFormLogin: PropTypes.func.isRequired,
    changeForm: PropTypes.func.isRequired,
    requestError: PropTypes.func.isRequired,
  };

  static defaultProps = {
    data: {}
  };
...
}

In order for this to work, you need to use the static keyword
According to babel, It should be the exact same thing, but for some reason, I cannot read the propTypes but only the defaultProps so where do they go ?
Is it Webpack/Babel that does remove them ?
Did anyone succeed to read is propTypes using this syntax ?

Seem to work fine


Windows: cannot open source file "app_links/app_links_plugin_c_api.h"

I am not sure if this is a bug or maybe a misunderstanding caused by the documentation. If the latter, the documentation should be extended in my opinion to help flutter beginners.
For the Windows Setup it says:

Add this inclusion in <PROJECT_DIR>\windows\runner\main.cpp
 #include "app_links/app_links_plugin_c_api.h"

but VS Code cannot find  "app_links/app_links_plugin_c_api.h", event after the package has been installed via the pubspec.
Instead, it is required to manually copy the source files found in https://github.com/llfbandit/app_links/tree/master/app_links into our flutter project. E.g. if I want to support Windows, I need to copy the "windows" folder into the root of my project.

You don't have to copy anything. But you need to enable Windows platform on your project first of course.


Replace exp/slices with slices from standard library

What this PR does:
https://github.com/grafana/dskit/blob/e27df29220ea69cf7335f653da18b8ea51ab27e4/ring/token_range.go#L7
We upgraded the minimum Go version to 1.21 in PR https://github.com/grafana/dskit/pull/540.
The experimental functions in golang.org/x/exp/slices are now included in the standard library since Go 1.21.
Reference: https://go.dev/doc/go1.21#slices
Which issue(s) this PR fixes:
Checklist

[N/A] Tests updated
[x] CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Animated tres files are not paused when the tree is paused

Godot version:
3.2.3
OS/device including version:
Linux mint 19.1
Issue description:
Animated Textures (tres) do not pause when game is paused via get_tree().paused=true
Observe the gif below, when the first dialog appears (this is set to process so does not pause) the game pauses ok however teh green conveyor below which is an animated tres file in the tilemap does not. Now, wait for the next screen and when the second dialog appears you will notice the blue graphics (animated texture in a tilemap) are still spinning despite the game being paused

It is irrelevant whether the tres is part of a tilemap or a texture to a sprite, the same thing happens, i.e. animated tres are not pausing. Note, particles pause just fine so it is not linked to #41582 that I raised a while ago regarding particles/tres not stopping during debug. This is a more serious bug as it affects gameplay.
Steps to reproduce:

create an animated tres file
add it to a sprite
pause the game

Duplicate of https://github.com/godotengine/godot/issues/44625.

Duplicate of https://github.com/godotengine/godot/issues/44625.

Have found a quick fix for this.
var alltiles =tile_set.get_tiles_ids()
grab tileset
for i in alltiles:
tile_set.tile_get_texture(i).pause = val
#for each id make pause = to val, where val is the paused val
Do not know it's performance, works well on a tileset with around 200 tiles, but haven't tried it on a low-spec machine.

@Atermnus I recommend posting the workaround on https://github.com/godotengine/godot/issues/44625 too, as people may not stumble upon this particular issue from a web search (it's a duplicate of another issue).


Consider adding this to AUR?

Hi, this script is very good would you consider adding it to the arch user repo?
This issue was a draft and I did it myself. So instead of opening a PR I'm just opening an issue
Link to the package

Is there anything I need to do? I'm not sure if this means I need to update the code/documentation of the script or something lol

Well, I opened this issue to just let you and the people who might use this script in the future know.
But if you'd like to maintain it yourself on AUR, I can hand it to you.
Or if you'd like, you can add arch-based distro installation under installation header like:
yay -S wayback-spn-script-git


chore: Bump eviction api to policy/v1

Fixes #N/A
Description
How was this change tested?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

14 of 14 (100.0%)  changed or added relevant lines in 1 file are covered.
22 unchanged lines in 5 files lost coverage.
Overall coverage decreased (-0.2%) to 78.862%

Files with Coverage Reduction
New Missed Lines
%

pkg/controllers/provisioning/scheduling/topologygroup.go
2
96.75%

pkg/scheduling/storageclass.go
2
81.82%

pkg/test/environment.go
5
89.71%

pkg/controllers/deprovisioning/pdblimits.go
6
76.47%

pkg/controllers/provisioning/scheduling/preferences.go
7
86.67%

Totals

Change from base Build<PHONE_NUMBER>:
-0.2%

Covered Lines:
8286

Relevant Lines:
10507

💛  - Coveralls


Setting a default value for ZoneInfo does not work

When setting a default value for ZoneInfo a different timezone is used.
Python 3.12.1 (main, Jan  8 2024, 05:53:39) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydantic_settings
>>> from zoneinfo import ZoneInfo
>>> from pydantic_settings import BaseSettings
>>> class Test(BaseSettings):
...     tz: ZoneInfo = ZoneInfo("Europe/London")
...
>>> Test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/acuthbert2/dev/ves/scripts/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 167, in __init__
    super().__init__(
  File "/home/acuthbert2/dev/ves/scripts/.venv/lib/python3.12/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Test
tz
  invalid timezone: :Asia/Shanghai [type=zoneinfo_str, input_value=':Asia/Shanghai', input_type=str]
>>> pydantic_settings.__version__
'2.6.1'

pydantic-settings version: 2.6.1
python version: 3.12.1
platform: DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

It seems that you have a tz env variable that has the value of :Asia/Shanghai. the :Asia/Shanghai is an invalid timezone and this is the reason of the error.

@hramezani I don't (or not that I know).
>>> import os
>>> os.getenv("tz")
>>>

However naming the variable something other than tz does work correctly, so it does look to be something to do with my specific environment rather than a bug. Thanks

Likely due to my environment, closing.


Geographic fix

The fix discussed in PR #99.
I added a version bump and change log. The latter would require some more work, either we could discuss that here or I'll revert those changes.

Thanks @mjziebarth ! I'll try to review in more details soon.
My main question about this is whether we can write a test that would check approximate equality of krigging in geographic coordinates, with the results that use euclidean coordinates? Possibly by taking some latitude, longitude where projecting beforehand wouldn't impact results too much?
Currently test_ok_geometric doesn't really check anything..

Yes, such an example should easily be possible somewhere close to the equator. Probably also close to the pole in some appropriate projection, which I think would be most important. I'll write something as soon as I have some spare time!

Thanks for the review!
Maybe the failing tests are also related to the matrix inversion improvement possibilities Mike mentioned in the original PR, though this is just guessing. For now, choosing 2e-5 circumvents the failing.
The git diff master...HEAD | flake8 --diff did not return anything so I guess the changes are compatible?
There's also the question about the version bump, I included that in a rough proposal, but it would need finishing.


there is no point in restarting the torque server each time

This was added due to issues with Torque not correctly scheduling parallel jobs after new instances are added. Unless that behaviour of Torque has changed, this is a required restart due to the elastic nature of CfnCluster.

scheduling --- I guess the problem is in pbs_sched, not pbs_server. Restarting the scheduler, not server, is less problematic. I do this, in fact, because sometimes pbs_sched dies on the restart of pbs_server:

restart pbs_sched
hold all jobs temporarily
release all jobs

pbs_sched is not actively maintained. Rather, the recommended way is to install Maui scheduler by Adaptive computing (licensed) which is more scalable and stable. I understand that due to the open source nature cfncluster should stick to pbs_sched. You should just restart pbs_sched.

Are you experiencing an issue with Torque running in CfnCluster?
On Tuesday, July 12, 2016, Masataro Asai<EMAIL_ADDRESS>wrote:

scheduling --- I guess the problem is in pbs_sched, not pbs_server.
Restarting the scheduler, not server, is less problematic. I do this, in
fact, because sometimes pbs_sched dies on the restart of pbs_server:

restart pbs_sched
hold all jobs temporarily
release all jobs

pbs_sched is not actively maintained. Rather, the recommended way is to
install Maui scheduler by Adaptive computing (licensed) which is more
scalable and stable. I understand that due to the open source nature
cfncluster should stick to pbs_sched. You should just restart pbs_sched.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/cfncluster-node/pull/3#issuecomment-232219992,
or mute the thread
https://github.com/notifications/unsubscribe/ACUVinXiPaQKIfXa4QLaUO2CFhRFmSRYks5qVC67gaJpZM4JJGV5
.

I'm writing scripts to handle the unexpected restart of pbs_server.

I'm not sure I understand, the restart of pbs_server is expected as part of
CfnCluster. Is the restart impacting the cluster?
Do you have a hard requirement for Torque. SGE performs the best in
CfnCluster.
On Tuesday, July 12, 2016, Masataro Asai<EMAIL_ADDRESS>wrote:

I'm writing scripts to handle the unexpected restart of pbs_server.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/cfncluster-node/pull/3#issuecomment-232221222,
or mute the thread
https://github.com/notifications/unsubscribe/ACUVioe07bBYNYqQSiYy9IsL5z51Ynitks5qVDC8gaJpZM4JJGV5
.

I've carried out testing with torque-6.0.1 and without a restart, there is still issues with jobs being scheduled to new nodes.

Yes, just stopping restarting the server is not a good idea. Instead, did you try restarting the scheduler?

Yes, that has been tested as well. Happy to look at pull requests that
removes the need for the restart.
The basic test, is to lauch a cluster with 1 or more instances in the
ComputeFleet. Then reduce the size to zero instances, followed by scaling
back up to 1 or more instances. After that simple batch and multi-node
parallel jobs should be submitted, and then scheduled to the new instances.
On Sunday, July 24, 2016, Masataro Asai<EMAIL_ADDRESS>wrote:

Yes, just stopping restarting the server is not a good idea. Instead, did
you try restarting the scheduler?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/awslabs/cfncluster-node/pull/3#issuecomment-234821883,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUVinCMG8OoHlxgRQlK64pcQ17Qa2q1ks5qZByEgaJpZM4JJGV5
.


Sphinx 2.1.2

PR title
Status
READY/IN DEVELOPMENT/HOLD
Description
A few sentences describing the overall goals of the pull request's commits.
Related PRs
List related PRs against other branches:

branch
PR

other_pr_production
link

other_pr_master
link

Todos

[ ] Tests
[ ] Documentation

Steps to Test or Reproduce
Outline the steps to test or reproduce the PR here.
git pull --prune
git checkout <feature_branch>
pytest --showlocals

Impacted Areas in Application
List general components of the application that this PR will affect:

:white_check_mark: Build ebook_homebrew 1.0.283 completed (commit https://github.com/tubone24/ebook_homebrew/commit/ffa1ae124d by @)


Provide more useful disabled message

Should relieve any potential confusion around whether FlakeHub Cache is disabled due to chosen configuration (use-flakehub set to false) vs. malfunction (missing token).

Ugh, I cannot get the CI to run to completion


Broken style setting

My other snippet doesn't show up in style settings. There is just Minimal. I think there is a problem with the formatting but I can't check the error for the moment.
Anyone can help me?

Okay, I found the culprit ! It's because of the last - for style settings : https://github.com/kepano/obsidian-minimal/blob/c4999a75c5ab6b97466b2af18a0ecfe455f35487/obsidian.css#L4829

This is fixed as of 3.3.2


Add the ability to dynamically ignore some sql errors on sql migration's scripts

Fixes #1175
I don't think it is a niche requirement.
In many DB provider such as PostgreSQL, there is no way to check for some object existence before creating them.
For example you cannot know if a column already exists before creating in one simple SQL line.
Moreover, SqlMigrationResolver being still impossible to extend, I think this feature is required.
Regards

In many DB provider such as PostgreSQL, there is no way to check for some object existence before creating them.

This assertion is dead wrong. I can think of three ways of handling this off hand:

9.6 adds the IF NOT EXISTS clause to ADD COLUMN.

Prior to 9.6, you can easily consult the system views:
DO $$
BEGIN
    IF EXISTS(SELECT * FROM information_schema.columns WHERE table_schema = 'public' AND table_name = 'my_table' AND column_name = 'blah') THEN
        RAISE NOTICE 'Yes';
    ELSE
        RAISE NOTICE 'No';
    END IF;
END
$$

You could also just catch the exception:
DO $$
BEGIN
    ALTER TABLE crop_type ADD COLUMN crop_type_id INTEGER;
EXCEPTION
    WHEN duplicate_column THEN RAISE NOTICE 'Column already exists';
END
$$

I really don't like the idea of this feature. Most DBs (all supported DBs?) make it possible in some way or another to catch specific errors in the procedural language (PL/pgSQL, PL/SQL, T-SQL), and those mechanisms, examining catalog tables to make checks, or things like IF NOT EXISTS are better suited for handling these errors than Flyway.

I fully agree with @bladeoflight16 .
Won't fix.


Refactor  and  components

Closes #221
Dear Reviewer, not exactly a small PR, happy to talk through changes!  ~ M
Changes

Remove react-table
Remove advanced Table capabilities (filter, sort)
Encapsulated Pagination logic into a reusable hook: usePagination
Sync Stories with Design System docs
[Test] Table: Shows pagination when isPaginated
[Test] usePagination: added unit tests

How to test this PR

yarn vitest Table
yarn vitest Pagination

@billhimmelsbach @shindigira A couple fixes for items I had missed:

Sync Pagination header info with DS - preview link

Add documentation for DSR's usePagination hook - preview link

Reduce the number of rows shown per page in the Table - Pagination story (from 20 to 5) to make it easier to interact with - preview link


I got the following error message when I used the command
docker exec mattermost-db su - postgres sh -c "/usr/bin/envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-push /var/lib/postgresql/data"
Traceback (most recent call last):   File "/usr/bin/wal-e", line 6, in <module>     from pkg_resources import load_entry_point   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3019, in <module>     @_call_aside   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3003, in _call_aside     f(*args, **kwargs)   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3032, in _initialize_master_working_set     working_set = WorkingSet._build_master()   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 657, in _build_master     return cls._build_from_requirements(__requires__)   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 670, in _build_from_requirements     dists = ws.resolve(reqs, Environment())   File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 849, in resolve     raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'cryptography>=2.1.4' distribution was not found and is required by azure-keyvault LOG:  archive command failed with exit code 1 DETAIL:  The failed archive command was: /usr/bin/wal-e wal-push pg_xlog/000000010000077D00000026 LOG:  incomplete startup packet ERROR:  relation "idx_teams_description" does not exist
Does anybody can help me ?

I found out maybe the main reason is as below
envdir '/etc/wal-e.d/env' does not exist

I found out the problems. When I pull the latest postgres:9.4-alpine, the error message Traceback (most recent call last): File "/usr/bin/wal-e", line 6, in <module> from pkg_resources import load_entry_point File ... was gone.
I think the remain problem is that /etc/wal-e.d/env does not exit.

I really don't know how this part (S3 backup) works. What is this /etc/wal-e.d/env file ?

@pichouk /etc/wal-e.d/env is a folder created by envdir.
we can use this envdir plugin to store the S3 environment variables.
you can see the code belowed, and it was written in wal-e official github page: https://github.com/wal-e/wal-e
`
Assumption: the group is trusted to read secret information
S3 Setup
$ umask u=rwx,g=rx,o=
$ mkdir -p /etc/wal-e.d/env
$ echo "secret-key-content" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
$ echo "access-key" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
$ echo 's3://some-bucket/directory/or/whatever' >
/etc/wal-e.d/env/WALE_S3_PREFIX
$ chown -R root:postgres /etc/wal-e.d
`
we can use backup command only with this settings, but it is disappeared now.
On the other hand, I think we need to change the backup command to
docker exec mattermost-db su - postgres sh -c "/usr/bin/envdir /etc/wal-e.d/env /usr/bin/wal-e backup-push /var/lib/postgresql/data"
because the path of wal-e has been changed from /usr/local/bin/wal-e to /usr/bin/wal-e
Does anybody can confirm this again?
Thanks.

Fixed at #324 PR.


Different name of authorisation cookie make login to AEM not possible

I found a need to customise cookie name during login to AEM. For now login is possible only if authorisation required default bobcat token.
And additionally, in this case I received Null value instead of message that name of cookie is different then expected.

Ready for release


Can't use cmd+a inside a sortable component

I have a textarea inside a component wrapped by @sortable and I can't use the cmd+a (which is supposed to "Select All" text inside the textarea). Removing the @sortable decorator fixes the cmd+a. I'm guessing sortable is preventing some keyboard events.

The only two event handlers sortable interfere with are onMouseDown and onTouchStart, maybe a working demo might help debugging.

Having this issue as well. @vhpoet, did you find anything?

Didn't have time to check it yet @jasonslyvia @olesse

It's related with selectstart event capture, on SortableItemMixin.js
 on(node, 'selectstart', (e) => {     if (e.preventDefault) {       e.preventDefault();     } else {       e.returnValue = false;     }   });

@hvergara yes, this is intended for legacy IEs, i guess we can remove it safely now.

I made a pull request to remove this prevention, but I got another dilemma, hope you guys can help. #63

can you guys confirm whether it's been settled or not in 1.6.1?

@jasonslyvia will check today and let you know

@jasonslyvia works! Thank you.


Feature/issue36

@sumeko is it a duplicate of code in feature/issue36?

duplicates #36
I'll add more test coverage to feature/issue36 branch and then merge

I am sorry, that was accidentally click :sleepy:

no probs. Check my other messages.
I've made phalcon env and you project works! :smile:
Check my PR out. It adds composer support.
Next 2 big steps:

tests, license, other small issues
migrate the trait


Using 'name' instead of 'property'

When checking the card on the Facebook website using the link https://developers.facebook.com/tools/debug/, I get an error:
Tag specified as 'name' instead of 'property'
The following meta tags are specified using 'name' instead of 'property' and will be ignored unless specified using the 'property' key: fb:app_id

Thanks!
I mixed up the meta tag key of Twitter and Facebook.
Only the key of Twitter's Open Graph is name.
I will fix it later.

Just released v2.1.2 to fix this issue.


✨Freemoji Stickers✨

Is your feature request related to a problem? Please describe.
Freemojis let us use emojis from other servers with links, and it works really well than its predecessor. However, it doesn't support Stickers.
Describe the solution you'd like
Support Stickers.
Describe alternatives you've considered
I tried using Free Stickers but whenever I go to Stickers, it crashes the client.
Additional context
Free Stickers

I tried using Free Stickers but whenever I go to Stickers, it crashes the client.
Have you reported this to the Free Stickers devs by opening an issue there?

The problem with stickers is animated stickers is that we can't just send a link to the sticker like the Freemoji plugin dose for Emojis because animated stickers are in APNG format and Discord does not animate APNG images. I assume the Free Stickers plugin actually has to convert the Stickers to a GIF before sending.
The plugin could always just send the link to the image and oh well if the animated stickers don't animate, but if Free Stickers exist and your issue can be fixed by informing the developers of the crash, then I don't see the point of this.

AFAIK there was a stickers plugin but it got taken down shortly after it was added to the BD website due to Discord saying they'll ban users who send stickers as links (at least that's the explanation I got)

AFAIK there was a stickers plugin but it got taken down shortly after it was added to the BD website due to Discord saying they'll ban users who send stickers as links (at least that's the explanation I got)

Anyone can send a stickers link, all you need is the developers tools that are built in all web browsers and then find the link to the sticker and then send it. Is that truly against TOS? They would ban people sending some links with no prior warning? Do you have any proof about this? Because, If this is actually true and not fearmongering, I will alert this sticker plugin project https://github.com/An00nymushun/DiscordFreeStickers and tell them they should add a disclaimer to their README.

Sticker support was not something I wanted to pursue. The plugin is also discontinued now.