Compatibility with Vue3

Hey, I tried to use it with the Vue3, but see that its not compatible.
With Vue3 its giving following error - _"Uncaught (in promise) TypeError: _vm.self is undefined".
It would be great if you can add the Vue3 compatibility.

Just copy/paste src/VGauge.vue in your project, this will work, and also import "gaugeJS": "^1.3.8",


Restore-PowerBIWorkspace throws  Not Found (404)

Hi Team,
I tried below to restore the deleted workspace. whereas when I tried get-workspaces I do see those workspaces still in delete state. Below is what I used.
 Get-PowerBIWorkspace -scope Organization -Deleted
OutPut
Id                    : 88888888-8888-8888-8888-888888888888
Name                  : testingPowerBIRestore
Type                  : Group
State                 : Deleted
IsReadOnly            : False
IsOrphaned            : False
IsOnDedicatedCapacity : False
CapacityId            :

 Restore-PowerBIWorkspace -Id "88888888-8888-8888-8888-888888888888" -RestoredName "testingPowerBIRestore" -AdminEmailAddress<EMAIL_ADDRESS>OutPut
WARNING: Only preview workspaces are supported when -Scope Organization is specified
Restore-PowerBIWorkspace : Operation returned an invalid status code 'NotFound'
At line:1 char:1
+ Restore-PowerBIWorkspace -Id "88888888-8888-8888-8888-888888888888" - ...
+
    + CategoryInfo          : WriteError: (Microsoft.Power...owerBIWorkspace:RestorePowerBIWorkspace) [Restore-PowerBI
   Workspace], HttpOperationException
    + FullyQualifiedErrorId : Operation returned an invalid status code 'NotFound',Microsoft.PowerBI.Commands.Workspac
   es.RestorePowerBIWorkspace

Resolve-PowerBIError -Last
OutPut
RequestId        : ac0a32c8-46a9-4419-8694-922f5ff0f359
Response         : Not Found (404): {"error":{"code":"PowerBIEntityNotFound","pbi.error":{"code":"PowerBIEntityNotFound
                   ","parameters":{},"details":[],"exceptionCulprit":1}}}
PowerBIErrorInfo :
ResponseDate     : Wed, 03 Jun 2020 12:54:09 GMT
RequestMethod    : POST
RequestUri       : https://api.powerbi.com/v1.0/myorg/admin/groups/88888888-8888-8888-8888-888888888888/restore
InvocationInfo   : {Restore-PowerBIWorkspace}
Line             : Restore-PowerBIWorkspace -Id "88888888-8888-8888-8888-888888888888" -RestoredName "testingPowerBIRestore"
                   -AdminEmailAddress<EMAIL_ADDRESS>Position         : At line:1 char:1
                   + Restore-PowerBIWorkspace -Id "88888888-8888-8888-8888-888888888888" - ...
                   +
StackTrace       :    at
                   Microsoft.PowerBI.Api.V2.Groups.<RestoreDeletedGroupAsAdminWithHttpMessagesAsync>d__21.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.PowerBI.Api.V2.GroupsExtensions.<RestoreDeletedGroupAsAdminAsync>d__33.MoveNext()
                   --- End of stack trace from previous location where exception was thrown ---
                      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                      at Microsoft.PowerBI.Api.V2.GroupsExtensions.RestoreDeletedGroupAsAdmin(IGroups operations,
                   String groupId, GroupRestoreRequest groupRestoreRequest)
                      at Microsoft.PowerBI.Common.Api.Workspaces.WorkspacesClient.RestoreDeletedWorkspaceAsAdmin(Guid
                   workspaceId, WorkspaceRestoreRequest restoreRequest)
                      at Microsoft.PowerBI.Commands.Workspaces.RestorePowerBIWorkspace.ExecuteCmdlet()
                      at Microsoft.PowerBI.Commands.Common.PowerBICmdlet.ProcessRecord()
HistoryId        : 5

MicrosoftPowerBIMgmt version 1.0.840
Is here someone faced with the same issue? Please help me

Can you please provide the output from a recent repro? We need a recent RequestId to investigate.


Stage 0.8.0

Codecov Report

Merging #193 (fde7116) into main (9f5a15c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #193   +/-   ##
=======================================
  Coverage   84.61%   84.61%
=======================================
  Files          38       38
  Lines        2898     2898
=======================================
  Hits         2452     2452
  Misses        354      354
  Partials       92       92

Help us with your feedback. Take ten seconds to tell us how you rate us.


[Map] DM-Door302

Hash: 7ad8065cca463d0d87d8127e2661d1f279261bb9
Current name: DM-Door302
For some reason, the author's name 	-=Chef Grolo =-  is truncated, both in the Map Information (though the full name appears when hovering the cursor over the truncated name) and in the Author's tab in Browse Authors.
This has happened quite a lot and makes the correct identification of authors more complicated.

Looks as though it's correct now: https://unrealarchive.org/maps/unreal-tournament/deathmatch/D/dm-door302_7ad8065c.html
Probably an old author display bug that's since been fixed.


PIR Konke cant join zigbee netowrk after update Z-Stack_Home_1.2 20201127/20201128

I´ve updated my cc2531.
Everything is working. The exception is a PIR Sensor Konke in ZHA.
I´ve tried channels 15, 20, 25 and nothig.
Anyone has some tips?

Well, I suspected flashing was wrong. So i did it again, and voila! Everything is working fine, and discovering device was faster the before.


Add a default value to generated parameter

Improve execute methods by providing a default value for generated methods (None)
Problem
An user used to execute run just plain queries in my database (like schema creation), just calling execute("Sql string). Now this user has to call execute("Sql string", None)
Checklist

[ ] Unit test all changes
[ ] Update README.md if applicable
[ ] Add [WIP] to the pull request title if it's work in progress
[ ] Squash commits that aren't meaningful changes
[ ] Run sbt scalariformFormat test:scalariformFormat to make sure that the source files are formatted

@getquill/maintainers

:+1:

:+1:

:+1: Travis seems to be stuck :-/

@onilton I think there was some issues with Travis, can you close and reopen your PR again?
https://twitter.com/traviscistatus/status/717320277391114240?lang=en


NamedRoutes are not resolving correctly

Describe the bug
namedRoutes are not resolving correctly. In this example if you go to any of these routes, you will be redirected to the last defined path (here FavoriteCocktailsView());
namedRoutes: {
	'/cocktails': GetRoute(page: CocktailsView()),
	'/cocktails/:id': GetRoute(page: CocktailDetailsView()),
	'/favorite/cocktails': GetRoute(page: FavoriteCocktailsView()),
}

To Reproduce
Steps to reproduce the behavior:

Set namedRoutes as follow:

namedRoutes: {
	'/cocktails': GetRoute(page: CocktailsView()),
	'/cocktails/:id': GetRoute(page: CocktailDetailsView()),
	'/favorite/cocktails': GetRoute(page: FavoriteCocktailsView()),
}

Get.toNamed('/cocktails/negroni')
See error FavoriteCocktailsView() is opening;

Expected behavior
Open CocktailDetailsView()
Flutter Version:
Flutter (Channel beta, v1.17.0-3.4.pre, on Mac OS X 10.15.4 19E287, locale en-GB)
Get Version:
get: ^2.0.9-dev
Describe on which device you found the bug:
ex: Emulator iPhone 11 (13.4.1)

Hi, thanks for opening this issue.
Initially, I need to clarify that you are not using the lib as I designed it to be used.
I really appreciate stability, and because of that, the library only has resources taken to branch stable that have been thoroughly tested in almost any situation.
I see that you are wanting to define two identical routes, one with parameters, and one without, and forward a different route when there is a parameter.
In theory, this can even be legal, because you delegate the responsibility to the route manager to check whether a route is null or not, or it can be legal for design reasons.
In practice, you would be creating a perfect route, and another ghost route, which would depend on a parameter sent by Get.toNamed to exist. In my view, this can have unwanted effects, simply because you have to ignore onUnknownRoute in some situations to have this functional API. Especially on the web, this could cause problems, and it simply cannot be tested.
namedRoutes: {
	'/cocktails': GetRoute(page: CocktailsView()),

         /// In this line you are basically creating two identical named routes, assigning one parameter
         /// null, and the other creating an object id.
	'/cocktails/:id': GetRoute(page: CocktailDetailsView()),
	'/favorite/cocktails': GetRoute(page: FavoriteCocktailsView()),
}

You can create other router:
'/cocktails/id/:id': GetRoute(page: CocktailDetailsView()),
out: url/cocktails/id/12345
or use web syntax
'/cocktails': GetRoute(page: CocktailsView()),
'/cocktails/id': GetRoute(page: CocktailDetailsView()),
'/favorite/cocktails': GetRoute(page: FavoriteCocktailsView()),
Get.toNamed("/cocktails/id?1234");
Keep in mind that creating an Object as a parameter does not mean creating a new route, and if you need to hide part of the parameter, just like in the web language, you can use url rewrites, like htaccess in apache, or nginx rewrites (in web side, if you're programming mobile / desktop, I can't think of a good reason to use a new route without typing it).
Obviously, there is a way to make a workaround to work that way, which I would not recommend (giving an abstract name for the route and replacing between the pushing of the route, and the receipt of it by the MaterialApp), as this can have side effects on onUnknownRoute and the general security of the route code, since we would have no way of predicting which routes could be used (maybe dynamic routes are not that dynamic).
Well, I'm going to do an update right now, and I'm going to add an undocumented super, mega, blaster, hyper experimental API, which you can use if you are in the beta / dev / master branch. I am not going to document this, because I know the risk involved, and I would be ashamed of anyone looking at the related code, because it is in fact a gambiarra ("if" /: "is present in the routes, create a phantom route and name it as soon as you receive the parameter sent in Get.toNamed)
If I ever find a safe way to do this, I backport to stable, but honestly, that for me will never be safe enough, as we are creating ghost routes, and giving the user the power to define them through parameters.


Skip invalid declarations in SourceLookupCache

Crashers 28598 and 28778 are creating invalid validation requests for the ITC
by using unqualified lookup to get the validator to jump inside of a
transitively invalid DeclContext.
Just don't load these members.
@swift-ci please smoke test

@swift-ci please smoke test

⛵


[SUPPORT]java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes

Tips before filing an issue

Have you gone through our FAQs?

Join the mailing list to engage in conversations and get faster support at<EMAIL_ADDRESS>

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
when I write hudi cow table to aws s3 concurrently by spark api, the except (java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes) happen. I use hudi ZookeeperBasedLockProvider occ.
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
1.Run 0.11.0 version Hudi  write concurrently to aws s3.
2.use the below configuration.
resultDF.write.format("hudi")
.option(DataSourceWriteOptions.OPERATION.key(), DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
.option(DataSourceWriteOptions.TABLE_TYPE.key(), DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL)
.option(DataSourceWriteOptions.RECORDKEY_FIELD.key(), "uq_id,_track_id,event,_flush_time")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD.key(), "process_time")
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD.key(), "p_day,p_hour,p_region,p_type")
.option(DataSourceWriteOptions.KEYGENERATOR_CLASS_NAME.key(), classOf[ComplexKeyGenerator].getName)
.option(HoodieWriteConfig.TBL_NAME.key(), sinkHudiTable)
.option(DataSourceWriteOptions.HIVE_URL.key(), "")
.option(DataSourceWriteOptions.HIVE_DATABASE.key(), "default")
.option(DataSourceWriteOptions.HIVE_TABLE.key(), sinkHudiTable)
.option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS.key(), "p_day,p_hour,p_region,p_type")
.option(DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS.key(), classOf[MultiPartKeysValueExtractor].getName)
.option(DataSourceWriteOptions.HIVE_SYNC_ENABLED.key(), "true")
.option(HoodieIndexConfig.BLOOM_INDEX_UPDATE_PARTITION_PATH_ENABLE.key(), "true")
.option(HoodieIndexConfig.INDEX_TYPE.key(), HoodieIndex.IndexType.GLOBAL_BLOOM.name())
.option(HoodieCompactionConfig.CLEANER_POLICY.key(), HoodieCleaningPolicy.KEEP_LATEST_COMMITS.name)
.option(HoodieCompactionConfig.ASYNC_CLEAN.key(), "true")
.option(HoodieCompactionConfig.CLEANER_COMMITS_RETAINED.key(), "240")
.option(HoodieCompactionConfig.MIN_COMMITS_TO_KEEP.key(), "250")
.option(HoodieCompactionConfig.MAX_COMMITS_TO_KEEP.key(), "260")
.option(HoodieCompactionConfig.FAILED_WRITES_CLEANER_POLICY.key(), HoodieFailedWritesCleaningPolicy.LAZY.name())
.option(HoodieWriteConfig.WRITE_CONCURRENCY_MODE.key(), WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL.name())
.option("hoodie.write.lock.provider", "org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider")
.option("hoodie.write.lock.zookeeper.url", "")
.option("hoodie.write.lock.zookeeper.port", "2181")
.option("hoodie.write.lock.zookeeper.lock_key", sinkHudiTable)
.option("hoodie.write.lock.zookeeper.base_path", "/hudi_multiwriter")
.option("hoodie.insert.shuffle.parallelism", "2")
.option("hoodie.upsert.shuffle.parallelism", "2")
.mode(SaveMode.Append)
.save(s3outPath)
3.
Expected behavior
Exception in thread "main" org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : 0.11.0

Spark version : 3.2.0

Hive version : 3.1.2

Hadoop version : 3.2.1

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

Additional context
Add any other context about the problem here.
the aws emr version is 6.7.0
Stacktrace
Add the stacktrace of the error.
Exception in thread "main" org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)
at org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)
at org.apache.hudi.client.SparkRDDWriteClient.preCommit(SparkRDDWriteClient.java:475)
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:233)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:122)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:678)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:313)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:165)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:115)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:112)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:108)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:519)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:83)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:519)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:495)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:108)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:95)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:136)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:303)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at EtlHudi2hudi.writeData(EtlHudi2hudi.scala:113)
at EtlHudi2hudi.run(EtlHudi2hudi.scala:53)
at EtlHudi2hudi$.main(EtlHudi2hudi.scala:32)
at EtlHudi2hudi.main(EtlHudi2hudi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1000)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1089)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1098)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
... 64 more

Can you job recover automatically from the failure?

Can you job recover automatically from the failure?

In my case,the exception is not root exception, so when this exception happens, my job is always running. so I can not catch the failure of the job. Do you have any other suggestions？ thanks very much.

this should be expected behavior when multiple writers write records into the same File group.
What behavior do you want? @maikouliujian

this should be expected behavior when multiple writers write records into the same File group. What behavior do you want? @maikouliujian

In my case，when this exception happens,my job is not failed,but always running . however ,the job
can not finish . so how can i know the job runs correctly.

how are you writing to hudi. Can you give us some reproducible script. Is it spark-datasource, or spark streaming, detalstreamer, spark-sql. atleast from spark-shell, when you are using spark-ds writer, we know the command fails for sure.

looks like its a spark datasource write. how do you claim that the job does not fail? are you executing it from spark-shell and the command to write to hudi is just stuck?

I tried multi-writers from two diff spark-shells, and one of them fails while writing to hudi.

scala> df2.write.format("hudi").
     |   options(getQuickstartWriteConfigs).
     |   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
     |   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
     |   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
     |   option(TABLE_NAME, tableName).
     |   option("hoodie.write.concurrency.mode","optimistic_concurrency_control").
     |   option("hoodie.cleaner.policy.failed.writes","LAZY").
     |   option("hoodie.write.lock.provider","org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider").
     |   option("hoodie.write.lock.zookeeper.url","localhost:2181").
     |   option("hoodie.write.lock.zookeeper.port","2181").
     |   option("hoodie.write.lock.zookeeper.lock_key","locks").
     |   option("hoodie.write.lock.zookeeper.base_path","/tmp/locks/.lock").
     |   mode(Append).
     |   save(basePath)
warning: there was one deprecation warning; re-run with -deprecation for details
[Stage 14:>                                                         (0 + 3) / 3]# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
23/01/23 10:00:20 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
  at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)
  at org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
  at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
  at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
  at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
  at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)
  at org.apache.hudi.client.SparkRDDWriteClient.preCommit(SparkRDDWriteClient.java:491)
  at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:234)
  at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:126)
  at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:698)
  at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:343)
  at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
  at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:696)
  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:696)
  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:696)
  at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:305)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:291)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:249)
  ... 75 elided
Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
  ... 109 more

scala>

Write to hudi fails and next command prompt it seen.
excerpt from my other shell which succeeded.
scala> df2.write.format("hudi").
     |   options(getQuickstartWriteConfigs).
     |   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
     |   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
     |   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
     |   option(TABLE_NAME, tableName).
     |   option("hoodie.write.concurrency.mode","optimistic_concurrency_control").
     |   option("hoodie.cleaner.policy.failed.writes","LAZY").
     |   option("hoodie.write.lock.provider","org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider").
     |   option("hoodie.write.lock.zookeeper.url","localhost:2181").
     |   option("hoodie.write.lock.zookeeper.port","2181").
     |   option("hoodie.write.lock.zookeeper.lock_key","locks").
     |   option("hoodie.write.lock.zookeeper.base_path","/tmp/locks/.lock").
     |   mode(Append).
     |   save(basePath)
warning: one deprecation; for details, enable `:setting -deprecation' or `:replay -deprecation'
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
23/01/23 10:00:19 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties

scala>

If you can provide us w/ reproducible script, would be nice. as of now, its not reproducible from our end

@maikouliujian : any updates please.

how are you writing to hudi. Can you give us some reproducible script. Is it spark-datasource, or spark streaming, detalstreamer, spark-sql. atleast from spark-shell, when you are using spark-ds writer, we know the command fails on conflicts.

In my case,I run my job by spark-ds writer.

looks like its a spark datasource write. how do you claim that the job does not fail? are you executing it from spark-shell and the command to write to hudi is just stuck?

In my case, I run my job by azkaban hourly scheduled. when multi jobs run, the azkaban job not fail, but always running .
I see the detail log, the exception happends. Why my azkaban job can not fail?

I tried multi-writers from two diff spark-shells, and one of them fails while writing to hudi.

scala> df2.write.format("hudi").
     |   options(getQuickstartWriteConfigs).
     |   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
     |   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
     |   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
     |   option(TABLE_NAME, tableName).
     |   option("hoodie.write.concurrency.mode","optimistic_concurrency_control").
     |   option("hoodie.cleaner.policy.failed.writes","LAZY").
     |   option("hoodie.write.lock.provider","org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider").
     |   option("hoodie.write.lock.zookeeper.url","localhost:2181").
     |   option("hoodie.write.lock.zookeeper.port","2181").
     |   option("hoodie.write.lock.zookeeper.lock_key","locks").
     |   option("hoodie.write.lock.zookeeper.base_path","/tmp/locks/.lock").
     |   mode(Append).
     |   save(basePath)
warning: there was one deprecation warning; re-run with -deprecation for details
[Stage 14:>                                                         (0 + 3) / 3]# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
23/01/23 10:00:20 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
  at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)
  at org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
  at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
  at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
  at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
  at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)
  at org.apache.hudi.client.SparkRDDWriteClient.preCommit(SparkRDDWriteClient.java:491)
  at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:234)
  at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:126)
  at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:698)
  at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:343)
  at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
  at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
  at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:83)
  at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:81)
  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:696)
  at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:696)
  at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:80)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:127)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:75)
  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:696)
  at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:305)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:291)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:249)
  ... 75 elided
Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
  ... 109 more

scala>

Write to hudi fails and next command prompt it seen.
excerpt from my other shell which succeeded.
scala> df2.write.format("hudi").
     |   options(getQuickstartWriteConfigs).
     |   option(PRECOMBINE_FIELD_OPT_KEY, "ts").
     |   option(RECORDKEY_FIELD_OPT_KEY, "uuid").
     |   option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
     |   option(TABLE_NAME, tableName).
     |   option("hoodie.write.concurrency.mode","optimistic_concurrency_control").
     |   option("hoodie.cleaner.policy.failed.writes","LAZY").
     |   option("hoodie.write.lock.provider","org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider").
     |   option("hoodie.write.lock.zookeeper.url","localhost:2181").
     |   option("hoodie.write.lock.zookeeper.port","2181").
     |   option("hoodie.write.lock.zookeeper.lock_key","locks").
     |   option("hoodie.write.lock.zookeeper.base_path","/tmp/locks/.lock").
     |   mode(Append).
     |   save(basePath)
warning: one deprecation; for details, enable `:setting -deprecation' or `:replay -deprecation'
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
23/01/23 10:00:19 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties

scala>

If you can provide us w/ reproducible script, would be nice. as of now, its not reproducible from our end
In hudi 0.11.0, it can not supports multiple writers on spark ds ?

I have the same issue of running multi-write in local console as well

"""
Install
https://dlcdn.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop2.tgz
hadoop2.7
https://github.com/soumilshah1995/winutils/blob/master/hadoop-2.7.7/bin/winutils.exe
pyspark --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.12.1 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
VAR
SPARK_HOME
HADOOP_HOME
PATH
%HAPOOP_HOME%\bin
%SPARK_HOME%\bin
Complete Tutorials on HUDI
https://github.com/soumilshah1995/Insert-Update-Read-Write-SnapShot-Time-Travel-incremental-Query-on-APache-Hudi-transacti/blob/main/hudi (1).ipynb
"""
import os
import sys
import uuid
import pyspark
from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from pyspark.sql.functions import col, asc, desc
from pyspark.sql.functions import col, to_timestamp, monotonically_increasing_id, to_date, when
from pyspark.sql.functions import *
from pyspark.sql.types import *
from datetime import datetime
from functools import reduce
from faker import Faker
from faker import Faker
import findspark
import datetime
time = datetime.datetime.now()
time = time.strftime("YMD%Y%m%dHHMMSSms%H%M%S%f")
SUBMIT_ARGS = "--packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.12.1 pyspark-shell"
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
findspark.init()
spark = SparkSession.builder
.config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer')
.config('className', 'org.apache.hudi')
.config('spark.sql.hive.convertMetastoreParquet', 'false')
.config('spark.sql.extensions', 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension')
.config('spark.sql.warehouse.dir', 'file:///C:/tmp/spark_warehouse')
.getOrCreate()
global faker
faker = Faker()
class DataGenerator(object):
@staticmethod
def get_data():
    return [
        (
            x,
            faker.name(),
            faker.random_element(elements=('IT', 'HR', 'Sales', 'Marketing')),
            faker.random_element(elements=('CA', 'NY', 'TX', 'FL', 'IL', 'RJ')),
            faker.random_int(min=10000, max=150000),
            faker.random_int(min=18, max=60),
            faker.random_int(min=0, max=100000),
            faker.unix_time()
        ) for x in range(5)
    ]

data = DataGenerator.get_data()
columns = ["emp_id", "employee_name", "department", "state", "salary", "age", "bonus", "ts"]
spark_df = spark.createDataFrame(data=data, schema=columns)
print(spark_df.show())
db_name = "hudidb"
table_name = "hudi_table"
recordkey = 'emp_id'
precombine = 'ts'
path = "file:///C:/tmp/spark_warehouse"
method = 'upsert'
table_type = "COPY_ON_WRITE"
hudi_options = {
'hoodie.table.name': table_name,
'hoodie.datasource.write.recordkey.field': 'emp_id',
'hoodie.datasource.write.table.name': table_name,
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'ts',
'hoodie.upsert.shuffle.parallelism': 2,
'hoodie.insert.shuffle.parallelism': 2,
'hoodie.schema.on.read.enable' : 'true', # for changing column names
'hoodie.write.concurrency.mode':'optimistic_concurrency_control',     #added for zookeeper to deal with multiple source writes
'hoodie.cleaner.policy.failed.writes':'LAZY',
# 'hoodie.write.lock.provider':'org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider',

'hoodie.write.lock.provider':'org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider',
'hoodie.write.lock.zookeeper.url':'localhost',
'hoodie.write.lock.zookeeper.port':'2181',
'hoodie.write.lock.zookeeper.lock_key':'my_lock',
'hoodie.write.lock.zookeeper.base_path':'/hudi_locks',

}
print("*"55)
print("over-write")
print(""*55)
spark_df.write.format("hudi").
options(**hudi_options).
mode("overwrite").
save(path)
print("*"55)
print("READ")
print(""*55)
read_df = spark.read.
format("hudi").
load(path)
print(read_df.show())
impleDataUpd = [
(6, "This is APPEND4", "Sales", "RJ", 81000, 30, 23000, 827307999),
(7, "This is APPEND4", "Engineering", "RJ", 79000, 53, 15000,<PHONE_NUMBER>),
]
columns = ["emp_id", "employee_name", "department", "state", "salary", "age", "bonus", "ts"]
usr_up_df = spark.createDataFrame(data=impleDataUpd, schema=columns)
usr_up_df.write.format("hudi").options(**hudi_options).mode("append").save(path)
print("*"55)
print("READ")
print(""*55)
read_df = spark.read.
format("hudi").
load(path)
print(read_df.show())

when I run the above code in 2 seperate notebooks to simulate the multi-write process, the error occurs. Just like what @maikouliujian had faced

@tomyanth
I tried to run with 0.13.0 version and I didn't had any issues like spark-shell job getting stuck.  Can you try with 0.13.0 if you still face the issue.

try using DynamoDB as lock Table

DYNAMODB_LOCK_TABLE_NAME = 'hudi-lock-table'
curr_session = boto3.session.Session()
curr_region = curr_session.region_name

def upsert_hudi_table(glue_database, table_name,
                      record_id, precomb_key, table_type, spark_df,
                      enable_partition, enable_cleaner, enable_hive_sync, enable_dynamodb_lock,
                      use_sql_transformer, sql_transformer_query,
                      target_path, index_type, method='upsert'):
    """
    Upserts a dataframe into a Hudi table.

    Args:
        glue_database (str): The name of the glue database.
        table_name (str): The name of the Hudi table.
        record_id (str): The name of the field in the dataframe that will be used as the record key.
        precomb_key (str): The name of the field in the dataframe that will be used for pre-combine.
        table_type (str): The Hudi table type (e.g., COPY_ON_WRITE, MERGE_ON_READ).
        spark_df (pyspark.sql.DataFrame): The dataframe to upsert.
        enable_partition (bool): Whether or not to enable partitioning.
        enable_cleaner (bool): Whether or not to enable data cleaning.
        enable_hive_sync (bool): Whether or not to enable syncing with Hive.
        use_sql_transformer (bool): Whether or not to use SQL to transform the dataframe before upserting.
        sql_transformer_query (str): The SQL query to use for data transformation.
        target_path (str): The path to the target Hudi table.
        method (str): The Hudi write method to use (default is 'upsert').
        index_type : BLOOM or GLOBAL_BLOOM
    Returns:
        None
    """
    # These are the basic settings for the Hoodie table
    hudi_final_settings = {
        "hoodie.table.name": table_name,
        "hoodie.datasource.write.table.type": table_type,
        "hoodie.datasource.write.operation": method,
        "hoodie.datasource.write.recordkey.field": record_id,
        "hoodie.datasource.write.precombine.field": precomb_key,
    }

    # These settings enable syncing with Hive
    hudi_hive_sync_settings = {
        "hoodie.parquet.compression.codec": "gzip",
        "hoodie.datasource.hive_sync.enable": "true",
        "hoodie.datasource.hive_sync.database": glue_database,
        "hoodie.datasource.hive_sync.table": table_name,
        "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
        "hoodie.datasource.hive_sync.use_jdbc": "false",
        "hoodie.datasource.hive_sync.mode": "hms",
    }

    # These settings enable automatic cleaning of old data
    hudi_cleaner_options = {
        "hoodie.clean.automatic": "true",
        "hoodie.clean.async": "true",
        "hoodie.cleaner.policy": 'KEEP_LATEST_FILE_VERSIONS',
        "hoodie.cleaner.fileversions.retained": "3",
        "hoodie-conf hoodie.cleaner.parallelism": '200',
        'hoodie.cleaner.commits.retained': 5
    }

    # These settings enable partitioning of the data
    partition_settings = {
        "hoodie.datasource.write.partitionpath.field": args['PARTITON_FIELDS'],
        "hoodie.datasource.hive_sync.partition_fields": args['PARTITON_FIELDS'],
        "hoodie.datasource.write.hive_style_partitioning": "true",
    }

    # Define a dictionary with the index settings for Hudi
    hudi_index_settings = {
        "hoodie.index.type": index_type,  # Specify the index type for Hudi
    }

    hudi_dynamo_db_based_lock = {
        'hoodie.write.concurrency.mode': 'optimistic_concurrency_control'
        , 'hoodie.cleaner.policy.failed.writes': 'LAZY'
        , 'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider'
        , 'hoodie.write.lock.dynamodb.table': DYNAMODB_LOCK_TABLE_NAME
        , 'hoodie.write.lock.dynamodb.partition_key': 'tablename'
        , 'hoodie.write.lock.dynamodb.region': '{0}'.format(curr_region)
        , 'hoodie.write.lock.dynamodb.endpoint_url': 'dynamodb.{0}.amazonaws.com'.format(curr_region)
        , 'hoodie.write.lock.dynamodb.billing_mode': 'PAY_PER_REQUEST'

    }

    hudi_file_size = {
        "hoodie.parquet.max.file.size": 512 * 1024 * 1024,  # 512MB
        "hoodie.parquet.small.file.limit": 104857600,  # 100MB
    }

    # Add the Hudi index settings to the final settings dictionary
    for key, value in hudi_index_settings.items():
        hudi_final_settings[key] = value  # Add the key-value pair to the final settings dictionary

    for key, value in hudi_file_size.items():
        hudi_final_settings[key] = value

        # If partitioning is enabled, add the partition settings to the final settings
    if enable_partition == "True" or enable_partition == "true" or enable_partition == True:
        for key, value in partition_settings.items(): hudi_final_settings[key] = value

    # if DynamoDB based lock enabled use dynamodb as lock table
    if enable_dynamodb_lock == "True" or enable_dynamodb_lock == "true" or enable_dynamodb_lock == True:
        for key, value in hudi_dynamo_db_based_lock.items(): hudi_final_settings[key] = value

    # If data cleaning is enabled, add the cleaner options to the final settings
    if enable_cleaner == "True" or enable_cleaner == "true" or enable_cleaner == True:
        for key, value in hudi_cleaner_options.items(): hudi_final_settings[key] = value

    # If Hive syncing is enabled, add the Hive sync settings to the final settings
    if enable_hive_sync == "True" or enable_hive_sync == "true" or enable_hive_sync == True:
        for key, value in hudi_hive_sync_settings.items(): hudi_final_settings[key] = value

    # If there is data to write, apply any SQL transformations and write to the target path
    if spark_df.count() > 0:

        if use_sql_transformer == "True" or use_sql_transformer == "true" or use_sql_transformer == True:
            spark_df.createOrReplaceTempView("temp")
            spark_df = spark.sql(sql_transformer_query)

        # Replace null values in all columns with default value 'unknown'
        default_value = 'n/a'
        for column in spark_df.columns:
            spark_df = spark_df.na.fill(default_value)

        print("**************************************************************")
        print(spark_df.show())
        print("**************************************************************")

        spark_df.write.format("hudi"). \
            options(**hudi_final_settings). \
            mode("append"). \
            save(target_path)

you can use dynamoDB lock tables

DYNAMODB_LOCK_TABLE_NAME = 'hudi-lock-table'
curr_session = boto3.session.Session()
curr_region = curr_session.region_name

    hudi_dynamo_db_based_lock = {
        'hoodie.write.concurrency.mode': 'optimistic_concurrency_control'
        , 'hoodie.cleaner.policy.failed.writes': 'LAZY'
        , 'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider'
        , 'hoodie.write.lock.dynamodb.table': DYNAMODB_LOCK_TABLE_NAME
        , 'hoodie.write.lock.dynamodb.partition_key': 'tablename'
        , 'hoodie.write.lock.dynamodb.region': '{0}'.format(curr_region)
        , 'hoodie.write.lock.dynamodb.endpoint_url': 'dynamodb.{0}.amazonaws.com'.format(curr_region)
        , 'hoodie.write.lock.dynamodb.billing_mode': 'PAY_PER_REQUEST'

    }

Hello @Jason-liujc @maikouliujian
I am seeing exact error, and also using dynamoDB lock same way as last comment, were you guys able to figure out the work around for it ?
Or anything to fix this issue ?, facing something very similar.

The main thing we did was to change our Hudi table structure to avoid concurrent writes to the same partition as much as possible (batch workload together, sequence the job etc)
For us, the DynamoDB lock provider wasn't able to to do any write retries, so it just fails the Spark job. We increased the yarn and spark retry to automatically retry from the cluster side.

@Jason-liujc
Can we just increase yarn.resourcemanager.am.max-attempts to higher number?
so we can retry to run hudi again if somehow it fails on  java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes.

I heard that DynamoDB lock provider doesn't work with retries, but zookeeper does ?
If anyone has knowledge about this, would mind sharing here ?

Can't speak to what the official guidance from Hudi is at the moment (seems like they will rollout the non-blocking concurent write feature in version 1.0+).
We had to increase yarn.resourcemanager.am.max-attempts and spark.yarn.maxAppAttempts (the spark specific config) to make it retry more and reoganize our tables to reduce concurrent writes. Any other lock provider wasn't an option for us since we are running different jobs from different clusters.

I'm seeing this with flink using the hive metastore for locking as well.

Do you have multi writers, do you config the lock providers explicitly?

Yes. This happens this seems to happens with multiple writers. I do specify the lock config:
         'compaction.delta_commits'='2',
         'connector' = 'hudi',
         'hive_sync.db'='{hudi_db}',
         'hive_sync.enable'='true',
         'hive_sync.jdbcurl'='{hive_jdbc_url}',
         'hive_sync.metastore.uris'='{hive_thrift_url}',
         'hive_sync.mode'='hms',
         'hive_sync.password'='',
         'hive_sync.table'='{hudi_table}',
         'hive_sync.username'='hive',
         'hoodie.cleaner.policy.failed.writes' = 'LAZY',
         'hoodie.database.name'='{hudi_db}',
         'hoodie.datasource.write.recordkey.field'='name',
         'hoodie.embed.timeline.server'='false',
         'hoodie.parquet.compression.codec' = 'snappy',
         'hoodie.payload.event.time.field' = 'event_time',
         'hoodie.payload.ordering.field' = 'event_time',
         'hoodie.table.name'='{hudi_table}',
         'hoodie.table.precombine.field' = 'event_time',
         'hoodie.table.recordkey.fields' = 'name',
         'hoodie.write.concurrency.mode' = 'optimistic_concurrency_control',
         'hoodie.write.lock.hivemetastore.database'='{hudi_db}',
         'hoodie.write.lock.hivemetastore.table'='{hudi_table}',
         'hoodie.write.lock.provider' = 'org.apache.hudi.hive.transaction.lock.HiveMetastoreBasedLockProvider',
         'path' = '{hudi_table_path}',
         'precombine.field'='event_time',
         'read.streaming.enabled'='true',
         'read.streaming.skip_compaction'='true',
         'read.tasks'='4',
         'recordkey.field'='name',
         'table.type' = 'MERGE_ON_READ',
         'write.operation' = 'upsert',
         'write.precombine.field' = 'event_time',
         'write.recordkey.field'='name',
         'write.tasks'='4'

You are using OCC, does the conflict really happens or not? If it happens, the job needs a retry to make the comit succeed.


Segfault on glfwSwapBuffers when diagonally resizing

Hello,
I am receiving a segfault when calling glfwSwapBuffers() from a window refresh callback in 3.3.2 while the window is being diagonally resized. It crashes @ wgl_context.c::340 i.e.
SwapBuffers(window->context.wgl.dc);

It only happens when the window is being diagonally resized (i.e. resized in both dimensions at the same time). You can reproduce it quickly by rapidly dragging the bottom-right corner of the window (to resize it) in circles until it crashes--it typically happens after about 4-5 calls to the refresh callback.
Compiler: mingw-w64
OS: Windows 10 Pro
CPU: AMD Ryzen 7 3800X
GPU: NVIDIA GeForce RTX 2080
Here's an example that causes the crash:
#include <glew.h>
#include <glfw3.h>
#include <iostream>

void refresh(GLFWwindow* glfwWindow)
{
    std::cout << "refresh\n";
    glfwSwapBuffers(glfwWindow);
}

int main()
{
    if (!glfwInit())
    {
        return 0;
    }

    glfwWindowHint(GLFW_OPENGL_FORWARD_COMPAT, GL_TRUE);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 4);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 3);
    glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);
    glfwWindowHint(GLFW_FOCUSED, GL_TRUE);

    GLFWwindow* glfwWindow = glfwCreateWindow(1024, 768, "resize crash test", NULL, NULL);

    if (glfwWindow == NULL)
    {
        return 0;
    }

    glfwMakeContextCurrent(glfwWindow);

    glewExperimental = true;
    if ( glewInit() != GLEW_OK )
    {
        return 0;
    }

    glfwSwapInterval(1);
    glfwSetWindowRefreshCallback(glfwWindow, refresh);

    while (glfwGetKey(glfwWindow, GLFW_KEY_ESCAPE ) != GLFW_PRESS
        && glfwWindowShouldClose(glfwWindow) == 0)
    {
        glfwSwapBuffers(glfwWindow);
        glfwPollEvents();
    }

    return 0;
}

Cannot reproduce this with 3.3.2 or pre-3.4. MinGW-w64, Windows 10, GeForce GTX 1050M.

I have the same problem that only diagonal resize will result in program crash, and I am using 2080 Super with driver 471.96 on WIndows 10 Education.
The problem also happens on my macOS (M1). Hence I don't think it is a driver or hardware issue but glfw.

Cannot reproduce this with 3.3.2 or pre-3.4. MinGW-w64, Windows 10, GeForce GTX 1050M.

For reference we encountered varying similar issues :(
Nahimic and some without Nahimic:
https://github.com/ocornut/imgui/issues/3321
Nahimic
https://github.com/ocornut/imgui/issues/3204
(Here) Asus's Sonic Suite:
https://github.com/glfw/glfw/issues/905
How ironic that two audio solutions are making OpenGL resize crash (maybe same underlying tech with same bugs).
But in https://github.com/ocornut/imgui/issues/3321 there are reports not involving either.

I can confirm, disabling Nahimic service fixed the issue.

This is terrifying but very good to know. Thank you both!


Feat: Slack bot integration

This pull request adds a QCoDeS slack bot that provides information about current measurements.
The user can communicate with the Slack bot via instant messaging (see image below).
The slack bot currently has the following functionality:

Give information about the current measurement (percentage done, print of dataset).
Upload the most recent plot
Notify when measurement is done
Adding custom commands and periodically-executed tasks

A few notes

I added the class attributes DataSet.latest_dataset and BasePlot.latest_plot. Every time a dataset/plot is instantiated, this attribute points to that object. This allows the slack bot to determine the latest dataset/plot.
I added to slack bot to qcodes/widgets, but I was not sure if this is the right place. If not, where should it go?
Periodic calls to Slack.update() must be made, and so the Slack bot works best as a widget in multiprocessing.
The package 'tempfile' is used to determine a temporary store location for any plot file. This isn't currently in the package requirements

@giulioungaretti @AdriaanRol

lol this is super cool!  May need more massaging though, as dataset.sync() may disappear soon, and same goes for the hiddenupdatewidget !
We can definitely work a way to insert the slack class into the code base,   I have the feeling that this should rather be a daemon-like process (such as monitoring) than a widget like thing :D
Terrific contribution !

@giulioungaretti I wasn't aware that dataset.sync() would disappear, will it not be necessary later on?
And I was initially also thinking of a subprocess, but with the current multiprocessing architecture I ran into problems with retrieving the latest plot. Also, I noticed some cool things you can do when qcodes is a widget instead of a subprocess. For instance, you can create a cell, fill it with a chunk of code, and then execute it. I'm using this to run measurements from Slack atm, although I'm not sure if this functionality should be part of the QCoDeS package

@giulioungaretti I realized this code doesn't need widgets nor multiprocessing. Instantiating it in a different thread is sufficient. Would this PR be useful for qcodes? If so I can make the changes. What folder would now be a good location for the code?

We can  definitely add this!  I would make an extra folder next to utils.

@giulioungaretti I upgraded the Slack class, it now is a subclass of threading.Thread. The main advantage here is that it operates as a separate thread without depending on ipython widgets.
Starting the Slack client is very easy, namely as such:
slack = Slack(**optional_commands)```

`optional_commands` are commands that can also be executed aside from the standard ones. For instance, it could be a parameter, which we can then perform get/set operations on via Slack.

Since the Slack class needs access to the latest measurement and plot, I also added some functionality to BasePlot and loops.py. BasePlot now has the class attribute `BasePlot.latest_plot`, which is updated every time a new BasePlot object is created. This way, `latest_plot` always refers to the latest created plot.

For loops I did something similar: I created ActiveLoop.active_loop. Using this, I added the two functions active_loop() and active_data_set(), which calls active_loop(). When running a loop with kwarg set_active=True (True by default), it will set ActiveLoop.active_loop to itself for the duration of the measurement. This also works with Measure. In the cases where you have a Loop in which a complex parameter itself creates a Loop or Measure, you can avoid overwriting active_loop by adding set_active = False.

I noticed that active_loop and active_data_set are also useful in other ways. For instance, you can use the folder of the active dataset as a base folder, and any parameters that create Loops can create subfolders in the base dataset folder. I added the two functions to qcodes init, so they can be accessed easier.

I finally also added slacker as an optional package in the setup

looks good to me! @jenshnielsen @WilliamHPNielsen  ?

Looks good to me

great will review the code and merge//ask for changes  today !


Integrate Optimizing Content Efficiency into new Loading section

Ilya previously wrote this great section on optimizing content efficiency: https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/ covering topics like image optimization.
Now that Loading is becoming its own section, we should either directly fold the child content into the new section as-is move across and keep optimizing content efficiency.

This is done.


releaseCmd: define additional variables

This patch sets the following environment variables when running the release command:

$GX_VERSION (replaces $VERSION)
$GX_NAME
$GX_AUTHOR
$GX_LICENSE
$GX_LANGUAGE
$GX_HASH

This makes it possible to include things like the hash in the release commit message/tag.

cc @raulk


Add retrieve submission summaries command

bors r+

Codecov Report

Merging #68 into master will decrease coverage by 0.00%.
The diff coverage is 0.62%.

@@            Coverage Diff            @@
##           master     #68      +/-   ##
=========================================
- Coverage    2.95%   2.95%   -0.01%
=========================================
  Files          23      24       +1
  Lines        4329    4438     +109
  Branches     1925    1964      +39
=========================================
+ Hits          128     131       +3
- Misses       4073    4184     +111
+ Partials      128     123       -5

Impacted Files
Coverage Δ

snowchains_core/src/web/atcoder.rs
0.00% <0.00%> (ø)

snowchains_core/src/web/codeforces.rs
0.00% <ø> (ø)

snowchains_core/src/web/mod.rs
0.00% <ø> (ø)

snowchains_core/src/web/yukicoder.rs
0.00% <ø> (ø)

src/commands/judge.rs
0.00% <ø> (ø)

src/commands/retrieve_submission_summaries.rs
0.00% <0.00%> (ø)

src/commands/submit.rs
0.00% <ø> (ø)

src/lib.rs
7.34% <14.28%> (-0.27%)
:arrow_down:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 753696b...0a63330. Read the comment docs.


Full pleblist removal

Pull request checklist:

[x] CHANGELOG.md was updated, if applicable
[x] Documentation in docs/ or install-docs/ was updated, if applicable
[ ] I have tested all changes

Full pleblist removal excluding the db-related code.

I can see that it might be worth removing the httplib and types-httplib2 dependencies. Any suggestions on how to modify the code to remove it?
https://github.com/pajbot/pajbot/blob/aa8dc95e5ac2f14ad90109ea7880d6c4ce6f431d/pajbot/models/pleblist.py#L85-L86

End of an era


Partial Resource Routes only() except() why throw an error instead of 404 ?

Laravel Version: 5.6
PHP Version: 7.1.19

Description:
https://laravel.com/docs/5.6/controllers#resource-controllers
when i create a controller by php artisan make:controller TestController --resource --model=Test
and use route:
Route::resource('/test', 'TestController')->only(['index', 'store', 'edit', 'update', 'destroy']);
or
Route::resource('/test', 'TestController')->except(['create', 'show']);
if go to http://example.com/test/1 (actually this is route test.show)
why throw an error instead of 404?
Symfony \ Component \ HttpKernel \ Exception \
MethodNotAllowedHttpException

but i following the Laravel 5.6 document
Partial Resource Routes
Route::resource('photos', 'PhotoController')->only([
    'index', 'show'
]);

Route::resource('photos', 'PhotoController')->except([
    'create', 'store', 'update', 'destroy'
]);

your url matches valid route patterns but not for get method, so method not allowed is correct i think
as for it being an exception, laravel uses exceptions for things like this, even for 404s.
it just comes down to how that exception is handled (this will also be different for html/ajax and development/production)

or catch the method not allowed message and swap it for a not found

One unrelated thing, the first parameter to "resource" isn't  a path but the resource name. I wouldn't expect to see a "/" in a resource name.

yep ok i removed the "/"
i tested 3 hours, and found some problem, i don't know what is this
i disabled the Route::resource, and use this instead
// Route::resource('test', 'TestController');

// Route::get('test', 'TestController@index')->name('test.index');
// Route::post('test', 'TestController@store')->name('test.store');
// Route::get('test/{test}/edit', 'TestController@edit')->name('test.edit');
// Route::patch('test/{test}', 'TestController@update')->name('test.update');
// Route::delete('test/{test}', 'TestController@destroy')->name('test.destroy');

ok i un-comment ONE line each time from top to bottom, no abc this method in controller
and go to (GET) http://localhost/test/abc
->name('test.index');    // 404 , because not match, so is correct
->name('test.store');    // 404 , because not match, so is correct
->name('test.edit');      // 404 , because not match, so is correct
->name('test.update'); // error MethodNotAllowedHttpException, i was use GET, but this is for PATCH, ??
->name('test.destroy'); // error same as ->name('test.update'), this is for delete, why ??

Routes are working as they should. The errors you are seeing are triggered because test.update and test.destroy don't accept GET method that your browser is using to request the pages, hence the MethodNotAllowedHttpException.
Usually GET and POST requests that can't be resolved, trigger 404 error (NOT FOUND), and MethodNotAllowedHttpException trigger 405 error (METHOD NOT ALLOWED).
This has been defined in \Symfony\Component\HttpKernel\Exception\MethodNotAllowedHttpException that is used by \Illuminate\Routing\RouteCollection.
Summa summarum: PUT, PATCH and DELETE don't allow GET requests what your browser is using to access, so it tells you that. If you want to use those routes, you need to use Form Method Spoofing in your POST requests.

thankyou for your reply @ivuorinen
i know the Form Method Spoofing
but my problem is , when i go to http://localhost/test/abc (via GET, type the url in browser address bar)
and also GET /test/{test} is NOT in the route:list
why the GET request trigger the test.update and test.destroy
i want GET /test/{test} goes to 404
actually not in the route list is goes to 404, right ?

Then you need to allow GET, catch the request, check the method and if GET return your 404 error page.
Something like this:
# In your routes/web.php
Route::get('/test/{test}', 'TestController@update');
Route::patch('/test/{test}', 'TestController@update')->name('test.update');

# In your Controller
public function update(\Illuminate\Http\Request $request)
{
    if ($request->isMethod('GET')) {
        return abort(404, 'Your error message');
    }

    // Your code...
}

I didn't test these, but the logic should be fine.

ref to https://laravel.com/docs/5.6/controllers#resource-controllers
i don't need create and show route
Route::get('/test/create', 'TestController@create'); // this is equal to create
Route::get('/test/{test}', 'TestController@show'); // this is equal to show
not in the route:list still trigger the route, i don't know is this a bug or?
now i was using something like this in the controller to ignore the error details showing on the browser
public function create()
{
    abort(404);
}

public function show($test)
{
    abort(404);
}

but this not make sense ;(

if you want to understands the problem, just do a simple test
create a controller, run
php artisan make:controller TestController --resource
add route
Route::resource('test', 'TestController')->except(['create', 'show']);
go to the url in your browser
http://localhost/test/1
or
http://localhost/test/create
you will see an error instead of a blank page

That is expected. The URL is valid, but is being accessed via a disallowed method.
A 404 means the resource doesn't exist. A 405 means the resource exists, but you're accessing it wrong.

so what the ->except used for ?

When you call except(['create', 'show']), the create and show routes simply aren't created.
The update route is still created, which uses the same URL structure as show, but a different method.
Since you didn't define a test/create url, test/create matches test/{test}, which is a valid URL defined for the PUT and PATCH methods, but not for GET.

yes @36864 get the point
which is a valid URL defined for the PUT and PATCH methods, but not for GET.
so if i put this line over on top of other routes, will this route still proccess ? even the form look like this
Route::get('/test/{test}', 'TestController@update'); // this is for update via GET

<form method="POST" action="/test/create">
    @method('delete')
    @csrf
</form>


Add support for model's with a binary uuid

In these commits I changed some behaviors that they work more with Laravel and Scout's default behavior. I needed this in order to be able to use binary uuids on my models. In detail I changed the following files:
ElasticEngine.php
I removed this manual result to model mapping function and replaced it with the one of Laravel Scout's Searchable trait.
DocumentPayload.php
Call the scout key method before serializing any documents. By default the getScoutKey method calls the getKey method.

Hi @manueldublanc,
Thank you for contribution it was solved in https://github.com/babenkoivan/scout-elasticsearch-driver/pull/209, which I just merged.


Disable weird lines (hover? focus?) in the integrated terminal

There are two lines in the terminal left margin. One, very bright white, line appears on hover on each command, and the other, gray, line appears I don't know when, sometimes. I don't know what they do, I don't need them, and they are very very distracting. Is there a way to disable them? Or at least change the color to transparent.

does disabling terminal.integrated.shellIntegration.enabled fix this for you? I am not reproducing, but we have command line decorations when one navigates using ctrlCmd+up/downArrow with shell integration, so wondering if that's what you're seeing?

This is called the "command guide" and doesn't seem to have a documentation entry yet. It uses the same theme key as the similar feature in notebooks right now: notebook.inactiveFocusedCellBorder. It looks like your theme is an atypical one where it's light everywhere except for the terminal, so when in notebooks it might be the same as the background color.
You can make it invisible with this setting:
"workbench.colorCustomizations": {
  "notebook.inactiveFocusedCellBorder": "#0000"
}

We should add a setting to disable this and a separate theme key.

The smaller line is command navigation. Those lines just stick around currently until they're moved again, you could clear the terminal, use command nav again or create a new terminal to move that one.

To verify make sure the terminal.integrated.shellIntegration.showCommandGuide setting and terminalCommandGuide.foreground theme keys work as expected.
The command guide should show when hovering a command or its output in a shell integration-enabled terminal (provided the command isn't very long):

terminal.integrated.shellIntegration.showCommandGuide seems to have no effect on already-open terminals, only new ones

@joyceerhl could you create an issue for that? It should

https://github.com/microsoft/vscode/issues/226868


Add karma-sabarivka-reporter to ensure all source files under coverage

This project is fantastic and it's unit test coverage on very high level. Therefore, I've decided to create this PR and improve it a little more.
By adding this simple plugin, developers/architects of the project can ensure no new functionality (as new source file) will be added to the project without testing it.
Explanation:
With karma and istanbul reporter, if someone simply forgets to create feature.spec.js file for feature.js file - coverage result won't include feature.js source file

TBH I don't see any big difference in our case.

@XhmikosR Yes, you're totally right. Currently - no difference at all. Anyway, reason for this is to automate one more thing and save time in the future, e.g. during code review.

I'm a bit mismatched here, I understand the need of this karma plugin, but unit test is a requirement for our JS code, so if there is a lack of unit test we'll see it during code review because we review unit tests too

@Johann-S, OK, got it. Feel free to close or merge this PR. If you think, you don't need this change – fine with me. In this repo, as I see, it's very little chance to get such tests regression. This repo is tested very well.  Thanks.

Thanks for the PR but we are gonna pass for now. :)


S3 tests

This adds integration tests for the list_warc_paths and extract_text jobs, using FakeS3 to mock out S3 - and more broadly, a general strategy for testing interactions with S3. Here's the idea:

To interact with FakeS3 in the tests, we need to change global settings that control the parameters passed to boto.connect_s3(). First, pytest-env is used to automatically set a OSP_ENV environment variable to test when pytest is run. In pytest.ini:

[pytest]
env =
  OSP_ENV=test

Then, a Config class uses anyconfig to load a hierarchy of YAML files. In ascending order of priority - first a default config.yml in the code, then /etc/osp/osp.yml. And, if OSP_ENV == test, it also loads /etc/osp/osp.test.yml, so that the testing values get applied automatically when the code is run under pytest. Once these files are assembled into a single dictionary by anyconfig, the dict is passed through a schema validator, to catch any errors in the config files. Config just subclasses dict, so, the values can be looked up like config['results']['texts'].

The osp.test.yml file (which is deployed to Vagrant by Ansible, so nothing manual needed) provides the overrides that point boto at FakeS3:

s3:
  is_secure: false
  host: localhost
  port: '{{ osp_fakes3_port }}'

In the tests, a config fixture provides an instance of the global config object, and automatically resets any changes made to the values during the test that requests the fixture:

from osp_pipeline import config as _config

@pytest.yield_fixture
def config():
    """Clear changes to the config dict.
    """
    # Copy settings.
    old = _config.copy()

    yield _config

    # Restore settings.
    _config.clear()
    _config.update(old)

Then, other fixtures can patch values in the config. Eg, to override the text parquet result path, which is necessary because FakeS3 doesn't speak the s3a protocol:

@pytest.yield_fixture
def texts_result(config, tmpdir):
    """Patch the WARC paths result.
    """
    path = str(tmpdir.join('texts.parquet'))
    config['results']['texts'] = path
    yield path

Then, tests can just pull in these fixtures to set up the local result paths, run the job functions (passing in the sc and spark we added before), and everything just works.

There's also a sync_s3_bucket testing helper that syncs a directory of fixture data into FakeS3. Here's the complete test for extract_text, which for now just checks that the ids in the extracted parquet rows match the fixture ids:

@pytest.fixture(autouse=True)
def sync_fixtures(s3, config):
    """Sync WARC fixtures.
    """
    fixture_dir = os.path.join(os.path.dirname(__file__), 'fixtures')

    sync_s3_bucket(fixture_dir, config['buckets']['scraper'])

def test_extract_text(sc, spark, warc_paths_result, texts_result):
    """extract_text should extract metadata + text from WARCs.
    """
    list_warc_paths(sc, spark)

    extract_text(sc, spark)

    texts = spark.read.parquet(texts_result)

    ids = set(texts.rdd.map(lambda t: t.id).collect())

    assert ids == set([
        '00023910-d894-11e6-93af-1234e7cd3e74',
        '0039d9b2-d88d-11e6-93af-1234e7cd3e74',
        '007f5e16-d88c-11e6-93af-1234e7cd3e74',
        '00d34786-d898-11e6-8963-1234cc5b983c',
        '011abf14-d896-11e6-8963-1234cc5b983c',
        '01aa9514-d899-11e6-8963-1234cc5b983c',
        '00b2895e-a654-11e6-a86a-12f2032911c8',
        '00bd9b74-a658-11e6-a86a-12f2032911c8',
        '0115f4c6-a659-11e6-a86a-12f2032911c8',
        '0162cc68-a656-11e6-82db-12f2032911c8',
        '01d33ba8-a654-11e6-82db-12f2032911c8',
        '033e83b8-a653-11e6-82db-12f2032911c8',
    ])

The comment on this PR is awesome documentation - can we get it into the README or somewhere? Otherwise looks good.

Going ahead and merging this now, will flesh out the documentation about writing tests in another PR.


[Satisfactory] Add fluids unpackaging recipes

At early tier 5 you can setup diluted packaged fuel loop, which solver never uses because of missing "unpackage fuel" recipe.
Adding this would help planning factories mid-game.

I don't usually include cycles like this in the solver data, since assuming the amount of fluid remains constant, it just adds more machines so the solver wouldn't ever include the loop. How do you imagine the solver would use the unpackaging recipes? I don't believe it would be reasonable to expect it to package and unpackage every fluid.

Diluted Packaged Fuel Loop uses alt recipe for fuel which creates fuel from water canister and oil residue, its more complicated but optimal solution early on before unlocking blender in late-game.

Diluted Packaged Fuel Loop

Normal recipe (Residual Fuel)

@Pepciorek thanks for the explanation. Seems reasonable given the context, should be easy enough to add them


Speech recognition sometimes not working

Speech recognition sometimes not working because of unknown reasons. It's either working from the beginning to end of Dragonfire or not working at all.
I think there are two possibilities:

[ ] Some how this process hanging even though Dragonfire is closed properly. I predicate this to that; every time after a I reboot my machine, at the first run of Dragonfire, speech recognition works seamlessly.
[ ] It could be an issue related to this package: https://github.com/Uberi/speech_recognition

With the migration from SpeechRecognition package to Kaldi, this issue has been resolved https://github.com/DragonComputer/Dragonfire/commit/a18930a53d6f1fe3aa371f59398f39607623cbd7


Allow where filter to evaluate dot

Currently, where appears to only filter variables but has no way to check the "dot"
{{where (where $crumb "IsHome" false) "Permalink" "ne" .Permalink}}

Would like to be able to do something like this though:
{{where (where $crumb "IsHome" false) "." "ne" .}}

What if there were an .IsCurrent method?
Example: On each blog page I want to display all the other blog pages (exclude the current page).
layouts/blog/single.html
{{ range where (where site.RegularPages "Type" "blog) "Path" "ne" .Path }}  vs.
{{ range where (where site.RegularPages "Type" "blog) "IsCurrent" false }}

It's a little easier to understand, but it doesn't save you any keystrokes.
I don't even know if this would be possible.

What if there were an .IsCurrent method on .Page?

There is already a ... .Page method on Page, so I'm guessing that this would work:
{{where (where $crumb "IsHome" false) "Page" "ne" . }}

That said, that would not work for, say, Site, and I think the original proposal is a good idea.

What if there were an .IsCurrent method on .Page?

There is already a ... .Page method on Page, so I'm guessing that this would work:
{{where (where $crumb "IsHome" false) "Page" "ne" . }}

I have tried evaluating Page and couldn't get it to work.


Question about ArcFaceNet

class BottleNeck_IR_SE(nn.Module):
def init(self, in_channel, out_channel, stride):
super(BottleNeck_IR_SE, self).init()
if in_channel == out_channel:
self.shortcut_layer = nn.MaxPool2d(1, stride)
else:
self.shortcut_layer = nn.Sequential(
nn.Conv2d(in_channel, out_channel, kernel_size=(1, 1), stride=stride, bias=False),
nn.BatchNorm2d(out_channel)
)
I wonder why you use nn.MaxPool2d(1, stride) when in_channel == out_channel? I didn't find it in the raw ResNet and ArcFaceNet. I think there should be none like you write in the cbam.py:
if dim_match:             self.shortcut_layer = None
Wating for your response and thanks a lot!

Yes, you are right about this.
MaxPool2d is a mechiansm that I saw it in somewhere else. Actually,  it has the same result with "if dim_match: self.shortcut_layer = None ", because when you do maxpool with kernel size 1*1, the feature maps remains unchanged.
To prevent an ambiguity about this， I suggest you using the latter implementation.

Got it~Thanks for your reply


fix url hosting url parse

The current logic to fetch the langsmith host url unintentionally turns the default langsmith url of https://api.smith.langchain.com into an invalid url of https:/.smith.langchain.com because it matches on the /api portion.
This PR changes the logic to exempt those cases where the subdomain begins with api so that the correct host url is returned.

cc @hinthornw


fix: Fix translation of application status. feat: Add BorrowerSector. Use gettext to translate enums.

#362

@yolile FYI this PR contains a fix for #357, where the capitalized application statuses in applications/export/{lang} wouldn't be translated. You can merge it already if needed. Otherwise I'll add more commits for #362

Pull Request Test Coverage Report for Build<PHONE_NUMBER>0
Details

30 of 42 (71.43%)  changed or added relevant lines in 2 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.2%) to 84.874%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

app/i18n.py
9
21
42.86%

Files with Coverage Reduction
New Missed Lines
%

app/utils/tables.py
2
31.71%

Totals

Change from base Build<PHONE_NUMBER>2:
-0.2%

Covered Lines:
1818

Relevant Lines:
2142

💛  - Coveralls

Feel free to add more commits

@yolile There are some error messages for you to translate (Or, we can let them continue to be returned in English, and relax the i18n.yml workflow). You can add your commits to finish #362 (or create a new PR).

I get this error when I run pybabel extract -F babel.cfg -o messages.pot .
extracting messages from README.rst (classes="ApplicationStatus,BorrowerDocumentType,BorrowerSize,BorrowerSector")
Traceback (most recent call last):
  File "/home/yolile/development/credere-backend/.ve/bin/pybabel", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/frontend.py", line 999, in main
    return CommandLineInterface().run(sys.argv)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/frontend.py", line 925, in run
    return cmdinst.run()
           ^^^^^^^^^^^^^
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/frontend.py", line 523, in run
    for filename, lineno, message, comments, context in extracted:
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/extract.py", line 217, in extract_from_dir
    yield from check_and_call_extract_file(
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/extract.py", line 281, in check_and_call_extract_file
    for message_tuple in extract_from_file(
                         ^^^^^^^^^^^^^^^^^^
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/extract.py", line 323, in extract_from_file
    return list(extract(method, fileobj, keywords, comment_tags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yolile/development/credere-backend/.ve/lib/python3.11/site-packages/babel/messages/extract.py", line 441, in extract
    raise ValueError(f"Unknown extraction method {method!r}")
ValueError: Unknown extraction method 'enum'

pip install -e . (mentioned earlier under setup)

I'm not sure why "Not authenticated" is not being translated, not sure if we are mocking JWT somewhere

We can create a follow-up issue to translate model names, if that's desired.


fix: Self Trade Protection ser/de

Hi,
A minor one but I think for the v2 websocket api the Self Trade Protection options should be snake case not kebab case.
Copied and pasted from here:
Self trade prevention behavior definition.
cancel_both
If self trade is triggered, both arriving and resting orders will be canceled.
cancel_newest
If self trade is triggered, arriving order will be canceled.
cancel_oldest
If self trade is triggered, resting order will be canceled.
Accepted Values
cancel_both
cancel_newest
cancel_oldest
Default Value
cancel_newest

Full disclosure: I have not tested this yet

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.83%. Comparing base (c462d73) to head (f399deb).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master       #3      +/-   ##
==========================================
- Coverage   90.35%   89.83%   -0.53%
==========================================
  Files          27       27
  Lines        1338     1338
==========================================
- Hits         1209     1202       -7
- Misses        129      136       +7

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks! Let me confirm tomorrow, this looks good to merge assuming behavior matches the docs. Thanks!

Sorry for the delay, I was able to confirm that "snake_case" is correct, and added the example code I had for trading on the V2 websockets that I used for further testing.


#12649 reconnect is connection is lost when we read procedure body

This should be done for all databases, not just postgresql.
Use tryExecuteRecover


Node-red docker synology crash after delay module

Current Behavior
upgrading to 4.0

after cast voice, sound and a delay of 5s nodered crashed and restart
Expected Behavior
No crash and continuous the following flows
Steps To Reproduce
After i pushed the inject button

_ndrd_logs_24062024.txt
Example flow
paste your flow here

Environment

Node-RED version: 4.0
Node.js version:
npm version:
Platform/OS: Synology 7.2
Browser: Brave

From your logs, the Node-RED crash is a bug in the node-red-contrib-cast node;
24 Jun 10:31:16 - [error] TypeError: node.error is not a function
    at errorHandler (/data/node_modules/node-red-contrib-cast/cast-to-client.js:52:10)
    at getSpeechUrl (/data/node_modules/node-red-contrib-cast/cast-to-client.js:192:9)
    at Timeout._onTimeout (/data/node_modules/node-red-contrib-cast/cast-to-client.js:836:33)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7)

Which has already been reported here: https://github.com/Hypnos3/node-red-contrib-cast/issues/416

Ok, i understands that, but why node-red crashed?

Because that is correct thing to do with a unhandled error, we do not know what state anything has been left in if the node does not handle the error.

I have to disable it?

The issue Nick linked to was raised back in April and it's not even been acknowledged, that and given that the node in question has not been updated in 4 years I think it is unlikely it will get fixed.
You may need to either sick with NR 3.1.x for now or look for a different cast node.

Thanks a lot. 🙏

Closing as this appears to be a node-specific issue


Question: how to make initial region to fit minimum dimension of map container?

at the moment a map fits to maximum dimension of its container and rest of the map is cut off. it looks like this (for ios platform)

is it possible to fit the map to minimum dimension of the container? like this

Not sure it's possible.
Closing as it's an old question


ValidDatabaseOwner check not working correctly

The dbacheck ValidDatabaseOwner should perform the validation based on the sid of the account and not the name. One of the best practice is to rename the sa user (hence your dbacheck "SaRenamed") and if that has been done then we will have failed status regardless if the owner is actually the original sa account.
I think it should validate if the database owner has the sid 0x01 which I believe is always the sa account.

Which version of dbachecks are you using ?
The latest version has been altered to accept a group of possible job owner logins by name as the valid or invalid job owner will not necessarily be the sa account or the one with a sid of 0x01

I believe I upgraded yesterday or day before and the version is 1.1.109.
PS C:\Windows\Temp\dbachecks> Get-Module Dbachecks -ListAvailable
Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    Name                                ExportedCommands

Script     1.1.109    dbachecks                           {Get-DbcConfig, Get-DbcConfigValue, Set-DbcConfig, Invoke-DbcCheck...}
Script     1.1.107    dbachecks                           {Get-DbcConfig, Get-DbcConfigValue, Set-DbcConfig, Invoke-DbcCheck...}

I just upgraded to 1.1.110 and it still reports invalid database owner when the owner is renamed sa user.

what is returned from  Get-DbcConfigValue agent.validjobowner.name ?

It is working as expected for me here
I wonder if you are not setting the expected name using
Set-DbcConfig -Name policy.validdbowner.name -Value sarenamed

and following a change of owner to the renamed sa account

You are absolutely right. I did not configure that, I just assumed it worked in similar way as the dbatools Test-DbaDatabaseOwner which I think recognizes if the sa has been renamed.
But yes, when I have set it as you described above it works fine for me.
Thanks for the lesson and sorry for the inconvenience.

Never a problem :-)


Hide certain apps or two folders (tabs)

Ability to hide apps that aren't needed. Though I understand it's the also a home launcher, so have maybe 2 tabs one for maybe put the games and then the rest?

Or just that it tracks app launches and sorts them by that so the most used are at the top

Hello again my friend. Thanks for your suggestions. You can hide apps  in version 65(1.1.39) now.


Support for Redux 3.0.0

Could you please upgrade the package,json to support redux 3.0.0 when you get a chance?

Already support redux 3.0 now:
https://github.com/gaearon/redux-devtools/blob/master/package.json#L47

It should work fine.


Demo does not work on IE11

Issue:
When opening https://grapesjs.com/demo-mjml.html in IE11 it should load. Currently it only shows a blank page.
Console output:

duplicate of https://github.com/artf/grapesjs-mjml/issues/77


fix/kskeleton-publish

Summary
Changes made:
*fix publish
PR Checklist

[x] Does not introduce dependencies
[x] Functional: all changes do not break existing APIs and if so, bump major version.
[x] Tests pass: check the output of yarn test packages/
[x] Naming: the files and the method and prop variables use the same naming conventions as other Kongponents
[x] Framework style: abides by the essential rules in Vue's style guide
[x] Cleanliness: does not have formatting issues, unused code (e.g., console.logs), or leftover comments
[x] Docs: includes a technically accurate README, uses JSDOC where appropriate

Going on holiday so wanted to drop this for visibility. kong-loader.gif is inlined so it does not need to be included re:
https://github.com/Kong/kongponents/pull/208


Add text that shows if there are no results

This adds a piece of text for if there are no search results when looking for applications.
Trello Card
Screenshot

We need to ask Wes about this. I don't think the content is correct. Fine for now though.

It was actually Wes who came up with the content! 😅


Minification breaks regenerator-runtime package and redux-saga is not usable

Do you want to request a feature or report a bug?
Bug
What is the current behavior?
The<EMAIL_ADDRESS>is not minified properly. Throwing the following error:
ExceptionsManager.js:73 uncaught at n at n
 ReferenceError: record is not defined
    at Generator._invoke (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:221:931)
    at Generator.t.(anonymous function) [as next] (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:221:1352)
    at t (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:869:536)
    at o (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:869:7374)
    at t.runSaga (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:867:806)
    at o.default (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:865:440)
    at http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:514:173
    at t (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:2:1273)
    at http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:2:614
    at n (http://localhost:8081/index.android.bundle?platform=android&dev=true&minify=true:2:622)

This is the original code:
var record = tryCatch(innerFn, self, context);
if (record.type === "normal") {

This is the minified:
if (o = b, "normal" === (record = n(t, r, e)).type) {

The regenerator-runtime/runtime.js has this header with strict mode enabled (this cause the error):
!(function(global) {
  "use strict";

With this error the redux-saga package is unusable and the whole app is broken.
If the current behavior is a bug, please provide the steps to reproduce and a minimal repository on GitHub that we can yarn install and yarn test.
What is the expected behavior?
The record variable is well defined. Work fine on:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Please provide your exact metro-bundler configuration and mention your metro-bundler, node, yarn/npm version and operating system.
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>OS: Ubuntu MATE 16.04.3 LTS

@navarroaxel I have just released<EMAIL_ADDRESS>with an upgraded uglify version, which hopefully will fix this.
You can force yarn to use the new metro version by forcing a resolution in your package.json:
"resolutions": {
  "metro-bundler": "0.20.1"
}

Please reopen the issue if this does not fix it


Bring back Docker images from Docker Hub

As suggested in #245, after that PR was merged we'll need to get back to the images from Docker Hub. This is what this PR does ; )

Testing this PR!

Clone the repository:

$ git clone git@github.com:datasciencebr/jarbas.git

Open the repo folder:

$ cd jarbas

Checkout to @cuducos branch:

$ git checkout -b cuducos-docker-images origin/cuducos-docker-images

Update the branch:

$ git merge master

Copy the .env file:

$ cp contrib/.env.sample env

Build and start services:

$ docker-compose up -d

Create the database and apply the migrations:

$ docker-compose run --rm django python manage.py migrate
$ docker-compose run --rm django python manage.py searchvector

Seeding it with sample data:

$ docker-compose run --rm django python manage.py reimbursements /mnt/data/reimbursements_sample.xz
$ docker-compose run --rm django python manage.py companies /mnt/data/companies_sample.xz
$ docker-compose run --rm django python manage.py suspicions /mnt/data/suspicions_sample.xz
$ docker-compose run --rm django python manage.py tweets

Open http://localhost:8000/ and http://localhost:8000/dashboard and do tests with functionalities, works! :)


Add your name to the contributor list

@ghenry @rlsutton1 I added a list of contributors to README.md and you can add yourself to the contributor list via a PR.

@cloudwebrtc @ghenry  done and thank you.


Threads display blank in Firefox

Threads display as blank pages for me in Firefox.
Example: https://bobbin.herokuapp.com/thread/1035584419929702402
The same link displays fine in Chrome.
Firefox 67.0b18 (64-bit) running on Windows 10
Extensions:
LastPass <IP_ADDRESS>
uBlock Origin 1.18.16
Plugins:
OpenH264 Video Codec provided by Cisco Systems, Inc.  1.8.1  (always activate)
Widevine Content Decryption Module provided by Google. Inc.  4.10.1146.0  (always activate)
Shockwave Flash  <IP_ADDRESS>  (ask to activate)
I also tried to disable all extensions and plugins, and even tried in a Private window (like incognito mode in Chrome) and the result was the same.
I have seen uBlock cause problems like this on some pages, because it had blocked a style sheet from loading or something like that, but every time, disabling it would fix it. In this case, that did not work.
It might be caused by Firefox, I heard they had different policies than Chrome related to loading content from different domains or something like that. I am not a web developer, though, so I would not know the details, nor a way to work around the issue.
Here are screenshots:
Firefox:

Chrome:

Thanks in advance,
J-S

The provided link is just an example, I tried bobbin on a few other threads first -- just figured I'd provide an example without linking to other people's content.


Allow further options in dind

The binderhub helm chart (when dind enabled) creates pods with fixed arguments:
        args:
          - dockerd
          - --storage-driver={{ .Values.dind.storageDriver }}
          - -H unix://{{ .Values.dind.hostSocketDir }}/docker.sock

For some deployments it may be necessary to add/change some of the options. Specifically in my case I need to specify a --mtu option for setting a smaller than the default docker MTU. It would be good to be able to specify extra options there.

Sounds like a sensible and useful extension. Is there a common pattern for doing this in the helm chart world? Something like extraEnv which is a frequently used key for extending the environment variables.
Maybe extraArgs?
Do you want to create a PR to implement this?

Might I suggest we finish/merge #838 before adding more dind features? 😉
+1 on adding something like extraArgs though.

I can create a PR for this with the extraArgs

extraArgs is just a guess so a short amount of research what other helm charts in for example https://github.com/helm/charts/ do would be good before spending too much time implementing something.

PR #916 should fix this issue.
I have found two main ways of using extraArgs in the stable helm charts:

as a list, so they basically get copied to the arguments of the pod, or
as a dict, where you create arguments in the form of --key=value.

I opted for the first one as I don't have quite clear if there will be any args that do not fit the --key=value schema. Also this option seems to have problems with values == false


Anti-DDoS Mechanism in openssh-8.5p1

Once, I also submitted a ddos question to the openssh community: https://bugzilla.mindrot.org/show_bug.cgi?id=3211
They added the configuration items PerSourceMaxStartups and PerSourceNetBlockSize to openssh-8.5p1. I think that the two parameters can be properly configured to prevent "dheat" from attacking OpenSSH.

Thank for the info. If I understand it well these configuration items should be added to the documentation of D(HE)ater, but I have no other thing to do.