HDFS doesn't complete Deployment on Testing Branch

The HDFS framework doesn't complete deployment according to Marathon on testing/continuous clusters.
The Framework is healthy from it's perspective.  It launches all needed Tasks.  It responds to all health check requests appropriately.  It responds to client requests as designed etc.
This defect makes it impossible for us to programmatically determine the health of an HDFS deployment.
https://mesosphere.atlassian.net/browse/MARATHON-317

@gabrielhartmann  I just created a DCOS cluster and started HDFS with dcos package install hdfs. I saw this in the error log which doesn't look healthy at all:
15/09/22 10:56:51 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /hdfs-framework-healthcheck/test1.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1504)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3065)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:615)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:188)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:476)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

	at org.apache.hadoop.ipc.Client.call(Client.java:1411)
	at org.apache.hadoop.ipc.Client.call(Client.java:1364)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
	at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:391)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1473)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1290)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:536)
put: File /hdfs-framework-healthcheck/test1.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
I0922 10:56:51.865679  1272 main.cpp:100] Ignoring failure as health check still in grace period

Furthermore, in the slave logs I found the following:
I0922 11:01:48.026366  1099 slave.cpp:2696] Handling status update TASK_RUNNING (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000 from executor(1)@<IP_ADDRESS>:35511
I0922 11:01:48.026757  1097 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000
I0922 11:01:48.026805  1097 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000
I0922 11:01:48.028033  1099 slave.cpp:2975] Forwarding the update TASK_RUNNING (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000 to master@<IP_ADDRESS>:5050
I0922 11:01:48.028142  1099 slave.cpp:2905] Sending acknowledgement for status update TASK_RUNNING (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000 to executor(1)@<IP_ADDRESS>:35511
I0922 11:01:48.031659  1095 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 of framework 20150922-104356-3892576266-5050-1219-0000
I0922 11:01:48.031724  1095 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 59f65583-165d-43c2-bd81-32b519898ec9) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000
I0922 11:01:48.125778  1095 slave.cpp:2696] Handling status update TASK_KILLED (UUID: d1c354e1-5826-4dbd-971b-58bac23ad618) for task hdfs.12d7cd56-6118-11e5-95c5-421f79ce4c65 in health state unhealthy of framework 20150922-104356-3892576266-5050-1219-0000 from executor(1)@<IP_ADDRESS>:35511

Therefore Mesos considers the task as unhealthy. If Mesos considers the task unhealthy, Marathon will as well. So please investigate why Mesos considers the task unhealthy.
If that's not the case in your setup, please provide all relevant logs to debug the problem:

Marathon
Mesos Master
The relevant slave
HDFS stderr/stdout
state.json

On a side note, I also so this which should probably optimized:
15/09/22 10:58:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

All those error messages you provide are expected and have always occurred, except for those in the Slave logs.  Furthermore, HDFS deployed flawlessly on Stable clusters, and is only now encountering the situation where it is constantly listed as Deploying by Marathon.  Prior to the the new 0.10.1 release HDFS worked well.  Post that release it doesn't work well.  This doesn't have the hallmarks of an HDFS problem.  Let me reiterate, that HDFS is working properly and I have observed no failed tasks.
I have provided a repro so I won't be spending a great deal of time collecting a lot of logs.  Deploying HDFS on any cluster is sufficient now (as stable has been upgraded to 0.10.1).  I'm deploying a repro now and will leave the cluster up for 24 hours and leave the address here.

We had a face-to-face call.  I'll re-file an issue when I get a repro today with the more detailed information you need.

Here is a live cluster with the repro.  I was wrong previously.  The HDFS scheduler is not continuously passing it's healthcheck, but the last thing it did was to pass it's healthcheck.  We can see this in the scheduler (hdfs task) stderr log.  Marathon notes that HDFS is still deploying.
In the past I have tried stopping the deployment and then reinstalling (restarting the scheduler essentially) and this sometimes solves the issue and sometimes doesn't.  I can also just restart the scheduler and this never resolves the problem, but the Scheduler will begin passing the healtcheck continuously.
The expected output for the healthcheck is:
15/09/22 17:06:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/22 17:06:51 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/09/22 17:06:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/22 17:06:55 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/09/22 17:06:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/22 17:07:00 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/09/22 17:07:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/22 17:07:04 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/09/22 17:07:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
W0922 17:07:07.336280  1341 main.cpp:375] Health check failed Command check failed with reason: status still pending after timeout 1mins
15/09/22 17:07:08 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/09/22 17:07:08 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted hdfs://hdfs/hdfs-framework-healthcheck

Sorry it's so ugly.  We'll be working on improving logging.  The cluster is set to expire in 5 days, which can be extended.  Please let me know if you need any additional information.  Please note that HDFS is operational now.  It will successfully respond to client queries, and has passed it's healthcheck, although this is no longer being run.

Just quick feedback @gabrielhartmann, I will continue my investigation. I was not able to retrieve the marathon logs or the slave logs of the relevant time interval. Either because of my lack of journalctl knowledge or because those logs have been rotated away. I am now trying to reproduce the problem by restarting the hdfs service. "Unfortunately", after restarting the service so far always starts up correctly. (I do a "stop" on the deployment in the UI and then "restart" the app in the UI)
Generally, you are seriously helping me a lot if you save the logs when the problem appears.

It would be great if you had saved the logs as described above when you reproduce the bug. Please let's do that in the future.
I could not reproduce the problem by restarting the hdfs service for several times. I did most of the restarts while having the spark app suspended (to keep the logs focussed) but just to check that the problem doesn't result from interference with spark, I started spark again and repeated some restarts of the hdfs service after that. That allowed me to reproduce the error, finally.
I will continue my investigation.

Unfortunately, hdfs just took a while until becoming healthy. When Mesos reported healthy status, the old task was stopped and the deployment stopped (excerpt from the Marathon log):
Sep 23 09:25:51 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [2015-09-23 09:25:51,179] INFO Received status update for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4feea61a:100)
Sep 23 09:26:50 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [2015-09-23 09:26:50,741] INFO Received status update for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4feea61a:100)
Sep 23 09:26:50 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [2015-09-23 09:26:50,745] INFO Received status update for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4feea61a:100)
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [2015-09-23 09:31:14,277] INFO Received status update for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4feea61a:100)
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [2015-09-23 09:31:14,278] INFO Received status for [hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5] with version [2015-09-23T09:25:35.547Z] and healthy [true] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$60c84ed1:165)
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [2015-09-23 09:31:14,279] INFO Forwarding health result [Healthy(hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5,2015-09-23T09:25:35.547Z,2015-09-23T09:31:14.278Z)] to health check actor [Actor[akka://marathon/user/$ab#874294670]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$60c84ed1:186)
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [INFO] [09/23/2015 09:31:14.279] [marathon-akka.actor.default-dispatcher-16] [akka://marathon/user/$ab] Received health result: [Healthy(hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5,2015-09-23T09:25:35.547Z,2015-09-23T09:31:14.278Z)]
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [INFO] [09/23/2015 09:31:14.279] [marathon-akka.actor.default-dispatcher-12] [akka://marathon/user/MarathonScheduler/$a/DeploymentManager/8823c0e4-3040-4bcb-b7fb-86defbdcc0d0/$a] Killing old task hdfs.ddadbd6c-61d4-11e5-84a9-1a46da8345c5 because hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 became reachable
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [INFO] [09/23/2015 09:31:14.279] [marathon-akka.actor.default-dispatcher-12] [akka://marathon/user/MarathonScheduler/$a/DeploymentManager/8823c0e4-3040-4bcb-b7fb-86defbdcc0d0/$a] All tasks for /hdfs are healthy
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [INFO] [09/23/2015 09:31:14.279] [marathon-akka.actor.default-dispatcher-19] [akka://marathon/user/MarathonScheduler/$a] Deployment of / successful
Sep 23 09:31:14 ip-10-0-6-210.us-west-2.compute.internal java[1352]: [INFO] [09/23/2015 09:31:14.279] [marathon-akka.actor.default-dispatcher-12] [akka://marathon/user/MarathonScheduler/$a/DeploymentManager] Removing 8823c0e4-3040-4bcb-b7fb-86defbdcc0d0 from list of running deployments

And the matching excerpt from the slave.log:
I0923 09:25:38.948685  1086 slave.cpp:1249] Got assigned task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 for framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:25:38.949239  1086 slave.cpp:1365] Launching task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 for framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:25:38.956179  1086 slave.cpp:4799] Launching executor hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/var/lib/mesos/slave/slaves/20150922-165631-3523608586-5050-1307-S5/frameworks/20150922-165631-3523608586-5050-1307-0000/executors/hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5/runs/3ea99853-19a3-4fea-ac96-4c335f5a70e1'
I0923 09:25:38.956701  1086 slave.cpp:1583] Queuing task 'hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5' for executor hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework '20150922-165631-3523608586-5050-1307-0000
I0923 09:25:38.957082  1086 containerizer.cpp:633] Starting container '3ea99853-19a3-4fea-ac96-4c335f5a70e1' for executor 'hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5' of framework '20150922-165631-3523608586-5050-1307-0000'
I0923 09:25:38.966505  1088 containerizer.cpp:855] Checkpointing executor's forked pid 2519 to '/var/lib/mesos/slave/meta/slaves/20150922-165631-3523608586-5050-1307-S5/frameworks/20150922-165631-3523608586-5050-1307-0000/executors/hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5/runs/3ea99853-19a3-4fea-ac96-4c335f5a70e1/pids/forked.pid'
I0923 09:25:51.171690  1088 slave.cpp:2358] Got registration for executor 'hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5' of framework 20150922-165631-3523608586-5050-1307-0000 from executor(1)@<IP_ADDRESS>:50363
I0923 09:25:51.173938  1088 slave.cpp:1739] Sending queued task 'hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5' to executor 'hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5' of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:25:51.176723  1086 slave.cpp:2696] Handling status update TASK_RUNNING (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000 from executor(1)@<IP_ADDRESS>:50363
I0923 09:25:51.176925  1086 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:25:51.177189  1086 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:25:51.180034  1085 slave.cpp:2975] Forwarding the update TASK_RUNNING (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000 to master@<IP_ADDRESS>:5050
I0923 09:25:51.180198  1085 slave.cpp:2905] Sending acknowledgement for status update TASK_RUNNING (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000 to executor(1)@<IP_ADDRESS>:50363
I0923 09:25:51.183645  1086 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:25:51.183753  1086 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: 6a0b387a-3ab4-4e7e-8471-7820d257d63e) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:31:14.271569  1085 slave.cpp:2696] Handling status update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000 from executor(1)@<IP_ADDRESS>:50363
I0923 09:31:14.271834  1085 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:31:14.274528  1085 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:31:14.277241  1089 slave.cpp:2975] Forwarding the update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000 to master@<IP_ADDRESS>:5050
I0923 09:31:14.277700  1089 slave.cpp:2905] Sending acknowledgement for status update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000 to executor(1)@<IP_ADDRESS>:50363
I0923 09:31:14.283279  1085 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 of framework 20150922-165631-3523608586-5050-1307-0000
I0923 09:31:14.283396  1085 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000

If you manage to reproduce the problem, please save the logs. I'll have to close this until then.

Here are the logs for the Agent hosting the HDFS scheduler.
Here are the stderr logs for from the scheduler itself.
Note that the stack traces etc. at the end of the scheduler logs are the health-check failing because HDFS hadn't recovered yet.  That's just spew from running hadoop commands.  The health check is this.
The cluster should be stuck in this state forever.  I've extended its lifespan for 5 days again.  In this case we're not stuck in a "Deploying" state.  We're just stuck in an indeterminate health state.  We never converge to Healthy or Unhealthy.  Note that the whole healthcheck failure took 45 seconds.  Our intervalSeconds is 60, and our timeoutSeconds is 60.  gracePeriodSeconds is 300 and maxConsecutiveFailures is 3.
I don't know how to get Marathon logs.  Let me know if they're lost by the time you get around to this.

Your theory is that mesos detects that the task is healthy and Marathon does not react appropriately, correct?
Then there should be something like
I0923 09:31:14.277241  1089 slave.cpp:2975] Forwarding the update TASK_RUNNING (UUID: a6c8b038-a677-4c3d-b2cd-5229e86931ab) for task hdfs.0da34453-61d5-11e5-84a9-1a46da8345c5 in health state healthy of framework 20150922-165631-3523608586-5050-1307-0000 to master@<IP_ADDRESS>:5050

in the slave log. But I am not seeing that. I get the only the following:
$ grep "Forwarding" slave.log | grep "hdfs."

Sep 23 22:09:47 ip-10-0-1-109.us-west-2.compute.internal mesos-slave[813]: I0923 22:09:47.318738   825 slave.cpp:2975] Forwarding the update TASK_RUNNING (UUID: 1135b032-2bf0-4055-941e-127cfbbb8ef7) for task hdfs.c5013396-623f-11e5-baf1-920390a3dcf7 of framework 20150923-213844-1862729738-5050-1392-0000 to master@<IP_ADDRESS>:5050

So Marathon gets the initial TASK_RUNNING notification but nothing else. I guess that mesos should either follow up with a TASK_RUNNING with healthy state or an unhealthy state.
So if I am not missing something and that is entirely possible, it looks like a Mesos bug. Or maybe the log is incomplete?

Seems we run into two issues here:

Why does the health-check run into a timeout
-> the current health-check script seems rather slow in execution - I timed a couple of attempts and it always took more than 24 seconds to run it entirely. As such the timeout of 60 seconds appears to be too optimistic for a machine with a higher workload. Seems you got at least two options here; A. adapt the health-check, maybe by putting it entirely into a single JAVA code invocation. B. raise the timeout to e.g. 2 or more minutes.

export PATH=$MESOS_DIRECTORY/hdfs-mesos-0.1.4/bin:$PATH && export JAVA_HOME=$MESOS_DIRECTORY/jre1.7.0_76 && hadoop fs -ls hdfs://hdfs/ && rm -rf hdfs-framework-healthcheck && hadoop fs -rm -r -f hdfs://hdfs/hdfs-framework-healthcheck && hadoop fs -mkdir hdfs://hdfs/hdfs-framework-healthcheck && mkdir hdfs-framework-healthcheck && echo "this is a test" > hdfs-framework-healthcheck/test1.txt && hadoop fs -put hdfs-framework-healthcheck/test1.txt hdfs://hdfs/hdfs-framework-healthcheck && hadoop fs -get hdfs://hdfs/hdfs-framework-healthcheck/test1.txt hdfs-framework-healthcheck/test2.txt && rm -rf hdfs-framework-healthcheck && hadoop fs -rm -r -f hdfs://hdfs/hdfs-framework-healthcheck

Why does the health-check not send any updates after it ran into a timeout
-> covered by https://issues.apache.org/jira/browse/MESOS-3479

@kolloch:  Thanks for having a look again Peter.  I'll be moving this investigation to the Mesos team.  Hopefully this bug doesn't live in some no-man's land between Marathon and Mesos.  I have a feeling there are at least two bugs here, or that the same bug manifests in different ways.
@tillt:  I can bump the timeout again, but as you can see from the timestamps in this case, execution fell well within the 60 second period.  So I'm not sure the referenced Mesos bug has anything to do with this particular failure.  Although it might be a different manifestation.  I'm not worried about production loaded systems at the moment.  The tests I'm running fall very often and they're run on totally pristine unloaded systems.

Apologies for opening this again.  I had a pretty close look from someone from the Core team.  It looks to us like this may be a Marathon issue.  Here is what we found:

The HDFS Scheduler comes up and immediately launches a health-check at about 19:02:52
[This fails pretty much immediately here](I0930 19:02:52.870311  1461 main.cpp:100] Ignoring failure as health check still in grace period)
It then fails 2 more times here and here separated at approximately 60s intervals as specified by intervalSeconds in the marathon.json.
It is attempted again here again pretty much exactly 60s later.  This time it's going to fail with stack traces spewing into stderr.  It has completed failed by here.  Note that the duration of the attempt / failure is ~40s here.  Well within the 60s timeout.
The health check is attempted one more time and succeeds here.

No more health check attempts are made, and Marathon's status regarding HDFS is that there 1 running task, with 0 healthy, 0 staging, and 0 unhealthy.  The above logs seem to indicate proper operation of Mesos.  There are two interesting observations I'd like to make:

When we see this failure it always follows the pattern above with the stack spew from the health-check followed by a single successful iteration of the health-check.
The final iteration of the health-check is near the end of the gracePeriod.  The gracePeriod is 300s.  We started at about 19:02:32.  We start the final successful health-check at 19:06:43 and complete it at 19:07:04

Here's a link to the huge Master logs in their entirety.
In an excerpt of that log here, we see the Marathon health checker getting started, but when looking at more logs with this ID, I never see any further mention of getting health check results from HDFS.  I see health check results from another framework,  ArangoDB,  but never HDFS.
Is it possible, the akka health-check actor is having problems?  I think the above investigation pretty much proves this isn't HDFS's problem.  I'm not sure if it's Mesos's problem.  I'd like not to bounce back and forth between Mesos and Marathon teams anymore.  If there's something I'm doing that's illegal in my health-check, let me know.  It's a simble cmd line command.  It produces an exit code.  It seems like the fault must lie elsewhere.  This problem has been haunting the HDFS framework for a long time now.  I'd like to see if somehow a resolution can be found.

Nevermind.  There are overlapping healthchecks going on.  It looks like the one spewing a stack trace did timeout.  This happened while the last one was running and then succeeded.  My bad.  Closed forever.

Hey @gabrielhartmann, even though it eventually turned out to be something else, thanks for excellent info provided with the last comment!
Just some comments to debug something like this in the future and to decide which component (Mesos, Marathon) is "responsible". The COMMAND health checks (as opposed to HTTP health checks) are executed by Mesos. The "actor" in Marathon doesn't really make much in this case. Marathon is dependent on getting the health info from Mesos via a task status update.
If you grep the master log with grep "hdfs\\." logs.txt | grep TASK, you will see:
Sep 30 19:02:48 ip-10-0-7-233.us-west-2.compute.internal mesos-master[1242]: I0930 19:02:48.484849  1251 master.cpp:4069] Status update TASK_RUNNING (UUID: 80d4134b-de09-4bc7-9631-c4bac2cb76fa) for task hdfs.ce956568-67a5-11e5-baa4-7643f8838a73 of framework 20150930-184507-3909550090-5050-1242-0000 from slave 20150930-184507-3909550090-5050-1242-S5 at slave(1)@<IP_ADDRESS>:5051 (ip-10-0-0-138.us-west-2.compute.internal)
Sep 30 19:02:48 ip-10-0-7-233.us-west-2.compute.internal mesos-master[1242]: I0930 19:02:48.484938  1251 master.cpp:4108] Forwarding status update TASK_RUNNING (UUID: 80d4134b-de09-4bc7-9631-c4bac2cb76fa) for task hdfs.ce956568-67a5-11e5-baa4-7643f8838a73 of framework 20150930-184507-3909550090-5050-1242-0000
Sep 30 19:02:48 ip-10-0-7-233.us-west-2.compute.internal mesos-master[1242]: I0930 19:02:48.485108  1251 master.cpp:5576] Updating the latest state of task hdfs.ce956568-67a5-11e5-baa4-7643f8838a73 of framework 20150930-184507-3909550090-5050-1242-0000 to TASK_RUNNING
Sep 30 19:02:48 ip-10-0-7-233.us-west-2.compute.internal java[1282]: [2015-09-30 19:02:48,485] INFO Received status update for task hdfs.ce956568-67a5-11e5-baa4-7643f8838a73: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4feea61a:100)

And then many times reconciliation status updates:
Sep 30 19:05:27 ip-10-0-7-233.us-west-2.compute.internal java[1282]: [2015-09-30 19:05:27,202] INFO Received status update for task hdfs.ce956568-67a5-11e5-baa4-7643f8838a73: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4feea61a:100)

None of these contain any health status. If Mesos doesn't pass health status information to Marathon, Marathon cannot act on them.


Regarding pitch_embedding and energy_embedding

HI,we are getting errors when pitch_embedding and energy_embedding set true during txt2vec  training

Hi,
When I try to enable the use_energy_embed: True in Config/txt2vec/MM6_IPA/model.yaml, i'm getting the following error (couldn't find the stats.json file in this repo as well.
Traceback (most recent call last):
File "/home/rafi/miniconda3/envs/ZMM-TTS/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "ZMM-TTS/txt2vec/train.py", line 56, in train
model, optimizer = get_model(args, configs, device, train=True)
File "ZMM-TTS/txt2vec/utils/model.py", line 14, in get_model
model = CompTransTTS(preprocess_config, model_config, train_config).to(device)
File "ZMM-TTS/txt2vec/model/CompTransTTS.py", line 57, in init
self.variance_adaptor = VarianceAdaptor(preprocess_config, model_config, train_config, 256)
File "ZMM-TTS/txt2vec/model/modules.py", line 795, in init
with open(
FileNotFoundError: [Errno 2] No such file or directory: 'Dataset/preprocessed_data/MM6/stats.json'
Note: I'm using the same configuration from this repo except enabling energy embedding.
Please let me know if any have solution for this. Thanks in advance.

@su4kk @pavanhitloop
Sorry for the late reply. It is feasible to use pitch and energy embeddings. You may refer to the repository https://github.com/keonlee9420/Comprehensive-Transformer-TTS for further details. It is important to note that you need to modify our preprocessing code to extract the pitch and energy features.

@gongchenghhu, thank you for your response. I'll try and update here if any issues.


Working Dockerfile for mountebank-grpc

Hi,
I was hoping someone could help me and provide me a working docker file for mountebank-grpc.
I've been poking at it for a couple of days now and always end up with error creating imposter: [{"code":"bad data","message":"the grpc protocol is not yet supported"}]
Current  attempt:
FROM node:8.2

EXPOSE 2525
#CMD ["mb", "--configfile", "/tmp/perf_sven_grpc_poc/grpc_impostor.ejs", "--protofile","/tmp/perf_sven_grpc_poc/protocols.json","--allowInjection",  "--logfile", "/var/log/mb_gprc.log", "--port", "6105" ]

ENV MOUNTEBANK_VERSION=1.15.0

RUN cd /usr/lib/node_modules; npm install -g mountebank@${MOUNTEBANK_VERSION} --production \
 && npm cache clean --force 2>/dev/null \
 && rm -rf /tmp/npm* \
 && npm install xml2js \
 && npm install dateformat \
 && npm install node-uuid \
 && npm install url \
 && npm install sax \
 && npm install xmlbuilder

RUN mkdir /tmp/perf_sven_grpc_poc
RUN cd /tmp/perf_sven_grpc_poc
RUN git clone https://github.com/cbrz/mountebank-grpc
RUN npm install

COPY ./protocols.json /tmp/perf_sven_grpc_poc/protocols.json
COPY ./01_grpc_unary_unary.ejs /tmp/perf_sven_grpc_poc/01_grpc_unary_unary.ejs
COPY ./grpc_impostor.ejs /tmp/perf_sven_grpc_poc/grpc_impostor.ejs
COPY ./start_mb_grpc.sh /tmp/perf_sven_grpc_poc/start_mb_grpc.sh

Hi @bobruub ,
I haven't had a lot of time to look into things (been busy with other projects)... There's been changes to grpc where some of the mock here was probably made obsolete. This is applicable to all custom protocols.
For mountebank-grpc it should go somehting like this (i did this on a win10 host, so adjust accordingly):
Directory Structure
C:\mydir\mountebank-grpc
C:\mydir\mountebank-grpc\mountebank-grpc    # cloned repo
C:\mydir\mountebank-grpc\Dockerfile                 # Dockerfile below
C:\mydir\mountebank-grpc\protocols.json           # protocols.json below

Dockerfile
FROM debian:buster

RUN set -ex; \
    apt-get update; \
    apt-get install -y --no-install-recommends ca-certificates dirmngr gnupg wget;\
    apt-get install -y --no-install-recommends curl; \
    curl -fsSL https://deb.nodesource.com/setup_12.x | bash -; \
    apt-get install -y nodejs; \
    \
    # add whatever you want here
    rm -rf /var/lib/apt/lists/*; \
    apt-get clean;

COPY . /etc/mountebank
WORKDIR /etc/mountebank

RUN set -ex; \
    # install mountebank global
    npm install -g mountebank; \
    # install mountebank-grpc local
    npm install ./mountebank-grpc;

ENTRYPOINT ["mb", "start", "--protofile", "protocols.json"]

protocols.json
{
    "grpc": {
        "createCommand": "node /etc/mountebank/mountebank-grpc/src/index.js"
    }
}

Build the image and run
docker build --tag=mb-grpc:latest
docker run -p 2525:2525 -p 4545:4545 --rm -it mb-grpc

After that I was able to use postman/curl to load the imposter from the README.md
PS C:\mydir\mountebank-grpc> docker run -p 2525:2525 --rm -it mb-grpc
info: [mb:2525] Loaded custom protocol grpc
info: [mb:2525] mountebank v2.4.0 now taking orders - point your browser to http://localhost:2525/ for help
info: [mb:2525] POST /imposters
info: [grpc:4545] Open for business...
info: [grpc:4545] [2021-08-21T21:14:08.925Z] server started on port '4545'

I didn't test the overall grpc functionality since i don't have the tools on this machine.
This should at least get you started. Hope it helps!


Definition provider not found error

CocInfo output
## versions

vim version: NVIM v0.4.3
node version: v14.4.0
coc.nvim version: 0.0.78-0c2ff5e505
term: alacritty
platform: linux

## Output channel: Julia Language Server Trace

## Output channel: languageserver.julia

ERROR:
MethodError: Cannot `convert` an object of type String to an object of type Function
Closest candidates are:
  convert(::Type{T}, !Matched::T) where T at essentials.jl:171

Stacktrace:

 [1]
convert
(
::
Type{Union
{
Nothing, Function
}}, ::String) at ./some.jl:34
 [2] LanguageServer.JSONRPCEndpoints.JSONRPCEndpoint(::Base.PipeEndpoint, ::Base.PipeEndpoint, ::String) at /home/direwolf/.julia/packages/LanguageServer/mpNvN/src/jsonrpcendpoint.jl:62
 [3] LanguageServerInstance(::Base.PipeEndpoint, ::Base.PipeEndpoint,
::Bool, ::String, ::String, ::Nothing) at /home/direwolf/.julia/packages/LanguageServer/mpNvN/src/languageserverinstance.jl:62
 (repeats
2 times)

 [4] top-level scope at none:7
[Info  - 1:04:35 AM] Connection to server got closed. Server will restart.
[Error  - 1:04:35 AM] /usr/bin/julia exited with code: 1
ERROR:

Looks like this LanguageServer error not the coc error
Closed #20


To support InfluxDB 2.0

Thanks

Does this library support InfluxDB 2.0 or not?

Actual question? Would be great to have support for InfluxDB for 2.0 and others.

#44


New PR to test SignCLA at 2020-02-26 08:51:26.706523815 -0800 PST m=+3.488253640

Body of PR

unsigned-fail-fork-staging: test succeeded


Refactor existing code and add basic tests

The existing code is not well designed and is tightly coupled. It should be refactored with right design pattern and sanitary tests should be added.

Part of https://github.com/maheshmnj/furdle/issues/10


Remove unneeded fetch of CSRF token

We don't need CSRF protection for the reset password token. It doesn't
make sense since you don't even have a session when you're doing reset
password.

@hanningni please review

:+1:


Feat/tbrom 241 public api refactoring

Refactor public methods signature
Remove useless public methods or methods
Reorganize test_tbrom module
Update example 06

Codecov Report

Merging #96 (83c7fa4) into main (2538282) will decrease coverage by 0.54%.
The diff coverage is 92.04%.

@@            Coverage Diff             @@
##             main      #96      +/-   ##
==========================================
- Coverage   88.04%   87.51%   -0.54%
==========================================
  Files          11       11
  Lines        2209     2194      -15
==========================================
- Hits         1945     1920      -25
- Misses        264      274      +10

@chrpetre This one is ready to merge with a lot of refactoring. I suggest we merge it and create a new one to incorporate your last dev from the PR95. What do you think?


How do we know if I am doing level 5 the right way?

If I get A for CodeCrunch, does it mean I am coding correctly?
I created a new class SingleServiceBooking, that takes in a service, request and driver. It is similar to Booking class, but it only takes in service.
I used it to make SingleServiceBooking for each services a driver can have.
For example, the test case,
findBestBooking(new Request(20, 3, 1000), List.of(new NormalCab("SHA1234", 5), new PrivateCar("SMA7890", 10)))
I will make 4 SingleServiceBooking objects, that will compare among themselves and sort based on lowest fare and time.
I would like to know if making extra classes when the question does not ask for it is allowed, and if it is the right way of coming up with the code for the question, since I am not using Booking class in level 5.

I have a similar worry as well, but I think its fine. I did it your way as well with the new class, but in the end I figured out how to incorporate it into the booking class itself by overloading the constructor.
Similarly in Lab 3 Level 6, where it said to modify level5.jsh minimally to complete level 6 and made no mention of creating a new class. I didn't think creating a new RecycledLoader class was allowed so I tried to do it all in the jsh script.
Turns out I was torturing myself for nothing and I just made a new RecycledLoader class anyway. If it makes sense to create a new class I don't see why not.

I asked my TA the same question and he said it is not necessary. In the end, I did what SpectralMusket did and had another constructor in my Booking class so that I can create 4 booking classes with 2 from each driver.
I think it should be allowed to make your own classes as long as you are not breaking any OO principles such as having cyclic dependencies.

Maybe you can create another booking class that subclass the booking class say call it other booking that calculates the fare that is more expensive on the same service. Doing so allows us to call the comparator method and sort them accordingly,since Booking class is superclass of the other booking class

I think an easier way would be to separate the 2 different options for 1 booking in the booking class instead. This can be done by creating a new constructor (driver,req,service), as well as using a getter method to separate the 2 different options in one booking. After which, u can use the comparator method thought in lecture 5 to sort the bookings in the level5.jsh file, which would store all the bookings (via an imlist). this was how i did it, probably would have other alternative methods as well

What I did was to have each driver return an ImList of services they provided.  I also overloaded the booking class to force a service. By iterating through the ImList of services, I created an ImList of bookings with each (driver, service) premutation, which I  sorted.

I think an easier way would be to separate the 2 different options for 1 booking in the booking class instead. This can be done by creating a new constructor (driver,req,service), as well as using a getter method to separate the 2 different options in one booking. After which, u can use the comparator method thought in lecture 5 to sort the bookings in the level5.jsh file, which would store all the bookings (via an imlist). this was how i did it, probably would have other alternative methods as well

Hi, may I know what properties you have initialized in the Booking class if you were to overload the constructor? Because if there's a new constructor (driver,req,service), I need to initialize service too right?

I think an easier way would be to separate the 2 different options for 1 booking in the booking class instead. This can be done by creating a new constructor (driver,req,service), as well as using a getter method to separate the 2 different options in one booking. After which, u can use the comparator method thought in lecture 5 to sort the bookings in the level5.jsh file, which would store all the bookings (via an imlist). this was how i did it, probably would have other alternative methods as well

Hi, may I know what properties you have initialized in the Booking class if you were to overload the constructor? Because if there's a new constructor (driver,req,service), I need to initialize service too right?

My booking class has an added service property, what i did was if a service was declared in the overloaded constructor, it will use the service, if it was just created with (driver,req) then it will find the optimal service to use and declare it with some calculations.

I think an easier way would be to separate the 2 different options for 1 booking in the booking class instead. This can be done by creating a new constructor (driver,req,service), as well as using a getter method to separate the 2 different options in one booking. After which, u can use the comparator method thought in lecture 5 to sort the bookings in the level5.jsh file, which would store all the bookings (via an imlist). this was how i did it, probably would have other alternative methods as well

Hi, may I know what properties you have initialized in the Booking class if you were to overload the constructor? Because if there's a new constructor (driver,req,service), I need to initialize service too right?

What I did was booking has a property service. In the original constructor,  this.service = driver.bestService(), where bestService() is a method that returns the lowest cost of the services available. On the new (driver,req,service) constructor, service is initialized directly this.service = service.

I think an easier way would be to separate the 2 different options for 1 booking in the booking class instead. This can be done by creating a new constructor (driver,req,service), as well as using a getter method to separate the 2 different options in one booking. After which, u can use the comparator method thought in lecture 5 to sort the bookings in the level5.jsh file, which would store all the bookings (via an imlist). this was how i did it, probably would have other alternative methods as well

Hi, may I know what properties you have initialized in the Booking class if you were to overload the constructor? Because if there's a new constructor (driver,req,service), I need to initialize service too right?

What I did was booking has a property service. In the original constructor,  this.service = driver.bestService(), where bestService() is a method that returns the lowest cost of the services available. On the new (driver,req,service) constructor, service is initialized directly this.service = service.

Got it. Thanks!!


Add documentation

@Mark-Simulacrum let me know if there are things you'd like to see covered in more details.
Rendered

Addressed review feedback.


Data block API

It would be nice to have an API for easily constructing learning methods as manually implementing all the methods can get tedious and the resulting methods don't compose well.
The API would be similar to fastai's data block API with the main difference that it is limited to learning methods, i.e. it keeps data container creation and task-specific data encoding separate.
Based on Blocks which represent a kind of data and Encodings, transformations that encode data and are optionally invertible allowing the decoding of outputs.
API
Best to give an example of what using it would look like. Below are reimplementations of some of FastAI.jl's computer vision methods.
ImageClassificationSingle(sz, classes) = Method(
    blocks=(Image{2}(), Label(classes)),
    encodings=[
        ProjectiveTransforms(sz),
        ImagePreprocessing(),
        OneHot()
    ]
)

ImageSegmentation(sz, classes) = Method(
    blocks=(Image{2}(), Mask{2}(classes)),
    encodings=[
        ProjectiveTransforms(sz),
        ImagePreprocessing(),
        OneHot()
    ]
)

SiameseSimilarity() = Method(
    blocks=((Image{2}(), Image{2}()), Label([true, false])),
    encodings=[
        ProjectiveTransforms(sz),
        ImagePreprocessing(),
    ],
)

Given just these short definitions and the block and encoding definitions and the right interfaces inplace, it would be possible to derive the following:

core interface (encoding, decoding, incl. buffered versions)
validation of input data
plotting interface
model building based on input and target blocks
loss functions based on target block

By grouping functionality by block or encoding, it would be much easier to compose and reuse different steps.
Status
I've started writing out the interface and giving some examples.

I crossposted your request for feedback to the FastAI discord and received the following response:

A couple thoughts:
From the design spec, it looks encodings cannot be applied separately to each input, and the API is limited to one set of encodings. For self supervised learning and other multi-image tasks, it's often necessary to apply different sets of augmentations to each image pair: one weakly augmented and the other strongly augmented. From the start I would design FastAI.jl's datablock to support this.
On a similar line of thought, in fastai2's datablock mixing input types, images + tabular for example, is not a straightforward task and requires creating separate image and tabular datablocks then mixing their dataloaders. Instead of adding an image and tabular block in one datablock. Would be nice if FastAI.jl's datablock could support this too.

The comment is now adressed above under "Apply an encoding to a specific block only"


HTML/ScrInject.B trojan in the sample code?

I didn't see this one already posted, so sorry if it is a dupe.
I am running through the migration steps (https://github.com/angular/angular-cli/wiki/Upgrading-from-Beta.10-to-Beta.14) and got to this step:
ng new migration-project

my AV software (on my mac) said there was a trojan in the examples file:
 /usr/local/lib/node_modules/.staging/redeyed-91a8e7f5/examples/browser/index.html

The trojan in question is called: HTML/ScrInject.B
I am skeptical to the validity of this claim, but I figured I'd pass it along for you all to check out.

Please provide us with the following information:

OS?

macOS Sierra 10.12.1

Versions.

Please run ng --version. If there's nothing outputted, please run in a Terminal: node --version and paste the result here:

$ ng --version
Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.
angular-cli: 1.0.0-beta.18
node: 6.3.1
os: darwin x64
Repro steps.

Was this an app that wasn't created using the CLI? What change did you do on your code? etc.
start a new project?

The AV software I am running is Open ESET Cyber Security ( https://www.eset.com/us/ )
The log given by the failure.

Normally this include a stack trace and some more information.
N/A

Mention any other details that might be useful.

Thanks! We'll be in touch soon.

I have the same problem too. The redeyed package is all used in react-scripts. The anti-virus software I used is System Center 2012 on OSX El Captan. It seems it's just an example file so it won't affect anything once. But this is really annoying.

same here with ESET Endpoint Antivirus, it pointet me to this: http://www.virusradar.com/en/HTML_ScrInject.B/description

There isn't anything we can do specifically here, but I'll keep the issue open so the community can pitch in.

@filipesilva redeyed is now 1.0.1 and solved the problem. See https://github.com/thlorenz/redeyed/issues/18.

@philiptzou good to know, thanks!


Add lowercase to colorSchemas

Hello,
first of all thank you very much for the msabrowser, it works very well! Nevertheless, I have seen a possible enhancement by adding lowercase letters to the colorSchemas. There are some alignment programs that output lowercase fasta files by default, such as MAFFT in the case you align nucleotides.
Best regards

Dear @dreyes17 ,
First of all, thank you very much for giving it a try on MSABrowser and your feedback is well appreciated.
Also, we are happy to hear that you like our tool!
As for your suggestion, we have added your input to our project plan and try to consider it during updates.
I think we could resolve this issue by converting all the inputs to uppercase rather than duplicating the keys in ColorSchemas dictionary.
Here is what I thought at Line 485 (https://github.com/thekaplanlab/msabrowser/blob/master/javascript/msabrowser.js#L485):
from
 var aaLetter = seq1.charAt(positionIndex);

to
 var aaLetter = seq1.charAt(positionIndex).toUpperCase();

What do you think @halilbilgin?
Yours sincerely,
Furkan

Dear @dreyes17,
Thank you very much again for informing us about the issue.
PR by @halilbilgin, as we discussed above, is on the way!
Best,
Furkan


Keyboard Remapping Win Right, Win Left, etc Makes CTRL Stick (Like it is always pressed)

Microsoft PowerToys version
0.35
Running as admin

[ ] Yes

Area(s) with issue?
Keyboard Manager
Steps to reproduce
Remapped CTRL + Arrow keys to work as Win + Arrow keys. Can't type afterward. Like the CTRL key is stuck.
✔ Expected Behavior
THe keys to be remapped
❌ Actual Behavior
Keys remapped, but keyboard not usable for typing.
Other Software
No response

@mrorange44
can you please post a screenshot showing the mapping configuration? Thanks.

I deleted it. But it was

Win + Right   -   Ctrl Right + Right    -   All
Win + Left    -    Ctrl Right + Left    -    All
Win + Up    -    Ctrl Right + Up    -    Alll
Win + Down    -    Ctrl Right + Down    -    All

@mrorange44
Does CTRL stuck after typing or right away? How often does it happen?

Here was the mapping i did.  I'm not seeing any 'sticking'.  I'm unable to repro and i'm on 0.36
This mapping makes WinKey with arrows act like the ctrlkey with arrows

This ctrl sticking happens to me also (just tested with 0.37).  Here's what I've tried to achieve and a short background explanation, why:

I want to use US layout (brackets etc are better available), but I also need accented characters when writing in Finnish or when typing people's names.
As a result, I always set my environment to use US International which allows me to access accented letters (https://en.wikipedia.org/wiki/QWERTY#/media/File:KB_US-International.svg)
The positioning of Ä (which I need most frequently) is cumbersome (altgr+Q).
Hence I've tried to remap e.g. ctrl+] to ctrl+alt+Q so that I have somewhat easier access.
If I try to use those remaps, ctrl gets stuck. For example typing A afterwards does 'select all'.
The stickiness of ctrl goes off sometimes if I press the ctrl key only (without any other key).

Some of my remaps below.  The behaviour is the same even without choosing a left/right ctrl/alt key:

Short screengrab of what I tried to explain:
https://user-images.githubusercontent.com/65907494/116905725-08c9d580-ac48-11eb-8d42-16356b9336ce.mp4

I solved my issue by mapping 'SHIFT + Win + A'  instead of 'Win + Left'   and 'SHIFT + Win + S' instead of 'Win + Right', etc... No more sticking CTRL key.

@arttu-karpinlehto
Your issue is tracked by #9690, leave your comment there.

PowerToysReport_2021-07-23-12-38-31.zip

Mapping Shift(left)+N/P => Ctrl(left)+Win(left)+Right/Left
switch back and forth between virtual desktops with Shift(left)+N/P
Ctrl eventually gets stuck (and default windows actions for ctrl+n and ctrl+p are activated...)

This happen with just shortcuts remapping, with no keys remapping the problem persists

Hi, some changes from https://github.com/microsoft/PowerToys/pull/12986 might have fixed this behavior.
Are you still experiencing this on the latest releases?

Hi, some changes from #12986 might have fixed this behavior. Are you still experiencing this on the latest releases?

Hi. Unfortunately, I can't confirm. When I had the initial issue, I revised my mapping choices to use other keys to achieve the same function.


[Dojo] issue with delete_entity command

I am facing an issue while using commands::delete_entity that leads to not delete my entity and allow the following test to pass:
let mut tile_sk: Query = (ctx.caller_account, col, row).into();
commands::<Tile>::delete_entity(tile_sk);
let tile = commands::<Tile>::entity(tile_sk);

By using the following work around, I succeeded to have my entity to be deleted
let mut tile_sk: Query = (ctx.caller_account, col, row).into();
ctx.world.delete_entity(ctx, 'Tile'.into(), tile_sk);
let tile = commands::<Tile>::entity(tile_sk);  <-- raised an error

@Bal7hazar can you confirm this is still an issue

@neotheprogramist can you confirm this is resolved with the new indexer logic

I don't think we addressed this issue while working on the indexer logic.
@Bal7hazar Could you provide broader context or some tests so I could check that on the newest dojo version, please?

I did not use it for a while, not sure if it is still an issue, better to close and reopen if needed probably.

This test fails,
    #[test]
    #[available_gas(30000000)]
    fn test_delete_entity() {
        let (caller, world, actions_) = init();

        let energy_init = Energy { amt: 10, id: 1 };

        set!(world, (energy_init));

        let energy = get!(world, (1), (Energy));
        assert(energy.amt == 10, 'incorrect energy');

        world.delete_entity('RPSType', array![1].span());

        let post_delete_energy = get!(world, (1), (Energy));
        assert(post_delete_energy.amt == 0, 'incorrect post delete energy');
    }

➜  contracts git:(main) ✗ sozo test
running 1 tests
test emojiman::actions::tests::test_delete_entity ... fail (gas usage est.: 6075238)
failures:
   emojiman::actions::tests::test_delete_entity - panicked with [11103201786955740868605512006434910093315610855676006929386402113401 ('incorrect post delete energy'), ].

error: test result: FAILED. 0 passed; 1 failed; 0 ignored

I'm happy to grab this, we need this...

how did you go with this @shramee ?

This is fixed and merged


Mobile collapse-ables are not behaving properly.

Describe the bug
Mobile version collapse-ables retain its status when moving back to desktop view.
To Reproduce
Steps to reproduce the behavior:

Go to '/members'
Go to responsive view
Click on member filter collapseable box
Go back to desktop view
See error
The box will retain its status of being collapsed without a way to open it again unless you go to mobile view and open it.

Expected behavior
On desktop it should always retain its status of being uncollapsed filter menu.
This issue occurs on /missions /members
Screenshots

Suggested solutions
I went on and found a package called @next/device which is not supported on nuxt3, however, there's a fork of it working on nuxt3 which I am working it out on our project with the maintainer.
Link to the discussion can be found here

Fixed in PR #76


Rename Request protocol to RequestType for consistency with Swift standard library

Resolves #79.

Done! :zap:

:+1:


Add Opendata France geocoder

I implemented support for forward and reverse city geocoding using opendata from french government portal.
Full documentation of this API is here : https://adresse.data.gouv.fr/api/
Unit testing included
Let me know if there's anything that needs to be changed in order to merge.
We will use this module with this geocoder soon in one of our project ;)

Thanks for the PR :+1:

Just published the version 3.8.0 with your geocoder

Thanks !!


Gae

Gae branch for "Generalized Advantage Estimation". Advantage is either R-V (normal), or GAE (using temporal difference in the advantage)
Gae branch fixes logits_p having relu func
Gae branch adds a "Config.Zoo" for personal folder for other training configs like "CartPole-v0" with no convnet for fast regression testing (under 1 minute) or "KarpathyPong" which makes a smart preprocessing for fast training (under 20 minutes).

undoing the " Environment.preprocess_karpathy_pong(frame)" is simple enough, but things like
GA3C/ga3c/Environment.py", line 77, in get_num_actions     return len(self.game.env._action_set) AttributeError: 'TimeLimit' object has no attribute '_action_set'
seem to indicate that @etienne87 has made additional changes that require a modified openai-universe.

thanks @nczempin, will take care of this asap! i'm gonna pull current version & resolve conflicts. I need to replace len(self.game.env._action_set) by  return self.game.env.action_space.n

well, the conflicts are not an issue for me for now, because I can just work with the branch.
Are you saying that the karpathy_pong thing and this one line 77 are the only two things I need to change, so I basically gave up too early?

okay, I made those two changes (rolling back the karpathy_pong thing and making the suggested change on line 77) and it is working on the branch.

@etienne87, in networkVP.py
log function is missing the newly added args.
def log(self, x, y_r, a, adv):
    feed_dict.update({self.y_r: y_r, self.action_index: a, self.advantages : adv})

why delete? Is the code wrong?

@tangbohu the code was not giving much better results...sorry about that; if you want any guidance i should be able to send you some old code.


D4B4 Peer-to-peer code review

Great work @MostafaShahat2021 💯
However, there are a few things you can improve on your code.

The social icons on the mobile version are a little bit squeezed, you can increase the gap or padding around them.
Kindly do not wrap an image into another div.
You can also include a box shadow on the hover effect on images.

[OPTIONAL]

Kindly add the acknowledgment parts to your README.md
Kindly use the flex gap property for your image.
Happy coding 👍🏿

Great work @MostafaShahat2021 👍
There are a few things you can improve in your code.

kindly decrease the animation in the card.
follow the Figma design in your card section.

appreciate your work 🥇


👉🏻 REQUEST: Made scroll back to top button smooth

Is there an existing issue for this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe.
when we add this feature it made better user experience
Describe the solution you'd like
Required to made scroll smooth

Describe alternatives you've considered
No response
Additional context
No response

pls assign me this issue with hacktoberfest label @wh0sumit

pls assign this issue to me?

@AnshuPathak-88825  & @Ankita-278  could you both explain how to create a smooth scrolling effect in simple words??

@wh0sumit  Added smooth scrolling feature pls merge it

Do not raise unnecessary pull requests, wait for maintainers to assign before submitting a PR.
this is the wrong way to contribute. You should first learn how to contribute if you're a beginner.

I'll assign this issue to you, but first, you have to explain to me how to create a smooth scrolling effect in simple word.


⚠ Bofrost.de has degraded performance

In d4fe60d, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1496 ms

Resolved: Bofrost.de performance has improved in 7bd7607 after 13 minutes.


module 'qtpy.QtCore' has no attribute 'pyqtBoundSignal'?

I was opening a GUI with an existed yml file,and have no idea what was going on

I ran into the same issue and notice someone else has the same problem: https://github.com/lneuhaus/pyrpl/issues/467
My workaround is to excute pip install qtpy==1.9.0 and it worked.

Also ran into this issue. My workaround was using the python3-only branch of pyrpl instead of master.


fix: undefined error property in json logging

Closes #1547. Relatively simple fix as the bug appears to just have been a typo of copying the wrong line i.e. the code block several lines above this change has the correct conditional.

It does fix the unhandled error but since the error occurred, it means that the logFull function was at some point called with level set to ERROR and options.error being null. This way we will just ignore that call, won't we?

There are some instances where we do log errors without an error, for example: https://github.com/api3dao/airnode/blob/74f33ea835d71f22152354ee8cff2e0747080c0b/packages/airnode-node/src/coordinator/calls/coordinated-execution.ts#L46
But that is somewhat beside the point here as the changed code block corresponds to providing an additional log when there is a stack available. If you look one line above the changed line, the error will still be logged in the case where there isn't an error: https://github.com/api3dao/airnode/blob/3034ad0b10e7cd9c464132bdcce50743029a31f5/packages/airnode-utilities/src/logging/logging.ts#L119-L121

Oh, right, must have missed that, sorry. Carry on :slightly_smiling_face:


FileSystem returns files and directories sorted by name

A change made last year to version 5.5 changed the behavior of the FileSystem files() method:
https://github.com/laravel/framework/commit/0665f952fdee73f0fe41c0b311ea82ac6718bcb3#diff-3aea724aa0501a58dc0f933de2be07e7
Previously files() would return the list of files in alphabetical order. Since that change, the returned array is no longer sorted alphabetically by name. As the returned object is an array of SplFileSystem objects, sorting the result set in userland isn't as simple as just calling sort(). However, sorting the result set inside the FileSystem class is dead easy.
Many developers may prefer the FileSystem class to return arrays of things sorted by name, as per the original behavior. At the very least it seems like a nice convenience that shouldn't negatively impact anybody who doesn't need this behavior.

Would this need any tests so that it stays this way?


Feature Request: Hide menu bar icon

I would like to be able to hide the menu bar icon.

(duplicate) See #24


fix left value not updating when scrolling horizontally with fixed position

I fixed two bugs that occur when you scroll horizontally:

When you have already scrolled to the right and you enter the VIEWPORT-BOTTOM affixType, the sidebar jumps to the right. (already worked correctly for VIEWPORT-TOP)
When scrolling horizontally in either the VIEWPORT-BOTTOM or VIEWPORT-TOP affixType, the sidebar stays fixed in one place.

You can see both bugs in this video:
https://user-images.githubusercontent.com/129282340/228540452-82836699-464c-4ba7-abcd-3439c87c4a1e.mp4

@blixhavn Could you take a look at this, please?

This PR is stalled? Or any chance for reviving the lib @blixhavn?


Ratings feedback failing for BOLD images

BOLD reports are set a wrong URL to submit ratings.

This was actually fixed in 0.15


Search results getting overlapped by Weekly Summary text

Please see below.

Fixed with<PHONE_NUMBER>f9b9eae4ca0bc060b9259059654047
Someone please verify

looks good.


Bug: Cannot reach last slide using navigation button

Bug Report
Current Behavior
Cannot reach the last slides by swiping.
Input Code and steps to reproduce
<carousel
          :per-page="2"
          :scroll-per-page="false"
          :navigation-enabled="true"
          :pagination-enabled="false"
>
...
</carousel>

Expected behavior/code
Before 0.16.0, it was not possible to reach the last slide by using navigation "next" button. Swiping everything worked fine. Now, after update to 0.16.0, this behavor is inverted, i.e., I can reach the last slider by pressing "next" button but not by swiping.

Having the same issue @SEE

This can be reproduced on the "Navigation buttons and scrollPerPage false" scenario on the project's Vue Play

@dfvc Thank you!

Hmm thanks for flagging guys, I can reproduce on my end. The last minor added some changes to scrolling logic, we'll have to investigate to see where the issue is. Clearly our fix didn't fully resolve! I'll mark as a bug - if anyone fixes in the mean time, feel free to make a pr!

@quinnlangille, there's already a pr for it: https://github.com/SSENSE/vue-carousel/pull/346

@quinnlangille there is already https://github.com/SSENSE/vue-carousel/pull/346

Ah, didn't see (still going thru notifications for the week and always do issues first). I'll review and continue conversation on pr - thanks for pointing it out :octocat:

This should be closed now with #346 merged. 💯

Hi! I'm still seeing evidence of this behavior.  I.e., I can reach the last slide by pressing "next" button but not by swiping.
I was able to reproduce on that same "Navigation buttons and scrollPerPage false" scenario mentioned above with some minor changes:  load 9 or more slides and change perPage to anything more than '2':
const images = [
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176006_1.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176005_2.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176003_3.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176004_4.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176002_5.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176001_2.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176006_1.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176005_2.jpg",
  "https://res.cloudinary.com/ssenseweb/image/upload/b_white,c_lpad,g_south,h_1086,w_724/c_scale,h_560/v588/171924M176003_3.jpg"
]
...
{ props: { navigationEnabled: true, scrollPerPage: false, perPage: 3 } }


markdown bug in readme

'_'
should be written as
'\_'

Could you provide more context. I generate MarkDown from Rakudoc using Raku-Pod-Render, so my template is incorrect. I'll need some more info to locate the error

I'm not sure what additional context I can provide, but: line 36 in README.md has the literal text:
'_'
This markdown is the beginning of italics markup, e.g. italics (look at the source of this comment). You can see this in the rendered README.md on github.
Instead, this should be escaped, as:
'\_'
which will then show a literal underscore with no italic effect. If you're using a tool, then this is a bug in the tool.

Thanks. I see that problem now, and another one later in the README.

Fixed in README


Not very important code optimization

I made same optimization changes of HubGroupList to HubConnectionList.

@Gurgen,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

Thanks @Gurgen !


Spelling mistake

When clicking the link in:
Roadmaps -> open source -> Getting Started -> 14 ways to contribute...genuis
It should be changed to genius.

I would like to fix this

I would like to fix this

I don't think you can unless you have the source code of the website.

@AnimeshKumar923 it would be really helpful if you add screenshots etc. It's hard to find the typo with current information provided :)

@AnimeshKumar923 it would be really helpful if you add screenshots etc. It's hard to find the typo with current information provided :)

This should help...@kaiwalyakoparkar

@AnimeshKumar923 this has been fixed. thank you so much for raising the issue 🙌


Crash in setDeallocHandler

I have a view controller which contains a UIWebView. I'm not using any BlocksKit methods with this web view.
In the view controllers dealloc method, when I set the webview's delegate to nil I get this crash:
#6	0x00022942 in -[WebViewController dealloc] at ViewControllers/WebViewController.m:266    <- set delegate to nil here
#5	0x00093e5a in __79+[NSObject(A2BlockDelegateBlocksKit) registerDynamicDelegateNamed:forProtocol:]_block_invoke_0 at BlocksKit/BlocksKit/A2BlockDelegate+BlocksKit.m:239
#4	0x00092781 in -[A2DynamicDelegate(A2BlockDelegate) setRealDelegate:] at BlocksKit/BlocksKit/A2BlockDelegate+BlocksKit.m:79
#3	0x0009335c in -[NSObject(A2BlockDelegateBlocksKitPrivate) setDeallocHandler:] at BlocksKit/BlocksKit/A2BlockDelegate+BlocksKit.m:174
The line that crashes is:
objc_setAssociatedObject(self, _cmd, nil, OBJC_ASSOCIATION_RETAIN_NONATOMIC);
with:
-[A2BlockDelegateDeallocHandler release]: message sent to deallocated instance 0x8088a40

Oh, my bad, that's fixed in 4161320fab. It'll be in the next tagged release if you're using those, otherwise just git pull and you're good. Have a good day.

One more for you :)
In A2BlockDelegate+BlocksKit.m, in dealloc you are calling block(), except block has just been set to nil.
Cheers

Fixed in 95640f1. I'll roll these all into a release soon.


hyp1f1 precision issue

Reported on the mailing list a while ago:
In [7]: scipy.__version__
Out[7]: '0.14.0.dev-7c99934'

In [8]: special.hyp1f1(0.5, 1.5, -709.7827128933)
Out[8]: 0.033264594357227797

In [9]: special.hyp1f1(0.5, 1.5, -709.7827128934)
Out[9]: inf

Analysis and suggested fix: http://article.gmane.org/gmane.comp.python.scientific.user/32789

This issue has been fixed, better to close it.

Also confirmed, closing.


Doesn't work on reversed layout.

It works fine when the layout it straight, but when we set
mLinearLayoutManager.setReverseLayout(reverse);
mLinearLayoutManager.setStackFromEnd(reverse);
then, the header gets drawn from the middle of the first item. I tried tinkering around with the code, but couldn't spend much time on figuring out the issue.

Library deprecated. No new development will be taking place.


Json actions module added Signed-off-by: Ioan Vranau<EMAIL_ADDRESS>

Do we really need json when we can write python actions?

This is our P1 priority according to this document:
https://connections.houston.hp.com/docs/DOC-97325
Luci, or Gabi could give us more details


npm audit bump packages version

Bump vulnerable package versions

Coverage remained the same at 78.886% when pulling 3a9a090c325caa6a2586152903acc11d6972b411 on k-egor-smirnov:master into<PHONE_NUMBER>dfcf2cd9ed2bba500cb8ee31c00b9c on jkphl:master.


Fixed key reset after app boot

Generation of captcha key after boot was causing that key submitted by form was invalid all the time

Hey! Thanks for your contribution. Diving back into this old code of mine, I’m starting to wonder if this call to Captcha::generate() is actually useful at all. Captcha::verify() checks if a phrase is stored in session (and returns false if there isn’t) before comparing anyway, so… What do you think?

I’ve just pushed some changes to master, does the problem still occur?

@fabschurt sorry I have shitload of work today. I'll check it tomorrow

@fabschurt I tested your latest changes and submitted another PR with fix for Twig.
Closing this PR

Nice, thanks for your input on this subject 👍


Hash History v5 Compatibility

This adds a check that should make preact-router compatible with both v4 and v5 of the history module (the API changed).
This fixes #378.

I'd like to find time to add a test for this before merging.


Config rule queries related to RDS DB instance should handle

Describe the bug
A clear and concise description of what the bug is.
The below controls are affected due to

RDS DB instance backup should be enabled - "db_instance_backup_enabled": "rds_db_instance_backup_enabled"
RDS DB instances should be in a backup plan - "rds_in_backup_plan": "rds_db_instance_in_backup_plan",
Database logging should be enabled - "rds_logging_enabled": "rds_db_instance_logging_enabled",
RDS DB instances should have deletion protection enabled - "rds_cluster_deletion_protection_enabled": "rds_db_cluster_deletion_protection_enabled",

Steampipe version (steampipe -v)
Example: v0.3.0
Plugin version (steampipe plugin list)
Example: v0.5.0
To reproduce
Steps to reproduce the behavior (please include relevant code and/or commands).
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

Not reproducible; it may be conflicting with tags branch testing. Closing, for now, will re-open in case repeatable.


A bit more help choosing which license?

Licensing is scary - it would be cool to have a series of questions that lead the user down a flowchart to the right license to use for a project. Or maybe just a really neutral summary of each on the selection screen, similar to http://choosealicense.com/

Hey @joerick, absolutely. It's on the plan for the next release.
Will leave it open until its done, thanks for pointing. 👍


Escape curly braces in twig

I have an issue with using twig site variables from site.yaml
Any pages which contain curly braces in code blocks are giving a server error.  How can I escape curly braces when using twig with Grav?

So the curly braces are in the page content or the site,yaml variables?
In either case if you have trig processing on these could be confused with trig tags.  You shokbe able to just wrap the code with twig verbatim tags.

(https://twig.sensiolabs.org/doc/2.x/tags/verbatim.html)

Thanks guys I got it fixed with verbatim tags


Proposal to add YAML support in addition to JSON

Would be nice if we had the option to use YAML as an alternative to JSON. More versatility would be great.
Also seems YAML can do things JSON cannot like : comments, extensible data types, strings without quotation marks and others.  And , according to Wikipedia, JSON can be parsed by YAML (ver 1.2) parsers making it a subset of YAML. So at second glance both of these are interchangeable and YAML does seem easier to write, read and more powerful than JSON.
Obvs this is a low priority feature
I will leave it up you to decide
Thanks. Keep up the good work 👍

Having comments will be good for temporarily adding and removing data blocks to game data like scenario and stats for testing easier. Also adding todo and programmer notes will be a plus. As a stepping stone https://hjson.org/ maybe used as well, since there is a Json parse readily available in godot.

@hubbyist I would still prefer YAML not just for the fact that is readable but it can also do weird things like define variable data types, extend data types. Essentially it can be used not just for data but for modding or scripting a game.  But HJson is easier to read and would be very much preferred over regular JSON.

I think YAML adds value for config files strictly due to commenting. +1ing this.

I think YAML adds value for config files strictly due to commenting. +1ing this.

Please do not bump issues when showing support for a feature request, use the 👍 reaction button on the original post instead.

I'm also interested in an alternative to JSON.  I'm wondering why a save format doesn't even support integers.  For example, saving an array with integer indexes to JSON, then loading them, converts those indexes into strings.  Leaving me to either: do convoluted loading reconstruction of arrays/dicts, or use strings instead of integers.  Both are bizarre solutions.  If YAML supports integers, it should be a thing.

Only keys require quotes in proper json, arrays of ints should save/load just fine.

For example, saving an array with integer indexes to JSON, then loading them, converts those indexes into strings.

I'm not sure what you mean by that. Arrays in JSON don't store the indexes and Godot's arrays don't support string indexes (only dictionaries do), so I don't even know what to make of the sentence. Can you can provide a simple example of those conversions so we can assess if it's a bug?

What about https://msgpack.org? Looks great for network and data compression

@Vaiktorg msgpack is for serializing messages in a compact binary-like format, this discussion has been more for supporting human readable and editable file formats that can be used as resources and loaded from the file system.

Bump.

does YAML have redundant keys like JSON or?

@gring what do you mean? YAML is a super set of JSON.

@buckle2000 well, if the redundant keys are still there, it's hard for me to justify why using YAML over json then. i was thinking it was more of a CSV parser, which is much better for large data storage import. which i am in favor for

@Gring YAML is really cool. For instance, it supports reference and binary data (base64-encoded).

@buckle2000
json is not for large data storage either, which is why i was trying to think of a reason to justify YAML over it. as a CSV parser would be the most beneficial for game devs. and have a real use case (importing large data, no redundant keys, faster, and a much lower filesize).

@naturally-intelligent  For example, saving an array with integer indexes to JSON, then loading them, converts those indexes into strings

i actually just did some testing.
var json = """
{"my_keys": [1,2,3,4,5]}
"""

func _ready():
	var obj_literal = JSON.parse(json).result["my_keys"]
	for hi in obj_literal:
		 print(typeof(hi) == TYPE_INT)

this actually prints False. that's what he was saying when he was referring to the indexes (@vnen).

but it can also do weird things like define variable data types

if YAML can define variable types, then i am willing to take back my original statement and say yes, it could have an advantage over regular json. the YAML gdscript parser will need to accommodate for the different types too

@gring you can specify types explicitly in YAML. Please read the specs yourself.
http://yaml.org/spec/1.2/spec.html#id2761292

@girng well, those aren't indexes, they are values. And this will always print false because they are imported as floats and never as ints (because the JSON spec only has a number type, it does not have any distinction between integers and real numbers). But they are not imported as strings, like @naturally-intelligent declared in the report.
Also, YAML is suggested as a replacement to JSON (since it's a super-set anyway). It's not a new format to fill some other gap if there's any. YAML is easier for a human to read and write than JSON, hence the proposal. CSV is not that good for large storage either and you can already import it using the File API. CSV is also mostly suitable for homogeneous data, while YAML/JSON works fine with ad-hoc structures.

@vnen  CSV is great for large storage because it doesn't use redundant keys. it also loads/parses much faster (which is essential for startup/load times). its also been proven in the industry with many games
they are definitely values i agree, but some people refer to them as indexes too, which i was just trying to show what he meant
@vnen, YAML seems better then. especially if we can define custom types in there, and not have to worry about traversing after the data gets loaded, just to convert to the  type it was. i have changed my vote, but i do urge you to look into a CSV.parse method (similar to JSON.parse) as it will have much more of a use case in the game dev world imo.

Yaml is more friendly to source control than JSON, which would make yaml ideal not only for configuration files, but all textual project files (.tscn, .tres, etc.).
Imagine a JSON dict serialized with sorted keys and indentation like:
"data": {
    "a": 1,
    "b": 2
}

Let's append item "c":
"data": {
    "a": 1,
    "b": 2,
    "c": 3
}

The diff will include two modified lines, since a comma must be added at the end of the "b" line. This does not seem like a big problem with these simple values, but imagine if "b" has a 1000 characters long string as value. It would pollute a patch with unrelated data just because of a single comma at the end not even visible on screen.
Always adding a comma after the last item would fix it, but that's not possible due to the JSON specification.
Same data serialized in yaml:
data:
  a: 1
  b: 2
  c: 3

Adding item "c" would show up as a single line of change in a diff (patch), which is more readable and includes information only on the change itself.
Considerations for maxing out source control friendly behavior by minimizing diffs:

Yaml serialization should be one field per line with some minimum indentation (like 2 spaces)
Serialization should sort keys for consistent order (no diff solely due to change of item order)
Either write all keys to each object or avoid making unnecessary changes

Hey guys I don't want to needlessly bump this thread but I feel this feature is probably the most supported feature request right now. I hope devs will consider this at some point. I think YAML fits perfectly with GDScript's "pythonesque" tab based philosophy.

If you really want this I suggest not to depend on the main devs and instead try to make a pull request. I probably would make use of this too, but I don't need it enough to justify spending time on it right now.

I'd highly vote on no yaml, it is a way overdesigned and broken spec.  If it were to be added then which of its ambiguous setups would be used?  I'm pretty sure TOML is taking over YAML's space, being conceptually similar and similar in syntax while being simple and shorter to parse without a broken spec.
(And as an aside, yaml not supporting tabs for indentation is wtfery, Tabs for Indentation, Spaces for Alignment...  >.<)

@gimg is right I meant to say dict keys, I was new to gdscript at the time.  I get data like this when I serialize to json:
{"characters":{"1":{"approved":false,"denied":true,"id":1,"name":"Richard Fennel"},"2":{"approved":false,"denied":true,"id":2,"name":"Margaret Gerblignzer"}
And then when I'm loading I have to reconstruct the dict in annoying fashion, looping through the data structure like:
character['id'] = int(character['id'])
I shouldn't have to do that, it should just rebuild it.

@naturally-intelligent that's normal because you have strings on the JSON (the numbers are surrounded by quotes). In JavaScript you'd get the same as GDScript, the difference is that JS doesn't care about the types and can mix strings with numbers without issues, while GDS won't let you add a string to a number.
That's not something that can be solved if we follow the JSON specification. You need to serialize the data properly or deal with type-casting when reading.

@vnen yep I just wasn't expecting it.  JSON may not be broken but it also means it isn't fully compatible with the datastructures of GDScript.  I've stopped using integers as keys in GDScript because JSON doesn't support it.  It's not worth the trouble when it comes time for save/load.  I can just convert the string key to integers in code when I need to.  But it feels hacky to not use a GDScript language feature because of JSON.  Doesn't seem right to me.  It's a "gotcha" that is not clear at all to the new developer

Attempting to collect the requirements:

Usability

Human readable
Behaves well with source control (intuitive, minimal diffs)
Non-ambiguous specification

Interoperability

Stable libraries to read/write programmatically
Libraries are available for all mainstream programming languages
Libraries can be compiled to all platforms supported by Godot

Data types

All data types used in Godot/GDScript can be encoded by default or with extension
Supports non-string values as keys

Any suggested additions/changes?
After finalizing the requirements we can create a list of candidate formats with their pros/cons.

Is too bad @vnen   is not onboard with this idea because he has a lot of sway among devs. So I feel since he wont support this idea no matter how many likes might never be implemented unless a contributor will implement it.
@viktor-ferenczi  I think the overall point here is yaml is much more in line with python and Python/GDScript. Json to me feels more like Java.

Well, everybody has some power of influence. The current proposals form is open, and there's one every month. If people add this as a suggestion and it is well voted, it'll certainly bring attention (while only patrons votes are "considered", everybody can vote and global vote stats are always shown, so it still makes an impact).
I didn't say I'm against this, I just said that I personally won't work on this anytime soon, since it's not a particular need of mine right now.

Json to me feels more like Java.

Funny, because JSON stands for JSON JavaScript Object Notation.
I also would appreciate this addition. I would like to export a web build of one of my projects, but I can't because GDNative doesn't work in html5 exports yet. This would solve that problem.

Note that the ConfigFile format partially matches TOML, which makes it possible to parse and write simple TOML files. The main issue is that comments starting with # won't be parsed correctly, as only ; can be used for comments. We could solve this by also allowing # for comments in addition to ;.

Main points of this change:

Enabling the reading and writing of all Godot project files with standard tools, processing with scripts and other similar uses.
Enforcing a normalized serialization format to prevent modifications due to white-space, comment or formatting changes while there is no actual change to the project. This is essential for efficient reviewing of changes before committing or merging.

So is someone working on it? vnen seems to be ok with this. So how and what do we plan to do?

Do we make all the changes from JSON to YAML or just make a json2yaml convertor? Note: If latter, then we could have a toggle in settings.
What files would need changes and which part/class of this should be implemented?
Anybody have any more suggestions or plans?
Anybody up to work listing out the candidate formats as suggested by @viktor-ferenczi?

@Anutrix Adding support for YAML likely means pulling in a YAML library, since it's a very complex format.

And each YAML library parses slightly differently because the format is ambiguous, be careful about choosing one that works well and doesn't have security concerns that some have.  YAML really is not a good format...

Anybody have any more suggestions or plans?

Just dropping these here for inspiration, no plans (yet?) to implement them.
It's often useful to provide "variables" to configuration, such as paths for example, like Symfony does : %kernel_path%.  Godot has some paths and config it could provide.  Injecting (replacing) the variables could be the responsibility of a wrapper of the YAML component?
Also, a function for batch-loading multiple YAML files or directories, in order (last wins) into a single object would allow plugins and modules to expose YAML configuration that the user may override with its own.

Didn't know something like this existed.
wikipedia.org/wiki/Comparison_of_data-serialization_formats
I know Godot has its own requirements for this but it could be a start.

I think only supporting a simplified spec of yaml would be sufficient for 99% of use cases, something similar to StrictYAML.
They also have a note against TOML, but I don't only want yaml for config files. It's great as a basic format that does all the same stuff as JSON but is way cleaner to read and write.

I think only supporting a simplified spec of yaml would be sufficient for 99% of use cases, something similar to StrictYAML. That is, if we don't just use an existing library.
They also have a note against TOML, but I don't only want yaml for config files. It's great as a basic format that does all the same stuff as JSON but is way cleaner to read and write.

I like the idea of supporting a strict subset of YAML. Links broken. Here is a github page
https://github.com/crdoconnor/strictyaml

I agree that StrictYAML would be a good candidate.

That would be essential for Godot integration.

Not really, given that the JSON implementation in Godot is fully custom.

Not really, given that the JSON implementation in Godot is fully custom.

JSON is also an extremely simple single-page spec, where YAML's spec is thousands of pages and still ambiguous...

IMHO, the only problem with hand editing json is that there is too much stiffness in the standard. Mandatory quotes for dictionary keys and string values, and absence of comments and the incapacity of ending arrays and dictionaries with a trailing comma is the main reason that makes json harder to write than yaml. Of course this could be solved outside the engine using external tools to convert from yaml/hjson/whatever to json, but would not solve for inline yaml inside other files or generated at runtime or gotten from outside sources (via an http request, for example).
Also, IMHO, yaml is a great data storage and transmission language, despite some problems here and there. Mainly for the better raw text readability, "writability" and ability to write comments, but also for other features anchors and references which are great. But writing a yaml parser from scratch would probably be impracticable and adding a third party parser maybe would not be optimal.
Suppose someone had the time and interest to modify the json parsing logic at core/io/json.cpp so that it could parse single line comments, trailing commas on arrays and dictionaries and dictionary keys and strings without quotes. Would this be a viable alternative? It would be a json superset, like yaml or hjson, but with the absolute bare minimum of features to allow for better readability, "writability" and "diffability", which would be only the three abovementioned features. Even if it is a viable alternative, would it be preferable than supporting and industry standard format with lots of features like yaml? with the bonus that, being a superset of json, the yaml parser could replace the json one? I know the json parser is custom made, it would take some work to get an external library and adapt it for godot needs, but it is still and option.

[...]
Suppose someone had the time and interest to modify the json parsing logic at core/io/json.cpp so that it could parse single line comments, trailing commas on arrays and dictionaries and dictionary keys and strings without quotes. Would this be a viable alternative?
[...]

I would be quite content with those things being added.

The issue with doing this is that it would make the JSON parser non-standard. If this is done, it must be an opt-in setting, not the default.

ConfigFile:

There seem to be no standard specification for the format, but it is reasonably simple
Implementations for Python, Haskell, C++ (headers only) and more

Also, IMHO, yaml is a great data storage and transmission language, despite some problems here and there. Mainly for the better raw text readability, "writability" and ability to write comments

That is why I think it would be a good addition, writing json is tedious and it would be nicer to store data in an to write format that doesn't need an external tool. For example we creating an item dictionary would be really simple and for anyone to read and modify, as opposed to json that you would need to understand its format.

But writing a yaml parser from scratch would probably be impracticable

That's why I proposed strictyaml, only supporting a subset of features that would fit most use cases. I would argue that anchors shouldn't be allowed in a stricter syntax, because they decrease readability

Suppose someone had the time and interest to modify the json parsing logic at core/io/json.cpp so that [...] Would this be a viable alternative?

IMO it would not be. YAML has a couple extra features that aren't supported in JSON, one example being block scalars to break text into multiple lines.

There seem to be no standard specification for the format, but it is reasonably simple

A standard specification is usually good, but what is more important IMHO are de-facto standards and already wide-spread implementations. Suppose the first most used 'dialect' of config file have, say, 50% of general use and the second most used have 20% or less usage, than the first dialect is good enough to be the one used and the only one supported.
The problem with, let's say, the configfile python implementation is that it does not support nested structures, it cannot represent object trees properly. Any non scalar value is written to the file as single line string, a python array or dictionary literal, and must be manually parsed on code after reading the configfile. Suppose this is the most widespread dialect of config file, then it would not be suitable for our purposes. One can manually put newlines inside the dictionary literal and still have it be parsed in python as long as one ident the next lines with at least one space or tab, but it seems there is not pretty-print for the dictionary literals using the standard configparser python module
Still talking about python implementations, there is a package named configobj wich implements a configfile dialect properly supporting nested structures without having to rely on strings with dictionary literals. This dialect is much more suitable for our needs.
This is a test script for comparison
test.py:
data = {}
data['anint'] = 1
data['afloat'] = 1.2
data['abool'] = True
data['anone'] = None
data['astring'] = 'asdf'
data['anarray'] = [1, 2, 3]
data['anotherarray'] = [',', ',', ',']
data['andict'] = {'a':1,'b':1.0,'c':True,'d':None,'e':'asdf','f':[1,2,3]}
data['nesteddict'] = {'j':1,'a':{'j':1,'b':{'j':1,'c':1}}}

import configobj # external pypi package: pip install configobj
config = configobj.ConfigObj()
config['root'] = data
config.filename = 'configobj.ini'
config.write()

import configparser # part of python standard library
config = configparser.ConfigParser()
config['root'] = {'key':data}
with open('configparser.ini', 'w') as configfile:
	config.write(configfile)

This is the output of the configobj module from the external package 'configobj'
configobj.ini
[root]
anint = 1
afloat = 1.2
abool = True
anone = None
astring = asdf
anarray = 1, 2, 3
anotherarray = ",", ",", ","
[[andict]]
a = 1
b = 1.0
c = True
d = None
e = asdf
f = 1, 2, 3
[[nesteddict]]
j = 1
[[[a]]]
j = 1
[[[[b]]]]
j = 1
c = 1

And this is the output of the standard configparser module
configparser.ini
[root]
key = {'anint': 1, 'afloat': 1.2, 'abool': True, 'anone': None, 'astring': 'asdf', 'anarray': [1, 2, 3], 'anotherarray': [',', ',', ','], 'andict': {'a': 1, 'b': 1.0, 'c': True, 'd': None, 'e': 'asdf', 'f': [1, 2, 3]}, 'nesteddict': {'j': 1, 'a': {'j': 1, 'b': {'j': 1, 'c': 1}}}}

And for comparison, this is how windows' registry editor exports nested structures
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\removeproperties]
"ProgrammaticAccessOnly"="Apartment"

[HKEY_CLASSES_ROOT\*\shell\removeproperties\DropTarget]
"CLSID"="{09a28848-0e97-4cef-b950-cea037161155}"

I should have also compared the toml output for the same data. Note that toml is the only dialect (in these specific implementations at least) where it is possible to put variables outside sections. Also toml nesting levels are much more legible than configobj
This is the test script for comparison
test.py:
data = {}
data['anint'] = 1
data['afloat'] = 1.2
data['abool'] = True
data['anone'] = None
data['astring'] = 'asdf'
data['anarray'] = [1, 2, 3]
data['anotherarray'] = [',', ',', ',']
data['andict'] = {'a':1,'b':1.0,'c':True,'d':None,'e':'asdf','f':[1,2,3]}
data['nesteddict'] = {'j':1,'a':{'j':1,'b':{'j':1,'c':1}}}

import configobj # external pypi package: pip install configobj
config = configobj.ConfigObj()
config['root'] = data
config.filename = 'configobj.ini'
config.write()

import configparser # part of python standard library
config = configparser.ConfigParser()
config['root'] = {'key':data}
with open('configparser.ini', 'w') as configfile:
	config.write(configfile)

import toml # external pypi package: pip install toml
with open('toml.ini', 'w') as configfile:
	toml.dump(data, configfile)

This is the output of the toml module from the external package 'toml'
toml.ini
anint = 1
afloat = 1.2
abool = true
astring = "asdf"
anarray = [ 1, 2, 3,]
anotherarray = [ ",", ",", ",",]

[andict]
a = 1
b = 1.0
c = true
e = "asdf"
f = [ 1, 2, 3,]

[nesteddict]
j = 1

[nesteddict.a]
j = 1

[nesteddict.a.b]
j = 1
c = 1

This is the output of the configobj module from the external package 'configobj'
configobj.ini
[root]
anint = 1
afloat = 1.2
abool = True
anone = None
astring = asdf
anarray = 1, 2, 3
anotherarray = ",", ",", ","
[[andict]]
a = 1
b = 1.0
c = True
d = None
e = asdf
f = 1, 2, 3
[[nesteddict]]
j = 1
[[[a]]]
j = 1
[[[[b]]]]
j = 1
c = 1

This is the output of the standard configparser module
configparser.ini
[root]
key = {'anint': 1, 'afloat': 1.2, 'abool': True, 'anone': None, 'astring': 'asdf', 'anarray': [1, 2, 3], 'anotherarray': [',', ',', ','], 'andict': {'a': 1, 'b': 1.0, 'c': True, 'd': None, 'e': 'asdf', 'f': [1, 2, 3]}, 'nesteddict': {'j': 1, 'a': {'j': 1, 'b': {'j': 1, 'c': 1}}}}

And for comparison, this is how windows' registry editor exports nested structures
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\removeproperties]
"ProgrammaticAccessOnly"="Apartment"

[HKEY_CLASSES_ROOT\*\shell\removeproperties\DropTarget]
"CLSID"="{09a28848-0e97-4cef-b950-cea037161155}"

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.
The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.
If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!