AttributeError: 'NoneType' object has no attribute 'get'

Hi, it seems this error is reported many times, but after checking the various issue, did not find anything that matches my use case.
So, I am working on cluster mode, on yarn.
My job is working fine for a while, and then crashes with the following error:
Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 12, hdpdatanode25): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 759, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 310, in _train
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 100, in _get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Driver stacktrace
My running command is as follow:
${SPARK_HOME}/bin/spark-submit
--master yarn
--deploy-mode cluster
--queue ${QUEUE}
--num-executors 4
--executor-memory 10G
--total-executor-cores 4
--py-files TensorFlowOnSpark/tfspark.zip,TensorFlowOnSpark/examples/criteo/spark/xxx_dist.py
--conf spark.dynamicAllocation.enabled=false
--conf spark.yarn.maxAppAttempts=1
--archives hdfs:///user/${USER}/Python.zip#Python
--conf spark.executorEnv.LD_LIBRARY_PATH="$LIB_HDFS:$LIB_JVM"
--conf spark.executorEnv.HADOOP_HDFS_HOME="$HADOOP_HDFS_HOME"
--conf spark.executorEnv.CLASSPATH="$($HADOOP_HOME/bin/hadoop classpath --glob):${CLASSPATH}"
TensorFlowOnSpark/examples/criteo/spark/xxx_spark.py
--mode train
--data ${TRAINING_DATA}
--validation ${VALIDATION_DATA}
--steps 1000000
--model ${MODEL_OUTPUT}
Copy/Paste from log:
Container: container_e82_1515444508016_3200860_01_000009 on hdpdatanode25_45454
LogType:stderr
Log Upload Time:Wed Mar 28 19:51:04 +0000 2018
LogLength:41976
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data2/hadoop/yarn/local/usercache/amantrach/filecache/4004/spark-assembly-<IP_ADDRESS>.4.2.0-258-hadoop<IP_ADDRESS>.
4.2.0-258.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/<IP_ADDRESS>-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/03/28 19:40:43 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
18/03/28 19:40:44 INFO SecurityManager: Changing view acls to: yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: Changing modify acls to: yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, am
antrach); users with modify permissions: Set(yarn, amantrach)
18/03/28 19:40:44 INFO SecurityManager: Changing view acls to: yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: Changing modify acls to: yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, am
antrach); users with modify permissions: Set(yarn, amantrach)
18/03/28 19:40:45 INFO Slf4jLogger: Slf4jLogger started
18/03/28 19:40:45 INFO Remoting: Starting remoting
18/03/28 19:40:45 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@hdpdatanode25:37929]
18/03/28 19:40:45 INFO Utils: Successfully started service 'sparkExecutorActorSystem' on port 37929.
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data8/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-0c51ff7c-5796-4eba-9c82-fbab024a3496
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data7/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-737f3482-ede7-4d9a-a2c9-2d7a22f13f96
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data9/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-a617b78a-5c59-483e-8f07-961e311fab80
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data2/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-97a31a14-bb28-4700-9361-3b018fa5e6bc
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-40834467-7b08-4a36-ab92-06fa5c4aede8
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data10/hadoop/yarn/local/usercache/amantrach/appcache/application_151544
4508016_3200860/blockmgr-3f10beb5-1fb0-417e-999f-886496fc75ad
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data6/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-c602423e-6b78-4a03-90ec-fe94fe8c83ac
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data1/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-7a36dd58-8b8a-4f1d-8fd0-b9a829bf3d09
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data5/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-2942c138-b7c1-4f9b-a87c-b0ce99e3896a
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at /data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/blockmgr-3c044739-e0ad-4de8-916d-46d4d11e9ecf
18/03/28 19:40:45 INFO MemoryStore: MemoryStore started with capacity 7.0 GB
18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@<IP_ADDRESS>:44573
18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
18/03/28 19:40:45 INFO Executor: Starting executor ID 5 on host hdpdatanode25
18/03/28 19:40:45 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52308.
18/03/28 19:40:45 INFO NettyBlockTransferService: Server created on 52308
18/03/28 19:40:45 INFO BlockManagerMaster: Trying to register BlockManager
18/03/28 19:40:45 INFO BlockManagerMaster: Registered BlockManager
18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Got assigned task 7
18/03/28 19:40:45 INFO Executor: Running task 1.1 in stage 1.0 (TID 7)
18/03/28 19:40:45 INFO TorrentBroadcast: Started reading broadcast variable 2
18/03/28 19:40:45 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 17.5 KB, free 17.5 KB)
18/03/28 19:40:45 INFO TorrentBroadcast: Reading broadcast variable 2 took 123 ms
18/03/28 19:40:46 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 70.5 KB, free 88.0 KB)
18/03/28 19:40:46 INFO HadoopRDD: Input split: hdfs://kandula/user/amantrach/criteo_dataset/training/part-r-00061.deflate:0+3530368287
18/03/28 19:40:46 INFO TorrentBroadcast: Started reading broadcast variable 0
18/03/28 19:40:46 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 28.0 KB, free 115.9 KB)
18/03/28 19:40:46 INFO TorrentBroadcast: Reading broadcast variable 0 took 17 ms
18/03/28 19:40:46 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 343.1 KB, free 459.0 KB)
18/03/28 19:40:46 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
18/03/28 19:40:46 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
18/03/28 19:40:46 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
18/03/28 19:40:46 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
18/03/28 19:40:46 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
18/03/28 19:40:46 INFO ZlibFactory: Successfully loaded & initialized native-zlib library
18/03/28 19:40:46 INFO CodecPool: Got brand-new decompressor [.deflate]
18/03/28 19:40:48 ERROR Executor: Exception in task 1.1 in stage 1.0 (TID 7)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 759, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 310, in _train
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 100, in _get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
    at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

18/03/28 19:40:48 INFO CoarseGrainedExecutorBackend: Got assigned task 9
18/03/28 19:40:48 INFO Executor: Running task 0.1 in stage 1.0 (TID 9)
18/03/28 19:40:48 INFO HadoopRDD: Input split: hdfs://kandula/user/amantrach/criteo_dataset/training/part-r-00060.deflate:0+3531024796
18/03/28 19:40:48 ERROR Executor: Exception in task 0.1 in stage 1.0 (TID 9)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 759, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 310, in _train
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 100, in _get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
18/03/28 19:40:48 INFO CoarseGrainedExecutorBackend: Got assigned task 10
18/03/28 19:40:48 INFO Executor: Running task 1.2 in stage 1.0 (TID 10)
18/03/28 19:40:48 INFO HadoopRDD: Input split: hdfs://kandula/user/amantrach/criteo_dataset/training/part-r-00061.deflate:0+3530368287
18/03/28 19:40:48 ERROR Executor: Exception in task 1.2 in stage 1.0 (TID 10)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444508016_3200860/container_e82_1515444508016_3200860_01_000001:
...
Attached the full log of another run
Also, when checking the run, the system automatically tries to run new executors...which is a problem and should not happened with maxattempt fixed to 1, and dynamicallocation set to false.
Any help is welcome,
Thanks
application_1515444508016_3202937.log

Hi Amin,
That's usually a downstream "symptom" vs. a root cause.  I searched through
your logs via:
egrep "ERROR|Exception|exceed" application_1515444508016_3202937.log
and found the following log statements:
18/03/28 20:42:34 WARN YarnAllocator: Container killed by YARN for
exceeding memory limits. 12.0 GB of 12 GB physical memory used. Consider
boosting spark.yarn.executor.memoryOverhead.
18/03/28 20:42:34 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
Container killed by YARN for exceeding memory limits. 12.0 GB of 12 GB
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
18/03/28 20:42:34 ERROR YarnClusterScheduler: Lost executor 1 on
hdpdatanode36: Container killed by YARN for exceeding memory limits. 12.0
GB of 12 GB physical memory used. Consider boosting
spark.yarn.executor.memoryOverhead.
So, if you're not using Spark RDD storage/caching, you can reduce
spark.memory.storageFraction and increase  spark.yarn.executor.memoryOverhead.
Lee
On Wed, Mar 28, 2018 at 1:51 PM, Amin Mantrach<EMAIL_ADDRESS>wrote:

Hi, it seems this error is reported many times, but after checking the
various issue, did not find anything that matches my use case.
So, I am working on cluster mode, on yarn.
My job is working fine for a while, and then crashes with the following
error:
Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most
recent failure: Lost task 1.3 in stage 1.0 (TID 12, hdpdatanode25):
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 759, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 310, in train
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01
000009/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 100, in
_get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1},
state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace
My running command is as follow:
${SPARK_HOME}/bin/spark-submit
--master yarn
--deploy-mode cluster
--queue ${QUEUE}
--num-executors 4
--executor-memory 10G
--total-executor-cores 4
--py-files TensorFlowOnSpark/tfspark.zip,TensorFlowOnSpark/examples/criteo/spark/xxx_dist.py
--conf spark.dynamicAllocation.enabled=false
--conf spark.yarn.maxAppAttempts=1
--archives hdfs:///user/${USER}/Python.zip#Python
--conf spark.executorEnv.LD_LIBRARY_PATH="$LIB_HDFS:$LIB_JVM"
--conf spark.executorEnv.HADOOP_HDFS_HOME="$HADOOP_HDFS_HOME"
--conf spark.executorEnv.CLASSPATH="$($HADOOP_HOME/bin/hadoop classpath
--glob):${CLASSPATH}"
TensorFlowOnSpark/examples/criteo/spark/xxx_spark.py
--mode train
--data ${TRAINING_DATA}
--validation ${VALIDATION_DATA}
--steps 1000000
--model ${MODEL_OUTPUT}
Copy/Paste from log:
Container: container_e82_1515444508016_3200860_01_000009 on
hdpdatanode25_45454
LogType:stderr
Log Upload Time:Wed Mar 28 19:51:04 +0000 2018
LogLength:41976
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data2/hadoop/yarn/
local/usercache/amantrach/filecache/4004/spark-assembly-
<IP_ADDRESS>.4.2.0-258-hadoop<IP_ADDRESS>.
4.2.0-258.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/<IP_ADDRESS>-
258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/
StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/03/28 19:40:43 INFO CoarseGrainedExecutorBackend: Registered signal
handlers for [TERM, HUP, INT]
18/03/28 19:40:44 INFO SecurityManager: Changing view acls to:
yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: Changing modify acls to:
yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(yarn, am
antrach); users with modify permissions: Set(yarn, amantrach)
18/03/28 19:40:44 INFO SecurityManager: Changing view acls to:
yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: Changing modify acls to:
yarn,amantrach
18/03/28 19:40:44 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(yarn, am
antrach); users with modify permissions: Set(yarn, amantrach)
18/03/28 19:40:45 INFO Slf4jLogger: Slf4jLogger started
18/03/28 19:40:45 INFO Remoting: Starting remoting
18/03/28 19:40:45 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkExecutorActorSystem@hdpdatanode25:37929]
18/03/28 19:40:45 INFO Utils: Successfully started service
'sparkExecutorActorSystem' on port 37929.
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data8/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-0c51ff7c-5796-4eba-9c82-fbab024a3496
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data7/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-737f3482-ede7-4d9a-a2c9-2d7a22f13f96
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data9/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-a617b78a-5c59-483e-8f07-961e311fab80
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data2/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-97a31a14-bb28-4700-9361-3b018fa5e6bc
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data4/hadoop/yarn/local/usercache/amantrach/appcache/application_1515444
508016_3200860/blockmgr-40834467-7b08-4a36-ab92-06fa5c4aede8
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data10/hadoop/yarn/local/usercache/amantrach/appcache/application_151544
4508016_3200860/blockmgr-3f10beb5-1fb0-417e-999f-886496fc75ad
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data6/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/blockmgr-c602423e-
6b78-4a03-90ec-fe94fe8c83ac
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data1/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/blockmgr-7a36dd58-
8b8a-4f1d-8fd0-b9a829bf3d09
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data5/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/blockmgr-2942c138-
b7c1-4f9b-a87c-b0ce99e3896a
18/03/28 19:40:45 INFO DiskBlockManager: Created local directory at
/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/blockmgr-3c044739-
e0ad-4de8-916d-46d4d11e9ecf
18/03/28 19:40:45 INFO MemoryStore: MemoryStore started with capacity 7.0
GB
18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Connecting to driver:
spark://CoarseGrainedScheduler@<IP_ADDRESS>:44573
18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Successfully
registered with driver
18/03/28 19:40:45 INFO Executor: Starting executor ID 5 on host
hdpdatanode25
18/03/28 19:40:45 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 52308.
18/03/28 19:40:45 INFO NettyBlockTransferService: Server created on 52308
18/03/28 19:40:45 INFO BlockManagerMaster: Trying to register BlockManager
18/03/28 19:40:45 INFO BlockManagerMaster: Registered BlockManager
18/03/28 19:40:45 INFO CoarseGrainedExecutorBackend: Got assigned task 7
18/03/28 19:40:45 INFO Executor: Running task 1.1 in stage 1.0 (TID 7)
18/03/28 19:40:45 INFO TorrentBroadcast: Started reading broadcast
variable 2
18/03/28 19:40:45 INFO MemoryStore: Block broadcast_2_piece0 stored as
bytes in memory (estimated size 17.5 KB, free 17.5 KB)
18/03/28 19:40:45 INFO TorrentBroadcast: Reading broadcast variable 2 took
123 ms
18/03/28 19:40:46 INFO MemoryStore: Block broadcast_2 stored as values in
memory (estimated size 70.5 KB, free 88.0 KB)
18/03/28 19:40:46 INFO HadoopRDD: Input split:
hdfs://kandula/user/amantrach/criteo_dataset/training/part-
r-00061.deflate:0+3530368287
18/03/28 19:40:46 INFO TorrentBroadcast: Started reading broadcast
variable 0
18/03/28 19:40:46 INFO MemoryStore: Block broadcast_0_piece0 stored as
bytes in memory (estimated size 28.0 KB, free 115.9 KB)
18/03/28 19:40:46 INFO TorrentBroadcast: Reading broadcast variable 0 took
17 ms
18/03/28 19:40:46 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 343.1 KB, free 459.0 KB)
18/03/28 19:40:46 INFO deprecation: mapred.tip.id is deprecated. Instead,
use mapreduce.task.id
18/03/28 19:40:46 INFO deprecation: mapred.task.id is deprecated.
Instead, use mapreduce.task.attempt.id
18/03/28 19:40:46 INFO deprecation: mapred.task.is.map is deprecated.
Instead, use mapreduce.task.ismap
18/03/28 19:40:46 INFO deprecation: mapred.task.partition is deprecated.
Instead, use mapreduce.task.partition
18/03/28 19:40:46 INFO deprecation: mapred.job.id is deprecated. Instead,
use mapreduce.job.id
18/03/28 19:40:46 INFO ZlibFactory: Successfully loaded & initialized
native-zlib library
18/03/28 19:40:46 INFO CodecPool: Got brand-new decompressor [.deflate]
18/03/28 19:40:48 ERROR Executor: Exception in task 1.1 in stage 1.0 (TID
7)
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 759, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 310, in train
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01
000009/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 100, in
_get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1},
state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

18/03/28 19:40:48 INFO CoarseGrainedExecutorBackend: Got assigned task 9
18/03/28 19:40:48 INFO Executor: Running task 0.1 in stage 1.0 (TID 9)
18/03/28 19:40:48 INFO HadoopRDD: Input split:
hdfs://kandula/user/amantrach/criteo_dataset/training/part-
r-00060.deflate:0+3531024796
18/03/28 19:40:48 ERROR Executor: Exception in task 0.1 in stage 1.0 (TID
9)
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 759, in func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 310, in train
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01
000009/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 100, in
get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1},
state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(
PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
18/03/28 19:40:48 INFO CoarseGrainedExecutorBackend: Got assigned task 10
18/03/28 19:40:48 INFO Executor: Running task 1.2 in stage 1.0 (TID 10)
18/03/28 19:40:48 INFO HadoopRDD: Input split:
hdfs://kandula/user/amantrach/criteo_dataset/training/part-
r-00061.deflate:0+3530368287
18/03/28 19:40:48 ERROR Executor: Exception in task 1.2 in stage 1.0 (TID
10)
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01
000009/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/data4/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000009/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/data3/hadoop/yarn/local/usercache/amantrach/appcache/
application_1515444508016_3200860/container_e82_1515444508016_3200860_01_
000001:
...
Attached the full log of another run
Also, when checking the run, the system automatically tries to run new
executors...which is a problem and should not happened with maxattempt
fixed to 1, and dynamicallocation set to false.
Any help is welcome,
Thanks
application_1515444508016_3202937.log
https://github.com/yahoo/TensorFlowOnSpark/files/1857722/application_1515444508016_3202937.log
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/yahoo/TensorFlowOnSpark/issues/257, or mute the
thread
https://github.com/notifications/unsubscribe-auth/ADgXrhPxHmdCAVN_bomNjqM9cKVDKlOWks5ti_fGgaJpZM4S_WAg
.

Hi, thanks. Can you be more explicit about the caching solution, any pointer in the examples? How would you use it.

Hi, it seems that actually tf_feed.next_batch(batch_size) is not reading data from a file buffer or in streaming mode but from a queue where the data is pre-loaded. In my case, the chunks are very big more than 10GB per chunk, and this explains why it requires that much memory, it's not a model issue that is very small. To solve this, I just did a preprocessing to have very small chunks. Hence no memory problem, but is there an alternative solution you would suggest? Thanks Lee!

By caching, I only meant Spark RDD persistence, i.e. if your upstream preprocessing of data requires caching/persisting the RDD into Spark storage memory... If not (i.e. your just reading from disk), then you can tune down the spark.memory.storageFraction, so less of your memory is "reserved" for RDD storage.
As for the queue, that was required to guarantee that the TF process (running in the background) could access the data being fed via Spark executor process (in the foreground).  A workaround would be to switch your code to InputMode.TENSORFLOW and use TF's built in tf.data APIs to read directly from HDFS.

You can just submit a PR, and we'll take a look at it.
Also, been thinking about limiting the size of the queue for your problem... if you are interested, you can modify the TFoS code here to set a "reasonable" maxsize... large enough so the TF consumer doesn't spend all it's time waiting on the Spark producer, and small enough that the Spark producer doesn't copy the entire partition into the queue.


Add commercial licence warning for NREL SPA algorithm

Add commercial limitation warning for NREL SPA code, until licensing is determined from NREL

Wondering if it would be better to use warnings.warn. This way you only get the message once instead of every time that you use the function.

I think the word ing on this needs to be changed anyways to be consistent with #9, so i'll ditch this one. warnings.warn would be better.


General assembly: excuse passives (fixes #417)

Thanks, I'll make the changes you mentioned.
I just saw #300 (which mentions that observers should be preferred), therefore I just pushed a commit that makes the necessary changes. Let me know if I misunderstood something and should undo the commit.

No it's good, I just did not want to say it as we you only used the "created" event here. Observers are best if you handle a lot of events there but at least now we are consistent.

I made the requested changes :)
I made two separate database migrations, because I figured a table rename is disruptive enough to warrant its own migration file. Let me know if I should combine the migrations instead.
The way the localization is handled for the comment column is sub-ideal: the backend fills the database with a localized string based on the locale of the user who creates the general assembly. Instead of storing a localized string, we could add some sort of flag column that determines whether the contents of the comment column should be displayed as-is or if it should be used as a localization key. Let me know if the current solution is adequate or if I should do something differently.


v0.1.1 release sync

incorporates the astroport tokenfactory-based liquid pooler contract into the main branch

One question, right now in this state the contract would not be compatible with covenants in main branch because it doesn't have the op_mode, this will be done later I guess?

One question, right now in this state the contract would not be compatible with covenants in main branch because it doesn't have the op_mode, this will be done later I guess?

Yes, we will update it with the op_mode. I created an issue for that.

@bekauz @keyleu Anything missing or this can be merged ?


businger b_m and b_h are unstable params

Purpose

This PR adds the unstable Businger parameters b_m and b_h to BusingerParams, and modifies other appropriate files to fit these additional parameters.
Content

This PR changes the hardcoded 15 and 9 parameters in the unstable regime to parameters b_m and b_h. This change is reflected in UniversalFunctions.jl. In addition, create_parameters is modified such that creating Businger parameters also fetches the default values of b_m and b_h from CLIMAParameters. Lastly, the tests are modified with the appropriate default values of b_m and b_h.

[ x ] I have read and checked the items on the review checklist.

cc @charleskawczynski your input here would be helpful! I think this is a breaking change - perhaps I can add another constructor that picks up default values from the CLIMAParameters toml files to mitigate interface issues.

cc @charleskawczynski your input here would be helpful! I think this is a breaking change - perhaps I can add another constructor that picks up default values from the CLIMAParameters toml files to mitigate interface issues.

I think I’d rather avoid that, all parameter changes are considered breaking unfortunately. I’m fine with this.
we’ve added the new parameters to clima parameters?
and, just to double check, these values are the same as what the hard-coded ones were in surface fluxes, right?
Finally, what was the motivation behind adding these parameters? Just calibration? Why these in particular?

cc @charleskawczynski your input here would be helpful! I think this is a breaking change - perhaps I can add another constructor that picks up default values from the CLIMAParameters toml files to mitigate interface issues.

I think I’d rather avoid that, all parameter changes are considered breaking unfortunately. I’m fine with this.
we’ve added the new parameters to clima parameters?
and, just to double check, these values are the same as what the hard-coded ones were in surface fluxes, right?
Finally, what was the motivation behind adding these parameters? Just calibration? Why these in particular?

(1) Noted - the corresponding parameter modifications have been made to the ClimaParameters repo - v0.7.7 . Once the calibration work is wrapped up we can notify people on upcoming version changes.
(2) Yes, the default values in ClimaParameters are those that were hard-coded previously in the UniversalFunctions module.
(3) The motivation here is to demonstrate the calibration pipeline against observed datasets, and high resolution large-eddy simulations with known parameters. Businger parameters are being modified in particular as a first step: we expect to update other such parameters in all UniversalFunctions eventually.
cc @wilsonduan10


Add addon version check with the repo

Problem:

Addon & repo chart version mis-matching (e.g. https://github.com/harvester/harvester/issues/5840)
Solution:

Check version matching in the step of build-bundle
In amd64 architecture, the mismatching will cause build failure
In arm64 architecture, the mismatching will print some WARNING messages but build continue
Related Issue:
https://github.com/harvester/harvester/issues/4937
https://github.com/harvester/harvester/issues/5840
Test plan:

make iso output:
amd64
...
charts packed in Harvester repo
total 1.3M
drwxr-xr-x 2 root root 4.0K Jun  3 18:35 .
-rw-r--r-- 1 root root  12K Jun  3 18:35 index.yaml
-rw-r--r-- 1 root root 2.5K Jun  3 18:35 nvidia-driver-runtime-0.1.1.tgz
-rw-r--r-- 1 root root 3.0K Jun  3 18:35 harvester-seeder-0.1.1.tgz
-rw-r--r-- 1 root root 2.8K Jun  3 18:35 harvester-pcidevices-controller-0.3.2.tgz
-rw-r--r-- 1 root root 2.6K Jun  3 18:35 harvester-vm-import-controller-0.1.8.tgz
-rw-r--r-- 1 root root  15K Jun  3 18:35 rancher-logging-103.0.0+up3.17.10.tgz
-rw-r--r-- 1 root root  82K Jun  3 18:35 rancher-logging-crd-103.0.0+up3.17.10.tgz
-rw-r--r-- 1 root root 398K Jun  3 18:35 rancher-monitoring-crd-103.0.3+up45.31.1.tgz
-rw-r--r-- 1 root root 416K Jun  3 18:35 rancher-monitoring-103.0.3+up45.31.1.tgz
-rw-r--r-- 1 root root  66K Jun  3 18:34 harvester-crd-0.0.0-master-f9464f06.tgz
-rw-r--r-- 1 root root 211K Jun  3 18:34 harvester-0.0.0-master-f9464f06.tgz
drwxrwxr-x 1 root root 4.0K Jun  3 18:34 ..
addon template files
total 16K
drwxr-xr-x 2 root root 4.0K Jun  3 18:34 .
drwxr-xr-x 4 root root 4.0K Jun  3 18:34 ..
-rw-r--r-- 1 root root 7.2K Jun  3 18:34 rancherd-22-addons.yaml
addon: harvester-vm-import-controller version: 0.1.8
addon: harvester-pcidevices-controller version: 0.3.2
addon: rancher-logging version: 103.0.0+up3.17.10
addon: rancher-monitoring version: 103.0.3+up45.31.1
addon: harvester-seeder version: 0.1.1
addon: nvidia-driver-runtime version: 0.1.1
all addons have matched version with the repo

arm64:
...
drwxr-xr-x 2 root root 4.0K Jun  3 18:37 .
-rw-r--r-- 1 root root  11K Jun  3 18:37 index.yaml
-rw-r--r-- 1 root root 2.5K Jun  3 18:37 nvidia-driver-runtime-0.1.1.tgz
-rw-r--r-- 1 root root  15K Jun  3 18:37 rancher-logging-103.0.0+up3.17.10.tgz
-rw-r--r-- 1 root root  82K Jun  3 18:37 rancher-logging-crd-103.0.0+up3.17.10.tgz
-rw-r--r-- 1 root root 398K Jun  3 18:37 rancher-monitoring-crd-103.0.3+up45.31.1.tgz
-rw-r--r-- 1 root root 417K Jun  3 18:37 rancher-monitoring-103.0.3+up45.31.1.tgz
-rw-r--r-- 1 root root  66K Jun  3 18:37 harvester-crd-0.0.0-master-f9464f06.tgz
-rw-r--r-- 1 root root 215K Jun  3 18:37 harvester-0.0.0-master-f9464f06.tgz
drwxr-xr-x 1 root root 4.0K Jun  3 18:37 ..
addon template files
total 16K
drwxr-xr-x 2 root root 4.0K Jun  3 18:37 .
drwxr-xr-x 4 root root 4.0K Jun  3 18:37 ..
-rw-r--r-- 1 root root 7.2K Jun  3 18:37 rancherd-22-addons.yaml
addon: harvester-vm-import-controller version: 0.1.8
WARNING: addon harvester-vm-import-controller is defined with version 0.1.8 but the chart is not packed into repo in arm64
addon: harvester-pcidevices-controller version: 0.3.2
WARNING: addon harvester-pcidevices-controller is defined with version 0.3.2 but the chart is not packed into repo in arm64
addon: rancher-logging version: 103.0.0+up3.17.10
addon: rancher-monitoring version: 103.0.3+up45.31.1
addon: harvester-seeder version: 0.1.1
WARNING: addon harvester-seeder is defined with version 0.1.1 but the chart is not packed into repo in arm64
addon: nvidia-driver-runtime version: 0.1.1
all addons have matched version with the repo

The upgrade path (amd64)
upgrade addon files
total 32K
drwxr-xr-x 5 root root 4.0K Jun  4 11:40 ..
drwxr-xr-x 2 root root 4.0K Jun  4 11:40 .
-rw-r--r-- 1 root root  388 Jun  4 11:40 harvester-seeder.yaml
-rw-r--r-- 1 root root 1.8K Jun  4 11:40 logging_addon.yaml
-rw-r--r-- 1 root root 2.3K Jun  4 11:40 monitoring_addon.yaml
-rw-r--r-- 1 root root  475 Jun  4 11:40 nvidia-driver-toolkit.yaml
-rw-r--r-- 1 root root  366 Jun  4 11:40 pcidevices-controller.yaml
-rw-r--r-- 1 root root  362 Jun  4 11:40 vm-import-controller.yaml
upgrade addon: harvester-seeder version: 0.1.1
upgrade addon: rancher-logging version: 102.0.0+up3.17.10
WARNING upgrade addon rancher-logging has version mis-matching: in repo is 103.0.0+up3.17.10 but in addon is 102.0.0+up3.17.10
upgrade addon: rancher-monitoring version: 100.1.0+up19.0.3
WARNING upgrade addon rancher-monitoring has version mis-matching: in repo is 103.0.3+up45.31.1 but in addon is 100.1.0+up19.0.3
upgrade addon: nvidia-driver-runtime version: 0.1.1
upgrade addon: harvester-pcidevices-controller version: 0.3.2
upgrade addon: harvester-vm-import-controller version: 0.1.8
all upgrade addons have matched version with the repo

note: the above warning message will be gone after https://github.com/harvester/harvester/pull/5954 is merged.

@tserong You are right, this PR can't help https://github.com/harvester/harvester/issues/5840. It can only check the matching in the current generated ISO.
For the upgrade path, let's try to find a practical solution, thanks.

@tserong The new function check_upgrade_addon_chart_version_matching will check the matching between upgrade addon definition and the chart repo, please take another look, thanks.

Will rebase code after https://github.com/harvester/harvester/pull/5633 is merged.

The current code is manually backported to v1.3 via https://github.com/harvester/harvester-installer/pull/754

for previous harvester releases, I think add these checks make sense, we don't want any mismatched version show up. But I'm not sure we still need this in master branch if https://github.com/harvester/harvester/pull/5633 got merged first since after then all addon yamls will be moved to the new addon repos which make sure all the addon version matched.

for previous harvester releases, I think add these checks make sense, we don't want any mismatched version show up. But I'm not sure we still need this in master branch if harvester/harvester#5633 got merged first since after then all addon yamls will be moved to the new addon repos which make sure all the addon version matched.

@brandboat
There are 3 places related to addon/chart versions:
(1) Addon manifest
(2) Addon upgrade
(3) Chart repo in Harvester-installer when build-bundle.
The #5633 will move (1) and (2) to same repo, but the files themselves are still split in different folders (?). If true, it it still possible that (1) and (2) are not updated at same time due to kind of mistake. And (1) (2) should be synced with (3) to have identical chart version.
This PR will help us to avoid any potential mismatching.

Rebased code after https://github.com/harvester/harvester/pull/6112.
The upgrade addon is generated from the single-source https://github.com/harvester/addons/blob/58ad21070c91bd875df901818c3caddd35a6eefa/pkg/templates/rancherd-22-addons.yaml#L1
This PR will just check the template file once, make sure the charts are matching.

@mergifyio backport v1.4

@w13915984028 Does this also work for v1.3/v1.2?

@bk201 Master-head has https://github.com/harvester/harvester/pull/6112, that's different with v1.3.0
Instead v1.3 needs PR https://github.com/harvester/harvester-installer/pull/754 to check version mis-matching.

I have also prepared this https://github.com/harvester/addons/pull/4, and should ensure that template and upgrade path manifests are all generated from a common list of environment variables


Deserialize tz

fixes #102 #106 #107

This pull requests seems to be very elegantly done for TZ support as it also supports current usage. @dominicrodger  There has been no review process and updated to this library since September. Things like Django 2.0 support and timezone support that has multiple tickets are critical for this library to be in a healthy state.
Time wise I certainly know how hard it's to support all the questions and pull requests. Has there been any discussion of adding core contributors to this package?
Besides this I would like to thank Dominic for writing this library. It has helped us out of the django-celery-beat hole that we were in and helps us support recurring tasks in a much more sustainable and dynamic way. Thank you!

All valid comments @madisvain, and very kindly and thoughtfully expressed.
First, this pull request - I need to take a closer look, but I think it's likely fine to merge, I'll try to get that done by the end of next week.
Secondly, I'm very open to more maintainers - I am not actually using django-recurrence any more, and haven't done since the Django 1.8 release, so it would be great to have other contributors on board. For now, if anyone wants write access I have no major objections, and we will leave it so only @emperorcezar has permission to push to PyPI. If anyone wants to step up - let me know either on here, or by email (internet@dominicrodger.com).

Dom

It would be highly appreciated if there was some documentation about the development process, who has to be prodded for code reviews, Pull requests and releases to Pypi

Hey @cleder,
Me for code reviews/pull requests, and @emperorcezar for pushes to PyPI. As mentioned - if others want to help out with reviews/merges here, I'm happy to add people.
Dom

Thanks for your work on this @cleder. Let me know if you're interested in being added as a contributor.

Please let's update the documentation. I had to search this pull request to find that this feature is hidden behind a setting change.


15832 Fixed null JSON properties

Issue #: bcgov/entity#15832
Description of changes:

app version = 5.0.3
updated browserslist
don't save applicationDate if it's empty
don't save noticeDate if it's empty

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the bcrs-entities-create-ui license (Apache 2.0).

thanks for the quick fix


Allow control to be deselected

Hello,
I have a use-case to allow a new user to select his/her gender. I want the control to be initially deselected to prevent the user from skipping the setup screen with the default "Male" option. SO an empty option would help force the user to select an option. UISegmentedControl allows this:
control.selectedSegmentIndex = UISegmentedControlNoSegment

perhaps you could allow index and setIndex() to accept -1 as an exception. This is common standard for indexes.

Hmm... that's interesting! It shouldn't be that big of a change but some logic needs to be changed since I'm using a UInt for the index and not an Int like UISegmentedControl does.
I'm adding this to the TODO list, thanks!

thanks for the update. Can't wait to get it!

Has this been implemented?

@pjebs No, not yet. When I implement something I link the Pull Request it was implemented in.

@gmarm Any luck?

@simoami unfortunately not.
I understand the need for this and I see it's a popular request, however I haven't had much time for OSS lately.
Will do my best to work on it soon.


Docker::Network.create does not pass through Docker::Connection

Hi,
third parameter from Docker::Network.create is a Docker Connection, however, after creating the network, the get call is not passing through the parameter.
def create(name, opts = {}, conn = Docker.connection)
      default_opts = {
        'Name' => name,
        'CheckDuplicate' => true
      }
      resp = conn.post('/networks/create', {},
                       body: default_opts.merge(opts).to_json)
      response_hash = Docker::Util.parse_json(resp) || {}
      get(response_hash['Id'], {}, conn) || {} # conn was missing here
    end

@ineb can you open a PR?

Yes.
https://github.com/swipely/docker-api/pull/367


Update travis

...although, it looks like the humann2 tests are erroring now. maybe Python3 issue? Do we want to fix for now?

I fixed the python3 sorting errors, the new one is
The database file for MetaPhlAn does not exist at /home/travis/build/qiita-spots/qp-shotgun/miniconda3/envs/qp-shotgun/bin/db_v20/mpa_v20_m200.pkl . Please provide the location with --metaphlan-options. Note that the latests version doesn't come with a DB (db_v20) so it's not being copied. Do you know where can we get it?

ah this is the bigass metaphlan db. I've made a streamlined mini db for
testing here:
https://github.com/biocore/oecophylla/blob/dev/test_data/test_dbs/mp2_mini.tar.gz
small enough to include in test files (3.9 MB)
On Fri, Dec 1, 2017 at 7:11 AM Antonio Gonzalez<EMAIL_ADDRESS>wrote:

I fixed the python3 sorting errors, the new one is
The database file for MetaPhlAn does not exist at
/home/travis/build/qiita-spots/qp-shotgun/miniconda3/envs/qp-shotgun/bin/db_v20/mpa_v20_m200.pkl
. Please provide the location with --metaphlan-options. Note that the
latests version doesn't come with a DB (db_v20) so it's not being copied.
Do you know where can we get it?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/qiita-spots/qp-shotgun/pull/32#issuecomment-348519432,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH6JAFSCqIU0RQgGG_c-pAZGHpmjMi_Aks5s8BccgaJpZM4QxWRR
.

closing in favor of: https://github.com/qiita-spots/qp-shotgun/pull/33


Npm build script

Created an npm build script (runnable via npm run build) which will among other little tweaks output the transpiled JS files to the dist folder.

@hoffmabc can you test this and see if it works with your builds?


Inconsistent/Incorrect results

Hi, first of all thanks for creating ort!
I'm trying to run inference of a very simple model but get drastically different (incorrect) results from onnxruntime in Python.

Model generation and reference (keras, tf2onnx and onnxruntime)
You can download the model here: onnx_model.zip
from tensorflow.keras import losses
from tensorflow.keras import Input, Model
from tensorflow.keras.layers import Dense, Flatten

import onnx
import onnxruntime
import tf2onnx
import numpy as np

input = Input(shape=(7,6,2,2), name='input')

x = Flatten()(input)
x = Dense(128, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(32, activation='relu')(x)
x = Dense(16, activation='relu')(x)
x = Dense(2, activation='softmax', name='output')(x)

model = Model(inputs=[input], outputs=[x])
model.compile(loss=[losses.CategoricalCrossentropy()], optimizer='adam', metrics=['accuracy'])
model.summary()

onnx_model, _ = tf2onnx.convert.from_keras(model)
onnx.save_model(onnx_model, f"onnx_model.onnx")

sess = onnxruntime.InferenceSession(f"onnx_model.onnx")
input_name = sess.get_inputs()[0].name
label_name = sess.get_outputs()[0].name

for i in range(500):
    input = np.zeros((1,7,6,2,2)).astype(np.float32)
    pred_onnx = sess.run([label_name], {input_name: input})[0]
    print(pred_onnx) # [[0.5 0.5]]

I'm using branch v2, and tried running the model in a loop, but get weird results which vary with the value I use in with_intra_threads:

Example with v2 branch
use ndarray::*;
use ort::{Environment, ExecutionProvider, SessionBuilder, Value};

fn main() -> ort::Result<()> {
    tracing_subscriber::fmt::init();

    let environment = Environment::builder()
        .with_execution_providers([ExecutionProvider::CPU(Default::default())])
        .build()?
        .into_arc();

    let session = SessionBuilder::new(&environment)?
        .with_intra_threads(8)?
        .with_model_from_file("onnx_model.onnx")?;

    loop {
        let input = Array5::<f32>::zeros((1, 7, 6, 2, 2));
        let output = Array2::<f32>::zeros((1, 2));

        let in_value = Value::from_array(input)?;
        let out_value = Value::from_array(output)?;

        let mut binding = session.create_binding()?;
        binding.bind_input("input", in_value)?;
        binding.bind_output("output", out_value)?;
        let outputs = binding.run()?;

        let data = outputs[0].extract_tensor::<f32>()?.view().t().into_owned();

        println!("{:?}", data.as_slice().unwrap());
    }
}

With .with_intra_threads(1), I never get something close to [0.5, 0.5]:
[3.0262524e29, 0.501777]
[6.2462477e28, 5.76e-43]
[3.0262524e29, 4.1777e-41]
[6.246444e28, 5.76e-43]
[6.2462175e28, 5.76e-43]
[6.246172e28, 5.76e-43]
[6.246852e28, 5.76e-43]
[3.0262524e29, 0.501777]
[6.246444e28, 5.76e-43]
[6.246535e28, 5.76e-43]
[3.0262524e29, 2.7915669e23]
[6.246157e28, 5.76e-43]
[3.0262524e29, 0.501777]
[6.2460966e28, 5.76e-43]

With .with_intra_threads(8), I sometimes get the expected result:
[0.5, 0.5]
[0.5, 0.5]
[3.0262524e29, 0.501777]
[3.4692473e27, 2.7908747e23]
[8.391229e34, 7.67e-43]
[8.392116e34, 7.67e-43]
[0.5, 0.5]
[0.5, 0.5]
[0.5, 0.5]
[0.5, 0.5]
[0.5, 0.5]
[0.5, 0.5]
[8.392338e34, 7.67e-43]
[0.5, 0.5]

With CUDA, is the same story (affected by threads too) but with more incorrect results than correct.
I'm not sure if I'm messing up the lifetime of the tensors or something. I tried it in two different computers and faced the same issue, so I decided to report it. Hope it can be replicated.

I also tried with the latest version (1.16.2). Each execution (session) gives a consistent result, but not across sessions. I came across: [[0.5], [0.5]], [[1], [0]] (most likely), [[0.57364064], [0.4263594]] (or close) and sometimes even [[NaN], [NaN]].

Example with 1.16.2
use ndarray::*;
use ort::{Environment, ExecutionProvider, SessionBuilder, Value, OrtResult, AllocatorType, MemType, MemoryInfo, AllocationDevice};

fn main() -> OrtResult<()> {
    tracing_subscriber::fmt::init();

    let environment = Environment::builder()
        .with_execution_providers([ExecutionProvider::CPU(Default::default())])
        .build()?
        .into_arc();

    let session = SessionBuilder::new(&environment)?
        .with_intra_threads(1)?
        .with_model_from_file("onnx_model.onnx")?;

    loop {
        let input = CowArray::from(Array5::<f32>::zeros((1, 7, 6, 2, 2)).into_dyn());

        let in_value = Value::from_array(session.allocator(), &input)?;
        let output_mem_info = MemoryInfo::new(AllocationDevice::CPU, 0, AllocatorType::Device, MemType::Default)?;

        let mut binding = session.bind()?;
        binding.bind_input("input", in_value)?;
        binding.bind_output("output", output_mem_info)?;

        session.run_with_binding(&binding)?;

        let outputs = binding.outputs()?;
        for (output_name, output_value) in outputs {
            let output_array = output_value.try_extract::<f32>()?;
            println!("{:}", output_array.view().t().into_owned());
        }
    }
}

Thanks!

With the v2 code on Windows I was getting a STATUS_HEAP_CORRUPTION exit code, which I fixed by removing the line to register the CPU execution provider. I'm able to reproduce the accuracy issues with IoBinding.

Foregoing `IoBinding` gives consistent `0.5`s.
use ndarray::*;
use ort::{Environment, ExecutionProvider, SessionBuilder};

fn main() -> ort::Result<()> {
	tracing_subscriber::fmt::init();

	let environment = Environment::builder().build()?.into_arc();

	let session = SessionBuilder::new(&environment)?
		.with_intra_threads(8)?
		.with_model_from_file("onnx_model.onnx")?;

	loop {
		let input = Array5::<f32>::zeros((1, 7, 6, 2, 2));

		let outputs = session.run(ort::inputs![input]?)?;

		let data = outputs[0].extract_tensor::<f32>()?.view().t().into_owned();

		println!("{:?}", data.as_slice().unwrap());
	}
}

For the segfault it seems like OrtSessionOptionsAppendExecutionProvider_CPU was silently removed in ONNX Runtime 1.16. I will backport the fix to ort 1.16 shortly.
As for the IoBinding issue, the output values are moved into the IoBinding in bind_output by adding them to a vector of values:
https://github.com/pykeio/ort/blob/84add5f0d22c36ed251e063e40a1457ea01ece69/src/io_binding.rs#L44-L50
When an IoBinding is run, it drops all values owned by the binding, including output values, before then returning those outputs:
https://github.com/pykeio/ort/blob/84add5f0d22c36ed251e063e40a1457ea01ece69/src/io_binding.rs#L60-L65
which is obviously going to cause an issue :^)
Changing bind_output to instead take a &mut Value is more ergonomic and fixes the lifetime issues, and I'm now getting consistent 0.5s with & without IoBinding. Unfortunately, this is a breaking change, so I cannot backport it to 1.16.
Thank you for reporting this issue! Potential lifetime & ergonomics issues with IoBinding was one of my few remaining concerns about v2. Finding and fixing this issue is going to help its release move much more swiftly.

Thanks for looking at this so fast! Glad it helped!


[security] do not log username and password in clear text (in case of errors)

Adapter logs incorret username and password (!) in cleartext.
Client [shellymotion2-nnnnnnnn] has invalid password(-password-) or username(-username-)
Ìn my oppinion security relevant data should NEVER be logged with cleartext. At least passwords should be masked (or omitted in this case). Even in case of incorrect data this information is sensitive as incorrect paswords often are incorrect by simply missing or mixed up characters only.
Adapter Release 2.5.1

see PR https://github.com/ioBroker/ioBroker.sonoff/pull/305


MSID: 87340 Version: 2 DOI: 2<IP_ADDRESS>2020

[PLACE MANUSCRIPT AND EDITOR DETAILS HERE WHEN AVAILABLE] See step 3
[PLACE PDF URL HERE WHEN AVAILABLE] See step 7
Step 1. Inform bioRxiv
Who can help: @QueenKraken, @nlisgo, @scottaubrey

[x] Manuscript is in data hub index (https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=87340)

Step 2. Create preview of manuscript
Who can help: @fred-atherden, @nlisgo, @scottaubrey

[ ] Manuscript is available for preview (https://prod--epp.elifesciences.org/preview/87340/v2)

[ ] Trigger import for staging: https://prod--epp.elifesciences.org/import
[ ] Sync data folder with s3://prod-elife-epp-data/data
[ ] Trigger import for staging: https://staging--epp.elifesciences.org/import
[ ] Sync data folder with s3://staging-elife-epp-data/data
[ ] Pull request created to enhanced-preprint-data
[ ] Post link to pull request below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-manuscript
Step 3: Awaiting public reviews and QC
Who can help: Production team

[ ] QC passed or QC not needed

Who can help: Editorial team

[x] Public reviews available on sciety (https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>2020)
[ ] Manuscript and editor details at the top of this issue (Supplied by Editorial team)
[ ] Reviews returned by EPP - https://prod--epp.elifesciences.org/api/reviewed-preprints/87340/v2/reviews

Example
"msas": "Genetics and Genomics", "Neuroscience"
"msid": "84628"
"version": "1"
"preprintDoi": "10.1101/2<IP_ADDRESS>4241"
"articleType": "Reviewed Preprint"
"status": "Published from the original preprint after peer review and assessment by eLife."

"Reviewed Preprint posted": "2023-01-02"
"Sent for peer review": "2022-10-28"
"Posted to bioRxiv": "2022-11-21" (link: "Go to bioRxiv": "https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>4241v1")

Editors:
Reviewing Editor
Michael B Eisen
University of California, Berkeley, United States
Senior Editor
Michael B Eisen
University of California, Berkeley, United States

Step 4: Modify manuscripts.json (no PDF)

[ ] Reviewed preprint is published to EPP (https://elifesciences.org/reviewed-preprints/87340)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[ ] Request that a doi be registered

[ ] Post link to #enhanced-preprint comment below

Pull request: [PLACE LINK TO PULL REQUEST HERE]
#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]

Instructions to modify manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/publish-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/334/files
Once the pull request is merged in it should be available a few minutes later.

Request that a doi
Post the following in #enhanced-preprint:
@Fred can you register a doi for https://elifesciences.org/reviewed-preprints/87340

Step 5: Awaiting search reindex

[ ] Reviewed preprint is avaliable on journal homepage (https://elifesciences.org)

The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.

Additional info
If needed, the jenkins pipeline to reindex search can be triggered sooner.
https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 6: Published! Request PDF generation

[ ] Post the link to the PDF url at the top of the issue

[ ] Post link to #enhanced-preprint comment below

#enhanced-preprint comment thread: [PLACE LINK TO COMMENT HERE]
Post the following to the #enhanced-preprint on slack:
@Ryan Dix-Peek please can you generate a PDF for https://elifesciences.org/reviewed-preprints/87340

Step 7: Introduce PDF to data folder and git repo

[ ] PDF is avaliable at https://github.com/elifesciences/enhanced-preprints-data/raw/master/data/87340/v2/87340-v2.pdf

Detailed instructions: https://github.com/elifesciences/enhanced-preprints-data#add-a-pdf
Step 8: Add PDF url to manuscripts.json

[ ] Reviewed preprint PDF is available for download (https://prod--epp.elifesciences.org/reviewed-preprints/87340)

[ ] Pull request created to enhanced-preprint-client
[ ] Pull request switched to "Ready for review"
[ ] Post link to pull request below

[PLACE LINK TO PULL REQUEST HERE]

Instructions to add PDF url to manuscripts.json

Visit: https://github.com/elifesciences/enhanced-preprints-client/actions/workflows/add-pdf-url-to-manuscript.yaml
Click: Run workflow
Complete the form and click "Run workflow"
A successful run should result in a new pull request at https://github.com/elifesciences/enhanced-preprints-client/pulls
Open the pull request and click the "Ready for review" button to trigger tests
Once the tests pass and you are happy with the changes the PR can be merged

Example pull request: https://github.com/elifesciences/enhanced-preprints-client/pull/397/files
Once the pull request is merged in it should be available a few minutes later.

Step 9: Done!

[ ] Kettle is on!

elocations missing from references


Control Structures Queries

args
binary
do
branch
for_each
error
default
expr
js
coerce_to
type_of
info
json
http
uuid
These are all new, except for branch which has an early implementation along with a lambda mapping. Other functions might also need a lambda mapping.
expr might be internal only, since we are going to allow native elixir data types everywhere.
Usual specs/tests/docs

do is a keyword. Additionally, the spec says it accepts FUNCTION(*), DATUM... Which is pretty hard to handle in Elixir. The current implementation is to take one data arg and one function. I think we should push weird stuff to the user by requiring them to use args.

expr not necessary since we have a prefix driver (expr basically just wraps an existing data object so it can be used as an infix driver)

Finished and pushed.
Most of the functions here are weird. I would like to eliminate their direct usage as much has possible. Anytime they are used, I think we have an opportunity to make this driver better suited for Elixir.

How could you convert such query to Elixir code?
r.branch(r.table("users").getAll(email, {index: "email"}).isEmpty(),
         r.table("users").insert(user),
         {})


Costly message caused by tokens exhausted limits breach for large messages.

Issue
Sent: a costly message with a big file.
Response:
Tokens: 1663k sent, 0 received. Cost: $4.99 message, $9.68 session.

Model claude-3-5-sonnet-20240620 has hit a token limit!
Token counts below are approximate.

Input tokens: ~1,663,395 of 200,000 -- possibly exhausted context window!
Output tokens: ~0 of 8,192
Total tokens: ~1,663,395 of 200,000 -- possibly exhausted context window!

A previously sent message with a larger file resulted in the following error:
BadRequestError: litellm.BadRequestError: AnthropicException - b'{"type":"error","error":{"type":"invalid_request_error","message":"too
many total text bytes: 9256098 > 9000000"}}'

The different api limits should be considered before sending a message, preventing limits breaching.
Maybe a pre-send validation hook could solve such a scenario.
Version and model info
Aider v0.58.1
Main model: claude-3-5-sonnet-20240620 with architect edit format, infinite output
Editor model: claude-3-5-sonnet-20240620 with editor-diff edit format
Weak model: claude-3-haiku-20240307

Thank you for filing this issue.
Please check if those costly messages really got billed. If the API returns an error, those requests should not get billed. Do not trust aider's cost estimation, it is (as stated) not reliable.
Also note that you can (and should) check your token usage before you send a prompt/request with /tokens


🐛Input type number allows for dashes -

Description

When input type is number, pressing dash (-) populates the field.
Same behavior happens with dots .
The reason is basically that HTML input type number allows for dots and dashes, however, I don't think it's useful in this component (people accidentally press - instead of 0 sometimes).

Expected behavior

- should not be valid to enter

Preview

Extra info
Version: "react-code-input": "^3.2.0",
Happy to open a PR!

@A-Tokyo  please feel free to contribute, thanx!


Change sample.env and generate install link in terminal

Hi,
I made some changes to the sample.env and edited next.config.js, so the install link for the dev store (see .env) is generated and shown in terminal, if server is started in dev mode

I don't accept contributions to the repo but please feel free to fork and build on it!


dbt testsql capability to test whether models have valid sql in both incremental and full-refresh modes.

Describe the feature
dbt testsql capability to test whether models have valid sql in both incremental and full-refresh modes.
Describe alternatives you've considered
right now as part of testing during development I start running the model in incremental mode and full-refresh mode on a small warehouse and then hit ctrl-C to stop the model running both times and rollback any transactions
Additional context
the method above isn't great as it's very manual and you can only test one model at a time. Also if a model runs too fast then it can easily commit before you can cancel the run!
I would imagine that it would work as follows:
dbt testsql -m model1 model2
and it will do a 10/20 second run (or other way of establishing the model hasn't thrown a db error) and rollback for each model in both incremental and full-refresh modes, because it automatically rolls back it doesn't matter if the model completes or not.
Who will this benefit?
This will benefit anyone who is trying to test new models or model changes.

Hey @djayatillake - cool suggestion! I don't think we'd want to implement exactly this, but I do like where your head is at. One of the challenges here is that dbt can't generate all of the possible permutations of a model -- if you did something like:
select * from {{ ref(model) }}

{% if modules.datetime.datetime.now().isoweekday() == 4 %}
where 1 / 0
{% endif %}

this models would fail to build.... but only on Thursdays! As such, it's not really possible or desirable for dbt to try to make a claim like "this model's code passes regardless of the supplied configs (eg. if --full-refresh is present or not).
Instead, what do you think of something like https://github.com/fishtown-analytics/dbt/issues/401? Now that Snowflake has explain capabilities, I think it would be awesome if we could do something like:
dbt run --explain -m model1 model2

This would run an explain plan on the specified models. If they contain syntax errors or invalid table references, then the explain plan should fail right away. Note that this would not help in cases like:
select 1 / 0

as it's a valid SQL statement that would encounter a runtime error.
I'm going to close this one out with a wontfix label as I don't imagine making the code change you've described exactly here, but I like the idea and I'm keep to discuss further in the comments below! Let me know what you think


Update aadAuthorizationEndpoint

ADAL 4.x does not accept https://login.windows.net/common/oauth2/authorize as a valid auth endpoint (see https://github.com/AzureAD/azure-activedirectory-library-for-dotnet/issues/1346). This PR updates DailyReporterPro and EmbeddedLab to use https://login.microsoftonline.com/common instead.
Note: this has not been tested with ADAL < 4.3, which may not work with this endpoint.

Thanks, I just ran into this on another project when moving from ADAL.NET 3.18 to 4.3. I found the same solution as you did.
Thanks


VTS L1 Enhancement - Test Profile Changes - Device Setting - Host

This task is to modify the L1 concerning the latest Test profile changes if any. If there are no changes concerning the test profile, this Jira can be closed with proper information
interface file:
https://github.com/rdkcentral/rdk-halif-device_settings/blob/develop/include/dsHost.h
L1:
https://github.com/rdkcentral/rdk-halif-test-device_settings/blob/develop/src/test_l1_dsHost.c

No changes in the interface file.  No need to address this task.

No changes required as all the API's are getting values from lower lever and we are not setting any values from yaml file.


add statement_timeout to db template

record the config we are using to timeout long sql queries at the db connection layer
I'm going to update our system database.yaml to include this config. The plan is to set the ENV variable for API nodes to just longer than the load balancer (60s). As the loadbalancer will terminate connections that surpass the timeout, we can safely cancel the db queries as the results won't make it to the client.
The 5min (300000ms) value here is for sidekiq and should handle any long running queries in this env. This default can be changed via ENV vars if needed.
Review checklist

[ ] First, the most important one: is this PR small enough that you can actually review it? Feel free to just reject a branch if the changes are hard to review due to the length of the diff.
[ ] If there are any migrations, will they the previous version of the app work correctly after they've been run (e.g. the don't remove columns still known about by ActiveRecord).
[ ] If anything changed with regards to the public API, are those changes also documented in the apiary.apib file?
[ ] Are all the changes covered by tests? Think about any possible edge cases that might be left untested.

this needs a rebase over #3279 to build correctly as a gem was yanked :(


Suggestion: Online players graph

Hi,
Just a small little suggestion, would it be possible to have an online players graph with time stamps added to the panel in the future?
Thanks,
- James

For now, as I understand, the stats Query haves been removed from the panel since there is no fully featured and maintained library to fetch thoses datas.
Ptero dev team will have to rewrite one I think, and this will take a lil bit more time :p

ref #98


Spigot server requires server.jar to exist before server is started

This may be a redundant or known issue, and I apologize if that is the case.
It seems like when selecting the Minecraft Spigot option, it requires a server.jar to already exist in the directory prior to the server being started, otherwise it won't start. I went online and downloaded a Spigot server.jar and simply upload this whenever a Spigot server is requested, but obviously it would be better if I didn't need to do this every time I spin up the server.

Closed as this is fixed by packs and better docker images.


Add integration test for WebDAV ImageServer and ImageInspector.Inspect() method

This PR adds Integration tests will attempt to pull and scan an image without throwing an error. In order to accomplish this with our CI (travis), travis now runs tests inside a docker container (as travis does not support centos-based builds).
The PR also adds a set of integration tests for the WebDAV server (ensures http endpoints behave as expected).
the success of docker build will now also be tested by travis.
Additional changes:

changes to some constant names
ImageServer interface now implements GetHandler which returns an http.Handler rather than ServeImage, allowing the image server to be tested, as well as for middleware to be more easily added to the http Handler chain.

The additional tests are implemented with the Ginkgo Test Framework. Ginkgo tests are compatible with go test

requesting review from @pweil- @mfojtik

@enoodle help with review? I think it's important we merge this to help prevent regressions & provide confidence when making changes

@enoodle done :) although i think the test should be renamed from "e2e" since they are more like unit tests for the CLI

@ilackarms if you can split out refactoring (e.g. variable names, mux.HandleFunc, etc.) I think it will be easier to review/merge.

I can split this PR into two; the original intent was to a bit of code cleanup. I'll keep this PR focused on testing, and move the refactoring to another branch

@simon3z i'd like to rebase this branch off of #64 instead of completely removing the refactor changes; they're necessary for webdav_test.go
is that alright?

closing this PR; see #64 (refactoring) and #65 (testing)


Fixed responsibility and color combination

Fixed all mobile responsibility. but....
BUG FOUND!! AND COULDN'T BE FIXED!!!!!!
-> Where??
-> In desktop mode drop down is not working. But in mobile mode its working just fine. Please check if you find the bug.

Good-job Dada :>
thanks a lot :)


Update name and links sailfishos-chum→sailfishos-chum-repo-config

https://build.sailfishos.org/package/show/sailfishos:chum/sailfishos-chum-gui is back online and we don't have to change it.

@mentaljam, can you please deploy this (i.e., current state of main branch) at chumrpm.netlify.app.
Or is anybody else able to do this?

WIP

Done!


sync types and languages with spec

closes #68
there are 77 types now in the spec, too many to keep in sync manually
for example, i noticed that norwegian and thai were missing from this list
this PR uses NPM to keep the types in sync
preserves extra OIAA types and languages not approved yet in the spec
also it drops the various UI translations that aren't supported yet by the spec, which most people aren't aware of anyway. still supported: english, spanish, french, dutch, slovak, swedish, japanese, portuguese

Cool cheers Josh - I’ll rebuild and push today
Appreciate your service
We help businesses develop integrated ecommerce digital solutions
Matthew Bylett
Founder Think New Ltd
022 029 2881
On Sun, 6 Oct 2024 at 5:07 AM, Josh Reisner @.***>
wrote:

closes #68
https://github.com/code4recovery/online-meeting-list/issues/68
there are 77 types now in the spec, too many to keep in sync manually
for example, i noticed that norwegian and thai were missing from this list
this PR uses NPM to keep the types in sync
also it drops the various UI translations that aren't supported yet by the
spec, which most people aren't aware of anyway. still supported: english,
spanish, french, dutch, slovak, swedish, japanese, portuguese
You can view, comment on, or merge this pull request online at:
https://github.com/code4recovery/online-meeting-list/pull/69
Commit Summary

5bdc238
https://github.com/code4recovery/online-meeting-list/pull/69/commits/5bdc238e977941b1a968fb40d4cd4c5e0c5558e6
sync types to spec

File Changes
(28 files
https://github.com/code4recovery/online-meeting-list/pull/69/files)

M package-lock.json
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-053150b640a7ce75eff69d1a22cae7f0f94ad64ce9a855db544dda0929316519
(11)
M package.json
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519
(1)
M src/helpers/config.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-b133de3707a2586076b97e4901fad59fbc6ea2f4a695c0b7885175855376dd01
(35)
D src/languages/am.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-b7091a4a90d64592a5e17769fbd1bcc8ae922a3314c28e6c41d7634547bbd601
(126)
D src/languages/bg.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-1221f200e2b4fc0eb51451b54cc5c997ec3dad31fb69a50829a3816396851def
(134)
D src/languages/da.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-dc3b6a0a0eee4ee69e906cde49b70e11e20cf61646d39d062283450dd0169beb
(134)
D src/languages/de.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-f0aefb75717f0fc8d18287fe8c13c9a1811436d7a4789cd2d8b11c166ef35ecf
(136)
D src/languages/el.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-6ed79ddf2a08622f7498db56fb129bed67ed8b0e357558cff44fdf8567577c57
(135)
M src/languages/en.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-2a9066512656fa29b1565e362127de5cc11ba95b28f143a9db26f76dc3750785
(93)
M src/languages/es.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-472528060691374654a2e2e866547d385eec3849efddca38cbfca1e5b3ba47d5
(93)
D src/languages/fa.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-bfb41e13d0f0921bb2732a91152fa470e453d2b275ea1b70823227d5b0a85a08
(135)
M src/languages/fr.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-c7c8cf8bbbc6135f2a6a4477774099c80a89be2acc7dd1db881c5c1769f7b019
(93)
D src/languages/hi.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-5a39d63092df98ec6911a53c932024d933921eb1a9447d4b3686df291b7295ff
(135)
D src/languages/hu.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-9aa2b2c4444d61343779229ac8ddb954f79cacfacee768772a8d1c06a37bf052
(135)
M src/languages/index.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-e12c368ec619330eb1b73933561a18038f1d771cfec0c78d8e3242ed1a6730b4
(17)
D src/languages/it.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-1dac1e07587ff8cc7181e11bbf7752472c8d4edc6fc6af40441836b4f68572ec
(134)
M src/languages/ja.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-12bffcf624f03fc8a28a69a1c7891dcf19bc7f9484038ca37106b98e02faf371
(93)
D src/languages/ko.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-d41e7c1d41572dbd3ad549abf1f281d79130a35dd90d3fc6476e8f5fbddfc4cb
(134)
D src/languages/mt.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-2ead3e341bb756fa704e371913ac6ab6d20dbe6f7fe6de120435c32d794b1c59
(134)
D src/languages/ne.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-5b7e88a1c8f3a66691939e6442bcb9b830f2f1f5787c0beb43eaec45905d0fae
(135)
A src/languages/nl.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-9a23046a029944e53daf037493531a530603ce3497412d82ca6940fbbcfe4c16
(78)
D src/languages/pa.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-3b6b7bd8b63607dd32af7f37d4dd02f764ff88a0f96892c1c7e40aa5ae4d50f6
(134)
D src/languages/pl.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-cce33ee9e654389363b894a40c6c88765d5c71edfd18db4270656887c6744aab
(134)
M src/languages/pt.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-d5a0b0d68824e62409a22d4a50c06a23137869bb3d8f8b1188a2e518aed313d3
(93)
D src/languages/ru.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-f9ddb58877d9303bc6f3b0ef22693c566752bdedf3e01e259c435d0a396f7571
(135)
M src/languages/sk.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-228093fe7f9c179b0bdb9921937e9d70e9099d87c2be4e6ac19134833026555c
(93)
M src/languages/sv.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-2c4ecfb6c2ec81a19765db32c67a230b93769455e64e5cea678969f25d1ed643
(93)
D src/languages/sw.ts
https://github.com/code4recovery/online-meeting-list/pull/69/files#diff-08b9a6eb5ed4d1127d8c1e25ed9a9b3d3a2e0d9967773f8eb03be95d0a51b4cc
(135)

Patch Links:

https://github.com/code4recovery/online-meeting-list/pull/69.patch
https://github.com/code4recovery/online-meeting-list/pull/69.diff

—
Reply to this email directly, view it on GitHub
https://github.com/code4recovery/online-meeting-list/pull/69, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AHNVISIG7U5FR6IFWC4BMZDZ2AFFDAVCNFSM6AAAAABPNOXONKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGU3DQMJZHA3TCNQ
.
You are receiving this because your review was requested.Message ID:
@.***>


Add constraints to update

add constraint to updateMessage in webController, only users with role ROLE_ADMIN can update any message, otherwise only the owner of the message can edit the message. closes #82

it works for me, it takes me back to messagelist if i try to change message id
but here is conflicts u have to resolve


ARC-36 add await to installation update

This is a secondary (and hopefully final) PR to fix the flaky test in the app. Turns out, we weren't awaiting update in the install method so our super speedy tests were receiving the returned valued for the existing subscription instead of the renamed one.

@tebriel
@rachellerathbone and I found the problem this morning while pairing.  Essentially it comes down to the tests being too fast and the Installation.install function not waiting for the subscriptions to be done updating before continuing.  Definitely a tricky one to solve :)


Can't get statistics after updating to v2.6.0

Describe the Bug
I have been using Umami in a Docker environment since 2.5.0. Everything was working comfortably until now, but after updating to 2.6.0, I can no longer get any statistics. From the logs, it looks like there is a fault in db, but the process itself is working fine.
The update was performed with docker-compose pull, followed by a restart with docker-compose down and docker-compose up -d.
It is possible that I may have made a careless error, but I would like to ask for a solution just in case.
Database
PostgreSQL
Relevant log output
umami-db-1     |
umami-db-1     |
umami-db-1     | PostgreSQL Database directory appears to contain a database; Skipping initialization
umami-db-1     |
umami-db-1     |
umami-db-1     | 2023-08-30 01:17:14.756 UTC [1] LOG:  starting PostgreSQL 15.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit
umami-db-1     | 2023-08-30 01:17:14.756 UTC [1] LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
umami-db-1     | 2023-08-30 01:17:14.756 UTC [1] LOG:  listening on IPv6 address "::", port 5432
umami-db-1     | 2023-08-30 01:17:14.758 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
umami-db-1     | 2023-08-30 01:17:14.762 UTC [23] LOG:  database system was shut down at 2023-08-30 01:16:58 UTC
umami-db-1     | 2023-08-30 01:17:14.769 UTC [1] LOG:  database system is ready to accept connections
umami-db-1     | 2023-08-30 01:17:22.336 UTC [35] ERROR:  relation "account" does not exist at character 15
umami-db-1     | 2023-08-30 01:17:22.336 UTC [35] STATEMENT:  select * from account limit 1
umami-db-1     | 2023-08-30 01:17:23.582 UTC [37] LOG:  could not receive data from client: Connection reset by peer
umami-umami-1  | yarn run v1.22.19
umami-umami-1  | $ npm-run-all check-db update-tracker start-server
umami-umami-1  | $ node scripts/check-db.js
umami-umami-1  | ✓ DATABASE_URL is defined.
umami-umami-1  | ✓ Database connection successful.
umami-umami-1  | ✓ Database version check successful.
umami-umami-1  | Prisma schema loaded from prisma/schema.prisma
umami-umami-1  | Datasource "db": PostgreSQL database "umami", schema "public" at "db:5432"
umami-umami-1  |
umami-umami-1  |
umami-umami-1  | 3 migrations found in prisma/migrations
umami-umami-1  |
umami-umami-1  |
umami-umami-1  |
umami-umami-1  |
umami-umami-1  | No pending migrations to apply.
umami-umami-1  |
umami-umami-1  |
umami-umami-1  | ✓ Database is up to date.
umami-umami-1  | $ node scripts/update-tracker.js
umami-umami-1  | $ node server.js
umami-umami-1  | - ready started server on <IP_ADDRESS>:3000, url: http://<IP_ADDRESS>:3000
umami-umami-1  | TypeError: Cannot destructure property 'url' of 'payload' as it is undefined.
umami-umami-1  |     at __WEBPACK_DEFAULT_EXPORT__ (/app/.next/server/pages/api/send.js:502:13)
umami-umami-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
umami-db-1     | 2023-08-30 01:22:14.860 UTC [21] LOG:  checkpoint starting: time
umami-db-1     | 2023-08-30 01:22:14.873 UTC [21] LOG:  checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.002 s, total=0.014 s; sync files=2, longest=0.001 s, average=0.001 s; distance=0 kB, estimate=0 kB

Which browser are you using? (if relevant)
No response
How are you deploying your application? (if relevant)
Docker

TypeError: Cannot destructure property 'url' of 'payload' as it is undefined.
Wondering what this is. Looking into it. Is the container stopping at all, or does the app still run, but no stats?

I reverted the version used back to v2.5.0 as a test and got the statistics successfully. but, there is no particular change in the content of the logs.

I too ran into this (temporarily and successfully reverted to v2.5.0) and am also using the PostgreSQL variant, in case there might be something specific to that going on.

I also noticed that no visits were recorded after upgrading :(
I reverted to 2.5.0 and it's now working. In my case it gave a 404 for the endpoint path.

@vitobotta with 2.6 the endpoint script was renamed from umami.js to script.js. If you don't want to update your websites to use the new script name you can set an environment variable in the docker-compose.prod.yml file to have the old script name served:
TRACKER_SCRIPT_NAME=umami.js

We have a fix for this, and will do another release soon (v2.6.1).

Is the aforementioned TRACKER_SCRIPT_NAME breaking change still part of v2.6.1, too? I didn't see any mention of it in the release notes of either release, so I'm not quite sure if this change still needs to be made to ensure all sites keep working as they did.
Thanks so much!

Updated to v2.6.1.
I think the problem is probably solved.
Thanks a lot!

I upgraded again but what's not working for me is not the script, which I already had renamed to script.js. It's the custom endpoint name set with the env variable COLLECT_API_ENDPOINT. If I unset that then it works. But I want to customise the name in case Umami is added to adblocker lists.

This applies to both env vars  https://github.com/umami-software/umami/issues/2252#issuecomment-1700414659
@jibsaramnim @vitobotta

Fixed in v2.6.2.


Create a stack through a compose file

I want to use the api to create a stack using docker compose file.
Basically trying to achieve through golang application docker stack deploy --config-file docker-compose.yml stackdemo passing in the authorization token.
I see https://github.com/docker/stacks/pull/21/files was merged and had something related to read and convert the compose file to spec necessary for stack. But that code is no longer there.
Why was it removed?
How can achieve creation of a stack using docker compose file.

Hi, this is not the docker stack repo, this is some old experimental code.


Why some form helper functions works only with forms sent with POST method?

Why set_value, set_select, set_checkbox and set_radio  works only with forms sent with POST method?

This is a bug tracker.
If you want to ask a question, do it on our forums.


Find all references permanently hangs while checking for references with the method declaration file open with calls outside of that file.

VSCode Version: Code 1.10.1 (653f8733dd5a5c43d66d7168b4701f94d72b62e5, 2017-03-02T00:27:09.350Z)
OS Version: Darwin x64 16.5.0
Extensions:

Extension
Author
Version

empty-indent
DmitryDorofeev
0.2.0

beautify
HookyQR
0.8.8

minify
HookyQR
0.3.0

vscode-paste-and-indent
Rubymaniac
0.0.8

html-snippets
abusaidm
0.1.0

vscode-theme-onedark
akamud
1.3.1

project-manager
alefragnani
0.13.2

path-intellisense
christian-kohler
1.2.0

githistory
donjayamanne
0.1.5

phpunit
emallin
1.0.0

php-debug
felixfbecker
1.10.0

php-intellisense
felixfbecker
1.1.3

auto-close-tag
formulahendry
0.3.9

code-runner
formulahendry
0.6.13

vs-code-phpunit-snippets
jameshrisho
0.0.2

theme-onedark-sublime
joshpeng
0.3.6

indent-rainbow
oderwat
0.4.1

Steps to Reproduce:

Open a php file
Call a method from a class outside of the class file
Find all references for that method while inside that class file.
It hangs, doesn't find the call you created in 2.

same version
C/C++ Find all references can not find the result, but not hangs, no matter inside or outside the file

This must have been using php-intellisense? If it's still an issue, file on that extension.


Question regarding position requirements

Hello, does this position requires knowledge in JavaScript and html?
I am asking because I did not mention in my CV I write JavaScript and the assignment is JavaScript based.
Thanks in advance.

The position is for a client developer, we do not require previous experience, but you are required to manage with this web related assignment, so some knowledge is required.