Job stuck in reduce phase

I'm running mnist data but get some problem.
My cluster has one master node and 6 data nodes, each data node with 128GB memory, and caffeOnSpark is running under CPU mode.
The job runs well for executor = 1, but it would get stuck in the reduce phase if I increase executor number, as shown below

For executor = 1 the UI turn out like this:

It finished success and show me some results, but the reduces are skipped.
The log where the program got stuck was like below
I1116 14:51:57.595015 12024 layer_factory.hpp:77] Creating layer data
I1116 14:51:57.595031 12024 net.cpp:100] Creating Layer data
I1116 14:51:57.595053 12024 net.cpp:408] data -> data
I1116 14:51:57.595067 12024 net.cpp:408] data -> label
I1116 14:51:57.595149 12024 net.cpp:150] Setting up data
I1116 14:51:57.595165 12024 net.cpp:157] Top shape: 100 1 28 28 (78400)
I1116 14:51:57.595181 12024 net.cpp:157] Top shape: 100 (100)
I1116 14:51:57.595190 12024 net.cpp:165] Memory required for data: 314000
I1116 14:51:57.595197 12024 layer_factory.hpp:77] Creating layer label_data_1_split
I1116 14:51:57.595208 12024 net.cpp:100] Creating Layer label_data_1_split
I1116 14:51:57.595216 12024 net.cpp:434] label_data_1_split <- label
I1116 14:51:57.595227 12024 net.cpp:408] label_data_1_split -> label_data_1_split_0
I1116 14:51:57.595239 12024 net.cpp:408] label_data_1_split -> label_data_1_split_1
I1116 14:51:57.595252 12024 net.cpp:150] Setting up label_data_1_split
I1116 14:51:57.595263 12024 net.cpp:157] Top shape: 100 (100)
I1116 14:51:57.595283 12024 net.cpp:157] Top shape: 100 (100)
I1116 14:51:57.595291 12024 net.cpp:165] Memory required for data: 314800
I1116 14:51:57.595299 12024 layer_factory.hpp:77] Creating layer conv1
I1116 14:51:57.595315 12024 net.cpp:100] Creating Layer conv1
I1116 14:51:57.595324 12024 net.cpp:434] conv1 <- data
I1116 14:51:57.595351 12024 net.cpp:408] conv1 -> conv1
I1116 14:51:57.595386 12024 net.cpp:150] Setting up conv1
I1116 14:51:57.595397 12024 net.cpp:157] Top shape: 100 20 24 24 (1152000)
I1116 14:51:57.595404 12024 net.cpp:165] Memory required for data: 4922800
I1116 14:51:57.595418 12024 layer_factory.hpp:77] Creating layer pool1
I1116 14:51:57.595429 12024 net.cpp:100] Creating Layer pool1
I1116 14:51:57.595437 12024 net.cpp:434] pool1 <- conv1
I1116 14:51:57.595446 12024 net.cpp:408] pool1 -> pool1
I1116 14:51:57.595460 12024 net.cpp:150] Setting up pool1
I1116 14:51:57.595470 12024 net.cpp:157] Top shape: 100 20 12 12 (288000)
I1116 14:51:57.595477 12024 net.cpp:165] Memory required for data: 6074800
I1116 14:51:57.595484 12024 layer_factory.hpp:77] Creating layer conv2
I1116 14:51:57.595497 12024 net.cpp:100] Creating Layer conv2
I1116 14:51:57.595505 12024 net.cpp:434] conv2 <- pool1
I1116 14:51:57.595515 12024 net.cpp:408] conv2 -> conv2
I1116 14:51:57.595755 12024 net.cpp:150] Setting up conv2
I1116 14:51:57.595768 12024 net.cpp:157] Top shape: 100 50 8 8 (320000)
I1116 14:51:57.595777 12024 net.cpp:165] Memory required for data: 7354800
I1116 14:51:57.595789 12024 layer_factory.hpp:77] Creating layer pool2
I1116 14:51:57.595799 12024 net.cpp:100] Creating Layer pool2
I1116 14:51:57.595808 12024 net.cpp:434] pool2 <- conv2
I1116 14:51:57.595816 12024 net.cpp:408] pool2 -> pool2
I1116 14:51:57.595830 12024 net.cpp:150] Setting up pool2
I1116 14:51:57.595840 12024 net.cpp:157] Top shape: 100 50 4 4 (80000)
I1116 14:51:57.595849 12024 net.cpp:165] Memory required for data: 7674800
I1116 14:51:57.595855 12024 layer_factory.hpp:77] Creating layer ip1
I1116 14:51:57.595866 12024 net.cpp:100] Creating Layer ip1
I1116 14:51:57.595875 12024 net.cpp:434] ip1 <- pool2
I1116 14:51:57.595887 12024 net.cpp:408] ip1 -> ip1
I1116 14:51:57.600900 12024 net.cpp:150] Setting up ip1
I1116 14:51:57.600922 12024 net.cpp:157] Top shape: 100 500 (50000)
I1116 14:51:57.600930 12024 net.cpp:165] Memory required for data: 7874800
I1116 14:51:57.600945 12024 layer_factory.hpp:77] Creating layer relu1
I1116 14:51:57.600957 12024 net.cpp:100] Creating Layer relu1
I1116 14:51:57.600966 12024 net.cpp:434] relu1 <- ip1
I1116 14:51:57.600976 12024 net.cpp:395] relu1 -> ip1 (in-place)
I1116 14:51:57.600988 12024 net.cpp:150] Setting up relu1
I1116 14:51:57.600997 12024 net.cpp:157] Top shape: 100 500 (50000)
I1116 14:51:57.601004 12024 net.cpp:165] Memory required for data: 8074800
I1116 14:51:57.601012 12024 layer_factory.hpp:77] Creating layer ip2
I1116 14:51:57.601024 12024 net.cpp:100] Creating Layer ip2
I1116 14:51:57.601032 12024 net.cpp:434] ip2 <- ip1
I1116 14:51:57.601043 12024 net.cpp:408] ip2 -> ip2
I1116 14:51:57.601111 12024 net.cpp:150] Setting up ip2
I1116 14:51:57.601122 12024 net.cpp:157] Top shape: 100 10 (1000)
I1116 14:51:57.601130 12024 net.cpp:165] Memory required for data: 8078800
I1116 14:51:57.601140 12024 layer_factory.hpp:77] Creating layer ip2_ip2_0_split
I1116 14:51:57.601151 12024 net.cpp:100] Creating Layer ip2_ip2_0_split
I1116 14:51:57.601166 12024 net.cpp:434] ip2_ip2_0_split <- ip2
I1116 14:51:57.601177 12024 net.cpp:408] ip2_ip2_0_split -> ip2_ip2_0_split_0
I1116 14:51:57.601189 12024 net.cpp:408] ip2_ip2_0_split -> ip2_ip2_0_split_1
I1116 14:51:57.601202 12024 net.cpp:150] Setting up ip2_ip2_0_split
I1116 14:51:57.601212 12024 net.cpp:157] Top shape: 100 10 (1000)
I1116 14:51:57.601222 12024 net.cpp:157] Top shape: 100 10 (1000)
I1116 14:51:57.601228 12024 net.cpp:165] Memory required for data: 8086800
I1116 14:51:57.601235 12024 layer_factory.hpp:77] Creating layer accuracy
I1116 14:51:57.601246 12024 net.cpp:100] Creating Layer accuracy
I1116 14:51:57.601254 12024 net.cpp:434] accuracy <- ip2_ip2_0_split_0
I1116 14:51:57.601263 12024 net.cpp:434] accuracy <- label_data_1_split_0
I1116 14:51:57.601274 12024 net.cpp:408] accuracy -> accuracy
I1116 14:51:57.601286 12024 net.cpp:150] Setting up accuracy
I1116 14:51:57.601296 12024 net.cpp:157] Top shape: (1)
I1116 14:51:57.601303 12024 net.cpp:165] Memory required for data: 8086804
I1116 14:51:57.601311 12024 layer_factory.hpp:77] Creating layer loss
I1116 14:51:57.601321 12024 net.cpp:100] Creating Layer loss
I1116 14:51:57.601330 12024 net.cpp:434] loss <- ip2_ip2_0_split_1
I1116 14:51:57.601337 12024 net.cpp:434] loss <- label_data_1_split_1
I1116 14:51:57.601347 12024 net.cpp:408] loss -> loss
I1116 14:51:57.601359 12024 layer_factory.hpp:77] Creating layer loss
I1116 14:51:57.601379 12024 net.cpp:150] Setting up loss
I1116 14:51:57.601389 12024 net.cpp:157] Top shape: (1)
I1116 14:51:57.601397 12024 net.cpp:160]     with loss weight 1
I1116 14:51:57.601408 12024 net.cpp:165] Memory required for data: 8086808
I1116 14:51:57.601415 12024 net.cpp:226] loss needs backward computation.
I1116 14:51:57.601424 12024 net.cpp:228] accuracy does not need backward computation.
I1116 14:51:57.601433 12024 net.cpp:226] ip2_ip2_0_split needs backward computation.
I1116 14:51:57.601439 12024 net.cpp:226] ip2 needs backward computation.
I1116 14:51:57.601447 12024 net.cpp:226] relu1 needs backward computation.
I1116 14:51:57.601454 12024 net.cpp:226] ip1 needs backward computation.
I1116 14:51:57.601461 12024 net.cpp:226] pool2 needs backward computation.
I1116 14:51:57.601469 12024 net.cpp:226] conv2 needs backward computation.
I1116 14:51:57.601475 12024 net.cpp:226] pool1 needs backward computation.
I1116 14:51:57.601483 12024 net.cpp:226] conv1 needs backward computation.
I1116 14:51:57.601491 12024 net.cpp:228] label_data_1_split does not need backward computation.
I1116 14:51:57.601500 12024 net.cpp:228] data does not need backward computation.
I1116 14:51:57.601506 12024 net.cpp:270] This network produces output accuracy
I1116 14:51:57.601514 12024 net.cpp:270] This network produces output loss
I1116 14:51:57.601531 12024 net.cpp:283] Network initialization done.
I1116 14:51:57.601585 12024 solver.cpp:60] Solver scaffolding done.
I1116 14:51:57.601702 12024 socket.cpp:221] Waiting for valid port [0]
I1116 14:51:57.601812 12041 socket.cpp:162] Assigned socket server port [50184]
I1116 14:51:57.602773 12041 socket.cpp:175] Socket Server ready [<IP_ADDRESS>]
I1116 14:51:57.611819 12024 socket.cpp:221] Waiting for valid port [50184]
I1116 14:51:57.611841 12024 socket.cpp:229] Valid port found [50184]
I1116 14:51:57.611863 12024 CaffeNet.cpp:262] Socket adapter: longzhou-hdp1.lz.dscc:50184
I1116 14:51:57.612177 12024 CaffeNet.cpp:402] 0-th Socket addr:
I1116 14:51:57.612201 12024 CaffeNet.cpp:402] 1-th Socket addr: longzhou-hdp1.lz.dscc:50184
I1116 14:51:57.612211 12024 CaffeNet.cpp:402] 2-th Socket addr: longzhou-hdp1.lz.dscc:50184
I1116 14:51:57.612221 12024 CaffeNet.cpp:402] 3-th Socket addr: longzhou-hdp1.lz.dscc:50184
I1116 14:51:57.612236 12024 JniCaffeNet.cpp:145] 0-th local addr:
I1116 14:51:57.612246 12024 JniCaffeNet.cpp:145] 1-th local addr: longzhou-hdp1.lz.dscc:50184
I1116 14:51:57.612253 12024 JniCaffeNet.cpp:145] 2-th local addr: longzhou-hdp1.lz.dscc:50184
I1116 14:51:57.612262 12024 JniCaffeNet.cpp:145] 3-th local addr: longzhou-hdp1.lz.dscc:50184
16/11/16 14:51:57 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 8). 1018 bytes result sent to driver
16/11/16 14:51:58 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 14
16/11/16 14:51:58 INFO executor.Executor: Running task 2.0 in stage 3.0 (TID 14)
16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 4
16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1584.0 B, free 18.7 KB)
16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Reading broadcast variable 4 took 12 ms
16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 2.6 KB, free 21.3 KB)
16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 3
16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 170.0 B, free 21.4 KB)
16/11/16 14:51:58 INFO broadcast.TorrentBroadcast: Reading broadcast variable 3 took 7 ms
16/11/16 14:51:58 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 1536.0 B, free 22.9 KB)
I1116 14:51:58.208997 12024 common.cpp:61] 0-th string is NULL
I1116 14:51:58.209087 12024 socket.cpp:257] Trying to connect with ...[longzhou-hdp3.lz.dscc:56137]
I1116 14:51:58.209530 12024 socket.cpp:321] Connected to server [longzhou-hdp3.lz.dscc:56137] with client_fd [374]
I1116 14:51:58.228672 12041 socket.cpp:188] Accepted the connection from client [longzhou-hdp2.lz.dscc]
I1116 14:51:58.230128 12041 socket.cpp:188] Accepted the connection from client [longzhou-hdp4.lz.dscc]
I1116 14:51:58.276432 12041 socket.cpp:188] Accepted the connection from client [longzhou-hdp3.lz.dscc]
I1116 14:51:59.209713 12024 socket.cpp:257] Trying to connect with ...[longzhou-hdp2.lz.dscc:51049]
I1116 14:51:59.210247 12024 socket.cpp:321] Connected to server [longzhou-hdp2.lz.dscc:51049] with client_fd [378]
I1116 14:52:00.210403 12024 socket.cpp:257] Trying to connect with ...[longzhou-hdp4.lz.dscc:52620]
I1116 14:52:00.210934 12024 socket.cpp:321] Connected to server [longzhou-hdp4.lz.dscc:52620] with client_fd [294]
16/11/16 14:52:01 INFO caffe.CaffeProcessor: Interleave enabled
I1116 14:52:01.218786 12048 CaffeNet.cpp:633] Interleaved
I1116 14:52:01.218842 12048 MemoryInputAdapter.cpp:15] MemoryInputAdapter is used
I1116 14:52:01.218858 12048 MemoryInputAdapter.cpp:15] MemoryInputAdapter is used
16/11/16 14:52:01 INFO caffe.CaffeProcessor: Start transformer for train in CaffeProcessor StartThreads
16/11/16 14:52:01 INFO caffe.CaffeProcessor: Start transformer for validation in CaffeProcessor StartThreads
16/11/16 14:52:01 INFO executor.Executor: Finished task 2.0 in stage 3.0 (TID 14). 908 bytes result sent to driver
16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 16
16/11/16 14:52:01 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 16)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 5
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 25.0 KB)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 5 took 10 ms
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.3 KB, free 28.3 KB)
16/11/16 14:52:01 INFO storage.BlockManager: Found block rdd_1_0 locally
16/11/16 14:52:01 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 16). 2830 bytes result sent to driver
16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 17
16/11/16 14:52:01 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 17)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 6
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2.0 KB, free 30.3 KB)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 6 took 10 ms
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.1 KB, free 33.5 KB)
16/11/16 14:52:01 INFO storage.BlockManager: Found block rdd_1_0 locally
16/11/16 14:52:01 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 17). 2016 bytes result sent to driver
16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 21
16/11/16 14:52:01 INFO executor.Executor: Running task 1.0 in stage 6.0 (TID 21)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 7
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.0 KB, free 35.5 KB)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 7 took 10 ms
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 3.1 KB, free 38.6 KB)
16/11/16 14:52:01 INFO storage.BlockManager: Found block rdd_3_1 locally
16/11/16 14:52:01 INFO executor.Executor: Finished task 1.0 in stage 6.0 (TID 21). 2015 bytes result sent to driver
16/11/16 14:52:01 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 25
16/11/16 14:52:01 INFO executor.Executor: Running task 2.0 in stage 7.0 (TID 25)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 8
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 1309.0 B, free 39.9 KB)
16/11/16 14:52:01 INFO broadcast.TorrentBroadcast: Reading broadcast variable 8 took 9 ms
16/11/16 14:52:01 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 2.0 KB, free 42.0 KB)
16/11/16 14:52:01 INFO executor.Executor: Finished task 2.0 in stage 7.0 (TID 25). 938 bytes result sent to driver
16/11/16 14:52:02 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 29
16/11/16 14:52:02 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 29)
16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 9
16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.5 KB, free 44.5 KB)
16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Reading broadcast variable 9 took 10 ms
16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 4.1 KB, free 48.6 KB)
16/11/16 14:52:02 INFO storage.BlockManager: Found block rdd_1_0 locally
16/11/16 14:52:02 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 29). 2211 bytes result sent to driver
16/11/16 14:52:02 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 31
16/11/16 14:52:02 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 31)
16/11/16 14:52:02 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 33
16/11/16 14:52:02 INFO executor.Executor: Running task 1.0 in stage 9.0 (TID 33)
16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 10
16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.6 KB, free 51.2 KB)
16/11/16 14:52:02 INFO broadcast.TorrentBroadcast: Reading broadcast variable 10 took 8 ms
16/11/16 14:52:02 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.4 KB, free 55.5 KB)
16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@<IP_ADDRESS>:2671)
16/11/16 14:52:02 INFO spark.MapOutputTrackerWorker: Got the output locations
16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Getting 4 non-empty blocks out of 4 blocks
16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Started 3 remote fetches in 15 ms
16/11/16 14:52:02 INFO storage.ShuffleBlockFetcherIterator: Started 3 remote fetches in 16 ms

And no more log for this container after it said "Started remote fetches"
Still my cluster runs well with other spark programs with reduce phase (not skipped).

Can you try disable the validation phase by setting the following in your solver.prototxt?
test_iter:0
test_interval: 0

@junshi15 Thanks this worked for me. So what's the reason behind and what could I do if I want to do the validation ? (though I still see some outputs like validation when finished after I set this two parameters to 0)

@junshi I am on the same boat here. Would you please be able to elaborate the reason for this problem? In particular, why disabling validation might help and any clue for a fix? Thanks!

Interleaving training and validation is quite delicate. The number of training batches and the number of validation batches have to be pre-computed, then interleaved. I suspect some of the numbers may not divide each other, hence hitting a corner case. To help us debug, please kindly provide your prototxt files, also the sizes of the train and validation dataset. I will have it looked after Thanksgiving.

@junshi15 thanks for reply.
I used the default lenet_memory_prototxts except the path setting and using the CPU mode.
As shown below
the solver prototxt:
# The train/test net protocol buffer definition
net: "lenet_memory_train_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
test_iter: 10
# Carry out testing every 100 training iterations.
test_interval: 100
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005
# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 2000
# snapshot intermediate results
snapshot: 5000
snapshot_prefix: "mnist_lenet"
# solver mode: CPU or GPU
solver_mode: CPU

the train_test prototxt:
name: "LeNet"
layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  source_class: "com.yahoo.ml.caffe.LMDB"
  memory_data_param {
    source: "hdfs:/user/zhanbohan/machine_learning/image_dataset/mnist_train_lmdb"
    batch_size: 64
    channels: 1
    height: 28
    width: 28
    share_in_parallel: false
  }
  transform_param {
    scale: 0.00390625
  }
}
layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  source_class: "com.yahoo.ml.caffe.LMDB"
  memory_data_param {
    source: "hdfs:/user/zhanbohan/machine_learning/image_dataset/mnist_test_lmdb/"
    batch_size: 100
    channels: 1
    height: 28
    width: 28
    share_in_parallel: true
  }
  transform_param {
    scale: 0.00390625
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 20
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 50
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "ip1"
  type: "InnerProduct"
  bottom: "pool2"
  top: "ip1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 500
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "ip1"
  top: "ip1"
}
layer {
  name: "ip2"
  type: "InnerProduct"
  bottom: "ip1"
  top: "ip2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 10
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "ip2"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "ip2"
  bottom: "label"
  top: "loss"
}

the submit script:
export CAFFE_ON_SPARK=/home/spark/distributed_ml/CaffeOnSpark

export SPARK_WORKER_INSTANCES=5
export DEVICES=1

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home//hadoop/hadoop/lib/native
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-7.0/lib64:/usr/local/mkl/lib/intel64/
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib

WORK_HOME=/home/zhanbohan/work/caffeOnSpark
memory=7g
hadoop fs -rm -f hdfs:///user/zhanbohan/caffe/mnist.model
hadoop fs -rm -r -f hdfs:///user/zhanbohan/caffe/mnist_features_result
spark-submit --master yarn --deploy-mode cluster \
        --num-executors ${SPARK_WORKER_INSTANCES} \
        --files ${WORK_HOME}/data/lenet_memory_solver.prototxt,${WORK_HOME}/data/lenet_memory_train_test.prototxt \
        --conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}" \
        --conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
        --class com.yahoo.ml.caffe.CaffeOnSpark  \
              ${WORK_HOME}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar \
        -train \
        -features accuracy,loss -label label \
        -conf lenet_memory_solver.prototxt \
        -devices ${DEVICES} \
        -connection ethernet \
        -model hdfs:///user/zhanbohan/caffe/mnist.model \
        -output hdfs:///user/zhanbohan/caffe/mnist_features_result
hadoop fs -ls hdfs:///user/zhanbohan/caffe/mnist.model
hadoop fs -cat hdfs:///user/zhanbohan/caffe/mnist_features_result/*

@junshi15 I download the cifar10 dataset and use the default settings run it with success. Maybe it's indeed the prototxt problem on mnist dataset.

@junshi15 cifar10 dataset again meetsthe same problem for instance=3, and runs well for instance=4,5 so far. For your locating the problem, I also provide the prototxt and submit script as below
cifar10_quick_solver.prototxt
# reduce the learning rate after 8 epochs (4000 iters) by a factor of 10

# The train/test net protocol buffer definition
net: "cifar10_quick_train_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
test_iter: 100
# Carry out testing every 100 training iterations.
test_interval: 100
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.001
momentum: 0.9
weight_decay: 0.004
# The learning rate policy
lr_policy: "fixed"
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 4000
# snapshot intermediate results
snapshot: 4000
#snapshot_format: HDF5
snapshot_prefix: "cifar10_quick"
# solver mode: CPU or GPU
solver_mode: CPU

cifar10_quick_train_test.prototxt
name: "CIFAR10_quick"
layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  source_class: "com.yahoo.ml.caffe.LMDB"
  memory_data_param {
    source: "hdfs:/user/zhanbohan/machine_learning/image_dataset/cifar10_train_lmdb/"
    batch_size: 100
    channels: 3
    height: 32
    width: 32
    share_in_parallel: false
  }
  transform_param {
    mean_file: "mean.binaryproto"
  }
}
layer {
  name: "data"
  type: "MemoryData"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  source_class: "com.yahoo.ml.caffe.LMDB"
  memory_data_param {
    source: "hdfs:/user/zhanbohan/machine_learning/image_dataset/cifar10_test_lmdb/"
    batch_size: 100
    channels: 3
    height: 32
    width: 32
    share_in_parallel: false
  }
  transform_param {
    mean_file: "mean.binaryproto"
  }

}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "data"
  top: "conv1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 32
    pad: 2
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "gaussian"
      std: 0.0001
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "pool1"
  top: "pool1"
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 32
    pad: 2
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: AVE
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "pool2"
  top: "conv3"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 64
    pad: 2
    kernel_size: 5
    stride: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "pool3"
  type: "Pooling"
  bottom: "conv3"
  top: "pool3"
  pooling_param {
    pool: AVE
    kernel_size: 3
    stride: 2
  }
}
layer {
  name: "ip1"
  type: "InnerProduct"
  bottom: "pool3"
  top: "ip1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 64
    weight_filler {
      type: "gaussian"
      std: 0.1
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "ip2"
  type: "InnerProduct"
  bottom: "ip1"
  top: "ip2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 10
    weight_filler {
      type: "gaussian"
      std: 0.1
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "ip2"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "ip2"
  bottom: "label"
  top: "loss"
}

submit script
export CAFFE_ON_SPARK=/home/spark/distributed_ml/CaffeOnSpark

export SPARK_WORKER_INSTANCES=3
export DEVICES=1

#export LD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home//hadoop/hadoop/lib/native
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-7.0/lib64:/usr/local/mkl/lib/intel64/
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib

WORK_HOME=/home/zhanbohan/work/caffeOnSpark
memory=7g
MODEL_PATH=hdfs:/user/zhanbohan/caffe/cifar10.model_3
RESULT_PATH=hdfs:/user/zhanbohan/caffe/cifar10_features_result_3
CONFIG_FILES=${WORK_HOME}/data/cifar10_quick_solver.prototxt,${WORK_HOME}/data/cifar10_quick_train_test.prototxt,${WORK_HOME}/data/mean.binaryproto

hadoop fs -rm -f $MODEL_PATH
hadoop fs -rm -r -f $RESULT_PATH
spark-submit --master yarn --deploy-mode cluster \
        --num-executors ${SPARK_WORKER_INSTANCES} \
        --files $CONFIG_FILES \
        --conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}" \
        --conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
        --class com.yahoo.ml.caffe.CaffeOnSpark  \
              ${WORK_HOME}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar \
        -train \
        -features accuracy,loss -label label \
        -conf cifar10_quick_solver.prototxt \
        -devices ${DEVICES} \
        -connection ethernet \
        -model $MODEL_PATH \
        -output $RESULT_PATH
hadoop fs -ls $MODEL_PATH
hadoop fs -cat $RESULT_PATH

Found the problem. Will send a fix soon. It has to do with even partition distribution to all executors.

@mriduljain great! looking forward to the patch

We need more information. Could you please run your testcase again by adding the following options to your  spark_submit command above i.e --executor-cores 1.
Please collect the following once your job starts running(i.e all executors have been allocated):

Screen shot of executors tab in your SparkUI - Need to see how many tasks show up per executor. It should be 1 task per executor
The complete yarn logs for the job using yarn logs -applicationId > log.txt. Do this at the end of the job run.

it ends successfully
here is the information required
spark UI executor tab:

the log: log.txt

You mean with executor-core 1 you don't see the failure?

In yarn mode, the default for driver and executor cores is 1. http://spark.apache.org/docs/latest/running-on-yarn.html#spark-properties
This is required for CaffeOnSpark to work properly. For some reason, it is not properly set in your env. Please check your spark config file and make sure it is set to 1 or provide the executor-core as 1 on spark-submit as suggested above.

ok I got this. I think it's better for me to take --executor-cores 1 every time.
Anyway thx and could I know why multiple cores would get stuck in reduce?

During any kind of training we need to make sure that every executor has only one task. executor-core takes care if this as it is essentially the max number of tasks per executor. If we have more than one task per executor, then we get into a situation where partition allocation happens based on whichever tasks are active at a particular time. So suppose spark needs to schedule 4 partitions and we have executor-core as 4; it can schedule all 4 partitions in 4 tasks(if they are ready/active) for an executor , leaving the other executors starving for partitions. As per the design of COS, all executors need to process their partitions in parallel and in sync with others. More tasks per executor creates an imbalance.

got it. Thx for ur help and answer.

Besides, I think it's better to note this setting suggestion in your wiki document. For not everyone start trying caffeOnSpark from scratch, and in our case --executor-cores has set to a preference long before.

@mriduljain Hello .If --executor-cores can only be set to 1, the training speed will become very slow,won't it?


Fix for Hype Cask

Fix for a symlink source error for Hype, where the app file is normally renamed during extraction (from "Hype2.app" to "Hype 3.app"), but this happens after homebrew checks for an app file.

Please squash your commits.

Thank you for the contribution. It needed some fixes, so they were made in 008edaff6dc4e96629864003cd5a995c01111f3a. Your contribution is still included (and still credited to you), with the appropriate modifications. Please feel free to ask about any of the changes.

Thank you for your help!


Wrong return type for findById passed with composite key

Type of issue
Bug in TS type definition
Description / Steps to reproduce
In the TS type definition of QueryBuilderBase, only these two overloads of findById are present:
{
  // ...
  findById(id: Id): QueryBuilderYieldingOneOrNone<QM>;
  findById(idOrIds: IdOrIds): this;
  // ...
}

Let's say a model has a composite primary key, the following use case would trigger a compile-time error due to the incorrect overload:
class SomeModel extends objection.Model {
  static idColumn = ['idField1', 'idField2']

  // ...
}

SomeModel.query()
  .findById([1, 100])
  .then((model: SomeModel | undefined) => {
    //   ^^^^^ error TS2740
  });

In the above case, the only eligible overload is
findById(idOrIds: IdOrIds): this;

await-ing / .then() on the return type of this overload only yields SomeModel[].
Expected result
The code should compile.
Actual result
error TS2740: Type 'SomeModel[]' is missing the following properties from type 'SomeModel': id, type, articles, $beforeInsert, and 31 more.

Thanks! PR is merged now.


Add 'contactable()' function to filter out non-accessible speakers

This adds a small function to the discovery module, to filter out non-contactable speakers. See the discussion in #828 for motivation.

Coverage decreased (-0.2%) to 53.959% when pulling d758cb99531643618494e780b6c32b5e9977a915 on pwt:contactable into bb71285d5d9255a2eca942b357f292d613891f93 on SoCo:master.


Fix typos

Fix typos (found using codespell).
There are some more but I think that they are in generated files or files that are not original from firebase-messaging:
carles@pinux:[fix-typos]~/git/firebase-messaging$ codespell
./firebase_messaging/proto/android_checkin_pb2.pyi:178: Miliseconds ==> Milliseconds
./firebase_messaging/proto/checkin.proto:113: supportes ==> supports
./firebase_messaging/proto/checkin_pb2.pyi:185: supportes ==> supports
./firebase_messaging/proto/android_checkin.proto:46: Miliseconds ==> Milliseconds
carles@pinux:[fix-typos]~/git/firebase-messaging$

I've checked them because I'm packaging firebase-messaging for Debian and saw one in the changelog.

Yes the files in ./firebase_messaging/proto are autogenerated. Presumably you'll need a new release to be cut after this gets merged?

coverage: 79.595%. remained the same
when pulling 7f29ee4d5bee27222d14a3aaa679cfacec0e9355 on cpina:fix-typos
into a80de6634e650d1045010d7420f940bbb59613dc on sdb9696:main.

Yes the files in ./firebase_messaging/proto are autogenerated. Presumably you'll need a new release to be cut after this gets merged?

Regarding a new release: it does not block me, but simplifies a bit the packaging (I would drop a patch in the Debian package that I did for this). Not a requirement though for this! :-) (if it was other things it would have been needed). Thanks for merging the PR!


Guarding the Realm/Arya Stark

When putting Arya into play from an opponent's discard pile with Guarding the Realm (presumably Yoren as well?) you're not prompted to play the top card of your deck as a duplicate.

You are only allowed to dupe cards you both control and own, so you cannot dupe a Yoren'd Arya.


Add ability to use via custom hook

Description :memo:
This PR adds a custom hook useScrollbarSize to the exports from the package.  The existing component syntax was retained for backwards compatibility, but will be deprecated in a future major release (likely 4.0).
Conceptually, the hook works identically to the component in that it will measure the scrollbar sizes and react to changes which might affect the size of the scrollbar (e.g. change to browser zoom factor).  This is intended merely as a usability enhancement as use of the hook is significantly easier than use of the component.
All examples and guides have been updated to favor using the hook instead of the component
Type of change :gem:

[X] New feature (non-breaking change which adds functionality)

Example code :scroll:
See changes to README.md for updated examples.
How Has This Been Tested? :vertical_traffic_light:

Unit tests were created and updated as needed.
Both examples (hook and component) were run to ensure everything works as intended.
Consumed the library from another project to ensure that it runs properly

Checklist :checkered_flag:

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] New and existing unit tests pass locally with my changes
[X] My changes pass lint, prettier, and TS checks
[X] I have added tests that prove my fix is effective or that my feature works
[X] I have commented my code, particularly in hard-to-understand areas

@kgregory I would appreciate it if you could give this one a review.

@notidart Tagging you as you had asked about this feature.

@Noitidart tagging you since you had asked about this feature.

Thank you. @shawnmcknight - checking now!

arning about referencing a ref's value in the useEffect cleanup.

Thanks for such great comments, may you please share something about refrencing ref's value in useEffect cleanup. I had not heard of this till now. I did some searching and find only results that are refrencing the ref's value - https://medium.com/@alicantorun/react-hooks-usestate-and-useeffect-useref-with-cleanups-166d9231b6e4

Thanks for the review @kgregory and @Noitidart !


Can't install: ptxas fatal: Ptx assembly aborted due to errors

Hello! Thank you for your great work.
As I saw in other issue – there are success installations of this package for python 3.5, cuda 9.2 and pytorch 0.5.0
But I have problem when try to build package:
~/warp-ctc/pytorch_binding$ python3 setup.py bdist_wheel
running bdist_wheel
running build
running build_deps
-- cuda found TRUE
CMake Warning at CMakeLists.txt:52 (FIND_PACKAGE):
  By not providing "FindTorch.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Torch", but
  CMake did not find one.

  Could not find a package configuration file provided by "Torch" with any of
  the following names:

    TorchConfig.cmake
    torch-config.cmake

  Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
  "Torch_DIR" to a directory containing one of the above files.  If "Torch"
  provides a separate development package or SDK, be sure it has been
  installed.

-- Torch found Torch_DIR-NOTFOUND
-- Building shared library with GPU support
-- Configuring done
-- Generating done
-- Build files have been written to: /home/saint/warp-ctc/pytorch_binding/build/lib
[ 11%] Building NVCC (Device) object CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o
ptxas /tmp/tmpxft_00000a80_00000000-12_ctc_entrypoint.compute_30.ptx, line 1931; fatal   : Parsing error near 'u': syntax error
ptxas <macro util>, line 9; error   : Illegal modifier '.sync' for instruction 'mov'
ptxas fatal   : Ptx assembly aborted due to errors
CMake Error at warpctc_generated_ctc_entrypoint.cu.o.cmake:266 (message):
  Error generating file
  /home/saint/warp-ctc/pytorch_binding/build/lib/CMakeFiles/warpctc.dir/src/./warpctc_generated_ctc_entrypoint.cu.o

CMakeFiles/warpctc.dir/build.make:403: recipe for target 'CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o' failed
make[2]: *** [CMakeFiles/warpctc.dir/src/warpctc_generated_ctc_entrypoint.cu.o] Error 1
CMakeFiles/Makefile2:141: recipe for target 'CMakeFiles/warpctc.dir/all' failed
make[1]: *** [CMakeFiles/warpctc.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    'clean': Clean,
  File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.5/dist-packages/wheel/bdist_wheel.py", line 204, in run
    self.run_command('build')
  File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 73, in run
    build_warpctc_so()
  File "setup.py", line 41, in build_warpctc_so
    assert res.returncode == 0, "Error"
AssertionError: Error

Do have an idea why is this happening? Maybe I can provide some other information to help you understand the issue?
I'm using gcc 5.4, cudnn 7.1.4 for cuda 9.2 and python 3.5.2 in Ubuntu 16.04

Hm I'm certainly using something very similar (master PyTorch, cuda 9.2, recent cudnn, gcc-5.5 from Debian). Does PyTorch build ok from source in your setup?

Yes, PyTorch master was built ok. There were breaking changes few days ago (https://github.com/pytorch/pytorch/commit/cffca2926b09970bd46a9fa49dda19f0a90d016d), I was able to build Sean's warp-ctc version only after building pytorch from previous commit. But as I can see in backtrace – here is the problem with some C++ code in bindings, but I'm not sure what exactly

PTXAS means that the error is in warp-ctc, not the bindings.
If it doesn't work after setup.py clean build , could you try building the lib from the main dir as with the instructions from baidu, please?

I have the same question as gazay. I also try building the lib from the main dir as with the instructions from baidu. but the same errors raise.

Hi,
I have finally reproduced it and the fix is simple enough. Apparently I had something cached that wasn't rebuilt during testing.
Thank you for your effort in reporting it!
I have attached a fix.
Best regards
Thomas


range iterator for ArchivedBTreeMap

Unfortunately this will only be able to iterate forward, but that should be alright.

The B-tree map implementation in 0.8 has been updated and is now more space-efficient, but at the cost of being more difficult to write a range-style iterator for it.


toString like function to serialise built transaction skeleton

Lucid had toString function which one could use to serialise built transaction skeleton and diagnose it (without requiring to peek into library internals as known CDDL specification is followed).
While doing migration to lucid-evolution, I am getting script failures and am not sure how to go about debugging it. It appears the field of interest is txBuilder but am not sure how to pretty print it, printing it from config just gives some pointer value.

I see so you want the serialized builder before complete() , will work on it


[SPARK-7586][ML][doc] Add docs of Word2Vec in ml package

CC @jkbradley.
JIRA issue.

Merged build triggered.

Merged build started.

Test build #32804 has started for   PR 6181 at commit 1c3f389.

Test build #32804 has finished for   PR 6181 at commit 1c3f389.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaWord2VecSuite

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32804/
Test FAILed.

Test build #812 has started for   PR 6181 at commit 1c3f389.

Test build #812 has finished for   PR 6181 at commit 1c3f389.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

@yinxusen  I tried the Scala example but got this error:
scala> val documentDF = sqlContext.createDataFrame(Seq(
     |   "Hi I heard about Spark".split(" "),
     |   "I wish Java could use case classes".split(" "),
     |   "Logistic regression models are neat".split(" ")
     | )).map(Tuple1.apply).toDF("text")
<console>:20: error: overloaded method value createDataFrame with alternatives:
  [A <: Product](data: Seq[A])(implicit evidence$4: reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame <and>
  [A <: Product](rdd: org.apache.spark.rdd.RDD[A])(implicit evidence$3: reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame
 cannot be applied to (Seq[Array[String]])
       val documentDF = sqlContext.createDataFrame(Seq(

Did these all run for you?
The Java file needs the Apache license at the top.

Merged build triggered.

Merged build started.

Test build #32935 has started for   PR 6181 at commit 57a4c07.

Test build #32935 has finished for   PR 6181 at commit 57a4c07.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaWord2VecSuite

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32935/
Test FAILed.

Merged build finished. Test FAILed.

@yinxusen  This needs the Apache license at the top of all files.  (That is causing the RAT test failure.)

@yinxusen Except for the RAT test issue, this LGTM.  You may encounter merge issues from other updates to ml-features.md, but they should be easy to fix.

@jkbradley Sorry for forgetting the Apache license again... fix it now.

@jkbradley Seems it has no merging problem, for now.

Merged build triggered.

Merged build started.

Test build #33050 has started for   PR 6181 at commit 77014c5.

Test build #33050 has finished for   PR 6181 at commit 77014c5.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaWord2VecSuite

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33050/
Test FAILed.

Test build #830 has started for   PR 6181 at commit 77014c5.

@jkbradley Are there more concrete error infos for Mima error? I search from the console output of the test, but cannot locate the error.

Test build #830 timed out     for PR 6181 at commit 77014c5     after a configured wait of 150m.

@yinxusen It's not actually a MIMA error; that's a bug in the script reporting on the tests.  It looks to me like Jenkins being flaky.

Test build #831 has started for   PR 6181 at commit 77014c5.

Test build #831 has finished for   PR 6181 at commit 77014c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaWord2VecSuite

LGTM merging into master and branch-1.4
@yinxusen Thank you!


Monthly Recurrence options

Critical Changes
Changes
Going forward when the users create new Service Schedules using the wizard and select the Monthly frequency they will see a dropdown that gives them options to

Create sessions on the day of the session start date.
Create sessions on the specific weeknumber of the day of the week
If the Session Start Date is on the last day of the month then the users will see another option in the dropdown to create sessions on the last day of the month

The values in the drop down are driven from the Session Start Date field
Issues Closed
New Metadata
Deleted Metadata
Definition of Done
Refer to Asteroids DoD document to see any additional details for the items below
- [ ] Place holder data is incorporated (Refer to Case Management Sample Data document)
- [ ] Base axe-core JEST test included for any new LWC (No critical violations)
- [] CRUD/FLS is enforced in Apex

[x] Permission sets & field sets are updated to account for CRUD/FLS/TABS for new object metadata
[x] All modified classes are unit tested (Apex) at 100% coverage
[x] UX approval or UX not necessary
[x] PR contains draft release notes
[x] Attach work item to the pull request with Lurch **Lurch:attach W-000000 or **Lurch:add
[ ] All acceptance criteria have been met

[x] Developer
[ ] Code Reviewer
[ ] QA

[ ] QE story level testing completed

W-8140897

@jjbennett , The last month of the day option should already be taken care of with the byMonthDays option that we already had

@jjbennett , The last month of the day option should already be taken care of with the byMonthDays option that we already had

No because we always step back but last day of month should step forward if 11/30 is the first session date and 12/31 is the next.

Let me look into it


impossible to delete project on Cloudflare Pages

impossible to delete project on Cloudflare Pages
Error Code:800013

Hi @Viteok we're aware of this issue and working on fixing it (hopefully early next week). It's not related to the docs so going to close this issue. Thanks for reporting!


native text display fails to render nested TTML cues

Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
2.5.6
Can you reproduce the issue with our latest release version?
Yes
Can you reproduce the issue with the latest code from master?
Yes
Are you using the demo app or your own custom app?
Custom app
If custom app, can you reproduce the issue using our demo app?
What browser and OS are you using?
Chrome on a laptop, Tizen and WebOS
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
webOS 3.2-4.7 and Tizen 3-5
What are the manifest and license server URIs?
Will be provided by email.
What did you do?
Watched the video and sometimes the subtitles are not showing.
What did you expect to happen?
The subtitles to be rendered.
What actually happened?
The first and last paragraphs are rendered but the rest are not. The rendering works with the same manifest on version 2.5.5 but not on 2.5.6. Couldn't see anything unusual on the logs.

    Lorem ipsum dolor,

    consectetur adipisicing elit.

    -Excepteur sint.

    -Duis aute.

    Excepteur sint...

    Aliquip ex ea commodo,

    Officia Deserunt.

    Mollit Fugiat Nulla Pariatur?

Possibly related to this https://github.com/google/shaka-player/pull/1962 ?

Thanks for reply and the fix. I tested with the latest version using the demo app. The problematic parts are now visible but are aligned oddly to the left and font size is smaller than on the other parts.
Example
Normal:

    Lorem ipsum dolor,

    consectetur adipisicing elit.

Aligned to the left and font size pretty small:

    -Excepteur sint.

    -Duis aute.

Looking more into this. Actually 2.5.8 did not fix the issue. All the subs work when using Shaka Controls in the demo app but after switching to native controls (build=compiled;nativecontrols), the problem is there and cue.text is empty for those problematic parts.
Reproduce:

https://shaka-player-demo.appspot.com/demo/#audiolang=en-US;textlang=fi-FI;uilang=en-US;asset=<PROVIDED_MANIFEST_THERE>;panel=CUSTOM%20CONTENT;build=uncompiled;nativecontrols
Seek to around 5:26

We use native controls in our app. Subtitles work fine on 2.5.5 but does not on 2.5.6-2.5.8. I believe this problem is affecting ChromeCast users as well since the ShakaPlayer was updated to version 2.5.6 in January.
Let me know if you need more information.

I've updated the title of the issue to reflect the fact that this is a problem specifically for native text display.  This does not yet support nested cues.

SimpleTextDisplayer should iterate through nestedCues and flatten them out.  This will not accurately represent TTML nested cues, but it will be as close as we can get with the VTTCue and TextTrackCue APIs in browsers today.


Copy the new Vec::retain implementation into the retain_mut function

In some previous PRs (#83, #85), we introduced the retain_mut helper function that permits the modification of the visited elements by giving a &mut of the elements. Now that the standard library Vec::retain function was rewritten and gives more than 20% performance improvement, it would be cool to patch our own function too!

FWIW, you can use my crate retain_mut. It hasn't updated the code, but I do plan to when the change reaches stable.
(Actually the change to Rust std was motivated by my inspect of the retain_mut crate for which I raised this inefficiency in a Rust group and another member decided to fix it :))


Update device-control-removable-storage-protection.md

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks

expanded "description" text to state macOS or Windows support.


Start Docker daemon inside setup script

Source Docker start function inside setup script.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhay-krishna, danbudris
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhay-krishna,danbudris]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/lgtm


Justin chen fix dist build

Pull Request Test Coverage Report for Build 1307

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 1303:
0.0%

Covered Lines:
333

Relevant Lines:
333

💛  - Coveralls


Insert 'REC_BASE' events during event creation

As part of the reporting and biomarker generation, there is a function that gets called to identify periods of time in the retrieval period that can serve as baseline retrieval events. These are inserted as 'REC_BASE' events. It seems like this process would make more sense to exist as part of event_creation.

This is definitely something that would be helpful for certain retrieval analyses, but I'm a bit concerned about doing too much analysis behind the scenes. Choosing baseline periods is non-trivial and shouldn't be done in a blackbox, to avoid less experienced analysts thinking that there is some "real" event happening there and not questioning the methodology of choosing these periods.


Will not run on Windows 11 + WSL-G

I'm trying to run Kui in Windows 11 with WSL-G (ubuntu 20.04).
I've confirmed that my wsl-g works by successfully running gimp and other graphical linux apps.
I have installed Kui (Kui-linux-x64.zip) from the github releases page, v10.8.2 (https://github.com/kubernetes-sigs/kui/releases/tag/v10.8.2), unzipped it into /usr/local/bin and added :/usr/local/bin/Kui-linux-x64 to my path.
But when I run
kubectl kui

or even just:
Kui

The response I get is:
/usr/local/bin/Kui-linux-x64/Kui: error while loading shared libraries: libatk-bridge-2.0.so.0: cannot open shared object file: No such file or directory

Howdy! Thanks for reporting this. There are probably some missing X-related dependencies that the underlying chromium relies on. Could you try
sudo apt install libatk-bridge2.0-0 libgtk-3.0

There may be a few others, but ... crossing fingers that should cover it?
It has been a TODO for a while to get a .deb package built for Kui. I think that would avoid this dependence issue, at least for debian-related linux.

You may run into another issue, reported a while back: https://github.com/kubernetes-sigs/kui/issues/5110

The futex facility returned an unexpected error code.Aborted

Let us know if you see that!

sudo apt install -y libatk-bridge2.0-0 libgtk-3.0

E: Unable to locate package libgtk-3.0
E: Couldn't find any package by glob 'libgtk-3.0'

The libatk-bridge partially installs as a separate thing, but I get an error too:
sudo apt install -y libatk-bridge2.0-0

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  at-spi2-core libatspi2.0-0
The following NEW packages will be installed:
  at-spi2-core libatk-bridge2.0-0 libatspi2.0-0
0 upgraded, 3 newly installed, 0 to remove and 89 not upgraded.
Need to get 171 kB of archives.
After this operation, 783 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libatspi2.0-0 amd64 2.36.0-2 [64.2 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 at-spi2-core amd64 2.36.0-2 [48.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libatk-bridge2.0-0 amd64 2.34.2-0ubuntu2~20.04.1 [58.2 kB]
Fetched 171 kB in 2s (96.4 kB/s)
Selecting previously unselected package libatspi2.0-0:amd64.
(Reading database ... 59326 files and directories currently installed.)
Preparing to unpack .../libatspi2.0-0_2.36.0-2_amd64.deb ...
Unpacking libatspi2.0-0:amd64 (2.36.0-2) ...
Selecting previously unselected package at-spi2-core.
Preparing to unpack .../at-spi2-core_2.36.0-2_amd64.deb ...
Unpacking at-spi2-core (2.36.0-2) ...
Selecting previously unselected package libatk-bridge2.0-0:amd64.
Preparing to unpack .../libatk-bridge2.0-0_2.34.2-0ubuntu2~20.04.1_amd64.deb ...
Unpacking libatk-bridge2.0-0:amd64 (2.34.2-0ubuntu2~20.04.1) ...
Setting up libatspi2.0-0:amd64 (2.36.0-2) ...
Setting up libatk-bridge2.0-0:amd64 (2.34.2-0ubuntu2~20.04.1) ...
Setting up at-spi2-core (2.36.0-2) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
/sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

Any more advice on this one? I'd really like to get it working.

I managed to start kui on w11 + WSL2 (ubuntu 20.04) with these packages:
sudo apt-get install -y libatk-bridge2.0-0  libgdk-pixbuf2.0-0  libgtk-3-0 libgbm1

only thing is, if I resize the window Kui freezes

errata corrige: if I just run kubectl kui, without any command, I can resize the window correctly and run kubectl commands inside kui.


trans rights are human rights

Tested versions
trans rights are human rights
System information
trans rights are human rights
Issue description
trans rights are human rights
Steps to reproduce
trans rights are human rights
Minimal reproduction project (MRP)
trans rights are human rights

This is a discussion area for the game engine not your body parts.

 Forget Godot. Ignore the trolls.

With these people in your midst, nothing can be done. Look at this, it's every day every minute. This is why it's important to fork well.

Yeah I should not have said anything just report ignore move on

With these people in your midst, nothing can be done. Look at this, it's every day every minute. This is why it's important to fork well.

Indeed.
That's why I am volunteering to remove these timewasters. I'm in Japan and the different timezone could help free up folks to focus on other more important things.

With these people in your midst, nothing can be done. Look at this, it's every day every minute. This is why it's important to fork well.

Indeed.
That's why I am volunteering to remove these timewasters. I'm in Japan and the different timezone could help free up folks to focus on other more important things.

Thanks for doing this for us

Tested versions
trans rights are human rights
System information
trans rights are human rights
Issue description
trans rights are human rights
Steps to reproduce
trans rights are human rights
Minimal reproduction project (MRP)
trans rights are human rights

Cool story bro but how exactly does that influence a game engine?

Spam


Unit of odometer

Version of the custom_component
Tesla custom Integration: Version 3.0.2
Home Assistant 2022.11.1
Supervisor 2022.10.2
Operating System 9.3
Describe the bug
Unit of sensors. "Odometer" and "Range" is mi and not kms
How can I setup with unit kms ?

Dupe of #301


ZDM-494 Make it easier to encrypt sensitive variables and remove them from intermediate files

This PR contains changes to:

Make it easier for users to encrypt sensitive variables that are passed to Ansible as part of the ZDM configuration.
Avoid storing cluster credentials in intermediate files used by the automation.

Note: encrypting variables is completely optional, and can be done using ansible-vault commands to encrypt any configuration files after populating them. With this PR:

The cluster-related configuration is moved into a separate file (zdm_proxy_cluster_config.yml) that the user can choose to encrypt if they wish. This configuration data can be considered sensitive and rarely needs to change.
The read mode, primary cluster and logging configuration remain in zdm_proxy_core_config.yml. These are not sensitive variables and need to change during the migration, so having them in an unencrypted file makes more practical sense.

The change is backwards-compatible. This version of the automation works even if a user provides the "old-style" zdm_proxy_core_config.yml file containing all the variables: in this case, the values in zdm_proxy_core_config.yml are used as normal (taking precedence over any value provided in zdm_proxy_cluster_config.yml, should the user populate both files).
Additionally, this PR removes the need for storing the cluster credentials into intermediate files used by the automation, therefore making the handling of these sensitive values more secure.

The --ask-vault-pass option needs to be passed to the run playbook command whenever an encrypted file is pulled in by the playbook, regardless of what that file actually contains. The reason for this is that Ansible cannot of course view the encrypted file, so it can't know whether its content is actually necessary or not without decrypting it and therefore needs to decrypt it in order to run the playbook.
Encrypting files is always something optional that the user explicitly chooses to do, so I don't see this as breaking backward compatibility. If the user had the old-style and the new-style files both in clear, they could run the playbook in the usual way and the old-style one would take precedence (as expected), so everything would work as before. If they decide to encrypt any file used by the playbook, they will always need to pass the --ask-vault-pass option and then the old-style file will still take precedence, if provided.


Updated the project for Maven migration

please let me know if I missed something or if something is off

Once ready, you can always do a test run deployment to Nexus and Drop the repository if there is something wrong.

Does this fix @aar? What was the issue?

main issue was in the line implementation fileTree(dir: 'libs', include: ['*.jar']) in prince.build which would cause the dependencies to be undefined for Prince


Look into caching travis builds

And only recompile if necessary, similar to running locally.

Included in https://github.com/bitpay/bitcoind.js/commit/2c8ae04af1afeb54507e34903ba2fa010978b286


Introduce GLM-PCA to scanpy.external

I agree with @adrianveres here: GLM-PCA by @willtownes seems like a rigorous way to deal with zero inflation, since we use PCA as base for all subsequent processing. It’s apparently much faster than e.g. ZINB-WaVE

twitter thread explaining the problem and solution
code

@flying-sheep I think that the code is here: https://github.com/willtownes/glmpca

Hi to all, thanks for your interest in glmpca. I have been thinking of doing a python package now that the R package is finished and it would be an honor to have it included in scanpy. Can you give me a sense of how urgently you would need the package (ie what is the typical release cycle)? Also let me note a few caveats about the method:

It does not handle zero inflation (which ZINB-WAVE does). However, we argue in our paper that despite large numbers of zeros, UMI data are not zero-inflated. We do not make any claim about the appropriateness of the glmpca model for non-UMI data (eg Smart-Seq read counts), which may actually be zero-inflated, although you could certainly run it with eg the negative binomial likelihood.
glmpca is an alternative to PCA but not necessarily a replacement to PCA. For example, it is at least 10x slower than PCA and we are still working on the big data implementation for sparse matrices (in other words, we assume you can load the data matrix in dense form, which can be limiting).
We describe a fast approximation to GLM-PCA in the paper which involves transforming raw counts to either Pearson or deviance residuals from a null model then applying standard PCA to that. This approach is just as fast as PCA as long as the null model can be computed in closed-form, which is what we have implemented here: https://github.com/willtownes/scrna2019/blob/master/util/functions.R#L164 . The idea is similar to the sctransform approach used by seurat, but the computation is simpler and faster.
We also provide a deviance-based gene filtering method which is an alternative to using highly variable genes. This and the residuals functions will be available as an R package on bioconductor.

I look forward to collaborating with you all to help make these methods available to a wider community!

Oops sorry here is the correct line for the null residuals function, please ignore the link in previous comment. https://github.com/willtownes/scrna2019/blob/master/util/functions.R#L206

OK, seems like I misunderstood the point about zero inflation here. You just meant “large number of zeroes” as in “pretty sparse” then?
A factor of 10 isn’t that bad for something that’s more complex, and I doubt PCA speed is the bottleneck for most datasets.
So not a replacement, but an enhancement. As such, it would probably live in scanpy.external except if you want to develop it within scanpy instead of as a separate package (which is possible, but would tie you to our – currently slow but we’ll get better) release cycle.

A rough implementation of glmpca in python is now available here: https://github.com/willtownes/glmpca-py . I will try to get it organized as an installable package tomorrow and add unit tests. Issues/ pull requests welcome.

The package is now available on pypi and there is an automated test suite.

Great! I’m sure this will soon find its way into scanpy.external!

@falexwolf @willtownes @LuckyMD Valentine Svensson suggests that log-transforming data could be responsible for the apparent zero inflation, and that the high number of zeros can be accurately modeled with a non-zero-inflated model: https://www.nature.com/articles/s41587-019-0379-5
Since GLM-PCA doesn’t model zero inflation, it’s probably a really good base for distance calculations in scanpy in cases where the performance is sufficient:

The multinomial model adequately describes negative control data, and there is no need to model zero inflation.

I agree that there is definitely no need to model zero inflation.

My point is: The paper suggests that the reason think that might be the log-transforming, so we should offer to transform data (or calculate distances) in a better way.
One path is to go via normally distributed “expression” values which can be used to calculate distances. Something like fold changes, i.e. what we currently do by log-transforming:

counts (→ normalization) → expressions (→ normalization) (→ embedding) → distances → analyses

We use PCA as a latent space embedding here for efficiency purposes but it’s not required, we could calculate distances directly from expressions.
The alternative is to stay with (possibly bias-normalized) counts, and create our latent space from those directly using a suitable model (like GLM-PCA):

counts (→ normalization) → embedding → distances → analyses

Hi, thanks again for your interest in GLM-PCA. We welcome its inclusion in scanpy, but some caveats are that it is about 10x slower than PCA and we are still working to improve its numerical stability and ability to handle sparse data matrices.
With that in mind, we have put together an implementation of Pearson and deviance residuals as an approximation to GLM-PCA via the scry R package. These residuals, based on binomial and poisson approximation to multinomial, can be computed in closed form so they are computationally as fast as log-transforming. The sctransform method uses a negative binomial likelihood which doesn't have a closed form solution and is more complicated to implement (although we do recomment it from a statistical validity standpoint).
In addition to the null residuals, the scry package has an implementation of feature selection via deviance, which may also be of interest as an alternative to highly variable genes. This is also a closed form computation. Both the feature selection and null residuals functions allow adjusting for categorical batch labels.
I do hope to implement both of these in python eventually but it's pretty far down my to-do list. Given the functions are fairly simple, I welcome anyone to go ahead and copy them into python if they find it potentially useful.

Sounds good! Maybe there’s takers in @theislab

We can compute any kind of residual in principle (NB right now, Poisson + log normal soon) in batchglm, this is also not restricted to categorical covariates and automatically selects whether closed form is available. The implmentation is numerically very stable. I think it could be a potential backend for this, @willtownes?

@davidsebfischer that sounds very appealing, I will probably use it whenever I do the python implementation. Thanks!


Dialog opening freezes on Ubuntu 20.04

VSCode:

Version: 1.50.1
Commit: d2e414d9e4239a252d1ab117bd7067f125afd80a
Date: 2020-10-13T14:44:48.716Z
Electron: 9.2.1
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.4.0-52-generic snap

VSCode insider:

Version: 1.51.0-insider
Commit: 7a3bdf4ee9588755d447aa1c3b5db4a123fc11a9
Date: 2020-10-23T07:34:54.891Z
Electron: 9.3.2
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: <IP_ADDRESS>-electron.0
OS: Linux x64 5.4.0-52-generic snap

OS: Ubuntu 20.04.1 LTS
Steps to Reproduce:

Click Help -> About or Help
It takes approx. 5-10 seconds to open the dialog, also closing the dialog is very slow.

Click File -> Open folder
File browsing is impossible. Either clicking a folder takes 10 s - 10 min, or does not open a folder at all.
File browser: Nautilus

Dialog opening or folder opening shows also "Visual Studio Code is not responding"

Tested also insider version, also very slow but maybe marginally faster.
I did not see significant changes in cpu usage or memory usage.

Does this issue occur when all extensions are disabled?: Yes

These are native dialogs (electron) since this is linux. Also note this is with the snap package, not sure if that has an impact.
/cc @joaomoreno

Yes, only vscode seems to misbehave though. I haven't had any problems with other electron+snap apps, e.g. Slack and GitKraken. I tested both with their file opening dialogs, which are native, no problem.
This problem is quite recent. I have successfully used vscode with the same setup for years.

I can't reproduce this.
If you run code --verbose, what errors come out during the crash?

Yes, I tested also the insider version initially as stated in the original report, it had the same problem.
Initially I also tried the --verbose flag, but it didn't report anything significant.
The odd thing is that this does not seem to happen constantly. I had the issue a few days ago again and it was solved by a reboot. At the moment I can't reproduce the issue at all.

So I managed to reproduce this again just now.   This is discouraged.
[154883:1104/131448.374638:INFO:CONSOLE(1707)] "%cTRACE color: #888 [storage state.vscdb] updateItems(): insert(Map(1)

Also immediately when trying to open a folder, the syslog shows this:
Nov  4 13:19:04 xxx dbus-daemon[1545]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.400' (uid=1000 pid=154883 comm="/snap/code/48/usr/share/code/code --verbose --no-s" label="snap.code.code (complain)")
Nov  4 13:19:04 xxx systemd[1]: Starting Hostname Service...
Nov  4 13:19:05 xxx rtkit-daemon[2181]: Successfully made thread 159317 of process 159317 owned by '1000' high priority at nice level -11.
Nov  4 13:19:05 xxx rtkit-daemon[2181]: Supervising 8 threads of 3 processes of 2 users.
Nov  4 13:19:05 xxx pulseaudio[159317]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:1
Nov  4 13:19:05 xxx dbus-daemon[1545]: [system] Successfully activated service 'org.freedesktop.hostname1'
Nov  4 13:19:05 xxx systemd[1]: Started Hostname Service.

And if I open Help -> About
[main 2020-11-04T11:25:34.689Z] menubarService#updateMenubar 1
[main 2020-11-04T11:25:36.434Z] menubar#runActionInRenderer { type: 'commandId', commandId: 'workbench.action.showAboutDialog' }
[154883:1104/132536.449838:INFO:CONSOLE(1707)] "%cTRACE color: #888 CommandService#executeCommand workbench.action.showAboutDialog", source: file:///snap/code/48/usr/share/code/resources/app/out/vs/workbench/workbench.desktop.main.js (1707)
[154883:1104/132536.454411:WARNING:x11_util.cc(1520)] X error received: serial 25783, error_code 3 (BadWindow (invalid Window parameter)), request_code 3, minor_code 0 (X_GetWindowAttributes)
[154917:1104/132536.465852:WARNING:x11_util.cc(1520)] X error received: serial 20707, error_code 3 (BadWindow (invalid Window parameter)), request_code 4, minor_code 0 (X_DestroyWindow)
Gtk-Message: 13:25:36.476: GtkDialog mapped without a transient parent. This is discouraged.

 This is discouraged.
[162737:1104/132736.676242:WARNING:x11_util.cc(1520)] X error received: serial 2146, error_code 3 (BadWindow (invalid Window parameter)), request_code 3, minor_code 0 (X_GetWindowAttributes)
[main 2020-11-04T11:27:39.577Z] update#checkForUpdates, state =  idle
[main 2020-11-04T11:27:39.577Z] update#setState checking for updates
[162766:1104/132739.597657:WARNING:x11_util.cc(1520)] X error received: serial 431, error_code 3 (BadWindow (invalid Window parameter)), request_code 4, minor_code 0 (X_DestroyWindow)
[main 2020-11-04T11:27:39.720Z] update#setState idle

And Help -> About with insiders:
[main 2020-11-04T11:29:04.157Z] menubarService#updateMenubar 1
[main 2020-11-04T11:29:05.096Z] menubar#runActionInRenderer { type: 'commandId', commandId: 'workbench.action.showAboutDialog' }
[162737:1104/132905.117894:INFO:CONSOLE(1742)] "%cTRACE color: #888 CommandService#executeCommand workbench.action.showAboutDialog", source: file:///snap/code-insiders/567/usr/share/code-insiders/resources/app/out/vs/workbench/workbench.desktop.main.js (1742)
[162766:1104/132905.121511:WARNING:x11_util.cc(1520)] X error received: serial 401, error_code 3 (BadWindow (invalid Window parameter)), request_code 4, minor_code 0 (X_DestroyWindow)
[162766:1104/132905.130112:WARNING:x11_util.cc(1520)] X error received: serial 800, error_code 3 (BadWindow (invalid Window parameter)), request_code 4, minor_code 0 (X_DestroyWindow)
Gtk-Message: 13:29:05.136: GtkDialog mapped without a transient parent. This is discouraged.
[162737:1104/132905.137474:WARNING:x11_util.cc(1520)] X error received: serial 4825, error_code 3 (BadWindow (invalid Window parameter)), request_code 3, minor_code 0 (X_GetWindowAttributes)

@deepak1556 Can you drive the issue for further details?

I did some additional research, I'm not sure if this is helpful.
When I click File -> Open Folder the syslog writes the following. I added some comments to map what happens in vscode at the times of logs.
<CLICK OPEN FOLDER, nothing happens>

2020-11-05T11:09:51.791738+02:00 xxx dbus-daemon[1514]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.355' (uid=1000 pid=52132 comm="/snap/code/48/usr/share/code/code --force-user-env" label="snap.code.code (complain)")
2020-11-05T11:09:51.797320+02:00 xxx systemd[1]: Starting Hostname Service...
2020-11-05T11:09:51.820530+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65584 of process 65584 owned by '1000' high priority at nice level -11.
2020-11-05T11:09:51.820729+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:09:51.830819+02:00 xxx pulseaudio[65584]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:2
2020-11-05T11:09:51.919331+02:00 xxx dbus-daemon[1514]: [system] Successfully activated service 'org.freedesktop.hostname1'
2020-11-05T11:09:51.919511+02:00 xxx systemd[1]: Started Hostname Service.

<FILE EXPLORER OPENS after ~5 seconds>

2020-11-05T11:09:55.531823+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:09:55.538169+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65615 of process 65584 owned by '1000' RT at priority 5.
2020-11-05T11:09:55.538358+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:09:55.544242+02:00 xxx pulseaudio[65584]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:0
2020-11-05T11:09:55.921876+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:09:55.927216+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65616 of process 65584 owned by '1000' RT at priority 5.
2020-11-05T11:09:55.927385+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:09:55.966216+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:09:55.971075+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65621 of process 65584 owned by '1000' RT at priority 5.
2020-11-05T11:09:55.971258+02:00 xxx rtkit-daemon[2107]: Supervising 12 threads of 3 processes of 2 users.
2020-11-05T11:09:55.992617+02:00 xxx pulseaudio[65584]: [pulseaudio] main.c: D-Bus name org.PulseAudio1 already taken.
2020-11-05T11:09:56.033853+02:00 xxx pulseaudio[65581]: [pulseaudio] main.c: Daemon startup failed.
2020-11-05T11:09:56.620708+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65629 of process 65629 owned by '1000' high priority at nice level -11.
2020-11-05T11:09:56.620907+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:09:56.634107+02:00 xxx pulseaudio[65629]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:2
2020-11-05T11:09:57.707866+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:09:57.714308+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65639 of process 65629 owned by '1000' RT at priority 5.
2020-11-05T11:09:57.714496+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:09:57.722430+02:00 xxx pulseaudio[65629]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:0
2020-11-05T11:09:58.099146+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:09:58.103386+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65644 of process 65629 owned by '1000' RT at priority 5.
2020-11-05T11:09:58.103509+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:09:58.137929+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:09:58.143317+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65645 of process 65629 owned by '1000' RT at priority 5.
2020-11-05T11:09:58.143400+02:00 xxx rtkit-daemon[2107]: Supervising 12 threads of 3 processes of 2 users.
2020-11-05T11:09:58.165537+02:00 xxx pulseaudio[65629]: [pulseaudio] main.c: D-Bus name org.PulseAudio1 already taken.
2020-11-05T11:09:58.210456+02:00 xxx pulseaudio[65626]: [pulseaudio] main.c: Daemon startup failed.
2020-11-05T11:09:58.259051+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65651 of process 65651 owned by '1000' high priority at nice level -11.
2020-11-05T11:09:58.259322+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:09:58.270283+02:00 xxx pulseaudio[65651]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:2
2020-11-05T11:09:59.331760+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:09:59.338301+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65657 of process 65651 owned by '1000' RT at priority 5.
2020-11-05T11:09:59.338504+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:09:59.346496+02:00 xxx pulseaudio[65651]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:0
2020-11-05T11:09:59.548691+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:09:59.553892+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65659 of process 65651 owned by '1000' RT at priority 5.
2020-11-05T11:09:59.553965+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:09:59.593939+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:09:59.598670+02:00 xxx rtkit-daemon[2107]: Successfully made thread 65660 of process 65651 owned by '1000' RT at priority 5.
2020-11-05T11:09:59.598777+02:00 xxx rtkit-daemon[2107]: Supervising 12 threads of 3 processes of 2 users.
2020-11-05T11:09:59.619397+02:00 xxx pulseaudio[65651]: [pulseaudio] main.c: D-Bus name org.PulseAudio1 already taken.
2020-11-05T11:09:59.666896+02:00 xxx pulseaudio[65648]: [pulseaudio] main.c: Daemon startup failed.

<no interaction or logging for a while>
<CLICK CANCEL, nothing happens>

2020-11-05T11:14:30.792587+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67576 of process 67576 owned by '1000' high priority at nice level -11.
2020-11-05T11:14:30.792935+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:14:30.813732+02:00 xxx pulseaudio[67576]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:2

<FILE EXPLORER CLOSES after ~4 seconds>

2020-11-05T11:14:34.492191+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:14:34.498832+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67598 of process 67576 owned by '1000' RT at priority 5.
2020-11-05T11:14:34.499053+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:14:34.506962+02:00 xxx pulseaudio[67576]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:0
2020-11-05T11:14:34.884631+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:14:34.889826+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67599 of process 67576 owned by '1000' RT at priority 5.
2020-11-05T11:14:34.889967+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:14:34.930009+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:14:34.935658+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67600 of process 67576 owned by '1000' RT at priority 5.
2020-11-05T11:14:34.935887+02:00 xxx rtkit-daemon[2107]: Supervising 12 threads of 3 processes of 2 users.
2020-11-05T11:14:34.956634+02:00 xxx pulseaudio[67576]: [pulseaudio] main.c: D-Bus name org.PulseAudio1 already taken.
2020-11-05T11:14:34.997993+02:00 xxx pulseaudio[67573]: [pulseaudio] main.c: Daemon startup failed.
2020-11-05T11:14:35.033313+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67608 of process 67608 owned by '1000' high priority at nice level -11.
2020-11-05T11:14:35.033495+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:14:35.045555+02:00 xxx pulseaudio[67608]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:2
2020-11-05T11:14:36.117016+02:00 xxx rtkit-daemon[2107]: Supervising 9 threads of 3 processes of 2 users.
2020-11-05T11:14:36.119518+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67618 of process 67608 owned by '1000' RT at priority 5.
2020-11-05T11:14:36.119593+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:14:36.122727+02:00 xxx pulseaudio[67608]: [pulseaudio] alsa-ucm.c: No UCM verb is valid for hw:0
2020-11-05T11:14:36.309145+02:00 xxx rtkit-daemon[2107]: Supervising 10 threads of 3 processes of 2 users.
2020-11-05T11:14:36.314767+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67620 of process 67608 owned by '1000' RT at priority 5.
2020-11-05T11:14:36.314945+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:14:36.355265+02:00 xxx rtkit-daemon[2107]: Supervising 11 threads of 3 processes of 2 users.
2020-11-05T11:14:36.359620+02:00 xxx rtkit-daemon[2107]: Successfully made thread 67621 of process 67608 owned by '1000' RT at priority 5.
2020-11-05T11:14:36.359727+02:00 xxx rtkit-daemon[2107]: Supervising 12 threads of 3 processes of 2 users.
2020-11-05T11:14:36.380090+02:00 xxx pulseaudio[67608]: [pulseaudio] main.c: D-Bus name org.PulseAudio1 already taken.
2020-11-05T11:14:36.426639+02:00 xxx pulseaudio[67605]: [pulseaudio] main.c: Daemon startup failed.

Syslog logging about rtkit-daemon and pulseaudio does not happen unless I interact with vscode and native dialogs.
Looking at htop when trying File -> Open Folder I see that every time /usr/bin/pulseaudio --start --log-target=syslog with nice value -11 is spawned for a short while and then disappearing.
Not sure if these are relevant but this is what I observed. I was also able to reproduce the issue this time with Slack too.

After a logout/logon the problem disappeared (probably temporarily) and syslog shows only this when trying the same File -> Open Folder, everything works snappy.
2020-11-05T13:14:52.328732+02:00 xxx dbus-daemon[1514]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.639' (uid=1000 pid=115330 comm="/snap/code/48/usr/share/code/code --force-user-env" label="snap.code.code (complain)")
2020-11-05T13:14:52.334210+02:00 xxx systemd[1]: Starting Hostname Service...
2020-11-05T13:14:52.477994+02:00 xxx dbus-daemon[1514]: [system] Successfully activated service 'org.freedesktop.hostname1'
2020-11-05T13:14:52.478294+02:00 xxx systemd[1]: Started Hostname Service.

Thanks for investigating the issue,

Looking at htop when trying File -> Open Folder I see that every time /usr/bin/pulseaudio --start --log-target=syslog with nice value -11 is spawned for a short while and then disappearing.

it is really surprising to see, I have no idea how that happens. As a quick check, can you uninstall pulseaudio and does file dialogs work fine ?
Can you also check if the issue happens with exploration build that is based on a newer runtime https://az764295.vo.msecnd.net/exploration/7af7d1c0c5af759c0e5bee5139491b3524a2ec8d/code-exploration-1605782755.snap ?


Reuse Metadata from Catalog

As a cataloger,
I would like to pull in descriptive metadata from the catalog,
so that I can repurpose existing metadata.

Duplicate of #157. Closing.


Fix plugin templates

Thanks!

How did you run into the bug?

No problem! It was in a plugin's package.json:
  "strider": {
    "templates": {
      "Brand": "TITLE"
    }
  }

Released as 1.7.1


feat(protocol-designer): add alert if x/y position is too close to edge

closes AUTH-252
Overview
Adds a PD alert banner to the tip position modal if the user adds an x or y position that is too close to the edge of the well
Test Plan
Create a flex or ot-2 protocol. Add a labware. Add a transfer or mix step. In the advanced settings, select the tip position icon and open the tip position modal. Add a custom offset with an x or y position that is at least 90% of the max/min positions. You should see the PD alert banner pop up. It should go away if you change your value
Changelog

add alert banner to pop up if the x or y value is >90% of the min or max
add test cases

Review requests
see test plan
Risk assessment
low

After changing the position, still the banner is staying in the modal. Is this expected behavior?

@koji the banner should display if the value inputted is greater than 90% of the max value/min value. But you should be allowed to submit as long as it is below the max value. I think i need to modify the min/max to only display up to 1 decimal value. I will address this, thank you for reviewing!


No way to have space separated abbreviation and not-separated currency symbol at the same time

There should be two separate config options for currency space and abbreviation space.
What I want to achieve is $10 k. With current configuration options it is only possible to achieve either $10k or $ 10 k.

in numbro 2.2.0


Multiple error messages in form fields

Is there a reason why we only display the first error on an invalid model attribute? (cf: https://github.com/kaliber5/ember-bootstrap/blob/536036adad90e70a4ac9add59bea881a99eff631/addon/templates/components/common/bs-form/element/errors.hbs#L2)
Wouldn't we want to loop through all items to show them?

I guess it depends. I think for many cases I want not want to display multiple lines of error text, causing all following form elements to move down. Also with just one message displayed, when the user types and the message does not apply anymore (say minimum length) but there is still another one, that would be shown then, so you still see the others until there are no errors anymore.
But I am ok with adding this as an opt-in, so will flag this as a feature request. PRs certainly welcome! ;)

I'll keep this in mind when I do #249


utxoAt -> utxosAt migration?

I've been using utxoAt (at tag plutus-starter-devcontainer/v1.0.8):
https://github.com/input-output-hk/plutus/blob/cc7bc06c4344cee6cd59bc170063fd627da25ed3/plutus-contract/src/Plutus/Contract/Request.hs#L222-L228
Which is now removed in tag plutus-starter-devcontainer/v1.0.9 in favor of utxosAt:
https://github.com/input-output-hk/plutus/blob/73b21ed30e982b51f3debf69c55d3affe8642eb7/plutus-contract/src/Plutus/Contract/Request.hs#L353-L364
However utxosAt carries less information. The originating Tx is not recoverable. UtxoMap contains TxOutTx, which has both Tx and TxOut. ChainIndexTxOut just lets me get TxOut.
So, how do you migrate?

There is
-- | Get unspent transaction outputs with transaction from address.
utxosTxOutTxAt ::
    forall w s e.
    ( AsContractError e
    )
    => Address
    -> Contract w s e (Map TxOutRef (ChainIndexTxOut, ChainIndexTx))

that also returns ChainIndexTx:
data ChainIndexTx = ChainIndexTx {
    _citxTxId            :: TxId,
    -- ^ The id of this transaction.
    _citxInputs          :: Set TxIn,
    -- ^ The inputs to this transaction.
    _citxOutputs         :: ChainIndexTxOutputs,
    -- ^ The outputs of this transaction, ordered so they can be referenced by index.
    _citxValidRange      :: !SlotRange,
    -- ^ The 'SlotRange' during which this transaction may be validated.
    _citxData            :: Map DatumHash Datum,
    -- ^ Datum objects recorded on this transaction.
    _citxRedeemers       :: Map RedeemerHash Redeemer,
    -- ^ Redeemers of the minting scripts.
    _citxMintingPolicies :: Map MintingPolicyHash MintingPolicy,
    -- ^ The scripts used to check minting conditions.
    _citxStakeValidators :: Map StakeValidatorHash StakeValidator,
    _citxValidators      :: Map ValidatorHash Validator
    } deriving (Show, Eq, Generic, ToJSON, FromJSON)

which is similar to the original Tx.

ChainIndexTx doesn't seem to contain all fields of Tx and I don't see what UtxoMap is for anymore.

ChainIndexTx got a field with the full original transaction _citxCardanoTx.
https://github.com/input-output-hk/plutus/blob/master/plutus-chain-index/src/Plutus/ChainIndex/Tx.hs#L76

this needs to be added to the documentation PLS! Not easy to find that out....


Doctrine connection not reset on DB::reconnect()

Laravel Version: 5.5.28
PHP Version: 7.0
Database Driver & Version: MySQL

Description:

Calling DB::reconnect() calls DB::refreshPdoConnections() which will update the PDOs for the connection.
Connection::getDoctrineConnection() sets up a new doctrine connection if it does not exist yet, passes the current PDO and stores it in $this->doctrineConnection.

If the connection's PDO gets changed via DB::reconnect() the doctrineConnection property does not get updated and still holds the old PDO information. Queries using the doctrine connection will execute using wrong settings.
Steps To Reproduce:
Assuming two databases with identical structure ('table' containing a column 'column1') and :
Config::set('database.connections.mysql.database', 'firstDb');
DB::reconnect();

Schema::table('table', function (Blueprint $table) {
	$table->renameColumn('column1', 'column2');
});

Config::set('database.connections.mysql.database', 'secondDb');
DB::reconnect();

Schema::table('table', function (Blueprint $table) {
	$table->renameColumn('column1', 'column2');
});

This results in:

There is no column with name 'column1' on table 'table'.

Explanation:
While the first schema change works correctly, the second one wrongly still uses the old PDO information (ie running on db 'firstDb') and sees there is no 'column1' on it because it already got renamed. If the PDO change would have been propagated to the doctrineConnection it would see that the 'column1' still exists on 'secondDb'.
Workaround:
** Note: ** This only works if database.connections.mysql.database in the initial database.php config file is set to null, although I am not sure why.
reset doctrineConnection as well whenever a reconnect or disconnect occurs:
\App\Database\MySqlConnectionWithPDOFix:
<?php

namespace App\Database;
use Illuminate\Database\MySqlConnection;

class MySqlConnectionWithPDOFix extends MySqlConnection {

	public function reconnect() {
		$this->doctrineConnection = null;
		return parent::reconnect();
	}

	public function disconnect() {
		$this->doctrineConnection = null;
		parent::disconnect();
	}
}

\App\Providers\AppServiceProvider:
Connection::resolverFor('mysql', function($connection, $database, $prefix, $config) {
  return new MySqlConnectionWithPDOFix($connection, $database, $prefix, $config);
});

Facing the same problem and can't find the solution anywhere.

You can set a custom reconnector for your connection to reset the doctrineConnection property:
/** @var \Illuminate\Database\Connection $connection */ $connection->setReconnector(function(\Illuminate\Database\Connection $connection) use ($databaseManager) { 	$reflectionProperty = new \ReflectionProperty($connection, 'doctrineConnection'); 	$reflectionProperty->setAccessible(true); 	$reflectionProperty->setValue($connection, null); 	/** @var \Illuminate\Database\DatabaseManager $databaseManager */ 	$databaseManager->reconnect($connection->getName()); });
I have no idea where to put this in the Laravel framework code, because I am using Illuminate/Database as a separate module in my project (so no Laravel Framework in the project).

[5.8] Still an issue, disconnect does not remove the doctineConnection and this causes a build up of database connections.
We have a RabbitMQ queue processor which connects to many different databases depending on the message. We disconnect the connection once the task has been complete assuming the DB connection will be closed, however after a while our process breaks due to too many database connections being open.
Following the advice of @calvinbaart works, I can see our connection counts dropping as we would expect. The PDO object passed into the DoctrineConnection object is not being nullified when disconnect is called on the Connection object.
Even using DB::purge which unsets the connection from the DatabaseManager cache, does not resolve the database build up. It seems that removing the Connection doesn't remove the PDO connection... which makes me believe this issue is due to the events or query logging keeping a copy of the connection. Even though the local PDO is nullified the DoctrineConnection copy of the PDO persists somewhere on an event queue.
This issue is easy to recreate using MySQL, simply create many database connections in the config. Count the rows returned from the process list to see your baseline.
DB::select('SHOW PROCESSLIST')->count()

Disconnect or purge those connections and count the rows from the process list again. They should reduce in number, but they do not.


[FeatureRequest] limit number of function parameters

It would be great to have an option to have a lint property which will error when the number of parameters exceeds the defined number:
max-num-of-params: 2

goo(a, b) {} // no error
foo(a, b, c) {} // error

@vlio20 parameters-max-number

Tangentially related: #3849

StealingCopying @chelmertz's suggestion from #4203: it would be good to suggest using an object with named parameters once the limit is reached.

I'd be into taking this - would like this rule as well :)

For context, ESLint supports this using 'max-params':
https://eslint.org/docs/rules/max-params

Somewhat relevant: TypeScript has a built-in refactoring to fix this! https://github.com/Microsoft/TypeScript/issues/23552

Note: per #4534, this issue will be closed in less than a month if no PR is sent to add the rule. If you really need the rule, custom rules are always an option and can be maintained outside this repo!

💀 It's time! 💀
TSLint is being deprecated and no longer accepting pull requests for new rules. See #4534. 😱
If you'd like to see this rule implemented, you have two choices:

Recommended: Use typescript-eslint to lint your TypeScript code with ESLint ✅
Legacy: Write it as a custom TSLint rule 🤷♂

👋 It was a pleasure open sourcing with you!
If you believe this message was posted here in error, please comment so we can re-open the issue!

🤖 Beep boop! 👉  TSLint is deprecated 👈 (#4534)  and you should switch to typescript-eslint! 🤖
🔒 This issue is being locked to prevent further unnecessary discussions. Thank you! 👋


Level storage mappings

First commit is copypasta from what Notch copypasta'd, which is public domain.
Second commit is my own work. Some of the names are ehh (OldRegionComparable) but they work for now.

ohno, i'm going to have to cherry pick these commits out because of the squash merge.

what's the problem with the squash merge?

My branch has the actual commit but master has a squashed one instead, so it was conflicted and resolved in the merge commit above. Should be an easy fix in IDEA's git UI.

can I merge?

also while you're at it can you bump the build number to 6 in build.gradle

Sure


Python docs for use_compliant_nested_typebool on ParquetWriter states default is both True and False.

Title and description is conflicts: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html#pyarrow-parquet-parquetwriter

use_compliant_nested_typebool, default True
Whether to write compliant Parquet nested type (lists) as defined here, defaults to False. [..]

Component(s)
Python

This is fixed in https://github.com/apache/arrow/pull/38471
And default is True now.


Improvement in Console

Need Improvement in Console and also need command register support

@Lablnet Do you have any idea in mind to improve it?