envoy: Use ipcache instead of xDS on supported kernels.

Use bpf ipcache instead of NetworkPolicyHosts xDS on kernels that
support LPM maps (4.11+).
xDS hosts cache is populated only if the LPM support from the kernel is not detected. This reduces the amount of work Cilium agent needs to do on large clusters even when no L7 policies are configured. This has the downside that Istio sidecar proxies need to be upgraded if running on Linux 4.11 or later.
Signed-off-by: Jarno Rajahalme<EMAIL_ADDRESS>Istio sidecar proxies from earlier Cilium releases must be upgraded if running on Linux 4.11 or later.

This change is

test-me-please

Coverage decreased (-0.01%) to 41.776% when pulling 7b7ece94a3614993a8c7963d037bb0c9b9824ef9 on pr/jrajahalme/envoy-use-ipcache into 81d485f26949d4f8aac3fbae3f164fff8c7ece30 on master.

test-me-please

test-me-please

Separated the minor bug fix into a PR of it's own: #8149

Note also the follow-on PR for Istio update: #8150


How to pretrain from scratch?

Hi, great work. And I am wondering did u provide the pretraining code?

sorry for bothering, I found that.


GPU OOM with Keras and Estimator, fine with Keras alone

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows10
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): tensorflowGPU_1.7
Python version: 3.5.4
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: CUDA 9.0/cuDNN7.0
GPU model and memory:NVidia Titan X (Pascal) 12GB
Exact command to reproduce:python TestKerasAndEstimator.py

Describe the problem
I have developed a Keras model and used it successfully with only Keras train and evaluate calls. Now I would like to use the same model in an Estimator context using tf.keras.estimator.model_to_estimator.
It's quite a large model (3 LSTM layers with 100 units each, input dimension 598), but I can run it in the Keras-alone context with a batch_size up to 10.  However, with an Estimator, I get an OOM on the GPU even with a batch_size of 1.  If I reduce the sequence length (SEQ_LEN) from 24000 to 5000 however, it will run with an Estimator.
Here is the output log with Keras alone (USE_ESTIMATOR=False)
[Console output redirected to file:C:\Users\philip.LMS\git\GIT_RD_Python\Ch2018\ch2018_train\TestEstimator_20180612_2256_log.txt]
Using TensorFlow backend.
2018-06-12 22:57:11.753350: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-12 22:57:12.396391: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:03:00.0
totalMemory: 12.00GiB freeMemory: 9.93GiB
2018-06-12 22:57:12.624894: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 1 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:a1:00.0
totalMemory: 12.00GiB freeMemory: 9.93GiB
2018-06-12 22:57:12.625753: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0, 1
2018-06-12 22:57:17.075745: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-12 22:57:17.076181: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:917]      0 1
2018-06-12 22:57:17.076457: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0:   N N
2018-06-12 22:57:17.076746: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 1:   N N
2018-06-12 22:57:17.077336: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9618 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:03:00.0, compute capability: 6.1)
2018-06-12 22:57:17.079930: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9619 MB memory) -> physical GPU (device: 1, name: TITAN X (Pascal), pci bus id: 0000:a1:00.0, compute capability: 6.1)
Creating Model
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_1 (LSTM)                (10, 24000, 100)          279600
_________________________________________________________________
batch_normalization_1 (Batch (10, 24000, 100)          400
_________________________________________________________________
lstm_2 (LSTM)                (10, 24000, 100)          80400
_________________________________________________________________
batch_normalization_2 (Batch (10, 24000, 100)          400
_________________________________________________________________
lstm_3 (LSTM)                (10, 24000, 100)          80400
_________________________________________________________________
batch_normalization_3 (Batch (10, 24000, 100)          400
_________________________________________________________________
dense_1 (Dense)              (10, 24000, 3)            303
=================================================================
Total params: 441,903
Trainable params: 441,303
Non-trainable params: 600
_________________________________________________________________
None
Epoch 1/2

10/20 [==============>...............] - ETA: 6:45 - loss: 0.0000e+00 - acc: 1.0000 - weighted_acc: 1.0000
20/20 [==============================] - 904s 45s/step - loss: 0.0000e+00 - acc: 1.0000 - weighted_acc: 1.0000
Epoch 2/2

10/20 [==============>...............] - ETA: 8:59 - loss: 0.0000e+00 - acc: 1.0000 - weighted_acc: 1.0000
20/20 [==============================] - 1059s 53s/step - loss: 0.0000e+00 - acc: 1.0000 - weighted_acc: 1.0000

10/20 [==============>...............] - ETA: 43s
20/20 [==============================] - 86s 4s/step

And here is the output log with Keras and Estimator (USE_ESTIMATOR=True)
[Console output redirected to file:C:\Users\philip.LMS\git\GIT_RD_Python\Ch2018\ch2018_train\TestEstimator_20180612_2338_log.txt]
Creating Model
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_1 (LSTM)                (1, 24000, 100)           279600
_________________________________________________________________
batch_normalization_1 (Batch (1, 24000, 100)           400
_________________________________________________________________
lstm_2 (LSTM)                (1, 24000, 100)           80400
_________________________________________________________________
batch_normalization_2 (Batch (1, 24000, 100)           400
_________________________________________________________________
lstm_3 (LSTM)                (1, 24000, 100)           80400
_________________________________________________________________
batch_normalization_3 (Batch (1, 24000, 100)           400
_________________________________________________________________
dense_1 (Dense)              (1, 24000, 3)             303
=================================================================
Total params: 441,903
Trainable params: 441,303
Non-trainable params: 600
_________________________________________________________________
None
2018-06-12 23:39:11.126586: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-12 23:39:11.751372: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 0 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:03:00.0
totalMemory: 12.00GiB freeMemory: 9.93GiB
2018-06-12 23:39:11.981018: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1344] Found device 1 with properties:
name: TITAN X (Pascal) major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:a1:00.0
totalMemory: 12.00GiB freeMemory: 9.93GiB
2018-06-12 23:39:11.981905: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0, 1
2018-06-12 23:39:17.295319: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-12 23:39:17.295766: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:917]      0 1
2018-06-12 23:39:17.296030: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0:   N N
2018-06-12 23:39:17.296314: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 1:   N N
2018-06-12 23:39:17.296888: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9618 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:03:00.0, compute capability: 6.1)
2018-06-12 23:39:17.299820: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9619 MB memory) -> physical GPU (device: 1, name: TITAN X (Pascal), pci bus id: 0000:a1:00.0, compute capability: 6.1)
2018-06-12 23:39:17.644506: I T:\src\github\tensorflow\tensorflow\core\kernels\cuda_solvers.cc:159] Creating CudaSolver handles for stream 000001E5C8B53C10
<MapDataset shapes: (<unknown>, <unknown>), types: (tf.float32, tf.float32)>
Dataset point 1:
<MapDataset shapes: ((24000, 598), (24000, 3)), types: (tf.float32, tf.float32)>
Dataset point 2:
<MapDataset shapes: ((24000, 598), (24000, 3)), types: (tf.float32, tf.float32)>
Dataset point 3:
<RepeatDataset shapes: ((24000, 598), (24000, 3)), types: (tf.float32, tf.float32)>
Batch features
Tensor("IteratorGetNext:0", shape=(?, 24000, 598), dtype=float32, device=/device:CPU:0)
Batch labels
Tensor("IteratorGetNext:1", shape=(?, 24000, 3), dtype=float32, device=/device:CPU:0)
2018-06-12 23:39:26.545626: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1423] Adding visible gpu devices: 0, 1
2018-06-12 23:39:26.546174: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-12 23:39:26.546739: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:917]      0 1
2018-06-12 23:39:26.547090: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 0:   N N
2018-06-12 23:39:26.547462: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:930] 1:   N N
2018-06-12 23:39:26.548091: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9618 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:03:00.0, compute capability: 6.1)
2018-06-12 23:39:26.549907: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 9619 MB memory) -> physical GPU (device: 1, name: TITAN X (Pascal), pci bus id: 0000:a1:00.0, compute capability: 6.1)
Input File:
b'record_0'
Decoded File:
record_0
Features:
[[ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 ...,
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]]
Labels:
[[ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]
 ...,
 [ 0.  0.  0.]
 [ 0.  0.  0.]
 [ 0.  0.  0.]]
2018-06-12 23:40:47.905054: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:967] failed to alloc<PHONE_NUMBER> bytes on host: CUDA_ERROR_OUT_OF_MEMORY
2018-06-12 23:40:47.905614: W T:\src\github\tensorflow\tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size:<PHONE_NUMBER>
2018-06-12 23:40:47.906068: E
...

Here is the python test file TestKerasAndEstimator.py:
import sys

import numpy as np
import tensorflow as tf

USE_ESTIMATOR = True
#USE_ESTIMATOR = False

if USE_ESTIMATOR:
  from tensorflow.python import keras
  from tensorflow.python.keras.models import Sequential
  from tensorflow.python.keras.layers import Dense, LSTM, BatchNormalization
  from tensorflow.python.keras.optimizers import SGD, RMSprop
  N_RECORDS = 1
  BATCH_SIZE = 1
else:
  import keras
  import keras.backend.tensorflow_backend as K
  from keras.models import Sequential
  from keras.layers import Dense, LSTM, BatchNormalization
  from keras.optimizers import SGD, RMSprop
  N_RECORDS = 20
  BATCH_SIZE = 10

#SEQ_LENGTH=1000
#SEQ_LENGTH=5000
#SEQ_LENGTH=10000
SEQ_LENGTH=24000

INPUT_DIM = 598
OUTPUT_DIM = 3

NP_DTYPE = np.float32
TF_DTYPE = tf.float32

TRAIN_EPOCHS = 2
DEVICE_ID = '/gpu:0'

class ModelLSTM():
  def __init__(self, batch_size, max_length=None, device_id='/cpu:0', n_input_dim=1, n_output_dim=2):

    self.batch_size = batch_size
    self.max_length = max_length
    self.device_id = device_id
    self.n_input_dim = n_input_dim
    self.n_output_dim = n_output_dim

    self.lstm_n_cell=[100, 100, 100]
    self.dropout=0.1
    self.recurrent_dropout=0.1

    self.create_model()

  def create_model(self):

    with tf.device(self.device_id):

      print('Creating Model')
      model = Sequential()
      model.add(LSTM(self.lstm_n_cell[0],
                      return_sequences=True,
                      stateful=False,
                      kernel_initializer='he_normal',
                      activation='tanh',
                      dropout = self.dropout,
                      recurrent_dropout = self.recurrent_dropout,
                      batch_input_shape=(self.batch_size, self.max_length, self.n_input_dim)))
      model.add(BatchNormalization(momentum=0.99, epsilon=0.001, center=True,
                                   scale=True, beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros', moving_variance_initializer='ones'))
      model.add(LSTM(self.lstm_n_cell[1],
                     return_sequences=True,
                     stateful=False,
                     kernel_initializer='he_normal',
                      activation='tanh',
                      dropout = self.dropout,
                      recurrent_dropout = self.recurrent_dropout))
      model.add(BatchNormalization(momentum=0.99, epsilon=0.001, center=True,
                                   scale=True, beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros', moving_variance_initializer='ones'))
      model.add(LSTM(self.lstm_n_cell[2],
                     return_sequences=True,
                     stateful=False,
                     kernel_initializer='he_normal',
                      activation='tanh',
                      dropout = self.dropout,
                      recurrent_dropout = self.recurrent_dropout))
      model.add(BatchNormalization(momentum=0.99, epsilon=0.001, center=True,
                                   scale=True, beta_initializer='zeros', gamma_initializer='ones', moving_mean_initializer='zeros', moving_variance_initializer='ones'))

      model.add(Dense(self.n_output_dim, kernel_initializer='he_normal',
                                      activation='softmax'))

      print (model.summary())

      opt = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)

      model.compile(loss='categorical_crossentropy',
                    optimizer=opt,
                    metrics=['accuracy'],
                    weighted_metrics=['accuracy'],
                    sample_weight_mode='temporal')

    self.model = model
    return self

class TestKerasAndEstimator():
  def __init__(self):
    self.device_id = DEVICE_ID

  def set_device(self, id):
    self.device_id = id

  def the_input_fn(self, filenames, perform_shuffle=False, repeat_count=1, batch_size=1):
    def _set_shapes(features, labels):
      features.set_shape([SEQ_LENGTH, 598])
      labels.set_shape([SEQ_LENGTH, 3])
      return features, labels

    def _my_parse_function(filename, label=None):

      print('Input File:')
      print(filename)

      dec_filename = filename.decode(sys.getdefaultencoding())
      print('Decoded File:')
      print(dec_filename)

      features = np.zeros((SEQ_LENGTH, INPUT_DIM), dtype=NP_DTYPE)
      labels = np.zeros((SEQ_LENGTH, OUTPUT_DIM), dtype=NP_DTYPE)

      print('Features:')
      print(features)
      print('Labels:')
      print(labels)

      return features, labels

    labels = [0]*len(filenames)
    labels = np.array(labels)
    labels = tf.constant(labels)
    labels = tf.cast(labels, TF_DTYPE)

    dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))

    dataset = dataset.map(
      lambda filename, label: tuple(tf.py_func(
        _my_parse_function, [filename, label], [TF_DTYPE, label.dtype])))

    print(dataset)

    dataset = dataset.map(_set_shapes)

    print("Dataset point 1:")
    print(dataset)

    if perform_shuffle:
        dataset = dataset.shuffle(buffer_size=batch_size)
    print("Dataset point 2:")
    print(dataset)
    dataset = dataset.repeat(repeat_count)  # Repeats dataset this # times
    print("Dataset point 3:")
    print(dataset)
    dataset = dataset.batch(batch_size)  # Batch size to use
    the_iterator = dataset.make_one_shot_iterator()
    batch_features, batch_labels = the_iterator.get_next()
    print('Batch features')
    print(batch_features)
    print('Batch labels')
    print(batch_labels)
    return batch_features, batch_labels

  def test_keras_estimator(self, n_records=1, batch_size=1):
    gpu_options = tf.GPUOptions(allow_growth=True)
    sess_config = tf.ConfigProto(gpu_options=gpu_options, allow_soft_placement=True, log_device_placement=False)
    run_config = tf.estimator.RunConfig(session_config=sess_config)

    train_model = ModelLSTM(batch_size=batch_size, max_length=SEQ_LENGTH, device_id=self.device_id,
                            n_input_dim=INPUT_DIM, n_output_dim=OUTPUT_DIM)
    self.estimator = tf.keras.estimator.model_to_estimator(keras_model=train_model.model,
                                                           model_dir='.', config=run_config)
    train_records = list()
    for i in range(0, n_records):
      train_records.append('record_' + str(i))

    train_spec = tf.estimator.TrainSpec(input_fn=lambda:
                                        self.the_input_fn(train_records, perform_shuffle=False, batch_size=batch_size),
                                        max_steps=TRAIN_EPOCHS)
    eval_spec = tf.estimator.EvalSpec(input_fn=lambda:
                                        self.the_input_fn(train_records, perform_shuffle=False, batch_size=batch_size))

    tf.estimator.train_and_evaluate(self.estimator, train_spec, eval_spec)

  def test_keras(self, n_records=1, batch_size=1):
    gpu_options = tf.GPUOptions(allow_growth=True)
    sess_config = tf.ConfigProto(gpu_options=gpu_options, allow_soft_placement=True, log_device_placement=False)
    K.set_session(tf.Session(config=sess_config))

    train_model = ModelLSTM(batch_size=batch_size, max_length=SEQ_LENGTH, device_id=self.device_id,
                            n_input_dim=INPUT_DIM, n_output_dim=OUTPUT_DIM)
    features = np.zeros((n_records, SEQ_LENGTH, INPUT_DIM), dtype=NP_DTYPE)
    labels = np.zeros((n_records, SEQ_LENGTH, OUTPUT_DIM), dtype=NP_DTYPE)

    train_model.model.fit(x=features, y=labels, batch_size=batch_size, epochs=TRAIN_EPOCHS, verbose=1)
    train_model.model.evaluate(x=features, y=labels, batch_size=batch_size, verbose=1)

if __name__ == "__main__":
  mt = TestKerasAndEstimator()
  if USE_ESTIMATOR:
      mt.test_keras_estimator(n_records=N_RECORDS, batch_size=BATCH_SIZE)
  else:
      mt.test_keras(n_records=N_RECORDS, batch_size=BATCH_SIZE)

Thanks!

Also, I see the same behaviour with tensorflowGPU_1.9rc0
Thanks for looking into this.

@pawarrick Can you post your TF graph other than model.summary() output? For a code snippet check https://github.com/tensorflow/tensorflow/issues/16468

Also with tensorflowGPU_1.9rc0, using
keras.layers.CuDNNLSTM instead of
keras.layers.LSTM
the Estimator version runs fine with SEQ_LEN=24000 and BATCH_SIZE=35

Here is the updated code with CuDNNLSTM added:
import sys

import numpy as np
import tensorflow as tf

USE_ESTIMATOR = True
#USE_ESTIMATOR = False

#USE_CUDNN_LSTM = True
USE_CUDNN_LSTM = False

if USE_ESTIMATOR:
  from tensorflow.python import keras
  from tensorflow.python.keras.models import Sequential
  from tensorflow.python.keras.layers import Dense, LSTM, BatchNormalization, CuDNNLSTM
  from tensorflow.python.keras.optimizers import SGD, RMSprop
  if USE_CUDNN_LSTM:
    N_RECORDS = 35
    BATCH_SIZE = 35
  else:
    N_RECORDS = 1
    BATCH_SIZE = 1

else:
  import keras
  import keras.backend.tensorflow_backend as K
  from keras.models import Sequential
  from keras.layers import Dense, LSTM, BatchNormalization
  from keras.optimizers import SGD, RMSprop
  N_RECORDS = 20
  BATCH_SIZE = 10

#SEQ_LENGTH=1000
#SEQ_LENGTH=5000
#SEQ_LENGTH=10000
SEQ_LENGTH=24000

INPUT_DIM = 598
OUTPUT_DIM = 3

NP_DTYPE = np.float32
TF_DTYPE = tf.float32

TRAIN_EPOCHS = 2
DEVICE_ID = '/gpu:0'

class ModelLSTM():
  def __init__(self, batch_size, max_length=None, device_id='/cpu:0', n_input_dim=1, n_output_dim=2):

    self.batch_size = batch_size
    self.max_length = max_length
    self.device_id = device_id
    self.n_input_dim = n_input_dim
    self.n_output_dim = n_output_dim

    self.n_layers = 3
    self.lstm_n_cell=[100, 100, 100]
    self.dropout=0.1
    self.recurrent_dropout=0.1

    self.create_model()

  def create_model(self):
    with tf.device(self.device_id):

      print('Creating Model')
      model = Sequential()

      for i_layer in range(0, self.n_layers):
        if USE_CUDNN_LSTM:
          model.add(keras.layers.CuDNNLSTM(self.lstm_n_cell[i_layer],
                          return_sequences=True,
                          stateful=False,
                          kernel_initializer='he_normal',
                          batch_input_shape=(self.batch_size, self.max_length, self.n_input_dim)))
        else:
          model.add(LSTM(self.lstm_n_cell[i_layer],
                          return_sequences=True,
                          stateful=False,
                          kernel_initializer='he_normal',
                          activation='tanh',
                          dropout = self.dropout,
                          recurrent_dropout = self.recurrent_dropout,
                          batch_input_shape=(self.batch_size, self.max_length, self.n_input_dim)))

        model.add(BatchNormalization(momentum=0.99, epsilon=0.001, center=True,
                                     scale=True, beta_initializer='zeros', gamma_initializer='ones',
                                     moving_mean_initializer='zeros', moving_variance_initializer='ones'))

      model.add(Dense(self.n_output_dim, kernel_initializer='he_normal',
                                      activation='softmax'))

      #print (model.summary())
      opt = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)

      model.compile(loss='categorical_crossentropy',
                    optimizer=opt,
                    metrics=['accuracy'],
                    weighted_metrics=['accuracy'],
                    sample_weight_mode='temporal')

    self.model = model
    return self

class TestKerasAndEstimator():
  def __init__(self):
    self.device_id = DEVICE_ID

  def set_device(self, id):
    self.device_id = id

  def the_input_fn(self, filenames, perform_shuffle=False, repeat_count=1, batch_size=1):
    def _set_shapes(features, labels):
      features.set_shape([SEQ_LENGTH, 598])
      labels.set_shape([SEQ_LENGTH, 3])
      return features, labels

    def _my_parse_function(filename, label=None):

      print('Input File:')
      print(filename)

      dec_filename = filename.decode(sys.getdefaultencoding())
      print('Decoded File:')
      print(dec_filename)

      # stub for testing, but normally read data from file here
      features = np.zeros((SEQ_LENGTH, INPUT_DIM), dtype=NP_DTYPE)
      labels = np.zeros((SEQ_LENGTH, OUTPUT_DIM), dtype=NP_DTYPE)

      print('Features:')
      print(features)
      print('Labels:')
      print(labels)

      return features, labels

    labels = [0]*len(filenames)
    labels = np.array(labels)
    labels = tf.constant(labels)
    labels = tf.cast(labels, TF_DTYPE)

    dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))

    dataset = dataset.map(
      lambda filename, label: tuple(tf.py_func(
        _my_parse_function, [filename, label], [TF_DTYPE, label.dtype])))

    print(dataset)

    dataset = dataset.map(_set_shapes)

    print("Dataset point 1:")
    print(dataset)

    if perform_shuffle:
        dataset = dataset.shuffle(buffer_size=batch_size)
    print("Dataset point 2:")
    print(dataset)
    dataset = dataset.repeat(repeat_count)  # Repeats dataset this # times
    print("Dataset point 3:")
    print(dataset)
    dataset = dataset.batch(batch_size)  # Batch size to use
    the_iterator = dataset.make_one_shot_iterator()
    batch_features, batch_labels = the_iterator.get_next()
    print('Batch features')
    print(batch_features)
    print('Batch labels')
    print(batch_labels)
    return batch_features, batch_labels

  def test_keras_estimator(self, n_records=1, batch_size=1):
    gpu_options = tf.GPUOptions(allow_growth=True)
    sess_config = tf.ConfigProto(gpu_options=gpu_options, allow_soft_placement=True, log_device_placement=False)
    run_config = tf.estimator.RunConfig(session_config=sess_config)

    train_model = ModelLSTM(batch_size=batch_size, max_length=SEQ_LENGTH, device_id=self.device_id,
                            n_input_dim=INPUT_DIM, n_output_dim=OUTPUT_DIM)

    self.estimator = tf.keras.estimator.model_to_estimator(keras_model=train_model.model,
                                                           model_dir='models', config=run_config)
    train_records = list()
    for i in range(0, n_records):
      train_records.append('record_' + str(i))

    train_spec = tf.estimator.TrainSpec(input_fn=lambda:
                                        self.the_input_fn(train_records, perform_shuffle=False, batch_size=batch_size),
                                        max_steps=TRAIN_EPOCHS)
    eval_spec = tf.estimator.EvalSpec(input_fn=lambda:
                                        self.the_input_fn(train_records, perform_shuffle=False, batch_size=batch_size))

    tf.estimator.train_and_evaluate(self.estimator, train_spec, eval_spec)
    #print (train_model.model.summary())

  def test_keras(self, n_records=1, batch_size=1):
    gpu_options = tf.GPUOptions(allow_growth=True)
    sess_config = tf.ConfigProto(gpu_options=gpu_options, allow_soft_placement=True, log_device_placement=False)
    K.set_session(tf.Session(config=sess_config))

    train_model = ModelLSTM(batch_size=batch_size, max_length=SEQ_LENGTH, device_id=self.device_id,
                            n_input_dim=INPUT_DIM, n_output_dim=OUTPUT_DIM)

    # stub for testing, but normally read data from file here
    features = np.zeros((n_records, SEQ_LENGTH, INPUT_DIM), dtype=NP_DTYPE)
    labels = np.zeros((n_records, SEQ_LENGTH, OUTPUT_DIM), dtype=NP_DTYPE)

    train_model.model.fit(x=features, y=labels, batch_size=batch_size, epochs=TRAIN_EPOCHS, verbose=1)
    train_model.model.evaluate(x=features, y=labels, batch_size=batch_size, verbose=1)

if __name__ == "__main__":
  mt = TestKerasAndEstimator()
  if USE_ESTIMATOR:
      mt.test_keras_estimator(n_records=N_RECORDS, batch_size=BATCH_SIZE)
  else:
      mt.test_keras(n_records=N_RECORDS, batch_size=BATCH_SIZE)

@bhack, I couldn't get your IPython code to display the graph (it ran without errors, but the HTML did not display in my Web browser--but I'm not very familiar with jupyter notebooks, so maybe I haven't configured something).
But here's the graph as viewed in Tensorboard in case that's equivalent to what your code does:

the graph above is for the failing keras.layers.LSTM case with an Estimator

@pawarrick Yes that code was made for colab. So seems that in your case there is any spurious branch right? Generally spurious branches appeared in the graph with model_to_estimator as you can see in my issue but I've not tested recently with tensorflowGPU_1.9rc0.

Nagging Assignee @fchollet: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

/cc @martinwicke @tanzhenyu  Has model_to_estimator struck again?

I have had the same (but less 'weird') issue before.
@pawarrick Have you tried with pure estimator implementation?

The OOM message is about trying to allocate<PHONE_NUMBER> (MAXINT) bytes, which is failing.
This points to a bug where something tries to allocate ((unsigned int) -1) bytes, which sounds like a bug (and a bug in the error message to go with it).
This is, btw., also about host memory. Does this work if you disable the GPU? It might well not be a GPU problem at all.

If you mean by ‘pure’ without Keras, no

On Jul 23, 2018, at 10:42 AM, tanzhenyu<EMAIL_ADDRESS>wrote:
I have had the same (but less 'weird') issue before.
@pawarrick Have you tried with pure estimator implementation?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

The model_to_estimator merely copies the model (which shouldn't matter too much in your case as you only have less than 1M parameters), and allocating so much memory doesn't seem right but that process shouldn't happen during the conversion -- that's why I'm asking if pure estimator implementation can be checked.

@martinwicke I've tried now https://github.com/tensorflow/tensorflow/issues/19967#issuecomment-398143577 on Colab/CPU ONLY.
Pure tf.keras USE_ESTIMATOR = False is working fine instead the estimator USE_ESTIMATOR = True code path it is crashing.

That's nice, at least we can rule out GPU.
One more thing to try (for better comparability) is to run the the_input_fn in the pure keras path. I believe it will keep at least three copies of the "features" in memory (still not enough that it should break, but it would be better to rule it out).

@martinwicke Is it enought adding the_input_fn to the fit callbacks= ?

It shouldn't be called each iteration. Start by calling it once and keeping
a reference to the result. If that works, add a callback which evaluates
the result with K.get_session in each iteration.

@pawarrick Have you time to test it?

@tanzhenyu  Do you have a little bit more context for your case https://github.com/tensorflow/tensorflow/issues/19967#issuecomment-407081003?

@tanzhenyu What was you case?

Nagging Assignees @fchollet, @tanzhenyu: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 31 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@bhack @pawarrick I suspect if it has something to do with the input fn you are using in Estimator? (I'm not very familiar with tf.Dataset but I imagine it could need to allocate extra large amount of memory for the map function you're using (as well as the shuffling).

@tanzhenyu Are the map executing gpu Ops? I have only pure python /numpy. My context was more on the Shared vision model. With shared vision you need to heavily reduce the batch size against the same not shared model.

Nagging Assignees @fchollet, @tanzhenyu: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @fchollet, @tanzhenyu: It has been 74 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

As we're approaching TF 2.0, can you try Keras natively? Or is there any other thing that you'd need that is only provided by Estimator?

Is the bug still there running in TF2.0?

I mean using Keras natively, not through model_to_estimator

Sorry I don't understand . In the original post, the code worked fine in with native Keras (with TF 1.7).  Are you asking if that's still the case in TF 1.13?  By the way, I stopped using Keras natively shortly after this post and began using the Keras within TF.  Later (in TF 1.12?), Keras supported Datasets directly without Estimator so I've been using that paradigm.

Sorry I should have clarified. By Keras natively I mean is there anything you need to do in TF.Estimator that you won't be able to do in TF.Keras?
Dataset is already supported, as you mention. Another popular ask is for distributed training, which is also supported. So as long as you don't have any thing specific required from Estimator (like warm start, scaffold, hooks, etc), I'd think your case is already covered?

I'm one of those who reques distributed support too. But because of a lack of support for sample loss weighting (see issue #24226), I haven't been able to take my Keras-with-Datasets approach to distributed training.

That's a valid request, I will respond on that issue.

thanks

Hi @pawarrick! we are checking to see if you still you need help in this issue , Have your tried TF 2.6 yet ? Please create a new issue if the issue persistss.

This was resolved a while ago when sample loss weighting was supported with datasets. Thanks

Are you satisfied with the resolution of your issue?
Yes
No


Unicode support for urlparse 4

@lopuhin, this PR should be clearer than the one in the original repository 😄

You are right! I will set up tox and travis in the next PRs :D


Remove indexer, denormalize database

Changes

Remove indexer
denormalize database
add lockTimeout limit
add getLogs limit

@pustovalov  I have setup near localnet and aurora engine on near localnet.Later I have setup aurora relayer(PR #338) for localnet.

Hi @TanyaSrivastava-Dipali
To get blocks with this PR, few other services should be running:

borealis-engine-lib
aurora-relayer-indexer


Using TAB to change the values crash the game…

it might be imgui related, is it going in infinite loop? Then it is.
If you are able, change imgui to some older version before low level backend function in wndproc.
I don't use any custom changed stuff in imgui so it will be easy.
I will check it myself in few hours or days

it might be imgui related, is it going in infinite loop? Then it is. If you are able, change imgui to some older version before low level backend function in wndproc. I don't use any custom changed stuff in imgui so it will be easy. I will check it myself in few hours or days

I don't think it's infinite loop, it's just crash the game when u press TAB on the functions that editable with it.

after digging, simple change without any code edit is to use IMGUI_DISABLE_WIN32_DEFAULT_IME_FUNCTIONS macro.
Will get rid of those backend issues and doesn't affect menu in any way.
Or to use imgui's branch without new backend logic.

fixed
macro to fix
Closing...


Refactoring Printer

The current Phel\Printer class is a "God Class" with too many responsibilities. Actually 19, one per each conditional inside the print() function. See:
public function print($form): string
{
    if ($form instanceof Tuple) {
        return $this->printTuple($form);
    }
    if ($form instanceof Keyword) {
        return ':' . $form->getName();
    }
    if ($form instanceof Symbol) {
        return $form->getName();
    }
   // ... etc

Suggestion 1: Using extract class refactoring

Create a new Module named Printer and there allocate each private function in a class. This way it will be easily testable.
A similar strategy as we already did for the Analyzer module.

Suggestion2: Using polymorphism

Instead of having all information about how to print anything inside this GodClass, consider moving the responsibility of printing inside each class. For example. Instead of:

if ($form instanceof Tuple) {
    return $this->printTuple($form);
}

Use $form->print() when required (usually inside their __toString() methods).
In my opinion, the ideal option is the second one, because each form type would be responsible for its own printing thing. BUT there is a problem with this approach, there are forms without types, also we are using native scalar types (this means not custom-defined types). This way I don’t find an easy way to do this second approach…
TL;DR: I would go for the first approach (Suggestion 1: Using extract class refactoring) as a first iteration to reduce the complexity of this existing class. And in another Issue/PR we could discuss the second option (adding custom-types for all forms).

Suggestion 1 is the better solution! The Printer must be able to print any PHP Type. This can also be classes of other library.

Implemented here https://github.com/jenshaase/phel-lang/pull/134 :D


Add solver workspace structure

This adds a workspace structure to SolverBase allowing reuse of the allocations for internal work vectors and arrays.
I wanted to add Cg only first, so we can discuss the interface before proceeding.
It not only allows eliminating almost all allocations inside a solver apply call, but also potentially gives loggers the possibility to access internal solver vectors and scalars, improving debuggability and behavior observability.
Closes #805

@MarcelKoch I think strings are not a good representation, as they can't be checked statically for correctness (typos) and require more complex data structures (i.e. map). It definitely makes sense to have a name for them, but I would keep name and ID separate and only use them when I actually have to print some output.
I added a draft for what I had in mind for named vectors

Sorry, I was a bit lax in my language. I didn't want to suggest adding the workspace to each solver, using the SolverBase is clearly the better idea. I just wanted to bring up how it seems to be done in petsc.
Your version with constexpr static members looks interesting, but I would want to think about that a bit.
Regarding the naming, I guess we just have different priorities, so we can leave it at that.

@MarcelKoch yes, I fully agree, the naming part is still a bit of a draft. But I think it would be safer than using strings directly

If we did any of that, we would lose the ability to use custom Dense vector types, more generally any information (number of right-hand sides, dynamic type of the vector) that is only available at the apply time. Turning this into a single allocation only works if we require all derived types to be initializable from existing data, which is a pretty strong requirement.

for the second, I will imagine something like the generate_with_hint(A, rhs_vector);

This is pretty similar to the discussion on reusable LinOpFactories, and many of the same arguments apply here - only the reuse happens on the LinOp level. In short, I don't think it's a good idea. It requires us to basically duplicate the vector initialization for generate and apply, which can involve more than 10 different LinOps. Would this be worth slightly decreasing the runtime of the first apply?

@pratikvn I don't think that's worth the complexity it involves, this would require some additional macro tooling (FOREACH like in Boost::Preprocessor) and obscure what is happening much more.

it seems to require more focus on keeping everything matching.

constexpr static int with the name (0 ~ n-1). no skipping
num_vector should return n
GKO_SOLVER_VECTOR(**name**) should use the same name as the constexpr static int name
(good: it can be checked in compiler time)
vector_name should match the order but it only infects the logger information

I agree this is a bit tedious, unfortunately C++ still lacks some reflection capabilities which would make this trivial.
Skipping numbers is not a big issue, since it just results in nullptr entries. We can also add a check to test/solver.cpp making sure that no workspace vectors are unused and the number of vector names is correct. If your num_vectors is too small, you get an assertion failure in debug builds.

cb_gmres will need to be restructured to deal with #861 anyways, because the accessor interfaces hides the array allocation. I wanted to do that when that PR is done. So you would suggest unpacking all the macros instead?

@yhmtsai We need to keep in mind that we can do a lot of complex stuff internally, but user-facing code should stay as simple and readable as possible. I think in that regard, forcing us to write out the IDs is better than hiding how they are declared and set up behind macros would be the worse approach.

format-rebase!

@upsj user will not use id directly. They will use ws::v or ws:: residual not a magical number 1 or 11. And will user touch the workspace easily? I thought they need to use logger

If all the workspace ID constants are hidden behind macros, they are not easily visible when looking at the code. The workspace can be useful from multiple perspectives. Accessing intermediate state during a solve requires a logger, but inspecting the vectors after a fixed number of iterations also works without it using a Iteration stopping criterion.


3rd party "Tag Documentation" links?

Is it possible to have a "Tag Documentation" link for custom field types?

@lindseydiloreto I've been thinking about this one for a few days now. I definitely want to be able to support 3rd party docs links in the same way we link to Craft docs.
I'm not sure what route to take yet but am open to suggestions.

Hard to say... It'll probably be a little easier when it's eventually converted into a standard plugin. It might be "acceptable losses" for that feature to not exist quite yet.
I haven't though about it in detail, so this suggestion might be garbage... but maybe something can be pulled off using Twig macros?

Interesting idea, the macros. I'll chew on this a bit after the 1.0 release and see where we go.
Thanks!


Contact info data

parse from sme.sk or fill some other way instead of existing mocks

DoD:

returns Daily Menu object with parsed data
returns error on parsing failure

(to be updated)


Build process doesn't symlink to the correct version of Create

Through the build process a symlink to an installed version of create is placed here \extern\nvidia\app
I have multiple versions of create installed, by default it is picking the latest (or first it finds - I'm not entirely sure of the logic)
The result is it is linking to a Kit 105.1 version of Create, which is causing issues with resolving dependencies when trying to run the dev app as I'm building a Kit 105 version of Cesium for Omniverse (0.10.0 tag)
If I manually create a symlink to a Kit 105.0 version of Create, it works as expected.
Not sure if there is an actual fix needed, however I suspect similar issues could be encountered if one was trying to build a Kit 105.1 version and only had Kit 105.0 versions of  create installed (haven't tested to confirm)

Not sure if there is an actual fix needed, however I suspect similar issues could be encountered if one was trying to build a Kit 105.1 version and only had Kit 105.0 versions of create installed (haven't tested to confirm)

I just checked on my desktop, and I'm building a Kit 105.1 version against a 105.0 version of Create. It's working. (But it does make me wonder if that's the source of some other issues.)


Creating an DateTime<Tz> instance from a DateTime<Local>

I'm trying to create a rrule::RRuleSet via its new() method, but can't figure out how to create a DateTime<Tz> instance (which is what this method requires).
I essentially have a DTSTART:19700101T000000 string as a starting point, and use the following to get a DateTime instance:
chrono::offset::Local.datetime_from_str(value, "%Y%m%dT%H%M%SZ")?

However, this returns a DateTime<Local>, and I can't quite figure out how to convert that to DateTime<Tz>. I intuitively think that new should maybe have the signature fn new<T>(dt_start: DateTime<T>) where T: TimeZone. I've no idea if this makes sense for all the rest of codebase, but it would allow using any TimeZone (including Local).
For context, I'm trying to parse VTIMEZONE entries, so I'm trying to construct this RRuleSet to then add other relevant fields to it (e.g.: RDATE).

I intuitively think that new should maybe have the signature fn new<T>(dt_start: DateTime<T>) where T: TimeZone.

Actually, this seems to be what's missing for my approach to work. This would also cover #24 and #36. I might have a PoC for this soon.

Hey!
Would this work for you?
rrule::Tz::LOCAL.datetime_from_str(s, "%Y%m%dT%H%M%SZ")?

Yup, it does, thanks. I think I was trying to use the wrong Local.

Yup, it does, thanks. I think I was trying to use the wrong Local.


Add padding methods by percentage of the screen width and dp

I also tidy up some part of the code, let me know if you don't agree with the convention

Example
<com.example.lib.Deck
        android:id="@+id/deck_pager"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:padding_percentage="80"
        app:padding_dp="8dp"/>

I just pushed the updates, please check again

That's good for me, good job! =)

Don't forget to update the release as well for jitpack dependency

Before update it I'm going to investigate that scroll feed back problem and change lib package name.

@bloderxd can you update the lib on jitpack? I think it's more important than change the lib package name :)


bug: Cody code lens doesn't show up for class method

From Discord:

Hi, I turn on Code Lenses feature in setting and I see that Cody icon only show up for Class and Function definition. As you can see in the picture, I think it good if the feature support constructor() and in class method (sleep(),...) too, I can use it for quick create jsdoc and other things.

Also I found that app.get() have Cody code lenses but app.listen() dont.

I think this is fixed, please reopen if it's still a concern


Replace ExtractCssChunkWebpackPlugin relative path with module path

I guess this is just a little typo but make my build fail. Thanks for the great job you have done.

Thanks brother


Impossible to define scalars for input parameters in Kotlin2 generated code

I am using dgs-codegen successfully since a couple of months. I saw that recently a new so called kotlin2 code generation has been added and wanted to give it a try.
I have to say I really like the way the code generation works and I'd like to use it instead of the "old" kotlin code generation.
But I have one problem :)
Currently, what I am doing to call my GQL APIs is the following:
val graphQLQueryRequest = GraphQLQueryRequest(
    query = CoursesGraphQLQuery.Builder()
        .build(),
    projection = CoursesProjectionRoot()
        .items().id(),
    scalars = scalars
)

With scalar being a mapOf(UUID::class.java to graphql.scalars.id.UUIDScalar.INSTANCE.coercing) and this works like charm.
What I would like to do, by leveraging kotlin2 implementation, is:
DgsClient.buildQuery {
    course(id = UUID.randomUUID()) {
        id // this is a UUID as well
    }
}

but when I try to do this, I get this error:
java.lang.NoSuchMethodError: 'graphql.language.Value com.netflix.graphql.dgs.client.codegen.InputValueSerializer.toValue(java.lang.Object)'

	at com.netflix.graphql.dgs.codegen.GraphQLProjection.arguments(Kotlin2Core.kt:108)
	at com.netflix.graphql.dgs.codegen.GraphQLProjection.field(Kotlin2Core.kt:138)
        TRUNCATED BY ME

If I try it without the input parameter, it works fine, that means that somehow the return parameter is able to deserialize a UUID to String.
Digging a little bit inside the kotlin2 implementation, I see that in the Kotlin2Core.kt class there is this private val inputSerializer = InputValueSerializer() instantiated in the companion object that can actually be instantiated with a custom map of scalars, but it's impossible to reach from the outside. I have no control over it.
My question is: would it be possible to make code generation for kotlin2 to create the code to pass a custom map of scalars as I do for the GraphQLQueryRequest in the first example? Maybe there is a way of doing it that I am not seeing?
Thanks a lot for your hard work!!

I had the same problem, would like to send a pr to fix this problem

Same problem brought be here. I just copied GraphQLProjection class into com.netflix.graphql.dgs.codegen package in my sources to override it but it's not a sustainable solution.
I peeked into your PR and I think it won't be convenient. Typically in app runtime you define scalars once for the whole application. Imagine having deep and nested projection - passing scalars map to each projection would be annoying. 😅  Maybe that approach would be more welcome.


File path completion

Should figure out how to get completion to pass through as normal. It seems relatively undoable to set the working directory of kittylauncher dynamically (at least it'd be incredibly arbitrary and confusing; it's likely to be a feature of kittylauncherrc in the future) but at least with my file organization it'd be perfectly fine if I could get, say, "kitty-launch featherpad documents/cakerecipe.txt" to work.

This is partially implemented in commit 48bc0f1. To be exact, now kitty will complete file paths under any kitty-launch regular command - even if you might want to complete something else! :> I myself only really use the default bash-completions (files and regular commands in $PATH), so until there's some evidence anyone except me is using kittylauncher and wants this feature, I won't do more work on it.
(Well, I say that, then I get bored and end up writing more code anyway.)


chore: Use namespace package and Buf SDKs

Remove the proto-to-python generation logic and instead use Buf generated Python SDKs.
This ultimately solves the grpc namespace collision issue as it removes all vendored proto code and passes the responsibility to the package manager (I think).
Closes https://github.com/cerbos/cerbos-sdk-python/issues/44

The final hurdle here is that it requires the users to specify and allow an additional repository in their configuration, e.g:
[[tool.pdm.source]]
name = "buf"
url = "https://buf.build/gen/python"

If they don't do this, the installation step will fail. The alternative method here is hilariously to vendor the dependencies ourselves (which is the very thing I'm trying to avoid). A little bit chicken<->egg.
Rather than requiring users to explicitly add the repo config to their projects, I wonder if we could instead mirror the wheels built by Buf and host them in our Pypi repository 🤔.
Alternatively, I finally give in and publish a "legacy" http package separate from the main branch.
@charithe - wondering if you have any thoughts on this?

I am a bit reluctant to have the BSR as a critical component in the user-facing distribution path because it's unclear whether it'll be free forever. Furthermore, the fact that users have to configure a new repository for their Python package manager adds a hurdle because some environments have strict rules about which third-party repositories are allowed.
If we can pull the wheels from BSR and distribute them as part of our Python package, that would be ideal. They have to be downloaded automatically as part of our package (without any additional configuration on the user side) though.

I managed to get it all running, vendoring the wheels directly and referencing as local dependencies, got to the point of pushing to the test PyPI index, and... 🥁
INFO     Response from https://test.pypi.org/legacy/:
         400 Invalid value for requires_dist. Error: Can't have direct dependency:
         'cerbos-cerbos-api-grpc-python@
         file:///Users/samuellock/cerbos/cerbos-sdk-python/libs/cerbos_cerbos_api_grpc_python-<IP_ADDRESS>.202
<PHONE_NUMBER>0%2B75004b0044b2-py3-none-any.whl'

😞. A bit of digging found a similar issue, and this particular comment:

PEP 440 says:

Public index servers SHOULD NOT allow the use of direct references in uploaded distributions.

PEP 508 says:

Firstly, PEP-440 direct references haven't actually been deployed in the wild, but they were designed to be compatibly added, and there are no known obstacles to adding them to pip or other tools that consume the existing dependency metadata in distributions - particularly since they won't be permitted to be present in PyPI uploaded distributions anyway.

As such, I'm going to close this issue as we have no plans to deviate from these PEPs, but thanks for the report.

Sigh

That's frustrating 😞
Taking a step back, I am confused about the root cause of #44 because I haven't done any advanced Python packaging. Am I right in saying that:

the protobuf code we generate has references to Google protobuf generated code
those references are not namespaced (because Python protobuf generator doesn't support that)
including the google dependencies in our package causes problems for users who already have a dependency on them through another route

If the above is correct:

how does the Buf generated code work around it? can we replicate it locally?
if we produce a package variant that doesn't require google protos, will that address #44?

Yeah your assumptions seem correct.
I think the crux of the issue was that googleapis-common-protos had the same namespace as the google code built via buf generate due to our own proto definition's dependency on it. This, coupled with the fact that the import paths generated in the code were absolute and strictly matched the proto module structure.
I'm not entirely sure why, but importing the Buf generated SDKs from their index resolved #44 , I'm initially thought due to smarter dependency resolution, but on further thought it could be due to namespace package magic or something else I'm unfamiliar with (namespace packages seem to allow for duplicate namespaces, providing there are no collisions in submodules..?).

if we produce a package variant that doesn't require google protos, will that address https://github.com/cerbos/cerbos-sdk-python/issues/44?

I think so. Our other generated namespaces are less likely to collide than google 😆.

Thanks! I think I (kind of) get it now. What if we use the buf option to restrict the set of generated types to avoid generating code for google and protoc_gen_openapiv2 and instead add dependencies on googleapis-common-protos and protoc-gen-openapiv2-protoc3-19?

Could we create a namespace package for google.api.expr? There is some precedent for grpc-gateway openapiv2 protos (https://github.com/unionai-oss/protoc-gen-openapiv2).

Yeah I think that works. If I add the two additional dependencies (googleapis-common-protos and protoc-gen-openapiv2), delete all generated dependency proto code apart from google/api/expr/, then the collision issue seems to go away. Perhaps this is the way to go for now, given there don't seem to be any other viable options.
buf generate type filters don't seem to work on imported modules:
Failure: filtering by type "google.api.expr.v1alpha1.CheckedExpr": type declared in imported module

So the way to approach this will be to generate all code into a temp directory and copy what we need into the top level 🤪. It's bonkers, the gymnastics (and ongoing maintenance costs) that have to be done to solve a seemingly simple issue!

So the way to approach this will be to generate all code into a temp directory and copy what we need into the top level

I think we can make it work by  removing the --include-imports flag from https://github.com/cerbos/cerbos-sdk-python/blob/341f84297f79e3e8a9ad79966779f446a262fbba/proto/generate_protos.sh#L12 and then adding the following line to the script:
buf generate --path google/api/expr buf.build/googleapis/googleapis

Yes! That works! Thanks so much for all the help. I went slightly off piste and opted for a two-step buf export -> buf generate step. It means that we filter out unused generated types.


On the meaning of validseqlen and seq_len

First of all, thanks for sharing your code. I'm trying to understand your code in order to adapt it to my task.  Can I ask why, in the language model, you set the value of "seq_len" two times the value of "validseqlen"? Is there any reason behind it? Since, clearly, I did not well understand this part, can you please explain me the exact role of these two parameters? Thank you so much.

Hope this helps: https://github.com/locuslab/TCN/tree/master/TCN/word_cnn#note

Hope this helps: https://github.com/locuslab/TCN/tree/master/TCN/word_cnn#note

Ok. Thank you!


remove coredns patch

This PR include:

Remove coredns patch because removing bufsize from /etc/coredns/Corefile
Get version from assisted installer URL.

After discuss it with @RobertKrawitz, understand that its no harm to leave it in case of installing OCP 4.10 or older versions.

Sounds good to leave as is, and glad to see the issue is resolved in newer OCP versions.

@RobertKrawitz, do you think we need to close this PR or leave it as is ?

Close due to maintenance


Scroll bar dark mode

Fixes #1132
I could not figure out how should I use ScrollView inside dialog so scrolls in dialog are not changed. I'll make another issue

this is just a UI changes so no changelog

Looks great!
Only thing I found is that the Unity Installations part of Settings doesn't scale in width.

Looks great!
Only thing I found is that the Unity Installations part of Settings doesn't scale in width.

Thank you for notice that! fixed now!


Fix failed drive-kop tests

Can someone merge this PR? My PR, https://github.com/openmessaging/benchmark/pull/370,  depends on this change. Thanks

ping @merlimat


Display size mismatch

Issue:

Display size is mismatched, images below for verification. It looks like the issue occurs when using
# or ##, causing it to be left shifted compared to the rest of the document, which then removes the side lines and causes the overflow.

Potential fix:
Add a css over ride for a right shift on headers, or wrap everything inside of a div to ensure it's consistent, and not overflowing to the left.
Personal solution
So I reinstalled multiple times. It resulted in the error on 3.5.1 the first install, error on 2.7.11 on the second install, but then fixed itself on 3.5.1 on the third install. Not sure if it's a result of all the reinstalls, but end result is the bug disappears. Opening this issue in case others run into the same issue, though you can close if I'm incorrect.
OS:

OSX 10.11.3 El Capitan

Browsers checked:

Firefox (45.0.2)
Chrome (49.0.2623.112 (64-bit))

Python versions checked:

2.7.11
3.5.1

Image verification - issue:
Python 2.7.11, Chrome example 1:

Python 2.7.11, Firefox example 1:

Python 2.7.11, Firefox example 2:

Image verification - fixed
Python 3.5.1, Chrome: fixed after 3rd reinstall

Python 3.5.1, Firefox: fixed after 3rd reinstall

Was the 3rd reinstall within the past half hour? I just published Grip 4.2.0, which contains a style fix. That might be what you saw.
Closing this, but feel free to re-open if that wasn't the problem afterall.
Also, thanks for all the detailed in this issue! It's really helpful when context and screenshots like you have here are provided :smile:

No worries, and yes, the third install was using the recently published 4.2.0, which appears to have solved the styling issues - great job pre-empting the fix!

@JasonYao :smiley:
Keep up those awesome bug reports.


bug: fixed trash labware details page not linked

Overview
The link to the labware details page for the Opentrons fixed trash does not work on opentrons.com
Steps to reproduce

Navigate to the fixed trash in the online labware library
Click on the hyperlinked blue text "Opentrons Fixed Trash"
See the dead page

Here are the relevant dimensions that should show up on the fixed trash details page:
Well Count
1
MAX VOLUME
3200 mL
Footprint (mm)
LENGTH
210.75
WIDTH
79.75
HEIGHT
93.50
Well Measurements (mm)
DEPTH
187.50
X-SIZE
107.11
Y-SIZE
165.67
Spacing (mm)
X-OFFSET
105.38
Y-OFFSET
39.88
X-SPACING
0.00
Y-SPACING
0.00
Current behavior
No labware details page shows up when clicking on the opentrons fixed trash in the online labware library.
Expected behavior

[ ] All labware details should show up when clicking the fixed trash link

New JIRA ticket: https://opentrons.atlassian.net/browse/RAUT-206


Not working when "produce()" object across iframes

When we have multiple iframes with the same origin, we can share objects between different iframe window. But immer produce has a check isPlainObject(), there is a condition "Object.getPrototype(xxx) === Object.prototype", which makes this scenario broken. Because the window.Object.prototype in different iframes have different instances. This condition will return false.
The basic repro example is:
in iframe, we can set window.parent.test = {a: 1}   (this can work when iframe and parent have same origin)
in parent window, we can access window.test and produce(window.test, (draft) => { } ), but this action will get exception "Not a plain object"
Can we change the condition "Object.getPrototype(xxx) === Object.prototype" in the function isPlainObject() ?

Feel free to verify whether your solution or the one mentioned here works and submit a PR :)

:tada: This issue has been resolved in version 8.0.4 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


processor.tokenize support non-lang-syms

We can assign --non-lang-syms in the old io framework, such as text2token.py and format_data. It leaves a special symbol([laugh] or [noise]) as it is.
But in uio we don't support this feature.
Do we have any plan for this? Or how do I achieve the same purpose?

I didn't go through the details of  --non-lang-syms. do you have the  bandwidth to support that?

I'd like to but I'm not free recently. Maybe next week or after.


CRS-21 update ships

OCISLY departed yesterday (2020-12-01) towed by Tug Hawk.

Updated

As always, thank you.

I think that UTC it's uncorrect. It should be around 13.45 UTC not 8.19 UTC.

Launch has been scrubbed. Now it's set to 2020-12-06 16:17 UTC.

Updated


warnings from internal use of apply_ufunc

Another follow up from #4060: quantile now emits a FutureWarning:
Minimal Complete Verifiable Example:
xr.DataArray([1, 2, 3]).quantile(q=0.5)

~/.conda/envs/ipcc_ar6/lib/python3.7/site-packages/xarray/core/variable.py:1866:
FutureWarning: ``output_sizes`` should be given in the ``dask_gufunc_kwargs``
parameter. It will be removed as direct parameter in a future version.
kwargs={"q": q, "axis": axis, "interpolation": interpolation},

We should probably check the warnings in the test suite - there may be others.

@mathause Then these internal uses should be adapted to the new use of apply_ufunc, right? As I feel somehow committed to #4060 I could work on this.

Then these internal uses should be adapted to the new use of apply_ufunc, right?

:+1:

I could work on this.

Only if you have time! No need to feel obligated :) but it should be relatively simple.

@dcherian Thanks, I'll come up with a PR over the weekend. But this should not block others to take over if this becomes urgent.

No worry this is only on master and not in a released version.


As a Validator, I want to have a way to attest Job execution, so that consensus can be achieved.

Acceptance Criteria

Attestator Go Routine able to attest TXn on Terra Network
Also needs ability to attest Txn NOT happening
Sends report

Closing as shipped.


[BUG] MonitorServiceImpl textarea value issue

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
if judgment condition have bug
 if (!StringUtils.hasText(param.getValue())) {}
Expected Behavior
No response
Steps To Reproduce
No response
Environment
HertzBeat version(s):

Debug logs
No response
Anything else?
text need to judge the value

If the textarea has a value, an exception will be thrown.


chore: refresh the infra

#915  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Brooooooklyn and the rest of your teammates on  Graphite


Fix lagging 3d view for big data in direct color mode

Description
This is PR on top of #6411.
It implements reducing dtype for direct color mode.
To see these PR changes, if #6411 is not merged, see https://github.com/Czaki/napari/compare/bugfix/draw_performance...Czaki:napari:bugfix/draw_performance_direct?expand=1

Ah, I understand now why it's harder than I initially thought. But I think we can make it less complex @Czaki and get the benefit for most use cases. (At least, I think that should be our initial approach, then implement this only if someone complains about performance and this could fix it.)
Here's my suggestion: find the maximum value in the color dict, then make the texture have the dtype needed for 1 + that value. (The 1+ is for the None colour.) Then, if the dtype of the labels is higher than that value, you can use np.clip() to cast the original volume to the texture.
So, in cases where you have the labels [0, 1, 2**35] and just those two colours, my method will be very inefficient, compared to what you have implemented here. However, that is a rare use case — for 99% of use cases, it will be enough. And the code will be much much easier. (I think???)
What do you think?

For me, labeling such data on only a few region types (few colors) could be a typical use case, and your solution will not provide performance improvement.

Um, yes it will, as it can use a uint16 texture for almost all region types except in the case when they include the one final value? Also, since they don't have values greater than 65535, they would have accessible to them the workaround of just passing in a uint16 array instead of uint32. (Indeed I think they can do this with a right-click?)
Also, there are more than 255 values there so if you try to colour them all you will need uint16 anyway.
In some cases you might get away with a uint8 texture, sure, but I suspect uint16 already provides a significant improvement anyway...

@jni. I know that it is not what you request, but see this massive code removal in napari/_vispy/layers/labels.py

@Czaki Simplification is great, but unfortunately I don't think this solves the problem — a 1D hash is limited by the maximum texture dimensions which is typically 2**15 ≈ 32K. The issues with direct colormaps come from users with millions of labels in the direct colormap. That's the whole reason we implemented a 2D hash table in #3308.
Apart from that I can't ever run one of the test scripts from those early issues about direct colormap aliasing:
import matplotlib
import numpy as np
import pandas as pd

import napari

# Set the number of steps
nb_steps = 10000

# Create a dummy label image
base = np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint16')
label_img = np.repeat(
        base.reshape([1, base.shape[0]]), int(nb_steps/10), axis=0
        )

# Add a harder test case: Randomly order the label values.
# But assign them monotonously increasing feature values
# (=> if it's off by one, the error is more visible)

shuffled = np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint16')
np.random.shuffle(shuffled)
label_img_shuffled = np.repeat(
        shuffled.reshape([1, shuffled.shape[0]]), int(nb_steps/10), axis=0
        )
df_shuffled = pd.DataFrame([shuffled, np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint16')]).T
df_shuffled.columns = ['label', 'feature']

# calculate the colormaps manually
lower_contrast_limit = 1
upper_contrast_limit = nb_steps

df = df_shuffled

df['feature_scaled_shuffled'] = (
    (df['feature'] - lower_contrast_limit)
    / (upper_contrast_limit - lower_contrast_limit)
)
colors = matplotlib.colormaps['viridis'](df['feature_scaled_shuffled'])
colormap_shuffled = dict(zip(df['label'].astype(int), colors))

df['feature_scaled_not_shuffled'] = (
    (df['label'] - lower_contrast_limit)
    / (upper_contrast_limit - lower_contrast_limit)
)
colors_ordered = (
    matplotlib.colormaps['viridis'](df['feature_scaled_not_shuffled'])
)
colormap_ordered = dict(zip(df['label'].astype(int), colors_ordered))

# Add to napari
viewer = napari.Viewer()
viewer.add_image(label_img, colormap='viridis')
labels_layer_shuffled = viewer.add_labels(label_img_shuffled, opacity=100)
#viewer.add_image(texel_pos_img, rgb=True)
labels_layer_ordered = viewer.add_labels(label_img, opacity=100)
viewer.grid.enabled = True
viewer.grid.shape = -1, 1

# Set the label image colormaps
labels_layer_shuffled.color = colormap_shuffled
labels_layer_ordered.color = colormap_ordered

# TMP debugging stuff
vlab = viewer.window._qt_viewer.layer_to_visual[viewer.layers[-1]]

napari.run()

On this branch, the script above fails with:
  File "/Users/jni/projects/napari/examples/dev/direct-colormap-aliasing-test.py", line 60, in <module>
    labels_layer_shuffled.color = colormap_shuffled
  File "/Users/jni/projects/napari/napari/layers/labels/labels.py", line 643, in color
    self.color_mode = color_mode
  File "/Users/jni/projects/napari/napari/layers/labels/labels.py", line 818, in color_mode
    self.events.colormap()  # If remove this emitting, connect shader update to color_mode
  File "/Users/jni/projects/napari/napari/utils/events/event.py", line 771, in __call__
    self._invoke_callback(cb, event if pass_event else None)
  File "/Users/jni/projects/napari/napari/utils/events/event.py", line 809, in _invoke_callback
    _handle_exception(
  File "/Users/jni/projects/napari/napari/utils/events/event.py", line 796, in _invoke_callback
    cb(event)
  File "/Users/jni/projects/napari/napari/_vispy/layers/labels.py", line 170, in _on_colormap_change
    val_texture = build_textures_from_dict(color_dict)
  File "/Users/jni/projects/napari/napari/_vispy/layers/labels.py", line 101, in build_textures_from_dict
    data[key] = value
IndexError: index 275 is out of bounds for axis 0 with size 258

(I think this is happening because we are assigning a new dict after the remapping has happened once?)
At any rate, I think we need to add a test with ~100K distinct colors in the dictionary, because we have already had that many reported...

Ok. I have wrongly created a color dict. Thanks for spotting it. This is single line bugfix. But I see a problem on thumbnail:

@Czaki but if you change n_steps to 100K, all hell breaks loose:

and the warnings in the terminal:
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.
data shape (10000, 100000) exceeds GL_MAX_TEXTURE_SIZE 16384 in at least one axis and will be downsampled. Rendering is currently in 2D mode.

(Edit: never mind the issue is on main too 🤯)

(Obviously the texture size makes sense on main. 🤦 But i don't know why the double hashing isn't working properly...)

Hmmm. Seems to work fine on this branch... But can you explain why @Czaki? You shouldn't be able to have more than 16K colors in the 1D texture?

This is not even working in with image layer and colormap applied.

Yeah sorry clearly I need to make more changes than just nb_steps to the script. But what do you think of my concern? Do you think you can accommodate more than GL_MAX_TEXTURE_SIZE colors in this implementation, and if so, how? 🤔

A short check shows that direct colormap from this example has only 169 elements. So there is no problem with the lookup texture size.

ah ok. So I need to fiddle with the example to add more colors. This can be done. 😈
Unfortunately I have to go to bed now. But I think we should be able to support large dicts, certainly more than 16K elements.

Few notes to the script.
Using uin16 as the data type broke any rendering. as it blocks any values above 2**16 and leads to wrong results.
Here fixed script that shows that labels are correct. The Image is wrongly rendered...
import matplotlib
import numpy as np
import pandas as pd

import napari

# Set the number of steps
nb_steps = 100000

# Create a dummy label image
base = np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint32')
label_img = np.repeat(
        base.reshape([1, base.shape[0]]), int(nb_steps/10), axis=0
        )

# Add a harder test case: Randomly order the label values.
# But assign them monotonously increasing feature values
# (=> if it's off by one, the error is more visible)

shuffled = np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint32')
np.random.shuffle(shuffled)
label_img_shuffled = np.repeat(
        shuffled.reshape([1, shuffled.shape[0]]), int(nb_steps/10), axis=0
        )
df_shuffled = pd.DataFrame([shuffled, np.linspace(start=1, stop=nb_steps, num=nb_steps).astype('uint32')]).T
df_shuffled.columns = ['label', 'feature']

# calculate the colormaps manually
lower_contrast_limit = 1
upper_contrast_limit = nb_steps

df = df_shuffled

df['feature_scaled_shuffled'] = (
    (df['feature'] - lower_contrast_limit)
    / (upper_contrast_limit - lower_contrast_limit)
)
colors = matplotlib.colormaps['viridis'](df['feature_scaled_shuffled'])
colormap_shuffled = dict(zip(df['label'].astype(int), colors))

df['feature_scaled_not_shuffled'] = (
    (df['label'] - lower_contrast_limit)
    / (upper_contrast_limit - lower_contrast_limit)
)
colors_ordered = (
    matplotlib.colormaps['viridis'](df['feature_scaled_not_shuffled'])
)
colormap_ordered = dict(zip(df['label'].astype(int), colors_ordered))
colormap_ordered[None] = "red"
colormap_ordered[0] = "blue"

# Add to napari
viewer = napari.Viewer()
viewer.add_image(label_img, colormap='viridis')
labels_layer_shuffled = viewer.add_labels(label_img_shuffled, opacity=100)
#viewer.add_image(texel_pos_img, rgb=True)
labels_layer_ordered = viewer.add_labels(label_img, opacity=100, name="ordered")
viewer.grid.enabled = True
viewer.grid.shape = -1, 1

# Set the label image colormaps
labels_layer_shuffled.color = colormap_shuffled
labels_layer_ordered.color = colormap_ordered

# TMP debugging stuff
vlab = viewer.window._qt_viewer.layer_to_visual[viewer.layers[-1]]

napari.run()

It is working now.
Here script for 2**32 colors:
import numpy as np
import napari
from itertools import product

print("Creating data...")
data = np.zeros((256, 256), dtype=np.uint32)
color = {}

b = 1
count = 0
skip = 0
steep = 1
for r, g in product(range(0, 256, steep), repeat=2):
    val = r * 256 + g
    count += 1
    data[r:r+steep, g:g+steep] = val
    color[val] = (1-r/255, 1-g/255, 0, 1)

print(f"Created {count} labels. Skipped {skip} labels.")
# assert np.sum(data == 0) <= 1, np.sum(data == 0)

color[None] = "pink"

print("Adding data to napari...")
viewer = napari.Viewer()

viewer.add_labels(data, color=color, name="labels", opacity=1)
# viewer.add_image(data.astype(np.float32), name="image", opacity=1, colormap="viridis")
print("Done.")

napari.run()

Some notes from a live discussion @Czaki, @melonora and I had during the community meeting and . Looking at the code, these things came to mind as a checklist to make sure this solution works as expected:

[ ] check all the original direct colormap issues closed by #3308, and make sure that they are addressed here.
[ ] make sure we have sufficient tests
[ ] check slicing performance in 0.4.18, after #3308, after @Czaki's shader improvements (#6179) and after #6411 and this one. (I think this will be a performance regression, but the question is how much. It's a tradeoff between how much work to do on the CPU and how much to do at interaction time in the GPU.)
[x] check the dtype conversion in gl.py — VisPy won't take uint32 as input to texture. @Czaki has done this in #6456
[ ] direct colormap is broken on main since #6411 — figure out why and why tests didn't catch it... 🤔

For last point, the #6461 fix colormaps. The reverting dict was hard...

@Czaki I've updated the list to make a note of that, thanks!

@jni I have updated the names of functions and cleanup code a little.

Looking on non Qt benchmarks it seems ok for me:

| Change   | Before [0312cacd]    | After [0944219f]    |   Ratio | Benchmark (Parameter)                                                                              |
|----------|----------------------|---------------------|---------|----------------------------------------------------------------------------------------------------|
| -        | 2.27±0.02ms          | 1.05±0.03ms         |    0.46 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(2048, <class 'numpy.uint8'>)          |
| -        | 1.72±0.03ms          | 572±3μs             |    0.33 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(1024, <class 'numpy.uint8'>)          |
| -        | 1.48±0.04ms          | 417±2μs             |    0.28 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(128, <class 'numpy.uint8'>)           |
| -        | 1.48±0.01ms          | 415±7μs             |    0.28 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(32, <class 'numpy.uint8'>)            |
| -        | 1.61±0.03ms          | 453±1μs             |    0.28 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(512, <class 'numpy.uint8'>)           |
| -        | 1.45±0.01ms          | 412±2μs             |    0.28 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(64, <class 'numpy.uint8'>)            |
| -        | 1.51±0ms             | 413±5μs             |    0.27 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(16, <class 'numpy.uint8'>)            |
| -        | 1.71±0.03ms          | 424±6μs             |    0.25 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(256, <class 'numpy.uint8'>)           |
| -        | 1.37±0.01ms          | 312±1μs             |    0.23 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(128, <class 'numpy.uint8'>)  |
| -        | 1.35±0.01ms          | 312±1μs             |    0.23 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(64, <class 'numpy.uint8'>)   |
| -        | 1.43±0.01ms          | 311±3μs             |    0.22 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(16, <class 'numpy.uint8'>)   |
| -        | 1.39±0.01ms          | 306±3μs             |    0.22 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(32, <class 'numpy.uint8'>)   |
| -        | 1.46±0.01ms          | 309±1μs             |    0.21 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(512, <class 'numpy.uint8'>)  |
| -        | 1.41±0.01ms          | 287±6μs             |    0.2  | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(1024, <class 'numpy.uint8'>) |
| -        | 1.40±0.03ms          | 281±4μs             |    0.2  | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(2048, <class 'numpy.uint8'>) |
| -        | 1.57±0.01ms          | 310±5μs             |    0.2  | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(256, <class 'numpy.uint8'>)  |
| -        | 1.46±0.03ms          | 280±2μs             |    0.19 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(4096, <class 'numpy.uint8'>) |
| -        | 96.4±2ms             | 180±4μs             |    0    | benchmark_labels_layer.Labels2DColorDirectSuite.time_fill(4096, <class 'numpy.uint8'>)             |
| Change   | Before [0312cacd]    | After [0944219f]    |   Ratio | Benchmark (Parameter)                                                                              |
|----------|----------------------|---------------------|---------|----------------------------------------------------------------------------------------------------|
| +        | 9.51±0.09μs          | 33.6±0.3μs          |    3.53 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(16, <class 'numpy.int32'>)   |
| +        | 10.7±0.1μs           | 35.7±0.5μs          |    3.33 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(32, <class 'numpy.int32'>)   |
| +        | 1.16±0.01ms          | 3.84±0.03ms         |    3.31 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(2048, <class 'numpy.int32'>) |
| +        | 1.18±0.01ms          | 3.89±0.03ms         |    3.3  | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(4096, <class 'numpy.int32'>) |
| +        | 1.15±0.02ms          | 3.75±0.02ms         |    3.26 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(1024, <class 'numpy.int32'>) |
| +        | 1.18±0.03ms          | 3.75±0.02ms         |    3.18 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(512, <class 'numpy.int32'>)  |
| +        | 1.20±0ms             | 3.77±0.01ms         |    3.15 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(16, <class 'numpy.int32'>)   |
| +        | 1.20±0.01ms          | 3.77±0.01ms         |    3.13 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(32, <class 'numpy.int32'>)   |
| +        | 1.23±0.01ms          | 3.80±0.01ms         |    3.1  | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(128, <class 'numpy.int32'>)  |
| +        | 1.22±0.01ms          | 3.76±0.01ms         |    3.08 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(64, <class 'numpy.int32'>)   |
| +        | 1.32±0.01ms          | 3.95±0.01ms         |    2.99 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(512, <class 'numpy.int32'>)           |
| +        | 13.4±0.2μs           | 39.9±0.2μs          |    2.98 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(64, <class 'numpy.int32'>)   |
| +        | 1.30±0.01ms          | 3.87±0.01ms         |    2.97 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(16, <class 'numpy.int32'>)            |
| +        | 1.32±0.01ms          | 3.92±0.02ms         |    2.97 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(64, <class 'numpy.int32'>)            |
| +        | 1.32±0.02ms          | 3.89±0.01ms         |    2.94 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(128, <class 'numpy.int32'>)           |
| +        | 1.33±0.02ms          | 3.90±0.01ms         |    2.94 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(32, <class 'numpy.int32'>)            |
| +        | 1.39±0.02ms          | 3.74±0.03ms         |    2.69 | benchmark_labels_layer.Labels2DColorDirectSuite.time_update_thumbnail(256, <class 'numpy.int32'>)  |
| +        | 1.51±0.02ms          | 3.92±0.03ms         |    2.6  | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(256, <class 'numpy.int32'>)           |
| +        | 1.63±0.05ms          | 4.22±0.03ms         |    2.59 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(1024, <class 'numpy.int32'>)          |
| +        | 22.7±0.1μs           | 57.1±0.3μs          |    2.51 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(128, <class 'numpy.int32'>)  |
| +        | 58.0±0.5μs           | 124±0.5μs           |    2.13 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(256, <class 'numpy.int32'>)  |
| +        | 3.03±0.09ms          | 5.71±0.08ms         |    1.88 | benchmark_labels_layer.Labels2DColorDirectSuite.time_refresh(2048, <class 'numpy.int32'>)          |
| +        | 197±1μs              | 368±2μs             |    1.87 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(512, <class 'numpy.int32'>)  |
| +        | 727±4μs              | 1.34±0.01ms         |    1.85 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(1024, <class 'numpy.int32'>) |
| +        | 3.18±0.01ms          | 5.12±0.1ms          |    1.61 | benchmark_labels_layer.Labels2DColorDirectSuite.time_raw_to_displayed(2048, <class 'numpy.int32'>) |

It may be nice to merge #6530 before to see how this PR impacts memory.

| -        | 2.29M                | 1.51M               |    0.66 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(512, <class 'numpy.int32'>)              |
| -        | 8.58M                | 5.44M               |    0.63 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(1024, <class 'numpy.int32'>)             |
| -        | 33.7M                | 21.2M               |    0.63 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(2048, <class 'numpy.int32'>)             |
| -        | 134M                 | 84.1M               |    0.63 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(4096, <class 'numpy.int32'>)             |
| -        | 520k                 | 274k                |    0.53 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(256, <class 'numpy.uint8'>)
| -        | 1.5M                 | 471k                |    0.31 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(512, <class 'numpy.uint8'>)              |
| -        | 5.43M                | 1.26M               |    0.23 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(1024, <class 'numpy.uint8'>)             |
| -        | 21.2M                | 4.4M                |    0.21 | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(2048, <class 'numpy.uint8'>)             |
| -        | 84.1M                | 17M                 |    0.2  | benchmark_labels_layer.Labels2DColorDirectSuite.mem_layer(4096, <class 'numpy.uint8'>)             |


Blazeborn are not very well adapted to the nether (not much food)

There's nothing to eat. Blazeborn should have the ability to eat things in the nether, besides zombie pigman flesh. Here are some suggestions:

nether wart
cooked nether wart? Or somesuch.
Ooh, maybe there could be netherwart bread?
might be a good idea to add umbral flesh blocks from Cinderscapes -- or perhaps just eating a cooked version.
 Mushrooms are also farmable. If you manage to find some fossils, you have access to quite a bit of bone meal, which can accelerate mushroom growth!
As a general rule, I do not like to add content (new items, blocks, etc) to this mod. If other mods add content with more food to the Nether, you are welcome to use those!
Surviving in the Nether is difficult, I agree, but after all it's supposed to be a drawback.

Aw, okay. That's fair enough! Maybe it works better in a more vanilla-y modpack, but I found huge difficulty with my modpack, during playtesting -- like, going hours without any food besides rotten flesh. I have a lot of stuff changing world generation, tho. Thinking about it, I haven't played vanilla in awhile, and it might be easier to get a hold of mushrooms in vanilla world generation.
I'll probably just make my own companion mod for your mod, for modpacks where getting hold of mushrooms is tougher! ^_^


can not delete Vampire: InternalServerError: column "dde51130-637a-11ec-81ea-854924cb81a5" does not exist

EdgeDB Version: EdgeDB 1.0-rc.3+147eb62 (repl 1.0.0-rc.4+eb50f45)
OS Version: Ubuntu 20.04.3

Steps to Reproduce:

Download project https://drive.google.com/file/d/1lkK-0AJRg_0QONZnGHiojdMygdoWTGsY/view?usp=sharing
unpack project
go to project directory and init project edgedb project init
in edgedb shell insert a Vampire object INSERT Vampire {name := 'Some Name'};
try to delete Vampire DELETE Vampire;
at this moment I get this error:
edgedb error: InternalServerError: column "dde51130-637a-11ec-81ea-854924cb81a5" does not exist Hint: This is most likely a bug in EdgeDB. Please consider opening an issue ticket at https://github.com/edgedb/edgedb/issues/new?template=bug_report.md Server traceback:    edb.errors.InternalServerError: column "dde51130-637a-11ec-81ea-854924cb81a5" does not exist

Current error is caused by one of migrations in the project. If I create migration from scratch for existing schema there are no problems with deleting Vampire.

I think the issue is going to be some busted trigger stuff.

Minimized test case:
create type Foo;
create type Bar { create link foo -> Foo; };
alter type Bar { drop link foo; };
insert Foo;
delete Foo;


CUDA: unspecified launch failure on CI Windows

Description
CI on windows keeps giving CUDA: unspecified launch failure. It has been retriggered many times.
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17400/20/pipeline/

@haojin2 The first failing test is test_operator_gpu.test_np_nan_to_num.

Please first disable then investigate.

#17630 still fails on windows-gpu with test_np_nan_to_num disabled, the error starts to occur on the test immediately after test_np_nan_to_num, meanwhile there were other PRs passing the windows-gpu build, as a result, I suspect the problem is not with the test_np_nan_to_num itself.

Another one for unrelated PR #17487  - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/PR-17487/24/pipeline

This issue appears fixed by the toolchain updgrade https://github.com/apache/incubator-mxnet/pull/17962


Added a loading spinner components in web components.

Added a loading spinner components in web components.
used the given color in #9 .

LGTM! Thanks for your contribution. But can you add a short video (screencast) or image showing how the spinner is working and looking?

Awesome! I am merging.


Fix typescript errors during build

There is a Typescript error when building because of an unassigned type

Closed via #251


Add pinch to zoom in all the circuit image. Some images are very small and viewing the text is difficult without zooming it.

Is your feature request related to a problem? Please describe.
The circuit displayed in the project detail page is very small and there is no option to zoom.
Describe the solution you'd like
Add a full screen icon with the image that redirects the user to another page with the full screen of the image, this view would allow pinch to zoom so that the user can easily zoom and view the entire circuit
Describe alternatives you've considered
Alternatively, I could add a pinch to zoom in the existing page that allows to expand the image.
Additional context

Are you working on this? (Yes/No)
Yes

Can I work on this? Can I add a full-screen view with pinch zoom on the image click?

Can I work on this? Can I add a full-screen view with pinch zoom on the image click?

Hi @NishantChandla I have already started working on this and will be creating a PR in the coming days. Thanks

Hi @NishantChandla I have already started working on this and will be creating a PR in the coming days. Thanks

Okay!

Okay!


Rancher deletes PV VMDK when node is deleted

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):

Provision a cluster with vSphere provider
Add vSphere as cloud provider & as storage class provisioner
Provision some disks using PVC
Mount those disks to containers using Pod and make a note of the node where the Pod is running
Delete that specific node from UI

Result:
Watch your precious data vanish when Rancher deletes the node together with the corresponding PV VMDK attached to it.
Other details that may be helpful:
Rancher should first deattach the VMDKs before deleting node so that no data loss occurs.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
v2.5-ac483705ad321e9bdfc868bcf72a0c6ca167d7e1-head
Installation option (single install/HA):
single install

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Infrastructure Provider
Machine type (cloud/VM/metal) and specifications (CPU/memory): VM
Kubernetes version (use kubectl version): v1.19.7-rancher1-1

We had the same problem two times now with Rancher v2.4.8
When removing a node from the cluster the PV for cluster monitoring got deleted.