Exponential Moving Average of trained variables in tf.contrib.learn.estimators.estimator

Hi,
As mentioned in the cifar10 tutorial, it is often useful to use an exponential moving average of the trained variables at prediction time.
From my understanding, in it's current form, the class BaseEstimator does not allow to do so.
The _infer_model method of BaseEstimator would need to create a saver object that restores the averaged version of the trained variables, and the saver object would then need to be cascaded into graph_actions.infer and ultimately down to graph_actions.run_feeds.
Whilst overriding a class method is not a big deal, I was wondering if there was a neat way of achieving the end goal without having to refactor the graph_actions.
Best Regards,
Marco

@martinwicke Can you comment please?

I think if you simply add the moving averages to the graph and explicitly
add their output to the VARIABLES collection they will be saved and you can
use them in the inference graph.
This is assuming you use the Estimator class. There should be no need to
override anything.
On Mon, Aug 15, 2016 at 16:37 Daniel W Mane<EMAIL_ADDRESS>wrote:

@martinwicke https://github.com/martinwicke Can you comment please?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/3460#issuecomment-239960552,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAjO_XL8t3C19ix8K12DAO0Uz5Ip8oOUks5qgPgbgaJpZM4JSlat
.

@martinwicke, is the gist close to what you had in mind?
Estimator and moving average of trained variables
A few points worth mentioning:

There  is no need to explicitly add the shadow variables to the VARIABLES collection. This is automatically done.

At inference time I assign the value of the shadow variables to the graph variables using tf.assign

With [tf.control_dependencies and tf.identity] (https://gist.github.com/marcoadurno/1359e49e50a9237b68181a29d104af8c#file-estimator_ema-py-L39) I can enforce the output of the linear layer to depend on the assign operation

Thanks
Marco

That looks right to me. Nicely done.
On Wed, Sep 7, 2016 at 3:28 AM, marcoadurno<EMAIL_ADDRESS>wrote:

@martinwicke https://github.com/martinwicke, is the gist close to what
you had in mind?
Estimator and moving average of trained variables
https://gist.github.com/marcoadurno/1359e49e50a9237b68181a29d104af8c
A few points worth mentioning:

There is no need to explicitly add the shadow variables to the
VARIABLES collection. This is automatically done.
https://gist.github.com/marcoadurno/1359e49e50a9237b68181a29d104af8c#file-estimator_ema-py-L31
At inference time
https://gist.github.com/marcoadurno/1359e49e50a9237b68181a29d104af8c#file-estimator_ema-py-L37
I assign the value of the shadow variables to the graph variables using
tf.assign
https://gist.github.com/marcoadurno/1359e49e50a9237b68181a29d104af8c#file-estimator_ema-py-L16
With tf.control_dependencies and tf.identity
https://gist.github.com/marcoadurno/1359e49e50a9237b68181a29d104af8c#file-estimator_ema-py-L39
I can enforce the output of the linear layer to depend on the assign
operation
Thanks
Marco
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/3460#issuecomment-245239464,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAjO_QmiaysDxARg7UDhZ1naQAKLvNxBks5qnpHMgaJpZM4JSlat
.

Thanks for the hint. I'll close the issue as the API already supports the requested functionality.

@marcoadurno
Although things have been going on for a long time, there is still something to explain.
The restoring operation should be executed only once, and before any inference.
In your code this operation will be executed every time _infer() is invoked, and if there are multiple layers (for example, your model has 3 dense layers), you can't restore the variables from the very beginning, you can only restore them after all layers (thus all variables) have been created. but when the variables are created, they have been calculated at least once, so the first batch will get the wrong result.
The way to solve this problem is restoring the variables in other place.

In the training phase, create ema instance and track moving averages:
def track_params_averages():
    """
    Track the moving averages of parameters.
    Must be invoked after `infer()` and before `train()`.

    :return:
            ema:                    `tf.train.ExponentialMovingAverage`
            params_averages_op:     operator that tracking averages
    """
    ema = tf.train.ExponentialMovingAverage(FLAGS.params_moving_average_decay)
    params = tf.trainable_variables()
    params_averages_op = ema.apply(params)
    return ema, params_averages_op

def model_fn_train(features, labels, params):
    model = Model(params)
    logits = model.infer(features)

    _, params_averages_op = track_params_averages()

    ....

    # Note that the UPDATE_OPS must be executed before applying gradients,
    # and tracking variables averages must be done after applying gradients.
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        gradients = opt.compute_gradients(total_loss)
        apply_gradients_op = opt.apply_gradients(gradients, global_step=global_step)

    with tf.control_dependencies([apply_gradients_op]):
        params_averages_op = tf.identity(params_averages_op)

    return tf.estimator.EstimatorSpec(
        mode=tf.estimator.ModeKeys.TRAIN, loss=loss, train_op=train_op)

In the evaluation phase, create a SessionRunHook to restore the averages:
class RestoreParametersAverageValues(tf.train.SessionRunHook):
   """
   Replace parameters with their moving averages.
   This operation should be executed only once, and before any inference.
   """
   def __init__(self, ema):
       """
       :param ema:         tf.train.ExponentialMovingAverage
       """
       super(RestoreParametersAverageValues, self).__init__()
       self._ema = ema
       self._restore_ops = None

   def begin(self):
       """ Create restoring operations before the graph been finalized. """
       ema_variables = tf.moving_average_variables()
       self._restore_ops = [tf.assign(x, self._ema.average(x)) for x in ema_variables]

   def after_create_session(self, session, coord):
       """ Restore the parameters right after the session been created. """
       session.run(self._restore_ops)

def model_fn_eval(features, labels, params):
   model = Model(params)
   logits = model.infer(features)

   ema, _ = track_params_averages()

   ...

   return tf.estimator.EstimatorSpec(
       mode=tf.estimator.ModeKeys.PREDICT,
       predictions={...},
       prediction_hooks=[RestoreParametersAverageValues(ema)],
   )

@martinwicke, is the gist close to what you had in mind?
Estimator and moving average of trained variables
A few points worth mentioning:

There  is no need to explicitly add the shadow variables to the VARIABLES collection. This is automatically done.
At inference time I assign the value of the shadow variables to the graph variables using tf.assign
With tf.control_dependencies and tf.identity I can enforce the output of the linear layer to depend on the assign operation

Thanks
Marco

what if you want to continue training the model after the model after evaluation?

@marcoadurno
Although things have been going on for a long time, there is still something to explain.
The restoring operation should be executed only once, and before any inference.
In your code this operation will be executed every time _infer() is invoked, and if there are multiple layers (for example, your model has 3 dense layers), you can't restore the variables from the very beginning, you can only restore them after all layers (thus all variables) have been created. but when the variables are created, they have been calculated at least once, so the first batch will get the wrong result.
The way to solve this problem is restoring the variables in other place.

In the training phase, create ema instance and track moving averages:def track_params_averages():
    """
    Track the moving averages of parameters.
    Must be invoked after `infer()` and before `train()`.

    :return:
            ema:                    `tf.train.ExponentialMovingAverage`
            params_averages_op:     operator that tracking averages
    """
    ema = tf.train.ExponentialMovingAverage(FLAGS.params_moving_average_decay)
    params = tf.trainable_variables()
    params_averages_op = ema.apply(params)
    return ema, params_averages_op

def model_fn_train(features, labels, params):
    model = Model(params)
    logits = model.infer(features)

    _, params_averages_op = track_params_averages()

    ....

    # Note that the UPDATE_OPS must be executed before applying gradients,
    # and tracking variables averages must be done after applying gradients.
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        gradients = opt.compute_gradients(total_loss)
        apply_gradients_op = opt.apply_gradients(gradients, global_step=global_step)

    with tf.control_dependencies([apply_gradients_op]):
        params_averages_op = tf.group(params_averages_op)

    return tf.estimator.EstimatorSpec(
        mode=tf.estimator.ModeKeys.TRAIN, loss=loss, train_op=train_op)

In the evaluation phase, create a SessionRunHook to restore the averages:class RestoreParametersAverageValues(tf.train.SessionRunHook):
   """
   Replace parameters with their moving averages.
   This operation should be executed only once, and before any inference.
   """
   def __init__(self, ema):
       """
       :param ema:         tf.train.ExponentialMovingAverage
       """
       super(RestoreParametersAverageValues, self).__init__()
       self._ema = ema
       self._restore_ops = None

   def begin(self):
       """ Create restoring operations before the graph been finalized. """
       ema_variables = tf.moving_average_variables()
       self._restore_ops = [tf.assign(x, self._ema.average(x)) for x in ema_variables]

   def after_create_session(self, session, coord):
       """ Restore the parameters right after the session been created. """
       session.run(self._restore_ops)

def model_fn_eval(features, labels, params):
   model = Model(params)
   logits = model.infer(features)

   ema, _ = track_params_averages()

   ...

   return tf.estimator.EstimatorSpec(
       mode=tf.estimator.ModeKeys.PREDICT,
       predictions={...},
       prediction_hooks=[RestoreParametersAverageValues(ema)],
   )

what is train_op  here?

@marcoadurno
Although things have been going on for a long time, there is still something to explain.
The restoring operation should be executed only once, and before any inference.
In your code this operation will be executed every time _infer() is invoked, and if there are multiple layers (for example, your model has 3 dense layers), you can't restore the variables from the very beginning, you can only restore them after all layers (thus all variables) have been created. but when the variables are created, they have been calculated at least once, so the first batch will get the wrong result.
The way to solve this problem is restoring the variables in other place.

In the training phase, create ema instance and track moving averages:def track_params_averages():
    """
    Track the moving averages of parameters.
    Must be invoked after `infer()` and before `train()`.

    :return:
            ema:                    `tf.train.ExponentialMovingAverage`
            params_averages_op:     operator that tracking averages
    """
    ema = tf.train.ExponentialMovingAverage(FLAGS.params_moving_average_decay)
    params = tf.trainable_variables()
    params_averages_op = ema.apply(params)
    return ema, params_averages_op

def model_fn_train(features, labels, params):
    model = Model(params)
    logits = model.infer(features)

    _, params_averages_op = track_params_averages()

    ....

    # Note that the UPDATE_OPS must be executed before applying gradients,
    # and tracking variables averages must be done after applying gradients.
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        gradients = opt.compute_gradients(total_loss)
        apply_gradients_op = opt.apply_gradients(gradients, global_step=global_step)

    with tf.control_dependencies([apply_gradients_op]):
        params_averages_op = tf.group(params_averages_op)

    return tf.estimator.EstimatorSpec(
        mode=tf.estimator.ModeKeys.TRAIN, loss=loss, train_op=train_op)

In the evaluation phase, create a SessionRunHook to restore the averages:class RestoreParametersAverageValues(tf.train.SessionRunHook):
   """
   Replace parameters with their moving averages.
   This operation should be executed only once, and before any inference.
   """
   def __init__(self, ema):
       """
       :param ema:         tf.train.ExponentialMovingAverage
       """
       super(RestoreParametersAverageValues, self).__init__()
       self._ema = ema
       self._restore_ops = None

   def begin(self):
       """ Create restoring operations before the graph been finalized. """
       ema_variables = tf.moving_average_variables()
       self._restore_ops = [tf.assign(x, self._ema.average(x)) for x in ema_variables]

   def after_create_session(self, session, coord):
       """ Restore the parameters right after the session been created. """
       session.run(self._restore_ops)

def model_fn_eval(features, labels, params):
   model = Model(params)
   logits = model.infer(features)

   ema, _ = track_params_averages()

   ...

   return tf.estimator.EstimatorSpec(
       mode=tf.estimator.ModeKeys.PREDICT,
       predictions={...},
       prediction_hooks=[RestoreParametersAverageValues(ema)],
   )

class RestoreParametersAverageValues(tf.train.SessionRunHook):
          def __init__(self, restore_weight_backups, assign_dep ):
              '''
              with tf.variable_scope('BackupVariables'):
                  backup_vars = [tf.get_variable(var.op.name, dtype=var.value().dtype, trainable=False,
                                                initializer=var.initialized_value())
                                for var in model_vars]

              save_weight_backups = tf.group(*(tf.assign(bck, var.read_value())
                                              for var, bck in zip(model_vars, backup_vars)))
              restore_weight_backups = tf.group(*(tf.assign(var, bck.read_value())
                                              for var, bck in zip(model_vars, backup_vars)))

              with tf.control_dependencies( [ save_weight_backups() ] ):
                assign_dep = tf.group(*[tf.assign(x, ema.average(x)) for x in model_vars])

              '''
              super(RestoreParametersAverageValues, self).__init__()
              self._assign_dep = assign_dep # assign ema var to var
              self._restore_weight_backups = restore_weight_backups # restore weight back

          def after_create_session(self, session, coord):
              """ Restore the parameters right before run. """
              session.run(self._assign_dep)

          def end(self, session):
               session.run(self._restore_weight_backups)

I was wondering, will this work? I am kinda confused

@marcoadurno
Although things have been going on for a long time, there is still something to explain.
The restoring operation should be executed only once, and before any inference.
In your code this operation will be executed every time _infer() is invoked, and if there are multiple layers (for example, your model has 3 dense layers), you can't restore the variables from the very beginning, you can only restore them after all layers (thus all variables) have been created. but when the variables are created, they have been calculated at least once, so the first batch will get the wrong result.
The way to solve this problem is restoring the variables in other place.

In the training phase, create ema instance and track moving averages:def track_params_averages():
    """
    Track the moving averages of parameters.
    Must be invoked after `infer()` and before `train()`.

    :return:
            ema:                    `tf.train.ExponentialMovingAverage`
            params_averages_op:     operator that tracking averages
    """
    ema = tf.train.ExponentialMovingAverage(FLAGS.params_moving_average_decay)
    params = tf.trainable_variables()
    params_averages_op = ema.apply(params)
    return ema, params_averages_op

def model_fn_train(features, labels, params):
    model = Model(params)
    logits = model.infer(features)

    _, params_averages_op = track_params_averages()

    ....

    # Note that the UPDATE_OPS must be executed before applying gradients,
    # and tracking variables averages must be done after applying gradients.
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        gradients = opt.compute_gradients(total_loss)
        apply_gradients_op = opt.apply_gradients(gradients, global_step=global_step)

    with tf.control_dependencies([apply_gradients_op]):
        params_averages_op = tf.group(params_averages_op)

    return tf.estimator.EstimatorSpec(
        mode=tf.estimator.ModeKeys.TRAIN, loss=loss, train_op=train_op)

In the evaluation phase, create a SessionRunHook to restore the averages:class RestoreParametersAverageValues(tf.train.SessionRunHook):
   """
   Replace parameters with their moving averages.
   This operation should be executed only once, and before any inference.
   """
   def __init__(self, ema):
       """
       :param ema:         tf.train.ExponentialMovingAverage
       """
       super(RestoreParametersAverageValues, self).__init__()
       self._ema = ema
       self._restore_ops = None

   def begin(self):
       """ Create restoring operations before the graph been finalized. """
       ema_variables = tf.moving_average_variables()
       self._restore_ops = [tf.assign(x, self._ema.average(x)) for x in ema_variables]

   def after_create_session(self, session, coord):
       """ Restore the parameters right after the session been created. """
       session.run(self._restore_ops)

def model_fn_eval(features, labels, params):
   model = Model(params)
   logits = model.infer(features)

   ema, _ = track_params_averages()

   ...

   return tf.estimator.EstimatorSpec(
       mode=tf.estimator.ModeKeys.PREDICT,
       predictions={...},
       prediction_hooks=[RestoreParametersAverageValues(ema)],
   )

what is train_op  here?

It doesn't matter, right?
Due to the confidentiality agreement, I can't upload the full project, the following is the newest version of model_fn_train():
def model_fn_train(features, labels, params):
    """
    All tensors must be created in the model_fn / input_fn function,
    otherwise tf.estimator.Estimator will throw an exception.
    """
    device_name = '/{}:{}'.format(DEVICE_CPU, DEVICE_ID_DEFAULT) if params.force_cpu else None
    with tf.device(device_name):
        model = Model(params)

        global_step = tf.train.get_or_create_global_step()

        logits = model.infer(features, mode=tf.estimator.ModeKeys.TRAIN)

        _, params_averages_op = model.track_params_averages()

        loss = model.get_loss(logits, labels)

        train_op = model.train(loss, global_step, params_averages_op)

        return tf.estimator.EstimatorSpec(
            mode=tf.estimator.ModeKeys.TRAIN, loss=loss, train_op=train_op)

after_create_session

Thanks for the explaning, But what if you want to restore the variable to the original variable to continue training?

class RestoreParametersAverageValues(tf.train.SessionRunHook):
          def __init__(self, restore_weight_backups, assign_dep ):
              '''
              with tf.variable_scope('BackupVariables'):
                  backup_vars = [tf.get_variable(var.op.name, dtype=var.value().dtype, trainable=False,
                                                initializer=var.initialized_value())
                                for var in model_vars]

              save_weight_backups = tf.group(*(tf.assign(bck, var.read_value())
                                              for var, bck in zip(model_vars, backup_vars)))
              restore_weight_backups = tf.group(*(tf.assign(var, bck.read_value())
                                              for var, bck in zip(model_vars, backup_vars)))

              with tf.control_dependencies( [ save_weight_backups ] ):
                assign_dep = tf.group(*[tf.assign(x, ema.average(x)) for x in model_vars])

              '''
              super(RestoreParametersAverageValues, self).__init__()
              self._assign_dep = assign_dep # assign ema var to var
              self._restore_weight_backups = restore_weight_backups # restore weight back

          def after_create_session(self, session, coord):
              """ Restore the parameters right before run. """
              session.run(self._assign_dep)

          def end(self, session):
               session.run(self._restore_weight_backups)

I was wondering, will this work? I am kinda confused
Thanks for the explaning, But what if you want to restore the variable to the original variable to continue training?

You don't need to save and restore the variables manually, 'tf.estimator.Estimator' will save and restore all the parameters automaticfally for you.
def train():
    config = tf.ConfigProto()
    config.intra_op_parallelism_threads = FLAGS.cpu_threads
    config.inter_op_parallelism_threads = FLAGS.cpu_threads
    config.gpu_options.allow_growth = True
    config.log_device_placement = FLAGS.log_device_placement

    estimator = tf.estimator.Estimator(
        model_fn=model_fn_train,
        model_dir=FLAGS.train_dir,
        config=tf.estimator.RunConfig(
            tf_random_seed=FLAGS.random_seed,
            log_step_count_steps=FLAGS.log_frequency,
            save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,
            session_config=config,
        ),
        params=FLAGS,
    )

    input_fn, iterator_initializer_hook = get_input(flags=FLAGS, mode=tf.estimator.ModeKeys.TRAIN)

    logging_hook = tf.train.LoggingTensorHook(tensors={
        'cross_entropy': 'loss/softmax_cross_entropy_loss/value',
    }, every_n_iter=FLAGS.log_frequency)
    hooks = [
        iterator_initializer_hook,
        logging_hook,
    ]
    if FLAGS.debug:
        hooks.append(tf_debug.LocalCLIDebugHook())

    estimator.train(input_fn=input_fn, steps=FLAGS.max_steps, hooks=hooks)

> ```python
> class RestoreParametersAverageValues(tf.train.SessionRunHook):
>           def __init__(self, restore_weight_backups, assign_dep ):
>               '''
>               with tf.variable_scope('BackupVariables'):
>                   backup_vars = [tf.get_variable(var.op.name, dtype=var.value().dtype, trainable=False,
>                                                 initializer=var.initialized_value())
>                                 for var in model_vars]
>
>               save_weight_backups = tf.group(*(tf.assign(bck, var.read_value())
>                                               for var, bck in zip(model_vars, backup_vars)))
>               restore_weight_backups = tf.group(*(tf.assign(var, bck.read_value())
>                                               for var, bck in zip(model_vars, backup_vars)))
>
>               with tf.control_dependencies( [ save_weight_backups ] ):
>                 assign_dep = tf.group(*[tf.assign(x, ema.average(x)) for x in model_vars])
>
>               '''
>               super(RestoreParametersAverageValues, self).__init__()
>               self._assign_dep = assign_dep # assign ema var to var
>               self._restore_weight_backups = restore_weight_backups # restore weight back
>
>           def after_create_session(self, session, coord):
>               """ Restore the parameters right before run. """
>               session.run(self._assign_dep)
>
>           def end(self, session):
>                session.run(self._restore_weight_backups)
> ```
>
>
> I was wondering, will this work? I am kinda confused
> Thanks for the explaning, But what if you want to restore the variable to the original variable to continue training?

You don't need to save and restore the variables manually, 'tf.estimator.Estimator' will save and restore all the parameters automatically for you.
```python
def train():
    config = tf.ConfigProto()
    config.intra_op_parallelism_threads = FLAGS.cpu_threads
    config.inter_op_parallelism_threads = FLAGS.cpu_threads
    config.gpu_options.allow_growth = True
    config.log_device_placement = FLAGS.log_device_placement

    estimator = tf.estimator.Estimator(
        model_fn=model_fn_train,
        model_dir=FLAGS.train_dir,
        config=tf.estimator.RunConfig(
            tf_random_seed=FLAGS.random_seed,
            log_step_count_steps=FLAGS.log_frequency,
            save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,
            session_config=config,
        ),
        params=FLAGS,
    )

    input_fn, iterator_initializer_hook = get_input(flags=FLAGS, mode=tf.estimator.ModeKeys.TRAIN)

    logging_hook = tf.train.LoggingTensorHook(tensors={
        'cross_entropy': 'loss/softmax_cross_entropy_loss/value',
    }, every_n_iter=FLAGS.log_frequency)
    hooks = [
        iterator_initializer_hook,
        logging_hook,
    ]
    if FLAGS.debug:
        hooks.append(tf_debug.LocalCLIDebugHook())

    estimator.train(input_fn=input_fn, steps=FLAGS.max_steps, hooks=hooks)

All you need is replacing the parameters with their moving averages.

Thanks! what is  get_input ?  I mean during  the online training, how do you restore the var back to original variable and continue training? Because we want to do evaluation during training.

> ```python
> class RestoreParametersAverageValues(tf.train.SessionRunHook):
>           def __init__(self, restore_weight_backups, assign_dep ):
>               '''
>               with tf.variable_scope('BackupVariables'):
>                   backup_vars = [tf.get_variable(var.op.name, dtype=var.value().dtype, trainable=False,
>                                                 initializer=var.initialized_value())
>                                 for var in model_vars]
>
>               save_weight_backups = tf.group(*(tf.assign(bck, var.read_value())
>                                               for var, bck in zip(model_vars, backup_vars)))
>               restore_weight_backups = tf.group(*(tf.assign(var, bck.read_value())
>                                               for var, bck in zip(model_vars, backup_vars)))
>
>               with tf.control_dependencies( [ save_weight_backups ] ):
>                 assign_dep = tf.group(*[tf.assign(x, ema.average(x)) for x in model_vars])
>
>               '''
>               super(RestoreParametersAverageValues, self).__init__()
>               self._assign_dep = assign_dep # assign ema var to var
>               self._restore_weight_backups = restore_weight_backups # restore weight back
>
>           def after_create_session(self, session, coord):
>               """ Restore the parameters right before run. """
>               session.run(self._assign_dep)
>
>           def end(self, session):
>                session.run(self._restore_weight_backups)
> ```
>
>
> I was wondering, will this work? I am kinda confused
> Thanks for the explaning, But what if you want to restore the variable to the original variable to continue training?

You don't need to save and restore the variables manually, 'tf.estimator.Estimator' will save and restore all the parameters automatically for you.
```python
def train():
    config = tf.ConfigProto()
    config.intra_op_parallelism_threads = FLAGS.cpu_threads
    config.inter_op_parallelism_threads = FLAGS.cpu_threads
    config.gpu_options.allow_growth = True
    config.log_device_placement = FLAGS.log_device_placement

    estimator = tf.estimator.Estimator(
        model_fn=model_fn_train,
        model_dir=FLAGS.train_dir,
        config=tf.estimator.RunConfig(
            tf_random_seed=FLAGS.random_seed,
            log_step_count_steps=FLAGS.log_frequency,
            save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,
            session_config=config,
        ),
        params=FLAGS,
    )

    input_fn, iterator_initializer_hook = get_input(flags=FLAGS, mode=tf.estimator.ModeKeys.TRAIN)

    logging_hook = tf.train.LoggingTensorHook(tensors={
        'cross_entropy': 'loss/softmax_cross_entropy_loss/value',
    }, every_n_iter=FLAGS.log_frequency)
    hooks = [
        iterator_initializer_hook,
        logging_hook,
    ]
    if FLAGS.debug:
        hooks.append(tf_debug.LocalCLIDebugHook())

    estimator.train(input_fn=input_fn, steps=FLAGS.max_steps, hooks=hooks)

All you need is replacing the parameters with their moving averages.

Thanks! what is  get_input ? I mean during the online training, how do you restore the var back to original variable and continue training? Because we want to do evaluation during training.

tf.estimator.Estimator will do this for you, of cause you can evaluate the model during training.
Maybe you need to read the reference of this class.

> > ```python
> > > ```python
> > > class RestoreParametersAverageValues(tf.train.SessionRunHook):
> > >           def __init__(self, restore_weight_backups, assign_dep ):
> > >               '''
> > >               with tf.variable_scope('BackupVariables'):
> > >                   backup_vars = [tf.get_variable(var.op.name, dtype=var.value().dtype, trainable=False,
> > >                                                 initializer=var.initialized_value())
> > >                                 for var in model_vars]
> > >
> > >               save_weight_backups = tf.group(*(tf.assign(bck, var.read_value())
> > >                                               for var, bck in zip(model_vars, backup_vars)))
> > >               restore_weight_backups = tf.group(*(tf.assign(var, bck.read_value())
> > >                                               for var, bck in zip(model_vars, backup_vars)))
> > >
> > >               with tf.control_dependencies( [ save_weight_backups ] ):
> > >                 assign_dep = tf.group(*[tf.assign(x, ema.average(x)) for x in model_vars])
> > >
> > >               '''
> > >               super(RestoreParametersAverageValues, self).__init__()
> > >               self._assign_dep = assign_dep # assign ema var to var
> > >               self._restore_weight_backups = restore_weight_backups # restore weight back
> > >
> > >           def after_create_session(self, session, coord):
> > >               """ Restore the parameters right before run. """
> > >               session.run(self._assign_dep)
> > >
> > >           def end(self, session):
> > >                session.run(self._restore_weight_backups)
> > > ```
> > >
> > >
> > > I was wondering, will this work? I am kinda confused
> > > Thanks for the explaning, But what if you want to restore the variable to the original variable to continue training?
> >
> >
> > You don't need to save and restore the variables manually, 'tf.estimator.Estimator' will save and restore all the parameters automatically for you.
> > ```python
> > def train():
> >     config = tf.ConfigProto()
> >     config.intra_op_parallelism_threads = FLAGS.cpu_threads
> >     config.inter_op_parallelism_threads = FLAGS.cpu_threads
> >     config.gpu_options.allow_growth = True
> >     config.log_device_placement = FLAGS.log_device_placement
> >
> >     estimator = tf.estimator.Estimator(
> >         model_fn=model_fn_train,
> >         model_dir=FLAGS.train_dir,
> >         config=tf.estimator.RunConfig(
> >             tf_random_seed=FLAGS.random_seed,
> >             log_step_count_steps=FLAGS.log_frequency,
> >             save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,
> >             session_config=config,
> >         ),
> >         params=FLAGS,
> >     )
> >
> >     input_fn, iterator_initializer_hook = get_input(flags=FLAGS, mode=tf.estimator.ModeKeys.TRAIN)
> >
> >     logging_hook = tf.train.LoggingTensorHook(tensors={
> >         'cross_entropy': 'loss/softmax_cross_entropy_loss/value',
> >     }, every_n_iter=FLAGS.log_frequency)
> >     hooks = [
> >         iterator_initializer_hook,
> >         logging_hook,
> >     ]
> >     if FLAGS.debug:
> >         hooks.append(tf_debug.LocalCLIDebugHook())
> >
> >     estimator.train(input_fn=input_fn, steps=FLAGS.max_steps, hooks=hooks)
> > ```
> >
> >
> > All you need is replacing the parameters with their moving averages.
>
>
> Thanks! what is ` get_input` ? I mean during the online training, how do you restore the var back to original variable and continue training? Because we want to do evaluation during training.

`tf.estimator.Estimator` will do this for you, of course you can evaluate the model during training.
Use different processes to train and evaluate the model. Every time you evaluate the model, create a new instance of `tf.estimator.Estimator`, then it will use the newest value of moving averages of the parameters.
Maybe you need to read the reference of this class.

Thanks for the explanation. So basically all I need is the following?

          assign_eval_op_list = [tf.assign(x, ema.average(x)) for x in model_vars]
          assign_eval_op_list.append( tf.assign(eval_flag[0], 1) )
          assign_eval_op = tf.group( *assign_eval_op_list )
          # # we use ema variable for testing
          with tf.control_dependencies([assign_eval_op]):
            output = tf.identity(output)
            loss   = tf.identity(loss)

> > ```python
> > > ```python
> > > class RestoreParametersAverageValues(tf.train.SessionRunHook):
> > >           def __init__(self, restore_weight_backups, assign_dep ):
> > >               '''
> > >               with tf.variable_scope('BackupVariables'):
> > >                   backup_vars = [tf.get_variable(var.op.name, dtype=var.value().dtype, trainable=False,
> > >                                                 initializer=var.initialized_value())
> > >                                 for var in model_vars]
> > >
> > >               save_weight_backups = tf.group(*(tf.assign(bck, var.read_value())
> > >                                               for var, bck in zip(model_vars, backup_vars)))
> > >               restore_weight_backups = tf.group(*(tf.assign(var, bck.read_value())
> > >                                               for var, bck in zip(model_vars, backup_vars)))
> > >
> > >               with tf.control_dependencies( [ save_weight_backups ] ):
> > >                 assign_dep = tf.group(*[tf.assign(x, ema.average(x)) for x in model_vars])
> > >
> > >               '''
> > >               super(RestoreParametersAverageValues, self).__init__()
> > >               self._assign_dep = assign_dep # assign ema var to var
> > >               self._restore_weight_backups = restore_weight_backups # restore weight back
> > >
> > >           def after_create_session(self, session, coord):
> > >               """ Restore the parameters right before run. """
> > >               session.run(self._assign_dep)
> > >
> > >           def end(self, session):
> > >                session.run(self._restore_weight_backups)
> > > ```
> > >
> > >
> > > I was wondering, will this work? I am kinda confused
> > > Thanks for the explaning, But what if you want to restore the variable to the original variable to continue training?
> >
> >
> > You don't need to save and restore the variables manually, 'tf.estimator.Estimator' will save and restore all the parameters automatically for you.
> > ```python
> > def train():
> >     config = tf.ConfigProto()
> >     config.intra_op_parallelism_threads = FLAGS.cpu_threads
> >     config.inter_op_parallelism_threads = FLAGS.cpu_threads
> >     config.gpu_options.allow_growth = True
> >     config.log_device_placement = FLAGS.log_device_placement
> >
> >     estimator = tf.estimator.Estimator(
> >         model_fn=model_fn_train,
> >         model_dir=FLAGS.train_dir,
> >         config=tf.estimator.RunConfig(
> >             tf_random_seed=FLAGS.random_seed,
> >             log_step_count_steps=FLAGS.log_frequency,
> >             save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,
> >             session_config=config,
> >         ),
> >         params=FLAGS,
> >     )
> >
> >     input_fn, iterator_initializer_hook = get_input(flags=FLAGS, mode=tf.estimator.ModeKeys.TRAIN)
> >
> >     logging_hook = tf.train.LoggingTensorHook(tensors={
> >         'cross_entropy': 'loss/softmax_cross_entropy_loss/value',
> >     }, every_n_iter=FLAGS.log_frequency)
> >     hooks = [
> >         iterator_initializer_hook,
> >         logging_hook,
> >     ]
> >     if FLAGS.debug:
> >         hooks.append(tf_debug.LocalCLIDebugHook())
> >
> >     estimator.train(input_fn=input_fn, steps=FLAGS.max_steps, hooks=hooks)
> > ```
> >
> >
> > All you need is replacing the parameters with their moving averages.
>
>
> Thanks! what is ` get_input` ? I mean during the online training, how do you restore the var back to original variable and continue training? Because we want to do evaluation during training.

`tf.estimator.Estimator` will do this for you, of course you can evaluate the model during training.
Use different processes to train and evaluate the model. Every time you evaluate the model, create a new instance of `tf.estimator.Estimator`, then it will use the newest value of moving averages of the parameters.
Maybe you need to read the reference of this class.

Thanks for the explanation. So basically all I need is the following?
     if mode != tf.estimator.ModeKeys.TRAIN:
          assign_eval_op_list = [tf.assign(x, ema.average(x)) for x in model_vars]
          assign_eval_op_list.append( tf.assign(eval_flag[0], 1) )
          assign_eval_op = tf.group( *assign_eval_op_list )
          # # we use ema variable for testing
          with tf.control_dependencies([assign_eval_op]):
            output = tf.identity(output)
            loss   = tf.identity(loss)

As mentioned above, please make sure you won't fall into this pit:
**The restoring operation should be executed only once, and before any inference.**

In your code this operation will be executed every time `_infer()` is invoked, and if there are multiple layers (for example, your model has 3 dense layers), you can't restore the variables from the very beginning, you can only restore them after all layers (thus all variables) have been created. but when the variables are created, they have been calculated at least once, so the first batch will get the wrong result.


Invalid translations - html entities

Hello.
I did clean install of LV 5.6 + this package. Unfortunately all icons are displayed as html entities not as parsed HTML which is default behaviour of blade templates. Could You fix it?

I am unable to replicate the issue. Can you please provide a screenshot and steps you took. Thank you.

It was standard installation. composer.json:
"require": {
        "php": "^7.1.3",
        "barryvdh/laravel-ide-helper": "^2.5",
        "fideloper/proxy": "^4.0",
        "jeremykenedy/laravel-users": "^3.1",
        "laravel/framework": "5.6.*",
        "laravel/tinker": "^1.0"
    },
    "require-dev": {
        "barryvdh/laravel-debugbar": "^3.2",
        "filp/whoops": "^2.0",
        "fzaninotto/faker": "^1.4",
        "mockery/mockery": "^1.0",
        "nunomaduro/collision": "^2.0",
        "phpunit/phpunit": "^7.0"
    },

composer create-project --prefer-dist laravel/laravel blog "5.6.*"
composer require jeremykenedy/laravel-users
php artisan make:auth
php artisan vendor:publish --tag=laravelusers
php artisan migrate

It like something with using @lang which equivalates to using {{ }}, as soon as time allows I will test with replacing the @lang with {!! !!} syntax to see if that may fix it.

This commit https://github.com/jeremykenedy/laravel-users/commit/c6af076bd5584119b386300bbd1c4ba81fbf6b8f has solved this. Thanks!


React and Chatkit Slack Clone

My team and I built a Slack clone using React and Chatkit. Here's a link to the repository: https://github.com/pusher/react-slack-clone
We have a number of "Good first issue"s, and would love to get you involved, and guide you through your first pull request: https://github.com/pusher/react-slack-clone/issues?q=is%3Aissue+is%3Aopen+label%3A"good+first+issue"

Hello @bookercodes,
I try to run the project, but I got this error when I click the add button to create a room.

@cand126  You seems to have clicked + button without filling Room Name in "Create a Room" input,first fill room name and then click + button.
Happy Coding

@EduardoSaverin Thank you :)


Carousel Scroll Jumps

On the learn page, where we have carousels - when I click on the pagination, it forces a scroll jump - we should avoid doing this and keep the user in place.

The card sizes should be the same.

FYI please see ongoing discussion https://github.com/onflow/next-docs-v1/pull/309#issuecomment-1165778655


Python server

use curl -X POST http://localhost:8000/ --data "texthere"

Great start!

Let's, maybe, keep it in a ./server dir and add a ./server/README.md there as well, to explain what is it for (and how to launch it)
Let's make at least the port number configurable, though a CLI arg using Argparse
Probably it's worth to keep API minimal: do we need anything besides GET and POST?


Update OOP.html

I add the Copy constructor topic in the oops.html folder u can check and merge it

if you want to contribute in that field at least cover the topic, contributions like this are not meaningful to use, please don't create PRs for the sake of it, closing this for the time being


Add contract type name to draft referral

What does this pull request do?
Adds the Contract Type name to the draft referral DTO to be passed to the frontend.
What is the intent behind these changes?
It was decided to add this information in the backend rather than making additional calls from the frontend.

Looks good - I've approved changes in their current state but of course they can't be approved until the pipelines are successful


Feat/sheet offer

Issues Addressed
closes #18
Additional Information
offer screen created

@sfaria27 left is spec, right device screenshot


We need to have "Thumbnail URL" for the migrated assets in the analyze report.

We are successfully migrated the assets from media services using the tool, and after i have checked the report and it shows the DASH and HLS URL for the assets but here i need one more URL from the same output storage account of these DASH and HLS urls which is its thumbnail url. Is there any possible way to add new column in the report for thumbnail URL.
Similarly i have one more issue after the migration few assets are not migrated those are "non_ism", but i can see that not all non_ism files have this issue.
This is the report and i need one more column in the report for thumbnail url

@dhanil94, thanks for reporting the issues.

unfortunately for thumbnail url generation, we don't have any way to find information about the thumbnail (both name, or existance) on an asset.  it seems existing api doesn't support this scenario as it is not in any of the server manifests / data.  So it would be difficult for us to add this feature in a reliable manner.  That said, if you know the thumbnail name, then you can compose the url by using the outputHlsURl and substitute .m3u8 to your thumbnail name.

for your other issue, I'm not sure I 100% understand what you are saying so in case I misinterpret it, please let me know.  For this issue, I think your screen shot shows that we are reporting hls/dash url for non_ism content (with /.m3u8 or /.mpd) which is incorrect.  We've submitted a PR to fix this issue.

https://github.com/Azure/azure-media-migration/commit/146d726f51d0afd46b3195cfd57d4231940c9f0a
so you can git pull and try again to see if it addresses your issue.

For the second issue, some assets were not migrated to the new storage account but in the report, it was showing as "completed".

I have checked the assets in the new storage account and have not been able to find the assets.

@dhanil94,

@dhanil94,
you mentioned 'new storage account' so I'm wondering whether the follow scenario happened:  you tried to migrate the same asset to two different output storage account?
I can't tell from your screenshot whether this happened or not so I wanted to clarify this with you.
The reason I ask is that the migration tool will mark the input asset that it migrated already with 'complete' and if you tried to run the tool a second time to a different storage location, it will skip the asset.  In case this is what's happening to you, you get around this by doing one of the two things

you can add try to force migrate a single asset by adding '--skip-migrated False' in the asset command, you can see command line usage if you do 'AMSMigrate.exe assets -h'

you can try to reset the state of all input asset to 'Not Migrated' by using 'AMSMigrate.exe reset', you can see the command line usage if you do 'AMSMigrate.exe -h', after that you can do a full migration of all assets again.

Also, I wanted to clarify that you didn't set '--copy-nonstreamable' to False in asset command, right?  That would prevent copying of the non_ism assets as well.
please let me know if this helps, if not, we'll need more information for you, like the command line, output logs to see what happened.


regression-test fix invalid case, two cases use one table in same db

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Refactor activity indicators

In this PR, we have done the following tasks.

Refactor enums to use IBEnum.
Refactor entire activity indicators feature to use Swift 3 APIs.
Refactor all factory method to use make*** instead of  generate*** or create***. According to API Design Guidelines
Reactor all M_PIs.
Fix some fileprivate to private

related to #221
@lastMove @tbaranes can you have a look when you have time? thanks.

🍩

Thanks @tbaranes


Use separate JSON quotes file for easier updates

Drop unrelated and duplicate quotes

@sebbASF How is this going? Can I assist you in any way?

I can do that, but it would be easier to approve in-system if it weren't marked as Draft.


Persist custom settings so can retry if deploy fails

Is your feature request related to a problem? Please describe.
Save the current settings locally in the event the cdk deploy fails so that I don't have to re-enter them to retry the deployment.
Describe the solution you'd like
save to a temp file until confirming that settings have been saved in Cloud Formation metadata?

This is a :rocket: feature request

Settings are saved in CloudFormation after the first successful deployment.


fly: 7.0.0 -> 7.1.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/concourse/concourse/releases.
meta.description for fly is: "A command line interface to Concourse CI"
meta.homepage for fly is: "https://concourse-ci.org"
meta.changelog for fly is: ""
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 7.1.0 with grep in /nix/store/94y6sh86nc7gn1vnhbr1bab7an6g3k6w-fly-7.1.0

directory tree listing: https://gist.github.com/7e99fbd29b32b082f20f99d36dcaaf1c

du listing: https://gist.github.com/e1fd06a048d5257f0b1193eefe180cd1

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
fly

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/94y6sh86nc7gn1vnhbr1bab7an6g3k6w-fly-7.1.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A fly https://github.com/r-ryantm/nixpkgs/archive/6fdb1772f35330430aab324a0a7df9648e0a9392.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/94y6sh86nc7gn1vnhbr1bab7an6g3k6w-fly-7.1.0
ls -la /nix/store/94y6sh86nc7gn1vnhbr1bab7an6g3k6w-fly-7.1.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

fly

Maintainer pings
cc @ivanbrennan @kalbasit for testing.

Result of nixpkgs-review pr 116614 at 6fdb1772 run on aarch64-linux 1

1 package built successfully:

fly

1 suggestion:

warning: unused-argument
Unused argument: writeText.
Near pkgs/development/tools/continuous-integration/fly/default.nix:1:48:
  |
1 | { buildGoModule, fetchFromGitHub, stdenv, lib, writeText }:
  |                                                ^

/me plugs his latest project to the other concourse user in the thread: https://github.com/risicle/nix-build-task


speedread + espeak

I am trying to figure out how to use
speedread with espeak. I think I have to
cat book.txt | ./speedread -w 230 | espeak
Works, but I cannot see the text words going by. I can
only hear espeak - speaking the book

Hello. Did you find out?

Hello. Did you find out?

nope. You can close the issue...

this: cat book.txt | tee >(espeak) | speedread
But syncing the actual output speeds with each other seems rather impossible (for example large numbers take longer to pronounce and it gets out of sync)

In 225 line on speedread file I add such line ( don't need remove anything, just create blank line and add there)
system ("espeak -s 200 "".$current_word."" 2>&1 >/dev/null &");
Then I run like that: speedread -w 200. But if someone want, they can change play with other numbers.


How to set date range using metroui date picker?

I am using datepicker to pick a from and to date. But i couldnot set a default date from/to? so we can choose any date period.

put to forum please

How to disable datepicker in particular input field?

https://github.com/olton/Metro-UI-CSS/issues/752


EF Core 1.1 LinQ and Skip

Hello,
Using VS 2015 and EntityFrameworkCore (1.1.0), I am trying to page my search results using the pagedList plugin from Sakura.
It doesn't work as I am getting an error with EF core.
The following query works fine:
IQueryable<SearchResultViewModel> SearchVM = (from item in _context.MyModel
                                                         select new SearchResultViewModel()
                                                         {
                                                             A = item.a,
                                                                B= item.b
                                                         });
            model.SearchResults = SearchVM.ToList();
            return model;

If I am adding a skip(10) to simulate the paging plugin for example:
IQueryable<SearchResultViewModel> SearchVM = (from item in _context.MyModel
                                                         select new SearchResultViewModel()
                                                         {
                                                             A = item.a,
                                                                B= item.b
                                                         }).Skip(10);

            model.SearchResults = SearchVM.ToList();
            return model;

I get the following error message:
An exception of type 'System.Data.SqlClient.SqlException' occurred in Microsoft.EntityFrameworkCore.dll
Additional information: Incorrect syntax near 'OFFSET'.

What should be the correct syntax to skip records in EF core?
Thanks
S.

OK found the solution.
As I am using SQL Server 2008 R2, I needed to add the following parameter for my context:
optionsBuilder.UseSqlServer(_config["ConnectionStrings:MyConnection"], b => b.UseRowNumberForPaging());
UseRowNumberForPaging being needed for mssql less than 2012 as suggested here:
Thank you
S.

@sylvaincaillot If you are still using RowNumberForPaging, could you please comment on issue #13959 and let us know why?

Thank you @ajcvickers for the followup. I have migrated the database to SQL Server 2017 so I am not using the UseRowNumberForPaging parameter anymore.

@sylvaincaillot Thanks for getting back to us. Appreciated.


question: what's required to build multi-arch with melange and github actions

i suppose i need qemu setup. so i use the following
- name: install qemu-user
   runs: apt install qemu-user

but then i still get this error. all is good if i build for amd64 only.
Error: failed to build package: unable to populate workspace: open /home/runner/work/test-repo/test-repo/workspace/x86_64/.git/objects/0e/a4be1961454522ac2eafb79b3da05b64627ea5: permission denied
839
2022/06/15 06:42:46 error during command execution: failed to build package: unable to populate workspace: open /home/runner/work/test-repo/test-repo/workspace/x86_64/.git/objects/0e/a4be1961454522ac2eafb79b3da05b64627ea5: permission denied
840

another question: to build multi-arch, i need to config both melange and apko right?

Use the docker/setup-qemu-action action to set up qemu.

i tried that before but it complains qemu-${arch} binary is not found so i change to installing qemu-user package.

You might try to use the empty-workspace: true setting.

Ok, I got to the bottom of this: you need to run the docker/setup-qemu-action and then install qemu-user afterward and it should work.


Tutorial 2 does not make use of BEAUti yet

Now that there is a BEAUti template, it would be nice to make use of it in these tutorials.
The layout in Tutorial 1 could be used as a basis for what this should look like.

Tutorial 3 now makes use of BEAUti to get the XML started but requires further tweaking.

there is so much work to do on the tutorials that this issue is no longer helpful


Improved rational curve fit for sheen albedo

This changelist improves the rational curve fit for sheen directional albedo, generating results that are significantly closer to Monte Carlo reference for low roughness values.
Additionally, it increases the size of the directional albedo table to 128, allowing a correct match for sheen direction albedo when table-based rendering is used.

Here's a comparison of roughness wedges for the Standard Surface Velvet material, using three different methods to compute sheen directional albedo:
Monte Carlo:

New Rational Curve, with difference from MC:

Old Rational Curve, with difference from MC:


BUG - mattermost crash when modify sections after user add tasks

Summary
when user modify the sections use /standupconfig after someone add tasks, the mattermost will crash when click raven button on channel top.
Steps To Reproduce
Steps to reproduce the behavior:

install plugin and set the config use /standupconfig ,for example set sections "yesterday" and "today"
add member use /standupaddmembers , for example add userA to group
userA add tasks by click raven button on channel top
modify sections use /standupconfig, for example delete "yestoday"
userA click the button and mattermost crash, must reload the view.

Expected behavior
I think maybe it should clear all tasks set before and let user add task again.
Observed Behavior
mattermost crash
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (if applicable):

OS: windows10 desktop app 4.2.0

@tgly307 huge thanks for reporting the bug!
Wondering if you'd be open to help with a fix for it?

I am very happy to help, but I am very strange to golang and have never done related development.

@tgly307 ah, no issue there! We'll look into it.

This happens in the client and Firefox (for linux).
Relevant JS console output from Firefox:

This may have a trigger: someone edited the questions after standups were submitted by the affected users. Not all users are seeing the issue.

@Routhinator you're correct! It happens when one of the section from current standup config doesn't exist n your standup, probably when someone changes the config. Will be pushing a fix soon.

This issue is fixed in #17 and will be available in the next release.


[Bug]

🐛 Bug
is_slow_wallet function was introduced in #994 when unlocked balance was always 0 due to GAS not being unlocked by the protocol and flowing to slow wallets. The introduction of allowing the locked balance to be dripped broke this function.
Expected Behavior
If a wallet is set as a slow wallet the function should return true. If it doesn't have a 0 balance this does not happen
Steps To reproduce
run is_slow_wallet function from ol/cli/src/node/query.rs

Merged into main


App server doesn't display on Tidbyt after docker compose up

I'm trying to test Pixbyt with only hello-world to see if it works with my setup. I think I followed the instructions in the readme correctly, but I'm totally new to Docker.
After running through all the steps and executing docker compose up, it completes successfully but then nothing happens on my Tidbyt. I'm running Docker Desktop on an Intel MacBook Air (maybe not the best idea long-term...) It looks like the Pixbyt container is pulling >100% CPU when it's active and then constantly restarts. Could that be the source of the issue? Is there a better way to allocate my resources?
Reproducing my workflow below:
Bens-MacBook-Air-3:Documents ben$ git clone https://github.com/bboatwright/pixbyt.git
Cloning into 'pixbyt'...
remote: Enumerating objects: 53, done.
remote: Counting objects: 100% (53/53), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 53 (delta 12), reused 51 (delta 10), pack-reused 0
Receiving objects: 100% (53/53), 98.11 KiB | 1.29 MiB/s, done.
Resolving deltas: 100% (12/12), done.
Bens-MacBook-Air-3:Documents ben$ echo $GIT_TOKEN | docker login ghcr.io -u bboatwright --password-stdin
WARNING! Your password will be stored unencrypted in /Users/ben/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
Bens-MacBook-Air-3:Documents ben$ cd pixbyt
Bens-MacBook-Air-3:pixbyt ben$ docker compose build
Bens-MacBook-Air-3:pixbyt ben$ nano .env
Bens-MacBook-Air-3:pixbyt ben$ docker compose up --build -d --pull "always"
[+] Running 39/19
 ✔ pixbyt 22 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                72.3s
 ✔ meltano-db 14 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                32.4s
 ✔ airflow-db Pulled                                                                                32.4s

[+] Running 3/4
 ⠙ Network pixbyt_default         Created                                                                                4.1s
 ✔ Container pixbyt-meltano-db-1  Started                                                                                3.2s
 ✔ Container pixbyt-airflow-db-1  Started                                                                                3.4s
 ✔ Container pixbyt               Started                                                                                2.5s
Bens-MacBook-Air-3:pixbyt ben$ docker images
REPOSITORY                   TAG       IMAGE ID       CREATED             SIZE
ghcr.io/bboatwright/pixbyt   main      e5cab0dc002a   About an hour ago   1.48GB
postgres                     14        5fbd2ea0d85b   6 days ago          422MB
Bens-MacBook-Air-3:pixbyt ben$ docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS         PORTS                    NAMES
b5729909532a   ghcr.io/bboatwright/pixbyt:main   "meltano invoke airf…"   10 seconds ago   Up 7 seconds                            pixbyt
061d139b82b8   postgres:14                       "docker-entrypoint.s…"   11 seconds ago   Up 8 seconds   <IP_ADDRESS>:5434->5432/tcp   pixbyt-airflow-db-1
b3b7bac714e0   postgres:14                       "docker-entrypoint.s…"   11 seconds ago   Up 8 seconds   <IP_ADDRESS>:5433->5432/tcp   pixbyt-meltano-db-1
Bens-MacBook-Air-3:pixbyt ben$

This error had to do with the incompatible airflow-ext dependencies, resolved by #15


Bootstrap devcontainer config to work with glci

Create a devcontainer config to work with glci.
The purpose is to make share a usable configuration and remove friction for people who contribute to glci.

This is work in progress
Missing: How to best configure required secrets?
I found this but it looks like this is not (yet?) part of the VS Code extension?


Service template cannot be instantiated

Description
Service template with a mandatory custom variable with empty description cannot be instantiated.
To Reproduce
Create the service template, and in the Custom variables section add a variable (say "MYTESTVAR") of type text, empty description, empty default value, and set it as mandatory.
Try to instantiate this service template.
Expected behavior
There should be an input field in the "Custom variables" section during instantiation. But the section is empty instead, no input field at all and indeed when I click on Instantiate, Sunstone complains that
Every custom_attrs key must have its value defined at custom_attrs_value
When edit the service template again, and fill in also a Description of that variable, the input field appears in the Instantiate screen as expected, the mandatory variable can be filled in, and the service template can be instantiated.
So either the Description field should be made mandatory when editing the service template, or in the Instantiate screen Sunstone should be able to present the input fields even for variables without Description.
Details

Affected Component: Sunstone
Hypervisor: KVM
Version: 6.4.0 CE

Progress Status

[ ] Code committed
[ ] Testing - QA
[ ] Documentation (Release notes - resolved issues, compatibility, known issues)

@vickmp - here is the template:

{
  "name": "Testservice",
  "deployment": "straight",
  "description": "",
  "roles": [
    {
      "name": "testrole",
      "cardinality": 1,
      "vm_template": 1205,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template_contents": ""
    }
  ],
  "ready_status_gate": false,
  "automatic_deletion": false,
  "custom_attrs": {
    "TESTVAR": "M|text|| |"
  },
  "registration_time":<PHONE_NUMBER>
}

FWIW, I also made a screen capture video of Sunstone, it is available here:
https://www.fi.muni.cz/~kas/tmp/opennebula-service-custom-var.mp4

Hi @Yenya, thanks for your update. We are going to proceed to close this issue as after further analysis we have come to the conclusion that this bug was fixed in 6.4.2 through this issue.


Computed bindings: Subproperties

Computed bindings are only called when their arguments change.
But what if the computed value depends on subproperties of the arguments...
Is it possibly to tell the binding to recompute on subproperty changes?
If not is there a way to manually trigger the computation?
I also made a fiddle.
PS: Notifing on the arguments (see fiddle) does not cause a recomputation.

In order to recompute a value on sub-property changes, you simply need to use the wildcard syntax for deep-path observation.
In your example, I changed your binding to
<div>{{computeValue(first.*, second)}}</div>

and note that wildcard arguments receive an info object describing the path that changed, so your computing function needed a small tweak:
    computeValue: function (firstInfo, second) {
        return firstInfo.base.join("-") + ", " + second;
    },

I also removed your call to notifyPath, which you should only need to use if you are not using the Polymer API for modifying data.
Working example:
https://jsfiddle.net/7ee702La/2/

I suppose to state this out explicitly in the docs.


[Roadmap] Provide more chart options

Provide more chart options:

Line graph
Side by side bar graphs
Should be able to toggle chart display with quick buttons similar to dates

@mikecao Is this something you want help with? I have some experience with visualizations and would like to give it a shot!

@axelekwall Sure you can try and tackle this. I think to start off the user should be able to toggle between line and bar charts.


Documentation about the project

Create a general documentation about CW and a simple examples about the usage of it.

Done: https://github.com/RobertoRojas/CandyWrappers/commit/ac2fd53eab13d88f4d5fc7f82283300b9830d9ae


Compile libmysql error on Windows

C:\programs\compa_libs\vcpkg>vcpkg install libmysql
The following packages will be built and installed:
    libmysql[core]:x86-windows
Starting package 1/1: libmysql:x86-windows
Building package libmysql[core]:x86-windows...
CMake Error at ports/libmysql/portfile.cmake:10 (message):
  Oracle has dropped support in libmysql for 32-bit Windows.
Call Stack (most recent call first):
  scripts/ports.cmake:71 (include)

But this is Win10 64 bits and I told  it nothing to build a 32 bits version (which in fact I would even like to do)

Oracle has dropped support in libmysql for 32-bit Windows.
default x86

libmysql[core]:x86-windows

built
vcpkg install libmysql:x64-windows

OK, thanks. But I actually found this issue when trying to build gdal and it failed because it depends on this lib.

Try to delete these lines, maybe the assembly will be successful
if(VCPKG_TARGET_ARCHITECTURE STREQUAL "x86" AND NOT CMAKE_SYSTEM_NAME OR CMAKE_SYSTEM_NAME STREQUAL "WindowsStore")
    message(FATAL_ERROR "Oracle has dropped support in libmysql for 32-bit Windows.")
endif()

Right, I try it latter. Meanwhile I'm waiting that mysql finishes to build but it's taking ages. The same thing happened yesterday when building gdal it took hours to build (had to remove the mysql dependency for it to build). Don't know why it's building icu and lots of boost stuff
...
-- Installing: C:/programs/compa_libs/vcpkg/packages/boost-algorithm_x64-windows/share/boost-algorithm/copyright
-- Performing post-build validation
-- Performing post-build validation done
Building package boost-algorithm[core]:x64-windows... done
Installing package boost-algorithm[core]:x64-windows...
Installing package boost-algorithm[core]:x64-windows... done
Elapsed time for package boost-algorithm:x64-windows: 1.32 s
Starting package 43/91: icu:x64-windows
Building package icu[core]:x64-windows...
-- Using cached C:/programs/compa_libs/vcpkg/downloads/icu4c-61_1-src.tgz
-- Applying patch C:/programs/compa_libs/vcpkg/ports/icu/disable-escapestr-tool.patch
-- Applying patch failed. This is expected if this patch was previously applied.
-- Applying patch C:/programs/compa_libs/vcpkg/ports/icu/remove-MD-from-configure.patch
-- Applying patch failed. This is expected if this patch was previously applied.
-- Acquiring MSYS Packages...
-- Acquiring MSYS Packages... OK
-- Configuring x64-windows-rel
-- Configuring x64-windows-rel done
-- Configuring x64-windows-dbg
-- Configuring x64-windows-dbg done
-- Package x64-windows-rel

Nope, removing those lines just pulls the error further down
-- CMAKE_GENERATOR: Ninja
CMake Error at cmake/os/Windows.cmake:56 (MESSAGE):
  32 bit Windows builds are not supported.  Clean the build dir and rebuild
  using -G "Ninja Win64"
Call Stack (most recent call first):
  CMakeLists.txt:238 (INCLUDE)

BTW, the libmysql build took 39 minutes which looks an awful long time for the task.

Changes were made in PR #4413 to allow gdal package to be built with mariadb or without mysql support.
By default, it still use mysql for compatibility but it can be disabled with package feature.
Example:
vcpkg install gdal[core]
vcpkg install gdal[core,mysql-libmariadb]

I also have this problem when tried to install poco[mysql]  or libmysql, the error is 'libmysql and libmariadb are incompatible', anyone encounter this problem?


Commons auth technical design updates

┆Issue is synchronized with this JIRA Task
┆Project Name: Boardwalk-UI
┆Issue Number: BOARD-64

Hi @coverbeck anything else to be done here for the time being? If not can you close this please.


DOC: Updated docstring for optimize.nnls

Added a simple example and 'See Also' section for the optimize.nnls docstring as mentioned in #7168.

Thanks @jconroy14


[devcenter] Fix incorrect command group name, update author details

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[ ] My extension description/summary conforms to the Extension Summary Guidelines.

About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

devcenter


Fix link to plausible script

We might need to use the script from view.scientific-python.org rather than the default one from plausible. At least I'm currently not seeing any data from this page and this seems to be the only difference.

Thanks @aoanla . It works now :)


intel xmx tensor core

Initial shot at introducing the intel tc. Currently only supports float16. Also need to optimize/improve perf.

I don't think we want to introduce ops_intel for this. Can you just init ops_gpu based on device name or vendor (CL_DEVICE_VENDOR)?

Right, I guess that makes sense, sorry about that. Was just following this tutorial which I guess was overkill.
Updated to just be based on CL_DEVICE_VENDOR= "Intel" or INTEL=1 (the 2nd one might mostly just for local testing/emulation)

I don't understand how this works at all...
Doesn't K=16 for the Intel XMX?  Is the layout really the same as METAL?

Can you paste a working generated kernel? I'll lock bounty when I see that.

Yes, will get a working kernel up as soon as I can. Will close this in the meantime since it's not fully baked.


Update to provider facing roadmap

Context
We haven't updated this in a while, we can now talk about references and API v1.2.
Changes proposed in this pull request
Copy updates.
Guidance to review
Does this make sense?
Link to Trello card
https://trello.com/c/1EQLeRvA/588-update-provider-facing-roadmap
Things to check

[ ] If the code removes any existing feature flags, a data migration has also been added to delete the entry from the database
[ ] This code does not rely on migrations in the same Pull Request
[ ] If this code includes a migration adding or changing columns, it also backfills existing records for consistency
[ ] If this code adds a column to the DB, decide whether it needs to be in analytics yml file or analytics blocklist
[ ] API release notes have been updated if necessary
[ ] Required environment variables have been updated added to the Azure KeyVault

Ready to merge once tests pass. ✅


Unable to scroll inside a fixed content when useLockBodyScroll is used.

What is the current behavior?
On iOS, It is not possible to scroll inside a fixed positioned content when useLockBodyScroll is used. It works fine in Desktops and android phones though.

For comparison, go to the demo without body lock to view all scrollable content

I recorded the issue below.
Using useLockBodyScroll from react-use

Without body scroll lock

What is the expected behavior?
Overlay should be scrollable on ios as well
A little about versions:

OS: IOS 15.1
Browser (vendor and version): safari 15.1
React: ^18.2.0
react-use:^18.2.0
Did this worked in the previous package version? : Looking at open issues, no

Source code:: https://github.com/AkromDev/fixed-position-ios-scroll

yea it looks like the hook disables all touch scroll events for IOS devices https://github.com/streamich/react-use/blob/master/src/useLockBodyScroll.ts#L58

For those in need of a quick fix while this issue is resolved:
Add onTouchMove={(e) => e.stopPropagation()} to the outermost element that should still be scrollable when useLockBodyScroll is active.
This will prevent the onTouchMove event from bubbling up and getting catched by useLockBodyScrolls scroll preventing event listener on iOS.

I ended up using usePreventScroll from<EMAIL_ADDRESS>I've just checked the docs and usePreventScroll is imported from react-aria directly but I did not test that one.
If you don't want to install react-aria, I would recommend checking its implementation.
import { usePreventScroll } from '@react-aria/overlays';

...

usePreventScroll({ isDisabled: !isMobileMenuOpened });

Same issue with usePreventScroll.
Attaching stopPropagation() on child element seem's not working anymore too.


Roll Engine from 399c19bb6766 to 69770a5f3cdb (3 revisions)

https://github.com/flutter/engine/compare/399c19bb6766...69770a5f3cdb
2021-06-26<EMAIL_ADDRESS>Roll Skia from 7bf799956d8b to 688d3180ab9d (8 revisions) (flutter/engine#26978)
2021-06-25<EMAIL_ADDRESS>Remove tech debt related to image disposal and layer GC (flutter/engine#26870)
2021-06-25<EMAIL_ADDRESS>Roll Dart SDK from 624322b41ddb to 4f1a93c0cd09 (2 revisions) (flutter/engine#26975)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


IP Pool Rules - Error

I, [2017-05-12T00:17:50.622475 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429] Started GET "/org/custserv/ip_pool_rules" for <IP_ADDRESS> at 2017-05-12 00:17:50 +0100
I, [2017-05-12T00:17:50.623596 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429] Processing by IPPoolRulesController#index as HTML
I, [2017-05-12T00:17:50.623703 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]   Parameters: {"org_permalink"=>"custserv"}
I, [2017-05-12T00:17:50.772439 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]   Rendering ip_pool_rules/index.html.haml within layouts/application
I, [2017-05-12T00:17:50.776300 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]   Rendered organizations/_nav.html.haml (3.3ms)
I, [2017-05-12T00:17:50.776977 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]   Rendered organization_ip_pools/_nav.html.haml (0.4ms)
I, [2017-05-12T00:17:50.787426 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]   Rendered ip_pool_rules/index.html.haml within layouts/application (14.8ms)
I, [2017-05-12T00:17:50.787823 #3111]  INFO -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429] Completed 500 Internal Server Error in 164ms (ActiveRecord: 137.6ms)
F, [2017-05-12T00:17:50.789577 #3111] FATAL -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]
F, [2017-05-12T00:17:50.789673 #3111] FATAL -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429] ActionView::Template::Error (undefined method `ip_pool' for nil:NilClass):
F, [2017-05-12T00:17:50.789974 #3111] FATAL -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]     60:             %dl.ipPoolRuleList__condition
[89f3c3f8-d022-4dc1-9add-bca449bc2429]     61:               %dt Will be sent using:
[89f3c3f8-d022-4dc1-9add-bca449bc2429]     62:               %dd= ip_pool_rule.ip_pool.name
[89f3c3f8-d022-4dc1-9add-bca449bc2429]     63:     - if @server.ip_pool
[89f3c3f8-d022-4dc1-9add-bca449bc2429]     64:       %p.ipPoolRuleListDefault.u-margin All mail that doesn't match a rule above will be sent using<EMAIL_ADDRESS>[89f3c3f8-d022-4dc1-9add-bca449bc2429]     65:     %p.u-center= link_to "Add another rule", [:new, organization, @server, :ip_pool_rule], :class => "button button--positive"
F, [2017-05-12T00:17:50.790055 #3111] FATAL -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429]
F, [2017-05-12T00:17:50.790140 #3111] FATAL -- : [89f3c3f8-d022-4dc1-9add-bca449bc2429] app/views/ip_pool_rules/index.html.haml:63:in `_app_views_ip_pool_rules_index_html_haml___1051384519009291652_52715760'

There are already rules which were added earlier but this error appears to only occur on the org that has rules. It loads OK on those without any rules.

Not sure this fix worked.
F, [2017-05-15T01:58:15.259185 #3041] FATAL -- : [47880832-20d9-4b63-bfb7-c1234d12b1dd] ActionView::Template::Error (undefined method ip_pool' for nil:NilClass): F, [2017-05-15T01:58:15.259475 #3041] FATAL -- : [47880832-20d9-4b63-bfb7-c1234d12b1dd]     60:             %dl.ipPoolRuleList__condition [47880832-20d9-4b63-bfb7-c1234d12b1dd]     61:               %dt Will be sent using: [47880832-20d9-4b63-bfb7-c1234d12b1dd]     62:               %dd= ip_pool_rule.ip_pool.name [47880832-20d9-4b63-bfb7-c1234d12b1dd]     63:     - if @server && @server.ip_pool [47880832-20d9-4b63-bfb7-c1234d12b1dd]     64:       %p.ipPoolRuleListDefault.u-margin All mail that doesn't match a rule above will be sent using<EMAIL_ADDRESS>[47880832-20d9-4b63-bfb7-c1234d12b1dd]     65:     %p.u-center= link_to "Add another rule", [:new, organization, @server, :ip_pool_rule], :class => "button button--positive" F, [2017-05-15T01:58:15.259579 #3041] FATAL -- : [47880832-20d9-4b63-bfb7-c1234d12b1dd] F, [2017-05-15T01:58:15.259668 #3041] FATAL -- : [47880832-20d9-4b63-bfb7-c1234d12b1dd] app/views/ip_pool_rules/index.html.haml:63:in _app_views_ip_pool_rules_index_html_haml___1796347729016451487_55497600'
Upgraded but that error still shows.
File confirmed on file system as having the changes in d33994c.

I think this is actually a different issue. I think you have a rule that belongs to an IP pool that no longer exists. I have made an adjustment to delete rules associated with a pool when it is deleted (9a76538f11064fb1bdb5f15aa8310ea04b68725c) but you'll need to track down the erroneous route and remove it manually I'm afraid. The following will do that in postal console:
IPPoolRule.all.select { |i| i.ip_pool.nil? }.each(&:destroy)

Did that.
irb(main):001:0> IPPoolRule.all.select { |i| i.ip_pool.nil? }.each(&:destroy)
=> []
irb(main):002:0>
but it still errors.
Have checked in the database in ip_pool_rules and they all match to existing items that exist in other referenced tables.


Fix ModTime for directories

Fixes https://github.com/restic/restic/issues/24.
But a similar issue exists with symlinks. It seems that it's currently not possible to call the lutimes system call to change symlink timestamps (without dereferencing the link). I'm gonna open a separate GitHub issue for that after this PR is merged.

A fix for this has been merged upstream 8 days ago (https://go-review.googlesource.com/#/c/9379/).

Looks like I made coverails unhappy :-/
I don't think it's worth it to add a unit test for this new function. The code has just as much test coverage as before via the shell scripts.

I've set the failure threshhold to 5%, I hope that this is the right value. The meaning of the fields in the coveralls settings panel aren't explained anywhere.
This should also be gone once #179 is done, afterwards the integration tests count towards the coverage.

Does this solution make sense to you? Iterate over the directories a second time at the end. Or is that too inefficient?

Looks good to me. We could also fix the timestamps for dirs right after all nodes have been restored to, before returning from restoreTo(), what do you think?

Hm not sure what you mean by that. We need to do this after all non-dirs have already been restored.

Yeah, I know, but we can do this directly after all the nodes in a dir have been created, no need to iterate a second time.

You mean right after this returns?

oh, you already did that.

Exactly. ;)

works for me :rocket:

since travis and coveralls are happy, I'm merging this.


[Feature]: Dynamic Gridlayout from array of json object

Is there an existing issue for this?

[X] I have searched the existing issues

Summary
Dynamic grid layout should be rendered from array of json objects.

Assuming we have an API that returns array of json objects.
Now I would like to render them in a dynamic grid layout.
A main dynamic Grid Layout component can be dragged to the form and the options can be set for 1, 3, 6 columns grid layout.
Then It should be possible to configure the boxes in layout with specific set of component, like image or video or some text with multiple headers etc.

Why should this be worked on?
The infinite scrolling with dynamic grid layout can make it possible to have beautiful SPA. Right now I don't see any components that can do this. You can take an example from ghost main page where there is dynamic grid and infinite scrolling.

@xoraingroup thank you for the feature request. We're tracking this feature here #7256


[SPARK-41029][SQL] Optimize constructor use of GenericArrayData for Scala 2.13

What changes were proposed in this pull request?
This pr change to use a more appropriate constructor when the input is ArrayBuffer or Empty Collection to improve the construction performance of GenericArrayData with Scala 2.13.
Why are the changes needed?
Minor performance improvement.
GenericArrayData  has the following constructor
https://github.com/apache/spark/blob/57d492556768eb341f525ce7eb5c934089fa9e7e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala#L30
When the input type is ArrayBuffer, the following code is similar in Spark
new GenericArrayData(arrayBuffer.toSeq)

For Scala 2.12, there will be no performance gap between new GenericArrayData(arrayBuffer.toSeq) and new GenericArrayData(arrayBuffer).
However, when Scala 2.13 is used, there will be a performance gap, because 'toSeq' will cause a redundant memory copy.
For the following test case:
 val valuesPerIteration: Long = 1000 * 1000 * 10
 val buffer = if (bufferSize == 0) {
    ArrayBuffer.empty[Any]
  } else {
    ArrayBuffer.fill[Any](bufferSize)(() => 1)
  }
  val benchmark = new Benchmark(s"constructor with buffer size = $bufferSize",
    valuesPerIteration, output = output)
  benchmark.addCase("toSeq and construct") { _ =>
    var n = 0
    while (n < valuesPerIteration) {
      new GenericArrayData(buffer.toSeq)
      n += 1
    }
  }

  benchmark.addCase("construct directly") { _ =>
    var n = 0
    while (n < valuesPerIteration) {
      new GenericArrayData(buffer)
      n += 1
    }
  }

When bufferSize=10, there is a performance gap of more than 5 times between a and b, and the performance gap increases almost linearly with the increase of bufferSize
There will be more than 5 times performance gap between new GenericArrayData(buffer.toSeq) and new GenericArrayData(buffer) when bufferSize = 10 and the performance gap will increase with the increase of bufferSize.
OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
constructor with buffer size = 10:        Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
toSeq and construct                                2617           2622           7          3.8         261.7       1.0X
construct directly                                  399            406          11         25.1          39.9       6.6X

OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
constructor with buffer size = 100:       Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
toSeq and construct                               12512          12554          60          0.8        1251.2       1.0X
construct directly                                  779            781           2         12.8          77.9      16.1X

OpenJDK 64-Bit Server VM 1.8.0_345-b01 on Linux 5.15.0-1022-azure
Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
constructor with buffer size = 1000:      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
toSeq and construct                              108882         109400         732          0.1       10888.2       1.0X
construct directly                                 5717           5731          20          1.7         571.7      19.0X

We can safely change new GenericArrayData(buffer.toSeq) to new GenericArrayData(buffer) due to ArrayBuffer is still scala.collection.Seq in Scala 2.13.
On the other hand, when input is an empty set, using Array.empty is 10% faster than using Seq.empty.
Does this PR introduce any user-facing change?
No
How was this patch tested?

Pass GitHub Actions
Manual tests catalyst and sql module with Scala 2.13 passed

Yes, there is no change, all inputs will be 'toArray`

Merged to master


[feat] Notification onclick event

Describe the problem
I am running my Tauri app in the tray and I only want to show the UI when the user responds to a notification. I can't seem to subscribe to any notification onclick event.
Describe the solution you'd like
I would like to be able to subscribe to a click event, like so:
let n = new Notification('title', { body: 'some text' });
n.onclick = function() { console.log('click happened') };

Alternatives considered
No response
Additional context
No response

Click events aren't implemented yet.

Is there any ETA on this?

Anything that can be done to develop this? New to Open Source if any references and guides to help then I would like to help with this feature.

@Huzaifa-MS the problem is that the upstream notification crates (https://github.com/hoodie/notify-rust and its dependencies) don't support this, so you'd have to work with ugly system APIs to implement this.

@FabianLars I see. I did get that vibe from searching for a bit. I also noticied that the web api for notification works on its own in the frontend. Is it possible to use that to create actions in notifications?
Gonna be looking into that later

I also noticied that the web api for notification works on its own in the frontend

It doesn't. We're injecting our own api (the same from the api module) here: https://github.com/tauri-apps/tauri/blob/dev/core/tauri/scripts/core.js#L163-L247

It doesn't.

I am sorry, I am new to this so perhaps I am not understanding.
So this code I wrote in the frontend portion of the Tauri app:
 const callWebNotificationAPI = async () => {
  Notification
  if (!("Notification" in window)) {
    // Check if the browser supports notifications
    alert("This browser does not support desktop notification")
  } else if (Notification.permission === "granted") {
    // Check whether notification permissions have already been granted;
    // if so, create a notification
    new Notification("Hi there!")
    // …
  } else if (Notification.permission !== "denied") {
    // We need to ask the user for permission
    Notification.requestPermission().then((permission) => {
      // If the user accepts, let's create a notification
      if (permission === "granted") {
        new Notification("Hi there!")
      }
    })
  }
}

Is getting transformed by Tauri automatically at compile time to use Tauri's notification library instead?
Does Tauri not support this web API yet?
Again apologies for my lack of knowledge on Tauri and the associated technologies.

Is getting transformed by Tauri automatically at compile time to use Tauri's notification library instead?

We don't touch your code at all. We overwrite the global Notification / window.Notification object/apis.

Does Tauri not support this web API yet?

Tauri does. But the underlying webviews don't, by design as far as i know. They expect the host process to handle/implement notifications, which we do by overwriting the js apis to use our rust api.

I see. I understand better now. Thank you.

@

@Huzaifa-MS the problem is that the upstream notification crates (https://github.com/hoodie/notify-rust and its dependencies) don't support this, so you'd have to work with ugly system APIs to implement this.

Do you of any guides or documentation I can look into to implement the feature? I would like to try my hand at it.

Is there any way to use the JS APIs instead?

There are no JS APIs to use. The web apis you know from normal browsers don't exist in webviews and we need to polyfill them ourselves. And the nodejs apis can't be used because there is no nodejs runtime in tauri.
Or did you mean something different?

The web apis you know from normal browsers don't exist in webviews and we need to polyfill them ourselves

Completely understood, thank you.

Any updates?


HttpClient cant authenticate when site is published Blazor Core hosted .Net 3.0 Preview7

Issue Title
HttpClient cant authenticate with Bearer when site is published and running in IIS works fine from visual studio.
General
Maybe im missing something new but i tried the token thats given from both my machine and the publised server in Postman and it works. But from the application it throws "Response status code does not indicate success: 401 (Unauthorized)."
This code was working fine in .net core 2.2 but i made a new project and moving over to .Net Core 3.0 Preview 7 (Both projects are Blazor ASP.NET Core hosted project)

.NET Core version on developing machine

.NET Core version on published machine

@Pilchie is this something your team can help with?

@mkArtakMSFT @danroth27 @blowdart ?

Move it to our repo?
There's no enough to go on here.

Where are tokens issued from?
What authentication and authorization services are configured and how?
Contents of startup.cs, with any secrets removed.

All of the above is needed.

This issue was moved to aspnet/AspNetCore#13139


OpenAPI 3.0 specs for user creation

Specs written by Gustavo and edited by Andrew. Covers the user creation endpoint.
See specs at http://localhost:9001/docs
Note: The endpoint is not yet implemented.

We could also just use the email as login if needed.


Support for multi core STM32 H7 boards

Problem:
There is no support for multi core STM32 boards in the Nucleo in the H77 and H75 families; these are quite powerful and offer interesting new possibilities.
I would like to know if it would be a quick operation to implement support for these board families, as there is already support for both Cortex M7 and M4 STM32 boards. Is there a guide for doing this type of implementation?

Hi @stevewells20,
Adding Dual core support for these soc/boards should not be a complex task.
You first need to add device tree for the targeted SoCs.
Then provide board support based on what was done on stm32h747i_disco.

Thank you for the response! I'll look into it. Are there any resources available for best practices or examples for adding new boards? I'm new to Zephyr, and would like to get my feet wet by diving in.

@stevewells20, you can look at https://github.com/zephyrproject-rtos/zephyr/pull/23160 for reference.
You don't need to provide full support for the board, but a minimum needs to be added like done in this example.
From what I understand, you want to introduce new SoC, so we'll wait 2 commits:

new SoC introduction
board introduction

For the board configuration, dual core support is a bit specific, so you can check how it is done for this board:
https://github.com/zephyrproject-rtos/zephyr/tree/master/boards/arm/stm32h747i_disco
If you feel stuck, don't hesitate to push a draft PR and ask for help: @FRASTM or @ABOSTM

I am also working on support for NUCLEO-H745ZI


Using hot-import cannot quit wechaty when quit the code

When I Using hot-import of wechaty,  when I quit the code， but it seems wechaty still work background, log as follows.
➜  ticket-bot git:(master) ✗ 16:34:41 WARN PuppetWeb initWatchdogForPuppet() dog.on(reset) last food:inited, timeout:120000
16:34:51 WARN PuppetWeb initWatchdogForPuppet() dog.on(reset) last food:inited, timeout:120000
(node:7403) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 watchdog listeners added. Use emitter.setMaxListeners() to increase limit
16:35:14 WARN PuppetWeb initWatchdogForPuppet() dog.on(reset) last food:inited, timeout:120000

➜  ticket-bot git:(master) ✗
➜  ticket-bot git:(master) ✗ 16:36:42 WARN PuppetWeb initWatchdogForPuppet() dog.on(reset) last food:inited, timeout:120000
(node:7544) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 scan listeners added. Use emitter.setMaxListeners() to increase limit
(node:7544) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 login listeners added. Use emitter.setMaxListeners() to increase limit
(node:7544) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 logout listeners added. Use emitter.setMaxListeners() to increase limit
16:36:52 WARN PuppetWeb initWatchdogForPuppet() dog.on(reset) last food:inited, timeout:120000
(node:7544) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 heartbeat listeners added. Use emitter.setMaxListeners() to increase limit

should add bot.stop() when exit the process.
const finis = require('finis')
finis(async (code, signal) => {
  const exitMsg = `Wechaty exit ${code} because of ${signal} `
  await bot.stop()
  console.log(exitMsg)
  bot.say(exitMsg)
})

It seems not wechaty doesn't quit, but some monitor thing doesn't quit.
When I quit the program, then change the file, I will get the following log
10:37:46 INFO Bot DingDong! 我上线啦，你可以来和我订机票啦！
10:37:46 INFO Wechaty announce(DingDong! 我上线啦，你可以来和我订机票啦！)
10:37:46 VERB Wechaty announce() got 0 rooms
10:37:46 VERB Wechaty announce() to self because no room found

^C
lijiarui:~/workspace (master) $ 10:38:01 WARN PuppetWebBridge quit() page.close() exception: Error: Protocol error (Target.closeTarget): Target closed.
Wechaty exit 130 because of SIGINT
(node:36952) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: no puppet

lijiarui:~/workspace (master) $ 10:38:41 ERR HotImport importFile(/home/ubuntu/workspace/src/listeners/message.ts) rejected: TSError: ⨯ Unable to compile TypeScript
src/listeners/message.ts (8,12): Cannot find name 'BOTBUILDERAPPID'. (2304)
src/listeners/message.ts (9,18): Cannot find name 'BOTBUILDERPSW'. (2304)
src/listeners/message.ts (12,7): 'MsBot' is declared but its value is never read. (6133)
src/listeners/message.ts (12,51): Block-scoped variable 'startBot' used before its declaration. (2448)
src/listeners/message.ts (12,51): Variable 'startBot' is used before being assigned. (2454)
10:38:41 ERR HotImport refreshImport(/home/ubuntu/workspace/src/listeners/message.ts) exception: TSError: ⨯ Unable to compile TypeScript
src/listeners/message.ts (8,12): Cannot find name 'BOTBUILDERAPPID'. (2304)
src/listeners/message.ts (9,18): Cannot find name 'BOTBUILDERPSW'. (2304)
src/listeners/message.ts (12,7): 'MsBot' is declared but its value is never read. (6133)
src/listeners/message.ts (12,51): Block-scoped variable 'startBot' used before its declaration. (2448)
src/listeners/message.ts (12,51): Variable 'startBot' is used before being assigned. (2454)
10:38:41 ERR HotImport refreshImport(/home/ubuntu/workspace/src/listeners/message.ts) keep using the latest usable version
10:38:41 ERR HotImport importFile(/home/ubuntu/workspace/src/listeners/message.ts) rejected: TSError: ⨯ Unable to compile TypeScript
src/listeners/message.ts (8,12): Cannot find name 'BOTBUILDERAPPID'. (2304)

I found the solution! should add process.exit(-1) at finis, as follows
finis(async (code, signal) => {
  const exitMsg = `Wechaty exit ${code} because of ${signal} `
  await bot.stop()
  console.log(exitMsg)
  process.exit(-1)
})

Actually, I think it should be done by wechaty itself instead by user,  because wechaty is a 'platform' solution.

What if you remove all the finis part of the code?

If I remove all finis part, when I quit the code, node and chrom also work background, as the following shows
https://login.weixin.qq.com/qrcode/4Y04lLO_YQ==
14:22:39 INFO Listeners/Scan https://login.weixin.qq.com/qrcode/4Y04lLO_YQ==
[0]
^C
lijiarui:~/workspace (master) $ ps -A
    PID TTY          TIME CMD
      1 ?        00:00:01 tini
      7 ?        00:00:00 micro-inetd
  34436 ?        00:00:00 tmux
  34437 pts/1    00:00:00 bash
  34438 pts/1    00:00:00 bash
  37393 pts/1    00:00:00 node
  37403 pts/1    00:00:44 node
  37647 pts/3    00:00:00 bash
  37648 pts/3    00:00:00 bash
  40297 ?        00:00:00 dropbear
  40298 ?        00:00:01 vfs-worker {"pi
  40465 pts/0    00:00:00 tmux
  40467 pts/2    00:00:00 tmux
  41227 pts/3    00:00:06 node
  41300 ?        00:00:02 chrome
  41305 ?        00:00:00 chrome
  41326 ?        00:00:00 chrome
  41333 ?        00:00:10 chrome
  41374 pts/3    00:00:00 ps

@zixia
Run the repo example can reproduce this.
run  ts-node example/roger-bot.ts (roger-bot)

I do not think this issue relates to hot-import.

Due to a typescript bug

@lijiarui (and maybe @zixia as well), when closing the issue, please use e.g., 'close #978' in the git comment  so that the fix can be linked back to the issue, and vice versa, people can use the issue to find the fix.
thanks


[POC] Update messaging with SCALE

Rework messaging (data protocol & application api) into same set of messaging enums, all serialized with scale codec
Rework control mode to correctly handle any data protocol or application api messages at any given time
Adds an example cpp utility demonstrating interop with generated rust library

This diff here is a little inaccurate b/c it includes work done in another PR.  Maybe we wait for that to merge before review this, or consider changing the merge target?

Great PR @plauche 💯   I just left a couple of comments, but nothing major to block a merge.