Can mxnet share GPU memory across all buckets?

I have implemented LSTM layer for speech task with utterance parallelization, and I use bucketing to deal with variable-length inputs. However, I notice that GPU memory usage will increase when I raise bucket number. So I think mxnet will allocate GPU memory for every bucket. Is it possible for mxnet to share GPU memory across all buckets?
This is the GPU memory usage vs. bucket number and parallel utterance number

GPU memory (MB)
parallel_num = 25
parallel_num = 40
parallel_num = 60

bucket_num = 10
3751
6016
8120

bucket_num = 20
4391
7085
9810

bucket_num = 50
6576
10549

bucket_num = 70
8058

bucket_num = 100
10021

@GrindstoneLZX The memory should already been shared across buckets -- essentially, the total memory consumption should be equal to the largest bucket (e.g. the longest sequence). Can you try to set the longest sequence to be of the same length in all the cases to see if the memory still grows noticeably with the number of buckets?

BTW: that mechanism are you using the measure the device memory? We are also interested in doing some more test on the memory consumption, but did not get good way to measure the device memory usage yet.

thank you @pluskid for your reply!
I use Tesla K40 for my experiments and I measure GPU memory usage by simply using 'nvidia-smi' to observe the maximum usage during training.
In all caces I use the same dataset. My dataset is a speech dataset with 4620 utterances, and the frame number (sequence length) of each utterance ranges from 54 to 487. I set buckets in a way that the number of utterances is roughly equal in each bucket. For example:

bucket[0]
bucket[1]
bucket[2]
...
bucket[N-2]
bucket[N-1]

89
126
178
...
321
487

the default_bucket_key of my DataIter is bucket[0]. During training, first I select one bucket and then select several utterances as one mini-batch. If the mini-batch is not full, I will fill it with zeros. I have tried two different way to select buckets.
The result I reported yesterday is the selecting buckets in an ascending order, which is 89, 126, 178, ... , 321, 487.
And I also tried descending order, which is 487, 321, ..., 178, 126, 89. The result is:

GPU memory (MB)
parallel_num = 25
parallel_num = 40
parallel_num = 60

bucket_num = 10
3321
5454
7080

bucket_num = 20
3638
5945
7846

bucket_num = 50
3568
5881
7699

bucket_num = 70
3553
5774
7658

bucket_num = 100
3537
5728
7626

And I also tried random order, the memory usage is random too.
I tried to set default_bucket_key to buckets[N-1] to pre-allocate maximum memory space, but when I set default_bucket_key = buckets[N-1], I get an error:
2016-04-07 11:49:12,823 Start training with [gpu(0)]
[11:49:13] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:13] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:13] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:13] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:13] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:13] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
bucket: 216
[11:49:20] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:20] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:20] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:20] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:20] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
[11:49:20] src/symbol/static_graph.cc:383: Memory efficient gradient aggregation on... to disable, set MXNET_EXEC_INPLACE_GRAD_SUM_CAP to big number
Traceback (most recent call last):
File "TrainSchedule2.py", line 162, in
epoch_end_callback=mx.callback.do_checkpoint(save_prefix),)
File "/home/zxlee/Downloads/miniconda2/envs/mxnet2/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/model.py", line 770, in fit
sym_gen=self.sym_gen)
File "/home/zxlee/Downloads/miniconda2/envs/mxnet2/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/model.py", line 221, in _train_multi_device
executor_manager.load_data_batch(data_batch)
File "/home/zxlee/Downloads/miniconda2/envs/mxnet2/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/executor_manager.py", line 374, in load_data_batch
shared_group=self.execgrp)
File "/home/zxlee/Downloads/miniconda2/envs/mxnet2/lib/python2.7/site-packages/mxnet-0.5.0-py2.7.egg/mxnet/executor_manager.py", line 221, in init
for i in self.param_idx]
IndexError: list index out of range
Do you have any ideas?

@GrindstoneLZX sorry I don't quite understand what your figure mean. Setting default bucket to be the largest one might be a good way to try. Please remember when you set the default bucket, the provide_data and provide_label on the DataIter should match that of the default bucket. Can you double check that? Thanks!

Your figure looks reasonable to me.
The error seems strange. Could you give a minimal script that reproduce this error?

@piiswrong @pluskid
I wrote a simple LSTM implementation script:

The LSTM layer module is lstm_layer, and 'lstm_layer_init_states()' is a function to get init_states for data iterator.

The DataIter simple_data_iterator generates data and labels: label = np.ones and data = label + np.random.normal.

fea_len is the feature length, bucket_num is the bucket number, batch_size is the utterance parallel number, max_seq_len is the maximum sequence length of all utterances.

order controls the way simple_data_iterator selects bucket when called next():

When order = 'ascend', bucket_key is set from 1 to max_seq_len,

When order = 'descend', bucket_key is set from max_seq_len to 1,

When order = random, bucket_key is set in random order.

simple_data_iterator.next() changes bucket_key according to order and returns one simple_batch each time.

I built a 3 LSTM layers network, each LSTM layer has 256 hidden nodes, I change max_seq_len, bucket_num and order in experiments.

Here is my script:

# -*- coding: utf-8 -*-
"""
Created on Tue Apr 12 17:32:32 2016

@author: zxlee
"""
import mxnet as mx
import numpy as np
from collections import namedtuple

LSTMState = namedtuple("LSTMState", ["c", "h"])
LSTMParam = namedtuple("LSTMParam", ["i2h_weight", "i2h_bias",
                                     "h2h_weight", "h2h_bias"])

''' one lstm cell '''
def lstm_cell(indata, prev_state, param, num_hidden, cell_idx, layer_name, dropout=0.):
    if dropout > 0.:
        indata = mx.sym.Dropout(data=indata, p=dropout)
    i2h = mx.sym.FullyConnected(data=indata,
                                weight=param.i2h_weight,
                                bias=param.i2h_bias,
                                num_hidden=num_hidden * 4,
                                name="%s_t%d_i2h" % (layer_name, cell_idx))
    h2h = mx.sym.FullyConnected(data=prev_state.h,
                            weight=param.h2h_weight,
                            bias=param.h2h_bias,
                            num_hidden=num_hidden * 4,
                            name="%s_t%d_h2h" % (layer_name, cell_idx))
    gates = i2h + h2h

    slice_gates = mx.sym.SliceChannel(gates, num_outputs=4,
                                      name="%s_t%d_slice" % (layer_name, cell_idx))
    in_gate = mx.sym.Activation(slice_gates[0], act_type="sigmoid")
    in_transform = mx.sym.Activation(slice_gates[1], act_type="tanh")
    forget_gate = mx.sym.Activation(slice_gates[2], act_type="sigmoid")
    out_gate = mx.sym.Activation(slice_gates[3], act_type="sigmoid")
    next_c = (forget_gate * prev_state.c) + (in_gate * in_transform)
    next_h = out_gate * mx.sym.Activation(next_c, act_type="tanh")
    return LSTMState(c=next_c, h=next_h)

''' one layer lstm '''
def lstm_layer(inputs, num_hidden, name, dropout=0.):

    layer_param = LSTMParam(i2h_weight=mx.sym.Variable("%s_i2h_weight" % name),
                            i2h_bias=mx.sym.Variable("%s_i2h_bias" % name),
                            h2h_weight=mx.sym.Variable("%s_h2h_weight" % name),
                            h2h_bias=mx.sym.Variable("%s_h2h_bias" % name))

    prev_state = LSTMState(c=mx.sym.Variable("%s_init_c" % name),
                           h=mx.sym.Variable("%s_init_h" % name))

    layer_out = []
    for i in range(len(inputs)):
        next_state = lstm_cell(inputs[i], prev_state, layer_param, num_hidden, i, name, dropout)
        prev_state = next_state
        layer_out.append(next_state.h)

    return layer_out

''' lstm init_states for data iterator '''
def lstm_layer_init_states(batch_size, num_hidden, name):
    init_c = ('%s_init_c'%name, (batch_size, num_hidden))
    init_h = ('%s_init_h'%name, (batch_size, num_hidden))
    return [init_c, init_h]

''' mxnet interface batch '''
class simple_batch(object):
    def __init__(self, data_names, data, label_names, label, bucket_key):
        self.data = data
        self.label = label
        self.data_names = data_names
        self.label_names = label_names
        self.bucket_key = bucket_key

    @property
    def provide_data(self):
        return [(n, x.shape) for n, x in zip(self.data_names, self.data)]

    @property
    def provide_label(self):
        return [(n, x.shape) for n, x in zip(self.label_names, self.label)]

class simple_data_iterator(mx.io.DataIter):
    def __init__(self, init_states, data_name='data', label_name='label',
                 fea_len=257, bucket_num=10, batch_size=10, order='ascend', max_seq_len=500):

        self.init_states = init_states
        self.data_name = data_name
        self.label_name = label_name
        self.fea_len = fea_len
        self.bucket_num = bucket_num
        self.batch_size = batch_size
        self.order = order
        self.max_seq_len = max_seq_len

        ''' for buckets '''
        self.buckets = np.linspace(1, max_seq_len, num=bucket_num, dtype='int32')
        self.bucket_indices = np.arange(len(self.buckets))
        if order is 'descend':
            self.bucket_indices = self.bucket_indices[::-1]

        ''' for mxnet interface '''
        self.init_states = init_states
        self.init_state_arrays = [mx.nd.zeros(x[1]) for x in init_states]
        self.init_state_names = [x[0] for x in self.init_states]
        self.default_bucket_key = self.buckets[0]
        self.provide_data = [('%s_t%d' % (self.data_name, t), (batch_size, fea_len))
                             for t in range(self.default_bucket_key)] + init_states
        self.provide_label = [('%s_t%d' % (self.label_name, t), (batch_size, fea_len))
                              for t in range(self.default_bucket_key)]

        ''' for next() '''
        self.bucket_load_idx = 0

    def reset(self):
        if self.order is 'random':
            np.random.shuffle(self.bucket_indices)
        self.bucket_load_idx = 0

    def next(self):
        if self.bucket_load_idx >= len(self.bucket_indices):
            raise StopIteration
        else:
            bucket_key = self.buckets[self.bucket_indices[self.bucket_load_idx]]
            print bucket_key
            random_label = np.ones((bucket_key, self.batch_size*self.fea_len), dtype='float32')
            random_data = random_label + np.random.normal(size=((bucket_key, self.batch_size*self.fea_len)))

            data_all = [mx.nd.array(random_data[t,:].reshape((self.batch_size, self.fea_len))) \
                        for t in range(bucket_key)] + self.init_state_arrays
            label_all = [mx.nd.array(random_label[t,:].reshape((self.batch_size, self.fea_len))) \
                        for t in range(bucket_key)]
            data_names = ['%s_t%d' % (self.data_name, t) for t in range(bucket_key)] + self.init_state_names
            label_names = ['%s_t%d' % (self.label_name, t) for t in range(bucket_key)]

            batch = simple_batch(data_names, data_all, label_names, label_all, bucket_key)

            self.bucket_load_idx += 1
            return batch

    def __next__(self):
        return self.next(self)

    def __iter__(self):
        return self

if __name__ == '__main__':
    gpus = [0]
    batch_size = 30

    # set bucket order: can be set to 'ascend', 'descend', 'random'
    order = 'descend'
    bucket_num = 40
    max_seq_len = 240
    do_val = False

    learning_rate = 0.001
    momentum = 0.9
    weight_decay = 0.00001

    num_epoch = 30

    fea_len = 257
    input_name = 'data'
    label_name = 'label'

    ''' define network: sym_gen and get_init_states '''
    num_hidden1 = 256
    num_hidden2 = 256
    num_hidden3 = 256

    def sym_gen(seq_len):
        reg_weight = mx.sym.Variable('reg_weight')
        reg_bias = mx.sym.Variable('reg_bias')
        input_sym = [mx.sym.Variable('%s_t%d' % (input_name, t)) for t in range(seq_len)]
        label_sym = [mx.sym.Variable('%s_t%d' % (label_name, t)) for t in range(seq_len)]
        label_concat = mx.sym.Concat(*label_sym, dim=0, name='label_concat')

        lstm1 = lstm_layer(input_sym, num_hidden1, 'lstm1')
        lstm2 = lstm_layer(lstm1, num_hidden2, 'lstm2')
        lstm3 = lstm_layer(lstm2, num_hidden3, 'lstm3')

        lstm_out_concat = mx.sym.Concat(*lstm3, dim=0, name='lstm_out_concat')
        fc = mx.sym.FullyConnected(data=lstm_out_concat,
                                   weight=reg_weight,
                                   bias=reg_bias,
                                   num_hidden=fea_len,
                                   name='fc')
        reg_all = mx.sym.LinearRegressionOutput(data=fc, label=label_concat, name='reg_all')
        reg_slice = mx.sym.SliceChannel(reg_all, num_outputs=seq_len, axis=0, name='reg_slice')
        return reg_slice

    def get_init_states():
        lstm1_init_states = lstm_layer_init_states(batch_size, num_hidden1, 'lstm1')
        lstm2_init_states = lstm_layer_init_states(batch_size, num_hidden2, 'lstm2')
        lstm3_init_states = lstm_layer_init_states(batch_size, num_hidden3, 'lstm3')
        return lstm1_init_states + lstm2_init_states + lstm3_init_states

    symbol = sym_gen
    init_states = get_init_states()

    ''' prepare data '''
    train_iterator = simple_data_iterator(init_states,
                                          fea_len=fea_len,
                                          bucket_num=bucket_num,
                                          batch_size=batch_size,
                                          max_seq_len=max_seq_len,
                                          order=order)
    if do_val:
        val_iterator = simple_data_iterator(init_states,
                                            fea_len=fea_len,
                                            bucket_num=bucket_num,
                                            batch_size=batch_size,
                                            max_seq_len=max_seq_len,
                                            order=order)
    else:
        val_iterator = None

    ''' training '''
    contexts = [mx.context.gpu(i) for i in gpus]

    optimizer = mx.optimizer.SGD(learning_rate=learning_rate,
                                 momentum=momentum,
                                 wd=weight_decay,
                                 clip_gradient=1)

    initializer = mx.initializer.Xavier(factor_type='in',magnitude=3)

    model = mx.model.FeedForward(symbol=symbol,
                                 ctx=contexts,
                                 num_epoch=num_epoch,
                                 optimizer=optimizer,
                                 initializer=initializer)

    import logging
    head = '%(asctime)-15s %(message)s'
    logging.basicConfig(level=logging.DEBUG, format=head)

    model.fit(X=train_iterator, eval_data=val_iterator, eval_metric='mse',
              batch_end_callback = mx.callback.Speedometer(batch_size, 1))

The memory usage results are:

max_seq_len=120
order=descend
order=ascend

bucket_num=20
395
461

bucket_num=40
395
597

bucket_num=60
395
720

max_seq_len=240
order=descend
order=ascend

bucket_num=20
575
723

bucket_num=40
608
1053

bucket_num=60
596
1320

max_seq_len=360
order=descend
order=ascend

bucket_num=20
790
1011

bucket_num=40
790
1468

bucket_num=60
823
1917

here is the figure:

order = descend:

order = ascend:

It seems that when order = ascend, memory usage increases at python/executor_manager.py in _bind_exec() at this line:

executor = sym.bind(ctx=ctx, args=arg_arrays, args_grad=grad_arrays,
                        aux_states=aux_arrays,
                        grad_req=grad_req, shared_exec=base_exec)

@GrindstoneLZX Thanks for the detailed test case. After inserting some print statements into _bind_exec() as
diff --git a/python/mxnet/executor_manager.py b/python/mxnet/executor_manager.py
index 6f5fcfd..628f0aa 100644
--- a/python/mxnet/executor_manager.py
+++ b/python/mxnet/executor_manager.py
@@ -115,7 +115,7 @@ def _bind_exec(sym, ctx, input_shapes, param_names, need_grad=False,
         raise AssertionError("need_grad must be boolean or set.")
     grad_req = {name:('write' if name in need_grad else 'null') for name in arg_names}

-
+    print('############ _bind_exec on %s' % ctx)
     # create or borrow arguments and gradients
     for i in range(len(arg_names)):
         name = arg_names[i]
@@ -124,12 +124,14 @@ def _bind_exec(sym, ctx, input_shapes, param_names, need_grad=False,
             if shared_data_arrays is not None and \
                     name in shared_data_arrays:
                 arg_arr = shared_data_arrays[name]
+                print('@@@ reusing %s...' % name)

                 # in bucketing, we want to be strict here to avoid
                 # potential bugs
                 assert(arg_shape[i] == arg_arr.shape)
                 assert(arg_types[i] == arg_arr.dtype)
             else:
+                print('### allocating %s...' % name)
                 arg_arr = nd.zeros(arg_shape[i], ctx, dtype=arg_types[i])
             arg_arrays.append(arg_arr)
             if shared_data_arrays is not None:
@@ -137,11 +139,13 @@ def _bind_exec(sym, ctx, input_shapes, param_names, need_grad=False,
         else:
             # model parameter
             if base_exec is None:
+                print('### allocating %s...' % name)
                 arg_arr = nd.zeros(arg_shape[i], ctx, dtype=arg_types[i])
                 if name in need_grad:
                     grad_arr = nd.zeros(arg_shape[i], ctx, dtype=arg_types[i])
                     grad_arrays[name] = grad_arr
             else:
+                print('@@@ reusing %s...' % name)
                 arg_arr = base_exec.arg_dict[name]
                 assert arg_arr.shape == arg_shape[i]
                 assert arg_arr.dtype == arg_types[i]

I can see that the memory sharing works as expected -- only the extra data and label blocks are allocated each time bucket length is increased.
...
2016-04-12 11:02:41,455 Epoch[0] Batch [2]      Speed: 650.74 samples/sec       Train-mse=1.004897
13
############ _bind_exec on gpu(0)
@@@ reusing data_t0...
@@@ reusing lstm1_i2h_weight...
@@@ reusing lstm1_i2h_bias...
@@@ reusing lstm1_init_h...
@@@ reusing lstm1_h2h_weight...
@@@ reusing lstm1_h2h_bias...
@@@ reusing lstm1_init_c...
@@@ reusing lstm2_i2h_weight...
@@@ reusing lstm2_i2h_bias...
@@@ reusing lstm2_init_h...
@@@ reusing lstm2_h2h_weight...
@@@ reusing lstm2_h2h_bias...
@@@ reusing lstm2_init_c...
@@@ reusing lstm3_i2h_weight...
@@@ reusing lstm3_i2h_bias...
@@@ reusing lstm3_init_h...
@@@ reusing lstm3_h2h_weight...
@@@ reusing lstm3_h2h_bias...
@@@ reusing lstm3_init_c...
@@@ reusing data_t1...
@@@ reusing data_t2...
@@@ reusing data_t3...
@@@ reusing data_t4...
@@@ reusing data_t5...
@@@ reusing data_t6...
### allocating data_t7...
### allocating data_t8...
### allocating data_t9...
### allocating data_t10...
### allocating data_t11...
### allocating data_t12...
@@@ reusing reg_weight...
@@@ reusing reg_bias...
@@@ reusing label_t0...
@@@ reusing label_t1...
@@@ reusing label_t2...
@@@ reusing label_t3...
@@@ reusing label_t4...
@@@ reusing label_t5...
@@@ reusing label_t6...
### allocating label_t7...
### allocating label_t8...
### allocating label_t9...
### allocating label_t10...
### allocating label_t11...
### allocating label_t12...
...

My best guess is that nvidia-smi is not very accurate at counting memory when you increasingly allocate a bunch of relative small memory blocks. Or @tqchen @antinucleon do we have a customized memory allocator for NDArray that might not be very optimized under this situation?

In current implementation, the memory will always be reused from the executor binded from default key. We need to make sure that the default key uses the maximum length. Otherwise the other executors will grab more memory

@pluskid @piiswrong
I rethinked about this issue, it was due to the fact the input arguments(data of different timestamps)(which are binded) is not shared across buckets because they are not allocated by the executor, instead passed in by outside, thus was not shard.

@tqchen They can still be shared, but the front end need to handle it. executor.reshape does this

Yes, bucketing also tries to share the inputs.

If it is the inputs with the same name and same shape, they are just shared.
If it is the inputs with the same name but smaller shape, then a subset of the existing array is used for sharing. (This is why we now encourage to set the default_bucket_key to correspond to the largest bucket.

I see, so I guess if user set default to largest key, this problem is kinda of already resolved? if so I will close this issue

Closing due to inactivity.


[TritonGEN] Add subgroup.local.id operation

Add operation querying the sub-group local invocation id. gpu.lane_id converts to this operation.
This is needed to enable some sub-group aware algorithms.

Related to #1637


Nick: Watch sandydhumale

Nick requests the watch of the watch_keyword sandydhumale. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

sandydhumale has been seen in 0 true positives, 1 false positive, and 0 NAAs.

Rejected by Jeff Schaller in Charcoal HQ. Reason: 'MS search shows 0 TP and 1 FP'


markdown list exclude from actual usage, table of contents

Currently, the recipe treats the first list created with markdown in the document as a table of contents item.
However, there are many situations in which one does not want to incorporate that list into the table of contents, or at least, wants the list to also appear in the text in addition to being parsed as a table of contents

In my case, I don't want to make use of this feature at all.
I have created a custom theme, and as part of the customisations I've included these overrides:
.documentation>ul:first-of-type {
    position: initial;
    width: initial;
    top: initial;
    right: initial;
    list-style: initial;
    padding: revert;
}

.documentation>ul:first-of-type li {
    border: unset;
    line-height: initial;
    padding: 0.25rem;
}

.documentation>ul:first-of-type li a {
    color: inherit;
    font-size: inherit;
}

.documentation>ul li,
.documentation>ol li {
    padding: 0.25rem;
}

.documentation li a {
    color: inherit;
}

So now this page:
# List example page

* The
* First
* List
* [And a link](#)

---

* The
* Second
* List
* [And a link](#)

looks like this:

If, like me, you don't want to use this at all, you may also want to add something like this to your theme for a full width documentation area:
.documentation {
    width: 100%;
    padding-right: 4rem;
}

Hope this helps someone!

yes it's designed like this (and later I found it's dump :D). will be fixed in v3 with a custom TBC component that can be injected in markdown without affecting the first list. For now you can fix it with custom CSS as @corazzi explained.


[material-ui][docs-infra] API header doesn't appear in the Button demo doc sidebar

Noticed this while working on #40446 - at first I thought it was something I broke in that PR, but then I saw that the link to the API section doesn't show up in the sidebar in production either:

It should be right there under "Unstyled" in the bottom right corner. I'm not sure why this would only be happening on this page—I haven't noticed the issue anywhere else.
Search keywords:

That's because the MarkdownV2 component filters API from the ToC because base UI don't need i (it's in the tab at the top of the page)
https://github.com/mui/material-ui/blob/3faf9c30ce3672430d9185763e1c9a1200394216/docs/src/modules/components/MarkdownDocsV2.js#L84


Proposal : thumbnail / original

Why not allow to differentiate the thumbnail of the zoomed photo?
For example enable a tiny or cropped thumbnail and display the original on zooming.

this is a duplicate proposal - it's a bit of a pain and complicates the plugin a great deal - which is why i didn't write it here… open to pr's tho

Ok.
Sorry for duplicate.


chore: fix image branch name with dot

The change a42d9cc83d29249797d02105ab3295aa16c4a644 replace dot . to - in image tags (e.g., v1.3-xxxx-head to v1-3-xxxx-head), ignore the dot because we publish branch images with tags like v1.3-xxx-head.
supposed to fix: https://github.com/harvester/harvester/actions/runs/9345571254/job/25727308736

@mergifyio backport v1.2 v1.3


Automated PR: Cookstyle Changes

Hey!
I ran Cookstyle 7.25.10 against this repo and here are the results.
This repo was selected due to the topics of chef-cookbook
Changes
Issues found and resolved with resources/global.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/pip.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/plugin.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/python.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/rehash.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/script.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/system_install.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

Issues found and resolved with resources/user_install.rb

1:1 refactor: Chef/Deprecations/ResourceWithoutUnifiedTrue - Set unified_mode true in Chef Infra Client 15.3+ custom resources to ensure they work correctly in Chef Infra Client 18 (April 2022) when Unified Mode becomes the default. (https://docs.chef.io/workstation/cookstyle/chef_deprecations_resourcewithoutunifiedtrue)

          1 Warning

:warning:
This Pull Request is probably missing tests.

  Generated by :no_entry_sign: Danger

@damacus until you merge #90 this will keep being reopened. I recommend you just leave this open until we get that merged.


Move block::Header validation checks into zebra-consensus.

Same rationale as #907, but per https://github.com/ZcashFoundation/zebra/pull/917#discussion_r472377610 there might be an obstacle.
There are currently two checks:

[ ] is_equihash_solution_valid

This function should be split into two parts:

the work of checking the solution should move to a new method on Solution in work/equihash.rs:#[non_exhaustive]
#[derive(Debug, thiserror::Error)]
#[error("invalid equihash solution for BlockHeader")]
pub struct Error(#[from] equihash::Error);

impl Solution {
    pub fn check(header: &block::Header) -> Result<(), Error> {
        // do the check
    }
}

the block::check module should havepub fn is_equihash_solution_valid(header: &block::Header) -> Result<(), work::equihash::Error> {
    header.solution.check(&header)
}

This might seem pointless, but it means that all of the consensus checks have the same interface: block::check::name_of_check.
the EquihashError in block/header.rs should be removed (it moves to work/equihash.rs above).

[ ] is_time_valid_at

This is also used in test code, specifically chain/src/block/tests/vectors.rs.
These tests should be moved into the consensus crate.
They can't be moved, because they rely on the generate code, which isn't exposed to the rest of the crate API
The generate code should be expressed as proptest strategies (#919) to fix this.
Then consensus tests can be written using the proptest strategies.
In fact, rather than writing unit tests, we could combine proptests with the transcript functionality to perform top-level integration tests.
Doing this correctly requires some amount of work but will save a lot of work compared to writing individual unit tests.
Rather than blocking correct code organization on the test refactoring, a better move would be to comment out the test code in zebra-chain and leave a note that it should be restored after moving to zebra-consensus.

Can I work on this?

Can I work on this?

absolutely!

@hdevalence, I'd like to suggest a different change for is_time_valid_at:
is_time_valid_at is used as a consensus rule, and in one particular test, but that particular test doesn't rely on generate:
https://github.com/ZcashFoundation/zebra/blob/8539302ad8a874747cd11a3bb3f73134c53ed995/zebra-consensus/src/block.rs#L110
https://github.com/ZcashFoundation/zebra/blob/8e15c16b425aa5dfe751be726bfa65b788c4c57f/zebra-chain/src/block/tests/vectors.rs#L163
So instead of commenting out the tests, we could:

move the tests that don't rely on generate to zebra-consensus
move is_time_valid_at to zebra-consensus
keep the tests that rely on generate in zebra-chain, and open a ticket to fix them

(I don't like to rely on code comments for future work, because they aren't actually tracked anywhere.)

Cool, that sounds like a better plan!

I have updated the issue description.


Hope to be able to analyze the ship destroyed data

Hope to be able to analyze the ship destroyed data

Thanks! Release v0.9.7-1 adds the ShipDestroyed packet, which looks like:
{
  "packet_size": 24,
  "packet_type": 8,
  "clock": 385.41638,
  "payload": {
    "ShipDestroyed": {
      "victim": 745158,
      "killer": 745190,
      "death_cause": "Secondaries",
      "unknown": 2
    }
  }
}

victim and killer are ship IDs. death_cause has a variety of different possible values.


Documentation request

I'd like to know what advantage there is to putting
eval "$(hub alias -s)"

in my .zshrc file, instead of just a plain ol'
alias git=hub

It appears to be the case—and I may be missing something, but that's where the documentation would be helpful—that hub alias -s simply executes alias git=hub, so why not just add the simpler and more direct version to my config file, instead of the very indirect version?

The hub alias -s expression may contain more configuration in the future than just alias. Having people add that into their shell means that we can update the hub alias output in the future to configure shell completion scripts for hub, for example. We're still working on improving bash and zsh completion scripts.
I hope that answers your question!


Build breaks on axios

If you have axios as dependency the build breaks. Rollup need also a json plugin to be able to parse package.json
I added
import json from 'rollup-plugin-json'
and enabled in the plugins
json(),
I had this issues only with axios but than again it is a very common library

Duplicated https://github.com/alexjoverm/typescript-library-starter/issues/76.
Although it might be a good idea to include it by default. In progres...


[HAMMER] Drop 2.3.8 support on Travis

Drop 2.3.8 support on Travis

We're not at ruby 2.4.x yet on the build machine. Will look into updating it.


Major change in 1.5.1 removes empty lines after class definitions

I updated to docformatter 1.5.1 from 1.5.0 expecting no major changes to take place. However, I found that running docformatter removes all empty lines following class definitions, even when there is no docstring present.
I'm not sure if this is intentional, but I expected docformatter to only modify places where there is a docstring present.
(I ran into this face-first because Django generates its migration files with an empty line after class definition, so a patch version upgrade resulted in 171 files changed, 113 insertions(+), 286 deletions(-) for me. Half of that is this issue, the other half that apparently the max line length was slightly shortened?)
Reproduction:
Format this file with docformatter==1.5.0 and docformatter==1.5.1:
class Test:

    default = True

1.5.0 will leave the file alone, while 1.5.1 will output
--- before/test.py
+++ after/test.py
@@ -1,3 +1,2 @@
 class Test:
-
     default = True

This change appears to be related to #130.

docformatter now has a method to determine whether blank lines after a class definition precede a docstring.  If so, the blank lines are removed.  If not, they are left alone.
Thanks @rixx for reporting the issue.  Sorry for the headaches it caused.


Initial Petri nets version

Includes

Petri nets implementation with label and weight support
Petri net fragment parser implementation for Python

See the Java implementation and this blog post for more context

Export to DOT format and basic utility functions for invoking dot executable
Export to PNML
Unit tests

Does not include:

Exporter to pm4py objects or direct PNML visualisation support

I have functions for these, but did not want to include pm4py in the koalas dependencies

Should we put them in a pmkoalas-pm4py bridge project?

This code was originally implemented for State Snapshot Miner. I have changed the module names and removed a few functions that would bring in extra dependencies.

Hi Adam,
Could you provide some code snippets of how one might create a "Petri net"?
For example could you finish the following:
from pmkoalas.dtlog import convert
from pmkoalas.discovery.alpha_miner import AlphaMinerInstance
from pmkoalas.models.petrinet import LabelledPetriNet

# create a log
log = convert(
    "a b e d z",
    "a c e f z",
    "a y e g z"
)

# run alpha miner
miner = AlphaMinerInstance()
places, transitions, flows = miner.mine_model(log)

# use given places, transitions and flows to create an
# instance of a petri net
net = LabelledPetriNet(places, transitions, flows)
# display net in some potential human readable manner
print(net)

However, running the above on the fork produces the following:
Traceback (most recent call last):
  File "F:\Projects\Koalas\working\net_test.py", line 20, in <module>
    print(net)
  File "F:\Projects\Koalas\pmkoalas\models\petrinet.py", line 59, in __str__
    return "LabelledPetriNet(" + self._label + ")\n" \
TypeError: can only concatenate str (not "NoneType") to str

Also I checked out the code, it seems mostly uncommented. We should at least strive to have the key entry points with a clear indication of how to use the function call.

Hi Adam,
Could you provide some code snippets of how one might create a "Petri net"?
For example could you finish the following:
from pmkoalas.dtlog import convert
from pmkoalas.discovery.alpha_miner import AlphaMinerInstance
from pmkoalas.models.petrinet import LabelledPetriNet

# create a log
log = convert(
    "a b e d z",
    "a c e f z",
    "a y e g z"
)

# run alpha miner
miner = AlphaMinerInstance()
places, transitions, flows = miner.mine_model(log)

# use given places, transitions and flows to create an
# instance of a petri net
net = LabelledPetriNet(places, transitions, flows)
# display net in some potential human readable manner
print(net)

However, running the above on the fork produces the following:
Traceback (most recent call last):
  File "F:\Projects\Koalas\working\net_test.py", line 20, in <module>
    print(net)
  File "F:\Projects\Koalas\pmkoalas\models\petrinet.py", line 59, in __str__
    return "LabelledPetriNet(" + self._label + ")\n" \
TypeError: can only concatenate str (not "NoneType") to str

See PetriNetFragmentTest.test_ptp_fragment()

Also I checked out the code, it seems mostly uncommented. We should at least strive to have the key entry points with a clear indication of how to use the function call.

Unit tests are better documentation than comments for new unstable APIs.

Also I checked out the code, it seems mostly uncommented. We should at least strive to have the key entry points with a clear indication of how to use the function call.

Unit tests are better documentation than comments for new unstable APIs.

Also I checked out the code, it seems mostly uncommented. We should at least strive to have the key entry points with a clear indication of how to use the function call.

Unit tests are better documentation than comments for new unstable APIs.

Adam, the tests are not in the module on pypi. You need to add a decent comment on the structure and its initial call.

It's not necessary for a version 0.01 and is creating unnecessary maintenance overhead at a time when interfaces are very fluid. All that said, it's not a hard point of disagreement for me. Since you feel very strongly about it, I can add a few more comments.

I found the entry point to petri net fragment somewhat difficult to follow. Before tackling the naming conventions (which I might leave alone for the pnfrag file), I wanted to see if you would allow an alternative function call to quickly consume fragments.
Based on the last commit, check out the following code:
import pmkoalas.models.petrinet as pn
import pmkoalas.models.pnfrag as pnfrag

# test exporting on simple net
net = pn.LabelledPetriNet(
          places=[
            pn.Place("p1",pid="f9da4b8b-2ebe-4384-a2f1-ae484cf898c8"),
            pn.Place("p2",pid="1da99779-a974-40b3-9573-bf34ab4c43cc"),
          ],
          transitions=[
            pn.Transition("t1",tid="24747a8f-0f98-4f2a-9694-1a6687c01334",
                       weight=1.0,silent=False),
          ],
          arcs=[
            pn.Arc(fromNode=pn.Place("p1",pid="f9da4b8b-2ebe-4384-a2f1-ae484cf898c8"),
                toNode=pn.Transition("t1",tid="24747a8f-0f98-4f2a-9694-1a6687c01334",
                                  weight=1.0,silent=False)
            ),
            pn.Arc(fromNode=pn.Transition("t1",tid="24747a8f-0f98-4f2a-9694-1a6687c01334",
                                    weight=1.0,silent=False),
                toNode=pn.Place("p1",pid="f9da4b8b-2ebe-4384-a2f1-ae484cf898c8")),
          ],
          name='Petri net'
        )
print(net)
pn.export_net_to_pnml(net, "./testing_import.pnml")
dotcode = pn.convert_net_to_dot(net)
with open("./test_dot.dot", "w") as f:
    f.write(dotcode)

# build a larger model
parser = pnfrag.PetriNetFragmentParser()
onet = parser.createNet("tester_net", "I -> [A] -> p1 -> [tau] -> F")
parser.addToNet(onet, "I -> [A] -> p1 -> [B] -> p2 -> [E] -> p3 -> [G] -> F")
parser.addToNet(onet, "I -> [A] -> p1 -> [C] -> p2 -> [E] -> p3 -> [H] -> F")
parser.addToNet(onet, "I -> [A] -> p1 -> [D] -> p2 -> [E] -> p3 -> [K] -> F")
onet = onet.create_net()
print(onet)
## alternative build pattern for fragments
net = pnfrag.parse_net_fragments("tester_net",
    "I -> [A] -> p1 -> [tau] -> F",
    "I -> [A] -> p1 -> [B] -> p2 -> [E] -> p3 -> [G] -> F",
    "I -> [A] -> p1 -> [C] -> p2 -> [E] -> p3 -> [H] -> F",
    "I -> [A] -> p1 -> [D] -> p2 -> [E] -> p3 -> [K] -> F"
)
print(f"equality test : {onet==net=}")
pn.export_net_to_pnml(net, "./testing_import_2.pnml")
dotcode = pn.convert_net_to_dot(net)
with open("./test_dot_2.dot", "w") as f:
    f.write(dotcode)

I found the entry point to petri net fragment somewhat difficult to follow. Before tackling the naming conventions (which I might leave alone for the pnfrag file), I wanted to see if you would allow an alternative function call to quickly consume fragments.
Based on the last commit, check out the following code:

onet = onet.create_net()
net = pnfrag.parse_net_fragments("tester_net",
"I -> [A] -> p1 -> [tau] -> F",
"I -> [A] -> p1 -> [B] -> p2 -> [E] -> p3 -> [G] -> F",
"I -> [A] -> p1 -> [C] -> p2 -> [E] -> p3 -> [H] -> F",
"I -> [A] -> p1 -> [D] -> p2 -> [E] -> p3 -> [K] -> F"
)

I think convenience function entry points are a good idea. I added some tests and fixed the style convention to fit the rest of the module. I removed the repetition of the grammar in the doc comment, as it is a technical spec that needs to be maintained in a single place.

At what point do we merge out of this long-lived branch and do the rest of the tinkering in the main project?

At what point do we merge out of this long-lived branch and do the rest of the tinkering in the main project?

I think it is ready to merge, did you want to shift the version number and make a new dist?

I will commit the version change separately


Polar score streak

I made a couple small changes to get the anchoring right and cap the streak score, but besides that it's all set and looks good.


Error messages (12)

the wording needs to be adapted:
Not: Reason: Empty or special character is detected in pair of square brackets.
But: Reason: A pair of square brackets is empty or contains not allowed characters.
Not: Slicing is currently not supported!
But: Slicing is not supported!
(no need to mention 'currently', because then customers will assume that slicing might be available in near future already)
Not: 'Invalid expression while handling the parameter '${listP.${keyP}}'.'
But: 'Invalid expression found: '${listP.${keyP}}'.'
But nevertheless: This expression itself is not really invalid. It's a data type issue: List indices are expected to be positive integers, and not strings. Later this should be reworked.
Originally posted by @HolQue in https://github.com/test-fullautomation/python-jsonpreprocessor/issues/238#issuecomment-2014728173

released with RobotFramework AIO 0.13.0


Dedicated vernacular search

searching on vernaculars can only return accepted taxa. mixing search with sciname might just lead to confusion and were better off to offer a dedicated common name search?

Is that not the same as specifying only VERNACULAR_NAMES for SearchContent? Or do you mean allowing for more common name search options like language?

in todays call we decided to remove the vernacular search from the current ES name usage search and reimplement it as a dedicated vernacular search directly on postgres.

simple postgres driven vernacular search added


Make package name configurable to be able to distinguish between inte…

…rnal and external builds
Make the default package name configurable but make the default 'openweave' to distinguish between internal and public package. Please see internal ticket for more context (can cc as requested).

This looks like a feature that belongs in the Google internal fork of OpenWeave.

This looks like a feature that belongs in the Google internal fork of OpenWeave.

Wasn't sure about where to make the change for a shared file but can definitely try make the change in internal weave. In that case, probably easier to just hardcode the package name, but it'll be cleaner in a way.

@githanwang1 : I'm not against making the build-openweave-wheel.py script customizable.  What I don't want to do is wire that logic into the OpenWeave build system.
I think there's a middle ground here: We can modify build-openweave-wheel.py to accept a package name as a command line option (--pkgname foo).   Then, after the OpenWeave build completes, your internal build script can run build-openweave-wheel.py again, supplying the package name it desires.
We can also add other options to further customize the behavior of the script if desired.  E.g. we could have an option that specifies where to put the output file.
This approach could also be a means to build both python2 and python3 wheel files in the same build.

Thanks for suggesting alternatives but not sure if I'm following correctly. Doesn't the makefile have to adjusted regardless based on the configured package name? Is the concern about modifying the makefiles (as part of the build system) vs modifying the build-openweave-wheel.py file?
The internal fork suggestion you mentioned earlier definitely seems tempting, since it's a simpler solution but maybe not as valuable.

I have been assuming that your internal build process involved some form of outer build script that called the OpenWeave Makefile (e.g. Makefile-Standalone).  Please correct me if I'm wrong.
Based on this, I was suggesting that we modify build-openweave-wheel.py so that it can be used as a stand-alone tool outside of the OpenWeave build process, in addition to being invoked during the build.
Then we modify your outer build script to invoke build-openweave-wheel.py a second time, after the OpenWeave build completes, passing it the necessary arguments to produce the package you desire.

Ah, it seems like I might need further clarification. What is meant by the OpenWeave build process?
Yes, we call make -f Makefile-Standalone which ends up invoking the OpenWeave build process (but I'm not sure where the separation is). I can see your point now that if we can get build-openweave-wheel.py to work standalone, there is no need to modify the makefiles as well, since we aren't calling make.

Well, to be a little more clear, you will still need to invoke make -f Makefile-Standalone, because you need the C++ code needs to be built into a shared library.  But after that completes, you can invoke build-openweave-wheel.py to package up the shared library, along with the python code, into a .whl file configured as you like.


API: Quota system which gives requests different weights for throttling purposes

I've tweaked the scope of issue #189 so that it includes basic rate limiting (e.g. throttle after x requests per hour), but does not distinguish the cost of different requests. This issue here will deal with the latter task.
Previously, I figured we could assign a weight to each different endpoint in the API (read metadata, create source, deploy, etc.) to account for the fact that, for example, any 'read' operation should be less expensive in terms of server resources than a 'deploy' operation. However, if we look at 'deploy' alone, this scheme wouldn't be nuanced enough. These characteristics could greatly influence the server resources required for a deploy:

Image count. A single deploy operation can request classification of 1 image, or multiple images, up to some upper limit that we define (for example, 100).
Point count per image. A deploy operation can include 1 point per image, or up to some upper limit that we define (for example, 1000 points per image).
Image resolution. A deploy can specify images that are 4 megapixels each, or 40 megapixels each.
Image filesize. A deploy can specify images that are 1 MB each, or 20 MB each.

So, I think it's crucial for us to weight different requests based on characteristics like these. For example, for deploy, we could say 1 MB of image filesize uses 5 quota units, and 1 point uses 1 quota unit. So if you call deploy on a 5 MB image with 20 points, that uses 5*5 + 20*1 = 45 quota units. And once you use 5000 quota units in one hour, you have to wait. Something like that.
Implementation-wise, this seems to go beyond django-rest-framework's basic rate limiting feature, since I think that feature can only count rates/quotas at the time an API endpoint is called. But for deploy, image resolution and filesize are evaluated when we download an image from the requested URL, and that download would happen in an asynchronous task. So, hopefully django-rest-framework provides some customization hook which lets us add to the quota at any time.

I've thought of a pretty simple idea which could account for most of the concerns above: Allow each user to have up to X active deploy requests at a time.
'Active' means the deploy jobs haven't completed yet; they're either pending or in progress. We could make X = 3, for example. Then if we keep the image limit per request at 100, that means the most efficient way to use the API would be to submit 300 images to begin with (3 deploy requests of 100 images each), check the job statuses periodically, and submit another 100 images each time a job finishes.
In general, this should make people wait longer when their requests take longer (larger images to download, larger images to process, more points, etc.), and when the spacer queue is more busy. So it seems to account for a lot of factors, without having to actually track all of that stuff in detail.

I haven't given this much thought but this does sound reasonable. Simple is usually better. I do wonder if we are putting the horse before the cart, though? Until we have a lot of users, does it make sense to even have limits? Initially, we can just configure some alerts on AWS that pings us if usage goes up a lot, and then we can see what is going on. One  way could be to make a "hidden url" similar to /backend_overview that lists the number of deploy requests by user, completed, pending, etc. Just to give us a dashboard to monitor usage in these early days. @kriegman : what are your thoughts?

I haven't given this much thought but this does sound reasonable. Simple is usually better. I do wonder if we are putting the horse before the cart, though? Until we have a lot of users, does it make sense to even have limits?

Yeah, I suppose it's not likely that we'll run into issues in the early stages. But unlimited deploy does seem fairly easy to abuse compared to 'traditional' image upload. Submitting a bunch of image URL strings is quite lightweight from the client side, while being potentially expensive on the server side (have to download all of those images, for a start). I guess that's my main concern. But if you think it's doable enough to catch any odd activity before it gets too expensive, then I'm fine with addressing this later.

Initially, we can just configure some alerts on AWS that pings us if usage goes up a lot, and then we can see what is going on. One way could be to make a "hidden url" similar to /backend_overview that lists the number of deploy requests by user, completed, pending, etc. Just to give us a dashboard to monitor usage in these early days.

That does sound like something we'd probably want either way.

Yeah, I suppose it's not likely that we'll run into issues in the early stages. But unlimited deploy does seem fairly easy to abuse compared to 'traditional' image upload. Submitting a bunch of image URL strings is quite lightweight from the client side, while being potentially expensive on the server side (have to download all of those images, for a start). I guess that's my main concern.

That is a great point. I didn't think about that. A minor coding error on their end can end up spawning millions of deploy calls which we very costly on our end. And since we don't charge for the service, that is bad news. I suppose I'm in favor, then, of your suggestion.
... and great that we agree on the monitor. Can you create an issue to that effect?

I haven't read the thread through, but I think we do need some controls on
the API. It could be very easy to consume a LOT of resources
fi it's free. e.g., it might be easier codingwise to just keep calling the
API rather than storing/caching results. Program errors as Oscar mentioned.
NOAA has plans for these mosaics, and they have thousands of images per
mosaic and hundreds of mosaics. Imagine if they start doing a
"Sweep" of say preprocessing algorithms etc.
I do think it's important to both have logs and guard rails, especially
once the API goes public.
There may be a need also to provide users with stats or a dashboard so they
know what's going on and if they're about to bump up against a limit.
David
On Tue, Feb 25, 2020 at 9:13 AM Oscar Beijbom<EMAIL_ADDRESS>wrote:

Yeah, I suppose it's not likely that we'll run into issues in the early
stages. But unlimited deploy does seem fairly easy to abuse compared to
'traditional' image upload. Submitting a bunch of image URL strings is
quite lightweight from the client side, while being potentially expensive
on the server side (have to download all of those images, for a start). I
guess that's my main concern.
That is a great point. I didn't think about that. A minor coding error on
their end can end up spawning millions of deploy calls which we very costly
on our end. And since we don't charge for the service, that is bad news. I
suppose I'm in favor, then, of your suggestion.
... and great that we agree on the monitor. Can you create an issue to
that effect?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/277?email_source=notifications&email_token=ABKA5ANXMEPXJW2DPNARPRDREVGU3A5CNFSM4KGMIX6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM4WCPY#issuecomment-590963007,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5ALIYEIIR4I2K2LDRFDREVGU3ANCNFSM4KGMIX6A
.

A minor coding error on their end can end up spawning millions of deploy calls which we very costly on our end. And since we don't charge for the service, that is bad news.

Exactly - even if we're not much of a target for intentional abuse, there can be loop-iteration bugs or something.

and great that we agree on the monitor. Can you create an issue to that effect?

Will do. Relatedly, I'll also change the title of this issue since I'm not favoring a complex weighting system anymore, at least for now.

There may be a need also to provide users with stats or a dashboard so they
know what's going on and if they're about to bump up against a limit.

That would be good to have, yeah. I'm actually not sure if user-facing usage stats/warnings are common in the big-name APIs out there, but if it's not that much effort then we should do it. I wonder if there's a standard for reporting usage stats as part of API responses; I'll look into that.

Well, if we shut off service, then they'll want to know why. Perhaps they
just have to log in and go to their profile page?
I believe that all API calls have to be from a registered user.
David
On Tue, Feb 25, 2020 at 12:18 PM StephenChan<EMAIL_ADDRESS>wrote:

A minor coding error on their end can end up spawning millions of deploy
calls which we very costly on our end. And since we don't charge for the
service, that is bad news.
Exactly - even if we're not much of a target for intentional abuse, there
can be loop-iteration bugs or something.
and great that we agree on the monitor. Can you create an issue to that
effect?
Will do. Relatedly, I'll also change the title of this issue since I'm not
favoring a complex weighting system anymore, at least for now.
There may be a need also to provide users with stats or a dashboard so they
know what's going on and if they're about to bump up against a limit.
That would be good to have, yeah. I'm actually not sure if user-facing
usage stats/warnings are common in the big-name APIs out there, but if it's
not that much effort then we should do it. I wonder if there's a standard
for reporting usage stats as part of API responses; I'll look into that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/277?email_source=notifications&email_token=ABKA5AIQDDVMJTIWHU4B3STREV4F5A5CNFSM4KGMIX6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM5K55Q#issuecomment-591048438,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKA5APGUHR6IIIBGI6FOGLREV4F5ANCNFSM4KGMIX6A
.

To clarify, yeah, it is standard for APIs to respond with HTTP code 429 (too many requests) and an appropriate error message when the limit has been exceeded. Actually, the simple throttling we currently have in place already does that. I agree that part is important.
So the question after that is, how much additional rate-related info should we show to the user beyond that.

PR #296 implemented:

Limiting concurrent deploy requests
Letting the user know when they've reached the concurrent limit

So the scope of this particular issue should be taken care of. The user-facing status monitor is for issue #294.

Hi, I have a few questions about making requests with the new public API. I'm using the base of Steve Miller's code, but I'm working with Hong Kong MarineGeo images that are 6000 x 6000 pixels, and I'm hoping to get ~1% of those labeled so I can convert them into dense labels using another script I made.
But, because the images are large and I'm asking for a decent amount of labels, I wanted to get y'alls feedback on how this can be done in a timely manner, but without bogging down your server with requests or stealing precious server time from other patrons (I understand this is a free service you're providing and I don't want to abuse it as @StephenChan mentioned).
What exactly is the most efficient method with what you currently have setup? Should I ask for all of the annotations I need for for all the images I need in a single request (up to 100 images, of course), or should I only send one request at a time asking for all of the annotations for a single image? Alternatively, I could send a single request asking for some of the annotations I need for that image, and send additional requests for the rest of the annotations (e.g. batches of annotations for a single image)?
Also, how long should this take ? Currently I send a request and wait 60 seconds before asking for a status update... Should I be waiting much longer? I see that you implemented the limits of 1000 points per image, and 5 active requests.
Anyways, any insight you can provide would be greatly appreciated, I love that you guys made this open and I'm really looking forward to seeing how CoralNet grows.

hi Jordan,
thanks for your interest in the new API!
So 1% of a 6000x6000 image is 360k points, which is a lot. How many images
did you want to annotate that way? This will likely approach quite sizable
cloud-costs, so we may have to resl;ve that before you submit more jobs.
We are actually planning to roll-out a semantic segmentation end-point
which does full image segmentation directly in a much more efficient way,
but we don't have a fixed timeline yet. We are also rolling out a new and
improved computer vision backend (for point annotations) next month. This
new backend is both more accurate and faster, so if you can wait for that
to get rolled out, it'd be a win-win.
Re. the recent jobs you submitted and which are stuck: it appears to me the
compute nodes have run out of memory.  But let's first
figure out how many points and images in total you had planned to annotate.
cheers
On Sat, Oct 3, 2020 at 5:04 PM Jordan Pierce<EMAIL_ADDRESS>wrote:

Hi, I have a few questions about making requests with the new public API.
I'm using the base of Steve Miller's code, but I'm working with Hong Kong
MarineGeo images that are 6000 x 6000 pixels, and I'm hoping to get ~1% of
those labeled so I can convert them into dense labels using another script
I made.
But, because the images are large and I'm asking for a decent amount of
labels, I wanted to get y'alls feedback on how this can be done in a timely
manner, but without bogging down your server with requests or stealing
precious server time from other patrons (I understand this is a free
service you're providing and I don't want to abuse it as @StephenChan
https://github.com/StephenChan mentioned).
What exactly is the most efficient method with what you currently have
setup? Should I ask for all of the annotations I need for for all the
images I need in a single request (up to 100 images, of course), or should
I only send one request at a time asking for all of the annotations for a
single image? Alternatively, I could send a single request asking for some
of the annotations I need for that image, and send additional requests for
the rest of the annotations (e.g. batches of annotations for a single
image)?
Also, how long should this take ? Currently I send a request and wait 60
seconds before asking for a status update... Should I be waiting much
longer? I see that you implemented the limits of 1000 points per image, and
5 active requests.
Anyways, any insight you can provide would be greatly appreciated, I love
that you guys made this open and I'm really looking forward to seeing how
CoralNet grows.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/277#issuecomment-703179494,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTF3L4JUERVVWACYDWDDSI63YFANCNFSM4KGMIX6A
.

hey Jordan,
some comments inline below. How many images will you need to annotate?
On Tue, Oct 6, 2020 at 8:34 AM Jordan Pierce<EMAIL_ADDRESS>wrote:

Hey Oscar, yeah I can definitely dial that back to .01% so each image
would only get 3,600 labels; based on your knowledge of the system do you
think this is doable without running into OOM errors or should I reduce
even further? If not, I can wait for the updated backend to become
available and try 'benchmarking' for you guys :)

3600 points per image is much more doable from a cost perspective. It would
cost approximately $0.05 to do 3600 points on one image -- the total cost
would depend on how many images you need. If it's less than, say, 1000
images we can absorb that cost.
Now, from a systems perspective we have a limit of 1000 points per image.
With the current system (soon to be deprecated) it seems you need to use
even fewer due to OOM problems.  So you'd need to make multiple calls per
image, each with perhaps 200 points or so. Then you can accumulate things
on your side once the jobs are complete.
Semantic segmentation, heck yes! Are you allowed/willing to disclose your

approach to this task, whether it's using the existing sparse points
somehow or if you're training an FCN?..

Yeah, it's a simple FCN. We still train on the point annotations, but then
make the network fully convolutional. It's pretty straight-forward. LMK if
you want to get involved. :)

Also, I'm a bit new to making requests to servers, but it is somehow
possible on y'alls end to embed a message in a status check indicating that
there was an OOM error as a result of my initial request? And if so, I
could from my end send a cancel request so you don't need to kill them
manually and it alleviates the bottleneck?

Yeah, when we roll out the new backend we will be able to give better
feedback. For now, if a job isn't complete within 24 hours you can assume
it's broken. The best we can do at the moment is sending an email to this
thread so we can cancel it on our end. Sorry about that.
@Stephen Chan<EMAIL_ADDRESS>: correct me if I'm wrong but it
seems we don't have a way for folks to cancel jobs?
I see 5 requests in the queue right now, all of which seem to be stuck.
Should I go ahead and cancel those?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/277#issuecomment-704355220,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTF2DMUEY4BHJ44ALRF3SJM2GXANCNFSM4KGMIX6A
.

@beijbom sounds good. I actually won't be doing all 900 as of now, maybe just 5 as a proof on concept. Also yes, if they're still active they can be killed and I'll start again with smaller batches, thanks.
I definitely am interested in how you plan on implementing the semantic segmentation component, but, are you doing something like Reus et al. 201 or Alsonso et al. 2019? Or are you literally taking the CNN that is currently trained on a source, converting it into some FCN-style architecture and then feeding it whole images? Or maybe with patches of the same size that it was originally trained on and then concatenating all of the output segmentation maps together?
Also, if users can't cancel it on their end, what do you think about a time limit of 24hrs or so with no amount of progress triggering an automatic kill? Again, that way you guys don't need to play clean up.

Ok, great. 900 images is no problem. Or indeed 5. haha. I was worried you'd
say 100k images. :)
Yeah, I just killed those old jobs, so go ahead and submit new ones.
Perhaps shoot for around 200 row/col locations per job and it should go
through fine.
We are just using the original FCN design [Long et al. 2015
https://arxiv.org/abs/1411.4038]. So we start with the base-CNN that is
shared across all sources, and then add the source-specific classiifer as a
1x1 conv layer, and then make the whole thing fully convolutional so that
we can feed in the whole image. Qimin, copied is working on if you want to
chat more.
cheers
On Wed, Oct 7, 2020 at 7:08 AM Jordan Pierce<EMAIL_ADDRESS>wrote:

@beijbom https://github.com/beijbom sounds good. I actually won't be
doing all 900 as of now, maybe just 5 as a proof on concept. Also yes, if
they're still active they can be killed and I'll start again with smaller
batches, thanks.
I definitely am interested in how you plan on implementing the semantic
segmentation component, but, are you doing something like Reus et al. 201
https://sci-hub.do/10.1109/oceanskobe.2018.8559302 or Alsonso et al.
2019 https://sci-hub.do/10.1002/rob.21915? Or are you literally taking
the CNN that is currently trained on a source, converting it into some
FCN-style architecture and then feeding it whole images? Or maybe with
patches of the same size that it was originally trained on and then
concatenating all of the output segmentation maps together?
Also, if users can't cancel it on their end, what do you think about a
time limit of 24hrs or so with no amount of progress triggering an
automatic kill? Again, that way you guys don't need to play clean up.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/beijbom/coralnet/issues/277#issuecomment-704961528,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAITTFZB5YV5Q5UYWGLXAWDSJRY6DANCNFSM4KGMIX6A
.

@beijbom @JordanMakesMaps yes, the idea is pretty straightforward, I tried to implement the dense prediction in Pytorch and had some results but I'm still working on it and verifying if my code is working correctly. I will let you know once I have something available. Let me know if you want to chat more, we can open another issue to discuss that.

Hey @qiminchen that sounds awesome, looking forward to seeing it! Yeah actually if you could create a new issue for that, that'd be cool because I'd like to learn more and maybe try to mimic what y'all are doing but in Keras. I'd like to see if it's better/more efficient than what I'm currently doing.
@beijbom sorry to keep spamming your inbox but I'm still getting a 429 response error; are they still showing up as active on your end?

@JordanMakesMaps : gotya. No worries. @StephenChan : can you mark the jobs as cancelled on the DB side?

@beijbom Right - just canceled the jobs on the database side. @JordanMakesMaps : Should work now, go ahead and try again.

Also, if users can't cancel it on their end, what do you think about a time limit of 24hrs or so with no amount of progress triggering an automatic kill? Again, that way you guys don't need to play clean up.

There can potentially be a queue if we get more API users later, so I'm not sure about a time limit, but I do think it makes sense to let users cancel.


add -gui and modify -app description in gui_build.py

Yes, I forgot to add it to rrv_tb_temp.sv to /.gitignore

From Methodology pont if view, git ignore should not have file that we manually create.. usually only files that are generated. (Target, logs, py_cache, etc.)
It's not a terrible thing to have in the original main a "temp" file. But it should serve a purpose..
Generaly if it's an "older" version of a file that you want to keep for reference, the aldest version should be accessible using git without adding a new temp file..

Got it! Git tracks the changes anyway so creating temp files for storing old data not really necessary


[Enhancement] Support avg rollup rewrite in MV

What type of PR is this：

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Problem Summary(Required) ：

Support AVG rollup rewrite into Sum/Count;
Ignore aggregation predicates, because aggregation predicates should be rewritten after aggregation functions' rewrite and should not be pushed down into mv scan operator;
Change fn's type  as 1st child has changed when rollup, otherwise physical plan will still use the old arg input's type
Add enable_materialized_view_single_table_view_delta_rewrite  config to indicate whether rewrite single table query into view-delta mvs or not.
Not rewrite single table query by default.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto backported to target branch

[x] 3.0
[x] 2.5
[ ] 2.4
[ ] 2.3

[FE PR Coverage Check]
:heart_eyes: pass : 80 / 87 (91.95%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[1822, 1823]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/materialization/AggregatedMaterializedViewRewriter.java
62
67
92.54%
[148, 149, 150, 151, 153]

:large_blue_circle:
com/starrocks/sql/optimizer/Optimizer.java
8
8
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/operator/scalar/CallOperator.java
5
5
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/tree/AddDecodeNodeForDictStringRule.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/materialization/MvUtils.java
2
2
100.00%
[]

run starrocks_fe_unittest

run starrocks_fe_unittest

run starrocks_admit_test

run starrocks_fe_unittest

[FE PR Coverage Check]
:heart_eyes: pass : 80 / 87 (91.95%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[1821, 1822]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/materialization/AggregatedMaterializedViewRewriter.java
62
67
92.54%
[148, 149, 150, 151, 153]

:large_blue_circle:
com/starrocks/sql/optimizer/Optimizer.java
8
8
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/operator/scalar/CallOperator.java
5
5
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/tree/AddDecodeNodeForDictStringRule.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/materialization/MvUtils.java
2
2
100.00%
[]


Ranged and Melee Defence Increase on Every Update

Using dev branch
Replication:

Create a character
Equip them with any armor that has >1 defence rating
Modify any field, or close and reopen the actor sheet
See that the R/M defence fields increase by armor value

Now fixed in the dev branch.


fail to create kubeconfig secret

What is version of KubeSphere DevOps has the issue?
v3.4.1
How did you install the Kubernetes? Or what is the Kubernetes distribution?
No response
What happened?
In kse version with 3.4.1, the jenkins plugins delete the kubernetes-cd plugin and  Azure Commons plugin，so when i create kubeconfig secret in devops project, it will be error in jenkins logs.

Relevant log output
No response
Additional information
No response

The kubernetes-cd plugin is not deleted in v3.4.1


Week number of datepicker

Is it possible to add week number to datepicker?

We don't have any plans to add a week number at this time.


[embedded] Add Set to the embedded stdlib

Mostly a straightforward addition of the Set-implementing source files to the embedded stdlib, except:

Builtin.BridgeObject in BridgeStorage causes usage of swift_bridgeObjectRelease/swift_bridgeObjectRetain, which we could probably implement in the embedded runtime, but it looks like a better idea to just avoid using Builtin.BridgeObject altogether
_swift_stdlib_Hashing_parameters are now needed, so they need to be initialized
ELEMENT_TYPE_OF_SET_VIOLATES_HASHABLE_REQUIREMENTS cannot be used in embedded Swift because it uses a metatype as an argument
_swiftEmptySetSingleton needs to be defined

@swift-ci please test

@swift-ci please test

@swift-ci please test

@swift-ci please test

@swift-ci please test

@swift-ci please test

I'm going to proceed and pursue merging this PR, even though there are some very valid points in the discussion above that deserve more explorations and investigations. Given that embedded Swift is experimental and there's no promise of stability (yet), we can always iterate on it later, and even revert implementation choices.
CC @phausler @lorentey


Dailasm/uifixes homepage

What
Fixed following Ui Issues . Also fixed snapshots for Home and Book (as FAQ section is same for both pages)

Font for link - verified that segoe is fine - done
Learn more text - fixed to show "…"
Font size 1rem - done
Icons - blue - done
2rem margin top above container with all text after calendar illustration - done
Flex width sizing for the buttons for while we have two they use all the container width and when we add the third one they resize - done
each of the links under "Learn more..." open a new tab - done
the alignment of the join from link button as it is not centered - done

Why
UI Issues
How Tested
locally and deployed -

Does this PR contain ARM Template changes?
NO
Is this a breaking change?
NO

It looks like we might need some space between the banner and Hello title.
This is from the Figma design:

Nvm fort this!


Upgraded mustache.js to v1.1.0

@CartoDB/frontend., this PR bumps the version of mustache.js from v0.5.2 to v1.1.0.
We (@javisantana and I) discovered a bug in mustache.js when investigating cartodb/cartodb#2420 and this bump fixes it.
PTAL @javisantana, @xavijam

New mustache! :dancer:

We should check mustache templates with different mustache libraries, shouldn't we?

basically the things that can break are the infowindows but if you tried every template we have in cartodb we are pretty safe

When I run grunt dist, the new version of mustache is not being included in the dist files :-(. It turns out, we are not including the file under vendor/mustache.js. Instead, we're relaying on the mustache.js that comes with wax.cartodb.js (v0.5.5).


Graph: index V3 markets  (missing Nfl Market on Graph)

Hi,
I am integrating Augur inside Mask.
It is like a component in Twitter with market data and actions like Buy/Sell/Add/Remove liquidity...
Until now, I was using a combination of Graph and contracts to fetch market data,
But seems the Nfl market query does not return any result.
Also, the Market schema missing the Nfl market.
Using this Graph: https://thegraph.com/legacy-explorer/subgraph/augurproject/augur-turbo-matic

Use the fetcher contracts to pull in V3 market factories. This will make V3 markets available to 3rd parties.

Use the fetcher contracts to pull in V3 market factories. This will make V3 markets available to 3rd parties.

Oh, thanks! where can I find them?

@iRhonin devs working on augur will pull in the market factories in the next week. If you could wait for their PR to be merged, this should just work for you without additional work from you.

Hi @iRhonin
I've added NFL and MMA markets that were missing in the latest deploy.
I also added nflMarkets to the market schema.

Hi @iRhonin
I've added NFL and MMA markets that were missing in the latest deploy.
I also added nflMarkets to the market schema.

Hi @bconfortin, thank you!

It seems nflMaket is not resovling any market:


cmd/pprof: spurious "Main binary filename not available." error

I just created a heap profile using go test -memprofile=x.prof. When I run pprof on it, the first line of output is:
Main binary filename not available.

I'm not 100% sure what this means, but it should not be printed. We just went through all the effort of making profiles stand alone so that the program binary is unnecessary for pprof's operation. The first thing it prints should not be a complaint about a file it does not need.
This print was introduced in Go 1.10:
$ go1.9 tool pprof x.prof
Type: alloc_space
Time: Jul 6, 2018 at 4:00pm (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

$ go1.10 tool pprof x.prof
Main binary filename not available.
Type: alloc_space
Time: Jul 6, 2018 at 4:00pm (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

Normally, a print introduced in Go 1.10 would not be a release blocker in Go 1.11. But now in Go 1.11 the print is in RED (or surrounded by line noise; see #26254):
$ go tool pprof x.prof
Main binary filename not available.
Type: alloc_space
Type: alloc_space
Time: Jul 6, 2018 at 4:00pm (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

It's as though pprof REALLY wants us to know that it doesn't have a file it doesn't need. We made a big deal of not needing the main binary anymore. I don't want to confuse users with a spurious RED message.

@hyangah Are you still taking care of pprof?

The only place I could find with git grep is https://go.googlesource.com/go/+/master/src/cmd/vendor/github.com/google/pprof/internal/symbolizer/symbolizer.go#321
Strange. The fix for #25743 sets the HasFunctions field and is supposed to prevent printing of the error message.
I tried to reproduce it by generating a memory profile with go test on the fmt pkg, and couldn't reproduce it.
Can you upload the profile, or add how to produce the profile?

I reproduced it on mac. There is no non-zero mapping at all.
@aalexand - I think it is too harsh to classify the lack of the file name as an error. What do you think?

Can the runtime put in the non-zero mapping corresponding to the binary, on Mac? I believe there isn't /proc filesystem on OSX which makes it difficult, is that why it's missing? Having it would be useful when profiling non-pure Go programs (to symbolize locations that couldn't be symbolized by the runtime).
What apparently happens now is that this code executes adding a fake mapping since the profile itself does not have any mappings and later the symbolizer complains about that. Maybe, similar to what was done in #25743, we could set HasFunctions in this fake mapping if all locations it points to already have the function information. We could also just silently ignore mappings with empty file name (as in https://go.googlesource.com/go/+/master/src/cmd/vendor/github.com/google/pprof/internal/symbolizer/symbolizer.go#329 where the code skips mappings with name like "[vdso]") but I am less sure about that. If some locations were not symbolized, user may specify the binary path on the pprof command line and that will be used as the mapping name. Without any diagnostics the user wouldn't know that they can do this.
@rauls5382 FYI, thoughts on this?

I think the ideal fix would be to extend #25743 to have the Go runtime generate in OSX and other non-linux OSs an empty mapping and set HasFunctions on it. This would be appropriate for Go as it does online symbolization, but other profile sources do not. This would tell pprof that local symbolization isn't needed and it will bypass all of these checks.
Having said that, I agree this message isn't very useful as is, and we could just remove it. That would trigger the more useful (but also more verbose): "Some binary filenames not available. Symbolization may be incomplete. Try setting PPROF_BINARY_PATH to the search path for local binaries."


feature request: detect msys on windows

Related to https://github.com/slog-rs/term/issues/3
And implementation in golang:
https://github.com/k-takata/go-iscygpty

I don't have access to a Windows machine so I will need someone to help me with this.


Update Profiler.php

When getProfileBetween() is invoked before start() an undefined index error raises when trying to access $startSnapshot['microtime'] or $endSnapshot['microtime'].

Can you please check the test failures? https://travis-ci.org/github/propelorm/Propel2/jobs/715807964#L539

We are also affected by these notices.
The tests appear to fail for two reasons:

start() isn't called, so $startSnapshot['microtime'] is not set.
the test runs getProfileBetween([], ['memoryUsage' => 343245]) , without specifying microtime in the stubbed $endSnapshot.

So, if microtime isn't specified by the test, it seems like the failure is actually correct? However, this doesn't account for the lack of ->start()

@SpikedCola Can you create an alternative PR please?
This once seems stale.

Should I open an issue instead? This appears to go beyond 'microtime', it also happens with the 'memoryUsage' key (again, because $startSnapshot is empty, all checks will fail). Submitting a PR that just adds isset() checks (so the test cases pass) doesn't seem like the right solution to me.

Either way works.

I think this error is related to PHP 7.4.
https://www.php.net/manual/en/migration74.incompatible.php "Array-style access of non-arrays"
I was in the process of upgrading our project from using php 7.0 to 7.4 while at the same time upgrading from a very old (2016) version of Propel2 to the latest version, and both the 2016 and the latest versions of Propel gave this or similar error messages.
That means that the Profiler has had this flaw where it checks for slow operations without a start snapshot for quite a while.
If, like me, that never bothered you in the past and all you care about is PHP 7.4 compatibility, then the change in this PR does the trick.
I'd recommend merging this or a similar PR as soon as possible, and then dealing with the underlying flaw with there being no start snapshot without the need for haste, so that at least it's not broken in PHP 7.4.

Please rebase or make a new PR.

tl;dr: The solution in this PR is the correct solution for the problem (though the condition is wrong, $endSnapshot is never empty).
From what I understand, Profiler.php has to be initialized with start(), which stores start time ('microtime'), initial memory usage and memory peak usage. For every log message, the difference between the initial values and the current status is printed as a header:
 [2021-03-17 18:57:39]      Time:  0.236ms | Memory:    900kB | Memory Delta:  +16.9kB | Memory Peak:    901kB | SELECT ...

The ProfilerConnectionWrapper class wraps the database connection object (ConnectionInterface), and starts the Profiler when prepare(), query() or exec() is called, in order to log query statistics with execution time and memory usage after execution.
However, during construction of the ConnectionWrapper (parent class of ProfilerConnectionWrapper), it tries to write a log message itself:

    /**
     * Creates a Connection instance.
     *
     * @param \Propel\Runtime\Connection\ConnectionInterface $connection
     */
    public function __construct(ConnectionInterface $connection)
    {
        $this->connection = $connection;
        if ($this->useDebug) {
            $this->log('Opening connection');
        }
    }

At that point, Profiler::start() has not been called, so when accessing the start time and memory values, they are not there, causing the warning.
There are four ways this can be fixed:

initialize the Profiler with start values
return empty header if no start values are set (this PR does that, by checking if $startSnapshot['microtime'] is set)
remove the 'Opening connection' message
use $this->getLogger()->info('Opening connection); in the constructor of ConnectionWrapper

Some observations:

The ConnectionWrapper is set to swallow all log messages not triggered by exec, query or execute (see ConnectionWrapper::log(). So its own  'Opening connection' message does not make it to the log
Once set, the stored start values are never removed, only reset. If they are not reset, profiler gives difference since last start(), i.e. during log of ProfilerConnectionWrapper::__destuct() (which is also swallowed).
The connection is given to the wrapper as an argument. Even if we wanted to measure time and memory usage during connection initialization, we couldn't, since it is already opened when the wrapper sees it.

Personally, I would remove the 'Opening connection' message, as it doesn't do anything anyway (unless a child class overrides the log function). Initializing the Profiler with start values or returning an empty header is a matter of taste, both have up- and downsides.

@mringler thanks for detailing the issue so clearly! Removing the 'Opening connection' message makes sense to me.

Hey @SpikedCola, thank you for the response!
I have created a new PR #1716 for this. If you have a minute to review and request changes or approve, it would help getting this finished. No worries if you don't, of course.
Best, Moritz

Replaced by https://github.com/propelorm/Propel2/pull/1716
Please approve the other PR.


IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes

Description
The function crashes during straightening. The segmentation looks OK: some irregularities on the derivatives, but no discontinuities.
Original post from user here.
Other notable issues which could be addressed in the same PR:

the usage does not specify "nurbs" in -param algo_fitting, even though it seems to always use this algorithm regardless the input param.
When using -v 2, the straightening command does not pass the verbose info. The sct.run() should be replaced with python call.
The linear interpolation before running nurbs is dysfunctional, because given that the smoothing window is null, it will return the intput without interpolation. Instead, the function numpy.interp could be used if smooth=0.

Steps to Reproduce
Download these data:
T2_seg.nii.gz
Run:
sct_smooth_spinalcord -i T2_seg.nii.gz -s T2_seg.nii.gz -smooth 0 0 8 -r 0 -v 2

SCT version: 4.0.2

The error is caused by the NURBS routine, not outputting the same vector length between x_centerline_fit or y_centerline_fit and z_ref, here: https://github.com/neuropoly/spinalcordtoolbox/blob/master/spinalcordtoolbox/centerline/core.py#L137

@benjamindeleener could that be a bug in this section? https://github.com/neuropoly/spinalcordtoolbox/blob/master/spinalcordtoolbox/centerline/nurbs.py#L420

Doing further investigations via the debugger, I noticed a strange behaviour. The normal call in this code produces:
P = nurbs.getCourbe3D()
len(P[0])
240  # PROBLEMATIC
len(z)
241

However, when manually running the sub-routine, I get:
courbe3D_bis, _ = nurbs.construct3D(nurbs.pointsControle, nurbs.degre, nurbs.precision)
len(courbe3D_bis[0])
241  # GOOD!

Pfiou that was a tricky one...
The issue was coming from a slight mistake in the reparametrization of the NURBS curve to make it uniform. This effect was not usually apparent but was accentuated by the segmentation inhomogeneities on the edges.
So the error comes from this line: https://github.com/neuropoly/spinalcordtoolbox/blob/master/spinalcordtoolbox/centerline/nurbs.py#L948
That we need to change for
distances_between_points = centerline.progressive_length[1:]

Pfiou that was a tricky one...

I knew it 😉
Thanks a lot @benjamindeleener! I can do the change and create the PR at this point (and bug you again if there are any more problems 😊)

Great, let me know.


Dialogs scaled incorrectly, no longer match size of their contents

Godot version: 3.1.alpha.official
OS/device including version: MacOS High Sierra, iMac Retina 5K
Issue description:
Some of the popup dialog boxes (Find in files, Go to Line) and results are not showing properly on Retina displays, see screenshots below. Other dialogs like Search Classes and Search Help show up fine.

For non-retina part, a cause was found in https://github.com/godotengine/godot/issues/21933#issuecomment-422490629


Getting JSON Unmarshal Errors from TF Plan using NR Provider

Description
I am getting some weird errors when I run my CI build which executes a Terraform Plan against my code that uses v 2.7.1 of the NR Provider. This started happening today only and I did not have any errors with my CI build before this.
Go Version
Not sure what the NR Provider uses for Go version
Current behavior
TF Plan fails with Go related errors
Expected behavior
No errors with JSON marshaling
Steps To Reproduce
Steps to reproduce the behavior:

tf plan on terraform code using NR Provider

Debug Output (if applicable)
Error: json: cannot unmarshal number into Go struct field entitySearchResponse.data.actor of type nrtime.Minutes
1349 |
1350 | on modules/synth_script_api/main.tf line 48, in data "newrelic_entity" "entity":
1351 | 48: data "newrelic_entity" "entity" {
1352 |
1353 |
1354 |
1355 | Error: json: cannot unmarshal number into Go struct field entitySearchResponse.data.actor of type nrtime.Minutes
1356 |
1357 | on modules/synth_script_api/main.tf line 48, in data "newrelic_entity" "entity":
1358 | 48: data "newrelic_entity" "entity" {
1359 |
1360 |
1361 |
1362 | Error: Service Exception: EntitySearchService.API.Error Server Error (504)
1363 |
1364 | on modules/synth_script_api/main.tf line 48, in data "newrelic_entity" "entity":
1365 | 48: data "newrelic_entity" "entity" {

Seems to happen even if I use the later NR Provider version of 2.14.1

Seems to happen even if I use the later NR Provider version of 2.14.1

Oh ouch.  Maybe the API changed underneath us.  I'll check on that.

Oh ouch.  Maybe the API changed underneath us.  I'll check on that.

Is this happening consistently?  Can I see more detail for the newrelic_entity resource please?

Is this happening consistently?  Can I see more detail for the newrelic_entity resource please?

Also, can you confirm the version.  I just checked, and the v2.7.1 of the provider wouldn't have included the nrtime.Minutes type yet.

Also, can you confirm the version.  I just checked, and the v2.7.1 of the provider wouldn't have included the nrtime.Minutes type yet.

@zlesnr - Yes it is happening consistently since today. I just re-ran our CI and still getting the same errors. Here's how we are using the newrelic_entity resource in our code:
`#entity tags (replaces synthetic labels)
data "newrelic_entity" "entity" {
name = var.name
}
resource "newrelic_entity_tags" "tags" {
guid = data.newrelic_entity.entity.guid
dynamic "tag" {
for_each = var.tags
content {
key    = tag.key
values = list(tag.value)
}
}
}`

@zlesnr - Yes it is happening consistently since today. I just re-ran our CI and still getting the same errors. Here's how we are using the newrelic_entity resource in our code:
`#entity tags (replaces synthetic labels)
data "newrelic_entity" "entity" {
name = var.name
}
resource "newrelic_entity_tags" "tags" {
guid = data.newrelic_entity.entity.guid
dynamic "tag" {
for_each = var.tags
content {
key    = tag.key
values = list(tag.value)
}
}
}`

Also, can you confirm the version. I just checked, and the v2.7.1 of the provider wouldn't have included the nrtime.Minutes type yet.

@zlesnr - yep that's the NR Provider version we are using in our repo. But even using the latest version 2.14.1 still gets the same error

Also, can you confirm the version. I just checked, and the v2.7.1 of the provider wouldn't have included the nrtime.Minutes type yet.

@zlesnr - yep that's the NR Provider version we are using in our repo. But even using the latest version 2.14.1 still gets the same error

Okay, synthetics specific was my expectation also.  I'll try to get a reproduction of this today or tomorrow and get back to you.

Okay, synthetics specific was my expectation also.  I'll try to get a reproduction of this today or tomorrow and get back to you.

This is also reproducible with newrelic entity search --name WebPortal, where WebPortal is our favorite demo app.

This is also reproducible with newrelic entity search --name WebPortal, where WebPortal is our favorite demo app.

This should be fixed in v0.55.5.  I'll work this through the CLI and Terraform next.

This should be fixed in v0.55.5.  I'll work this through the CLI and Terraform next.

Thank you for reporting the issue.  This was a lack of test coverage on our part, and not due to a change in the API.

Thank you for reporting the issue.  This was a lack of test coverage on our part, and not due to a change in the API.

Thank you @zlesnr

Thank you @zlesnr


[Behaviour] Minecraft-like command autocompletion

Hi,
Would it be possible that you switch the commands with tab? (like for example in Minecraft)
You have these commands:
sayhello
seeversion
So if you type S, a small HBoxContainer will show all commands with S, and if you type TAB, the command line will show "sayhello", and if you type TAB a second time, the command line will show "seeversion.
Otherwise the chat will be crowded with all commands. (If you use it also as chat)
Thanks,
Gamemap

Hello,
I can't say this would be supported out of the box. Currently console tries to mimic unix terminal that's why it behaves like that.
Right now I'm working on decoupling the console into smaller modules to allow people to add/replace/remove those modules without modifying the core of the console.
Right now I would suggest you go right away and modify console sources and add the features you want. Current autocomplete functionality is in godot/addons/quentincaffeino-console/src/ConsoleLine.gd on line 66. Please note that I'm unaware of how Godot acts when there's an update for an addon, so to prevent yourself from loosing your changes I suggest you to use git or etc.
After migration from monolith to module-based will be over we can include your autocomplete as a separate module for people to use.
Best)

Hello,
I can't say this would be supported out of the box. Currently console tries to mimic unix terminal that's why it behaves like that.
Right now I'm working on decoupling the console into smaller modules to allow people to add/replace/remove those modules without modifying the core of the console.
Right now I would suggest you go right away and modify console sources and add the features you want. Current autocomplete functionality is in godot/addons/quentincaffeino-console/src/ConsoleLine.gd on line 66. Please note that I'm unaware of how Godot acts when there's an update for an addon, so to prevent yourself from loosing your changes I suggest you to use git or etc.
After migration from monolith to module-based will be over we can include your autocomplete as a separate module for people to use.
Best)

Thank you

Thank you

I have two new ideas:

Maybe you canalso change the opening to a variable (toggel or open with key close with enter) (If TextLine empty no print)
Maybe add chatcompability (multiplayer) (type a special key (var commandkey = "/") for command otherwise chattext)
(for current  behaviour var commandkey = "")

What do you think about this?
Gamemap

I have two new ideas:

Maybe you canalso change the opening to a variable (toggel or open with key close with enter) (If TextLine empty no print)
Maybe add chatcompability (multiplayer) (type a special key (var commandkey = "/") for command otherwise chattext)
(for current  behaviour var commandkey = "")

What do you think about this?
Gamemap

Hey, those are great ideas but only for a specific project but the purpose of this project is to allow people to create any type of console.
So as I said earlier, you can do that within your project)
Those features could be implemented but as a separate modules so that people that don't need them could disable em and vice versa, but  rn this addon is implemented as a monolith so it is impossible to create and use separate modules without affecting the core of the addon. I'm working on splitting it but it is not essy)

Hey, those are great ideas but only for a specific project but the purpose of this project is to allow people to create any type of console.
So as I said earlier, you can do that within your project)
Those features could be implemented but as a separate modules so that people that don't need them could disable em and vice versa, but  rn this addon is implemented as a monolith so it is impossible to create and use separate modules without affecting the core of the addon. I'm working on splitting it but it is not essy)

I also thought of separate modules for these ideas.
You build a really good addon.
Good luck
Gamemap

I also thought of separate modules for these ideas.
You build a really good addon.
Good luck
Gamemap