from_generator feedback and questions

@mrry, thank you for implementing the from_generator method in tf.data. I just wanted to provide some feedback and ask a few more questions.
Interface
In addition to having generator be a callable that returns an iterator, would it be possible to support iterators that aren't wrapped in a callable? E.g. instead of
pool = multiprocessing.Pool()
dataset = tf.contrib.data.Dataset.from_generator(
    lambda: pool.imap(some_function, some_data), dtypes, shapes
)

also support
pool = multiprocessing.Pool()
dataset = tf.contrib.data.Dataset.from_generator(
    pool.imap(some_function, some_data), dtypes, shapes
)

Return types
from_generator does not seem to support unpacking numpy arrays at the moment. I don't think it's essential but would be a nice-to-have. E.g. this fails
def generator():
    while True:
        yield np.zeros(2, np.float32)

dataset = tf.contrib.data.Dataset.from_generator(generator, (tf.float32, tf.float32))
x, y = dataset.make_one_shot_iterator().get_next()
session = tf.Session()
session.run([x, y])

but this runs smoothly
def generator():
    while True:
        yield tuple(np.zeros(2, np.float32))

dataset = tf.contrib.data.Dataset.from_generator(generator, (tf.float32, tf.float32))
x, y = dataset.make_one_shot_iterator().get_next()
session = tf.Session()
session.run([x, y])

Performance
I've played around with a very naive example using the generator API (in the hope to eventually leverage ipyparallel or some other distributed computing framework). Unfortunately, I can't achieve the same performance that I would get using feed_dicts. The setup is as follows
import tensorflow as tf
from tensorflow.contrib import data as tfdata
import numpy as np
from time import time

num_batches = 1000
batch_size = 100

class Generator:
    def __init__(self):
        self.times = []

    def __iter__(self):
        while True:
            x = np.random.normal()
            y = 3 + 5 * x
            x, y = np.asarray([x, y], np.float32)
            self.times.append(time())
            yield x, y

generator_state1 = Generator()

dataset = tfdata.Dataset.from_generator(
    lambda: generator_state1,
    (tf.float32, tf.float32),
    (tf.TensorShape([]), tf.TensorShape([]))
)
prefetched = dataset.prefetch(3 * batch_size)
batches = prefetched.batch(batch_size)
iterator = batches.make_one_shot_iterator()

x, y = iterator.get_next()

w = tf.Variable([0, 0], dtype=tf.float32)
prediction = w[0] + w[1] * x
loss = tf.losses.mean_squared_error(y, prediction)
optimizer = tf.train.AdamOptimizer(0.1)
train_op = optimizer.minimize(loss)
init_op = tf.global_variables_initializer()

session = tf.Session()
session.run(init_op)

Running the optimisation gives me
losses = []

start = time()
for _ in range(num_batches):
    _, _loss = session.run([train_op, loss])
    losses.append(_loss)
time() - start  # about seven seconds

Doing the same using feed_dicts gives
losses = []

generator_state2 = Generator()
iterator = iter(generator_state2)

start = time()
for _ in range(num_batches):
    _x, _y = np.transpose([next(iterator) for _ in range(batch_size)])
    _, _loss = session.run([train_op, loss], {x: _x, y: _y})
    losses.append(_loss)
time() - start  # about one second

It seems that the dataset created using the from_generator method isn't fetching from the generator fast enough:
np.mean(np.diff(generator_state1.times))  # 7.1533812683949508e-05
np.mean(np.diff(generator_state2.times))  # 1.0633696558370612e-05

Thanks again for this, and looking forward to hearing your thoughts.

My 2 cents of feedback: I have been trying to use the Dataset.from_generator API (thanks) to return an iterator to another dataset, constructed in the generator - but I can't really make it work without passing the session in:
import tensorflow as tf

range5 = tf.contrib.data.Dataset.range(5).make_one_shot_iterator()
with tf.Session() as sess:
    print(tf.__version__)
    def _dataset_generator():
        while True:
            try:
                yield sess.run(range5.get_next())
            except tf.errors.OutOfRangeError:
                return
    gen_dataset = tf.contrib.data.Dataset.from_generator(_dataset_generator,
                                                         tf.int64)
    gen_it = gen_dataset.make_one_shot_iterator()
    while True:
        try:
            print(sess.run(gen_it.get_next()))
        except tf.errors.OutOfRangeError:
            break

I may be missing something obvious but there seems to be no way to "chain" the iterators - so just issue yield range5.get_next() . Moreover the pattern above does not (easily ?) generalize to more complex datasets. Implementing the generator as a python generator over a typical Dataset needs not use the session (but we still must explicitly loop in python)
And a note on errors - I erroneously wrote Dataset.zip((Dataset.from_tensors(...), tf.range(count)) and I got the rather cryptic errors_impl.UnknownError: AttributeError: 'Tensor' object has no attribute 'output_types'. It seemed rather natural to pass a tensor as argument - I guess many will be tripped by this. Would help to have some type checks in zip for a clearer error message maybe?
Thanks!

Hi folks, thanks for the feedback. I started with a pretty conservative interface and implementation for Dataset.from_generator(), so we should be able to evolve it towards a better state. Responding to @tillahoffmann's and @Utumno's  individual points:

In addition to having generator be a callable that returns an iterator, would it be possible to support iterators that aren't wrapped in a callable?

Yes, we could do this, but I wasn't able to figure out a sensible set of type checks to make this work smoothly. There are also some annoying wrinkles: e.g. a generator expression is a Python iterator so you can only iterate over it once, unless you wrap it in a callable. Of course, this only matters if you try to .repeat() the resulting dataset! I think the workaround (explicit wrapping) is better than the confusion that could arise here, but I'm open to better suggestions... perhaps these would involve better static information about whether it's possible to repeat a dataset?

from_generator does not seem to support unpacking numpy arrays at the moment. I don't think it's essential but would be a nice-to-have.

I suppose we could extend the nest library so that it would delve into NumPy arrays when it does nest.flatten_up_to(output_types, values) on the elements yielded from the generator. It doesn't seem incompatible with how we already use it, but I'm not sure what it would take. Let's call that "contributions welcome". :)

I've played around with a very naive example using the generator API (in the hope to eventually leverage ipyparallel or some other distributed computing framework). Unfortunately, I can't achieve the same performance that I would get using feed_dicts.

This is a bit surprising, and we'll need to do some more investigation. It's possible that there's some overhead from acquiring and releasing the GIL in tf.py_func() and I don't think it's been heavily optimized. One quick question though: is the steady-state performance 7x worse, or does the difference in mean shrink if you exclude the first step when the iterator is constructed? (That's not an excuse of course, but it would help to focus our efforts!)

I have been trying to use the Dataset.from_generator API (thanks) to return an iterator to another dataset, constructed in the generator - but I can't really make it work without passing the session in[.]

This is not really the expected use for Dataset.from_generator() and as we discussed on Stack Overflow, you can use Dataset.map() if you need to capture a different iterator. However, I'm not quite sure what the use case is here... why don't you want to use the original iterator directly?

I erroneously wrote Dataset.zip((Dataset.from_tensors(...), tf.range(count)) and I got the rather cryptic errors_impl.UnknownError: AttributeError: 'Tensor' object has no attribute 'output_types'.

I coincidentally fixed this in 3c00952c6680d77ee2f10def35fbc7cbd138aea3.
Thanks again for using the new API, and let me know if you have more comments or thoughts on these answers!

This is not really the expected use for Dataset.from_generator() and as we discussed on Stack Overflow, you can use Dataset.map() if you need to capture a different iterator. However, I'm not quite sure what the use case is here... why don't you want to use the original iterator directly?

Use case: I have a dataset (d1, lengths) of variable length tensors (read from tfrecord files) and I want to produce from it another dataset d2 with slices of the tensors in d1.

I coincidentally fixed this in 3c00952.

I was bitten by the list vs tuple in Dataset.zip and took some to figure that out too - good job!

@Utumno Would Dataset.flat_map() work for this? For example, if you have a dataset of variable-length tensors called var_len_dataset, you can get all the slices as follows:
var_len_dataset = ...  # Element shape (?, x, y, z)
all_slices = var_len_dataset.flat_map(
    lambda x: tf.data.Dataset.from_tensor_slices(x))  # Element shape (x, y, z)

I have the zip(d1, num_slices) dataset and I want to produce another dataset that slices each tensor in d1 into num_slices slices (segments of  equal length). I posted a SO question - I know the code there is not correct (I should be using the dummy dataset) but still I am not sure the approach is at all possible.

I posted an answer: Dataset.flat_map() should work for your case.

@mrry, thanks for the feedback.

There are also some annoying wrinkles: e.g. a generator expression is a Python iterator so you can only iterate over it once, unless you wrap it in a callable.

Yes, very good point.
Regarding the performance issues: https://gist.github.com/tillahoffmann/53f902212c1c511e6c3d45abb5f37b9b

@mrry, do you have any more thoughts on the speed differences with feed_dicts, queues and the dataset API?

I haven't had time to dig into this in depth, but I did notice that when I dial the batch size down to 1, the Dataset.from_generator() version is faster than using a feed_dict. This suggests that the overhead might be coming from the Dataset.batch() transformation. I'm surprised that batching a std::vector<Tensor> (even by a pretty naive approach) would be slower than stacking together a batch of ndarrays (which implicitly happens when you feed the list), but perhaps there's something to be learned from the NumPy implementation?
/cc @jsimsa who has looked at batch performance in other contexts.

@tillahoffmann give map_and_batch, providing identity for the map a try. Dataset.batch() is serializing memory copies during batch creation, which can affect performance for large batches. Dataset.map_and_batch() is doing the memory copies in parallel.

@mrry, @jsimsa, thank you for the updates.
I have changed the line
batches = dataset.batch(BATCH_SIZE).prefetch(PREFETCH_FACTOR * BATCH_SIZE)

to
batches = dataset.apply(map_and_batch(lambda a, b: (a, b), BATCH_SIZE)).prefetch(PREFETCH_FACTOR * BATCH_SIZE)

in the notebook linked above. But the code hangs indefinitely when I try to train.

@tillahoffmann apply the following fix to your version of TF: https://github.com/jsimsa/tensorflow/pull/1 and try again ... if that does not help, please use gdb to connect to the running process using sudo gdb -p <PID>, then run thread apply all bt to collect all stack traces, and share them here

@jsimsa, thank you for the fix. It seems to have made it into master because I can run the cod using the nightly build. Unfortunately, the performance is slightly worse than the original code that used batch instead of map_and_batch. I have updated the notebook.

@tillahoffmann I digged into this a bit more and my conclusion is that the slowdown is due to the (constant) overhead of tf.data runtime. In particular, I  have done the following:
import time
import numpy as np
import tensorflow as tf

from tqdm import tqdm
from tensorflow.contrib import data as tfdata

BATCH_SIZE = 128
FEATURE_SHAPE = (16, 16)
NUM_BATCHES = 1024*2
PREFETCH_FACTOR = 4

weights = np.random.normal(0, 1, FEATURE_SHAPE)

def data_generator():
  for _ in range(NUM_BATCHES * BATCH_SIZE):
    feature = np.random.normal(0, 1, FEATURE_SHAPE)
    value = np.sum(feature * weights) + np.random.normal(0, 1)
    yield feature.astype(np.float32), value.astype(np.float32)

def data_generator_batch():
  for _ in range(NUM_BATCHES):
    features, values = np.empty((BATCH_SIZE, 16, 16)), np.empty((BATCH_SIZE))
    for i in range(BATCH_SIZE):
      features[i] = np.random.normal(0, 1, FEATURE_SHAPE)
      values[i] = np.sum(features[i] * weights) + np.random.normal(0, 1)
    yield features, values

def build_graph(batchX, batchY):
  weights_tensor = tf.get_variable('weights', FEATURE_SHAPE, tf.float32)
  predictions = tf.reduce_sum(weights_tensor * batchX, axis=(1, 2))
  loss_tensor = tf.losses.mean_squared_error(batchY, predictions)

  optimizer = tf.train.AdamOptimizer()
  train_op = optimizer.minimize(loss_tensor)
  return loss_tensor, train_op

def train(sess, train_op, loss_tensor):
  times = []
  losses = []
  for i in tqdm(range(NUM_BATCHES)):
    start = time.time()
    _, loss = sess.run([train_op, loss_tensor])
    times.append(time.time() - start)
    losses.append(loss)
  return np.asarray(times), np.asarray(losses)

times = {}
losses = {}

# Dataset without batch training
print("Dataset with batching generator")
with tf.Graph().as_default():
  dataset = tfdata.Dataset.from_generator(
      data_generator_batch, (tf.float32, tf.float32), ((BATCH_SIZE, 16, 16),
                                                       (BATCH_SIZE)))
  batches = dataset.prefetch(PREFETCH_FACTOR)
  iterator = batches.make_one_shot_iterator()
  batch_X, batch_y = iterator.get_next()

  loss_tensor, train_op = build_graph(batch_X, batch_y)

  sess = tf.Session()
  sess.run(tf.global_variables_initializer())
times['datasets_batch'], losses['datasets_batch'] = train(
    sess, train_op, loss_tensor)

# Using generator
print("Feed with batching generator")
with tf.Graph().as_default():
  batch_X = tf.placeholder(tf.float32, (BATCH_SIZE, ) + FEATURE_SHAPE)
  batch_y = tf.placeholder(tf.float32, (BATCH_SIZE, ))

  loss_tensor, train_op = build_graph(batch_X, batch_y)

  sess = tf.Session()
  sess.run(tf.global_variables_initializer())
  tf.train.start_queue_runners(sess)

data = data_generator_batch()
_times = []
_losses = []

for i in tqdm(range(NUM_BATCHES)):
  X, y = next(data)
  start = time.time()
  _, loss = sess.run([train_op, loss_tensor], {batch_X: X, batch_y: y})
  _times.append(time.time() - start)
  _losses.append(loss)
times['generator_batch'] = np.asarray(_times)
losses['generator_batch'] = np.asarray(_losses)

# Dataset with batch training
print("Dataset without batching generator")
with tf.Graph().as_default():
  dataset = tfdata.Dataset.from_generator(
      data_generator, (tf.float32, tf.float32), (FEATURE_SHAPE, ()))
  batches = dataset.batch(BATCH_SIZE).prefetch(PREFETCH_FACTOR)
  iterator = batches.make_one_shot_iterator()
  batch_X, batch_y = iterator.get_next()

  loss_tensor, train_op = build_graph(batch_X, batch_y)

  sess = tf.Session()
  sess.run(tf.global_variables_initializer())
times['datasets_no_batch'], losses['datasets_no_batch'] = train(
    sess, train_op, loss_tensor)

# Using generator
print("Feed without batching generator")
with tf.Graph().as_default():
  batch_X = tf.placeholder(tf.float32, (BATCH_SIZE, ) + FEATURE_SHAPE)
  batch_y = tf.placeholder(tf.float32, (BATCH_SIZE, ))

  loss_tensor, train_op = build_graph(batch_X, batch_y)

  sess = tf.Session()
  sess.run(tf.global_variables_initializer())
  tf.train.start_queue_runners(sess)

data = data_generator()
_times = []
_losses = []

for i in tqdm(range(NUM_BATCHES)):
  X = []
  y = []

  for i in range(BATCH_SIZE):
      _X, _y = next(data)
      X.append(_X)
      y.append(_y)

  start = time.time()
  _, loss = sess.run([train_op, loss_tensor], {batch_X: X, batch_y: y})
  _times.append(time.time() - start)
  _losses.append(loss)
times['generator_no_batch'] = np.asarray(_times)
losses['generator_no_batch'] = np.asarray(_losses)

which results in:
Dataset with batching generator
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2048/2048 [00:09<00:00, 214.56it/s]
Feed with batching generator
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2048/2048 [00:07<00:00, 291.40it/s]
Dataset without batching generator
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2048/2048 [01:05<00:00, 31.11it/s]
Feed without batching generator
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2048/2048 [00:08<00:00, 227.84it/s]

In particular, if a generator that produces whole batches is used (as opposed to a generator that produces single elements batched by tf.data), the throughput of tf.data-based execution is comparable to the throughput of feed-based execution.
Normally, I would expect the cost of fetching a single element from a pipeline to be include a possible expensive tf.data.map operation, over which the constant overhead of executing the tf.data a pipeline (which for instance involves scheduling TF ops) would be amortized. For pipelines that do very little computation (such as the one used in your examples), the overhead of the general purpose pipeline execution engine can be non-negligible (as illustrated by your experiment) and pushing the batching logic into the generator is one possible solution to avoid the slowdown.

@jsimsa Somebody is suggesting this approach on stackoverflow. Is it a workaround or a best practice?

/cc @matpalm

I commented on the Stack Overflow thread; if possible, moving expensive processing from a generator to a map will make it possible to parallelize the processing. I find that preferrable to creating multiple identical generators and combining them using interleave.
On a related note, the extreme version of the approach discussed on the Stack Overflow page is to replace  tf.data.Dataset.from_generator with tf.data.Dataset.range(N).map(...) where map(n) generates the n-th element. If possible, this will maximize the opportunity for parallelism as the only computation that needs to be serialized is the generation of the map function inputs.

@jsimsa I think that generator is an important parallel case to handle in the docs cause frequently it needs to handle itself the concept of epoch.

@jsima Using tf.data.Dataset.range(N).map(...) instead of tf.data.Dataset.from_generator does not alwais work. As I mentioned in #14448, tf.data.Dataset.from_generator wrapps tf.py_func to have a generalized tf.py_func (i.e. allow nested structures as output instead of only flat outputs. Also it assigns the shapes. In #15121 is a PR that extracts the generalized tf.py_func).
On Stack Overflow I gave an example how you can replace tf.data.Dataset.from_generator with something that can be parallized from tensorflow.

@boeddeker So mainly  num_parallel_calls will be independent from dataset.batch: I.e. num_parallel_calls could be set to the number of available core and dataset.batch to the device capacity? It this use case is dataset.batch %  num_parallel_calls needed to be 0?

@bhack Afaik is dataset.batch independent from num_parallel_calls. Using the example code for from_indexable from Stack Overflow, below a small example. The execution time is time to get one item devided by num_parallel_calls and is independent from batch size.
import time

class PyDataSet:
    def __len__(self):
        return 20

    def __getitem__(self, item):
        time.sleep(0.5)  # <----------------- Takes 0.5 seconds to "load" the data
        return np.random.normal(size=(10))

ds = from_indexable(PyDataSet(), output_types=tf.float64, output_shapes=[None, 10], num_parallel_calls=4)
ds = ds.batch(9)
it = ds.make_one_shot_iterator()
entry = it.get_next()
with tf.Session() as sess:
    start = time.perf_counter()
    try:
        while True:
            print(sess.run(entry).shape)
    except tf.errors.OutOfRangeError:
        pass
    delta = time.perf_counter() - start
    print(delta)  # 2.5277008840057533

Yes what I meant is that if you make 4 parallel calls (i.e. you have 4 cores) for each get_next how much get_next internally will be called to fill a batch of 9?

I added a print to __getitem__ and it printed 12 times for 4 cores with batch size 9. So there are 12 __getitem__ calls.
Further, I changed the sleep to only apply to one entry per num_parallel_calls
        if item in [0, 4, 8]:
            time.sleep(1)

and the slowest entry defined the runtime (i.e. 3 seconds, ideally it would take 1 second).
So my conclusion is that num_parallel_calls follows the fork and join idea. It starts num_parallel_calls and waits until all of them have finished before launch the next num_parallel_calls.
Maybe @mrry knows more about the internals, my knowledge is based on toy examples.

Yes what I am guessing if there is not an explicitly data.prefetch where dataset.batch % num_parallel_calls is going? Is it in the next batch? Does it make sense to maintain dataset.batch % num_parallel_calls == 0?

I am not sure if I understand your question correctly.
My example does not contain data.prefetch and dataset.batch % num_parallel_calls is 1.
My experiments showed that the iterator items are independent of the value num_parallel_calls.
The only reason that I can find for dataset.batch % num_parallel_calls == 0 is when you are only interested in a single call and does not want to burn/waste computation power and/or RAM.
map caches the not required calculations for the next batch.
Here an example what I understand that happens:
num_parallel_calls == 4
batch == 9
launch __getitem__[i] for i in [0, 1, 2, 3]
wait until all __getitem__ calls finished, launch __getitem__[i] for i in [4, 5, 6, 7]
wait until all __getitem__ calls finished, launch __getitem__[i] for i in [8, 9, 10, 11]
wait until __getitem__[8] finished than build a batch
Note: In the background the remaining calls to __getitem__ may still compute)
nest session run
wait until all __getitem__ calls finished
wait until all __getitem__ calls finished, launch __getitem__[i] for i in [12, 13, 14, 15]
...

The only reason that I can find for dataset.batch % num_parallel_calls == 0 is when you are only interested in a single call and does not want to burn/waste computation power and/or RAM.

Yes.
Where 13, 14, 15  will go? In the next batch?

Yes, the next batch will contain the items 9, 10, 11, 12, 13, 14, 15, 16, 17.

So there is some kind of buffer that store 9, 10, 11 from the third __getitem__ call?

Yes, from my experiments it seems to be so.

@boeddeker your understanding is almost complete ... the only difference is that in reality there is no barrier of size num_parallel_calls. The is __getitem__[i + num_parallel_calls] launched only after __getitem__[i] finishes but possibly before __getitem__[i + 1] finishes. Internally, the map implementation has num_parallel_calls cyclic buffer that is used for storing the invocation results.

@jsimsa Thanks for the explanation.
I had a bad example, where the element 0, num_parallel_calls, ... are the slowest.
Do you also know, why calling 20 times time.sleep(1) with  num_parallel_calls=20 takes 5 seconds on a computer with 4 cores (I expect 1 second)?
It seems that the "prefetching" rule in map follows your explanation, but the numbers of threads are limited to min(num_parallel_calls, cpu_core_count).

@jsimsa So basically it has a internal auto prefecthing for the next batch. Right?

Each map invocation itself does not have a thread pool, instead it schedules the invocations using the interop_thread_pool (which has a default size of cpu_core_count). Try increasing the size of the interop_thread_pool to 20 and see what effect it has on the runtime.

Thanks, inter_op_parallelism_threads work. But the second tf.Session requires also use_per_session_threads to overwrite the value.
config = tf.ConfigProto(
    inter_op_parallelism_threads=20,
    # use_per_session_threads=True  # required when not first Session in python interpreter
)

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I had opened another issue (https://github.com/tensorflow/tensorflow/issues/16343) which is also related to feedback of the generator process. I think it's beneficial to move those comments here and close that issue in order to consolidate the related discussions to one issue.

We're given hundreds of data files, each containing many gigabytes worth of sample data in a custom format. As far as I can tell there are only two approaches to extract samples from this using Dataset:

tf.data.Dataset.from_generator(generator=my_custom_reader, ...)

Create a generator which produces samples. This approach is not ideal because this method must be the first dataset in the chain. The generator cannot accept a tensor. Therefore you can't batch and shuffle your list of 100's of filenames (or anything more complex). You also can't make use of interleave(...) because the generator can't accept a tensor, and this use case is begging to use interleave(...).
A solution here might be to provide a method for a generator to accept a tensor, as tf.py_func(...) does for functions.

tf.data.Dataset.map(map_func=tf.py_func(my_custom_reader, ...), ...)

The map function does allow us to shuffle and parallelize the filenames using all of the functionality of the Dataset pipeline, however, with map, the files must be read into memory completely, and these files are large. Reading numerous files into memory is infeasible.
A solution here might be to extend the map function to support generators.
Unless there's an alternative approach, which I didn't glean from the docs or stackoverflow, then this seems to be an inherent limitation and a seemingly reasonable use case on which to base a feature request.
I've implemented this process using map functions by splitting large files into filename/index pairs so as to specify a subset of the file to read in each map call. This was not a trivial task. A final sample solution can be found on stack overflow below. I continue to believe that Dataset should be extended to better support this workflow. In particular with respect to supporting generators throughout the pipeline.
https://stackoverflow.com/questions/48471688/using-tensorflows-dataset-pipeline-how-do-i-name-the-results-of-a-map-oper/48589464#48589464
Update to this post:
The solution referenced above doesn't work in the case where each sample needs to be different shapes. This code example demonstrates an attempt to return an ndarray of ndarray's which produces an unsupported object type numpy.ndarray error in tensorflow. The goal of the example is to demonstrate reading 2 samples of variable length from a file.
import tensorflow as tf
import numpy as np

def mypymap(data_numpy):
  result = np.array([
    np.array([100,101,102]),
    np.array([200,201,202,203])
  ])
  return result

data_input = ['fileA','fileB','fileC']

ds = tf.data.Dataset.from_tensor_slices(data_input)
ds = ds.map(lambda data_tensor: tf.py_func(mypymap, [data_tensor], Tout=[tf.int64], stateful=False))

element = ds.make_one_shot_iterator().get_next()

with tf.Session() as sess:
  for _ in range(3):
    print(sess.run(element))

So that leaves the only solution being to incorporate a bunch of logic into my generator (for interleaving, parallel file reads, and shuffling) and putting the generator up at the beginning of the preprocessing pipeline. All things that are better done as part of the Dataset pipeline.

Can we document the relationship between from_generators and Dataset cache?

Nagging Assignee @mrry: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@mrry I saw dataset.batch(BATCH_SIZE).prefetch(FACTOR * BATCH_SIZE) and dataset.batch(BATCH_SIZE).prefetch(FACTOR), which ones is correct usage?

Nagging Assignee @mrry: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@mrry Is it possible to add inp argument like tf.py_func for Dataset.from_generator?
It would be more flexible for this approach

I'm interested in something like this too @ychfan.

Nagging Assignee @mrry: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Has there been any discussion on or a conclusion to adding num_parallel_calls to from_generator()?

@RylanSchaeffer the function from_generator will never get a num_parallel_calls becasuse a python generator is something that cannot be parallelized. In #14448 I suggested to add an alternative to from_generator that can have a num_parallel_calls (Use __getitem__ instead if of __iter__), but the feedback was that it is not necessary.
I updated my answer in https://stackoverflow.com/a/47884927/5766934, so using that code is now simpler.

Assuming I/O is the bottleneck, does it suffice to chain from_generator with prefetch?
You could always do multiple from_generator in a flat_map too. Particularly tf.contrib.data.parallel_interleave is sweet.

Nagging Assignee @mrry: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@mrry
Hi Derek;
seems to me that the from_generator can not catch up the requirement of the data feed for the convolution neuron network processing on gpu.
following is the image of the gpu performance

I am using the dataset.from_generator to get the input data from slicing of pandas dataframe. set the prefetech as 5. the image shows the teeth of the gpu utilization which implies the gpu is waiting for the data feed in, wonder why the from_generator can not catch up the requirement via multithreading of cpu.
pls indicate if there is any trick to improve the feed speed of the from_generator.
tks.

@mrry
further will from_generator introduce the parameter as num_parallel_calls which will parallel the process of the generator function.
tks.

@jianxucsm As I mentioned earlier, a python generator cannot run in parallel. So from_generator will never get a num_parallel_calls, as long the input is a generator.

Nagging Assignee @mrry: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 74 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 89 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 106 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @mrry: It has been 120 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@boeddeker So what is the preferred method?

@zh794390558 It depends, when you have a lightweight generator, then using tf.data.Dataset.from_generator is fine.
When your generator gets expensive, I would rewrite it to something that is indexable and use my code that I posted in https://stackoverflow.com/a/47884927/5766934.
Alternatives are splitting your generator in a lightweight generator and a map function (https://stackoverflow.com/a/47089278/5766934) or use tf.contrib.data.parallel_interleave (https://stackoverflow.com/a/47874044/5766934). I personally do not like tf.contrib.data.parallel_interleave in cases where loading the examples is independent.

https://stackoverflow.com/a/47884927/5766934 this need the dataset must be a iterator, how to using the yield pipeline?
The tf.contrib.data.parallel_interleave  will working easy for yield pipeline, and I think the tf.contrib.data.parallel_interleave has a random case  to feed data to shuffle buffer.

In my opinion is a yield pipeline not necessary. The code that you write in your generator is usually something like:
def gen():
  # pre calculations
  for i in range(10000):
    # example calculations
    yield (i, [1] * i)

where you have a for loop over an index or a file list.
The counterpart code where this is indexable is then:
class gen:
  def __init__(self):
    # pre calculations
  def __len__(self):
    return 10000
  def __getitem__(self, i):
    # example calculations
    return (i, [1] * i)

So most yield pipelines could be rewritten with a class that has __getitem__.
Main advantages:

__getitem__ can be parallelized, while a generator can not be parallelized
you can shuffle without actually loading the data (i.e. no buffer is needed)

Since I do not use tf.contrib.data.parallel_interleave, TFRecord, I have no performance comparison.
I like to do a shuffle across all my data for each epoch and TFRecord does not fit this use case.

Can we add something in the doc/faq?

IMHO would be the best place to document something like that at a from_indexable method in the Dataset source code.
But @mrry argued in #14448 that from_indexable would be to trivial and the user can write the code himself.

@boeddeker why not to using tf.data.Dataset.shuffle ？If using it, then we not need to control shuffle manually in class gen.

@zh794390558 Because that function uses a buffer.
Usually the buffer size is significant smaller than your number of examples, because the size of your dataset is far away from fitting in memory (you can overcome this when your actual data loading happens in tensorflow).
When this is the case, the shuffle is only a local shuffle.
So when your complete dataset is sorted (e.g., the deprecated sklearn.datasets.fetch_mldata('MNIST original') was sorted) the shuffle can only shuffle the similar examples that are close together.
Toy example:
Let's consider your dataset is sorted and the labels are:
[1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 9]
When the buffer size is 4 you can never observe a 3, 4, 5, 6, 7, 8 or 9 at the beginning of an epoch.

Because that function uses a buffer.

Speaking of which, @mrry mentioned on a related issue (maybe StackOverflow, don't remember sorry!) that the tf.data team is looking into a disk shuffle of the entire index. Have there been any more news/work around this? I guess there are complications with distributed training but it doesn't feel like it should be that hard to write a permutation.index file per shard and maintain close to random access in the underlying data store.
tf.data is great but one of the big missing features is definitely the possibility of shuffling an entire Dataset per epoch. After porting some keras.utils.Sequence code to tf.data I get somewhat worse final test accuracy on a bunch of standard benchmarks (FashionMNIST, Boston house pricing, etc.) and it's greatly affected by shuffle buffer size.

So I  think tf.data should using iterator which same to class gen, then do index shuffle by tf.data.

In my pipeline dataset shuffling is the bottleneck. Is my use-case (large dataset, small networks. In my case 3e8 samples, 9D input space, 1D output space. Networks 2 layers of 128 neurons) any consideration? Found this way of shuffling 4x faster then TF dataset shuffle in some cases:
perm = np.arange(self._num_examples)
np.random.shuffle(perm)
self._data = np.take(self._data, perm, axis=0)

how do you take TFRecords of sequence examples and turn them into ragged tensors? It seems like this might be supported but there are no docs about it.

hello, if my generator yield x and y, x has shape(batch, 257, 250,1) and y has shape(batch, class_num), what value should output_type be ? i try and get generator yielded an element that did not match the expected structure may times

Thanks for all the great feedback on this thread! Since this is now an old issue and the Dataset.from_generator() API and implementation have evolved, I'm going to close the thread now. Please feel free to open new issues if you have comments about Dataset.from_generator() or how it could be made more useful.

@mrry @aselle @davidparks21 @tillahoffmann @jsimsa
Hi , I'm not familiar with tf.data api and huge data training tricks，in my case, I found the IO is a problem: GPU uti is low, about 15%. I use a generator function to read local multiple pickle files(about 200K),so each batch is a local file in my case, examples are bellow:
def train_data():
    while True:
        for i in shuffle(train_id):
            df = pd.read_pickle(df_train_val['file_name'].iloc[i],
                                compression='gzip')
            feature = df.to_numpy().reshape(df.shape[0], -1)
            yield feature.reshape(-1, 56),feature.reshape(-1, 56)

Then
gn = tf.data.Dataset.from_generator(train_data, output_types=(tf.float32, tf.float32),
                                          output_shapes=((None, 56),(None, 56))
                                         ).prefetch(tf.data.experimental.AUTOTUNE)

I found prefetch not work and IO bottleneck is very high, so any advice for this case, how can I speed up the training process?

@Anhaoxu, this kind of question should be moved to stack overflow, could you repost it there and put a link to the topic here?
In brief, for high IO data Python is a major obstacle here. The Python GIL (global interpreter lock) limits you to one CPU core when processing is done in python and not external libraries like tensorflow (which handles processing in C external to the Python runtime). I run a model that chews on about 400 MB/sec of data and that requires about 3 cores to handle IO. If your main process tries to do all the work it can delay the training loop, and one core might be enough for the IO period. There are 2 solutions as I have come to understand. 1) write your data to tfrecords files in subprocesses/multiprocessing (what I currently do). Tensorflow can deserialize tfrecords files in more than one core outside of Python so you're good. But that's a little cumbersome to implement. 2) Use python 3.8 (for which tensorflow is just now starting to support) and use System V shared memory where you use subprocesses (Python multiprocessing) to do the IO and prep work, and hand off a pointer to the prepared sample to the main process. All other forms of Python serialization between processes engage the Python GIL and don't work under high IO scenarios. I have not tested this approach yet, I just installed Python 3.8 and TF 2.2RC2 to do so in the past few weeks.

In brief, for high IO data Python is a major obstacle here. The Python GIL (global interpreter lock) limits you to one CPU core when processing is done in Python

I have to correct that statement. The Python GIL limits you to execute the python code on a single CPU, not the C code.
When a function is implemented in C, it can release the GIL. When the GIL is released, you can do parallel IO with threads in python.
A famous example for a function that releases the GIL is open, so an IO operation usually releases the GIL. Another example are many numpy functions. So using a threadpool for numpy operations can utilizes all cores of your system (Assuming, the arrays are large enough)
@Anhaoxu The problem in your example is, that a generator cannot be executed in parallel. When you start your dataset with a range and do the IO in a tensorflow map function, the IO can be executed in parallel. That is the reason, why I proposed to have a from_indexable.

@boeddeker, your clarifications make an excellent point and clarifies something I wasn't clear about. Many libraries, Tensorflow, in particular, release the GIL and can process on multiple cores. However, at high IO data rates (a good fraction of a GB/sec) the problem becomes CPU bound not IO bound because just the deserialization process inside of Python drives up the CPU utilization past 1 core.
For example, TensorFlow utilizes 3-4 cores to deserialize the tfrecords files I pass it, and those are in a nice efficient protobuf format. Python cannot read at those rates because the GIL becomes CPU bound on the basic serialization processes that are required to prepare the data for tensorflow, not on the IO-wait part.
I've spent many iterations trying to solve this problem for a relatively small CNN model that reads raw uncompressible scientific data from a 3TB dataset. With threads, I maybe get 120% CPU use out of Python. Subprocesses can't pass data to the main process because serialization becomes CPU bound in Python (GIL issue). AsyncIO with multiple event threads is all GIL based, even using c-based uvloop.
If someone is using Python/Tensorflow where you see more than about 1.2 cores being used by the main process (the max I've ever seen) I would absolutely love to know about it. Perhaps the only difference in my case is that my data has to be read over a remote filesystem, so it's coming in over HTTP from a local S3-compliant interface, perhaps that changes the equation for me.
I went around and around with this problem and the only thing that worked was dumping to tfrecords and letting tensorflow handle the multi-core deserialization outside of Python. And I believe shared memory is another solution. But trying to push a good fraction of a GB/sec of data through the same python interpreter as the training loop is hopeless in my experience (at least pre-py3.8 shared-memory).

@boeddeker @davidparks21
I just want to use map api to yield parallel generators, but the map api not supported generator type, in my case, I would like to know how to realize this page https://www.tensorflow.org/guide/data_performance#parallelizing_data_extraction, or how to use from_indexable to realize parallel generators, it's not clear to me, and I can't find related information.
in fact, I have multiple files, they are from s3, I just found reading from s3 is slower than read from local file system, so I compressed these files, maybe 200K files, in this process  I also can concat them to be less number, maybe just 1K files, but I think this operation not solve the problem fundamentally, the problem is how can not let the GPU wait for data?
for example:
I want to shuffle their file id to make every epoch training using all files, so I write this function:
def train_data():
    while True:
        for i in shuffle(train_id):
            df = pd.read_pickle(df_train_val['file_name'].iloc[i],
                                compression='gzip')
            feature = df.to_numpy().reshape(df.shape[0], -1)
            yield feature.reshape(-1, 56),feature.reshape(-1, 56)

this function is very clear, right? so how this function be parallel? if there are some tutorial about parallel and distributed data transformation or training, I would be very appreciate.

As others have already mentioned, please ask this question on stackoverflow @Anhaoxu

your clarifications make an excellent point and clarifies something I wasn't clear about. Many libraries, Tensorflow, in particular, release the GIL and can process on multiple cores. However, at high IO data rates (a good fraction of a GB/sec) the problem becomes CPU bound not IO bound because just the deserialization process inside of Python drives up the CPU utilization past 1 core.

@davidparks21 good point. Yes, it highly depends on the data that you use. I mainly work with wav files on a remote file system. Wav files are usually relative large ~ 5MB.
Because of that (and the file format) I have a purely IO bottleneck.
Depending on the file format and the reader also other data can be efficiently loaded (e.g. decompress images could be done without the GIL in C).
I have some good experiences for parallel deserialization of uncompressed data with pickle.loads (note with an s).

I've spent many iterations trying to solve this problem for a relatively small CNN model that reads raw uncompressible scientific data from a 3TB dataset. With threads, I maybe get 120% CPU use out of Python. Subprocesses can't pass data to the main process because serialization becomes CPU bound in Python (GIL issue). AsyncIO with multiple event threads is all GIL based, even using c-based uvloop.

I am not sure, how you moved the data from one subprocess to another, but probably you used pickle.
Did you try to save your data as pickle, read all data with open and then call pickle.dumps? This can be done in threads and need only low CPU time (pickle data is already prepared for python).
@davidparks21: When you want to continue this discussion, should we move this discussion to somewhere else?


Code - macOS: Check if Launch Service is enabled before creating persistence through Electron

Won't fix


Revert "pyros_setup: 0.0.9-0 in 'indigo/distribution.yaml' [bloom]"

Reverts ros/rosdistro#10230
The sourcedeb is failing to build and it consequently keeps trying to rebuild.

Yep, lets revert that. I thought it would have been easier than it is, to integrate dh-virtualenv. Not sure about the buildfarm setup, I ll try ros-buildbot (and maybe a prerelease) on a local server to see if I can catch something...

FYI just installing dh-virtualenv package on my buildbot-ros server is enough to get the sourcedeb step to succeed. However it fails later on because of : https://github.com/spotify/dh-virtualenv/issues/127


TextField autofill for common user properties taken from iOS

Use case
Autofill values that can be automatically retrieved from iOS.
Proposal
It'd be great to add a new property to TextField/TextFormField that will allow to control over iOS' textContentType field. For example it is handy to auto fill the user's name, email or phone number.
More info:
https://developer.apple.com/documentation/uikit/uitextcontenttype
https://developer.apple.com/documentation/security/password_autofill/enabling_password_autofill_on_a_text_input_view

I thought that Flutter might already do this via keyboardType, but it looks like textContentType.  This would probably mean adding a field to CupertinoTextField, passing it through to the engine, and then setting textContentType in the engine.  That's doable and I think this is a very good feature to have.
There may be something equivalent to do for Android too.

Related: https://github.com/flutter/flutter/issues/13015

#https://github.com/flutter/engine/pull/13878

I have a suggestion of add TextContent.username this properity to let iOS to know that this is a filed of Username (beacuse password textcontenttype already known by ios secure text entry.)


Paths for plugging in editor need update

The Paths in the editor still refer to TBDF Engine, instead of the DCAF Engine

In new install this is the only one that appears with DCAF and Tag Editor Core


Shipping line can't be blank

I only got this error at the final stage of the test, which means I made the payment and I try to complete the checkout.
I didn't find a way to define the shipping rate.
onPaymentResult(Map<String, dynamic> result) async {
    debugPrint('Payment Result $result');

    final shopifyCheckout = ShopifyCheckout.instance;

    var items = List<LineItem>.empty(growable: true);
    items.add(LineItem(
        quantity: 1,
        variantId: products[0].productVariants[0].id,
        title: products[0].title,
        id: products[0].id));

    var address = Address(
        address1: '11 Hinkler Avenue',
        city: 'Sydney',
        country: 'Australia',
        countryCode: 'AU',
        firstName: 'Anderson',
        lastName: 'Fetter',
        phone: '046444444',
        zip: '2229');

    final checkout = await shopifyCheckout.createCheckout(
      lineItems: items,
      shippingAddress: address,
      email<EMAIL_ADDRESS>    );

    await shopifyCheckout.shippingAddressUpdate(checkout.id, address);

    String token = result['token'][0] as String;

    final idempotencyKey = UniqueKey().toString();

    await shopifyCheckout.checkoutCompleteWithTokenizedPaymentV3(
        checkout.id,
        checkout: checkout,
        token: token,
        paymentTokenType: PaymentTokenType.APPLE_PAY,
        idempotencyKey: idempotencyKey,
        amount: '1.00',
        currencyCode: 'AUD');
  }

[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: checkoutUserErrors during checkoutCompleteWithTokenizedPaymentV3: [{__typename: CheckoutUserError, code: BLANK, field: null, message: Billing address can't be blank}, {__typename: CheckoutUserError, code: BLANK, field: null, message: Shipping rate can't be blank}, {__typename: CheckoutUserError, code: BLANK, field: null, message: Shipping line can't be blank}]
#0      ShopifyError.checkForError (package:shopify_flutter/mixins/src/shopfiy_error.dart:14:9)
#1      ShopifyCheckout.checkoutCompleteWithTokenizedPaymentV3 (package:shopify_flutter/shopify/src/shopify_checkout.dart:607:5)
<asynchronous suspension>
#2      TestTabState.onPaymentResult (package:fighter_comics/screens/test_tab.dart:95:5)
<asynchronous suspension>

According to Shopify documentation, you won't be able to checkout the products that require shipping.
Shopify Reference

What I didn't find it is place to set the shipping method. Or a place to get he shipping methods as well.

Thank you so much mate, it was an error in my code.
the process requires:

 var checkout = await shopifyCheckout.createCheckout(
      lineItems: items,
      shippingAddress: address,
      email: 'anderson',
    );
//missed part to get available Shipping Rates
checkout = await shopifyCheckout.getCheckoutInfoQuery(checkout.id);
    var shippingRates = checkout.availableShippingRates?.shippingRates;

//missed part, set shipping rate
    await shopifyCheckout.checkoutShippingLineUpdate(
        checkout.id, shippingRates?[0].handle ?? '');

Glad to hear you found the fix to the issue


请问有没有试过backbone 为resnet的蒸馏

我尝试实现用resnet101去蒸resnet50，发现收敛很慢，而且map很低，请问你尝试过吗

@cjt222 How is the effect of your model training now?

I try to use res50 based FPN to teach res18 based FPN, but the mAP is not so good. Is that normal?


Git Action fails after connection creation

My Git Action workflow fails after connection is created with Azure DevOps service, getting Error: Could not queue the build because there were validation errors or warnings. as it shows in the screenshot below.

Hi @talalalamdar-tomtom , is this still an issue for you ?

Yeah it was, i had to change my implementation, by not using git action anymore and just use auto trigger from the azure pipeline it self

Same issue here. No real way to see what is happening

Ran into this issue and was able to resolve it by making sure my Azure Pipeline had a default value set for all input parameters.  When being triggered by the GitHub Action, the Pipeline is forced to use default values for all input parameters.

Since parameters cannot be passed in when invoked from this action, one would have to ensure that all the parameters are set in a different manner - aka defaults. Closing this issue.

There is an open PR to be able to specify parameters: #63


run edgex with nats messagebus in docker get “already connected to NATS” error

I use the edgex-compose project to generate the docker-compose.ymal file, the generation command is cd compose-builder&& make gen no-secty nats-bus ds-virtual, the command to generate the docker image is cd edgex-go&& make docker-nats. After running docker-compose up, the edgex-app-rules-engine container has been restarting. Check the logs and find that the application in the edgex-app-rules-engine container has connected to the nats service twice, resulting in an error“already connected to NATS”when connecting for the second time。
🌍  Your Environment
Deployment Environment: ubuntu 20.04 ,docker 23.0.1
EdgeX Version [REQUIRED]: 3.0.0

@zouyuxuan please use English to describe your problem because most people in this community are English speaker.  Using English could get more help.
Also, please don't just paste your log with the context.  Please fill out all the section in the issue template, including version, config, and steps, or we can't help you.

@zouyuxuan you also have to run make docker-nats for app-service-configurable and for device-virtual. I assume you did this?
Also please provide the log file for edgex-app-rules-engine container

@zouyuxuan , have you been able to resolve the error? If so, please close this issue.


breaking(diagnostics): make zebrad diagnostics into optional compile-time features

Motivation
This PR makes a bunch of zebrad diagnostics optional by default, including most tracing layers.
Turning off dependencies helps improve compilation speed, runtime speed, and supply-chain security.
Depends-On: #4519
Designs
Make diagnostic features optional at compile time, and turn them off by default.
Solution
Compile-time features:

Disable the flamegraph feature by default at compile time
Disable the journald feature by default at compile time
Disable the prometheus metrics exporter feature by default at compile time
Disable the tracing filter reload feature by default at compile time
Minimise dependencies and dependency features

Documentation:

Mention the Rust feature flag in the tracing and metrics setup docs:

https://github.com/ZcashFoundation/zebra/blob/optional-deps/book/src/user/tracing.md
https://github.com/ZcashFoundation/zebra/blob/optional-deps/book/src/user/metrics.md

Review
Anyone can review this PR.
Reviewer Checklist

[ ] Existing tests pass

I'm running a speed test here, ideally we're looking for 5h35m:
https://github.com/ZcashFoundation/zebra/actions/runs/2413137208

gustavovalverde changed the title ~breaking(features): Make zebrad diagnostics into optional compile-time features~ feat(diagnostics)!: make zebrad diagnostics into optional compile-time features 2 days ago

@gustavovalverde I'd like to make sure this change gets included in the first paragraph of the release notes. Since we don't have automatic detection of !, we're relying on the changelog author to do that manually. So I think breaking is a bit more obvious.

@mergify update.

@upbqdn I'm not sure if mergifyio update works on PRs that are based on other branches.
I'll rebase it manually later today.

@upbqdn there was a bug in the tests - I needed to turn off the conflict tests when those features are disabled, as well as the "port is working" tests.
CI should pass now.

This seems to have just been a few slow syncs, I'm going to restart and see if that helps.
(Disabling unused features shouldn't change our sync performance or reliability, if it does, that's a very weird bug.)

@mergifyio update

I think lightwalletd is deadlocking again... retrying

I think GitHub automatically requested a review from you when I merged #4623.


[GS][W-7976392] Change button label

Critical Changes
Changes
Change Label value
Issues Closed
Community Ideas Delivered
New Metadata
Deleted Metadata

W-7976392


How do we describe enums in generic types?

This was raised in #480.
Consider:
public class C<T>
{
    public enum E { Xyz }
}

That's entirely valid, and C<string>.E and C<int>.E are different types... even though an enum declaration itself can't have type parameters.
Is it reasonable to consider the declaration of C<>.E to be a "generic enum declaration"?

I'll offer “No” – it’s a non-generic member of a generic type.
Disclaimer: I haven’t looked into the implications for the Standard text either way.

That would be inconsistent with:
public class C<T>
{
    public class D {}
}

... where D is described in 14.2.1 as a generic class declaration.

It would be inconsistent with §15.2.1 as an enum is a value type aka struct. But…

§14.2.1 says classes inside generic classes are also generic
§15.2.1 says structs inside generic structs are also generic
But neither says anything about a struct/class nested inside of a generic class/struct…

Can meet worms ;-)

<IP_ADDRESS> Nested types in generic classes says
https://github.com/dotnet/csharpstandard/blob/53e081b393ea33d9a8b3dbe955ff50984ba09ab9/standard/classes.md#L1100
So yeah, it's a generic enum type declaration. I suggest linking from all the recently added generic-type paragraphs to <IP_ADDRESS> and adding a similar paragraph for enums.
53e081b393ea33d9a8b3dbe955ff50984ba09ab9

@Nigel-Ecma: Both 14.2.1 and 15.2.1 do talk about a struct/class nested inside a generic class/struct, as far as I can see:
From 14.2.1:

Additionally, any class nested inside a generic class declaration or a generic struct declaration is itself a generic class declaration, since type arguments for the containing type shall be supplied to create a constructed type.

From 15.2.1:

A struct declaration that supplies a type_parameter_list is a generic struct declaration. Additionally, any struct nested inside a generic class declaration or a generic struct declaration is itself a generic struct declaration, since type arguments for the containing type shall be supplied to create a constructed type.

Have I misunderstood your concern?

@jskeet – You haven’t misunderstood, I obviously wasn’t reading the version I thought I was, and I double checked because I didn’t quite believe we’d let such a simple error stand. So mea culpa, both those clauses are correct in this regard and have been for quite some time (at least 7 years!) in the Standard.

On 18/03/2022, at 10:32 pm, Jon Skeet @.***> wrote:
@Nigel-Ecma https://github.com/Nigel-Ecma: Both 14.2.1 and 15.2.1 do talk about a struct/class nested inside a generic class/struct, as far as I can see:
From 14.2.1:
Additionally, any class nested inside a generic class declaration or a generic struct declaration is itself a generic class declaration, since type arguments for the containing type shall be supplied to create a constructed type.
From 15.2.1:
A struct declaration that supplies a type_parameter_list is a generic struct declaration. Additionally, any struct nested inside a generic class declaration or a generic struct declaration is itself a generic struct declaration, since type arguments for the containing type shall be supplied to create a constructed type.
Have I misunderstood your concern?
—
Reply to this email directly, view it on GitHub https://github.com/dotnet/csharpstandard/issues/482#issuecomment-1072233757, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSYVW5SESIDZQZXF5WG533VARES3ANCNFSM5QJTOQBQ.
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.


Fix ResolveToParents, which seem to broke https://github.com/TiVo/intellij-haxe/issues/196 Also it used to throw error on loading of my test project

Fix ResolveToParents, which seem to broke https://github.com/TiVo/intellij-haxe/issues/196
Also it used to throw error on loading of my test project

Yay tests passed

Approved.  Are you intending to make this a release or just check in with added release notes?

When @fzzr- will test if it fixes his issues, then I think it might be a good idea to make a quick release, because issue is pretty critical

OK.  I am adding a new section to CONTRIBUTING.md that describes the release process.  It's actually a good bit of work to do it right.

The release process has been documented and pushed to master: 1b9652cf3aa33ecb3bd1d7a48b26675314503821


docs: update iOS release and re-add code documentation

What
doc: update iOS release and re-add code documentation

Codecov Report

Merging #3384 (6054ae4) into develop (246436b) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 6054ae4 differs from pull request most recent head f024ac2. Consider uploading reports for the commit f024ac2 to get more accurate results

@@           Coverage Diff            @@
##           develop    #3384   +/-   ##
========================================
  Coverage    10.33%   10.33%
========================================
  Files          260      260
  Lines        12588    12588
========================================
  Hits          1301     1301
  Misses       11287    11287

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

some conflicts here


Pack config ignores '.yml' files

When placing *.yml config file in /opt/stackstorm/configs, - it's ignored.
After renaming it to *.yaml, - it works.
Both .yaml and .yml file extensions should be supported, according to:
https://github.com/StackStorm/st2/blob/b5f170d26d4a9f2bccd36e4570d92ca283f4930f/st2common/st2common/constants/meta.py#L23

Hey is anyone working on this issue?  I would like to work it.

@mack7121 No one currently working on this issue, but if you'd like to contribute the fix, - it's very welcome! :+1:

To add some context - in the past we supported both (*.yml and *.yaml), but a while ago we decided that going forward we want consistently and only support one way of doing that (that is, only support *.yaml files).
That's also the reason why that and some of the new code doesn't support it. If anything, we should deprecate and drop support for *.yml files :)
@mack7121 You are also welcome to work on that issue.


Chore: bump chokidar to v4

Feature Request Checklist

[x] I have read and agree to Mocha's Code of Conduct and Contributing Guidelines
[x] I have searched for related issues and issues with the faq label, but none matched my issue.
[x] I want to provide a PR to resolve this

Overview
I've made a PR to bump chokidar v4. It drops 12 dependencies and has better types.
Suggested Solution
Bump chokidar to v4.
Alternatives
Not bump chokidar to v4.
Additional Info
No response

It drops 12 dependencies and has better types.

Dependencies are updated as we go. Issues can of course be raised if there are special needs such as bugs etc. This is not a special need.
So: Closing, we will update dependencies together with Mocha 11, #5216, no need for separate PR:s.


CloudKit syncing is currently limited to 750 rows per outline

This is a limit placed on the number of CloudKit records associated with a parent record.

This will be fixed in Zavala 2.7.4.


Train model from scratch

Hi,
I m trying to create a deep-learning in golang which is able to recognize emotion from a face picture.
I would like to use gorgonia.
Do think is possible to create and train a model with using your package ?
Thanks

As long as you have the training data, yes

ok thanks for your answer i will try !


使用github账号登录错误

errInfo:
Get https://api.github.com/user: dial tcp <IP_ADDRESS>:443: connect: connection refused
err request:
Request URL: https://studygolang.com/oauth/github/login
Request Method: GET
Status Code: 303 See Other
Remote Address: <IP_ADDRESS>:443

过段时间 登录ok 关闭issue


Issue with instantly rerendering table with setState and dynamic Columns

Griddle version
"griddle-react": "^1.3.1",
Expected Behavior
Assume that a .csv has been imported, converted to JSON, and contains data composed of six columns: App Icon, App Store, App Title, Detected, Client, and Keyword. Columns Temp1 through Temp4 for are not included and thus they are not populated.
Columns temp1 through temp4 should be hidden, leaving the table with only 6 columns.

Actual Behavior
Columns temp1 through temp4 are appearing at the end of the table with blank cells.

Steps to reproduce
Here is the code:

I know that setState() doesn't always instantly rerender but I am at loss as to how I could instantly update my table. Also, I would like to know how I can make the number of columns dynamic and based on the number of headers given in the .csv file. Any feedback would be much appreciated.

@DannyTakeuchi Column Definitions are set only when the component mounts currently.
See #671.


quit confirmation, navigation appearance update

Thanks for your PR. To make it translatable, extract string resources to separate file. In Kotlin, we have lambdas, so there's no need to implement Dialog OnClickListener listeners.


Pane jumping with ^+⌘+1-9 shortcut.

This PR implements #1295
I've chosen to use ^+⌘+1-9 to jump to a pane because ``⌘+1-9` is used to jump to a tab.
Shortcut appears on top left of each term (in margin) from 2 panes in tab.

Possible improvements:

Make shortcut appears only when ^+⌘ keys are down
Choose an another shortcut?

Is it better to tranform this PR to a plugin ?

@chabou I think it might be better to make it a plugin 🤔 We're trying to If something can be done as a plugin, it should be a plugin instead of a core Hyper feature. Will confirm with @rauchg and get back to you ASAP 🙌

What I'd like to have is a very simple way for plugins to configure new keymappings. This would be trivial to write, in that case. So, when we merge the keymap feature we need to keep in mind that it should be easy for @chabou to extend that config from his plugin to accomplish this.
Thoughts?

@chabou Can you point me to the pane numbering order?

@ppot I made a selector: https://github.com/chabou/hyper/blob/pane_jumping/lib/selectors.js#L16-L20
It use existing findChildSession() function (moved in utils/term-groups.js: https://github.com/chabou/hyper/blob/pane_jumping/lib/utils/term-groups.js#L8-L21) used for next/previous pane calculation. It was the same order than iTerm2: "first child descendent".
I needed to add a special meaning of 9 number which is "the last":
here https://github.com/chabou/hyper/blob/pane_jumping/lib/components/term-group.js#L52
and here https://github.com/chabou/hyper/blob/pane_jumping/lib/actions/ui.js#L209
I don't know yet if your work about Session Restoring (and state refactoring) will impact this.

@chabou I was saying the docs about the order of the pane.

@chabou I will supply pane numing. So you will be able to navigate with pane number!

I will adapt my plugin to use your work ;)

@rauchg my plugin use mousetrap for keybindings inspired by https://github.com/iamstarkov/hyper-panes-iterm2-hotkeys. I think it is a better way to achieve this than hacking hterm like this plugin: https://github.com/gyandeeps/hypernpm
It would be really great if, in next keymap feature, handlers registered by a plugin for a key combination will receive dispatch() function directly.


Cannot filter by column when data is from snmp.tables

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

bring data into bucket from  snmp.tables

Expected behavior:
When trying to filter data collected in the bucket from a column that is data from snmp.tables the query should return matches based on the data in that column, just as it does for snmp.fields.
Actual behavior:
When trying to filter based on the column that is data from snmp.tables the query returns "No Results" even though the data is clearly there in the table. When filtering columns that have snmp data that came from snmp.fields the filtering works as expected.
Environment info:

System info: Linux 4.18.0-147.el8.x86_64 x86_64
InfluxDB version: InfluxDB 2.0.8 (git: e91d41810f) build_date: 2021-08-13T18:22:30Z

Here is additional info...
Basic query used against data in bucket from snmp.tables.
from(bucket: "InfluxDB_Server")   |> range(start: v.timeRangeStart, stop: v.timeRangeStop)   |> filter(fn: (r) => r["_measurement"] == "Volume")   |> drop (columns: ["_start", "_stop", "_measurement", "host"])   |> filter(fn:(r) => r.agent_host == "nas-01" and r._field == "volumeTableDescription")
This basically gives me a list of volume names that are mounted on the host nas-01 in the _value column, but I don’t want all the volumes I want a specific volume by name but when I add…
|> filter(fn:(r) => r._value == "Volume_1")
It returns No Results even though the volume “Volume_1” IS on the list of volumes returned.
This is the RAW data in Influx...

I don't know if this is a bug or not but finally figured out I can use the following to filter data in the _value field.
|> filter(fn: (r) => (strings.containsStr(v: r["_value"], substr: "Volume_1")))


Getting ROUTE_NOT_FOUND when authenticating with twitter in version ~0.9.2

This is my code:
    var auth = $firebaseAuth(ref);
    auth.$authWithOAuthPopup("twitter").then(function(authData) {
      console.log("Logged in as:", authData.uid);
    }).catch(function(error) {
      console.error("Authentication failed: ", error);
    });

but when i try to run it, it outputs:
   {"error":{"code":"ROUTE_NOT_FOUND","message":"Route not found."}}

I tried google and its working.

Huh, interesting. I'm not sure exactly what went wrong and things appear to be working for me locally. But I do have a couple questions for you to see if we can get to the bottom of this:

What version of the Firebase client are you using? If you aren't using the latest (2.2.0 as of this comment), try upgrading and see if that fixes anything.
What do you have specified as your callback URL in your Twitter application? It should be https://auth.firebase.com/v2/<YOUR-FIREBASE>/auth/twitter/callback as specified in our docs.

yes. i think its the firebase version.
i used bower to install it and this is the config:
{
  "name": "angularfire",
  "description": "The officially supported AngularJS binding for Firebase",
  "version": "0.9.2",
  "authors": [
    "Firebase<EMAIL_ADDRESS>(https://www.firebase.com/)"
  ],
  "homepage": "https://github.com/firebase/angularfire",
  "repository": {
    "type": "git",
    "url": "https://github.com/firebase/angularfire.git"
  },
  "license": "MIT",
  "keywords": [
    "angular",
    "angularjs",
    "firebase",
    "realtime"
  ],
  "main": "dist/angularfire.js",
  "ignore": [
    "**/.*",
    "src",
    "tests",
    "node_modules",
    "bower_components",
    "firebase.json",
    "package.json",
    "Gruntfile.js",
    "changelog.txt"
  ],
  "dependencies": {
    "angular": "1.2.x || 1.3.x",
    "firebase": "2.1.x"
  },
  "devDependencies": {
    "lodash": "~2.4.1",
    "angular-mocks": "~1.2.18",
    "mockfirebase": "~0.7.0"
  }
}

I think i found the reason why.
https://github.com/firebase/angularfire/releases
the releases was not updated with the latest fix.

What latest fix would that be? I don't see anything in the changelogs that suggests we fixed an error with routing (which we don't control or modify, so there wouldn't be anything to fix)

Yeah I'm not sure what you are pointing out in the release changelogs regarding this issue. Were you able to get the issue resolved? From the command line, if you type bower list, we can see exactly what versions of Firebase and AngularFire you have installed.

I installed it normally using bower install angularfire .The bower.json from my comment is actually the one being installed by the bower command.
anyway, here's the bower list
├── angular#1.3.13 (1.3.14-build.51+sha.5fbac74 available, latest is 1.4.0-build.3841+sha.ef894c8)
├─┬ angularfire#0.9.2
│ ├── angular#1.3.13 (1.3.14-build.51+sha.5fbac74 available, latest is 1.4.0-build.3841+sha.ef894c8)
│ └── firebase#2.1.2 (latest is 2.2.0)
├── firebase#2.1.2 incompatible with ~2.2.0 (2.2.0 available)
└─┬ ionic#1.0.0-beta.14 (latest is 1.0.0-beta.9-nightly-279)
├── angular#1.3.13 incompatible with 1.3.6 (1.3.6 available, latest is 1.4.0-build.3841+sha.ef894c8)
├─┬ angular-animate#1.3.6 (latest is 1.4.0-build.3841+sha.ef894c8)
│ └── angular#1.3.13
├─┬ angular-sanitize#1.3.6 (latest is 1.4.0-build.3841+sha.ef894c8)
│ └── angular#1.3.13
└─┬ angular-ui-router#0.2.13
└── angular#1.3.13 (1.4.0-build.3841+sha.ef894c8 available)

You still haven't explained how you came to the conclusion that AngularFire had anything to do with your missing route.

Closing this due to lack of actionable information. @gianebao, if you can explain more about how you got this error response, I will gladly track down the underlying problem. Until, then, I'm closing this issue.

no problem @jwngr. Thanks!

Getting this same issue.
When I first tried to authenticate w/ twitter, I was using a callback URL for the wrong Firebase instance.
Switched my callback url to the appropriate instance and now getting:
{"error":{"code":"ROUTE_NOT_FOUND","message":"Route not found."}}
When I turn off twitter auth I get the expected 'not allowed' page, so I'm definitely attempting to authenticate w/ the correct instance.
Could this be some sort of caching issue on the Firebase side?

@chinedufn try checking your callback URL in twitter. it should be https://auth.firebase.com/v2//auth/twitter/callback

Can either of you provide the exact URL that you put as your callback URL which caused this issue to occur? I assume the issue is that you put in a URL which pointed to an invalid endpoint. So you just got our generic ROUTE_NOT_FOUND error. To me, that seems like the correct error since we have no easy way to catch that you put in an invalid callback URL (since it is something owned by Twitter, not us). Updating your callback URL to what is mentioned in the previous comment should fix the problem.

As a sanity check.. I copied and pasted my working callback url from Facebook directly over to Twitter, (and changed the 'facebook' portion of URL to 'twitter')
Still no dice.
Even with security rules, I'd prefer not to provide the exact URL here if it can be at all avoided. Here is a copy paste with my firebase name substituted:
https://auth.firebase.com/v2/my-firebase-name/auth/twitter/callback

Also this leads me to think that the url might not be the issue

When I turn off twitter auth for my instance I get the expected 'not allowed' page, so I'm definitely attempting to authenticate w/ the correct instance.

But I'd be happy to email the exact URL

And here is the URL in the OAuth popup window:
https://auth.firebase.com/v2/my-firebase-name/auth/twitter?&v=js-2.2.2&transport=popup&suppress_status_codes=true

Did a bit more poking around and I seem to be having more of a Firebase issue than an AngularFire issue.
For anyone that might be following this, I've posted on Stack Overflow here - http://stackoverflow.com/questions/29101530/firebase-instances-can-be-improperly-spun-up

@chinedufn - Please email me at<EMAIL_ADDRESS>

After investigating this through email and by digging into the Firebase auth code, we have discovered that Twitter is throwing a Could not authenticate you error. This probably means that your Twitter app key and secret are incorrect in your Firebase dashboard. Please double check that they match what you see on twitter.com. If that doesn't fix the issue, maybe Twitter has blocked your app for some reason. You may need to create a new Twitter app.
The 404 not found error is due to us not properly bubbling up the Twitter error from deep within our codebase. We will work on improving this and turn this from a 404 to a 401 or something of the sort, so that it is clear that this is not a Firebase problem, but a problem with accessing the provider. We've added this to our internal tracker.
Thanks for all the help from everyone here in tracking this down!

@jwngr you were right, I let a whitespace char sneak in front of my secret. Fixed now. Thank you

We will fix the whitespace issue as well by stripping whitespace. Hopefully you are the last person who runs into that issue. Thank you!

@jwngr hi! I would like to know if this issue was fixed? ,I have the same problem but I getting this same issue  when I used "firebase-name" but if I use "firebasename" it´s right!

this issue is still open!

This issue has been fixed and is definitely marked as closed. If you are having issues, please open another issue and explain what problem you are having.


Reduce false positives

Without this change, the "Test helm chart (install) action" will
rightfully succeed when our helm chart gets installed and has
no syntax issues. However, it doesn't test if kured is properly
installed. For example, the helm chart can try to install a
yet unpublished image, and our test will succeed, as the syntax
is still valid.
This is a problem, as everything looks green, but it's not
effectively working. Our other jobs are focusing on code changes,
so it effectively overrides the image tag, which is not what
we want in this "Test helm chart" action.
This fixes it by adding an extra command in the test.

See also failure in https://github.com/weaveworks/kured/pull/356

DNM yet. Caught an issue.

I am not confident to merge this yet.

I guess I was too careful. It's rightfully working! :)


utPLSQL suitepath is case-insensitive

in Oddgen (Run test) window, suite paths are shown as duplicated if they are typed with upper/lower case in unit test package spec.
So from those two suitepaths:
--%suitepath(org.utplsql.core)
...
--%SUITEPATH(ORG.UTPLSQL.CORE)

We get separate trees/branches in navigator.
Those should be converted into common (lower) case and displayed as one tree.

Thanks for reporting. I will fix it a.s.a.p.


Cannot use a Duration as a timeout in SyncSession.get()

If I pass e.g Duration.ofMillis(1000) as a timeout to SyncSession.get(), it throws an UnsupportedTemporalTypeException.
It seems to originate from MessageBusSyncSession.SyncSend(), which calls timeout.get(ChronoUnit.MILLIS).
I cannot find any standard TemporalAmount implementation that supports this, is there a special time unit I'm supposed to use?

No, this is indeed a bug. The API is intended to accept Duration instances. Thanks for the report.


Vb deeksha covid 19

Description
I have done pull request for the issue Analysis of Covid_19_data and I have uploaded the file for this issue.
I have been analyzed the data using python libraries like numpy, pandas, seaborn, matplotlib.
Checklist

[] I've read the contribution guidelines.
[] I've checked the issue list before deciding what to submit.
[] I've edited the README.md and link to my code.

Related Issues or Pull Requests
Analysis of Covid_19 #241

Hi, this is an empty PR.


Migrate to ES6

This is a large PR with some opinionated changes, so I'm not sure if it would be accepted, but I'd try anyway :)

Deprecate non-string coercion (major version upgrade): it is difficult to implement this functionality when each function is used separately.
Rewrite validation functions as ES6 modules, so we can now require each function separately without having to include the whole library, this will be helpful to anyone using validator on the client side, and will make it possible in the future to use tree shaking (i.e. removing unused code) when Node supports ES6 modules.

Functions are available on the root folder for convenience Functions are available in the lib/ dir:
// ES5
var isEmail = require('validator/lib/isEmail');

// ES6
import isEmail from 'validator/lib/isEmail';

// ES6 with tree shaking
import { isEmail } from 'validator';

Of course you can still include the entire library as usual:
var validator = require('validator');
validator.isEmail('email@domain.com'); // => true

Now to the opinionated changes:

Move to ESLint: it is extensible, has better support for ES6 and has awesome integration extensions for popular code editors.

Use npm scripts to build, clean and test the package as they are more compatible with non-*nix platforms and easier to use.
npm run clean # removes all builds
npm run clean:browser
npm run clean:node
npm run build
npm run build:browser
npm run build:node
npm run lint # runs ESLint
npm run test

Thanks for the PR! :sparkles: This will obviously take some time to review but I'm liking everything so far.
I'm happy to move from jshint to eslint, and also happy to drop the makefile in favour of npm scripts.
Dropping coercion of non-string input is already planned for v5, although I'd like for the deprecation notice to be out there in the wild for a bit longer, as there are undoubtedly projects and libraries that depend on the latest version of the library rather than ^4

Great!
I've add a new function assertString that is called by every validation function and currently throws an error if the input is not a string, toString is still available but it should be called manually. We could print the deprecation notice in assertString and throw the error, or even restore the coercion behavior by simply calling toString inside assertString.

Just wanted to point out that accessing individual functions using validator/<functionName> is not currently implemented, instead one has to use validator/lib/<functionName>. I'm trying to figure out the best way to do it without polluting the root dir with too many files. We may need to add tests for these files too.

I'm trying to figure out the best way to do it without polluting the root dir with too many files

Try export-files, set just one index.js in lib :)

The point of the deprecation notice was that it still coerced input but warned when it did so, giving you time to update your code. Printing a deprecation notice and then throwing an error defeats the purpose.
I'm happy for you to drop the deprecation notice entirely. Once this PR Is merged I'll tag and publish v5.

I've removed depd and all related babel plugins and env checks.

Hmm I just realised extend (docs) probably won't work in its current state.
The alternatives are to replace validator[name] with this[name] or to just drop it entirely. I was never really happy with modifying the global validator object anyway. The only reason it was there was to automatically coerce the first argument to a string, which will be going away in v5 anyway. Thoughts?

@chriso I do not see any use for extend now that the validator object is just a collection of individual functions, additional validation functions can be imported from other libraries or just created manually.
The only case for extend is wrapping new functions with assertString:
const validator = {
  isEmail, isMobilePhone,
  toString, // ...
  extend(name, fn) {
    this[name] = (input) => {
      assertString(input);
      return fn(input);
    }
  }
}

Would something like this work?

@chriso Here's a quick example https://jsbin.com/pecojit/edit?js,console

On second thoughts, let's just kill extend. Can you remove the existing function and also the README entry?

It was not implemented in the new code. I just removed it from README.md.

:+1:
Is everything ready from your end? I can do another review pass.

I would like to upgrade to ESLint 2.0 but eslint-config-airbnb is not compatible yet with v2 so this can wait. Also, test code need to be adapted to the new code style, but that will take sometime.
Everything else looks good to me.

@chirso Tests are linted now. I've also added:

isMobilePhone validation for ar-SY (Arabic (Syria))
isAlpha and isAlphanumeric validation for various Arabic locales.

@chriso upgraded to ESLint 2 :+1:

:+1: we'll need to merge in the changes from master in the past week and then I'll look at merging this PR.

Do you want me to do that?

If you don't mind. It won't let me merge otherwise.

@chriso Done!

Apart from the README changes, this is ready to merge ✨

v5 will land shortly.

Awesome! :smile:
Thank you!

5.0.0 has been published. Thanks for the PR!

I dont understand why it does not work for Ember by default? Something like:
import validator from 'bower_components/validator-js';
I had to do this
bower install validator-js --save-dev

Then inside ember-cli-build.js, add  this line of code app.import('bower_components/validator-js/validator.js');
@chriso Thanks for the help.


Not Connecting, No Web API entry

Hello,
Working through making this connection. All firmware is up to date.
When I open the connector app for the first time I am not prompted to enter an API key.
I am able to connect the LM to my phone, authenticate Awesome Golf, then close the Rapsodo app on the phone to get back to solid red light; however the connecter app only shows Connected to GSPro and looking for device in the LM panel.
I am making the phone-lm connection via direct wifi, not through my home network
Please let me know your thoughts

same issue here

the newest update no longer requires you to enter the API key

oh also. for my connector you need to have the device paired with windows. is it paired?
i think the newer firmware update requires you to unpair and repair it with windows

it is not connected to windows. It does not appear in the bluetooth window


[WIP] Modifications after extensively iterations on Virtual Machine

Hey there :wave:
The overall code was almost ready to use, it wasn't necessary to tweak the modules, only the module configuration and the cohort analysis workflow.
Very interestingly, the parallel run wasn't very optimized so it took the same time as normal run.
Tasks done:

Add specific processes names to modules.config
Modify genome names handling on cohort workflow

Tasks in progress:

Generate a new overflow:

MTBSEQ_NF:QC:FASTQC
MTBSEQ_NF:REPORT:MULTIQC
MTBSEQ_NF:NORMAL_MODE:TBFULL
MTBSEQ_NF:PARALLEL_MODE:TBBWA
MTBSEQ_NF:REPORT:MULTIQC

change medium and high labels to match how the process are run

Thanks @Mxrcon , please do let me know when it is ready to be reviewed and I'll take a deeper look :)


Kahoot style session management of hubs

Work started here https://github.com/Tejpbit/talarlista/tree/feature/kahoot-style-joining


Error parsing Capability statement

Not sure exactly how to describe this one, but I hope the image gives you enough info.

The response being parsed is:
{
  "resourceType": "CapabilityStatement",
  "status": "active",
  "date": "2024-03-14T13:31:55+00:00",
  "publisher": "Boston Children's Hospital",
  "kind": "instance",
  "instantiates": [
    "http://hl7.org/fhir/uv/bulkdata/CapabilityStatement/bulk-data"
  ],
  "software": {
    "name": "SMART Sample Bulk Data Server",
    "version": "2.1.1"
  },
  "implementation": {
    "description": "SMART Sample Bulk Data Server"
  },
  "fhirVersion": "3.0.2",
  "acceptUnknown": "extensions",
  "format": [
    "json"
  ],
  "rest": [
    {
      "mode": "server",
      "security": {
        "extension": [
          {
            "url": "http://fhir-registry.smarthealthit.org/StructureDefinition/oauth-uris",
            "extension": [
              {
                "url": "token",
                "valueUri": "https://bulk-data.smarthealthit.org/auth/token"
              },
              {
                "url": "register",
                "valueUri": "https://bulk-data.smarthealthit.org/auth/register"
              }
            ]
          }
        ],
        "service": [
          {
            "coding": [
              {
                "system": "http://hl7.org/fhir/restful-security-service",
                "code": "SMART-on-FHIR",
                "display": "SMART-on-FHIR"
              }
            ],
            "text": "OAuth2 using SMART-on-FHIR profile (see http://docs.smarthealthit.org)"
          }
        ]
      },
      "resource": [
        {
          "type": "Patient",
          "operation": [
            {
              "extension": [
                {
                  "url": "http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation",
                  "valueCode": "SHOULD"
                }
              ],
              "name": "export",
              "definition": {
                "reference": "http://hl7.org/fhir/uv/bulkdata/OperationDefinition/patient-export"
              }
            }
          ]
        },
        {
          "type": "Group",
          "operation": [
            {
              "extension": [
                {
                  "url": "http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation",
                  "valueCode": "SHOULD"
                }
              ],
              "name": "export",
              "definition": {
                "reference": "http://hl7.org/fhir/uv/bulkdata/OperationDefinition/group-export"
              }
            }
          ]
        },
        {
          "type": "OperationDefinition",
          "profile": {
            "reference": "http://hl7.org/fhir/Profile/OperationDefinition"
          },
          "interaction": [
            {
              "code": "read"
            }
          ],
          "searchParam": []
        }
      ],
      "operation": [
        {
          "name": "get-resource-counts",
          "definition": {
            "reference": "OperationDefinition/-s-get-resource-counts"
          }
        },
        {
          "extension": [
            {
              "url": "http://hl7.org/fhir/StructureDefinition/capabilitystatement-expectation",
              "valueCode": "SHOULD"
            }
          ],
          "name": "export",
          "definition": {
            "reference": "http://hl7.org/fhir/uv/bulkdata/OperationDefinition/export"
          }
        }
      ]
    }
  ]
}

To reproduce, run it with the SMART Server preset.
P.S. I am making some changes to the Bulk Data Reference Server which may require you to update the preset:

Fhir base url with optional auth: https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1IjozLCJkZWwiOjAsInNlY3VyZSI6MH0/fhir
Fhir base url with required auth (what you want): https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1IjozLCJkZWwiOjAsInNlY3VyZSI6MX0/fhir
Group ID: 5033ae08-fb5b-40fb-ab9a-23674bc1b42b. This is 3 patients only, therefore good for testing. However, group IDs might change if I re-generate the patient data, in which case I'll let you know.

Hmm, strange.   This is breaking because our code is expecting a string for operation.definition, not an object with a reference key.   I just ran it, and it passed because it is a string as our test expects.   We'll take a closer look. Regardless, you should never see those 'purple runtime errors' and our tests need to be more tolerant to unexpected content in responses... so we'll clean that up.
Regarding groups, it would be nice if the group ids were static and printed in the interface, as it is something you need to precoordinate with clients so would be helpful for them to have, right?  No problem if not though, just an idea.
And while I'm just looking at the capability statement, looks like you have capabilitystatement-expectations still in there, but those don't make a lot of sense for 'instance' kind CapabilityStatements.  They are more for kind=requirements CapabilityStatements (as you would find provided in an IG) I believe.
Thanks for the feedback, much appreciated.

operation.definition is a canonical which is basically a string.

Right, I seem to recall pre-R4 FHIR was a reference and not a canonical, and your server supports pre-r4, so seems like maybe we are running tests expecting R4 but are configured to run against a DSTU2 or STU3 simulated endpoint.

Ups, yes I somehow ran it with wrong server URL pointing to the R3 version of the server, where that is a reference. Seems to work fine with the right R4 URL (https://bulk-data.smarthealthit.org/eyJlcnIiOiIiLCJwYWdlIjoxMDAwMCwiZHVyIjoxMCwidGx0IjoxNSwibSI6MSwic3R1Ijo0LCJkZWwiOjAsInNlY3VyZSI6MX0/fhir/metadata). Perhaps you can still add some protection against incorrect input, but otherwise feel free to close this. Thanks!

Regarding groups, it would be nice if the group ids were static and printed in the interface, as it is something you need to communicate with clients out-of-band (of the bulk data spec) so would be helpful for them to have, right? No problem if not though, just an idea.

Very good idea! Groups are now listed on the home page https://bulk-data.smarthealthit.org/index.html. Thanks!


Ignore PRs with label kind/cherry-pick when --all is used

These PRs start with "Automated cherry pick of" are already included by default with original PRs. They should be ignored when --all is passed.

These PRs start with "Automated cherry pick of" are already included by default with original PRs. They should be ignored when --all is passed.

I didn't get it. In which situation a cherry-pick PR and its original PR could be in the same release?

Hi @tnqn , I got below data locally when I tried to run ./hack/release/prepare-changelog.sh --release 1.12.1 --all. I didn't recall that these PRs started with Automated cherry pick were included before. Not sure if there is a github API change.
### Changed

- [ci][flexible-ipam] Use k8s 1.27.2 and containerd for e2e. ([#5116](https://github.com/antrea-io/antrea/pull/5116), [@gran-vmv])
- Fix incorrect FlowMod message passing in openflow client modifyFlows. ([#5125](https://github.com/antrea-io/antrea/pull/5125), [@Dyanngg])
- Fix Egress IP not advertised in some cases . ([#5127](https://github.com/antrea-io/antrea/pull/5127), [@tnqn])
...
...
### Unlabelled (Remove this section eventually)

- Fix ClickHouse service not found in FA e2e test. ([#5148](https://github.com/antrea-io/antrea/pull/5148), [@wsquan171])
- Automated cherry pick of #5116: Use k8s 1.27.2 and containerd for e2e. ([#5151](https://github.com/antrea-io/antrea/pull/5151), [@luolanzone])
- Automated cherry pick of #5125: Fix incorrect FlowMod message passing in openflow client. ([#5145](https://github.com/antrea-io/antrea/pull/5145), [@luolanzone])
- Automated cherry pick of #5127: Fix Egress IP not advertised in some cases. ([#5141](https://github.com/antrea-io/antrea/pull/5141), [@tnqn])
...

I see. It's because https://github.com/antrea-io/antrea/pull/3789 added a step to search cherry-pick PRs and include their original PRs to "Changed" section. Then it makes sense to exclude them in "Unlabelled".

/skip-all


Add gunicorn to requirements.txt for PaaS deploys

Add gunicorn to requirements.txt for running the app on Cloud Foundry
(possibly not ideal see [1]).
Explicitly add cffi to requirements so that Cloud Foundry spots it and
adds libffi.
There doesn't seem to be an easy way to use a different requirements
file for Cloud Foundry, so we'll have to add the dependencies
everywhere.
[1] https://github.com/etianen/django-herokuapp/issues/9

When you say "not ideal" - am I right in thinking this will only be an issue for when apps are deployed to PaaS?  The new dependencies won't have any affect when running in AWS, will they?

@TheDoubleK yeah, the issue is about gunicorn vs alternatives when running on PaaS. The dependencies aren't used when running in AWS, but they do get installed, which might have an effect on deployments. Ideally, there'd be an easy way to have a different requirements file for PaaS, but I couldn't think of one that wouldn't involve writing wrapper code.

Yeah it's a bit weird but like you say, no easy way round it.
:+1:


Typos and phrasing in quickstart.md

Ref: https://github.com/openjournals/joss-reviews/issues/6915


Appstudio update rhtap-demo-component-yhvi

Pipelines as Code configuration proposal
To start the PipelineRun, add a new comment with content /ok-to-test
For more detailed information about running a PipelineRun, please refer to Pipelines as Code documentation Running the PipelineRun
To customize the proposed PipelineRuns after merge, please refer to Build Pipeline customization

Pipelines as Code CI has failed.There was an error creating the PipelineRun: rhtap-demo-component-yhvi-on-pull-request-
creating pipelinerun rhtap-demo-component-yhvi-on-pull-request- in namespace rhtap-demo-dev-sjpy-tenant has failed.
Tekton Controller has reported this error: the namespace of the provided object does not match the namespace sent on the request


Add CMakeList.txt support

Atom Beautify could not determine a supported beautifier to handle this file with grammar "CMake Listfile" and extension "txt". If you would like to request support for this file and its language, please create an issue for Atom Beautify at https://github.com/Glavin001/atom-beautify/issues

Hi! There is a nice formater here: https://github.com/cheshirekow/cmake_format
Thanks!

cmake-format works pretty well for me (mostly).
#554 discusses this, so I'll link this issue in that issue.