Incorrect usage of StagingArea.

Hey Alex!
Awesome that you're using the StagingArea. However, the usage of it is incorrect, and you won't see any speedup.
You're correctly aggregating a list of the staging area ops (you call enqueue ops). Those enqueue ops you'll eventually pass to the run function in your monkey-patched back-end.
What's happening now is, is that you're not pre-queueing anything. That means that if you run:
run([enqueue_ops, your_model])
Is that your_model will try to fetch from the staging_area, which is empty, and it will block. It will block, until your enqueue_ops is actually done putting things in the StagingArea. Once something is in there, your_model will be unblocked, and get from your StagingArea. You see, this is no different (or actually even more inefficient!) than without a StagingArea.
The intended way of using it, is having a list of enqueue_ops. And you should pre-stage n samples. I don't think you're prestaging anything now, which means you don't have a buffer, which means you essentially don't really have any form of queue at all, and won't benefit.
Notice that often you want to use a cascade of queues (StagingAreas) and that means you'll need to do something like this:
for i, op in enumerate(enqueue_ops):
    sess.run(enqueue_ops[:i])

The above loop will first enqueue on the first staging area, then the first+second, then the first+second+third, etc, unwill all your staging area cascade is filled. Then you can run the full enqueue_ops function n times, where n is your buffer size.
What complicates the above, is that if you want to empty your queues (and not use a continuous feed), then for the last few batches, you need to _stop+ using the enqueueing of (1) stop fetchign the first queue, (2) stop fetching the first+second queue (3) etc.
Monkey patching of the backend
We could consider making a Keras MR that allows some kwargs with extra ops to be passed to the backend Function so that you don't need to monkey patch it. The way you're currently doing it (monkey patching) is indeed the only way.
Lastly, multi-gpu and multi-node won't help mnist of cifar probably. It doesn't matter at all, because those aren't real use cases anyway. In more practical use-cases (imagenet, automotive, etc) the models are deeper and the input data larger, and that's where your keras experiment will really shine. I'll see if I can help you there.

Don't want to change the subject, but

In more practical use-cases (imagenet, automotive, etc) the models are deeper and the input data larger, and that's where your keras experiment will really shine.

Yes, indeed! I'm using the multi-gpu code with a WRN 16-4 and 3D data (13x18x50) and using 4 GTX1080 in a single node instead of one scales x4 :) Thanks a lot for your efforts @avolkov1

@TimZaman I see. Yea, I was not really sure what I was doing with the StagingArea. I put it there so that hopefully someone might fix it or tell me what I'm doing wrong. Thank you for that. I would greatly appreciate it if you helped me in getting that to work properly. I'm not sure how to implement the prestaging. There are very few examples and documentation is lacking.
With regards to mnist and cifar examples, I know those are silly and don't scale, but I'm using them purely for illustrating implementation details. Cifar scales a little bit and the reason I use Cifar10 is because the dataset for it is small and easy to download. Everyone is familiar with Cifar10.
I would really appreciate any help on porting this to imagenet with ResNet50 especially on our internal psgcluster (refer to this repo on yagr). I haven't had a chance to implement the ResNet50 benchmark which is what I really need to compare with native Tensorflow approach and perflab benchmarks. My goal is to have on par performance for ResNet50 with native Tensorflow by using Keras with tfrecords approach and this simple multigpu parallelization. I also want it to scale when I run distributed.
When I run Tensorflow with RDMA on psgcluster my current distributed implementation is not too bad given the very simple approach, but it can be much better I believe.

@ViaFerrata Awesome, I'm glad it's helping. Are you using fit_tfrecords approach, or fit with feed_dict?

@avolkov1 I'm currently running your code with imagnet+resnet18 and get 3x speedup with 4x titan X. I'll see if the stagingarea's prequeueing can bump this up. 3x is already pretty fair.

@avolkov1 I'm using the Keras fit_generator function with hdf5 files, so feed_dict. Currently, I'm waiting for https://github.com/fchollet/keras/pull/6928, since the data feeding could be made (a lot?) faster with native framework input. Quite a shame that TF graph rewiring is needed for that :/
I think getting a super fast data pipe to work (TF-like performance) should be of high priority for Keras, especially for people that use multi-gpu or distributed training. Unfortunately, I can't really contribute to that.

@TimZaman Any progress. I would really like to see the resnet implementation and if you get the stagingarea working that would be awesome.
I will push an example today with distributed (on SLURM) training implementation where I achieved really good scaling and performance. 2 nodes each with 4 GPUs, reduced the training epoch to 2 sec./epoch on Cifar10 down from 12 sec./epoch for 1 GPU. I know, the Cifar10 example is a toy example, but I think the distribution will scale properly with more stressing examples.

Yeah I even got 3.6x scaling now. I hit a big snag with the stagingarea. I realized that both the label and the image need to be in the same queue (make sure things stay connected deterministically.
It makes no sense to have just a queue for one, and not for the other. That'll result is very nasty issues and data/label discrepancies.
How can we make sure we somehow enqueue the labels too? Sure, we can enqueue both at the same time before we declare the tensors into keras.layers.Input(tensor=..), but that would put the stagingarea/queue on the CPU. Although surely that should give a benefit, we want one per GPU, resident on the GPUs that'll use the data.
Not sure yet how to proceed there.

@ViaFerrata if you are looking for fast data reading pipeline, dataflow in tensorpack could be of interest. Although written in pure Python it can read ImageNet data fast enough to keep multigpu busy. A common misunderstanding is that native TFRecord make things faster -- not really, if your python side is already fast enough.
I understand people may prefer keras-style APIs or hand-crafted training loops, that's why dataflow is written for use outside tensorpack as well. I noticed this repo actually contains some code I wrote there ... 😄

@ppwwyyxx Hey, yes, I took the dataflow portion out of tensorpack that you wrote. The repo here was meant for me to experiment with and share. I have an implementation using dataflow:
https://github.com/avolkov1/keras_experiments/blob/master/examples/cifar/cifar10_cnn_mgpu_dflow.py
I did not see an immediate speedup. Looking at tensorpack code it was difficult to figure out how the code works for QueueInputTrainer, QueueInput, and EnqueueThread. I would love to understand it better. My implementation is here:
https://github.com/avolkov1/keras_experiments/blob/master/keras_exp/mixin_models/dataflow.py
Any hints you might have I would appreciate it.

@ppwwyyxx Depends on how much you care about speed. If you want good performance in TF, you need queues natively inside TF, and preferably queues that are already on the GPU. The latter constraint leaves you only with a StagingArea. The TF's Queue(Runners) to will also be deprecated in TF 2.0 because their perf is not satisfactory.
If you really care about that last +10~20% speed, a tf-native queue is a must. Then still you can use tensorpack, but that should then push into the queue natively, hence, you'll need tensors as inputs in Keras anyway.

@avolkov1 The problem is, for dataset as small as cifar, the bottleneck is probably not in Python-side data loading -- in my experience hiding the latency with queues will make more visible differences in the overall speed. But for dataset such as ImageNet, loading them in Python-side is already an issue, (and you'll also need queues and stagingareas).
@TimZaman I understand the correction, that's why it's a misunderstanding.
I care about speed very much. But using native queues and StagingAreas doesn't mean I have to use tfrecord, as long as Python-side data loader is fast enough -- that's the point I want to make. tensorpack has included queues and StagingAreas already, btw.
I'm not saying dataflow alone will make training faster. tensorpack.dataflow is a fast Python loader. You can use it with tensorpack to connect to native queues/stagingareas as well, you can also use it in other places (e.g. keras) to simply load data and connect to other input pipeline, then it's hard to say whether there will be an improvement or not.

@avolkov1:

I would really appreciate any help on porting this to imagenet with ResNet50

I tried to apply your multi-gpu code to Resnet50 and InceptionV3 from keras.applications with synthetic imagenet dataset:
https://github.com/rossumai/keras-multi-gpu/blob/master/keras_tf_multigpu/examples/benchmark_inception3_resnet50.py
On Azure 2xTesla M60:
For Resnet50 with PS=CPU 1.72x speed-up on 2 GPUs, with PS=GPU 1.65x speed-up, for InceptionV3 the numbers are almost the same.

@bzamecnik Thank you. Great stuff on your blog. I didn't know that Chollet added the GPU parallelization to the Keras utils. I don't know why he didn't fix the part with slicing on CPU. Maybe my repo here might become obsolete soon.
I'm going to move the enqueue/StagingArea stuff and monkey patching hack into a dev branch to clean up the code.

Thanks. The code evolves so quickly... So far fchollet's version is just in master, not released yet. Yeah, I'm also missing the CPU slicing and also correct saving/loading. Maybe a space for another PR.
So far I'm trying to summarize the measurements. Then I'll try to measure new packages like horovod and tensorpack. But anyway we'd like to have a clean Keras code without other dependencies and with nice API.
So far the result is that double-buffering at GPU with StagingArea is the key. I'll try to make some experiments with that as well. It seem that it need only queue of size 2 for one GPU - one input batch being copied while another is being consumed by computation. The only serial part would then be only gradient and weight exchange.

I tried to fiddling with StagingArea and Keras. To summarize: For at least double-buffering at GPU we need a queue of size 2. Ie. first we need to put(), then at each batch within model.fit() we need both put() and get() within the same session.run(), at the last batch we need to only get() or possibly put() a dummy value.
I tried to provide values for StagingArea via a placeholder fed with feed_dict (for simplicity). It turns out that in Keras it's possible to combine a tensor input and feed_dict input! Thus we can use tensors StagingArea.get() as main input and target_tensors labels, StagingArea.put() as additional operation to fetches argument and values for next batch as x argument to model.fit(). Note that features and labels are represented as a list of tensors and are transferred together in single get/put call, not separately.
Whole rudimentary code snippet: https://gist.github.com/bzamecnik/54104644baddc614230102e65f02b593
Problems to overcome:

the API usage is very hacky, really not nice (the usage of pipelining should be encapsulated)
we can call model.fit() on single batch (steps_per_epoch=1) since Keras doesn't automatically slice the batches in the tensor input mode
since the session.run() is wired once in compile we should provide a dummy value for the last batch
we need to switch between training and validation sets if validation set is provided to model.fit()
it needs patch of K.function() (I rewrote the monkey patch to a Keras fork that will go in a PR): https://github.com/bzamecnik/keras/commit/8593309c371ce716fd039e33ed5ae4079096ee0f

TF queues API might help in cleaner API - we only add a few more ops to fetches and run Python->TF transfer in another thread. But it's deprecated. Newer Dataset API in TF 1.3 doesn't support Python memory as data source, except for baking numpy arrays as constant (no-go). In TF 1.4 we have in addition Dataset.from_generator() which seems to be a viable alternative, but TF 1.4 is not out yet.

tf 1.4 is already in rc. If you want to use stagingarea just.

make sure your model input is from the stagingarea's .get()

@bzamecnik  @TimZaman
If you can illustrate with an example and/or PR how I could enable this for multigpu via my mini-framework here I would really appreciate that. Thank you.

If you're interested in multi-gpu or multi-node with keras, just use horovod, it has the best scaling by far: https://github.com/uber/horovod

Another proof-of-concept step: https://gist.github.com/bzamecnik/3c2b5279a5949d694421d7cfbe813557
Inputs to StagingArea provided via Variable assignment which can be decoupled from Keras' Session.run(). The pipelining logic is encapsulated in a callback. It's able to slice an input numpy array to batches.
Still it relies on feed_dict, but we don't have to hack with Keras inputs. It still needs to take care of validation set/prediction etc. And it also assumes one feature input and one label output.

@TimZaman Yeah, I'd like to try. So far I didn't find time for that. Their results look impressive.

I installed and played around with horovod. I like their synchronous optimizer and their simplification of Tensorflow API is nice, but if one can get away without synchronizing/averaging gradients, then it's not speeding up Keras training. Here are the results I got for 2 GPUs mnist example that they posted (I split the data set of 60,000 samples into two data sets of 30,000 samples processed on each GPU):
https://github.com/uber/horovod/blob/master/examples/keras_mnist.py
# my code modification
nsamples = x_train.shape[0]
mysamples = nsamples // hvd.size()
start_sam = hvd.local_rank() * mysamples
stop_sam = min((hvd.local_rank() + 1) * mysamples, nsamples)
x_train = x_train[start_sam:stop_sam, ...]
y_train = y_train[start_sam:stop_sam, ...]

# mpirun -np 2 python keras_mnist.py
Train on 30000 samples, validate on 10000 samples
Train on 30000 samples, validate on 10000 samples
Epoch 1/6
30000/30000 [==============================] - 10s - loss: 0.3384 - acc: 0.8965 - val_loss: 0.0729 - val_acc: 0.9769
Epoch 1/6
30000/30000 [==============================] - 10s - loss: 0.3381 - acc: 0.8974 - val_loss: 0.0729 - val_acc: 0.9769
Epoch 2/6
30000/30000 [==============================] - 6s - loss: 0.1007 - acc: 0.9697 - val_loss: 0.0468 - val_acc: 0.9834
Epoch 2/6
30000/30000 [==============================] - 6s - loss: 0.1039 - acc: 0.9690 - val_loss: 0.0468 - val_acc: 0.9834
Epoch 3/6
30000/30000 [==============================] - 6s - loss: 0.0747 - acc: 0.9787 - val_loss: 0.0372 - val_acc: 0.9876
Epoch 3/6
30000/30000 [==============================] - 6s - loss: 0.0749 - acc: 0.9769 - val_loss: 0.0372 - val_acc: 0.9876
Epoch 4/6
30000/30000 [==============================] - 6s - loss: 0.0604 - acc: 0.9816 - val_loss: 0.0318 - val_acc: 0.9889
Epoch 4/6
30000/30000 [==============================] - 6s - loss: 0.0672 - acc: 0.9801 - val_loss: 0.0318 - val_acc: 0.9889
Epoch 5/6
30000/30000 [==============================] - 6s - loss: 0.0504 - acc: 0.9853 - val_loss: 0.0363 - val_acc: 0.9872
Epoch 4/6
30000/30000 [==============================] - 6s - loss: 0.0549 - acc: 0.9829 - val_loss: 0.0363 - val_acc: 0.9872
Epoch 6/6
30000/30000 [==============================] - 6s - loss: 0.0434 - acc: 0.9860 - val_loss: 0.0308 - val_acc: 0.9897
Epoch 6/6
30000/30000 [==============================] - 6s - loss: 0.0478 - acc: 0.9853 - val_loss: 0.0308 - val_acc: 0.9897
Test loss: 0.0308260866315
Test accuracy: 0.9897
Test loss: 0.0308260866315
Test accuracy: 0.9897

Compare to mnist_cnn.py that I prallelized on 2 GPUs using asynchronous training:
# CUDA_VISIBLE_DEVICES=0,1 python mnist_cnn_mgpu.py
Train on 60000 samples, validate on 10000 samples
Epoch 1/6
60000/60000 [==============================] - 7s - loss: 0.3254 - acc: 0.9017 - val_loss: 0.0725 - val_acc: 0.9764
Epoch 2/6
60000/60000 [==============================] - 3s - loss: 0.1113 - acc: 0.9672 - val_loss: 0.0509 - val_acc: 0.9827
Epoch 3/6
60000/60000 [==============================] - 3s - loss: 0.0826 - acc: 0.9753 - val_loss: 0.0419 - val_acc: 0.9852
Epoch 4/6
60000/60000 [==============================] - 3s - loss: 0.0712 - acc: 0.9784 - val_loss: 0.0365 - val_acc: 0.9874
Epoch 5/6
60000/60000 [==============================] - 3s - loss: 0.0622 - acc: 0.9812 - val_loss: 0.0337 - val_acc: 0.9884
Epoch 6/6
60000/60000 [==============================] - 4s - loss: 0.0544 - acc: 0.9831 - val_loss: 0.0326 - val_acc: 0.9887
Test loss: 0.0326202042509
Test accuracy: 0.9887

For single GPU each epoch runs in ~5 sec. on P40s that I am testing on. Asynchronous training doesn't work in all cases, so in those cases horovod should be very valuable.
I'm curious to compile Tensorflow with NCCL2 and perform gradient synchronization without MPI hooks. For distributed I get pretty good results using GRPC with RDMA (protocol='grpc+verbs' and trying to get `protocol='grpc+gdr' in tf1.4 working which should improve distributed training even more).

I don’t see a real-world usecase for multigpu/node for networks as shallow
as LeNet and MNIST datasets. Try resnet18 and up/inception/vgg and you will
get amazing numbers. 7x scaling with 8 GPUs is the last I’ve seen with
resnet50 equivalent complexity
On Thu, 26 Oct 2017 at 15:49 avolkov1<EMAIL_ADDRESS>wrote:

I installed and played around with horovod. I like their synchronous
optimizer and their simplification of Tensorflow API is nice, but if one
can get away without synchronizing/averaging gradients, then it's not
speeding up Keras training. Here are the results I got for 2 GPUs mnist
example that they posted (I split the data set of 60,000 samples into two
data sets of 30,000 samples processed on each GPU):
https://github.com/uber/horovod/blob/master/examples/keras_mnist.py
my code modification
nsamples = x_train.shape[0]
mysamples = nsamples // hvd.size()
start_sam = hvd.local_rank() * mysamples
stop_sam = min((hvd.local_rank() + 1) * mysamples, nsamples)
x_train = x_train[start_sam:stop_sam, ...]
y_train = y_train[start_sam:stop_sam, ...]
mpirun -np 2 python keras_mnist.py
Train on 30000 samples, validate on 10000 samples
Train on 30000 samples, validate on 10000 samples
Epoch 1/6
30000/30000 [==============================] - 10s - loss: 0.3384 - acc: 0.8965 - val_loss: 0.0729 - val_acc: 0.9769
Epoch 1/6
30000/30000 [==============================] - 10s - loss: 0.3381 - acc: 0.8974 - val_loss: 0.0729 - val_acc: 0.9769
Epoch 2/6
30000/30000 [==============================] - 6s - loss: 0.1007 - acc: 0.9697 - val_loss: 0.0468 - val_acc: 0.9834
Epoch 2/6
30000/30000 [==============================] - 6s - loss: 0.1039 - acc: 0.9690 - val_loss: 0.0468 - val_acc: 0.9834
Epoch 3/6
30000/30000 [==============================] - 6s - loss: 0.0747 - acc: 0.9787 - val_loss: 0.0372 - val_acc: 0.9876
Epoch 3/6
30000/30000 [==============================] - 6s - loss: 0.0749 - acc: 0.9769 - val_loss: 0.0372 - val_acc: 0.9876
Epoch 4/6
30000/30000 [==============================] - 6s - loss: 0.0604 - acc: 0.9816 - val_loss: 0.0318 - val_acc: 0.9889
Epoch 4/6
30000/30000 [==============================] - 6s - loss: 0.0672 - acc: 0.9801 - val_loss: 0.0318 - val_acc: 0.9889
Epoch 5/6
30000/30000 [==============================] - 6s - loss: 0.0504 - acc: 0.9853 - val_loss: 0.0363 - val_acc: 0.9872
Epoch 4/6
30000/30000 [==============================] - 6s - loss: 0.0549 - acc: 0.9829 - val_loss: 0.0363 - val_acc: 0.9872
Epoch 6/6
30000/30000 [==============================] - 6s - loss: 0.0434 - acc: 0.9860 - val_loss: 0.0308 - val_acc: 0.9897
Epoch 6/6
30000/30000 [==============================] - 6s - loss: 0.0478 - acc: 0.9853 - val_loss: 0.0308 - val_acc: 0.9897
Test loss: 0.0308260866315
Test accuracy: 0.9897
Test loss: 0.0308260866315
Test accuracy: 0.9897
Compare to mnist_cnn.py
https://github.com/fchollet/keras/blob/master/examples/mnist_cnn.py
that I prallelized on 2 GPUs using asynchronous training:
CUDA_VISIBLE_DEVICES=0,1 python mnist_cnn_mgpu.py
Train on 60000 samples, validate on 10000 samples
Epoch 1/6
60000/60000 [==============================] - 7s - loss: 0.3254 - acc: 0.9017 - val_loss: 0.0725 - val_acc: 0.9764
Epoch 2/6
60000/60000 [==============================] - 3s - loss: 0.1113 - acc: 0.9672 - val_loss: 0.0509 - val_acc: 0.9827
Epoch 3/6
60000/60000 [==============================] - 3s - loss: 0.0826 - acc: 0.9753 - val_loss: 0.0419 - val_acc: 0.9852
Epoch 4/6
60000/60000 [==============================] - 3s - loss: 0.0712 - acc: 0.9784 - val_loss: 0.0365 - val_acc: 0.9874
Epoch 5/6
60000/60000 [==============================] - 3s - loss: 0.0622 - acc: 0.9812 - val_loss: 0.0337 - val_acc: 0.9884
Epoch 6/6
60000/60000 [==============================] - 4s - loss: 0.0544 - acc: 0.9831 - val_loss: 0.0326 - val_acc: 0.9887
Test loss: 0.0326202042509
Test accuracy: 0.9887
For single GPU each epoch runs in ~5 sec. on P40s that I am testing on.
Asynchronous training doesn't work in all cases, so in those cases horovod
should be very valuable.
I'm curious to compile Tensorflow with NCCL2 and perform gradient
synchronization without MPI hooks. For distributed I get pretty good
results using GRPC with RDMA (protocol='grpc+verbs' and trying to get
`protocol='grpc+gdr' in tf1.4 working which should improve distributed
training even more).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avolkov1/keras_experiments/issues/2#issuecomment-339823025,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHXSRFsBsN522H-gD8A3SsPMi9ApMuI0ks5swQyVgaJpZM4OoPGl
.

I'll test horovod with resnet (trust but verify). Their metrics are based on images/sec., but each epoch is actually slower. If the walltime to convergence is faster, then that's what matters. In more complicated networks such as the variational autoencoder examples, LSTM/RNNs (ptb lstm example based on Keras), the asynchronous training sped up the training, but did not work as far as convergence. I briefly tested horovod on those examples and it worked, but did not speed up the training (single GPU performance was faster and achieved the same convergence rate or even faster).
The asynchronous training I have implemented speeds up epochs or iterations, but I have observed that the async approach sometimes messes with loss and accuracy metrics, and converges slowly or doesn't converge at all. Hyperparameter adjustments with async training are more complicated. I also tried implementing the synchronous gradient averaging (OptimizerMultiGPUMixin), but I don't think it's working correctly (Note: sync doesn't seem to work right even with the fix to issue #12 for compatibility with Keras 2.0.8). I verified that the horovod gradient averaging is definitely working as opposed to my attempt at it.

Hi, I made some experiments to figure out behavior of StagingArea. The results are cheering :)

it supports variable batch size, while enforcing data sample shape
it's possible to to pipelining and join get() and put() into a single operation with tf.tuple()

this removed the need for patching K.function() since put() operation is hidden in the input tensor
but I'm not sure whether this would allow memcpy parallel to computation, most probably not

Also I tried to run make_parallel on 4xM60 (on Azure NV24) and got nice results:
|speedup|inception3|cifar10 convnet|
|tf_cnn_benchmarks|3.86x|3.54x|
|avolkov1|3.04x|2.57x|

speedup on 4x M60 wrt. 1x M60
cifar10 with batch size 512
inception3 with batch size 32
PS=CPU

Speedup for Keras is significant, but still there's room for improvement.

There is no need to use tf.tuple with control_inputs, which adds an unnecessary op dependency: control_inputs contains additional ops that have to finish before this op finishes, but whose outputs are not returned..
You can just tf.group the get and put op.

@ppwwyyxx Yeah, true, that I don't want that dependency. But unfortunately, tf.group doesn't return any value (in this case from get()): "This op has no output.". I checked this behavior.

You can still use get for the training, and use tf.group to group get, put as well as the training op, just to make sure put will be run.

@ppwwyyxx Unfortunately we can't get value from StaginArea.get() through tf.group(). Check this example: https://gist.github.com/bzamecnik/44a3642c05afc2ac09135687edce8a9d Do you see any flaw in that or you have a different mechanism in mind? Thanks.
Yeah, probably the way to go to enable parallel put() would be using the patch of K.function().

I know. You can use the output of t = get() for training, and use tf.group(t, put()) just for dependencies. get() will not be evaluated twice.

Hooray. I managed to get the pipelining to work and observe the desired effect on a basic CIFAR10 convnet. :)
https://gist.github.com/bzamecnik/b9dbd50cdc195d54513cd2f9dfb7e21b

It works well with variable-sized batches. Also the throughput on a single GPU increased a tiny bit.
So far still the data is fed into the StagingArea in a separate session.run() via an intermediate Variable. This is responsible for the initial gap. It can be reduced by TF queue or possibly Dataset
API (?).
A known limitation for tensor inputs to Keras Model is that they do not support validation data yet. We can still perform validation the classic way via feed_dict.

The used models and datasets make little sense to me work benchmarking;
they are too small and narrow and therefore barely reveal meaningful
numbers and bottlenecks.
I don’t see how the stagingarea’s put command is used in the snippet in a
way that would speed things up, you are calling it synchronously it seems;
whereas you should be calling it simultaneously with your get command in
the same session.run fetch statement to have both put and get run
independently and asynchronously.
On Sat, 28 Oct 2017 at 12:41 Bohumír Zámečník<EMAIL_ADDRESS>wrote:

Hooray. I managed to get the pipelining to work and observe the desired
effect on a basic CIFAR10 convnet. :)
https://gist.github.com/bzamecnik/b9dbd50cdc195d54513cd2f9dfb7e21b
[image: screenshot 2017-10-28 v 21 29 45]
https://user-images.githubusercontent.com/446124/32137825-9cba6cd8-bc27-11e7-9cb5-58ce727ea04d.png
It works well with variable-sized batches. Also the throughput on a single
GPU increased a tiny bit.
So far still the data is fed into the StagingArea in a separate
session.run() via an intermediate Variable. This is responsible for the
initial gap. It can be reduced by TF queue or possibly Dataset
API (?).
A known limitation for tensor inputs to Keras Model is that they do not
support validation data yet. We can still perform validation the classic
way via feed_dict.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/avolkov1/keras_experiments/issues/2#issuecomment-340215037,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHXSRFMM_Qkelej_aitMuFQPhz_o7GVzks5sw4NvgaJpZM4OoPGl
.

@TimZaman Of course I'd like to try it also on Inception3/Resnet50 on imagenet-synth, but so far adapting a smaller CIFAR10 convnet was easier. Indeed get() and put() are called in the same session.run(): tensor from get() goes to Input layer and put() is passed in model.compile() to fetches. Copy from Python to TF memory is still synchronous (it was not the point of this experiment) and nvprof shows that the host-to-device memcpy is really asynchronous. The gap between computations was reduced (in this case roughly from 7 to 4 ms).

In order to make experiments with the StagingArea pipelining easier I extracted the code into StagingAreaCallback and added a few examples - trivial MNIST, simple CIFAR10 and ResNet50.

A known limitation for tensor inputs to Keras Model is that they do not support validation data yet. We can still perform validation the classic way via feed_dict.

It is supported: I added that myself to Keras months ago: https://github.com/fchollet/keras/commit/135efd66d00166ca6df32f218d086f98cc300f1e#diff-299cfd5886683a4b012f286403769fc1
The problem you're solving here isn't that big, and your solution won't give great results, if any at all, as your numpy code is still in the critical path. It would make sense if you would use an example that natively uses tensorflow tensors, which would be the only logical usecase of the stagingarea. furthermore, try to place the stagingarea on the gpu for extra juice. I'll be unsubscribing from this thread as there are way too many messages posted here.

Thanks. Sorry, for too much noise. I was pointing at validation_data in fit(), not evaluate() for testing. The StagingArea is located on GPU, otherwise it makes no sense.

Why wouldn't the stagingarea on the CPU make no sense? In typical high-performance jobs you've got a tensorflow-native graph, meaning there's no feed_dicts and no slow numpy stuff going on.
Anyway, I'm adding support for everyone to add their stagingArea to upstream keras with this MR: https://github.com/fchollet/keras/pull/8285 . There's some pseudocode in that MR of how to use the StagingArea.

Thank you guys very much. I see you both created PRs fchollet/keras#8285 and fchollet/keras#8286. Could you guys come to an agreement and combine the PRs into one PR? But regardless, I'd like to apply the PRs in my monkey patch temporary code here until it gets incorporated (in some fashion) into mainstream Keras.
Thank you both again. Very much appreciated!

And our article Towards Efficient Multi-GPU Training in Keras with TensorFlow about multi-GPU training on Keras+TensorFlow has been published: https://twitter.com/RossumAi/status/924985495851012096

I benchmark a ton; and multi-gpu training for TF backends doesn't get much better than just using Horovod for it.

@TimZaman Yes indeed! I've been testing horovod over the weekend. For some reason on my install on psgcluster I can't get horovod to run well. Is there some trick in installing horovod without containers and getting it to work well? I compiled tf1.4 (whatever was current two weeks ago for tf1.4) with CUDA9, and installed horovod with NCCL2, but something must have been off.
I installed horovod in our dgx containers, and indeed, it scaled even for the trivial mnist example. I'll add horovod examples to my repo. We don't have any dockerfiles or containers available publicly with NCCL2 yet, so it will be kind of difficult for other to replicate. I repeated my test that I ran before and this is what I'm getting on P40s:
        GPU0    GPU1    GPU2    GPU3    mlx5_0  CPU Affinity
GPU0     X      SOC     SOC     SOC     SOC     0-15
GPU1    SOC      X      PHB     PHB     PHB     16-31
GPU2    SOC     PHB      X      PIX     PHB     16-31
GPU3    SOC     PHB     PIX      X      PHB     16-31
mlx5_0  SOC     PHB     PHB     PHB      X

# CUDA_VISIBLE_DEVICES=1,2 mpirun --report-bindings -np 2 -H localhost  -rf rankfile_hswp100  python mnist_horovod.py
Epoch 1/12
30000/30000 [==============================] - 3s - loss: 0.3377 - acc: 0.8963 - val_loss: 0.0713 - val_acc: 0.9763
Epoch 1/12
30000/30000 [==============================] - 3s - loss: 0.3330 - acc: 0.8957 - val_loss: 0.0713 - val_acc: 0.9763
Epoch 2/12
30000/30000 [==============================] - 2s - loss: 0.1048 - acc: 0.9690 - val_loss: 0.0478 - val_acc: 0.9843
Epoch 2/12
30000/30000 [==============================] - 2s - loss: 0.1069 - acc: 0.9688 - val_loss: 0.0478 - val_acc: 0.9843
Epoch 3/12
30000/30000 [==============================] - 2s - loss: 0.0779 - acc: 0.9775 - val_loss: 0.0387 - val_acc: 0.9866
Epoch 3/12
30000/30000 [==============================] - 2s - loss: 0.0805 - acc: 0.9757 - val_loss: 0.0387 - val_acc: 0.9866
Epoch 4/12
30000/30000 [==============================] - 2s - loss: 0.0584 - acc: 0.9821 - val_loss: 0.0365 - val_acc: 0.9872
Epoch 4/12
30000/30000 [==============================] - 2s - loss: 0.0650 - acc: 0.9811 - val_loss: 0.0365 - val_acc: 0.9872
Epoch 5/12
30000/30000 [==============================] - 2s - loss: 0.0518 - acc: 0.9848 - val_loss: 0.0307 - val_acc: 0.9894
Epoch 5/12
30000/30000 [==============================] - 2s - loss: 0.0555 - acc: 0.9833 - val_loss: 0.0307 - val_acc: 0.9894
Epoch 6/12
30000/30000 [==============================] - 2s - loss: 0.0447 - acc: 0.9863 - val_loss: 0.0287 - val_acc: 0.9900
Epoch 6/12
30000/30000 [==============================] - 2s - loss: 0.0494 - acc: 0.9847 - val_loss: 0.0287 - val_acc: 0.9900
Epoch 7/12
30000/30000 [==============================] - 2s - loss: 0.0411 - acc: 0.9875 - val_loss: 0.0291 - val_acc: 0.9892
Epoch 7/12
30000/30000 [==============================] - 2s - loss: 0.0440 - acc: 0.9856 - val_loss: 0.0291 - val_acc: 0.9892
Epoch 8/12
30000/30000 [==============================] - 2s - loss: 0.0382 - acc: 0.9885 - val_loss: 0.0285 - val_acc: 0.9902
Epoch 8/12
30000/30000 [==============================] - 2s - loss: 0.0402 - acc: 0.9876 - val_loss: 0.0285 - val_acc: 0.9902
Epoch 9/12
30000/30000 [==============================] - 2s - loss: 0.0346 - acc: 0.9897 - val_loss: 0.0282 - val_acc: 0.9903
Epoch 9/12
30000/30000 [==============================] - 2s - loss: 0.0346 - acc: 0.9896 - val_loss: 0.0282 - val_acc: 0.9903
Epoch 10/12
30000/30000 [==============================] - 2s - loss: 0.0315 - acc: 0.9904 - val_loss: 0.0254 - val_acc: 0.9910
Epoch 10/12
30000/30000 [==============================] - 2s - loss: 0.0316 - acc: 0.9899 - val_loss: 0.0254 - val_acc: 0.9910
Epoch 11/12
30000/30000 [==============================] - 2s - loss: 0.0282 - acc: 0.9910 - val_loss: 0.0267 - val_acc: 0.9910
Epoch 11/12
30000/30000 [==============================] - 2s - loss: 0.0294 - acc: 0.9901 - val_loss: 0.0267 - val_acc: 0.9910
Epoch 12/12
30000/30000 [==============================] - 2s - loss: 0.0246 - acc: 0.9922 - val_loss: 0.0285 - val_acc: 0.9908
Epoch 12/12
30000/30000 [==============================] - 2s - loss: 0.0285 - acc: 0.9909 - val_loss: 0.0285 - val_acc: 0.9908
Test loss: 0.0285155087762
Test accuracy: 0.9908
Test loss: 0.0285155087762
Test accuracy: 0.9908

The above for single GPU runs at 4s per epoch, and for 4 GPUs 1s. It seems to scale linearly at least within a single node (I haven't been able to get it to work distributed yet).
I've tested it with more complicated keras (ptb lstm, variation autoencoder, ...) networks and it generalizes pretty well. I've had issues running the asynchronous training approach with such networks (loss/acc diverge for multi-GPUs with asynchronous training), but horovod worked.

@avolkov1 That would be great!
Can you use it directly with Keras objects like they describe it beneath 'Usage'?
E.g. a Keras optimizer in hvd.DistributedOptimizer ? I'm also a bit clueless how to handel "5." in e.g. fit_generator.
Also, as far as I get it, the gradients in Horovod are averaged synchronously? And in your (great!) code it is asynchronous?

Aleks, come on, stop using MNIST for distributed training.
Are you in Endeavor any time soon, to meet?

Sorry about resorting to mnist and cifar, but these datasets are universally available.
@ViaFerrata Yea, I'll post some code. I'm not sure about fit_generator. I'll be traveling for the next day, but once I get a chance I'll add some examples. Do this:
hvd.init()  # globally up top is a good idea

def main():  # in main do
    # Pin GPU to be used to process local rank (one GPU per process)
    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    config.gpu_options.visible_device_list = str(hvd.local_rank())
    K.set_session(tf.Session(config=config))

    # define some optimizer
    opt = tf.train.RMSPropOptimizer()
    # Add Horovod Distributed Optimizer.
    opt = hvd.DistributedOptimizer(opt)  # , use_locking=True)

    # run prior to fit
    sess.run(hvd.broadcast_global_variables(0))

And don't miss any steps, because if you do it might run, but results will be messed up. Yes, horovod implements synchronous training whereby it averages gradients.
Maybe once I understand the code well enough I could add a proper synchronous gradient mode to my code base without resorting to MPI.
The code is a mess, but if you want to see a more involved example I've been working with I attached it.
variational_autoencoder_deconv_tfqueue_horovod.txt
There I'm using tf-queue, horovod, keras, etc. For mnist it runs faster but doesn't really converge faster because it converges fast enough with 1 GPU, but for larger datasets (with large images), it should run a lot faster when scaling to multi-gpus.
Here's mnist_horovod.txt a basic mnist example I benchmarked. I'll modify it with tfqueue. It runs a little bit faster with tfqueue, but with tfqueue I don't have to split the data I just reduce steps per epoch so the random sampling is done correctly. I do this because I prefer to decrease the epoch walltime time instead of increasing imgs/sec metric. With img/sec increase overall training time is reduced the same I suppose, as long as convergence is faster with less epochs, but I prefer to reduce epoch iteration time and "split" data across GPUs. This makes it clear that 12 epochs multigpu is faster than 12 epochs 1 gpu.

I tested uber's benchmark script (https://github.com/tensorflow/benchmarks/pull/73) on a single DGX1. For resnet50 horovod is about 95% the speed of native tensorflow (replicated mode), which is really good considering it's in multi-process.


added vertical scrollbar

Issue #88

Thanks for making a vertical slider.

@rx112358
Thanks for making a vertical scrollbar. 😊
@LucasPilla
We now have one!


Change plan strategy for generate search path

When the function design with lower altitude (fd_generate_search_path_low) is selected it is never adapted again for two reasons:

If the water visibility is lower than the expected for fd_generate_search_path_low there are no alternatives to it
When the water visibility starts growing, it keeps fulfilling the expected value for fd_generate_search_path_low, so there is no reason to choose the ones with higher altitude

Reason 1 is not a problem, but reason 2 is.
A solution is to re-plan the function f_generate_search_path at every cycle

My idea for now is to create a sub-class of RosReasoner and override the plan method.

I have no idea how much implementation work this would be, but could we design something like a QA "efficiency" that is the difference between the expected water visibility of the current function design and the measured one? Then it is negative if the expected water visibility is less than the measured one (i.e. in the case where we want to choose another function design). If it is negative, reconfiguration is triggered and another function design with higher expected water visibility is chosen

From the meeting:
Important metrics:

Time for searching pipeline
Time for inspecting the pipeline (minimize it)
Sum both times
Distance of the pipe detected and inspected (if we set a time limit) (maximize it)
Relationship between them

the difference between the expected water visibility of the current function design and the measured one?

Good idea, I will go for this. Might be possible to do it with swrl rules but I think I go for a python solution to be faster.

Done at 66e4923caf5b0788172aa3a3483651847c87f775

the difference between the expected water visibility of the current function design and the measured one?

This basically results in the adaptation being triggered unless the measured water visibility is equal to the expected water visibility. Because if it is lower, it is already being triggered in the regular analyze process. But anyway, this is not a problem.


Fix Guard source code link

What changes to the docs does this PR provide?
Fixes the link to the Guard source code
PR Checklist
Please check if your PR fulfills the following requirements:

[x] Correctly picked the right branch to base the change off (dev for new features, master for typos/improvements)
[ ] For new pages, used the provided template
[ ] For new features, added an entry in the Table of Contents
[ ] Ran against a spell and grammar checker
[x] Contains NO breaking changes

Docs Build status updates of commit 448aeef:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/developer-tools/Guard.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Issue with Integration starting

When I first discovered the integration, it worked perfectly. Then, I bought some more Govee accessories, paired via Govee like usual, and then all my other Govee devices stopped working.
I tried new API keys, tried removed/rebooted/reinstalling the integration, everything I could think of, but nothing got the Integration to work again and doesn't return any devices.
I looked at the logs and I'm seeing the following error:
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 289, in _async_setup_platform
    await asyncio.shield(task)
  File "/config/custom_components/govee_lan/light.py", line 150, in async_setup_entry
    await controller.query_http_devices()
  File "/usr/local/lib/python3.10/site-packages/govee_led_wez/govee.py", line 228, in query_http_devices
    devices = await http_get_devices(self.api_key)
  File "/usr/local/lib/python3.10/site-packages/govee_led_wez/http.py", line 52, in http_get_devices
    return [
  File "/usr/local/lib/python3.10/site-packages/govee_led_wez/http.py", line 60, in <listcomp>
    properties=d["properties"],
KeyError: 'properties'

Is that enough to diagnose? Is there more I can share? Is there something I'm missing?

I'm assuming that one of those devices is missing properties and we need to react accordingly, so it would be great to see what we're dealing with!
Could you try manually curl'ing the HTTP endpoint to retrieve the device list, and sharing it here?
curl -H Govee-API-Key:$GOVEE_KEY "https://developer-api.govee.com/v1/devices"

Sure thing!
{
  "data": {
    "devices": [
      {
        "device": "DEVICE:ID:1",
        "model": "H6199",
        "deviceName": "DeviceName1",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn", "brightness", "color", "colorTem"],
        "properties": { "colorTem": { "range": { "min": 2000, "max": 9000 } } }
      },
      {
        "device": "DEVICE:ID:2",
        "model": "H6008",
        "deviceName": "DeviceName2",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn", "brightness", "color", "colorTem"],
        "properties": { "colorTem": { "range": { "min": 2700, "max": 6500 } } }
      },
      {
        "device": "DEVICE:ID:3",
        "model": "H6008",
        "deviceName": "DeviceName3",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn", "brightness", "color", "colorTem"],
        "properties": { "colorTem": { "range": { "min": 2700, "max": 6500 } } }
      },
      {
        "device": "DEVICE:ID:4",
        "model": "H6008",
        "deviceName": "DeviceName4",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn", "brightness", "color", "colorTem"],
        "properties": { "colorTem": { "range": { "min": 2700, "max": 6500 } } }
      },
      {
        "device": "DEVICE:ID:5",
        "model": "H6008",
        "deviceName": "DeviceName5",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn", "brightness", "color", "colorTem"],
        "properties": { "colorTem": { "range": { "min": 2700, "max": 6500 } } }
      },
      {
        "device": "DEVICE:ID:6",
        "model": "H5080",
        "deviceName": "DeviceName6",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn"]
      },
      {
        "device": "DEVICE:ID:7",
        "model": "H5080",
        "deviceName": "DeviceName7",
        "controllable": true,
        "retrievable": true,
        "supportCmds": ["turn"]
      }
    ]
  },
  "message": "Success",
  "code": 200
}

I believe it's the wifi plugs that were the new additions, that don't have properties is what borked it all

I see; those H5080 smart plugs only support on/off and not the other light controls.
I can make a quick fix to ignore those, but we could probably also support switches separately in the future.

I pushed the minimum effort first pass at this, which is to just pretend that properties are empty when missing. You'll probably see your plugs show up as lights. Hopefully that will unblock the use of the actual lights.
This should be available to update already in HACS.
I don't know what will happen if you try to control the plugs as lights; it's possible that it might allow turning them on/off, but it may error out.  Please let me know!

I've updated my integration, and it works! And the plugs work too!

Great! I've opened a separate issue to track making them show up as proper switches


Fix: Add new display method for test notebooks to fix false-positive E2E test runs in Synapse

Related Issues/PRs

#xxx
What changes are proposed in this pull request?
We currently have a gap in test coverage, where in some circumstances, an example notebook will pass in Synapse E2E test but fail while running it as an interactive notebook. Notably this happens when a dataframe is created inside a call to notebookutils.display().
This change adds a way to force dataframes to materialize in calls to display in our example notebooks while running in Synapse E2E tests using the spark collect operator. This strategy should not be used on large datasets as it forces the driver node to collect all data pages from workers.
How is this patch tested?
I created a small reproduction notebook, which uses the TTS transformer. I set the output file value to use an invalid filesystem. With this patch, you can see this spark app fails in the E2E tests:

This same notebook will pass without this change, when it should have failed:

Does this PR change any dependencies?

[x] No. You can skip this section.
[ ] Yes. Make sure the dependencies are resolved correctly, and list changes here.

Does this PR add a new feature? If so, have you added samples on website?

[x] No. You can skip this section.
[ ] Yes. Make sure you have added samples following below steps.

Find the corresponding markdown file for your new feature in website/docs/documentation folder.
Make sure you choose the correct class estimators/transformers and namespace.
Follow the pattern in markdown file and add another section for your new API, including pyspark, scala (and .NET potentially) samples.
Make sure the DocTable points to correct API link.
Navigate to website folder, and run yarn run start to make sure the website renders correctly.
Don't forget to add <!--pytest-codeblocks:cont--> before each python code blocks to enable auto-tests for python samples.
Make sure the WebsiteSamplesTests job pass in the pipeline.

/azp run

Codecov Report

Merging #1670 (eebfc7f) into master (dd9e5d2) will decrease coverage by 83.23%.
The diff coverage is n/a.

 Take ten seconds to tell us how you rate us. Have a feature suggestion?


ga4gh api support

requires my ga4gh-schema branch:
https://github.com/jpdna/ga4gh-schemas/tree/dev1

not asking for merge just yet - but looking for comments.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/377/
Test FAILed.

It seemed to produced by the shade plug-in, or at least I didn't create it directly myself, I was hoping one of you would recommend how to deal with it :)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/380/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/381/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/382/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/383/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/384/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/mango-prb/385/
Test FAILed.


apply policy

sample yaml policy is as below
policy_yaml: version: 1
databaseType: mysql
databaseName: volume_test_db299
objectType: Table
policy:
- users:
- sa,user1,user2
tables:
- name: personal_info
columns:
- Medical Beneficiary Number
- TaxpayerNumber
- SocialSecurity_Number
- PhoneNumber
- PassportNumber
- NationalInsuranceNumber
- NationalHealthServiceNumber
- National_Provider_Identifier
- National_Drug_Code
- BankAccount_Numbers
- id
- Health_Insurance_Claim_Number
- GPS_Coordinates
- Email_Address
- ElectoralRollNumber
- Drug_Enforcement_Agency_Registration_Number
- Drivers_LicenseID Number
- Credit_CardNumber
rules:
- Audit Read
- Audit Delete
- Audit Modification


Network Interfaces - The text is not aligned in 'Return to previous page'

In the Network Interfaces the button 'Return to previous page' is not looking good. It look like this:

This is how it should look like:

OK, I'll fix it after I finish working on theme configuration for non-root users.

This doesn't seem to be the theme issue. First of all, I simply can't reproduce it.
The theme just re-renders data supplied by the server. In case you see this, you must have to provide ways to reproduce it and report it to Webmin repo. It has to be fixed on the server side.
The reason why this is happening is broken output on error page. Jamie should fix it in no time. He would guess very quickly what is missing.

The real issue looks like it's due to an error message appearing in the middle of a table. I'll fix this in the next Webmin release, and also fix the underlying cause of that error.

Thank you, Jaime! :wink:


Dart >3.0.0

Any chance you can open up the pubspec SDK version upper limit to include the latest dart versions?

Anyone home?


Typo in comparisons guide

In the "Performance" subsection, comparing Vue to React, it seems like there might be a typo: Both React and Vue offer comparable performance in most commonly seen use cases, with Vue usually slightly head due to its lighter-weight Virtual DOM implementation Seems like head should be ahead? Not a huge deal but figured I'd let you know! Cheers.

Looks like someone just submitted a PR to fix this! Thanks. 😄


Migrate from ntpdate to ntpd

Is your feature request related to a problem? Please describe.
ntpdate is deprecated and is replace by ntpd or chrony.
It will be remove in next major version of FreeBSD
https://man.freebsd.org/cgi/man.cgi?query=ntpdate&sektion=8
(Section Description on top)
FreeBSD will use NTPD ...
Describe the solution you'd like
Implemant NTPD like the True Unix way, NTPD make a good job , and can replace ntpdate easylly with -q option.
https://docs.freebsd.org/en/books/handbook/network-servers/#network-ntp
Describe alternatives you've considered
Chrony is the big winner of they last years, but have no port on FreeBSD
https://gitlab.com/chrony/chrony
Additional context
Date and Time.app have as target NTPD and require it migration

The ntpd package should be install inside ISO
The ntpdate package should be remove from ISO
That specific file https://github.com/lattera/freebsd/blob/master/etc/ntp.conf should be imposed by default
Date and Time.app will use that default ntp.conf as model, then impose/restore each time a file as close to original file.

I dont know where edit what for got the result but that the plan.

That file also says:
# Don't forget to enable ntpd in /etc/rc.conf with:
# ntpd_enable="YES"

Why do we need a file from https://github.com/lattera, how is it different from the default in the ntpd package?

that the default package, no diffrence

Did you make the change on your local system? Did it change boot speed?

My Dev system in under GNU/Linux and got it package and file by default . Yet i havent make any performance test under helloSystem.


feat(LinearAlgebra/QuadraticForm): add QuadraticForm.Isometry

Also use it to tidy the API of CliffordAlgebra.map.

I think this was broken by #6786, I've merged master, fixed the error, and tidied I file I forgot about.

bors merge


Permission denied opening temp file

Hi,
I'm just starting to play with onsets_frames_transcription model from drums.  I am attempting to transcribe a single hit of a one-shot sample.  While debugging, I can see a temporary file created in C:\Users\micha\AppData\Local\Temp (with zero bytes).  However, the process cannot open that file.  I'm running Python 3.8.3 under Windows 10.  What am I doing wrong?
Thanks.
See below:
(base) C:\Users\micha>onsets_frames_transcription_transcribe --model_dir=E-GMD\e-gmd-v1.0.0\e-gmd_checkpoint --config=drums --load_audio_with_librosa "Tom Wave 3.wav"
2020-10-13 22:26:28.882284: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-10-13 22:26:28.882475: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
c:\programdata\anaconda3\lib\site-packages\librosa\util\decorators.py:9: NumbaDeprecationWarning: ←[1mAn import was requested from a module that has moved location.
Import requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.←[0m
from numba.decorators import jit as optional_jit
c:\programdata\anaconda3\lib\site-packages\librosa\util\decorators.py:9: NumbaDeprecationWarning: ←[1mAn import was requested from a module that has moved location.
Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0.←[0m
from numba.decorators import jit as optional_jit
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:tensorflow:AutoGraph could not transform <function preprocess_example at 0x000001FD33B21B80> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output.
Cause: module 'gast' has no attribute 'Index'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
W1013 22:26:33.223956 17772 ag_logging.py:146] AutoGraph could not transform <function preprocess_example at 0x000001FD33B21B80> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output.
Cause: module 'gast' has no attribute 'Index'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\data.py:134: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
W1013 22:26:33.223956 17772 deprecation.py:317] From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\data.py:134: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.
WARNING:tensorflow:AutoGraph could not transform <function input_tensors_to_model_input at 0x000001FD33B22430> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output.
Cause: module 'gast' has no attribute 'Index'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
W1013 22:26:33.302016 17772 ag_logging.py:146] AutoGraph could not transform <function input_tensors_to_model_input at 0x000001FD33B22430> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output.
Cause: module 'gast' has no attribute 'Index'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2020-10-13 22:26:33.343892: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-10-13 22:26:33.344098: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-10-13 22:26:33.356562: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-H0T5JPS
2020-10-13 22:26:33.369591: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-H0T5JPS
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\data.py:656: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(dataset).
W1013 22:26:34.879807 17772 deprecation.py:317] From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\data.py:656: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(dataset).
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\train_util.py:87: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.
W1013 22:26:34.973538 17772 module_wrapper.py:136] From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\train_util.py:87: The name tf.estimator.tpu.RunConfig is deprecated. Please use tf.compat.v1.estimator.tpu.RunConfig instead.
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\train_util.py:88: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.
W1013 22:26:34.989150 17772 module_wrapper.py:136] From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\train_util.py:88: The name tf.estimator.tpu.TPUConfig is deprecated. Please use tf.compat.v1.estimator.tpu.TPUConfig instead.
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\train_util.py:99: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.
W1013 22:26:35.004811 17772 module_wrapper.py:136] From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\train_util.py:99: The name tf.estimator.tpu.TPUEstimator is deprecated. Please use tf.compat.v1.estimator.tpu.TPUEstimator instead.
INFO:tensorflow:Using config: {'_model_dir': 'E-GMD\e-gmd-v1.0.0\e-gmd_checkpoint', '_tf_random_seed': None, '_save_summary_steps': 300, '_save_checkpoints_steps': 300, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': None, '_keep_checkpoint_every_n_hours': 1, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=300, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=2, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
I1013 22:26:35.020538 17772 estimator.py:191] Using config: {'_model_dir': 'E-GMD\e-gmd-v1.0.0\e-gmd_checkpoint', '_tf_random_seed': None, '_save_summary_steps': 300, '_save_checkpoints_steps': 300, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': None, '_keep_checkpoint_every_n_hours': 1, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=300, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=2, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1), '_cluster': None}
INFO:tensorflow:_TPUContext: eval_on_tpu False
I1013 22:26:35.020538 17772 tpu_context.py:217] _TPUContext: eval_on_tpu False
WARNING:tensorflow:From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\onsets_frames_transcription_transcribe.py:102: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through tf.compat.v1. In all other situations -- namely, eager mode and inside tf.function -- you can consume dataset elements using for elem in dataset: ... or by explicitly creating iterator via iterator = iter(dataset) and fetching its elements via values = next(iterator). Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use tf.compat.v1.data.make_initializable_iterator(dataset) to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.
W1013 22:26:35.036062 17772 deprecation.py:317] From c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\onsets_frames_transcription_transcribe.py:102: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through tf.compat.v1. In all other situations -- namely, eager mode and inside tf.function -- you can consume dataset elements using for elem in dataset: ... or by explicitly creating iterator via iterator = iter(dataset) and fetching its elements via values = next(iterator). Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use tf.compat.v1.data.make_initializable_iterator(dataset) to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.
2020-10-13 22:26:35.057591: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-13 22:26:35.073695: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1fd34d77e00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-13 22:26:35.073825: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
INFO:tensorflow:Starting transcription for Tom Wave 3.wav...
I1013 22:26:35.145374 17772 onsets_frames_transcription_transcribe.py:112] Starting transcription for Tom Wave 3.wav...
INFO:tensorflow:Processing file...
I1013 22:26:35.160952 17772 onsets_frames_transcription_transcribe.py:118] Processing file...
c:\programdata\anaconda3\lib\site-packages\librosa\core\audio.py:161: UserWarning: PySoundFile failed. Trying audioread instead.
warnings.warn('PySoundFile failed. Trying audioread instead.')
Exception %s [Errno 13] Permission denied: 'C:\Users\micha\AppData\Local\Temp\tmprlo72l7m.wav'
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\programdata\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\Scripts\onsets_frames_transcription_transcribe.exe_main.py", line 7, in
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\onsets_frames_transcription_transcribe.py", line 155, in console_entry_point
tf.app.run(main)
File "c:\programdata\anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "c:\programdata\anaconda3\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "c:\programdata\anaconda3\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\onsets_frames_transcription_transcribe.py", line 150, in main
run(argv, config_map=configs.CONFIG_MAP, data_fn=data.provide_batch)
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\onsets_frames_transcription_transcribe.py", line 121, in run
create_example(filename, hparams.sample_rate,
File "c:\programdata\anaconda3\lib\site-packages\magenta\models\onsets_frames_transcription\onsets_frames_transcription_transcribe.py", line 73, in create_example
assert len(example_list) == 1
AssertionError

Duplicate of #16


Pre-commit hook and pyproject-flake8

Hi :slightly_smiling_face:
Thanks for this nice addition to the flake8 ecosystem!
I'm currently working on a project relying on pre-commit-config.yaml to insure consistency in the code style.
Since the default commit hook does not include your package, I tried using the method described here:
repos:
    # Flake8
    - repo: https://github.com/pycqa/flake8
      rev: 3.9.2
      hooks:
        - id: flake8
          additional_dependencies: [pyproject-flake8]

My config for flake8 in pyproject.toml is:
[tool.flake8]
ignore = "E203, E266, E501, W503, F403, F401"
exclude = ".git, .mypy_cache, .pytest_cache, build, dist"
max-line-length = 89
max-complexity = 18
select = "B,C,E,F,W,T4,B9"

Yet, when running the commit hook, I get:
E501 line too long (87 > 79 characters)

This means that the configuration is not parsed properly from the pyproject.toml.
Do you have any idea to solve this?
Thanks :slightly_smiling_face:

Sorry for the auto-reply, but if someone comes across the same issue, here is a configuration that works. The problem was that the hook used the flake8 command and not pflake8:
- repo: https://github.com/pycqa/flake8
      rev: 3.9.2
      hooks:
        - id: flake8
          entry: pflake8
          additional_dependencies: [pyproject-flake8]

Indeed someone else may need such a clarification, I'll see if I'll add a section to the README how to use pflake8 with pre-commit.

I just saw that the repository includes a pre-commit hook so my solution might not be the recommended one. Thanks :slightly_smiling_face:

I just saw that the repository includes a pre-commit hook so my solution might not be the recommended one. Thanks 🙂

I still like the above configuration because the version of flake8 is explicitly fixed (while the pre-commit hook in the repository does not specify the version of flake8; probably the latest one will be used, which potentially leads to some incompatibility in future).

Probably obvious to most,  but something I just ran into after settings this up on a new project.
If your using vscode in WSL and you get the error saying it cant find pflake8 when you do a commit,  you need to start vscode from inside your poetry venv.


Other Possible Business Logics?

Hello everyone! I’m kinda interested to see if anyone had any thoughts on an alternate logic for the project as I think my current solution does not seem very smooth…
So originally with the service times supplied for all customers at the start, the Arrive -> Wait -> Serve chain was much smoother imo; that is, from the simulator (my manager) I could just directly call all the necessary ProcessEvent methods, i.e. Arrive returns a Wait Event and Wait Event returns a Serve Event and so on…
However, in order to account for the delayed process of getting the service times and anticipate the possibility that my queue for any server can be very long, I severed the Wait -/-> Serve chain. Instead, from the simulator, I additionally check if the polled event is a Done Event with the respective Server with a non-empty queue (unsmooth thing 1) and directly construct a Serve Event in the simulator itself (unsmooth thing 2) to be passed back into PQ, and so this just does not seem really optimal though it’s functional, but I can’t seem to think of an alternate logic that is smoother or more “elegant”. Any thoughts are appreciated! Thank you!

Hi,
Perhaps you can refer to #176 for some inspiration.

Hi,
Perhaps you can refer to #176 for some inspiration.

Hey thanks for linking the post, I must have missed it! I will read the comments more deeply again later but on a first reading my worry is that it seems the focus on the max queue size being 2 (?) (kindly correct me if I am reading the post wrongly) may not account for say, a server with a max queue size of >2 like 100. I’ll leave my post open for awhile more!

I think what you are saying about one queue being too long is a valid consideration in reality but I remember in the project#1 document it did state the criteria of scanning from 1 to N and "not necessarily the shortest" so I dont think we have to worry about optimisation for this project.
As for implementation, I did the same as you for project#1 submission and I agree that it is way more elegant than what I have after lab#5. I tried both your current method (Done -> Serve) and repeated Wait event and I personally find that repeated Wait seems smoother since in simulator I still only deal with events without dealing with servers and customer whereas when I did (Done -> Serve) I had to deal with them in the simulator. I can't keep the abstraction otherwise it results in cyclic dependency. I'm sure there must be a better way but repeated Wait is my best one so far.


Swift Macros support

I haven't seen any discussion related to Swift Macros support in XcodeGen. Since macros are meant to be distributed as Swift packages, I assumed that I could import a local package containing my macro and that this would just work. However, XcodeGen seems to treat local and remote packages differently (i.e. local packages are not part of the "Package Dependencies" section in the Project navigator), which is causing the compiler to not find my macro when building. If instead I use "Add Package Dependency..." in Xcode and add my local package there manually, the compiler is able to see the macro and everything works fine.
Curious if anyone else has encountered this issue when working with swift macros.

I've been using macros in packages both remote and local, and has been working 🤷
Local dependencies are in the file navigator instead of the Package Dependencies section in Xcode

Help please with this problem
How do you describe macros importing locally and remotely?
https://github.com/yonaskolb/XcodeGen/issues/1501


fix: build fixtures show logs and exit process when error occurred

Codecov Report

Merging #3112 into dev will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff          @@
##            dev   #3112   +/-   ##
====================================
  Coverage   100%    100%
====================================
  Files         4       4
  Lines        19      19
  Branches      2       2
====================================
  Hits         19      19

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 992f18a...15296bf. Read the comment docs.


Sample does not work on v1 of Azure Function

This only works in the beta version of the Azure Function Proxy. I am using authentication.
Once I switch to beta, it started working.
Runtime version: 2.0.11933.0 (beta)
{
    "$schema": "http://json.schemastore.org/proxies",
    "proxies": {
        "proxy1": {
            "matchCondition": {
                "methods": [ "GET" ],
                "route": "/api/{test}"
            },
            "responseOverrides": {
                "response.body": "Hello, {test}",
                "response.headers.Content-Type": "text/plain"
            }
        }
    }
}

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c59024a4-9208-9b00-43ee-f6ba936c95a6
Version Independent ID: 5ac7705c-35e3-b211-ebba-3798dd949409
Content: Work with proxies in Azure Functions
Content Source: articles/azure-functions/functions-proxies.md
Service: functions
GitHub Login: @alexkarcher-msft
Microsoft Alias: alkarche

This was caused by another proxy in the same file which was invalid.

Thanks for the update @masters3d


Allow group/role based authentication using LdapRealm [ZEPPELIN-2539]

What is this PR for?
Currently allowing authentication for selected roles/groups of an LDAP realm is not possible. The LDAPRealm allows for mapping of roles to groups but only allows authorization on URLs with respect to groups. No group based checks are carried out during authentication. This PR allows for group based authentication using LdapRealm.
What type of PR is it?
[Improvement]
Todos

[ ] - Merge #932 - This PR also merges changes from 932 so that needs to be merged first.

What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2539
How should this be tested?
Build and configure shiro.ini to use the LdapRealm and verify that the realm works as before along with the added functionality of allowing only certain user groups for authentication if the allowedRolesForAuthentication config is set in the init. If this configuration is absent authentication should work as before without verifying roles. A sample shiro.ini is pasted here for testing purposes.
[main]
ldapRealm = org.apache.zeppelin.realm.LdapRealm
ldapRealm.userDnTemplate = uid={0},ou=people,dc=my-company,dc=net
ldapRealm.searchBase = dc=my-company,dc=net
ldapRealm.userSearchBase = ou=people,dc=my-company,dc=net
ldapRealm.groupSearchBase = ou=groups,dc=my-company,dc=net
ldapRealm.contextFactory.url = ldaps://auth.my-company.net:636
ldapRealm.contextFactory.authenticationMechanism = simple
ldapRealm.userObjectClass = posixAccount
ldapRealm.groupObjectClass = posixGroup
ldapRealm.authorizationEnabled = true
ldapRealm.memberAttribute = memberUid
ldapRealm.memberAttributeValueTemplate=uid={0},ou=people,dc=my-company,dc=net
ldapRealm.rolesByGroup = GLOBAL_ADMINS:admin,HKG_USERS:user
ldapRealm.allowedRolesForAuthentication = admin,user
ldapRealm.userSearchAttributeName = uid
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
shiro.loginUrl = /api/login
securityManager.sessionManager = $sessionManager
securityManager.sessionManager.globalSessionTimeout = 86400000
securityManager.realms = $ldapRealm

[urls]
/api/version = anon
/api/login = authc
/api/login/logout = authc
/** = authc, roles[admin,user]

Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? Y (documentation updated in PR)

\cc @khalidhuseynov

@sohaibiftikhar thanks for adding option for authentication with role check, I was wondering if these changes are related to the issue in #932, from first look seems a bit unrelated and i would assume it should work from master as well, WDYT?

@khalidhuseynov At front it seems that the issues are unrelated however for reasons I do not know #2182 is triggered when using the LdapRealm for me. #932 fixes that issue and hence for testing my build with this option I had included the changes for that.

thanks for clarification, then i may look into #932 as well, and after finalising that we can rebase this one

Hi @khalidhuseynov!
Thanks for looking into it. Is there a way to support you?

Hi @FRosner, I think best way would be to help review and test #932 first. i was going to do it as well, but didn't get much time recently. thanks for help, and i'll check it this week too.

@sohaibiftikhar i guess we can rebase it from master now

CI looks good now it was initially now working before as for some reason:
curl -sSL "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x51716619E084DAB9" | sudo -E apt-key add -
was returning with no gpg key found.
Build is now fine at:
https://travis-ci.org/sohaibiftikhar/zeppelin/builds/238652785

CI is green. Earlier errors seem to have been erratic and resolved themselves after estart: https://travis-ci.org/sohaibiftikhar/zeppelin/builds/238705120

LGTM. Thanks @sohaibiftikhar for the contribution.
Merge to master if no further discussions here.


Add extra path in Engagement endpoint for checklists

Add extra path in Engagement endpoint for checklists because there was not a way to get or create those checklists from the api

@Maffooch, I'm just working on #9751 and I found out that this PR is using the non-existing function get_authorized_engagement_checklists. Can you check it, please?

@kiblik Thanks for the heads up. I found this the other day as well. I believe at the time of implementation, I attempted to add a full endpoint with model viewsets for Engagement Checklists, but then backed up to only add a few new paths to the Engagement viewset. Since the viewset was not actually called anywhere, there was never an error thrown for trying to use the non existent get_authorized_engagement_checklists method.
In a later PR (#10503) I removed the EngagementCheckListViewset  viewset that calls the get_authorized_engagement_checklists  method. You could do that same in your PR, or just pull in the latest changes from the release when it drops later today


Power BI integration with Entity store

Hi,
This is an amazing solution for Power BI integration to AX.
I just want to know which way that Power BI load data from entity store, the SQL server or others?
Also, can you provide an example that I can follow to setup data, build my own report and embed to AX?
Thanks!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: da145b17-ccd8-7bf7-7495-0ddb1bf5829f
Version Independent ID: 2a24d715-ecac-1c65-d2d4-8d14b55b8765
Content: Power BI integration with Entity store - Finance & Operations | Dynamics 365
Content Source: articles/fin-ops-core/dev-itpro/analytics/power-bi-integration-entity-store.md
Service: dynamics-ax-platform
Product: ****
Technology: ****
GitHub Login: @MilindaV2
Microsoft Alias: milindav

@April-Cai, thanks for the feedback.
@MilindaV2, please review.

Bug #483141 logged to track this issue.

Pl see the document here: https://docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/analytics/add-analytics-tab-workspaces?toc=/dynamics365/commerce/toc.json


LEFT_SHIFT key up event is not caught if the window it not focused

This is refined version of #444.
Environment:

macOS BigSur (11.0.1)
Japanese QWERTY keyboard

Steps to reproduce:

Press left shift key and keep it held down
While keeping the key pressed, click the mouse outside the window so that the window becomes inactive
Finally release the left shift key, then KEY_UP event doesn't come

It can be tested with for example cimgui-sokol-starterkit with a custom event function:
static void event(const sapp_event *ev) {
+    if (ev->type == SAPP_EVENTTYPE_KEY_UP || ev->type == SAPP_EVENTTYPE_KEY_DOWN) {
+        printf("%d\n", ev->type);
+    }
    simgui_handle_event(ev);
}

You can now check for the SAPP_EVENTTYPE_UNFOCUSED event (happens when the window becomes inactive), and "cancel" any potentially pressed keys. It's not super-convenient, because you need to keep track of pressed keys yourself, but it should fix the problem of keys getting stuck when the window loses input focus.
Closing the ticket.


Assert - leave test status at default (hence invisible) until statement parses (& compiles?)

This is now how it works, as a result of other changes


TPR name unclear

I'm very excited about this operator and giving it a try. In https://github.com/pires/nats-operator/blob/master/pkg/controller/controller.go#L38 the TPR name is set to management.nats.io which also seems to be the TPR that's created when I run the operator.
The example however defines the kind as NatsCluster https://github.com/pires/nats-operator/blob/master/example/example-nats-cluster.yaml
However there is no natscluster TRP.
What am I missing?

Possibly some mess up when merging. Unfortunately, I haven't had the time to move forward with this implementation but would be great to have someone else helping out as well.

I'm working on a site that's basically a curated catalog of operators. I've also written the Memcached operator there and am working on the one for MongoDB (very early). My Go isn't very strong though, my own operators are in Python. I won't be much help due to my limited Go and the fact that I don't know anything about NATS.
I will give this TPR name issue a try though. That should be easy enough to fix. I would assume you agree that NatsCluster is the better name?

I would assume you agree that NatsCluster is the better name.

Correct. management.nats.io is or should be the API group, while NatsCluster is the API object.
I paused this work because of some TPR bugs. Need to check if they've been fixed or not.


Lots of repos missing

Hi do you know any of alternative lists that are regularly updated? For instance there are a lot of good repos missing in this one (e.g. r_idn, Wizage, Antique, etc)

I don't know of any other up to date repository lists, unfortunately. I usually just ask devs to make a PR here to add their own repo. I post it semi-regularly (every 6 months) on /r/jailbreak, but I don't want to spam it that much.
I might post it again soon though!


Update native extensions article

Can we mention the need to define DART_SHARED_LIB in the Linux and Mac build instructions as well?
https://www.dartlang.org/articles/native-extensions-for-standalone-dart-vm/

Here's the source for the article: https://github.com/dart-lang/www.dartlang.org/blob/master/src/site/articles/native-extensions-for-standalone-dart-vm/index.markdown
If possible, open the file in the GitHub editor and send us a pull request.
Thanks!

Thanks to Ryan https://github.com/dart-lang/www.dartlang.org/pull/1380


Prepare release-1.4 branches

This issue aims to track all steps needed to prepare the release 1.4 branches/builds.
Please add all steps taken here so we can document for future releases

[ ] (Optional) Update go dependencies. This must be done before branching so that PRs can be easily backported later.

[ ] istio/istio
[ ] istio/pkg
[ ] istio/api
[ ] istio/cni
[ ] istio/operator
[ ] istio/installer
[ ] istio/tests
[ ] istio/client-go

[x] Set up release manager group (owner: @geeknoid)
[ ] Create a release-1.4 branch in every Istio repo - run commands export org=xyz ; export repo=xyz ; (git clone git@github.com:${org}/${repo}.git && cd $repo && git checkout -b release-1.4 && git push --set-upstream origin release-1.4).

[ ] istio/istio
[ ] istio/pkg
[ ] istio/api
[ ] istio/cni
[ ] istio/proxy
[ ] istio/envoy
[ ] istio/operator
[ ] istio/installer
[ ] istio/client-go
[x] Explicitly skipped: istio-releases/pipeline, istio/tests, istio/istio.io

[ ] Commit a PR into every release-1.4 branch to modify every CODEOWNERS to contain only the release managers for this release. Command: export org=xyz ; export repo=xyz ; (git clone git@github.com:${org}/${repo}.git && cd $repo && git checkout release-1.4 && git checkout -b release-1.4-codeowners && echo '* @istio/release-managers-1-4' > CODEOWNERS && git add CODEOWNERS && git commit -m 'Set release managers as CODEOWNERS for release-1.4' && git push --set-upstream origin release-1.4-codeowners). Those PRs need to be force-merged using admin privileges.

[ ] istio/istio
[ ] istio/pkg
[ ] istio/api
[ ] istio/cni
[ ] istio/proxy
[ ] istio/envoy
[ ] istio/operator
[ ] istio/installer
[ ] istio/client-go

[ ] Set up prow on release branches (owner: @howardjohn)

[ ] istio/istio
[ ] istio/pkg
[ ] istio/api
[ ] istio/cni
[ ] istio/proxy
[ ] istio/envoy
[ ] istio/operator
[ ] istio/installer
[ ] istio/client-go

[ ] Set up branch protection (owner: @howardjohn) https://github.com/istio/test-infra/pull/1649
[ ] Update all repos that point to master branch of another repo to point to the release branch

[ ] istio/istio
[ ] istio/pkg
[ ] istio/api
[ ] istio/cni
[ ] istio/proxy
[ ] istio/envoy
[ ] istio/operator
[ ] istio/installer
[ ] istio/client-go

[ ] Run the long-running tests on the dev builds
[ ] Send announcement on Discuss/Slack/Twitter
[ ] Publish an alpha release

@howardjohn s/1-3/1-4/

To upgrade to the latest version for all direct and indirect dependencies, using the latest patch releases, of the current module, the following can be run from the module root directory: go get -u=patch -t ./...

I did do some initial testing of updating the go modules across the repos listed above with draft PRs:

istio is still failing some tests. I suspect it might not have liked prometheus updates. Continuing to look at that.
installer and operator had no new updates
I can't get cni to get through a go mod tidy or go get -u ...
pkg and api are passing
tests is still being done

Steps we missed

update common files in master first
apply branch protection before doing the branch, not after

Also

rebuild the base docker image to pick up security patches

* rebuild the base docker image to pick up security patches

+1 for doing that even for z releases

I think we're doing a couple things incorrectly.

We should not be updating common files before branching.
We should not be updating go.mod dependencies before branching.

In general, we want to update the tools dependencies in build-tools, update common-files, and update go.mod in master right after the release branch has been created. That is, update all of this stuff at the start of a release cycle, not as the start of a release branch.

update go.mod in master right after the release branch has been created

We thought that the incompatible dependencies could make cherry-picks difficult.

One comment as I was looking at some of the module updates that I didn't do because I was concerned, and a pro for doing in master after the branches are cut. I noticed that some packages have changed, I think one example was github.com/golang/sync went to golang.org/x/sync. Best time to do that would be just after the branch is cut in master but it will affect cherry picks.

Note: Remember to change release-builder specified branch to CHANGE_THIS_TO_RELEASE_BRANCH in master after branching.

We still need to branch Envoy and Proxy @PiotrSikora @mandarjog can we do this soon?

Retrospective comments:

Google CLA bot and non-Googlers on cherry picks
Late change to which installer to document which invalidated some testing event testing

Multi-repo is a pain. Changes in a chart need to travel through 3 repos to reach final destination, also add the fact that most PR's need to be cherry-picked which gives 6 PR's!

Closing as all repos are branched

Another thing we may/may not be doing is the preliminary.io uses scripts from the master branch while once it's moved to istio.io it'll use the release branch. It may be that the release branch script isn't correct and wasn't necessarily tested.


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v4.9.295",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=b10c7d745615a092a50c2e03ce70446d2bec2aca"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "b10c7d745615a092a50c2e03ce70446d2bec2aca",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "phonet/pep: refuse to enable an unbound pipe\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.9.295 by commit b10c7d745615a092a50c2e03ce70446d2bec2aca. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002779


Should support month and years

var parseMs = require("parse-ms");
parseMs(133700000000);

{
days: 1547
hours: 10
milliseconds: 0
minutes: 53
seconds: 20
}

Same answer as https://github.com/sindresorhus/pretty-ms/issues/16#issuecomment-315037501

Closing as we can't accurately calculate months and years without defining a set point in time and that's outside the scope of this module.


refactor: remove address requirement from nc registering process

Motivation
Liberate a Nano Contract registration while using Wallet Service. It is not possible now because the address is checked as belonging to the wallet, but Wallet Service's wallet doesn't implement this check.
Acceptance Criteria

Remove address requirement to register a Nano Contract

[!WARNING]
It is lacking:

[ ] Load wallet's firstAddress on wallet initialization
[ ] Adapt NanoContractDetailsScreen to consume the firstAddress as the default address of signature in case of an operation
[ ] Adapt NanoContractDetailsScreen to show a feedback of first address request failure in case it happens

Security Checklist

[x] Make sure you do not include new dependencies in the project unless strictly necessary and do not include dev-dependencies as production ones. More dependencies increase the possibility of one of them being hijacked and affecting us.

We decided to not go ahead with this refactoring because there is another refactoring that would do this changes unecessary.


Task/hrt306 incomplete applications

Link to JIRA ticket
Add a link to the JIRA ticket that the changes in this PR describe.
Describe this PR
What is the problem we're trying to solve
Add a summary of the problem and any additional context that helps explain what the issue we're trying to solve is.
What changes have we introduced
Add descriptions of what changes were made to address the problem and provide a solution.
Include any links to commits that introduce significant additions of code if they help explain the process of coming to the solution. e.g Addition of test database setup, addition of first test and production class, removal of buggy code, etc.
Checklist

[ ] Added end-to-end (i.e. integration) tests where necessary e.g to test complete functionality of a newly added feature
[ ] Added tests to cover all new production code
[ ] Added comments to the README or updated relevant documentation (add link to documentation), where necessary.
[ ] Checked all code for possible refactoring
[ ] Code pipeline builds correctly

Follow up actions after merging PR
Are there any next steps that need to addressed after merging this PR? Add them here.

Added some comments on linked PR (needs closing). At a glance, this looks good to me.


Set faked-system-time in gpg.conf

Old packages would previously fail to build if they depended on a
package that was signed with an expired key, from the perspective of
the keyring they were signed with. We can fix this by setting
faked-system-time to control in gpg.conf to the package build time.
This reveals a further issue that the keyring master key will be
generated, and sign the archlinux master keys, at the time repro
builds that keyring, which may be in the future relative to the build
time of packages that depend on it. As gpg ignores signatures and keys
which appear to be from the future, this will also break the build. We
can fix this by also setting faked-system-time to the keyring
package build time when setting up the keyring.
Finally, to ease transition, we discard and rebuild any keyrings in
the cache that weren't built in this way. This should allow recovery
from this issue without manual intervention.
Fixes #87

Instead of keeping workaround in the code, I think it's better to bump the repro chroot version and just ask people to delete the entire directory. It's much easier for them and us to ensure we don't end up adding a ton of work arounds and much rather declare changes as incompatible.
https://github.com/archlinux/archlinux-repro/blob/master/repro.in#L232
Could maybe write a manpage for this as well.
Other then that the patch looks great. Just need to set off some time to test it :)

Instead of keeping workaround in the code, I think it's better to bump the repro chroot version and just ask people to delete the entire directory. It's much easier for them and us to ensure we don't end up adding a ton of work arounds and much rather declare changes as incompatible.
https://github.com/archlinux/archlinux-repro/blob/master/repro.in#L232
Could maybe write a manpage for this as well.
Other then that the patch looks great. Just need to set off some time to test it :)

Nice work! @fergus-dall !

Nice work! @fergus-dall !

Merged. Because of glibc issues I'm having some issues reproducing the error. But it looks fine enough


chore(deps): update dependency cypress to v3.8.2

This PR contains the following updates:

Package
Type
Update
Change

cypress
devDependencies
patch
3.8.1 -> 3.8.2

Release Notes

cypress-io/cypress
v3.8.2
Compare Source

Renovate configuration
:date: Schedule: At any time (no schedule defined).
:vertical_traffic_light: Automerge: Enabled.
:recycle: Rebasing: Whenever PR becomes conflicted, or if you modify the PR title to begin with "rebase!".
:no_bell: Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

Available now for Enterprise: Renovate Pro with real-time webhook handling and priority job queue.

:tada: This PR is included in version 2.154.9 :tada:
The release is available on:

master+v2.154.9
GitHub release

Your semantic-release bot :package::rocket:


Web UI downgrade to 0.88 in 0.92 CLI

[Urgent] I have 0.88 installed, then I rpm -e it. and install .92. But the Web UI is still connect to v0.88 version
[root@sijr12mxdb2-iot influxdb]# ./influx
Connected to http://localhost:8086 version InfluxDB v0.8.8 (git: afde71e) (leveldb: 1.15)
InfluxDB shell 0.9.2

How can I get the 8086 to 0.92??

Can any one help me?

Try restart influx

It appears that you still have InfluxDB 0.8.8 running on the box. Shut down the process and make sure that you are explicitly launching the new 0.9.2 binary. The CLI just connects to whatever answers at port 8086, so somehow you've still got the old binary running.