ref(*): update DEIS_PLATFORM_VERSION to be identical to DEIS_API_VERSION since components now have an independant release cycle

I think the change is good, but breaking backwards compatibility is a little concerning. That being said, I would hope that people would be using api version rather than platform version for checks.
We may want to expose the controller version, because if there is a bug in a specific release, people's apis might want to be able to account for it.

Current coverage is 85.44% (diff: 100%)

Merging #930 into master will increase coverage by 0.06%

@@             master       #930   diff @@
==========================================
  Files            30         28     -2
  Lines          3049       3043     -6
  Methods           0          0
  Messages          0          0
  Branches        512        512
==========================================
- Hits           2603       2600     -3
+ Misses          296        294     -2
+ Partials        150        149     -1

Powered by Codecov. Last update 1e90c0c...74ebe1d

Yeah personally I'd keep this in for backwards compatibility reasons, especially since it's still useful for features like https://github.com/deis/workflow-cli/issues/25

We have to key it off something else then since we are not longer releasing Controller lockstep with Workflow.
My suggestion was to use api.__version__ and bump that every time we do a release - That would cause API versions that basically have no public changes but would hold the overall API (Controller) to a single release
If we did that we could keep this header intact, albeit only showing the minor release (like the API version) or we could go even more granular, but even that to me is its own security risk

I'm overall open to ideas - All I know is that right now the platform version would never stay in true sync with the world, given we are talking about a potential 2.4 Controller release for the 2.3 Workflow release :)

Yeah sounds like a plan - I amended a commit locally (will fix commit message) already to work that way. I wanted to try to pull in the git sha if people did a checkout but didn't get far

Okay let's keep both the API and PLATFORM headers available, but let's just change the two to be identical.

I think that's the best compromise that maintains backward-compatibility. 👍


YOLO loss

Hi, I've been spending the last few days trying to make a loss_yolo layer for dlib, in particular the loss presented in the YOLOv3 paper.
I think I came up with a pretty straightforward implementation but, as of now, it still does not work.
I wondered if you could have a look. I am quite confident the loss implementation is correct, however I think I might be making some assumptions about the dlib API when the loss layer takes several inputs from the network.
I tried to make the loss similar to the loss_mmod in the way you set the options of the layer, etc.
So, my question is, does this way of coding the loss in dlib make sense for multiple outputs? Or is dlib doing something I don't expect?
There's also a simple example program that takes a path containing a training.xml file (like the one from the face or vehicle detection examples).
Thanks in advance :)

Honestly, after debugging the code and running many experiments, I can't see what I am doing wrong.
I have even loaded the pretrained backbone, initialized the anchors weights to 0.5 as in here, used the same scales for the loss and the gradient updates as in the official YOLO code, but still couldn't make it work... maybe @pfeatherstone is right, and it's not as straightforward as I thought. I couldn't see any weird tricks in the training process in the official repository, though...
The training loss at the beginning goes down quite fast, but then it stays constant and pushes the detection confidences to 0.
I will keep checking, but I am open to suggestions. I think we're close to having this loss in dlib (unless I am making some terrible assumption about how dlib works, or a big conceptual mistake).
The most significant change that I made is having a sigmoid layer before the detector output.
As you may know, the box coordinates are computed like this:

And the objectness and classifier use a (multi) binary log loss, which means that a sigmoid is applied everywhere, except for the width and height outputs.
So, I decided to apply it everywhere, and then I can recover the original width and height outputs, by inverting the sigmoid:
y = σ(x) = 1 / (1 + e^(-x)) → x = log(y / (1 - y))
Then, when I apply the exponential to get the final width and height, it simplifies to just:
y / (1 - y)
Where y is the output after the sigmoid (0 < y < 1, so the training is stable). This allows me to apply the sigmoid everywhere using the built-in cuDNN function, which happens in place, while still being mathematically equivalent. I even made that change to Darknet and reproduced the results on VOC2012, so the problem is not there, I am a bit surprised that nobody is doing it this way...
So, any help or insights would be appreciated :)

Thank you for all this hard work. Unfortunately I don't have time to spend on this at the moment. Maybe you could try a very simple backbone with a single yolo layer and therefore a single yolo loss function. Maybe that will help to debug. Maybe the yolo layer could be at stride 16 and you could train on VOC which is easier than COCO.

Maybe a VGG backbone. You could even train it on darknet first. Then port the weights using your visitor. Then check the forward pass works correctly in dlib. Then go from there trying to restart the training in dlib. Maybe you could re-randomise the last couple conv layers before the yolo layer and train those.

This is why autograd is so awesome. There are no headaches when writing custom modules and custom loss functions.

Thank you for those suggestions :)
I didn't try a simpler backbone, but I tried with a single YOLO output to see if there was a problem with multiple outputs, but that didn't work. I might try that again, since I addressed several problems in the loss function since then.
Frankly, I am not too worried about the gradients, it's all MSE and binary log loss, so quite straightforward. What bothers me is that, at least the objectness confidence should be easy to learn, but it gets pushed to 0 (I tried weighing differently, even though in the official implementation all weights (obj, bbr and class) are set to 1).
I am still motivated to make this work, I hope not to lose the momentum...

Well, after a week of trying stuff, I could not make it work...
At first, I thought it was a problem of weighing the different parts of the loss, then I managed to get the objectness not to go to zero, but the network predicts the same bounding boxes for all images...
I will keep working on it, but @davisking, do you think you can have a look at some point?
I know it's a lot to ask, and I understand it's not a priority.
My main concern is that I am not understanding how dlib works well enough to use multiple tensors in a single loss layer, and I am making a conceptual mistake...
Thank you in advance, and please, don't feel pressured or anything :)

Yeah I’ll look in a bit when I get a chance :)

Thank you :D there's no hurry

@pfeatherstone since you seemed a bit skeptical, here are some results on this implementation trained from scratch on the COCO dataset (I didn't even initialize the input_rgb_image with (0, 0, 0), just the default values from dlib.)

The bounding box regression is not perfect, but it's been training for about 12h only. I think it's showing promising results, even if they are from the training set. I tried it on my webcam, and it also works.
So, @davisking, no need to spend time on this now, I will run some extra tests and prepare this PR for review when I have time (maybe, for the example program, a simpler backbone would be better) :)
I am thrilled to have YOLO detectors working in dlib! As always, thank you for creating this outstanding library :)

That looks awesome. Great work! Do you have some idea of training benchmarks vs darknet or pytorch (ultralytics repos)? It would be interesting to know memory usage when training at say 416x416 and the number of epochs required before plateau'ing.

I'm quite excited by this as this might draw more dlib users and maybe more PRs

That looks awesome. Great work! Do you have some idea of training benchmarks vs darknet or pytorch (ultralytics repos)? It would be interesting to know memory usage when training at say 416x416 and the number of epochs required before plateau'ing.

Thank you!
Frankly, I've never used YOLOv5... I've only used Darknet and recently YOLOR (which is based on Scaled-YOLOv4, which in turn is based on YOLOv5, I think).
I've never liked the experience of those toolkits/repos. Everything is overly complicated to bypass the shortcomings of Python (performance and multi-threading). That is the main reason why I was motivated to add this to dlib, as it has the best user experience, for me... and I won't have to use those anymore.
In terms of memory, I think dlib uses slightly more memory than Darknet, but I can't tell for the others, since I didn't use them with the same backbones...
Regarding the plateauing, it's hard to compare, since each toolkit might normalize the gradient differently (e.g. using the batch size, the layer dimensions, etc.), so the learning rate has different meanings in each toolkit/repo.
Anyway, using the default scheduler in dlib makes me have one less thing to worry about.
I never liked training with fixed number of steps/epochs and decreasing the learning rate at arbitrary positions, which have to be decided beforehand.
That's why, in this PR, I use burn-in at the beginning (not sure if it's needed, though) and then fallback to the default learning rate scheduler.

You might get away without burn-in, but typically with yolo models, gradients are unstable at the beginning. That's why burn-in is typically used. However, in my experience, when using CIOU loss, the gradients are way more stable and I can use a default scheduler. I wouldn't implement CIOU now though. Maybe a future PR.

I think it's ready for review! It's already working, I've tried in on some datasets (dlib cars, VOC2012, COCO2017) and it seems to work reasonably well. So, @davisking, you can have a look when you have time :)

Sweet.  I'll take a look in a little bit.  About adding the image width and height as a metadata field to dlib::image_dataset_metadata::image, yeah, that seems like a good idea given the use case you described.  Which is a pretty important use case :)

Done, I've added a simple example on how one would recompute the anchor boxes. :)

I'll update the other dlib datasets to have width and height annotations so this stuff will work on them too.

I'll update the other dlib datasets to have width and height annotations so this stuff will work on them too.

Great! I made a small program to do that already... I am sure you don't need it, but here it is anyway
https://gist.github.com/arrufat/d5fac2340d2a165d40627cdf338aac76

I'll update the other dlib datasets to have width and height annotations so this stuff will work on them too.

Great! I made a small program to do that already... I am sure you don't need it, but here it is anyway
https://gist.github.com/arrufat/d5fac2340d2a165d40627cdf338aac76
EDIT: Oops, I didn't see that you added that already to imglab...

Yeah no worries :)
I'm reuploading the last of the datasets.  Should be done in a few minutes.

Also, after seeing how you computed the anchor boxes in the example program, maybe we don't need an anchor_box_details struct, and the same can be achieved with point or dpoint.

Maybe.  I haven't gone over the code yet really.  I ran the example and
played around a bit (and it worked great).
But there is something to be said for having explicit types for specific
ideas.  Since it makes it much more clear what the arguments to functions
that use them are about.
On Sat, Jul 10, 2021 at 12:20 PM Adrià Arrufat @.***>
wrote:

Also, after seeing how you computed the anchor boxes in the example
program, maybe we don't need an anchor_box_details struct, and the same
can be achieved with point or dpoint.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-877663678, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPYFR5365SUS2QGUNQ2AWLTXBXN3ANCNFSM46GDQCUA
.

Are you sure about this? Reading the paper:
YOLOv3 predicts an objectness score for each bounding box using logistic regression.
This should be 1 if the bounding box prior overlaps a ground truth object by **more** than any
other  bounding  box  prior.
If  the  bounding  box  prior is  **not**  the  best  but  does  overlap  a  ground  truth  object
 by more than some threshold we **ignore** the prediction. We use the threshold of.5. Our system
only assigns one bounding box prior for each ground truth object.
If a bounding box prior is not assigned to a ground truth object
it incurs **no** loss for coordinate or class predic-tions, only objectness.

So the best match is always used, regardless of IOU : agreed.
I think those matches whose IOU is greater than iou_anchor_threshold, are NOT used in the bbox loss and class loss.

Yes, I am quite confident about this. This part is a new addition from the Alexey's fork:
https://github.com/AlexeyAB/darknet/blob/1be96802a030ae2216c7e19b7523e1e2f0948aaf/src/gaussian_yolo_layer.c#L590-L622
Here @AlexeyAB explains it: https://github.com/AlexeyAB/darknet/issues/7087#issuecomment-753706790.
As I said this part was not present in YOLOv3, but all the new models that achieve state-of-the-art results: Scaled YOLOv4, YOLOv4-CSP, YOLOR, use this setting with a value close to 0.2.

Oh right, I'd never noticed that. I've always just implemented yolo loss as the paper described and it worked fine. Thanks for bringing this to my attention.

As a general discussion, nothing to do with this implementation, I find yolo loss particularly faffy given all the hyper parameters and the enormous number of candidate predictions. I'd had high hopes for DeTR and other direct set prediction models using bipartite matching loss, but they are far too slow to train. It's a real shame coz they are so elegant, easy to write and intuitive. Anyone else dissatisfied with the state of SOTA detection models?

I'd wondered if you could combine bipartite matching with yolo layers. So currently each cell has tree "slots", each one assigned to an anchor. Then matching is done based on best IOU with anchors. Instead of that, make each cell have N "slots", maybe 5 of so, and the matching is done using bipartite matching, which you could do using dlib::max_cost_assignment(). Maybe there would be a lot of overhead calling dlib::max_cost_assignment() thousands of times per batch, but at least you wouldn't have to use anchors anymore and you wouldn't have hyper parameters like trush_threshold and stuff like that. Just a thought. Bye the way, this discussion is not really relevant to the PR, but i'm interested to read other people's thoughts.

Honestly, I haven't given a deep thought to this, I'd like to do some research at some point on how to improve detectors, and being able to implement YOLO from scratch is an essential step for me. I agree on DeTR paper, the approach seems so simple, and with less engineering, but I never got to try it for myself.

@arrufat If you're collating ideas for other detection heads, i've been experimenting with fcos in pytorch and you get pretty decent results and it's super easy to implement. That could be a cool future PR if you're interested.

Hi guys 2 quick questions

Is there any timeline when this will be reviewed/merged?
since its a long read, are thoe yolov4 features already implemented or is it "only" yolov3 ?

Huge pr !

I'll get to it next week.

@Compaile,
Hi! Regarding the second question, it started as YOLOv3 (actually the loss function is almost identical in YOLOv4 and others).
I just added an extra setting that the new YOLO-based state-of-the-art models use, the iou_anchor_threshold, called just iou_threshold in the YOLOv4 repository.
It's disabled by default (set to 1), so you get YOLOv3 if you don't change that setting.
Let me know if what the setting does is clear from the documentation :)

@pfeatherstone https://arxiv.org/abs/2107.08430
I am thinking about trying this out. It has a nice ablation study on the YOLO loss and it's anchor free.
I'm not sure whether we should put this in this PR or even in dlib.
I think it's too early, dlib is more about well known stuff that has survived the test of time (YOLO certainly has).
Maybe we could add a loss_yolox later on, and it doesn't seem that hard to implement on top of this one.
I don't know, these things move so fast... it's hard to know what's worth adding...

Ohh. I'll have a look. Thanks @arrufat

The paper looks interesting.  I wonder if the results they present are a bit lucky. Like when comparing results with other models, +1 AP score could be attributed to a well tuned optimiser and scheduler and may well have nothing to do with the network architecture. Ultralytics repo proves this with yolov3. With no changes whatsoever to the architecture, they got substantially better results than darknet. So I'm not convinced all these improvements can be attributed to detection heads or label assignments. A really fair setup would be to also retrain yolov5 from scratch using their setup, training loops, schedulers, the lot. I still think the most elegant and simplest solution is what DETR did. It's a shame it doesn't really work...

Also looking at the yolox code, it sets batchnorm parameters to unusual values. Like a momentum of 0.3, i have never seen. I don't think that's mentioned in the paper. It's the type of thing you do when you're mini-batching in torch... A few of the layers have specific methods for initialising parameters, rather than using pytorch defaults. So there are obvious engineering tricks which have been applied that hint that a lot of fine tuning has been done, all of that to get a couple AP points over yolov5. So it doesn't seem to be a dramatic improvement over the traditional yolo detection heads and loss layers. Maybe I'm wrong, I don't know.

The biggest take away for me from the paper is decouple heads. The paper suggests it accelerates training which is always desirable, even if the end result is the same.

There is already a thread on darknet about yolox. Alexey is all over it. He claims some of the yolov4 variants have equal if not better performance in terms of latency and AP score. So who knows. It seems to me you trade one faffy mechanism for another and get almost identical results. Yes it's publishable but not ground breaking

Thank you for your insights, it's always good having someone else's point of view on this. Either way, I want to try an anchor-less training with this loss, I'm really curious to see if that works.

Maybe just have one detection per cell. So have a total of 3 anchors, all of them set to 1.

Maybe just have one detection per cell. So have a total of 3 anchors, all of them set to 1.

I am currently trying this:

predict center offsets x, y as in YOLO
predict the height and width directly without any anchors
filter the truth boxes by strides (from FCOS), each stride should be in charge of detecting different sizes:

stride 8: boxes with sides smaller than 64
stride 16: boxes with sides between 64 and 128
stride 32: boxes with larger sides

The modification in the code is pretty straightforward, just multiply the prediction by the stride of that feature map instead of the anchor. Which seems very similar to what you were saying: training with one anchor per stride.
So far, it seems to work

It would be great if the example program trained on COCO and included an AP score function. That way you can directly compare with other frameworks and hopefully convince the user that using dlib isn't a bad shout. I think the COCO labels, as downloaded from the COCO website, not darknet, consists of big JSON files. So a COCO dataset loader in dlib would probably require a JSON parser. We could simply pinch nlohmann's single header JSON library and put that in a 'third party' or 'external' folder.

Oh and a mosaic data augmentation function would also be awesome (sorry if I'm always adding stuff to the wish list). I think that's one of the big contributions from ultralytics yolov3 repo which added +4 AP points without any model architecture changes. With that, and all the other stuff you've done, the example program is pretty close to a fully fleshed object detection training framework.

Actually, I am using this at work, and I already implemented my version of mosaic. It's quite straightforward in dlib:
const dlib::rectangle r(x, y, x + tile_size, y + tile_size);
auto si = dlib::sub_image(canvas, r);
resize_image(image, si);

Then you put that in a loop with 4 iterations which gets one image at a time and provide the correct x and y values at each iteration.

Actually, I am using this at work, and I already implemented my version of mosaic. It's quite straightforward in dlib:
const double scale = 0.5;
const dlib::rectangle r(x, y, x + tile_size, y + tile_size);
auto si = dlib::sub_image(canvas, r);
resize_image(image, si);
for (auto& b : boxes)
    b.rect = translate_rect(scale_rect(b.rect, scale), x, y);

Then you put that in a loop with 4 iterations which gets one image at a time and provide the correct x and y values at each iteration.

Thank you for spending time reviewing it, I've added most of your suggestions (I should improve my documentation-writing skills)...
I left a couple of comments, let me know what you think :)

Sweet. Thanks for another great PR :)

Thank you, it's my pleasure!

Could you compare the accuracy of dnn_yolo with dnn_face (with mmod)?
Using a smaller learning rate works better in dnn_yolo, whereas in mmod starting with 0.1 is the best. I can't explain it.

I am not going to do it, but you can do it if you're interested. Note that the results you get from the example programs are not representative of what's achievable with either technique. They are just “executable documentation” to easily illustrate how to use a certain part of dlib, and they provide a good starting point. However, I doubt the example programs will give good results for any production-ready application, whatsoever.
I guess this is just how it is, YOLO-style detectors are famous for being unstable at the beginning of the training, hence the small learning rate. That being said, the example uses a small batch size, while the MMOD face detector uses a way larger batch size. Increasing the batch size might allow for a larger learning rate. So, I'd say it's hard to compare them, the network architectures and the loss functions are completely different.
What do you mean? If you are talking about the random cropper in the example, you should go and read its documentation. The example only focuses on explaining the newly introduced features. But if something is not clear, we can try to improve the documentation. However, when I made that example, I added (too) many options so that people could visualize the data augmentation, intermediate results, mAP, etc… I think that example is a bit too complicated, but to get sensible results with the YOLO loss, you usually need to do more complicated stuff. In that sense, I really like the MMOD loss: there's way less fidling.

Thank you very much for your explanations.
3. Yes I mean random cropper. You have changed the default parameters, for example, .set_translate_amount(0.5); (0.5 is too high?) set_background_crops_fraction(0) (it is 0.5 in the default). I wonder the reasons why you changed them in such a way.

Well, I was trying the COCO dataset when I was preparing the example, and in COCO, you are expected to detect cropped objects, so that's why I set a relatively big translation value. Regarding the background fraction, all other training methods I've seen don't use object-empty images, so that's why I did it. I didn't check that those were the optimal parameters, they are very task-specific.

Hi Arrufat, I have two more questions.

By default options.iou_anchor_threshold = 0.2. What about setting it to
0.1?. (decreases the false positive rate?)
How can I interpret the "average loss: 0.232008"? Is it good learning,
or not? Does it mean that 23% of the time, it can't find the object?
Thanks.

On Tue, Mar 8, 2022 at 3:41 AM Adrià Arrufat @.***>
wrote:

Well, I was trying the COCO dataset when I was preparing the example, and
in COCO, you are expected to detect cropped objects, so that's why I set a
relatively big translation value. Regarding the background fraction, all
other training methods I've seen don't use object-empty images, so that's
why I did it. I didn't check that those were the optimal parameters, they
are very task-specific.
—
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-1061288117, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AW5C34MS5M32GCHBKTXNMSTU62O3RANCNFSM46GDQCUA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you commented.Message ID:
@.***>

Hi @carkaci,

By default options.iou_anchor_threshold = 0.2. What about setting it to 0.1?. (decreases the false positive rate?)

By decreasing the yolo_options::iou_anchor_threshold you are telling the network that any anchor that has an IoU higher than that value is also responsible for detecting that object. So, what you are effectively doing, is increasing the number of updates to the network (most of these detections will be suppressed by the NMS, anyway, but you are updating the network more often).
That's why this setting is usually meant to be used with high-capacity networks.
It's not all good, though. If you set this value to anything lower than 1, you are going to lose the ability to detect different things in the same grid cell. But maybe your dataset does not have many objects overlapping each other, so that's fine.

How can I interpret the "average loss: 0.232008"? Is it good learning, or not? Does it mean that 23% of the time, it can't find the object? Thanks.

I am afraid you can't interpret the loss like this, since it's just computed as the squared norm of the gradient in this code (and in the official Darknet YOLO version, too).
The loss will scale with the number of detection heads, the number of classes and also the number of grids cell in each feature map (image resolution). I've seen that YOLOv5 scales all these as if we were always using 3 outputs for with a fixed input size so that we can compare the losses between detectors. However, there's no easy way to know what it really means.

Thank you for your quick response.

) By decreasing the yolo_options::iou_anchor_threshold you are telling
the network that any anchor that has an IoU higher than that value is also
responsible for detecting that object.
Does it implicitly force the anchors to lower than the value of
yolo_options::iou_anchor_threshold
, that may result in lower false positives?
2.) What about setting options.iou_ignore_threshold  (by default it is 0.7) to
0.9 ?

On Sun, Jul 10, 2022 at 9:26 AM Adrià Arrufat @.***>
wrote:

Hi @carkaci https://github.com/carkaci,

By default options.iou_anchor_threshold = 0.2. What about setting
it to 0.1?. (decreases the false positive rate?)

By decreasing the yolo_options::iou_anchor_threshold you are telling the
network that any anchor that has an IoU higher than that value is also
responsible for detecting that object. So, what you are effectively doing,
is increasing the number of updates to the network (most of these
detections will be suppressed by the NMS, anyway, but you are updating the
network more often).
That's why this setting is usually meant to be used with high-capacity
networks.
It's not all good, though. If you set this value to anything lower than 1,
you are going to lose the ability to detect different things in the same
grid cell. But maybe your dataset does not have many objects overlapping
each other, so that's fine.

How can I interpret the "average loss: 0.232008"? Is it good
learning, or not? Does it mean that 23% of the time, it can't find the
object? Thanks.

I am afraid you can't interpret the loss like this, since it's just
computed as the squared norm of the gradient in this code (and in the
official Darknet YOLO version, too).
The loss will scale with the number of detection heads, the number of
classes and also the number of grids cell in each feature map (image
resolution). I've seen that YOLOv5 scales all these as if we were always
using 3 outputs for with a fixed input size so that we can compare the
losses between detectors. However, there's no easy way to know what it
really means.
—
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-1179665932, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AW5C34LNXHVBNDYIMRFH763VTJULBANCNFSM46GDQCUA
.
You are receiving this because you were mentioned.Message ID:
@.***>

My take is that, since you are using multiple anchors for the same object, if their shapes happen to be different (because the regression didn't work as expected) then you might have more false positives.
If you set the yolo_options::iou_ignore_threshold to 0.9, then all predictions that have an IoU with the truth below 0.9 will experience loss, forcing them not to detect the object. I like to think of it as a heatmap. The grid that contains the center will always predict the object, and the surrounding ones, only if they generate good predictions. What a good prediction means depends on the IoU ignore threshold. The lower the value, the larger the heatmap, the higher the value, the closer to a delta function. Note that if you set the IoU ignore threshold too high, maybe even the centered prediction will have an IoU below that value and will experience loss...

Presumably everyone has seen the two new yolov7 repos. I love the naming wars. Lots of cool new stuff.

@pfeatherstone yes, I've already implemented the yolov7 backbone, but I am debugging it to make sure there are no mistakes.

Just to add some more feedback.
I played around a lot with your yolo implementation. First of all huge thanks to build it in dlib.
The darknet yolo code is so hard to manage and adjust.
I tried some samples with the base implementation, my own and also recently your dlibml/yolo-object-detector.
However i cant get it to work- it means i can, it works but it isn't as good as if i do my usual yolov5 c++ pipeline.  The training it self around the nn should be much better (better pre processing, better load balance etc). but the network itself is just always quite a bit worse and so is the map for example on coco.
Mostly it has tho many false positives but also the conversion itself during training is way slower (which might be ok since its not deterministic and the sdg solver works different)
I still suspect there are some minor bugs in the yolo implementation but i wasnt able to pin point them yet exactly. I have some vague ideas but those might be wrong.
Have you been able to produce map values close to the paper?

i also tried many different datasets (the dlib ones, coco, my own) results in regards of comparing with the  https://github.com/AlexeyAB/darknet seems to be the same

I also have some gpu resources i can use to help you guys testing.
(currently the biggest setup is 4x3080ti).
So my questions would be:

what is your best map on the coco data
could you provide your converted training and testing xml data for the coco2017 set?
just to make sure they are identical
(i think thats the best dataset to validate it on since we can bench vs other Framework)

Great job. I am very excited to see yolo-v7 in dlib. Thank you for your valuable effort.
Sent from my iPhone

On Jul 10, 2022, at 13:46, Adrià Arrufat @.***> wrote:

@pfeatherstone yes, I've already implemented the yolov7 backbone, but I am debugging it to make sure there are no mistakes.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

@VisionEp1 Here's the COCO dataset in XML format: http://dlib.net/files/data/dlib_coco_annotations_trainval2017.tar.bz2
I also haven't been able to reproduce the training results on COCO using this implementation with YOLOv5l.
In particular, using this dataset and this code, I obtained these results:
overall performance
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.330
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.543
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.348
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.197
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.363
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.391
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.274
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.433
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.457
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.297
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.495
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.548

I used 4 V100 (32 GiB each), but I no longer have access to those :(
I can't see any significant difference between this implementation and the official one from Alexey (YOLOv4).
There are some differences between this one and YOLOv5. Notably, the scaling factors between the outputs in YOLOv5.
You are welcome to experiment with the code I shared, and suggest improvements.

Also, I have some improvements that I've been testing for YOLO over the last few months, you can check them here: https://github.com/arrufat/dlib/tree/yolo-improvements
Notably:

focal loss for objectness and classification
smoothed l1 loss for regression (to avoid having unbounded gradients)
use the truth.detection_confidence as a positive weight to make up for the class imbalance in the dataset
set iou_anchor_threshold to 0 to automatically find a good threshold based on ATSS.

thanks!
33 to 49 is quite a margin :/..  I also didnt see to many differences but then again the bulky c code from Alexey is very hard to follow (imo). and clearly science focused :).
Even yolov3 had higher map values then the current dlib implementation. So i guess there has to be some small weird bug somewhere.  I also tried many different dlib yolo architectures which didnt make a difference. My current guess is eigther:
a) some small bug in the yolo loss layer
b) the dlib solver has somehow a hard time with those object detection type loss functions.(i also had bad results with the previous dlib object detectors and spent many hours on it without major improvements (cnn mmod), but excellent results with the dlib classification models).
before i start another run with your xml files. would you recommend the default parameters in your training code. (i would put batch and workers high enough but other then that)
Feel free if you have any ideas how i could speed up my search
i will keep you updated

Since the yolov4 training tends to be super unstable i don't know if its better for me to use as many cool new features to get the best map, or start with the most base version (for example the previous more yolov3ish setup) and try from there. What do you think?
Also if i shall train a specific model for example the yolov7 feel free to tell me and i'll run the training.

From memory, the ultralytics repos have a special SGD optimizer where weights are grouped and each group has slightly different settings. so I think biases have momentum, no weight decay and quite high learning rate. Convolutional weights have lower learning rate and some weight decay, and another group for batchnorm weights which had some other slightly different settings. And i think all groups had nerestov variant on.  Furthermore, the yolo backbones were pretrained on imagenet.

Also the learning rate scheduler has a slow burning, which I'm pretty sure @arrufat's code already does, but the momentum is inversely cycled as well.  Maybe that would be useful for initializing yolo networks.

I dont think the pretraining is the issue. In yolov4 its mostly used to make training
a) faster b) stable. But does not change a lot in the accuracy. I trained many networks in the other repo with and without pretrained weights. Of course your findings might differ.
You can set a burn-in in @arrufat code. I cant say anything about the ultralytics repos i didnt use them.  I am completely with you that those adjustment in various ways can improve map.
But when we discussed on the other repos those things always where like +-0.5 -2% map. not something like 20%+

Sorry, I gave you the wrong configuration.
The darknet yolo works with subdivision of batches and is very unstable with smaller batches.
You think that might be the reason? (since i will use 12gb ram gpus i will start training now with even lower batches and compare the results)-

It could be, however, I've also tried with smaller YOLOv5 variants so that I could increase the batch size (dlib uses a lot of VRAM, for some reason) but same tendency.

alright thanks. just another quick question, what did you use to visualize your training loss like in the last picture.
That way i could compare it better with my results

Since the yolov4 training tends to be super unstable i don't know if its better for me to use as many cool new features to get the best map, or start with the most base version (for example the previous more yolov3ish setup) and try from there. What do you think?
Also if i shall train a specific model for example the yolov7 feel free to tell me and i'll run the training.

I would not use my YOLOv7 implementation, it's not ready for prime time yet.
The YOLOv3 implementation and YOLOv5 are properly implemented (they have the exact same architectures as the original implementations) so I would go with those. Probably with YOLOv5, since you can choose a smaller variant to make the training faster :)
In case you're curious about YOLOv7, here's the current state of it (I am sure it has bugs, since I didn't test it yet)
yolov7.zip
yolov7.pdf

thanks a lot again!. i will start with the yolo5 and report my findings.

alright thanks. just another quick question, what did you use to visualize your training loss like in the last picture. That way i could compare it better with my results

I used this gnuplot script (it's in the repo as well)
#!/usr/bin/env gnuplot

reset session

set term png size 1800,960
set output 'loss.png'
set grid

set style line 1 lc rgb '#C00000' lt 1 lw 2 pt 7 pi -1 ps 0.5
set style line 2 lc rgb '#00C000' lt 1 lw 2 pt 7 pi -1 ps 0.5
set style line 3 lc rgb '#0000C0' lt 1 lw 2 pt 7 pi -1 ps 0.5

set multiplot title 'YOLO'

set size 0.5,0.5
set origin 0,0.5
set yrange [0:20]
set title 'loss'
set xlabel 'step'

plot "< awk '$1==\"step#:\" {print $2, $8}' training.log" using 1:2 title 'train' w lines ls 1

set size 0.5,0.5
set origin 0.5,0.5
set title 'learning rate'
set xlabel 'step'
set autoscale y
plot "< awk '$1==\"step#:\" {print $2, $5}' training.log" using 1:2 title '' w lines ls 1

set size 0.25,0.5

set key right bottom
set origin 0,0
set title<EMAIL_ADDRESS>set yrange [0:1]
set xlabel 'epoch'
plot "< awk '$1==\"EPOCH\" {print $2, $3}' training.log" using 1:2 title '' w linespoints ls 1

set origin 0.25,0
set title 'macro'
plot "< awk '$1==\"EPOCH\" {print $2, $4}' training.log" using 1:2 title 'precision' w linespoints ls 3, \
     "< awk '$1==\"EPOCH\" {print $2, $5}' training.log" using 1:2 title 'recall' w linespoints ls 2, \
     "< awk '$1==\"EPOCH\" {print $2, $6}' training.log" using 1:2 title 'f1-score' w linespoints ls 1

set origin 0.5,0
set title 'micro'
plot "< awk '$1==\"EPOCH\" {print $2, $7}' training.log" using 1:2 title 'precision' w linespoints ls 3, \
     "< awk '$1==\"EPOCH\" {print $2, $8}' training.log" using 1:2 title 'recall' w linespoints ls 2, \
     "< awk '$1==\"EPOCH\" {print $2, $9}' training.log" using 1:2 title 'f1-score' w linespoints ls 1

set origin 0.75,0
set title 'weighted'
plot "< awk '$1==\"EPOCH\" {print $2, $10}' training.log" using 1:2 title 'precision' w linespoints ls 3, \
     "< awk '$1==\"EPOCH\" {print $2, $11}' training.log" using 1:2 title 'recall' w linespoints ls 2, \
     "< awk '$1==\"EPOCH\" {print $2, $12}' training.log" using 1:2 title 'f1-score' w linespoints ls 1
unset multiplot

pause 60; refresh; reread

It will generate the loss.png image every 60 seconds, then, if you put his HTML file next to it, you can have the poor man's version of Tensorboard:
<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="refresh" content="60" >
        <title>YOLO Training</title>
    </head>
    <body>
        <center>
            <img src="loss.png"/>
        </center>
    </body>
</html>

You could even run:
python -m http.server <port>

And then connec to localhost:. The port is optional and it defaults to 8000. I usually do this to monitor a training on a remote machine. As I said, the poor man's version of Tensorboard :sweat_smile:

Hi Arrufat. I have used your candidate yolov7.h. However, even for
"--batch-size=1" (RTX3060
https://www.hepsiburada.com/lenovo-legion-5-intel-core-i7-11800h-16gb-1tb-ssd-rtx3060-freedos-173-fhd-tanabilir-bilgisayar-82jm0012tx--p-HBCV00000VZJFB?magaza=Hepsiburada-6GB
VRAM), I am getting an insufficient memory error.
On Sun, Jul 10, 2022 at 3:53 PM Adrià Arrufat @.***>
wrote:

Since the yolov4 training tends to be super unstable i don't know if its
better for me to use as many cool new features to get the best map, or
start with the most base version (for example the previous more yolov3ish
setup) and try from there. What do you think?
Also if i shall train a specific model for example the yolov7 feel free to
tell me and i'll run the training.
I would not use my YOLOv7 implementation, it's not ready for prime time
yet.
The YOLOv3 implementation and YOLOv5 are properly implemented (they have
the exact same architectures as the original implementations) so I would go
with those. Probably with YOLOv5, since you can choose a smaller variant to
make the training faster :)
In case you're curious about YOLOv7, here's the current state of it (I am
sure it has bugs, since I didn't test it yet)
yolov7.zip https://github.com/davisking/dlib/files/9079077/yolov7.zip
yolov7.pdf https://github.com/davisking/dlib/files/9079082/yolov7.pdf
—
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-1179723488, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AW5C34LP2Z7PN7VAEEWNBN3VTLBVTANCNFSM46GDQCUA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hi Arrufat. I have used your candidate yolov7.h. However, even for "--batch-size=1" (RTX3060 https://www.hepsiburada.com/lenovo-legion-5-intel-core-i7-11800h-16gb-1tb-ssd-rtx3060-freedos-173-fhd-tanabilir-bilgisayar-82jm0012tx--p-HBCV00000VZJFB?magaza=Hepsiburada-6GB VRAM), I am getting an insufficient memory error.

Yes, I told you there were bugs.

Thank you everyone for the hard work! Not easy doing all of this in our free time, even harder to convince employer to do this at work. I'm excited for all of this. I do believe it will attract more users to dlib and therefore more PRs and more goodies.

Just a quick update i attached a part of my training loops as an example
training.log
.\train.exe --name yolov5l-c [4.log](https://github.com/davisking/dlib/files/9121092/4.log) oco2017 --size 512 --batch-gpu 6 --learning-rate 0.01 --min-learning-rate 0.0001 --gpus 4 --epochs 300 --cosine --iou-anchor 0.2 --iou-ignore 0.7 --gamma-obj 0.5 --gamma-cls 1.0 --mosaic 0.5 --angle 3 --min-coverage 0.5 --hsi 0.3 0.2 0.1 --scale 0.5 --shift 0.2 --perspective 0.01 --test-period 30 --mixup 0.01 X:\Datasets\coco2017  | tee -Append -FilePath "training.log"
I run the code on windows just fyi.
So the map is still increasing.
Whats interesting is when training yolo models you usually get to around 80% of the accuracy quite fast and then it takes a lot of time to gain the last % of map (which makes total sense). But already this area is at a much higher map for the same dataset as we have it with the dlib implementation.
Another finding is that (as u can also see in the loss).
images.zip
It performs worse on the train data without augmentation in testing than on the actual validation data  (i attached some images). Which is a bit wierd and i have to check if the data augmentation is buggy?
I get that the loss value is better on the testdata (no agumentation vs with) but when both are loaded without the train data should perform way better.
it might also be an idea to get start with a very basic v3 model to mimic the results and the training and see how it compares.
So not any solutions yet just sharing my findings in between.

Just a thought, it might be a good idea to benchmark against other 'vanilla' repositories that haven't optimised the training loops and the image augmentation, such as frgfm/Holocron. If you get similar performance then it's likely there is nothing wrong with the loss function but more an issue with optimising training loops, augmentation, hyper-parameter tuning and things like that.

i might also run some training with very low batch sizes on the alexay repo just to make sure thats not the issue.
Another thing which i need to try  is changing stuff like: multi gpu vs 1 gpu, maybe different cuda versions etc.
I had hardware related bugs in the past on many different ml freamworks and sometimes they are hard to detect.
However that doesnt help if neither of us can reproduce map values which are close to the other repos. Did any of you try a small yolo network with 1 gpu only to test this as a possible cause?
I also considered a more "bruto" force method, like the ones used in the automl context (but not on the architecture) where i create a tiny yolo net and load in pretrained weights and run loads of different Trainings regarding the most common settings.
Obviously this might also lead to nothing but do you think the following could give some insights:
I define the most common settings mainly the yolo options, the data augmentation options and scheduler options
with a set of plausible values. Then i start training with random datapoints out of the possible configurations and use the dlib tools to get some insights what the best combinations might be.
Do you have a ready dlib config which matches any known tiny yolo networks exactly?
for example https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny.cfg
so following the @pfeatherstone approach i can stay as close as possible to one of those networks.
i think the tiny ones make more sense(even tho currently i train the normal one) since we can get some insights faster

@VisionEp1 So @arrufat created a whole bunch of network definitions. https://github.com/dlibml/dnn/blob/master/src/detection/yolov5.h contains yolov5_n which is small.

@VisionEp1 I am pretty confident about those YOLOv5 definitions, I see no need on trying with the YOLOv3 or YOLOv4 ones.
From my experience, on an internal dataset, the training code I shared achieves an mAP of almost 90%, while the test reaches only about 66%.
If I train the same YOLOv5l model in PyTorch on that dataset, I get about 74% mAP on the test set. So, for me, it was a problem of the network not generalizing well enough. Sadly, these days, I won't have much time to play with it.

Also I would look at the optimizer. @arrufat correct me if i'm wrong but the optimizer is just vanilla SGD. The pytorch repos group weights into 3 different groups depending on what requires a weight decay and biases take higher lr I think. That could make a difference.

Yes, what I am trying to say is that I don't think the performance drop comes from the network architecture. But you don't have to believe me, in the official dlib example, that network is exactly the same as the one presented in the YOLOv3 paper: I even tried at some point to fine-tune from the official Darknet53 weights.
I would say that the main differences are:

upscaling: dlib uses bilinear, yolos use nearest neighbors
input image: YOLOv3 uses a 0-mean initialized input layer, in dlib you can achieve this by instantiating the network as darknet::yolov3_train_type net(options, input_rgb_image(0, 0, 0));

So, you could use that network in the training code I shared with you, however, I don't think that'll make a difference.

Oh I agree the network definitions are correct. I would use any architecture, even something that isn't official yolo model, maybe with only 1 head, and experiment with different training setups.

I completely agree I was not trying to find bugs where no bugs are.  I am very sorry if that came across wrong
Thanks for the answers I will go ahead and try different training setups

@VisionEp1 no, you are right to question everything, I don't even trust the 1-year-ago version of myself :P

People do this already but using genetic algorithms to find optimal parameters. i don't know how dlib::find_min_global() fairs against genetic algorithms, but they have the same goal.

A genetic algorithm will be much worse.

@VisionEp1 @carkaci I fixed the YOLOv7 version, I've started a training a couple of hours ago, and it seems to work... Feel free to try it out: https://github.com/dlibml/dnn/blob/master/src/detection/yolov7.h

Thank you, I will test it.
On Mon, Aug 15, 2022 at 6:34 PM Adrià Arrufat @.***>
wrote:

@VisionEp1 https://github.com/VisionEp1 @carkaci
https://github.com/carkaci I fixed the YOLOv7 version, I've started a
training a couple of hours ago, and it seems to work... Feel free to try it
out: https://github.com/dlibml/dnn/blob/master/src/detection/yolov7.h
—
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-1215174121, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AW5C34O5BURD6MP5CND5QZLVZJPQ5ANCNFSM46GDQCUA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I have recently started a training. Everything seems Ok.
On Mon, Aug 15, 2022 at 6:36 PM Abdurrahman Carkacioglu @.***>
wrote:

Thank you, I will test it.
On Mon, Aug 15, 2022 at 6:34 PM Adrià Arrufat @.***>
wrote:

@VisionEp1 https://github.com/VisionEp1 @carkaci
https://github.com/carkaci I fixed the YOLOv7 version, I've started a
training a couple of hours ago, and it seems to work... Feel free to try it
out: https://github.com/dlibml/dnn/blob/master/src/detection/yolov7.h
—
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-1215174121,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AW5C34O5BURD6MP5CND5QZLVZJPQ5ANCNFSM46GDQCUA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I have recently started a training. Everything seems Ok.

Yes, me too. I started yesterday and after 22 epochs, I got an mAP of 36.91%... Let's see how it goes, but I have the feeling it learns faster than YOLOv5 equivalent models.

@arrufat Do you have an update on training? I'm very interested. Do also have the same curves for yolov5, v4 and v3? It's a lot to ask I know :) ! If they're all similar, then it will probably indicate that the augmentation, training recipes and things like that are what contribute the most to performance.

@pfeatherstone I do not have curves for all those models. Currently, I only have them for YOLOv5m and YOLOv7 under same training settings, except for the batch size, 48×2 vs 14×2, but using the same learning rate scheduler, so YOLOv7 has way more steps per epoch.
 Thanks a lot

Maybe we can continue the discussion here: https://github.com/dlibml/yolo-object-detector/discussions/2

Like Arrufat, I also have the feeling it learns faster than YOLOv5.  I set
"--patience=200000", that is why training takes 5-6 days. I will let you
know the result when finished.
On Tue, Aug 16, 2022 at 10:24 PM pfeatherstone @.***>
wrote:

@arrufat https://github.com/arrufat Do you have an update on training?
I'm very interested. Do also have the same curves for yolov5, v4 and v3?
It's a lot to ask I know :) ! If they're all similar, then it will probably
indicate that the augmentation, training recipes and things like that are
what contribute the most to performance.
—
Reply to this email directly, view it on GitHub
https://github.com/davisking/dlib/pull/2376#issuecomment-1217064825, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AW5C34LLIM67EVGNGCVROW3VZPTE5ANCNFSM46GDQCUA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Like Arrufat, I also have the feeling it learns faster than YOLOv5.  I set
"--patience=200000", that is why training takes 5-6 days. I will let you
know the result when finished.

Oh, --patience uses the number of epochs, not the training steps in the YOLO repository. You can set it to a floating point number, though, but 200000 is definitely way too high.

@carkaci @arrufat
if possible can we move it to your discussion post?
so we have all the "insights" over there as well


fix(helm): Regenerate go types from proto

What this PR does / why we need it:
The checked in types generated from proto files is not in sync with code that is generated with the documented version of protoc and the golang generator. This PR updates the go types to bring this back in sync. It also fixes some issues that arise as a result of updating the generated types.
Fixes: #5658
Special notes for your reviewer:
The current code uses DeepEqual to compare Release objects in some tests. This no longer works with the updated go types as protobuf adds some extra fields. This PR replaces the DeepEqual with a shallow equal for release objects that only checks some of the fields. I think this is sufficient for the tests and I'm not sure we want an equal function that traverses every object referenced from a release. But I'm open to suggestions on how we want to do this.
If applicable:

[x] this PR contains documentation
[x] this PR contains unit tests
[x] this PR has been tested for backwards compatibility

Looking forward to seeing this merged :).


Alamofire Embedded Framework Support on iOS 8.0

ld: embedded dylibs/frameworks are only supported on iOS 8.0 and later (@rpath/Alamofire.framework/Alamofire) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@LightD Thanks!

i just created a test project you can look here https://github.com/leviyehonatan/cocoapods-lib-swift and it targets 7.1 and i t works, why won't alamofire target 7.0? because even though compiler supports it, the store doesn't wupport it?

@leviyehonatan It won't pass Itunes Connect Validation

Hi guyz
I have got an official word from Apple "There is no way to use Embedded framework code in iOS7".
Although if your embedded framework code is only used in iOS8 there seems to be a way of conditional loading of embedded framework in iOS8.
Thanks
Neeraj

Check this: https://github.com/devxoul/CocoaSeeds


accounting code: add total reward units to indexer schema; migration

This is not necessary, there is an easier way to compute the rewards pool balance.


Request for Bot Adding

Discord Server ID:347709902715748353
Discord Server Name: EXO  Enforcers
Room ID for newsfeed:<PHONE_NUMBER>90865152
My DiscordID: @Rockswift#3124
Newsfeed: Would be great to get Elite Dangerous Community Goal Weekly information sent to this channel.

I set this up with galnet (since that includes CG info), the EliteCG reddit and the EliteCG FDev forum.

Brilliant, thank you
Regards Daniel
On 11 Mar 2018, at 23:55, Eric Eisenhart <notifications<EMAIL_ADDRESS>wrote:
I set this up with galnet (since that includes CG info), the EliteCG reddit and the EliteCG FDev forum.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/freiheit/discord_feedbot/issues/62#issuecomment-372161202, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ahg4G3TnmBCNaJ2T1dzn5zU24X489fyBks5tdblfgaJpZM4RUglj.


[Cosmos] beta version release since we need api review

didnt realize we would need an api review, will release in beta

API change check
API changes are not detected in this pull request.


NSErrorFailingURLStringKey: reports.crashlytics.com

[REQUIRED] Step 1: Describe your environment

Xcode version:  Not using Xcode, running Xamarin with MS Virtual Studio 2019 (16.6.2)
Firebase SDK version: Xamarin.Firebase.iOS.Core (6.6.6)
Firebase Component: Xamarin.Firebase.iOS.Crashlytics (<IP_ADDRESS>)
Component version: <IP_ADDRESS>
Installation method: NuGet Package Manager

[REQUIRED] Step 2: Describe the problem
While running my app, it's constantly looping a failed URL attempt:
2020-06-12 12:05:30.044721-0700 ERPXamarin.iOS[30657:2178586]
Task <75F8446B-3124-4986-B02B-1158F1AA798E>.<507> finished with error [-1] Error Domain=NSURLErrorDomain Code=-1 "unknown error"
UserInfo={
	NSErrorFailingURLStringKey=https://reports.crashlytics.com/sdk-api/v1/platforms/ios/apps/com.perpetua.enroutepro3/reports,
	NSErrorFailingURLKey=https://reports.crashlytics.com/sdk-api/v1/platforms/ios/apps/com.perpetua.enroutepro3/reports,
	_NSURLErrorRelatedURLSessionTaskErrorKey=(
    		"BackgroundUploadTask <75F8446B-3124-4986-B02B-1158F1AA798E>.<507>"
	),
	_NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <75F8446B-3124-4986-B02B-1158F1AA798E>.<507>,
	NSLocalizedDescription=unknown error
}

The url reports.crashlytics.com does not appear in my code.
When I search this repo for reports.crashlytics.com I'm seeing it in "firebase-ios-sdk/Crashlytics/Shared/FIRCLSConstants.m".
When trying https://reports.crashlytics.com in a browser it returns a 404.

Steps to reproduce:
Just running my code with these 2 Firebase libraries installed causes the issue.

Looks like something went wrong in the background task to upload Crashlytics reports, but it's a generic (unknown) error. Are you using background tasks in your app at all?

No, only foreground tasks. We are only using Firebase for the crash reporting. We just converted from Fabric, and trying to do only as much as required for error reporting.
            //Old crash reporting configs
            //string apikey = "[...]";
            //MonoTouch.Fabric.Crashlytics.Crashlytics.StartWithAPIKey(apikey);

            //Firebase configs
            var configuration = Configuration.SharedInstance;
            App.Configure();  //<-- Including this line causes the the issue.
            Crashlytics.Configure();

We have the same issue. We've setup a new project a few days ago. And we also configure Crashlytics for Xamarin.iOS.

Are either of you proxying your device's internet traffic?

No proxying. I've tried to enable Background Mode for iOS, but it didn't help.

This is the error I get:
An error occurred on the xpc connection to setup the background session: Error Domain=NSCocoaErrorDomain Code=4097 "connection to service on pid 0 named com.apple.nsurlsessiond" UserInfo={NSDebugDescription=connection to service on pid 0 named com.apple.nsurlsessiond}
An error occurred on the xpc connection to setup the background session: Error Domain=NSCocoaErrorDomain Code=4097 "connection to service on pid 0 named com.apple.nsurlsessiond" UserInfo={NSDebugDescription=connection to service on pid 0 named com.apple.nsurlsessiond}
Task <1F7A6FB0-F0F5-4761-A45E-724D9FE78F7B>.<18> finished with error [-1] Error Domain=NSURLErrorDomain Code=-1 "unknown error" UserInfo={NSErrorFailingURLStringKey=https://reports.crashlytics.com/sdk-api/v1/platforms/ios/apps/com.myapp/reports, NSErrorFailingURLKey=https://reports.crashlytics.com/sdk-api/v1/platforms/ios/apps/com.myapp/reports, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "BackgroundUploadTask <1F7A6FB0-F0F5-4761-A45E-724D9FE78F7B>.<18>"
), _NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <1F7A6FB0-F0F5-4761-A45E-724D9FE78F7B>.<18>, NSLocalizedDescription=unknown error}

Ok. Referencing the Entitlements.info file for the iPhoneSimulator configuration fixed the issue for me (previously the file is referenced only for the iPhone configuration).
The content of the file is the following:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>keychain-access-groups</key>
	<array>
		<string>com.myapp</string>
	</array>
</dict>
</plist>

Are you able to reproduce this in a non-Xamarin setup? It seems like it may be a generalized entitlements issue.
Though it is definitely strange that missing a keychain access group entitlement is causing url session failures.

It was the same issue for us. The problem only happened on the simulator and there was no reference to the entitlements in the simulator section of the .csproj file.
So this fixed it for us:
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|iPhoneSimulator' ">
    [...]
    <CodesignEntitlements>Entitlements.plist</CodesignEntitlements>
    [...]
</PropertyGroup>

Our project is not set up for non-Xamarin project build, so I can't answer your question on that front.

I'm going to close this issue since it seems to be a Xamarin-only bug with a consistent and non-obtrusive workaround, but feel free to file a new issue if you run into any more Xamarin problems.


Fix POS Example in READEME

Fixes https://github.com/NaturalNode/natural/issues/328

Thanks! Is similar to #342 but repairs more issues with paths.
Hugo

Didn't see that one there, yeah this one does both

thanks!


403 /atm not working

The cookie introduced during November has now expire on session.
I don't know a way to retriever the cookie for every post request.
Maybe the only way is to simulate browser interaction using selenium or similar tools.

This is open to help from anyone.
modules/atm_feature.py involved.

I'll close for what said here https://github.com/MasterCruelty/robbot/issues/114#issuecomment-2038125292


lazer guy is out

i did a few things and did stuf and i got this yay

@haywardherlihy Need help?
Anything I'm here, you can call me :)

I think this is a really interesting remix Hayward! Nice work. There seem to be some little bugs though which can confuse the game play. Is the objective to hit the target with the laser or to stand on top of it?
Maybe if you wrote out what the different game mechanics are we could help you iron out the code and refactor it to be more clear. The level design seems really good and I love how you made laser tag your own by changing the laser shooter into the player and by moving in increments of two.

Interesting game, I love it! The concept is really cool and indeed an awesome remix.
However, there seems to be errors popping up and one of the levels does not contain the player.
Polish it up, and I'd say it's ready to merge! Maybe also add a few more levels while you're at it. :D

Hey @haywardherlihy !
Fix bugs, if you need help just call :)
After that we can reevaluate.


Fixed gather method to work with distributed training in metrics.py and loss.py

Description

In the DDP branch, the gather method uses a .get_state() method that does not exist in the torch_runstats package. I implemented a simple work around to directly access the state of the running statistics (_state) and the number of samples (_n).
In the metrics.py version of gather, I also explicitly made sure the tensors are on the same device prior to accumulating the state.
Motivation and Context

Resolves: #449
How Has This Been Tested?

I tested my changes by comparing the output metrics from minimal.yaml and minimal_distributed.yaml.
aspirin_pr_test.txt
distributed_aspirin_pr_test.txt
Training time with 2 GPUs (Quadro RTX6000) was 248 seconds with
minimal_distributed.txt
:
  Train      #    Epoch      wal       LR       loss_f         loss        f_mae       f_rmse
! Train              50  248.091     0.01      0.00573      0.00573         1.76          2.4
! Validation         50  248.091     0.01      0.00808      0.00808         2.01         2.85
Wall time:<PHONE_NUMBER>005127
! Best model       50    0.008
! Stop training: max epochs
Wall time:<PHONE_NUMBER>4859304
Cumulative wall time:<PHONE_NUMBER>4859304

Training time with 1 GPU (Quadro RTX6000) was 566 seconds with
minimal.txt
:
  Train      #    Epoch      wal       LR       loss_f         loss        f_mae       f_rmse
! Train              50  566.047     0.01      0.00684      0.00684         1.95         2.62
! Validation         50  566.047     0.01      0.00963      0.00963         2.23         3.11
Wall time:<PHONE_NUMBER>491964
! Stop training: max epochs
Wall time:<PHONE_NUMBER>825745
Cumulative wall time:<PHONE_NUMBER>825745

I tried another training attempt with 1 GPU
aspirin_pr_test_t2.txt
:
  Train      #    Epoch      wal       LR       loss_f         loss        f_mae       f_rmse
! Train              50  461.492     0.01      0.00684      0.00684         1.95         2.62
! Validation         50  461.492     0.01      0.00963      0.00963         2.23         3.11
Wall time:<PHONE_NUMBER>860399
! Stop training: max epochs
Wall time:<PHONE_NUMBER>779209
Cumulative wall time:<PHONE_NUMBER>779209

Using the state-run runstats branch
minimal_distributed_fixed_runstats.txt
:
  Train      #    Epoch      wal       LR       loss_f         loss        f_mae       f_rmse
! Train              50  258.864     0.01      0.00573      0.00573         1.76          2.4
! Validation         50  258.864     0.01      0.00808      0.00808         2.01         2.85
Wall time:<PHONE_NUMBER>475767
! Best model       50    0.008
! Stop training: max epochs
Wall time:<PHONE_NUMBER>095519
Cumulative wall time:<PHONE_NUMBER>095519

The configs were setup such that the distributed_batch_size / world_size = batch_size. Not sure what happened between the first and second single gpu test, but perhaps I should repeat the trainings a few times to compare the loss and wall times.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds or improves functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation improvement (updates to user guides, docstrings, or developer docs)

Checklist:

[x] My code follows the code style of this project and has been formatted using black.
[ ] All new and existing tests passed, including on GPU (if relevant).
[ ] I have added tests that cover my changes (if relevant).
[ ] The option documentation (docs/options) has been updated with new or changed options.
[ ] I have updated CHANGELOG.md.
[ ] I have updated the documentation (if relevant).

Hi @mstapelberg!
For the DDP version of nequip, the state-reduce branch of torch_runstats needs to be used as it has the get_state() methods defined. This should definitely be made more clear as I ran into the same issue too (the ddp branch is still under development so the documentation/instructions are still a work in progress).
Not sure if this is a better way of implementing though. @Linux-cpp-lisp will know!

Hi @kavanase thanks for your reply and help! I'll give that a go, likely will be a much better implementation than what I hacked together. Will try the new branch for pytorch_runstats now

Hi @kavanase I gave the updated pytorch_runstats a go; it works (I've updated my initial pull request). However when I try it on a more realistic problem the following error happens (full log here -
distributed_error_log_2gpus_vcrti_100.txt
training
# Epoch batch         loss       loss_f       loss_e        f_mae       f_rmse     Ti_f_mae      V_f_mae     Cr_f_mae  psavg_f_mae    Ti_f_rmse     V_f_rmse    Cr_f_rmse psavg_f_rmse        e_mae      e/N_mae
Traceback (most recent call last):
  File "/home/myless/.mambaforge/envs/allegro-ddp/bin/nequip-train", line 8, in <module>
    sys.exit(main())
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/scripts/train.py", line 119, in main
    trainer.train()
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/train/trainer.py", line 841, in train
    self.epoch_step()
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/train/trainer.py", line 997, in epoch_step
    self.batch_step(
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/train/trainer.py", line 874, in batch_step
    out = self.model(data_for_loss)
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1026, in forward
    if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by
making sure all `forward` function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 10
 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
Traceback (most recent call last):
  File "/home/myless/.mambaforge/envs/allegro-ddp/bin/nequip-train", line 8, in <module>
    sys.exit(main())
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/scripts/train.py", line 119, in main
    trainer.train()
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/train/trainer.py", line 841, in train
    self.epoch_step()
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/train/trainer.py", line 997, in epoch_step
    self.batch_step(
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/nequip/train/trainer.py", line 874, in batch_step
    out = self.model(data_for_loss)
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1026, in forward
    if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by
making sure all `forward` function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 1: 10
 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1196340) of binary: /home/myless/.mambaforge/envs/allegro-ddp/bin/python3.10
Traceback (most recent call last):
  File "/home/myless/.mambaforge/envs/allegro-ddp/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/myless/.mambaforge/envs/allegro-ddp/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/home/myless/.mambaforge/envs/allegro-ddp/bin/nequip-train FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-16_14:11:51
  host      : gpu-rtx6000-04.psfc.mit.edu
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1196341)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-16_14:11:51
  host      : gpu-rtx6000-04.psfc.mit.edu
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1196340)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

I'm fairly novice when it comes to Github so would it be best to open another issue with this? Or is this something you have seen in the past as well?
Thanks!
Myles

Hmm ok I haven't seen that error yet with my tests for the ddp branch (though I am using nequip ddp rather than allegro ddp, so not sure if this will also cause some differences). There's discussion in the linked issue (https://github.com/mir-group/nequip/issues/210) there on using the ddp branches, so that might help solve this issue?
Maybe worth trying that with the un-edited ddp and state_reduce branches of nequip & torch_runstats and seeing if it works? Then can close this PR and keep the discussion to https://github.com/mir-group/nequip/issues/210?


Ability to use RancherOS

Kinda trolling kinda not :)
http://rancher.com/rancher-os/
Having a way to use this would be pretty neat.

kinda RTFM, and kinda not:
[sven@t440s ~]$ boot2docker --help
Usage: boot2docker [<options>] <command> [<args>]

Boot2Docker management utility.

Commands:
   init                Create a new Boot2Docker VM.
   up|start|boot       Start VM from any states.
   ssh [ssh-command]   Login to VM via SSH.
   save|suspend        Suspend VM and save state to disk.
   down|stop|halt      Gracefully shutdown the VM.
   restart             Gracefully reboot the VM.
   poweroff            Forcefully power off the VM (may corrupt disk image).
   reset               Forcefully power cycle the VM (may corrupt disk image).
   delete|destroy      Delete Boot2Docker VM and its disk image.
   config|cfg          Show selected profile file settings.
   info                Display detailed information of VM.
   ip                  Display the IP address of the VM's Host-only network.
   shellinit           Display the shell commands to set up the Docker client.
   status              Display current state of VM.
   download            Download Boot2Docker ISO image.
   upgrade             Upgrade the Boot2Docker ISO image (restart if running).
   version             Display version information.

Options:
      --basevmdk="": Path to VMDK to use as base for persistent partition
      --clobber=false: overwrite Docker client binary on boot2docker upgrade
      --dhcp=true: enable VirtualBox host-only network DHCP.
      --dhcpip=<IP_ADDRESS>: VirtualBox host-only network DHCP server address.
  -s, --disksize=20000: boot2docker disk image size (in MB).
      --dockerport=0: host Docker port (forward to port 2376 in VM). (deprecated - use with care)
      --driver="virtualbox": hypervisor driver.
      --force-upgrade-download=false: always download on boot2docker upgrade, never skip
      --hostip=<IP_ADDRESS>: VirtualBox host-only network IP address.
      --iso="/home/sven/.boot2docker/boot2docker.iso": path to boot2docker ISO image.
      --iso-url="https://api.github.com/repos/boot2docker/boot2docker/releases": source URL to provision the boot2docker ISO image.
      --lowerip=<IP_ADDRESS>: VirtualBox host-only network DHCP lower bound.
  -m, --memory=2048: virtual machine memory size (in MB).
      --netmask=ffffff00: VirtualBox host-only network mask.
      --no-dummy="": Example parameter for the dummy driver.
      --retries=75: number of port knocking retries during 'start'
      --serial=false: try serial console to get IP address (experimental)
      --serialfile="": path to the serial socket/pipe.
      --ssh="ssh": path to SSH client utility.
      --ssh-keygen="ssh-keygen": path to ssh-keygen utility.
      --sshkey="/home/sven/.ssh/id_boot2docker": path to SSH key to use.
      --sshport=2022: host SSH port (forward to port 22 in VM).
      --upperip=<IP_ADDRESS>: VirtualBox host-only network DHCP upper bound.
      --vbm="VBoxManage": path to VirtualBox management utility.
      --vbox-share=[]: List of directories to share during 'up|start|boot' via VirtualBox Guest Additions, with optional labels
  -v, --verbose=false: display verbose command invocations.
      --vm="boot2docker-vm": virtual machine name.
      --waittime=300: Time in milliseconds to wait between port knocking retries during 'start'
error in run: config error: pflag: help requested

(hint: use boot2docker init --iso=/some/path/to/some.iso)
let us know how it goes :)

:D

it requires lots of manual work to make it work since it is just different convention in rancherOS.
firstly, the default user is rancher not docker. so it requires manual step to get boot2docker start or docker-machine start work, where they hard code user docker in ssh commands out of their virtualbox driver.
Booted rancherOS VM from the given ISO runs OS in memory only, so everything performed by boot2docker/docker-machine (or yourself manually) will be gone when restart although it creates a persistent device for this VM by docker-machine. So you need to persist state to device /dev/sda after mounting it.
However, I need to study rancher os configuration more because it's pretty neat to configure everything via a single YAML file and rancher supports cloud-init which is in the roadmap of docker-machine.

to make this clear for anyone reading this - the best place for this kind of work will be in docker-machine - and I think some initial work has gone into looking at it.

@SvenDowideit you're right if talking about docker/machine#693


[virt-operator] admission control for the infra and workloads placement config in Kubevirt CR

Added validation for the Spec.infra and Spec.workloads sections in Kubevirt CR.
Validation code placed in the virt-operator webhook code, and it will happen upon CR update.
Added functional test coverage as well.
What this PR does / why we need it:
Since we weren't fail fast on incorrect configuration, the apply of this configuration on the control place deployments/DS
would fail and we will have inconsistent state of the control plane deployment.
Fixes #
https://bugzilla.redhat.com/show_bug.cgi?id=1938262
Special notes for your reviewer:
More refactoring in the virt-operator functest area will come in following PRs..
Release note:
NONE

/hold
need to adjust operator unit tests

/unhold

/retest

Thanks for the PR @enp0s3!
See several comments inline, and another general comment (though not directly related to this PR): if I understand correctly, this webhook is invoked only when the KV object is updated, but not when it's created. Do you know why is it so?

@zcahana Hi. Thank you for your time! AFAIK the reason we don't invoke the webhook on creation is because we don't want to introduce an usability issue i.e. deploying the virt-operator may take time (pulling the operator image), during that time the webhooks doesn't exist yet, and if you have a creation webhook then you couldn't apply the kubevirt CR until virt-operator will be deployed.

AFAIK the reason we don't invoke the webhook on creation is because we don't want to introduce an usability issue i.e. deploying the virt-operator may take time (pulling the operator image), during that time the webhooks doesn't exist yet, and if you have a creation webhook then you couldn't apply the kubevirt CR until virt-operator will be deployed.

It seems to me that the webhook definitions are registered programmatically by virt-operator, i.e. only after it's up and running - implying that the webhook service itself is up and running and can serve both UPDATEs and CREATEs. Either way this leaves a hole in which an invalid KubeVirt configuration can slip-in bypassing validation. But as said before this isn't really related to this PR so shouldn't be blocking it. 😄

/lgtm
/hold
@enp0s3 I believe all review comments have been addressed. I've set a hold since I believe you wanted to de-include the unit-tests removal from this PR. Feel free to /unhold when you see fit.
Thanks!

/unhold

/lgtm
/hold
@enp0s3 I believe all review comments have been addressed. I've set a hold since I believe you wanted to de-include the unit-tests removal from this PR. Feel free to /unhold when you see fit.
Thanks!

@zcahana Thank you for your time!

Hey! This looks good, I'm just curious about why you are using mock objects for this? Wouldn't it be easier to use a dryrun update on the intended resources?

Hey! This looks good, I'm just curious about why you are using mock objects for this? Wouldn't it be easier to use a dryrun update on the intended resources?

@ashleyschuett Hello :) actually that option didn't come to my mind, this is an interesting approach. Currently I see two options to achieve the approach:

From the webhook context call the k8s client to read the deployement/DS resource, append the configuration to be validated, and update the resource.
Try to use the same logic as in strategy.go meaning that I need to use components and config

In general I wanted to reduce network operations to minimum in the webhook context i.e. minimum amount of REST calls.
Is there any other way to read the intended resources, WDYT?

From the webhook context call the k8s client to read the deployement/DS resource, append the configuration to be validated, and update the resource (prefer not to use informers in webhook).
Try to use the same logic as in strategy.go meaning that I need to use components and config

In general I wanted to reduce network operations to minimum in the webhook context i.e. minimum amount of REST calls.
Is there any other way to read the intended resources, WDYT?

I think #2 makes sense, we would then get the desired definition of the resource and compute the new desired state. Then run a dry run update with that resource. If it passes then we can be very confident that it will be successfully updated on reconciliation.

@enp0s3 What needs to be done to move this forward?

@xpivarc @ashleyschuett @stu-gott Addressed review comments. PTAL, thanks!

/hold
observing failed e2e cases

/hold cancel

/cc @davidvossel

It seems to me that the webhook definitions are registered programmatically by virt-operator, i.e. only after it's up and running - implying that the webhook service itself is up and running and can serve both UPDATEs and CREATEs. Either way this leaves a hole in which an invalid KubeVirt configuration can slip-in bypassing validation. But as said before this isn't really related to this PR so shouldn't be blocking it. smile

This is a problem. There's not an easy solution.
The issue here is that we want someone to be able to install KubeVirt in a single step. This means posting both the virt-operator manifest and the corresponding kubevirt CR all in one go as described here [1]. If we have a required validation webhook on CREATE of the kubevirt cr, then the commands outlined in [1] will fail because the webhook server won't be online yet.
That said, we have another usability issue if someone creates a kubevirt CR that isn't valid. Any attempt to update the CR will fail due to a validation issue that was actually at Create time.
it's not a great situation. We might have to revise our install steps to account for waiting for the virt-operator deployment to be ready before posting the kubevirt CR. That would allow us to validate at Creation.

http://kubevirt.io/user-guide/operations/installation/#installing-kubevirt-on-kubernetes

If we're going to validate our deployments/daemonsets, then it would need to somehow be the values generated by the installstrategy for the kubevirt version being installed for this to be completely accurate.

I looked at this a bit. Luckily the install strategy is cached and the affinity rules are layered on top of the deployments/daemonsets in the install strategy, so technically it is possible to construct an accurate daemonset/deployment during validation if you create a way to retrieve the install strategy cache from the kubevirt reconcile loop.
That said, this all feels overly complicated. Is there not a way to reasonably validate the workload placement without doing a dry run?

If we're going to validate our deployments/daemonsets, then it would need to somehow be the values generated by the installstrategy for the kubevirt version being installed for this to be completely accurate.

I looked at this a bit. Luckily the install strategy is cached and the affinity rules are layered on top of the deployments/daemonsets in the install strategy, so technically it is possible to construct an accurate daemonset/deployment during validation if you create a way to retrieve the install strategy cache from the kubevirt reconcile loop.
That said, this all feels overly complicated. Is there not a way to reasonably validate the workload placement without doing a dry run?

@davidvossel First of all thanks a lot for the effort :) What I was trying to do in my first attempt is simply taking the validation code from k8s, but during the work I noticed that I need to take a plenty of k8s code. Another disadvantage of it is that I will need to manually update this code every time the k8s CRD will be changed in the area of nodeSelector, Affinity, Tolerations. Therefore I wanted to do "code re-use" by running the dry run, thus utilizing the underlying k8s API validation.
For the short term I can suggest to replace the dry run with a simple check of the toleration operator, that will address the BZ issue.
WDYT?

@davidvossel Actually I am not sure why do we need accurate definition of our deployment/DS resources for the dry dun? I only want to validate Toleration/Affinity/NodeSelector fields, so for that purpose I can create a mocked resources, like SSP operator does here

/cc @davidvossel

/retest

I believe this PR has converged and is ready to go in.
Thank you @enp0s3!
/lgtm
/retest

/cherrypick release-0.44


Installation

I get an error when trying to install. I assume I missed something?
Running on Ubuntu 16.04.2 LTS
Node v8.2.0
npm 5.3.0
Commands used:
npm install
npm run-script build

gives
ERROR in ./main.js
Module build failed: Error: /home/anders/Code/schema-designer/.eslintrc:
	Configuration for rule "flowtype/require-return-type" is invalid:
	Value "[object Object]" is the wrong type.

    at validateRuleOptions (/home/anders/Code/schema-designer/node_modules/eslint/lib/config/config-validator.js:109:15)
    at Object.keys.forEach.id (/home/anders/Code/schema-designer/node_modules/eslint/lib/config/config-validator.js:156:13)
    at Array.forEach (native)
    at Object.validate (/home/anders/Code/schema-designer/node_modules/eslint/lib/config/config-validator.js:155:35)
    at Object.load (/home/anders/Code/schema-designer/node_modules/eslint/lib/config/config-file.js:559:19)
    at loadConfig (/home/anders/Code/schema-designer/node_modules/eslint/lib/config.js:63:33)
    at getLocalConfig (/home/anders/Code/schema-designer/node_modules/eslint/lib/config.js:130:29)
    at Config.getConfig (/home/anders/Code/schema-designer/node_modules/eslint/lib/config.js:260:26)
    at processText (/home/anders/Code/schema-designer/node_modules/eslint/lib/cli-engine.js:224:33)
    at CLIEngine.executeOnText (/home/anders/Code/schema-designer/node_modules/eslint/lib/cli-engine.js:754:26)
    at lint (/home/anders/Code/schema-designer/node_modules/eslint-loader/index.js:218:17)
    at Object.module.exports (/home/anders/Code/schema-designer/node_modules/eslint-loader/index.js:213:21)

Seeing this might be just a jslint issue I commented out that rule so I could build, but I still got other errors when accessing port 3000 so I suspect I have some fundamental part missing?
Thank you and looking forward to contributing.

flow-bin needs to be installed globally in your system. Then use npm start to start the dev server.

thanks still not there though. Build successful, then get the following errors in console when serving
Tables.js:73 Uncaught TypeError: _jsplumb2.default.ready is not a function
    at Tables.__makeTablesDraggable__REACT_HOT_LOADER__ (Tables.js:73)
    at Tables.__makeTablesDraggable__REACT_HOT_LOADER__ (createPrototypeProxy.js:44)
    at Tables._this.makeTablesDraggable (Tables.js:46)
    at Tables.componentDidMount (Tables.js:54)
    at Tables.proxiedComponentDidMount (createPrototypeProxy.js:66)
    at eval (ReactCompositeComponent.js:264)
    at measureLifeCyclePerf (ReactCompositeComponent.js:75)
    at eval (ReactCompositeComponent.js:263)
    at CallbackQueue.notifyAll (CallbackQueue.js:76)
    at ReactReconcileTransaction.close (ReactReconcileTransaction.js:80)

Make sure you're using jsplumb 2.2.10 version. Also if you have yarn installed, use it instead of npm. It'll make sure the right versions are installed for all packages.

npm silently fails but yarn did the trick! Thats awesome. No need to change jsplumb ^2.2.10 either.