WIP Adding Object store

Currently when controller detects new CRD for NetworkService, it does not store them. This PR adds an object store for NetworkServices and adds several basic methods.
Signed-off-by: Serguei Bezverkhi<EMAIL_ADDRESS>

So this one and #87 are tackling the problem from different angles. In this one, you're looking to store CRDs in a datastore inside of NSM. With #87, I was having clients register handlers, and passing CRDs directly to them as they come in. That pushes the datastore problem to the clients. Lets discuss these approaches.

Also, we should look at the datastore Ligato provides and likely use one of those. See here:
https://github.com/ligato/cn-infra/tree/master/db

I'd suggest we may want to layer a Plugin on top of the CRD plugin that does local caching of the CRDs.
I like very much having a CRD plugin that just takes handlers, its handy modularity, but I think having a CRDCache plugin is also a good idea :)  Layers.  We are networking people, its what comes naturally :)

@edwarnicke I like that as well. I propose we layer this with the handler work in #87 to provide maximum layering flexibility. I'd still like us to look at using the Ligato datastore if possible, I'll discuss with @sbezverk :)

@mestery I think the first phase is ready to be merged, you will need it for testing changes in CRD controller.


Mixed-up names in docs explaining relationships

I believe that section detailing relationships has a naming mix-up in the model definitions.
At first there is:
class Person(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(50), nullable=False)
addresses = db.relationship('Address', backref='person', lazy=True)

But further down it changes to this, where backrefs are explained in more detail:
class User(db.Model): # this should read: Person(db.Model)...
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(50), nullable=False)
addresses = db.relationship('Address', lazy='select',
backref=db.backref('person', lazy='joined'))

I think this is quite misleading and it's easy to think that this is a third table, which somehow connects to the two others defined earlier. It certainly had me confused for a while.

@pfabri thank you for the report.  We will get this fix in the 2.4 release, which should be coming soon.

closed in #718


Shifted to new URL

@karthikb351 just saw it there were a number of people who sent the same pull request, however you accepted the one who sent it first but what about changing <IP_ADDRESS> -> <IP_ADDRESS>
In that case, i think you should have accepted @aksaren


always keep sort_table symbol for debugging

This hasn't really fixed the issue.
If I kompile normally, I don't get gdb.error: Missing ELF symbol "sort_table"
But I get <optimized out> terms.
If I use -O0 or -O1 the error is back.

I don't know what to tell you; I was no longer able to reproduce this issue with this change. Are you sure you're using the correct version of the llvm backend?


Multiple clusters concurrency

From the docs it is stated that I can have multiple clusters on multiple machines, working on the same queue. But I'm missing understanding from architecture https://django-q2.readthedocs.io/en/latest/architecture.html how multiple Pushers are synchronized, so they don't push same task to different task queues. Maybe it is worth to improve doc or can you explain here?

The pusher gets the task from the broker. We make sure that it either gets popped from the broker, so that it won't exist anymore after the pusher got it, or we lock the task to a certain time so that other pushers can't take it.
In the case of the ORM for example, we lock the task:
https://github.com/django-q2/django-q2/blob/7add68f7709856f875bf145d59374adc80112009/django_q/brokers/orm.py#L66-L83
Let me know if you have questions.

The pusher gets the task from the broker. We make sure that it either gets popped from the broker, so that it won't exist anymore after the pusher got it, or we lock the task to a certain time so that other pushers can't take it.
In the case of the ORM for example, we lock the task:
https://github.com/django-q2/django-q2/blob/7add68f7709856f875bf145d59374adc80112009/django_q/brokers/orm.py#L66-L83
Let me know if you have questions.

Thank you for reply. That is pretty clear for me now.


No such file or directory when doing miRNA analysis

Hello,
When running the miRNA module with the command: XICRA miRNA --input datafolder --software miraligner optimir sRNAbench
I get the following error:
Can you please help me troubleshoot the issue? Thank you!
--------- Starting Process ---------
09/19/2022, 09:22:31

Software for miRNA analysis selected:
['miraligner', 'optimir', 'sRNAbench']

Getting files from input folder...

Mode: join.

Extension:
[_joined.fastq]

Input folder exists
12 files selected...
12 samples selected...
Single end mode selected...

Species provided: hsa

Create folder to store results:  /home/ubu/miniconda3/envs/XICRA/lib/python3.7/site-packages/XICRA/modules/db_files
+ miRNA gff file available
+ hairpin fasta file available
+ miRNA gff file available
+ miRBase str file available

Create a miRNA analysis for each sample retrieved...
***ERROR:
<Future at 0x7fe75ee55550 state=finished raised FileNotFoundError>
'CONTROL_1' generated an exception: [Errno 2] No such file or directory: '/home/ubu/datafolder/data/CONTROL_1/miRNA/miraligner/CONTROL_1-tab.freq.txt'

same error here for all samples......

miRNA analysis is finished...
Let's summarize all results...
Input folder exists

**ERROR: No samples were retrieved for this option. Continue processing...

Works now!

Hi there,
Happy to hear it worked.
thanks for the feedback!
Best regards


Update Podfile to IQKeyboardManager 6.0.0

Can we upgrade the Podfile to use IQKeyboardManager 6.0.0? v5 doesn't work well on the latest version of iOS, and v6 seems to work fine.

Out of curiosity: can you elaborate on what's not working correctly with the latest iOS (11 or 12?)?

On iOS 11, the "soft keyboard" (the row that appears above the regular keyboard) was appearing in the in the simulator, but not on the actual device. Also, the keyboard would cover the field that was selected.
Seems related to: #26 #39

+1 to get this merged to hopefully fix some unusual issues I'm having in iOS 11+.

@EddyVerbruggen @tjvantoll
Please update the repo to 6.0.0+
It fixes various bugs, but most importantly it fixes the navigation bar movement with the textfield. I expect the navigation bar to be fixed at top, but in 5.0.0 it moves with the content up, making in unreachable. This is bad UX for chat apps for example.

Hi @benediktveith, that navbar movement issue probably depends on the structure of the app, because in the demo app in this repo I can't reproduce it. That being said, bumping to 6.x doesn't seem to hurt either, so I'll merge this and even bump to 6.2.0.

@EddyVerbruggen Yep, done! Thanks for following up on this.

Thanks @tjvantoll!
I've merged #52 as well, and tagged the 1.4.0 release. Would you mind updating and running cd publish && ./publish.sh or adding me as an owner for the npm package? Thx!


Is windows 10 supported - getting usb_detach_kernel_driver_np error

Is it true that python-escpos 3.0a3 is not compatible with windows 10 - as this stackoverflow comment  suggests?  And as suggested by the python-escpos todo readme?
from escpos import *
Epson = printer.Usb(0x04b8, 0x0e03)

I get the error function 'usb_detach_kernel_driver_np' not found.
Am running Python 3.6.  Printer Epson TM-TS20.  Am running in vmware, windows 10 (hosted by a mac).  I can attatch the Epson printer to the windows virtual machine ok and print to it using the official Epson drivers etc.
Re python-escpos on windows 10, I worked around the lack of /tmp directory (saw there was a fix for this recently merged - nice) by simply creating a c:/tmp directory.  Then I ran into missing backend issues, and as I couldn't find libusb on the system after pip install pyusb thus assumed that I needed to find and install these backend dlls myself.  There are several confusing choices as to where to get these dlls from and how to install them.  I copied libusb dlls into windows/system32 using both the zadig installer and also by copying in the dlls downloaded from https://libusb.info/.
My normal Epson TM-TS20 driver (via ESDPRT001) seems to have been broken by the zadig step (which created a printer on port USB001). I currently have these dlls: (list made using git bash shell which is why it look Linux-like):
-rwxr-xr-x 1 Andy 197121  76384 May 17 19:06 /c/windows/system32/libusb0.dll*
-rwxr-xr-x 1 Andy<PHONE_NUMBER>56 Mar 25 12:00 /c/windows/system32/libusb-1.0.dll*
-rw-r--r-- 1 Andy 197121  46686 Mar 25 12:00 /c/windows/system32/libusb-1.0.lib
-rw-r--r-- 1 Andy<PHONE_NUMBER>92 Mar 25 12:00 /c/windows/system32/libusb-1.0.pdb
-rwxr-xr-x 1 Andy 197121  99128 May 17 19:06 /c/windows/system32/libusbk.dll*

So do I have a bad dll situation or is it simply that python-escpos 3.0a3 is not currently compatible with windows 10 and I should stop wasting my time trying to get this to work, till windows 10 is officially supported?

Hi @abulka thank you for your question.
I am not 100% sure, since I am developing and testing on a Linux-environment. As of 3.0a4 things like usage of /tmp should be done. @juanmitaboda looked into further improvements and it seems that the usb-backend has to be replaced. His work can be found #274.
I haven't merged it yet since I don't like to completely replace the backend, but rather prefer to support both backends.
If you find out anything more, feel free to add it to the discussion. You could try to use the changes in #274. Once I (or somebody else) had the time to change this PR to support both usb-libraries I will happily merge it and then it can also be included in a release.

Note here are some helpful tips on getting python-escpos working on windows 10
https://github.com/pyusb/pyusb/issues/200

I am developing the hotel billing system but stuck on receipt printing, i used escpos library
installed the required package and library but it shows [Errno None] Other error while trying to print the text
p = Usb(0x0416, 0x5011)
p.text("Hello World\n")
# p.image("logo.gif")
# p.barcode('1324354657687', 'EAN13', 64, 2, '', '')
p.cut()
i am testing on windows 10 , can anybody solve this problem?

Hi @kabindra-kc, it looks to me as this is not related to the issue in which you are posting. Could you please create a new issue and add full information: the version of python-escpos, your system version, (and the other information requested in the issue-template), the full stack-trace, your type of printer?

I am developing the hotel billing system but stuck on receipt printing, i used escpos library installed the required package and library but it shows [Errno None] Other error while trying to print the text i have installed pyusb 1.1.0, libusb 1.0.23b6, copied libusb-1.0.dll to system32,
p = Usb(0x0416, 0x5011) p.text("Hello World\n") # p.image("logo.gif") # p.barcode('1324354657687', 'EAN13', 64, 2, '', '') p.cut() i am testing on windows 10 , can anybody solve this problem?

did you solve the issue? im stuck as well

Hi @gabri3l0 !
Did you install zadig?
Some users found the solution by installing the usb driver via the zadig installer on Windows at this issue.

Hi @gabri3l0 !
Did you install zadig? Some users found the solution by installing the usb driver via the zadig installer on Windows at this issue.

Yes it works! thanks! i install libusbk driver and it works! thanks!


[BUG]

What happened?
A bug happened!
Selector doesn't draw anything and gives this error
NullReferenceException: Object reference not set to an instance of an object UnityEngine.GUILayoutEntry.ApplyStyleSettings (UnityEngine.GUIStyle style) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/LayoutEntry.cs:72) UnityEngine.GUILayoutEntry.set_style (UnityEngine.GUIStyle value) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/LayoutEntry.cs:31) UnityEngine.GUILayoutEntry..ctor (System.Single _minWidth, System.Single _maxWidth, System.Single _minHeight, System.Single _maxHeight, UnityEngine.GUIStyle _style, UnityEngine.GUILayoutOption[] options) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/LayoutEntry.cs:61) UnityEngine.GUILayoutUtility.DoGetRect (System.Single minWidth, System.Single maxWidth, System.Single minHeight, System.Single maxHeight, UnityEngine.GUIStyle style, UnityEngine.GUILayoutOption[] options) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/GUILayoutUtility.cs:482) UnityEngine.GUILayoutUtility.GetRect (System.Single minWidth, System.Single maxWidth, System.Single minHeight, System.Single maxHeight, UnityEngine.GUIStyle style, UnityEngine.GUILayoutOption[] options) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/GUILayoutUtility.cs:476) UnityEditor.EditorGUILayout.GetControlRect (System.Boolean hasLabel, System.Single height, UnityEngine.GUIStyle style, UnityEngine.GUILayoutOption[] options) (at /Users/bokken/build/output/unity/unity/Editor/Mono/EditorGUILayout.cs:2104) Sirenix.Utilities.Editor.SirenixEditorGUI.ToolbarSearchField (System.String searchText, System.Boolean forceFocus, System.Single marginLeftRight) (at <d5a36a96b2ad42b898b2ec8b30fb6453>:0) Sirenix.Utilities.Editor.InstanceCreatorWindow.OnGUI () (at <d5a36a96b2ad42b898b2ec8b30fb6453>:0) UnityEditor.HostView.OldOnGUI () (at /Users/bokken/build/output/unity/unity/Editor/Mono/HostView.cs:268) UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize, System.Action onGUIHandler, System.Boolean canAffectFocus) (at /Users/bokken/build/output/unity/unity/ModuleOverrides/com.unity.ui/Core/IMGUIContainer.cs:351) UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&) (at /Users/bokken/build/output/unity/unity/Modules/IMGUI/GUIUtility.cs:189)
Package Version
1.1.9
Unity Version
2021.3.33f

Hi @whynames .
It is difficult to determine the cause because the stack trace does not seem to contain any lines of code for this library.
Can you provide a video of this bug happening?

Yeah, I noticed it. Weird thing is it was working when I first tried, but doesn't now
https://github.com/mackysoft/Unity-SerializeReferenceExtensions/assets/73838734/714dd772-6f00-4d11-8126-d93328e94ebb
I can see it detects the classes but can't draw, because vertical sizes of the drawers are like size of how many of the inherited ones would fit

Umm...
From the video, the interface does not appear to be from this library.
I have tried drawing with same SerializeReferenceExtensions library and Unity version and it works fine.

For anyone wondering, problem was odin inspector editor override, just also add [DrawWithUnity] attribute to your field. It is hard to spot since it is inconsistent.

@whynames
Okay, I have never used Odin Inspector, but if I give it the DrawWithUnity attribute, will the SubclassSelector functionality work perfectly?
For example, as follows.
[SerializeReference, SubclassSelector, DrawWithUnity]
private IHoge m_Hoge;

It looks as your screenshot, but my classes didn't have many subclasses, only classes inherit from the interface

Thanks. If the subclass selection is working properly, there should be no problem.


build without text no longer work

Have you read the FAQ and checked for duplicate open issues?
yes
What version of Shaka Player are you using?
v3.3.0
Can you reproduce the issue with our latest release version?
yes
Can you reproduce the issue with the latest code from master?
yes
What did you do?
git clone https://github.com/google/shaka-player
cd shaka-player
python build/build.py +@complete -@polyfill -@text

What did you expect to happen?
it should work
What actually happened?
[INFO] Compiling the library (ui, release)...
/tmp/shaka-player/lib/player.js:37:0: ERROR - [JSC_MISSING_MODULE_OR_PROVIDE] Required namespace "shaka.text.WebVttGenerator" never defined.
  37| goog.require('shaka.text.WebVttGenerator');
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

1 error(s), 0 warning(s)
[ERROR] Build failed

Respected sir/Madam,
I am Madhusudan Pareek , a computer science undergraduate , I have just entered to my 4th semester at TC College affiliated with Savitri Bai Phule Pune University. I am new to open source but I am well aware of JavaScript , Nodejs, PostgreSQL, MySQL, HTML,CSS, bootstrap, React and also good knowledge of making rest api’s. I would love to contribute to your organization. Currently I had worked with an organization ‘IBKAAS’ as an intern and developed their backend with NodeJS ( express )and PostgreSQL and now I’m looking for open source..
Hoping to hear from you soon.
Regards
Madhusudan

To add some context, the error in its current form appeared in 0845843b651c32b3081772a39947b0202f044094.
Before that, there was already dependency on @text, which was for the first time introduced in faa1e52902bcf2b6d9d6b7067794236c41ec0cee.


Support INT8 mixed-precision training from torchao?

Recently I worked on INT8 mixed-precision training in torchao. The relevant PR is here https://github.com/pytorch/ao/pull/748
Preliminary results show that with torchtitan, it improves speed by 20% on 8x A100 with no noticeable difference in loss curve. See the PR for more details.
Would you be open to add an experimental flag for this in torchtitan? Similar to Float8 training. This can also help to profile and improve INT8 training performance directly in torchtitan for future perf optimization.
cc @msaroufim

cc: @weifengpy

@gau-nernst nice work! I took a look at the original torchao PR

did you land it as a torchtune full finetuning recipe? I am mostly curious about gaps for checkpointing if any
if performing rowwise scaling, how did you deal with backward in FSDP2? asking becuase rowwise became column wise in the backward.
is perf the main reason to adopt int8? the PR mentioned memeory savings. Is it because of int8 casting in fsdp hooks? otherwise it's bfloat16 casting

thanks for explaining everything in detail

What do you mean by this?

I thought model.parameters() might be some INT8 related tensor subclass

when saving state dict, whether we convert the sharded tensor subclass to higher precision
when loading state dict, whether we qunatize on full tensor firs then chunk

Right now all-gather is still in BF16, so quantization will be done after all-gather

yeah. INT8 all-gather might be the main reason to justify landing into torchtitan, since this repo is used to demonstrate composability with distributed api
for rowwise, if backward is too hard, are you comfortable with supporting INT8 all-gather with  fully_shard(reshard_after_forward=False) ? In that case, we do not have all-gather in the backward

For pre-training, it might be possible to do INT8 tensor-wise scaling

if the numerics does not become too bad witht tensor-wise scaling, it's a great demonstration for INT8 all-gather

I thought model.parameters() might be some INT8 related tensor subclass

Oh yea right now I don't have any special logic with it. So that state_dict will be a tensor subclass wrapper holding the original high precision weight (NOT int8). For INT8 mixed-precision training, I only inject custom matmul logic, weights stay the same (same as FP8 training).

supporting INT8 all-gather with fully_shard(reshard_after_forward=False) ? In that case, we do not have all-gather in the backward

Does that mean INT8 post-all-gathered weights remain in memory starting from forward until backward? If that's the case, we can just do what I suggested earlier?

use row-wise quantized weight (from forward) for column-wise scaling in backward (i.e. dequant and re-quant)

More concretely:

Pass
Original
Suggested change

Forward
FP32 weight -> all-gather -> row-wise quantize to INT8
FP32 weight -> row-wise quantize to INT8 -> all-gather

Backward
FP32 weight -> all-gather -> column-wise quantize to INT8
FP32 weight -> row-wise quantize to INT8 -> all-gather -> dequant -> column-wise quantize to INT8

In other words, it differs in which version of the weight will be used for column-wise quantization in backward: whether to use the original weight, or use the row-wise quantized weight used in forward.
Otherwise, to just demonstrate INT8 all-gather, I think it is easier (and save efforts) to do INT8 tensor-wise scaling 🤣.

Otherwise, to just demonstrate INT8 all-gather, I think it is easier (and save efforts) to do INT8 tensor-wise scaling

agree, having tensor-wise scaling is already a good thing. I will bring this topic for discussion with the team

I think long term it's great to unify training APIs in torchao, to enable torchtitan to work with float8/int8/mx/etc training in the same way.  I'm working on this, no ETA yet.
Short term if someone wants to add int8 training to torchtitan as an experimental feature - SGTM personally, but I'll also defer to torchtitan folks on that.

I will bring this topic for discussion with the team

we would love to have this feature after discussion. we can start with tensor-wise scaling. it's also consistent with our float8  offering
@mori360


[Android][0.27.2] Fetch send cookie but sever can not receive

code:
let headers = new Headers({
    'Cookie': options.cookie,
    ...
});
fetch(url, {
    method: 'POST',
    headers: headers,
}).then(data => {
    ...
})

rn 0.25 is OK，but in 0.27.2 sever can not receive cookie

Are you debugging in chrome?  Just curious

no...

We are experiencing the same problem.

Hi there! This issue is being closed because it has been inactive for a while.
But don't worry, it will live on with ProductPains! Check out its new home: https://productpains.com/post/react-native/android0272-fetch-send-cookie-but-sever-can-not-receive
Product Pains has been very useful in highlighting the top bugs and feature requests:
https://productpains.com/product/react-native?tab=top
Also, if this issue is a bug, please consider sending a pull request with a fix.


Support notifying subscribers despite the reference not changing

I'm looking into making an adapter from watermelon DB observables to Legend. In watermelon, when the record changes, the reference never changes. In legend, if the reference doesn't change, the subscribers won't be notified when setting the value. Is there a way to work around this?
https://github.com/LegendApp/legend-state/blob/d183ab68d3d6bf98ee87a74f843b511331f7a997/src/ObservableObject.ts#L571
Perhaps a second parameter such as observable.set(value, { forceNotify: true })

Hmm, that's an interesting one.
The downside of a forceNotify is that it would need to notify at every single in a hierarchy. We currently have optimizations to skip processing children if the child value is unchanged from the previous child value, so  setting a huge object would only need to process the nodes that did not change. But with forceNotify it would have to process every single node.
Of course even though it might be slow it's better than nothing... But I wonder if there's a better way to do it? Can you share some of your adapter code so I can see what it looks like and maybe we can find a more optimized solution?

Sure, here's a draft, right now this is untested code:
import { useObservable } from '@legendapp/state/react'
import { Model, Query } from '@nozbe/watermelondb'
import { useLayoutEffect } from 'react'

export const WatermelonLegendStateAdapter = {
  useModel: <T extends Model>(model: T) => {
    const o = useObservable(model)
    useLayoutEffect(() => {
      // A value is emitted synchronously when the observable is first subscribed to
      return model.experimentalSubscribe(isDeleted => {
        if (isDeleted) {
          // TODO: handle deletion
        } else {
          // HACK: Since the model reference will never change we use a proxy to signal changes
          o.set(new Proxy(model, {}))
        }
      })
    }, [model, o])
    return o
  },
  useQuery: <T extends Query<U>, U extends Model>(query: T) => {
    const o = useObservable([] as U[])
    useLayoutEffect(() => {
      // A value is emitted synchronously when the observable is first subscribed to
      return query.experimentalSubscribe(records => o.set(records))
    }, [query, o])
    return o
  },
}

Also, just thought it might be worth adding that this could be quite a big opportunity for Legend state and for Watermelon. Watermelon is stuck in the age HOCs, they've wanted to move to HOCs since 2018 but haven't managed yet

Ok, I have an idea:

Save the previous value from the query. You may need to clone if WatermelonDB is updating the object in place.
On change, set the raw value of the observable to the previous value (without notifying)
Set the new value, which will then notify anywhere it changed

Setting the raw value at the root of an observable is currently only possible in an experimental new feature: https://legendapp.com/open-source/state/experiments/#enabledirectpeek. But alternatively you could just create it with a child, like const o = useObservable({ value: [] as U[] }). Then you can set the raw value with o.peek().value = ...
const o = useObservable({ value: [] as U[] })
let previousValue

experimentalSubscribe(records => {
    if (previousValue) {
        const data = o.peek() // Get the raw object
        data.value = previousRecords // Set the raw object to the previous value
    }

    o.value.set(records) // Set the new value onto the observable

    previousValue = clone(records) // Save the previous value for next time
})

The clone may not be great for performance if these are huge objects, but it would probably be faster than iterating every node of the object and notifying all listeners...
Does that approach make sense?

Pretty sure we don't need to worry about cloning arrays as the query will emit a new array instance each time based on this line and this line. So unless I'm missing something the useQuery suggestion I posted should just work?
Side note on observable queries in Watermelon, query results only "re-render" if an item is created or deleted, so we don't need to worry about the references of the individual items staying constant. I believe this is similar to how Legend deals with lists anyway?
What do you think about the Proxy hack, it will be proxy on proxy on proxy, could this have perf consequences?

Do you think it's worth changing the issue title to Feature: Watermelon Adapter or do you think leave as forceNotify?

The downside of a forceNotify is that it would need to notify at every single in a hierarchy.

Even if all the subchildren references have not changed?

Pretty sure we don't need to worry about cloning arrays as the query will emit a new array instance each time based on this line and this line. So unless I'm missing something the useQuery suggestion I posted should just work?
Side note on observable queries in Watermelon, query results only "re-render" if an item is created or deleted, so we don't need to worry about the references of the individual items staying constant. I believe this is similar to how Legend deals with lists anyway?

Oh, it doesn't update if something is changed? In that case it seems like it should be fine.

What do you think about the Proxy hack, it will be proxy on proxy on proxy, could this have perf consequences?

I think that wouldn't work because although it sees the outer object as difference the references inside the object would still be the same. But I do think the idea of setting previous should work there, but with the model instead of the records array? I'm not super familiar with WatermelonDB, and from some research it seems like you're using undocumented experimental subscribers? So I'm guessing how it should work, but I think that should be right?

Do you think it's worth changing the issue title to Feature: Watermelon Adapter or do you think leave as forceNotify?
Let's keep talking to figure out what the actual thing to be done is, then change it :)

The downside of a forceNotify is that it would need to notify at every single in a hierarchy.

Even if all the subchildren references have not changed?

I think a forceNotify would have to force notifying at every single node in the object? Or maybe I'm misunderstanding what you're wanting it to do. From your description I'd thought that WatermelonDB is updating the model in place so setting has no effect because it's not changed. But it does that check at every node in the object so it would need to go through all child nodes recursively and notify them. If it only notified for the root and not the children it would break the core behavior of notifying sub-nodes when they change.

I'm currently getting a recursive error with the following code
const o = observable([] as any[])
      const q = database.collections.get<Post>('posts').query()
      q.experimentalSubscribe(x => {
        o.set(x) // Causes RangeError: Maximum call stack size exceeded
      })

Looking into the root cause now

Ah I found opaqueObject seems to be working now?

Hmm, when trying to update the object with the following standard watermelon code
          await database.write(async () => {
            await result.update(x => {
              x.title = `Updated ${counter.current++}`
            })
          })

I get the error
TypeError: Proxy set returned false for property '_isEditing'

Likely this is because of my use of opaqueObject. Here is my latest code:
export const WatermelonLegendStateAdapter = {
  useRecord: <T extends Model>(record: T) => {
    const o = useObservable(record)
    useLayoutEffect(() => {
      // A value is emitted synchronously when the model is first subscribed to
      return record.experimentalSubscribe(isDeleted => {
        if (isDeleted) {
          // TODO: handle deletion
        } else {
          // HACK: Since the model reference will never change we use a proxy to signal changes
          o.set(opaqueObject(record))
        }
      })
    }, [record, o])
    return o
  },
  useQuery: <T extends Model>(query: Query<T>) => {
    const o = useObservable([] as T[])
    useLayoutEffect(() => {
      // A value is emitted synchronously when the query is first subscribed to
      return query.experimentalSubscribe(records => {
        o.set(records.map(x => opaqueObject(x)))
      })
    }, [query, o])
    return o
  },
}

I imagine this is because opaqueObject blocks mutation? But then I also need opaqueObject to avoid circular issues. How can I get around this?

opaqueObject doesn't specifically block mutations, but direct modifications on an observable are blocked (because that's usually an error). Where is _isEditing being set from? It may be that you're saving the observable object into WatermelonDB which is internally changing _isEditing?
Also, opaqueObject would solve the circular references problem because it stops it from iterating through nodes looking for listeners to update, but that also means you wouldn't be able to listen to any child properties.

https://github.com/Nozbe/WatermelonDB/blob/7ad34099b863054da26bf705b64c8cba3a840990/src/Model/index.js#L127 there you are

You were right, it was a direct modification of an observable. I did a hack to make the circular reference not enumerable which seems to work, this is where I'm at
import { isObservable } from '@legendapp/state'
import { useObservable, useObserveEffect } from '@legendapp/state/react'
import { Model, Query } from '@nozbe/watermelondb'
import { useLayoutEffect } from 'react'

export const WatermelonLegendStateAdapter = {
  useRecord: <T extends Model>(record: T) => {
    const o = useObservable<T>(isObservable(record) ? record.peek() : record)
    useLayoutEffect(() => {
      // A value is emitted synchronously when the model is first subscribed to
      return record.experimentalSubscribe(isDeleted => {
        console.log('>>> record title', (record as any).peek().title)
        if (isDeleted) {
          // TODO: handle deletion
        } else {
          // HACK: create a new reference to force update
          o.set(new Proxy(isObservable(record) ? record.peek() : record, {}))
        }
      })
    }, [record, o])
    return o
  },
  useQuery: <T extends Model>(query: Query<T>) => {
    const o = useObservable([] as T[])
    useLayoutEffect(() => {
      // A value is emitted synchronously when the query is first subscribed to
      return query.experimentalSubscribe(records => {
        o.set(records.map(x => x))
      })
    }, [query, o])
    return o
  },
}

It does seem to be reactive and working for the limited use cases that I've tried

Hey so this approach seems to be working nicely. I just have one question, Legend does seem to be significantly slower than Watermelon's homegrown withObservables solution. The extreme case I have profiled is with useQuery with 6000 items returned. Legend ~12ms vs watermelons ~4ms. It seems that Legend traverses every property of every item in the array inside updateNodes. Why is this? The reference for every item should be the same so it should opt out of this? My understanding is that a node is a proxy over the underlying tree data structure used for tracking listeners. But why would there should not be a listener on every item since I'm just doing useQuery?

Are you sure the reference is the same? It should skip over any elements that are the same: https://github.com/LegendApp/legend-state/blob/86d66f3c3ea92400ff931ed3a27df824dcf0afb9/src/ObservableObject.ts#L216
Is this on the initial load where it's adding all new data to the observable or is it an update that's changing them?

Thanks for the quick response, I'm fairly sure it's not from the initial response. I'll verify this when back at a keyboard later this week.

I dug a bit deeper. You were right, the recursive updateNodes cost was on the initial launch not subsequent updates. Though it's not clear to me why there would needs to be a recursive updateNodes when creating the observable, couldn't all the nodes be lazy? I couldn't track down the root of the performance difference between watermelon and legend on updates.
However, I did realize I was on an old version of Legend (0.21.18). I upgraded to latest (1.11.1) which caused all my tests to fail because my proxy hack above, didn't result in the observable updating. I changed the following line
https://github.com/LegendApp/legend-state/blob/86d66f3c3ea92400ff931ed3a27df824dcf0afb9/src/ObservableObject.ts#L705
to
hasADiff = updateNodes(childNode, newValue, prevValue) || prevValue !== newValue;

and all my tests passed again. Also, now that I've upgraded the performance differences between Legend and Watermelon are negligible that I can see, which is great news!
So without using patch-package Legend wouldn't work for my use case.
Do you think it's worth adding a parameter to set would be worthwhile?

Hmm, that's an interesting issue. I can't think of a better way to solve it at the moment, but I'll think about it some more.
That hack would just notify at the root level but not at any of the children, right? So it would work for the initial load but would not update child nodes that they've changed in later updates?

Hmm yeah so the actual updates to the underlying record happen outside of legend so when it comes to doing the diffing it's too late! useComputed(() => record.get().column) does seem to work well though. What could break? Probably useComputed(() => record.column.get()) won't work. These records aren't deeply nested though so that's probably acceptable for my use case.
Maybe I'm just using Legend wrong ¯_(ツ)_/¯

Perhaps a better approach would be to wrap the prepareUpdate method on all the models? 🤔
https://github.com/Nozbe/WatermelonDB/blob/0974c47a559fa4f102f0026afb62d0ee368d01ca/src/Model/index.js#L124

Hey @mfbx9da4 Why are you wrapping legend state with watermelon db??


testRegression test for external hive orc table

Proposed changes
Add regression test for external hive orc table. This PR has generated all basic types support by hive orc, and create a hive external table to touch them in docker environment.
Functions to be tested:

Ensure that all types are parsed correctly
Ensure that the null map of all types are parsed correctly
Ensure that the SearchArgument of OrcReader works well
Only select partition columns

Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.23 seconds
load time: 573 seconds
storage size:<PHONE_NUMBER>0 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221028174646_clickbench_pr_35279.html


Update CHANGELOG.md

5.8.0 doesn't exist in versions. So it is removed from the ChangeLog file.

@leventkantaroglu Thanks


[VS Code] Collaboration session not found.

error.zip
Error:
Collaboration session not found.
Steps to Reproduce:

Version Data

extensionName
VSLS

extensionVersion
0.3.735

protocolVersion
2.2

applicationName
VSCode

applicationVersion
1.25.1

platformName
Linux

platformVersion
4.13.0-45-generic

This error generally indicates that the session you're trying to join has ended. That said, I'm going to close this as a duplicate of #1003, so that we can discuss it via a single issue. Thanks!


Consider using @NotNull annotations

@NotNull annotations are not used in jbanking : so far keeping the library completely dependency-free was preferred. But this annotation helps static code analysis, can be really helpful in IDEs or with some languages like Kotlin.
Should we consider using @NotNull annotations ?

A few links :

https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use
https://jspecify.dev/
https://github.com/marcwrobel/jbanking/pull/223#discussion_r1000376897

with some languages like Kotlin.

Or switch to Kotlin for its null-safely and less boilerplate, targeting the JVM only first, but eventually opening the target audience to multi-platform?


Marky/mongodb update

Fix volume permissions for mongodb
Two commits [*] got lost on the update to the newer mongodb chart; putting them back in again.
[*]
b1b506ad0e9fa4457cee9db9f14d65dd3c9fad0e
f32c426dc47c3e3b5fe7228974cce83cb6cccf2e

Sorry, pulling against the wrong repo.


KeyError: "j" in latest version of dask/dask and dask/distributed

What happened:
I ran into https://github.com/dmlc/xgboost/issues/6577 trying to build a distributed xgboost example - full stack trace available there, will post here as well.
After investigation, the issue is not related to xgboost and is reproducible with dask, distributed, and a few others when:

dask==2020.12.0
distributed==2020.12.0

Downgrading both to 2.30.0 fixes the issue.
What you expected to happen:
Dask arrays can be persisted and used in xgboost.dask
Minimal Complete Verifiable Example:
import pandas as pd
import dask.dataframe as dd

from distributed import Client
from adlfs import AzureBlobFileSystem

# setup variables
container_name = "malware"
storage_options = {"account_name": "azuremlexamples"}

# create distributed client
c = Client()

# create Azure filesystem
fs = AzureBlobFileSystem(**storage_options)

# list files
files = fs.ls(f"{container_name}/processed")

# read in training data
for f in files:
    if "train" in f:
        df = dd.read_parquet(f"az://{f}", storage_options=storage_options)

# advanced feature engineering
cols = [col for col in df.columns if df.dtypes[col] != "object"]

# define system input and output
## FAILS HERE
X = df[cols].drop("HasDetections", axis=1).values.persist()
y = df["HasDetections"].values.persist()

# print something
print(len(X))
print(len(y))

Anything else we need to know?:
Slightly more realistic code with environment.yml used to reproduce available here: https://github.com/Azure/azureml-examples/pull/324
Environment:

Dask version: 2020.12.0
Python version: 3.8
Operating System: macos, linux
Install method (conda, pip, source): pip

Stack Trace:

KeyError: 'j'
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-21-87b8ab28e9c9> in <module>
      1 import xgboost as xgb
      2
----> 3 dtrain = xgb.dask.DaskDMatrix(c, X, y)
      4 dtrain
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/xgboost/dask.py in __init__(self, client, data, label, missing, weight, base_margin, label_lower_bound, label_upper_bound, feature_names, feature_types)
    225         self.is_quantile = False
    226
--> 227         self._init = client.sync(self.map_local_data,
    228                                  client, data, label=label, weights=weight,
    229                                  base_margin=base_margin,
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    829             return future
    830         else:
--> 831             return sync(
    832                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    833             )
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    338     if error[0]:
    339         typ, exc, tb = error[0]
--> 340         raise exc.with_traceback(tb)
    341     else:
    342         return result[0]
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/utils.py in f()
    322             if callback_timeout is not None:
    323                 future = asyncio.wait_for(future, callback_timeout)
--> 324             result[0] = yield future
    325         except Exception as exc:
    326             error[0] = sys.exc_info()
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/tornado/gen.py in run(self)
    760
    761                     try:
--> 762                         value = future.result()
    763                     except Exception:
    764                         exc_info = sys.exc_info()
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/xgboost/dask.py in map_local_data(self, client, data, label, weights, base_margin, label_lower_bound, label_upper_bound)
    255                 ' chunks=(partition_size, X.shape[1])'
    256
--> 257         data = data.persist()
    258         for meta in [label, weights, base_margin, label_lower_bound,
    259                      label_upper_bound]:
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in persist(self, **kwargs)
    250         dask.base.persist
    251         """
--> 252         (result,) = persist(self, traverse=False, **kwargs)
    253         return result
    254
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in persist(*args, **kwargs)
    757             else:
    758                 if client.get == schedule:
--> 759                     results = client.persist(
    760                         collections, optimize_graph=optimize_graph, **kwargs
    761                     )
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in persist(self, collections, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, **kwargs)
   2974         assert all(map(dask.is_dask_collection, collections))
   2975
-> 2976         dsk = self.collections_to_dsk(collections, optimize_graph, **kwargs)
   2977
   2978         names = {k for c in collections for k in flatten(c.__dask_keys__())}
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in collections_to_dsk(collections, *args, **kwargs)
   3960     @staticmethod
   3961     def collections_to_dsk(collections, *args, **kwargs):
-> 3962         return collections_to_dsk(collections, *args, **kwargs)
   3963
   3964     def get_task_stream(
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in collections_to_dsk(collections, optimize_graph, **kwargs)
    330             dsk, keys = _extract_graph_and_keys(val)
    331             groups[opt] = (dsk, keys)
--> 332             _opt = opt(dsk, keys, **kwargs)
    333             _opt_list.append(_opt)
    334
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/array/optimization.py in optimize(dsk, keys, fuse_keys, fast_functions, inline_functions_fast_functions, rename_fused_keys, **kwargs)
     46     dsk = optimize_blockwise(dsk, keys=keys)
     47     dsk = fuse_roots(dsk, keys=keys)
---> 48     dsk = dsk.cull(set(keys))
     49
     50     if not config.get("optimization.fuse.active"):
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/highlevelgraph.py in cull(self, keys)
    625         """
    626
--> 627         all_ext_keys = self.get_all_external_keys()
    628         ret_layers = {}
    629         ret_key_deps = {}
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/highlevelgraph.py in get_all_external_keys(self)
    516             self._all_external_keys = set()
    517             for layer in self.layers.values():
--> 518                 self._all_external_keys.update(layer.get_output_keys())
    519         return self._all_external_keys
    520
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/blockwise.py in get_output_keys(self)
    283             (self.output, *p)
    284             for p in itertools.product(
--> 285                 *[range(self.dims[i]) for i in self.output_indices]
    286             )
    287         }
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/blockwise.py in <listcomp>(.0)
    283             (self.output, *p)
    284             for p in itertools.product(
--> 285                 *[range(self.dims[i]) for i in self.output_indices]
    286             )
    287         }
KeyError: 'j'

driver log from AML which fails on persist() instead of the xgboost.dask call:
[2021-01-07T17:33:10.126644] Entering context manager injector.
[context_manager_injector.py] Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError', 'UserExceptions:context_managers.UserExceptions'], invocation=['run.py'])
This is an MPI job. Rank:1
Script type = None
Starting the daemon thread to refresh tokens in background for process with pid = 121
Entering Run History Context Manager.
[2021-01-07T17:33:12.549591] Current directory: /mnt/batch/tasks/shared/LS_root/jobs/default/azureml/xgboost-cpu-tutorial_1610040108_f29decf7/mounts/workspaceblobstore/azureml/xgboost-cpu-tutorial_1610040108_f29decf7
[2021-01-07T17:33:12.557936] Preparing to call script [run.py] with arguments:[]
[2021-01-07T17:33:12.558039] After variable expansion, calling script [run.py] with arguments:[]

initializing...
<Client: 'tcp://<IP_ADDRESS>:41157' processes=1 threads=1, memory=3.58 GB>
creating dataframes...
processing data...
Starting the daemon thread to refresh tokens in background for process with pid = 121

[2021-01-07T17:33:18.136621] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
1 items cleaning up...
Cleanup took 0.0627279281616211 seconds
Starting the daemon thread to refresh tokens in background for process with pid = 121
Traceback (most recent call last):
  File "run.py", line 33, in <module>
    X = df_train[cols].drop("HasDetections", axis=1).values.persist()
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/base.py", line 252, in persist
    (result,) = persist(self, traverse=False, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/base.py", line 759, in persist
    results = client.persist(
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/distributed/client.py", line 2976, in persist
    dsk = self.collections_to_dsk(collections, optimize_graph, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/distributed/client.py", line 3962, in collections_to_dsk
    return collections_to_dsk(collections, *args, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/base.py", line 332, in collections_to_dsk
    _opt = opt(dsk, keys, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/array/optimization.py", line 48, in optimize
    dsk = dsk.cull(set(keys))
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/highlevelgraph.py", line 627, in cull
    all_ext_keys = self.get_all_external_keys()
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/highlevelgraph.py", line 518, in get_all_external_keys
    self._all_external_keys.update(layer.get_output_keys())
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/blockwise.py", line 285, in get_output_keys
    *[range(self.dims[i]) for i in self.output_indices]
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/blockwise.py", line 285, in <listcomp>
    *[range(self.dims[i]) for i in self.output_indices]
KeyError: 'j'

[2021-01-07T17:33:18.397146] Finished context manager injector with Exception.

driver log from AML which fails on persist() instead of the xgboost.dask call:
[2021-01-07T17:33:10.126644] Entering context manager injector.
[context_manager_injector.py] Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError', 'UserExceptions:context_managers.UserExceptions'], invocation=['run.py'])
This is an MPI job. Rank:1
Script type = None
Starting the daemon thread to refresh tokens in background for process with pid = 121
Entering Run History Context Manager.
[2021-01-07T17:33:12.549591] Current directory: /mnt/batch/tasks/shared/LS_root/jobs/default/azureml/xgboost-cpu-tutorial_1610040108_f29decf7/mounts/workspaceblobstore/azureml/xgboost-cpu-tutorial_1610040108_f29decf7
[2021-01-07T17:33:12.557936] Preparing to call script [run.py] with arguments:[]
[2021-01-07T17:33:12.558039] After variable expansion, calling script [run.py] with arguments:[]

initializing...
<Client: 'tcp://<IP_ADDRESS>:41157' processes=1 threads=1, memory=3.58 GB>
creating dataframes...
processing data...
Starting the daemon thread to refresh tokens in background for process with pid = 121

[2021-01-07T17:33:18.136621] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
1 items cleaning up...
Cleanup took 0.0627279281616211 seconds
Starting the daemon thread to refresh tokens in background for process with pid = 121
Traceback (most recent call last):
  File "run.py", line 33, in <module>
    X = df_train[cols].drop("HasDetections", axis=1).values.persist()
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/base.py", line 252, in persist
    (result,) = persist(self, traverse=False, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/base.py", line 759, in persist
    results = client.persist(
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/distributed/client.py", line 2976, in persist
    dsk = self.collections_to_dsk(collections, optimize_graph, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/distributed/client.py", line 3962, in collections_to_dsk
    return collections_to_dsk(collections, *args, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/base.py", line 332, in collections_to_dsk
    _opt = opt(dsk, keys, **kwargs)
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/array/optimization.py", line 48, in optimize
    dsk = dsk.cull(set(keys))
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/highlevelgraph.py", line 627, in cull
    all_ext_keys = self.get_all_external_keys()
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/highlevelgraph.py", line 518, in get_all_external_keys
    self._all_external_keys.update(layer.get_output_keys())
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/blockwise.py", line 285, in get_output_keys
    *[range(self.dims[i]) for i in self.output_indices]
  File "/azureml-envs/azureml_ffbb10db4a6af4f6397c992e16ad5e69/lib/python3.8/site-packages/dask/blockwise.py", line 285, in <listcomp>
    *[range(self.dims[i]) for i in self.output_indices]
KeyError: 'j'

[2021-01-07T17:33:18.397146] Finished context manager injector with Exception.

Are there any errors on the worker ?  Perhaps something which looks like:

TypeError: can not serialize 'CreateArraySubgraph' object

?

Are there any errors on the worker ?  Perhaps something which looks like:

TypeError: can not serialize 'CreateArraySubgraph' object

?

this reproduces locally as well, would that error still be possible?

this reproduces locally as well, would that error still be possible?

local stack trace looks the same, and I don't see anything in dask-worker-space to really check?
KeyError: 'j'
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-8333897085d4> in <module>
----> 1 X = df_train[cols].drop("HasDetections", axis=1).values.persist()
      2 X
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in persist(self, **kwargs)
    250         dask.base.persist
    251         """
--> 252         (result,) = persist(self, traverse=False, **kwargs)
    253         return result
    254
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in persist(*args, **kwargs)
    757             else:
    758                 if client.get == schedule:
--> 759                     results = client.persist(
    760                         collections, optimize_graph=optimize_graph, **kwargs
    761                     )
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in persist(self, collections, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, **kwargs)
   2974         assert all(map(dask.is_dask_collection, collections))
   2975
-> 2976         dsk = self.collections_to_dsk(collections, optimize_graph, **kwargs)
   2977
   2978         names = {k for c in collections for k in flatten(c.__dask_keys__())}
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in collections_to_dsk(collections, *args, **kwargs)
   3960     @staticmethod
   3961     def collections_to_dsk(collections, *args, **kwargs):
-> 3962         return collections_to_dsk(collections, *args, **kwargs)
   3963
   3964     def get_task_stream(
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in collections_to_dsk(collections, optimize_graph, **kwargs)
    330             dsk, keys = _extract_graph_and_keys(val)
    331             groups[opt] = (dsk, keys)
--> 332             _opt = opt(dsk, keys, **kwargs)
    333             _opt_list.append(_opt)
    334
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/array/optimization.py in optimize(dsk, keys, fuse_keys, fast_functions, inline_functions_fast_functions, rename_fused_keys, **kwargs)
     46     dsk = optimize_blockwise(dsk, keys=keys)
     47     dsk = fuse_roots(dsk, keys=keys)
---> 48     dsk = dsk.cull(set(keys))
     49
     50     if not config.get("optimization.fuse.active"):
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/highlevelgraph.py in cull(self, keys)
    625         """
    626
--> 627         all_ext_keys = self.get_all_external_keys()
    628         ret_layers = {}
    629         ret_key_deps = {}
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/highlevelgraph.py in get_all_external_keys(self)
    516             self._all_external_keys = set()
    517             for layer in self.layers.values():
--> 518                 self._all_external_keys.update(layer.get_output_keys())
    519         return self._all_external_keys
    520
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/blockwise.py in get_output_keys(self)
    283             (self.output, *p)
    284             for p in itertools.product(
--> 285                 *[range(self.dims[i]) for i in self.output_indices]
    286             )
    287         }
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/blockwise.py in <listcomp>(.0)
    283             (self.output, *p)
    284             for p in itertools.product(
--> 285                 *[range(self.dims[i]) for i in self.output_indices]
    286             )
    287         }
KeyError: 'j'

local stack trace looks the same, and I don't see anything in dask-worker-space to really check?
KeyError: 'j'
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-8333897085d4> in <module>
----> 1 X = df_train[cols].drop("HasDetections", axis=1).values.persist()
      2 X
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in persist(self, **kwargs)
    250         dask.base.persist
    251         """
--> 252         (result,) = persist(self, traverse=False, **kwargs)
    253         return result
    254
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in persist(*args, **kwargs)
    757             else:
    758                 if client.get == schedule:
--> 759                     results = client.persist(
    760                         collections, optimize_graph=optimize_graph, **kwargs
    761                     )
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in persist(self, collections, optimize_graph, workers, allow_other_workers, resources, retries, priority, fifo_timeout, actors, **kwargs)
   2974         assert all(map(dask.is_dask_collection, collections))
   2975
-> 2976         dsk = self.collections_to_dsk(collections, optimize_graph, **kwargs)
   2977
   2978         names = {k for c in collections for k in flatten(c.__dask_keys__())}
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/distributed/client.py in collections_to_dsk(collections, *args, **kwargs)
   3960     @staticmethod
   3961     def collections_to_dsk(collections, *args, **kwargs):
-> 3962         return collections_to_dsk(collections, *args, **kwargs)
   3963
   3964     def get_task_stream(
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/base.py in collections_to_dsk(collections, optimize_graph, **kwargs)
    330             dsk, keys = _extract_graph_and_keys(val)
    331             groups[opt] = (dsk, keys)
--> 332             _opt = opt(dsk, keys, **kwargs)
    333             _opt_list.append(_opt)
    334
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/array/optimization.py in optimize(dsk, keys, fuse_keys, fast_functions, inline_functions_fast_functions, rename_fused_keys, **kwargs)
     46     dsk = optimize_blockwise(dsk, keys=keys)
     47     dsk = fuse_roots(dsk, keys=keys)
---> 48     dsk = dsk.cull(set(keys))
     49
     50     if not config.get("optimization.fuse.active"):
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/highlevelgraph.py in cull(self, keys)
    625         """
    626
--> 627         all_ext_keys = self.get_all_external_keys()
    628         ret_layers = {}
    629         ret_key_deps = {}
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/highlevelgraph.py in get_all_external_keys(self)
    516             self._all_external_keys = set()
    517             for layer in self.layers.values():
--> 518                 self._all_external_keys.update(layer.get_output_keys())
    519         return self._all_external_keys
    520
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/blockwise.py in get_output_keys(self)
    283             (self.output, *p)
    284             for p in itertools.product(
--> 285                 *[range(self.dims[i]) for i in self.output_indices]
    286             )
    287         }
~/miniconda3/envs/dkdc/lib/python3.8/site-packages/dask/blockwise.py in <listcomp>(.0)
    283             (self.output, *p)
    284             for p in itertools.product(
--> 285                 *[range(self.dims[i]) for i in self.output_indices]
    286             )
    287         }
KeyError: 'j'

cc @rjzamora

cc @rjzamora

Thanks for raising this @lostmygithubaccount - There are known serialization issues with the new BlockwiseIO implementation (used by read_parquet) that absolutely need to be fixed.  The problem may require a slight redesign of BlockwiseIO, but I am working on it.  Your issue may be a separate problem, but it probably makes sense to wait on the redesign... Sorry for the trouble.

Thanks for raising this @lostmygithubaccount - There are known serialization issues with the new BlockwiseIO implementation (used by read_parquet) that absolutely need to be fixed.  The problem may require a slight redesign of BlockwiseIO, but I am working on it.  Your issue may be a separate problem, but it probably makes sense to wait on the redesign... Sorry for the trouble.

@lostmygithubaccount could you try with the latest release of xgboost / dask / distributed? There's been several updates since you originally ran into this issue. I wonder if this issue still exists

this issue is no longer present

Great, thanks for following up @lostmygithubaccount!


Neg and Dec

Hi. Thank you for your code. This might be out of scope, but I wondered if Dec() should be prefixed with a "-" when Sign < 0.
package main

import (
    "testing"

    "github.com/holiman/uint256"
)

func TestDec(t *testing.T) {
    a := uint256.MustFromHex("0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff")
    if a.Dec() != "-1" {
        t.Errorf("want: -1 got: %s", a.Dec())
    }
    b := uint256.NewInt(0).Neg(uint256.NewInt(1))
    if b.Dec() != "-1" {
        t.Errorf("want: -1 got: %s", b.Dec())
    }
}

--- FAIL: TestDec (0.00s)
    main_test.go:12: want: -1 got:<PHONE_NUMBER>37316195423570985008687907853269984665640564039457584007913129639935
    main_test.go:16: want: -1 got:<PHONE_NUMBER>37316195423570985008687907853269984665640564039457584007913129639935
    ```

The Neg function does not alter the fact that uint256 Ints are inherently unsigned. The Neg simply transforms it into another big unsigned number, which happens to be the negative complement.
But the Int has no internal notion of being negative, and cannot be made to output -


Refactor integration test playbooks

Instead of doing the checkout of the code to test in the pull request
job, the code should be checked out before the job begins.
Signed-off-by: Steve Kuznetsov<EMAIL_ADDRESS>/cc @mrunalp

+1
This is the "standard" way of doing things in CI testing.  Where the testing-system is in control of putting the repo. in the proper state.
Later: These (removed) steps can always be brought in as a role, allowing alternate playbooks to make use of them.  If/when that's ever important.  Until then, good riddance.

Broke things out a little bit as reading through a sea of shell: was making me cry -- @runcom @mrunalp

@stevekuznetsov  The changes look great :)

@stevekuznetsov eek, include files?  Why not go straight to roles: https://github.com/kubernetes-incubator/cri-o/pull/916

Also I foresee some conflicts w/  #928 (needed to divorce PB from "hidden" dependencies).

(though I'm super-duper in favor of breaking down the wall-of-tasks in general)

After having worked with Ansible for over a year I feel like I still don't "get" roles. I am not missing any functionality I need in this approach, so unless there is something concrete to gain with roles, it feels like too much effort. There will be conflicts with that playbook, but hopefully we can reconcile them here -- all of the additions you made should layer into system.yml AFAICT.

@runcom could you please explain something -- I am getting through all the builds but failing at:
TASK [enable and start CRI-O] ***********************************************************************************************************
task path: /data/src/github.com/kubernetes-incubator/cri-o/contrib/test/integration/build/cri-o.yml:80
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to start service crio: Job for crio.service failed because the control process exited with error code. See \"systemctl status crio.service\" and \"journalctl -xe\" for details.\n"}
	to retry, use: --limit @/data/src/github.com/kubernetes-incubator/cri-o/contrib/test/integration/main.retry

I install runc like this:
- name: install runc
  make:
    target: "install"
    chdir: "{{ ansible_env.GOPATH }}/src/github.com/opencontainers/runc"

This results in:
$ which runc
/usr/local/sbin/runc

However, starting cri-o fails with:
Sep 20 23:46:06 ip-172-18-8-146.ec2.internal crio[19580]: time="2017-09-20 23:46:00.587178580-04:00" level=fatal msg="invalid --runtime v

@runcom FWIW the code to install runc was the same before:
https://github.com/kubernetes-incubator/cri-o/blob/master/contrib/test/crio-integration-playbook.yaml#L228-L231

@stevekuznetsov Missing this https://github.com/kubernetes-incubator/cri-o/blob/master/contrib/test/crio-integration-playbook.yaml#L409
Another option is to change the runtime in /etc/crio/crio.conf to point to where runc is installed.

@mrunalp ah thanks, good catch

After having worked with Ansible for over a year I feel like I still don't "get" roles.

FWIW (and respectfully) It's not important distinction now, only a suggestion - The "modular" concept is more imporant, so 👍 and no wounded feelings either way.
---details---
Roles are just "include" level 2 😀.
My experience is they become more important down the road as these things evolve (IMHO).  With complexity comes the need for performance and to define (and document) default and static variable-values, dependencies[*], and to organize files and templates.  Roles are also required for using the "free" strategy, which can have a large performance benefit - no hosts will block on other hosts tasks completing.
Otherwise, roles look exactly like 'includes', just with a different path and filename - That's why I suggest them out of the gate.
[*]Role dependencies support a clever technique:  Dynamically notifying  if/when any role fails on one host,  while the play keeps running on another. That makes for easier debugging, showing the engineer exactly what failed, when, and where.  Otherwise, role-dependencies are pretty confusing in practice, and I tend to avoid them.

Looks like there is no way to async include a task file ... so we will have no parallelism on the builds. That's not the best.

@cevich yeah so once (if, and I really don't think so) we need to organize templates, defaults, etc for these things, sure, roles make sense. Today, seems like overkill. Especially since 99% of this thing is non-idempotent shell tasks / make tasks anyway.

Looks like there is no way to async include a task file ... so we will have no parallelism on the builds. That's not the best.

No but...you can async execute a script and maybe most of what's here could just be a script (haven't looked)?
e.g.
./roles/somename/files/script.sh:
    ...stuff...

./roles/somename/tasks/main.yml:
- name: Do some stuff
  script: 'script.sh'
  async: #whatever

There's also a way you can run a bunch of scripts (assuming they don't need synchronization) then wait for them all to complete:
Example from http://docs.ansible.com/ansible/latest/playbooks_async.html
---
# Requires ansible 1.8+
- name: 'YUM - fire and forget task'
  yum: name=docker-io state=installed
  async: 1000
  poll: 0
  register: yum_sleeper

...other stuff...

- name: 'YUM - check on fire and forget task'
  async_status: jid={{ yum_sleeper.ansible_job_id }}
  register: job_result
  until: job_result.finished
  retries: 30

shell tasks / make tasks anyway.

Yep, np at all m8.

N/B: The free-strategy will do async roles, but the output will absolutely drive you nutz trying to debug.  it really needs the callback_plugin from #935  Again...nothing we need to worry about now, just an FYI.

The code I had before was using that example @cevich ... but you can't use async and register on include ... and I want to be able to break this up into smaller, more manageable chunks. I don't like 1k+LOC playbooks.

@cevich can free strategy do synchronous tasks, then a set of roles in parallel, then after they're all done another set of synchronous tasks?

@cevich can free strategy do synchronous tasks, then a set of roles in parallel, then after they're all done another set of synchronous tasks?
Yes...kind of...the 'strategy' scope is just the "play", and you can have more than one in a playbook.  e.g.
- hosts: "{{ hosts_a | default(all) }}"
  strategy: linear  # the default
  ...stuff...

- hosts: "{{ hosts_b | default(all) }}"
  strategy: free
  ...stuff...

- hosts: "{{ hosts_c | default(all) }}"
  strategy: linear
  ...stuff...

In the above, the middle play will block on the first one completing.

I don't like 1k+LOC playbooks.

You and me both.

Hmm, ok. If we see this take forever and a day to finish, we can investigate the parallel approach.

/test rhel

lol

I OWN you @openshift-ci-robot, don't get sassy with me

/test rhel

/test rhel

/test rhel

/test rhel

/test rhel

/test rhel

@mrunalp it's live :)

@stevekuznetsov Awesome :)

@mrunalp moving forward I think we need to do three things:

make sure this works on Fedora
bake all of the set-up into the AMI
break this up into two jobs (per OS) -- integration and e2e

The last two points should bring runtime down, right now it's too long.

/test rhel

@mrunalp saw an integration test fail -- is this known? logs:

not ok 27 ctr oom
# (in test file ./ctr.bats, line 726)
#   `[[ "$output" =~ "OOMKilled" ]]' failed
# 0
# time="2017-09-22 01:47:14.073775586-04:00" level=debug msg="[graphdriver] trying provided driver "overlay2""
# time="2017-09-22 01:47:14.075605969-04:00" level=warning msg="Using pre-4.0.0 kernel for overlay, mount failures may require kernel update"
# time="2017-09-22 01:47:14.077775605-04:00" level=info msg="CNI network crionet (type=bridge) is used from /tmp/tmp.KihhEGanJw/cni/net.d/10-crio.conf"
# time="2017-09-22 01:47:14.077897268-04:00" level=info msg="CNI network crionet (type=bridge) is used from /tmp/tmp.KihhEGanJw/cni/net.d/10-crio.conf"
# time="2017-09-22 01:47:14.087823775-04:00" level=debug msg="Golang's threads limit set to 114030"
# time="2017-09-22 01:47:14.088815078-04:00" level=debug msg="sandboxes: []"
# time="2017-09-22 01:47:15.212205167-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:redis:alpine,},}"
# time="2017-09-22 01:47:15.212401540-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/library/redis:alpine""
# time="2017-09-22 01:47:15.213698169-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:735ca8b5afae9accc85f7216de371698d3cdedeaec697374c24fe6852ffd92dd,RepoTags:[docker.io/library/redis:alpine redis:alpine],RepoDigests:[],Size_:27500208,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.285254636-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:redis:alpine,},}"
# time="2017-09-22 01:47:15.285359007-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/library/redis:alpine""
# time="2017-09-22 01:47:15.285727041-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:735ca8b5afae9accc85f7216de371698d3cdedeaec697374c24fe6852ffd92dd,RepoTags:[docker.io/library/redis:alpine redis:alpine],RepoDigests:[],Size_:27500208,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.347240768-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:mrunalp/oom,},}"
# time="2017-09-22 01:47:15.347356507-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/mrunalp/oom:latest""
# time="2017-09-22 01:47:15.347657144-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:64b399673fc200859d677177df69e26453997a697306bc6b46e3a3fcfdd8350a,RepoTags:[docker.io/mrunalp/oom:latest mrunalp/oom docker.io/library/mrunalp/oom],RepoDigests:[],Size_:5719481,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.413787526-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:redis@sha256:03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b,},}"
# time="2017-09-22 01:47:15.413932983-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/library/redis:latest@03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b""
# time="2017-09-22 01:47:15.414288126-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b,RepoTags:[docker.io/library/redis:latest redis@sha256:03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b docker.io/library/redis@sha256:03789f402b2ecfb98184bf128d180f398f81c63364948ff1454583b02442f73b],RepoDigests:[],Size_:19826844,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.476858904-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:runcom/stderr-test,},}"
# time="2017-09-22 01:47:15.476971116-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/runcom/stderr-test:latest""
# time="2017-09-22 01:47:15.477190939-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:663e758a245c01644231c2fa1e9aab16362a86000292fe61891a865344f84ffe,RepoTags:[docker.io/runcom/stderr-test:latest runcom/stderr-test:latest],RepoDigests:[],Size_:4906035,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.549447626-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:runcom/stderr-test,},}"
# time="2017-09-22 01:47:15.549551558-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/runcom/stderr-test:latest""
# time="2017-09-22 01:47:15.549775628-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:663e758a245c01644231c2fa1e9aab16362a86000292fe61891a865344f84ffe,RepoTags:[docker.io/runcom/stderr-test:latest runcom/stderr-test:latest],RepoDigests:[],Size_:4906035,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.610605344-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:busybox,},}"
# time="2017-09-22 01:47:15.610694876-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/library/busybox:latest""
# time="2017-09-22 01:47:15.610892977-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:27aeace1040eb5f22ce51fca99236425abff8ac52eb18b891845d21f39e04c69,RepoTags:[docker.io/library/busybox:latest busybox:latest],RepoDigests:[],Size_:1131313,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.682125063-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:busybox,},}"
# time="2017-09-22 01:47:15.682231284-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/library/busybox:latest""
# time="2017-09-22 01:47:15.682449150-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:27aeace1040eb5f22ce51fca99236425abff8ac52eb18b891845d21f39e04c69,RepoTags:[docker.io/library/busybox:latest busybox:latest],RepoDigests:[],Size_:1131313,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.742357624-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:mrunalp/image-volume-test,},}"
# time="2017-09-22 01:47:15.742453366-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/mrunalp/image-volume-test:latest""
# time="2017-09-22 01:47:15.742671735-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:9769c418f4c18ebea9dd30d74b5e4610dc950a2cf82b88d8052cf9d92c8d12b4,RepoTags:[docker.io/mrunalp/image-volume-test:latest mrunalp/image-volume-test docker.io/library/mrunalp/image-volume-test],RepoDigests:[],Size_:1095628,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.812702581-04:00" level=debug msg="ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:mrunalp/image-volume-test,},}"
# time="2017-09-22 01:47:15.812783759-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/mrunalp/image-volume-test:latest""
# time="2017-09-22 01:47:15.812959150-04:00" level=debug msg="ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:9769c418f4c18ebea9dd30d74b5e4610dc950a2cf82b88d8052cf9d92c8d12b4,RepoTags:[docker.io/mrunalp/image-volume-test:latest mrunalp/image-volume-test docker.io/library/mrunalp/image-volume-test],RepoDigests:[],Size_:1095628,Uid:nil,Username:,},}"
# time="2017-09-22 01:47:15.874142938-04:00" level=debug msg="RunPodSandboxRequest &RunPodSandboxRequest{Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:podsandbox1,Uid:redhat-test-crio,Namespace:redhat.test.crio,Attempt:1,},Hostname:crioctl_host,LogDirectory:,DnsConfig:&DNSConfig{Servers:[],Searches:[<IP_ADDRESS>],Options:[],},PortMappings:[],Labels:map[string]string{group: test,},Annotations:map[string]string{owner: hmeng,security.alpha.kubernetes.io/seccomp/pod: unconfined,security.alpha.kubernetes.io/sysctls: kernel.shm_rmid_forced=1,net.ipv4.ip_local_port_range=1024 65000,security.alpha.kubernetes.io/unsafe-sysctls: kernel.msgmax=8192,},Linux:&LinuxPodSandboxConfig{CgroupParent:/Burstable/pod_123-456,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{HostNetwork:false,HostPid:false,HostIpc:false,},SelinuxOptions:&SELinuxOption{User:system_u,Role:system_r,Type:svirt_lxc_net_t,Level:s0:c4,c5,},RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,},Sysctls:map[string]string{},},},}"
# time="2017-09-22 01:47:15.874243275-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/kubernetes/pause:latest""
# time="2017-09-22 01:47:15.874581958-04:00" level=debug msg="exporting opaque data as blob "sha256:686426c1fd47f774a042ebbcb2daeeb1c8a391375d68b89b71385cfbc3e24d42""
# time="2017-09-22 01:47:15.890404316-04:00" level=debug msg="created pod sandbox "670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112""
# time="2017-09-22 01:47:15.894099725-04:00" level=debug msg="pod sandbox "670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112" has work directory "/tmp/tmp.KihhEGanJw/crio/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata""
# time="2017-09-22 01:47:15.894188914-04:00" level=debug msg="pod sandbox "670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112" has run directory "/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata""
# time="2017-09-22 01:47:16.010816214-04:00" level=debug msg="mounted container "670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112" at "/tmp/tmp.KihhEGanJw/crio/overlay2/f026dc9dfd179fee2b17ef9f82862ab172eac082a2483ac2c68bf5f7e5cec512/merged""
# time="2017-09-22 01:47:16.010980030-04:00" level=info msg="Got pod network {Name:podsandbox1 Namespace:redhat.test.crio ID:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 NetNS:/var/run/netns/k8s_podsandbox1_redhat.test.crio_redhat-test-crio_1-4735adb0 PortMappings:[]}"
# time="2017-09-22 01:47:16.010992935-04:00" level=info msg="About to add CNI network cni-loopback (type=loopback)"
# time="2017-09-22 01:47:16.014160055-04:00" level=info msg="Got pod network {Name:podsandbox1 Namespace:redhat.test.crio ID:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 NetNS:/var/run/netns/k8s_podsandbox1_redhat.test.crio_redhat-test-crio_1-4735adb0 PortMappings:[]}"
# time="2017-09-22 01:47:16.014179465-04:00" level=info msg="About to add CNI network crionet (type=bridge)"
# time="2017-09-22 01:47:16.087725025-04:00" level=debug msg="running conmon: /data/src/github.com/kubernetes-incubator/cri-o/conmon/conmon" args=[-c 670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 -u 670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 -r /bin/runc -b /tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata -p /tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata/pidfile -l /var/log/crio/pods/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112.log --exit-dir /var/run/crio/exits]
# time="2017-09-22 01:47:16.153408565-04:00" level=debug msg="Received container pid: 11686"
# time="2017-09-22 01:47:18.144100677-04:00" level=debug msg="RunPodSandboxResponse: &RunPodSandboxResponse{PodSandboxId:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112,}"
# 670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112
# time="2017-09-22 01:47:18.232623739-04:00" level=debug msg="CreateContainerRequest &CreateContainerRequest{PodSandboxId:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112,Config:&ContainerConfig{Metadata:&ContainerMetadata{Name:container1,Attempt:1,},Image:&ImageSpec{Image:mrunalp/oom,},Command:[/oom],Args:[],WorkingDir:/,Envs:[&KeyValue{Key:PATH,Value:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,} &KeyValue{Key:TERM,Value:xterm,} &KeyValue{Key:TESTDIR,Value:test/dir1,} &KeyValue{Key:TESTFILE,Value:test/file1,}],Mounts:[],Devices:[],Labels:map[string]string{batch: no,type: small,},Annotations:map[string]string{daemon: crio,owner: dragon,},LogPath:,Stdin:false,StdinOnce:false,Tty:false,Linux:&LinuxContainerConfig{Resources:&LinuxContainerResources{CpuPeriod:10000,CpuQuota:20000,CpuShares:512,MemoryLimitInBytes:5120000,OomScoreAdj:30,},SecurityContext:&LinuxContainerSecurityContext{Capabilities:&Capability{AddCapabilities:[setuid setgid],DropCapabilities:[],},Privileged:false,NamespaceOptions:nil,SelinuxOptions:&SELinuxOption{User:system_u,Role:system_r,Type:svirt_lxc_net_t,Level:s0:c4,c5,},RunAsUser:nil,RunAsUsername:,ReadonlyRootfs:false,SupplementalGroups:[],ApparmorProfile:,},},},SandboxConfig:nil,}"
# time="2017-09-22 01:47:18.232862501-04:00" level=debug msg="setting container's log_path" ctr.logfile= log_path="/var/log/crio/pods/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1.log" sbox.logdir="/var/log/crio/pods/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112"
# time="2017-09-22 01:47:18.232962302-04:00" level=debug msg="pod container state &{State:{Version:1.0.0 ID:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 Status:running Pid:11686 Bundle:/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata Annotations:map[group:test io.kubernetes.cri-o.MountPoint:/tmp/tmp.KihhEGanJw/crio/overlay2/f026dc9dfd179fee2b17ef9f82862ab172eac082a2483ac2c68bf5f7e5cec512/merged io.kubernetes.cri-o.PrivilegedRuntime:false owner:hmeng io.kubernetes.cri-o.LogPath:/var/log/crio/pods/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112.log io.kubernetes.cri-o.SandboxID:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 security.alpha.kubernetes.io/seccomp/pod:unconfined security.alpha.kubernetes.io/unsafe-sysctls:kernel.msgmax=8192 io.kubernetes.cri-o.ContainerType:sandbox io.kubernetes.cri-o.HostName:crioctl_host io.kubernetes.cri-o.HostnamePath:/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata/hostname io.kubernetes.cri-o.IP:<IP_ADDRESS> io.kubernetes.cri-o.ShmPath:/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata/shm security.alpha.kubernetes.io/sysctls:kernel.shm_rmid_forced=1,net.ipv4.ip_local_port_range=1024 65000 io.kubernetes.cri-o.ContainerName:k8s_POD_podsandbox1_redhat.test.crio_redhat-test-crio_1 io.kubernetes.cri-o.Created:2017-09-22T01:47:15.897251294-04:00 io.kubernetes.cri-o.Metadata:{"name":"podsandbox1","uid":"redhat-test-crio","namespace":"redhat.test.crio","attempt":1} io.kubernetes.cri-o.Name:k8s_podsandbox1_redhat.test.crio_redhat-test-crio_1 io.kubernetes.cri-o.Labels:{"group":"test","io.kubernetes.container.name":"POD"} io.kubernetes.cri-o.ResolvPath:/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata/resolv.conf io.kubernetes.cri-o.TrustedSandbox:true io.kubernetes.container.name:POD io.kubernetes.cri-o.Annotations:{"owner":"hmeng","security.alpha.kubernetes.io/seccomp/pod":"unconfined","security.alpha.kubernetes.io/sysctls":"kernel.shm_rmid_forced=1,net.ipv4.ip_local_port_range=1024 65000","security.alpha.kubernetes.io/unsafe-sysctls":"kernel.msgmax=8192"} io.kubernetes.cri-o.ContainerID:670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112 io.kubernetes.cri-o.KubeName:podsandbox1]} Created:2017-09-22 05:47:16.149210201 +0000 UTC Started:2017-09-22 01:47:18.134306329 -0400 EDT Finished:0001-01-01 00:00:00 +0000 UTC ExitCode:0 OOMKilled:false Error:}"
# time="2017-09-22 01:47:18.233050322-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/mrunalp/oom:latest""
# time="2017-09-22 01:47:18.233519328-04:00" level=debug msg="parsed reference into "[overlay2@/tmp/tmp.KihhEGanJw/crio+/tmp/tmp.KihhEGanJw/crio-run:overlay2.override_kernel_check=1]docker.io/mrunalp/oom:latest""
# time="2017-09-22 01:47:18.233753618-04:00" level=debug msg="exporting opaque data as blob "sha256:259cb9ee8ccba33f36ea25dab0224b602790b3e982788b55fd95bd47b5202684""
# time="2017-09-22 01:47:18.244141853-04:00" level=debug msg="created container "11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1""
# time="2017-09-22 01:47:18.248141075-04:00" level=debug msg="container "11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1" has work directory "/tmp/tmp.KihhEGanJw/crio/overlay2-containers/11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1/userdata""
# time="2017-09-22 01:47:18.248245418-04:00" level=debug msg="container "11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1" has run directory "/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1/userdata""
# time="2017-09-22 01:47:18.360611907-04:00" level=debug msg="mounted container "11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1" at "/tmp/tmp.KihhEGanJw/crio/overlay2/12e70e9c607997c30047be47c41d5c4168cf07d628e595df40491c378a620a7a/merged""
# time="2017-09-22 01:47:18.360650599-04:00" level=debug msg="OCI process args [/oom]"
# time="2017-09-22 01:47:18.360680297-04:00" level=debug msg="CONTAINER USER: "
# time="2017-09-22 01:47:18.360874856-04:00" level=debug msg="UID: 0, GID: 0, Groups: [0 1 2 3 4 6 10 11 20 26 27]"
# time="2017-09-22 01:47:18.361740957-04:00" level=debug msg="running conmon: /data/src/github.com/kubernetes-incubator/cri-o/conmon/conmon" args=[-c 11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1 -u 11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1 -r /bin/runc -b /tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1/userdata -p /tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1/userdata/pidfile -l /var/log/crio/pods/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1.log --exit-dir /var/run/crio/exits]
# time="2017-09-22 01:47:18.413027388-04:00" level=debug msg="Received container pid: 11758"
# time="2017-09-22 01:47:20.394553778-04:00" level=debug msg="CreateContainerResponse: &CreateContainerResponse{ContainerId:11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1,}"
# 11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1
# time="2017-09-22 01:47:20.461385839-04:00" level=debug msg="StartContainerRequest &StartContainerRequest{ContainerId:11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1,}"
# time="2017-09-22 01:47:20.470151442-04:00" level=debug msg="StartContainerResponse &StartContainerResponse{}"
# 11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1
# time="2017-09-22 01:49:00.562185047-04:00" level=debug msg="ContainerStatusRequest &ContainerStatusRequest{ContainerId:11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1,}"
# time="2017-09-22 01:49:00.562283891-04:00" level=debug msg="ContainerStatusResponse: &ContainerStatusResponse{Status:&ContainerStatus{Id:11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1,Metadata:&ContainerMetadata{Name:container1,Attempt:1,},State:CONTAINER_RUNNING,CreatedAt:1506059238410070809,StartedAt:1506059240470125514,FinishedAt:0,ExitCode:0,Image:&ImageSpec{Image:mrunalp/oom:latest,},ImageRef:64b399673fc200859d677177df69e26453997a697306bc6b46e3a3fcfdd8350a,Reason:,Message:,Labels:map[string]string{batch: no,type: small,},Annotations:map[string]string{daemon: crio,owner: dragon,},Mounts:[],LogPath:,},}"
# ID: 11cf94a97050cd8815963a1b15d65f9e64d8c8f408bbd99938e786dae6850ed1
# Name: container1
# Attempt: 1
# Status: CONTAINER_RUNNING
# Created: 2017-09-22 01:47:18.410070809 -0400 EDT
# Started: 2017-09-22 01:47:20.470125514 -0400 EDT
# Finished: 1969-12-31 19:00:00 -0500 EST
# Exit Code: 0
# Reason:
# Image: mrunalp/oom:latest
# rm: cannot remove ‘/tmp/tmp.KihhEGanJw/crio-run/overlay2-containers/670ba5accf66c23e79d14d57e432d016884d61ff5d76a463fbb08fd158e7c112/userdata/shm’: Device or resource busy
# rm: cannot remove ‘/tmp/tmp.KihhEGanJw/crio/overlay2/f026dc9dfd179fee2b17ef9f82862ab172eac082a2483ac2c68bf5f7e5cec512/merged’: Device or resource busy
# rm: cannot remove ‘/tmp/tmp.KihhEGanJw/crio/overlay2/12e70e9c607997c30047be47c41d5c4168cf07d628e595df40491c378a620a7a/merged’: Device or resource busy

forward I think we need to do three things

@mrunalp Those smell like features to me.  IMHO, let's get past the "working on all platforms" and "passes consistently" finish-lines.  Those are a pre-requesite to supporting rapid code-changes by > 3 engineers 😀
WRT baking in setup:  I think this is premature optimization.  It also brings a low-visibility / high-consequence caveat:  It allows "silent" divergence between the testing operations (e.g. playbook) and reality*.  My preference would be to leave the images as vanilla as possible and rely on having the setup as part of testing.  If/when that setup occupies a significant % of the execution time vs. executing tests.  That would be a good time to consider how it could be made faster.
WRT 'break this up into two jobs (per OS)': I respectfully (firmly) disagree 😀 (with love).  This is going backwards-  It's easier on Steve[**] and the automation-side if they have a single entry-point and don't need to worry about anything else.  Ansible is more than happy to manage sequencing, parallel tasks, result-analysis, feedback, etc.  Plus, everybody here has some basic understanding of Ansible, however very few understand the automation (Jenkins) details.   That means we can make changes to operations in parallel instead of relying on steve (one person) to do it.
[*] Reality - The end-user systems actually running cri-o will most certainly NOT be starting from the pre-setup image.  Therefor, in some respects test-environment != use-environment.
[**] Steve hasn't seen the 'CI Priorities' google doc. 😉 Nor is he on our scrum-team.  It's best retain our ability to maintain and enhance operations via the single entry-point, than having to ask Steve every time (he's only one person).

Those smell like features to me.

Yes

My preference would be to leave the images as vanilla as possible and rely on having the setup as part of testing

There is less than zero reason to waste (expensive) developer time waiting to install wget on RHEL every single time they run a test. CI exists to give feedback to developers quickly and succinctly. it is not doing it's job if we do this.

they have a single entry-point and don't need to worry about anything else.

Same argument. Literally no reason to waste dev time for this. Extremely simple to have multiple jobs. Look at k8s or Origin -- >10 jobs, all separate responsibilities.

@mrunalp @stevekuznetsov we should probably discuss this further via a call instead of in a PR.  There are some related red-hat internal items I can't mention in a public comment.

FWIW setup today:
real	32m29.203s
user	46m5.288s
sys	5m22.200s

@stevekuznetsov That test was racy but we added a 100 seconds sleep for the container to OOM. Surprising that it went past 100 seconds as well.

/test rhel

Left TODO:

ensure we fail if tests fail
ensure we have jUnit output by default for both tests

lookering...

@stevekuznetsov LGTM 😀

ensure we have jUnit output by default for both tests

IMHO, that can come via another PR, maybe like #939 (dunno yet).  I don't think this PR should block on it, but others may feel differently.  However the conversion-to-jUnit is done, the tasks in #939 will to the "collection" business (I tested it) safely.

Integration looks like it's passing OK

/test e2e_rhel

customcluster.service failing to start ... from apiserver logs:
I0922 17:11:52.022147   11483 insecure_handler.go:118] Serving insecurely on <IP_ADDRESS>:8080
Error: failed to listen on <IP_ADDRESS>:8080: listen tcp <IP_ADDRESS>:8080: bind: cannot assign requested address
Error: failed to listen on <IP_ADDRESS>:8080: listen tcp <IP_ADDRESS>:8080: bind: cannot assign requested address

... interesting. that's new

/test e2e_rhel

/test e2e_rhel

@stevekuznetsov Found the issue. The IP that we are populating in createcluster.sh and /etc/hosts is the external IP address that doesn't correspond to any interface inside the VM, so the bind fails. I changed it to the eth0 IP Address and was able to start the cluster. I have also kicked off the tests on the VM.

/test fedora

/test fedora

@kargakis -- we had a ci/openshift-jenkins/integration_rhel context that was green on this PR but it seems to be missing now. Is there some test-infra component that prunes these?

/test all

@kargakis -- we had a ci/openshift-jenkins/integration_rhel context that was green on this PR but it seems to be missing now. Is there some test-infra component that prunes these?

None that I know of.

e2e jobs filed waiting for cluster to come up, but they were stlll building binaries .. bumping the timeout

/test e2e

@stevekuznetsov I'm working on getting jUnit analysis of the results in #939
It's written as a playbook soas to be independent of the main playbook exit status.  My thinking is that would help identify the cause, if/when the job itself fails.  Do you think that's complicating things and I should make it into an include-file instead?
Also, I'm putting all the interesting artifacts under a results sub-directory of $WORKSPACE for easy consumption.  Shall I dump results.xml there as well?
Let me know when you have a chance.

It's written as a playbook soas to be independent of the main playbook exit status. My thinking is that would help identify the cause, if/when the job itself fails. Do you think that's complicating things and I should make it into an include-file instead?

Seems reasonable -- we can run it as a post-job step in Jenkins.

Also, I'm putting all the interesting artifacts under a results sub-directory of $WORKSPACE for easy consumption. Shall I dump results.xml there as well?

Jenkins doesn't much care, but if we want to get Gubernator working for these the jUnit files need to be named junit_*.xml

junit_*.xml

Sure thing, no problem.  Though that opens another question:  Currently I'm writing all the suites ('integration' and 'e2e') into a single file.  I know Jenkins doesn't care, but what about Gubernator*?
*I have no clue what that is/for.

Any one job only runs one suite of tests. Gubernator doesn't care about the content of the jUnit files, just needs all of the files to have the naming scheme.

@mrunalp changed the Ansible play to use ansible_eth0.ipv4.address but still seeing in kube-apiserver.log:
I0925 10:35:46.193153   11473 insecure_handler.go:118] Serving insecurely on <IP_ADDRESS>:8080
Error: failed to listen on <IP_ADDRESS>:8080: listen tcp <IP_ADDRESS>:8080: bind: cannot assign requested address
Error: failed to listen on <IP_ADDRESS>:8080: listen tcp <IP_ADDRESS>:8080: bind: cannot assign requested address

Also we're still doing 20+ minutes of building kubectl and hyperkube in the local-up-cluster.sh run ... will look into doing that in the AMI bake

/test e2e

/test e2e_rhel

/test e2e

@mrunalp lol, sorry -- the customcluster.service was written before the AMI was created, so the eth0 that was written was for a VM that was dead ... since we started a new one from the AMI to run the cluster on. whoopsie

/test e2e

/test e2e

jUnit --report-dir="artifacts/" flag seemed to be wrong... /cc @cevich
Removing it, everything else looks good.
/test all

Integration tests passed for fedora and rhel :tada:

RHEL e2e only 1 known failure with guestbook.

Fedora e2e also same guestbook issue.

LGTM

jUnit --report-dir="artifacts/" flag seemed to be wrong... /cc @cevich

Ya, we need #939 to both collect the results from the VMs and possibly sanitize the jUnit.  Until then, we'll just have to manually look at the output to know.