customFieldsManager  casued None type has no attribute 'field'

When setting fetch_custom_attributes to False, it should not initialize allCustomAttributesNames in helpers.py which will lead to a None type has no attribute 'field' from line "for f in content.customFieldsManager.field", after I commented that code, everything work fine.

@rmontenegroo Could you take a peek at this?

@knight-ni would paste the complete error message here? I can't make it happen here.
What is your vsphere version?

I have the same problem on a vSphere 6.5 host, I tried it with VSPHERE_PROD_FETCH_CUSTOM_ATTRIBUTES=True and without, no luck.
This is the log output using the latest docker image:

2020-05-26 12:48:44,303 INFO:Start collecting metrics from <IP_ADDRESS>
2020-05-26 12:48:44,303 INFO:Starting vm metrics collection
2020-05-26 12:48:44,303 INFO:Fetching vim.VirtualMachine inventory
2020-05-26 12:48:44,303 INFO:Retrieving service instance content
2020-05-26 12:48:44,304 INFO:START: _vmware_get_vm_perf_manager_metrics
2020-05-26 12:48:44,304 INFO:Fetching vim.Datastore inventory
2020-05-26 12:48:44,304 INFO:Starting host metrics collection
2020-05-26 12:48:44,304 INFO:Fetching vim.HostSystem inventory
2020-05-26 12:48:44,304 INFO:START: _vmware_get_host_perf_manager_metrics
2020-05-26 12:48:44,432 INFO:Retrieved service instance content
2020-05-26 12:48:44,563 ERROR:Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 1638, in _async_render_GET
    yield self.generate_latest_metrics(request)
twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'twisted.internet.defer.FirstError'>: FirstError[#1, [Failure instance: Traceback: <class 'twisted.internet.defer.FirstError'>: FirstError[#0, [Failure instance: Traceback: <class 'AttributeError'>: 'NoneType' object has no attribute 'field'
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:501:errback
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:568:_startRunCallbacks
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:654:_runCallbacks
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:1475:gotResult
--- <exception caught here> ---
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/local/lib/python3.6/site-packages/twisted/python/failure.py:512:throwExceptionIntoGenerator
/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py:623:vm_inventory
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/local/lib/python3.6/site-packages/twisted/python/failure.py:512:throwExceptionIntoGenerator
/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py:476:batch_fetch_properties
/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py:250:inContext
/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py:266:<lambda>
/usr/local/lib/python3.6/site-packages/twisted/python/context.py:122:callWithContext
/usr/local/lib/python3.6/site-packages/twisted/python/context.py:85:callWithContext
/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py:26:batch_fetch_properties
]]
--- <exception caught here> ---
/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py:930:vm_labels
/usr/local/lib/python3.6/site-packages/vmware_exporter/defer.py:99:parallelize
]]
--- <exception caught here> ---
/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py:1320:_vmware_get_vms
/usr/local/lib/python3.6/site-packages/vmware_exporter/defer.py:99:parallelize
]]

Unhandled error in Deferred:

Traceback (most recent call last):
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 930, in vm_labels
    virtual_machines, host_labels = yield parallelize(self.vm_inventory, self.host_labels)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/defer.py", line 99, in parallelize
    results = yield defer.DeferredList(args, fireOnOneErrback=True)
twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'AttributeError'>: 'NoneType' object has no attribute 'field'
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:501:errback
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:568:_startRunCallbacks
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:654:_runCallbacks
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:1475:gotResult
--- <exception caught here> ---
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/local/lib/python3.6/site-packages/twisted/python/failure.py:512:throwExceptionIntoGenerator
/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py:623:vm_inventory
/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/usr/local/lib/python3.6/site-packages/twisted/python/failure.py:512:throwExceptionIntoGenerator
/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py:476:batch_fetch_properties
/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py:250:inContext
/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py:266:<lambda>
/usr/local/lib/python3.6/site-packages/twisted/python/context.py:122:callWithContext
/usr/local/lib/python3.6/site-packages/twisted/python/context.py:85:callWithContext
/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py:26:batch_fetch_properties
]]

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 623, in vm_inventory
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 476, in batch_fetch_properties
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py", line 26, in batch_fetch_properties
    for f in content.customFieldsManager.field
builtins.AttributeError: 'NoneType' object has no attribute 'field'

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 623, in vm_inventory
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 476, in batch_fetch_properties
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py", line 26, in batch_fetch_properties
    for f in content.customFieldsManager.field
builtins.AttributeError: 'NoneType' object has no attribute 'field'

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 623, in vm_inventory
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)

    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py", line 26, in batch_fetch_properties
    for f in content.customFieldsManager.field
builtins.AttributeError: 'NoneType' object has no attribute 'field'

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 561, in host_system_inventory
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 476, in batch_fetch_properties
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py", line 26, in batch_fetch_properties
    for f in content.customFieldsManager.field
builtins.AttributeError: 'NoneType' object has no attribute 'field'

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 561, in host_system_inventory
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 476, in batch_fetch_properties
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py", line 26, in batch_fetch_properties
    for f in content.customFieldsManager.field
builtins.AttributeError: 'NoneType' object has no attribute 'field'

Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 506, in datastore_inventory
    properties
  File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/vmware_exporter.py", line 476, in batch_fetch_properties
    properties,
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/usr/local/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/usr/local/lib/python3.6/site-packages/vmware_exporter/helpers.py", line 26, in batch_fetch_properties
    for f in content.customFieldsManager.field
builtins.AttributeError: 'NoneType' object has no attribute 'field'


Test flake in 870_weave_recovers_unreachable_ips

Failing fairly often, e.g. https://circleci.com/gh/weaveworks/weave/12212.
Generally succeeds on re-run, e.g. 12213 is the re-run of 12212.
>>> Test /home/circleci/src/github.com/weaveworks/weave/test/870_weave_recovers_unreachable_ips_on_relaunch_3_test.sh finished after 193.8 secs with error: exit status 1
Test weave-net deallocates from IPAM on node failure
Tearing down kubernetes cluster
test-12212-1-0.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-0.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-0.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-0.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-0.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-0.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-0.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-1.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-1.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-1.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-1.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-1.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-1.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-2.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-2.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-2.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-2.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-2.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-2.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
Setting up kubernetes cluster
I0814 14:11:35.707956    3994 feature_gate.go:230] feature gates: &{map[]}
test-12212-1-0.us-central1-a.weave-net> [init] using Kubernetes version: v1.11.0
test-12212-1-0.us-central1-a.weave-net> [preflight] running pre-flight checks
I0814 14:11:35.733298    3994 kernel_validator.go:81] Validating kernel version
I0814 14:11:35.733412    3994 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
test-12212-1-0.us-central1-a.weave-net> [preflight/images] Pulling images required for setting up a Kubernetes cluster
test-12212-1-0.us-central1-a.weave-net> [preflight/images] This might take a minute or two, depending on the speed of your internet connection
test-12212-1-0.us-central1-a.weave-net> [preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
test-12212-1-0.us-central1-a.weave-net> [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
test-12212-1-0.us-central1-a.weave-net> [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
test-12212-1-0.us-central1-a.weave-net> [preflight] Activating the kubelet service
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated ca certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated apiserver certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] apiserver serving cert is signed for DNS names [test-12212-1-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [<IP_ADDRESS> <IP_ADDRESS>]
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated apiserver-kubelet-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated sa key and public key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated front-proxy-ca certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated front-proxy-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/ca certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/server certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] etcd/server serving cert is signed for DNS names [test-12212-1-0 localhost] and IPs [<IP_ADDRESS> ::1]
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/peer certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] etcd/peer serving cert is signed for DNS names [test-12212-1-0 localhost] and IPs [<IP_ADDRESS> <IP_ADDRESS> ::1]
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated etcd/healthcheck-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] Generated apiserver-etcd-client certificate and key.
test-12212-1-0.us-central1-a.weave-net> [certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
test-12212-1-0.us-central1-a.weave-net> [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
test-12212-1-0.us-central1-a.weave-net> [controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
test-12212-1-0.us-central1-a.weave-net> [controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
test-12212-1-0.us-central1-a.weave-net> [controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
test-12212-1-0.us-central1-a.weave-net> [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
test-12212-1-0.us-central1-a.weave-net> [init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
test-12212-1-0.us-central1-a.weave-net> [init] this might take a minute or longer if the control plane images have to be pulled
test-12212-1-0.us-central1-a.weave-net> [apiclient] All control plane components are healthy after 45.019342 seconds
test-12212-1-0.us-central1-a.weave-net> [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
test-12212-1-0.us-central1-a.weave-net> [kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
test-12212-1-0.us-central1-a.weave-net> [markmaster] Marking the node test-12212-1-0 as master by adding the label "node-role.kubernetes.io/master=''"
test-12212-1-0.us-central1-a.weave-net> [markmaster] Marking the node test-12212-1-0 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
test-12212-1-0.us-central1-a.weave-net> [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test-12212-1-0" as an annotation
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] using token:<PHONE_NUMBER>667788990000
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
test-12212-1-0.us-central1-a.weave-net> [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
test-12212-1-0.us-central1-a.weave-net> [addons] Applied essential addon: CoreDNS
test-12212-1-0.us-central1-a.weave-net> [addons] Applied essential addon: kube-proxy
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-0.us-central1-a.weave-net> Your Kubernetes master has initialized successfully!
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-0.us-central1-a.weave-net> To start using your cluster, you need to run the following as a regular user:
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-0.us-central1-a.weave-net> mkdir -p $HOME/.kube
test-12212-1-0.us-central1-a.weave-net> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
test-12212-1-0.us-central1-a.weave-net> sudo chown $(id -u):$(id -g) $HOME/.kube/config
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-0.us-central1-a.weave-net> You should now deploy a pod network to the cluster.
test-12212-1-0.us-central1-a.weave-net> Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
test-12212-1-0.us-central1-a.weave-net> https://kubernetes.io/docs/concepts/cluster-administration/addons/
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-0.us-central1-a.weave-net> You can now join any number of machines by running the following on each node
test-12212-1-0.us-central1-a.weave-net> as root:
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-0.us-central1-a.weave-net> kubeadm join <IP_ADDRESS>:6443 --token<PHONE_NUMBER>667788990000 --discovery-token-ca-cert-hash sha256:333b69a994b194c5f8aad732cd16aa7bf8b9f00a4efe17cf513e5feeec24b41d
test-12212-1-0.us-central1-a.weave-net>
test-12212-1-1.us-central1-a.weave-net> [preflight] running pre-flight checks
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0814 14:12:35.440398    3844 kernel_validator.go:81] Validating kernel version
I0814 14:12:35.440513    3844 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
test-12212-1-1.us-central1-a.weave-net> [discovery] Trying to connect to API Server "<IP_ADDRESS>:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Created cluster-info discovery client, requesting info from "https://<IP_ADDRESS>:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Failed to connect to API Server "<IP_ADDRESS>:6443": token id "112233" is invalid for this cluster or it has expired. Use "kubeadm token create" on the master node to creating a new valid token
test-12212-1-1.us-central1-a.weave-net> [discovery] Trying to connect to API Server "<IP_ADDRESS>:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Created cluster-info discovery client, requesting info from "https://<IP_ADDRESS>:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "<IP_ADDRESS>:6443"
test-12212-1-1.us-central1-a.weave-net> [discovery] Successfully established connection with API Server "<IP_ADDRESS>:6443"
test-12212-1-1.us-central1-a.weave-net> [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
test-12212-1-1.us-central1-a.weave-net> [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
test-12212-1-1.us-central1-a.weave-net> [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
test-12212-1-1.us-central1-a.weave-net> [preflight] Activating the kubelet service
test-12212-1-1.us-central1-a.weave-net> [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
test-12212-1-1.us-central1-a.weave-net> [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test-12212-1-1" as an annotation
test-12212-1-1.us-central1-a.weave-net>
test-12212-1-1.us-central1-a.weave-net> This node has joined the cluster:
test-12212-1-1.us-central1-a.weave-net> * Certificate signing request was sent to master and a response
test-12212-1-1.us-central1-a.weave-net> was received.
test-12212-1-1.us-central1-a.weave-net> * The Kubelet was informed of the new secure connection details.
test-12212-1-1.us-central1-a.weave-net>
test-12212-1-1.us-central1-a.weave-net> Run 'kubectl get nodes' on the master to see this node join the cluster.
test-12212-1-2.us-central1-a.weave-net> [preflight] running pre-flight checks
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0814 14:12:44.130369    3904 kernel_validator.go:81] Validating kernel version
I0814 14:12:44.130508    3904 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
test-12212-1-2.us-central1-a.weave-net> [discovery] Trying to connect to API Server "<IP_ADDRESS>:6443"
test-12212-1-2.us-central1-a.weave-net> [discovery] Created cluster-info discovery client, requesting info from "https://<IP_ADDRESS>:6443"
test-12212-1-2.us-central1-a.weave-net> [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "<IP_ADDRESS>:6443"
test-12212-1-2.us-central1-a.weave-net> [discovery] Successfully established connection with API Server "<IP_ADDRESS>:6443"
test-12212-1-2.us-central1-a.weave-net> [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
test-12212-1-2.us-central1-a.weave-net> [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
test-12212-1-2.us-central1-a.weave-net> [kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
test-12212-1-2.us-central1-a.weave-net> [preflight] Activating the kubelet service
test-12212-1-2.us-central1-a.weave-net> [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
test-12212-1-2.us-central1-a.weave-net> [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test-12212-1-2" as an annotation
test-12212-1-2.us-central1-a.weave-net>
test-12212-1-2.us-central1-a.weave-net> This node has joined the cluster:
test-12212-1-2.us-central1-a.weave-net> * Certificate signing request was sent to master and a response
test-12212-1-2.us-central1-a.weave-net> was received.
test-12212-1-2.us-central1-a.weave-net> * The Kubelet was informed of the new secure connection details.
test-12212-1-2.us-central1-a.weave-net>
test-12212-1-2.us-central1-a.weave-net> Run 'kubectl get nodes' on the master to see this node join the cluster.
test-12212-1-0.us-central1-a.weave-net> serviceaccount/weave-net created
test-12212-1-0.us-central1-a.weave-net> clusterrole.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> clusterrolebinding.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> role.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> rolebinding.rbac.authorization.k8s.io/weave-net created
test-12212-1-0.us-central1-a.weave-net> daemonset.extensions/weave-net created
Checking unreachable IPs
Dropping node test-12212-1-1.us-central1-a.weave-net with 'sudo kubectl delete node'
test-12212-1-0.us-central1-a.weave-net> node "test-12212-1-1" deleted
Checking unreachable IPs
Tearing down kubernetes cluster
test-12212-1-0.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-0.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-0.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-0.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-0.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-0.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-0.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-1.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-1.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-1.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-1.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-1.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-1.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-1.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test-12212-1-2.us-central1-a.weave-net> [preflight] running pre-flight checks
test-12212-1-2.us-central1-a.weave-net> [reset] stopping the kubelet service
test-12212-1-2.us-central1-a.weave-net> [reset] unmounting mounted directories in "/var/lib/kubelet"
test-12212-1-2.us-central1-a.weave-net> [reset] removing kubernetes-managed containers
test-12212-1-2.us-central1-a.weave-net> [reset] cleaning up running containers using crictl with socket /var/run/dockershim.sock
test-12212-1-2.us-central1-a.weave-net> [reset] failed to list running pods using crictl: exit status 1. Trying to use docker instead[reset] no etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
test-12212-1-2.us-central1-a.weave-net> [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
test #5 "unreachable_ip_addresses_count test-12212-1-0.us-central1-a.weave-net" failed:
	expected "0"
	got "1"
test #6 "unreachable_ip_addresses_count test-12212-1-2.us-central1-a.weave-net" failed:
	expected "0"
	got "1"
2 of 9 tests failed in 191.354s.

I got some Weave Net pod logs for a fail: the interesting thing is that HOST2 seems to reconnect after the node delete notification.
HOST1:
INFO: 2019/10/11 14:48:11.965897 Command line options: map[http-addr:<IP_ADDRESS>:6784 metrics-addr:<IP_ADDRESS>:6782 expect-npc:true ipalloc-init:consensus=2 port:6783 conn-limit:100 datapath:datapath ipalloc-range:<IP_ADDRESS>/12 nickname:test-12587-0-0 no-dns:true db-prefix:/weavedb/weave-net docker-api: host-root:/host name:d6:c0:35:6b:e7:4b]
INFO: 2019/10/11 14:48:11.966251 weave  unreleased
INFO: 2019/10/11 14:48:12.063254 Bridge type is bridged_fastdp
INFO: 2019/10/11 14:48:12.063272 Communication between peers is unencrypted.
INFO: 2019/10/11 14:48:12.067329 Our name is d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:12.067358 Launch detected - using supplied peer list: [<IP_ADDRESS> <IP_ADDRESS>]
INFO: 2019/10/11 14:48:12.143382 Unable to fetch ConfigMap kube-system/weave-net to infer unique cluster ID
INFO: 2019/10/11 14:48:12.143422 Checking for pre-existing addresses on weave bridge
INFO: 2019/10/11 14:48:12.176741 [allocator d6:c0:35:6b:e7:4b] No valid persisted data
INFO: 2019/10/11 14:48:12.187334 [allocator d6:c0:35:6b:e7:4b] Initialising via deferred consensus
INFO: 2019/10/11 14:48:12.187387 Sniffing traffic on datapath (via ODP)
INFO: 2019/10/11 14:48:12.189301 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:12.192990 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:12.200517 ->[<IP_ADDRESS>:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.200604 overlay_switch ->[ba:2e:11:69:a5:e2(test-12587-0-2)] using fastdp
INFO: 2019/10/11 14:48:12.200632 ->[<IP_ADDRESS>:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.214164 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.214256 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:12.214295 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.218161 Listening for HTTP control messages on <IP_ADDRESS>:6784
INFO: 2019/10/11 14:48:12.226111 Listening for metrics requests on <IP_ADDRESS>:6782
INFO: 2019/10/11 14:48:12.230320 ->[<IP_ADDRESS>:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: connection fully established
INFO: 2019/10/11 14:48:12.230733 sleeve ->[<IP_ADDRESS>:6783|ba:2e:11:69:a5:e2(test-12587-0-2)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.239238 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using sleeve
INFO: 2019/10/11 14:48:12.239263 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:12.239372 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:12.240481 sleeve ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.342832 Discovered remote MAC ba:2e:11:69:a5:e2 at ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:12.386941 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.440924 Discovered remote MAC 0a:24:dd:13:04:03 at ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:12.642827 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.706906 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.854272 Discovered remote MAC 2e:11:03:d5:22:91 at ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:13.090702 [kube-peers] Added myself to peer list &{[{ba:2e:11:69:a5:e2 test-12587-0-2} {52:9b:77:28:8d:4f test-12587-0-1} {d6:c0:35:6b:e7:4b test-12587-0-0}]}
DEBU: 2019/10/11 14:48:13.132990 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:13.133137 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:13.139167 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:13.139420 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:13.147498 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:13.147703 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
<IP_ADDRESS>
<IP_ADDRESS>
<IP_ADDRESS>
DEBU: 2019/10/11 14:48:13.265264 registering for updates for node delete events
INFO: 2019/10/11 14:48:17.214124 Discovered remote MAC 8e:83:29:e2:60:0e at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:17.312144 Discovered remote MAC 8a:a3:fd:36:eb:b4 at 52:9b:77:28:8d:4f(test-12587-0-1)
DEBU: 2019/10/11 14:48:19.560619 [kube-peers] Nodes that have disappeared: map[52:9b:77:28:8d:4f:{52:9b:77:28:8d:4f test-12587-0-1} d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
DEBU: 2019/10/11 14:48:19.560644 [kube-peers] Preparing to remove disappeared peer 52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.563497 [kube-peers] Noting I plan to remove  52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.569059 weave DELETE to http://<IP_ADDRESS>:6784/peer/52:9b:77:28:8d:4f with map[]
INFO: 2019/10/11 14:48:19.574313 [kube-peers] rmpeer of 52:9b:77:28:8d:4f: 262144 IPs taken over from 52:9b:77:28:8d:4f

INFO: 2019/10/11 14:48:19.583977 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 <IP_ADDRESS>:56677-><IP_ADDRESS>:6783: read: connection reset by peer
INFO: 2019/10/11 14:48:19.584188 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:19.586630 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
WARN: 2019/10/11 14:48:19.592494 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:19.599114 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:19.599201 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:19.952364 [kube-peers] Nodes that have disappeared: map[d6:c0:35:6b:e7:4b:{d6:c0:35:6b:e7:4b test-12587-0-0}]
WARN: 2019/10/11 14:48:19.952615 [kube-peers] not removing myself {d6:c0:35:6b:e7:4b test-12587-0-0}
DEBU: 2019/10/11 14:48:20.153754 weave POST to http://<IP_ADDRESS>:6784/connect with map[replace:[true] peer:[<IP_ADDRESS>]]
INFO: 2019/10/11 14:48:21.237874 ->[<IP_ADDRESS>:39755] connection accepted
INFO: 2019/10/11 14:48:21.238789 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:21.273076 ->[<IP_ADDRESS>:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:21.273351 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.273411 ->[<IP_ADDRESS>:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added
INFO: 2019/10/11 14:48:21.274418 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:21.274606 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.274660 ->[<IP_ADDRESS>:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:21.274745 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added
INFO: 2019/10/11 14:48:21.275029 ->[<IP_ADDRESS>:39755|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: Multiple connections to 52:9b:77:28:8d:4f(test-12587-0-1) added to d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:21.276800 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:21.280127 sleeve ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:21.338833 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.384071 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.954871 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:23.875303 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 <IP_ADDRESS>:38063-><IP_ADDRESS>:6783: read: connection reset by peer
INFO: 2019/10/11 14:48:23.875720 ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:23.876413 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:24.393358 Discovered remote MAC 4e:de:38:ef:c3:d7 at ba:2e:11:69:a5:e2(test-12587-0-2)

HOST3:
INFO: 2019/10/11 14:48:11.774943 Command line options: map[db-prefix:/weavedb/weave-net docker-api: ipalloc-range:<IP_ADDRESS>/12 metrics-addr:<IP_ADDRESS>:6782 no-dns:true conn-limit:100 datapath:datapath ipalloc-init:consensus=2 host-root:/host name:ba:2e:11:69:a5:e2 port:6783 expect-npc:true http-addr:<IP_ADDRESS>:6784 nickname:test-12587-0-2]
INFO: 2019/10/11 14:48:11.775139 weave  unreleased
INFO: 2019/10/11 14:48:11.849084 Bridge type is bridged_fastdp
INFO: 2019/10/11 14:48:11.849102 Communication between peers is unencrypted.
INFO: 2019/10/11 14:48:11.854314 Our name is ba:2e:11:69:a5:e2(test-12587-0-2)
INFO: 2019/10/11 14:48:11.854346 Launch detected - using supplied peer list: [<IP_ADDRESS> <IP_ADDRESS>]
INFO: 2019/10/11 14:48:11.987378 Unable to fetch ConfigMap kube-system/weave-net to infer unique cluster ID
INFO: 2019/10/11 14:48:11.987481 Checking for pre-existing addresses on weave bridge
INFO: 2019/10/11 14:48:12.014364 [allocator ba:2e:11:69:a5:e2] No valid persisted data
INFO: 2019/10/11 14:48:12.019424 [allocator ba:2e:11:69:a5:e2] Initialising via deferred consensus
INFO: 2019/10/11 14:48:12.019637 Sniffing traffic on datapath (via ODP)
INFO: 2019/10/11 14:48:12.020265 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:12.020486 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:12.021385 ->[<IP_ADDRESS>:6783] error during connection attempt: dial tcp4 :0-><IP_ADDRESS>:6783: connect: connection refused
INFO: 2019/10/11 14:48:12.021873 ->[<IP_ADDRESS>:6783] error during connection attempt: dial tcp4 :0-><IP_ADDRESS>:6783: connect: connection refused
INFO: 2019/10/11 14:48:12.023099 Listening for HTTP control messages on <IP_ADDRESS>:6784
INFO: 2019/10/11 14:48:12.023393 Listening for metrics requests on <IP_ADDRESS>:6782
INFO: 2019/10/11 14:48:12.135510 ->[<IP_ADDRESS>:42053] connection accepted
INFO: 2019/10/11 14:48:12.140825 ->[<IP_ADDRESS>:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.141110 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:12.141177 ->[<IP_ADDRESS>:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.151662 ->[<IP_ADDRESS>:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:12.175300 sleeve ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.193481 ->[<IP_ADDRESS>:54651] connection accepted
INFO: 2019/10/11 14:48:12.200750 ->[<IP_ADDRESS>:54651|d6:c0:35:6b:e7:4b(test-12587-0-0)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:12.201065 overlay_switch ->[d6:c0:35:6b:e7:4b(test-12587-0-0)] using fastdp
INFO: 2019/10/11 14:48:12.201125 ->[<IP_ADDRESS>:54651|d6:c0:35:6b:e7:4b(test-12587-0-0)]: connection added (new peer)
INFO: 2019/10/11 14:48:12.204539 ->[<IP_ADDRESS>:54651|d6:c0:35:6b:e7:4b(test-12587-0-0)]: connection fully established
INFO: 2019/10/11 14:48:12.240779 sleeve ->[<IP_ADDRESS>:6783|d6:c0:35:6b:e7:4b(test-12587-0-0)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:12.386919 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.561471 Discovered remote MAC 52:20:0f:40:c4:8a at d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:12.625454 Discovered remote MAC 86:38:7c:a6:0b:4e at d6:c0:35:6b:e7:4b(test-12587-0-0)
INFO: 2019/10/11 14:48:12.642866 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.708530 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:12.836941 [kube-peers] Added myself to peer list &{[{ba:2e:11:69:a5:e2 test-12587-0-2}]}
DEBU: 2019/10/11 14:48:12.845751 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:12.845886 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:12.851978 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:12.852094 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:12.857592 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:12.857707 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
<IP_ADDRESS>
INFO: 2019/10/11 14:48:13.010188 Discovered remote MAC d6:c0:35:6b:e7:4b at d6:c0:35:6b:e7:4b(test-12587-0-0)
DEBU: 2019/10/11 14:48:13.112876 registering for updates for node delete events
<IP_ADDRESS>
<IP_ADDRESS>
INFO: 2019/10/11 14:48:17.214010 Discovered remote MAC 8e:83:29:e2:60:0e at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:17.312161 Discovered remote MAC 8a:a3:fd:36:eb:b4 at 52:9b:77:28:8d:4f(test-12587-0-1)
DEBU: 2019/10/11 14:48:19.573833 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2} 52:9b:77:28:8d:4f:{52:9b:77:28:8d:4f test-12587-0-1}]
WARN: 2019/10/11 14:48:19.574563 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:19.574651 [kube-peers] Preparing to remove disappeared peer 52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.578360 [kube-peers] Existing annotation d6:c0:35:6b:e7:4b
INFO: 2019/10/11 14:48:19.584272 ->[<IP_ADDRESS>:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 <IP_ADDRESS>:6783-><IP_ADDRESS>:42053: read: connection reset by peer
INFO: 2019/10/11 14:48:19.584402 ->[<IP_ADDRESS>:42053|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:19.585810 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:19.586118 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:19.587108 ->[<IP_ADDRESS>:6783] error during connection attempt: dial tcp4 :0-><IP_ADDRESS>:6783: connect: connection refused
DEBU: 2019/10/11 14:48:19.595230 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2} 52:9b:77:28:8d:4f:{52:9b:77:28:8d:4f test-12587-0-1}]
WARN: 2019/10/11 14:48:19.595349 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:19.595440 [kube-peers] Preparing to remove disappeared peer 52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.600546 [kube-peers] Noting I plan to remove  52:9b:77:28:8d:4f
DEBU: 2019/10/11 14:48:19.604745 weave DELETE to http://<IP_ADDRESS>:6784/peer/52:9b:77:28:8d:4f with map[]
INFO: 2019/10/11 14:48:19.605669 [kube-peers] rmpeer of 52:9b:77:28:8d:4f: 0 IPs taken over from 52:9b:77:28:8d:4f

DEBU: 2019/10/11 14:48:20.150639 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:20.150664 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:20.551138 [kube-peers] Nodes that have disappeared: map[ba:2e:11:69:a5:e2:{ba:2e:11:69:a5:e2 test-12587-0-2}]
WARN: 2019/10/11 14:48:20.551159 [kube-peers] not removing myself {ba:2e:11:69:a5:e2 test-12587-0-2}
DEBU: 2019/10/11 14:48:20.750484 weave POST to http://<IP_ADDRESS>:6784/connect with map[peer:[<IP_ADDRESS>] replace:[true]]
INFO: 2019/10/11 14:48:21.237178 ->[<IP_ADDRESS>:55775] connection accepted
INFO: 2019/10/11 14:48:21.237926 ->[<IP_ADDRESS>:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection ready; using protocol version 2
INFO: 2019/10/11 14:48:21.238025 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.238063 ->[<IP_ADDRESS>:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection added (new peer)
INFO: 2019/10/11 14:48:21.270916 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using sleeve
INFO: 2019/10/11 14:48:21.271204 ->[<IP_ADDRESS>:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection fully established
INFO: 2019/10/11 14:48:21.271838 sleeve ->[<IP_ADDRESS>:6783|52:9b:77:28:8d:4f(test-12587-0-1)]: Effective MTU verified at 1398
INFO: 2019/10/11 14:48:21.338816 Discovered remote MAC c6:06:f1:95:ea:59 at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.385987 Discovered remote MAC 52:9b:77:28:8d:4f at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:21.739486 overlay_switch ->[52:9b:77:28:8d:4f(test-12587-0-1)] using fastdp
INFO: 2019/10/11 14:48:21.954937 Discovered remote MAC 12:c2:af:c9:69:7c at 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:23.875162 ->[<IP_ADDRESS>:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection shutting down due to error: read tcp4 <IP_ADDRESS>:6783-><IP_ADDRESS>:55775: read: connection reset by peer
INFO: 2019/10/11 14:48:23.875228 ->[<IP_ADDRESS>:55775|52:9b:77:28:8d:4f(test-12587-0-1)]: connection deleted
INFO: 2019/10/11 14:48:23.875526 ->[<IP_ADDRESS>:6783] attempting connection
INFO: 2019/10/11 14:48:23.875802 ->[<IP_ADDRESS>:6783] error during connection attempt: dial tcp4 :0-><IP_ADDRESS>:6783: connect: connection refused
INFO: 2019/10/11 14:48:23.876399 Removed unreachable peer 52:9b:77:28:8d:4f(test-12587-0-1)
INFO: 2019/10/11 14:48:24.494536 Discovered remote MAC 3a:5b:7e:32:ee:de at d6:c0:35:6b:e7:4b(test-12587-0-0)

In the kubelet logs on host2 we can see that it evicts all pods, then starts (some of) them again.
I think shutting down Kubernetes on the node before deleting it would be better.


Prohibit Futures.addCallback(Future, Callback)

Fixes https://github.com/apache/incubator-druid/issues/8756
@leventov

Mechanically adding Execs.directExecutor() doesn't solve the problem. You should actually thoughtfully check every case and verify that Execs.directExecutor() is an acceptable option in each of them. A case should meet at least one of three criteria, as described in https://github.com/code-review-checklists/java-concurrency#cf-beware-non-async:

The callback is lightweight and non-blocking (as is the case in your PR #8700); or
The callback is added from the same executor as the future is completed; or
The callback attachment is preceded with if (future.isDone()) check.

Then, the specific criterion should be identified in a comment, similarly to what I asked you to do here: https://github.com/apache/incubator-druid/pull/8700#discussion_r341482427.
Otherwise (if the case meets neither of these criteria), directExecutor() should not be used.

A forbidden-apis entry should also be added. See an example here: https://github.com/apache/incubator-druid/commit/15fbf5983d9d6abaeda54296f97fe65f1d6bba98#diff-4cfb6baf06c34e537a8666d9a0f3aebd

thank you, @leventov
comments addressed
did a force push, put this work on top of https://github.com/apache/incubator-druid/pull/8700, so that compiles with forbidden-apis
only the 2nd commit for review

thank you, @leventov
get comments addressed

Hi  @leventov I rebased this PR to resolve code conflicts. Are you free to take a look?


RuntimeError: Error(s) in loading state_dict for DPTDepthModel

As of commit d348fc, following the instructions in the README I am able to install all submodules and required packages and run the example test script. It seems to fail while attempting to load the weights for MiDaS (https://github.com/isl-org/MiDaS).
Example output:
$ PYTHONPATH=./utils python3 run.py   --image "examples/photo-1667788000333-4e36f948de9a.jpeg"   --prompt "a street with traditional buildings in Kyoto, Japan"   --output_path "./output.ply"   --mode "stage"
img_size [384, 512]
Using cache found in ~/.cache/torch/hub/intel-isl_MiDaS_master
Traceback (most recent call last):
  File "~/invisible-stitch/run.py", line 221, in <module>
    fire.Fire(generate_scene)
  File "~/invisible-stitch/venv/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "~/invisible-stitch/venv/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "~/invisible-stitch/venv/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "~/invisible-stitch/run.py", line 181, in generate_scene
    zoe_dc_model = get_zoe_dc_model(ckpt_path=hf_hub_download(repo_id="paulengstler/invisible-stitch", filename="invisible-stitch.pt")).to(device)
  File "~/invisible-stitch/utils/models.py", line 66, in get_zoe_dc_model
    return ZoeD_N(vanilla=vanilla, ckpt_path=ckpt_path, **kwargs)
  File "~/invisible-stitch/utils/models.py", line 39, in ZoeD_N
    model = build_model(zoedepth_config)
  File "~/invisible-stitch/zoedepth/models/builder.py", line 51, in build_model
    return get_version(config.version_name).build_from_config(config)
  File "~/invisible-stitch/zoedepth/models/zoedepth/zoedepth_v1.py", line 262, in build_from_config
    return ZoeDepth.build(**config)
  File "~/invisible-stitch/zoedepth/models/zoedepth/zoedepth_v1.py", line 252, in build
    core = MidasCore.build(midas_model_type=midas_model_type, use_pretrained_midas=use_pretrained_midas,
  File "~/invisible-stitch/zoedepth/models/base_models/midas.py", line 346, in build
    midas = torch.hub.load("intel-isl/MiDaS", midas_model_type,
  File "~/invisible-stitch/venv/lib/python3.10/site-packages/torch/hub.py", line 568, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "~/invisible-stitch/venv/lib/python3.10/site-packages/torch/hub.py", line 597, in _load_local
    model = entry(*args, **kwargs)
  File "~/.cache/torch/hub/intel-isl_MiDaS_master/hubconf.py", line 51, in DPT_BEiT_L_384
    model.load_state_dict(state_dict)
  File "~/invisible-stitch/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DPTDepthModel:
	Unexpected key(s) in state_dict: "pretrained.model.blocks.0.attn.relative_position_index", "pretrained.model.blocks.1.attn.relative_position_index", ...

Ubuntu 22.04.4, GTX 1080Ti, Nvidia drivers 535.171.04, Cuda 12.1
Maybe this is a known issue or a configuration problem?

Could you double-check if you have timm==0.6.7 installed? This is related to an issue with MiDaS requiring certain version of timm (see https://github.com/isl-org/MiDaS/issues/245).

Thank you, confirmed fixed by updating to 8156c79, installing timm==0.6.7 and also installing torch==2.2.0 torchvision==0.18.0 and xformers==0.0.24 following the instructions from https://github.com/paulengstler/invisible-stitch/issues/6

Adding a +1 to the solution, and  wanted to note that it was just the timm version that fixed it for me, I didn't need to change anything else.


atd not starting

Issue description
Nov 10 17:06:51 terry atd[31578]: Cannot change to /var/spool/atjobs: Permission denied

Steps to reproduce
services.atd.enable = true;

and sudo nixos-rebuild test.
Technical details

Nix version: nix-env (Nix) 1.11.15
Nixpkgs version: "18.03pre119245.5cfd049a03"
Sandboxing enabled: build-use-sandbox = true

NixOS test seems to be working on that revision (nix-build nixos/tests/atd.nix). Not sure if that means something imperative happened to your system, or there is a race condition bug somewhere :-/

Does /var/spool exists on your system?
It looks like the same issue  services.locate has with creating /var/cache

Yes, it exists:
drwx------  3 root root 4096  1. Feb 2017  spool

Perhaps do sudo chmod 755 /var/spool and see if it the problem reappears?

I believe there was a few revisions of NixOS where services.locate created wrong permissions in some directory under /var. Perhaps if mlocate was selected. (Don't remember.)

Well, the chmod step should be done by the service itself, right?

Ideally, yes, but in a way that doesn't introduce race. Consider if multiple services have their own copy of "chmod xyz /var/spool" and one of them starts diverging from the other. Now we have a race. So I think if we agree that each service should chmod, it should be done via an utility function provided by NixOS.

The service module does create /var/spool/{atjobs,atspool} and chmod it. Just tested on a nixos demo VM with no /var/spool in the first place.
So yes I'd suspect a conflict with another service module. Do you have services.locate enabled?

Do you have services.locate enabled?

No, a quick ag locate on my configuration directory did not yield anything.


added sdets as codeowners for tests directory

LGTM :rocket:

@alexzbusko the CF build failed because of some configuration that I forgot on the preview builds last night. Sorry about this

I pushed a commit that fixes the CF build. This is g2g
@mweichert @jevonearth, please review and approve when you can


Failure when Xcode is installed but the user has not signed the Xcode EULA

So I installed XCode, and tried flutter start again. Now I get:
idevice_id not found. To work with iOS devices, please install ideviceinstaller.
If you use homebrew, you can install it with "$ brew install ideviceinstaller".
Error executing simctl: 69

Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.

Downloading darwin-x64 artifacts from the cloud, one moment please...
Downloading https://storage.googleapis.com/mojo_infra/flutter/9e1e1fba5474b458458b65f1aa1d0d730af2705b/darwin-x64/artifacts.zip.
Unable to run application - no connected devices.

[ ] We don't seem to detect that XCode failed, instead, we blindly continue on into guaranteed doom.
[ ] We don't provide the user with any help in handling this license issue.

There is no way to sign the EULA from the command line without sudo. We can handle this case better via the messaging in the tooling. For now, the Getting Started guide has been updated https://github.com/flutter/website/pull/168

Can we catch this failure case? If we can, then we have several options:

tell the user what the problem is and what command to run to fix it
tell the user what the problem is and run the command to fix it

Also, can we detect this ahead of time so we can include it in the readiness checks?

Looking at the brew source, we can detect that the eula hasn't been signed if the output of this command: /usr/bin/xcrun clang has the work 'license' in either stdout or stderr (https://github.com/Homebrew/homebrew/blob/master/Library/brew.sh#L97). We can then prompt them to agree by opened xcode or by running that cli command (sudo xcodebuild -license).

We have a check in doctor now for the eula: https://github.com/flutter/flutter/pull/2078. We still need to add a eula check in run.

We now show:
$ flutter run

Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.

and
$ sudo /Users/danrubel/bin/flutter run

You have not agreed to the Xcode license agreements, please run 'xcodebuild -license' (for user-level acceptance) or 'sudo xcodebuild -license' (for system-wide acceptance) from within a Terminal window to review and agree to the Xcode license agreements.

Following the -license instructions above displays the license agreement and prompts you to sign it as expected. I believe this issues has been addressed.

Should we show that second message instead of the first?

Both of those messages are not our own and they are both reasonably clear as to what the user should do. We could add code to figure it out and display our own message to save the user a step but I don't think the savings is worth the added complexity in flutter_tools.


Update svelte and svelte-kit creation script

Summary by CodeRabbit

Documentation

Updated the installation guide for the SanS UI library.
Changed the command for initializing a SvelteKit project to npx sv create my-app.
Removed the section for initializing a Svelte project, recommending SvelteKit instead.
Maintained emphasis on configuring Tailwind CSS prior to installing SanS UI.

umm....the PR description was not automatically generated Lol

umm....the PR description was not automatically generated Lol

I saw it now~


Add login-logout API

Added login logout API
PS :Added  password and activated field in User model
closes #12

Made respective changes as per https://github.com/SPO-Career-Portal/SCP-Backend/pull/16#pullrequestreview-681583720


v-model is slow (hangs) when used on a big select[multiple] element

Vue version
3.4.5
Link to minimal reproduction
https://play.vuejs.org/#eNqNUk1v2zAM/SuCLt2Axg6W9eKlQbehh+2wDetuVQ+ZzcRq9QWJdlIY/u+jpCX1sqLoTSQfHx+fOPCPzhV9B7ziy1B76ZAFwM6thJHaWY9sYB42bGQbbzU7I+iZMMLU1oQIVVAjNOwygt7c3r0VZllmHmKgAEE7tUagiLFlxrN+pm0D6lLwA4HgTHcKpVMZSVjrUFpD2I31hJRMGvbuYj6fE7bq16qDmI3BAzzm52oY5Dguy9yaR5Z5RFJDYSP71c3foRUbhqcVYmOsxg0mqikM+JhkZejtQegdGyLlTjbYVmwxn7v9h5hoQW5brBhpTZkxeZI4+DnHQNZt5La4D9aQ64lD8NpqJxX470l6EJzEZSMEXytld19TDn0H54d83UL98Ez+PuxjTvAfHgL4HgQ/1nDtt4C5fH3zDfb0PhbpVzpF6BeKPyFY1UWNGfapMw3JnuCS2i/pdqTZ/grXewQTDktFoRE5JrzgdE+fX1j9Se6iWKQ+8pNczLc502t34mMu/EtCM7LYFtGFqizrxlAbXaDsfWEAS+N0eUWw0ncGpYZZY/XVonhfXNBJBJymCwh69tvbHTlLJJPFY3+y2888mAZ8tOV1Y0/apqNPSv+NP5oy/gHIRUlh
Steps to reproduce

Create a select[multiple] element with a thousands of options (e.g. 25K options)
Use v-model on it
Try to select a big chunk of options (> 500)

What is expected?
Quick v-model update
What is actually happening?
A very noticeable hang
https://github.com/vuejs/core/assets/2431842/dd3a6f1a-81fe-42bd-9f73-19a1f685c276
System Info
No response
Any additional comments?
I'm not just testing performance, I'm actually using this in production, for reasons I won't get into.
I should mention that managing the state manually (e.g. using @change) is fast.

And a similar test using @change event instead of v-model
It's probably not doing exactly the same things as v-model but just to demonstrate that it updates immediately and doesn't hang
https://play.vuejs.org/#eNqNU8Fu2zAM/RXClyZAIwfLevGSIN3Qw3bYhnW3qgfPZmK1tiRIspPA8L+PkprUy4ZiN5F8j3x8kvrkVmvWtZhkydIWRmgHFl2r11yKRivjoAeDWxhga1QDVwS94pLLQknroTUWDktYedDk4XHqa9tWFk4oCUWVyx1OsEPpptBzCWcG6/K6ReLdGpMfmW8ecczlZoeOnYDftG9lp6zJ9UTBag0qcqcfuBy4XKZRNgmmwGGj69whRQDL2AM2UceKJ/HAE2ja2gldRxwhVZgC3WyrDOEECAnvbubzOWGzMM5nffCMx3hc970YhmUaqXFgGicGLRSWolvfv+yRQd+/+uWJvur1jzRTaN0xyIrQh5PQx2jfXpSuymAxn+sD7Q9QodhVLgPSGjLRkdAjuU6cpXvaih17skrSFYceZINqtKjRvHjLExIXjeBJXtdq/yXknGnx+pQvKiye/5F/sgef48l3gxZNR+6ea/EqY/nu/ise6HwuNqpsa0K/UfyBVtWt1xhhH1tZkuwRLqj9HB6qkLuf9u7gUNrTUl6oRw4BzxN6vJ/eWP1V7oItAo/8JBfjR5jRA7zwMRb+bEIzotjKOW2zNC1KSbQSa9EZJtGlUjfphmCpaaUTDc5K1WwW7D27oSdh3TjN0DazX0btyVlqMlrc84PdZmZQlmi8Lf839oI2Hn1R+mv82ZThN1djcKg=
https://github.com/vuejs/core/assets/2431842/7d8e84cd-54e3-4583-b6fd-afde68c5b8e7

To a degree, that's expected.
We need to  set the selected prop on each affected <option> when the parent sets the value during the re-render that was triggered by the change event of the select. To do that, we have to loo over all option elements, and for each element, look up the option's value in the list of currently selected values passed through v-model.
This is essentially a controlled input, where the component always controls which options are selected.
Your (totally valid) workaround only uses the @change event, which 1) doesn't cause a re-render and 2) thereby skips the work of re-applying the selected attributes to all those options through this bad loop.
We might have some room for optimization here, but to a degree v-model will always be inherently be slower than an uncontrolled select due to the additional work it does.

@LinusBorg I agree, but I think this could optimized to the extent that it becomes unnoticeable. I believe the slowness here is caused by a slow array search in the v-model implementation.
Here's an example where I've attempted to replicate the behavior of v-model, maintaining the original value type and updating the 'selected' attribute for each option.
I could optimize it even further, but this is just an example to demonstrate how fast this can be, despite looping over all options.
https://play.vuejs.org/#eNqNVE1v2kAQ/SsjX2JUWNPSXCiJklY5tFI/VHrDqHLtATax19bu2hAh//fO7thAKIp6Ynfmzcx7bxbvg/uqEk2NwTSYmVTLyoJBW1e3sZJFVWoLe9C4GkJaFlVtMYMWVros4IqKrmIVq7RUxkJZWUkHuHHo8F7r5Fk4XLiHHNXabqbw7no8HkM7hPD3EOQAbm5Bwht4Oxj0TQzmmLoZ3GWxpMzZgK9JRdmeTBj6Ngq3QImww4gmyWsUBUX8yUEWc6ulWnNgMAT/uxwcZ/uAmaO93J4SYU+P+7vSWK1qlbqhkG4StcYQG1R2APtYwUEO46nvQgjhAcImeo1W9IjvTHzpObMKN/iouZNENV36yKGN1Szi1dHS6GKxqPLEIt0AZjwC7pjfTRzwIQ6gqHMrq5xxhOzm8g2gGa1KTXgmL1XPJg56xNSneshJ/AmfL0R7sX3KmS02ienX5mQ6gS82RQq7Dh1NgP2elwUtSffMI+7AgiOe472gayab23k3eepqD4+sbWeRyzr/Tjyjq7HP3haGLnqjlrzWrczcc56Mx9XugwtsUK43dgr0vn2EN+J7BMPAGnpgK7kWj6ZU9DfzPWgN9MZkjrpbfRwQOZYTB0mel9svPmZ1jcM+nm4wfboQfzQ7F4uDHxoN6oZcP+T4pXH6Yf4Nd3Q+JIsyq3NCv5L8iabMa8eRYR9rlRHtE5xn+9l/LGhvv8zDzqIyvShH1CFbj6fV1/jpFelHuhMx8XXkJ7nIH6MR/T/OfOTEyyY0g8lurK3MNIrSTFFZhrlstFBoI1UV0R3BIl0rKwscZWVxNxHvxTU9CWNPwwJNMfqjyy05S01OhLt6b7ceaVQZamfL/409KzsdfZb6Z/zBlPYvOzfxQg==

using a Set would indeed be more performant, but it doesn't cover all use cases, unfortunately.
We currently use a looseEqual() function to compare values, because v-model selects do support objects, and also support inline-defined objects like this:
<option :value="{ size: 'small' }"></option>
<option :value="{ size: 'medium' }"></option>
<option :value="{ size: 'large' }"></option>

an implementation with Set could not support that, as these inline objects only match in shape, but not in identity.
We could, in theory, think about a modifier for some kine of "performance" mode that would then use a Set and leave it to the developer to insure to define values in a way that selected values can be identified through identity comparison.

Appreciate the input @LinusBorg. I actually forgot v-model supports objects as a value.
A different strategy could be using a Set to track the indexes of selected options, rather than their values.
I've tested and seems to work fine, it even simplifies my previous implementation.
Possibly there are more edge-cases to consider, but this could be a move in the right direction.
Not sure about the inline-defined objects though, since I'm not aware of a way to access their value.
I assume v-model can internally access it, so possibly this solution could work in this case as well.
https://play.vuejs.org/#eNqNVMtu2zAQ/JWtLpEBmzLi5qLaadIih/bQFk1vVlCo0tpiQlECSck2DP17l6TkKk4R5GRyZ/Yxw7WOwW1ds7bBIA6WOlO8NqDRNPV1InlZV8rAERRuppBVZd0YzKGDjapKuKCki0QmMqukNlDVhtMBVpYd3iqVHpjlhUcQKLemiOHyaj6fQzeF8PcU+ARW10Bom4oGY+DQTSaToZpGgZlt5sutH14gl2NoAF0tfY+GwGHeMHSdJO6AgHDIZ45rOyZy08jMTg9ZkcothtiiNBM4JhJO/Tyf6q4ZY47ATKq2aFiv/MGyAViZ1mHoY4NKf2MnUR9JbQyyEYLau6QNFwZVyC2bw7vV6hlqSzqob+VnWXMrvUvkMvIPR09GF4NlLVKDdANY+p5w45WtksAfkgDKRhheC88jpq8N/gbQzjaVIn7oejklXA4DJMFAix1MPPc7ij/hgaJ8FBnkD2T7TqxIdW+Wdnp6ej8UwLFfEOhIqJsz8nQvL/JFnXK65ry9vu/bxDb3ZHnXLSOLWrdGDtFVm4MzwVPXgy0P/vl3PLeru5jP6/0HGyiQbwsTA+2yi3j/XY1gGhhNi7jhW/aoK0l/KVeDTKdd5ALV98E+Gs7LSYJUiGr31cWMIqeHeFZg9vSf+KPe21gS/FCoUbVk+gnzG+nhu/tvuKfzCSyrvBHEfgX8iboSjZ3R0z41MqexRzw37Rf3YeBy+0vf7Q1KPYiyg1pm5/j0zg1+fkX6v3EXbOHyyE9y0X94ZrT2Zz564HkR6uGHLYypdRxFWS4pLUfBW8UkmkjWZXRDtEg10vASZ3lV3izYe3ZFK6HNOMxQl7M/qtqRs1RkJNzmO7vVTKHMUVlb3tb2LG3c+gx60f5kSvcXusLjtg==

https://play.vuejs.org/#eNqNVMFu2zAM/RXOl9pAIgfNevGSrF3Rw3bYhrW3OBg8m0nUyrIhyU6KwP8+SopTpx2KnWyRj4+Pz5QPwU1ds7bBIAlmOle8NqDRNPUilbysK2XgAArXI8irsm4MFtDBWlUlXFDRRSpTmVdSG6hqw+kF5hYd3iiVPTOLCw8gUG7MNoHLq8lkAt0Iwt8j4BHMF0DZNhMNJsChi6KoZ9MoMLfNPN1y9SZzOUwJNGB4iVVjbreZ3OCLLkev7wkwP40Qhq65xB1QIuwpmcNaEalcNzK3A0Hu+EJsUZoIDqkEyAVm6sG3C8/aUimcC6GuZGcPDqP54rwb5ZeMMUfPTKY2aNjRypUlA2BlVoehj/W2+RM7ufSZ7EtANkI4BVS05sKgCrlFc/gwn59lLaVLHVt5LUu+iiKAVHapnMV+GWgN6GCwrEVmkE4AM98Wrr018zTwL2kAZSMMr4XHEdLTE6M9AbTjdaUIH7p2bhguew1p0MMSlyacew7iT/hMUT6I9A70YPuh2TbTR7+0HamHH0UBHI5LBx0N6nTGHu7Hiz2pm5yOBW8X98c2ia09ud51s9hmrVsDh+iozbMzwUOXvS0rvz87XtjrMJ1M6v0nG9gi32xNAnQ/XMT77ziCUWA0bfKab9ijriRdU8dBptMyc4HqR28fifPjpEEmRLX75mJGkdN9PN9i/vSP+KPe21ga/FSoUbVk+innl9Kn7+6/457eT8myKhpB6HeSv1BXorEaPexLIwuSPcA5tV/dz4bLzYO+2xuUuh/KCrXIzuHpOzd4+87oL3KnbOrqyE9y0f/MxrT5r3z0iXMS6uHFbo2pdRLHeSGprEDBW8UkmljWZXxNsFg10l74cVGV11P2kV3RSmgzDDPU5fiPqnbkLJEMBrf1zm41VigLVNaW/2v7qmzY+lXqTfuTKd1f+boCMQ==


Inline validation in contact from

Related Issue
Closes #563 Contact Us Form Validation
Description
This PR contains the code to implement inline validation in the contact form.
Sample Output video
https://github.com/rohansx/informatician/assets/86224794/7469a893-39cc-402a-a037-41baed77f830
Checklist

[x] My code adheres to the established style guidelines of the project.
[x] have included comments in areas that may be difficult to understand.
[x] My changes have not introduced any new warnings.
[x] I have conducted a self-review of my code.

@AlfiyaSiddique looks good


Requesting access to earthindex Project on Google Cloud

Upon running the gee pipeline notebook I get this error:
 Caller does not have required permission to use project earthindex. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=earthindex and then retry. Propagation of the new permission may take a few minutes.

Then I opened the Google Cloud Console link which led me to this error:

Is there a way that I can get the necessary permissions for this project?

To run the code you will need to establish your own project on Google Earth Engine and edit the project name accordingly.  Good luck!


Heavy's compiler uses a too old version of Wwise

Hi,
I have just realised that Heavy compiles not only the Wwise plugin itself but the runtime part too + the Visual Studio solution, which is great.
Although, having to use an old version of Visual Studio (2013) and Wwise (v2014.1.3) is complicated, especially if I want to copy/paste stuff from old projects. Also, versions of Wwise anterior to 2014.1.4 can't be used with Unity 5 and requires Unity 4, which is an even bigger issue.
Would you mind updating the Heavy compiler to the latest version of Wwise? That would be awesome :)

@martinbussyparis we're just working on a better solution for v2015/2016 and having an option to have static runtime plugins or dynamic runtime plugins. This is definitely coming soon.

Hey Joe,
What would be a dynamic runtime plugin? Would we be able to live edit the Wwise plugin in PD?
Good to hear about the coming update anyway :)

Heavy now supports Wwise 2016 and later.


Google Calendar integration

implement the integration with Google Calendar for notify and go along the future movements of accounts

Change of plans: now this integration is only a export of a .ical file. This solve the need of integrate with calendar platforms like google


nrf52_bsim: Remove no longer needed header stubs

These headers were here only because some drivers were
using them directly.
But they are not needed anymore, so remove them.
Signed-off-by: Alberto Escolar Piedras<EMAIL_ADDRESS>

@aescolar please, rebase to get a fresh CI run before merging the PR


KTX Basis textures fail to load on all iOS browsers

Hi @atteneder , new bug report for ya.
On WebGL export projects, KTXUnity is loading some textures as black (no error, but not being displayed properly). Happens on our iPhone SE 2020 and iPhone 12 pro max. This happens on Safari, Chrome, and Firefox for iOS.

Using Chrome://inspect on iOS Chrome browser, you can see my OnSuccess messages for KTXUnity are firing off, I also have error debug logs in this project, but Unity thinks everything is fine and I'm only seeing success callbacks.
Built using Unity 2020.1.1f1, WebGL 1.0, uncompressed (no brotli or gzip compression). KTXUnity is set to 1.0.0 from February 03, 2021.

I'll update this thread if I find out anything else about this. From memory, this wasn't happening on older versions of KTXUnity, but I'll keep investigating. Let me know if you've ever seen this bug or can confirm it on your end!

@atteneder I've created a new repo for testing this bug, you can access it here.

Hi @DerrickBarra ,
Thanks for the detailed report plus test repo. I'm still on paternity leave so I won't fix it soon, but I'll have a look in the next weeks.

I observed that Safari is quite picky when it comes to textures. Is the resolution of the non-working textures by chance:

non power of two
non multiple of four
non square

and do they come with mipmaps?
If not, please try to adhere to those rules and try them again.

@atteneder so looking at one of the textures that failed to load, the rabbit in the magic hat...

Here's a link to the .basis version.
This is a 1024x1024 image (divisible by 2, multiple of 4, square, no mipmaps). And it should be working...

So as a sanity test, I retested the project on my iOS SE 2020 and now everything works as expected even after multiple cache clearings, so I guess Apple released a Safari update that fixed it? That's nice!

Great to hear!
In KtxUnity 1.1.0 I fixed a bug that potentially caused this trouble, but when I tested on my iOS devices, I still ran into troubles (but only on non-square non-multiple-of-four).
I'm waiting on 2021.2 (which will support WebGL mobile) to fix a build/linking error. After that I'll do more tests on iOS 14/15 before I call this one closed.
Thanks for your feedback

Despite the various standards, iOS devices will only support square and power of 2 multiple of four textures, I think this is due to compatibility with metal apis

Yeah, today I also ran into problems on macOS-metal with non-multiple-of-4 textures and BC7

I just tested a dev version of the upcoming 2.2.0 release and it worked successfully on iOS 16 and iPadOS 15.7 respectively.
This version chooses formats differently, so that might have solved it  (or the iOS updates).
Will close this ticket now. Releases imminent


Monitoring app - need to install rancher-monitoring-crd chart before installing monitoring

rancher/rancher:master-3411-head
Steps:

install rancher in HA mode
go to the dashboard
install monitoring from the builtin chart

Results:

the app is failed to install, the following error shows on the pod helm-operation-kbvrf

Wed, Aug 5 2020 11:20:12 am
helm install --create-namespace=true --version=8.16.100 rancher-monitoring /home/shell/helm/rancher-monitoring-8.16.100.tgz
Wed, Aug 5 2020 11:20:13 am
Error: execution error at (rancher-monitoring/templates/validate-install-rancher-monitoring-crd.yaml:15:7): Required CRDs are missing. Please install the rancher-monitoring-crd chart before installing this chart.

More info:

this error is occurring since the CRD chart has been introduced but the CRD chart is not installed automatically when you try to install rancher-monitoring as @aiyengar2 explained.
It is confirmed that there are no CRDs for monitoring is created in the cluster.
since there will be a custom UI for monitoring, I am not sure how it will influence the flow, and if it will solve this issue.
But this issue is created here for tracking the behavior.

the bug fix is validated on master-d19f179c4ff15bb782e27ed50d33d169dad79f80-head
rancher-monitoring-crd installed automatically before installing rancher-monitoring

Still getting this.
 Installing it now.
creating 4 resource(s)
beginning wait for 4 resources with timeout of 10m0s
creating 1 resource(s)
Watching for changes to Job rancher-monitoring-crd-create with timeout of 10m0s
Add/Modify event for rancher-monitoring-crd-create: MODIFIED
rancher-monitoring-crd-create: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Add/Modify event for rancher-monitoring-crd-create: MODIFIED
Starting delete for "rancher-monitoring-crd-create" Job
NAME: rancher-monitoring-crd
LAST DEPLOYED: Thu Nov 26 21:51:31 2020
NAMESPACE: cattle-monitoring-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
---------------------------------------------------------------------
SUCCESS: helm upgrade --install=true --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-crd-9.4.201.yaml --version=9.4.201 --wait=true rancher-monitoring-crd /home/shell/helm/rancher-monitoring-crd-9.4.201.tgz
---------------------------------------------------------------------
helm upgrade --install=true --namespace=cattle-monitoring-system --timeout=10m0s --values=/home/shell/helm/values-rancher-monitoring-9.4.201.yaml --version=9.4.201 --wait=true rancher-monitoring /home/shell/helm/rancher-monitoring-9.4.201.tgz
Release "rancher-monitoring" does not exist. Installing it now.
Error: execution error at (rancher-monitoring/templates/validate-install-crd.yaml:16:7): Required CRDs are missing. Please install the rancher-monitoring-crd chart before installing this chart.
  on .terraform/modules/k8s_cluster/main.tf line 268, in resource "rancher2_app_v2" "applications":
 268: resource "rancher2_app_v2" "applications" {

Running the second time it works.

@mikekuzak, I'm finding the same problem, but the app is installed correctly, it seems. The same error message is shown, but the app is working.
I'm using Pulumi instead of Terraform directly (the pulumi provider use provider from Terraform)

@mikekuzak @gabomgp4 after installing rancher-monitoring-crd, can you ensure the following CRDs exist? That is what validate-install-crds.yaml is doing:
https://github.com/rancher/charts/blob/4476584d9b9b09d6393585b3e60b47d3cf4a32e4/charts/rancher-monitoring/templates/validate-install-crd.yaml#L3-L8


Update ubi-minimal base image to ubi-minimal:9.2-750.1696515534

Update ubi-minimal base image to ubi-minimal:9.2-750.1696515534

Already merged #2391


Different dataset to train and predict

Hi, May i know is that possible to train and test SVC on different dataset ?

Of course it is possible. The only question is whether these data will be properly prepared so that correct conclusions can be drawn from the results.


setup.py: incorrect dependency link

Setuptools’ setup.py adds a link for certifi to dependency_links.  The link it adds (https://pypi.python.org/packages/source/c/certifi/certifi-2016.9.26.tar.gz) is broken (404.)
I'm not totally sure what's going on here.  My guess is that PyPI has changed how downloads are filed.  (Older packages seem to still be available under /packages/source, but recently released packages appear not to be.)
In any case, this breaks zc.buildout — at least for me.  Buildout picks up the broken dependency link from the setuptools egg and uses it (unsuccessfully) if an attempt is made to install certifi==2016.9.26.

Work-around for zc.buildout: Assuming that the use of dependency_links is not required for your project, set use-depedency-links = false in buildout.cfg.

@dstufft Do you know what URL setuptools should use for these dependency links?

https://files.pythonhosted.org/packages/source/c/certifi/certifi-2016.9.26.tar.gz

Thank you!


Can't find the option to set the starting day of the week

I'm not sure if it's just me (newbie in VSTS) or there is no option to set that your week starts with Monday.
If there is no such option, it would be good to have it as it is a standard option of any calendar.

I tested to switch my regional settings in my ms live account and in Azure DevOps - but the start of week never changed to monday. It seems it is fix coded for US needs.
Hope to support regional settings some day - because there are regions outside US ;-)

We're closing old issues so we can get back to actively maintaining the extension. Feel free to reopen if this is still an issue for you.

@MOlausson
You may label it as a feature request - the need to have a calendar start with monday has half of the world. (Regional settings did have an effect on the calendar).

@ndeeH, I'm reopening the issue for triaging based on your feedback.


Internationalization/Translation of docs

How to translate the docs? This is a very tough problem.

This was discussed briefly in Socorro - the decision was to wait a little to see what is done by e.g. Numpy/NumFOCUS, since it was reported that there are some efforts underway in that direction.

Duplicate of https://github.com/astropy/project/issues/13


popup not working without including transitions

The popup documentation page states: "Named transition to use when animating menu in and out. Fade and slide down are available without including ui transitions"
One would infer it thus works without including ui transitions, by using transition: 'fade'.or just the default 'slide down', but neither work. But keeps throwing the following error:
"Popup: This module requires ui transitions"
Working example of issue:
https://jsfiddle.net/clemsontiger/9ddpudpw/12/

Fallback javascript animations were removed from most components. This was to reduce the filesize of js components, and to avoid having to support multiple branches of animation code.
For a lighter weight solution you might want to try tooltip (css only popups) available soon in 2.2

You'll want to change your documentation then, as the popup page documentation under Usage > Settings > transition specificly states "Fade and slide down are available without including ui transitions"

Thanks

Removed note from docs

Not sure why but I can still see that note here https://semantic-ui.com/modules/popup.html#/settings

@lulalala Thank you, I opened a PR.


Image not found error

I've installed using Cocoapods, and my app written in swift.
Everything works fine in the simulator but when loading on a device, it crashes with the following error.
dyld: Library not loaded: @rpath/PKHUD.framework/PKHUD
  Referenced from: /var/mobile/Containers/Bundle/Application/457898E0-D0D7-4733-AE87-3AA67134785A/forexer.app/forexer
  Reason: image not found

I've tried adding the framework to the 'copy bundle resources' but that doesn't even build.
Any suggestions?

Not a PKHUD relates error. Please consult CocoaPods Issue #3661.


[Enhancement] Modify the default value of nullsFraction/avgRowSize/distinctCount for DeltaLake

Why I'm doing:
What I'm doing:
The optimizer currently does not handle NaN well, so set the default value for delta lake.
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[ ] This is a backport pr

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.3
[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

@Mergifyio rebase

@Mergifyio backport branch-3.3


Adding new sites

Am I the only one experiencing voting issues on new repos? There seems to be a  database connection problem with the netlify functions, I have never used them so I am not sure how to debug it.
To recreate:
go to tokenlog.xyz/liminalvillage/governance
vote with DAI
Error:
Unable to cast vote. Please check your account and available voting power.
in the console:
In the console:
POST https://tokenlog.xyz/.netlify/functions/vote?org=liminalvillage&repo=governance 502
(anonymous) @ xhr.js:184
e.exports @ xhr.js:13
e.exports @ dispatchRequest.js:50
Promise.then (async)
u.request @ Axios.js:61
r.forEach.u.
This is using a different format in preparation for the latest development/release.
That's not public yet and won't work for any other repos.
I've submitted a PR on your repo with a correct tokenlog.json.
Or check out the readme/docs if you want to configure any other.
Closing this issue.


Add unit tests

Add some tests for the backend.
Inbound message handlers will not be covered here; they are just a bit too messy right now.

I discovered that the token expiry mechanism didn't work. This is why I should have put unit tests in from the start!

Trying to figure out how to test the broadcast is turning out to be a pain.

Since I have some tests, I'm going to leave this as is for now.


[RTM] use only fixed image mask for reports

Fixes https://github.com/poldracklab/fmriprep/issues/222

This is how it looks like now:
https://126-63891310-gh.circle-artifacts.com/0/home/ubuntu/scratch/testRobustMNINormalizationRPT_masked.svg
This is how it used to looks if we mask only the fixed image:
https://125-63891310-gh.circle-artifacts.com/0/home/ubuntu/scratch/testRobustMNINormalizationRPT_masked.svg
This is how it used to look like without masking:
https://125-63891310-gh.circle-artifacts.com/0/home/ubuntu/scratch/testANTSRegistrationRPT.svg
WDYT?

I would use the fixed mask for both images (reference and moving), in this case.

Agreed.
On Mon, Dec 5, 2016 at 2:12 PM, Oscar Esteban<EMAIL_ADDRESS>wrote:

I would use the fixed mask for both images (reference and moving), in this
case.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/poldracklab/niworkflows/pull/68#issuecomment-264994633,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp0eE6wNO1teUZ0sYnB23cmU983Jaks5rFIxkgaJpZM4LEknc
.


Improve Traefik secuity settings, add security documentation

Description

Update Traefik settings to increase TLS strength, hide server software and enforce the same security Django uses for Flower (when it is used). Also add new documentation on security and explain any caveats in compatibilty caused by security settings.
Checklist:

[x] I've made sure that tests/test_cookiecutter_generation.py is updated accordingly (especially if adding or updating a template option)
[x] I've updated the documentation or confirm that my change doesn't require any updates

Rationale

Fixes: #2841 fixes: #1992
The security settings used by Traefik were previously as default, this PR forces Traefik to use better TLS ciphers plus various other small security tweaks. When using docker, some HTTP secutity headers set by django are now set by Traefik so that they are also used by Flower (if celery is used) and any other parts of the site not served by Django.
Traefik has also been configured to run as non-root and to mount the traefik.yml rather than copy in the Dockerfile so that live reloading can be used when making changes.
Testing
I am running the exact same non-root Dockerfile in prod currently, no issues so far (I did see on the original issue about some kind of redirect issues in the logs but I am not seeing this?). Also am running the same TLS/headers settings, works as expected.

@browniebroke I think given there is already a fair bit of diff, CSP makes sense as its own issue. That would also need a lot of manual testing to ensure it doesn't block needed site features.

I haven't looked into details yet, but from a quick glance, it looks like some very solid work here @jameswilliams1!
Since this is touching with security settings, I don't want to rush the review and give it a good look.
One question I have (which might be a stupid one) is to know whether it makes sense to have some security settings both in Django and Traefik? You mention the Django deploy checks will raise false positives for example...

I think given there is already a fair bit of diff, CSP makes sense as its own issue.

Agree

@browniebroke yeah thats something I was deliberating with. On one hand I guess it's good to have a fallback, say if someone messes with the traefik.yml and removes the security and then django still catches it. But on the other config in 2 places gets confusing (as traefik will completely overwrite the django settings regardless of changes to production.py). I guess an option is to add the django settings back with a NOTE saying these are a fallback and won't affect anything when traefik is in use?

@Andrew-Chen-Wang completely forgot about this PR, have just rebased and ci has passed now, could you review again?

Shouldn't production.yml have the ports remapped for traefik?
e.g. "<IP_ADDRESS>:80:8080"