remote control plane: round-robin between istiod in multiple clusters

Bug description
When deploying Istio in two clusters, one control plane cluster and one remote cluster, the remote cluster inconsistently receives config from the control plane cluster.
Expected behavior
Envoys in the remote cluster should only receive configuration from the control plane cluster.
Steps to reproduce the bug
First, deploy sleep service to cluster2:
$ kubectl apply --context=${CTX_2} \
  -f ${ISTIO}/samples/sleep/sleep.yaml -n sample

Looking at the istiod logs in cluster1, config was pushed properly:

$ kubectl --context=$CTX_1 -n istio-system logs istiod-7c9fb7cc59-2kzhc  | grep sleep
2020-05-07T00:46:23.034999Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
        "outbound_.80_._.sleep.sample.svc.cluster.local": {},
        "outbound|80||sleep.sample.svc.cluster.local": {}
2020-05-07T00:46:29.018095Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:46:29.024914Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:46:42.332937Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:46:42.735265Z	info	ads	ADS:CDS: REQ <IP_ADDRESS>:35380 sidecar~<IP_ADDRESS>~sleep-f8cbf5b76-s5zgq.sample~sample.svc.cluster.local-5 469.94µs version:
2020-05-07T00:46:42.736631Z	info	ads	CDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample clusters:33 services:11 version:2020-05-07T00:46:34Z/17
2020-05-07T00:46:42.772257Z	info	ads	EDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample clusters:28 endpoints:50 empty:1
2020-05-07T00:46:43.424483Z	info	ads	LDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample listeners:30
2020-05-07T00:46:43.479825Z	info	ads	RDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample routes:12
2020-05-07T00:46:43.548809Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:46:43.564136Z	info	Handle EDS: 1 endpoints for sleep in namespace sample
2020-05-07T00:46:43.564310Z	info	ads	Full push, new service sleep.sample.svc.cluster.local
2020-05-07T00:46:43.564381Z	info	ads	Full push, service accounts changed, sleep.sample.svc.cluster.local
2020-05-07T00:46:43.666785Z	info	ads	Pushing sidecar~<IP_ADDRESS>~sleep-f8cbf5b76-s5zgq.sample~sample.svc.cluster.local-5
2020-05-07T00:46:43.668093Z	info	ads	CDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample clusters:33 services:11 version:2020-05-07T00:46:43Z/18
2020-05-07T00:46:43.670330Z	info	ads	EDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample clusters:28 endpoints:51 empty:0
2020-05-07T00:46:43.729299Z	info	ads	LDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample listeners:30
2020-05-07T00:46:43.729772Z	info	ads	RDS: PUSH for node:sleep-f8cbf5b76-s5zgq.sample routes:12

Below, we check the istiod logs for the remote cluster and the pushes aren't present, which is expected.

$ kubectl --context=$CTX_2 -n istio-system logs istiod-86597465d4-cdw4t | grep sleep
2020-05-07T00:46:23.051039Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:46:23.218346Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=sample Name= (sleep-f8cbf5b76-***** (actual name not yet known)) UID=8f490f5e-f024-46fb-8cf0-fa4a3b19757c Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller bc0813d1-494a-41d3-a387-27ed3be3e789 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2020-05-07T00:46:29.031274Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:46:29.039420Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:46:42.346367Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:46:43.561637Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:46:43.579486Z	info	Handle EDS: 1 endpoints for sleep in namespace sample
2020-05-07T00:46:43.579677Z	info	ads	Full push, new service sleep.sample.svc.cluster.local
2020-05-07T00:46:43.579750Z	info	ads	Full push, service accounts changed, sleep.sample.svc.cluster.local

Traffic from sleep in cluster2 to the helloworld gets routed to both cluster1 and cluster2 properly.
for i in $(seq 10); do kubectl --context=$CTX_2 -n sample exec -it sleep-f8cbf5b76-s5zgq --container=sleep curl helloworld.sample:5000/hello; done
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v1, instance: helloworld-v1-578dd69f69-9g8bw
Hello version: v1, instance: helloworld-v1-578dd69f69-9g8bw
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v1, instance: helloworld-v1-578dd69f69-9g8bw
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v1, instance: helloworld-v1-578dd69f69-9g8bw
Hello version: v1, instance: helloworld-v1-578dd69f69-9g8bw
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh

When a new sleep pod gets created we can start to see issues:
$ kubectl --context=$CTX_2 -n sample scale deployment sleep --replicas=0
$ kubectl --context=$CTX_2 -n sample scale deployment sleep --replicas=1

Looking at the logs again, cluster1 doesn't push config for the new sleep-f8cbf5b76-hbxxx pod.

$ kubectl --context=$CTX_1 -n istio-system logs istiod-7c9fb7cc59-2kzhc  | grep sleep | grep -v 'T00:46'
        "outbound_.80_._.sleep.sample.svc.cluster.local": {},
        "outbound|80||sleep.sample.svc.cluster.local": {}
2020-05-07T00:52:32.397746Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:32.408319Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:52:32.408360Z	info	ads	Incremental push, service sleep.sample.svc.cluster.local has no endpoints
2020-05-07T00:52:37.470053Z	info	ads	ADS: "<IP_ADDRESS>:35380" sidecar~<IP_ADDRESS>~sleep-f8cbf5b76-s5zgq.sample~sample.svc.cluster.local-5 terminated rpc error: code = Canceled desc = context canceled
2020-05-07T00:52:47.787978Z	info	Handling event update for pod sleep-f8cbf5b76-hbxxx in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:47.804926Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:52:47.805003Z	info	ads	Incremental push, service sleep.sample.svc.cluster.local has no endpoints
2020-05-07T00:52:50.827904Z	info	Handling event update for pod sleep-f8cbf5b76-hbxxx in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:53.461863Z	info	Handling event update for pod sleep-f8cbf5b76-hbxxx in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:53.474278Z	info	Handle EDS: 1 endpoints for sleep in namespace sample
...

However, cluster2's istiod does push config:

$ k --context=$CTX_2 -n istio-system logs istiod-86597465d4-cdw4t | grep sleep | grep -v 'T00:46'
2020-05-07T00:52:32.414591Z	info	Handling event update for pod sleep-f8cbf5b76-s5zgq in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:32.424371Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:52:32.424692Z	info	ads	Incremental push, service sleep.sample.svc.cluster.local has no endpoints
2020-05-07T00:52:34.135487Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=sample Name= (sleep-f8cbf5b76-***** (actual name not yet known)) UID=283e02dd-e768-4b87-895f-40b8fd82fbda Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller bc0813d1-494a-41d3-a387-27ed3be3e789 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2020-05-07T00:52:47.805472Z	info	Handling event update for pod sleep-f8cbf5b76-hbxxx in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:47.820711Z	info	Handle EDS: 0 endpoints for sleep in namespace sample
2020-05-07T00:52:47.820982Z	info	ads	Incremental push, service sleep.sample.svc.cluster.local has no endpoints
2020-05-07T00:52:50.844291Z	info	Handling event update for pod sleep-f8cbf5b76-hbxxx in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:51.480792Z	info	ads	ADS:CDS: REQ <IP_ADDRESS>:55268 sidecar~<IP_ADDRESS>~sleep-f8cbf5b76-hbxxx.sample~sample.svc.cluster.local-2 528.933µs version:
2020-05-07T00:52:51.512787Z	info	ads	EDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample clusters:23 endpoints:23 empty:1
2020-05-07T00:52:51.750595Z	info	ads	LDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample listeners:26
2020-05-07T00:52:51.798644Z	info	ads	RDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample routes:11
2020-05-07T00:52:53.477554Z	info	Handling event update for pod sleep-f8cbf5b76-hbxxx in namespace sample -> <IP_ADDRESS>
2020-05-07T00:52:53.489225Z	info	Handle EDS: 1 endpoints for sleep in namespace sample
...

istiod running in cluster2 has no knowledge of cluster1 endpoints. If we try hitting helloworld a few times, we will always get v2 which runs in cluster2:
$ for i in $(seq 10); do kubectl --context=$CTX_2 -n sample exec -it sleep-f8cbf5b76-hbxxx --container=sleep curl helloworld.sample:5000/hello; done
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh
Hello version: v2, instance: helloworld-v2-776f74c475-tt2xh

Looking at the clusters on the ingress gateway, there are multiple endpoints for istiod.
The remote cluster uses ingressgateway to connect to the primary cluster's istiod, but we end up round-robining between the two.
$ kubectl --context=$CTX_1 -n istio-system exec istio-ingressgateway-8b94c68db-fgjch curl <IP_ADDRESS>:15000/clusters | grep 15012 | grep istiod
k --context=$CTX_1 -n istio-system exec istio-ingressgateway-8b94c68db-fgjch curl <IP_ADDRESS>:15000/clusters | grep -E '15012.*istiod.*cx_active'
outbound|15012||istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::1
outbound|15012||istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::2
outbound_.15012_._.istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::0
outbound_.15012_._.istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::0

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
1.5.2
$ istioctl --context=$CTX_1 version --remote
client version: 1.5.2
control plane version: 1.5.2
data plane version: 1.5.2 (5 proxies)
$ istioctl --context=$CTX_2 version --remote
client version: 1.5.2
control plane version: 1.5.2
data plane version: 1.5.2 (3 proxies)

How was Istio installed?
Multicluster setup with a control plane cluster (cluster1) and remote cluster (cluster2). Both clusters are on the same network.
Generated certs for each cluster using samples/certs and pushed cacerts secrets to each cluster.
Primary Cluster
istioctl manifest apply --context=${CTX_1} \
    --set profile=default \
    --set values.global.multiCluster.clusterName=${CLUSTER_1} \
    --set values.global.meshExpansion.enabled=true

Remote Cluster
$ export DISCOVERY_ADDRESS=$(kubectl \
    --context=${CTX_1} \
    -n istio-system get svc istio-ingressgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ istioctl manifest apply --context=${CTX_2} \
    --set profile=remote \
    --set values.global.multiCluster.clusterName=${CLUSTER_2} \
    --set values.global.remotePilotAddress=${DISCOVERY_ADDRESS}

Helloworld Service was installed with v1 on cluster1 and v2 on cluster2
Environment where bug was observed (cloud vendor, OS, etc)
GKE 1.17

cc @linsun @GregHanson @nmittler

A fix for this @howardjohn suggested was to make the istiod in the primary cluster cluster-local.

Looking at the logs again, cluster1 doesn't push config for the new sleep-f8cbf5b76-hbxxx pod.

Can you make sure this new pod connects to the primary cluster's istiod?

Can you make sure this new pod connects to the primary cluster's istiod?
I don't believe it does, which is the issue.

The envoy is configured with istiod-remote for discovery:
$ kubectl --context=$CTX_2 -n sample logs sleep-f8cbf5b76-hbxxx istio-proxy | grep 'PilotSAN\|discoveryAddress'
2020-05-07T01:17:12.573450Z	info	FLAG: --discoveryAddress="istiod-remote.istio-system.svc:15012"
2020-05-07T01:17:12.575846Z	info	PilotSAN []string(nil)
discoveryAddress: istiod-remote.istio-system.svc:15012
2020-05-07T01:17:12.628747Z	info	PilotSAN []string{"istiod-remote.istio-system.svc"}

The following gives no results:
kubectl --context=$CTX_1 -n istio-system logs istiod-7c9fb7cc59-2kzhc  | \
    grep -E 'PUSH for node:sleep-f8cbf5b76-hbxxx'

however cluster2 does seem to have performed the push:
landow@Stevens-MacBook-Pro:certs $ k --context=$CTX_2 -n istio-system logs istiod-86597465d4-cdw4t | grep 'PUSH for node:sleep-f8cbf5b76-hbxxx'
2020-05-07T00:52:51.482038Z	info	ads	CDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample clusters:29 services:10 version:2020-05-07T00:46:43Z/11
2020-05-07T00:52:51.512787Z	info	ads	EDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample clusters:23 endpoints:23 empty:1
2020-05-07T00:52:51.750595Z	info	ads	LDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample listeners:26
2020-05-07T00:52:51.798644Z	info	ads	RDS: PUSH for node:sleep-f8cbf5b76-hbxxx.sample routes:11

Is istiod-remote.istio-system.svc correctly resolved to the ip address of remote istiod?

This is interesting finding @stevenctl!  Not sure how making istiod cluster-local on the main cluster would solve this?
It feels to me it is best we don't run istiod on the remote cluster to avoid 2 istiod pushing configs.

@hzxuzhonghu

Is istiod-remote.istio-system.svc correctly resolved to the ip address of remote istiod?

It correctly resolves to <IP_ADDRESS> which is the ingressgateway IP on the primary cluster.
NAME                   TYPE           CLUSTER-IP    EXTERNAL-IP
istio-ingressgateway   LoadBalancer   <IP_ADDRESS>   <IP_ADDRESS>

@linsun

It feels to me it is best we don't run istiod on the remote cluster to avoid 2 istiod pushing configs.

Don't we need istiod on the remote cluster for SDS/CA purposes?
Makin istiod cluster-local would ensure traffic from ingressgateway->istiod always goes to the main istiod.

It correctly resolves to <IP_ADDRESS> which is the ingressgateway IP on the primary cluster.

That's weird, then how does it connect to local istiod?

@hzxuzhonghu
$ kubectl --context=$CTX_1 -n istio-system exec istio-ingressgateway-8b94c68db-fgjch curl <IP_ADDRESS>:15000/clusters | grep 15012 | grep istiod
k --context=$CTX_1 -n istio-system exec istio-ingressgateway-8b94c68db-fgjch curl <IP_ADDRESS>:15000/clusters | grep -E '15012.*istiod.*cx_active'
outbound|15012||istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::1
outbound|15012||istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::2
outbound_.15012_._.istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::0
outbound_.15012_._.istiod.istio-system.svc.cluster.local::<IP_ADDRESS>:15012::cx_active::0

Notice that there are two endpoints for the istiod cluster. They correspond to the Pod IPs istiod in each cluster.
$ kubectl --context=$CTX_1 -n istio-system get pod istiod-7c9fb7cc59-2kzhc -o jsonpath='{.status.podIP}'; echo
<IP_ADDRESS>
$ kubectl --context=$CTX_2 -n istio-system get pod istiod-86597465d4-cdw4t  -o jsonpath='{.status.podIP}'; echo
<IP_ADDRESS>

The xds server address is static from the bootstrap config like this.
        "load_assignment": {
          "cluster_name": "xds-grpc",
          "endpoints": [{
            "lb_endpoints": [{
              "endpoint": {
                "address":{
                  "socket_address": {"address": "istiod.istio-system.svc", "port_value": 15012}
                }
              }
            }]
          }]
        },

@linsun @GregHanson
@howardjohn and I chatted offline and we think making istiod cluster-local may be the right solution here. The only requirement is that istiod be accessed via ingress from all remote clusters (this is already planned to be the recommended solution for both single and multi-network in 1.6).
@stevenctl mind giving this a try to see if it works?

@nmittler I hacked it into https://github.com/istio/istio/pull/23208 and ran the test 3 times without failing. I'll give it a go on the GKE cluster as well.

@stevenctl seems you are making istiod on remote cluster to be cluster local?  if so, that is reasonable.
Now, for the main cluster's istiod, do you need to make it cluster local?  BTW, we have central istiod coming (there is a flag centralIstiod that guards it)where users can configure remote plane not to run istiod.
Also, @stevenctl do you only see this prob with flat network?

@linsun This actually applies the same configuration to both clusters in the test. The main cluster is the only one that should need cluster-local istiod.
I haven't tested it in multinetwork, but I think things would function correctly since the remote cluster's istiod is unreachable by the main cluster's ingressgatway.

@stevenctl I think we should make istiod.istio-system... cluster local in general. Let's split that out into a separate PR, though.

@stevenctl nice work @howardjohn and @nmittler!  This makes sense, I understood your issue wrong but got it now (hopefully!):  the prob is istiod on remote cluster pushes config to sidecar on the main cluster.
I think @hzxuzhonghu probably read it wrong as well.

So I'm thinking that as part of the solution we set the configured "discoveryAddress" for the cluster to be cluster-local.
In a scenario where we have 1 master cluster (a) and 2 remotes (b, c): both of those remotes will have an istiod-remote service. I'm wondering if we'd run into this issue, minus the impact, where b and c will intermittently use eachother's istiod-remote service. It doesn't really matter in the end I guess, since they should both resolve to the same end host.

Thanks for the explanation @linsun, got it now

I remember that there will be only one istiod in the primary cluster eventually.


Retry actual connection to the server

Currently (9.3.1) it's the last part that is not covered by retry mechanism.

https://github.com/keboola/db-extractor-common/issues/77


Pre-release PR

[Breaking change] Replace OutModules enum by custom compiler option mjsModule.
[Breaking change] Drop support for Pug, Sass, Angular & Webpack.
[Feature] Expose custom registries for each target.
[Fix] Disable deprecated TsLint rules from the default config
[Internal] Fix continuous deployment script (stop confusing PRs to master
with push to master)
[Internal] Update dependencies

Codecov Report

Merging #113 into master will increase coverage by 0.35%.
The diff coverage is 74.19%.

@@            Coverage Diff             @@
##           master     #113      +/-   ##
==========================================
+ Coverage   67.58%   67.94%   +0.35%
==========================================
  Files          34       34
  Lines         870      886      +16
  Branches       98       96       -2
==========================================
+ Hits          588      602      +14
- Misses        213      216       +3
+ Partials       69       68       -1

Impacted Files
Coverage Δ

src/lib/utils/npm-publish.ts
25% <ø> (ø)
:arrow_up:

src/lib/options/tslint.ts
100% <ø> (ø)
:arrow_up:

src/lib/project-tasks/lint.ts
76.19% <100%> (+0.58%)
:arrow_up:

src/lib/project-tasks/tslint-json.ts
82.35% <100%> (ø)
:arrow_up:

src/lib/task-generators/nyc.ts
100% <100%> (ø)
:arrow_up:

src/lib/index.ts
100% <100%> (ø)
:arrow_up:

src/lib/utils/matcher.ts
68.75% <100%> (ø)
:arrow_up:

src/lib/task-generators/mocha.ts
90.9% <100%> (ø)
:arrow_up:

src/lib/utils/nyc.ts
69.23% <100%> (ø)
:arrow_up:

src/lib/task-generators/clean.ts
73.33% <100%> (ø)
:arrow_up:

... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1677b1...f1fb14c. Read the comment docs.


how can I view the loss by tensorboard.

I want to view the loss by chart ,but I do not which folder should I position to open tensorboard?

I have solved my question.


PsAll is missing

Takes misses Pass implementation that will require each provided Pass to success in order to the target Pass to success.
The concept is similar to python all builtin.
The contract could be like this:
public PsAll(final Iterable<Pass> list, final int index)

The index parameter will indicate which Identity to return once all suceeded.

@ekondrashev I can't imagine where such a class may be useful. Can you give a real life example?

@yegor256 Sure, for example in scope of netbout#685 there is an need in so called PsTwice which is actually the corner case of PsAll.
Here is the quote:

PsTwice which use two a Pass instances. Inside PsTwice.enter() we have to check the first Pass and if it passed succesfully then continue with second Pass and return Identity from it.

@ekondrashev hm.. make sense, thanks

@ekondrashev I am aware of the task, give me some time to find a developer...

@ekondrashev I set milestone here to 1.0, let me know if it is wrong

@ekondrashev thanks a lot for reporting, 15 mins added to your acc, pmt ID 59564195

@lautarobock please proceed, it's yours

PR#350 is ready for review

@lautarobock you're holding this ticket for 15 days already. If it is not finished (and closed) in the next 48 hours, it will be re-assigned to someone else, see No Obligations principle (this article should help if you're stuck). -30 added to your rating, current score is: -270

@suseika please proceed,it's yours

@davvd I need more time.

@davvd I need more time.

@suseika of course, take your time

@rultor @ekondrashev please take a look, rultor closed it automatically

@Suseika thank you!

@davvd Closed with #366

@elenavolokhova please, review this ticket for compliance with our QA rules

@davvd Looks good!

@davvd Looks good!

@elenavolokhova thanks for the review and a good remark

@suseika I added 10 mins to @elenavolokhova (for QA review) in transaction 61671588... thank you, added 30 mins to your acc, payment num is AP-1JC27519LF7076547, 208 hours and 34 mins spent in total to complete... added +30 to your rating, now it is equal to +413


styling.opacity is meanless

Hi there,
You implemented animation of toast like this,
    [UIView animateWithDuration:CSToastFadeDuration
                          delay:0.0
                        options:(UIViewAnimationOptionCurveEaseOut | UIViewAnimationOptionAllowUserInteraction)
                     animations:^{
                         toast.alpha = CSToastOpacity;
                     } completion:^(BOOL finished) {
                         NSTimer *timer = [NSTimer scheduledTimerWithTimeInterval:duration target:self selector:@selector(toastTimerDidFinish:) userInfo:toast repeats:NO];
                         // associate the timer with the toast view
                         objc_setAssociatedObject (toast, &CSToastTimerKey, timer, OBJC_ASSOCIATION_RETAIN_NONATOMIC);
                     }];

toast.alpha = CSToastOpacity; will go to predefined value 0.8, so our styling.opacity value is meanless. Can you set toast.alpha to "styling.opacity" value ? When I set to styling.opacity : 1.0 toast view is still transparent because of this reason.
Thanks.

Let me explain with screenshots, I set opacity to 1.0 in styling object, but toast looks like that because of
toast.alpha = CSToastOpacity;

code in your implementation. (alpha goes from 0.0 to 0.8)

I'll look into it, thanks!

Fixed in 2.4.2

You are just awesome, you always answer issues and solve them very fast, please add a donate button. Keep being amazing.

Thanks man, I'll do my best :)
Donate button has been added to the readme (thanks for the idea!).


No data for this layer

Quiver start normally, model and images are all visiable, but when I clicked the Input Layer (or other layers), got an error:
Any ideas?
E tensorflow/core/client/tensor_c_api.cc:485] input_1:0 is both fed and fetched.
[2016-11-21 10:02:37,298] ERROR in app: Exception on /layer/input_1/ILSVRC2012_val_00000610.jpg [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/nali/Workspace/quiver/quiver_engine/server.py", line 103, in get_layer_outputs
    layer_outputs = output_generator(input_img)[0]
  File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1197, in predict
    batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 896, in _predict_loop
    batch_outs = f(ins_batch)
  File "/usr/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1040, in __call__
    updated = session.run(self.outputs + [self.updates_op], feed_dict=feed_dict)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
InvalidArgumentError: input_1:0 is both fed and fetched.
<IP_ADDRESS> - - [2016-11-21 10:02:37] "GET /layer/input_1/ILSVRC2012_val_00000610.jpg HTTP/1.1" 500 444 0.065454
[2016-11-21 10:02:37,323] ERROR in app: Exception on /predict/ILSVRC2012_val_00000610.jpg [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/nali/Workspace/quiver/quiver_engine/server.py", line 127, in get_prediction
    input_img = load_img(input_path, single_input_shape, grayscale=is_grayscale)
  File "/Users/nali/Workspace/quiver/quiver_engine/util.py", line 27, in load_img
    img = image.load_img(input_path, target_size=target_shape, grayscale=grayscale)
  File "/usr/local/lib/python2.7/site-packages/keras/preprocessing/image.py", line 173, in load_img
    img = Image.open(path)
  File "/usr/local/lib/python2.7/site-packages/PIL/Image.py", line 2272, in open
    fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: u'ILSVRC2012_val_00000610.jpg'
<IP_ADDRESS> - - [2016-11-21 10:02:37] "GET /predict/ILSVRC2012_val_00000610.jpg HTTP/1.1" 500 444 0.006262

Did you install with pip or clone the git repo?

I cloned the repo and builded from master yesterday.

Fixed after reinstalled.


Add extra tracing to kafka init failure.

Signed-off-by: Ozzy Osborne<EMAIL_ADDRESS>

This change is

Reviewed 2 of 2 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable