[WIP] Local Cost Functions

Title:
Summary:
Relevant references: https://arxiv.org/abs/2001.00550
Possible Drawbacks:
Related GitHub Issues:

Website build
Commit: edd4206ac2d97bccc97b8515c6b5ab44d68330c6
Circle build number: 764
Built demos directory: https://764-214003948-gh.circle-artifacts.com/0/tmp/qml_demos.zip
Backreferences zip: https://764-214003948-gh.circle-artifacts.com/0/tmp/qml_backreferences.zip
Website zip: https://764-214003948-gh.circle-artifacts.com/0/tmp/qml_html.zip
Website url: https://764-214003948-gh.circle-artifacts.com/0/_build/html/index.html
Please double check the rendered website build to make sure everything is correct.

Hi @Tstorwick, thanks for the PR contribution! Looking forward to having a deeper look.

Hi @Tstorwick, thanks for the PR contribution! Looking forward to having a deeper look.

No problem! Thanks for taking a look at it :)

@josh146 This new version still has the wrong incoding, but for the life of me i cant seem to fix it. I've tried even manually replacing the characters and it still doesnt work.

Hey @Tstorwick, thanks for contributing the tutorial. The idea is great, and your explanations clear, but I think there is a bit more work needed.
@josh146 and @trbromley have already left such a detailed review. Let me only add some high-level comments.
Something is still off in the way things are designed. If the RX and RY rotation turns a qubit into state a|0> + b|1>, then the "local circuit" produces |a|^2. The global circuit should be just (|a|^2)*6 (since you use the same rotation angle on every qubit, and the chance of each qubit being in state 0 is |a|^2). I checked, and this is consistent with your code.
But this is why I don't think you are actually reproducing the results from the paper. Your results may not a property of a local or global measurement, but a property of smaller and larger probabilities, if I am not mistaken. (But I could be wrong). So I think it would be good to go back to the paper and really check that what you are doing contains the complexity of what they prove. Especially since it is not clear if the entire observation could be made on circuits where the qubits are not entangled.
And here a few more things that sprung to my mind:

Why do you initialise all rotation angles for RX in the same value, and also for RY?
the original circuit ansatz proposed in the paper is actually implemented as a PennyLane template...would that help? You are using unentangled qubits in the tutorial. So would you consider using the ansatz instead, which uses entanglers and actually corresponds to the paper?
my local build fails due to this line missing for 3D plotting: from mpl_toolkits.mplot3d import Axes3D Include in the imports?
It would be great to add a sentence explaining P_{|0>} (and what quantum observable it corresponds to, since the paper uses expectation values as outputs).
The tutorial takes very long to build. I like the idea to "artificially shorten" the runtime (i.e. number of training steps), and load pre-trained parameters in the tutorial, telling the user that they can reproduce by increasing the number of steps to XYZ. (See embeddings_metric_learning tutorial)

Hi Maria! I couldn't figure out how to reply directly, but the quote option will do :).
So, as I understood the paper, the local vs global measurement advantage was linked to smaller and large probabilities. A local function means a simpler state, and therefore a more "drastic" gradient when training. I thought it was just trivially easier to see when the qubits arent entangled. That said, a proper implementation should have entanglements, so I added them. I originally had a more complex problem after this toy version, that included entanglements and non-trivial search spaces, but runtimes were getting extreme (~2-3 hours) so I cut it.
"Why do you initialise all rotation angles for RX in the same value, and also for RY?", Originally, I was showing all the starting points on a plot, but decided to cut it for readability. This is just a relic I missed. Good catch!
"my local build fails due to this line missing for 3D plotting: from mpl_toolkits.mplot3d import Axes3D Include in the imports?" I tried on a clean python environment locally, and it seemed to work for me. Any more information on the error?
"The tutorial takes very long to build. I like the idea to "artificially shorten" the runtime (i.e. number of training steps), and load pre-trained parameters in the tutorial, telling the user that they can reproduce by increasing the number of steps to XYZ. (See embeddings_metric_learning tutorial)" Good Idea, I'll give implementing this a shot!

@Tstorwick Sorry, I missed your reply from a while back (must be because I wasn't mentioned by @). But it makes a lot of sense, thanks for the comments. Is this ready for final review?

@mariaschuld It is! Let me know what you think.

Hey @Tstorwick , with your explanation about where to find the example and how it is meant in the bigger context, I finally understand the scope of the tutorial. Really nice!
Since I blocked you for so long I approve the PR, but would recommend these two minor changes:

As mentioned I'd put the repeated plotting code into a function and make calls to that function, so the reader can focus on the important bits
Thanks for changing the title - I am now not so sure any more my request was so clever. "Alleviating" is more accurate but much less catchy. If you prefer "Avoid" I'm happy for you to change back, but something like "Getting out of barren plateaus" could also work.

Good job!


Can't build npm on mac

building with  ./build.sh fails on mac:
2609 verbose stack Error: command failed
2609 verbose stack     at ChildProcess.<anonymous> (/Users/philippk/.nvm/versions/node/v16.5.0/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/index.js:64:27)
2609 verbose stack     at ChildProcess.emit (node:events:394:28)
2609 verbose stack     at maybeClose (node:internal/child_process:1067:16)
2609 verbose stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
2610 verbose pkgid<EMAIL_ADDRESS>2611 verbose cwd /Users/philippk/Dropbox/se_projects/July21_Hackweek/FloorPlanner/sstk
2612 verbose Darwin 20.5.0
2613 verbose argv "/Users/philippk/.nvm/versions/node/v16.5.0/bin/node" "/Users/philippk/.nvm/versions/node/v16.5.0/bin/npm" "install"
2614 verbose node v16.5.0
2615 verbose npm  v7.19.1
2616 error code 1
2617 error path /Users/philippk/Dropbox/se_projects/July21_Hackweek/FloorPlanner/sstk
2618 error command failed
2619 error command sh -c node getdata.js && npm run bootstrap
2620 verbose exit 1

Would be awesome to have dockerized version...

The SSTK currently requires an older version of node (node v10.x.x).  The next update to the SSTK will support newer versions of node.


Sorts the scene entries by path before serializing.

Objective
Fixes: https://github.com/bevyengine/bevy/issues/14515
Solution
Sorts the iterator with itertools' sorted_by function. This is required given that 'self.entries' is an immutable &[Box<dyn PartialReflect] which also doesn't implement Clone  or Copy.
Testing
The modifications passed the unit testing only after they were edited to ensure that the items were in alphabetical order.
I haven't checked for performance implications.

Welcome, new contributor!
Please make sure you've read our contributing guide and we look forward to reviewing your pull request shortly ✨


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from  to before v4.9.273",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=d6a21a3fb03300fbaa9fc3ed99f8b0962ce28362"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "d6a21a3fb03300fbaa9fc3ed99f8b0962ce28362",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "net/nfc/rawsock.c: fix a permission check bug\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.9.273 by commit d6a21a3fb03300fbaa9fc3ed99f8b0962ce28362. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1000835


Refresh OpenFin runtime cache when app.json changes

Is the circleci job hanging because of the changes? Although https://github.com/tallarium/openfin-tests/pull/33 is passing just fine and it is rebased on top of this, so that shouldn't be the reason

Is the circleci job hanging because of the changes? Although #33 is passing just fine and it is rebased on top of this, so that shouldn't be the reason

The problem was that CircleCi didn't pick up the build at all.  Rebasing has fixed this.


Chore: Adapt Compose Gradle Plugin

Hi! @sridhar-sp
Here is a suggestive commit to adapt compose gradle plugin for this repo.
This will help to maintain compose library versions.
Please review this PR.
V/r
Thank you.

Hope this is tested locally,
Will review and reply by tomorrow.

Thank you for the rapid response!


ionic:localhost problem

Hi,
I am going crazy to resolve an issue.
All my GET from my app to server A are stopped by CORS
All my GET from my app to server B are OK
I check the .htaccess and is fine,very similar . One is Prestashop, the other is Wordpress.
The same app on the browser works fine, installed to the phone work only server B
They works fine in the last 4 months, now the supplier move the website to another hosting, but i check everythings and are fine on server-side.
I not understand what happen, i think the server but i am not sure that le GET from app starts with ionic:localhost before this issue
Ionic:
Ionic CLI                     : 5.4.16
Ionic Framework               : @ionic/angular 4.11.10
@angular-devkit/build-angular : 0.801.3
@angular-devkit/schematics    : 8.1.3
@angular/cli                  : 8.1.3
@ionic/angular-toolkit        : 2.0.0
Cordova:
Cordova CLI       : 9.0.0<EMAIL_ADDRESS>Cordova Platforms : android 8.1.0, ios 5.1.1
Cordova Plugins   : cordova-plugin-ionic-keyboard 2.2.0, cordova-plugin-ionic-webview 4.2.0, (and 17 other plugins)
Utility:
cordova-res (update available: 0.12.1) : 0.8.0
native-run (update available: 1.0.0)   : 0.2.9
System:
ios-deploy : 1.9.4
ios-sim    : 8.0.2
NodeJS     : v10.15.3 (.nvm/versions/node/v10.15.3/bin/node)
npm        : 6.13.0
OS         : macOS Catalina
Xcode      : Xcode 11.4.1 Build version 11E503a

sorry, looks like they did something wrong when configuring CORS, but it's not a problem on the plugin


Add annotations to renderer for internal google compatibility.

Codecov Report

Merging #155 (b58467d) into main (d1f0e37) will increase coverage by 0.03%.
The diff coverage is 76.92%.

@@            Coverage Diff             @@
##             main     #155      +/-   ##
==========================================
+ Coverage   86.74%   86.78%   +0.03%
==========================================
  Files          16       16
  Lines        2415     2414       -1
==========================================
  Hits         2095     2095
+ Misses        320      319       -1

Impacted Files
Coverage Δ

jax_md/colab_tools/renderer.py
2.20% <0.00%> (+0.01%)
:arrow_up:

jax_md/nn.py
96.00% <ø> (ø)

jax_md/partition.py
87.19% <75.00%> (ø)

jax_md/energy.py
92.04% <80.00%> (ø)

jax_md/interpolate.py
78.37% <100.00%> (ø)

jax_md/simulate.py
99.01% <100.00%> (ø)

jax_md/smap.py
89.16% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1f0e37...b58467d. Read the comment docs.


bump dependencies

I see the tests are failing

I see the tests are failing

yes, sorting it out

yes, sorting it out

fixed


JDK8/HotSpot/arm32 builds not completing successfully

Thank you for reporting a broken build! Please fill in the following details:

Jenkins link to the failing log: https://ci.adoptopenjdk.net/view/Failing Builds/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-linux-arm-hotspot/
Which platforms (OS+arch) are affected: Linux/arm32
If you look at the historic logs, does the problem appear to be specific to one machine? 489 and 492 were on different Scaleway build boxes and both exhibit the problem
Link to any upstream project bug report (e.g. OpenJ9/JBS) Initally recorded at https://github.com/AdoptOpenJDK/openjdk-build/issues/1450#issuecomment-731107791

SInce JDK8/aarch32 is build from a separate repository it may be a tagging issue in there. Since it has only just shown up it is unlikely to be a result of the changes that we put in to resolve issues related to the <IP_ADDRESS>+1 release last week however https://github.com/AdoptOpenJDK/openjdk-build/pull/2234 was merged three days ago (FYA @andrew-m-leonard - although I think https://ci.adoptopenjdk.net/view/Failing Builds/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-linux-arm-hotspot/488/consoleFull went through with your latest change included so it superficially looks unlikely that's affected it)
12:57:53  WARNING: Failed to identify latest tag in the repository
12:57:54  From /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-linux-arm-hotspot/workspace/./build//src
12:57:54   * branch                  HEAD       -> FETCH_HEAD
12:57:54  WARNING: Failed to identify latest tag in the repository
12:57:54  OpenJDK JDK path will be . JRE path will be -jre
12:57:54  Empty dir passed to be archived

It looks like the build is being repeated !!

ah getOpenJDKversion() gets called several times in build.sh

Fixed by https://github.com/AdoptOpenJDK/openjdk-build/pull/2252

Green: https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-linux-arm-hotspot/500/

Thanks @andrew-m-leonard :-D


doc: update docs for register collection changed callback. Fixes 42

Missing update in docs to reflect the changes in cubeb_register_device_collection_changed.

Thanks!


New version: QuantEcon v0.16.2

Registering package: QuantEcon
Repository: https://github.com/QuantEcon/QuantEcon.jl
Created by: @sglyon
Version: v0.16.2
Commit: 30fcd74abb6bbe761fd046591b90755016ef213b
Reviewed by: @sglyon
Reference: https://github.com/QuantEcon/QuantEcon.jl/commit/30fcd74abb6bbe761fd046591b90755016ef213b#commitcomment-34951726

TagBot tag


paper issue

Could you please share the paper of this code? It doesn't seem like the paper in dcase 2022 Task 3 which your team provides.

Since our paper was rejected at the ICASSP 2023, now we revise the manuscripts and implementations.
So, we will upload the paper after completing the revision.
Well, our team participated in DCASE 2022(Kim_KU entry), but the technical report(ConvNeXt ...) is NOT about this project.
This year, we also participate in DCASE 2023 task 3 based on this project. Therefore, the upgraded implementation and technical report will be publicly opened when the challenge ends.
In summary, we will provide a new GitHub project and paper after DCASE 2023 challenge.
Thank you for your interest in this project.

thank you very much


GatewayResourceCacheExpiry time or DefaultCacheTimeout is not working for resource cache as expected.

Description:
We can configure the Gateway cache expiry time with the GatewayResourceCacheExpiry parameter in the api-manager.xml file. If this parameter is not set, then the DefaultCacheTimeout value has to be applied as the cache timeout period. However, it seems both of these parameters are not working properly with the resource cache.
This issue can be observed while evaluating CacheProvider.getResourceCache() method in the APIAuthenticationService class. The resource cache persists even after with the GatewayResourceCacheExpiry time and DefaultCacheTimeout period.
Affected Product Version:
APIM 2.6.0

When we conduct tests with running product we couldn't observe this behaviour and the resource cache expired after the timeout period as expected. When we test holding the debugger, we can observe that at some debug points the resource cache didn't cleared. but after debugging further it's getting cleared and updated.  Since this is not reproduced in the running product, we are not currently addressing the issue with a fix.  If this issue occurred at the running product we'll look into this again.


6.5.2

Thought I might make a small contribution in keeping this repo upto date!
Always thankful for your great work :)


Fixes Clang Error

This PR fixes #265. It's r2g.

This fixes the problem, but I noticed that one of the unit tests fail. See issue #246.


Provide a template-engine based way of rendering pandas data objects

to_string(), to_html(), to_latex(), formatters, float_format, etc.
related #459

The existing methods are not as flexible as users sometimes wish,
see #167, #2502, https://github.com/pydata/pandas/pull/2919#issuecomment-14067605, #2942, #2924,#3195, #3196,#3264, #3467, #3197,#4578,#4315, ml1., #6052
Pandas' core competence is it's data structures and data ops DSL.
the output formats were added for convenience, but don't see much
ongoing work (formatting tables isn't sexy).
Users roll their own using custom code, there's no uniform way to do it
in a way that naturally extends pandas.

Template engines [jinja2, mako] are the accepted way of converting data into documents,
and the web dev community has provided great libraries to do it.
We should piggyback.
This would be good:

Give users the power to scratch their own itch, avoiding the need for pandas devs
do it, or ignoring "fringe" use cases.
Provides users with a readable baseline , which is easy to modify and adapt to their needs.
Having a standard way to do it (by customizing a template), users can more easily
have their output blend into python, and they can contribute / manage / source-control
document templates in a clean, self-contained way.
Escape the "%d" not "{:d}" silliness and format strings in general, which is a relic of
olden 2.5 support days. No more parameter hell necessary to make the existing methods
more expressive.  I never ever want to format html tables using function arguments.
Use a better way to express formatting logic via a DSL, rather then piecing together
python strings.
Unify the handling of textual representations, not different code, only different template files.
Can refactor this out of core (but not really,  see below)
Can provide features such as conditional formatting, styling.

However,

Can't really remove existing methods.
Adding a template engine dependency to pandas is clearly feature-creep (IMO).

So,

Make it a standalone project out of core (or optional dependency?)
Translate the existing styles into template form.
Provide hooks in pandas to allow users to optionally install the
library and use it for rendeing
Let users know it's available for their use.

Looks like this hasn't progressed in more than a year now. Is the unanimous view still to only accept a formatter architecture that fits all use cases? Or would a step change be acceptable?
I think it would already be a huge improvement if the to_<format> methods would allow overriding their formatter as an optional argument. I mean not the column formatters (they can stay as is), but e.g. to_html accepts a formatter which is then used instead of the default HTMLFormatter so users can subclass HTMLFormatter and pass it in.

@kynan this will be closed by #10250 shortly

@jreback I'm not so sure if this issue has the same things in mind as the new style one. While the styling issue is in #10520, a proper template based rendering infrastructure for more than HTML would be really nice (e.g. to implement a to_markdown() method https://github.com/pydata/pandas/issues/11052).

Great to hear this is nearing completion! Might be worth adding a reference to #10250 in the opening post for those digging through issues around output formatting (I don't have the required permissions).


support iphone x layout

This adds support for the iPhone X layout

[x] adds new routetag 'root' for the root container
[x] change root nav to pad correctly
[x] change profile padding (the only component that uses underStatusbar tag)
[x] pad tabbar

To test you likey want to get xcode beta 11.1

@keybase/react-hackers

Unclear. This is used in one place really so /shrug
On Thu, Sep 28, 2017 at 4:44 PM Marco Munizaga<EMAIL_ADDRESS>wrote:

@MarcoPolo approved this pull request.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/keybase/client/pull/8699#pullrequestreview-65993789,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfhUwbTYQMWnsbcWoH_USUlGX-Rkdhjks5snAUlgaJpZM4PnhOs
.


log.tqdm does not work when breaking

I added some documentation that says it shouldn't be used together with breaking


503 errors when scaling down, or rolling out a new application version

Describe the bug
Hey,
We are noticing blips in services under load during kubernetes rollouts.  We observe a handful of 503 errors from istio-proxy on the pod being removed (either because of a rollout, or a scale down).  This screenshot is from three separate "scale downs":

When scaling down, this is the sequence of events we observe:

pod goes into TERMINATING state and is removed from kubernetes endpoints
A handful of the last requests to the pod are reported by istio-proxy as 503
those requests are also logged in the upstream calling service as 503
application exits
istio-proxy exits

As you can see here:

At the moment, our only saving grace is that we have configured a retry policy which means our end users experience a bit of a slow request, but not a failure - however relying on a retry mechanism in this scenario doesn't feel right.
Expected behavior
The isito-proxy on the application being scaled down should not receive any requests after it has entered a TERMINATING state.
Steps to reproduce the bug
As above, but I can get on hangouts and show you this in detail.
The application itself gracefully handles sigterms and drains and have confirmed this with load tests without istio-proxy in play.  I have also added a preStop hook to the application with istio, to ensure the app doesn't receive a SIGTERM until well after istio-proxy shuts down.
Version
gke 1.10.5, istio 1.0
Is Istio Auth enabled or not?
Yes
Environment
GKE

@mandarjog @PiotrSikora it sounds like we need Envoy to properly enter a lameduck mode upon receiving SIGTERM. Is this something that Envoy currently supports? A quick look at the admin interface didn't reveal anything obvious.

I agree with @PiotrSikora - from my observations - envoy and the app drain fine when they get a SIGTERM however envoy is still getting routed requests from other envoys after it's entered a "terminating" state.
Is there anyone actively looking at this work by the way?  This is a real stickler for us as we fundamentally can't move any high traffic applications and use a kubernetes rolling deployment.  We would have to hand roll some traffic switching to mitigate these issues and I'd much rather not do that.
I can actively demonstrate this on our pre-prod cluster and can dedicate as much time as needed to helping whoever can work on this problem, or even give them access to our test cluster.

@mandarjog IIRC you were looking into this ... can you confirm or re-assign as appropriate?

I checked the kube client code as well as the docs. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-and-container-status
There is no way for pilot to know whether a pod is being terminated, as that status doesn't exist in the API. If someone finds it, please let me know. Otherwise, retry seems like the best solution so far

@rshriram I don't think you need to know if the other container is being terminated.  That's not really what I was trying to get across in the original issue.
Think about the kubernetes lifecycle, when a pod is terminated, both containers, eg the app and istio-pilot receive a SIGTERM, the pod is also marked as TERMINATING, at which point it is removed from the kubernetes endpoint list.
Both the app, and istio-proxy handle the SIGTERM asynchronously, and drain accordingly, however observe upstream istio-proxys on other applications still attempting to send traffic to the now draining container that has been SIGTERMd.
At the point if termination; the pod should be removed from istios registry, and any other istio-pilots trying to talk to it should send their traffic elsewhere.

OK, apologies - I think I miss read your comment.  In relation to pilot not knowing a pod is being terminated, wouldn't it get that event from the endpoint change?
As I mentioned before, when a pod is being terminated it is removed from the endpoints.

I'm having the same issue.
Is there any workaround?

@diegomarangoni it sucks :-( but it is improved in istio 1.0.1
You can see these green lines as our deployments here and the blips are quite minor:

We then use a RetryRule on the services:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: sauron-web
  namespace: sauron-web
spec:
  hosts:
  - app
  http:
  - appendHeaders:
      x-envoy-upstream-rq-timeout-ms: "0"
    retries:
      attempts: 3
      perTryTimeout: 10s
    route:
    - destination:
        host: app
      weight: 99
    timeout: 30s

Which further smoothes it.
But yes, I'd much rather we had a proper solution to this :-(

A chosen endpoint not being available can happen on the data path at any time. The sending proxy knows there is a problem when it receives conn reset, or 503 from upstream.  The datapath must be able to handle the case where it has slightly stale endpoints. Pilot cannot guarantee removal of endpoints in a timely manner. This is because 1. The event may be delivered late to pilot
2. Pilot may debounce many events
3. There may be many sidecars
We can tighten this interval but cannot eliminate it. I think retry is the right solution,
Even after we implement quiesce.

@Stono your workaround is working, increased the latency as expected, but working.
At least no more 503s, I am running in production :)

Glad it helped @diegomarangoni.  @mandarjog I largely agree with you, but we have to be very careful not to rely/fallback to "just do retries".   As retries can very quickly cascade (in a large microservice architecture like ours), and cause other problems.  They can also mask other issues.
I'm not sure how best to approach it, and I agree that 0 503's is probably unattainable, but there should be some sort of measured target/SLO that is tested as part of the istio release lifecycle, or we risk more and more sneaking in.

@rshriram just a question related to this - does istio proxy use draining during rolling updates? If it uses, is it via /healthcheck/fail endpoint supported by envoy? If it does not, how does it take care of inflight requests that might require egress calls? Sorry if it has already been explained and I missed that.

@rshriram when you get time, can you please help with this?

@Stono Can't the circuit-breaking feature be used in this issue?

I've set up a simple POC to reproduce this bug using minikube.

@thspinto maybe combined with retries.  Circuit breaker would only kick in after a few fails

@ramaraochavali sorry I missed your comment.
We don't do any draining, because we don't use envoy's active healthchecks. May be thats part of the problem, but K8S users seem to want the platform's passive health checks (readiness/liveness probes). So we/pilot is at the mercy of those checks - and the statuses that they report to kube API server. In terms of the lifecycle, the two containers/processes are just like anything else in k8s. Both get sigterm. We have the default envoy behavior when sigterm is received. We could trap sigterm and convert it to /healthcheck/fail but as I stated above, its pointless as we don't use envoy's active health checks at all.
As @nmittler mentioned, we need a way for Envoy to immediately enter into a lameduck mode where it stops accepting inbound connections. We also need a more principled approach to container termination (terminate envoy and then the app - per nathan's K8S feature request for dependency driven container start in a pod).

@rshriram how about using POD's Delete hook where you could run a command/binary or a better option a finalizer. With finalizer POD does not get removed until finalizer removes itself form the list. So the controller which implements a finalizer could track the actual app state and once everything is settled to let k8s cluster to remove pod.

That is a nice idea. A bit more machinery but allows for a cleaner way to remove pods.

@rshriram if you already tracking POD's with envoy in some controller, then it will be really easy to add,  otherwise it will require a new controller tracking all envoy enabled pods.

@rshriram No problem. Thanks. We are in the same situation where we do not use active health checks and hence can not use the draining feature.  During k8s rolling upgrades we see spikes in 503s. We are experimenting to see if combination of retry/outlier detection can help reduce the 503s. Probably Envoy should have the same behaviour on SIGTERM or some other equivalent of /healthcheck/fail for those who do not use active health checks?

Yes sure. retries for the specific reason you mentioned may not fully  help here. Did you get any positive results with retries? How much improvement did you see in your tests?

I beg you to find a solution haha.  The idea of sending back a x-envoy-shutting-down and then making it retry to another envoy seems to make sense

It only works for http :(

Selfishly that's all I care about :D haha.  I would actually suspect it is
the majority of use cases too?
My gut is there are going to have to be protocol specific ways of dealing
with things, probably unrealistic to be able to solve all issues in a
procotol agnostic way.
On Tue, 18 Sep 2018, 4:57 pm Shriram Rajagopalan<EMAIL_ADDRESS>wrote:

It only works for http :(
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-422448522, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABavicH92DN811rdjtpfNUbEXLDdpoziks5ucRfagaJpZM4Vwg1y
.

@ramaraochavali It worked at first, but some services where which I do file uploading/processing, where the request can take from 1s or 10m or random, it does not work because the istio retry timeout is fixed.

We are facing the same problem as well - whenever there is a pod restart, there is a spike in 503 responses on our gateway. Our gateway communicates with all the services using gRPC, and the retry solution above also doesn't seem to work.
@rshriram Does HTTPRetry work with gRPC services?

Facing the same problem like @ayushpateria for grpc...

I used these headers in my virtual service to avoid retry on legit 5xx:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: test
  namespace: test
spec:
  http:
  - appendHeaders:
      x-envoy-max-retries: "10"
      x-envoy-retry-on: connect-failure
      x-envoy-upstream-rq-timeout-ms: "0"
    route:
    - destination:
        host: some-test

Still only helps with http requests though.

Probably linked: https://github.com/istio/istio/issues/9113
@thspinto  - that didn't fix it for me :'(

I've uploaded a recording showing just how bad this is, under 20 or so requests a second, during a rollout we got around a 10% error rate:
https://asciinema.org/a/bVrgP5eL80R7qU2uSco3bOa75
Our VirtualService:
  spec:
    hosts:
    - app
    http:
    - appendHeaders:
        x-envoy-max-retries: "10"
        x-envoy-retry-on: connect-failure
        x-envoy-upstream-rq-timeout-ms: "0"
      route:
      - destination:
          host: app
        weight: 100
      timeout: 10s

In our case it was  mainly due to
--drainDuration 45s  --parentShutdownDuration   1m0s  --connectTimeout 10s parameters passed to istio-proxy.
We run istio 1.0.1, and the above values are the default ones generated by helm template.
I think istio 0.8 uses these default values:

connectTimeout: 1s
drainDuration: 2s
parentShutdownDuration: 3s

I used some values more suitable for us and the 503s dramatically reduced in number.
Then using appendHeaders in the virtual service as suggested by @thspinto almost removed them completely, it's more manageable now.

A relevant upstream issue: https://github.com/envoyproxy/envoy/issues/3837

How about having a kubernetes webhook that signals pilot when a pod IP is removed from a kubernetes service endpoints list, so it should not be used in load balancing anymore? Then pilot pushes updates to envoy pods.

@yoz2326 that doesn't help for in-flight requests though. The proxy shouldn't just drop these connections to the floor.

Doing a bit more reading, envoy does have something like a hot-reloading script, which also supports SIGTERM based exit.
https://github.com/istio/istio/pull/6734 was a PR that kinda attempted to replicate that in the pilot-agent. I'm happy to revive that PR and push it to completion if that's deemed useful. My current view of the situation:
The pilot agent would send SIGTERM to the envoy process and wait for it to be shut down before shutting down itself.

@markusthoemmes it already captures sigterms and drains, I believe the 503s however are caused by upstream istio-proxies not knowing that the downstream proxy has been drained, because it takes  to push the updates to all upstream proxies.
Eg:

user talks to a

a talks to b

b sends response, all good

user talks to a

b sigterms, and exits

a tries to talk to b because it hasn't had an update from pilot yet that b has gone

a 503s

pilot detects b has gone and pushes update to a

user talks to a

a talks to c instead as b has gone

@yoz2326 that doesn't help for in-flight requests though. The proxy shouldn't just drop these connections to the floor.

Wouldn't pod terminationGracePeriodSeconds setting allow for in flight connections to finish?

@yoz2326 that's set to 30 by default IIRC, so requests from 0 to ~30s should be fine even with default behavior I think.

@yoz2326 that doesn't help for in-flight requests though. The proxy shouldn't just drop these connections to the floor.

Wouldn't pod terminationGracePeriodSeconds setting allow for in flight connections to finish?

Nope.... doesn't seem to help.

@Stono what settings do you use for drainDuration, ``parentShutdownDurationanddiscoveryRefreshDelay`  on istio-proxy?

@Stono are you certain that the pilot-agent does drain already? It does seem to drain properly on configuration updates etc. Regarding shutdown logic, it looks more like a hard shutdown to me.

nginx-ingress has a similar problem. It might update endpoints a few seconds later than the start of termination. In order for the pod to respond, it is advised to use sleep 15 in lifecycle.preStop.
However as @Stono reported, it does not work for the istio-proxy sidecar. When using sleep 15, I find this in the upstream istio-proxy, confirming that the 503 NR happens right after start of pod termination, not when the SIGTERM kicks in.
200 OK requests...
[2018-11-20T12:34:47.642Z] "GET /ping HTTP/1.1" 503 NR
503 NR NOT OK for ~1 second
[2018-11-20 12:35:01.785][18][warning][main] external/envoy/source/server/server.cc:373] caught SIGTERM
[2018-11-20 12:35:01.785][18][info][main] external/envoy/source/server/server.cc:436] main dispatch loop exited

So to me it seems the issue is not about the downstream envoy not having updated the endpoints, but that upstream envoy does not pass the traffic to service during the sleep period for some reason.

Hmm interesting @mikksoone.  We've tried so many combinations of sleeps etc in the downstreams but got nowhere with this, so very frustrating!

So I've been working on another issue today and had a bit of a thought process around these 503s.
Take a scenario on a larger cluster like mine where it can take up to 10seconds for pilot to push config to all the proxies:
2018-11-26T22:56:36.877043Z     info    ads     Push finished: 10.120342411s {
    "Start": "2018-11-26T22:56:26.756693989Z",
    "End": "2018-11-26T22:56:36.876978375Z"
}

So; take an app that is up and running.  You come along and you do a rolling deployment of the application, which starts a new endpoint.
That endpoint comes up, as soon as it becomes "ready" from a kubernetes perspective:

Pilot detects it, waits PILOT_CACHE_SQUASH (couple of seconds), then starts pushing that endpoint our via EDS to all other proxies
However, kubernetes has already triggered the previous version to TERMINATE.

This can result in a small window where upstream proxies have a config which points to terminated pods.
Ways to work around this that I can think of are:

Ensure that all pods have a minReadySeconds setting that is greater than the possible maximum duration of istio pilot push
Ensure that pods (both application and istio-proxy) have a preStop hook which effectively sleeps for a duration that is greater than the maximum duration of istio pilot push

Thoughts?
CC. @costinm @rshriram @mandarjog

Thoughts @costinm @rshriram @mandarjog?  We need a plan for 1.1.  If there is work to be done other than graph pruning we need to capture it here and get someone on it :)

When a  pod is terminating, it is removed from endpoint and a push will be triggered to remove it from outbound clusters. So as soon as a proxy applies this update, it will not ship traffic to the not existing cluster. But as @mandarjog said https://github.com/istio/istio/issues/7665#issuecomment-416634476 There is a delay, which can not be eliminated.
But IMO, if sidecar has a graceful termination period, like default 30s, after which it quit, instead of currently quit immediately when receive a SIGTERM.  By this, this can be mitigated.
The grace termination period can help do:

let the in-flight request done

mitigate the configuration delay, new request can still come until all sidecars configuration take effect.

Quick question: couldn't the Gateway just ignore a 503 provided it has other routes to try out?

I think yes, if HTTP retry is configured.  But we have retry limit, like timeout.

Retries aren't fully effective and tbh are just a plaster on the problem of
stale cluster config
On Thu, 29 Nov 2018, 9:10 am Zhonghu Xu<EMAIL_ADDRESS>wrote:

couldn't the Gateway just ignore a 503 provided it has other routes to try
out?
I think yes, if HTTP retry is configured. But we have retry limit, like
timeout.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-442759297, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABavidr5kDFJC3cKfDwgDonIDf13NKSLks5uz6SQgaJpZM4Vwg1y
.

@hzxuzhonghu Then I'm afraid the issue is whether the retry guarantees idempotency.

@Stono Weird, because in my case I have one cluster with 1.0.3 where I was facing this issue, and then another with 1.0.4 where it's working properly. But here I'm strictly speaking about #9480.

Perhaps x-envoy-retry-on: gateway-error would help here?

We use:
    http:
    - appendHeaders:
        x-envoy-max-retries: "10"
        x-envoy-retry-on: gateway-error,connect-failure,refused-stream

More feedback on this matter...
After a few days suffering from more consistent 503s, today I found out that it was due to the istio-pilot not being able to cope (apparently) with the load of change in my cluster (although we're only talking here about ~350 pods).
I could notice that when starting up new pods, the clusters configuration in their sidecar proxy was only having the static routes, but not the dynamic (so to say) ones; in other words, the sidecar proxy was not receiving the CDS information from the istio-pilot somehow.   It's worth noting HPA basically does nothing and we have also disabled it in favour of fixed instances.
See https://github.com/istio/istio/issues/7878 for more details but at a high level, HPA scales up a new pilot but nothing goes to it anyway as everything has created long lived connections to the overloaded pilots.  It's only when a pod restarts that it can go to the new pilot, and even then it's round robin, not least conn.
Right now, to keep things happy we run this cronjob nightly:
    #!/bin/bash
    set -e
    if kubectl --namespace=istio-system get svc istio-pilot &>/dev/null; then
      kubectl --namespace=istio-system patch svc istio-pilot -p "{\"spec\":{\"selector\": {\"istio\": \"pilot-deploying\"}}}"
      sleep 3
    fi

    # Restart pilot
    kubectl -n istio-system patch deployment istio-pilot -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"RESTARTED_ON\":\"`date +'%s'`\"}}}}}"
    kubectl -n istio-system rollout status deployment/istio-pilot

    # Patch it back so things can connect to pilot
    if kubectl --namespace=istio-system get svc istio-pilot -o yaml | grep "pilot-deploying" &>/dev/null; then
      sleep 3
      kubectl --namespace=istio-system patch svc istio-pilot -p "{\"spec\":{\"selector\": {\"istio\": \"pilot\"}}}"
    fi

Which loosely rebalances all pods across our available pilots.

@Stono I see the point in your nightly restart of istio-pilots; however, it still does not prevent the regular restart of pods during the day to also be impacted, right?
In our case, this is a development cluster which is constantly creating/destroying pods continuously.

Yeah I'm finding that a day's worth of entropy is managable with periodic
rebalancing
On Fri, 7 Dec 2018, 11:46 am Enrique Medina Montenegro <
<EMAIL_ADDRESS>wrote:

@Stono https://github.com/Stono I see the point in your nightly restart
of istio-pilots; however, it still does not prevent the regular restart
of pods during the day to also be impacted, right?
In our case, this is a development cluster which is constantly
creating/destroying pods continuously.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-445208682, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABavibKksdJOY6ID0CJJMvtzTsaPXa5qks5u2lT_gaJpZM4Vwg1y
.

What really surprises me is the fact that Istio has been announced recently as production-ready, but this seems like a very sensitive issue yet.

IIUC, there are a couple of issues that are being discussed here:

When an endpoint in a cluster goes down, Envoy doesn't appear to retry properly, leading to 503s being seen by the client.
High pod churn seems to be exposing a Pilot issue where new endpoints are not propagated, requiring a reboot of Pilot to resolve the issue.

For 1, I believe I'm able to replicate this running locally.  In my experiment, I had 3 Envoys: 1 frontend that has endpoints for 2 backends , each of which talks to a simple application that just returns 200. When I bring 1 backend down, requests going to the frontend alternate between 200 and 503 (using round-robin). Even if the requests use the header x-envoy-retry-on: gateway-error, I still see the alternating (possibly an Envoy bug?). However, if I set the retry policy directly on the cluster, I get all 200s as expected.  This is something we can certainly look at doing for 1.1.
Number 2 sounds more like an ADS bug and falls more into @costinm's domain. Can you take a look?

seems to be high priority for 1.1, and if the change is small I don't mind having it in 1.0.6 given the release expectations of 1.1

For 2 - I can take a look, we had a bug problem in 1.0.3 which is fixed in 1.0.4. Not sure which logs are associated with this case.
There is also 3 - which Josh is handling - if Mixer policy is enabled and in 'fail close' mode, an upgrade of mixer policy can cause 503 for any current request. He will add a retry for the Check call, to hit the new mixer instance.

@emedina  Can you open a separate bug on pilot ADS not coping and failing to push, and add more logs/info if you have them ?

@costinm I just opened #10360 as you requested.
Not sure if related to #8554 as well, though.

I could reproduce again the issue of the pilot not properly handling the restart of one of my pods and still keeping the old IP by using the bookinfo application.
# kubectl get all -owide
NAME                                  READY   STATUS            RESTARTS   AGE   IP               NODE           NOMINATED NODE
pod/curl-74f57bfd66-7wlhs             1/1     Running           0          4d    <IP_ADDRESS>    orhlp-dck015   <none>
pod/details-v1-6764bbc7f7-xc5cc       2/2     Running           0          2d    <IP_ADDRESS>    orhlp-dck015   <none>
pod/nfs-busybox-7fd4f7fdd8-vcrlq      1/1     Running           0          19d   <IP_ADDRESS>    orhlp-dck015   <none>
pod/productpage-v1-54b8b9f55-vmhfm    2/2     Terminating       0          2d    <IP_ADDRESS>   orhlp-dck011   <none>
pod/productpage-v1-77758fb579-srwqj   0/2     PodInitializing   0          23s   <IP_ADDRESS>   ochlp-dck018   <none>
pod/ratings-v1-7bc85949-28q5f         2/2     Running           0          2d    <IP_ADDRESS>     orhlp-dck015   <none>
pod/reviews-v1-fdbf674bb-kfmw4        2/2     Running           0          2d    <IP_ADDRESS>    orhlp-dck015   <none>
pod/reviews-v2-5bdc5877d6-967bp       2/2     Running           0          2d    <IP_ADDRESS>   ochlp-dck018   <none>
pod/reviews-v3-dd846cc78-qhpgj        2/2     Running           0          2h    <IP_ADDRESS>    orhlp-dck017   <none>

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/details       ClusterIP   <IP_ADDRESS>   <none>        9080/TCP   2d    app=details
service/kubernetes    ClusterIP   <IP_ADDRESS>       <none>        443/TCP    21d   <none>
service/productpage   ClusterIP   <IP_ADDRESS>   <none>        9080/TCP   2d    app=productpage
service/ratings       ClusterIP   <IP_ADDRESS>   <none>        9080/TCP   2d    app=ratings
service/reviews       ClusterIP   <IP_ADDRESS>     <none>        9080/TCP   2d    app=reviews

Notice above that the old pod (productpage-v1-54b8b9f55-vmhfm) had an IP <IP_ADDRESS> and the new one (productpage-v1-77758fb579-srwqj) is <IP_ADDRESS>. However, when I query the clusters endpoint from the sidecar proxy inside the new pod:
# kubectl exec -ti productpage-v1-77758fb579-srwqj -c istio-proxy -- curl localhost:15000/clusters|grep productpage|grep outbound

outbound|9080||productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::added_via_api::true
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::added_via_api::true
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1

As you can see, the old IP <IP_ADDRESS> is still there, therefore resulting in getting one 503 every two invocations to the productpage.
As usual, just patching either the service or pilot itself does the trick:
# kubectl edit svc productpage <--- Simply added a label to force a refresh of the service.
service/productpage edited

# kubectl exec -ti productpage-v1-77758fb579-srwqj -c istio-proxy -- curl localhost:15000/clusters|grep productpage

outbound|9080||productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::added_via_api::true
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::added_via_api::true
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1

Also double-checked the RBAC permissions for the pilot just in case, but they allow it to be notified of changes in either pods, endpoints or services:
- apiGroups: [""]
  resources: ["endpoints", "pods", "services"]
  verbs: ["get", "list", "watch"]

To me, this is definitively not production-ready, and I would like to get in touch with people or organizations that claim to be using this in production to understand why it doesn't work for me with the simplest example (I've been trying out Istio since v0.1 and never found it ready, unfortunately).

Hey,
What version are you doing that testing on?  What you just described feels
like the 1.0.2 bug which is now fixed?
FYI I lead the infrastructure team at AutoTrader and we use Istio in
production (1.04) with circa 135 services.
Karl
On Sat, 8 Dec 2018, 1:53 pm Enrique Medina Montenegro <
<EMAIL_ADDRESS>wrote:

I could reproduce again the issue of the pilot not properly handling the
restart of one of my pods and still keeping the old IP by using the
bookinfo application.
kubectl get all -owide
NAME                                  READY   STATUS            RESTARTS   AGE   IP               NODE           NOMINATED NODE
pod/curl-74f57bfd66-7wlhs             1/1     Running           0          4d    <IP_ADDRESS>    orhlp-dck015
pod/details-v1-6764bbc7f7-xc5cc       2/2     Running           0          2d    <IP_ADDRESS>    orhlp-dck015
pod/nfs-busybox-7fd4f7fdd8-vcrlq      1/1     Running           0          19d   <IP_ADDRESS>    orhlp-dck015
pod/productpage-v1-54b8b9f55-vmhfm    2/2     Terminating       0          2d    <IP_ADDRESS>   orhlp-dck011
pod/productpage-v1-77758fb579-srwqj   0/2     PodInitializing   0          23s   <IP_ADDRESS>   ochlp-dck018
pod/ratings-v1-7bc85949-28q5f         2/2     Running           0          2d    <IP_ADDRESS>     orhlp-dck015
pod/reviews-v1-fdbf674bb-kfmw4        2/2     Running           0          2d    <IP_ADDRESS>    orhlp-dck015
pod/reviews-v2-5bdc5877d6-967bp       2/2     Running           0          2d    <IP_ADDRESS>   ochlp-dck018
pod/reviews-v3-dd846cc78-qhpgj        2/2     Running           0          2h    <IP_ADDRESS>    orhlp-dck017
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE   SELECTOR
service/details       ClusterIP   <IP_ADDRESS>           9080/TCP   2d    app=details
service/kubernetes    ClusterIP   <IP_ADDRESS>               443/TCP    21d
service/productpage   ClusterIP   <IP_ADDRESS>           9080/TCP   2d    app=productpage
service/ratings       ClusterIP   <IP_ADDRESS>           9080/TCP   2d    app=ratings
service/reviews       ClusterIP   <IP_ADDRESS>             9080/TCP   2d    app=reviews
Notice above that the old pod (productpage-v1-54b8b9f55-vmhfm) had an IP
<IP_ADDRESS> and the new one (productpage-v1-77758fb579-srwqj) is
<IP_ADDRESS>. However, when I query the clusters endpoint from the
sidecar proxy inside the new pod:
kubectl exec -ti productpage-v1-77758fb579-srwqj -c istio-proxy -- curl localhost:15000/clusters|grep productpage|grep outbound
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::added_via_api::true
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::added_via_api::true
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
As you can see, the old IP <IP_ADDRESS> is still there, therefore
resulting in getting one 503 every two invocations to the productpage.
As usual, just patching either the service or pilot itself does the trick:
kubectl edit svc productpage <--- Simply added a label to force a refresh of the service.
service/productpage edited
kubectl exec -ti productpage-v1-77758fb579-srwqj -c istio-proxy -- curl localhost:15000/clusters|grep productpage
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080||productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080||productpage.default.svc.cluster.local::added_via_api::true
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080||productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::default_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_connections::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_requests::1024
outbound|9080|v1|productpage.default.svc.cluster.local::high_priority::max_retries::3
outbound|9080|v1|productpage.default.svc.cluster.local::added_via_api::true
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_connect_fail::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::cx_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_active::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_error::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_success::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_timeout::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::rq_total::0
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::health_flags::healthy
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::weight::1
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::region::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::sub_zone::
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::canary::false
outbound|9080|v1|productpage.default.svc.cluster.local::<IP_ADDRESS>:9080::success_rate::-1
Also double-checked the RBAC permissions for the pilot just in case, but
they allow it to be notified of changes in either pods, endpoints or
services:

apiGroups: [""]
resources: ["endpoints", "pods", "services"]
verbs: ["get", "list", "watch"]

To me, this is definitively not production-ready, and I would like to get
in touch with people or organizations that claim to be using this in
production to understand why it doesn't work for me with the simplest
example (I've been trying out Istio since v0.1 and never found it ready,
unfortunately).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-445460885, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABavibKZSGuNiRkJjRtCqEIF7JOQcyMmks5u28RQgaJpZM4Vwg1y
.

Are you using Istio 1.0.4 ? And you didn't use the istioctl proxy-status command (which is known to expose a bug - fixed in the upcoming 1.0.5) ?
I understand your concern - but 'production-ready' doesn't always mean bug free (or we would never run anything in production). For almost all prod launches I've worked on, some bugs were found and it takes
few iterations to get rid of them. What is important to me is to prioritize prod-related bugs and find fixes or workaround quickly.
One problem we have with Istio is that we rely a lot on automated testing - plus some semi-automated stability and perf testing. Most Istio developers are not 'on-call' for a large production system. @Stono
has been a critical help - but we need more people who can run pre-release builds on a realistic staging environment and provide actionable data.
What works very well in Google is running all releases on staging and canary - and in some cases long-running 'experiments' and 'dogfood' - before cutting the official prod. Unfortunately we don't have that yet, there are few efforts in progress to fix it.

Keep in mind, many of us experience much higher error rates than 0.004%
with no explanation as to the cause. I'm still hopeful it'll get worked
out, but my simple tests (basically scaling up bookinfo) produce a large
numbers of 503s -- and other users have confirmed the results. It works for
clusters of a certain size but once you hit some thresholds it sorta' falls
apart, so I definitely understand the skepticism around production-ready.
On Sun, Dec 9, 2018 at 5:26 AM Enrique Medina Montenegro <
<EMAIL_ADDRESS>wrote:

@costinm https://github.com/costinm Yes, I'm not expecting a 0 bugs
software, of course, just expressing my frustration about no having yet
been able to run Istio in my PROD due to the issues already discussed in
this thread due to the serious impact they would have had in my specific
use cases. I've been betting for Istio since the very beginning (find my
name in Google Groups, for reference) and I'm a strong believer that this
is the definitive service mesh we need, but somehow I always run into a
blocking issue once I try to deploy in PROD. But be sure I'll keep on
trying. And let me know if I can help in testing here.
@Stono https://github.com/Stono Thanks so much for throwing in such a
proof of trust here, so I have better evidence than self practice to keep
pushing about Istio in my organization (yes, I also have bosses who want
clear evidence about the reliability of any tool I deploy iin our PROD
cluster ;-))
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-445525810, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALnFDHKVyrMUEq1TObytxwpFNIpClUpeks5u3OVLgaJpZM4Vwg1y
.

@jaygorrell it would be great if you could update this thread with logs and findings please, to help the istio folk debug.  As another user I'm also interested to see your findings

All of my findings as I've been going along are documented in
https://github.com/istio/istio/issues/9043
On Sun, Dec 9, 2018 at 9:41 AM Karl Stoney<EMAIL_ADDRESS>wrote:

@jaygorrell https://github.com/jaygorrell it would be great if you
could update this thread with logs and findings please, to help the istio
folk debug. As another user I'm also interested to see your findings
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-445543396, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALnFDGY6LRgPB_IWg5e-6-3I9ifPUyMcks5u3SEkgaJpZM4Vwg1y
.

Brill, ta.  I will have a read
On Sun, 9 Dec 2018, 2:43 pm Jay Gorrell<EMAIL_ADDRESS>wrote:

All of my findings as I've been going along are documented in
https://github.com/istio/istio/issues/9043
On Sun, Dec 9, 2018 at 9:41 AM Karl Stoney<EMAIL_ADDRESS>wrote:

@jaygorrell https://github.com/jaygorrell it would be great if you
could update this thread with logs and findings please, to help the istio
folk debug. As another user I'm also interested to see your findings
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-445543396, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/ALnFDGY6LRgPB_IWg5e-6-3I9ifPUyMcks5u3SEkgaJpZM4Vwg1y
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7665#issuecomment-445543513, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABaviWLCv1BI4T8zL8_h8UyUsKKshYIjks5u3SGGgaJpZM4Vwg1y
.

https://github.com/istio/istio/pull/10420 may decrease 503s in some degree

@nmittler can this be closed?

I think so

Any reason Istio doesn't plan to incorporate Envoy active health checks? I don't think there's any reason to keep an unhealthy endpoint in the load balancing pool.

For anyone arriving here with similar issues in 1.1.x, I've opened a new issue: https://github.com/istio/istio/issues/13616

Hi...We are using Istio 1.18.5 and facing 503 error during scale down. Because of this we have currently fixed number of pods deployed based on peak load to avoid 503's. Can anyone help me fix this issue.


AQAvit Meeting March 10, 2021

AQAvit Meeting March 10, 2021
Wednesday, March 10 11:30am – 12:00pm EST (+5hrs for Hursley/London)
Description: Bi-weekly meeting to discuss AdoptOpenJDK Quality Assurance (AQA) topics.
Join via https://bluejeans.com/724807178/2833?src=calendarLink
Add the recurring bi-weekly AQAvit meeting to your calendar.
Proposed Agenda:

Auto-generating test jobs from build pipelines - @sophia-guo
jdk16 release plan - @lumpfish
AQAvit repos update (see https://github.com/AdoptOpenJDK/TSC/issues/205) - @smlambert

plan to silently swap out the AdoptOpenJDK repos for the Eclipse repos in Jenkins once they are live (and during jdk16 code freeze)

Outreachy update (details on project proposals, mentors, timelines and co-ordination) - @smlambert
AOB

Minutes:

Sophia gives a quick overview of https://github.com/AdoptOpenJDK/ci-jenkins-pipelines/pull/75 that autogenerates test jobs if they are not already created (will help especially as we add new versions and do not need to manually run the Test_Auto_Gen_Job ahead of the first time tests get enabled... which is not always communicated clearly, so now no need for improved comms ;) )
Overview of JDK16 release plan outlined here: https://github.com/AdoptOpenJDK/TSC/issues/206 and that we will begin our code freeze now for new feature work, fixes and test exclusions still okay (2 reviewers as always)
During code freeze, we will try to create all of the test repos over in the Adoptium organization and attempt to do the switch over so that we can move to working in and using the repos out of Adoptium versus those in AdoptOpenJDK immediately after the release and when the freeze is lifted.

This swap out will have to be coordinated, not just at our Jenkins server, but at OpenJ9 and with any vendor using the AQA stuff, otherwise they will be working out of the AdoptOpenJDK repos which will be switched to read only and no longer receiving new features and fixes.
Need to make sure all contributors, including CanOSP students get ECAs so they can continue to make contributions.  For current PRs, depending on what is easiest, we may merge them into existing repo and then port to new repo, or just reconstruct it in the new repo (if its small/easy).
We intend to keep all history and issues in this migration, so it will hopefully not be too disruptive.

Outreachy, contribution period will start in a few weeks, so between now and then, team effort to create more "good first issues" and review existing ones (here)
AOB

there is now an Adoptium Community call scheduled bi-weekly on Mondays (find past and present ones here) for broader issues
next AQAvit meeting topics may include TRSS roadmap, other ideas welcome

Link to recording
Link to next meeting details and proposed agenda: https://github.com/AdoptOpenJDK/openjdk-tests/issues/2367
(please add agenda suggestions to that issue)


Feature/nirmal agnihotri/redux sagas structure

Tracking Info
Resolves #
Make sure your branch name conforms to: <feature|staging|bugfix|...>/<username>/<3-4 word description separated by dashes>. Otherwise, please rename your branch and create a new PR.
Changes
What changes did you make?

Added redux saga to store
Created example saga, see https://redux-saga.js.org/docs/introduction/BeginnerTutorial
Moved slices to slice folder

Testing
How did you confirm your changes worked?

Existing features (i.e. Navbar) still work as they do in main and Redux state is maintained

Confirmation of Change
N/A

LGTM, as all existing features still work and the tutorial seems to be followed correctly!


Fix guest additions vagrant up not working on Ubuntu.

See the troubleshooting in the NUbots/NUbots readme for the solution: https://github.com/NUbots/NUbots

Not using Vagrant anymore


add Element.getAttributeNames(): Array

according to MDN Element has a method getAttributeNames which returns and Array of strings.
https://developer.mozilla.org/en-US/docs/Web/API/Element/getAttributeNames
It seems supported quite a ways back in every browser documented

Some of the tests need re-recording. To do so, you need to compile flow, and then do ./runtests.sh -r -b path/to/compiled/flow-bin and ./runtests.sh -rn -b path/to/compiled/flow-bin

@SamChou19815 I updated the PR, seems a task is blocked still though :)


undefined variable in _iter_from_tabulator function

The value variable on line 325 in resource.py is undefined.
I encountered it while running the example code in the README.
import datapackage

dp = datapackage.DataPackage('http://data.okfn.org/data/core/gdp/datapackage.json')
brazil_gdp = [{'Year': int(row['Year']), 'Value': float(row['Value'])}
              for row in dp.resources[0].data if row['Country Code'] == 'BRA']

This throws a NameError for value in the _iter_from_tabulator function. Note: this error is not caught by Travis.
I'm not familiar with jsontableschema and can't quickly see how to fix this. Any help is greatly appreciated as I'd like to address #63.

In addition, the example code will always fail because the dates in the referenced data package are formatted as yyyy which cannot be cast by jsontableschema.types.date.DateType as there is no cast_yyyy() function.
Adding one is easy but I'm not sure if that's the desired solution.

@ndkv Thanks for flagging!  As of the current spec (http://specs.frictionlessdata.io/json-table-schema/#date), that datapackage.json should actually use this for the date:
"pattern": "fmt:%Y"

However, this is not particularly well liked and is currently under discussion for replacement: https://github.com/frictionlessdata/specs/issues/260 (your thoughts welcome!)
We should update the gdp dataset to at least match the current spec, but it might break the old OpenSpending (https://openspending.org/) which relies on this formatting and might be pulling in that dataset.

value is not defined but used here https://github.com/frictionlessdata/datapackage-py/blob/master/datapackage/resource.py#L325

@danfowler Right, thanks for explaining the issue. I have to agree that fmt:%Y doesn't feel right.


Invalidate cache on DefaultLanguage/DefaultPaperSize/ReadyPaperSizes change

Hi
I had some trouble getting my iOS device to show the correct "media type" (A4 instead of US letter) and tweaking DefaultLanguage, DefaultPaperSize or ReadyPaperSizes didn't affect the reported "media type" on iOS. After a lot of trial and error I figured the "media type" is apparently cached in /var/cache/cups, and after emptying that directory the correct "media type" was reported.
This was very confusing, so perhaps changing any of the mentioned values should invalidate the cache? or at least this behavior should be documented (or maybe it already is and I just missed that?).
The printing just works after fixing the "media type", so good work on the AirPrint and Mopria support! :) (https://github.com/OpenPrinting/cups/pull/126)

[master 41fbc18c0] Invalidate PPD cache if cupsd.conf is newer (Issue #371)


Make Request Logging Format Configurable

We want to analyze access logs of the oauth2 proxy in our centralized logging system where we use a slightly different access log format.
This PR makes the request logging format configurable via a template string passed as -request-logging-format. The default format stays the same of course.

Are you interested in this feature? ping @jehiah (sorry to bother you personally, I just found you in the contributor list)


Info

Hi, I see you create jwpub files, how can I learn? Thank you

Hi @tichico
You can start by reading the first few paragraphs in the Readme
Eryk


Ability to ignore column order

In legacy production databases with multiple instances, it's not always feasible to wait for a column reorder operation to execute against a large table, particularly if there's a chance of it being interrupted by a  constraint violation or the like. A flag to ignore column order when diffing/pushing would be very helpful.

Thank you for the feature request!
Just to confirm I understand the use-case: you have multiple shards in one environment (for example using host-wrapper), and some shards have differing column orders for some tables, but aside from that there's no other real drift/differences between shard state?
This could be added in theory, but I'm not sure how common this situation is, if I understand it correctly. I've encountered this situation once before, but a lot of other shard drift was present in that case, so unfortunately a new option just for ignoring column order wouldn't have been sufficient there.
Longer-term, options/commands to ignore drift (e.g. a way to only push the changes represented in a single git commit, ignoring all other preexisting state differences) may be explored, which might solve this as well. But no ETA yet, it's still in the early planning stages.

It's a distributed system across multiple datacenters, with an entirely separate DB server per datacenter but identical service code. We're trying to finally unify the DB schemas (in addition to the ordering differences, some have different column sizes in places) but I've ended up having to manually swap in different versions of the Really Big Table definition before doing a skeema push, because those column reorders simply won't finish.

Thanks, I see. So I'm guessing you're using different environment names, or maybe swapping out the .skeema file, rather than using something like host-wrapper since you can't safely push the same .sql to multiple instances at once.
If your company is a premium subscriber on a tier which includes technical support, please reach out and we can try to determine an ETA for implementing this and discuss some workarounds in the meantime. Can't guarantee it will appear in the Community Edition though, it depends on whether it's beneficial for enough users for that to make sense.

This feature will be present in Skeema v1.11.0: to suppress generation of ALTERs that only reorder columns, you can enable new option lax-column-order. Target release date is in September.

Skeema v1.11.0 is now available, containing the new lax-column-order option. Thanks again for the excellent feature request!


testnet network support - 25h5gpewvkza495 - validator,rpc,api

@sotukker, Your Moniker and validator addresses are required, please update the PR with those details.

@sotukker, Your Moniker and validator addresses are required, please update the PR with those details.

@PrathyushaLakkireddy
My moniker sotukker  address akashvaloper1s2a4y8fn4sv6sl3pcxceeqkzxtud6w7xpnnqc4

@PrathyushaLakkireddy
My moniker sotukker  address akashvaloper1s2a4y8fn4sv6sl3pcxceeqkzxtud6w7xpnnqc4


SMD Dataset - Default settings same as results reported in paper?

Hi authors,
I am trying to replicate the results reported in your paper. I would like to confirm whether the default hyperparameter settings provided in the code are exactly the same as the experiments you reported in your paper?
I think another issue I'm having in replicating similar Pot-F1 scores is the issue mentioned in #17, how do you combine the Pot-F1 from 28 different entities into a single score? I have attached the results I by running the code: omnianom_results.txt.
Thank you!

Hi authors,
I am trying to replicate the results reported in your paper. I would like to confirm whether the default hyperparameter settings provided in the code are exactly the same as the experiments you reported in your paper?
I think another issue I'm having in replicating similar Pot-F1 scores is the issue mentioned in #17, how do you combine the Pot-F1 from 28 different entities into a single score? I have attached the results I by running the code: omnianom_results.txt.
Thank you!

The settings provided in the code are not reported in my paper due to limited space.

Hi authors,
I am trying to replicate the results reported in your paper. I would like to confirm whether the default hyperparameter settings provided in the code are exactly the same as the experiments you reported in your paper?
I think another issue I'm having in replicating similar Pot-F1 scores is the issue mentioned in #17, how do you combine the Pot-F1 from 28 different entities into a single score? I have attached the results I by running the code: omnianom_results.txt.
Thank you!

In our evaluations, we have repeated the experiments many times, and there indeed have high and low result, because the model training and anomaly scores are not very stable for each experiment


Various Improvements, Simplified

Remove unnecessary null suppressions
Support deserialization of indexes with different patch levels
Configure awaits

Pulled the basic changes out of the other PR. Ready for review.

Thanks for the contribution!


summary markdown is being truncated

LUCI summary markdown supports 4332 characters only and the current 200 lines are adding more characters. This is causing the summary markdown to be truncated.

Fixed on this CL: https://flutter-review.googlesource.com/c/recipes/+/12163


store: Tolerate a missing source for a copy

When we check whether we should perform on_sync actions, tolerate the source having been deleted.

Would be great to ensure this is covered by a test. If possible

Would be great to ensure this is covered by a test. If possible

I just added a test, and even made sure it fails on master :)


Add the Aurelia plugin to the README

Nice community plugin -
https://github.com/oribella/aurelia-sortable

oribella/aurelia-sortable — this solution had nothing to do with Sortable.

Sorry I confused the one I linked with this one - https://github.com/eriklieben/aurelia-sortablejs which seems to be deprecated anyway, my bust.


support custom executables

should resolve #9
@Bromeon

Just tested it, and it works!
My project-local .ftw file:
[ftw]
godot-exe=D:/path/to/godot/bin/godot.windows.tools.64.exe

Also pointing to godot.bat does the job.
Maybe worth noting that backslashes in the path on Windows do not work. I think that's OK, should just be documented.


General usage issue and missunderstanding

Hey there,
I'm developing a CS:GO coinflipping website and I have set up the login with steam system and have all of my users in a database.
I'm now working on the part that lets them deposit their skins into the site and they get exchanged for coins relative to the skin's value.
I've been looking around for how to send tradeoffers via JS or PHP and have come across node-steam and it's addons like node-steam-tradeoffers.
I have afew questions about it's usage:

How does node-steam login to the bot's account if the bot has a mobile authenticator on? Which it has to so that it doesn't have to wait 3 days for the trade.
I npm-installed a whole bunch of the modules, I have (steam, steam-crypto, steam-trade, steam-tradeoffers, steam-web-api, steam-web-api-key and steam-weblogon). I've written the following code within the "steam" folder in my "node-modules" folder within my project:

var fs = require('fs');
var Steam = require('steam');
// if we've saved a server list, use it
if (fs.existsSync('servers')) {
Steam.servers = JSON.parse(fs.readFileSync('servers'));
}
var steamClient = new Steam.SteamClient();
var steamUser = new Steam.SteamUser(steamClient);
var steamFriends = new Steam.SteamFriends(steamClient);
steamClient.connect();
steamClient.on('connected', function() {
steamUser.logOn({
account_name: '.',
password: '.'
});
});
steamClient.on('logOnResponse', function(logonResp) {
if (logonResp.eresult == Steam.EResult.OK) {
console.log('Logged in!');
steamFriends.setPersonaState(Steam.EPersonaState.Online); // to display your bot's status as "Online"
steamFriends.setPersonaName('Haruhi'); // to change its nickname
//steamFriends.joinChat('103582791431621417'); // the group's SteamID as a string
}
});
Obviously I have my account username an dpassword within the account_name and password values.
Every time I run the code by cding into the "steam" folder and running "node example.js" I always get the same error output:
Jamess-MacBook-Pro:steam James$ node example.js
events.js:141
throw er; // Unhandled 'error' event
^
Error: Disconnected
at SteamClient._disconnected (/Users/James/Sites/CSFlip/node_modules/steam/lib/steam_client.js:186:24)
at emitOne (events.js:77:13)
at Connection.emit (events.js:169:7)
at TCP._onclose (net.js:469:12)
Jamess-MacBook-Pro:steam James$
This is a really key part of my site and if I can't get past this obstacle I won't be able to complete it, thanks for any help on this issue!

Please wrape you code in code tags, backticks.
You don't write your code in the node_modules folder.
This isn't a helpdesk, however if you move your code out into the proper directory, read the proper documentation, I'm sure you can fix it. :smiley:

As per README, you need to handle the error event.

First, this isn't a help desk. Second, you need to read a Node.js tutorial to understand how modules work in general. Third. you should read the READMEs of modules you use before creating issues - as your stack trace clearly shows, you didn't handle the 'error' event.


[FR] Upate formules

I'd love if there was an option to update outed formulas along with casks.
Right now if when enter brew cu -a -y it updates homebrew, tells me what formulas are out of date, and then updates casks.
What I was actually looking for when I found cu is the ability to just type one command that updates everything homebrew related, in the correct order. cu is a big improvement for updating casks vs vanilla homebrew, but imo it's just missing this one feature that would make it perfect.
So it would be great if there was a way to enable this behavior by default or to add it is an option. Maybe -bf short for brew formulas?

Hey @qcksilver ,
Isn't that something you can simply achieve by using brew upgrade or brew upgrade --greedy (for auto-update & non-versioned casks)?

Yup. But doesn't brew upgrade --greedy pretty much do the same things as brew cu -a -y?  brew cu just prints more information and looks much better (great job btw, I love how cu looks).
I guess I was just surprised that cu didn't automatically upgrade formulas even though it tells me about outdated formulas and auto updates homebrew.
I thought it would be cool if cu could just do it all lol, but idk if there is some technical reason it can't?

This repo is called homebrew-cask-upgrade, brew cu stands for cask upgrade. Messages about outdated formulas are printed automatically by upstream brew update command, which is being called under the hood.

i had the same feature request.
also because it also includes MAS updates ( with --include-mas)
i would like to also see the normal home bew formulas here - so i get an "take it all"
and with the options to update interactive or pinning versions i get additional features compared to brew upgrade

What I was actually looking for when I found cu is the ability to just type one command that updates everything homebrew related, in the correct order.

Personally i prefer to use this thing:
brew update && brew upgrade --formulae && yes | brew cu --all && brew cleanup --prune=all
as an alias brew u to update everything brew related. It's also possible to add --include-mas option to this approach.
And as one more thing i configured this alias to output command before executing, so i always know what executed under hood:
echo "brew update && brew upgrade --formulae && yes | brew cu --all && brew cleanup --prune=all" && brew update && brew upgrade --formulae && yes | brew cu --all && brew cleanup --prune=all

and you can add it by executing brew alias u --edit and pasting upper text and saving it. From now you can do brew u.
I think quite often people come to this tool when they want to update everything homebrew related using just one command while also been able to pin casks, so how about to provide them some instructions in README.md about how to do this? I think i can make PR, let me know if it's acceptable.