Switch to Go 1.12 & replace dep by go mod

Follow up of #602.
CC @domgreen

How much effort would it be to switch to 1.12? It brings some new improvements in memory management which may be (very) useful for us: https://golang.org/doc/go1.12#runtime.

@GiedriusS I'll take a look 👍

So I've just downloaded 1.12 and it builds as it with my branch. So I guess we just need to change the golang version in circle-ci.

You mean dropping dep ?
FWIU go mod tries to honour versions specified in the Gopkg.toml (it did so for minio-go). However when it reconciles projects also using go mod it can(does ?) chose to download the version required by the dependency.
FYI information I tried hard to compile Thanos with a soon to be merged prometheus patch (https://github.com/prometheus/prometheus/pull/5009) which will enlarge the dependency tree of projects importing prometheus and I couldn't make it work with dep. Moving to go mod and vendoring made it compile almost at first attempt (just needed to make this patch 5880dbb).

@domgreen as per discussion on K8s slack:

We need rebase as well

OK I will, should we jump directly to go 1.12 ?

After talking with the amazing @myitcv it turns out that with mod the requirements graph that is generated means that the transitive dependencies need to be pulled from the VCS of the dependency so for our case we are seeing that there is a dependency that needs to be pulled via bzr ... normally the case is that we need git installed to pull most dependencies.
It turns out that the dependency in question is actually a dep of Prometheus and therefore doing the same go mod vendor in Prometheus also breaks.
Lets add this to our documentation that if you see this error you will need to install bzr.

@domgreen
sylvain@ubuntu-1604-dev:~/gopath/src/github.com/improbable-eng/thanos[go-1.11]$ GO111MODULE=on go mod why launchpad.net/gocheck
# launchpad.net/gocheck
github.com/improbable-eng/thanos/pkg/cluster
github.com/hashicorp/memberlist
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/go-msgpack/codec.test
labix.org/v2/mgo/bson
labix.org/v2/mgo/bson.test
launchpad.net/gocheck

Can we add a check in make that the user has bzr installed and tell them to install it for deps if not as well as in the contributing docs.

 The final step in your CI run should then be something like test -z "$(git status --porcelain)" || (git status; git diff; false) to fail the build if any files in the working tree differ from the commit being CI-ed (this also catches untracked files). Else you would have no guarantee that the committed go.{mod,sum} (or any other files for that matter) are identical to the CI's run of go mod tidy.

Thanks, @myitcv good to have an experienced go modder 🤘 take a look over this.
@sylr have just checked locally we can replace the need to use go mod vendor and rm -rf vendor we are not checking it in so let's not use go mod vendor.
So can we refactor the dep & vendor functions into a mod function that checks what Paul mentions above in CI.
Locally I think make just needs to ensure it calls go mod tidy before we commit.

@myitcv I've just one question about go mod vendor, if all the libs are already in the $GOPATH it will still ensure that the checked out version in $GOPATH are the ones defined in go.mod before vendoring them ?

I prefer to keep go mod vendor as it does not mess with what you already have in your $GOPATH.

@sylr

I've just one question about go mod vendor, if all the libs are already in the $GOPATH it will still ensure that the checked out version in $GOPATH are the ones defined in go.mod before vendoring them ?

Modules neither touches nor uses GOPATH/src at all (unless you specifically add a directory-based replace directive at a path that happens to be within GOPATH/src; that is not the case here).
Modules use the read-only modules cache (which happens to be located at GOPATH/pkg/mod, but that's an implementation detail).
go mod vendor is documented as follows:

Vendor resets the main module's vendor directory to include all packages needed to build and test all the main module's packages. It does not include test code for vendored packages.

So go mod vendor also does not touch your GOPATH/src; it simply updates the vendor directory of your main module accordingly.

I prefer to keep go mod vendor as it does not mess with what you already have in your $GOPATH.

Can you explain what you think go mod vendor is doing? Because as I explained above, it will not touch your GOPATH/src at all.

After removing the go mod vendor and experimenting locally it does not have any impact to our dev workflow ... all I think we need from a go mod point of view is go mod tidy as @myitcv points out earlier in the thread.
We do not check in /vendor so should not need to put anything into that folder, if for a better dev experience down the line we wish to iterate on this idea lets wait till we have a concrete use case.

@myitcv All right, it seems I've been really misunderstanding the internals of go mod. I'll clear all go mod vendor then.

The latest release of Thanos is still built with go 1.11 I think:
$ ./thanos --version
thanos, version 0.4.0 (branch: HEAD, revision: a676095877e7a87576b0f7776fccd4b3f8b65188)
  build user:       root@3e7111579ad1
  build date:       20190504-09:31:11
  go version:       go1.11.9

Does the CI system need to change?


Checkboxes are rendering with a bulletpoint underneath

With the example given in the docs, checkboxes render with a bullet point underneath.

Taking the <li>s out removes the bulletpoints but then the checkbox is then not vertically-aligned to the middle. It's also bad HTML.

Added a style suppressing bullet points for lists inside the main form.


Fix #105

I also faced same issue #105

Here what changes

implemented lock mechanism following redis.io's handling deadlock
hostname lookup only once

Hi, thanks a lot. Just to be sure I understand it correctly, are you having issues with Spring Redis 1 (which I am aware of) or with Spring Redis 2?

Having issues with Spring Redis 1.
It doesn't have an interface to support SETNX with TTL in one command.
https://github.com/spring-projects/spring-data-redis/blob/1.0.x/src/main/java/org/springframework/data/redis/connection/RedisStringCommands.java#L37

Cool, in such case I will create a separate module with Spring Redis 1 lock provider and move your code there. Thanks again.

I have taken your code and copied it to a new module https://github.com/lukas-krecan/ShedLock/pull/199. The tests are passing so once you say that the code looks good for you, I will release it.

This PR code was released as 4.3.1!


Deploying event-driven sample to a server

Dear Alex,
I have been working on an event-based Office Add-In that is now working in Outlook on the web with a local host for debugging. However, it fails to work when I try to deploy it to a server.
I am told that the events.html file refered to in the manifest file would usually have a

Hi Dylan,
Thank you for reaching out. GitHub issues are for reporting potential bugs or errors in our documentation.
I would like to suggest you raise the same clarification on https://stackoverflow.com/questions/tagged/office-js
Hope this helps.


Can't work

hello, I download the this sdk and test with the guide.
but i can't get success result. always get errowcode 1131 InvalidAPIEndpoint.

Apologies, we are no longer supporting this SDK and will not be issuing any fixes. Feel free to fork this SDK, fix any issues, and develop in your own account.


Zero-downtime deployments with rolling upgrades

Hey all,
Usecase
I have a front-end proxy which listens on ports 80 and 443. My applications are deployed as microservices behind this proxy, which acts as a layer-7 load balancer for those applications. These applications are, for example, php:7-apache containers which simply listen on port 80 for HTTP requests. The idea is to have at least 2 replicas for a service, so they can be upgraded incrementally using the rolling upgrade functionality that comes with Docker Swarm.
Issue
At this point, I'm not sure if zero-downtime deployments using the rolling update functionality of Docker Swarm are even possible, at least for my use case. There is one major issue for me here.
When containers are stopped during the rolling update, they are always stopped using the same signal (SIGTERM, or SIGKILL after a certain period). Many images, like the aforementioned apache-based image, won't gracefully shutdown with a SIGTERM, but need a different signal to be sent for the container to shutdown in a graceful way. I created an issue (#25696) for this as well, but this didn't make the 1.13 release. I don't see how the current rolling upgrade system can work in any use case, except for the cases where containers actually are designed specifically to shutdown gracefully when receiving a SIGTERM. In my situation, upgrading the service leads to intermittent HTTP-502 errors until the upgrade is complete. I can't imagine this not being a problem for anyone else, unless I'm missing something obvious.
Possible workaround
Wrap the main command of an image that needs to be able to shutdown gracefully in a wrapper script:
shut_down() {
  kill -SIGWINCH ${SCRIPT_PID}
}

trap 'shut_down' SIGTERM SIGINT

start_apache &

SCRIPT_PID = "$!"
wait ${SCRIPT_PID}

This would immediately fix the issue I'm having, since any SIGINT or SIGTERM that reaches the container would simply be relayed as a SIGWINCH. In this case, this would gracefully shutdown my apache container. However, this would mean that I would have to modify every image I'm using to use this script. It's also a non-standard and to be fair, a nasty solution.
What's the recommended course of action here? Am I missing a piece of the puzzle, or simply overseeing something? Also: even if this issue would be solved, would I get true zero downtime deployments with the rolling upgrade functionality of Docker Swarm Mode? In other words, are containers actually removed from the ingress load balancing pool prior to sending the stop_signal during the upgrade, or would I still get HTTP-502 errors from containers that are still being load-balanced to, but would be in the process of shutting down?

@thaJeztah Can you point me in the right direction?

@mvdstam if that image should always be stopped with SIGWINCH, it would make sense to set it in the Dockerfile (using the STOPSIGNAL command), e.g.;
FROM busybox
STOPSIGNAL WINCH

Although you cannot override the default stop-signal when creating a service, you can still define it in the image, and Swarm will take that into account when stopping the container;
INFO[0120] Container 52aca76dd0c25e733f43121b4097103aae5f2dd125a36c083407d5b61eb8ae50 failed to exit within 10 seconds of signal 28 - using the force

Does that help?
It may be worth checking what exactly Apache is doing during a "graceful" shutdown, and if those steps are required if you're running in a container. For example, when using the official PHP image, logs are written to stdout/stderr, so any task that Apache would do to finish writing logs, or log-rotation are not needed.

@thaJeztah Thanks a lot, I had no idea one could do this with the STOPSIGNAL command in a Dockerfile.
I still get in trouble when I update a stack with docker stack deploy though. For example, consider the following VIP: <IP_ADDRESS>. I can confirm that this VIP is correctly load-balancing between multiple containers. The containers are simply apache-based services that serve a simple HTTP page. I can also confirm that the containers are stopped gracefully with a SIGWINCH when I run docker stack deploy:
[Mon Jan 23 14:14:44.767350 2017] [mpm_prefork:notice] [pid 1] AH00170: caught SIGWINCH, shutting down gracefully
However, during the rolling update, it seems containers that are in the shutting down state are still being load-balanced to. The starting state combined with the HEALTHCHECK directive seems to work perfectly as starting containers that are not yet defined as healthy are not placed in the load balancing pool.
In my head, the rolling update should function as follows:

Start rolling update for a service that is replicated with N containers
Do the following in batches of update_parallelism for those N containers:

Remove container from ingress load balancing pool for the respective service so it will not receive any more incoming traffic
Stop that container
Wait for the container to exit (ie give it time to finish any running requests), and send a SIGKILL after 10 seconds
Start new container
If the container has a healthcheck configured:

Wait for the new container to be healthy

Add container to ingress load balancing pool

Rolling update complete

Can you confirm that this is roughly the flow regarding rolling updates? I get the feeling that the container is stopped prior to it being removed from the ingress load balancing pool. This would explain the results I'm getting.

Your summary looks reasonable; I'm not 100% familiar with the exact order in which these events take place, so let me ping @aaronlehmann and @aboch, who may be able to answer this (without me having to dive into the code 😅). Possibly this proposal is relevant here as well, so let me link it; https://github.com/docker/docker/pull/30261
On a side-note; if SIGWINCH is the expected stop-signal for httpd (apache); it may be worth opening a pull request, or feature request for the official image (https://github.com/docker-library/httpd/issues, and/or https://github.com/docker-library/php/issues)

Thanks a bunch @thaJeztah, I'll look into making PR's for those images. It seems logical to have the the STOPSIGNAL necessary for graceful shutdowns configured by default, but perhaps there was a reason not to do that.

Adding @dongluochen so he can comment on the details of how health checks interact with load balacning.

In 1.13 libnetwork provides APIs to separate endpoint deletion (ie: remove the network plumbing) and removing the container from LB. Looking at the usage  I think we trigger the LB removal when the container is Shutdown. To achieve the behavior @mvdstam wants (which is clearly better for rolling upgrade) LB removal should happen before container shutdown.

@mvdstam  Assuming we do this..
Remove container from ingress load balancing pool for the respective service so it will not receive any more incoming traffic
If we pick some default for the interval between removal from LB and the container stop it still won't help the apps that are serving long lived sessions. So this by itself won't guarantee a hitless upgrade unless the interval is made configurable ?

@thaJeztah I think this should be marked an enhancement in area/swarm because we need a swarmkit API to trigger the LB removal for the task before its shutdown.

@mvdstam The rolling update procedure is like what you described but there is some subtlety to clarify.

Remove container from ingress load balancing pool for the respective service so it will not receive any more incoming traffic

This happens when Swarm tries to shutdown the task. In a distributed environment, this action needs time to propagate to every node in the cluster. If the container is shutdown before this state is propagate to every node in the cluster, requests may be lost.

Stop that container
Wait for the container to exit (ie give it time to finish any running requests), and send a SIGKILL after 10 seconds

By default Docker daemon sends SIGTERM and after 10 seconds sends SIGKILL. The container shouldn't exit on SIGTERM because of the race condition described above. I want to clarify if this is where you are seeing request lost.
@sanimej I think it's all right to remove endpoints from LB at Shutdown. The question is if we can get confirmation this has propagated to every node of the cluster. In even worse case, what should be done if some nodes are busy and not responsive.

Hey guys,
Thanks for your comments and clarification.
@sanimej @dongluochen I understand the race-condition issue you are bringing up. From what I understand, the issue lies in the fact that the removal of an endpoint from the LB needs to propagate through all of the nodes before the container may be shut down. As @dongluochen mentioned, there are two questions here:

Is it possible at this time to get confirmation of this being propagated successfully to every node of the cluster?
In the case of nodes that are busy, unresponsive, breaking down, etc: what happens then?

The question I have in addition to that: how many time passes for the endpoint removal to propagate to all nodes in the cluster? Does this time increase with O(n) when the number of nodes increase within a cluster, or even worse: O(n²)?
My guess is that this time is highly dependent on several factors such as network latency, load on the servers, number of nodes running in the cluster, etc. Having a (albeit configurable) static interval (delay) between removing the endpoint from the LB and stopping the container seems like a bad idea; this process should be synchronous.
The issue of "what to do" when nodes are unresponsive supersedes the issue at hand in my opinion, since non-responsive nodes affect all operations within the swarm cluster and not only updating services. In such a case, the node should just be removed from the cluster and its services rescheduled. I do think that some kind of timeout per node would be appropriate to avoid deadlocks. Just like healthchecks on containers that simply fail after certain periods of time without the expected result.

@mvdstam @dongluochen @sanimej Would it make sense to add a delay between when we broadcast the LB removal and when we actually start stopping the container?
That delay could be configurable with a sensible default.
Gossip converges really fast and I think even a one-second delay would possibly fix 80+% of those problems.
@mvdstam The time increase is more in the order of O(log(N)) - every node spreads the message to its peers.

@aluzzardi @dongluochen @sanimej Thanks for clarifying, the logarithmic increase in time makes sense.
I don't think there should be a static delay between the LB removal and stopping the container. During LB removal, a hundred things may happen causing the gossip propagation to take longer than expected. I think a callback-based system is the best way to go, since deployments (especially if we're talking about zero downtime deployments) should be 100% reliable and consistent. Also, if one of the servers fails to deliver the callback, then something serious is wrong and we really, really shouldn't want to update anything. Having a static delay between the update steps would also mean that this value would need to be adjusted when the number of nodes in the cluster change.
The reason why this is so important for me, and I can imagine many other folks, is the following.
In my situation for example, I am working on a application which consists of many services (database, redis, perhaps some worker processes, etc). One of those services is a webapplication, which will constantly handle incoming HTTP requests. This webservice may be updated once a week, or many times per day depending on what kind of work we're doing on the application. Updates are done with a CI pipeline, where the entire process is automated.
Especially in the case of patches, it is very important that we can rollout updates to the application during the day where many requests are being handled each second (and thus: the servers may be under significant load and may take longer than usual to actually perform updates and/or propagate network gossip).
Right now, we're using a (custom built) Docker version of Gantry with some IP tables magic to guarantee our front-end facing services' deployments have zero downtime (and zero packet loss thanks to this article). Although this works really well, it is still a very complex solution with a lot of bells and whistles. And it doesn't work with Docker Swarm, so we can't scale our application horizontally at all (at least, not without creating an even more complex situation). This is literally the only missing piece of the puzzle for us to migrate fully to Docker Swarm, and I can't imagine this not being the case for many other people who are in the same boat as we are.
I'm eager to hear all of your thoughts about this. I really hope we can get some traction going for this issue as well! 😄

Also, if one of the servers fails to deliver the callback, then something serious is wrong and we really, really shouldn't want to update anything.

I'm not sure I agree with this part. In a large-scale distributed system, failures will be a fact of life, and blocking the update in those cases may not be a scalable strategy.

@aaronlehmann In that case, would it be better to reschedule the service we're trying to update to a different node and consider the "failing" node as "down"?

I may have misunderstood what you were suggesting, but I thought you were talking about waiting until all nodes had updated their load balancer configurations to shut down an old version of a container. This involves getting confirmation from all the nodes in the swarm, not just the one running the container.

I discussed this issue with @mavenugo and @aboch. We think that adding delay to wait for cluster convergence doesn't work. Gossip is an eventual consistent system with no guarantee on convergence delay. We would like to add a confirmation mechanism for network updates. While confirming each gossip event individually is not scalable, we can put network event confirmation into batches, each node sends batch confirmation to managers. Managers processes the confirmations from each node and signal when an event has been processed by every node.
In load balancer use case, before shutting down a container, a network event to remove the sandbox from IPVS table is sent thru gossip. The agent waits for confirmation for this event. When confirmation is received, it moves on to shutdown the container. The container upon receiving stop signal (e.g. SIGTERM), it should finish existing requests and exit gracefully.
Here are some initial thoughts.

Network event confirmation needs to be reliable. The confirmation should be saved in raft so that leader switch would not lose information. (We may want to distribute confirmation processing to all the managers to reduce load on leader.)
An agent can subscribe to an network change and wait for confirmation. At the same time, the confirmation can be queried in case leader swap.
If some nodes are slow, they would slow down the whole cluster significantly. Node eviction might be needed. Network failure has similar impact on performance.

Thanks for your thoughts on this @dongluochen!
Any chance we can get this on the list for the next milestone? This is literally the only missing piece of the puzzle for us, since webservices make up a large part of our infrastructure and should be deployed without service interruption.

cc @mavenugo @aboch. This requires changes on libnetwork.

@dongluochen Gossip is an eventual consistent system with no guarantee on convergence delay. - Yes, there are no hard guarantees. But Gossip converges really fast, its logarithmic with a base more than 2 because each node sends the event to multiple peers.
For network events we use gossip because getting a consistency guarantee across the cluster for the chatty nature of network events just doesn't scale. So the approach of patching some consistency mechanism on top of gossip is not a good idea. As @aaronlehmann mentioned we should expect nodes failing randomly. So waiting for response from all nodes needs a timeout and a node failure detection and appropriate error handling (we fail the upgrade ?) for the failures. Network events most definitely shouldn't go into raft state.
@mvdstam Your earlier comment was However, during the rolling update, it seems containers that are in the shutting down state are still being load-balanced to.  - Can you be specific here, are you seeing requests from new clients going to containers that are about to be shutdown ? Unless you have a cluster with 1000s of nodes gossip propagation should be really quick. May be there is some additional delay in removing the state from the kernel. But you can confirm this with a quick experiment.. If you see new requests going to the containers being upgraded, can you add a 1 or 2 seconds sleep in the script you had originally and see if it helps ?
shut_down() {
  sleep 2 <<<<<<
  kill -SIGWINCH ${SCRIPT_PID}
}

trap 'shut_down' SIGTERM SIGINT

start_apache &

SCRIPT_PID = "$!"
wait ${SCRIPT_PID}

As @aluzzardi mentioned the right approach here is to add a delay and may be make it configurable eventually.

@sanimej

Can you be specific here, are you seeing requests from new clients going to containers that are about to be shutdown

Yes, during a simple test using siege while performing the rolling update, I got loads of HTTP-502 messages during the update process when performing HTTP requests to a webservice. The HTTP-502 originated from another container acting as a reverse proxy which routes incoming traffic to the VIP of the webservice as assigned by Docker.
I suspect the container receives the shutdown signal prior to being removed from the LB pool, which means the container still receives incoming HTTP requests while it doesn't listen for them anymore and is actually in the process of shutting down (ie letting child processes finish up but not accepting any new requests).
That said, I'm not sure how adding a sleep in this script will help unless both the removal of the container (task) from the LB pool as well as stopping the container happens in parallel instead of in a serial fashion.
I'll do a test with the delay and report back!

@mvdstam Call to remove from the LB pool is invoked before the shutdown..
https://github.com/docker/docker/blob/master/daemon/cluster/executor/container/controller.go#L319
@dongluochen In the rolling update case is it possible for a task to get shutdown without LB removal ?
If there is some latency in all the nodes getting the gossip message and doing the actual removal adding a delay should help.

In the rolling update case is it possible for a task to get shutdown without LB removal ?

@sanimej Rolling update goes thru the same Shutdown function.
Do you think we should just add a delay between r.adapter.deactivateServiceBinding() and r.adapter.shutdown(ctx)?
https://github.com/docker/docker/blob/master/daemon/cluster/executor/container/controller.go#L324

@dongluochen Do you think we should just add a delay between r.adapter.deactivateServiceBinding() and r.adapter.shutdown(ctx)? - Yes. But lets see if we can first pin down exactly what is happening in @mvdstam's environment.

@sanimej @dongluochen I have created a simple testcase with the added sleep delay as suggested by @sanimej: https://github.com/mvdstam/docker-rolling-upgrade-test.
As you can see in this screencast, this seems to work perfectly in combination with the healthcheck configuration of this webservice. I can perform upgrades of the webservice without any packet loss, so this seems like a nice (albeit temporary) workaround.

@mvdstam Thanks for confirming. To start with we can add a 2 seconds delay which should be sufficient for small to medium size clusters and later add a configurable option for it.
cc @dongluochen @aluzzardi

@sanimej Great, thank you and you're welcome. Can the change you are suggesting be patched into 1.13?

@sanimej Shutdown is in the path of task restart/rolling updates, etc. Adding delay here would increase service converge time. For containers that are not load balancer endpoints this delay is unnecessary. Can sandbox.DisableService() returns an error NoServiceBinding for such cases so that Shutdown can use it to decide if delay should be inserted.

@mvdstam The change in #31142 is very simple. It'd be great if you can test it in your environment.

@dongluochen Definitely -- what's the best way to get a Docker version with your commit in it? Do I have to build Docker locally myself?

@mvdstam You can download x86_64 Linux version here https://github.com/dongluochen/container-notes/raw/master/docker/docker-f0650a8.tgz, or build it yourself.

@dongluochen Thanks, I'll test using the binary you've provided and report back!

@dongluochen I have done some testing with the binary you provided in a cluster of 3 nodes (which I created just this morning on Digitalocean, so clean installs without anything else happening), with the webservice replicated globally. The image used as webservice is as follows:
FROM php:7.1-apache
STOPSIGNAL WINCH

COPY ./html/index.php /var/www/html/index.php

The docker-compose.yml file used:
version: '3'

services:
  webservice:
    image: mvdstam/apache-rolling-update
    ports:
      - "80:80"
    deploy:
      mode: global
      update_config:
        parallelism: 1
        delay: 5s
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 2s
      timeout: 5s
      retries: 3

When I did the first couple of tests, it seemed to work precisely as expected. But then this happened 😄  As you can see, I even got packet loss with 3 tasks in the running state. I suspect this is an issue unrelated to the one we're discussing, as I've seen this problem earlier (packet loss after services are updated).
If you'd like, I could give you access to those droplets so you can check it out yourself.

@dongluochen I've removed the droplets as of this morning. Please let me know if you need any more information to clarify this issue.

@mvdstam Thanks a lot for the testing. I'll look into this.

But then this happened :smile: As you can see, I even got packet loss with 3 tasks in the running state.

Yeah, I'm getting this too, running 1.13, 3 nodes, health checks, no other confounding factors.

@mvdstam I repeat your test and see similar request loss. The interesting part I find is the request loss happens after a container becomes healthy. In the test I add a global service on a 2-node cluster. Using the return ID I can see which container is failing the request. In the following example, at 02:52:00 a new container (7c2f8aa729cc on node ip-172-19-241-144) became healthy and was added to IPVS rotation. But for some reason requests to this new container were not delivered until 02:52:09. Between 02:52:00 and 02:52:09 half the requests were lost (the other container is responding). During this time the new container 7c2f8aa729cc were responding to healthcheck probes. It didn't see requests from IPVS.
@sanimej From node ip-172-19-241-144 daemon log (/var/log/syslog) I see following messages at the same period. Could it be MAC learning for the new containers is delayed so request delivery fails?
Feb 22 02:52:01 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:01.561353647Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"

I run arp command during failing period. I saw one case of incomplete record. Otherwise it looks fine.
Address                  HWtype  HWaddress           Flags Mask            Iface
<IP_ADDRESS>               ether   02:42:b3:b2:f3:e0   C                     eth1
<IP_ADDRESS>               ether   02:42:0a:ff:00:05   C                     eth0
<IP_ADDRESS>               ether   02:42:0a:ff:00:03   C                     eth0
<IP_ADDRESS>              ether   02:42:0a:ff:00:0e   C                     eth0
<IP_ADDRESS>                      (incomplete)                              eth0

Here are the detailed info. External query:
Wed Feb 22 02:51:59 UTC 2017
Server hostname: 5b9c26855175563
Wed Feb 22 02:51:59 UTC 2017
Server hostname: 5b9c26855175564
Wed Feb 22 02:52:00 UTC 2017
Server hostname: 5b9c26855175565
Wed Feb 22 02:52:00 UTC 2017
Server hostname: 5b9c26855175566
Wed Feb 22 02:52:00 UTC 2017
Server hostname: 5b9c26855175567
Wed Feb 22 02:52:00 UTC 2017
curl: (28) Connection timed out after 1000 milliseconds

Wed Feb 22 02:52:01 UTC 2017
Server hostname: 5b9c26855175569
Wed Feb 22 02:52:01 UTC 2017
curl: (28) Connection timed out after 1001 milliseconds

Wed Feb 22 02:52:02 UTC 2017
Server hostname: 5b9c26855175571
Wed Feb 22 02:52:02 UTC 2017
curl: (28) Connection timed out after 1001 milliseconds

Wed Feb 22 02:52:04 UTC 2017
Server hostname: 5b9c26855175573
Wed Feb 22 02:52:04 UTC 2017
curl: (28) Connection timed out after 1000 milliseconds

Wed Feb 22 02:52:05 UTC 2017
Server hostname: 5b9c26855175575
Wed Feb 22 02:52:05 UTC 2017
curl: (28) Connection timed out after 1000 milliseconds

Wed Feb 22 02:52:06 UTC 2017
Server hostname: 5b9c26855175577
Wed Feb 22 02:52:06 UTC 2017
curl: (28) Connection timed out after 1001 milliseconds

Wed Feb 22 02:52:07 UTC 2017
Server hostname: 5b9c26855175579
Wed Feb 22 02:52:07 UTC 2017
curl: (28) Connection timed out after 1001 milliseconds

Wed Feb 22 02:52:09 UTC 2017
Server hostname: 5b9c26855175581
Wed Feb 22 02:52:09 UTC 2017
Server hostname: 7c2f8aa729cc582
Wed Feb 22 02:52:09 UTC 2017
Server hostname: 5b9c26855175583

Logs from container that became healthy at 02:52:00. Requests from <IP_ADDRESS> were healthcheck probes. Requests from <IP_ADDRESS> were the request from IPVS.
ip-172-19-241-144:/# docker logs 7c2f8aa729cc
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using <IP_ADDRESS>. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using <IP_ADDRESS>.
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.533793548Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"d66bb47d28734d0c4497e1dd58f21c6959f23e557edcf349edebc5847840dc0e\", Timestamp:(*timestamp.Timestamp)(0xc42096ed40)}"
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.534343890Z" level=debug msg="attach: stderr: end"
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.534762771Z" level=debug msg="attach: stdout: end"
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.535181624Z" level=debug msg="Health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 done (exitCode=0)"
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.536363619Z" level=debug msg="state changed" module="node/agent/taskmanager" service.id=wpitottykfr3xd3v5wxavjm45 state.desired=RUNNING state.transition="STARTING->RUNNING" task.id=xd58plku9nzwi2l7saq144igl
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.537280144Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=xd58plku9nzwi2l7saq144igl
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.539099278Z" level=debug msg="task status reported" module="node/agent"
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.539630680Z" level=debug msg="waiting on events" module="node/agent/taskmanager" service.id=wpitottykfr3xd3v5wxavjm45 task.id=xd58plku9nzwi2l7saq144igl
Feb 22 02:52:00 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:00.561695528Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:01 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:01.561353647Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:01 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:01.806450029Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.535883478Z" level=debug msg="Running health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 ..."
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.536741396Z" level=debug msg="starting exec command e51426c6b16f720b40e734b173845459253dc1f15800905b504ff636895c6b7c in container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.540178455Z" level=debug msg="attach: stdout: begin"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.540783981Z" level=debug msg="attach: stderr: begin"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.656330179Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"e51426c6b16f720b40e734b173845459253dc1f15800905b504ff636895c6b7c\", Timestamp:(*timestamp.Timestamp)(0xc4208c7ad0)}"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.656965077Z" level=debug msg="libcontainerd: event unhandled: type:\"start-process\" id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\" pid:\"e51426c6b16f720b40e734b173845459253dc1f15800905b504ff636895c6b7c\" timestamp:<seconds:1487731922 nanos:655520437 > "
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.692145431Z" level=debug msg="containerd: process exited" id=7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 pid=e51426c6b16f720b40e734b173845459253dc1f15800905b504ff636895c6b7c status=0 systemPid=28585
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.693231938Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"e51426c6b16f720b40e734b173845459253dc1f15800905b504ff636895c6b7c\", Timestamp:(*timestamp.Timestamp)(0xc4208c7dd0)}"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.693861492Z" level=debug msg="attach: stdout: end"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.694272777Z" level=debug msg="attach: stderr: end"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.694685811Z" level=debug msg="Health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 done (exitCode=0)"
Feb 22 02:52:02 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:02.805644170Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:03 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:03.083633408Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.081299333Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.328502543Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.496662064Z" level=debug msg="ip-172-19-241-144-6b580491f376: Initiating bulk sync with nodes [ip-172-19-147-51-f229826a1bf1]"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.497410147Z" level=debug msg="ip-172-19-241-144-6b580491f376: Initiating unsolicited bulk sync for networks [wh47x1oulc3isethn9hgtpcyk mdt0ovna6m34r5qd0hykxsj0p] with node ip-172-19-147-51-f229826a1bf1"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.501262652Z" level=debug msg="memberlist: TCP connection from=<IP_ADDRESS>:53952"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.502247597Z" level=debug msg="ip-172-19-241-144-6b580491f376: Bulk sync to node ip-172-19-147-51-f229826a1bf1 took 3.286285ms"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.695315004Z" level=debug msg="Running health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 ..."
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.696243317Z" level=debug msg="starting exec command fcd3ae2fa53e084a933162eacff82e59d83cb2b741eac155bb52397475eb254c in container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.697513753Z" level=debug msg="attach: stdout: begin"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.697975493Z" level=debug msg="attach: stderr: begin"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.816291722Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"fcd3ae2fa53e084a933162eacff82e59d83cb2b741eac155bb52397475eb254c\", Timestamp:(*timestamp.Timestamp)(0xc4207a64e0)}"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.816939634Z" level=debug msg="libcontainerd: event unhandled: type:\"start-process\" id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\" pid:\"fcd3ae2fa53e084a933162eacff82e59d83cb2b741eac155bb52397475eb254c\" timestamp:<seconds:1487731924 nanos:815283476 > "
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.856092531Z" level=debug msg="containerd: process exited" id=7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 pid=fcd3ae2fa53e084a933162eacff82e59d83cb2b741eac155bb52397475eb254c status=0 systemPid=28612
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.857278337Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"fcd3ae2fa53e084a933162eacff82e59d83cb2b741eac155bb52397475eb254c\", Timestamp:(*timestamp.Timestamp)(0xc4207a6790)}"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.857877705Z" level=debug msg="attach: stderr: end"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.858270397Z" level=debug msg="attach: stdout: end"
Feb 22 02:52:04 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:04.858679903Z" level=debug msg="Health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 done (exitCode=0)"
Feb 22 02:52:05 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:05.325221199Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:05 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:05.588145239Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:06 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:06.585226257Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:06 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:06.833008950Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:06 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:06.859296618Z" level=debug msg="Running health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 ..."
Feb 22 02:52:06 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:06.877506579Z" level=debug msg="starting exec command 807d6c8c19fdfe57ef5b78881673806ee1b8c62228aa0a4d1860aabfcd294114 in container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612"
Feb 22 02:52:06 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:06.885114788Z" level=debug msg="attach: stdout: begin"
Feb 22 02:52:06 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:06.893674277Z" level=debug msg="attach: stderr: begin"
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.024604879Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"807d6c8c19fdfe57ef5b78881673806ee1b8c62228aa0a4d1860aabfcd294114\", Timestamp:(*timestamp.Timestamp)(0xc42085f8c0)}"
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.025286349Z" level=debug msg="libcontainerd: event unhandled: type:\"start-process\" id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\" pid:\"807d6c8c19fdfe57ef5b78881673806ee1b8c62228aa0a4d1860aabfcd294114\" timestamp:<seconds:1487731927 nanos:23559331 > "
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.057639666Z" level=debug msg="containerd: process exited" id=7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 pid=807d6c8c19fdfe57ef5b78881673806ee1b8c62228aa0a4d1860aabfcd294114 status=0 systemPid=28634
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.059132725Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"807d6c8c19fdfe57ef5b78881673806ee1b8c62228aa0a4d1860aabfcd294114\", Timestamp:(*timestamp.Timestamp)(0xc42085ff40)}"
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.059773729Z" level=debug msg="attach: stdout: end"
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.060223947Z" level=debug msg="attach: stderr: end"
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.060634034Z" level=debug msg="Health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 done (exitCode=0)"
Feb 22 02:52:07 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:07.829214313Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:08 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:08.091344853Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.061234377Z" level=debug msg="Running health check for container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 ..."
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.062074069Z" level=debug msg="starting exec command 9d0a254ec84ea77ca4e72a2c98994f20321611344405d305b607b0767ab2d44b in container 7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612"
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.073483158Z" level=debug msg="attach: stdout: begin"
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.073971127Z" level=debug msg="attach: stderr: begin"
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.098365248Z" level=debug msg="Miss notification, l2 mac 02:42:0a:ff:00:03"
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.204198744Z" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\", Status:0x0, Pid:\"9d0a254ec84ea77ca4e72a2c98994f20321611344405d305b607b0767ab2d44b\", Timestamp:(*timestamp.Timestamp)(0xc4210f8300)}"
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.204835618Z" level=debug msg="libcontainerd: event unhandled: type:\"start-process\" id:\"7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612\" pid:\"9d0a254ec84ea77ca4e72a2c98994f20321611344405d305b607b0767ab2d44b\" timestamp:<seconds:1487731929 nanos:203301022 > "
Feb 22 02:52:09 ip-172-19-241-144 dockerd[9168]: time="2017-02-22T02:52:09.227820903Z" level=debug msg="containerd: process exited" id=7c2f8aa729cc7954c81388efa7ac5f583403ef5265c290699f7422051c04b612 pid=9d0a254ec84ea77ca4e72a2c98994f20321611344405d305b607b0767ab2d44b status=0 systemPid=28655

@dongluochen Thanks for the feedback -- please let me know if you need me to do more testing on my end!

@dongluochen Are you scaling down and then up while the client is continuously trying to access the service and thats when you are noticing the drop for few seconds ?

@sanimej I was using a global service and did force upgrade. I saw request loss for ~10 seconds right after a new container is added to rotation. The container was healthy but didn't get these requests. I suspect the requests were not routed to the new container due to switching/routing issue.

@sanimej I narrow down the issue https://github.com/docker/docker/issues/30321#issuecomment-281557854 to the communication between ingress_sbox and the sandbox of a container on the same host. It looks like VTEP not knowing where to forward a packet to the new container for about 10 seconds. It looks like it takes this time to learn the destination.
In the following example, when the new container becomes healthy, its IP <IP_ADDRESS> is added to IPVS rotation at 23:28:38.769. But the sandbox of the new container gets first packet at 23:28:47.242.
Here is the tcpdump taken from ingress_sbox.
23:28:38.769048 IP <IP_ADDRESS>.48858 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68136927 ecr 0,nop,wscale 7], length 0
23:28:39.766182 IP <IP_ADDRESS>.48858 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68137177 ecr 0,nop,wscale 7], length 0
23:28:40.018383 IP <IP_ADDRESS>.48862 > <IP_ADDRESS>.8001: Flags [S], seq 740944373, win 43690, options [mss 65495,sackOK,TS val 68137240 ecr 0,nop,wscale 7], length 0
23:28:41.018196 IP <IP_ADDRESS>.48862 > <IP_ADDRESS>.8001: Flags [S], seq 740944373, win 43690, options [mss 65495,sackOK,TS val 68137490 ecr 0,nop,wscale 7], length 0
23:28:41.262852 IP <IP_ADDRESS>.48866 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68137551 ecr 0,nop,wscale 7], length 0
23:28:42.262186 IP <IP_ADDRESS>.48866 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68137801 ecr 0,nop,wscale 7], length 0
23:28:42.495023 IP <IP_ADDRESS>.48872 > <IP_ADDRESS>.8001: Flags [S], seq 889758315, win 43690, options [mss 65495,sackOK,TS val 68137859 ecr 0,nop,wscale 7], length 0
23:28:43.494179 IP <IP_ADDRESS>.48872 > <IP_ADDRESS>.8001: Flags [S], seq 889758315, win 43690, options [mss 65495,sackOK,TS val 68138109 ecr 0,nop,wscale 7], length 0
23:28:43.750138 IP <IP_ADDRESS>.48876 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68138172 ecr 0,nop,wscale 7], length 0
23:28:44.750172 IP <IP_ADDRESS>.48876 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68138423 ecr 0,nop,wscale 7], length 0
23:28:44.992814 IP <IP_ADDRESS>.48882 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68138483 ecr 0,nop,wscale 7], length 0
23:28:45.990182 IP <IP_ADDRESS>.48882 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68138733 ecr 0,nop,wscale 7], length 0
23:28:46.242924 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68138796 ecr 0,nop,wscale 7], length 0
23:28:47.242224 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68139046 ecr 0,nop,wscale 7], length 0
23:28:47.242275 IP <IP_ADDRESS>.8001 > <IP_ADDRESS>.48886: Flags [S.], seq<PHONE_NUMBER>, ack<PHONE_NUMBER>, win 27960, options [mss 1410,sackOK,TS val 68139046 ecr 68139046,nop,wscale 7], length 0
23:28:47.242309 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [.], ack 1, win 342, options [nop,nop,TS val 68139046 ecr 68139046], length 0
23:28:47.247406 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [F.], seq 1, ack 1, win 342, options [nop,nop,TS val 68139047 ecr 68139046], length 0
23:28:47.247477 IP <IP_ADDRESS>.8001 > <IP_ADDRESS>.48886: Flags [F.], seq 1, ack 2, win 219, options [nop,nop,TS val 68139047 ecr 68139047], length 0

Displaying the Ethernet header (from another capture) shows ingress_sbox has the correct MAC address.
02:42:0a:ff:00:04 (oui Unknown) > 02:42:0a:ff:00:06 (oui Unknown), ethertype IPv4 (0x0800), length 74: <IP_ADDRESS>.51430 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68584659 ecr 0,nop,wscale 7], length 0

Here is the tcpdump taken from the sandbox of the target container.
root@ip-172-19-241-144:/#  tcpdump -i eth0 tcp port 8001
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:28:47.242238 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [S], seq<PHONE_NUMBER>, win 43690, options [mss 65495,sackOK,TS val 68139046 ecr 0,nop,wscale 7], length 0
23:28:47.242269 IP <IP_ADDRESS>.8001 > <IP_ADDRESS>.48886: Flags [S.], seq<PHONE_NUMBER>, ack<PHONE_NUMBER>, win 27960, options [mss 1410,sackOK,TS val 68139046 ecr 68139046,nop,wscale 7], length 0
23:28:47.242314 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [.], ack 1, win 342, options [nop,nop,TS val 68139046 ecr 68139046], length 0
23:28:47.247415 IP <IP_ADDRESS>.48886 > <IP_ADDRESS>.8001: Flags [F.], seq 1, ack 1, win 342, options [nop,nop,TS val 68139047 ecr 68139046], length 0
23:28:47.247471 IP <IP_ADDRESS>.8001 > <IP_ADDRESS>.48886: Flags [F.], seq 1, ack 2, win 219, options [nop,nop,TS val 68139047 ecr 68139047], length 0

On this host, docker has Miss notification at exactly the same period.
 Do you see this intermittent drop only for local containers ? Does it work ok for remote containers ?

@sanimej Yes I only see this drop for local containers. My tests used 2 nodes and the remote container is always fine. It's quite consistent that leads me to think it a local vtep update problem.

@dongluochen For local containers the VTEP is not used. I will look into this.

@dongluochen @sanimej Have you made any progress regarding this issue? Let me know if you need me to run more tests! 👍

@mvdstam I tested a patch on docker-17.03.0-ce and it fixes my problem of request loss at container startup https://github.com/docker/docker/issues/30321#issuecomment-281557854. You can find a build at https://github.com/dongluochen/container-notes/raw/master/docker/docker-d3bc200.tgz, which also includes the 2 second delay for gossip convergence. It'd be great if you can run your tests to validate it.
diff --git a/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go b/vendor/github.com/docker/libnetwor
index adc370f..4a7174a 100644
--- a/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go
+++ b/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go
@@ -273,6 +273,11 @@ func (d *driver) peerAdd(nid, eid string, peerIP net.IP, peerIPMask net.IPMask,
        if sbox == nil {
                return nil
        }
+       // only add remote entries to peerdb
+       // this is a test. The proper fix should update entry.isLocal.
+       if vtep.Equal(net.ParseIP(d.advertiseAddress)) || vtep.IsLoopback() {
+               return nil
+       }

        IP := &net.IPNet{
                IP:   peerIP,

@dongluochen Definitely, I'll test your build as soon as I have the chance and I'll report back to you after. Maybe we can include this in 17.04?

@dongluochen I've done some tests, and there is still some packet loss when performing rolling updates. However, the problem regarding massive packet loss seems to have been fixed.
I've tested with the following docker-compose.yml:
version: '3'

services:
  webservice:
    image: mvdstam/apache-rolling-update
    ports:
      - "80:80"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 1s
      timeout: 10s
      retries: 3
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 5s
      restart_policy:
        condition: on-failure

Dockerfile contents:
FROM php:7.1-apache
STOPSIGNAL SIGWINCH

COPY ./html/index.php /var/www/html/index.php

See also this repository for more information.
It seems we're getting really close to a 100% zero-downtime update solution. Can you verify on your end that packet loss still occurs?

@mvdstam Thanks for the tests!
I deployed your service into my test environment. I don't see request loss with my test. What's your siege program? Do you see request loss with my test?
$for i in `seq 1 10000`; do curl -s -S http://target_IP/ ; echo ; done
Server hostname: 29c1c2cc3992
Server hostname: 615e83f49237
Server hostname: eb86b578dee9
Server hostname: 29c1c2cc3992

@dongluochen Siege is a http load tester which pretty much does the same as your one-liner -- except it's multithreaded and does way more HTTP requests than cURL would do. This tool is commonly used to simulate load on a website/webapplication and is perfect for checking if HTTP requests are dropped during updates.
Could you try and run your tests using siege? It's installed through apt-get install -y siege for example. Usage is simple: $ siege http://TARGET_IP.
I'll run the same tests using your method on my end as well to see if I see request loss.

@dongluochen I've done some more testing. I can confirm that using cURL doesn't seem to result in errors (I suspect cURL can handle dropped TCP/IP packets to some degree and will simply retry under the hood when necessary). When I continuously refresh a browser window pointing to the target IP, I don't seem to get any connectivity problems as well. In other words, it seems that it's now possible to update webapplications without downtime.
What are your conclusions in this?

@mvdstam When do connection failures happen, when a task is shutdown, when a task is added to load balancer, or any time? Understanding this may help the investigation.

@dongluochen I've done some more testing on my end using siege. It seems the problem occurs more frequently with subsequent rolling updates. The first update pretty much always goes without problems. In this screencast, you can see packets being dropped starting from around the 2:00 mark, and the more containers are being swapped out, the more packets seem to be dropped.
From what I can tell, the packets seem to drop the moment a container is brought into the LB pool (i.e. when it's healthy). You can see it clearly on 2:28, when the top container changes state from starting to running.

@mvdstam Thanks for the detailed analysis! Can you confirm that all your tests are run with healthcheck enabled? It looks removing container from LB is done cleanly, while adding container to LB might still has glitch. I'll try to reproduce it on my side.

@dongluochen You're welcome! All my tests are done with this docker-compose.yml file. So yes, all my tests so far had healthchecks enabled.

As far as I understand this issue is about making new connections being routed to tasks which won't be shut down in the next step.
There might be long running connections, though. Adding a short delay and then shutting down a task with existing connections would still lead to a broken connection.
It would be great if there's a way to check a task for existing connections or "active traffic" and wait for some grace period before cutting those connections. I would extend the suggests flow of https://github.com/docker/docker/issues/30321#issuecomment-274501853 to something like this:

Start rolling update for a service that is replicated with N containers
Do the following in batches of update_parallelism for those N containers:

Remove container from ingress load balancing pool for the respective service so it will not receive any more incoming traffic
if there's still traffic to that task, then wait  delay
Stop that container
Wait for the container to exit (ie give it time to finish any running requests), and send a SIGKILL after 10 seconds
Start new container
If the container has a healthcheck configured:

Wait for the new container to be healthy

Add container to ingress load balancing pool

Rolling update complete

The relevant bit is to wait some time before sending the stop signal. I would suggest to disable that step by default, and allow to configure the delay as part of the service spec or set the delay as flag to a docker service update command.
Do you see that aspect as a separate issue than the one you already discussed here?

Hey @gesellix!
First of all, I believe you're mistaking service (an n-amount of containers, where each container is a task) for a task (a single container that belongs to a service). See also this page. When a task (container) is removed from the LB during a rolling update, all incoming traffic is simply redirected to other tasks belonging to that same service (and thus, no connections are being "cut off" or "lost"). A delay between removing the container from the LB and sending the STOPSIGNAL, is therefore not necessary, since the container that receives the STOPSIGNAL has a full 10 seconds to finish up any running processes with the guarantee that no new connections will be sent to that same container.
Finally, in the event of long running connections, such as WebSockets, clients should always be written in a way where timeouts and/or close() events are handled correctly (ie. the connection is re-established when the connection is lost). Servers that are in the process of shutting down handling long-running connections, should inform/advise clients listening on those connections that the connection will be closed, and a new one should be established shortly by the client. Since this is very application-specific, this is well out of the scope of Docker.

Hi @mvdstam, thanks for your feedback.
I would guess that I got the difference between services vs. task right. I'll give you an example where I think that redirecting traffic won't be possible: I'm thinking of file uploads, where the actual task handling the file upload shouldn't be shut down until the request is finished.
Maybe I'm missing something here?

I'm thinking of file uploads, where the actual task handling the file upload shouldn't be shut down until the request is finished.

I can see the use case, but I'm not sure if docker would be able to handle all situations. How would a "regular", non-containerised service handle this in the event of an upgrade / maintenance? Thinking of;

a streaming service, or download
a huge upload on a slow connection, taking days to complete
interactive sessions (websockets), as mentioned

Requiring those connections to complete would potentially lead to never being able to update a service. These situations can be very specific to the use-case. Perhaps this should be regarded something "separate" from docker's responsibility (i.e., if updating an application can lead to short service interruptions, send out an e-mail to customers to inform them on scheduled maintenance).
Not saying its "not a thing", just genuinely wondering what can, or needs to be done (or not).

@thaJeztah you're right that a simple wait for finished connections can lead to never updating a service. That's why I mentioned the wait some time before sending the stop signal aspect. Maybe it would be enough for us to configure the already proposed delay of 2 seconds, so I would suggest to make it configurable from the start.

@dongluochen Were you able to reproduce at your end? Is there anything you need me to do?

any news about this issue?

@gesellix Why wouldn't the process in the container handle this?
ie, catch SIGTERM, wait for connections to be finished, exit.
The one hindrance here is the container will be forcefully killed after some amount of time.... maybe it shouldn't fallback to SIGKILL for upgrades, or provide a way to just pause the upgrade or skip (and re-queue) a task for the upgrade.
If a container does not gracefully shutdown on SIGTERM then what do you expect Docker to do? We adding something like --stop-signal that is currently on docker run... but this is already supported at least from the image viaSTOPSIGNAL <sig> in the dockerfile build.

@cpuguy83 As you already mentioned, we would see the forced SIGKILL as main issue. Is it possible to configure the delay between SIGTERM/SIGKILL like for the docker stop command?
To clarify a bit more, I can answer to @thaJeztah's question with our current way of handling long-running requests:

How would a "regular", non-containerised service handle this in the event of an upgrade / maintenance?

By implementing blue-green deployment combined with a reverse proxy, we route requests to the newly deployed containers/processes. The proxy keeps existing connections alive and we keep the old containers/processes running for a custom duration (e.g. 5 minutes). Stopping and removing them is only triggered, but not an actual step of the deployment. We don't wait for every connection to be finished, because we also don't want old containers to live forever.
You'll see that it's not relevant whether we use containers or plain processes. The difference lies in the way we clean up during or after the service upgrade. The process (containerised or not) doesn't need to know about deployments, and it doesn't need a custom handler for SIGTERM.
We would prefer our process to be as unaware of the orchestrating/deployment aspect as possible, but I admit that it would be an improvement to make our process gracefully handle SIGTERM. Would something like pausing the upgrade for a configurable delay (or continuing with another task, and retry the blocked task later) be possible?

Is it possible to configure the delay between SIGTERM/SIGKILL like for the docker stop command?

There is a --stop-grace-period flag.

Thanks @aaronlehmann, @cpuguy83 - we'll try if that fits our needs.

@thaJeztah @dongluochen @sanimej
Hey guys, it's been almost a month since the last activity regarding this issue. Is this issue still on the radar? Is there something I can do to help you fix this issue?
Thanks! 😄

@mvdstam Sorry for the delay partially due to preparation of DockerCon. It's definitely under our radar. As you already know there are 2 known issues and might be another. The discussion of gossip convergence has been going on for a while. We are working to validate them and find feasible solutions. At the same time we are actively working on end-to-end tests (repo docker/docker-e2e) to make sure we can detect such issue/regression.

@dongluochen Thanks for the reply, I understand. Please let me know if you need me to do any more testing. This is an important issue for me and I'd be more than happy to help you in any way I can.
I see that @thaJeztah removed this from the 17.04 milestone; could you place this issue back at another milestone so we don't forget about this one, and we'd have a general idea about when this potentially could be fixed?

@mvdstam removed it from 17.04 because it didn't make it there, but happy to add to another milestone if we have a possible ETA. 17.06 code-freeze is 2-3 weeks away, and was not sure if all things could expected to be resolved before that ( @dongluochen ?)

@thaJeztah I want to get the changes to 17.06. Most of the changes for this issue do not require API/CLI change so they can get in after freeze. The only change may add configuration is https://github.com/moby/moby/pull/31142.

Thanks 17.06 it is 👍

:+1: looking very much forward to that change for zero-downtime rolling updates

I think we should keep this open. We have identified at least 2 issues in this thread. Gossip convergence delay which is mitigated by #31142. The other issue (https://github.com/moby/moby/issues/30321#issuecomment-283823137) is not resolved yet.

@dongluochen We can open a new issue if the traffic loss during container coming up is reproducible.

We can open a new issue if the traffic loss during container coming up is reproducible.

@mvdstam and I were able to reproduce this problem. It has been a while. I'll retry it on the latest build.

@aaronlehmann @sanimej @dongluochen Indeed -- this issue is not yet fixed as far as I understand and thus shouldn't be closed yet, since rolling updates without packet loss still isn't possible.

I think the loss when a container comes back is related to MAC learning on vxlan. See https://github.com/moby/moby/issues/33076#issuecomment-303849173.

@mvdstam The issue of packet drop to the container that is on the same host is fixed by libnetwork#1792. This fix will be included in Docker 17.06 RC3. Please give it a try when its available.

@sanimej Awesome, I can't wait to try it out. Thanks so much for the effort and getting back to me as well; I'll try 17.06 RC3 when it's available and report back regarding this issue. Would be great if we can close this issue!
Also thanks again to @dongluochen for keeping traction on this issue!

@vieux Shall we keep this issue open until it has been confirmed to be fixed? I haven't been able to take a look at this just yet.

It auto closed because https://github.com/moby/moby/issues/33555 was merged; usually we close if a fix is merged to master, but this issue consists of multiple parts, so let me reopen until we confirmed nothing more is needed

Cheers @thaJeztah!

@sanimej @dongluochen @thaJeztah I've just tested with 17.06-rc3 on a Swarm Cluster consisting of 3 (equal) nodes, again using the same rolling update testsuite. There is still a lot of packet loss when updating, as you can see here. What definitely seems fixed, however, is the major traffic loss that occurred when all services were healthy. From what I can see, traffic gets routed too early to containers not ready for traffic, or to containers that are in the process of shutting down and shouldn't receive new incoming traffic.

@dongluochen @sanimej Can you confirm my findings?

@mvdstam added a fix here docker/libnetwork#1824

@abhinandanpb Great, thanks for the update!

For those stumbling over this issue in a Java/Tomcat/Spring Boot context, I created a proof of concept incorporating the suggestions from @cpuguy83 here: https://github.com/gesellix/graceful-shutdown-spring-boot

Stoking the flame on this topic to see if there is an update. I'm experiencing this as well.

Working with 1 node, 17.06.0-ce, I have made a service that sleeps 10 seconds before exiting when it receives a signal, having a stop-grace-period of 30s.
When I scale from 4 services to 1, services go unhealthy fast, and IPVS rules remain until containers are stopped.
laughing_johnson.1.jrroub6i2ch1@ubuntu    | Wed Aug  2 20:58:01 UTC 2017:we received a signal, sleeping 10 seconds
laughing_johnson.3.r331c9hc36ne@ubuntu    | Wed Aug  2 20:58:01 UTC 2017:we received a signal, sleeping 10 seconds
laughing_johnson.4.jla0we3vc3ql@ubuntu    | Wed Aug  2 20:58:01 UTC 2017:we received a signal, sleeping 10 seconds

Wed Aug 2 20:58:06 UTC 2017 (5 seconds later)
  -> <IP_ADDRESS>:0                 Masq    1      0          0
  -> <IP_ADDRESS>:0                 Masq    1      0          0
  -> <IP_ADDRESS>:0                 Masq    1      0          0
  -> <IP_ADDRESS>:0                 Masq    1      0          0

Real life workload (siege) shows connections refused just after the 10s sleep, when the web server actually terminates. But the stop was requested 10s earlier and IVPS rules should have been updated by this time.
At least, scaling up with proper healthcheck show no issues with this test.
That's far from everything that has been said here. especially regarding convergence time. Am I missing something ?
Test can be found here : https://github.com/hjdr4/docker-scale-downtime

I have tested this patch https://github.com/abhinandanpb/libnetwork/commit/fbb526e4575f782dcf2b1fe5167db9c91f3c2770  from 17.06-dev and it's working as expected : backends get removed when asking to scale down.
Also, I have deleted the previously introduced delay, and it's still working as expected.
Let's hope this patch get released soon so we can start using Swarm in production...

@hjdr4 Sounds great! So no traffic drops at all during and after rolling updates?

I have tested both rolling updates and scale down (both has the same issue in 17.06 release). It's working as expected : 0 error while using Siege.

I played around with Docker Swarm (17.06.0-ce) and I noticed the following behavior:

I deploy an application which waits 60 seconds and returns ok. I use 2 replicas

I connect to this application with curl

I deploy a new version of this application with parallelism: 1 and delay: 10s

As soon as the second container is updated curl returns:
curl: (52) Empty reply from server

Which means that the connection was dropped.

Is there a way to I avoid the dropped connections?
It would be cool if I could specify for how long I want to keep the connection to the old version.
Sometimes the drop happens when the first container was stopped.

Below are required files and instructions how to reproduce it.
Here is the docker-compose.yml I use:
---

version: '3.3'

services:

  app:
    image: czerasz/timeout-service
    ports:
      - 8080:8080
    environment:
      TIMEOUT: ${TIMEOUT}
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/ping"]
      interval: 2s
      timeout: 5s
      retries: 3

I deploy the stack with:
TIMEOUT=60000 docker stack deploy --compose-file docker-compose.yml timeout

I connect to the service with curl:
$ curl '<IP>:8080/'

Then I switch to another terminal and run the next deployment:
TIMEOUT=15000 docker stack deploy --compose-file docker-compose.yml timeout

In the previous tab I see after a while:
curl: (52) Empty reply from server

Here are more details about the Swarm Cluster:
$ docker info
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 14
Server Version: 17.06.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 70
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: xcg9ufx4w6db55eiflpboqdud
Is Manager: true
ClusterID: 1di4f2m2lrboh5pldv0qnlcq1
Managers: 1
Nodes: 3
Orchestration:
 Task History Retention Limit: 5
Raft:
 Snapshot Interval: 10000
 Number of Old Snapshots to Retain: 0
 Heartbeat Tick: 1
 Election Tick: 3
Dispatcher:
 Heartbeat Period: 5 seconds
CA Configuration:
 Expiry Duration: 3 months
 Force Rotate: 0
Root Rotation In Progress: false
Node Address: <IP>
Manager Addresses:
 <IP>:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
seccomp
 Profile: default
Kernel Version: 4.9.0-3-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.957GiB
Name: docker-swarm-01
ID: 54Q3:XTRK:3OJI:M6A3:ZCL4:VBMU:6RQ7:WCKY:TZCA:XKY7:BHCU:XQJT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: czerasz
Registry: https://index.docker.io/v1/
Labels:
provider=digitalocean
Experimental: false
Insecure Registries:
<IP_ADDRESS>/8
Live Restore Enabled: false

WARNING: No swap limit support

Looks there is an official flag for this see: https://github.com/moby/moby/issues/34111
This will be our friend (hopefully soon!)
docker service update \
--update-order=start-first \
...

@czerasz As for how long the old version of the service sticks around, you've got the services.app.stop_grace_period key (--stop-grace-period flag to docker service create) and, unfortunately not (yet?) in docker stack deploy with a compose file, the --stop-signal flag to docker service create.
So if you make your service respond gracefully to SIGTERM or whatever you configured as a stop signal, you can allow old connections to keep the old service versions' tasks/containers running while they service the long-running requests - provided that the fix for this issue is implemented, that is.

Hey @hjdr4 in your comment you said that no request is lost. Do you know when this fix is going to be released? We are using 17.06.2 and in this version, we see requests lost while testing with JMeter
Thanks!

I don't work at Docker. I just tested a variation of @abhinandanpb code.
While it works with my simple test (updating and scaling on one node, which is currently broken), it looks like it has a good chance to break stuff in some
obscure circumstances.
Code and docs are so poorly written that I did'nt go further, I let the libnetwork guys deal with it.
PR https://github.com/docker/libnetwork/pull/1824 has been saying "soon" for months. 17.09-rc still does not have a patch.
Updating and scaling are broken, but it looks like they are not a priority. Some comments suggest that even enterprise support customers don't have solutions.
If you need to run production stuff, Traefik may help for HTTP workload.

Thanks, @hjdr4!
We will wait until swarm provides the solution in future releases.
In our case, we have detected that some requests are lost when the tasks are stopped in the update process ( scale down ).
We hope that docker solve it "soon"

@hjdr4 @sirlatrom @thaJeztah @abhi
Docker 17.10 seems to fix a lot of network issues. For instance, I'm not seeing major packet loss anymore after a service has been updated, However, it's still not possible as of this time of writing to perform rolling updates without packet loss. During rolling updates, there still are many connection refused messages popping up as described earlier in this thread.
https://github.com/docker/libnetwork/pull/1824 seemed to fix that particular problem, but seems to be abandoned completely. What can we do to get this issue to get more traction?

Do we have any timeline for this fix?

Hey guys, any idea when this issue will be fixed?

Still its not solving the zero downtime deployment issue. When new version is deployed, connections to old container get cut off

@anojansivarajah what are you expecting? Maybe you can do it in-app or with proxy layer.

What i expect is when a new version is deployed,
all new connections go to the new container
existing connections to old container are not interrupted

Maybe you can do it in-app or with proxy layer.
It will increase time to get response?

OK but what if it's a websocket connection or a persistent DB connection?
I'm just drilling in because at some point, you have to cut off connections if they are persistent... so if you gave me a specific use case we might be able to solve it without relying on the ingress VIP (which is I assume how you're expecting swarm networking to handle it).
Things like https://github.com/stevvooe/sillyproxy or --stop-grace-period plus --stop-signal plus --update-order could do the same thing.

I've done this for half an hour straight, and there was zero packet loss!

Great news, thanks!!

SIGINT (which is the default signal sent by the Docker daemon when telling a service to stop)

Just wanted to chime in to avoid confusion that the default signal sent is in fact SIGTERM.

@sirlatrom You're absolutely right. For the sake of avoiding confusion, I've updated my earlier comment.

Hey guys, Have you tried with streaming connections?

@mvdstam Your test correctly tests the bug of ingress routing new connections to the container that is in process of being shut down. This seems to be fixed now, however, what we see is a problem of ingress network cutting off existing streaming connections.
We've modified your test case to include a streaming test here: https://github.com/YarekTyshchenko/docker-rolling-upgrade-test/tree/streams
The container is correctly configured to gracefully shut down, meaning it will wait indefinitely for the stream to finish (in our case infinite stream is finished when the client cuts the connection).
I have tested this without swarm, simply doing a docker stop -t 9999 <container> and the streaming connection continues working until I ^C curl.

@YarekTyshchenko This issue was brought up earlier by @thaJeztah in https://github.com/moby/moby/issues/30321#issuecomment-291427406:

Requiring those connections to complete would potentially lead to never being able to update a service. These situations can be very specific to the use-case. Perhaps this should be regarded something "separate" from docker's responsibility (i.e., if updating an application can lead to short service interruptions, send out an e-mail to customers to inform them on scheduled maintenance).

In my opinion, this is a separate problem that is tackled per use case and not specific to Docker. You can hook into the shutting-down process of containers (again, by listening to the shutdown signal which gets sent to containers upon shutting down) and handle application-specific logic (such as shutting down long-running connections, forcing clients to reconnect to other healthy containers for that task in the LB pool) yourself.

@mvdstam I was under the impression that the time swarm waits before cutting the connection is configurable by stop_grace_period, which would give an upper bound to how long it would take to update the service.
If stop_grace_period is not honoured in this case, then I'd like to know what is the current timeout? If its something that isn't configured somewhere and comes from libnetwork internals then I see it cutting off genuine connections, your test case might just serve requests too fast.
Our usecase isn't with some custom streaming protocol, its simply that some of our web pages take several seconds to transfer data to the client (think data exports) which is an intended use case.

I still appear to be getting downtime with 502 errors from Swarm when doing rolling updates between two components with an embedded server component and healthcheck.
I set up the UpdateConfig to use a start-first strategy so that the newer version of the service is started first and then the old one is closed after.
I upgraded to 18.03.0-ce which should have the fix. Is the code change in the client code or the server code? I create the service programmatically [0]
[0] https://github.com/openfaas/faas-swarm/blob/master/handlers/update.go#L82
Thanks,
Alex

I think all things for this are now resolved; let me close this: thank you all!

This is still a problem for me. It does work properly (first removes the container from the ingress load balancer and only after that sends a SIGTERM signal to the application) locally (when the whole swarm runs on one node) but doesn't work when it is distributed among multiple nodes. I'm using Docker version 18.09.6, build 481bc77 on Ubuntu 18.04.2 LTS

@stafichuk if you have more details and steps to reproduce, could you open a new ticket for that?

It's been a while since I saw this issue! I confirm the rolling upgrades are working perfectly on my side.

Just to clarify, do the fixes for this issue also supposed to solve service to service packet losses during rolling upgrades? While accessing a service via ingress indeed works fine, I'm seeing Connection reset by peer errors between services within swarm.
$ docker run --rm -it --network=testing_mynet1 alpine
/ # apk add apache2-utils
...
/ # ab -k -c 5 -n 300000 testing_exp:8080/call
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking testing_exp (be patient)
Completed 30000 requests
Completed 60000 requests
Completed 90000 requests
apr_socket_recv: Connection reset by peer (104)
Total of 113672 requests completed

$ docker version
Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:39 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:13:06 2019
  OS/Arch:          linux/amd64
  Experimental:     true

@gsserge No, and theres no way to do that. When addressing an individual service explicitly who do you expect to answer when you shut it down?

@gsserge How is your service "testing_exp" defined? Could you provide some more information?


[BOT PR] to update dependencies of istio

This PR will be merged automatically once checks are successful.
none

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: istio-testing
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: rshriram
If they are not already assigned, you can assign the PR to them by writing /assign @rshriram in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@istio-testing: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
6b8291d361632e1d6293f22a07b7bf2291752c09
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
6b8291d361632e1d6293f22a07b7bf2291752c09
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
6b8291d361632e1d6293f22a07b7bf2291752c09
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


fix(github): unthemed mentions

🔧 What does this fix? 🔧
as said in the title

🗒 Checklist 🗒

[X] I have read and followed Catppuccin's contributing guidelines.
[X] I have updated the version appropriately in the ==UserStyle== header of the catppuccin.user.css file.

As per the lint error that variable is already themed and themed correctly.

As per the lint error that variable is already themed and themed correctly.
i guess i'll search for duplicate variables the next time i contribute


Navbar Icon looks good but it can be more stylish

Is your feature request related to a problem? Please describe.
the icons on navbar is preety good but it can be made more stylish
Describe the solution you'd like.
No response
Describe alternatives you've considered.
No response
Additional context.
No response
Show us the magic with screenshots
No response
Checklist

[X] I have checked the Existing Issues
[X] I have read the Contributing Guidelines
[X] I want to work on this issue. (optional)

@aditya-bhaumik plz assign me this


[ALLUXIO-3251] Fix journal association and add test

https://alluxio.atlassian.net/browse/ALLUXIO-3251

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/20692/
Test PASSed.


Add more Date Recognizer Cases

Change Description
Add more cases for the date recognizer, including shorter MMM patterns and two digit patterns for years.
Checklist

[x] I have reviewed the contribution guidelines
[x] I have signed the CLA
[x] My code includes unit tests
[x] All unit tests and lint checks pass locally
[x] My PR contains documentation updates / additions if required

@omri374 Would be happy for a re-approval.
By the way, we have already documented DATE_TIME as 'Pattern match and context' detection Method

Will push a commit on (1) and (2) in a few minutes

@omri374 done


FreeBSD jobs occassionally fail when ejecting the disk

In the last few weeks I've been observing FreeBSD jobs failing when trying to eject a disk, with the error being that the resource is busy. https://github.com/inko-lang/inko/actions/runs/6520144623/job/17707484871 is one such job where this happened:
2023-10-14T22:02:10.0083480Z Waiting for partitions to activate
2023-10-14T22:02:10.0365870Z Formatting disk2s1 as MS-DOS (FAT32) with name RES
2023-10-14T22:02:20.8279870Z 512 bytes per physical sector
2023-10-14T22:02:20.8280650Z /dev/rdisk2s1: 76594 sectors in 76594 FAT32 clusters (512 bytes/cluster)
2023-10-14T22:02:20.8282080Z bps=512 spc=1 res=32 nft=2 mid=0xf8 spt=32 hds=16 hid=2048 drv=0x80 bsec=77824 bspf=599 rdcl=2 infs=1 bkbs=6
2023-10-14T22:02:20.9370120Z Mounting disk
2023-10-14T22:02:24.9493860Z Finished partitioning on disk2
2023-10-14T22:02:25.0276190Z [command]/usr/bin/sudo umount /Volumes/RES
2023-10-14T22:02:26.1238370Z [command]/usr/bin/hdiutil detach /dev/disk2
2023-10-14T22:02:27.0086020Z hdiutil: couldn't eject "disk2" - Resource busy
2023-10-14T22:02:27.0565870Z
2023-10-14T22:02:27.0569630Z /Users/runner/work/_actions/cross-platform-actions/action/v0.19.1/webpack:/cross-platform-action/node_modules/@actions/exec/lib/toolrunner.js:574
2023-10-14T22:02:27.0571900Z                 error = new Error(`The process '${this.toolPath}' failed with exit code ${this.processExitCode}`);
2023-10-14T22:02:27.0573230Z ^
2023-10-14T22:02:27.0573860Z Error: The process '/usr/bin/hdiutil' failed with exit code 16
2023-10-14T22:02:27.0576500Z     at ExecState._setResult (/Users/runner/work/_actions/cross-platform-actions/action/v0.19.1/webpack:/cross-platform-action/node_modules/@actions/exec/lib/toolrunner.js:574:1)
2023-10-14T22:02:27.0579510Z     at ExecState.CheckComplete (/Users/runner/work/_actions/cross-platform-actions/action/v0.19.1/webpack:/cross-platform-action/node_modules/@actions/exec/lib/toolrunner.js:557:1)
2023-10-14T22:02:27.0584190Z     at ChildProcess.<anonymous> (/Users/runner/work/_actions/cross-platform-actions/action/v0.19.1/webpack:/cross-platform-action/node_modules/@actions/exec/lib/toolrunner.js:451:1)
2023-10-14T22:02:27.0586440Z     at ChildProcess.emit (node:events:513:28)
2023-10-14T22:02:27.0587160Z     at maybeClose (node:internal/child_process:1100:16)
2023-10-14T22:02:27.0588120Z     at Socket.<anonymous> (node:internal/child_process:458:11)
2023-10-14T22:02:27.0588840Z     at Socket.emit (node:events:513:28)
2023-10-14T22:02:27.0589440Z     at Pipe.<anonymous> (node:net:301:12)

A restart of the job typically fixes it, though sometimes a second or third retry may be necessary. I'm not sure yet how to reliably reproduce it, as it happens sporadically.

I've started to see this myself as well. Also for OpenBSD. Not sure what to do about it. Catch the error and try the command again perhaps.

Looks like it was possible to fix/mitigate this issue by retrying: https://github.com/cross-platform-actions/action/actions/runs/6749533591/job/18350090001#step:3:236. Fixed in https://github.com/cross-platform-actions/action/releases/tag/v0.21.1.

@jacob-carlborg Thanks! I'll give it a try! :tada:


General documentation and document configuration options

Document configuration options - how to configure the service and possibly add some documentation for how to use the service as well.
If one can utilize the /redoc or /docs endpoints for this, it should become easier to document the API in general.
An example could be to build the /redoc webpage, create a new branch (gh-pages) that includes only the built files and have that as the documentation.

Perhaps something like jekyll-openapi (see rendered for test openapi file) would be nice?
Here's also a nice list of possible services to use to document OpenAPI files.


⚠ Bofrost.at has degraded performance

In 2706bd3, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1428 ms

Resolved: Bofrost.at performance has improved in c72a231 after 9 minutes.


Can not build avatar with AAO version 1.7.5~ imported

英語を使ってごめんなさい
When I import AAO: Avatar Optimizer (after version 1.7.5),build avatar will fail.
And I need to change AAO's version to 1.7.4 to upload my avatar.
I use new unity project and only import liltoon, AAO and avatar's package.
Unity log by ConsoleLogSaver : https://67.gigafile.nu/1112-c067011b00db5b5023a29c05c6f64bda3

There's no problem with English.
The main error is the following error and it looks bug of burst compiler, C# compiler, or Unity.
Could you try reinstalling Unity?
Processing assembly Library/Bee/artifacts/1900b0aP.dag/com.anatawa12.avatar-optimizer.runtime.dll, with 109 defines and 201 references
processors: Unity.Jobs.CodeGen.JobsILPostProcessor, zzzUnity.Burst.CodeGen.BurstILPostProcessor
running Unity.Jobs.CodeGen.JobsILPostProcessor
running zzzUnity.Burst.CodeGen.BurstILPostProcessor
zzzUnity.Burst.CodeGen.BurstILPostProcessor: ILPostProcessor has thrown an exception: System.InvalidOperationException: Internal compiler error for Burst ILPostProcessor on com.anatawa12.avatar-optimizer.runtime. Exception: System.BadImageFormatException: Format of the executable (.exe) or library (.dll) is invalid.
   at Mono.Cecil.PE.ImageReader.ReadImage()
   at Mono.Cecil.PE.ImageReader.ReadImage(Disposable`1 stream, String file_name)
   at Mono.Cecil.ModuleDefinition.ReadModule(Stream stream, ReaderParameters parameters)
   at Mono.Cecil.AssemblyDefinition.ReadAssembly(Stream stream, ReaderParameters parameters)
   at zzzUnity.Burst.CodeGen.AssemblyResolver.LoadFromStream(Stream peStream, Stream pdbStream, ISymbolReaderProvider customSymbolReader)
   at zzzUnity.Burst.CodeGen.BurstILPostProcessor.Process(ICompiledAssembly compiledAssembly)
   at zzzUnity.Burst.CodeGen.BurstILPostProcessor.Process(ICompiledAssembly compiledAssembly)
   at Unity.ILPP.Runner.PostProcessingPipeline.PostProcessAssemblyAsync(PostProcessAssemblyRequest request, Action`2 progressSink)
PostProcessing failed: System.InvalidOperationException: Internal compiler error for Burst ILPostProcessor on com.anatawa12.avatar-optimizer.runtime. Exception: System.BadImageFormatException: Format of the executable (.exe) or library (.dll) is invalid.
   at Mono.Cecil.PE.ImageReader.ReadImage()
   at Mono.Cecil.PE.ImageReader.ReadImage(Disposable`1 stream, String file_name)
   at Mono.Cecil.ModuleDefinition.ReadModule(Stream stream, ReaderParameters parameters)
   at Mono.Cecil.AssemblyDefinition.ReadAssembly(Stream stream, ReaderParameters parameters)
   at zzzUnity.Burst.CodeGen.AssemblyResolver.LoadFromStream(Stream peStream, Stream pdbStream, ISymbolReaderProvider customSymbolReader)
   at zzzUnity.Burst.CodeGen.BurstILPostProcessor.Process(ICompiledAssembly compiledAssembly)
   at zzzUnity.Burst.CodeGen.BurstILPostProcessor.Process(ICompiledAssembly compiledAssembly)
   at Unity.ILPP.Runner.PostProcessingPipeline.PostProcessAssemblyAsync(PostProcessAssemblyRequest request, Action`2 progressSink)
   at Unity.ILPP.Runner.PostProcessingService.PostProcessAssembly(PostProcessAssemblyRequest request, IServerStreamWriter`1 responseStream, ServerCallContext context)
Unhandled Exception: System.InvalidOperationException: Post processing failed
   at Unity.ILPP.Trigger.TriggerApp.<ProcessArgumentsAsync>d__1.MoveNext() + 0xf74
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
   at Unity.ILPP.Trigger.TriggerApp.<ProcessArgumentsAsync>d__1.MoveNext() + 0x1149
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb6
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task) + 0x42
   at Unity.ILPP.Trigger.TriggerApp.<RunAsync>d__0.MoveNext() + 0xc7
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb6
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task) + 0x42
   at Program.<<Main>$>d__0.MoveNext() + 0x1a3
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x20
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xb6
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task) + 0x42
   at Program.<Main>(String[]) + 0x20
   at Unity.ILPP.Trigger!<BaseAddress>+0x47849b

I try to reinstall unity, VCC and repair visual studio tool, it still can't work.
And I update burst from 1.8.12 to 1.8.17, the problem is solved.
Thank you for your help.

Oh it was burst side of problem...
It might be good to clear global upm cache since burst compiler of your global cache might be broken: https://docs.unity3d.com/2022.3/Documentation/Manual/upm-cache.html


bump to 3.1-snapshot

Description
Please list all relevant dependencies in this section and provide summary of the change, motivation and context.
Fixes # (issue)
Type of change

[ ] Bugfix
[ ] Feature

Checklist

[ ] I have read the contribution guide: https://vaadin.com/docs/latest/guide/contributing/overview/
[ ] I have added a description following the guideline.
[ ] The issue is created in the corresponding repository and I have referenced it.
[ ] I have added tests to ensure my change is effective and works as intended.
[ ] New and existing tests are passing locally with my change.
[ ] I have performed self-review and corrected misspellings.

Additional for Feature type of change

[ ] Enhancement / new feature was discussed in a corresponding GitHub issue and Acceptance Criteria were created.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.