Duplicate containers with agent endpoint

Bug description
Sometimes (not always) the Containers page has duplicate entries for many containers. Not every container is duplicated. If I reload the page several times, the problem occurs for some page reloads only.
The links are also identical except for ?nodeName=... in the querystring. So Portainer thinks the same container (with the same task-id/slot) has been deployed to multiple nodes in the swarm.
When I click on one, I get the expected container detail page. When I click on another (with a different ?nodeName=... in the querystring, I get an error about being unable to retrieve the container.
Expected behavior
One entry in the list per container.
Steps to reproduce the issue:
Happens intermittently. When it happens, page refresh might or might not fix it. Even if it does, a few more reloads will show the problem again.
 Might be related, but still seems buggy. I would have thought that if an agent is unable to successfully execute Docker CLI/API due to constraints, it wouldn't show any containers from the affected node, or would show an error (unhealthy agent)?

@deviantony I spoke too soon. It just happened again, and I noticed right after updating/redeploying a stack. Though all containers are duplicated, not just the ones that were recreated by the update.
Here's my portainer log:
2018/05/08 10:43:25 Instance already has defined endpoints.

I'm not able to reproduce this issue, either with a cluser with manager/workers or full managers.
Could you retry with the latest development version of Portainer (portainer/portainer:develop), I believe I've fixed the following panic error:

2018/05/08 10:44:53 http: panic serving <IP_ADDRESS>:35034: interface conversion: interface {} is map[string]interface {}, not []interface {},

Could you give me instructions on how to reproduce your environment? I believe that your env is using AWS, any chance to tell me how to setup a similar env so that I can try to reproduce?

I'm seeing this problem also with version 1.17.0 on two swarms that I control. Let me know if I can test something.

@Duvel what's the clusters topology? Could you share Portainer/agent logs?

portainer:
2018/05/11 13:22:04 Instance already has defined endpoints. Skipping the endpoint defined via CLI.
2018/05/11 13:22:04 Starting Portainer 1.17.0 on :9000
2018/05/11 13:23:00 http error: Get https://tasks.portainer_agent/_ping: dial tcp <IP_ADDRESS>:443: getsockopt: connection refused (code=500)

agent:
2018/05/11 13:15:56 [INFO] serf: EventMemberJoin: 49beaf1a58ce <IP_ADDRESS>,
2018/05/11 13:15:56 [INFO] - Starting Portainer agent version 1.0.0 on <IP_ADDRESS>:9001,
2018/05/11 13:15:56 [INFO] serf: EventMemberJoin: b6003670c089 <IP_ADDRESS>,
2018/05/11 13:15:57 [INFO] serf: EventMemberJoin: 2f961ca89b4b <IP_ADDRESS>,
2018/05/12 00:49:50 [WARN] memberlist: Refuting a suspect message (from: b6003670c089),
2018/05/11 13:15:57 [INFO] serf: EventMemberJoin: 2f961ca89b4b <IP_ADDRESS>,
2018/05/11 13:15:57 [INFO] serf: EventMemberJoin: 49beaf1a58ce <IP_ADDRESS>,
2018/05/11 13:15:57 [INFO] serf: EventMemberJoin: b6003670c089 <IP_ADDRESS>,
2018/05/11 13:15:57 [INFO] - Starting Portainer agent version 1.0.0 on <IP_ADDRESS>:9001,
2018/05/11 13:15:56 [INFO] serf: EventMemberJoin: b6003670c089 <IP_ADDRESS>,
2018/05/11 13:15:56 [INFO] serf: EventMemberJoin: 49beaf1a58ce <IP_ADDRESS>,
2018/05/11 13:15:56 [INFO] - Starting Portainer agent version 1.0.0 on <IP_ADDRESS>:9001,
2018/05/11 13:15:57 [INFO] serf: EventMemberJoin: 2f961ca89b4b <IP_ADDRESS>,
2018/05/12 00:49:50 [INFO] memberlist: Suspect 49beaf1a58ce has failed, no acks received,

What exact information do you need for the topology?

@Duvel number of nodes in the cluster (workers/manager count as well).

Ah, one cluster has one manager and two worker nodes, the other has one manager and one node. The first cluster is deployed on Ubuntu Server 16.04, the second on HypriotOS 1.8.

I think it happens when I reload the page, when it is on the container overview after a day of not using portainer. If I do a second reload all is normal again. It looks like that all containers running on the manager get duplicated to the worker node.
Just trying to give some info to try to succesfully reproduce this one...

@Duvel thanks for the update, really appreciate it, I'm unable to reproduce so far. Anything new in the agent logs?

The logs are scaringly empty. My best guess at the moment would be a caching issue on the browser side, but I haven't been able to have a browser running with developer tools enabled.

@Duvel I've added some debugging and update the image portainer/agent:develop (linux amd64 only), you can enable it by setting the environment variable LOG_LEVEL=debug when starting the agent.

Could you also do an ARM build? I won't be seeing the amd64 cluster for a week.

I'm on it.

@Duvel
Done, just pushed portainer/agent:develop-arm64 and portainer/agent:develop-arm. Do not forget to start it with the LOG_LEVEL=debug environnment variable set.

Just give it a try, but the arm build doesn't seem to work: "unsupported platform on 1 node".

Nope, the image is tagged with "Architecture": "amd64".

The architecture represents the arch of the building platform, I'm pretty sure I build these 2 images for arm & arm64. Let me retry.

Yup, I'm pretty sure :( I forced the deployment with resolve image never. Let's not go in these architecture things for now, as they are quit difficult...
But some good (or bad?) new, I immediately saw double containers and the logging is up!
2018/05/19 18:46:53 [DEBUG] - Using agent port: 9001
2018/05/19 18:46:53 [DEBUG] - Using cluster address: tasks.portainer_agent
2018/05/19 18:46:53 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>
2018/05/19 18:46:53 [DEBUG] - Generating TLS files...
2018/05/19 18:47:29 [DEBUG] - Agent details: map[NodeRole:manager AgentPort:9001 NodeName:pi]
2018/05/19 18:47:32 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>
2018/05/19 18:47:32 [INFO] serf: EventMemberJoin: 8c7e8525d586 <IP_ADDRESS>
2018/05/19 18:47:32 [DEBUG] - Will join cluster via: tasks.portainer_agent
2018/05/19 18:47:32 [INFO] serf: EventMemberJoin: 0d6cd4f4494d <IP_ADDRESS>
2018/05/19 18:47:32 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001
2018/05/19 18:48:02 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8
2018/05/19 18:48:02 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15
2018/05/19 18:48:02 [DEBUG] - Aggregated resource count: 23
2018/05/19 18:48:15 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8
2018/05/19 18:48:15 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 21
2018/05/19 18:48:15 [DEBUG] - Aggregated resource count: 29

The logging of the worker node:
2018/05/19 18:46:51 [DEBUG] - Using agent port: 9001,
2018/05/19 18:46:51 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/19 18:46:51 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/19 18:46:51 [DEBUG] - Generating TLS files...,
2018/05/19 18:47:10 [DEBUG] - Agent details: map[NodeName:node1 NodeRole:worker AgentPort:9001],
2018/05/19 18:47:13 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/19 18:47:13 [INFO] serf: EventMemberJoin: 0d6cd4f4494d <IP_ADDRESS>,
2018/05/19 18:47:13 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/19 18:47:13 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/19 18:47:32 [INFO] serf: EventMemberJoin: 8c7e8525d586 <IP_ADDRESS>,
2018/05/19 18:48:02 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 18:48:02 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 18:48:02 [DEBUG] - Aggregated resource count: 23,
2018/05/19 18:48:02 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 18:48:02 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/19 18:48:02 [DEBUG] - Aggregated resource count: 31,
2018/05/19 18:48:15 [DEBUG] - Processing node response for /images/json on node node1, resource count: 9,
2018/05/19 18:48:15 [DEBUG] - Processing node response for /images/json on node pi, resource count: 14,
2018/05/19 18:48:15 [DEBUG] - Aggregated resource count: 23,
2018/05/19 18:48:18 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 21,
2018/05/19 18:48:18 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 18:48:18 [DEBUG] - Aggregated resource count: 29,
2018/05/19 18:48:18 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 18:48:18 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 29,
2018/05/19 18:48:18 [DEBUG] - Aggregated resource count: 37,
2018/05/19 18:53:58 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 18:53:58 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 21,
2018/05/19 18:53:58 [DEBUG] - Aggregated resource count: 29,

Interesting, I'm guessing that the duplicates are all associated to the pi node?

I was just counting them:

8 on node 1
21 on pi

Now I have the idea, that the containers from node 1 are duplicated on pi. Maybe because the agent on pi started first.

pi
2018/05/19 19:25:12 [DEBUG] - Using agent port: 9001,
2018/05/19 19:25:12 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/19 19:25:12 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/19 19:25:12 [DEBUG] - Generating TLS files...,
2018/05/19 19:25:20 [DEBUG] - Agent details: map[AgentPort:9001 NodeName:pi NodeRole:manager],
2018/05/19 19:25:23 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/19 19:25:23 [INFO] serf: EventMemberJoin: 360a06207074 <IP_ADDRESS>,
2018/05/19 19:25:23 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/19 19:25:23 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/19 19:25:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:25:36 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:25:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:25:36 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:25:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:25:36 [DEBUG] - Redirecting request to pi,
2018/05/19 19:25:36 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:25:36 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:25:36 [DEBUG] - Aggregated resource count for /containers/json: 15,
2018/05/19 19:25:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:25:36 [DEBUG] - Redirecting request to pi,
2018/05/19 19:25:37 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:25:37 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:25:37 [DEBUG] - Aggregated resource count for /containers/json: 15,
2018/05/19 19:25:47 [INFO] serf: EventMemberJoin: 0fa33ebae343 <IP_ADDRESS>,
2018/05/19 19:26:05 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:05 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:05 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:26:05 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:05 [DEBUG] - Redirecting request to pi,
2018/05/19 19:26:05 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:05 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:05 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:26:05 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:26:05 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:26:05 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 19:26:05 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:05 [DEBUG] - Redirecting request to pi,
2018/05/19 19:26:05 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:26:06 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:26:06 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:26:06 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:26:06 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 19:26:47 [DEBUG] - Cluster operation for /networks,
2018/05/19 19:26:47 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:47 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /networks,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to pi,
2018/05/19 19:26:48 [DEBUG] - Response size for node pi: 14,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 19:26:48 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /volumes,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to pi,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to pi,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /images/json,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to pi,
2018/05/19 19:26:51 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:51 [DEBUG] - Redirecting request to pi,
2018/05/19 19:28:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:11 [DEBUG] - Redirecting request to pi,
2018/05/19 19:28:12 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:12 [DEBUG] - Redirecting request to pi,
2018/05/19 19:28:32 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:32 [DEBUG] - Redirecting request to pi,
2018/05/19 19:28:32 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:32 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:28:32 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:32 [DEBUG] - Redirecting request to pi,
2018/05/19 19:28:32 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:28:33 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:28:33 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:28:33 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:28:33 [DEBUG] - Aggregated resource count for /containers/json: 23,

node1
2018/05/19 19:25:18 [DEBUG] - Using agent port: 9001,
2018/05/19 19:25:18 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/19 19:25:18 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/19 19:25:18 [DEBUG] - Generating TLS files...,
2018/05/19 19:25:44 [DEBUG] - Agent details: map[NodeName:node1 NodeRole:worker AgentPort:9001],
2018/05/19 19:25:47 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/19 19:25:47 [INFO] serf: EventMemberJoin: 0fa33ebae343 <IP_ADDRESS>,
2018/05/19 19:25:47 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/19 19:25:47 [INFO] serf: EventMemberJoin: 360a06207074 <IP_ADDRESS>,
2018/05/19 19:25:47 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/19 19:26:05 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:05 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:05 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:05 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:47 [DEBUG] - Cluster operation for /images/json,
2018/05/19 19:26:47 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:47 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:47 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:47 [DEBUG] - Cluster operation for /volumes,
2018/05/19 19:26:47 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:47 [DEBUG] - Cluster operation for /networks,
2018/05/19 19:26:47 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /volumes,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:48 [DEBUG] - Cluster operation for /images/json,
2018/05/19 19:26:48 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:48 [DEBUG] - Response size for node node1: 1,
2018/05/19 19:26:48 [DEBUG] - Response size for node node1: 10,
2018/05/19 19:26:48 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:26:48 [DEBUG] - Response size for node pi: 6,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /volumes on node node1, resource count: 1,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /volumes on node pi, resource count: 6,
2018/05/19 19:26:48 [DEBUG] - Aggregated resource count for /volumes: 7,
2018/05/19 19:26:48 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /images/json on node node1, resource count: 10,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /images/json on node pi, resource count: 15,
2018/05/19 19:26:48 [DEBUG] - Aggregated resource count for /images/json: 25,
2018/05/19 19:26:48 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:26:48 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:26:48 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 19:26:51 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:51 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:26:51 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:26:51 [DEBUG] - Redirecting request to node1,
2018/05/19 19:26:51 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:26:52 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:26:52 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:26:52 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:26:52 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 19:28:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:28:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:11 [DEBUG] - Redirecting request to node1,
2018/05/19 19:28:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:28:11 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:28:11 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:28:11 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:28:11 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 19:28:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:28:12 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:12 [DEBUG] - Redirecting request to node1,
2018/05/19 19:28:12 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:28:12 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:28:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:28:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:28:12 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 19:28:32 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:32 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:28:32 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:28:32 [DEBUG] - Redirecting request to node1,
2018/05/19 19:28:33 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:28:33 [DEBUG] - Response size for node node1: 23,
2018/05/19 19:28:33 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:28:33 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 23,
2018/05/19 19:28:33 [DEBUG] - Aggregated resource count for /containers/json: 38,
2018/05/19 19:30:58 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:30:58 [DEBUG] - Scheduling cluster operation,
2018/05/19 19:30:58 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 19:30:58 [DEBUG] - Redirecting request to node1,
2018/05/19 19:30:58 [DEBUG] - Response size for node pi: 15,
2018/05/19 19:30:58 [DEBUG] - Response size for node node1: 8,
2018/05/19 19:30:58 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 19:30:58 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 19:30:58 [DEBUG] - Aggregated resource count for /containers/json: 23,

2018/05/19 19:59:11 [DEBUG] - Using agent port: 9001,
2018/05/19 19:59:11 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/19 19:59:11 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/19 19:59:11 [DEBUG] - Generating TLS files...,
2018/05/19 19:59:25 [DEBUG] - Agent details: map[NodeRole:worker AgentPort:9001 NodeName:node1],
2018/05/19 19:59:28 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/19 19:59:28 [INFO] serf: EventMemberJoin: 566cf2976e5b <IP_ADDRESS>,
2018/05/19 19:59:28 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/19 19:59:28 [INFO] serf: EventMemberJoin: 62791ed819b4 <IP_ADDRESS>,
2018/05/19 19:59:29 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:10 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:10 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:10 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:10 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 12,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node pi: 17,
2018/05/19 20:00:11 [DEBUG] - Processing node response for /images/json on node node1, resource count: 12,
2018/05/19 20:00:11 [DEBUG] - Processing node response for /images/json on node pi, resource count: 17,
2018/05/19 20:00:11 [DEBUG] - Aggregated resource count for /images/json: 29,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 14,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 6,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /volumes on node node1, resource count: 1,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /volumes on node pi, resource count: 6,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /volumes: 7,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 14,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 23,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 23,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:24 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:24 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Response size for node pi: 14,
2018/05/19 20:00:24 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:24 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 20:00:24 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:24 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Response size for node node1: 1,
2018/05/19 20:00:24 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:00:24 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:00:24 [DEBUG] - Response size for node pi: 6,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /volumes on node node1, resource count: 1,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /volumes on node pi, resource count: 6,
2018/05/19 20:00:24 [DEBUG] - Aggregated resource count for /volumes: 7,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:25 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:25 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:25 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:25 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:25 [DEBUG] - Response size for node node1: 12,
2018/05/19 20:00:25 [DEBUG] - Response size for node pi: 17,
2018/05/19 20:00:25 [DEBUG] - Processing node response for /images/json on node node1, resource count: 12,
2018/05/19 20:00:25 [DEBUG] - Processing node response for /images/json on node pi, resource count: 17,
2018/05/19 20:00:25 [DEBUG] - Aggregated resource count for /images/json: 29,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Redirecting request to node1,
2018/05/19 20:01:36 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:01:36 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:01:36 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Redirecting request to node1,
2018/05/19 20:01:36 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:01:36 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:01:36 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:02:01 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:01 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:02:01 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:01 [DEBUG] - Redirecting request to node1,
2018/05/19 20:02:01 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:02:01 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:02:01 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:02:01 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:02:01 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:02:09 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:09 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:02:09 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:09 [DEBUG] - Redirecting request to node1,
2018/05/19 20:02:09 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:02:09 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:02:09 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:02:09 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:02:09 [DEBUG] - Aggregated resource count for /containers/json: 23,

2018/05/19 19:59:11 [DEBUG] - Using agent port: 9001,
2018/05/19 19:59:11 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/19 19:59:11 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/19 19:59:11 [DEBUG] - Generating TLS files...,
2018/05/19 19:59:25 [DEBUG] - Agent details: map[NodeRole:worker AgentPort:9001 NodeName:node1],
2018/05/19 19:59:28 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/19 19:59:28 [INFO] serf: EventMemberJoin: 566cf2976e5b <IP_ADDRESS>,
2018/05/19 19:59:28 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/19 19:59:28 [INFO] serf: EventMemberJoin: 62791ed819b4 <IP_ADDRESS>,
2018/05/19 19:59:29 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:10 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:10 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:10 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:10 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:10 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 12,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:11 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:11 [DEBUG] - Response size for node pi: 17,
2018/05/19 20:00:11 [DEBUG] - Processing node response for /images/json on node node1, resource count: 12,
2018/05/19 20:00:11 [DEBUG] - Processing node response for /images/json on node pi, resource count: 17,
2018/05/19 20:00:11 [DEBUG] - Aggregated resource count for /images/json: 29,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 14,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 6,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /volumes on node node1, resource count: 1,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /volumes on node pi, resource count: 6,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /volumes: 7,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 14,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 23,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:00:12 [DEBUG] - Response size for node pi: 23,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:12 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/19 20:00:12 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:24 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:24 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Response size for node pi: 14,
2018/05/19 20:00:24 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:24 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /networks on node pi, resource count: 14,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /networks on node node1, resource count: 8,
2018/05/19 20:00:24 [DEBUG] - Aggregated resource count for /networks: 22,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:24 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Response size for node node1: 1,
2018/05/19 20:00:24 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:00:24 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:00:24 [DEBUG] - Response size for node pi: 6,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /volumes on node node1, resource count: 1,
2018/05/19 20:00:24 [DEBUG] - Processing node response for /volumes on node pi, resource count: 6,
2018/05/19 20:00:24 [DEBUG] - Aggregated resource count for /volumes: 7,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /networks,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /volumes,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:24 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:00:24 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:25 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:25 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:00:25 [DEBUG] - Cluster operation for /images/json,
2018/05/19 20:00:25 [DEBUG] - Redirecting request to node1,
2018/05/19 20:00:25 [DEBUG] - Response size for node node1: 12,
2018/05/19 20:00:25 [DEBUG] - Response size for node pi: 17,
2018/05/19 20:00:25 [DEBUG] - Processing node response for /images/json on node node1, resource count: 12,
2018/05/19 20:00:25 [DEBUG] - Processing node response for /images/json on node pi, resource count: 17,
2018/05/19 20:00:25 [DEBUG] - Aggregated resource count for /images/json: 29,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Redirecting request to node1,
2018/05/19 20:01:36 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:01:36 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:01:36 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:01:36 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:01:36 [DEBUG] - Redirecting request to node1,
2018/05/19 20:01:36 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:01:36 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:01:36 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:01:36 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:02:01 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:01 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:02:01 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:01 [DEBUG] - Redirecting request to node1,
2018/05/19 20:02:01 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:02:01 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:02:01 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:02:01 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:02:01 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:02:09 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:09 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:02:09 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:02:09 [DEBUG] - Redirecting request to node1,
2018/05/19 20:02:09 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:02:09 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:02:09 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:02:09 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:02:09 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:04:49 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:04:49 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:04:49 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:04:49 [DEBUG] - Redirecting request to node1,
2018/05/19 20:04:49 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:04:49 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:04:49 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:04:49 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:04:49 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:04:49 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:04:49 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:04:49 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:04:49 [DEBUG] - Redirecting request to node1,
2018/05/19 20:04:49 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:04:49 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:04:49 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:04:49 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:04:49 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:05:02 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:02 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:05:02 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:02 [DEBUG] - Redirecting request to node1,
2018/05/19 20:05:02 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:05:03 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:05:03 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:05:03 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:05:03 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:05:47 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:47 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:05:47 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:47 [DEBUG] - Redirecting request to node1,
2018/05/19 20:05:47 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:05:47 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:05:47 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:05:47 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:05:47 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:05:47 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:47 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:05:47 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:47 [DEBUG] - Redirecting request to node1,
2018/05/19 20:05:47 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:05:47 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:05:47 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:05:47 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:05:47 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:05:59 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:59 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:05:59 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:05:59 [DEBUG] - Redirecting request to node1,
2018/05/19 20:05:59 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:05:59 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:05:59 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:05:59 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:05:59 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:06:08 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:06:08 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:06:08 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:06:08 [DEBUG] - Redirecting request to node1,
2018/05/19 20:06:08 [DEBUG] - Response size for node node1: 8,
2018/05/19 20:06:08 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:06:08 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/19 20:06:08 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:06:08 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/19 20:06:08 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:06:08 [DEBUG] - Scheduling cluster operation,
2018/05/19 20:06:08 [DEBUG] - Cluster operation for /containers/json,
2018/05/19 20:06:08 [DEBUG] - Redirecting request to node1,
2018/05/19 20:06:09 [DEBUG] - Response size for node pi: 15,
2018/05/19 20:06:09 [DEBUG] - Response size for node node1: 23,
2018/05/19 20:06:09 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/19 20:06:09 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 23,
2018/05/19 20:06:09 [DEBUG] - Aggregated resource count for /containers/json: 38,

@Duvel
I just updated the portainer/agent:develop-arm image with more debugging. Keen to give it a try and paste the logs? Thanks !

On the first hit it already went wrong:
2018/05/20 07:57:05 [DEBUG] - Using agent port: 9001,
2018/05/20 07:57:05 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/20 07:57:05 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:05 [DEBUG] - Generating TLS files...,
2018/05/20 07:57:21 [DEBUG] - Agent details: map[NodeName:node1 NodeRole:worker AgentPort:9001],
2018/05/20 07:57:24 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:24 [INFO] serf: EventMemberJoin: db9cb4feb12a <IP_ADDRESS>,
2018/05/20 07:57:24 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/20 07:57:24 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/20 07:57:26 [INFO] serf: EventMemberJoin: 8a8ea756b55b <IP_ADDRESS>,
2018/05/20 07:57:26 #########,
2018/05/20 07:57:26 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 07:57:26 #########,
2018/05/20 07:57:26 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 11337,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 3845,
2018/05/20 07:57:27 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 07:57:27 [DEBUG] - Response body length for node node1: 3845,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node node1: 10,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 07:57:27 [DEBUG] - Response body length for node pi: 5507,
2018/05/20 07:57:27 [DEBUG] - Response body length for node pi: 28710,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node pi: 14,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /images/json on node node1, resource count: 10,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /images/json on node pi, resource count: 14,
2018/05/20 07:57:27 [DEBUG] - Aggregated resource count for /images/json: 24,
2018/05/20 07:57:27 #########,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 07:57:27 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 07:57:27 #########,
2018/05/20 07:57:31 #########,
2018/05/20 07:57:31 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 07:57:32 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:32 #########,
2018/05/20 07:57:32 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 07:57:32 [DEBUG] - Proxy response body length: 11337,
2018/05/20 07:57:32 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 07:57:32 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 07:57:32 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:33 [DEBUG] - Proxy response body length: 11337,
2018/05/20 07:57:33 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 07:57:33 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 07:57:33 [DEBUG] - Response body length for node pi: 28710,
2018/05/20 07:57:33 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 07:57:33 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 07:57:33 #########,
2018/05/20 07:57:33 [DEBUG] - Response body length for node pi: 41078,
2018/05/20 07:57:33 [DEBUG] - JSON response size for node pi: 23,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/20 07:57:33 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/20 07:57:33 #########,
2018/05/20 07:58:14 [DEBUG] - Proxy response body length: 5065,
2018/05/20 07:58:14 [DEBUG] - Proxy response body length: 4578,
2018/05/20 07:58:17 [DEBUG] - Proxy response body length: 4649,
2018/05/20 07:58:20 [DEBUG] - Proxy response body length: 4720,

pi
2018/05/20 07:57:05 [DEBUG] - Using agent port: 9001,
2018/05/20 07:57:05 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/20 07:57:05 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:05 [DEBUG] - Generating TLS files...,
2018/05/20 07:57:23 [DEBUG] - Agent details: map[NodeName:pi NodeRole:manager AgentPort:9001],
2018/05/20 07:57:26 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:26 [INFO] serf: EventMemberJoin: 8a8ea756b55b <IP_ADDRESS>,
2018/05/20 07:57:26 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/20 07:57:26 [INFO] serf: EventMemberJoin: db9cb4feb12a <IP_ADDRESS>,
2018/05/20 07:57:26 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to pi,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to pi,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 28710,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 5507,
2018/05/20 07:57:32 [DEBUG] - Redirecting request to pi,
2018/05/20 07:57:33 [DEBUG] - Proxy response body length: 28710,
2018/05/20 08:00:09 [DEBUG] - Proxy response body length: 5065,
2018/05/20 08:00:09 [DEBUG] - Proxy response body length: 1284,
2018/05/20 08:00:13 [DEBUG] - Proxy response body length: 1355,
2018/05/20 08:00:16 [DEBUG] - Proxy response body length: 1426,
2018/05/20 08:00:18 [DEBUG] - Proxy response body length: 1497,
2018/05/20 08:00:21 [DEBUG] - Proxy response body length: 1568,
2018/05/20 08:00:24 [DEBUG] - Proxy response body length: 1639,
2018/05/20 08:00:28 [DEBUG] - Proxy response body length: 1710,
2018/05/20 08:00:31 [DEBUG] - Proxy response body length: 1781,

@Duvel could you paste the logs of the service instead of the logs of each agent? That might help me to correlate the data.
Also, simple question, do you have dupes with other resources (networks, images or volumes) ?

2018/05/20 07:57:05 [DEBUG] - Using agent port: 9001,
2018/05/20 07:57:05 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/20 07:57:05 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:05 [DEBUG] - Generating TLS files...,
2018/05/20 07:57:21 [DEBUG] - Agent details: map[NodeName:node1 NodeRole:worker AgentPort:9001],
2018/05/20 07:57:24 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:24 [INFO] serf: EventMemberJoin: db9cb4feb12a <IP_ADDRESS>,
2018/05/20 07:57:24 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/20 07:57:24 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/20 07:57:26 [INFO] serf: EventMemberJoin: 8a8ea756b55b <IP_ADDRESS>,
2018/05/20 07:57:26 #########,
2018/05/20 07:57:26 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 07:57:26 #########,
2018/05/20 07:57:26 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:05 [DEBUG] - Using agent port: 9001,
2018/05/20 07:57:05 [DEBUG] - Using cluster address: tasks.portainer_agent,
2018/05/20 07:57:05 [DEBUG] - Using advertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:05 [DEBUG] - Generating TLS files...,
2018/05/20 07:57:23 [DEBUG] - Agent details: map[NodeName:pi NodeRole:manager AgentPort:9001],
2018/05/20 07:57:27 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 11337,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 3845,
2018/05/20 07:57:27 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 07:57:27 [DEBUG] - Response body length for node node1: 3845,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node node1: 10,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 07:57:27 [DEBUG] - Response body length for node pi: 5507,
2018/05/20 07:57:27 [DEBUG] - Response body length for node pi: 28710,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node pi: 14,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /images/json on node node1, resource count: 10,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /images/json on node pi, resource count: 14,
2018/05/20 07:57:27 [DEBUG] - Aggregated resource count for /images/json: 24,
2018/05/20 07:57:27 #########,
2018/05/20 07:57:27 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 07:57:27 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 07:57:27 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 07:57:27 #########,
2018/05/20 07:57:31 #########,
2018/05/20 07:57:31 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 07:57:32 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:32 #########,
2018/05/20 07:57:32 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 07:57:32 [DEBUG] - Proxy response body length: 11337,
2018/05/20 07:57:32 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 07:57:32 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 07:57:32 [DEBUG] - Redirecting request to node1,
2018/05/20 07:57:26 [DEBUG] - Serf configured with AdvertiseAddr: <IP_ADDRESS>,
2018/05/20 07:57:26 [INFO] serf: EventMemberJoin: 8a8ea756b55b <IP_ADDRESS>,
2018/05/20 07:57:26 [DEBUG] - Will join cluster via: tasks.portainer_agent,
2018/05/20 07:57:26 [INFO] serf: EventMemberJoin: db9cb4feb12a <IP_ADDRESS>,
2018/05/20 07:57:26 [INFO] - Starting Portainer agent version 1.0.0-dev on <IP_ADDRESS>:9001,
2018/05/20 07:57:33 [DEBUG] - Proxy response body length: 11337,
2018/05/20 07:57:33 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 07:57:33 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 07:57:33 [DEBUG] - Response body length for node pi: 28710,
2018/05/20 07:57:33 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to pi,
2018/05/20 07:57:27 [DEBUG] - Redirecting request to pi,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 07:57:33 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 07:57:33 #########,
2018/05/20 07:57:33 [DEBUG] - Response body length for node pi: 41078,
2018/05/20 07:57:33 [DEBUG] - JSON response size for node pi: 23,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 07:57:33 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 28710,
2018/05/20 07:57:33 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/20 07:57:33 #########,
2018/05/20 07:58:14 [DEBUG] - Proxy response body length: 5065,
2018/05/20 07:58:14 [DEBUG] - Proxy response body length: 4578,
2018/05/20 07:58:17 [DEBUG] - Proxy response body length: 4649,
2018/05/20 07:58:20 [DEBUG] - Proxy response body length: 4720,
2018/05/20 07:58:23 [DEBUG] - Proxy response body length: 4791,
2018/05/20 07:58:26 [DEBUG] - Proxy response body length: 4862,
2018/05/20 07:58:29 [DEBUG] - Proxy response body length: 4933,
2018/05/20 07:59:53 [DEBUG] - Proxy response body length: 5004,
2018/05/20 08:01:50 #########,
2018/05/20 08:01:50 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 08:01:50 #########,
2018/05/20 07:57:27 [DEBUG] - Proxy response body length: 5507,
2018/05/20 07:57:32 [DEBUG] - Redirecting request to pi,
2018/05/20 07:57:33 [DEBUG] - Proxy response body length: 28710,
2018/05/20 08:00:09 [DEBUG] - Proxy response body length: 5065,
2018/05/20 08:00:09 [DEBUG] - Proxy response body length: 1284,
2018/05/20 08:00:13 [DEBUG] - Proxy response body length: 1355,
2018/05/20 08:00:16 [DEBUG] - Proxy response body length: 1426,
2018/05/20 08:00:18 [DEBUG] - Proxy response body length: 1497,
2018/05/20 08:00:21 [DEBUG] - Proxy response body length: 1568,
2018/05/20 08:00:24 [DEBUG] - Proxy response body length: 1639,
2018/05/20 08:01:50 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 08:01:51 [DEBUG] - Redirecting request to node1,
2018/05/20 08:01:51 [DEBUG] - Redirecting request to node1,
2018/05/20 08:01:51 [DEBUG] - Proxy response body length: 11336,
2018/05/20 08:01:51 [DEBUG] - Response body length for node node1: 11336,
2018/05/20 08:01:51 [DEBUG] - Proxy response body length: 3845,
2018/05/20 08:00:28 [DEBUG] - Proxy response body length: 1710,
2018/05/20 08:01:51 [DEBUG] - Response body length for node node1: 3845,
2018/05/20 08:01:51 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:01:51 [DEBUG] - JSON response size for node node1: 10,
2018/05/20 08:01:51 [DEBUG] - Response body length for node pi: 5507,
2018/05/20 08:01:51 [DEBUG] - JSON response size for node pi: 14,
2018/05/20 08:01:51 [DEBUG] - Processing node response for /images/json on node node1, resource count: 10,
2018/05/20 08:01:51 [DEBUG] - Processing node response for /images/json on node pi, resource count: 14,
2018/05/20 08:01:51 [DEBUG] - Aggregated resource count for /images/json: 24,
2018/05/20 08:01:51 #########,
2018/05/20 08:01:51 [DEBUG] - Response body length for node pi: 28709,
2018/05/20 08:01:51 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 08:01:51 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 08:01:51 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 08:01:51 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 08:01:51 #########,
2018/05/20 08:02:07 [DEBUG] - Proxy response body length: 118,
2018/05/20 08:02:08 [DEBUG] - Proxy response body length: 277,
2018/05/20 08:02:08 #########,
2018/05/20 08:02:08 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 08:02:08 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:08 [DEBUG] - Proxy response body length: 3524,
2018/05/20 08:02:08 [DEBUG] - Response body length for node node1: 3524,
2018/05/20 08:02:08 [DEBUG] - JSON response size for node node1: 9,
2018/05/20 08:02:08 [DEBUG] - Response body length for node pi: 5186,
2018/05/20 08:02:08 [DEBUG] - JSON response size for node pi: 13,
2018/05/20 08:02:08 [DEBUG] - Processing node response for /images/json on node node1, resource count: 9,
2018/05/20 08:00:31 [DEBUG] - Proxy response body length: 1781,
2018/05/20 08:02:08 [DEBUG] - Processing node response for /images/json on node pi, resource count: 13,
2018/05/20 08:00:33 [DEBUG] - Proxy response body length: 1852,
2018/05/20 08:00:36 [DEBUG] - Proxy response body length: 1923,
2018/05/20 08:02:08 [DEBUG] - Aggregated resource count for /images/json: 22,
2018/05/20 08:02:08 #########,
2018/05/20 08:02:09 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:09 [DEBUG] - Proxy response body length: 11336,
2018/05/20 08:02:16 [DEBUG] - Proxy response body length: 792,
2018/05/20 08:02:16 [DEBUG] - Proxy response body length: 2397,
2018/05/20 08:02:19 [DEBUG] - Proxy response body length: 49,
2018/05/20 08:02:19 [DEBUG] - Proxy response body length: 758,
2018/05/20 08:02:19 [DEBUG] - Proxy response body length: 2363,
2018/05/20 08:02:22 [DEBUG] - Proxy response body length: 417,
2018/05/20 08:02:22 #########,
2018/05/20 08:02:22 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 08:02:23 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:23 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:23 [DEBUG] - Proxy response body length: 11336,
2018/05/20 08:02:23 [DEBUG] - Proxy response body length: 3058,
2018/05/20 08:02:23 [DEBUG] - Response body length for node node1: 3058,
2018/05/20 08:02:23 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:23 [DEBUG] - Response body length for node pi: 5186,
2018/05/20 08:02:23 [DEBUG] - JSON response size for node pi: 13,
2018/05/20 08:02:23 [DEBUG] - Processing node response for /images/json on node node1, resource count: 8,
2018/05/20 08:02:23 [DEBUG] - Processing node response for /images/json on node pi, resource count: 13,
2018/05/20 08:02:23 [DEBUG] - Aggregated resource count for /images/json: 21,
2018/05/20 08:02:23 #########,
2018/05/20 08:02:30 #########,
2018/05/20 08:02:30 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 08:02:30 #########,
2018/05/20 08:02:30 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 08:02:31 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:31 #########,
2018/05/20 08:02:31 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 08:02:31 [DEBUG] - Proxy response body length: 11336,
2018/05/20 08:02:31 [DEBUG] - Response body length for node node1: 11336,
2018/05/20 08:02:31 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:31 [DEBUG] - Redirecting request to node1,
2018/05/20 08:00:40 [DEBUG] - Proxy response body length: 1994,
2018/05/20 08:02:31 [DEBUG] - Proxy response body length: 3058,
2018/05/20 08:02:31 [DEBUG] - Response body length for node node1: 3058,
2018/05/20 08:02:31 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:31 #########,
2018/05/20 08:02:31 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 08:02:31 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:31 [DEBUG] - Proxy response body length: 11336,
2018/05/20 08:02:31 [DEBUG] - Response body length for node node1: 11336,
2018/05/20 08:01:48 [DEBUG] - Proxy response body length: 2065,
2018/05/20 08:02:31 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:32 [DEBUG] - Response body length for node pi: 28709,
2018/05/20 08:02:32 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 08:02:32 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 08:02:32 #########,
2018/05/20 08:02:32 [DEBUG] - Response body length for node pi: 41076,
2018/05/20 08:02:32 [DEBUG] - Redirecting request to node1,
2018/05/20 08:02:32 [DEBUG] - JSON response size for node pi: 23,
2018/05/20 08:01:51 [DEBUG] - Redirecting request to pi,
2018/05/20 08:01:51 [DEBUG] - Redirecting request to pi,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 23,
2018/05/20 08:02:32 [DEBUG] - Aggregated resource count for /containers/json: 31,
2018/05/20 08:02:32 #########,
2018/05/20 08:02:32 [DEBUG] - Proxy response body length: 3058,
2018/05/20 08:02:32 [DEBUG] - Response body length for node node1: 3058,
2018/05/20 08:02:32 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:32 [DEBUG] - Response body length for node pi: 4720,
2018/05/20 08:02:32 [DEBUG] - JSON response size for node pi: 12,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /images/json on node node1, resource count: 8,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /images/json on node pi, resource count: 12,
2018/05/20 08:02:32 [DEBUG] - Aggregated resource count for /images/json: 20,
2018/05/20 08:02:32 #########,
2018/05/20 08:02:32 [DEBUG] - Response body length for node pi: 8629,
2018/05/20 08:01:51 [DEBUG] - Proxy response body length: 5507,
2018/05/20 08:02:32 [DEBUG] - JSON response size for node pi: 20,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /images/json on node node1, resource count: 8,
2018/05/20 08:01:51 [DEBUG] - Proxy response body length: 28709,
2018/05/20 08:02:32 [DEBUG] - Processing node response for /images/json on node pi, resource count: 20,
2018/05/20 08:02:32 [DEBUG] - Aggregated resource count for /images/json: 28,
2018/05/20 08:02:32 #########,
2018/05/20 08:10:48 #########,
2018/05/20 08:10:48 [DEBUG] - Scheduling cluster request for /volumes,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 3,
2018/05/20 08:02:08 [DEBUG] - Proxy response body length: 118,
2018/05/20 08:02:08 [DEBUG] - Proxy response body length: 277,
2018/05/20 08:02:08 #########,
2018/05/20 08:02:08 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 08:02:08 [DEBUG] - Redirecting request to pi,
2018/05/20 08:10:48 #########,
2018/05/20 08:02:08 [DEBUG] - Proxy response body length: 5186,
2018/05/20 08:10:48 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 2371,
2018/05/20 08:10:48 [DEBUG] - Redirecting request to node1,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 210,
2018/05/20 08:10:48 [DEBUG] - Response body length for node node1: 210,
2018/05/20 08:10:48 [DEBUG] - JSON response size for node node1: 1,
2018/05/20 08:10:48 [DEBUG] - Response body length for node pi: 1758,
2018/05/20 08:10:48 [DEBUG] - JSON response size for node pi: 6,
2018/05/20 08:10:48 [DEBUG] - Processing node response for /volumes on node node1, resource count: 1,
2018/05/20 08:10:48 [DEBUG] - Processing node response for /volumes on node pi, resource count: 6,
2018/05/20 08:10:48 [DEBUG] - Aggregated resource count for /volumes: 7,
2018/05/20 08:10:48 #########,
2018/05/20 08:10:49 [DEBUG] - Redirecting request to node1,
2018/05/20 08:10:49 [DEBUG] - Proxy response body length: 11337,
2018/05/20 08:10:49 [DEBUG] - Response body length for node node1: 11337,
2018/05/20 08:10:49 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:10:49 [DEBUG] - Redirecting request to node1,
2018/05/20 08:10:49 [DEBUG] - Proxy response body length: 3058,
2018/05/20 08:10:49 [DEBUG] - Response body length for node pi: 28710,
2018/05/20 08:10:49 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 08:10:49 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 08:10:49 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 08:10:49 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 08:10:49 #########,
2018/05/20 08:02:09 [DEBUG] - Redirecting request to pi,
2018/05/20 08:02:09 [DEBUG] - Response body length for node node1: 11336,
2018/05/20 08:02:09 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:09 [DEBUG] - Proxy response body length: 28709,
2018/05/20 08:02:09 [DEBUG] - Response body length for node pi: 28709,
2018/05/20 08:02:09 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 08:02:09 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 08:02:09 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 08:02:09 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 08:02:09 #########,
2018/05/20 08:02:22 #########,
2018/05/20 08:02:22 [DEBUG] - Scheduling cluster request for /containers/json,
2018/05/20 08:02:23 [DEBUG] - Redirecting request to pi,
2018/05/20 08:02:23 [DEBUG] - Redirecting request to pi,
2018/05/20 08:02:23 [DEBUG] - Response body length for node node1: 11336,
2018/05/20 08:02:23 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:02:23 [DEBUG] - Proxy response body length: 5186,
2018/05/20 08:02:23 [DEBUG] - Proxy response body length: 28709,
2018/05/20 08:02:23 [DEBUG] - Response body length for node pi: 28709,
2018/05/20 08:02:23 [DEBUG] - JSON response size for node pi: 15,
2018/05/20 08:02:23 [DEBUG] - Processing node response for /containers/json on node node1, resource count: 8,
2018/05/20 08:02:23 [DEBUG] - Processing node response for /containers/json on node pi, resource count: 15,
2018/05/20 08:02:23 [DEBUG] - Aggregated resource count for /containers/json: 23,
2018/05/20 08:02:23 #########,
2018/05/20 08:02:26 [DEBUG] - Proxy response body length: 2397,
2018/05/20 08:02:26 [DEBUG] - Proxy response body length: 792,
2018/05/20 08:02:28 [DEBUG] - Proxy response body length: 49,
2018/05/20 08:02:29 [DEBUG] - Proxy response body length: 2363,
2018/05/20 08:02:29 [DEBUG] - Proxy response body length: 758,
2018/05/20 08:02:30 [DEBUG] - Proxy response body length: 417,
2018/05/20 08:02:31 [DEBUG] - Redirecting request to pi,
2018/05/20 08:02:32 [DEBUG] - Proxy response body length: 28709,
2018/05/20 08:02:32 [DEBUG] - Redirecting request to pi,
2018/05/20 08:02:32 [DEBUG] - Proxy response body length: 4720,
2018/05/20 08:10:28 [DEBUG] - Proxy response body length: 1543,
2018/05/20 08:10:28 [DEBUG] - Proxy response body length: 57669,
2018/05/20 08:10:35 [DEBUG] - Proxy response body length: 3649,
2018/05/20 08:10:35 [DEBUG] - Proxy response body length: 43849,
2018/05/20 08:10:36 [DEBUG] - Proxy response body length: 57669,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 970,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 3649,
2018/05/20 08:10:48 #########,
2018/05/20 08:10:48 [DEBUG] - Scheduling cluster request for /images/json,
2018/05/20 08:10:48 [DEBUG] - Redirecting request to pi,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 1758,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 189,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 2529,
2018/05/20 08:10:48 [DEBUG] - Proxy response body length: 3094,
2018/05/20 08:10:49 [DEBUG] - Redirecting request to pi,
2018/05/20 08:10:49 [DEBUG] - Redirecting request to pi,
2018/05/20 08:10:49 [DEBUG] - Response body length for node node1: 3058,
2018/05/20 08:10:49 [DEBUG] - JSON response size for node node1: 8,
2018/05/20 08:10:49 [DEBUG] - Proxy response body length: 28710,
2018/05/20 08:10:49 [DEBUG] - Proxy response body length: 4720,
2018/05/20 08:10:49 [DEBUG] - Response body length for node pi: 4720,
2018/05/20 08:10:49 [DEBUG] - JSON response size for node pi: 12,
2018/05/20 08:10:49 [DEBUG] - Processing node response for /images/json on node node1, resource count: 8,
2018/05/20 08:10:49 [DEBUG] - Processing node response for /images/json on node pi, resource count: 12,
2018/05/20 08:10:49 [DEBUG] - Aggregated resource count for /images/json: 20,
2018/05/20 08:10:49 #########,
2018/05/20 08:10:51 [DEBUG] - Proxy response body length: 970,
2018/05/20 08:10:51 [DEBUG] - Proxy response body length: 22668,

@Duvel just updated the image to use go 1.10 instead of 1.9, could you give a try and just tell me if it solves the issue?

I'll give it a try this evening.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

remco_portainer_agent.0.w69tzwl8n398@pi    | 2018/05/20 19:19:34 [DEBUG] - Redirecting request to pi
remco_portainer_agent.0.w69tzwl8n398@pi    | 2018/05/20 19:19:34 [DEBUG] - Proxy response body length: 24459
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Proxy response body length: 11336
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Response body length for node node1: 11336
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - JSON response size for node node1: 8
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Response body length for node pi: 24454
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - JSON response size for node pi: 13
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Aggregated resource count for /containers/json: 21
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 #########
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Redirecting request to node1
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Proxy response body length: 11336
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Response body length for node node1: 11336
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - JSON response size for node node1: 8
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Response body length for node pi: 24454
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - JSON response size for node pi: 13
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Aggregated resource count for /containers/json: 21
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 #########
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Response body length for node pi: 36741
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - JSON response size for node pi: 21
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 [DEBUG] - Aggregated resource count for /containers/json: 29
remco_portainer_agent.0.yn8087vhj9op@node1    | 2018/05/20 19:21:22 #########

It should be 8 + 13 = 21.

@Duvel Ok, rollback to 1.9 but I've updated the codebase, keen to tell if the bug is still here?

I've also this issue (swarm on a x4 raspberry pi cluster).

Just getting an update on this, are you still seeing this problem with the latest version of Portainer/Agent?

I think it has improved with the latest version, but in some totals for the swarm I still see wrong number of containers, like the endpoint dashboard and the endpoint picker.

I'm running version 1.20.0 and am experiencing the same issue.  It will duplicate containers, images and volumes from time to time.  When it duplicates resources it will usually report the container or image on multiple nodes.  One of the resources is correct while the other reported resources are ghosts.  Refreshing the screen will usually get rid of them.
I'm running an 11 node swarm with 3 managers and 8 workers.  Believe all the node servers are Ubuntu 16.04.

FWIW I've been seeing this behavior as well, on three (dev/stage/prod) of my Swarms and since the beginning of my company's use of agent.
5/7 managers + 5-20 workers on the three Swarms.
I can fairly easily confirm the behavior by refreshing my endpoint's Dashboard page:

Actually, i have checked (my references above) the real situation with puppet.
docker seems to be OK (except the dead agent containers somewhere on nodes), maybe GUI troubles?
Haven't discovered duplicated agent images or running constrained containers.
Also, situation with agent restarting bothers me. In my mind agent should start once and live forever :)

It is an agent related problem that seems to sometime duplicate resources.
Regarding your situation where the agent keeps restarting, please open a separate issue.

This issue could be related to an agent being registered multiple times in the agent cluster. It might be worth exposing a status API endpoint in the agent to provide useful informations about the agent status (agents in the cluster, signature validation, etc...)

Is there a way to clear and re-register the agents?

@houldsg simply re-deploy the agent via docker service update --force <agent_service_id>

@deviantony Wow. speedy response. Thanks. I did try that. Do I need to restart the portainer container as well?
Maybe the issue is just re-occurring. In my scenario the duplicated rows are from an agent that runs in another datacenter, so maybe the issue could be caused by flakey network connectivity.

@houldsg you don't need to restart the Portainer container.
This issue seems to randomly appear but I'm pretty sure it is related to network issues (an agent being seen multiple times by another agent...).
We have created a specific build allowing us to retrieve details about the state of an agent via the image portainerci/agent:feat87-status-api.
This image expose a public status API over /status on port 9777. If you can give a go to this image, expose port 9777 and issue a GET request to :9777/status when you can reproduce the resource duplication problem it should help us diagnose if the agent is seeing "double agent entries" in the cluster.

@deviantony deploying the feat87 build. I'll let you know when I get any useful info out of it

So it looks like, in my scenario, there is an extra agent running for no apparent reason. The agent service is set to run globally, so there should only ever be one instance on each node. The extra agent was not removed when the service was updated to deploy the feat87 image.

The portainer ui screenshot shows that the agent appearing twice has the same ip but they actually don't (confirmed by the json below from the status endpoint and by actually logging into the node and doing "docker ps")
{
  "cluster": {
    "members": [
      {
        "IPAddress": "<IP_ADDRESS>",
        "Port": "9001",
        "NodeName": "app1245",
        "NodeRole": "manager",
        "EdgeKeySet": false
      },
      {
        "IPAddress": "<IP_ADDRESS>",
        "Port": "9001",
        "NodeName": "app1245",
        "NodeRole": "manager",
        "EdgeKeySet": false
      },
      {
        "IPAddress": "<IP_ADDRESS>",
        "Port": "9001",
        "NodeName": "app1244",
        "NodeRole": "manager",
        "EdgeKeySet": false
      },
      {
        "IPAddress": "<IP_ADDRESS>",
        "Port": "9001",
        "NodeName": "app1246",
        "NodeRole": "manager",
        "EdgeKeySet": false
      }
    ]
  },
  "edge": {
    "enabled": false
  },
  "nodeName": "app1245"
}

@deviantony I haven't removed the extra agent for now. Swarm should have removed it though.

@houldsg yes, exactly what I thought. It seems that there is a "ghost" agent here... or maybe the agent running on that node app1245 has these 2 IPs in the agent overlay network? Can you inspect that agent from the Portainer UI and get more details? See if it has 2 IPs addresses?
Super strange though. I'm still trying to figure out how a single agent could be registered in the same cluster 2 times with different ips.

There are literally 2 agent containers with different ip addresses running on the same node. Swarm should have removed the old one when when it deployed the "feat87-status-api" image. Not sure why it didn't. Maybe network connectivity issues when the deployment was happening.
dev@app1245:~$ docker ps
CONTAINER ID        IMAGE                                 COMMAND     CREATED        STATUS     PORTS                   NAMES
a80e49659cb1        portainerci/agent:feat87-status-api   "./agent"   2 days ago     Up 2 days  <IP_ADDRESS>:9777->9777/tcp  portainer_agent.yxj811im1vhuao5htu88bt854.4tqtmnonsrqepqm786i1dpsr1
bbd097da1cfb        portainer/agent:latest                "./agent"   6 months ago   Up 3 days                          portainer_agent.yxj811im1vhuao5htu88bt854.dnkbpmstnu6vj9vbseo17bqai

@houldsg ah ! Yes, seems that there is a Swarm issue here. Don't know why Swarm did not clean-up the previous tasks though. Which version of Docker are you using? Might be worth investigating/opening an issue in Swarm

@deviantony It's a slightly older version. 18.06.1-ce. I think I'll try updating docker first. It might be solved already.

I've updated docker to 19.03.4, build 9013bf583a. If the issue happens again, I'll report it here.

Haven't been able to reproduce this on 1.23.2 with agent 1.5.1. I think this may be tied to agent resiliency with a node drain or with the swarm split brain issue

I was able to recreate this problem with 1.23.2 by doing the following:
Set up a swarm.
Deploy the portainer stack.
Deploy a container.
Everything is fine and the container replicates to all nodes.
Reboot one of the swarm hosts.
The host comes back online and its agent starts but it is missing from the manager view.
Now suddenly Portainer on the manager complains "Unable to find a agent on any manager."
Restart the agent on the manager node.
Now on the container list we have duplicates of every (non portainer related) containers for the manager node.
There is now also two agents running and I am unable to kill one. They keep respawning.
Restarting the agents on other workers results in even more duplicates.
Maybe this helps or maybe I have some bad configuration although I have literally just started from scratch.

@ncresswell are you able to try and reproduce?

The current agent architecture can't handle node restarts you describe above. Closing this as we will address this in this issue on the agent repo


Error 400 when I try to authorize with my stored request token?

The following code is giving me Error: HTTP Error 400: Bad Request and I don't understand why (client and token replaced for obvious reasons). I saved the refresh token from a successfully authorized session yesterday and can't seem to figure out how to obtain a new access token from it. How do I do it?
import deviantart

client_id = 'CLIENT_ID_HERE'
client_secret = 'CLIENT_SECRET_HERE'

da = deviantart.Api(
    client_id,
    client_secret,

    redirect_uri="REDIRECT_URI_HERE",
    standard_grant_type="authorization_code",
)

refresh_token = 'REFRESH_TOKEN_HERE'

# PROBLEM HERE
try:
    da.auth(refresh_token = refresh_token)
except deviantart.api.DeviantartError as e:
    print ("Couldn't authorize user. Error: {}".format(e))

if da.access_token:

    print ("The access token {}.".format(da.access_token))
    print ("The refresh token {}.".format(da.refresh_token))

    #the User object of the authorized user
    user = da.get_user()

    print ("The name of the authorized user is {}.".format(user.username))

I hope this provides some useful insight:
https://www.deviantart.com/developers/authentication

Refreshing An Access Token
All access_token's expire after one hour, after expiration you either need to re-authorize the app or refresh your access token using the refresh_token from the /token request.
The refresh_token will expire after 3 months, after that time you must re-authorize the app.


Format and lint all Python files in src/ using Flake 8

This process is necessary for me to begin to understand this codebase.  I'm doing it manually, which is labor intensive, but makes it possible to preserve the intentions of the original authors as well as possible.  It has uncovered a large number of possible bugs, some of which may be serious.

This is complete now, with no changes in the automated test results.


Fix freeze issue again

Quick patch for the issues that came up as a result of the last PR. It's hard to tell exactly, but probably the changes enabled some new race condition to occur.

Still some stuff to test.. will focus on other things for a bit though.


Default user changed

By this commit: https://github.com/olafz/percona-clustercheck/commit/245f1e38a572c7821b8de7a77dab44a6418201f9
The default user is now: -clustercheckuser instead of clustercheckuser  (the "-" has been added).

My assumption is that it should be reverted to just clustercheckuser as who would set up a user starting with a dash???


Meeting test

Related issues
For #39
Other information
Abstracted out the logic behind Meeting::getNextRecurrence to a public method to test. Side effect is that the method is now public, when it would be better off as private.

Codecov Report

Merging #202 (83cb8ba) into master (ea8bf7a) will increase coverage by 0.44%.
The diff coverage is 100.00%.

:exclamation: Current head 83cb8ba differs from pull request most recent head 05f867f. Consider uploading reports for the commit 05f867f to get more accurate results

@@             Coverage Diff              @@
##             master     #202      +/-   ##
============================================
+ Coverage     71.61%   72.05%   +0.44%
- Complexity      560      565       +5
============================================
  Files            94       94
  Lines          1878     1879       +1
  Branches        202      202
============================================
+ Hits           1345     1354       +9
+ Misses          447      443       -4
+ Partials         86       82       -4

Impacted Files
Coverage Δ

...main/java/seedu/address/model/meeting/Meeting.java
94.44% <100.00%> (+11.34%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea8bf7a...05f867f. Read the comment docs.


Hidden Channels Info

At the moment there is zero information displayed when tapping on a hidden channel, I'm told this is a bug?  All available API information should be visible tho (channel description, last message date, view permissions, ect).

It indeed is a bug, I'll take a look

fixed in the new update


How can I set sizeLimit dynamically?

Hi, I want to set sizeLimit dynamically and now do something like following but not working!
UploaderConfig.options.sizeLimit = 20;
How can I do?

You cannot change options after they have been set. In this case you must choose a value and stick with it for the instance you have created


BUG: When select euro offramp bot in my bots (twap) i get 404 Error

BUG: When select euro offramp bot in my bots (twap) i get 404 Error

@jamie088 is this caused by this https://github.com/VolumeFi/paloma/issues/1138


Fix indentations in srnaseq resources of BDGP6

Fixing the bug indentations preventing the execution of smallRNA-seq analysis for BDGP6 (see this comment of issue #3112 )

Thanks for fixing this!


dependency cchardet cannot be installed on python 3.11

Hi,
since HomeAssistant v2023.3.0, we can install HomeAssistant with python 3.11
Your sensors has a dependency that is not working with python 3.11 (cchardet).
see https://github.com/thewhiteh4t/nexfil/issues/24 stating that cchardet seems to be abandonned.
You could easily migrate to charset_normalizer.
I changed the import cchardet in core_async.py  to
from charset_normalizer import detect
and changed the detect line to:
encoding = detect(content).get("encoding")
and everyhing works fine.
Would be nice if you could add this modification so people migrating to v3.11 will be able to use your sensors.
Thanks

Thank you @pleblancq
It has worked well with my LG AC.
Regards,

Thanks for this suggestion. I'm busy with work and far from home on this period, I will implement suggested changes as soon I will be back home.

I'm running HA on Python 3.10.10 and also got an error logged when HA tried to install cchardet as a dependency of smartthinq-sensors. OP's suggestion (and updating manifest.json) got me to the account information screen. \o/
Let's see how the rest works out. :)

@pleblancq I've tried your solution but still is complaining about cchardet.
I've tried searching for cchardet in the whole config folder but didn't had any matches, but still failing.
Is there anything more to be done?

Hi,
You need to change manifest.json do not try to install cchardet. Try something like that:
$ cat manifest.json
{
"domain": "smartthinq_sensors",
"name": "SmartThinQ LGE Sensors",
"integration_type": "hub",
"documentation": "https://github.com/ollo69/ha-smartthinq-sensors",
"issue_tracker": "https://github.com/ollo69/ha-smartthinq-sensors/issues",
"dependencies": [],
"codeowners": @.***"],
"requirements": ["pycountry>=20.7.3", "xmltodict>=0.12.0", "charset-normalizer>=2.1.1"],
"config_flow": true,
"iot_class": "cloud_polling",
"version": "0.31.4"
}
Regrets,

On 16 Mar 2023, at 17:06, Sergio Rius @.***> wrote:
@pleblancq https://github.com/pleblancq I've tried your solution but still is complaining about cchardet.
I've tried searching for cchardet in the whole config folder but didn't had any matches, but still failing.
Is there anything more to be done?
—
Reply to this email directly, view it on GitHub https://github.com/ollo69/ha-smartthinq-sensors/issues/507#issuecomment-1472666103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ3MP5JKWWBKLCSOGJI23TTW4NXCVANCNFSM6AAAAAAVMRUOZ4.
You are receiving this because you commented.

You need to change manifest.json do not try to install cchardet.

Thank you very much Kleytonn.

Change implemented in last release, I close this issue. Please open a new one in case of problems.


Unable to specify file type from command line

I need to validate some YAML files which do not have yaml extension, with names like .ansiblelint and at this moment validation fails and there is no option to ajv to use a specific loader (yaml) for these files.

Possibly, there could be some option for a default file format: --format=... (and it would support any recognised file extension that would be applied as file type in case of unrecognised extension). PR is welcome.
The workaround is to copy/rename the file in the script and then rename it back.

Hi, same here
The default '.json' is blocking non extension file (like Jenkinsfile ok it's not json to validate just for example)
I have doubt a maintainer still here but may be :)


grammar fix

Changes
Changed you're to your in a readme
Checklist

[x] Added markup tests, or they don't apply here because... it's just a one-word markdown grammar fix
[ ] Updated the changelog at CHANGES.md
[ ] Added myself to AUTHORS.txt, under Contributors

I don't really think a readme one-word grammar fix warrants CHANGES.md or AUTHORS.txt, much less markup tests

I don't really think a readme one-word grammar fix warrants CHANGES.md or AUTHORS.txt, much less markup tests

I don't really think a readme one-word grammar fix warrants CHANGES.md or AUTHORS.txt, much less markup tests

Correct, typically "chores" do not.

I don't really think a readme one-word grammar fix warrants CHANGES.md or AUTHORS.txt, much less markup tests

Correct, typically "chores" do not.


Fail to login user whose user record has asset in Android SDK

I open issues here seems it is related to server, since the problem only happens in server v1.6.0. Works fine for 1.5.0 server.

Skygear Server Date/Version: 1.6.0
Application Platform: Android

Expected Results

Should be enable to login successfully

Actual Results

Got error Malformed server response

Steps to reproduce

Login through android SDK

Check the login payload, the user profile's asset miss the $content-type cause the problem.

The login problem is solved, but I found we have similar problem in whoami. And we should also check the signup.


fonts

VSCode Version:
OS Version:

Steps to Reproduce:

Reproduces without extensions: Yes/No

I need this as an artist.