Cluster unable to sustain load when restoring a large number of entries

We have been deploying an etcd cluster (etcd-operator 0.9.2 / etcd 3.2.18) in our project opencord/voltha and have failed to keep a reliable cluster running.
Brief description of the problem
While doing some performance tests with our deployed system, we noticed that the cluster was showing reliability issues when getting to a certain loaded state.
Our test scenario consists in loading our system to 80% of its saturation state.  Once the system achieves that state, we delete one component from our system to force a recovery.  A new instance is spawned to replace the deleted one and that instance starts retrieving information from the etcd cluster to restore its internal information.  At this point, we start seeing some degradation with the etcd cluster.  The etcd-operator detects the issue, removes the problematic member and a new etcd instance gets spawned.  The etcd instance then tries to restore the data from the previous member.  At times, this restore phase may succeed and the cluster comes back to normal, but most times, it will just fail, get terminated and other instances will follow the same path until the operator declares the cluster as being dead.
Reproducing the problem
I simplified my deployment and was able to reproduce the problem by only running the etcd cluster along with a cluster of test containers.  The following sections explain how to go about deploying my environment.
Environment
Our system is deployed in a virtual environment hosted on a physical server.
Physical server:

Ubuntu 16.04.4 (Xenial)
Libvirt 1.3.1 / QEMU 2.5.0
Vagrant 2.0.4

Virtual environment:

3 hosts allocated with 2 CPU, 6GB of memory and 50GB of storage
bento/ubuntu-16.04 converted to a libvirt vagrant box (using vagrant-mutate)
Kubernetes 1.9.5 (deployed using kubespray 2.5.0 framework)

1 master, 3 workers

glusterfs (using gluster-kubernetes framework)
etcd-operator 0.9.2 / etcd 3.2.18
voltha/voltha-tester to run python script

Configuration and Installation
Kubernetes cluster installation
The kubernetes cluster was deployed using the voltha-k8s-playground project with some modifications to run in a libvirt based environment.
In order to use the bento box mentioned above, you will need to issue the following instructions
Install some plugins
vagrant plugin install vagrant-libvirt
vagrant plugin install vagrant-mutate

Download and convert vagrant box
vagrant box add bento/ubuntu-16.04
vagrant mutate bento/ubuntu-16.04 libvirt

The following Vagrantfile was used
# -*- mode: ruby -*-
# # vi: set ft=ruby :

ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip

Vagrant.configure(2) do |config|

  (1..3).each do |i|
    config.vm.define "k8s#{i}" do |s|
      s.ssh.forward_agent = true
      s.vm.box = "bento/ubuntu-16.04"
      s.vm.hostname = "k8s#{i}"
      s.vm.synced_folder ".", "/vagrant", type: "nfs"

      s.vm.network "private_network", ip: "<IP_ADDRESS>#{i}", netmask: "<IP_ADDRESS>",
        auto_config: true

      s.vm.provider "virtualbox" do |v|
        v.name = "k8s#{i}"
        v.memory = 3072
        v.cpus = 2
        v.gui = false
      end
      s.vm.provider "libvirt" do |v|
        v.memory = 6144
        v.cpus = 2
        #v.storage :file, :size => '50G', :type => 'raw', :bus => 'virtio'
        v.storage :file, :size => '50G', :type => 'qcow2'
      end

      s.vm.provision "file", source: "#{Dir.home}/.ssh/id_rsa", destination: "/home/vagrant/.ssh/id_rsa"
      s.vm.provision :shell, :inline =>"
     	echo '#{ssh_pub_key}' >> /home/vagrant/.ssh/authorized_keys
      ", privileged: false

      s.vm.provision :shell, path: "scripts/bootstrap_ansible.sh"

      if i == 3
        s.vm.provision :shell, :inline =>"
     	  echo '#{ssh_pub_key}' >> /home/vagrant/.ssh/authorized_keys
        ", privileged: false

        s.vm.provision "ansible_local" do |ansible|
          ansible.playbook = "kubespray/cluster.yml"
          ansible.config_file = "kubespray/ansible.cfg"
          ansible.inventory_path = "kubespray/inventory/voltha/hosts.ini"
          ansible.become = true
          ansible.become_user = "root"
          ansible.verbose = true
          ansible.limit = "all"
        end
      end
    end
  end

  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
  end

end

GlusterFS installation
Login
vagrant ssh k8s1

Wipe storage and load module
cd /vagrant && ansible kube-node -a "wipefs -af /dev/vda" -b --become-user=root -i kubespray/inventory/voltha/hosts.ini
cd /vagrant && ansible kube-node -a "modprobe dm_thin_pool" -b --become-user=root -i kubespray/inventory/voltha/hosts.ini

Install glusterfs client and dependencies
cd /vagrant && ansible kube-node -a "add-apt-repository ppa:gluster/glusterfs-3.12" -b --become-user=root -i kubespray/inventory/voltha/hosts.ini
cd /vagrant && ansible kube-node -a "apt-get update" -b --become-user=root -i kubespray/inventory/voltha/hosts.ini
cd /vagrant && ansible kube-node -a "apt-get install -y glusterfs-client" -b --become-user=root -i kubespray/inventory/voltha/hosts.ini

Clone gluster-kubernetes framework
cd /vagrant && git clone https://github.com/gluster/gluster-kubernetes.git

Create a topology
cat <<EOF >> /vagrant/gluster-kubernetes/deploy/topology.json
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s1"
              ],
              "storage": [
                "<IP_ADDRESS>"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/vda"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s2"
              ],
              "storage": [
                "<IP_ADDRESS>"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/vda"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "k8s3"
              ],
              "storage": [
                "<IP_ADDRESS>"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/vda"
          ]
        }
      ]
    }
  ]
}
EOF

Install GlusterFS server cluster
cd /vagrant/gluster-kubernetes/deploy && ./gk-deploy -gyv --object-capacity 150Gi

Setup kubernetes manifests
Create storage class manifest.  Update the resturl based on what was returned in the previous command.
cat <<EOF >> /vagrant/glusterfs-storage.yml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://<IP_ADDRESS>:8080"
  restuser: "admin"
  restuserkey: "My Secret"
  volumetype: "replicate:2"
EOF

Create cluster role manifests
cat <<EOF >> /vagrant/cluster_role_binding.yml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: etcd-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: etcd-operator
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
EOF

cat <<EOF >> /vagrant/cluster_role.yml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: etcd-operator
rules:
- apiGroups:
  - etcd.database.coreos.com
  resources:
  - etcdclusters
  verbs:
  - "*"
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - "*"
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - endpoints
  - persistentvolumeclaims
  - events
  verbs:
  - "*"
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - "*"
EOF

Create operator manifest
cat <<EOF >> /vagrant/etcd_operator.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: etcd-operator
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: etcd-operator
      annotations:
        cni: "calico"
    spec:
      containers:
      - name: etcd-operator
        image: quay.io/coreos/etcd-operator:v0.9.2
        command:
        - etcd-operator
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
EOF

Create etcd cluster manifest
cat <<EOF >> /vagrant/etcd_cluster.yml
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
  name: etcd
spec:
  size: 3
  version: "3.2.18"
  pod:
    annotations:
      cni: "calico"
    etcdEnv:
      - name: ETCD_QUOTA_BACKEND_BYTES
        value: "6442450944"
      - name: ETCD_MAX_WALS
        value: "0"
      - name: ETCD_HEARTBEAT_INTERVAL
        value: "250"
      - name: ETCD_ELECTION_TIMEOUT
        value: "5000"
      - name: ETCD_SNAPSHOT_COUNT
        value: "5000"
    persistentVolumeClaimSpec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi
      storageClassName: glusterfs-storage
EOF

Create etcd tester manifests
cat <<EOF >> /vagrant/etcd_tester_config.yml
apiVersion: v1
data:
  leader_assert.py: |
    import argparse
    import logging
    import os
    import random
    from twisted.internet.defer import Deferred, inlineCallbacks
    from txaioetcd import Client, KeySet, Transaction, CompCreated, CompValue, CompVersion, OpGet, OpSet, Failed

    log = logging.getLogger('etcd-client')
    logging.basicConfig(level=os.environ.get('LOGLEVEL', 'INFO'))

    def asleep(dt):
        """
        Async (event driven) wait for given time period (in seconds)
        :param dt: Delay in seconds
        :return: Deferred to be fired with value None when time expires.
        """
        d = Deferred()
        reactor.callLater(dt, lambda: d.callback(None))
        return d

    class EtcdClient(object):

        def __init__(self, host, port, reactor):
            log.info('__init__')
            self.host = host
            self.port = port
            self.kv_client = None
            self.reactor = reactor
            self.startup_components()

        def startup_components(self):
            log.info('startup_components')
            url = u'http://' + self.host + u':' + self.port
            log.info('Etcd endpoint {}'.format(url))
            self.kv_client = Client(self.reactor, url)
            log.info('getting all keys')

            for idx in range(0, 499):
                self.reactor.callLater(10, self._reconcile_data, 'group/1/entry', idx)
            for idx in range(500, 999):
                self.reactor.callLater(10, self._reconcile_data, 'group/2/entry', idx)
            for idx in range(1000, 1499):
                self.reactor.callLater(10, self._reconcile_data, 'group/3/entry', idx)

            #    try:
            #        key = b''.join(['group1_', str(i)])
            #        #result = yield self.kv_client.get(KeySet(key, prefix=True))
            #        result = yield self.kv_client.get(key)
            #        log.info('Result - key:{} len:{}'.format(key, len(result.kvs)))
            #    except Exception as e:
            #        log.exception('Problem while getting group 1 keys')

            log.info('got all keys')

        @inlineCallbacks
        def _reconcile_data(self, prefix, idx):
            try:
                key = b''.join([prefix, str(idx)])
                log.info('getting key:{}'.format(key))
                result = yield self.kv_client.get(key)
                log.info('got key:{} len:{}'.format(key, len(result.kvs)))
            except Exception as e:
                log.exception('Problem while getting key:{}'.format(key))

        @inlineCallbacks
        def _kill_leadership(self):
            log.info('_kill_leadership')

            status = yield self.kv_client.status()
            log.info('status:')
            log.info(status)

            # Delete old key
            yield self.kv_client.delete(b'service/voltha/leader')
            log.info('Deleted leader key...confirming:')
            result = yield self.kv_client.get(b'service/voltha/leader')
            log.info(result)

            txn = Transaction(
                compare=[
                    CompVersion(b'service/voltha/leader', '==', 0)
                ],
                success=[],
                failure=[]
            )

            try:
                result = yield self.kv_client.submit(txn)
            except Failed as failed:
                log.info('FAILED to delete leader key!')
                for response in failed.responses:
                    print(response)
            else:
                log.info('Leader key DELETED successfully')
                for response in result.responses:
                    log.info(response)

            self.reactor.callLater(5, self._leadership_tracking_loop)

        @inlineCallbacks
        def _leadership_tracking_loop(self):
    #        log.info('_leadership_tracking_loop')
            result = yield self.kv_client.get(b'service/voltha/leader')
            if result.kvs:
                kv = result.kvs[0]
                log.info('_leadership_tracking_loop: Leader = "%s"', kv.value)
            else:
                # Initial startup or leader failure, try to grab
                log.info('_leadership_tracking_loop: Leader not found')
                instance_id = os.environ.get('HOSTNAME', 'voltha-core')

                # Create lease
                lease_duration = 9 * random.randint(1,10)
                log.info('%s trying to assert leadership for %d sec', instance_id, lease_duration)
                lease = yield self.kv_client.lease(lease_duration)

                # Test-and-set
                txn = Transaction(
                    compare=[
                        CompVersion(b'service/voltha/leader', '==', 0)
                    ],
                    success=[
                        OpSet(b'service/voltha/leader', instance_id, lease=lease),
                        OpGet(b'service/voltha/leader')
                    ],
                    failure=[]
                )
                try:
                    result = yield self.kv_client.submit(txn)
                except Failed as failed:
                    log.info('Leader key PRESENT')
                    for response in failed.responses:
                        log.info(response)
                else:
                    log.info('Leader key ABSENT')
                    for response in result.responses:
                        log.info(response)

                # Check if assertion succeeded
                log.info('Confirming assertion')
                result = yield self.kv_client.get(b'service/voltha/leader')
                if result.kvs:
                    kv = result.kvs[0]
                    leader = kv.value
                    log.info('service/voltha/leader = "%s"' % leader)
                    log.info('instance_id = "%s"' % instance_id)
                    if leader == instance_id:
                        log.info("************* I'M THE LEADER! *************")
                    else:
                        log.info("************* FALSE ALARM: I'm NOT the leader *************")

            loop_interval = random.randint(2,11)
            self.reactor.callLater(loop_interval, self._leadership_tracking_loop)

    if __name__ == '__main__':
        parser = argparse.ArgumentParser()

        _help = 'Etcd host name'
        parser.add_argument('-H', '--host', dest='host', action='store', help=_help)

        _help = 'Etcd port'
        parser.add_argument('-P', '--port', dest='port', action='store', help=_help)

        args = parser.parse_args()
        host = args.host
        port = args.port

        from twisted.internet import reactor
        etcd = EtcdClient(host, port, reactor)
        reactor.callLater(0, etcd._kill_leadership)
        reactor.run()

kind: ConfigMap
metadata:
  name: etcd-tester-config
EOF

cat <<EOF >> /vagrant/etcd_tester.yml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: etcd-tester
spec:
   replicas: 3
   template:
     metadata:
       labels:
         app: etcd-tester
       annotations:
         cni: "calico"
     spec:
       containers:
         - name: rg
           image: voltha/voltha-tester
           env:
             - name: POD_IP
               valueFrom:
                 fieldRef:
                   fieldPath: status.podIP
             - name: NAMESPACE
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
           command: [ "/usr/bin/python", "/root/leader_assert.py", "--host", "etcd.default.svc.cluster.local", "--port", "2379" ]
           imagePullPolicy: IfNotPresent
           volumeMounts:
             - name: etcd-tester-config
               mountPath: /root/leader_assert.py
               subPath: leader_assert.py
       volumes:
         - name: etcd-tester-config
           configMap:
             name: etcd-tester-config
EOF

Deployment
Start the etcd cluster
kubectl apply -f /vagrant/glusterfs-storage.yml
kubectl apply -f /vagrant/cluster_role.yml
kubectl apply -f /vagrant/cluster_role_binding.yml
kubectl apply -f /vagrant/etcd_operator.yml
kubectl apply -f /vagrant/etcd_cluster.yml

Start the etcd tester cluster.  The etcd-tester cluster is and etcd client that simulates the leadership functionality of our system.
kubectl apply -f /vagrant/etcd_tester_config.yml
kubectl apply -f /vagrant/etcd_tester.yml

Execute script to produce 1500 keys in the etcd cluster.  This script is executed within and etcd instance.
Enter the etcd instance
kubectl exec etcd-<instance id> -ti sh

Copy the following script in the etcd instance session to execute it
counter=0
while [ 1 ]; do
  if [ "$counter" -eq 1500 ]; then
    break
  elif [ "$counter" -ge 0 ] && [ "$counter" -lt 500 ]; then
    key_prefix="group/1/entry"
  elif [ "$counter" -ge 500 ] && [ "$counter" -lt 1000 ]; then
    key_prefix="group/2/entry"
  elif [ "$counter" -ge 1000 ] && [ "$counter" -lt 1500 ]; then
    key_prefix="group/3/entry"
  fi
  key=$key_prefix$counter
  dd if=/dev/urandom bs=1024 count=1024  | ETCDCTL_API=3 etcdctl put $key || break > /dev/null;
  echo $key
  counter=$((counter+1))
done

Trigger Cluster Failure
Once the key generation script has completed, you can issue the following command to try and trigger the cluster failure.
Delete one of the etcd-tester instances
kubectl delete pod etcd-tester-<instance id>

When a new etcd-tester instance is spawned to replace the deleted one, it will start loading the data from the etcd cluster.  The etcd instances will show a ready status of 0/1 and the logs will show some connection errors.  The operator usually re-spawns a new instance, but it will eventually fail.  The cluster failure may occur with the first try, or you may need to issue the command once again.
Logs
Etcd Operator
...

time="2018-06-06T14:21:57Z" level=info msg="start running..." cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:05Z" level=info msg="skip reconciliation: running ([]), pending ([etcd-8vfjdbfvnx])" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:13Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:13Z" level=info msg="running members: etcd-8vfjdbfvnx" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:13Z" level=info msg="cluster membership: etcd-8vfjdbfvnx" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:18Z" level=info msg="added member (etcd-qlrsndshdv)" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:18Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:26Z" level=info msg="skip reconciliation: running ([etcd-8vfjdbfvnx]), pending ([etcd-qlrsndshdv])" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:34Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:34Z" level=info msg="running members: etcd-8vfjdbfvnx,etcd-qlrsndshdv" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:34Z" level=info msg="cluster membership: etcd-8vfjdbfvnx,etcd-qlrsndshdv" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:36Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:36Z" level=error msg="failed to reconcile: fail to add new member (etcd-7wwrg6c947): etcdserver: unhealthy cluster" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:44Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:44Z" level=info msg="running members: etcd-8vfjdbfvnx,etcd-qlrsndshdv" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:44Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-8vfjdbfvnx" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:44Z" level=info msg="added member (etcd-vk2zv2bbsl)" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:44Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:22:52Z" level=info msg="skip reconciliation: running ([etcd-8vfjdbfvnx etcd-qlrsndshdv]), pending ([etcd-vk2zv2bbsl])" cluster-name=etcd pkg=cluster
time="2018-06-06T14:23:00Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:23:00Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster

...

time="2018-06-06T14:49:10Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="running members: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-8vfjdbfvnx,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="removing one dead member" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="removing dead member \"etcd-8vfjdbfvnx\"" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="removed member (etcd-8vfjdbfvnx) with ID<PHONE_NUMBER>922386385)" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:18Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:26Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:26Z" level=info msg="running members: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:26Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:31Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:31Z" level=error msg="failed to reconcile: fail to add new member (etcd-6v8h5d8974): context deadline exceeded" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:39Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:39Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:39Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:39Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:39Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:47Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:47Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:47Z" level=info msg="cluster membership: etcd-vk2zv2bbsl,etcd-qlrsndshdv" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:47Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:47Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:55Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:55Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:55Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:55Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:49:55Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:03Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:03Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:03Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:03Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:03Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:11Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:11Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:11Z" level=info msg="cluster membership: etcd-vk2zv2bbsl,etcd-qlrsndshdv" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:11Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:11Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:19Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:19Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:19Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:19Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:19Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:27Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:27Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:27Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:27Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:27Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:35Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:35Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:35Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:35Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:35Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:43Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:43Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:43Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:43Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:43Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:51Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:51Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:51Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:51Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:51Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:59Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:59Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:59Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:59Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:50:59Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:07Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:07Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:07Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:07Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:07Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:15Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:15Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:15Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:15Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:15Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:23Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:23Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:23Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:23Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:23Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:31Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:31Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:31Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:31Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:31Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:39Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:39Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:39Z" level=info msg="cluster membership: etcd-vk2zv2bbsl,etcd-qlrsndshdv" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:39Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:39Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:47Z" level=info msg="Start reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:47Z" level=info msg="running members: etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:47Z" level=info msg="cluster membership: etcd-qlrsndshdv,etcd-vk2zv2bbsl" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:47Z" level=info msg="Finish reconciling" cluster-name=etcd pkg=cluster
time="2018-06-06T14:51:47Z" level=error msg="failed to reconcile: lost quorum" cluster-name=etcd pkg=cluster
time="2018-06-06T14:52:00Z" level=error msg="failed to update members: list members failed: creating etcd client failed: dial tcp <IP_ADDRESS>:2379: connect: connection refused" cluster-name=etcd pkg=cluster
time="2018-06-06T14:52:08Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-06-06T14:52:16Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster

Etcd
2018-06-06 14:42:42.692364 W | etcdserver: apply entries took too long [268.466851ms for 1 entries]
2018-06-06 14:42:42.692391 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:43.560049 W | etcdserver: apply entries took too long [710.87362ms for 1 entries]
2018-06-06 14:42:43.560164 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:44.097188 W | etcdserver: apply entries took too long [366.497657ms for 1 entries]
2018-06-06 14:42:44.099156 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:44.299414 I | wal: segmented wal file /var/etcd/data/member/wal/0000000000000018-0000000000000625.wal is created
2018-06-06 14:42:44.819691 W | etcdserver: apply entries took too long [469.074347ms for 1 entries]
2018-06-06 14:42:44.819719 W | etcdserver: avoid queries with large range/delete range!

...

2018-06-06 14:46:41.886089 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 25.979255ms)
2018-06-06 14:46:41.886220 W | etcdserver: server is likely overloaded
2018-06-06 14:46:41.886357 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 27.885743ms)
2018-06-06 14:46:41.886442 W | etcdserver: server is likely overloaded
2018-06-06 14:46:44.152179 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 29.027882ms)
2018-06-06 14:46:44.152205 W | etcdserver: server is likely overloaded
2018-06-06 14:46:44.152213 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 29.076613ms)
2018-06-06 14:46:44.152217 W | etcdserver: server is likely overloaded
WARNING: 2018/06/06 14:46:58 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:08 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-06-06 14:47:11.757937 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 13.330617ms)
2018-06-06 14:47:11.758680 W | etcdserver: server is likely overloaded
2018-06-06 14:47:11.758699 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 14.101156ms)
2018-06-06 14:47:11.758704 W | etcdserver: server is likely overloaded
2018-06-06 14:47:12.991020 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 1.925275ms)
2018-06-06 14:47:12.991235 W | etcdserver: server is likely overloaded
2018-06-06 14:47:12.991364 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 2.256268ms)
2018-06-06 14:47:12.991409 W | etcdserver: server is likely overloaded
WARNING: 2018/06/06 14:47:19 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:23 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:23 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
2018-06-06 14:47:23.491639 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 506.094197ms)
2018-06-06 14:47:23.491661 W | etcdserver: server is likely overloaded
2018-06-06 14:47:23.491670 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 506.130679ms)

...

2018-06-06 14:48:48.928082 I | raft: 6398a0da206f9fd1 [logterm: 3, index: 1663, vote: 6398a0da206f9fd1] ignored MsgVote from 3338a611114a25c6 [logterm: 3, index: 1661] at term 3: lease is not expired (remaining ticks: 3)
2018-06-06 14:48:48.928123 I | raft: 6398a0da206f9fd1 [term: 3] received a MsgHeartbeat message with higher term from 3338a611114a25c6 [term: 4]
2018-06-06 14:48:48.930402 I | raft: 6398a0da206f9fd1 became follower at term 4
WARNING: 2018/06/06 14:48:48 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-06-06 14:48:49.561528 I | raft: raft.node: 6398a0da206f9fd1 changed leader from 6398a0da206f9fd1 to 3338a611114a25c6 at term 4
2018-06-06 14:48:49.561717 I | raft: found conflict at index 1662 [existing term: 3, conflicting term: 4]
2018-06-06 14:48:49.561739 I | raft: replace the unstable entries from index 1662
2018-06-06 14:48:49.561910 N | pkg/osutil: received terminated signal, shutting down...
2018-06-06 14:48:49.564003 W | rafthttp: closed an existing TCP streaming connection with peer 3338a611114a25c6 (stream Message writer)
2018-06-06 14:48:49.564180 I | rafthttp: established a TCP streaming connection with peer 3338a611114a25c6 (stream Message writer)
2018-06-06 14:48:49.564292 W | rafthttp: closed an existing TCP streaming connection with peer 6f80204d3efb278c (stream Message writer)
2018-06-06 14:48:49.565098 I | rafthttp: established a TCP streaming connection with peer 6f80204d3efb278c (stream Message writer)
2018-06-06 14:48:49.570320 W | rafthttp: closed an existing TCP streaming connection with peer 6f80204d3efb278c (stream MsgApp v2 writer)
2018-06-06 14:48:49.570406 I | rafthttp: established a TCP streaming connection with peer 6f80204d3efb278c (stream MsgApp v2 writer)
2018-06-06 14:48:49.570523 W | rafthttp: closed an existing TCP streaming connection with peer 3338a611114a25c6 (stream MsgApp v2 writer)
2018-06-06 14:48:49.570565 I | rafthttp: established a TCP streaming connection with peer 3338a611114a25c6 (stream MsgApp v2 writer)
WARNING: 2018/06/06 14:48:49 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:48:50 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:48:52 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:48:53 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-06-06 14:48:55.647774 W | etcdserver: timed out waiting for read index response
WARNING: 2018/06/06 14:48:55 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:49:00 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
2018-06-06 14:49:05.621050 I | etcdserver: skipped leadership transfer for stopping non-leader member
WARNING: 2018/06/06 14:49:05 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:05 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"

...

WARNING: 2018/06/06 14:49:06 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:06 grpc: addrConn.transportMonitor exits due to: context canceled
WARNING: 2018/06/06 14:49:06 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"

2018-06-06 14:22:31.458261 I | pkg/flags: recognized and used environment variable ETCD_ELECTION_TIMEOUT=5000
2018-06-06 14:22:31.458372 I | pkg/flags: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=250
2018-06-06 14:22:31.458392 I | pkg/flags: recognized and used environment variable ETCD_MAX_WALS=0
2018-06-06 14:22:31.458444 I | pkg/flags: recognized and used environment variable ETCD_QUOTA_BACKEND_BYTES=6442450944
2018-06-06 14:22:31.458474 I | pkg/flags: recognized and used environment variable ETCD_SNAPSHOT_COUNT=5000
2018-06-06 14:22:31.458614 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_SERVICE_HOST=<IP_ADDRESS>
2018-06-06 14:22:31.458627 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_PORT_2379_TCP_PORT=2379
2018-06-06 14:22:31.458635 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_PORT_2379_TCP=tcp://<IP_ADDRESS>:2379
2018-06-06 14:22:31.458644 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_PORT=tcp://<IP_ADDRESS>:2379
2018-06-06 14:22:31.458648 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_PORT_2379_TCP_PROTO=tcp
2018-06-06 14:22:31.458771 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_SERVICE_PORT_CLIENT=2379
2018-06-06 14:22:31.458786 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_SERVICE_PORT=2379
2018-06-06 14:22:31.458794 W | pkg/flags: unrecognized environment variable ETCD_CLIENT_PORT_2379_TCP_ADDR=<IP_ADDRESS>
2018-06-06 14:22:31.458829 I | etcdmain: etcd Version: 3.2.18
2018-06-06 14:22:31.458887 I | etcdmain: Git SHA: eddf599c6
2018-06-06 14:22:31.458898 I | etcdmain: Go Version: go1.8.7
2018-06-06 14:22:31.458961 I | etcdmain: Go OS/Arch: linux/amd64
2018-06-06 14:22:31.458975 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-06-06 14:22:31.461681 I | embed: listening for peers on http://<IP_ADDRESS>:2380
2018-06-06 14:22:31.461755 I | embed: listening for client requests on <IP_ADDRESS>:2379
2018-06-06 14:22:31.509915 I | pkg/netutil: resolving etcd-8vfjdbfvnx.etcd.voltha.svc:2380 to <IP_ADDRESS>:2380
2018-06-06 14:22:31.511601 I | pkg/netutil: resolving etcd-8vfjdbfvnx.etcd.voltha.svc:2380 to <IP_ADDRESS>:2380
2018-06-06 14:22:31.513389 I | pkg/netutil: resolving etcd-qlrsndshdv.etcd.voltha.svc:2380 to <IP_ADDRESS>:2380
2018-06-06 14:22:31.515269 I | pkg/netutil: resolving etcd-qlrsndshdv.etcd.voltha.svc:2380 to <IP_ADDRESS>:2380
2018-06-06 14:22:31.516040 I | etcdserver: name = etcd-qlrsndshdv
2018-06-06 14:22:31.516089 I | etcdserver: data dir = /var/etcd/data
2018-06-06 14:22:31.516117 I | etcdserver: member dir = /var/etcd/data/member
2018-06-06 14:22:31.516141 I | etcdserver: heartbeat = 250ms
2018-06-06 14:22:31.516174 I | etcdserver: election = 5000ms
2018-06-06 14:22:31.516195 I | etcdserver: snapshot count = 5000
2018-06-06 14:22:31.516244 I | etcdserver: advertise client URLs = http://etcd-qlrsndshdv.etcd.voltha.svc:2379
2018-06-06 14:22:31.542118 I | etcdserver: starting member 3338a611114a25c6 in cluster ca10d9cda4aabbc0
2018-06-06 14:22:31.542175 I | raft: 3338a611114a25c6 became follower at term 0
2018-06-06 14:22:31.542189 I | raft: newRaft 3338a611114a25c6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018-06-06 14:22:31.542195 I | raft: 3338a611114a25c6 became follower at term 1
2018-06-06 14:22:31.586788 W | auth: simple token is not cryptographically signed
2018-06-06 14:22:31.603990 I | rafthttp: started HTTP pipelining with peer 6398a0da206f9fd1
2018-06-06 14:22:31.604027 I | rafthttp: starting peer 6398a0da206f9fd1...
2018-06-06 14:22:31.604041 I | rafthttp: started HTTP pipelining with peer 6398a0da206f9fd1
2018-06-06 14:22:31.606347 I | rafthttp: started streaming with peer 6398a0da206f9fd1 (writer)
2018-06-06 14:22:31.606377 I | rafthttp: started streaming with peer 6398a0da206f9fd1 (writer)
2018-06-06 14:22:31.607317 I | rafthttp: started peer 6398a0da206f9fd1
2018-06-06 14:22:31.607343 I | rafthttp: added peer 6398a0da206f9fd1
2018-06-06 14:22:31.607358 I | etcdserver: starting server... [version: 3.2.18, cluster version: to_be_decided]
2018-06-06 14:22:31.607369 I | rafthttp: started streaming with peer 6398a0da206f9fd1 (stream MsgApp v2 reader)
2018-06-06 14:22:31.608130 I | rafthttp: started streaming with peer 6398a0da206f9fd1 (stream Message reader)
2018-06-06 14:22:31.609097 I | rafthttp: peer 6398a0da206f9fd1 became active
2018-06-06 14:22:31.609114 I | rafthttp: established a TCP streaming connection with peer 6398a0da206f9fd1 (stream Message writer)
2018-06-06 14:22:31.609410 I | rafthttp: established a TCP streaming connection with peer 6398a0da206f9fd1 (stream MsgApp v2 writer)
2018-06-06 14:22:31.619210 I | rafthttp: established a TCP streaming connection with peer 6398a0da206f9fd1 (stream MsgApp v2 reader)
2018-06-06 14:22:31.619240 I | rafthttp: established a TCP streaming connection with peer 6398a0da206f9fd1 (stream Message reader)
2018-06-06 14:22:36.443092 I | raft: 3338a611114a25c6 [term: 1] received a MsgVote message with higher term from 6398a0da206f9fd1 [term: 3]
2018-06-06 14:22:36.443351 I | raft: 3338a611114a25c6 became follower at term 3
2018-06-06 14:22:36.443451 I | raft: 3338a611114a25c6 [logterm: 0, index: 0, vote: 0] cast MsgVote for 6398a0da206f9fd1 [logterm: 2, index: 5] at term 3
2018-06-06 14:22:36.453171 I | raft: raft.node: 3338a611114a25c6 elected leader 6398a0da206f9fd1 at term 3
2018-06-06 14:22:36.460030 I | etcdserver/membership: added member 6398a0da206f9fd1 [http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380] to cluster ca10d9cda4aabbc0
2018-06-06 14:22:36.460351 N | etcdserver/membership: set the initial cluster version to 3.2
2018-06-06 14:22:36.460577 I | etcdserver/api: enabled capabilities for version 3.2
2018-06-06 14:22:36.460858 I | etcdserver/membership: added member 3338a611114a25c6 [http://etcd-qlrsndshdv.etcd.voltha.svc:2380] to cluster ca10d9cda4aabbc0
2018-06-06 14:22:36.465676 I | etcdserver: published {Name:etcd-qlrsndshdv ClientURLs:[http://etcd-qlrsndshdv.etcd.voltha.svc:2379]} to cluster ca10d9cda4aabbc0
2018-06-06 14:22:36.465878 I | embed: ready to serve client requests
2018-06-06 14:22:36.466402 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
2018-06-06 14:22:36.467522 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:b5087f8b5b58ee92 RaftAttributes:{PeerURLs:[http://etcd-7wwrg6c947.etcd.voltha.svc:2380]} Attributes:{Name: ClientURLs:[]}}
2018-06-06 14:22:44.502644 I | etcdserver/membership: added member 6f80204d3efb278c [http://etcd-vk2zv2bbsl.etcd.voltha.svc:2380] to cluster ca10d9cda4aabbc0
2018-06-06 14:22:44.502687 I | rafthttp: starting peer 6f80204d3efb278c...
2018-06-06 14:22:44.502726 I | rafthttp: started HTTP pipelining with peer 6f80204d3efb278c
2018-06-06 14:22:44.503252 I | rafthttp: started streaming with peer 6f80204d3efb278c (writer)
2018-06-06 14:22:44.503529 I | rafthttp: started streaming with peer 6f80204d3efb278c (writer)
2018-06-06 14:22:44.503999 I | rafthttp: started peer 6f80204d3efb278c
2018-06-06 14:22:44.504038 I | rafthttp: added peer 6f80204d3efb278c
2018-06-06 14:22:44.504786 I | rafthttp: started streaming with peer 6f80204d3efb278c (stream MsgApp v2 reader)
2018-06-06 14:22:44.505312 I | rafthttp: started streaming with peer 6f80204d3efb278c (stream Message reader)
2018-06-06 14:22:49.506575 W | rafthttp: health check for peer 6f80204d3efb278c could not connect: dial tcp: lookup etcd-vk2zv2bbsl.etcd.voltha.svc on <IP_ADDRESS>:53: no such host
2018-06-06 14:22:54.507003 W | rafthttp: health check for peer 6f80204d3efb278c could not connect: dial tcp: lookup etcd-vk2zv2bbsl.etcd.voltha.svc on <IP_ADDRESS>:53: no such host
2018-06-06 14:22:59.507213 W | rafthttp: health check for peer 6f80204d3efb278c could not connect: dial tcp <IP_ADDRESS>:2380: getsockopt: connection refused
2018-06-06 14:23:00.223315 I | rafthttp: peer 6f80204d3efb278c became active
2018-06-06 14:23:00.223857 I | rafthttp: established a TCP streaming connection with peer 6f80204d3efb278c (stream Message writer)
2018-06-06 14:23:00.224294 I | rafthttp: established a TCP streaming connection with peer 6f80204d3efb278c (stream MsgApp v2 writer)
2018-06-06 14:23:00.231870 I | rafthttp: established a TCP streaming connection with peer 6f80204d3efb278c (stream Message reader)
2018-06-06 14:23:00.237835 I | rafthttp: established a TCP streaming connection with peer 6f80204d3efb278c (stream MsgApp v2 reader)
2018-06-06 14:25:15.259015 W | etcdserver: apply entries took too long [164.019066ms for 1 entries]
2018-06-06 14:25:15.259090 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:28:16.612753 W | etcdserver: apply entries took too long [122.239226ms for 1 entries]
2018-06-06 14:28:16.612799 W | etcdserver: avoid queries with large range/delete range!

...

2018-06-06 14:42:44.188171 W | etcdserver: apply entries took too long [259.61503ms for 1 entries]
2018-06-06 14:42:44.188212 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:44.548649 I | wal: segmented wal file /var/etcd/data/member/wal/0000000000000018-0000000000000625.wal is created
2018-06-06 14:42:44.944672 W | etcdserver: apply entries took too long [387.834872ms for 1 entries]
2018-06-06 14:42:44.944735 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:45.596188 W | etcdserver: apply entries took too long [398.203983ms for 1 entries]
2018-06-06 14:42:45.596225 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:46.386990 W | etcdserver: apply entries took too long [618.439161ms for 1 entries]
2018-06-06 14:42:46.387090 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:48.385658 W | etcdserver: apply entries took too long [124.894069ms for 1 entries]
2018-06-06 14:42:48.385730 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:49.773485 W | etcdserver: apply entries took too long [184.624159ms for 1 entries]
2018-06-06 14:42:49.773535 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:50.354581 W | etcdserver: apply entries took too long [119.126028ms for 1 entries]
2018-06-06 14:42:50.354615 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:51.130465 W | etcdserver: apply entries took too long [196.798654ms for 1 entries]
2018-06-06 14:42:51.130508 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:51.723926 W | etcdserver: apply entries took too long [250.724299ms for 1 entries]
2018-06-06 14:42:51.724000 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:42:52.495984 W | etcdserver: apply entries took too long [413.399996ms for 1 entries]
2018-06-06 14:42:52.496060 W | etcdserver: avoid queries with large range/delete range!
WARNING: 2018/06/06 14:47:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:12 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:37 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:37 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:38 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:43 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:48 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:47:58 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:48:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:48:03 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:48:08 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-06-06 14:48:10.437708 W | etcdserver: apply entries took too long [1m21.966261143s for 1 entries]
2018-06-06 14:48:10.437754 W | etcdserver: avoid queries with large range/delete range!
2018-06-06 14:48:46.290893 W | rafthttp: lost the TCP streaming connection with peer 6398a0da206f9fd1 (stream Message reader)
2018-06-06 14:48:46.292886 E | rafthttp: failed to read 6398a0da206f9fd1 on stream Message (read tcp <IP_ADDRESS>:43826-><IP_ADDRESS>:2380: i/o timeout)
2018-06-06 14:48:46.292973 I | rafthttp: peer 6398a0da206f9fd1 became inactive
2018-06-06 14:48:46.298757 W | rafthttp: lost the TCP streaming connection with peer 6398a0da206f9fd1 (stream MsgApp v2 reader)
WARNING: 2018/06/06 14:48:47 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-06-06 14:48:48.426884 I | raft: 3338a611114a25c6 is starting a new election at term 3
2018-06-06 14:48:48.427392 I | raft: 3338a611114a25c6 became candidate at term 4
2018-06-06 14:48:48.431828 I | raft: 3338a611114a25c6 received MsgVoteResp from 3338a611114a25c6 at term 4
2018-06-06 14:48:48.431911 I | raft: 3338a611114a25c6 [logterm: 3, index: 1661] sent MsgVote request to 6398a0da206f9fd1 at term 4
2018-06-06 14:48:48.431945 I | raft: 3338a611114a25c6 [logterm: 3, index: 1661] sent MsgVote request to 6f80204d3efb278c at term 4
2018-06-06 14:48:48.432006 I | raft: raft.node: 3338a611114a25c6 lost leader 6398a0da206f9fd1 at term 4
2018-06-06 14:48:48.473680 I | raft: 3338a611114a25c6 received MsgVoteResp from 6f80204d3efb278c at term 4
2018-06-06 14:48:48.474272 I | raft: 3338a611114a25c6 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
2018-06-06 14:48:48.474332 I | raft: 3338a611114a25c6 became leader at term 4
2018-06-06 14:48:48.474788 I | raft: raft.node: 3338a611114a25c6 elected leader 3338a611114a25c6 at term 4
2018-06-06 14:48:49.303383 I | rafthttp: peer 6398a0da206f9fd1 became active
2018-06-06 14:48:49.303517 I | rafthttp: established a TCP streaming connection with peer 6398a0da206f9fd1 (stream Message reader)
2018-06-06 14:48:49.304236 I | rafthttp: established a TCP streaming connection with peer 6398a0da206f9fd1 (stream MsgApp v2 reader)
2018-06-06 14:48:53.116213 W | etcdserver: timed out waiting for read index response
2018-06-06 14:48:55.857435 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 57.154996ms)
2018-06-06 14:48:55.858095 W | etcdserver: server is likely overloaded
2018-06-06 14:48:55.858491 W | etcdserver: failed to send out heartbeat on time (exceeded the 250ms timeout for 59.794219ms)
2018-06-06 14:48:55.858796 W | etcdserver: server is likely overloaded
2018-06-06 14:49:04.182721 N | pkg/osutil: received terminated signal, shutting down...
WARNING: 2018/06/06 14:49:04 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:49:05 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:49:06 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
WARNING: 2018/06/06 14:49:07 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:09 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
2018-06-06 14:49:11.741700 W | rafthttp: lost the TCP streaming connection with peer 6398a0da206f9fd1 (stream MsgApp v2 reader)
2018-06-06 14:49:11.741740 E | rafthttp: failed to read 6398a0da206f9fd1 on stream MsgApp v2 (unexpected EOF)
2018-06-06 14:49:11.741761 I | rafthttp: peer 6398a0da206f9fd1 became inactive
2018-06-06 14:49:11.741891 W | rafthttp: lost the TCP streaming connection with peer 6398a0da206f9fd1 (stream Message reader)
2018-06-06 14:49:12.355510 W | rafthttp: lost the TCP streaming connection with peer 6398a0da206f9fd1 (stream Message writer)
WARNING: 2018/06/06 14:49:13 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp <IP_ADDRESS>:2379: getsockopt: connection refused"; Reconnecting to {<IP_ADDRESS>:2379 0  <nil>}
2018-06-06 14:49:14.615069 W | etcdserver: failed to reach the peerURL(http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380) of member 6398a0da206f9fd1 (Get http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380/version: dial tcp <IP_ADDRESS>:2380: i/o timeout)
2018-06-06 14:49:14.615110 W | etcdserver: cannot get the version of member 6398a0da206f9fd1 (Get http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380/version: dial tcp <IP_ADDRESS>:2380: i/o timeout)
2018-06-06 14:49:16.805076 W | rafthttp: lost the TCP streaming connection with peer 6398a0da206f9fd1 (stream MsgApp v2 writer)
2018-06-06 14:49:19.356282 I | etcdserver: 3338a611114a25c6 starts leadership transfer from 3338a611114a25c6 to 6f80204d3efb278c
WARNING: 2018/06/06 14:49:19 grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2018-06-06 14:49:19.367598 I | raft: 3338a611114a25c6 [term 4] starts to transfer leadership to 6f80204d3efb278c
2018-06-06 14:49:19.367631 I | raft: 3338a611114a25c6 sends MsgTimeoutNow to 6f80204d3efb278c immediately as 6f80204d3efb278c already has up-to-date log
WARNING: 2018/06/06 14:49:19 grpc: addrConn.transportMonitor exits due to: context canceled
2018-06-06 14:49:19.386449 I | raft: 3338a611114a25c6 [term: 4] received a MsgVote message with higher term from 6f80204d3efb278c [term: 5]
2018-06-06 14:49:19.386492 I | raft: 3338a611114a25c6 became follower at term 5
2018-06-06 14:49:19.386507 I | raft: 3338a611114a25c6 [logterm: 4, index: 1668, vote: 0] cast MsgVote for 6f80204d3efb278c [logterm: 4, index: 1668] at term 5
2018-06-06 14:49:19.386515 I | raft: raft.node: 3338a611114a25c6 lost leader 3338a611114a25c6 at term 5
2018-06-06 14:49:19.510260 I | raft: raft.node: 3338a611114a25c6 elected leader 6f80204d3efb278c at term 5
WARNING: 2018/06/06 14:49:19 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
2018-06-06 14:49:19.789617 I | etcdserver: 3338a611114a25c6 finished leadership transfer from 3338a611114a25c6 to 6f80204d3efb278c (took 433.325885ms)
WARNING: 2018/06/06 14:49:19 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:20 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:20 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:20 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:20 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
2018-06-06 14:49:20.841437 W | etcdserver: failed to reach the peerURL(http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380) of member 6398a0da206f9fd1 (Get http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380/version: dial tcp <IP_ADDRESS>:2380: i/o timeout)
2018-06-06 14:49:20.841468 W | etcdserver: cannot get the version of member 6398a0da206f9fd1 (Get http://etcd-8vfjdbfvnx.etcd.voltha.svc:2380/version: dial tcp <IP_ADDRESS>:2380: i/o timeout)
WARNING: 2018/06/06 14:49:20 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:21 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"
WARNING: 2018/06/06 14:49:33 grpc: Server.processUnaryRPC failed to write status connection error: desc = "transport is closing"

This issue has not yet been acknowledged.  Can anyone provide recommendations?

Can you provide more info about underlying hardware of the system which hosts libvirt machines?
From the logs it explictly displays disk performance issues: apply entries took too long
Also putting k8s, glusterfs and etcd together on 2core/6GB vms on qcow2 will lead to generally poor performance.
Things to try:

for each vm give at least 4core and 12GB
ensure to disable kernel same page merging on host and guests
try with raw disks instead of qcow2, at least for gluster
try to add more vms as k8s nodes (keep kubespray master nodes on separate vms) and use taints to spread etcd and gluster pods to avoid keeping them together
ensure to set container limits for gluster/etcd to avoid resource saturation

But first of all, don't use gluster for etcd backend storage.


typo in installed remove_initd

It installs as remove_intid... Please tell me if you need a PR

Merged. Thanks.


Consolidate REST API meeting information content with the wiki information

Current REST API meeting add process information to their meeting invites. For now, we are including it in the message of the meeting invite that comes from the scheduler tool. (See specifics here).
This information is similar to the one in the wiki https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/665/Azure-REST-API-Stewardship-Review .
We need to consolidate this information and make sure that we don't have dups or inconsistencies between them.

The majority of the information should be in the Wiki and the content of the email should point to it. That way we don't have multiple sources of information.
With that present, the proposed content of the email is:
Hi All,

This review is designed to help service owners create developer-friendly and consistent REST (data plane)
APIs for your service. For information about the meeting format, the prerequisites, and more, see <link to wiki>.

Here if the information provided:
- Service name: <Service name>
- Key contact for this review: <name person that created the meeting> <email person that created the meeting>

- Review material:
  - Issue tracking the meeting
  - Swagger PR link
  - Link to any docs uploaded

Need approval from meeting admin before implementing the changes

@markweitzel you ok if we add this content to the meeting invite content?
Note that us modifying the content is temporary until the Admin app supports modifications to templates

@maririos -- the issue that I have with the proposed format is that it requires the reader to take another action--click to the wiki--and then read. The success rate of this will be very low. I would keep the information in the email--and make sure to put the CTAs at the top. We should still provide links to the wiki as well.
I want to make sure that we optimize around the experience of the recipient of the information. Agree it would be easier for us to manage a single source of information, but significantly harder for the recipient to consume this.

I have identified the problem with our email template. I added a correct one for API meetings and have the following content:
i.e.

I still need to add the information requested about the process

Process added.
Looks like I am still missing including the PR link if provided

I opened issue https://github.com/Azure/azure-sdk-tools/issues/3986 for the PR link


(Chore) #2189 phase0 to phase1 upgrade popup

Description

Move upgrade popup display functionality to the RouterSelector.web.js
Style upgrade popup according to the phase1 design
show phase 1 upgrade design when phase1 env enabled

About #2189 #1739
The link to the GoodServer PR - https://github.com/GoodDollar/GoodServer/pull/221
How Has This Been Tested?
Try to open the app after deployment of new version being logged out - the upgrade popup should be shown.
Checklist:

[x] PR title matches follow: (Feature|Bug|Chore) Task Name
[x] My code follows the style guidelines of this project
[ ] I have followed all the instructions described in the initial task (check Definitions of Done)
[x] I have performed a self-review of my own code
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] I have added reference to a related issue in the repository
[x] I have added a detailed description of the changes proposed in the pull request. I am as descriptive as possible, assisting reviewers as much as possible.
[x] I have added screenshots related to my pull request (for frontend tasks)
[x] I have pasted a gif showing the feature.
[x] @mentions of the person or team responsible for reviewing proposed changes

Video: https://www.screencast.com/t/dXmLxKk9
Screenshots:

Browser
Simulators

@LiavGut Please see updateds screenshots:

Browser
Simulator (small)

@yaroslav-fedyshyn-nordwhale
perform the following changes on server

add RELEASE_PHASE env var
refactor phase0/1 env vars the same way i did on dapp (e.g. if no RELEASE_PHASE in the env but the one of old env vars are set - use them to determine phase)
refactor server config the same way (e.g. Config.phase is the main source of trust but also create isPhaseZeno isPhaseOne for compatibility)
add GET /phase endpoint (accessed without authentication) which will just return json {success:true, phase: <phase value as number 0,1,2,....> }

@LiavGut I applied fixes you asked:

larger margin between the text on small devices
align 'what's new?' button with the text by the left border.
See screenshot:

@serdiukov-o-nordwhale The link to the GoodServer PR - https://github.com/GoodDollar/GoodServer/pull/221

@Everyone please check
Here is the video of how it is working after all the changes - https://www.screencast.com/t/dXmLxKk9

@yaroslav-fedyshyn-nordwhale @serdiukov-o-nordwhale
Why is there a delay between the loading of the visual to the animation start?

Animation starts on mounted. That's a time React needs to render the Splash component. Or probably do some other actions (related to process two lazy component via Suspense) - JS in single-threaded anyway.
In the code rendering of Splash and preloading of routers components starts the same time. How react will process this - it's internal React implementation and we couldn't do anything with that. Also we couldn't start preloading after animation started because React.lazy and Suspense aren't working this way

?@sirpy @LiavGut could we merge this

@yaroslav-fedyshyn-nordwhale how did we handle the animation delay previously in other screens with animation?

@yaroslav-fedyshyn-nordwhale how did we handle the animation delay previously in other screens with animation?

@sirpy we did it only for Feed animated icons with setTimeout
like this:
onMount() {
  const { delay } = this.props
  setTimeout(() => this.anim && this.anim.play(), delay)
}

But there no animation delay for this (Jumping People) animation. I think its stuck a little on video because my computer was overloaded while rendering animation.

@yaroslav-fedyshyn-nordwhale we need a PR for production also

The link to the duplicated PR to production branch:
GoodDapp: https://github.com/GoodDollar/GoodDAPP/pull/2224
GoodServer: https://github.com/GoodDollar/GoodServer/pull/224


xidia_gold_final_fixed_nonumodzip_352c9826

Add content:

[Unreal MAP_PACK] Xidia Gold Final Fixed NonUmod

Submission log: https://unrealarchive.org/submit/#352c9826

This is a UT99 map pack


changed from connectvault to certmanager in osm_deployment

Description:

Affected area:

New Functionality      [ ]
Documentation          [ ]
Install                [ ]
Control Plane          [ ]
CLI Tool               [ ]
Certificate Management [X]
Networking             [ ]
Metrics                [ ]
SMI Policy             [ ]
Security               [ ]
Tests                  [ ]
CI System              [ ]
Performance            [ ]
Other                  [ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?
No

Codecov Report

Merging #2307 (bf9f3b0) into main (b7337ab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2307   +/-   ##
=======================================
  Coverage   57.25%   57.25%
=======================================
  Files         148      148
  Lines        6655     6655
=======================================
  Hits         3810     3810
  Misses       2840     2840
  Partials        5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7337ab...b4cd8a4. Read the comment docs.

Codecov Report

Merging #2307 (bf9f3b0) into main (b7337ab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2307   +/-   ##
=======================================
  Coverage   57.25%   57.25%
=======================================
  Files         148      148
  Lines        6655     6655
=======================================
  Hits         3810     3810
  Misses       2840     2840
  Partials        5        5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7337ab...b4cd8a4. Read the comment docs.


[Script]: Create Text Files of desired size

Aim
We can create text files of any desired size we want to. The user enters the size and a text file of that size would be created.
Details
The script will:

Take the user input for size
Define size
Create file

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSoC'23 contributor
[X] I want to work on this issue

Wanting to work on this issue, please assign it to me


You can‘t predict H+ & Be++

It seems that you can‘t predict H+ ，Be++ and maybe other molecular

Dont talk big


Provide a way to update Gemfile version limits

This is a feature request.

Sorry, premature enter..


Replace zendframework/zend-filter by laminas/laminas-filter

As Zend Filter is abandoned and not compatible with PHP 8, this PR replaces it with laminas/laminas-filter.

Nice, thank you for merging. I was afraid that this library was abandoned. Could you tag / release the current master branch so that we can install the library with composer as stable version?

Done!


Support for multiple view types

Maybe I'm missing something here, but as far as I can tell there isn't a way of successfully overriding getViewType(), since it's impossible to tell what item lies at what position in the adapter due to the headers.
Am I missing something? If not, would it be possible to provide a method e.g. getItemViewType() to allow us to use different view types as well as the subheaders?

Nevermind, I've done it like this:
override fun getViewType(position: Int): Int {
    val subheaderCount = getSectionIndex(position) + 1
    val item = myItems[position - subheaderCount]
    // Decide on viewType and return
}


2.0-wip: fix typo

This is me trying out proper pull requests and actually fixing something small.

Thanks, good catch. Merged!


built docker image fails

RancherOS Version: (v1.4.0)
Where are you running RancherOS? (PI 3B)

built the aarch64 version image fails
the detail is here built docker image fails
i have  no idea about that

Did you try this?
https://rancher.com/docs/os/v1.x/en/installation/running-rancheros/server/raspberry-pi/

yes, i did it
here is the ros config
mounts:
- - /dev/mmcblk0p3
  - /mnt/docker
  - ext4
  - ""
rancher:
  cloud_init:
    datasources:
    - configdrive:/media/config-2
  console: debian
  docker:
    extra_args:
    - -g
    - /mnt/docker
  environment:
    EXTRA_CMDLINE: /init


Update polyline

Is it possible to update a polyline?
<MapView
	style={styles.map}
	initialRegion={{
	  latitude: parseFloat(User.location.coords.latitude),
	  longitude: parseFloat(User.location.coords.longitude),
	  latitudeDelta: 1,
	  longitudeDelta: 1
	}}>
	{this._renderMarker([...Spots.campsites, ...Spots.parkingSpaces])}
	{polyline}
</MapView>

And {polyline} is a dynamic line from Google Directions Api which i generate everytime i change my origin and destination TextInputs.
I call this after i change my TextInputs.
const self = this;
Backend.getRouteBetween(origin, destination)
	   .then((route)=> {
		   self.setState({lastRoute: route})
	   });

Then the map gets rendered. But no polyline. Only after i move the map a bit. But only for the first time. If i change the Destination and move the map again, the old polyline is still visible.
if (this.state.lastRoute) {
	var route = {};
	var directions = {};
	if (this.state.lastRoute.directions) {
		directions = this.state.lastRoute.directions;
		if (directions.routes.length) {
			route = this.state.lastRoute.directions.routes[0];
			var steps = [];
			if (route.legs.length) {
				steps = route.legs[0].steps;
				distance = route.legs[0].distance;
				duration = route.legs[0].duration;

				coordinates = _.chain(steps).map('start_location').value();
				coordinates.push(steps[steps.length - 1].end_location);

				coordinates = _.map(coordinates, (c)=> {
					return {
						latitude: c.lat,
						longitude: c.lng
					}
				});

				polyline = <Polyline coordinates={coordinates}/>
			}
		}
	}
}

@maluramichael can you try https://github.com/lelandrichardson/react-native-maps/pull/157 and let me know if that fixes it?

I tried npm install https://git@github.com:lelandrichardson/react-native-maps#9aaa9a539fc7f347d8a8e958b798322f2896802d restarted the packager with --reset-cache and got the same problem as above described.

hmm strange this fix worked for me. Just double checking, did you rebuild the iOS app?

@jrichardlai awesome. i don't know why it is now working. Thanks for your help.


Installing plugin in OutSystems

Following this article explaining how to install a custom Cordova plugin on the OutSystems platform I've tried to install the qrscanner and got the following error:

Failed to install 'cordova-plugin-qrscanner':Error: Uh oh!
ENOENT: no such file or directory, open '/opt/NativeBuilder/builds/5b82b857-b4a9-4266-ac61-4fb769ff0077/source/plugins/cordova-plugin-qrscanner/www/www.min.js'

I've included the full error Log in the attached file.
AndroidBuildLog.txt
Any help is welcome.

i facing the same error.. anyone have any updates regarding this issue?

I have find solution for my case:
the www/www.min.js file is automated generated when we do
cordova plugin add cordova-plugin-qrscanner
then i just use that file and create the www folder, and it's working fine.

Same problem with cordova 8.0.0. Does anyone have solution?

I just had the exact same problem with version 2.6.0... :-(
I use Visual Studio 2017, added the plugin and when I build the solution, I get this error:
ENOENT: no such file or directory, open 'C:\Users\Gert\source\repos\BlankCordovaApp1\BlankCordovaApp1\plugins\cordova-plugin-qrscanner\www\www.min.js'

cordova 9.0.0 facing the same error again.

Failed to install 'cordova-plugin-qrscanner': Error: Uh oh!
ENOENT: no such file or directory, open '/Users/xxx/projectXXX/cordova/plugins/cordova-plugin-qrscanner/www/www.min.js'

you can somehow fix this problem, when going into the plugin folder and run
npm i
and then gulp
I also upgraded gulp since old gulp pulls a lot of vulnerable npm packages. In this case you need to migrate gulpfile.js
which results in some sort of
const prepackTask = (cb) => {
  webpack('webpack.prepack.config.js', cb);
}

const webpackCordovaTask = (cb) => {
  webpack('webpack.cordova.config.js', cb);
}

const distTask = (cb) => {
  webpack('webpack.library.config.js', cb);
}

const remapTask = () => {
  return gulp.src(['dist/plugin.min.js', 'dist/www.min.js'])
      .pipe(insert.prepend(remap))
      .pipe(gulp.dest('dist'));
}

const pluginTask = () => {
    return gulp.src(['dist/plugin.min.js'])
        .pipe(gulp.dest('src/browser'));
}

const wwwTask = () => {
  return gulp.src(['dist/www.min.js'])
      .pipe(gulp.dest('www'));
}

gulp.task('prepack',prepackTask)
gulp.task('webpack-cordova',gulp.series(prepackTask,webpackCordovaTask))
gulp.task('dist',gulp.series(prepackTask,distTask))
gulp.task('remap',gulp.series(webpackCordovaTask,remapTask))
gulp.task('plugin',gulp.series(remapTask,pluginTask))
gulp.task('www',gulp.series(remapTask,wwwTask))
gulp.task('default',gulp.series(distTask,pluginTask,wwwTask))

(tho this doesn'ts eem to work full yet, it might help you)


Set Default Filter Value of 500 for Row Selection

This PR introduces a default filter value of 500 rows for the "Rows" selection box in the SE16 Cloud interface. The change is intended to prevent the accidental selection of all records from large tables, which could result in performance issues or slowdowns when handling significant data volumes.

Thanks


[Task] Add typing indicator inline style

Description:
There is currently no style section for modifying typing indicators. Users have to target the CSS classes directly. That's not great, and it would be nice to have inline styles for typing indicator as described in #91

Hello, could I take on this issue? We talked on discord the other day ( name: dgmanisch ).

Hello, could I take on this issue? We talked on discord the other day ( name: dgmanisch ).

Sure thing! 😊

Resolved by #107


Schemer: no need to migrate schema from pod which hasn't been created

In a concile process, it's no need to migrate schema from the hosts that we know it's not exist and will be created later;
Schemer can really hang there for some time to wait invisible nodes respond.

Schemer uses the load balancer and skip_unavailable_shards setting, so it should not go to inactive pods.

the process spent too a lot  time，i think when we create new cluster，no need to migrating schema objects to host

flag that mean the old chi is nil.

@alex-zaitsev

Please read this log screenshots, it did ask invisible pods to answer.

ok. For some reason it does not fail fast in your environment and waits 10 seconds timeout every time.

@SalleryWang @slimtom95 , could you answer to following questions:

according to your comments with code https://github.com/Altinity/clickhouse-operator/issues/479#issuecomment-664735922
do you use a custom fork for clickhouse-operator?

according to shared PDF, do you use clickhouse cluster with 32 shards with 3 replicas inside each shard?

look like your fork contains error in schemer.go and you tried get schema objects from unavailable hosts which not exists now
for example from shared PDF file
I0727 01:21:53.570763 1 schemer.go:247] Migrating schema objects to host 28-2
I0727 01:21:53.570839 1 schemer.go:193] Extracting replicated table definitions from [chi-ck-bo7s4lw7ax-31-0.ckbo7s4lw7ax.svc.cluster.local chi-ck-bo7s4lw7ax-31-1.ckbo7s4lw7ax.svc.cluster.local]
I0727 01:21:53.623319 1 connection.go:95] FAILED Query(http://
***:***@chi-ck-bo7s4lw7ax-0-0.ck-bo7s4lw7ax.svc.cluster.local:8123/)
clickhouse: Code: 90, e.displayText() = DB::Exception: Empty list of
columns passed (version <IP_ADDRESS> (official build))
 for SQL: SELECT DISTINCT
database AS name,
concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS
create_db_query
FROM remote('chi-ck-bo7s4lw7ax-31-0.ckbo7s4lw7ax.svc.cluster.local,chi-ck-bo7s4lw7ax-31-1.ckbo7s4lw7ax.svc.cluster.local', system, tables)
WHERE database != 'system'
SETTINGS skip_unavailable_shards = 1

I don't know why your custom operator fork tried to connect to chi-ck-bo7s4lw7ax-0-0 instead of chi-ck-bo7s4lw7ax-28-2?

not a fork，it‘s origin

@SalleryWang which version?

@Slach v0.10.0

@SalleryWang in v0.10.0
part of code which you share in https://github.com/Altinity/clickhouse-operator/issues/479#issuecomment-664735922
looks different than part of worker.go in current 0.10.0
https://github.com/Altinity/clickhouse-operator/blob/master/pkg/controller/chi/worker.go#L398

ok. after internal discussion we decided to improve schemer.go to avoid create tables when creating ClickHouse cluster from scratch

fine

I just made a little change to avoid migrating schema from pod which hasn't been created, The premise is i use cloud disk. so I think this is fine

@Slach

Fixed in 0.11.0

https://github.com/Altinity/clickhouse-operator/releases/tag/0.11.0


*: build multiple stage1s simultaneously

The build workflow currently requires the selection of a single stage1 image at ./configure time, as the --with-stage1=type flag is a singleton. This means that to do things like rkt releases, we need to do multiple configure/makes to spit out the from-coreos and from-kvm variants.
Ideally --with-stage1 becomes variadic so that I can do something like:
./configure --with-stage1=coreos --with-stage1=kvm && make
and generate both of those stage1s at the same time.

To be clear I really don't care about the exact syntax but the simultaneous build should be possible.

Implemented in https://github.com/krnowak/rkt/tree/multiple-stage1-build, splitted into six batches:

https://github.com/krnowak/rkt/tree/ms-1st-batch (PR #1612)
https://github.com/krnowak/rkt/tree/ms-2nd-batch (PR #1632)
https://github.com/krnowak/rkt/tree/ms-3rd-batch (PR #1638)
https://github.com/krnowak/rkt/tree/ms-4th-batch (no PR yet)
https://github.com/krnowak/rkt/tree/ms-5th-batch (no PR yet)
https://github.com/krnowak/rkt/tree/ms-6th-batch (no PR yet)

All above PR are merged, closing.

\o/
On Oct 26, 2015 9:26 AM, "Krzesimir Nowak"<EMAIL_ADDRESS>wrote:

Closed #1420 https://github.com/coreos/rkt/issues/1420.
—
Reply to this email directly or view it on GitHub
https://github.com/coreos/rkt/issues/1420#event-445745553.


feat(semantic): apply ImportSpecifier's binder and remove ModuleDeclaration's binder

Current dependencies on/for this PR:

main

PR #2307   👈

This stack of pull requests is managed by Graphite.


propel init reverse engineering wrong schema?

I'm expecting this to be down to my environment but I can't figure out why it's happening. After successfully running through propel init I am left with the following:
[leonard@ombrelle returns]$ ls -al
total 52
drwxrwxr-x.  4 leonard developers  4096 May 26 13:12 .
drwxrwxr-x. 18 leonard developers  4096 May 26 13:13 ..
-rw-r--r--.  1 leonard developers    63 May 26 12:46 composer.json
-rw-r--r--.  1 leonard developers 20133 May 26 12:47 composer.lock
drwxrwxr-x.  7 leonard developers  4096 May 17 14:58 .git
lrwxrwxrwx.  1 leonard developers    17 May 26 12:48 propel -> vendor/bin/propel
-rw-rw-rw-.  1 leonard developers   505 May 26 13:00 propel.php
-rw-rw-rw-.  1 leonard developers  1488 May 26 13:00 propel.php.dist
-rw-rw-rw-.  1 leonard developers  2280 May 26 13:00 schema.xml
drwxr-xr-x.  7 leonard developers  4096 May 26 12:47 vendor

propel.php is exactly as you'd expect - the correct credentials. schema.xml, however, shows the definition for a different database! I wondered if maybe propel was running from somewhere else?
[leonard@ombrelle returns]$ whereis propel
propel:
[leonard@ombrelle returns]$ type -a propel
propel is a function
propel ()
{
    ./propel "$@"
}

This seems OK. The schema it has details for is actually for a project located at /var/www/subdomains/commitr and has it's own unique credentials.
The values in propel.php are for user le_returns, and to confirm:
mysql> SHOW GRANTS FOR 'le_returns'@'localhost';
+-------------------------------------------------------------------------------------------------------------------+
| Grants for le_returns@localhost                                                                                |
+-------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'le_returns'@'localhost' IDENTIFIED BY PASSWORD '*AAAAAAAAAAAAAAAAAAAAAAAAAA' |
| GRANT ALL PRIVILEGES ON `leonard_returns`.* TO 'le_returns'@'localhost'                                           |
+-------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)
mysql> SHOW TABLES IN `leonard_returns`;
+---------------------------+
| Tables_in_leonard_returns |
+---------------------------+
| email                     |
| fault                     |
| note                      |
| priority                  |
| product                   |
| return                    |
| status                    |
| type                      |
+---------------------------+
8 rows in set (0.00 sec)

But this is the output in schema.xml in the same folder:
<?xml version="1.0" encoding="utf-8"?>
<database name="default" defaultIdMethod="native" defaultPhpNamingMethod="underscore">

...

...

</database>

Hence:
mysql> SHOW TABLES IN `commitr`;
+-------------------+
| Tables_in_commitr |
+-------------------+
| query             |
| repository        |
| user              |
+-------------------+
3 rows in set (0.00 sec)

Finally:
[leonard@ombrelle returns]$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/leonard/bin

None of these locations exist or contain propel.
Any ideas why propel is working on the wrong database?

Can you please show propel.php? The DSN is important.

'dsn' => 'mysql:host=localhost;port=3306;dbname=leonard_returns',

Can you please try using the database:reverse command? If this is working fine, then init is the issue.

Thanks @marcj, this led me to looking at the init command. Long story short, I found that in /tmp there already exists a file called schema.xml owned by another user, and I can confirm the content of this file matches what I found in my own schema.xml.
I removed the file and ran init again in its entirety and it worked as expected.
Is it worth having some form of check in there to see if it can be written to and/or clean up afterwards?

Is someone able to make a PR here with suggested changes?


Timestamps wrong/missing in 6.5.x-release.zip

Describe the bug
Timestamps are missing / incorrect from dependency-check-6.5.x-release.zip.
It is causing confusion as it is displayed as either 1980-00-00, or a date far into the future!
Version of dependency-check used
The problem occurs using version dependency-check-6.5.0-release.zip or higher.
To Reproduce
Steps to reproduce the behavior:
[/home/user]$ wget https://github.com/jeremylong/DependencyCheck/releases/download/v6.4.1/dependency-check-6.4.1-release.zip
[/home/user]$ unzip -l dependency-check-6.4.1-release.zip | head
Archive:  dependency-check-6.4.1-release.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  10-11-2021 21:23   dependency-check/
        0  10-11-2021 21:23   dependency-check/bin/
        0  10-11-2021 21:23   dependency-check/lib/
        0  10-11-2021 21:23   dependency-check/plugins/
        0  10-11-2021 21:02   dependency-check/licenses/
        0  10-11-2021 21:02   dependency-check/licenses/commons-cli/
     4139  10-11-2021 21:23   dependency-check/bin/completion-for-dependency-check.sh
[/home/user]$ wget https://github.com/jeremylong/DependencyCheck/releases/download/v6.5.0/dependency-check-6.5.0-release.zip
[/home/user]$ unzip -l dependency-check-6.5.0-release.zip | head
Archive:  dependency-check-6.5.0-release.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  00-00-1980 04:08   dependency-check/
        0  00-00-1980 04:08   dependency-check/bin/
        0  00-00-1980 04:08   dependency-check/lib/
        0  00-00-1980 04:08   dependency-check/plugins/
        0  00-00-1980 04:08   dependency-check/licenses/
        0  00-00-1980 04:08   dependency-check/licenses/commons-cli/
     4189  00-00-1980 04:08   dependency-check/bin/completion-for-dependency-check.sh
[/home/user]$ wget https://github.com/jeremylong/DependencyCheck/releases/download/v6.5.3/dependency-check-6.5.3-release.zip
[/home/user]$ unzip -l dependency-check-6.5.3-release.zip | head
Archive:  dependency-check-6.5.3-release.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  00-00-1980 04:08   dependency-check/
        0  00-00-1980 04:08   dependency-check/bin/
        0  00-00-1980 04:08   dependency-check/lib/
        0  00-00-1980 04:08   dependency-check/plugins/
        0  00-00-1980 04:08   dependency-check/licenses/
        0  00-00-1980 04:08   dependency-check/licenses/commons-cli/
     4189  00-00-1980 04:08   dependency-check/bin/completion-for-dependency-check.sh

When unzipping onto RHEL system, we see a date in the future:
[/home/user]$ ls -lago dependency-check
total 44
drwxr-xr-x 6  4096 Aug  5  2092 .
drwxrwxr-x 7  4096 Jan 18 10:06 ..
drwxr-xr-x 2  4096 Aug  5  2092 bin
drwxr-xr-x 2  4096 Aug  5  2092 lib
drwxr-xr-x 3  4096 Aug  5  2092 licenses
-rw-r--r-- 1 11358 Aug  5  2092 LICENSE.txt
-rw-r--r-- 1  1003 Aug  5  2092 NOTICE.txt
drwxr-xr-x 2  4096 Aug  5  2092 plugins
-rw-r--r-- 1  1500 Aug  5  2092 README.md

In 7-Zip, we see nothing:

With 6.5.0 we started producing reproducible builds. See the release notes: https://github.com/jeremylong/DependencyCheck/blob/main/RELEASE_NOTES.md#version-650-2021-11-08. In order to make a Java build reproducible the timestamps on the files must be set to a consistent value.
I highly recommend the use of reproducible builds as it allows independent verification of the build integrity. Reproducible builds using Maven or gradle is simple and requires setting a property or two:

Maven: https://maven.apache.org/guides/mini/guide-reproducible-builds.html
Gradle: https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives
sbt: https://github.com/raboof/sbt-reproducible-builds

I understand the "reproducible builds" concept and that " In order to make a Java build reproducible the timestamps on the files must be set to a consistent value."
However, that should be a "valid timestamp".
FAQ

Q. Can the project.build.outputTimestamp property in pom.xml be updated automatically at release time?
A. Yes.

The Apache maven documentation probably needs clarification on the feature usage. (separate issue)
I would suggest the time be the time of the last git commit:

To set SOURCE_DATE_EPOCH to the last modification of a git repository you can use ‘git log’, for example in shell:
SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

I just know that as a user, seeing an invalid (and fixed? - ie: "1" ) timestamp is confusing, misleading and I'd suggest wrong. I'd classify  it is a bug as it could impact backup / restore operations of the systems (ie: time stamp 2092-08-05 is  2,226,182,400 seconds  in the future).
Not entirely sure how you'd go about implementing that in the context of this project (am just a user here).

Thank you for the feedback. I was originally following the referenced maven documentation. I did a little more research and found the git-commit-id-plugin so we can easily reference the data from git.
See https://github.com/jeremylong/DependencyCheck/pull/4007


Throttle request rate

Added ability to throttle request rate through each connection. If throttle is set to 10 requests per second and pool size is set to 10 connections then 100 requests per second would be allowed.
Also added #checked_out to give visibility into pool utilization from application space.

Hi Eric, thanks for the pull request.  I just have a couple of questions.  I was curious about the use case for this?  Also, why throttle existing connections but not new connections?  When throttling, it seems like it would be going into a pretty cpu intensive loop while it waits for the throttling delay?

Hey Brad,
My particular use case is a RESTFul API that allows a certain request rate which is pretty common these days. Most RESTFul APIs want the user to use persistent connections with a certain limit on requests per second per connection.
I only added this logic to throttle existing connections because my thinking was that a new connection will only be a new connection when it is first created and first used. After it is checked in from its first usage, it becomes an existing connection. It is that reuse of existing connections that needs throttling.
Lastly, I agree that the nullifying of the connection and allowing the until loop to be the blocker may spike CPU but only if the delta is large since there is some meat in the loop (connections - checked out and sort). Could call a @condition.wait(interval - (1.0 / @throttle)) to prevent this instead of just nullifying connection.

Hi Eric, sorry to not be responsive.  Having one of those weeks!
Could you add the condition.wait like you suggested?  Also, I would like to keep the examples as simple as possible with the very common options so could you remove throttle from the README.md example but add it to the GenePool#initialize documentation?
Thanks!

Made the updates but used a sleep instead of @condition.wait. The monitor wait call releases the lock while waiting allowing other threads to process which was causing a race condition on stale connections. All Travis tests are passing except in jruby. Not sure why this would not throttle correctly in jruby but maybe because jruby has true concurrency?

Heh, seems jruby is just too fast! Updated unit test to assert delta is greater than or equal to allowed interval. Before it was just greater than and jruby was too fast so failed because it was exactly equal.

Conceptually I like the idea of throttling either each connection, or even the entire
connection pool based on usage rates.
This particular use-case of throttling the number of times a connection is used over a time
interval could be useful to others. In general however, IT Administrators are more likely to
implement rate limiting at a firewall based on source ip address rather than by connection.
Although, by connection is still useful.
Source IP address throttling is far more likely since it would be trivial to exceed the requests
per second by just creating a new connection and using that one until the over-used connection
can be used again.
Currently the connection pool implements throttling based on duration rather than throughput.
It is achieved by limiting the number of connections in the pool.
In order to throttle based on connection usage each connection will need to keep a count of the number
of times it has been used over a specific time-period. It could be x requests per second, or
even y requests per 5 minute interval.
Looking at this proposed solution it assumes that every call takes exactly the same amount of time,
which in general is not very likely. In this particular use-case the calls may be similar, but
the solution will need to handle requests where some calls take longer than others.
For example, 80% of calls may take 0.01s to complete, and then 20% take 0.1s to complete.
The connections where calls are taking longer will be under-utilized. Making each connection calculate
it's actual rate would result in greater throughput.
There is also a call to sleep if a particular connection is re-used too quickly. This should
be replaced with a mechanism that returns "no connections available", so that a new connection
can be created and used immediately instead of just sleeping the current thread. This would
continue until the max pool size limit is reached.
Implementing a mechanism where each connection calculates its actual usage rate could be
fairly complex though. Additionally, the solution should support plug-able algorithms
for determining if a connection is available for use. This plug-able approach would allow
the Rate based algorithm to be added to a pool without affecting the performance of
existing pools that do not use rate limiting.
The first step is to implement logic where each connection calculates its actual rate, and
then allow the checkout process to skip these "un-available" connections.

I don't necessarily disagree with anything you have said but I do believe your analysis of this proposed solution is a bit off. It merely prevents any single connection from exceeding a configurable rate by sorting on last used and checking interval. It is not to maximize throughput but to limit throughput. It also does not limit the number of connections in any way. The pool size is what has and still limits the number of connections. If a request comes in for a connection when one is not available and there is room left in the pool, a new one is created.
I do agree that the pool should be exhausted prior to throttling so that could be interesting though I suspect it would require a somewhat largish rewrite of #checkout. Your expressed preference, in its entirety, would require a complete rewrite of the gem which is currently only 322 lines including comments. Not to say that your idea does not sound useful but that maybe creating your own gem would be a better route?
I look forward to whatever you choose, keep us updated.


ENH: associated Legendre function of second kind with types 2 and 3

This PR replaces #3903  where I was unable to resolve a merge conflict going back to 090b85d which I have resolved here. All changes of #3903 have been applied here to a recent version of upstream/master so that the present version corresponds to a0e7f51 discussed in #3903 as far as the associated Legendre function of second kind is concerned.

@gertingold still interested in this one?

shouldn't it be enough to check in basic.py

It seems like as long as the check is in Python or Fortran, it should be good enough. Maybe it would be slightly better to have it in the Fortran code in case something else needs to use the function.

Yes, I am still interested even though it took me some time to respond (sorry about that). I agree that doing the checks in Fortran is slightly preferable, but do you understand why the test fails with an out-of-bounds error? Before changing any code, I would prefer to fully understand where the problem actually lies. Why does the problem in the Fortran code pointed out by @pv actually surface when the function is called via a Python script with appropriate checks implemented. Obviously, I am missing a point here...

@gertingold: I think the failure is because specfun.clpmn is also called from def clqmn --- sometimes with mm > nn at basic.py:1651.

@pv: Thanks for pointing this out. You are right of course.
One option would be to impose the constraint on mm and nn also in clqmn. On the other hand, the associated Legendre function of the second kind is defined also for mm > nn. So, may be a more general approach is needed here. On the other hand, the associated Legendre function of the second kind for complex arguments is probably not in high demand. I will look into it if I find some spare time. If somebody wants to take over, that is also fine with me.

Closing; if someone wants to take this up again we can revisit.