Certificates do not install properly, always regenerate

I am using Docker Toolbox 1.8.2c with a local build of docker-machine using PR #1951. That PR fixes the ssh problems but now the generation/validation of certificates is broken. I do not know if the problem is due to the PR or is present on master.
After creating a machine, any attempt to use the certificates, e.g. running env causes docker-machine to detect that the certs are invalid and regenerate them. The certs are never regenerated and copied successfully so all attempts to connect to the machine and use docker fail. I attempted debugging a bit and the certificate validation is failing in cert.go, line 205 _, err = tls.DialWithDialer(dialer, "tcp", addr, tlsConfig).
See https://gist.github.com/carolynvs/d98baf90172d386561e1 for the full output from calling docker-machine create default --driver virtualbox on Windows 10.
The machine can't ever get its certificates installed properly:
$ docker-machine env default
Invalid certs detected; regenerating for <IP_ADDRESS>:2376
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://<IP_ADDRESS>:2376"
export DOCKER_CERT_PATH="C:\Users\caro8994\.docker\machine\certs"
export DOCKER_MACHINE_NAME="default"
# Run this command to configure your shell:
# eval "$(C:\Program Files\Docker Toolbox\docker-machine.exe env default)"

caro8994@CAROLYNVANS87E4 MINGW64 ~
$ docker-machine env default
Invalid certs detected; regenerating for <IP_ADDRESS>:2376
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://<IP_ADDRESS>:2376"
export DOCKER_CERT_PATH="C:\Users\caro8994\.docker\machine\certs"
export DOCKER_MACHINE_NAME="default"
# Run this command to configure your shell:
# eval "$(C:\Program Files\Docker Toolbox\docker-machine.exe env default)"

Here is the output from running docker-machine -D env default https://gist.github.com/carolynvs/778e4533a26fd612732d.
Here is the output from running docker-machine -D regenerate-certs default https://gist.github.com/carolynvs/ad82eb5fb9d7c42a3ed0

Thanks for the detailed summary.  I've seen issues like this before as well and I'll look into it.

Are you on the latest VirutalBox?  i.e. 5.0.6?

I was using the 5.0.4 which ships with the latest version of Docker Toolbox (1.8.2c). I just removed that version, installed 5.0.6 and I am experiencing the same behavior.

OK thanks.

@carolynvs If you remove the host only network that you have (can do this in VirtualBox GUI) and try again, does it work?

I deleted the machine, removed the adapter and tried again with the same result.

OK thanks.  Very peculiar behavior.  I might make a test build which dumps more information about the certs and suggest that you try that if you're agreeable.

Of course! I'm happy to help out however I can.
If you want to just make a branch and point me to it, I can build it myself (:heart: containerized builds!). That way you don't need to throw multiple builds over the wall if this takes more than one attempt.

Another thing to possibly consider while fixing this, some folks like myself actually write out the contents of docker-machine env to a file which I'll source for each new terminal session (as it's a little faster than running docker-machine env). If the output of this command contains anything that cannot be evald, it's obviously going to cause problems.
So lines like the following will cause issues:
Invalid certs detected; regenerating for <IP_ADDRESS>:2376
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...

I experienced this issue on 0.5.0-dev, but haven't experienced it since downgrading to 0.4.1.

I experienced exactly the same behavior today on the release candidate.

Hi @carolynvs @blaggacao, thanks a lot for your feedback.
I'm trying to reproduce/fix this bug. Could you try this PR (https://github.com/docker/machine/pull/2006) that I created to help investigate the bug

Looks like I'm seeing this too. I'm using the latest master build on OS X using the digitalocean driver, so this definitely isn't anything to do with the environment. I think the area/windows and area/driver-virtualbox tags are irrelevant here :)

Hi @hairyhenderson, can you try build PR #2006 and tell me the output for docker-machine -D env default?

@dgageot - will do when I get a chance.
I'm also thinking a bit more about this and realizing that I've been doing a local build (i.e. make build on OS X, without using a container). One of the areas where go build has behaved differently in the past is around certificates (esp. root CA certs), so this might be related to that... I dunno.
But I'll rebuild with #2006 and try it out. Thanks!

@hairyhenderson That's a good point. I'll run my tests with a cross-compiled docker-machine

@dgageot Here is the failed output https://gist.github.com/carolynvs/e2473d21c3376f1ebec2 from docker-machine -D env default for a brand new machine.
I built #2006 and copied docker-machine.exe and docker-machine-driver-virtualbox.exe to the Docker Toolbox installation directory. I am using Docker Toolbox 1.8.2c on Windows 10.

I'm not sufficiently proficient as to know how to build, maybe I will have a look on it tis evening, if I can figure it out.

@carolynvs Thanks a lot. I still don't understand what's going on but your logs will help me.

@carolynvs Can you provide the output of:
VBoxManage list hostonlyifs
VBoxManage list dhcpservers

C:\Program Files\Oracle\VirtualBox>VBoxManage list hostonlyifs
Name:            VirtualBox Host-Only Ethernet Adapter
GUID:            3729f60a-d9c3-4daa-96ca-7ce7bae4ddcc
DHCP:            Disabled
IPAddress:       <IP_ADDRESS>
NetworkMask:     <IP_ADDRESS>
IPV6Address:     fe80:0000:0000:0000:9d6d:4449:fce1:e1cb
IPV6NetworkMaskPrefixLength: 64
HardwareAddress: 0a:00:27:00:00:00
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter

Name:            VirtualBox Host-Only Ethernet Adapter #2
GUID:            99076a32-c9e5-4930-895a-a35ee45c2542
DHCP:            Disabled
IPAddress:       <IP_ADDRESS>
NetworkMask:     <IP_ADDRESS>
IPV6Address:     fe80:0000:0000:0000:118b:39e1:36b9:a336
IPV6NetworkMaskPrefixLength: 64
HardwareAddress: 0a:00:27:00:00:00
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter #2

C:\Program Files\Oracle\VirtualBox>VBoxManage list dhcpservers
NetworkName:    HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter
IP:             <IP_ADDRESS>
NetworkMask:    <IP_ADDRESS>
lowerIPAddress: <IP_ADDRESS>
upperIPAddress: <IP_ADDRESS>
Enabled:        Yes

NetworkName:    HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter #2
IP:             <IP_ADDRESS>
NetworkMask:    <IP_ADDRESS>
lowerIPAddress: <IP_ADDRESS>
upperIPAddress: <IP_ADDRESS>
Enabled:        Yes

I have found that I still occasionally get double host only adapters. I just deleted them both and created a new machine. The certs are still regenerating when I run docker-machine env default.
Here is the output of the VBoxManage commands the second time around (with only 1 host adapter).
C:\Program Files\Oracle\VirtualBox>VBoxManage list hostonlyifs
Name:            VirtualBox Host-Only Ethernet Adapter
GUID:            2883b47a-862d-454e-9db7-42c3789585eb
DHCP:            Disabled
IPAddress:       <IP_ADDRESS>
NetworkMask:     <IP_ADDRESS>
IPV6Address:     fe80:0000:0000:0000:90ff:fd25:e5f0:8c92
IPV6NetworkMaskPrefixLength: 64
HardwareAddress: 0a:00:27:00:00:00
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter

C:\Program Files\Oracle\VirtualBox>VBoxManage list dhcpservers
NetworkName:    HostInterfaceNetworking-VirtualBox Host-Only Ethernet Adapter
IP:             <IP_ADDRESS>
NetworkMask:    <IP_ADDRESS>
lowerIPAddress: <IP_ADDRESS>
upperIPAddress: <IP_ADDRESS>
Enabled:        Yes

@carolynvs I have no idea so far.
I pushed a couple more commits on the PR to print more information and try things.
If you have time to update the output you get, that'd be just great.
ping @nathanleclaire @dmp42 any idea?

Here's the new output: https://gist.github.com/carolynvs/84cd140bcbf9b696e20f.
Let me know if there's another way to go about debugging the connection problem. I'm not quite sure what docker-machine is detecting that is causing it to regenerate the certs but am happy to poke around in /var/lib/boot2docker on the host or compare certs between windows and the host, etc if I knew what to look for.

@carolynvs That would be awesome. As you pointed out, the problem arises in cert.go:
Certs are not valid: read tcp <IP_ADDRESS>:49755-><IP_ADDRESS>:2376: wsarecv: An established connection was aborted by the software in your host machine.

Either the certificate are not properly copied onto the vm.
Or the vm is not reachable on port <IP_ADDRESS>:2376 (host network config? firewall, vpn? vm network config?)
Or there's a problem in the way we check.
If you export the env variables given by docker-machine env and ignore the errors, are you able to connect to the docker daemon?

I can ping the docker host and ssh into it. When I ignore the messages about regnerating certs from docker-machine env and set the variables manually, I am still unable to connect with the docker client.
An error occurred trying to connect: Get https://<IP_ADDRESS>:2376/v1.20/containers/json: WSARecv tcp <IP_ADDRESS>:50072: An established connection was aborted by the software in your host machine.

The certs on the host in /var/lib/boot2docker/tls/ do not match those locally in ~/.docker/machine/machines/default/. The certs in /var/lib/boot2docker/ match what is on my local machine. Also the certs in ~/.docker/machine/certs/ matches what is in ~/.docker/machine/machines/default/.
I'm guessing that the issue lies with the certs not matching, which prevents docker-machine from securely connecting to the docker daemon, thus triggering a cert regen?
I've verified that the docker daemon is running:
docker@default2:/var/log$ ps aux | grep docker
root      2439  0.1  1.9<PHONE_NUMBER>2 ?        Sl   13:23   0:00 /usr/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp://<IP_ADDRESS>:2376 --label provider=virtualbox --tlsverify --tlscacert=/var/lib/boot2docker/ca.pem --tlscert=/var/lib/boot2docker/server.pem --tlskey=/var/lib/boot2docker/server-key.pem -s aufs

Also here are the logs from boot2docker and docker: https://gist.github.com/carolynvs/f7965455ebbceb85d4e6

:+1: Thanks! I feel we're getting closer :smile:

IIRC, the certs in /var/lib/boot2docker/tls are generated server side by a startup script in the boot2docker OS and not used for anything in the current machine model (they are a relic of how boot2docker-cli historically expected the certificates to be set up).

@carolynvs @nathanleclaire I have no idea then. The only difference I have in my logs is that I'm using VBox version 5.0.6 and a more recent boot2docker.

@carolynvs Can you try connecting to the docker daemon using curl. We might get a better feedback on what's going wrong. I think you are on windows, so I don't really know how to achieve that but here's how I did it on OSX:
$ openssl pkcs12 -export -in ~/.docker/machine/certs/cert.pem -inkey ~/.docker/machine/certs/key.pem -out ~/.docker/machine/certs/cert.pfx -password pass:supersecret
$ curl -v --cacert ~/.docker/machine/machines/default/ca.pem --cert ~/.docker/machine/certs/cert.pfx --pass supersecret https://<IP_ADDRESS>:2376/version

*   Trying <IP_ADDRESS>...
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 2376 (#0)
* WARNING: SSL: Certificate type not set, assuming PKCS#12 format.
* Client certificate: dgageot
* WARNING: using IP address, SNI is being disabled by the OS.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: default
* Server certificate: dgageot
> GET /version HTTP/1.1
> Host: <IP_ADDRESS>:2376
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Server: Docker/1.8.3 (linux)
< Date: Tue, 20 Oct 2015 14:47:14 GMT
< Content-Length: 192
<
{"Version":"1.8.3","ApiVersion":"1.20","GitCommit":"f4bf5c7","GoVersion":"go1.4.2","Os":"linux","Arch":"amd64","KernelVersion":"4.1.10-boot2docker","BuildTime":"Mon Oct 12 18:01:15 UTC 2015"}
* Connection #0 to host <IP_ADDRESS> left intact

FTR, here's the tutorial I used to make it work: http://opensolitude.com/2015/07/12/curl-docker-remote-api-os-x.html

@dgageot Ooh, yes this seems to be a problem on my machine (using curl/openssl from Git for Windows so all the commands are the same).
$ openssl pkcs12 -export -in ~/.docker/machine/certs/cert.pem -inkey ~/.docker/machine/certs/key.pem -out ~/.docker/machine/certs/cert.pfx -password pass:supersecret
Loading 'screen' into random state - done

caro8994@CAROLYNVANS87E4 MINGW64 ~
$ docker-machine ip default
<IP_ADDRESS>

caro8994@CAROLYNVANS87E4 MINGW64 ~
$ curl -v --cacert ~/.docker/machine/machines/default/ca.pem --cert ~/.docker/machine/certs/cert.pfx --pass supersecret https://<IP_ADDRESS>:2376/version
* timeout on name lookup is not supported
*   Trying <IP_ADDRESS>...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 2376 (#0)
* ALPN, offering http/1.1
* could not load PEM client certificate, OpenSSL error error:0906D06C:PEM routines:PEM_read_bio:no start line, (no key found, wrong pass phrase, or wrong file format?)
* Closing connection 0
curl: (58) could not load PEM client certificate, OpenSSL error error:0906D06C:PEM routines:PEM_read_bio:no start line, (no key found, wrong pass phrase, or wrong file format?)

I checked all the certs in ~/.docker/machine/certs using vi -b path/to/cert and verified that it has unix line endings. I also used the following command to try to check if openssl was able to read them.
$ openssl x509 -in .docker/machine/certs/cert.pem -inform PEM -text -noout

I'll keep poking around with certs, as this seems like the issue. Maybe try it out on another machine and see if it's just a Windows 10 thing.

@carolynvs Great job! I'll check that tomorrow morning (Paris time)

Hi @carolynvs, have you tried this command on ca.pem too?
openssl x509 -in ~/.docker/machine/machines/default/ca.pem -inform PEM -text -noout

Can you check that it properly starts with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----. Nothing before and after.

@carolynvs I must admit I don't know what's going on. Have you tried this PR which seems vaguely related.

If you don't mind confirming this intermediate summary, so I can silently spend brain on this:

Certs are copied, but cannot be read?

I;m sure, you already checked: http://stackoverflow.com/questions/20837161/openssl-pem-routinespem-read-biono-start-linepem-lib-c703expecting-truste
I put it for reference for others.

I just tried a different curl command using --cert and --key instead of the generated pfx file and it is able to connect.
$ curl --cacert ~/.docker/machine/machines/bugtest/ca.pem --cert ~/.docker/machine/machines/bugtest/cert.pem --key ~/.docker/machine/machines/bugtest/key.pem https://$(docker-machine ip bugtest):2376/version
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   192  100   192    0     0   1761      0 --:--:-- --:--:-- --:--:--  1761{"Version":"1.8.3","ApiVersion":"1.20","GitCommit":"f4bf5c7","GoVersion":"go1.4.2","Os":"linux","Arch":"amd64","KernelVersion":"4.1.10-boot2docker","BuildTime":"Mon Oct 12 18:01:15 UTC 2015"}

Looking more closely at the output of docker-machine env I see that it is exporting what I think is a bad cert path. On my mac this points to .docker/machines/machine/ while in the output below, it points to .docker/machine/certs.
$ docker-machine env bugtest
Certs are not valid: remote error: bad certificate
Invalid certs detected; regenerating for <IP_ADDRESS>:2376
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://<IP_ADDRESS>:2376"
export DOCKER_CERT_PATH="C:\Users\caro8994\.docker\machine\certs"
export DOCKER_MACHINE_NAME="bugtest"
# Run this command to configure your shell:
# eval "$(C:\Program Files\Docker Toolbox\docker-machine.exe env bugtest)"

After manually setting the environment variables, changing the cert path to what I think it should have been, I can connect with the docker client.
Perhaps when docker-machine is testing if it can connect, it is using the wrong certs?
I added some debug info when validating certs and then tried manually connecting using first what docker-machine is using then what I think should be used.
$ docker-machine env bugtest
HOST URL=<IP_ADDRESS>:2376
CA CERT PATH=C:\Users\caro8994\.docker\machine\certs\ca.pem
SERVER CERT PATH=C:\Users\caro8994\.docker\machine\machines\bugtest\server.pem
SERVER KEY PATH=C:\Users\caro8994\.docker\machine\machines\bugtest\server-key.pem
Certs are not valid: read tcp <IP_ADDRESS>:50658-><IP_ADDRESS>:2376: wsarecv: An established connection was aborted by the software in your host
Invalid certs detected; regenerating for <IP_ADDRESS>:2376
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://<IP_ADDRESS>:2376"
export DOCKER_CERT_PATH="C:\Users\caro8994\.docker\machine\certs"
export DOCKER_MACHINE_NAME="bugtest"
# Run this command to configure your shell:
# eval "$(C:\Program Files\Docker Toolbox\docker-machine.exe env bugtest)"

caro8994@CAROLYNVANS87E4 MINGW64 ~
$ curl --cacert ~/.docker/machine/certs/ca.pem --cert ~/.docker/machine/machines/bugtest/server.pem --key ~/.docker/machine/machines/bugtest/serverachine ip bugtest):2376/version
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert ba

caro8994@CAROLYNVANS87E4 MINGW64 ~
$ curl --cacert ~/.docker/machine/certs/ca.pem --cert ~/.docker/machine/machines/bugtest/cert.pem --key ~/.docker/machine/machines/bugtest/key.pem  bugtest):2376/version
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   192  100   192    0     0    472      0 --:--:-- --:--:-- --:--:--   472{"Version":"1.8.3","ApiVersion":"1.20","GitCommit":"f4bf5c7","GoVersi"Arch":"amd64","KernelVersion":"4.1.10-boot2docker","BuildTime":"Mon Oct 12 18:01:15 UTC 2015"}

So I see 2 suspicious things:

DOCKER_CERT_PATH is using .docker/machine/certs instead of .docker/machine/machines/
Validate is using server.pem and server-key.pem instead of cert.pem and key.pem. I don't know what each cert is for but that doesn't seem right.

Thanks a lot @carolynvs that should really help. Before I digest everything you reported, can you try the latest version of https://github.com/docker/machine/pull/2006 It should print the certificates being used for validation. That should help

Here are the certs it is using
HOST URL=<IP_ADDRESS>:2376
CA CERT PATH=C:\Users\caro8994.docker\machine\certs\ca.pem
SERVER CERT PATH=C:\Users\caro8994.docker\machine\machines\bugtest\server.pem
SERVER KEY PATH=C:\Users\caro8994.docker\machine\machines\bugtest\server-key.pem
That's from my own debug info, not the PR which takes a long time to build now that it's building all the plugins. :smile:

OK, now I'm confused, so I'll just try a recap.
Can you confirm that:

~/.docker/machine/certs/ca.pem is the same as ~/.docker/machine/machines/bugtest/ca.pem
~/.docker/machine/certs/cert.pem is the same as ~/.docker/machine/machines/bugtest/cert.pem
~/.docker/machine/certs/key.pem is the same as ~/.docker/machine/machines/bugtest/key.pem
You managed to make docker cli reach the server. Which value for DOCKER_CERT_PATH did you use then?
On your mac, docker-machine env bugtest prints export DOCKER_CERT_PATH="~/.docker/machine" and not DOCKER_CERT_PATH="~/.docker/machine/certs"

Thanks again for the support!

@carolynvs FTR, cross-building only docker-machine, only for windows should be much faster: TARGET_ARCH=amd64 TARGET_OS=windows make build-x-machine

Sorry for the brain dump!

ca.pem, cert.pem and key.pm are the same in both ~/.docker/machine/certs and ~/.docker/machine/machines/bugtest
The docker  client worked when I set DOCKER_CERT_PATH to ~.docker/machine/machines/bugtest
On my mac (which works) docker-machine env sets DOCKER_CERT_PATH="~/.docker/machine/machines/bugtest". On Windows 10 (which doesn't), the same command is resulting in `DOCKER_CERT_PATH="~/.docker/machine/certs"

This was in my brain dump but may have gotten lost. When docker-machine is validating the certificates, it is attempting to connect to the docker daemon using server.pem and server-key.pem. This seems super fishy.

OK. Let's call @nathanleclaire and @ehazlett to the rescue. I think you nailed it but now, I'm too new to the project to understand why we have so many duplicated certificates and why we don't use the right ones.

Thanks for the build tip!
Below is the relevant output from the latest build of PR #2006 and here's the full output: https://gist.github.com/carolynvs/8b7034c26fe3a764c537
Reading CA certificate from C:\Users\caro8994\.docker\machine\certs\ca.pem
Reading server certificate from C:\Users\caro8994\.docker\machine\machines\bugtest\server.pem
Reading server key from C:\Users\caro8994\.docker\machine\machines\bugtest\server-key.pem

Sorry for the closed/reopened noise. I fumbled

Oi vey. @carolynvs @dgageot you all are champs for continuing to chase this one down.  I think Carolyn's suspicion is correct: if the DOCKER_CERT_PATH is not getting set correctly, then communication with the daemon will not work properly.  It sounds like it may be an issue with paths that I introduced inadvertently in the libmachine changes.  I'll continue investigating this and poking around at your findings so far.

May the gateway to the culprit be this line, then?
https://github.com/docker/machine/blob/8aa1572e0dcd75762a7627e1056ef104317f44b9/libmachine/persist/filestore.go#L155

@blaggacao Definitely strongly in the realm of possibility - that code tends to be a little brittle and has been problematic in the past.

I don't get how this can be different on windows and mac os, though as @carolynvs confirmed.
To me it clearly constructs the .docker\machine\certs path.

diff .docker/machine/certs/ca.pem .docker/machine/machines/oca/ca.pem
diff .docker/machine/certs/cert.pem .docker/machine/machines/oca/cert.pem
diff .docker/machine/certs/key.pem .docker/machine/machines/oca/key.pem
``` remains silent.

@blaggacao clearly, I don't have the same behaviour as @carolynvs on mac. So there is something fishy.

Yeah, the certs get copied over to that machine's directory during the provisioning bit.

@dgageot Apologies for the confusion. My mac is running docker-machine 0.4.1. I'm only running the PR build on my Windows machine as I've been testing out fixes there as they are merged into master.
I can do a build and run again on my mac right now.

docker ps
Could not read CA certificate "~\\.docker\\machine\\machines\\oca\\ca.pem": open ~\.docker\machine\machines\oca\ca.pem: The system cannot find the path specified.

Is this all about wrong path formatting on cygwin, gitbash?

cat ~/.docker/machine/machines/oca/ca.pem succeeds

nope, I deny, this was legacy from runing the command once in powershell probably..

$ export DOCKER_CERT_PATH="C:\Users\David\.docker\machine\certs"
$ docker ps
An error occurred trying to connect: Get https://<IP_ADDRESS>:2376/v1.20/containers/json: dial tcp <IP_ADDRESS>:2376: ConnectEx tcp: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

and
$ export DOCKER_CERT_PATH="C:\Users\David\.docker\machine\machines\oca"
$ docker ps
An error occurred trying to connect: Get https://<IP_ADDRESS>:2376/v1.20/containers/json: dial tcp <IP_ADDRESS>:2376: ConnectEx tcp: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

I resume:

diffs are not showing difference between /machine/certs and  /machine/machines/certs
I cannot reproduce carlynvs workarround to solve the problem.

When manually setting DOCKER_CERT_PATH on windows (in bash), you should use UNIX style paths. For example, export DOCKER_CERT_PATH="~./docker/machine/machines/oca".
I can confirm that on my (wonky) machine the certs do match between /machine/certs and /machine/machines/certs.

This result in
$ export DOCKER_CERT_PATH="~./docker/machine/machines/oca"
$ d ps
Could not read CA certificate "~.\\docker\\machine\\machines\\oca\\ca.pem": open ~.\docker\machine\machines\oca\ca.pem: The system cannot find the path specified.

I can confirm, by manual copying, as scp doesn't work:
diff ca.pem.local ca.pem.vm FALSE
diff server.pem.local server.pem.vm TRUE
diff key.pam.local key.pem.vm TRUE

the second ones differ between /machines/oca and oca:~/.docker

Which paths in the VM are you using for the certs @blaggacao ?

I just realized, it was the wrong one...
I checked against ~/.docker I'll check again against /var/lib/boot2docker

I can definitley confirm that

the certs in /machines/oca and vm:/var/lib/boot2docker/ are the same
( I run dos2unix on all 3 fiels ca.pem, server.pem, sever-key.pem` on the virtual machine)

I additionally get this timeout error: https://github.com/docker/machine/blob/6a5219b879db52698ccb2b7e0aafd516b34df839/libmachine/provision/boot2docker.go#L193
every time I run env

Yeah, @blaggacao it also looks like the host only IP assigned to the VM is not reachable on your computer.  Can you ping $(docker-machine ip vmname)?

nope, doesn't work either...

docker-machine ssh vmname works though

Yeah, ssh goes through localhost.  But it seems that you cannot contact the assigned host only VM IP, so I wouldn't expect env to function correctly.  Are you using any VPN or proxy?

not, that I would be aware of, just doublechecked the task manager...

Closing does't change anything, but this is another issue, I think...

https://github.com/docker/machine/blob/56f457c2ef6e306fb1815b6b125f98c85a6e92ec/commands/config.go#L88
leads me to
https://github.com/docker/machine/blob/56f457c2ef6e306fb1815b6b125f98c85a6e92ec/libmachine/cert/cert.go#L177
As I don't get any of: https://github.com/docker/machine/blob/56f457c2ef6e306fb1815b6b125f98c85a6e92ec/libmachine/cert/cert.go#L22
the only remaining candidates are:
https://github.com/docker/machine/blob/56f457c2ef6e306fb1815b6b125f98c85a6e92ec/libmachine/cert/cert.go#L198-L205

This smells like a connection between both problems. Can you interprete my line of thought?

Remote digging: https://github.com/leo-stone/fix-go-1.5-tls-issue/blob/master/main.go

I didn't trust my Windows environment anymore so I started over and rebuilt Windows then put on #2006.
In the docker.log file I see this error
2015/10/21 17:06:23 http: TLS handshake error from <IP_ADDRESS>:50386: tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid

so I checked the cert's dates
$ openssl x509 -in server.pem -noout -dates
notBefore=Oct 21 22:00:00 2015 GMT
notAfter=Oct  5 22:00:00 2018 GMT

Could the problem be that the certificate is future dated? That would explain why originally my curl commands didn't work but a few hours later they did.

same here:
$ openssl x509 -in .docker/machine/machines/oca/server.pem -noout -dates
notBefore=Oct 21 22:00:00 2015 GMT
notAfter=Oct  5 22:00:00 2018 GMT

docker@oca:~$ time
BusyBox v1.23.1 (2015-02-22 15:53:49 UTC) multi-call binary.

As stated here: https://github.com/docker/docker/issues/11534#issuecomment-89405874
docker-machine ssh vmname
sudo ntpclient -s -h pool.ntp.org

yielded me a different error (one step at a time :)

I think this is it, the rest is my virtualbox.

I am going to eat dinner and check back in 5 hours when I suspect my cert will be valid and everything will just work. :smile:

Bad news, I've to do this on every vm restart.

:smile: I guess you hit the root cause! Thanks!

:clap: :clap: :clap: :clap: :clap: :clap: :clap:

@carolynvs did the fix i posted work for you?

I just wanted to confirm that after waiting 5 hours until the cert was valid, docker-machine env works. No clue why I am getting certs that are future dated but maybe should update issue to reflect the real root cause now that we know.

In my case not the certs was the issue, but the time setting on boot2docker... As I can see on your github profile, you're from Chicago, that's a similar timezone to Bogota, maybe the boot2docker gets setup wrongly in our timezones...

After syncing the time using your workaround, I still receive the same error (certificate has expired or is not yet valid) when using those certs to connect to my docker host.
On my mac, this is what I see after making a new box and checking its time.
docker@bugtest:~$ time
BusyBox v1.23.1 (2015-02-22 15:53:49 UTC) multi-call binary.

docker@bugtest:~$ hwclock
Thu Oct 22 15:54:29 2015  0.000000 seconds

docker@bugtest:~$ date
Thu Oct 22 15:54:06 UTC 2015

docker@bugtest:~$ openssl x509 -in /var/lib/boot2docker/server.pem -noout -dates
notBefore=Oct 22 15:48:00 2015 GMT
notAfter=Oct  6 15:48:00 2018 GMT

Here is the same commands on a new host on windows:
docker@bugtest:~$ time
BusyBox v1.23.1 (2015-02-22 15:53:49 UTC) multi-call binary.

docker@bugtest:~$ hwclock
Thu Oct 22 15:58:56 2015  0.000000 seconds

docker@bugtest:~$ date
Thu Oct 22 10:58:58 UTC 2015

docker@bugtest:~$ openssl x509 -in /var/lib/boot2docker/server.pem -noout -dates
notBefore=Oct 22 15:45:00 2015 GMT
notAfter=Oct  6 15:45:00 2018 GMT

The date is showing my local time but thinks that it is UTC and I do not know how to update it to match the hwclock. I've tried manually changing date but there is something about either busybox or virtualbox that is immediately undoing any change.

docker@oca:~$ time
BusyBox v1.23.1 (2015-02-22 15:53:49 UTC) multi-call binary.
docker@oca:~$ hwclock
Thu Oct 22 10:10:46 2015  0.000000 seconds
docker@oca:~$ date
Thu Oct 22 16:28:19 UTC 2015
docker@oca:~$
docker@oca:~$ openssl x509 -in /var/lib/boot2docker/server.pem -noout -dates
notBefore=Oct 21 22:32:00 2015 GMT
notAfter=Oct  5 22:32:00 2018 GMT

cc @tianon @SvenDowideit PTAL at the above RE: boot2docker time/date issues ^^

Some code I've found might be contributing to the timestamp issue:
https://github.com/docker/machine/blob/master/libmachine/cert/cert.go#L53-L56
But it's always been working fine before.

@carolynvs @blaggacao Have you run into these issues still?

For me it is working after the referenced work-around. This, in turn, indicates, that some boot2docker time parameter where not set correctly. Typically it would then occur only during a limited time frame right after the machine creation. (Probably only in some time zones).

I did stumble over this again on my rc, but after update to 5.0 everything seems to work. We could probably close this for now. Anyhow, As soon as I notice strange behavior I would reopen it.

https://gist.github.com/damontic/bd60b6a18cacf635dc9c
I have this problem too. Don't close it.

@damontic That looks like a different issue than the one being discussed here.

Creating machine...
(consul) OUT | Creating SSH key...
(consul) OUT | Creating Digital Ocean droplet...
(consul) OUT | Waiting for IP address to be assigned to the Droplet...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: docker-machine env consul
Unable to find image 'progrium/consul:latest' locally
latest: Pulling from progrium/consul
3b4d28ce80e4: Pull complete
...
d9125e9e799b: Pull complete
Digest: sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274
Status: Downloaded newer image for progrium/consul:latest
ab964fd70394d34f8d1de5c76246490b5857adaffbc1c02235bdc53663c33b37
Running pre-create checks...
Creating machine...
(demo0) OUT | Creating SSH key...
(demo0) OUT | Creating Digital Ocean droplet...
(demo0) OUT | Waiting for IP address to be assigned to the Droplet...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening:        Maximum number of retries (5) exceeded
Running pre-create checks...
Creating machine...
(demo1) OUT | Creating SSH key...
(demo1) OUT | Creating Digital Ocean droplet...
(demo1) OUT | Waiting for IP address to be assigned to the Droplet...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning created instance...
 They have been ignored, or old ones used instead.

Before running this, I updated to Machine 0.5.1
$ docker-machine -v
docker-machine version 0.5.1 (7e8e38e)

I can move to the context of machine "consul" but not to the "demo0" or "demo1"
$ docker-machine env consul
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://<IP_ADDRESS>:2376"
export DOCKER_CERT_PATH="/Users/luc/.docker/machine/machines/consul"
export DOCKER_MACHINE_NAME="consul"
# Run this command to configure your shell:
# eval "$(/usr/local/bin/docker-machine env consul)"

$ docker-machine env    demo0
Error running connection boilerplate: Error checking and/or regenerating the certs: There was an error   validating certificates for host "<IP_ADDRESS>:2376": dial tcp <IP_ADDRESS>:2376: getsockopt: connection  refused
You can attempt to regenerate them using 'docker-machine regenerate-certs name'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

$ docker-machine env demo1
Error running connection boilerplate: Error checking and/or regenerating the certs: There was an error  validating certificates for host "<IP_ADDRESS>:2376": open   /Users/luc/.docker/machine/machines/demo1/server.pem: no such file or directory
You can attempt to regenerate them using 'docker-machine regenerate-certs name'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

@lucj If provisioning fails, the created instances will be "invalid".  Try removing them and starting again from scratch.

@nathanleclaire I've just deleted the machines (is 'docker-machine rm consul demo0 demo1' enough or should I manually delete some other stuff ?) and rerun with setup file and I got the same certs problem (when creating on DigitalOcean). Strange thing is that there is no problem with the 'consul' machine, but only with the swarm ones (demo0, demo1).
When creating the swarm on VirtualBox (5.0.10) it's working fine though.

i'm seeing this when using aws driver

I've done several tests (a lot actually), after having deleted the VM and recreated them (with a swarm) I still have the same problem.

I now have this issue after upgrading from version 1.8 to 1.9.1 using the docker toolbox on MacOSX 10.10.5
Error running connection boilerplate: Error checking and/or regenerating the certs: There was an error validating certificates for host "<IP_ADDRESS>:2376": dial tcp <IP_ADDRESS>:2376: getsockopt: connection refused
You can attempt to regenerate them using 'docker-machine regenerate-certs name'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

command failed; 1

This is happening to me periodically too. Docker v1.9.1

Same problem here with azure driver. Every time that we I create a new azure machine it fails with the error:
Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "testcargo2-prefapp-in.cloudapp.net:2376": tls: DialWithDialer timed out
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'

After running docker-machine regenerate-certs the certs validations works ok.
In docker-machine v0.5.5 there is no problem, and the creation of a docker host works ok:
Running pre-create checks...
Creating machine...
(testcargo3-prefapp-in) Creating Azure machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning with ubuntu(upstart)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect Docker to this machine, run: docker-machine env

@alambike You're hitting this issue wtih 0.6.0?

Sure, from 0.5.5 onwards. I have test this with 0.5.6 and 0.6.0.

same for me on 0.6.0 with aws driver (constantly). Not happening with virtual box driver.

fixed after changing --engine-opt="cluster-advertise=eth1:2376" to --engine-opt="cluster-advertise=eth0:2376" using docker-machine 0.6.0 (docker-machine 0.5.4 still fails)

I think im battling the same issue on my machine.  I'm using ubuntu 14.04
docker-machine version 0.5.5, build 02c4254
Running host on RHEL 7.1
Server Version: 1.10.2-cs1-rc3
Tried everything suggested with time on the machines, here is output i get from curl
curl -v --cacert ~/.docker/machine/certs/ca.pem --cert ~/.docker/machine/machines/$NODE_NAME/cert.pem --key ~/.docker/machine/machines/$NODE_NAME/key.pem https://$(docker-machine ip $NODE_NAME):2376/version

Hostname was NOT found in DNS cache
Trying <IP_ADDRESS>...
Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 2376 (#0)
successfully set certificate verify locations:
CAfile: /home/eraigosa/.docker/machine/certs/ca.pem
CApath: /etc/ssl/certs
SSLv3, TLS handshake, Client hello (1):
SSLv3, TLS handshake, Server hello (2):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Server key exchange (12):
SSLv3, TLS handshake, Request CERT (13):
SSLv3, TLS handshake, Server finished (14):
SSLv3, TLS handshake, CERT (11):
SSLv3, TLS handshake, Client key exchange (16):
SSLv3, TLS handshake, CERT verify (15):
SSLv3, TLS change cipher, Client hello (1):
SSLv3, TLS handshake, Finished (20):
SSLv3, TLS alert, Server hello (2):
error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
Closing connection 0
curl: (35) error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate

@nathanleclaire I have found the cultprit! prltoolsd from boot2docker is constantly setting my date/timezone incorrectly.
$ date
<the current local time with the timezone is set to UTC>

$ date -s '<the correct time in UTC>'
<prints the correct time>

$ date
<the date/time is now broken again>

$ /usr/local/etc/init.d/prltoolsd stop

$ date -s '<the correct time in UTC>'
<prints the correct time>

$ date
<prints the correct time and stays put>

After stopping prltoolsd and resetting the date, all my docker-machine commands work as expected and my certificates do not regenerate.
I still don't know why the timezone is set to UTC and the time to localtime after making a new machine, so this is just a workaround, not a fix.

Nice @carolynvs !
@tianon @legal90 FYI ^^

@carolynvs Wow :fearful: . It looks really weird, because prltoolsd process shouldn't start on any other virtualization system except Parallels Desktop. The daemon will start only if /usr/bin/prlvmcheck returns 0 exit-code, which means that we are on Parallels VM.
Have you reproduced this issue on Virtualbox VM? What Boot2Docker version are you using?
P.s. Also, if we assume that prltoolsd is the only reason, then Docker Machine version should not make sense. However, other comments above (link) tells that the issue appears in Machine 0.5.5+

@legal90  That makes more sense. My environment is a bit wonky, but it did used to work just fine:

I'm on a Mac running Parallels.
Inside Parallels I run Windows, then the latest docker toolbox installation. I do this because I write documentation and tutorials for Docker, and need to target Mac, Linux and Windows users.

This explains why prltoolsd is attempting to manage my docker host clock. It must be picking up on being nested inside Parallels. Does that also explain why the system clock is set to local time but thinks it is UTC?
That is the root problem that caused me to open this bug. I create a new docker machine at 10 AM CST (-6). The system clock (date) on the new machine thinks that it is 10 AM UTC, so the timestamps on the certificates are "in the future". hwclock reports the correct time.
Looking at the boot2docker Dockerfile, I noticed that it is setting /etc/timezone to UTC and should have set /etc/localtime to UTC as well.
see https://github.com/boot2docker/boot2docker/blob/master/Dockerfile#L311
RUN echo 'UTC' > $ROOTFS/etc/timezone \
	&& cp -L /usr/share/zoneinfo/UTC $ROOTFS/etc/localtime

But on my docker machine host, the tzdata package is not installed, so /usr/share/zoneinfo doesn't exist and neither does /etc/localtime. I have built my own boot2docker from the latest Dockerfile just to verify that I'm not using an old iso. I wonder if missing the /etc/localtime file is is contributing to the incorrect time problem?

@carolynvs Ah, now got it.

This explains why prltoolsd is attempting to manage my docker host clock. It must be picking up on being nested inside Parallels.

Yeah, that's the root of issue. prltoolsd runs in Virtualbox VM nested into Parallels VM. I've reproduced this and reported to responsible people at Parallels. I'll let you know as soon as it's fixed.

Does that also explain why the system clock is set to local time but thinks it is UTC?

Well, it's hard to commit but it is a known issue of Parallels Desktop (and its guest tools). It was originally reported here: https://github.com/Parallels/vagrant-parallels/issues/186.
It was worked around in PD 11 by additional option for prlctl utility, but it doesn't help in your rare case, because you are actually running Virtualbox VM on Windows.
I'm sorry, but the only solution I can suggest you at the moment is to prevent prltoolsd to be running in your VM on the boot. If you use a custom Boot2Docker ISO build, you can remove parallels-related lines from Dockerfile and rebuild the ISO. Or comment out this line: https://github.com/boot2docker/boot2docker/blob/master/rootfs/rootfs/bootscript.sh#L101

Thanks for the extra info about how prltoolsd works! I'll do as you suggest and make a custom iso for my setup. :beer:
I would close this issue, as this fixes my problem, but I'll leave that up to you since other people seem to be hitting it (though probably for different reasons!).

I think we can treat it as effectively resolved; we can re-open if any new issues are discovered.
Thanks everyone for your contributions in reporting and triaging this epically long issue!

I am using DockerToolbox 1.10.3 on Windows.  It was working fine until I restarted, and I am now having this same issue.  I am also not that familiar with Docker, so can someone tell me what the fix is?

@mtrtm Does docker-machine regenerate-certs -f not work?

Yes, docker-machine regenerate-certs -f does.  It also seems to do it every time I start up Docker Quickstart Terminal

+1
I'm using docker mainly on a Redhat server and everything works just fine. I'm not an expert but I know what I'm doing. On Windows with virtualbox, however, every time the docker VM restarts I need to regenerate-certs. I'm on toolbox 1.11.1

+1
Macbook late 2009
2,26 GHz Intel Core 2 Duo
Mac OS Sierra 10.12
Docker Tollbox 1.2.1
VirtualBox 5.0.26
$ docker-machine ls
NAME        ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
vbox-test   -        virtualbox   Running   tcp://<IP_ADDRESS>:2376           Unknown   Unable to query docker version: Get https://<IP_ADDRESS>:2376/v1.15/version: x509: certificate has expired or is not yet valid
$ docker-machine env vbox-test
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "<IP_ADDRESS>:2376": x509: certificate has expired or is not yet valid
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.
$ docker-machine regenerate-certs vbox-test
Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Waiting for SSH to be available...
Detecting the provisioner...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
$ docker-machine env vbox-test
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "<IP_ADDRESS>:2376": x509: certificate has expired or is not yet valid
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

I had this on the default install of the Docker Tookit. The error went away after running:
docker-machine regenerate-certs

Having this issue on macOS. docker-machine env complains:
$ docker-machine env docker1
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "<IP_ADDRESS>:2376": x509: certificate has expired or is not yet valid
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which might stop running containers.

Regenerating the certificates (even with -f) does not help. docker-machine ssh docker1 date shows the correct date and time.
Any ideas?

@paddor Regenerating the certificates incl. client certificates (docker-machine regenerate-certs -f --client-certs) fixed it for me.


Eureka seems to be counting 1 extra instance for 'renews threshold'

I'm using eureka in a standalone mode. On the start up, eureka starts with default renews threshold value of 1.
I look at the log on the start up and it says:
[      Thread-10] c.n.e.r.PeerAwareInstanceRegistryImpl    : Got 1 instances from neighboring DS node
[      Thread-10] c.n.e.r.PeerAwareInstanceRegistryImpl    : Renew threshold is: 1

I have 9 instances later registered with the eureka, therefore, I assume threshold should be 15.
9  instance *  2 times renewal per minute (30 sec interval by default) * 0.85 (default ratio) = 15.3
But it gives Renews threshold: 17. I don't know how it exactly rounds up, but I'm sure its counting 1 extra instance.
It is causing the EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE. very easily since the threshold is higher than expected.

What is the value of "Got X instances from neighboring DS node" before you get "Renew threshold: 17"?

public class TestApp {
	public static void main(String[] args) {
		int count = 9;
		double renewalPercentThreshold = 0.85;
		int expectedNumberOfRenewsPerMin = count * 2;
		int numberOfRenewsPerMinThreshold =
				(int) (expectedNumberOfRenewsPerMin * renewalPercentThreshold);
		System.out.println("Got " + count + " instances from neighboring DS node");
		System.out.println("Renew threshold is: " + numberOfRenewsPerMinThreshold);
	}
}

gives
Got 9 instances from neighboring DS node
Renew threshold is: 15

So either count=10 or eureka.instance.renewalPercentThreshold is 0.95. Either of those changes give 17 as the answer. The calculation happens in netflix PeerAwareInstanceRegistryImpl.

I've encountered a similar issue in standalone mode. From my debugging with 1.2.0-RELEASE:
On startup, EurekaServerBootstrap.initEurekaServerContext() is called. This calls register.syncUp() which in a standalone server returns a registryCount of 0 (line 153).
Next, registry.openForTraffic() is called. This first hits InstanceRegistry.openForTraffic where the count is evaluated before calling super.openForTraffic().
During this evaluation (line 75-76 of InstanceRegistry), count is 0, so the super method is called with defaultOpenForTrafficCount - which is 1. So when PeerAwareInstanceRegistryImpl.openForTraffic() is called, count is now 1, setting expectedNumberOfRenewsPerMin to be 2, and expectedNumberOfRenewsPerMin to be 1.
Now we have a situation where there are no clients registered, but we are expecting 1 renew per minute. When the next (i.e. the first real) client registered, AbstractInstanceRegistry.register() is called, incrementing expectedNumberOfRenewsPerMin by 2, and updating the threshold as appropriate.
This is all discovered by running the debugger, so sorry if my understanding is flawed, but I don't see how this could ever be recovered. Surely the threshold will always be set expecting two more renews/min than it should be.

@peterjhall setting eureka.server.defaultOpenForTrafficCount=0 should change that behaviour.

Would still like to hear back from @beku8. Also see #380

@spencergibb Thanks! What is the expected behaviour with that property left at it's default value?

Early on (eureka 1.1.x) there was trouble canceling leases in a standalone server. It may be worth revisiting the default.


A (may be?) fork of a brilliant win-32/Linux/BSD i386/amd64/arm  GUI library & IDE as GUI lib for Rust

The library has excellent database support, only needs core OS & GUI libraries to work etc, RPC & remote form features, secure channels, etc
The web site: http://msegui.com/
The main repo: https://gitlab.com/mseide-msegui/mseide-msegui.git
This (Free Pascal) project were created and maintained by a single person (Martinn Schreiber) which suddenly died at Nov  2018
https://gitlab.com/mseide-msegui/mselang.git
This is  Martin's 90% made work on more suitable language for the msee+msegui.
http://gitorious.org/mseuniverse/mseuniverse
This is a multiple commmiters repo of some documentation, working & example projects , extensions etc.
The famliy of Martin has passwords to his repos & agrees to share these to new maintainers & forkers if needed.
The current community around this library ins't skilled enough to continue developing the existing (Free Pascal) code base

This repository is about the language design and changes to the compiler and thus is a wrong place to discuss the wider ecosystem. A better place for such an RFC would be users.rlo or, perhaps, reddit.


fin vs fisherman?

The fin design is very elegant, I like it. The fin is "prove of concept" or will replace the fisherman?

Thanks! No and no.

Is fin a proof of concept?

Not anymore. I use it.

Will it replace fisherman?

No. fisherman (~2000 loc) has a ton of features that fin (~250 loc) doesn't (and won't) have.
Right off the top of my head.

feature
fin
fisherman

Runs Wahoo/OMF/Tackle plugins
No
Yes

Uninstall hooks
No
Yes

set_color_custom
No
Yes

fish_user_key_bindings parser
No
Yes

Customize $fisher_cmd_name
No
Yes

Install/Update/Remove plugins via CLI
No
Yes

List installed plugins
No
Yes

List/auto-complete remote plugins (omf, fisherman, etc.)
No
Yes

Using git
No
Yes

Self-uninstall
No
Yes

Include custom script files in plugin (awk, py, etc.)
No
Yes

Install gists
No
Yes

Support fish-shell 2.2
No
Yes

Support omf dependency plugins (bundle)
No
Yes

Builtin man page
No
Yes

Customize output format with ls-remote --format="..."
No
Yes

Fetch plugins by tagged<EMAIL_ADDRESS>version
Yes
Yes

Use tar.bz archives
Yes
No

Thanks!

Is fin a proof of concept?

No. It's what I'm using now.

Will it replace fisherman?

No. Never.

Bonus question: Should I use fin or fisherman?

It's up to you. If you need support for key bindings, omf plugins and other complex non-standard stuff and favor using the CLI to install, update and remove plugins, fisherman is for you. If you want a minimal setup that does not deviate so much from bare bones fish-shell, then fin is a worthy option.
The fisherman (~2000 loc) has a large number of "luxury" features that fin doesn't (and won't) have.
On the other hand, fin is tiny (~250 loc) and supports most fisherman plugins.
Right off the top of my head.

Feature
fin
fisherman

Support Wahoo / OMF / Tackle plugins and modules
No
Yes

Support uninstall hooks
No
Yes

set_color_custom
No
Yes

fish_user_key_bindings parser
No
Yes

Customize command name fisher_cmd_name
No
Yes

Install / Update / Remove plugins via CLI
No
Yes

List installed plugins
No
Yes

List / Tab-complete remote plugins (OMF, fisherman, etc.)
No
Yes

Customize output format with ls-remote --format="..."
No
Yes

Self-uninstall
No
Yes

Include custom script files in plugin (awk, py, etc.)
No
Yes

Install GitHub Gists
No
Yes

Install local (file system) directories as plugins
Yes
Yes

Support fish-shell 2.2
No
Yes

Support OMF dependency plugins (parse bundle file)
No
Yes

Built-in man page
No
Yes

Use git
No
Yes

Use tar.bz archives
Yes
No

Fetch plugins by tagged<EMAIL_ADDRESS>version
Yes
Yes

Fetch plugins concurrently
Yes
Yes

Manage nested dependencies recursively
Yes
Yes


[6]SPIKE: Normalize Hebrew strings for canonical equivalence according to Unicode standard

Based on https://unicode.org/reports/tr15/#Canonical_Equivalence and following. Note that: The outputs are not required to be identical, only canonically equivalent.
DoD: Opening a canonically equivalent, but different Hebrew text highlights the same as another Hebrew text. Text UHB version 2.1.9
Needs a definition of what characters can be re-ordered.  Consult @jag3773

Needs to be discussed with @klappy

@birchamp - It looks like this should be a tC PR?  Or is this tC-create?

@PhotoNomad0 If we address this in a common library there will be parity between the applications usage of normalization and reduction of duplicated effort on a highly complex task. String normalization appears simple at first but quickly becomes a maintenance headache.

@klappy agreed.  Do we have a plan on how to do this that won't bring the app performance to a crawl on slow machines?  I'm guessing we don't have detail on how this would be implemented for Hebrew.  The link looks very general.  Are we mostly talking about Hebrew words where the accent characters are in different order?  Or are there other cases?

My understanding is that there is a set list of these entities that are identical and there may be a preferred rendered output that we want it to be.
The tokenizer has a new feature for normalization in the latest release to allow for custom normalization. We can continue working on the implementation and refining it as we have more use cases.

Oh, and for performance, normalization should only be run when absolutely necessary. Since our use case is Original languages then that should be easy to run that once on the Hebrew text prior to parsing it. That is way more efficient that running it on every verse, word, or token that gets rendered.

@klappy - makes sense, but requirements are still vague.  Can you get that set list of these entities and attach them here.  And it does seem strange that we are doing word normalization in a string tokenizer. Also it would seem better to have a rule set then a list of identical entities since that could cause a lot of churn as they keep finding one more identity that they noticed.

Also, haven't read all comments here, but would doing any normalization on the DCS side through a git hook work at all so all apps are using the same data and get it massaged correctly? Go (the language DCS is written in) does support text normalization: https://blog.golang.org/normalization

https://unicode.org/faq/normalization.html#10
Q: But isn't there is still a problem with Biblical Hebrew?
A: There was a problem, but it has been addressed. Because the Hebrew points are defined to have distinct combining classes, their character semantics is such that their ordering is immaterial in the standard. To handle those cases where visual ordering is material, see the discussion of the Combining Grapheme Joiner (CGJ) in Section 23.2, Layout Controls, in the Unicode Standard.

article: a bit dated
only useful for "Appendix: Keyboard Charts" pp. 16-17
http://www.ntresources.com/blog/documents/Unicode4BibStudies.pdf

SBL: info on normalization & a helpful examples / test cases (e.g., Ps 27:13):
also includes "recommended mark ordering"
https://www.sbl-site.org/Fonts/SBLHebrewUserManual1.5x.pdf

StackOverflow article seems to be related to the [old] Safari problem mentioned on the phone:
https://stackoverflow.com/questions/11176603/how-to-avoid-browsers-unicode-normalization-when-submitting-a-form-with-unicode

Screenshot of a normalization problem from a text of 1 Chr. 13:13.
I have not verified which method UHT uses.
Also, I would like to discuss HOW this issue would manifest in tc-create?  Translating from biblical Hebrew to a modern Hebrew text??

TODO: confirm with @klappy  if Greek normalization is is still required

My opinion is that yes, we need this for Greek too.


Convert Slider to Variants

Variants in Figma

[x] Add documentation link
[x] Test out nested variants (Stepper as main. Dot, handle, indicator, and track as nested)
[x] Divide into 4 different component sets (dot, handle, indicator, and track)

Variant properties:
Dark theme and state

Looking good!


Support custom viem transport in network config

Description
Took a quick stab at https://github.com/0xOlias/ponder/issues/325 by introducing a viem transport option to the configuration options.
Supports optionally specifying a transport. If the transport exists, the rpcUrl will not be used to create one.
Wanted to get validation on the approach before adding tests.
Todo

[ ] Add test

There was consensus in the chat that we should just replace rpcUrl with transport.
Questions/todo:

It's probably worth trying out the websocket and fallback transports to make sure they work.
Now that viem is required (to import http or another transport), do we need to add it to package.json peerDependencies? We currently automatically add it in devDependencies because we use some types, but the transport is runtime code.
We'll need to update the examples/ projects, the docs, some of the tests, and the create-ponder templates to reflect this change


Have Problem in style overwrite

I want to overwrite the default styles, how can i do it

@alivelee All the styles are on the CSS file, if you want to override, you just have to make a custom one


[aes] Consider adding a repetition check for the EDN interface

As part of #14240 we were thinking to adding repetition checks for the EDN interface of AES. At the moment there is no repetition check as AES doesn't require high-quality entropy. However, it would make it harder for someone to replicate the masking off analysis which requires all 32-bit LFSRs in the PRNG to be seeded to the same specific seed.
For future versions, we should consider adding this repetition check and connect it to a recoverable/info alert.

yeah, we could increase the resolution at which edn does the repetition checks? I think it's 64b right now (could be wrong), and we could make it 32b for example.  Which is probably the width that most blocks fetch.

Backlog sounds right for this. We don't need this for M2.5.


Add MODULE_VERSION constant to have an ability to determine available crud features

Good idea. AFAIS, it is usually named VERSION.

@Totktonada Doesn't matter how it will be named. And seems that this must be standardized and extended to all modules in ecosystem!

Closed by #342


Fix Layout Mode for  Category pages

### Expected Behaviour
When you change layout perspectives (desktop, tablet, iphone) that the page still has content
### Actual Behaviour
When you change the layout perspective, the query for content to Magento is lost and page goes blank.
Reproduce Scenario (including but not limited to)
Steps to Reproduce
view a category page which has products in preview mode, switch to layout mode and then change perspective. Poof, products are disappear.
View of category page in desktop mode after arriving in preview mode

Click on tablet or mobile to change perspective and products disappear

Platform and Version
Sample Code that illustrates the problem
Logs taken while reproducing problem

I can reproduce this and this is unfortunately working as designed. The editor reloads the page if one changes the layout preview, that makes removes the current category from the URL (see browser URL before / after). Without the category present in the URL the product list does not render anything, adding it back will fix it.
We work on making the product list more robust in editing mode.

At some level, it will still only be somewhat useful as the product component appears to have no ability to be reallocated across the grid. In fact I see no grid at this time, but hoping that while the product component will have spacing defined by CSS, the rest of the content on a product page tied to a slug or slugs could be authored with components including managing the changes across perspectives.....hope all that makes sense.  Thanks M.

Ok, yes there is a grid underneath but the pages with products lists are not respecting the fact that there should be editiable areas of the page.  In authoring environment, go to a category with products in it and there are no editable area. There is an inherit area, but if you put something on a blank page upstream, like the catalog page, it is not inherited on a category page.

The current category page templates have a grid (https://github.com/adobe/aem-cif-project-archetype/blob/master/src/main/archetype/samplecontent/src/main/content/jcr_root/content/contentFolderName/us/en/products/category-page/.content.xml#L19) but the product list component is not part of that, it is next to it. That is intentionally since it is "not made for authors", the component itself is hidden. If authors remove it from page they can not add it back.
This was a decision we made, but can be changed. WDYT should PDP & category templates be fully author-able?

Yes I do think Category and Product Pages should be fully authorable.  I believe that people will want to move things around and I believe that Target is big on talking about testing layouts etc. They should also allow for insertion of additional UI blocks as things like payment methods and source based inventory etc are needed to find space on the page in a way that flow with the product content. HTH


is that any example for using it?

and I would like to ask, can this library be cross-compiled for Android?

Library is extensively documented, you can find examples right in the docs, see https://docs.rs/cggmp21/. Hope that helps!

and I would like to ask, can this library be cross-compiled for Android?

We didn't check. If you can make GMP library work on Android, then you might be able to do the same for cggmp21 implementation. Implementation is almost pure-Rust except for GMP lib which is usually a blocker when you need to port it somewhere.

I try ios\andriod, it works. thanks

@ChiaoTeo Sorry to resurrect this closed issue, but I am also trying to cross compile to android but I am having issues with the GMP lib, can you share more details on how you managed to make it cross compile?


Add an About section

In the About section I suggest writing "simple Logger for Fortran" and adding the topic tags logger, fortran, as explained at https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/classifying-your-repository-with-topics

I appreciate your kind advice!


Need help adding IR climate device in ESPHome to ha-bridge

Need help adding IR climate device in ESPHome to ha-bridge
At first I thought I will add Home assistant in bridge config and bulk add Home Assistant devices, which is listing the IR Climate device which controls my Daikin AC. But ha-bridge doesnt list Home Assistant devices.
Looked for Web API to add a custom device in ha-bridge, could some one please help, ha-bridge is my hub for Alexa integration, hence need to add this AC control device in HA bridge.
Ref link: https://esphome.io/components/climate/ir_climate.html

Take a look at this link https://github.com/esphome/esphome-docs/blob/current/web-api/index.rst


data can never be lost, metadata can never be lost and always recovered

we should verify following behavior

data can never be lost, we can always rebuild to a working state even if it means e.g. loosing last 15min of data
we have all data required in TFChain KVS, so a user can always get back to a working config by having nothing more than private key
possibility have multiple storage configs, we can use this for multiple storage usecases at once by 1 user

In the context of QSFS, I can confirm that it's possible to recover with nothing more than the zstor config file. The hook script already stores the zdb metadata (index and namespace files), so recovering the QSFS is just a matter of restoring those files manually in order to create the necessary conditions for zdb to start requesting data blocks from zstor. We also have the auto rotate feature in zdb which enables setting a max time that data can potentially be lost from the frontend before getting uploaded.
I think we can forget about storing config in tfchain. It's simple enough for the user to store their entire zstor config in a password manager or secrets vault.
As for supporting multiple configs, I think this can already be achieved by running multiple copies of the zstor daemon, each with its own socket. If something more is needed in that regard we can define it in a new issue.


Management interface no listed by get_facts/get_interfaces/get_interfaces_ip methods

Hello,
The last 'napalm-dellos10' version doesn't report some interfaces type in the interface list table because the 'show interface status' doesn't report it.
This is the case for:

mgmt
port-channel

OS10# show interface
  ethernet      Ethernet interface number
  loopback      Loopback interface number
  mgmt          Management interface number
  null          Null interface number
  phy-eth       Ethernet IEEE 802.3
  port-channel  Port-channel number
  status        Show interface status
  vlan          VLAN ID number

OS10# show interface | grep "line protocol"
Ethernet 1/1/1 is up, line protocol is up
Ethernet 1/1/2 is up, line protocol is up
...
Ethernet 1/1/50 is down, line protocol is down
Ethernet 1/1/51 is down, line protocol is down
Ethernet 1/1/52 is down, line protocol is down
Ethernet 1/1/53 is up, line protocol is up
Ethernet 1/1/54 is up, line protocol is up
Management 1/1/1 is up, line protocol is up
Vlan 1 is up, line protocol is down
Port-channel 1 is up, line protocol is up
Port-channel 2 is up, line protocol is up
Port-channel 3 is up, line protocol is up
Port-channel 4 is up, line protocol is up
Port-channel 5 is up, line protocol is up
Port-channel 46 is up, line protocol is down
Port-channel 47 is up, line protocol is down
Port-channel 48 is up, line protocol is down
NULL is up, line protocol is up

This is because of the following command in the code:
output = self._send_command('show interface status | display-xml')
So if I change the command to test if againt the management interface, it is reported by the
get_interfaces NAPALM method:
       output = self._send_command('show interface mgmt | display-xml')
Result:
$ python napalm-dellos10-get_interfaces.py
collecting from device <IP_ADDRESS>
{   u'mgmt1/1/1': {   'description': u'',
                      'is_enabled': True,
                      'is_up': True,
                      'last_flapped': -1.0,
                      'mac_address': u'N/A',
                      'speed': -1}}

Could you add a second loop in the following methods?

get_facts
get_interfaces
get_interfaces_ip

Regards,
Ludovic

Thanks, I will test 1.0.6 version ASAP.


Allow passing S3 credentials via Environment Variables

Priority: ENV Variables -> AWS_PROFILE -> default

Will passing the region as an environment variable AWS_DEFAULT_REGION=... work in conjunction with the account id + secret supported in this PR?

Will passing the region as an environment variable AWS_DEFAULT_REGION=... work in conjunction with the account id + secret supported in this PR?

Sure. I have added support for AWS_DEFAULT_REGION as well in this PR.

👍 Thanks!


London | saqib_javed | Module-User-Focused-Data | week 1

wip.. working on my wireframe

Learners, PR Template
Self checklist

[ ] I have committed my files one by one, on purpose, and for a reason
[ ] I have titled my PR with COHORT_NAME | FIRST_NAME LAST_NAME | REPO_NAME | WEEK
[ ] I have tested my changes
[ ] My changes follow the style guide
[ ] My changes meet the requirements of this task

Changelist
Briefly explain your PR.
Questions
Ask any questions you have for your reviewer.

fix css path and wip.

@saqibjvd I've set your PR to draft as it's a WIP. Press "ready for review" when it's ready


Update PDF.js version

We're using an old version of PDF.js(1.5), which it have some problems to render certain pdfs. And we need to refactor the code to use the new features provided with the version 2.0. For this we need to use webpack

Done


Make daemonbuilder.Docker leaner.

Currently builder.Backend is implemented by daemonbuilder.Docker{} for
the daemon. This registration happens in the API/server code. However,
this is too implementation specific. Ideally we should be able to specify
that docker daemon (or any other) is implementing the Backend and abstract
the implementation details. So we should remove package daemonbuilder
dependency in build_routes.go
With this change, daemonbuilder.Docker is nothing more than the daemon.
A follow on change will remove the daemonbuilder package and move relevant
methods under daemon, so that API only knows about the backend.
Also cleanup code in api/client/build.go. docker cli always performs build
context tar download for remoteURLs and sends an empty remoteContext. So
remove relevant dead code.
Signed-off-by: Anusha Ragunathan<EMAIL_ADDRESS>

for the record: build timed out on userns: https://jenkins.dockerproject.org/job/Docker-PRs-userns/5038/console
Restarting it.

In addition to getting rid of daemonbuilder, I would also like to look into refactoring that output parameter.
LGTM

LGTM


Add a table for F# wrappers for popular .NET libraries

@Happypig375 this list already contains Giraffe and few other.
Section Wrappers around dotnet libs is not correct, couse F# is already .net, please use existing categories

@Neftedollar They are mixed within a lot of other smaller libraries, making the "standard wrappers" harder to find.

@Happypig375 You can add this smaller libraries without duplication and without category "wrapper". If you will do it, please send another PR or change this PR.

@Neftedollar What I mean is that

Person would like to look for the F# wrapper of some C# library.
This list is too long for effective searching.

@Neftedollar so what are the current thoughts? I think it's a useful list. Libraries made in C# are normally much more popular for software/desktop etc. development, so people might want to search by them, and seeing what F# equivalents those have.

merged, thanx


Feature suggestion: Connect to google via proxy

It would be great to have the option to connect to google via proxy server. This way google cannot see the IP (and therefore additional information) of the device.

Duplicate of #173


Should epoch/modifiedat be changed when no properties are updated?

when a PUT/POST/PATCH is done such that the entity doesn't have any properties updated (e.g. PUT .../GROUPs/gID. + {}), should the epoch and modifiedat properties be updated?
Proposal: yes for two reasons. 1) I think having consistent processing (ie. do a loop over all incoming props, and the when done update the epoch/modifiedat) is good - even if the loop exits immediately, 2) I think having a "touch"-like feature could be handy
I think we need to be explicit whatever way we decide so that we have interop (we're consistent) on whether a client's cache of an entity (based on epoch) will become stale or not based on this empty update operation.

Agreed on the 5/17 call

fixed by #124


raftstore: testcase test_check_pending_admin is unstable

This test case assumes that the region leader is always on store 1. However, the assumption does not hold.
The test case may fail with certain possibility. For example
https://ci.pingcap.net/blue/organizations/jenkins/tikv_ghpr_test/detail/tikv_ghpr_test/28025/tests

/type enhancement


Update to swift 403

Update to swift403

Codecov Report

Merging #10 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #10   +/-   ##
=======================================
  Coverage   56.52%   56.52%
=======================================
  Files           2        2
  Lines         184      184
  Branches       65       65
=======================================
  Hits          104      104
  Misses         80       80

Flag
Coverage Δ

#KituraContracts
56.52% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b90b39a...231fa27. Read the comment docs.


ZEVA - Error when analyst tries to submit credit agreement to Director

Describe the Bug
In the test environment, when an analyst ties to submit a credit agreement to the Director they receive a 'we're sorry' message. It appears to the user that the recommendation has failed.
https://zeva-test.apps.silver.devops.gov.bc.ca/credit-agreements/107
Expected Behaviour
The analyst click 'submit to director' for a credit agreement and it fails to submit and the user get a 'we're sorry' error message.
Actual Behaviour
The user is given a 'we're sorry' message when they click to recommend a credit agreement to the director.
Implications
It is not possible to submit a credit agreement to the Director so credit agreement cannot be issued to suppliers.
Steps To Reproduce
Steps to reproduce the behaviour:
User/Role:

Go to https://zeva-test.apps.silver.devops.gov.bc.ca/credit-agreements
Click on a draft credit agreement
Click to submit the agreement to the Director
See error message, and that it has failed to submit to director.

Dev Notes
This is a breaking organization reference in the send_email.py file so we just need to rework that bit of logic for this function to work.
Dev Logs
File "/opt/app-root/src/api/services/send_email.py", line 255, in get_subscribed_user_emails
Q(organization_id__in=[obj.organization, govt_org.id]) &
AttributeError: 'str' object has no attribute 'organization'
notifications_credit_agreement None

Hey team! Please add your planning poker estimate with Zenhub @AlexZorkin @jig-patel @JulianForeman @prv-proton @tim738745

Fixed as part of zeva card 1783


Should not ask user to clean cache each time the code upgrades.

shoud set some attributes
such as

to avoid it.

I think we could trigger browser to reload upgrades by changing JS file harbor.app.min.js name with random content which included in header of HTML at the UI image building stage, and also need to update the tag script attribute src against to the changed name.

Fixed with PR #1081


New docker image for cellranger 6.0.0

This PR addresses #55 which requires addition of a new docker image for cellranger 6.0.0 to be used for benchmarking pre-processing of ScPCA data.
The old dockerfile used for cellranger 4.0.0 was moved to a new file named cellranger4.docker and the new Dockerfile contains the image for running cellranger 6.0.0. The docker image was built and tested using the following command: docker run -it --rm scpca-cellranger:6.0.0 cellranger testrun --id=tiny to ensure that it successfully can run cellranger 6.0.0.
Additionally, it appears that the reference files for cellranger have not been updated since July 7, 2020 and the reference files located on S3 were added in October, 2020. The STARsolo website also doesn't mention any need for a new reference index. I think it is safe to assume that we do not need to update the reference. https://support.10xgenomics.com/single-cell-gene-expression/software/downloads/latest
Things that have not been done:

this updated image has not been pushed to AWS ECR
the nextflow workflow workflows/cellranger-quant/run-cellranger.nf has not been updated yet and will be addressed in a new issue

Actually... I realized that there is one more step that needs to be done... updating images/cellranger/README.md to reflect the changes. You can either do that in this PR as well before merging, or in a separate PR.


HtmlAttributeCollection: Implementation for ICollection.Remove(HtmlAttribute) violates interface contract

1. Description
When trying to remove an attribute instance that's NOT in a HtmlAttributeCollection through the explicit interface implementation for ICollection<HtmlAttribute>.Remove(HtmlAttribute), it is expected that the method returns false.
However, as implemented in HAP 1.11.41 (implementing a fix for https://github.com/zzzprojects/html-agility-pack/issues/463), it throws an exception instead of returning false.
The ICollection<HtmlAttribute>.Remove(HtmlAttribute) implementation in HtmlAttributeCollection thus violates the ICollection<T>.Remove(T item) interface contract. The contract mandates (https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.icollection-1.remove?system-collections-generic-icollection-1-remove(-0)):

Returns true if item was successfully removed from the ICollection<T>; otherwise, false.
This method also returns false if item is not found in the original ICollection<T>.

2. Exception
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at HtmlAgilityPack.HtmlAttributeCollection.Remove(HtmlAttribute attribute) in C:\Repos\HtmlAgilityPack\HtmlAgilityPack.Shared\HtmlAttributeCollection.cs:line 343
   at HtmlAgilityPack.HtmlAttributeCollection.System.Collections.Generic.ICollection<HtmlAgilityPack.HtmlAttribute>.Remove(HtmlAttribute item) in C:\Repos\HtmlAgilityPack\HtmlAgilityPack.Shared\HtmlAttributeCollection.cs:line 233
   at HtmlAttributeCollection_RemoveAndClearNotUpdatingHashItems.Program.Main(String[] args)

3. Fiddle or Project
dotnetfiddle: https://dotnetfiddle.net/lKA3kg

Click to expand the source code
var html = @"
Hello world!
";

var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(html);
var divNode = htmlDoc.DocumentNode.SelectSingleNode("//div");

var attr = divNode.Attributes["class"];

ICollection<HtmlAttribute> attributeCollectionAsICollection = (ICollection<HtmlAttribute>) divNode.Attributes;

//
// Remove an attribute instance that is in the Attributes collection
// Expected return value: true
//
var res1 = attributeCollectionAsICollection.Remove(attr);
Console.WriteLine($"Return value after successful removal: {res1}   (should be true)");

//
// Now trying to remove an attribute instance that is not in the Attributes collection (anymore)
// Expected return value: false
//
var res2 = attributeCollectionAsICollection.Remove(attr);
Console.WriteLine($"Return value after successful removal: {res2}   (should be false)");

4. Any further technical details

HAP version: 1.11.41
.NET 6.0

Hello @elgonzo ,
Thank you for reporting, you are right.
We updated the method to return false correctly when the item is not or doesn't exists in the latest version v1.11.42
Best Regards,
Jon

I can confirm that the issue is fixed in 1.11.42. :-)


Distinct event trigger for Cron pipelines

I can't skip some pipelines/steps during cron scheduling.
When I use cron jobs, I can set pipeline trigger with cronjob name to include this pipline during cron execution.
trigger:
  cron: ['cron']

But I can't exclude some pipelines/steps from all cronjobs.
I tried to use something like
trigger:
  cron:
    exclude: '*'

to exclude cronjobs from common pipelines for pushing code to repo.
But when i commit to repo, I'm getting trigger: skipping pipeline, does not match cron job.
It would be nice to add event type 'cron' to avoid this problem.
I think it'll look like
trigger:
  event:
    exclude: 'cron'

Thanks

I noticed that DRONE_COMMIT_BEFORE and DRONE_SOURCE_BRANCH are empty when build is triggered by cron. Could this be used to reliably distinguish between regular builds and cron-triggered ones ? Or this is something that will be changed in future versions ?

added support for event: cron in master.


Get-RsCatalogItemRole: fix bug that would always return false on inherited security

Small bugfix to fix the fact that Get-RsCatalogItemRole always returns false when not used with -Recurse on the ParentSecurity property, even if permissions are inherited.

I was having this issue on Version <IP_ADDRESS>.  After uninstalling and then reinstalling to Version <IP_ADDRESS>, I am getting the correct results!  Thank you!!

Ah, I understand why - this is querying the current roles.   Oops.


Alert properties list not complete

The list under the Alert "Properties" section does not include "rbacGroupName", "mitreTechniques", and "relatedUser"; however, these all show in the example.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f71b65fe-28c7-42c1-ec6c-cffb63014eb9
Version Independent ID: e23e1eac-de5e-7c99-2e46-89b4ffd32d4b
Content: Get alerts API
Content Source: microsoft-365/security/defender-endpoint/alerts.md
Product: m365-security
GitHub Login: @mjcaparas
Microsoft Alias: macapara

@MastaDisasta Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/6036
Thanks
Sri

@msbemba Thank you very much for the contribution and updating the documentation with PR. @MastaDisasta Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Refactor install to use a function table

Some more refactoring can still be done, but hopefully a good start.

@asalkeld evidently there are additional reasonably straightforward steps that can build on top of this PR - let's get this one merged as a starting point as it's in good shape.  You may want to flesh out the steps in question and co-ordinate with an onboarding team member (@nilsanderselde?) to do some or all of them?
Examples:

pull the client setups out of the mini-functions and get the clients into the Installer struct; probably the graph as well
write unit tests, starting with the easy mini-functions, merge PRs bit by bit
there are more places where the bigger functions can be broken down into smaller pieces and unit tests added

@asalkeld evidently there are additional reasonably straightforward steps that can build on top of this PR - let's get this one merged as a starting point as it's in good shape. You may want to flesh out the steps in question and co-ordinate with an onboarding team member (@nilsanderselde?) to do some or all of them?
Examples:
* pull the client setups out of the mini-functions and get the clients into the Installer struct; probably the graph as well

* write unit tests, starting with the easy mini-functions, merge PRs bit by bit

* there are more places where the bigger functions can be broken down into smaller pieces and unit tests added

Sure, I'll clean this up and write up a work item.

should be ready to merge

/lgtm


Fixed CUDA 11.1 compilation.

Fixes:

::floor() -> ::floorf() which should be same as float is used, yet otherwise compiler complains host function can't be used from global.

Which compiler are you using? Just wondering because I've been using CUDA 11.1 with GCC 10.2 without any issues.

Good point, I assumed this affects every compiler on top of CUDA. I use MSVC 16.9.0 Preview 2.0, 19.28.29617.
Also VCPKG version won't compile with now (with e.g. dlib[core,cuda]:x64-windows).

Yeah, different compilers and platforms are the bane of cross-platform code :)


Any plans to add auth?

Does anyone have any good resources on an auth example, the only one I know that exists is really messy.

We could certainly extend this to include auth. Do you have any specific requirements? Are you just focussed on username/password auth, or use of 3rd party oauth services?

Just basic Auth, and with Stormpath - since I love this stack so much, I'd like to do some ports with Vue.js as well - I will let you know!

it would be cool to use passport + jwt auth

Hi guys! We have added jwt-auth to our fork. Check this out


Compass light theme

Compass stays dark gray color in white theme mode which makes it hard to read in bright sunlight on the trail even with max brightness. Maybe use black, red, and blue text? In Dark mode everything is readable in use cases such as ie. a cave.

Developer notes:

Should be configurable in settings (Navigation)
Default to following app theme
Ensure the color works with all beacon and path colors


4.6.0

:x: Build TheIdServer 4.5.4-ci1-0034 failed (commit https://github.com/Aguafrommars/TheIdServer/commit/07b1e2b446 by @aguacongas)

:white_check_mark: Build TheIdServer 4.5.4-ci.1+34.build.2103 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/07b1e2b446 by @aguacongas)


Swift 4 support

Compilation fails with error method 'initialize()' defines Objective-C class method 'initialize', which is not permitted by Swift

Yes, actually you don't need this anymore, cause in Xcode9, there is a main thread checker enabled by default, function is same as this one


[Tool] add health check for minio and zeppelin containers in starrocks-stack-compose

…s-stack-compose
What type of PR is this：

[ ] BugFix
[ ] Feature
[X] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[X] Tool

Which issues of this PR fixes ：
add health check for minio and zeppelin containers in starrocks-stack-compose
Fixes
Problem Summary(Required) ：
ditto
Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)
[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[X] I have checked the version labels which the pr will be auto backported to target branch

[X] 2.5
[ ] 2.4
[ ] 2.3
[ ] 2.2

https://github.com/Mergifyio backport branch-2.5

https://github.com/Mergifyio backport branch-2.5

https://github.com/Mergifyio backport branch-2.5

@mergify backport branch-2.5


chore: add check for navigator locks

Description
Currently, apps do not work as expected when users use an insecure context (e.g. http://<IP_ADDRESS>) which results in confusing behaviour for users (e.g. https://discord.com/channels/1138230179878154300/1291056749197131796).
This PR adds a check to confirm that navigator locks are present and throws an error otherwise. This should avoid users experiencing the above mentioned issue.
This also includes vitest updates as they are required to avoid false positives see here (https://github.com/vitest-dev/vitest/issues/6187)
Testing
I've added unit tests to confirm the behaviour

@rkistner based on this https://discord.com/channels/1138230179878154300/1291056749197131796 the general issue is an unsecure context being used (e.g. 192.168...) which results in hard to diagnose issues for users. The idea with this is to warn users when navigator locks are not available and to then have a fallback so they can still continue albeit with more limited functionality.

Ok, I wasn't aware of the secure context limitation.
Is it feasible to limit this to only "safe" setups, such as when using a SharedWorker? I'm concerned that silent data corruption because of a lack of cross-tab locks could be even more difficult to debug.

I did some digging and found that WA-SQLite also uses the navigator.locks API internally. From what I could see Navigator locks are currently unavoidable if we use the IDBBatchAtomicVFS VFS which uses WebLocks under the hood.

HI there, is this work complete? And if so do you have any idea when it will be released? I think I am in a situation where this would benefit me, I am building a nextjs application and then converting it to ios and android app using capacitor. Powersync seems to work well when the apps are bundled fully, but during the development process I am pointing to local host and using the nextjs instance that is running there, which enables hot reloading of my changes in xcode and android studio, this method doesn't work with powersync, I get an error that mentions navigator locks.
Apologies if this is unrelated, but would be a massive help if your changes solve it! happy to test them on this use case if you can point me in the direction of a package with the changes implemented.

This is released already, but it only checks that navigator.locks are present, giving a better error message when it isn't.
Even if we removed usage of the locks here, wa-sqlite still uses the same navigator.locks internally, so it is something that is required.
You say you're pointing to local host in development - is that http://localhost:<port>, or for example using your WiFi IP? localhost should work as far as I know, but the WiFi IP won't. At least for Android you can use adb reverse to expose your development server as an actual http://localhost URL on the device. Not sure if there is an equivalent for iOS.

@rkistner Thanks very much for the response, I will give that a try


use instrinsics for simd equality

this does exactly what the //TODO comments planned to do.
special thanks to @bjorn3 for helping me understand the simd instrinsics and the compiler errors, and proposing solutions (#208).

I think it's worth waiting for #206, since that will definitely affect this implementation.

Closed in #274.