pihole doesn't respond to DNS queries without container restart

After a full restart of my RPi3 (shutdown and power off/on), pihole doesn't respond to DNS queries until I restart the pihole container again.  Logs/testing output below:
Logs from power-on:
20.02.21 12:50:19 (-0600) Supervisor starting
20.02.21 12:49:55 (-0600)  pihole  [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
20.02.21 12:49:56 (-0600)  pihole  [s6-init] ensuring user provided files have correct perms...exited 0.
20.02.21 12:49:56 (-0600)  pihole  [fix-attrs.d] applying ownership & permissions fixes...
20.02.21 12:49:56 (-0600)  pihole  [fix-attrs.d] 01-resolver-resolv: applying...
20.02.21 12:49:56 (-0600)  pihole  [fix-attrs.d] 01-resolver-resolv: exited 0.
20.02.21 12:49:56 (-0600)  pihole  [fix-attrs.d] done.
20.02.21 12:49:56 (-0600)  pihole  [cont-init.d] executing container initialization scripts...
20.02.21 12:49:56 (-0600)  pihole  [cont-init.d] 20-start.sh: executing...
20.02.21 12:49:57 (-0600)  pihole   ::: Starting docker specific checks & setup for docker pihole/pihole
20.02.21 12:49:57 (-0600)  pihole
20.02.21 12:49:57 (-0600)  pihole    [i] Installing configs from /etc/.pihole...
20.02.21 12:49:58 (-0600)  pihole    [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
20.02.21 12:49:58 (-0600)  pihole    [i] Copying 01-pi  [â] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf
20.02.21 12:49:59 (-0600)  pihole  Converting DNS1 to PIHOLE_DNS_
20.02.21 12:49:59 (-0600)  pihole  Converting DNS2 to PIHOLE_DNS_
20.02.21 12:49:59 (-0600)  pihole  Setting DNS servers based on PIHOLE_DNS_ variable
20.02.21 12:49:59 (-0600)  pihole  ::: Pre existing WEBPASSWORD found
20.02.21 12:49:59 (-0600)  pihole  DNSMasq binding to default interface: eth0
20.02.21 12:50:00 (-0600)  pihole  Added ENV to php:
20.02.21 12:50:00 (-0600)  pihole                    "PHP_ERROR_LOG" => "/var/log/lighttpd/error.log",
20.02.21 12:50:00 (-0600)  pihole                    "ServerIP" => "<IP_ADDRESS>",
20.02.21 12:50:00 (-0600)  pihole                    "VIRTUAL_HOST" => "<IP_ADDRESS>",
20.02.21 12:50:00 (-0600)  pihole  Using IPv4 and IPv6
20.02.21 12:50:00 (-0600)  pihole  ::: Preexisting ad list /etc/pihole/adlists.list detected ((exiting setup_blocklists early))
20.02.21 12:50:00 (-0600)  pihole  https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
20.02.21 12:50:00 (-0600)  pihole  ::: Testing pihole-FTL DNS: sudo: unable to resolve host 7ed1bcc: Temporary failure in name resolution
20.02.21 12:50:06 (-0600)  pihole  FTL started!
20.02.21 12:50:07 (-0600)  pihole  ::: Testing lighttpd config: Syntax OK
20.02.21 12:50:07 (-0600)  pihole  ::: All config checks passed, cleared for startup ...
20.02.21 12:50:07 (-0600)  pihole  ::: Enabling Query Logging
20.02.21 12:50:07 (-0600)  pihole    [i] Enabling logging...
20.02.21 12:50:07 (-0600)  pihole    [i] Restarting DN  [â] Restarting DNS server
  [â] Logging has been enabled!
20.02.21 12:50:07 (-0600)  pihole   ::: Docker start setup complete
20.02.21 12:50:08 (-0600)  pihole    [i] Neutrino emissions detected...
  [â] Pulling blocklist source list into range
20.02.21 12:50:08 (-0600)  pihole
20.02.21 12:50:08 (-0600)  pihole    [i] Preparing new  [â] Preparing new gravity database
20.02.21 12:50:08 (-0600)  pihole    [i] Using libz compression
20.02.21 12:50:08 (-0600)  pihole
20.02.21 12:50:08 (-0600)  pihole    [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
20.02.21 12:50:08 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:08 (-0600)  pihole    [â] List download failed: using previously cached list
20.02.21 12:50:10 (-0600)  pihole    [i] Received 62904 domains
20.02.21 12:50:10 (-0600)  pihole
20.02.21 12:50:10 (-0600)  pihole    [i] Target: https://mirror1.malwaredomains.com/files/justdomains
20.02.21 12:50:10 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:10 (-0600)  pihole    [â] List download failed: no cached list available
20.02.21 12:50:10 (-0600)  pihole
20.02.21 12:50:10 (-0600)  pihole    [i] Target: http://sysctl.org/cameleon/hosts
20.02.21 12:50:10 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:10 (-0600)  pihole    [â] List download failed: using previously cached list
20.02.21 12:50:11 (-0600)  pihole    [i] Received 20566 domains
20.02.21 12:50:11 (-0600)  pihole
20.02.21 12:50:11 (-0600)  pihole    [i] Target: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
20.02.21 12:50:11 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:11 (-0600)  pihole    [â] List download failed: using previously cached list
20.02.21 12:50:11 (-0600)  pihole    [i] Received 0 domains
20.02.21 12:50:11 (-0600)  pihole
20.02.21 12:50:11 (-0600)  pihole    [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
20.02.21 12:50:12 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:12 (-0600)  pihole    [â] List download failed: using previously cached list
20.02.21 12:50:12 (-0600)  pihole    [i] Received 34 domains
20.02.21 12:50:12 (-0600)  pihole
20.02.21 12:50:12 (-0600)  pihole    [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
20.02.21 12:50:12 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:12 (-0600)  pihole    [â] List download failed: using previously cached list
20.02.21 12:50:12 (-0600)  pihole    [i] Received 2701 domains
20.02.21 12:50:12 (-0600)  pihole
20.02.21 12:50:12 (-0600)  pihole    [i] Target: https://hosts-file.net/ad_servers.txt
20.02.21 12:50:13 (-0600)  pihole    [i] Status: Pendi  [â] Status: Connection Refused
20.02.21 12:50:13 (-0600)  pihole    [â] List download failed: no cached list available
20.02.21 12:50:13 (-0600)  pihole
20.02.21 12:50:14 (-0600)  pihole    [i] Storing downl  [â] Storing downloaded domains in new gravity database
20.02.21 12:50:15 (-0600)  pihole    [i] Building tree  [â] Building tree
20.02.21 12:50:15 (-0600)  pihole    [i] Swapping data  [â] Swapping databases
20.02.21 12:50:17 (-0600)  pihole    [i] Number of gravity domains: 86205 (74489 unique domains)
20.02.21 12:50:17 (-0600)  pihole    [i] Number of exact blacklisted domains: 1
20.02.21 12:50:17 (-0600)  pihole    [i] Number of regex blacklist filters: 0
20.02.21 12:50:17 (-0600)  pihole    [i] Number of exact whitelisted domains: 108
20.02.21 12:50:17 (-0600)  pihole    [i] Number of regex whitelist filters: 0
20.02.21 12:50:17 (-0600)  pihole    [i] Flushing DNS   [â] Flushing DNS cache
20.02.21 12:50:17 (-0600)  pihole    [i] Cleaning up s  [â] Cleaning up stray matter
20.02.21 12:50:17 (-0600)  pihole
20.02.21 12:50:18 (-0600)  pihole    [â] DNS service is listening
20.02.21 12:50:18 (-0600)  pihole       [â] UDP (IPv4)
20.02.21 12:50:18 (-0600)  pihole       [â] TCP (IPv4)
20.02.21 12:50:18 (-0600)  pihole       [â] UDP (IPv6)
20.02.21 12:50:18 (-0600)  pihole       [â] TCP (IPv6)
20.02.21 12:50:18 (-0600)  pihole
20.02.21 12:50:18 (-0600)  pihole    [â] Pi-hole blocking is enabled
20.02.21 12:50:18 (-0600)  pihole    Pi-hole version is v5.2.4 (Latest: v5.2.4)
20.02.21 12:50:18 (-0600)  pihole    AdminLTE version is v5.4 (Latest: v5.4)
20.02.21 12:50:18 (-0600)  pihole    FTL version is v5.7 (Latest: v5.7)
20.02.21 12:50:18 (-0600)  pihole  [cont-init.d] 20-start.sh: exited 0.
20.02.21 12:50:18 (-0600)  pihole  [cont-init.d] done.
20.02.21 12:50:18 (-0600)  pihole  [services.d] starting services
20.02.21 12:50:18 (-0600)  pihole  Starting crond
20.02.21 12:50:18 (-0600)  pihole  Starting lighttpd
20.02.21 12:50:18 (-0600)  pihole  Starting PADD
20.02.21 12:50:18 (-0600)  pihole  Starting pihole-FTL (no-daemon) as root
20.02.21 12:50:18 (-0600)  pihole  [services.d] done.
20.02.21 12:49:59 (-0600)  unbound  [1613846999] unbound[1:0] notice: init module 0: validator
20.02.21 12:49:59 (-0600)  unbound  [1613846999] unbound[1:0] notice: init module 1: iterator
20.02.21 12:49:59 (-0600)  unbound  [1613846999] unbound[1:0] info: start of service (unbound 1.13.0).
20.02.21 12:50:21 (-0600) Starting service 'dbus sha256:ca6de21b82870c1b7278e017977d90be0fcf1c1bdb61ac7e059a6b194881cea5'
20.02.21 12:58:54 (-0600) Started service 'dbus sha256:ca6de21b82870c1b7278e017977d90be0fcf1c1bdb61ac7e059a6b194881cea5'
20.02.21 12:58:54 (-0600)  dbus  method return time=1613847534.437242 sender=:1.0 -> destination=:1.89 serial=585 reply_serial=2
20.02.21 12:58:54 (-0600)  dbus     object path "/org/freedesktop/systemd1/job/707"
20.02.21 12:58:54 (-0600)  dbus  method return time=1613847534.437242 sender=:1.0 -> destination=:1.89 serial=585 reply_serial=2
20.02.21 12:58:54 (-0600)  dbus     object path "/org/freedesktop/systemd1/job/707"
20.02.21 12:58:56 (-0600) Service exited 'dbus sha256:ca6de21b82870c1b7278e017977d90be0fcf1c1bdb61ac7e059a6b194881cea5'

DNS query:
bash-3.2$ nslookup
> server <IP_ADDRESS>
Default server: <IP_ADDRESS>
Address: <IP_ADDRESS>#53
> google.com
;; connection timed out; no servers could be reached
>

Logs following pihole container restart:
20.02.21 13:02:07 (-0600) Killing service 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:07 (-0600)  pihole  Stopping PADD
20.02.21 13:02:07 (-0600)  pihole  Stopping lighttpd
20.02.21 13:02:07 (-0600)  pihole  Stopping pihole-FTL
20.02.21 13:02:07 (-0600)  pihole  padd.sh: no process found
20.02.21 13:02:07 (-0600)  pihole  Stopping cron
20.02.21 13:02:07 (-0600)  pihole  [cont-finish.d] executing container finish scripts...
20.02.21 13:02:07 (-0600)  pihole  [cont-finish.d] done.
20.02.21 13:02:07 (-0600)  pihole  [s6-finish] waiting for services.
20.02.21 13:02:08 (-0600)  pihole  [s6-finish] sending all processes the TERM signal.
20.02.21 13:02:11 (-0600)  pihole  [s6-finish] sending all processes the KILL signal and exiting.
20.02.21 13:02:12 (-0600) Service exited 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:12 (-0600) Killed service 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:13 (-0600) Installing service 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:13 (-0600) Installed service 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:13 (-0600) Starting service 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:15 (-0600) Started service 'pihole sha256:4f00e2485051978fffdfb56d069f40da2b1e83ad1ef399926cc5ac157cc48264'
20.02.21 13:02:15 (-0600)  pihole  [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
20.02.21 13:02:15 (-0600)  pihole  [s6-init] ensuring user provided files have correct perms...exited 0.
20.02.21 13:02:15 (-0600)  pihole  [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
20.02.21 13:02:15 (-0600)  pihole  [s6-init] ensuring user provided files have correct perms...exited 0.
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] applying ownership & permissions fixes...
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] 01-resolver-resolv: applying...
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] 01-resolver-resolv: exited 0.
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] done.
20.02.21 13:02:15 (-0600)  pihole  [cont-init.d] executing container initialization scripts...
20.02.21 13:02:15 (-0600)  pihole  [cont-init.d] 20-start.sh: executing...
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] applying ownership & permissions fixes...
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] 01-resolver-resolv: applying...
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] 01-resolver-resolv: exited 0.
20.02.21 13:02:15 (-0600)  pihole  [fix-attrs.d] done.
20.02.21 13:02:15 (-0600)  pihole  [cont-init.d] executing container initialization scripts...
20.02.21 13:02:15 (-0600)  pihole  [cont-init.d] 20-start.sh: executing...
20.02.21 13:02:15 (-0600)  pihole   ::: Starting docker specific checks & setup for docker pihole/pihole
20.02.21 13:02:16 (-0600)  pihole
20.02.21 13:02:16 (-0600)  pihole    [i] Installing configs from /etc/.pihole...
20.02.21 13:02:16 (-0600)  pihole    [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
20.02.21 13:02:16 (-0600)  pihole    [i] Copying 01-pihole.  [â] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf
20.02.21 13:02:17 (-0600)  pihole  Converting DNS1 to PIHOLE_DNS_
20.02.21 13:02:17 (-0600)  pihole  Converting DNS2 to PIHOLE_DNS_
20.02.21 13:02:17 (-0600)  pihole  Setting DNS servers based on PIHOLE_DNS_ variable
20.02.21 13:02:17 (-0600)  pihole  ::: Pre existing WEBPASSWORD found
20.02.21 13:02:17 (-0600)  pihole  DNSMasq binding to default interface: eth0
20.02.21 13:02:17 (-0600)  pihole  Added ENV to php:
20.02.21 13:02:17 (-0600)  pihole                       "PHP_ERROR_LOG" => "/var/log/lighttpd/error.log",
20.02.21 13:02:17 (-0600)  pihole                       "ServerIP" => "<IP_ADDRESS>",
20.02.21 13:02:17 (-0600)  pihole                       "VIRTUAL_HOST" => "<IP_ADDRESS>",
20.02.21 13:02:18 (-0600)  pihole  Using IPv4 and IPv6
20.02.21 13:02:18 (-0600)  pihole  ::: Preexisting ad list /etc/pihole/adlists.list detected ((exiting setup_blocklists early))
20.02.21 13:02:18 (-0600)  pihole  https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
20.02.21 13:02:18 (-0600)  pihole  ::: Testing pihole-FTL DNS: sudo: unable to resolve host 7ed1bcc: Name or service not known
20.02.21 13:02:19 (-0600)  pihole  FTL started!
20.02.21 13:02:19 (-0600)  pihole  ::: Testing lighttpd config: Syntax OK
20.02.21 13:02:19 (-0600)  pihole  ::: All config checks passed, cleared for startup ...
20.02.21 13:02:19 (-0600)  pihole  ::: Enabling Query Logging
20.02.21 13:02:19 (-0600)  pihole    [i] Enabling logging...
20.02.21 13:02:19 (-0600)  pihole    [i] Restarting DNS se  [â] Restarting DNS server
  [â] Logging has been enabled!
20.02.21 13:02:19 (-0600)  pihole   ::: Docker start setup complete
20.02.21 13:02:20 (-0600)  pihole    [i] Neutrino emissions detected...
  [â] Pulling blocklist source list into range
20.02.21 13:02:20 (-0600)  pihole
20.02.21 13:02:20 (-0600)  pihole    [i] Preparing new gra  [â] Preparing new gravity database
20.02.21 13:02:20 (-0600)  pihole    [i] Using libz compression
20.02.21 13:02:20 (-0600)  pihole
20.02.21 13:02:20 (-0600)  pihole    [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
20.02.21 13:02:20 (-0600)  pihole    [i] Status: Pending..  [â] Status: Retrieval successful
20.02.21 13:02:21 (-0600)  pihole    [i] Received 62904 domains
20.02.21 13:02:21 (-0600)  pihole
20.02.21 13:02:21 (-0600)  pihole    [i] Target: https://mirror1.malwaredomains.com/files/justdomains
20.02.21 13:02:21 (-0600)  pihole    [i] Status: Pending..  [â] Status: Not found
20.02.21 13:02:21 (-0600)  pihole    [â] List download failed: no cached list available
20.02.21 13:02:21 (-0600)  pihole
20.02.21 13:02:21 (-0600)  pihole    [i] Target: http://sysctl.org/cameleon/hosts
20.02.21 13:02:22 (-0600)  pihole    [i] Status: Pending..  [â] Status: No changes detected
20.02.21 13:02:22 (-0600)  pihole    [i] Received 20566 domains
20.02.21 13:02:22 (-0600)  pihole
20.02.21 13:02:22 (-0600)  pihole    [i] Target: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
20.02.21 13:02:23 (-0600)  pihole    [i] Status: Pending..  [â] Status: Retrieval successful
20.02.21 13:02:23 (-0600)  pihole    [i] Received 0 domains
20.02.21 13:02:23 (-0600)  pihole
20.02.21 13:02:23 (-0600)  pihole    [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
20.02.21 13:02:23 (-0600)  pihole    [i] Status: Pending..  [â] Status: No changes detected
20.02.21 13:02:23 (-0600)  pihole    [i] Received 34 domains
20.02.21 13:02:23 (-0600)  pihole
20.02.21 13:02:23 (-0600)  pihole    [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
20.02.21 13:02:24 (-0600)  pihole    [i] Status: Pending..  [â] Status: No changes detected
20.02.21 13:02:24 (-0600)  pihole    [i] Received 2701 domains
20.02.21 13:02:24 (-0600)  pihole
20.02.21 13:02:24 (-0600)  pihole    [i] Target: https://hosts-file.net/ad_servers.txt
20.02.21 13:02:24 (-0600)  pihole    [i] Status: Pending..  [â] Status: Not found
20.02.21 13:02:24 (-0600)  pihole    [â] List download failed: no cached list available
20.02.21 13:02:24 (-0600)  pihole
20.02.21 13:02:25 (-0600)  pihole    [i] Storing downloade  [â] Storing downloaded domains in new gravity database
  [â] Building tree
20.02.21 13:02:26 (-0600)  pihole    [i] Swapping database  [â] Swapping databases
20.02.21 13:02:27 (-0600)  pihole    [i] Number of gravity domains: 86205 (74489 unique domains)
20.02.21 13:02:27 (-0600)  pihole    [i] Number of exact blacklisted domains: 1
20.02.21 13:02:27 (-0600)  pihole    [i] Number of regex blacklist filters: 0
20.02.21 13:02:27 (-0600)  pihole    [i] Number of exact whitelisted domains: 108
20.02.21 13:02:27 (-0600)  pihole    [i] Number of regex whitelist filters: 0
20.02.21 13:02:27 (-0600)  pihole    [i] Flushing DNS cach  [â] Flushing DNS cache
20.02.21 13:02:27 (-0600)  pihole    [i] Cleaning up stray  [â] Cleaning up stray matter
20.02.21 13:02:27 (-0600)  pihole
20.02.21 13:02:28 (-0600)  pihole    [â] DNS service is listening
20.02.21 13:02:28 (-0600)  pihole       [â] UDP (IPv4)
20.02.21 13:02:28 (-0600)  pihole       [â] TCP (IPv4)
20.02.21 13:02:28 (-0600)  pihole       [â] UDP (IPv6)
20.02.21 13:02:28 (-0600)  pihole       [â] TCP (IPv6)
20.02.21 13:02:28 (-0600)  pihole
20.02.21 13:02:28 (-0600)  pihole    [â] Pi-hole blocking is enabled
20.02.21 13:02:28 (-0600)  pihole    Pi-hole version is v5.2.4 (Latest: v5.2.4)
20.02.21 13:02:28 (-0600)  pihole    AdminLTE version is v5.4 (Latest: v5.4)
20.02.21 13:02:29 (-0600)  pihole    FTL version is v5.7 (Latest: v5.7)
20.02.21 13:02:29 (-0600)  pihole  [cont-init.d] 20-start.sh: exited 0.
20.02.21 13:02:29 (-0600)  pihole  [cont-init.d] done.
20.02.21 13:02:29 (-0600)  pihole  [services.d] starting services
20.02.21 13:02:29 (-0600)  pihole  Starting lighttpd
20.02.21 13:02:29 (-0600)  pihole  Starting pihole-FTL (no-daemon) as root
20.02.21 13:02:29 (-0600)  pihole  Starting PADD
20.02.21 13:02:29 (-0600)  pihole  Starting crond
20.02.21 13:02:29 (-0600)  pihole  [services.d] done.

DNS query following pihole container restart:
bash-3.2$ nslookup
> server <IP_ADDRESS>
Default server: <IP_ADDRESS>
Address: <IP_ADDRESS>#53
> google.com
Server:		<IP_ADDRESS>
Address:	<IP_ADDRESS>#53

Non-authoritative answer:
Name:	google.com
Address: <IP_ADDRESS>
>

Thanks for the logs, that's super helpful! I see that on first boot it is unable to update any of the blocklists (Status: Connection Refused) but on restart the updates work (Status: No changes detected).
On it's own this wouldn't be a problem but it indicates that on first start the pihole container is having connection issues. Have you modified the docker-compose file at all? Can you share some of your env vars?

DNSMASQ_LISTENING
INTERFACE
DNS1
DNS2
others? (not webpassword)

Also could you try setting the ServerIP variable to the LAN IP of your device and see if the behavior changes?

Wow, good catch - I'm not sure how I missed that difference.  I saw some "Connection Refused" messages but didn't notice that it was for every blocklist and kind of got used to seeing them.
No modification to docker-compose, especially while testing these branches for you.  Once I get a stable version, I do change some stuff in pihole/Dockerfile to pull in a hosts file and change the font size, but again I've been careful not to do that while testing.
Environment variables:
DNS1 <IP_ADDRESS>
DNS2 <IP_ADDRESS>
DNSMASQ_LISTENING eth0
INTERFACE eth0
TZ America/Chicago
Just added ServerIP and I'm testing now.  Is that a recent addition?

No luck after adding ServerIP - I was really hoping for it to be that simple :-)

Just to add - there's nothing special about my network:
The RPi3b is wired directly into a Cisco 9200 switch with no auth configured, so the port comes up instantly.

ServerIP has been required since v4.2.2. I'll remove the optional label from the README. I also see a bunch of other env vars that are being deprecated so I'll have to clean that up soon.
https://github.com/pi-hole/docker-pi-hole#deprecated-environment-variables

Starting with the v4.1.1 release your Pi-hole container may encounter issues starting the DNS service unless ran with the following setting: --dns=<IP_ADDRESS> --dns=<IP_ADDRESS>
The second server can be any DNS IP of your choosing, but the first dns must be <IP_ADDRESS>

https://github.com/pi-hole/docker-pi-hole#docker-pi-hole-v411
You're not blocking <IP_ADDRESS> on your network in any way?

Oh, man - Their Docker hub docs are a bit out of sync with the GutHub docs - https://hub.docker.com/r/pihole/pihole.
I need to revisit my "new" pihole device now (even though it's working fine).
Also, I'm reading this as ServerIP not being required unless we're running host mode.  Does Balena run host mode by default?

Docker Pi-Hole v4.2.2
ServerIP no longer a required environment variable unless you run network 'host' mode! Feel free to remove it unless you need it to customize lighttpd

As for the DNS requirement - This is where my knowledge (or lack of) of Balena breaks down.  Looking at docker-compose.yml it seems that we're covered:
dns:
      - "<IP_ADDRESS>"
      - "<IP_ADDRESS>"
    environment:
      DNSMASQ_LISTENING: eth0
      INTERFACE: eth0
      DNS1: <IP_ADDRESS>#5053
      DNS2: <IP_ADDRESS>#5053

Now, I'm guessing that the environment variables in the GUI override these settings (where does that magic happen?).
The thing is, I want to run OpenDNS as my upstream DNS (I work for Cisco).  How do we reconcile that with the <IP_ADDRESS> requirement?
With the new method, I guess it would be like this:
PIHOLE_DNS_: <IP_ADDRESS>#5053;<IP_ADDRESS>;<IP_ADDRESS>
Now, do I have to wait until Balena Pihole gets an update before I can use that?
(<IP_ADDRESS> is fully accessible in my network, I use it all the time to test web redirect portals)

I decided to try to set DNS1 to <IP_ADDRESS> and am still having the problem, so it wasn't that.

FYI - on my dedicated RPi running pihole, this syntax worked for me and pihole was even "smart" enough to recognize OpenDNS as the upstream
PIHOLE_DNS_ =  <IP_ADDRESS>;<IP_ADDRESS>;<IP_ADDRESS>

Also, I'm reading this as ServerIP not being required unless we're running host mode. Does Balena run host mode by default? (Should it?)

Balena does not run in host mode by default, but this Pi-Hole stack does in order to avoid port conflicts.
https://github.com/klutchell/balena-pihole/blob/2c028ecc409e42431d4f57c75bf5066bbed7fac7/docker-compose.yml#L23
^^ So as you can see from the line above, ServerIP should be provided for this stack.
https://github.com/klutchell/balena-pihole/blob/2c028ecc409e42431d4f57c75bf5066bbed7fac7/docker-compose.yml#L15-L17
^^ These lines specify the container DNS, and aren't directly related to Pi-Hole, but according to their docs we want values similar to this to make sure Pi-Hole can start correctly (needs to resolve before it can resolve!).
In fact, since these are only used at container startup and not to perform actual runtime queries, you could replace the <IP_ADDRESS> with <IP_ADDRESS> to use OpenDNS there as well. As long as <IP_ADDRESS> is first.
https://github.com/klutchell/balena-pihole/blob/2c028ecc409e42431d4f57c75bf5066bbed7fac7/docker-compose.yml#L21-L22
^^ These DNS lines are overwritten if you provide alternate values in the dashboard. So by providing <IP_ADDRESS> and 1<IP_ADDRESS> you are effectively using OpenDNS for your upstream (not related to dns: above).

Now, I'm guessing that the environment variables in the GUI override these settings (where does that magic happen?).

The balena supervisor will apply dashboard environment variables to the services and override existing ones.
You can go ahead and start using PIHOLE_DNS_=<IP_ADDRESS>;<IP_ADDRESS>  and remove the two DNS lines from your compose file and dashboard now if you want. I already have for my setup.

Thanks for the thorough explanation, @klutchell !

You can go ahead and start using PIHOLE_DNS_=<IP_ADDRESS>;<IP_ADDRESS> and remove the two DNS lines from your compose file and dashboard now if you want. I already have for my setup.

Done.
So it looks like after all of this we've:
a) taught Roddie a lot about how Balena works
b) figured out that we need to update some environment variables and docs in the project
Not a waste for me at all, but it seems like I'm still at square one and you ended up working this weekend anyway.

Nah dude, I still got lots of gaming and dog walks in this weekend. We're all good.
https://github.com/klutchell/balena-pihole/blob/2c028ecc409e42431d4f57c75bf5066bbed7fac7/docker-compose.yml#L15-L17
What if you changed the second entry here to be something else, like google or opendns? Any difference?

Once I get a stable version, I do change some stuff in pihole/Dockerfile to pull in a hosts file and change the font size

Can you clarify these steps a bit? Anything that would impact the host OS that you would know of? I wonder if this would happen with a fresh image, or another device, to isolate the issue to the environment/network or the device.
Pro Tip: if you want to learn a ton more about how balena works, these masterclasses are killer and actually pretty fun.
https://www.balena.io/docs/learn/more/masterclasses/overview/

Nah dude, I still got lots of gaming and dog walks in this weekend. We're all good.

Ha!  As long as the wife isn't blaming me for gaming, we're ok.

What if you changed the second entry here to be something else, like google or opendns? Any difference?

No difference.

Can you clarify these steps a bit? Anything that would impact the host OS that you would know of? I wonder if this would happen with a fresh image, or another device, to isolate the issue to the environment/network or the device.

Harmless, I believe:
RUN echo "addn-hosts=/etc/pihole/lan.list" | sudo tee /etc/dnsmasq.d/02-lan.conf
RUN curl -sSL https://x.x.x.x/roddie/hosts -o /etc/pihole/lan.list

ENV FONTFACE Terminus
ENV FONTSIZE 10x20

Before I even opened the initial display issue I started a new application with a new device to make sure it wasn't something left-over from the last year or three.  Happy to try again and only start with the dtoverlay branch.  It'll take me a half hour or so.

Pro Tip: if you want to learn a ton more about how balena works, these masterclasses are killer and actually pretty fun.
https://www.balena.io/docs/learn/more/masterclasses/overview/

Excellent!  Can't wait to go through them!

@klutchell Fresh application and device, using the dtoverlay branch with no modifications.  Same problem.

Okay, thanks for testing. You're correct that your production modifications are harmless as well. Though if you wanted you could also set FONTFACE=Terminus and FONTSIZE=10x20 in the balena dashboard and skip changing the Dockerfile for those. The supervisor will use the dashboard values above all else.
For your lan hosts those can be added manually via the Pi-Hole dashboard under Local DNS -> DNS Records but your way is faster since you already have the list hosted somewhere.
Let me think about your container startup issue some more, nothing obvious is coming to me right now.
But remember that we aren't at square one cause we fixed your PADD display (and mine) and we noticed that some Pi-Hole env vars are being deprecated so we are tracking that. Also a learning experience isn't a waste in this line of work.

Though if you wanted you could also set FONTFACE=Terminus and FONTSIZE=10x20 in the balena dashboard and skip changing the Dockerfile for those. The supervisor will use the dashboard values above all else.

Good to know!  Is that documented somewhere?  I only noticed those settings in pihole/Dockerfile a couple of months ago.  It's nice to finally use all of my screen real estate :-)
For the LAN hosts there was a whole discussion about it in the comments section of the original balena-pihole blogpost from a few years ago and a few different options were thrown around.  It's my entire network, which is about 90 hosts and this is just a hosts file that I have always maintained on my "main" server so that I can quickly find something if needed.  It keeps things semi-automated, at least.
Related: I learned what Docker was due to that blog post and the related comment thread, and ended up writing a whole blog series about it.

Let me think about your container startup issue some more, nothing obvious is coming to me right now.

I appreciate it - Let me know if you want me to test anything.  Two things are nagging at me:

I don't like host mode - I'm sure it's not causing this problem, but I really don't like host mode.

Something in the back of my mind keeps telling me that I see this connectivity issue whenever we have display-related changes.  I'll play with some of them to see if I can get it to "unbreak."

But remember that we aren't at square one cause we fixed your PADD display (and mine) and we noticed that some Pi-Hole env vars are being deprecated so we are tracking that. Also a learning experience isn't a waste in this line of work.

True true - And I got my 3rd or 4th balena project PR submitted.

If you'd like to try it without host mode you can replace the following line...
https://github.com/klutchell/balena-pihole/blob/c2dd4c64fc0f48727ab5fbf96808e4896e06a300/docker-compose.yml#L22
... with the below lines using your device's LAN IP.
    ports:
      - "LAN_IP:53:53/tcp"
      - "LAN_IP:53:53/upd"

This is to prevent the balena engine from binding to 53 on ALL interfaces, because 53 is already in use on the balena VPN interface.
Then you should also remove the following lines from the pihole Dockerfile:
https://github.com/klutchell/balena-pihole/blob/c2dd4c64fc0f48727ab5fbf96808e4896e06a300/pihole/Dockerfile#L17-L20
After that you'll no longer be running in host mode and we can see if the behaviour changes.
You can also try resolving this error ...
::: Testing pihole-FTL DNS: sudo: unable to resolve host 7ed1bcc: Temporary failure in name resolution

... by adding <IP_ADDRESS> 7ed1bcc to your hosts file or in the Pi-hole dashboard. Normally that error doesn't seem to have any impact but that's how I cleared it on my devices.

After that you'll no longer be running in host mode and we can see if the behaviour changes.

So, I actually ended up trying this last night (I added 80/443 as well for the GUI), and I couldn't get it past this state:

Just redid it using only the 53s and got the same results.  I actually had to do a "Purge data" to get it to take another push and reload again.  I don't want to spend too much time on the host mode thing.  If anything it should make the network connectivity from the container better, not worse.  Just a sticking-point with me since networking is actually my happy place.

::: Testing pihole-FTL DNS: sudo: unable to resolve host 7ed1bcc: Temporary failure in name resolution

Got rid of that last night, too - No change.  That one has always irritated me.

When you tried it did you specify the LAN IP in your ports? This part is very important to not conflict with balena services running on port 53 and may be why you ended up in a bad state?
    ports:
      - "80:80/tcp"
      - "LAN_IP:53:53/tcp"
      - "LAN_IP:53:53/upd"

I tried both ways with every combination - Maybe there's another port that balena is expecting? I may come back to the host mode thing later once we figure out the problem, but in my mind host mode should make things a bit more open here and I don't want to get too distracted with it.

I'm currently in a non-working state and here's what I'm seeing:

local DNS queries from the pihole container work
DNS queries from outside the pihole container to not work (as in the OP)
the GUI works from the outside (?!)
from the host I can do a DNS query to localhost but not to the LAN_IP

root@57e7e72:~# nslookup google.com
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS> 57e7e72

Name:      google.com
Address 1: <IP_ADDRESS> lax17s02-in-f14.1e100.net
Address 2: 2607:f8b0:4000:814::200e dfw25s27-in-x0e.1e100.net
root@57e7e72:~# nslookup google.com <IP_ADDRESS>
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS> 57e7e72

nslookup: can't resolve 'google.com'
root@57e7e72:~# nslookup google.com <IP_ADDRESS>
Server:    <IP_ADDRESS>
Address 1: <IP_ADDRESS> 57e7e72

Name:      google.com
Address 1: <IP_ADDRESS> dfw25s27-in-f14.1e100.net
Address 2: 2607:f8b0:4000:814::200e dfw25s27-in-x0e.1e100.net

Docker doesn't seem to think 53 is bound anywhere else:
root@57e7e72:~# balena ps -a --format="table {{.ID}}\t{{.Names}}\t{{.Ports}}\t{{.Status}}"
CONTAINER ID        NAMES                     PORTS                                            STATUS
71fa5916cc18        pihole_3310990_1706164                                                     Up 35 minutes (healthy)
6221682a7f82        dbus_3310993_1706164                                                       Exited (0) 34 minutes ago
da3f01c7f402        fbcp_3310992_1706164                                                       Up 35 minutes
d6762440786d        unbound_3310991_1706164   <IP_ADDRESS>:5053->5053/tcp, <IP_ADDRESS>:5053->5053/udp   Up 35 minutes
f108ef13311c        resin_supervisor                                                           Up 34 minutes (healthy)

Simply restarting the pihole container makes everything work.

Very odd, sorry if it feels like we are spinning our wheels here but I really want to find out what changes in the container between restarts. Can you check the content of /etc/dnsmasq.conf before and after the restart?
Also, I'm not sure what state you are in but if we are still using host networking what happens if you just remove the following line from your Dockerfile which will allow binding to all interfaces?
https://github.com/klutchell/balena-pihole/blob/c2dd4c64fc0f48727ab5fbf96808e4896e06a300/pihole/Dockerfile#L17-L20

Oh no, I absolutely appreciate this and since it does seem to be network-adjacent, I am enjoying it.  Plus I only use two balena apps, so the more I get to tinker with them the better.  I think we're getting closer.

Check my logic here:
ServerIP:53 isn't open to the container.  Docker doesn't seem to see a problem, but because we're using host networking, we don't see the explicit port as open.
ServerIP:80 works fine, so it's not an IP-binding issue.
<IP_ADDRESS>:53 works fine, so it's not a DNS process issue.

Reproducing this is just a matter of doing a shutdown and power-on, so it's easy for me to test whatever can think of.
/etc/dnsmasq.conf content is the same regardless of state:
conf-dir=/etc/dnsmasq.d
bind-interfaces

user=root

Commenting out that line puts the pihole container into a very unhappy loop:
22.02.21 13:25:10 (-0600)  pihole  dnsmasq: failed to create listening socket for port 53: Address already in use
22.02.21 13:25:10 (-0600)  pihole  [cont-init.d] 20-start.sh: exited 1.
22.02.21 13:25:10 (-0600)  pihole  [cont-finish.d] executing container finish scripts...
22.02.21 13:25:10 (-0600)  pihole  [cont-finish.d] done.
22.02.21 13:25:10 (-0600)  pihole  [s6-finish] waiting for services.
22.02.21 13:25:10 (-0600)  pihole  [s6-finish] sending all processes the TERM signal.
22.02.21 13:25:13 (-0600)  pihole  [s6-finish] sending all processes the KILL signal and exiting.
22.02.21 13:25:15 (-0600) Service exited 'pihole sha256:761dcf61b473bc430c0f2c473ea4ed0303a8ee8da9f56b33a65c3b0b527f5b4d'

Okay, that's the behaviour I was expecting and why that line exists. Can you confirm that the interface provided for INTERFACE and DNSMASQ_LISTENING matches the interface name on the host for your wired connection?
It seems almost like dnsmasq is binding to the wrong interface on first startup (like localhost only), but finding the correct one on a container restart. That would be very weird though, and somehow related to a race condition on startup or a delayed network event? I don't suppose you have a secondary network where you could connect one of these devices? Or even try wifi to see if the behaviour changes? I know I'm grasping at straws now.

Confirmed they're both eth0
You read my mind on dnsmasq (I think we got there at the same time) - I'm collecting some lsof output at the moment.
One of these IPv4 TCP:53 listeners is missing when things are broken (I don't know why there are two):
pihole-FT 7254     root    5u  IPv4  40101      0t0  TCP 57e7e72:domain (LISTEN)
pihole-FT 7254     root    7u  IPv4  40103      0t0  TCP 57e7e72:domain (LISTEN)

More importantly, this UDP listener is missing when things are broken:
pihole-FT 7254   root    4u  IPv4  40100      0t0  UDP <IP_ADDRESS>:53

In both cases, dnsmasq is listening on the containerip:53but that doesn't change and shouldn't matter.
If dnsmasq is getting in the way at some point, it's gone by the time I have access to the console.
I also don't like that whatever is failing is doing so silently.  Are there any other logs that I can collect that might help?

Good findings, I think we are narrowing it down.
I've never tried this before now, but there seems to be a lot of info in the debug logs tool in the Pi-hole dashboard.
https://discourse.pi-hole.net/t/how-do-i-debug-my-pi-hole-installation/3104
Might be worth grabbing these before and after a restart and comparing them. The farther we get into FTL and dnsmasq territory the less help I'll be, but I'm happy to learn!

Yeah, I think we're close.  I've been trying various things while on calls, but I don't think I've learned anything.
That page looks to be more about secure transport of the logs /var/log than anything else - Unless I'm missing something.  I'll have to dig some more and see exactly where I need to submit a debug token.
FWIW I have looked at /var/log/pihole.log and /var/log/pihole-FTL.log - nothing interesting.
Also nothing in dmesg of the pihole container - nothing.
Unsurprisingly, host networking is still nagging at me - I wonder if we can eliminate that, if we'd be able to see the binding in Docker (or an error)?
The fact that we can see a difference between lsof outputs is encouraging, I just can't help but think that somewhere somehow there's a Docker log that is waiting for us to look at it.

No need to submit with a token, just click the button to generate them and a surprising amount of information is collected. Not just logs but simple diagnostic tests. Then you can copy them somewhere and compare.

The balena engine logs can be viewed with journalctl -u balena -a.

The balena engine logs can be viewed with journalctl -u balena -a.

I bet that's covered in the Masterclass which is tab number 28 in my browser :-)
Nothing interesting there - Everyone thinks everyone else is happy.

No need to submit with a token, just click the button to generate them and a surprising amount of information is collected. Not just logs but simple diagnostic tests. Then you can copy them somewhere and compare.

Duh - That makes a whole lot more sense than what I was thinking.  Ok, I'll grab them and go through them tonight/tomorrow and let you know if anything jumps out.
Thanks again for all of your help here!
BTW - This is from within the container:
Not working:
root@57e7e72:/usr/src/app#  lsof -iUDP -P -n | egrep -v '(127|::1)'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 526 root    6u  IPv6  26222      0t0  UDP [fe80::84ae:1c5f:e361:59d2]:53

Working:
root@57e7e72:/usr/src/app#  lsof -iUDP -P -n | egrep -v '(127|::1)'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 544 root    4u  IPv4  89624      0t0  UDP <IP_ADDRESS>:53
pihole-FT 544 root    8u  IPv6  89628      0t0  UDP [fe80::84ae:1c5f:e361:59d2]:53

So, a lot of good info in these logs - Nothing too in the weeds and everything is pretty self-explanatory.
The only difference is kind of disappointing, though.  An extra port "in use" which we already know about.
Not working:
*** [ DIAGNOSING ]: Ports in use
[80] is in use by lighttpd
[80] is in use by lighttpd
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[4711] is in use by pihole-FTL
[4711] is in use by pihole-FTL

Working:
*** [ DIAGNOSING ]: Ports in use
[80] is in use by lighttpd
[80] is in use by lighttpd
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[4711] is in use by pihole-FTL
[4711] is in use by pihole-FTL

The rest of the debug output is virtually identical.  I'll try again tomorrow to make sure I'm not just really tired and missing something, but it seems like whatever is happening is being ignored by everything.

A few ideas --

Is there a way to delay the pihole container's start and manually start it to see if it gives an error?  If it works, we'll know that it's more of a timing/sequence issue.

I figure out how to disable host mode and troubleshoot directly with the ports via docker.

I strip all of the display-related stuff from my config and eliminate that as a variable.

Is there a way to delay the pihole container's start and manually start it to see if it gives an error? If it works, we'll know that it's more of a timing/sequence issue.

Adding the following line to the bottom of your pihole service definition in the docker-compose file will not delay the container start, but it will delay any services running in the container.
    entrypoint: /bin/bash
    command:
      - -c
      - "sleep 60 && /s6-init"

Delaying the container start is a bit more tricky but we will see if that is even required.

I figure out how to disable host mode and troubleshoot directly with the ports via docker.

The following changes worked for my device:

Set the ports to only bind to the lan IP (160 in my case)

    # network_mode: host
    ports:
      - <IP_ADDRESS>:53:53/tcp
      - <IP_ADDRESS>:53:53/udp
      - <IP_ADDRESS>:80
      - :80/tcp

Set DNSMASQ_LISTENING and INTERFACE to all in the dashboard

I strip all of the display-related stuff from my config and eliminate that as a variable.

Just remove the fbcp and dbus services from docker-compose and delete pihole/services/padd/run. You can also remove the unbound service definition since you aren't using it.

Oh, good - A few things to try today.  Thank you!  I'll re-run the pihole debugs first to make sure I didn't miss anything, and then I'll decide which order to try the rest in.
Did I read the last main commit correctly?  Is it all caught up and safe for me to use now?

Yeah, I'm waiting on the balenablocks/fbcp PR to be merged but I'd like someone else to take a look at it first.

Here are the debugs in case you want to check my work - I'm actually impressed with how these are laid-out and annotated.  There's some really cool logic here that even I understand.
pihole_debug_not_working.txt
pihole_debug_working.txt
I'm going to tackle this in this order:

Pare back services (PADD, unbound, fbcp, dbus) one at a time

Delay the services in the pihole container to see if it works.  If it doesn't, I'll attempt to run them manually and see if anything complains.

Disable host mode and try to collect some Docker output.

So - Good news, I suppose.  It's not related to any of the other services.  I'm only running pihole right now and it's still breaking.
I really hope it's nothing in my environment, but I can't see how it would be.  Connectivity is there, the GUI works even when DNS doesn't.
Next step is to delay the services in the container itself.

More "good" news, only because it gets us closer. The sleep works.
So now we have a timing issue somewhere... All of the other containers are gone, so what's left?  The Supervisor and the pihole container?
The problem I have right now is that I can't get into the container to manually run s6-init before whatever condition goes away.
I'm currently messing with the sleep timer to see if I can find the "breaking" point, but that's probably not going to help much.
On the bright side - host mode is in the clear (for now).

I would still try without host mode if possible. If something in the network stack or environment is changing in the first few seconds of container start, it will be nice to isolate that a little more.
Knowing how long of a delay will workaround the issue is interesting but not super helpful as you said.

Knowing how long of a delay will workaround the issue is interesting but not super helpful as you said.

~11 seconds heh. That took me way too long to narrow down, but that's about where it is.
Going for non-host mode now.

So, remember that nagging thought that I've mentioned once or twice?
Disabling host mode gets rid of the issue.  I'm on my tenth test just to make sure, but yeah... everything is working fine following a shutdown/restart.  I started with a fresh archive and only disabled host mode, so everything else is in-tact.
Now the question is: what can/should we do about it?  Whatever is happening should show up in the Docker/balena logs, right?

Also, just to eliminate my environment a bit more - I took my other RPi3B running Pi-hole in Docker (non-Balena), and switched it to host networking.  No problems.

Yup, that tells me that some other interface is being created or binding at the same time the Pi-hole container is starting on boot. Maybe something related to the supervisor and/or balena vpn but I'll have to ask internally to get a hint of which direction to look.
I wish we could push the non-host network option to everyone but manually entering the ServerIP in the docker-compose.yml is a lot of friction for a new user. We also don't have any way to substitute environment variables in the compose file at runtime either.
Perhaps we should try some tests with non-host mode as we have now, but don't specify the ServerIP in the port binding? In my experience that usually results in one of two things:

balena engine complaining that port 53 is in use on one of the interfaces and refusing to start the pihole service
binding succeeds but the device stops communicating with the balena API

So if this is indeed a race condition, and we can trigger the 2nd outcome, I would expect more useful logs in the balena supervisor.
journalctl -u resin-supervisor

I can try this test as well to try to reproduce the 2nd case.

Maybe something related to the supervisor and/or balena vpn but I'll have to ask internally to get a hint of which direction to look.

Right - That was my next question: does belena do anything special with UDP 53 at startup?
I also don't like the fact that I seem to be the only one having a problem.

I wish we could push the non-host network option to everyone but manually entering the ServerIP in the docker-compose.yml is a lot of friction for a new user. We also don't have any way to substitute environment variables in the compose file at runtime either.

I've been thinking about this one.  Host networking effectively binds * to *, right?  Earlier you had me specify LAN_IP so we wouldn't step on the 53 binding that Balena VPN needed.  Wouldn't they be fighting over it with host networking enable?  (maybe leading to my problem?)
I agree that we can't push it to the user, but there's gotta be an answer here somewhere.

Perhaps we should try some tests with non-host mode as we have now, but don't specify the ServerIP in the port binding?

Yep, that makes sense - I'll work on this tomorrow.
Sleep is long-gone - I'm currently sitting at dtoverlay with host networking disabled.
Thanks again for your help!  I think we're there and just need a good answer :-)

I've been thinking about this one. Host networking effectively binds * to *, right? Earlier you had me specify LAN_IP so we wouldn't step on the 53 binding that Balena VPN needed. Wouldn't they be fighting over it with host networking enable? (maybe leading to my problem?)

I think the issue is in this area, yes. The dnsmasq conf entry we are adding is supposed to ensure dnsmasq only binds to the interfaces we specify (eth0 in this case) and up until now no one has reported any issue with that not working as intended.
I pinged the whole balena team internally so I may have some suggestions by tomorrow. I'm also hoping that if we can get the container to bind to 53 on all interfaces the device logs will give us a clue as to where the conflict is.

One idea that was floated was to run lsof -nP -iTCP:53 -sTCP:LISTEN in a tight loop on startup. Preferably on the host but maybe easier in the pihole container (or a new service) initially.
I'll leave it to you to replace the pihole init command with a little bash script, or you can wait for me to give it a try tomorrow and I can provide my steps.

One idea that was floated was to run lsof -nP -iTCP:53 -sTCP:LISTEN in a tight loop on startup. Preferably on the host but maybe easier in the pihole container (or a new service) initially.

I thought of that a couple of days ago, but I was hoping for a way to run it manually before things settle.  Your idea is much better, but I don't believe lsof inside of a container will see ports on the host.
Let me flip to host networking test that first.

Yeah, no good inside of a container -
Host:
root@57e7e72:~# lsof -nP -iTCP:5053 -sTCP:LISTEN
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
balena-en 1840 root    6u  IPv6  22086      0t0  TCP *:5053 (LISTEN)

Container:
root@57e7e72:/usr/src/app# lsof -nP -iTCP:5053 -sTCP:LISTEN
root@57e7e72:/usr/src/app#

So now we're getting a bit above my pay grade, but I'll try to read through this thread and this thread to see what I can figure out/learn.
Running the script is one challenge, but also... where will the output go?  I guess I could redirect it to /tmp/something.log
I also need to figure out how to fork said script.
I have a headache.

I got as far as "build your own balena image" and stopped.  We might need a better mousetrap.

I think I have a better idea of what's happening now. The dnsmasq on the host is only supposed to bind to the VPN interface and a special loopback, so there shouldn't be an issue there. I did consider that the host is not using the bind-interfaces option and we could try adding that option your device only as a test if we want. I can provide quick instructions.
root@2cac2ee:~# lsof -nP -iTCP:53 -sTCP:LISTEN
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dnsmasq 1366 nobody    5u  IPv4  15269      0t0  TCP <IP_ADDRESS>:53 (LISTEN)
dnsmasq 1366 nobody    7u  IPv4  15271      0t0  TCP <IP_ADDRESS>:53 (LISTEN)

On my devices there is a delay in the logs between using the default balena upstream DNS (<IP_ADDRESS>) and my DHCP assigned DNS (<IP_ADDRESS>). This delay probably explains the race condition, since your container must be starting before an updated DHCP ack is received.
root@2cac2ee:~# journalctl -u dnsmasq
-- Logs begin at Tue 2020-08-04 14:03:36 UTC, end at Wed 2021-02-24 14:00:32 UTC. --
Feb 24 10:14:05 2cac2ee dnsmasq[1361]: dnsmasq: syntax check OK.
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: started, version 2.78 cachesize 150
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: DBus support enabled: connected to system bus
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: reading /etc/resolv.dnsmasq
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: using nameserver <IP_ADDRESS>#53
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: read /etc/hosts - 6 addresses
Feb 24 10:14:05 2cac2ee dnsmasq[1366]: dnsmasq[1366]: using nameserver <IP_ADDRESS>#53
Feb 24 10:14:14 2cac2ee dnsmasq[1366]: dnsmasq[1366]: reading /etc/resolv.dnsmasq
Feb 24 10:14:14 2cac2ee dnsmasq[1366]: dnsmasq[1366]: using nameserver <IP_ADDRESS>#53
Feb 24 10:14:14 2cac2ee dnsmasq[1366]: dnsmasq[1366]: using nameserver <IP_ADDRESS>#53

I'd like to see your dnsmasq logs before anything else, since this may indicate why your devices behave differently. It crossed my mind that you are using a 10.x.x.x subnet but I sure hope that's not related or we have much bigger problems.

If you want to try adding the bind-interfaces option to the host you can do the following. Note that this is just a test and it's not recommended to modify the host OS as all changes are lost on an OS upgrade.
mount -o remount,rw /
cp /etc/dnsmasq.conf /etc/dnsmasq.conf.orig
echo bind-interfaces > /etc/dnsmasq.conf
sync
reboot

My dnsmasq logs:
-- Logs begin at Wed 2021-02-24 15:51:59 UTC, end at Wed 2021-02-24 16:30:01 UTC. --
Feb 24 15:52:05 57e7e72 dnsmasq[1524]: dnsmasq: syntax check OK.
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: started, version 2.78 cachesize 150
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: DBus support enabled: connected to system bus
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: reading /etc/resolv.dnsmasq
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: read /etc/hosts - 6 addresses
Feb 24 15:52:05 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 15:52:42 57e7e72 dnsmasq[1530]: dnsmasq[1530]: reading /etc/resolv.dnsmasq
Feb 24 15:52:42 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 15:52:42 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver 10.10.1010#53
Feb 24 15:52:42 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53

using nameserver 10.10.1010#53

Is this a typo on your DHCP server or just bad parsing of the journalctl logs? If your device is trying to use 10.10.1010#53 as the primary DNS server on boot, and eventually falling back to <IP_ADDRESS>#53 that would explain the delay and the race condition.
Your device took 37s to get an updated DNS server via DHCP and my devices took 4s and 9s.

Is this a typo on your DHCP server or just bad parsing of the journalctl logs? If your device is trying to use 10.10.1010#53 as the primary DNS server on boot, and eventually falling back to <IP_ADDRESS>#53 that would explain the delay and the race condition.

It's a typo when I was masking out my home subnet.  :-) The actual IP is sane.
No change after bind-interfaces - I assume you meant >> instead of > but I tried both ways.
I noticed that delay - I mean, it could be for many reasons, but it shouldn't matter, right?  I could span the switch interface and look at it with Wireshark to see what the actual delay is.
I'm about to edit my resin files to set a static IP/DNS.  I thought I'd done this the other day, but I did it incorrectly.

There's no reason that <IP_ADDRESS> wouldn't work on your network? I know I asked about <IP_ADDRESS> previously. Since at the time your container is starting that would be the primary and secondary DNS.

There's no reason that <IP_ADDRESS> wouldn't work on your network?

No, not at all - It's used on some of my devices to get around pihole sometimes.  There are really no restrictions on my network to speak of.
Static IP works -- about to run a test:
-- Logs begin at Wed 2021-02-24 17:28:48 UTC, end at Wed 2021-02-24 17:32:25 UTC. --
Feb 24 17:28:54 57e7e72 dnsmasq[1527]: dnsmasq: syntax check OK.
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: started, version 2.78 cachesize 150
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua>
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: DBus support enabled: connected to system bus
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: reading /etc/resolv.dnsmasq
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: using nameserver <IP_ADDRESS>#53
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: read /etc/hosts - 6 addresses
Feb 24 17:28:54 57e7e72 dnsmasq[1532]: dnsmasq[1532]: using nameserver <IP_ADDRESS>#53

So... setting a static IP seems to have resolved it. (Which makes me annoyed at myself, because I tried to set a static IP two days ago but I did it incorrectly.)
I set it to OpenDNS' servers instead of my local ones, but I'll try those next.
If this is it, there are a few questions :-)

Well yeah, if this fixes it then I have to wonder why my pihole project depends on a speedy DHCP response. Not the most robust project in that case.

Well yeah, if this fixes it then I have to wonder why my pihole project depends on a speedy DHCP response. Not the most robust project in that case.

It fixed it - I'm using my local DNS servers and everything works fine after a shutdown/power on.
-- Logs begin at Wed 2021-02-24 18:00:01 UTC, end at Wed 2021-02-24 18:16:54 UTC. --
Feb 24 18:00:07 57e7e72 dnsmasq[1468]: dnsmasq: syntax check OK.
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: started, version 2.78 cachesize 150
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua>
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: DBus support enabled: connected to system bus
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: reading /etc/resolv.dnsmasq
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: using nameserver <IP_ADDRESS>#53
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: read /etc/hosts - 6 addresses
Feb 24 18:00:07 57e7e72 dnsmasq[1473]: dnsmasq[1473]: using nameserver <IP_ADDRESS>#53

Now, I get that we're in a special area because we're relying on DHCP for DNS and we're also trying to start a DNS service, but yeah... 37 seconds is not an unreasonable amount of time.  My balenaSound device doesn't have any problems.
I'm not even convinced that the 37 seconds is something caused by my network, though I'll try to figure it out.  I run DHCP snooping on the upstream switch, but that shouldn't add delay.  Let me try turning portfast on the port to see if that helps.
It's only happening after a power down and not with a release/renew or a reboot, so it can only be a couple of things.
Ok - while I try to figure this out, where do you think we should go from here?  Happy to offer any support/help you need.

Ok hold up - This is interesting.
I don't think there is a "delay" at all.
After power-up, the device is reachable on the network after 56-57 seconds regardless of whether it's running DHCP or static.  I've tested this ten times and the results are consistent.
I'm just pinging it from my workstation and starting the stopwatch as soon as I turn power on, but I think this is a valid test.  If the device is reachable, it's gotten an IP from DHCP and presumably the DNS settings as well.
So now... the problem might not be the delay at all, but the actual change in DNS servers?
The last test - There's not even a delay:
root@57e7e72:~# journalctl -u dnsmasq
-- Logs begin at Wed 2021-02-24 18:39:27 UTC, end at Wed 2021-02-24 18:44:21 UTC. --
Feb 24 18:39:33 57e7e72 dnsmasq[1526]: dnsmasq: syntax check OK.
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: started, version 2.78 cachesize 150
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua>
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: DBus support enabled: connected to system bus
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: reading /etc/resolv.dnsmasq
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: read /etc/hosts - 6 addresses
Feb 24 18:39:33 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 18:40:08 57e7e72 dnsmasq[1530]: dnsmasq[1530]: reading /etc/resolv.dnsmasq
Feb 24 18:40:08 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 18:40:08 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53
Feb 24 18:40:08 57e7e72 dnsmasq[1530]: dnsmasq[1530]: using nameserver <IP_ADDRESS>#53

> google.com
;; connection timed out; no servers could be reached

Ok, so ignore some of the above.
It is delay related - I forced the switchport to access mode and enabled portfast (basically disabling Spanning Tree Protocol (STP) and Dynamic Trunking Protocol (DTP), and the device is reachable after about 25 seconds (with either static or DHCP), and everything works.
Great, I guess - It explains the 30 second difference, but it still doesn't help us figure out why this is happening.
No more delay in the dnsmasq logs.
-- Logs begin at Wed 2021-02-24 18:49:03 UTC, end at Wed 2021-02-24 18:54:08 UTC. --
Feb 24 18:49:09 57e7e72 dnsmasq[1525]: dnsmasq: syntax check OK.
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: started, version 2.78 cachesize 150
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: compile time options: IPv6 GNU-getopt DBus no-i18n no-IDN DHCP DHCPv6 no-Lua>
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: DBus support enabled: connected to system bus
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: reading /etc/resolv.dnsmasq
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: using nameserver <IP_ADDRESS>#53
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: read /etc/hosts - 6 addresses
Feb 24 18:49:09 57e7e72 dnsmasq[1531]: dnsmasq[1531]: using nameserver <IP_ADDRESS>#53
Feb 24 18:49:13 57e7e72 dnsmasq[1531]: dnsmasq[1531]: reading /etc/resolv.dnsmasq
Feb 24 18:49:13 57e7e72 dnsmasq[1531]: dnsmasq[1531]: using nameserver <IP_ADDRESS>#53
Feb 24 18:49:13 57e7e72 dnsmasq[1531]: dnsmasq[1531]: using nameserver <IP_ADDRESS>#53
Feb 24 18:49:13 57e7e72 dnsmasq[1531]: dnsmasq[1531]: using nameserver <IP_ADDRESS>#53

I'm currently reading the Pi-hole startup functions and I've already found a couple things that wouldn't work well on balena. Still a ways to go and I already have a few things to fix, will keep you posted.

Anything interesting in /var/log/pihole-FTL.log when the state is broken vs working? We should have looked at this log sooner I think.

Nope, nothing - I looked at them early-on, back when I thought the debug logs were just a way to send those files.  Nothing useful in any of them.

In fact, I just remembered this - /var/log/pihole-FTL.log isn't even written to in a broken state.  It's not even opened.
/var/log/pihole.log doesn't have anything useful, either.
Thought - Obviously pihole isn't able to bind to LAN_IP:53, but we learned earlier when discussing lsof that the container can't actually see what is using the port on the host, so we might not ever be able to see anything useful from the container perspective.  I'm amazed that we don't at least see generic error there, though.

/var/log/pihole-FTL.log isn't even written to in a broken state. It's not even opened.

That seems serious, I'll have a look at the scripts to see what could cause it not to be started or not to write to the log.

lsof that the container can't actually see what is using the port on the host

Only because lsof is only aware of processes that were started in it's own namespace. The container can still see all the same interfaces as the host, but can't list port bindings except the ones that were created by processes in the same kernel namespace.
I have an easy way to add startup scripts to the pihole container (that would run before pihole tries to start) but I haven't decided yet what would be the most useful information to collect from the container perspective. I was thinking about listing interfaces in /proc/net/dev and pinging a public url/ip for 1min so see if we can catch a change but I don't know if there's something else I should be looking for.
Whatever we want to do can be added in a new executable file called /etc/cont-init.d/10-custom.sh and it will run before 20-start.sh
Example off the top of my head, we can change this to whatever:
#!/usr/bin/with-contenv bash

for i in {1..10}
do
    cat /proc/net/dev
    nslookup cloudflare.com
    ping <IP_ADDRESS>
done

That seems serious, I'll have a look at the scripts to see what could cause it not to be started or not to write to the log.

You'd think it would show up in those debug logs, right?
This seems significant from 20-start.sh BTW.
# Kill dnsmasq because s6 won't like it if it's running when s6 services start
kill -9 $(pgrep pihole-FTL) || true # TODO: REVISIT THIS SO AS TO NOT kill -9

Only because lsof is only aware of processes that were started in it's own namespace. The container can still see all the same interfaces as the host, but can't list port bindings except the ones that were created by processes in the same kernel namespace.

Ok, that makes much more sense to me.  Thanks!

Whatever we want to do can be added in a new executable file called /etc/cont-init.d/10-custom.sh and it will run before 20-start.sh

Ha, this is why I'm sticking to networking and keeping this as a hobby - My plan was just hack up /s6-init
Trying the example above in /etc/cont-init.d/10-custom.sh - Adding a -c 10 otherwise the ping will never stop.

Here are the logs with that script in place - I grabbed them from the host using journalctl so I could get them right from boot, so they're a bit cloudy.
balena-logs.txt

Sorry for the delayed response, I needed a break to clear my head.
So what can we learn from these logs? Here's what I see:

script start: Feb 24 21:22:16 57e7e72<PHONE_NUMBER>b8[1534]: [cont-init.d] 10-custom.sh: executing...
+0 interfaces present: Feb 24 21:22:16 57e7e72<PHONE_NUMBER>b8[1534]:   eth0:       0       0    0    0    0     0          0         0     3088      16    0    0    0     0       0          0
+1m16s nslookup response: Feb 24 21:23:32 57e7e72<PHONE_NUMBER>b8[1534]: Non-authoritative answer:
+3s pings response: Feb 24 21:23:35 57e7e72<PHONE_NUMBER>b8[1534]: --- <IP_ADDRESS> ping statistics ---
+37s scripts repeats 9x times
+1m56s script end: Feb 24 21:24:12 57e7e72<PHONE_NUMBER>b8[1534]: [cont-init.d] 10-custom.sh: exited 0.

all interfaces are present from the moment the container starts and don't change
the pihole container starts before the balena VPN connection is established (normal I think)
first dns query takes over a minute
10 tests are performed and no transmit errors reported on eth0
by the time testing is done the pihole-FTL service starts successfully (I assume)

I bet we aren't seeing any traffic in or out of eth0 until about that 1m mark. We could try again with a slightly modified version to see if ping has any success before nslookup. Also we could install time in the container so I don't have to calculate these diffs.
#!/usr/bin/with-contenv bash

for i in {1..10}
do
    time cat /proc/net/dev
    time ping -c 4 <IP_ADDRESS>
    time nslookup google.com
done

Did you try capturing network packets on device startup? It would be interesting to see if any packets leave the device when trying to use ping/nslookup in that first minute.

Sorry for the delayed response, I needed a break to clear my head.

I could tell, man - no worries!

by the time testing is done the pihole-FTL service starts successfully (I assume)

Yep - our script acts like our sleep command.  Everything works fine once it completes.

I bet we aren't seeing any traffic in or out of eth0 until about that 1m mark.

This makes sense - I'm not sure how long it takes for the system to boot and the container to start, but it's about 56 seconds from power-on until connectivity, so whatever the difference is.
Installing the script now - Is there a cleaner way to get the output only from the pihole container for you?

Did you try capturing network packets on device startup? It would be interesting to see if any packets leave the device when trying to use ping/nslookup in that first minute because /proc/net/dev seems to think we are transmitting, but not receiving until later.

I haven't captured anything switch-side - I could, but I already know that the delay is just Spanning Tree doing its thing, and I know that the switch won't put the port into "forwarding" state until it's done. This wouldn't happen with wifi, and I would guess most of our other users are using consumer switches that wouldn't have STP fully-enabled.
The end-device wouldn't have any idea what is happening since it's not doing anything with BPDUs or speaking STP, so it will just send traffic as if everything is normal.  The switch will go through its STP states and pretty much ignore everything until the port goes into "forwarding."

Backing up a bit - Is there where we're at?
The container comes up expecting full connectivity and is trying to start pihole-FTL but it can't bind to LAN_IP:53 because the host isn't finished doing its thing due to the connectivity delay.
I just want to make sure it's still right in my head.

Is there a cleaner way to get the output only from the pihole container for you?

What you provided is perfect for me as long as it's not too much trouble for you.

The container comes up expecting full connectivity and is trying to start pihole-FTL but it can't bind to LAN_IP:53 because the host isn't finished doing its thing due to the connectivity delay.

Still unclear. I think FTL can bind to 53 because the interface is present and there are no errors in the logs regarding conflicts. But something is preventing it from starting cleanly because as you pointed out there are no FTL logs in a broken container. I'm considering adding some more logging around FTL start/stop actions because lack of internet connectivity shouldn't block port binding (but it would explain not downloading updated blocklists).
Maybe FTL doesn't print any message if the port is in use and just dies quietly? I intend to find out.

What you provided is perfect for me as long as it's not too much trouble for you.

Oh not at all, it's a few jumps from Pi to workstation, but it only takes me a minute.  I just see stuff from other processes and I didn't know if you wanted it in a cleaner way.
Logs attached - You can see the "Network is unreachable" during the first run.
balena-logs.txt

Still unclear. I think FTL can bind to 53 because the interface is present and there are no errors in the logs regarding conflicts. But something is preventing it from starting cleanly because as you pointed out there are no FTL logs in a broken container.

Oh, that's a really good point - Leave it to the network guy to jump to a network idea first.

Maybe FTL doesn't print any message if the port is in use and just dies quietly? I intend to find out.

Honestly, considering how much info is in the debug log, this really surprises me.

Could you try the following when you get a chance?

try FROM pihole/pihole:v5.6 if we haven't already
add rm /dev/shm/FTL* before the kill -9 call in both /run/s6/services/pihole-FTL/finish and /etc/cont-init.d/20-start.sh
set SKIPGRAVITYONBOOT=1 in the dashboard

Obviously not all at the same time, just in case one of them works :)

Could you try the following when you get a chance?

Yep - I assume you want the script removed prior to testing, right?

add rm /dev/shm/FTL* before the kill -9 call in both /run/s6/services/pihole-FTL/finish and /etc/cont-init.d/20-start.sh

This fixed it.  I did say the line looked significant a few posts ago, right?  Haha....
On my fourth test now.

Well, it worked for five tests and now it's failed twice in a row and I'm a bit confused.

I did say the line looked significant a few posts ago

You called it dude, you didn't even need me.

Well, it worked for five tests and now it's failed twice in a row and I'm a bit confused.

Did you try the other 2 suggestions as well or just this one? I'd be interested in the results of all 3, separate from each other.
https://github.com/pi-hole/docker-pi-hole/issues/794
According to this it should be fixed in the re-release of v5.7 from 9 days ago, but we only opened this issue 5 days ago. So either it's not quite fixed for our environment, or we have a caching issue. Testing v5.6 would also answer this.
What's your deploy pipeline look like? Are you using balena push from the command line? If so you could try appending --nocache to that and see if it works without the additional FTL cleanup lines.
You could also change the top of the Dockerfile to hardcode the sha256 of the latest armv7 v5.7 image.
FROM pihole/pihole:v5.7@sha256:6f8e902adb26cf10220763d1ab027b0b2c64e2df3a397e4f015ead7a1c40d96c

@klutchell Ok - I did the changes, shutdown, restarted and things worked.  Times five.
On the sixth try and beyond, it stopped working.  I logged into the pihole container and the changes to /run/s6/services/pihole-FTL/finish are gone, but the ones to /etc/cont-init.d/20-start.sh are still there.
Which has me all sorts of confused.  Should I be injecting the modified files via Dockerfile instead?

According to this it should be fixed in the re-release of v5.7 from 9 days ago, but we only opened this issue 5 days ago. So either it's not quite fixed for our environment, or we have a caching issue. Testing v5.6 would also answer this.

5.6 did not work and I didn't make it to the SKIPGRAVITYONBOOT=1 test.  Let me backup for a minute, try to push 5.7 again with --nocache, and see where I end up.  I'm afraid I messed-up our results with the "disappearing" FTL cleanup lines, so I don't want to get too far ahead of my skis.
I'll test with 5.7 and then try SKIPGRAVITYONBOOT=1 - I do need your help to make sure I get those FTL lines in-place and sticky.

You called it dude, you didn't even need me.

Yeah, right!!!  Look at the mess I made above!  haha

Which has me all sorts of confused. Should I be injecting the modified files via Dockerfile instead?

Yes, I should have clarified or provided instructions since the container may be recreated and only our data volumes are guaranteed to persist. I also should have provided the correct path /etc/services.d/pihole-FTL/finish but let's not point fingers.
Would you like some instructions to modify those scripts via the Dockerfile?

I swear I didn't notice that we inject a PADD file into that structure and then think "Why didn't he tell me to put it there?"  I promise that didn't happen at least twice.
5.7 pushed with the static sha256 seems to be working.  Let me try a few more times before embarrassing myself again.
I'm having about a 50% success rate now.  This is not making me feel better haha

Ok, I'm going to call 5.7 (and 5.6 earlier) no good - I may have been power-cycling the RPi too quickly recently because I'm multitasking, which may have given me false positives (a quick power-cycle may bypass the delay).
Trying SKIPGRAVITYONBOOT now.

Ok, so - I am not 100% sure why, but I've had a bunch of false positives today.  This issue has been with me for a while, and we've been working on it closely for almost a week and I've managed to make it fail consistently.
Anyway, here's where I am - I did a purge data and redid the last few tests (5.6, SKIPGRAVITYONBOOT, and FTL stuff).  None of those fix the problem.
Looking closer at https://github.com/pi-hole/docker-pi-hole/issues/794, I don't think we're having that issue as pihole-FTL is definitely running on my setup even on a failure.
Output below shows during a failure -

pihole-FTL running
pihole-FTL.log not updated at all since it started running
current time.

root@57e7e72:/var/log# ps -ef | grep -i FTL
root       496     1  0 17:13 pts/0    00:00:00 s6-supervise pihole-FTL
root       522   502  0 17:13 ?        00:00:04 pihole-FTL no-daemon
root     11483 11027  0 17:21 pts/1    00:00:00 grep -i FTL
root@57e7e72:/var/log# tail pihole-FTL.log
[2021-02-25 17:10:14.999 522M]  -> Known forward destinations: 2
[2021-02-25 17:10:14.999 522M] Successfully accessed setupVars.conf
[2021-02-25 17:10:15.015 522M] PID of FTL process: 522
[2021-02-25 17:10:15.016 522M] INFO: FTL is running as root
[2021-02-25 17:10:15.016 522/T530] Listening on port 4711 for incoming IPv4 telnet connections
[2021-02-25 17:10:15.016 522/T532] Listening on Unix socket
[2021-02-25 17:10:15.018 522/T531] Listening on port 4711 for incoming IPv6 telnet connections
[2021-02-25 17:10:15.021 522M] Reloading DNS cache
[2021-02-25 17:10:15.021 522M] Blocking status is enabled
[2021-02-25 17:10:15.591 522/T533] Compiled 0 whitelist and 0 blacklist regex filters for 11 clients in 1.0 msec
root@57e7e72:/var/log# date
Thu Feb 25 17:21:36 CST 2021

After restarting the container (DNS working):
root@57e7e72:/usr/src/app# ps -ef | grep FTL
root       499     1  0 17:26 pts/0    00:00:00 s6-supervise pihole-FTL
root       529   501  1 17:26 ?        00:00:00 pihole-FTL no-daemon
root       833   826  0 17:26 pts/1    00:00:00 grep FTL
root@57e7e72:/usr/src/app# tail /var/log/pihole-FTL.log
[2021-02-25 17:26:38.472 529M]  -> Known forward destinations: 2
[2021-02-25 17:26:38.472 529M] Successfully accessed setupVars.conf
[2021-02-25 17:26:38.479 529M] PID of FTL process: 529
[2021-02-25 17:26:38.481 529M] INFO: FTL is running as root
[2021-02-25 17:26:38.481 529/T542] Listening on port 4711 for incoming IPv4 telnet connections
[2021-02-25 17:26:38.482 529/T544] Listening on Unix socket
[2021-02-25 17:26:38.490 529/T543] Listening on port 4711 for incoming IPv6 telnet connections
[2021-02-25 17:26:38.491 529M] Reloading DNS cache
[2021-02-25 17:26:38.491 529M] Blocking status is enabled
[2021-02-25 17:26:38.540 529/T545] Compiled 0 whitelist and 0 blacklist regex filters for 11 clients in 1.0 msec
root@57e7e72:/usr/src/app# date
Thu Feb 25 17:27:01 CST 2021

Good points, at least we know the FTL service is running but not generating any output. Which bring me to my next test...
There is a dnsmasq timeout setting on the host OS resolv.conf that is copied and updated into the container by the engine. We have very little control over this file except by adding the dns: entry in our docker-compose.
So while on the host we have:
root@2cac2ee:~# cat /etc/resolve.conf
# we use dnsmasq at <IP_ADDRESS> so that user containers can run their own dns cache and forwarder and not conflict with dnsmasq on the host
nameserver <IP_ADDRESS>
options timeout:15

In the container we have (mounted by the engine, it is not writable):
root@2cac2ee:/usr/src/app# cat /etc/resolv.conf
nameserver <IP_ADDRESS>
options timeout:15

So, knowing that your network has a longer than average connection time, what if we bumped that timeout on the host and thus it would be copied into the container? My thinking is that it FTL is running but not logging, maybe it timed out after 15s of waiting for a response?
mount -o remount,rw /
cp /etc/resolv.conf /etc/resolv.conf.orig
sed 's/options timeout:15/options timeout:45/' -i /etc/resolv.conf
sync
reboot

We should have all other hacks removed at this point. I'm assuming you are running on the tip of dtoverlay.
Also, for bonus points on the exam: What if we didn't specify custom dns: entries in docker compose? It didn't break my device when I tried it and I wonder if it changes the behaviour of yours at all.
services:
  # https://hub.docker.com/r/pihole/pihole
  pihole:
    build: ./pihole
    privileged: true
    volumes:
      - "pihole_config:/etc/pihole"
      - "dnsmasq_config:/etc/dnsmasq.d"
    # dns:
    #   - "<IP_ADDRESS>"
    #   - "<IP_ADDRESS>"
    environment:
      DNSMASQ_LISTENING: eth0
      INTERFACE: eth0
      PIHOLE_DNS_: <IP_ADDRESS>#5053;<IP_ADDRESS>#5053
    network_mode: host
    depends_on:
      - unbound

Ok, so I'm trying to kind of take a step back and revaluate where we are.  Sorry for the confusion yesterday, but I still think we're learning some stuff.
At a high-level I have three workarounds:

Reduce the extra ~30 second connectivity delay by forcing access mode and disabling spanning-tree (enabling portfast) on my switch port.

This is easy and would work fine for me longterm.
!
interface GigabitEthernet1/0/3
 description Balena pi-hole (pihole1)
 switchport access vlan 101
 switchport mode access <--- new
 spanning-tree portfast <--- new
end

Add a delay before the script startup in the pihole container.  Effectively accomplishes the same thing as #1, but from the other direction.

Easy to do, but doesn't scale and isn't a great permanent workaround.
 entrypoint: /bin/bash
    command:
      - -c
      - "sleep 30 && /s6-init"

Disable host networking for the container and explicitly define the ports to be forwarded to the container.

A simple fix by itself, and arguably a Docker best practice, however does not scale to the balena project as it is a heavy lift for the user.  No way to automate this fix using the current system (ie. environment variables, etc.)
#    network_mode: host
    ports:
      - LAN_IP:53:53/tcp
      - LAN_IP:53:53/udp
      - LAN_IP:80:80/tcp

DNSMASQ_LISTENING=all
INTERFACE=all

#1 and #2 are pure workarounds to avoid the issue.  #3 is also a workaround, but I think it holds the key to what the root cause is.  Have we figured out why host mode triggers the problem?  We've kind of been everywhere, so if we have, I have forgotten.  Does host mode give too much power to the container?
(I just had an idea.)
(You just updated the post so I'll leave it here for now)

We should have all other hacks removed at this point. I'm assuming you are running on the tip of dtoverlay.

Yeah, this is one mistake I haven't made yet haha

So, knowing that your network has a longer than average connection time, what if we bumped that timeout on the host and thus it would be copied into the container?

No luck with this one.

Also, for bonus points on the exam: What if we didn't specify custom dns: entries in docker compose? It didn't break my device when I tried it and I wonder if it changes the behaviour of yours at all.

No good.

The idea I had earlier was inspecting the Pi-hole debug output between host networking mode and non-host networking mode (with working configurations) to see what the behavior difference is and working backwards.  I know it's a long-shot because the debug output doesn't even differ between working/non-working.

I'm still extremely annoyed that pihole-FTL doesn't write to its log at all during a failure scenario, so I tried to be clever and actually wrote my own sed line (my first) to do this:
RUN sed 's/>\/dev\/null\ 2>\&1//' -i /etc/services.d/pihole-FTL/run
I figured it would give us some more output to the console/log - I've attached it, but nothing is really jumping out at me.
balena-FTL.txt
--
pihole-FTL isn't failing, or at least doesn't know that it's failing, because the failure is being masked by Docker?  But at the same time, Docker doesn't appear to think anything is wrong, either.

Some interesting differences between a host-networking environment and a non-host networking environment:

With host networking, the container's default route is the same as the host's default route (LAN_IP_GATEWAY).  With explicit ports, the container default gateway is the Docker network.

root@57e7e72:/usr/src/app# ip route | grep default
default via <IP_ADDRESS> dev eth0 proto dhcp metric 100

In non-host networking, pihole-FTL binds itself to a lot less ports:

non-host networking
*** [ DIAGNOSING ]: Ports in use
[80] is in use by lighttpd
[80] is in use by lighttpd
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[4711] is in use by pihole-FTL

host networking
*** [ DIAGNOSING ]: Ports in use
[80] is in use by lighttpd
[80] is in use by lighttpd
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[53] is in use by pihole-FTL
[4711] is in use by pihole-FTL
[4711] is in use by pihole-FTL

The number of interfaces seen by the container is drastically different between the two modes:

non-host networking
   DHCP packets received on interface eth0: 0
   DHCP packets received on interface lo: 0

host networking
   DHCP packets received on interface resin-dns: 0
   DHCP packets received on interface vethfd9335e: 0
   DHCP packets received on interface balena0: 0
   DHCP packets received on interface supervisor0: 0
   DHCP packets received on interface br-efe19f716f8f: 0
   DHCP packets received on interface lo: 0
   DHCP packets received on interface wlan0: 0
   DHCP packets received on interface veth16dceb8: 0
   DHCP packets received on interface eth0: 3

I'm not sure if any of that is helpful, but there are some real differences to consider.
Notice that the container doesn't see DHCP packets when in non-host networking mode, too.

Some lsof output from the container just to add some color to the above.
Container lsof output non-host networking:
root@72e8e9120a4f:/usr/src/app# lsof -iTCP -sTCP:LISTEN
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
lighttpd  525 www-data    4u  IPv4  29214      0t0  TCP *:80 (LISTEN)
lighttpd  525 www-data    5u  IPv6  29215      0t0  TCP *:80 (LISTEN)
pihole-FT 527     root    5u  IPv4  30012      0t0  TCP 72e8e9120a4f:53 (LISTEN)
pihole-FT 527     root    7u  IPv4  30014      0t0  TCP localhost:53 (LISTEN)
pihole-FT 527     root   11u  IPv4  29208      0t0  TCP localhost:4711 (LISTEN)

root@72e8e9120a4f:/usr/src/app# lsof -iUDP -P -n | egrep -v '(127|::1)'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 527 root    4u  IPv4  30011      0t0  UDP <IP_ADDRESS>:53

Container lsof output host networking (working):
root@57e7e72:/usr/src/app# lsof -iTCP -sTCP:LISTEN
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
lighttpd  521 www-data    4u  IPv4  73511      0t0  TCP *:80 (LISTEN)
lighttpd  521 www-data    5u  IPv6  73512      0t0  TCP *:80 (LISTEN)
pihole-FT 530     root    5u  IPv4  73492      0t0  TCP 57e7e72:53 (LISTEN)
pihole-FT 530     root    7u  IPv4  73494      0t0  TCP 57e7e72:53 (LISTEN)
pihole-FT 530     root    9u  IPv6  73496      0t0  TCP [fe80::84ae:1c5f:e361:59d2]:53 (LISTEN)
pihole-FT 530     root   11u  IPv6  73498      0t0  TCP localhost:53 (LISTEN)
pihole-FT 530     root   14u  IPv6  74295      0t0  TCP localhost:4711 (LISTEN)
pihole-FT 530     root   16u  IPv4  73502      0t0  TCP 57e7e72:4711 (LISTEN)

root@57e7e72:/usr/src/app# lsof -iUDP -P -n | egrep -v '(127|::1)'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 530 root    4u  IPv4  73491      0t0  UDP <IP_ADDRESS>:53
pihole-FT 530 root    8u  IPv6  73495      0t0  UDP [fe80::84ae:1c5f:e361:59d2]:53

Container lsof output non-host networking (non-working):
root@57e7e72:/usr/src/app# lsof -iTCP -sTCP:LISTEN
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
lighttpd  511 www-data    4u  IPv4  26432      0t0  TCP *:80 (LISTEN)
lighttpd  511 www-data    5u  IPv6  26433      0t0  TCP *:80 (LISTEN)
pihole-FT 527     root    5u  IPv4  26418      0t0  TCP 57e7e72:53 (LISTEN)
pihole-FT 527     root    7u  IPv6  26420      0t0  TCP [fe80::84ae:1c5f:e361:59d2]:53 (LISTEN)
pihole-FT 527     root    9u  IPv6  26422      0t0  TCP localhost:53 (LISTEN)
pihole-FT 527     root   14u  IPv6  26845      0t0  TCP localhost:4711 (LISTEN)
pihole-FT 527     root   17u  IPv4  26847      0t0  TCP 57e7e72:4711 (LISTEN)

root@57e7e72:/usr/src/app# lsof -iUDP -P -n | egrep -v '(127|::1)'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
pihole-FT 527 root    6u  IPv6  26419      0t0  UDP [fe80::84ae:1c5f:e361:59d2]:53

Another idea.
Are we overthinking this?  Is it simply a matter of pihole-FTL not being able to bind to an address that doesn't exist yet?

Looking at the official docs, it looks like we're behind on something else, too:

If I set DNSMASQ_LISTENING to gibberish, it behaves the same as setting it to eth0
DNSMASQ_LISTENING=adsgfdsfasd
26.02.21 16:20:44 (-0600)  pihole  DNSMasq binding to default interface: eth0
I'll open a PR this weekend to update it.  I guess for now we should just set it to single and completely remove the option from the docs since it's of no use unless people disable host networking.
With host networking enabled, local and all put the container into a loop.
26.02.21 16:27:42 (-0600)  pihole  dnsmasq: failed to create listening socket for <IP_ADDRESS>: Address already in use
(I assume <IP_ADDRESS> is the VPN address.)

One other thing I tried was to replicate this on my other Pi-hole device which is a RPi3b running Raspbian and the "official" Docker image, but I couldn't get it to break.  I tried host mode on it earlier, but this time I actually moved it to the Cisco switch so that we'd have the delay.
I'm guessing that Balena introduces some timing differences and overhead with the Supervisor, VPN, etc.  This device gets connectivity ~36 seconds from a power up vs. ~56 seconds with the Balena device.

Amazing data collection above, dude. Thanks for doing all that!

I guess for now we should just set it to single and completely remove the option from the docs since it's of no use unless people disable host networking.

Yup, that's my bad. I don't think they ever changed that behaviour, I've just been using that field incorrectly from the start.

I assume <IP_ADDRESS> is the VPN address.

Yup.

Are we overthinking this? Is it simply a matter of pihole-FTL not being able to bind to an address that doesn't exist yet?

That was my initial thought, but the eth0 interface exists on container start as we tested with cat /proc/net/dev.
I guess we could check if it has a valid IP but I believe it does, even if it hasn't heard back from the switch yet? Though that would be a useful difference because in bridge networking mode the engine would give it a valid bridge IP immediately and never change it.

Container lsof output non-host networking (non-working)

Do you mean host networking (non-working) for this last one?
Feel free to use host vs bridge network modes going forward to avoid confusion. Bridge is the default network mode and requires we expose the ports. If you edit the post above I'll review it again but it sure looks like you are in host mode and UDP 53 is only binding to IPv6.

Amazing data collection above, dude. Thanks for doing all that!

Well, I felt bad for yesterday's silliness, plus it is my problem, and obviously I'm not the type to let things go or take a break from it.  It's in my brain until we get it fixed.

Yup, that's my bad. I don't think they ever changed that behaviour, I've just been using that field incorrectly from the start.

Haha - I shouldn't be laughing, but that's kind of funny.  I'll get it cleaned up.

That was my initial thought, but the eth0 interface exists on container start as we tested with cat /proc/net/dev.

Ok, but note the difference in bindings above - pihole-FTL is the only process binding to <IP_ADDRESS>:53.  Everything else is binding to * or a hostname.
We run pihole-FTL with $FTL_CMD argument but so far I haven't been able to find where that's set.  Maybe we could try messing with that and bending pihole-FTL to our will (if we can find it - I've looked everywhere).

I guess we could check if it has a valid IP but I believe it does, even if it hasn't heard back from the switch yet?

At best, it would have an APIPA address, but I don't know for sure if Linux does that - I could modify 10-custom.sh to grab an ip addr show dev eth0 if you're really curious.

Though that would be a useful difference because in bridge networking mode the engine would give it a valid bridge IP immediately and never change it.

Yep!

Do you mean host networking (non-working) for this last one?

I did - will fix in a sec.

Feel free to use host vs bridge network modes going forward to avoid confusion. Bridge is the default network mode and requires we expose the ports.

Yeah... I mean, it's not like I wrote an entire blog post about this topic or anything hahahaha - Getting old sucks, man.

Yeah, no v4 address to bind to, which might explain why there's no error - nothing is actually failing.
Feb 27 03:32:30 57e7e72 c2b944d7c437[1532]: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group defau
lt qlen 1000
Feb 27 03:32:30 57e7e72 c2b944d7c437[1532]:     link/ether b8:27:eb:1c:7e:1e brd ff:ff:ff:ff:ff:ff
Feb 27 03:32:30 57e7e72 c2b944d7c437[1532]:     inet6 fe80::84ae:1c5f:e361:59d2/64 scope link noprefixroute
Feb 27 03:32:30 57e7e72 c2b944d7c437[1532]:        valid_lft forever preferred_lft forever

pihole-FTL binds to all active IPs on the container, my IP isn't active by the time it runs, and that's that.

What would happen if you booted your pi up with the ethernet cable disconnected, connecting it after a couple of minutes?
pihole-FTL may fail because the interface is down and then just restart when it comes up, so it might not work.
But if you, say, disconnected your DHCP server for a couple of minutes and tested, I bet you'd get the same result.

Getting old sucks, man.

You're telling me. I have a wicked hangover today from 3 drinks last night. 3 drinks?! I'm only 35.

pihole-FTL binds to all active IPs on the container, my IP isn't active by the time it runs, and that's that.

I was able to reproduce it by disconnecting my unmanaged switch from the LAN, so eth0 was still present but no LAN access or DHCP server. I even used the active wifi connection to verify that eth0 was present but had no IP, and FTL was not binding to anything on that interface. Upon reconnecting the switch to the LAN, eth0 got an address but FTL did not restart or retry bind or anything, cause how would it know to do that?
So, you solved it! We now know what is happening behind the scenes to explain this behaviour and we have several workarounds for cases where it may take more than a few seconds for an IP to be assigned to the selected interface.
I'm happy to put a permanent fix in to 10-custom.sh going forward, in case anyone else has network behavior similar to this. As a network guy yourself, what kind of command would be the most reliable, and fast, way to determine if a valid address has been assigned to INTERFACE?
while ! [ condition ]
do
   echo "waiting for IPv4 address on ${INTERFACE}..."
   sleep 2
done

Once we have this script in place and I've tested it, I can open a PR and you can retest it from there. I'll also roll in the DNSMASQ_LISTENING=single fix as well.
I expected a bit more fanfare at this point, but maybe I'm still in disbelief. Oh well 🍻

You're telling me. I have a wicked hangover today from 3 drinks last night. 3 drinks?! I'm only 35.

I'm 50 this year, I wish I could tell you that it gets better.  Haha - You're hungover and I've been thinking about this issue all night.

I was able to reproduce it by disconnecting my unmanaged switch from the LAN, so eth0 was still present but no LAN access or DHCP server. I even used the active wifi connection to verify that eth0 was present but had no IP, and FTL was not binding to anything on that interface. Upon reconnecting the switch to the LAN, eth0 got an address but FTL did not restart or retry bind or anything, cause how would it know to do that?

I just took this a bit further and tried to reproduce it on my non-Balena Rpi running the pi-hole/docker-pi-hole but couldn't do it.  I changed it to use DHCP, on the switch I put the interface on a different VLAN without a DHCP server and left it there for a couple of minutes after bootup.  After moving it back to the regular, it got an IP address ~40ish seconds later (STP will run on a VLAN change) and it was responding to DNS queries.
Not worth worrying about right now, but I would expect it to behave the same way.  I'll throw a monitor on it later and figure it out.

I'm happy to put a permanent fix in to 10-custom.sh going forward, in case anyone else has network behavior similar to this. As a network guy yourself, what kind of command would be the most reliable, and fast, way to determine if a valid address has been assigned to INTERFACE?

So, there are probably a few ways to workaround this one, but first want to say that this seems (to me) to be a pihole-FTL limitation.  Why is it binding to active IPs instead of an interface?  Is it because of dnsmasq?  lighttpd is in the same container and isn't having a problem.
Per the docs, in host mode we're feeding it INTERFACE so it should be able to bind to that.  In fact, we're even feeding it ServerIP in host mode, so it should fail (and retry) if it can't bind to it.

This is partly why I want to reproduce this on my non-Balena Rpi.  I looked at the pihole-FTL CLI options and there's really nothing helpful there.
Ok... now to workarounds - We have a lot of options (I'm going to cover the ones that aren't realistic for you or me to fix, just because it's been on my mind all night):

A DNS server should always have a static IP address and in theory we should recommend this in the docs.  The problem is that setting a static in Balena is non-trivial.  It would be really cool to be able do this during the "Add new device" workflow similar to how we configure Wifi parameters.  Above our pay grades maybe, but to me this really is a best practice.

Fix pihole-FTL to either bind to INTERFACE or to fail if it can't bind to ServerIP

Off my soapbox now, sorry.  More realistic workarounds:

Test for the presence of an inet4 address on INTERFACE (preferably one that matches ServerIP) by either evaluating ip -o -4 addr show dev eth0 assuming it gives a result (hopefully I'm using the right words here), or using a grep on that command for inet (using regest, or making sure we specify -4 so that v6 doesn't get in the way).

Testing the result of a ping -c 1 <ServerIP> would be the most reliable.

There are probably a few others, but those two were the first ones I could think of.

I expected a bit more fanfare at this point, but maybe I'm still in disbelief. Oh well 🍻

Hahaha - Same!  I think we're both tired of this, and honestly the problem was in front of our faces the whole time.  Nothing really obscure or complex or tricky.  But we did it, and I'm kind of relieved that it wasn't really because of anything in the project or that either of us did.

I just took this a bit further and tried to reproduce it on my non-Balena Rpi running the pi-hole/docker-pi-hole but couldn't do it

Possibly because your docker version is different than the one the current balena engine is based on. Perhaps the hand-off of interfaces in host mode has changed in ways I can't begin to understand.

lighttpd is in the same container and isn't having a problem.

I don't know why lighttpd works. It's binding to all interfaces so maybe *:80 works differently than eth0:53.

Why is it binding to active IPs instead of an interface? Is it because of dnsmasq?

As far as I understand, pihole-FTL is a fork of dnsmasq so that's why we see config files for dnsmasq but pihole-FTL is the actual process. Also I think the binding behaviour is as follows:

provide an interface name
bind to 53 for each address on that interface
no addresses? keep running as if everything is fine

we're even feeding it ServerIP in host mode, so it should fail (and retry) if it can't bind to it

Not a terrible idea, you could run in past the pi-hole/pi-hole-docker project and see what they think.

Testing the result of a ping -c 1  would be the most reliable

Up until now, I've been treating ServerIP as optional, so a lot of existing users may not have that env var set. In fact, the only behaviour difference I see without it is no longer being able to resolve http://pi.hole/admin on my LAN. So I agree it would be the most reliable I don't think I'm ready to force that env var on existing setups yet.
Interestingly, if we were running in host mode with proper docker and provided ServerIP, lighttpd would only bind to ServerIP. There's a snippet of code on startup that tries to identify if we are in host mode by listing interface names.
https://github.com/pi-hole/docker-pi-hole/blob/75328c63f86f3232b83e97cca0ede0613e7c5abd/bash_functions.sh#L160-L168
This has never impacted us because our bridge interface name is balena0 not docker0. Fortunate as well because we want to bind lighttpd to all interfaces for the Public Device URL functionality that uses our VPN.

Perhaps the hand-off of interfaces in host mode has changed in ways I can't begin to understand.

That may be a possible reason, but I'll try to confirm in a little bit.  I've never run in host mode so I don't know what "normal" looks like.  Now it's just a curiosity thing.

I don't know why lighttpd works. It's binding to all interfaces so maybe *:80 works differently than eth0:53.

This is more of an OS thing than a network thing, so I'm only going by what I've seen over the years, but there seems to be four "levels" of bindings:

all interfaces (lighttpd is here)
specific interfaces
all IP addresses (pihole-FTL is here)
specific IP addresses

Just speculation above.

provide an interface name
bind to 53 for each address on that interface
no addresses? keep running as if everything is fine

This is where I think they can optimize and bind to a specific interface.  I'm the furthest thing from a developer, so I don't know what the exact possibilities are here when it comes to the code/OS.
I guess from a Balena perspective there's always the possibility of conflicting with this:
dnsmasq   1531 nobody    4u  IPv4  20593      0t0  UDP <IP_ADDRESS>:53
So... I don't know if there is a "right" answer here that would work universally.

Not a terrible idea, you could run in past the pi-hole/pi-hole-docker project and see what they think.

Yeah, I want to do that once I can get my other RPi to fail.  I'd rather open an issue/feature request based on an actual use-case vs. a theoretical issue, and I'm trying to avoid any finger-pointing.

I don't think I'm ready to force that env var on existing setups yet.

Ok, that makes perfect sense - I think what you've got in the PR so far looks great and should scale well since we're forcing INTERFACE.
Are we worried about wlan0 users forgetting to change INTERFACE?  Does it fail today if they forget to do that?

Interestingly, if we were running in host mode with proper docker and provided ServerIP, lighttpd would only bind to ServerIP. There's a snippet of code on startup that tries to identify if we are in host mode by listing interface names.

Oh that is interesting - Just tested it here.  That's clever.  It's really not that different than our new test :-)
Thanks again for all of your help this (and last) week!  Let me know when you want me to test the changes when you're happy with them.  Feel free to roll them into dtoverlay :-)

Are we worried about wlan0 users forgetting to change INTERFACE? Does it fail today if they forget to do that?

Yeah, it fails in the main branch, but it's obvious in the logs why it fails. I do it all the time when I test on wifi and forget to change the interface name. Something along the lines of unknown interface eth0 and then I remember I need to set wlan0. But honestly, no one should be running a DNS server on wifi or with a dynamic IP address).

Let me know when you want me to test the changes when you're happy with them.

I think that PR is ready to go, just leave a comment on the PR once you've tested it and I'll merge. Your screen won't work while testing this PR of course.

Feel free to roll them into dtoverlay

Once we merge wait-for-ipv4 to main I'll rebase the dtoverlay branch so you can have the best of both worlds. The dtoverlay branch is still in alpha since my colleagues have to test the new fbcp image with a bunch of screens so we can avoid regression. Can't wait to get both of these PRs into main though!

Thanks again for all of your help this (and last) week!

No thank YOU good sir. It's been a pleasure and don't be a stranger in my repos.

The dtoverlay branch has been rebased on main so feel free to use that until we confirm the fbcp changes.

Postscript for you, @klutchell.
While discussing another project's issue here: https://github.com/wouterdebie/locast2tuner/issues/32
I started having flashbacks to this issue and decided to check how balena does things.
Sure enough, balena will start without the network being online where Docker will wait.
balena's systemd file looks like this:
[Unit]
Description=Balena Application Container Engine
Documentation=https://www.balena.io/docs/getting-started
Wants=dnsmasq.service
Requires=balena-engine.socket var-lib-docker.mount bind-etc-docker.service bind-home-root-.docker.service
After=network.target balena-engine.socket var-lib-docker.mount bind-etc-docker.service bind-home-root-.docker.service dnsmasq.service rollback-altboot.service

I'm wondering if we had these two options along with everything else if we would have ever run into this problem:
After=network-online.target
Wants=network-online.target

This is what Docker's systemd file looks like for a comparison:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket containerd.service

You are likely correct, but we don't want to use network-online in the IoT space as there are many use cases where the network may not be available but we expect the engine and services to start. Especially if user applications are expected to configure the network in some way.

network-online.target is a target that actively waits until the nework is "up", where the definition of "up" is defined by the network management software. Usually it indicates a configured, routable IP address of some kind. Its primary purpose is to actively delay activation of services until the network is set up. It is an active target, meaning that is may be pulled in by the services requiring the network to be up, but is not pulled in by the network management service itself. By default all remote mounts defined in /etc/fstab pull this service in, in order to make sure the network is up before it is attempted to connect to a network share. Note that normally, if no service requires it, and if no remote mount point is configured this target is not pulled into the boot, thus avoiding any delays during boot should the network not be available. It is strongly recommended not to pull in this target too liberally: for example network server software should generally not pull this in (since server software generally is happy to accept local connections even before any routable network interface is up), its primary purpose is network client software that cannot operate without network.

Hmmm interesting - So, this may reflect my lack of understanding of some of the other stuff that balena does.  There are user containers that would set/modify the host network settings?  How typical is this?  (All of these questions are just my curiosity.)
Not that my scenario appears to be overly common, either, but would it make sense to have base images for "Internet" apps and base images for IoT/Other?

We can optionally expose the host dbus, plus a number of other things like the host engine socket, to perform host OS tasks. I see it in support all the time, so it's not that rare.

https://www.balena.io/docs/learn/develop/multicontainer/#labels
https://www.balena.io/docs/learn/develop/runtime/#d-bus-communication-with-host-os
https://www.balena.io/docs/reference/OS/network/2.x/#changing-the-network-at-runtime

I don't see the need for multiple images, if a user application requires certain features to be available at startup, that's usually trivial to add a condition to the container, vs maintaining and entire separate image for a minor change in a service file.

You should do those masterclasses and ask questions like this on the forums so others can see your answers and we can improve our docs!

I don't see the need for multiple images, if a user application requires certain features to be available at startup, that's usually trivial to add a condition to the container, vs maintaining and entire separate image for a minor change in a service file.

Yeah, that makes perfect sense - Thanks for the explanation!
I love this:
"Warning: Making changes to the networking of a device is extremely dangerous and can lead to a device being unrecoverable, so exercise caution with any of the following."

You should do those masterclasses and ask questions like this on the forums so others can see your answers and we can improve our docs!

I really should.  Look man, I've had the tab open in my browser since you told me about it :-)

Re-opening this for discussion and to possibly re-introduce the fix.
This issue occurs when the pi-hole device is connected to a switch that is running a proper implementation of the spanning-tree protocol (STP.)  With STP there is a delay (Forwarding Delay) when the interface comes up of ~15 seconds while STP factors in the new "path" and makes sure that it will not cause any bridging loops before putting the interface into forwarding mode.  This delay impacts how quickly DHCP gets an IP address for the interface.  Because we run in Docker "host" mode for scalability reasons, this results in pihole-FTL not being able to bind to the interface when it is launched because there is not yet an IP address on it.
A workaround on these switches is to enable "STP Portfast" (if supported), which bypasses the STP forwarding delay on the interface and puts the port into a forwarding state immediately.  While enabling portfast is a best practice when you know an interface is connected to a host (vs. another switch/potential loop), it is not enabled by default on most commercial switches, and a lot of people leave it alone unless the forwarding delay is causing problems.
Consumer switches generally do run STP, but automatically set their ports to "portfast" mode.  The assumption with these switches is that they will not be connected to other switches and therefore there should not be any bridging loops in the topology.
In my case, I moved my pi-hole to a consumer switch a couple of years ago, so when this fix was removed, I wasn't impacted.  I brought my pi-hole home yesterday and connected it to my Cisco switch and the issue has resurfaced.

Any improvements over this shell to wait for IP assignments?
while [ -z "$(ip -o -4 addr show dev "${INTERFACE}")" ]
do
   echo "Waiting for IPv4 address on ${INTERFACE}..."
   sleep 5
done

Maybe something that also supports IPv6, or does some other network magic to determine when it's okay to bind?

I don't run v6 at home myself, but it would make sense to have this work for both v4 and v6 if we can do it without too much complexity.  I'm not sure how many people out there are running only v6, but I'm sure they exist.
The only thing we can really test for is a proper IP address (v4/v6) being assigned to the interface, but maybe we can ping a hostname (www.google.com) and make assumptions based on the results.
One of the other alternatives we tried above was to just force a 30 second wait before starting FTL which seemed to work and also wouldn't rely on anything or have any kinds of unpredictable failures.
I just checked Adguard to see if I have the same problem there, but I think the fact that we bind with - "<IP_ADDRESS>:80" is why it works fine.  Not 100% here.  Adguard might also not be as sensitive as pihole-FTL.
I can't believe it's been 2 1/2 years since this discussion. A lot of this is coming back to me now.

I thought about the ping of a public URL but I think Pi-hole itself is designed to start even with no internet access.
Maybe we can ping the default gateway? That should always exist and maybe fails when STP is pending? Do you want to try it?

Maybe we can ping the default gateway? That should always exist and maybe fails when STP is pending? Do you want to try it?

We don't have a default gateway if we don't have an IP address, so we don't know what to ping.

Maybe that's the test then? Check routes for a default gateway until one exists?

That should work.
Is it much different processing wise than what we're doing with ip -o -4 addr show dev "${INTERFACE}" or is this more or less the same?
Stepping-back: I'm the only one to ever report this as an issue, since I'm sure most people don't use $30k switches in their homes.  Do you think this is a problem we need to solve?

Do you think this is a problem we need to solve?

Honestly no, but I wasn't going to stand in your way if you wanted to persue it.
We can leave this issue open indefinitely, or close it with the STP workaround highlighted as the solution.

Sounds fair - I might update the docs to put a caveat in as well instructing people to enable portfast (and pointing them to this novel) if they encounter issues.    I figure if people have a switch capable of portfast then they'll know what it means.