device_tracker/asuswrt gives false positive on RT-AC86U

Home Assistant release with the issue:
0.82
Last working Home Assistant release (if known):
unknown
Operating environment (Hass.io/Docker/Windows/etc.):
RPi3, Raspbian, Python 3.6.3 virtual_env
Component/platform:
device_tracker/asuswrt
Description of problem:
For some devices, the component gives a false positive home state when they are disconnected from the network. After rebooting the router they have not_home state but after reconnecting and disconnecting the state remains home for next router restart. Some problematic devices are wire, some wireless. My router model is RT-AC86U with Merlin firmware 384.7_2.
All my devices have a static IP address assigned by DHCP.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
- platform: asuswrt
  host: !secret router_hostname
  username: !secret router_username
  ssh_key: '/home/homeassistant/.ssh/id_rsa'
  protocol: ssh

Additional information:
One of the devices that have such problems is Dune media player. This is the result of the commands for this device when it is disconnected from the network.
arp -n | grep 00:16:xx:xx:xx:xx
? (<IP_ADDRESS>) at 00:16:xx:xx:xx:xx [ether]  on br0

cat /var/lib/misc/dnsmasq.leases | grep 00:16:xx:xx:xx:xx
36939 00:16:xx:xx:xx:xx <IP_ADDRESS> dune 01:00:16:xx:xx:xx:xx

ip neigh | grep 00:16:xx:xx:xx:xx
<IP_ADDRESS> dev br0 lladdr 00:16:xx:xx:xx:xx STALE

Resault is false positive but the router in the web interface shows that the device isn't connected. I asked about it some time ago on the asuswrt-merlin forum and I got the answer that this method will not work on this router model (and all new models I think).

#18427

Closing since #18427 was merged

@cgarwood @kennedyshead I installed aioasus 1.1.11. Unfortunately, it didn't help. All devices physically disconnected from the network have in HA state home. Issue should be reopen.

Are you 100% sure you do use 1.1.11? If you have an issue I think it should be dealt with in the component: https:/github.com/kennedyshead/aioasuswrt
Install that test it with a script (you could find test in issues that are closed), any error would be great to have filed as bugs in that repo

I installed HA 0.83.0.dev0:
pip list | grep aioasuswrt
aioasuswrt              1.1.11

Devices marked with a green arrow are physically disconnected from the network but in HA they have state home. If I change the state of these devices manually to not_home, then the component restores the home state after a while.
My configuration:
asuswrt:
    host: <IP_ADDRESS>
    username: !secret router_username
    protocol: ssh
    mode: router
    ssh_key: ~/.ssh/id_rsa

The result of running the script:
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:__main__:wl
INFO:asyncssh:Opening SSH connection to <IP_ADDRESS>, port 22
INFO:asyncssh:[conn=0] Connection to <IP_ADDRESS>, port 22 succeeded
INFO:asyncssh:[conn=0]   Local address: <IP_ADDRESS>, port 57732
DEBUG:asyncssh:[conn=0] Requesting key exchange
DEBUG:asyncssh:[conn=0] Received key exchange request
DEBUG:asyncssh:[conn=0] Beginning key exchange
DEBUG:asyncssh:[conn=0] Completed key exchange
INFO:asyncssh:[conn=0] Beginning auth for user _redacted_
DEBUG:asyncssh:[conn=0] Trying password auth
INFO:asyncssh:[conn=0] Auth for user _redacted_ succeeded
DEBUG:asyncssh:[conn=0, chan=0] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=0] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=0]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && for dev in `nvram get wl_ifnames`; do wl -i $dev assoclist; done
INFO:asyncssh:[conn=0, chan=0] Received exit status 0
INFO:asyncssh:[conn=0, chan=0] Received channel close
INFO:asyncssh:[conn=0, chan=0] Channel closed
DEBUG:__main__:['assoclist F0:B4:xx:xx:xx:xx', 'assoclist 28:6C:xx:xx:xx:xx', 'assoclist 28:6C:xx:xx:xx:xx', 'assoclist 0C:E7:xx:xx:xx:xx', 'assoclist 0C:8B:xx:xx:xx:xx', 'assoclist A8:6B:xx:xx:xx:xx', 'assoclist 34:CE:xx:xx:xx:xx', 'assoclist 34:CE:xx:xx:xx:xx', 'assoclist 60:01:x:xx:xx:xx', 'assoclist 5C:CF:xx:xx:xx:xx', 'assoclist EC:FA:xx:xx:xx:xx', 'assoclist DC:4F:x:xx:xx:xx', 'assoclist 00:04x:xx:xx:xx', 'assoclist 28:6C:xx:xx:xx:xx', 'assoclist 78:11:xx:xx:xx:xx', 'assoclist 64:A2:xx:xx:xx:xx', 'assoclist 98:E0:xx:xx:xx:xx', 'assoclist D4:A3:xx:xx:xx:xx', 'assoclist 00:9E:xx:xx:xx:xx', '']
DEBUG:asyncssh:[conn=0, chan=1] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=1] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=1]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && for dev in `nvram get wl1_vifs && nvram get wl0_vifs && nvram get wl_ifnames`; do if type wlanconfig > /dev/null; then wlanconfig $dev list | awk 'FNR > 1 {print substr($1, 0, 18)}'; else wl -i $dev assoclist; fi; done
INFO:asyncssh:[conn=0, chan=1] Received exit status 0
INFO:asyncssh:[conn=0, chan=1] Received channel close
INFO:asyncssh:[conn=0, chan=1] Channel closed
DEBUG:__main__:{'F0:B4:xx:xx:xx:xx': Device(mac='F0:B4:xx:xx:xx:xx', ip=None, name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip=None, name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip=None, name=None), '0C:E7:xx:xx:xx:xx': Device(mac='0C:E7:xx:xx:xx:xx', ip=None, name=None), '0C:8B:xx:xx:xx:xx': Device(mac='0C:8B:xx:xx:xx:xx', ip=None, name=None), 'A8:6B:xx:xx:xx:xx': Device(mac='A8:6B:xx:xx:xx:xx', ip=None, name=None), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip=None, name=None), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip=None, name=None), '60:01:x:xx:xx:xx': Device(mac='60:01:x:xx:xx:xx', ip=None, name=None), '5C:CF:xx:xx:xx:xx': Device(mac='5C:CF:xx:xx:xx:xx', ip=None, name=None), 'EC:FA:xx:xx:xx:xx': Device(mac='EC:FA:xx:xx:xx:xx', ip=None, name=None), 'DC:4F:x:xx:xx:xx': Device(mac='DC:4F:x:xx:xx:xx', ip=None, name=None), '00:04x:xx:xx:xx': Device(mac='00:04x:xx:xx:xx', ip=None, name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip=None, name=None), '78:11:xx:xx:xx:xx': Device(mac='78:11:xx:xx:xx:xx', ip=None, name=None), '64:A2:xx:xx:xx:xx': Device(mac='64:A2:xx:xx:xx:xx', ip=None, name=None), '98:E0:xx:xx:xx:xx': Device(mac='98:E0:xx:xx:xx:xx', ip=None, name=None), 'D4:A3:xx:xx:xx:xx': Device(mac='D4:A3:xx:xx:xx:xx', ip=None, name=None), '00:9E:xx:xx:xx:xx': Device(mac='00:9E:xx:xx:xx:xx', ip=None, name=None)}
DEBUG:__main__:arp
DEBUG:asyncssh:[conn=0, chan=2] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=2] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=2]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && arp -n
INFO:asyncssh:[conn=0, chan=2] Received exit status 0
INFO:asyncssh:[conn=0, chan=2] Received channel close
INFO:asyncssh:[conn=0, chan=2] Channel closed
DEBUG:__main__:['? (<IP_ADDRESS>) at 00:11:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 0C:8B:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 00:9E:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 64:A2:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 0C:E7:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at A0:F3:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at DC:4F:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 28:6C:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at <incomplete>  on br0', '? (<IP_ADDRESS>) at A0:F3:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at A8:6B:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 14:60:x:xx:xx:xx [ether]  on eth0', '? (<IP_ADDRESS>) at <incomplete>  on br0', '? (<IP_ADDRESS>) at B8:27:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at <incomplete>  on br0', '? (<IP_ADDRESS>) at 00:05:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at <incomplete>  on br0', '? (<IP_ADDRESS>) at EC:FA:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at D4:A3:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 60:01:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 78:11:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 34:CE:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 98:E0:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at F0:B4:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 00:04x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at C0:EE:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 34:CE:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at <incomplete>  on br0', '? (<IP_ADDRESS>) at F8:46:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 5C:CF:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 28:6C:xx:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at F0:23:x:xx:xx:xx [ether]  on br0', '? (<IP_ADDRESS>) at 28:6C:xx:xx:xx:xx [ether]  on br0', '']
DEBUG:asyncssh:[conn=0, chan=3] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=3] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=3]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && arp -n
INFO:asyncssh:[conn=0, chan=3] Received exit status 0
INFO:asyncssh:[conn=0, chan=3] Received channel close
INFO:asyncssh:[conn=0, chan=3] Channel closed
DEBUG:aioasuswrt.asuswrt:Could not parse row: ? (<IP_ADDRESS>) at <incomplete>  on br0
DEBUG:aioasuswrt.asuswrt:Could not parse row: ? (<IP_ADDRESS>) at <incomplete>  on br0
DEBUG:aioasuswrt.asuswrt:Could not parse row: ? (<IP_ADDRESS>) at <incomplete>  on br0
DEBUG:aioasuswrt.asuswrt:Could not parse row: ? (<IP_ADDRESS>) at <incomplete>  on br0
DEBUG:aioasuswrt.asuswrt:Could not parse row: ? (<IP_ADDRESS>) at <incomplete>  on br0
DEBUG:__main__:{'F0:B4:xx:xx:xx:xx': Device(mac='F0:B4:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '0C:E7:xx:xx:xx:xx': Device(mac='0C:E7:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '0C:8B:xx:xx:xx:xx': Device(mac='0C:8B:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A8:6B:xx:xx:xx:xx': Device(mac='A8:6B:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '60:01:x:xx:xx:xx': Device(mac='60:01:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '5C:CF:xx:xx:xx:xx': Device(mac='5C:CF:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'EC:FA:xx:xx:xx:xx': Device(mac='EC:FA:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'DC:4F:x:xx:xx:xx': Device(mac='DC:4F:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:04x:xx:xx:xx': Device(mac='00:04x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '78:11:xx:xx:xx:xx': Device(mac='78:11:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '64:A2:xx:xx:xx:xx': Device(mac='64:A2:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '98:E0:xx:xx:xx:xx': Device(mac='98:E0:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'D4:A3:xx:xx:xx:xx': Device(mac='D4:A3:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:9E:xx:xx:xx:xx': Device(mac='00:9E:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:11:x:xx:xx:xx': Device(mac='00:11:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A0:F3:x:xx:xx:xx': Device(mac='A0:F3:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A0:F3:x:xx:xx:xx': Device(mac='A0:F3:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '14:60:x:xx:xx:xx': Device(mac='14:60:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'B8:27:x:xx:xx:xx': Device(mac='B8:27:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:05:x:xx:xx:xx': Device(mac='00:05:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'C0:EE:x:xx:xx:xx': Device(mac='C0:EE:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'F8:46:x:xx:xx:xx': Device(mac='F8:46:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'F0:23:x:xx:xx:xx': Device(mac='F0:23:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None)}
DEBUG:__main__:neigh
DEBUG:asyncssh:[conn=0, chan=4] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=4] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=4]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && ip neigh
INFO:asyncssh:[conn=0, chan=4] Received exit status 0
INFO:asyncssh:[conn=0, chan=4] Received channel close
INFO:asyncssh:[conn=0, chan=4] Channel closed
DEBUG:__main__:['<IP_ADDRESS> dev br0 lladdr 00:11:x:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0 lladdr 0C:8B:xx:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0 lladdr 00:9E:xx:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0 lladdr 64:A2:xx:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0 lladdr 0C:E7:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr A0:F3:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr DC:4F:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 28:6C:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0  FAILED', '<IP_ADDRESS> dev br0 lladdr A0:F3:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr A8:6B:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev eth0 lladdr 14:60:x:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0  FAILED', '<IP_ADDRESS> dev br0 lladdr B8:27:x:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0  FAILED', '<IP_ADDRESS> dev br0 lladdr 00:05:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0  FAILED', '<IP_ADDRESS> dev br0 lladdr EC:FA:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr D4:A3:xx:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0 lladdr 60:01:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 78:11:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 34:CE:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 98:E0:xx:xx:xx:xx REACHABLE', '<IP_ADDRESS> dev br0 lladdr F0:B4:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 00:04x:xx:xx:xx DELAY', '<IP_ADDRESS> dev br0 lladdr C0:EE:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 34:CE:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0  FAILED', '<IP_ADDRESS> dev br0 lladdr F8:46:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 5C:CF:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 28:6C:xx:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr F0:23:x:xx:xx:xx STALE', '<IP_ADDRESS> dev br0 lladdr 28:6C:xx:xx:xx:xx STALE', '']
DEBUG:asyncssh:[conn=0, chan=5] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=5] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=5]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && ip neigh
INFO:asyncssh:[conn=0, chan=5] Received exit status 0
INFO:asyncssh:[conn=0, chan=5] Received channel close
INFO:asyncssh:[conn=0, chan=5] Channel closed
DEBUG:__main__:{'F0:B4:xx:xx:xx:xx': Device(mac='F0:B4:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '0C:E7:xx:xx:xx:xx': Device(mac='0C:E7:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '0C:8B:xx:xx:xx:xx': Device(mac='0C:8B:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A8:6B:xx:xx:xx:xx': Device(mac='A8:6B:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '60:01:x:xx:xx:xx': Device(mac='60:01:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '5C:CF:xx:xx:xx:xx': Device(mac='5C:CF:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'EC:FA:xx:xx:xx:xx': Device(mac='EC:FA:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'DC:4F:x:xx:xx:xx': Device(mac='DC:4F:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:04x:xx:xx:xx': Device(mac='00:04x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '78:11:xx:xx:xx:xx': Device(mac='78:11:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '64:A2:xx:xx:xx:xx': Device(mac='64:A2:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '98:E0:xx:xx:xx:xx': Device(mac='98:E0:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'D4:A3:xx:xx:xx:xx': Device(mac='D4:A3:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:9E:xx:xx:xx:xx': Device(mac='00:9E:xx:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:11:x:xx:xx:xx': Device(mac='00:11:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A0:F3:x:xx:xx:xx': Device(mac='A0:F3:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A0:F3:x:xx:xx:xx': Device(mac='A0:F3:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '14:60:x:xx:xx:xx': Device(mac='14:60:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'B8:27:x:xx:xx:xx': Device(mac='B8:27:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), '00:05:x:xx:xx:xx': Device(mac='00:05:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'C0:EE:x:xx:xx:xx': Device(mac='C0:EE:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'F8:46:x:xx:xx:xx': Device(mac='F8:46:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'F0:23:x:xx:xx:xx': Device(mac='F0:23:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None)}
DEBUG:__main__:leases
DEBUG:asyncssh:[conn=0, chan=6] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=6] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=6]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && cat /var/lib/misc/dnsmasq.leases
INFO:asyncssh:[conn=0, chan=6] Received exit status 0
INFO:asyncssh:[conn=0, chan=6] Received channel close
INFO:asyncssh:[conn=0, chan=6] Channel closed
DEBUG:__main__:['70168 44:85:xx:xx:xx:xx <IP_ADDRESS> laptop-edyta 01:44:85:xx:xx:xx:xx', '69987 d4:xx:xx:xx:xx:xx <IP_ADDRESS> iphone 01:d4:xx:xx:xx:xx:xx', '64245 00:11:x:xx:xx:xx <IP_ADDRESS> synology 01:00:11:x:xx:xx:xx', '47570 F0:23:x:xx:xx:xx <IP_ADDRESS> fingbox ff:54:cc:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx', '84942 B8:27:x:xx:xx:xx <IP_ADDRESS> raspberry 01:B8:27:x:xx:xx:xx', '73278 A0:F3:x:xx:xx:xx <IP_ADDRESS> switch *', '68204 00:05:x:xx:xx:xx <IP_ADDRESS> denon 01:00:05:x:xx:xx:xx', '68420 00:04x:xx:xx:xx <IP_ADDRESS> harmony 01:00:04x:xx:xx:xx', '57128 34:CE:xx:xx:xx:xx <IP_ADDRESS> shelly1 *', '85999 F0:B4:xx:xx:xx:xx <IP_ADDRESS> migateway *', '67320 28:6c:xx:xx:xx:xx <IP_ADDRESS> yeelight-lightstrip-1 *', '66839 78:11:xx:xx:xx:xx <IP_ADDRESS> yeelight-lightstrip-2 *', '60676 F8:46:x:xx:xx:xx <IP_ADDRESS> playstation 01:F8:46:x:xx:xx:xx', '62287 A8:6B:xx:xx:xx:xx <IP_ADDRESS> brother 01:A8:6B:xx:xx:xx:xx', '83403 64:a2:xx:xx:xx:xx <IP_ADDRESS> oneplus 01:64:a2:xx:xx:xx:xx', '68481 00:9e:xx:xx:xx:xx <IP_ADDRESS> mibox 01:00:xx:xx:xx:xx:xx', '53285 5C:CF:xx:xx:xx:xx <IP_ADDRESS> sonoff-basic-1 *', '79231 DC:4F:x:xx:xx:xx <IP_ADDRESS> wemos-d1-mini *', '53294 60:01:x:xx:xx:xx <IP_ADDRESS> sonoff-basic-3 *', '70706 98:e0:xx:xx:xx:xx <IP_ADDRESS> ipad 01:98:xx:xx:xx:xx:xx', '52400 EC:FA:xx:xx:xx:xx <IP_ADDRESS> sonoff-basic-2 *', '63639 34:CE:xx:xx:xx:xx <IP_ADDRESS> dafang 01:34:CE:xx:xx:xx:xx', '70204 0C:E7:xx:xx:xx:xx <IP_ADDRESS> lumia 01:0C:E7:xx:xx:xx:xx', '84156 28:6C:xx:xx:xx:xx <IP_ADDRESS> yeelight-color-bulb-1 *', '84156 28:6C:xx:xx:xx:xx <IP_ADDRESS> yeelight-color-bulb-2 *', '54354 40:16:xx:xx:xx:xx <IP_ADDRESS> stacjonarny 01:40:16:xx:xx:xx:xx', '70194 0C:8B:xx:xx:xx:xx <IP_ADDRESS> laptop 01:0C:8B:xx:xx:xx:xx', '']
DEBUG:asyncssh:[conn=0, chan=7] Set write buffer limits: low-water=16384, high-water=65536
INFO:asyncssh:[conn=0, chan=7] Requesting new SSH session
INFO:asyncssh:[conn=0, chan=7]   Command: PATH=$PATH:/bin:/usr/sbin:/sbin && cat /var/lib/misc/dnsmasq.leases
INFO:asyncssh:[conn=0, chan=7] Received exit status 0
INFO:asyncssh:[conn=0, chan=7] Received channel close
INFO:asyncssh:[conn=0, chan=7] Channel closed
DEBUG:__main__:{'F0:B4:xx:xx:xx:xx': Device(mac='F0:B4:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='migateway'), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='yeelight-color-bulb-2'), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='yeelight-color-bulb-1'), '0C:E7:xx:xx:xx:xx': Device(mac='0C:E7:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='lumia'), '0C:8B:xx:xx:xx:xx': Device(mac='0C:8B:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='laptop'), 'A8:6B:xx:xx:xx:xx': Device(mac='A8:6B:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='brother'), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='shelly1'), '34:CE:xx:xx:xx:xx': Device(mac='34:CE:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='dafang'), '60:01:x:xx:xx:xx': Device(mac='60:01:x:xx:xx:xx', ip='<IP_ADDRESS>', name='sonoff-basic-3'), '5C:CF:xx:xx:xx:xx': Device(mac='5C:CF:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='sonoff-basic-1'), 'EC:FA:xx:xx:xx:xx': Device(mac='EC:FA:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='sonoff-basic-2'), 'DC:4F:x:xx:xx:xx': Device(mac='DC:4F:x:xx:xx:xx', ip='<IP_ADDRESS>', name='wemos-d1-mini'), '00:04x:xx:xx:xx': Device(mac='00:04x:xx:xx:xx', ip='<IP_ADDRESS>', name='harmony'), '28:6C:xx:xx:xx:xx': Device(mac='28:6C:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='yeelight-lightstrip-1'), '78:11:xx:xx:xx:xx': Device(mac='78:11:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='yeelight-lightstrip-2'), '64:A2:xx:xx:xx:xx': Device(mac='64:A2:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='oneplus'), '98:E0:xx:xx:xx:xx': Device(mac='98:E0:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='ipad'), 'D4:A3:xx:xx:xx:xx': Device(mac='D4:A3:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='iphone'), '00:9E:xx:xx:xx:xx': Device(mac='00:9E:xx:xx:xx:xx', ip='<IP_ADDRESS>', name='mibox'), '00:11:x:xx:xx:xx': Device(mac='00:11:x:xx:xx:xx', ip='<IP_ADDRESS>', name='synology'), 'A0:F3:x:xx:xx:xx': Device(mac='A0:F3:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'A0:F3:x:xx:xx:xx': Device(mac='A0:F3:x:xx:xx:xx', ip='<IP_ADDRESS>', name='switch'), '14:60:x:xx:xx:xx': Device(mac='14:60:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'B8:27:x:xx:xx:xx': Device(mac='B8:27:x:xx:xx:xx', ip='<IP_ADDRESS>', name='raspberry'), '00:05:x:xx:xx:xx': Device(mac='00:05:x:xx:xx:xx', ip='<IP_ADDRESS>', name='denon'), 'C0:EE:x:xx:xx:xx': Device(mac='C0:EE:x:xx:xx:xx', ip='<IP_ADDRESS>', name=None), 'F8:46:x:xx:xx:xx': Device(mac='F8:46:x:xx:xx:xx', ip='<IP_ADDRESS>', name='playstation'), 'F0:23:x:xx:xx:xx': Device(mac='F0:23:x:xx:xx:xx', ip='<IP_ADDRESS>', name='fingbox')}

How long time is it online, there is an unfortunate "delay" in the connection.

Device dune (wire) is offline for 2 days (so shows Fingbox which scans devices on the network):

In HA with asuswrt component it has state home almost all the time (w domu means home):

Another (wireless) device jerzy is offline for 2 days:

I na HA this device is almost online all the time:

@bieniu did you ever solve this? I've have the same issue and been searching for answers. I just got a Asus ac86u and flashed Merlin's latest stable and latest home assistant 84.2.  The issue is with an iPhone, if I turn off WiFi it will switch to away but whenever the iPhone leaves the house it always shows home. Android devices work perfectly.  Any help would be appreciated.

@bieniu i agree i would like to see this reopened, however im starting to think that this is related to merlin firmware possibly.  im unsure how to use the aioasuswrt test scripts so i used several methods to do some testing.  i took and older version of the device tracker from another issue thread so i could more easily modify it in custom_components for testing(https://pastebin.com/JHew4djg).  i checked each individual command by removing all but one from
        devices.update(self._get_wl())
        devices.update(self._get_arp())
        devices.update(self._get_neigh(devices))
        if not self.mode == 'ap':
            devices.update(self._get_leases(devices))

get_wl  gives output like "assoclist xx:xx:xx:xx:A0:B0" and since there is no ip listed with it, it is ignored by this block
        for key in devices:
            if self.mode == 'ap' or devices[key].ip is not None:
                ret_devices[key] = devices[key]
        return ret_devices

so i removed the "if" line and retested then i get accurate detection from wifi, i then moved on to try get_neigh which worked as expected however devices come and go offline frequently when not sending traffic.  so i then ran the device_tracker for a day using just
       devices.update(self._get_wl())
       devices.update(self._get_neigh(devices))

and now i was getting accurate detection from my wifi devices.  so next i started playing with just using get_arp, this is where i think the problem lies.  first i found that some devices were not reporting correctly in hass and i saw
DEBUG:aioasuswrt.asuswrt:Could not parse row: ? (<IP_ADDRESS>) at <incomplete>  on br0
in your logs, so the devices that are incomplete should be away but nothing seems to be happening so i edited the line in
def _get_arp(self):

if device['mac'] is not None:

to
if device['mac'] is not '<incomplete>':

this seemed to help but i was still having issues with an iphone that would not show away if it leaves the house.  so after monitoring the commands during the events i found that while the device is no longer connected, "arp -n" still shows
? (<IP_ADDRESS>) at xx:xx:xx:xx:xx:xx [ether]  on br0

note that it does not show as "incomplete" like other disconnected devices, but it is gone from get_wl and stale from get_neigh, so something may be wrong with the way the router is handling a device that does not disconnect from network "gracefully".  i am starting to exhaust my limited ability to understand these things and was hoping to further a dialog to help figure this out.  i know my approach is "ugly" as im just hacking apart on ald device_tracker but at this point thats all i could figure out.  so for now im just using wl and neigh for wireless presence and all is working.  i hope this can help move forward with this. i apologize if this was too long or should be posted else where, i am open to such feedback.

This is not a bug in home assistant, its a bug in aioasuswrt.


Correctly reading the filename from disposition

Hello,
this issue is related to #834.
the problem relates in this code
const fileNameMatch = contentDisposition ? /filename="?([^"]*)"?;/g.exec(contentDisposition) : undefined;

you are matching all the filenames with a ; but if the content-disposition ONLY has a normal filename, there's no ; on the http-header value, this meaning on a fail Regex
example:
Content-Disposition:attachment; filename=Belege.pdf

hope it's clear :)

So the simplest solution would be to just add a ;?
const fileNameMatch = contentDisposition ? /filename="?([^"]*)"?;/g.exec(contentDisposition + “;”) : undefined;

Or better:
/filename="?([^"]*?)"?(;|$)/g.exec(contentDisposition)

Can you test the second regex in your app?

works ;)


Search Result Podcast Date (when background green): Elements must have sufficient color contrast

💥 Impact rating
serious
👣 Steps to reproduce
Add a podcast to the queue, start playing it, pause it, and then run the aXe Accessibility Test.
⚛ Element(s) with issue
Element location
.hTfpdO > section > b > span
Element source
<span>(2018-08-21)</span>
🛠 Recommended fix
Fix the following:
Element has insufficient color contrast of 2.79 (foreground color: #157169, background color: #7fbfb8, font size: 9.6pt, font weight: bold). Expected contrast ratio of 4.5:1
Related node:
.hTfpdO
📚 Reference information
https://dequeuniversity.com/rules/axe/3.0/color-contrast?application=AxeChrome

Closed by #237


Allow deployment on kubernetes 1.16+

Hi, I discovered an issue trying to deploy OSBA on newer Kubernetes releases (e.g. 1.16+), due to the promoted Deployment object API. I made deployment apiVersion parametric based on kubernetes version, and changed the predefined image tag in values.yaml

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: rkapoz-tech sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

@norshtein would be good to take a look at this ^^
FYI we had the same issue, and also had to update the redis version this chart depends on:
https://github.com/Azure/open-service-broker-azure/compare/master...hmcts:master


Fix/find context correction (#9949)

trigger input change on new notebooks

return to avoid further executions

remove unnecessary check

Port #9949
Fixes #9551

/merge

Merging PR based on approval from @chlafreniere


Broken link for "Pair"

The current link is "https://https//github.com/jiggy-ai/pair", should be "https://github.com/jiggy-ai/pair".

Thank you!


[SPARK-31913][SQL] Fix StackOverflowError in FileScanRDD

What changes were proposed in this pull request?
Resolve the recursive calls in FileScanRDD.
Why are the changes needed?
FileScanRDD throw a StackOverflowError when read a mass of empty files.
There are two ways to scan a file source, i.e. BucketedRead and NonBucketedRead. When scan a file source in NonBucketedRead way, spark will calculate how much files will be read in each partition with two important configuration: spark.sql.files.maxPartitionBytes and spark.sql.files.openCostInBytes. With these two config, we can calculate the maxSplitBytes in each partition. If there are lots of input files and low parallelism, the maxSplitBytes will be spark.sql.files.maxPartitionBytes. In my failed job, I set spark.sql.files.maxPartitionBytes with a large number, like 5GB. It means a spark task will read 5GB files at most. But accidentally,
a mass of empty files was produced in my environment. Then a spark task will read a mass of empty files. But the FileScanRDD read this files in recursive way:
     def hasNext: Boolean = {
        // Kill the task in case it has been marked as killed. This logic is from
        // InterruptibleIterator, but we inline it here instead of wrapping the iterator in order
        // to avoid performance overhead.
        context.killTaskIfInterrupted()
        (currentIterator != null && currentIterator.hasNext) || nextIterator()
      }

     private def nextIterator(): Boolean = {
         ...
         try {
           hasNext
         } catch {
         ...
     }

(currentIterator != null && currentIterator.hasNext) always is false, and finally this will trigger a StackOverflowError.
!!! Have to admit, it is a corner case as  a mass of empty files was produced in my environment accidentally.
java.lang.StackOverflowError
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.getSubject(Subject.java:297)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:648)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2828)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
	at org.apache.parquet.hadoop.util.HadoopInputFile.fromPath(HadoopInputFile.java:38)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:640)
	at org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:148)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:143)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat.$anonfun$buildReaderWithPartitionValues$2(ParquetFileFormat.scala:326)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:169)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)

Does this PR introduce any user-facing change?
No
How was this patch tested?
Add new UT.

Test build #123561 has started for PR 28737 at commit 2fac4a8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28184/
Test PASSed.

Test build #123561 has finished for PR 28737 at commit 2fac4a8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123561/
Test FAILed.

pending on fix ut failure

cc @cloud-fan @xuanyuanking

Test build #123611 has started for PR 28737 at commit 1e17fd0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28234/
Test PASSed.

Test build #123611 has finished for PR 28737 at commit 1e17fd0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123611/
Test FAILed.

retest this please.

Test build #123630 has started for PR 28737 at commit 1e17fd0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28254/
Test PASSed.

Test build #123630 has finished for PR 28737 at commit 1e17fd0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123630/
Test PASSed.

Could you clearly describe what is a root cause of this issue in the PR description?

@maropu I have updated the PR description, take a review please.

Let me clarify. The issue is the recursive calls in FileScanRDD will cause StackOverflowError while we have too many empty files. Could you please quantify the number of empty files in your env?

@xuanyuanking Thanks for clarifying. Unfortunately,  those empty files has been removed, I remember there are thousands of empty files at least.

ping @xuanyuanking

Same question with Takeshi here https://github.com/apache/spark/pull/28737#discussion_r437831211

Yea, we need env-independent tests for this issue...


First pass at sniff to confirm inline docs for filters are present

Fixes #424
First pass at sniff to confirm inline docs for hooks are present.
I'm adding this into Jetpack via https://github.com/Automattic/jetpack/pull/15693 since I wanted to check for some Jetpack-specific things. Per suggestion, wanted to offer the "base" to core WPCS for everyone's benefit.
In the initial commit, it finds a matching the various hook function (L27) using WPCS' AbstractFunctionRestrictionsSniff and checks for a preceding comment. The preceding comment need to be on the previous line (noting this is an open question from the Jetpack PR https://github.com/Automattic/jetpack/pull/15693#pullrequestreview-406406725) and confirm it is using the docblock format.
For the initial commit, it is verifying it has a @since tag within the docblock and that the value starts with a X.Y.Z version number. I'm not sure about WP itself, but Jetpack requires three levels to both confirm the exact version (e.g. 8.5 vs 8.5.0) and ensure that all hooks in the same major release (X.Y) will have the same value to avoid our docs parser tagging it as two separate versions.
If Core WPCS doesn't need that, I can remove that to just verify a @since tag is present and move the version check itself to Jetpack as I did with our @module tag in https://github.com/Automattic/jetpack/pull/15693/files#diff-eb7bc4afcdf23e6f99ae1f793afd119bR57

Hi @kraftbj Welcome to WPCS ;-)
Thanks for this PR. I haven't looked at the code yet, but would like to ask you whether you have run the sniff over WP Core to see what it throws up.
Any false positives from WP Core should be handled, or when in doubt whether something is a false positive, let's discuss it.

For the initial commit, it is verifying it has a @since tag within the docblock and that the value starts with a X.Y.Z version number. I'm not sure about WP itself, but Jetpack requires three levels to both confirm the exact version (e.g. 8.5 vs 8.5.0) and ensure that all hooks in the same major release (X.Y) will have the same value to avoid our docs parser tagging it as two separate versions.

Core requires three levels too as per the Documentation guidelines, so checking for three levels is fine.
FYI: I expect WPCS 3.0.0 to contain a trait for version number checking (written for another sniff), so that can probably be re-used.
With that in mind and the last 2.x release expected soon, after which the pulls for WPCS 3.0.0 will start, I'd like to earmark this PR for WPCS 3.0.0.

Sounds good. New to PHPCS overall, so appreciate the guidance on naming scheme. Took a guess that WP seem to fit best based on the existing directories.
I'll get some unit tests worked up early next week since it sounds like this is a sniff that would be considered for inclusion.

New to PHPCS overall, so appreciate the guidance ...

Please feel free to ask for as much guidance as you need from me and the others in the team.
In my case, my remarks will often presume you know what you're doing with PHPCS (as why would you send in a sniff otherwise?), please don't let that intimidate you and ask for as much clarification as you need.

it sounds like this is a sniff that would be considered for inclusion.

Definitely. There are long standing open issues about this check missing from WPCS, though I also have to caution you that it may not be the easiest issue to tackle. Can be a great way to learn more about writing sniffs though ;-)

Sounds good. I appreciate the tip on running it against WP to find false positives. I'll make a couple of tweaks when sending up the unit tests.

For the initial commit, it is verifying it has a @since tag within the docblock and that the value starts with a X.Y.Z version number.

Maybe @since Unknown should also be permitted? I believe that has/had some instances in core function/methods DocBlocks as well at one point (example in SimplePie package).

Maybe @since Unknown should also be permitted?

I don't think we should allow that. Both SVN as well as Git allow to trace things through history, so I can't think of a valid reason to allow @since Unknown.

I added unit tests and documentation, but I'm not sure how to unit test when the ruleset would be passing a property as an array.
With the X.Y.Z. format, there are two instances currently in Core which does not follow this which I am not keen at automatically changing:

0.71 -- the initial release did not follow the X.Y.Z format.
MU (3.0.0) -- to denote items that came in when MU was merged into Core. We could modify these to be 3.0.0 (MU) as an alternative.

To allow these, I set a rule property of an array that could be defined in the ruleset XML for any "exceptional" versions. I just don't know how to pass that to the unit test via phpcs:set.
Any pointers?

Maybe @since Unknown should also be permitted?

For Core, I would disagree, but for implementing projects, I did add a way to set an exceptional version. If an implementing project wanted to allow that, they could in their own phpcs.xml.dist file as the PR stands now.

I'm not sure how to unit test when the ruleset would be passing a property as an array.

See: https://github.com/WordPress/WordPress-Coding-Standards/blob/develop/WordPress/Tests/NamingConventions/PrefixAllGlobalsUnitTest.1.inc#L6 for an example of setting an array property.
Please also remember to reset the property after the test using it and/or at the end of the test file (see bottom of the above linked test file for an example).

@jrfnl I really appreciate the pointers and direction. I think this is at a state where you can ~rip it to shreds~ review it.

@kraftbj Wowie! I'll try and have a look some time this week ;-)

This is wonderful. Thank you so much for the review. Today is a standard holiday in the U.S. so a short week for me. I'll try to make a good dent on this during the week, but may get bumped to next. If so, my delay doesn't mean I'm discouraged! 🙂

While it would be nice to have a sniff for this, the PR as-is is not mergable with way too many open review comments and no action taken on them in over two years, so I'm going to close this PR.


Required again on sierra

Looks like macos sierra requires this hack again. If others can confirm this, I will provide a patch to the readme to explaining this.

It seems to be broken (again) under macOS Sierra.

100% confirmed.
😞

The current fix that worked was to use: reattach-to-user-namespace pbcopy always.
So it is probably reattaching every time, but works.
Do you have any better workarounds?

Nothing besides what's already mentioned in the README. I sent feedback to Apple about the regression, but I am pessimistic that will do anything.


New ticket notification emails

I do not receive any new ticket notification emails. If I assign a ticket, I get that notification. Is it possible to auto assign new tickets? Or how do I enable notifications on new tickets?

Hi
There is the notification settings in your account section, you can enable/disable the ones you want

Yes, I have them all checked

How do you create the ticket?
If you create a new ticket from handesk itself you won't receive a notification (as you created it)
To auto assign new tickets, you just need to update the status and if the ticket is not assigned it will automatically auto assign to you


net: tcp: Set timer for last ACK

If we do not receive last ACK when the connection is tore down,
then do not wait forever as that would eat all the resources in
the network stack. So when we enter the LAST_ACK state, we setup
a timer that will unref the connection if the last ACK is not
received.
Signed-off-by: Jukka Rissanen<EMAIL_ADDRESS>

Maybe make the ACK_TIMEOUT a configurable setting with 1s as the default, like CONFIG_NET_TCP_2MSL_TIME is?

Maybe make the ACK_TIMEOUT a configurable setting with 1s as the default, like CONFIG_NET_TCP_2MSL_TIME is?

Sure, good idea. I will send a separate patch for that.


Fix convertToBase value of gist:_minute

Base unit is gist:_second, conversion factor was 1.0, should be 60.0.

PR withdrawn, issue to be fixed in a later commit.

@uscholdm Michael, now I'm confused about why the change needs to be made in e6, since you told me that not everyone is using e6. How are they making edits?

Not everyone uses e6 for client projects (I might be the only one, now that Dave does very little ontology development).
But the only way to evolve gist for now is using e6.
From: Rebecca Younes<EMAIL_ADDRESS>Sent: Thursday, August 8, 2019 5:52 AM
To: semanticarts/gist<EMAIL_ADDRESS>Cc: Michael Uschold<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [semanticarts/gist] Fix convertToBase value of gist:_minute (#83)
@uscholdmhttps://github.com/uscholdm Michael, now I'm confused about why the change needs to be made in e6, since you told me that not everyone is using e6. How are they making edits?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/semanticarts/gist/pull/83?email_source=notifications&email_token=ACSHHSMV5OHYWP77ZJN34XTQDQJGPA5CNFSM4IITHJE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD33QHUQ#issuecomment-519504850, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ACSHHSO27ZYMZ2DH4KAXGBTQDQJGPANCNFSM4IITHJEQ.


Add link to testing guide

cc @cpojer, added a link to the new testing guide I just released!

Current coverage is 90.31% (diff: 100%)

Merging #1614 into master will not change coverage

@@             master      #1614   diff @@
==========================================
  Files            28         28
  Lines          1125       1125
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1016       1016
  Misses          109        109
  Partials          0          0

Powered by Codecov. Last update 046e665...228bf1a

Thank you!
I'm probably going to turn this into a bullet list later but for now this is good.


/bin/sh: 1: Syntax error: "(" unexpected outfile: results/output.mp4

Getting this issue while running google colab:
/bin/sh: 1: Syntax error: "(" unexpected
outfile: results/output.mp4
Can't generate and save output mp4 file.

/bin/sh: 1: ffmpeg: not found
Can't generate and save output mp4 file too


CosmosAsyncStoredProcedure / CosmosStoredProcedure APIs and Docs

Change Object[] to List of Objects in execute() API

This is done.


hide commented out blocks

commented out (I keep it hide yaml from note metadata but for stuff like navigation, clean up) doesn't work on perlite

Describe the solution you'd like
It should like obsidian

Obsidian Metadata (front matter) is using yaml syntax, I cant find anything about using %% as a comment indicator. Yaml only supports # for commenting as far as I know.
also in your third screenshot (I suggest this shows obsidian) there is no front matter at all, so I suggest this is the obsidian view mode and you don`t have enabled the display of metadata. In Perlite you can also hide the metadata by clicking on the header or just hide it by default by changing this setting


Skip defaultGw check if sandbox is being deleted

On Sandbox deletion, during Leave from each
connected endpoint, avoid the default gw
check, which may create an unnecessary
connection to the default gateway network.

Especially during sandbox cleanup on start
after an ungraceful shutdown of the docker
daemon, the unnecessary connection to the
default gw network is likely to happen, if a
container was connected to multiple network.

Signed-off-by: Alessandro Boch<EMAIL_ADDRESS>

LGTM

LGTM


Could you also consider adding example on how to add custom API's? I think this approach might be generalizable to other kind of API's, such as search API's.

          Could you also consider adding example on how to add custom API's? I think this approach might be generalizable to other kind of API's, such as search API's.

Originally posted by @torronen in https://github.com/microsoft/FLAML/issues/954#issuecomment-1528163090

@torronen I'd like to understand more about what you mean by "add custom API's". Could you elaborate?

I believe this should already be doable, so maybe just needs a sample.
General examples:
1.users could add themselves other engines, such as gpt-3.5-turbo-0301 or their own finetune OpenAI model names
2.users could use other LLM providers than OpenAI
3.users could "finetune" any kind of endpoints that provide any kind of parameters
I may think about this in a too complex way, but I am thinking something like this
 default_search_space = {
    "url": tune.choice([
        "https://api.example.org",
         "https://api-alternative.example.org"
    ]),
    "verb": "POST",
    "json_structure": "temperature: {temperature}, top_p: {top_p}",   // optional, if not specified, then flat JSON
    "temperature_or_top_p": tune.choice(
        [
            {"temperature": tune.uniform(0, 1)},
            {"top_p": tune.uniform(0, 1)},
        ]
    ),
    "max_tokens": tune.lograndint(50, 1000),
    "n": tune.randint(1, 100),
    "prompt": "{prompt}",
}

My specific case:
I am working on degree and occupation suggestion problem. Users (students, people looking for a new career) etc. will provide a list of data (personality test results, work history, general interests, preferences, goals etc.) and my task is to provide them potential occupation from ESCO classification. In the next step, I will need to find suitable degrees for the selected occupation. For example, user may select "software developer" so the app should suggest a list of university, college and vocational degrees, potentially ranking them. It is more challenging for other occupation, such as primary school teacher, where there is only 1 or few correct answers but there are many other degrees with similar or related content.
The first version is complete. The endpoint uses mixture of models to give the predictions. For each model and data preprocessing there are a few parameters. There are also other parameters for the matching service, such as spellchecking tolerance, weight given for each data point and so on. I have also a few options for the search strategy, such as rank everything, or first drop 90% then rank etc.
When reading about the OpenAI integration, I thought I might be able to use this to get "finetune" this service. It is not a model but I could expose all parameters and then finetune it like it were a model. Perhaps it could even one way to build different user profiles in  the future.

I believe this should already be doable with FLAML, so maybe just needs a sample.
General examples: 1.users could add themselves other engines, such as gpt-3.5-turbo-0301 or their own finetune OpenAI model names 2.users could use other LLM providers than OpenAI 3.users could "finetune" any kind of endpoints that provide any kind of parameters
I may think about this in a too complex way, but I am thinking something like this
 default_search_space = {
    "url": tune.choice([
        "https://api.example.org",
         "https://api-alternative.example.org"
    ]),
    "verb": "POST",
    "json_structure": "temperature: {temperature}, top_p: {top_p}",   // optional, if not specified, then flat JSON
    "temperature_or_top_p": tune.choice(
        [
            {"temperature": tune.uniform(0, 1)},
            {"top_p": tune.uniform(0, 1)},
        ]
    ),
    "max_tokens": tune.lograndint(50, 1000),
    "n": tune.randint(1, 100),
    "prompt": "{prompt}",
}

My specific case: I am working on degree and occupation suggestion problem. Users (students, people looking for a new career) etc. will provide a list of data (personality test results, work history, general interests, preferences, goals etc.) and my task is to provide them potential occupation from ESCO classification. In the next step, I will need to find suitable degrees for the selected occupation. For example, user may select "software developer" so the app should suggest a list of university, college and vocational degrees, potentially ranking them. It is more challenging for other occupation, such as primary school teacher, where there is only 1 or few correct answers but there are many other degrees with similar or related content.
The first version is complete. The endpoint uses mixture of models to give the predictions. For each model and data preprocessing there are a few parameters. There are also other parameters for the matching service, such as spellchecking tolerance, weight given for each data point and so on. I have also a few options for the search strategy, such as rank everything, or first drop 90% then rank etc.
When reading about the OpenAI integration, I thought I might be able to use this to get "finetune" this service. It is not a model but I could expose all parameters and then finetune it like it were a model. Perhaps it could even one way to build different user profiles in the future.

Thanks for the elaboration. (1) is easy to do with the current flaml.oai.tune by specifying model. (2) and (3) are feasible with flaml.tune but not with flaml.oai.tune currently. Are you familiar with how flaml.tune works?

@sonichi not yet, but the docs seem clear enough as a starting point.  I think I tried it a bit a long time ago, but I am not sure. Thanks for the tip!
(1) Can we put any model name as parameter and it should work, including finetuned ones?
BTW, I am not sure if it is interesting or not, but I learned about a project that aims to re-create OpenAI API for local models, such as LLaMa. Only thing that needs updating is the endpoint address. I have not yet tried it and seems I now misremember the name.

@sonichi not yet, but the docs seem clear enough as a starting point. I think I tried it a bit a long time ago, but I am not sure. Thanks for the tip!
(1) Can we put any model name as parameter and it should work, including finetuned ones?

According to OpenAI's documentation, (1) should work for fine-tuned models. I have only tested non-fine-tuned models. If you get a chance to test fine-tuned model, I'll be curious to know.

BTW, I am not sure if it is interesting or not, but I learned about a project that aims to re-create OpenAI API for local models, such as LLaMa. Only thing that needs updating is the endpoint address. I have not yet tried it and seems I now misremember the name.

It sounds interesting to me. That'll make things a lot easier.

@torronen Is this the project you mentioned? https://github.com/go-skynet/LocalAI
It's written in Go.

A python project is https://github.com/lhenault/simpleAI

@sonichi Yes,  that is the one. I just remembered it was something to do with word "village" 🥇 Close enough to localAI 😆

@sonichi Yes, that is the one. I just remembered it was something to do with word "village" 🥇 Close enough to localAI 😆

I added some documentation about it: https://microsoft.github.io/FLAML/docs/Use-Cases/Auto-Generation/#api-unification and added a feature to try multiple endpoints in turn: https://microsoft.github.io/FLAML/docs/Use-Cases/Auto-Generation/#error-handling
Let me know if it works!

@sonichi Amazing, thanks!


Buy Grid Trade not filled after buy order filled

Version

v0.0.96
Description

After buy order has been filled, bot got stuck with ...you have enough BTC to sell.
To Reproduce

Random issue. I can't reproduce by will.
Expected Behaviours

All buy orders have to be processed correctly.
Screenshots

Order on Binance:

Docker log:

Trade on Binance:

Docker log:

Bot stuck:

Additional context

Unfortunately it was too late to export log as it passed more than 30 minutes already but I have at least docker log. Interesting thing is there is no event logged at 19:20:16 (18:20:16 UTC) when order has been filled.

@chrisleekr There is unused checkOrderExecutePeriod configuration parameter. Does that mean bot was able to check for orders periodically and this functionality has been removed in the past or was this planned to be implemented in the future?

@uhliksk checkOrderExecutePeriod was used before implementing WebSocket to receive order notifications.
Now it is deprecated. Should remove this variable from the configuration.
For the issue you raised, two scenarios can be guessed.

The filled order event was not received from Binance. So the bot may not record the last buy price.
The filled order event was sent from Binance, but the bot ignored processing it.

Unfortunately, it's hard to find out what happened without the log.
If happens again, try to get the log.
By the way, you can now export the log without any issue? So the fix is working?

@chrisleekr

The filled order event was not received from Binance. So the bot may not record the last buy price.
The filled order event was sent from Binance, but the bot ignored processing it.

Can we check periodically if orders are still present on Binance? Or at least when enough SYMBOL to sell is detected the option to set last buy price from last buy order (or orders up to sum of current balance) automatically will be great solution I think. Or for simplicity we can use current price as last buy price if asset is transfered to spot wallet from funding wallet or by Binance Convert. I don't use spot wallet for nothing else but bot trading so everytime there is some balance it should be traded automatically without waiting to put last buy price manually.

By the way, you can now export the log without any issue? So the fix is working?

Yes, log export is working flawlessly now.

@chrisleekr

Yes, log export is working flawlessly now.

But, not working for big logs:
{"name":"binance-api","version":"0.0.96","hostname":"94d1c8d5d371","pid":46,"gitHash":"unspecified","level":50,"err":{"message":"Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.","name":"MongoServerError","stack":"MongoServerError: Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.\n    at MessageStream.messageHandler (/srv/node_modules/mongodb/lib/cmap/connection.js:463:30)\n    at MessageStream.emit (events.js:400:28)\n    at processIncomingData (/srv/node_modules/mongodb/lib/cmap/message_stream.js:108:16)\n    at MessageStream._write (/srv/node_modules/mongodb/lib/cmap/message_stream.js:28:9)\n    at writeOrBuffer (internal/streams/writable.js:358:12)\n    at MessageStream.Writable.write (internal/streams/writable.js:303:10)\n    at Socket.ondata (internal/streams/readable.js:731:22)\n    at Socket.emit (events.js:400:28)\n    at addChunk (internal/streams/readable.js:293:12)\n    at readableAddChunk (internal/streams/readable.js:267:9)\n    at Socket.Readable.push (internal/streams/readable.js:206:10)\n    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)","code":96},"msg":"Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.","time":"2023-01-05T15:03:01.125Z","v":0}


Update Vectr to 0.1.8

Thank you for the contribution. It was merged directly as 16d83bcfa5010c7a2970071879499e28fae882d2 to keep commit history cleaner. Your contribution is still credited to you.


Random Battle: Remove Toxic from Reshiram

Seems a logic change but I will wait for @TheImmortal on this

No. Toxic is not only used on its Leftovers Ubers' set but also on its Scarf set! I would argue it's more useful in Random Battle because its attacks are weaker due to being at a lower level than everything else.


Spells are not being imported, though spell-likes seem to work.

In general, spells from Hero Lab statblocks don't seem to be importing.  I've tried several over time and none have worked.
The latest example is the statblock for "Pyromaniac Mage" and it's only CR 12:
Pyromaniac Mage      CR 12
XP 19,200
Male human evoker 13
CE Medium humanoid (human)
Init +6; Senses Perception +12
Defense
AC 19, touch 14, flat-footed 16 (+4 armor, +1 deflection, +2 Dex, +1 dodge, +1 natural)
hp 81 (13d6+33)
Fort +9, Ref +11, Will +12
Resist fire 30
Offense
Speed 40 ft.
Melee club +6/+1 (1d6)
Special Attacks intense spells (+6 damage)
Arcane School Spell-Like Abilities (CL 13th; concentration +19)
At will—elemental wall (13 rounds/day)
9/day—force missile (1d4+6)
Evoker Spells Prepared (CL 13th; concentration +19)
7th—delayed blast fireball (DC 25), statue (DC 23)
6th—greater dispel magic, elemental body III, maximized fireball (DC 21), maximized fireball (DC 21)
5th—passwall, maximized scorching ray, maximized scorching ray, telekinesis (DC 21), waves of fatigue
4th—maximized burning hands (2, DC 19), greater invisibility, phantasmal killer (DC 20), shout (DC 22), wall of fire
3rd—dispel magic, explosive runes, fireball (DC 21), flame arrow, fly, protection from energy
2nd—flaming sphere (DC 20), gust of wind (DC 20), pyrotechnics (2, DC 18), resist energy, scorching ray, web (DC 18)
1st—burning hands (DC 19), expeditious retreat, grease, mage armor, magic missile, magic missile, ray of enfeeblement (DC 17)
0 (at will)—bleed (DC 16), dancing lights, mage hand, open/close (DC 16)
Opposition Schools Divination, Enchantment
Tactics
Before Combat The wizard casts mage armor, protection from energy (fire), resist energy (fire), and statue.
During Combat The wizard casts greater invisibility, fly, and maximized fireball. He follows up with phantasmal killer, shout, and necromantic rays.
Statistics
Str 10, Dex 14, Con 14, Int 22, Wis 9, Cha 12
Base Atk +6; CMB +6; CMD 20
Feats Combat Casting, Craft Wondrous Item, Dodge, Greater Spell Focus (evocation), Improved Initiative, Iron Will, Lightning Reflexes, Maximize Spell, Point-Blank Shot, Scribe Scroll, Spell Focus (evocation)
Skills Acrobatics +2 (+6 to jump), Climb +5, Craft (alchemy) +19, Fly +18, Knowledge (arcana) +21, Knowledge (engineering) +19, Knowledge (geography) +14, Knowledge (history) +14, Knowledge (local) +14, Knowledge (planes) +19, Knowledge (religion) +14, Perception +12, Sense Motive +4, Spellcraft +19, Stealth +12, Survival +4
Languages Abyssal, Common, Draconic, Dwarven, Goblin, Ignan, Infernal
SQ arcane bond (ring of protection +1)
Combat Gear fire elemental gem, potion of cure moderate wounds (2), potion of invisibility, scroll of protection from energy (CL 4th) (2), scroll of resist energy (2), wand of burning hands (CL 5th, 20 charges), alchemist's fire (10); Other Gear club, amulet of natural armor +1, boots of striding and springing, bracers of armor +3, cloak of resistance +3, headband of vast intelligence +4, ring of protection +1, spellbook, 1,100 gp
Special Abilities
Arcane Bond (Ring of protection +1) (1/day) (Sp) Use object to cast any spell in your spellbook. Without it, Concentration required to cast spells (DC20 + spell level).
Combat Casting +4 to Concentration checks to cast while on the defensive.
Divination You must spend 2 slots to cast spells from the Divination school.
Elemental Wall (13 rounds/day) (Sp) Create wall of one chosen energy type, otherwise as wall of fire.
Enchantment You must spend 2 slots to cast spells from the Enchantment school.
Energy Resistance, Fire (30) You have the specified Energy Resistance against Fire attacks.
Evocation Evokers revel in the raw power of magic, and can use it to create and destroy with shocking ease.
Force Missile (1d4+6, 9/day) (Sp) As a standard action, magic missile strikes a foe.
Greater Spell Focus (Evocation) +1 to the Save DC of spells from one school.
Intense Spells (+6 damage) (Su) Evocation spells deal listed extra damage.
Maximize Spell All variable effects of a spell are maximized. +3 Levels.
Point-Blank Shot +1 to attack and damage rolls with ranged weapons at up to 30 feet.
Spell Focus (Evocation) Spells from one school of magic have +1 to their save DC.
These mages love to cause collateral damage with fires.
Additional Details
Base Statistics Without mage armor, protection from energy, and resist energy, the wizard's statistics are AC 15, touch 14, flat-footed 12; Immune none; Resist none.

I'm using the latest releases for FVTT, PF1e game system, and sbc module.

I've attached the text for the statblock below:
Pyromaniac Mage.txt

This is the only major issue I'm aware of.  Fixing it would greatly improve my FVTT experience and I would stop crying in my sleep.  :)

Oh, i don't want you crying in my sleep :D I'll try to replicate that.


pain with setting base url when needed access from outside by public ip and inside network by internal ip

Description

when starting configuration server listens localhost and not <IP_ADDRESS> as stating in its log, so
2022/08/31 10:29:54 cmd/web.go:217:listen() [I] [630f0df2] Listen: http://<IP_ADDRESS>:3000
2022/08/31 10:29:54 cmd/web.go:221:listen() [I] [630f0df2] AppURL(ROOT_URL): http://localhost:3000/
is just confusing. User cannot connect to server from outside to initial configuration

when setting addres from local net as AppURL e.g. <IP_ADDRESS>:3000 server actually does not listen to that address, but continues response on localhost:3000.

user cannot connect to both external and internal addresses of a server

Gitea Version
1.17.1
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
initial run via command line, next was planned to run via systemd
Database
No response

set ROOT_URL as your expected outside address

then it does not work from inside. And I cannot set that url from outside via web console because I cannot access to console from remote. | can see that server works via wget localhost:3000 but not from <IP_ADDRESS>:3000

That's another issue which has been reported in #19345

please help: what should i do when internal access URL  is http://<IP_ADDRESS>:3000/
and external is http://<IP_ADDRESS>:8000/ ?

I will try to answer some questions, and at the same time, you could understand more about your system and environment.

1. when starting configuration server listens localhost and not <IP_ADDRESS> as stating in its log, so
   2022/08/31 10:29:54 cmd/web.go:217:listen() [I] [630f0df2] Listen: http://<IP_ADDRESS>:3000
   2022/08/31 10:29:54 cmd/web.go:221:listen() [I] [630f0df2] AppURL(ROOT_URL): http://localhost:3000/
   is just confusing. User cannot connect to server from outside to initial configuration

Since Gitea shows Listen: http://<IP_ADDRESS>:3000, then it's already listening on all IP addresses.
AppURL is just for UI display and URL construction, it doesn't nothing to the Listen address.
If user can not access it, please check your firewall (nat, etc) or other rules.

2. when setting addres from local net as AppURL e.g. <IP_ADDRESS>:3000 server actually does not listen to that address, but continues response on localhost:3000.

Which address?

3. user cannot connect to both external and internal addresses of a server

If Gitea is listening on "<IP_ADDRESS>:3000", it should work. Check your firewall and environment. Plenty of users use the same setting and do not have such problem, and there doesn't seem to be a problem indeed.


[RFC] Integrate TVM into Apache MXNet

Problem Statement
Currently in MXNet we implement operator kernels in C++. Developers need to specify the detail logic of each computation, which slows down the development process. Given the fact that we’re moving forward to be numpy-compatible[1], a large amount of operators are to be implemented. Moreover, we also have various of backend to support, including CPUs and GPUs from AMD, ARM, Intel, Nvidia, etc. It requires great effort to implement efficient kernels for each of these hardwares, so as writing test cases for each of the operator+backend combination.
Proposal
Thus I would like to propose to integrate Apache TVM into Apache MXNet, so that we can leverage its ability to easily implement (in Python) high-performance operator kernels. Here’re some of the advantages,

We devised a new approach to implement MXNet kernels in Python, (see PR https://github.com/apache/incubator-mxnet/pull/15345 ). For example, to implement broadcast add, people can write pure Python,

@defop(name="vadd", target="cpu", auto_broadcast=True,
       dtype=AllTypes, ndim=list(range(6)))
def vadd(dtype, ndim):
    A = tvm.placeholder(shape=[tvm.var() for _ in range(ndim)], name='A', dtype=dtype)
    B = tvm.placeholder(shape=[tvm.var() for _ in range(ndim)], name='B', dtype=dtype)
    C = tvm.compute(shape=[tvm.var() for _ in range(ndim)],
                    lambda *index: A[index] + B[index], name='C')
    s = tvm.create_schedule(C.op)
    axes = [axis for axis in C.op.axis]
    fused = s[C].fuse(*axes)
    s[C].parallel(fused)
    return s, [A, B, C]

The code above will be compiled to binary and linked into MXNet as a regular function. Note that the same piece of compute definition can be shared across the multiple backends (cpu, gpu, etc.). As it is much more concise than that defined in C++, it can lower the bar of implementing high-performance kernels and improve the development experience. We expect people can develop more efficiently with such approach.

Operators in current TVM are already numpy-compatible, we can leverage the efforts there to help our Numpy project.

Approach

Build and link TVM dynamic library into MXNet.
Build infrastructure to write operator kernels in Python, including:

The approach for registering TVM operator into MXNet
Modify CI to enable TVM operator build, and pack the operator library together with the release binary. We can enable automatic performance tuning[2] later to further improve the performance.

Currently Apache TVM has already been integrated as a 3rdparty repository, and some of the nnvm header files are included in MXNet source code.
FAQ
Q: Does it increase the binary size of MXNet release?
A: libtvm_runtime.so is roughly 750K, it is fairly small compared to libmxnet.so (60MB for cpu, ~500MB for gpu)
Q: Are TVM operators going to replace current operators in MXNet?
A: No. It is an alternative way to write kernels. For new operators which are easy to be written in this approach, we can benefit from the advantages mentioned above.
Q: Any license problem?
A: TVM is provided under Apache 2.0 License, and it’s currently incubating at Apache Software Foundation: https://tvm.ai/2019/03/18/tvm-apache-announcement
Background of TVM
Apache TVM is an open deep learning compiler stack for CPUs, GPUs, and specialized accelerators. It aims to close the gap between the productivity-focused deep learning frameworks, and the performance- or efficiency-oriented hardware backends.

TVM has showed its ability to get decent performance not only on end2end neural network, but also on single kernels. Recently papers and benchmarks showed it can achieve better performance even comparing to vendors’ acceleration libraries: https://tvm.ai/2018/10/03/auto-opt-all
TVM supports a large number of backend, including Intel, AMD, ARM CPUs and GPUs, as well as Nvidia GPUs and FPGAs. Reusing TVM's optimized kernels could benefit MXNet's backend support a lot.
TVM provides a flexible and convenient tool to route data structure and function call between C++ and other frontend language, which would be helpful for general purposes. See https://docs.tvm.ai/dev/runtime.html#tvm-node-and-compiler-stack for reference.

Reference
[1] [RFC] Introducing NumPy-compatible coding experience into MXNet - https://github.com/apache/incubator-mxnet/issues/14253
[2] [TVM] Automatic Kernel Optimization for Deep Learning on All Hardware Platforms - https://tvm.ai/2018/10/03/auto-opt-all

Good to see we have another option to implement operators with high performance! Here's a quick question:
How to deal with TVM dependence with LLVM ? LLVM is an important code generator for many CPU backends. If we integrate TVM into MXNet, then shall we integrate LLVM into MXNet? If the answer is yes, then we have binary size issue. If not, then how to make sure TVM can works together with any host installed LLVM?

@ZhennanQin This is a good point. LLVM is required during compile time, while for binary release, only tvm runtime is needed (instead of tvm compiler plus llvm). We don't do JIT.

Oh, it's a bit different from what I thought. If we don't do jit, then tvm can't be used to implement custom op unless rebuilding MXNet.
Also, as tvm generated kernel is complied along with compling MXNet, avx512 instructions won't be generated from release version of MXNet, making integrated tvm can't provide best performance comparing with standalone tvm.
Seems jit is the solution to address above problem, then why don't we support it?

I actually like the idea. But as jit requires big surgery to MXNet's engine, I suggest to do aot first, then make another design for jit later.

@yzhliu Nice proposal. I look forward to this integration of TVM into MXNet backend. It would be great if you could also add a section in the project plan for performance comparison with existing backend, so we have a more quantitative measure.

@ZhennanQin It would be possible to allow this to be an easy plugin of MXNet. Imagine the following scenario: MXNet itself is a small library with not many operators, and can be extended by downloading extra TVM ops, specialized for different CPUs, without re-compilation.
By detecting the feature set of the CPUs used (whether supporting avx512, etc), we could instruct users to use the most efficient operator library.

@ZhennanQin btw, JIT is something requires runtime external dependency, and extra effort for compiling/caching/... We may consider it later after AOT is done :-)

dumb question: Are operators written this way available in any way to non-python languages such as lua, scala, etc?

@cjolivier01 yep! The TVM compiler generates llvm IR which are compiled as binary and loaded as a C++ module, then could be used in all frontend languages

@junrushao1994 The plugin idea looks too fantastic for me. Of course, If we can achieve that, then integrated TVM is able to provide very good performance.
Here's another question about threading management. As you may know, MXNet is using openmp as threading management protocol. Also threaded engine can create many threaded workers to execute operators in parallel. So the total thread number on threaded engine is (total_workers * OMP_NUM_THREADS). AFAIK, TVM runtime has its own threading pool, will TVM switch to use openmp as threading management protocol after integration? can we run 2 TVM generated operators in different worker at same time?

That python code looks pretty intense compared to JAX/XLA, which looks like normal python/numpy code. What’s the difference between the end results/use cases of these packages?
https://github.com/google/jax/blob/master/README.md

@cjolivier01 basically you can consider TVM serves similar functionality as XLA. JAX doesn’t seem to be relevant in this case.

@yzhliu i am not super familiar with the threading management in TVM. Could you answer @ZhennanQin’s question on your side?

@yzhliu If TVM operators have to be compiled and then would need to be recompiled with MXNet, have you thought about TVM being an "accelerator" that could be used with the Accelerator API proposal: https://cwiki.apache.org/confluence/display/MXNET/Bring+your+own+Accelerator ? Maybe using this interface we can build that "auto-download" the appropriate operator-set feature. And keep the operator libraries compiled separate from MXNet.

@ZhennanQin tvm will still be using its own thread pool, the thread pool is shared across the workers. I don't see this will be a problem as anyhow the max parallelism we can achieve is the number of physical cores.
@apeforest unfortunately I don't have the performance number for new-written operators as for now.
@samskalicky Yes this can be an option. while it's a little bit unclear to me how infer-shape/dtype/etc is registered to mxnet under the approach. Maybe we can keep in mind the use-cases when designing the interface, and port tvm as an "accelerator" later once it has been implemented.

Thanks @yzhliu for the answers. How will users control the number of threads used by TVM in MXNet? they already have a few knobs to turn, and some use cases like MMS set minimum thread pool sizes to prioritize throughput over latency. Are we overcomplicating things by adding this additional threadpool?
The question about multiple OMP libs still stands though. Will TVM compile with the same OMP as MXNet?

@yzhliu Actually this will be a performance problem. OMP threads will spin when parallel block is completed, this can help to avoid thread switch off before executing next parallel block. If a TVM operator follows with an OMP operator, then there will be 2x threads activate, half are TVM threads, and the other half are the spined OMP threads. This will cause physical core race, result in low performance onTVM operator.

@ZhennanQin Does it make sense to parallelize two CPU operators both of which eat all threads?

@junrushao1994 No, it doesn't make sense. But currently we can specify OMP_NUM_THREADS = physical_core_num / 2 for two paralleled operators to let each of them to use half physical cores.

@ZhennanQin If we could somehow instruct TVM's threading backward how many threads it would utilize, will it be something similar? Or what if we could instruct TVM to use omp?

@junrushao1994 Let me clarify my concerns for TVM thread pool.
1>
On default, threaded engine will only create single worker for normal CPU operator. At this scenario, user will specify OMP_NUM_THREADS = physical_core_num to maximize the computing capacity of CPU. A OMP operator followed by TVM operator will have omp thread spinning issue, causing TVM operator executes slowly. This is a major issue need to address because it's a typical scenario in MXNet.
2>
when user specify MXNET_CPU_WORKER_NTHREADS=2, then 2 CPU workers are activated. It's possible to run 2 independent CPU operators in parallel. For this case, user can set OMP_NUM_THREADS = physical_core_num / 2  to let each worker to use half physical cores. I want to know if TVM can do the same thing.
3>
Even if we resolved above problem, there's a thread switching penalty when maintaining  2 different threading pool. The penalty can't be ignored for latency sensitive task(e.g. BS=1 inference).
From my perspective, mixed using 2 kinds of thread pool isn't a good idea. It's hard to manage thread binding for multi-workers, and will have thread switching overhead between different thread pools. It's better to use same threading management as MXNet.

Re: Threadpool
Technically it is easy to drop in openmp as a threadpool backend of TVM. It is just not the default one. Historically tvm defaulted to openmp then evolved its own version of thread pool for better perf. The current thread pool already has accepts the OMP env variable, making it consistent with the omp settings.
Most of the operators touched so far are single threaded so won’t be affected by the discussion.
Given the current discussion, we could have the runtime in Mxnet default to omp if we see it is necessary, for end to end inference on Embedded devices like android, I would still recommend to use tvm’s Threadpool

We will never want to use two thread pools simultaneously. We should use the one which gives best-to-date performance. Technically it is not difficult for TVM to switch among different thread pool implementations. So we just need to benchmark them and decide which one to use. As @tqchen said, our benchmarking results show that the current TVM thread pool is better than the ones using OpenMP APIs. Please check out this paper for details.

@tqchen I agree with you that this is a technical issue and shouldn't block this PFC. As thread management should be considered from high level design, I bring this topic here.

(1) and (2) likely won't be an issue because: TVM's thread number configurations and core settings are consistent with OMP(OMP_NUM_THREADS).

OMP_NUM_THREADS can't help for either of them. For (1), consider the executing sequence: OMP operator -> TVM operator. When OMP operator finished, OMP threads won't sleep immediately, but spin for a while. Then when TVM operator starts, there will be 2x threads active, making TVM threads can't fully occupy all physical cores. You can check OMP_WAIT_POLICY for details.
For (2), it's about how to manage threads. OMP manages threads per engine worker. If TVM uses single global thread pool, then it will has problem for multi-worker usage.
Another thing worth to mention is, as OMP is the MXNet thread management protocol, user may use many OMP environment to change their thread policy. For example, KMP_AFFINITY and OMP_WAIT_POLICY  are used frequently. I doubt if TVM threads pool can support them.
It's good to know TVM can support OMP as a threadpool backend. Then let's performance data make the decision if integrated TVM will switch to use OMP. Also thanks @yidawang for the explanation.

@ZhennanQin I guess we should first make sure that we don't use more than one thread pool implementation in the runtime. Also, KMP_AFFINITY only works for Intel CPUs, we have many more other platforms.

@yidawang If we all agree that we should avoid using more than one thread pool implementation in the runtime, then I guess OMP is the only choice, is that true?

@yidawang If we all agree that we should avoid using more than one thread pool implementation in the runtime, then I guess OMP is the only choice, is that true?

Sorry for the late reply, traveling these days. In this case, I think we should also benchmark to compare the performance of between different scenarios to decide. Theoretically, if a model runs both TVM ops and original MXNet ops on CPUs, I agree that using OpenMP may be a short term solution.

Can it fuse different operators in the graph? I think this would be badly needed for us.

@larroy In this proposal it cannot. It requires more deeply integrated with tvm/relay to run graph-level optimization, which @junrushao1994 is planning.

Could we add more information on how will it impact the build systems and dependencies? While integrating TVM is exciting, I think is fair to gauge the increase of complexity, build and release implications. Would it be possible to load operators compiled with TVM dynamically? This would open a path to modularize MXNet and reduce the library size / target inference only version in a more streamlined way than with amalgamation hacks.

Sure,

Build system

Download pre-built LLVM
Compile 3rdParty/tvm
use compiled tvm to compile tvm operator kernels and generate libtvmop.so

Runtime

libtvm_runtime.so linked with libmxnet.so
MXNet loads libtvmop.so during runtime (sort of dynamic loading)

While for "fully" supported dynamic loading, the major problem is how to register op metadata, basically InferShape/InferDtype/etc. It'll be great if the accelerator API solves the problem

Thanks everyone comments on the RFC. To summarize,

We agree to integrate TVM into MXNet
The first step is to use TVM write operator kernels, do ahead-of-time compile.
It might have performance issue if we mix TVM's threadpool and openmp together in MXNet. We need to benchmark and if it does appear to be a problem, we can switch to use openmp in TVM as well.
Later: whether and how to integrate the approach with Accelerator API so that tvm operators can be dynamically loaded.

Would it be possible to keep it decoupled, say the infrastructure to load the tvm operator is included in MXNet but we don't need to couple with LLVM and TVM, (even though we are using it through NNVM). I think irrespective of the technical merit and benefits of TVM which are undeniable, less coupling makes software easier to maintain. @samskalicky knows more about Accelerator APIs.

If we're doing this on a per-operator basis and treating TVM operators like CPU operators then maybe we should be considering something similar to the dynamic customOp proposal instead of the accelerator API.
@yzhliu how are you thinking about presenting TVM ops here? will they run on CPU or GPU (or both)?


soci update to 4.0.0

Library name: soci
New version number: 4.0.0
https://github.com/SOCI/soci/releases/tag/4.0.0

Thanks for posting this issue. This issue has been fixed. So I am closing this issue for now.


Add support for GoImports

What does this PR do?
This PR adds support for GoImports.
Why is it important?
It allows GoImports targets to be used by all projects, this target fixes some lint issues regarding imports ordering.
Checklist

[x] My code follows the style guidelines of this project
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have added an entry in CHANGELOG.md

## Author's Checklist
## Related issues

:broken_heart: Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-08-10T08:16:37.067+0000

Duration: 2 min 12 sec

Steps errors
Expand to view the steps failures

Check out from version control

Took 0 min 9 sec . View more details here
Description: [2022-08-10T08:18:05.564Z] The recommended git tool is: NONE
[2022-08-10T08:18:12.488Z] using crede

Check out from version control

Took 0 min 1 sec . View more details here
Description: [2022-08-10T08:18:24.693Z] The recommended git tool is: NONE
[2022-08-10T08:18:24.707Z] using crede

Check out from version control

Took 0 min 2 sec . View more details here
Description: [2022-08-10T08:18:36.148Z] The recommended git tool is: NONE
[2022-08-10T08:18:36.161Z] using crede

:robot: GitHub comments
To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

/test


API doc introduction confuse

Version
README
Test Case
Introduction Code
Steps to reproduce
I try to make the practice according to the doc code, but when coder the following code, I get a error message.(it's in Adding More UI h2)
// Make a new component to render inside of Inbox
const Message = React.createClass({
  render() {
    return
Message

  }
})

const Inbox = React.createClass({
  render() {
    return (

Inbox

        {/* Render the child route component */}
        {this.props.children}

    )
  }
})

render((
  <Router history={history}>
    <Route path="/" component={App}>
      <IndexRoute component={Home} />
      <Route path="about" component={About} />
      <Route path="inbox" component={Inbox}>
        {/* add some nested routes where we want the UI to nest */}
        {/* render the stats page when at `/inbox` */}
        <IndexRoute component={InboxStats}/>
        {/* render the message component at /inbox/messages/123 */}
        <Route path="messages/:id" component={Message} />
      </Route>
    </Route>
  </Router>
), document.body)

Error Msg
Uncaught ReferenceError: InboxStats is not defined

Expected Behavior
no error message, and the application is run well.
Finally
I'think should declaration the InboxStats component. like this:
const InboxStats = React.createClass({
  render() {
    return (

        some DOM code

    )
  }
})

thanks!

Thanks for your question!
We want to make sure that the GitHub issue tracker remains the best place to track bug reports and feature requests that affect the development of React Router, and per the issue template, we are not handling support requests here.
If you have a question or otherwise need help, please post on Stack Overflow with the #react-router tag at https://stackoverflow.com/questions/ask?tags=react-router, or drop in on the #react-router room on Reactiflux at https://discord.gg/0ZcbPKXt5bYaNQ46.

Oh, I know! I'm feel sorry about my issue.


replace load_current_value with hab status

Currently the hab_service resource uses the Habitat supervisor's API to get the service state. In Habitat 0.22.0, a new hab sup status command that will report the status of the given service names. At this time, the output is not super friendly for machine consumption. Chiefly, there are  two problems:

The output uses color escape sequences that can make parsing problematic
The output is an English sentence that means we have to use a regular expression to get the data we want

It is preferable that the output report the service name / fully qualified Habitat package identifier with its current state as two separate fields in a string that has a stable API so we don't have to update the load_current_value method again later.
Once this issue is resolved, we should use hab sup status to get the status of services in the supervisor.

I'd actually prefer to stick to the endpoint. Parsing out shellouts is problematic and probably slower.