Scanner rewrite

It's not finished yet, I have a bunch of stuff commented out as it hasn't been updated yet, but this is the direction I'm heading.  Doing a force-scan adds 4-7 seconds (depending on if it's local or on the other side of a router) to the scan time (which I have defaulting to 18 seconds) using 150 parallel requests and a 3-second connect timeout.  Matching scan-detected devices to IDs still needs work as mentioned in #172 .  Closes #159 and closes #172 and closes #232.

Well, I had it all nice and neat, but after I added broadcast-discovered device polling back in it kinda spaghetti'd.  I may make some classes to better handle how the different socket types process stuff.

Parallel-socket theory of operation:
There are 3 different types of sockets: force-scan connection attempts, broadcast-discovered status polling, and the UDP broadcast receiver.  All 3 are stuffed into either the "waiting to write" list or the "waiting to read" list, both of which are fed into select.select().  select() then returns all the sockets which are writable or readable.  I currently have a fixed sleep time (200ms during the force-scan phase and 100ms during the broadcast-discovery phase) before the select() call with select() itself getting 0 in order to keep the spinny thing spinning at a constant rate.
When a force-scan connection is first opened, it is dropped into the "write" list.  select() then returns it when it either connects, or the connection was actively refused.  (Timeouts are handled elsewhere.)  If the connection succeeds, it is marked as discovered, a DP_QUERY is sent, and the sock is moved into the "read" list.  If the connection failed then it tries to determine if it was refused, or if it connected but then dropped.  If it was dropped then it is retried exactly 1 more time.  If it was refused, or if the retry fails, it is closed and ignored.  When the DP_QUERY response is received, it is decoded and saved if it has payload data.  It is then left open in read mode to see if the device broadcasts unsolicited status updates; this is to try and detect which version it is as the DP_REQUEST response does not include it.
When a broadcast is received with polling enabled, a connection is created and it is placed into the "write" list just like force-scan sockets.  A connection failure is retried 3 times.  Once connected, a status request is sent and the socket is moved into the "read" list.  When a status response is received or it exhausts the failure retries, the device info is printed and the socket is closed.
The UDP broadcast receiver is always in the "read" list.  When a broadcast is received, a status polling socket is opened if polling is enabled or the device details are immediately displayed if not.
Currently, force-scanned device details are not displayed until the very end after all scanning has finished in hopes that an unsolicited status update is received.  I may change this so it displays earlier and then updates should it get one.

Great research here @uzlonewolf I can't wait to try it myself.  Also, I'm traveling this week without good acces so I may be a bit more silent than normal.

The scanner branch may be out of sync.  Testing this draft, it is fast, but also getting errors on all devices:
python3 -m tinytuya scan

Polling <IP_ADDRESS> Failed: Send Poll failed

It is.  I've basically rewritten it yet again since that last commit but haven't pushed it yet as it's not done.  I got passive broadcast listening working again so I'll go ahead and push it, but force scanning isn't going to work yet.
New features:
Scans end after a certain amount of time instead of a count of packets
If you pass forceips=[...] or forceids=[...] the scan will terminate after all of the passed IPs or GW IDs are found even if there is time left
If a device is found at the last second, the scan time is extended by the lesser of "the time it takes to poll the device" or (connect_timeout * retries)
getmac is used if installed, but it is no longer required
Force scanning features which are not working yet:
When forcescan=True is passed, the netifaces module is used to get a list of every IP address/network attached to the machine and scans all of them.  If netifaces is not available it falls back to the original single-IP getmyIP() function.
When forcescan=[...] then all listed networks are scanned.
Because Layer 2 information is not available, force scanned devices are fingerprinted to try and determine the device version and gwId:
When sent an unencrypted status request,
v3.1 devices will return the status
v3.2 devices will ???
v3.3 devices will return an encrypted rejection message which we can then brute-force to find the key
v3.4 devices will slam the door in our face by dropping the connection
@jasonacox What do v3.1 devices do when sent a CONTROL command with the incorrect local key?  Do they send a rejection message or do they just ignore the request completely?

With 10 devices in my devices.json file (all online), python3 -m tinytuya devices now takes ~6 seconds and snapshot/json are under 2 seconds.  If there are offline devices then devices takes ~18 seconds with snapshot/json remaining sub-2 seconds due to them skipping devices without IPs.

@uzlonewolf - here is 3.1 device command behavior using a bad key. Note real lkey = '78c2e271b3dcfae4' but I couldn't get anything out of the payload using the bogus or real key.
Code
import tinytuya

tinytuya.set_debug(True)

d = tinytuya.OutletDevice('03200160ecfabc8d9700', '<IP_ADDRESS>', '1234567890abcdef', version=3.1)

print(" > Fetch Status < ")
print(d.status())

print(" > Turn on < ")
d.turn_on()

Output
DEBUG:TinyTuya [1.7.0]

 > Fetch Status <
DEBUG:status() entry (dev_type is default)
DEBUG:building command 10 payload=b'{"gwId":"03200160ecfabc8d9700","devId":"03200160ecfabc8d9700","uid":"03200160ecfabc8d9700","t":"1663207188"}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000010000000a000000747b2267774964223a223033323030313630656366616263386439373030222c226465764964223a223033323030313630656366616263386439373030222c22756964223a223033323030313630656366616263386439373030222c2274223a2231363633323037313838227d4db799a10000aa55'
DEBUG:received data=b'000055aa000000010000000a0000005c000000007b226465764964223a223033323030313630656366616263386439373030222c22647073223a7b2231223a747275652c2232223a302c2234223a3136312c2235223a3136382c2236223a313139347d7dafacce430000aa55'
DEBUG:received message=TuyaMessage(seqno=1, cmd=10, retcode=(0,), payload=b'{"devId":"03200160ecfabc8d9700","dps":{"1":true,"2":0,"4":161,"5":168,"6":1194}}', crc=2947337795, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=1, cmd=10, retcode=(0,), payload=b'{"devId":"03200160ecfabc8d9700","dps":{"1":true,"2":0,"4":161,"5":168,"6":1194}}', crc=2947337795, crc_good=True)
DEBUG:decode payload=b'{"devId":"03200160ecfabc8d9700","dps":{"1":true,"2":0,"4":161,"5":168,"6":1194}}'
DEBUG:decoded results='{"devId":"03200160ecfabc8d9700","dps":{"1":true,"2":0,"4":161,"5":168,"6":1194}}'
DEBUG:status() received data={'devId': '03200160ecfabc8d9700', 'dps': {'1': True, '2': 0, '4': 161, '5': 168, '6': 1194}}
{'devId': '03200160ecfabc8d9700', 'dps': {'1': True, '2': 0, '4': 161, '5': 168, '6': 1194}}
 > Turn on <
DEBUG:building command 7 payload=b'{"devId":"03200160ecfabc8d9700","uid":"03200160ecfabc8d9700","t":"1663207188","dps":{"1":true}}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa00000002000000070000009b332e31663535313137396430306432636330336d337664587a4f6b445058324a32686e3843715858616369454b68466834436b45654e56534d5859325755436d4a474e612b486944512b2f792b6c6c68682b726b2b4d6b41327347375648654f4941767834566b487558456f454435434a47337131533377643358594e466d4265322f5568575250357a4a656767596d6c623450b8b89f0000aa55'
DEBUG:received data=b'000055aa00000000000000080000008b00000000332e3138343032376239396630333762623037702f726c75342b76564c422b5450505234462b36557333397a387a52686974705550434d4f6945732f753831755833306843712b362b6475636a57494178336e66462b3064426e524d51384b464355645941436953446b7946586f4977496b456a68534d4c4f37544865383dc3a8ce650000aa55'
DEBUG:received message=TuyaMessage(seqno=0, cmd=8, retcode=(0,), payload=b'3.184027b99f037bb07p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u81uX30hCq+6+ducjWIAx3nfF+0dBnRMQ8KFCUdYACiSDkyFXoIwIkEjhSMLO7THe8=', crc=3282620005, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=0, cmd=8, retcode=(0,), payload=b'3.184027b99f037bb07p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u81uX30hCq+6+ducjWIAx3nfF+0dBnRMQ8KFCUdYACiSDkyFXoIwIkEjhSMLO7THe8=', crc=3282620005, crc_good=True)
DEBUG:decode payload=b'3.184027b99f037bb07p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u81uX30hCq+6+ducjWIAx3nfF+0dBnRMQ8KFCUdYACiSDkyFXoIwIkEjhSMLO7THe8='
DEBUG:error unpacking or decoding tuya JSON payload
Traceback (most recent call last):
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 779, in _send_receive
    result = self._decode_payload(msg.payload)
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 820, in _decode_payload
    payload = cipher.decrypt(payload[16:])
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 239, in decrypt
    raw = self._unpad(raw, verify_padding)
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 259, in _unpad
    raise ValueError("invalid padding length byte")
ValueError: invalid padding length byte
DEBUG:ERROR Unexpected Payload from Device - 904 - payload: null
DEBUG:set_status received data={'Error': 'Unexpected Payload from Device', 'Err': '904', 'Payload': None}

Hmm, that response is command 8 (STATUS) despite the request being command 7 (CONTROL).  It also seems to decode properly with the real key to b'{"devId":"03200160ecfabc8d9700","dps":{"4":162},"t":1663207186,"s":84738}' so I suspect it was a coincidental async status update.
Do they respond to status requests with no payload at all?  Something like print( d._send_receive( MessagePayload(tinytuya.DP_QUERY, b'') ) ) ?
I really need to find a 3.1 device for myself....

@uzlonewolf - If you will allow me, email me your address and I'll send you a 3.1 plug.
Here is your test with good and bad lkey:
>>> d = tinytuya.OutletDevice('03200160ecfabc8d9700', '<IP_ADDRESS>', '78c2e271b3dcfae4', version=3.1)
>>> print( d._send_receive( tinytuya.MessagePayload(tinytuya.DP_QUERY, b'') ) )
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000010000000a00000008354862b10000aa55'
DEBUG:received data=b'000055aa000000010000000a0000001d000000016461746120666f726d6174206572726f7271bf61a40000aa55'
DEBUG:received message=TuyaMessage(seqno=1, cmd=10, retcode=(1,), payload=b'data format error', crc=1908367780, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=1, cmd=10, retcode=(1,), payload=b'data format error', crc=1908367780, crc_good=True)
DEBUG:decode payload=b'data format error'
DEBUG:Unexpected payload=b'data format error'
DEBUG:ERROR Unexpected Payload from Device - 904 - payload: ""

>>> d = tinytuya.OutletDevice('03200160ecfabc8d9700', '<IP_ADDRESS>', '1234567890abcdef', version=3.1)
>>> print( d._send_receive( tinytuya.MessagePayload(tinytuya.DP_QUERY, b'') ) )
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000010000000a00000008354862b10000aa55'
DEBUG:received data=b'000055aa000000010000000a0000001d000000016461746120666f726d6174206572726f7271bf61a40000aa55'
DEBUG:received message=TuyaMessage(seqno=1, cmd=10, retcode=(1,), payload=b'data format error', crc=1908367780, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=1, cmd=10, retcode=(1,), payload=b'data format error', crc=1908367780, crc_good=True)
DEBUG:decode payload=b'data format error'
DEBUG:Unexpected payload=b'data format error'
DEBUG:ERROR Unexpected Payload from Device - 904 - payload: ""

You are right... It seems like sending any commend results in a null cmd 7 then a cmd 8 response (this with valid lkey).
DEBUG:building command 7 payload=b'{"devId":"03200160ecfabc8d9700","uid":"03200160ecfabc8d9700","t":"1663212818","dps":{"1":true}}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa00000002000000070000009b332e3137303538653333666338656336393461702f726c75342b76564c422b5450505234462b36557333397a387a52686974705550434d4f6945732f752f5042454c6357776a417463575249684c745661624d317047417936327278466b5a7158617669365247727047374f4d5963764f76317457724e446d3336473947593455586979564a477641446d6e70366636645363f4f1925a0000aa55'
DEBUG:received data=b'000055aa00000002000000070000000c0000000018cfc5da0000aa55'
DEBUG:received null payload (TuyaMessage(seqno=2, cmd=7, retcode=(0,), payload=b'', crc=416269786, crc_good=True)), fetch new one - retry 0 / 5
DEBUG:received data=b'000055aa00000000000000080000008b00000000332e3132366633613532613762636539393838702f726c75342b76564c422b5450505234462b36557333397a387a52686974705550434d4f6945732f752f3770525844687a4236775550712f6b5572643136684b5833634d336e526b6267715372736f526d6e6b6a5a5474515a665a5747614f496979664b6f546d4c6e383da0e8c8fd0000aa55'
DEBUG:received message=TuyaMessage(seqno=0, cmd=8, retcode=(0,), payload=b'3.126f3a52a7bce9988p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u/7pRXDhzB6wUPq/kUrd16hKX3cM3nRkbgqSrsoRmnkjZTtQZfZWGaOIiyfKoTmLn8=', crc=2699610365, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=0, cmd=8, retcode=(0,), payload=b'3.126f3a52a7bce9988p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u/7pRXDhzB6wUPq/kUrd16hKX3cM3nRkbgqSrsoRmnkjZTtQZfZWGaOIiyfKoTmLn8=', crc=2699610365, crc_good=True)
DEBUG:decode payload=b'3.126f3a52a7bce9988p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u/7pRXDhzB6wUPq/kUrd16hKX3cM3nRkbgqSrsoRmnkjZTtQZfZWGaOIiyfKoTmLn8='
DEBUG:decoded results='{"devId":"03200160ecfabc8d9700","dps":{"1":true},"t":1663212814,"s":86361}'
DEBUG:set_status received data={'devId': '03200160ecfabc8d9700', 'dps': {'1': True}, 't':<PHONE_NUMBER>, 's': 86361}

With bad lkey - it basically ignores the command 7 and just sends back an update as you noticed.
>>> d.turn_on()
DEBUG:building command 7 payload=b'{"devId":"03200160ecfabc8d9700","uid":"03200160ecfabc8d9700","t":"1663213360","dps":{"1":true}}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa00000003000000070000009b332e31366134306637616630333233623036316d337664587a4f6b445058324a32686e3843715858616369454b68466834436b45654e56534d5859325755436d4a474e612b486944512b2f792b6c6c68682b726b2b4d6b41327347375648654f4941767834566b486d5951474c4647526a506b4c366e4b63706b6c475a746d4265322f5568575250357a4a656767596d6c6234fdc289990000aa55'
DEBUG:received data=b'000055aa00000000000000080000008b00000000332e3139633837356534343230396631383632702f726c75342b76564c422b5450505234462b36557333397a387a52686974705550434d4f6945732f752b4e3234466d6e576f36627155785a307976686c7872517032594252714b4d3973654c646e72614d6c38554d6148624e7a7273497a2b31455449616f4d6b44776b3de51bbc480000aa55'
DEBUG:received message=TuyaMessage(seqno=0, cmd=8, retcode=(0,), payload=b'3.19c875e44209f1862p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u+N24FmnWo6bqUxZ0yvhlxrQp2YBRqKM9seLdnraMl8UMaHbNzrsIz+1ETIaoMkDwk=', crc=3843800136, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=0, cmd=8, retcode=(0,), payload=b'3.19c875e44209f1862p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u+N24FmnWo6bqUxZ0yvhlxrQp2YBRqKM9seLdnraMl8UMaHbNzrsIz+1ETIaoMkDwk=', crc=3843800136, crc_good=True)
DEBUG:decode payload=b'3.19c875e44209f1862p/rlu4+vVLB+TPPR4F+6Us39z8zRhitpUPCMOiEs/u+N24FmnWo6bqUxZ0yvhlxrQp2YBRqKM9seLdnraMl8UMaHbNzrsIz+1ETIaoMkDwk='
DEBUG:error unpacking or decoding tuya JSON payload
Traceback (most recent call last):
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 779, in _send_receive
    result = self._decode_payload(msg.payload)
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 820, in _decode_payload
    payload = cipher.decrypt(payload[16:])
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 239, in decrypt
    raw = self._unpad(raw, verify_padding)
  File "/Users/jason/Code/tinytuya/sandbox/tinytuya/core.py", line 259, in _unpad
    raise ValueError("invalid padding length byte")
ValueError: invalid padding length byte
DEBUG:ERROR Unexpected Payload from Device - 904 - payload: null
DEBUG:set_status received data={'Error': 'Unexpected Payload from Device', 'Err': '904', 'Payload': None}

I'll run a loop to ensure I'm not just missing a cmd 7 error response...

I actually went hunting for some right after I made that post, so hopefully I have some on the way.  If they show up without v3.1 on them I'll take you up on that offer.  I also have some coming which I hope have v3.2 on them; if they do I wouldn't mind sending you a couple if you want.
Thanks for running those commands for me.  Looks like I'll at least be able to determine the version, and possibly get the gwId if they happen to send a status update while the connection is open.

Force scanning now (mostly) works.
Device identification for v3.1 devices is not fully implemented yet.
For v3.3 devices it provokes an encrypted error message and then starts trying to decrypt that response using known keys until it finds one which works.
For v3.4 devices it starts sending the device commands encrypted with each key until it gets a response (which means it found the correct one).
To run a force scan:
python3 -m tinytuya -h scan -force scans every network attached to the machine if the netifaces package is installed, or just the default network if it is not.
python3 -m tinytuya -h scan -force <IP_ADDRESS>/24 scans just <IP_ADDRESS>/24
python3 -m tinytuya -h scan -force <IP_ADDRESS>/24 <IP_ADDRESS>/24 <IP_ADDRESS>/24 scans <IP_ADDRESS>/24, <IP_ADDRESS>/24, and <IP_ADDRESS>/24.
All force scans also listen for broadcasts in addition to the force-scanned networks.

This is fantastic @uzlonewolf !   The speed improvement is absolutely incredible and I love the -force options!

python3 -m tinytuya scan - I don't have 3.2 devices (still need to find one) but do have 3.1, 3.3 and 3.4 and all are showing up and polling in the scan.
python3 -m tinytuya scan force - I get "Starting Scan for network <IP_ADDRESS>/8" so it is looking only at localnet instead of my WLAN (on macos) and I sure didn't wait for the /8 :-) - for machines with multiple interfaces I wonder if we should prompt for selection?
python3 -m tinytuya scan -force <IP_ADDRESS>/24  - I got a great scan but also seems to consistently mis-identify one 3.3 device as 3.4 (also picked up broadcast):

Unknown v3.4 Device   Product ID = ?  [Failed to Force-Scan]:
    Address = <IP_ADDRESS>,  Device ID = , Local Key = ,  Version = 3.4, MAC =
    Polling <IP_ADDRESS> Failed: Device stopped responding before key was found
Chandelier   Product ID = qcgkaqmaivuwfwz4  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = x, Local Key = x,  Version = 3.3, MAC = x
    Status: {'1': False, '3': 71, '101': False}
Scanned in 18.364540815353394

Scan Complete!  Found 43 devices.
Broadcasted: 25
Force-Scanned: 18  - Matched GWID: 0 Matched Key: 17 Unmatched: 1
Force-Scan Errors: Connection Errors: 1 Version Detect Failed: 1

Can you change scanner.py line 940 and change debug_ips = [] to debug_ips = ["<IP_ADDRESS>"] and let me know what it's saying about that mis-identifying device?

I haven't pushed it yet but I added a filter for 127...* and made it prompt "Y/n" for each network when verbose=True.

Nice!  Love it!
On the debug request -  Using debug_ips = ['<IP_ADDRESS>']   I ran it multiple times and the output varies a bit, but this seems to capture most of it.  Debug doesn't seem to be working?
    Running Scan...
  Starting Scan for network <IP_ADDRESS>/24
Connecting to debug ip <IP_ADDRESS>
 Force-Scan Found Device <IP_ADDRESS>
Chandelier Upstairs   Product ID = keyefm5gggjg4qch  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = x, Local Key = x,  Version = 3.3, MAC = bc:dd:c2:59:44:ea
    Status: {'1': False, '101': 0}

....[clip - occasionally a device would fail]....

Stand Lamp by Fireplace   Product ID = AiHXxAyyn7eAkLQY  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = x, Local Key = x,  Version = 3.3, MAC = 24:62:ab:3c:56:31
    Polling <IP_ADDRESS> Failed: No response
Debug sock <IP_ADDRESS> socket send failed
Connecting to debug ip <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS><PHONE_NUMBER>.853455<PHONE_NUMBER>.853657

....[clip]....

Unknown v3.1 Device   Product ID = ?  [Failed to Force-Scan]:
    Address = <IP_ADDRESS>,  Device ID = , Local Key = ,  Version = 3.1, MAC =
Debug sock <IP_ADDRESS> socket send failed
Connecting to debug ip <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS><PHONE_NUMBER>.9384892<PHONE_NUMBER>.9387279

....[clip]....

Minecraft1 Server   Product ID = keym9qkuywghyrvs  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = x, Local Key = x,  Version = 3.3, MAC = 68:57:2d:fd:94:a0
    Status: {'1': True, '9': 0, '17': 89, '18': 1466, '19': 1790, '20': 1212, '21': 1, '22': 1123, '23': 6986, '24': 7260, '25': 4960, '26': 0}
closed thrice: <IP_ADDRESS>
keygen gave: <tinytuya.scanner.KeyObj object at 0x103391210> <IP_ADDRESS>
Connecting to debug ip <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS><PHONE_NUMBER>.0284312<PHONE_NUMBER>.0287411
v3.4 trying key <IP_ADDRESS> b'377d01b7c86e970c'
v3.4 session key neg start, debug ip <IP_ADDRESS>

....[clip]....

Unknown v3.4 Device   Product ID = ?  [Failed to Force-Scan]:
    Address = <IP_ADDRESS>,  Device ID = , Local Key = ,  Version = 3.4, MAC =
    Polling <IP_ADDRESS> Failed: Device stopped responding before key was found
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.237886
SmartBulb   Product ID = keycuag84ttsx3fm  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = x, Local Key = x,  Version = 3.3, MAC = b8:f0:09:01:3c:c3
    Status: {'20': True, '21': 'white', '22': 1000, '24': '000003e803e8', '25': '07464602000003e803e800000000464602007803e803e80000000046460200f003e803e800000000464602003d03e803e80000000046460200ae03e803e800000000464602011303e803e800000000', '26': 0}
Connecting to debug ip <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.266581
WD New timeo:<PHONE_NUMBER>.3685791
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.3685791
Chandelier   Product ID = qcgkaqmaivuwfwz4  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = 00712485d8f15bc9d55b, Local Key = x,  Version = 3.3, MAC = d8:f1:5b:c9:d5:5b
    Status: {'1': False, '3': 71, '101': False}
Scanned in 18.056277990341187

Minor add - we should add a check for no netblocks on -force and either disable force mode or exit cleanly.
Scan network <IP_ADDRESS>/24 from interface en0? (Y/n): n
Traceback (most recent call last):
  File "/Users/jason/Code/sandbox/tinytuya/tinytuya/scanner.py", line 1011, in devices
    networks.append( ip )
AttributeError: 'dict_keys' object has no attribute 'append'

Hmm, that debug output is interesting. It is working, it just doesn't print much for individual devices (your screen would be absolutely flooded if it was enabled for every device though).
It seems that particular 3.3 device closes the connection if it receives a payload-less DP_QUERY packet instead of emitting an error, and then stops responding completely when fed a bunch of v3.4 requests.  I'll tweak the brute force trying a bit to see if I can detect that.
This also reminds me, I need to add get_mac_address() back in for failed devices.  I had removed it because calling it takes 100ms and results in a UDP packet getting sent to port 55555.

In case this help - I tried to separate the debug thread:
python3 -m tinytuya scan -force <IP_ADDRESS>/24 -nocolor | egrep "<IP_ADDRESS>|Chandelier" | grep -v Resetting
Connecting to debug ip <IP_ADDRESS>
 Force-Scan Found Device <IP_ADDRESS>
Debug sock <IP_ADDRESS> socket send failed
Connecting to debug ip <IP_ADDRESS>
Chandelier Upstairs   Product ID = ?  [Failed to Force-Scan]:
Debug sock <IP_ADDRESS> socket send failed
Connecting to debug ip <IP_ADDRESS>
Chandelier Upstairs   Product ID = keyefm5gggjg4qch  [Valid Broadcast]:
closed thrice: <IP_ADDRESS>
keygen gave: <tinytuya.scanner.KeyObj object at 0xb5b245e0> <IP_ADDRESS>
Connecting to debug ip <IP_ADDRESS>
v3.4 trying key <IP_ADDRESS> b'9b7eeb55d0c7819e'
v3.4 session key neg start, debug ip <IP_ADDRESS>
Connecting to debug ip <IP_ADDRESS>
trying next key <IP_ADDRESS> 9541e603af7266a4
Connecting to debug ip <IP_ADDRESS>
v3.4 trying key <IP_ADDRESS> b'9541e603af7266a4'
v3.4 session key neg start, debug ip <IP_ADDRESS>
Connecting to debug ip <IP_ADDRESS>
Debug sock <IP_ADDRESS> failed!
Debug sock <IP_ADDRESS> failed!
Timeout for debug ip <IP_ADDRESS> - reconnecting, retries 2
Connecting to debug ip <IP_ADDRESS>
Debug sock <IP_ADDRESS> failed!
Debug sock <IP_ADDRESS> failed!
Timeout for debug ip <IP_ADDRESS> - reconnecting, retries 1
Connecting to debug ip <IP_ADDRESS>
Debug sock <IP_ADDRESS> failed!
failed 1 <IP_ADDRESS> 104 104
Debug sock <IP_ADDRESS> failed!
Final timeout for debug ip <IP_ADDRESS> aborting
Chandelier   Product ID = qcgkaqmaivuwfwz4  [Valid Broadcast]:
    Address = <IP_ADDRESS>,  Device ID = 00712485d8f15bc9d55b, Local Key = x,  Version = 3.3, MAC = d8:f1:5b:c9:d5:5b
    Polling <IP_ADDRESS> Failed: Connect Failed
Connecting to debug ip <IP_ADDRESS>
trying next key <IP_ADDRESS> b980a651f7b76a13
Connecting to debug ip <IP_ADDRESS>
v3.4 trying key <IP_ADDRESS> b'b980a651f7b76a13'
v3.4 session key neg start, debug ip <IP_ADDRESS>
    Address = <IP_ADDRESS>,  Device ID = , Local Key = ,  Version = 3.4, MAC =
    Polling <IP_ADDRESS> Failed: Device stopped responding before key was found

That does help - it looks like the force-scan and broadcast-poll are fighting for control over the device.  For now I'll make it so broadcasts are ignored when a device is being force-scanned, but the final code will do the opposite and abort the force-scan (which is most likely attempting to brute-force the key) if a broadcast is received.

Wait, that device doesn't happen to be a device22 does it?  Those use CONTROL_NEW instead of DP_QUERY which would explain it not responding to DP_QUERY.
Also, slight change of plans.  Instead of ignoring broadcasts I added a -no-broadcasts flag and went ahead and made the broadcast processing abort a force-scan.

Ok, here is some more info... This is not a device22, here is the debug output from a status():
DEBUG:TinyTuya [1.7.0]

DEBUG:status() entry (dev_type is default)
DEBUG:building command 10 payload=b'{"gwId":"00712485d8f15bc9d55b","devId":"00712485d8f15bc9d55b","uid":"00712485d8f15bc9d55b","t":"1663477738"}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000010000000a000000787cc7957550f90d601cab003851e0e6e549b9463a65d3c914741b032258e4fd363312cd3bdce296e87c2d10abea1e9e158ff708776dd328a4ed750c5be5ad406661977e0d5adea2884bd304d1211fc3680597285f950ce9e15997fe05db97c4290943d12e664685f8e879590f84481a38022148f80000aa55'
DEBUG:received data=b'000055aa000000010000000a0000005c0000000067e10dcf3dcd76635cd90aec077455252bbbb850e42164957f260914aa7e5a5dc4c242078f83b4e4731137d9af3c8a7cb308798bef2dc865b1e396352a8db9e7ebd5e344ac68d7f4610fcb176201ca813dc124840000aa55'
DEBUG:received message=TuyaMessage(seqno=1, cmd=10, retcode=0, payload=b'g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]\xc4\xc2B\x07\x8f\x83\xb4\xe4s\x117\xd9\xaf<\x8a|\xb3\x08y\x8b\xef-\xc8e\xb1\xe3\x965*\x8d\xb9\xe7\xeb\xd5\xe3D\xach\xd7\xf4a\x0f\xcb\x17b\x01\xca\x81', crc=1036067972, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=1, cmd=10, retcode=0, payload=b'g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]\xc4\xc2B\x07\x8f\x83\xb4\xe4s\x117\xd9\xaf<\x8a|\xb3\x08y\x8b\xef-\xc8e\xb1\xe3\x965*\x8d\xb9\xe7\xeb\xd5\xe3D\xach\xd7\xf4a\x0f\xcb\x17b\x01\xca\x81', crc=1036067972, crc_good=True)
DEBUG:decode payload=b'g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]\xc4\xc2B\x07\x8f\x83\xb4\xe4s\x117\xd9\xaf<\x8a|\xb3\x08y\x8b\xef-\xc8e\xb1\xe3\x965*\x8d\xb9\xe7\xeb\xd5\xe3D\xach\xd7\xf4a\x0f\xcb\x17b\x01\xca\x81'
DEBUG:decrypting=b'g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]\xc4\xc2B\x07\x8f\x83\xb4\xe4s\x117\xd9\xaf<\x8a|\xb3\x08y\x8b\xef-\xc8e\xb1\xe3\x965*\x8d\xb9\xe7\xeb\xd5\xe3D\xach\xd7\xf4a\x0f\xcb\x17b\x01\xca\x81'
DEBUG:decrypted 3.x payload='{"devId":"00712485d8f15bc9d55b","dps":{"1":false,"3":166,"101":false}}'
DEBUG:payload type = <class 'str'>
DEBUG:decoded results='{"devId":"00712485d8f15bc9d55b","dps":{"1":false,"3":166,"101":false}}'
DEBUG:status() received data={'devId': '00712485d8f15bc9d55b', 'dps': {'1': False, '3': 166, '101': False}}

Same but using a bogus/invalid local key:
DEBUG:TinyTuya [1.7.0]

DEBUG:status() entry (dev_type is default)
DEBUG:building command 10 payload=b'{"gwId":"00712485d8f15bc9d55b","devId":"00712485d8f15bc9d55b","uid":"00712485d8f15bc9d55b","t":"1663478057"}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000010000000a000000788942e53e7b68baf064364cd88521507c4a686d6563edc5d21b57feb0d7544ae47136bff29382e14351568d31fa9535fc5945955e96b7d87eb54ab5df294f12c39854da1cab9c309b7cc45897dd28b020bc3e1422598843137b5bf0e5edbb0d2d6f9d963b5c024afd0aa90a5d651bd956e6ad865c0000aa55'
DEBUG:received data=b'000055aa000000010000000a0000002c00000001a58708f81a1671cf579242c9e373aaa2fd6865e2369eb6f71d82747f03ca74f0e5cfcb760000aa55'
DEBUG:received message=TuyaMessage(seqno=1, cmd=10, retcode=1, payload=b'\xa5\x87\x08\xf8\x1a\x16q\xcfW\x92B\xc9\xe3s\xaa\xa2\xfdhe\xe26\x9e\xb6\xf7\x1d\x82t\x7f\x03\xcat\xf0', crc=3855600502, crc_good=True)
DEBUG:raw unpacked message = TuyaMessage(seqno=1, cmd=10, retcode=1, payload=b'\xa5\x87\x08\xf8\x1a\x16q\xcfW\x92B\xc9\xe3s\xaa\xa2\xfdhe\xe26\x9e\xb6\xf7\x1d\x82t\x7f\x03\xcat\xf0', crc=3855600502, crc_good=True)
DEBUG:decode payload=b'\xa5\x87\x08\xf8\x1a\x16q\xcfW\x92B\xc9\xe3s\xaa\xa2\xfdhe\xe26\x9e\xb6\xf7\x1d\x82t\x7f\x03\xcat\xf0'
DEBUG:decrypting=b'\xa5\x87\x08\xf8\x1a\x16q\xcfW\x92B\xc9\xe3s\xaa\xa2\xfdhe\xe26\x9e\xb6\xf7\x1d\x82t\x7f\x03\xcat\xf0'
DEBUG:incomplete payload=b'\xa5\x87\x08\xf8\x1a\x16q\xcfW\x92B\xc9\xe3s\xaa\xa2\xfdhe\xe26\x9e\xb6\xf7\x1d\x82t\x7f\x03\xcat\xf0' (len:32)
Traceback (most recent call last):
  File "/home/pi/sandbox/tinytuya/tinytuya/core.py", line 834, in _decode_payload
    payload = cipher.decrypt(payload, False)
  File "/home/pi/sandbox/tinytuya/tinytuya/core.py", line 240, in decrypt
    raw = self._unpad(raw, verify_padding)
  File "/home/pi/sandbox/tinytuya/tinytuya/core.py", line 260, in _unpad
    raise ValueError("invalid padding length byte")
ValueError: invalid padding length byte
DEBUG:ERROR Unexpected Payload from Device - 904 - payload: null
DEBUG:status() received data={'Error': 'Unexpected Payload from Device', 'Err': '904', 'Payload': None}
{'Error': 'Unexpected Payload from Device', 'Err': '904', 'Payload': None}

I ran a test with -no-broadcasts - The force-scan found most of the devices but still can't figure out <IP_ADDRESS>.  Also note some SyntaxWarning we should fix (doesn't show up if you omit the netblock).
pi@piserver:~/sandbox/tinytuya $ python3 -m tinytuya scan -force <IP_ADDRESS>/24 -no-broadcasts
/home/pi/sandbox/tinytuya/tinytuya/scanner.py:122: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if ask is not 2:

TinyTuya (Tuya device scanner) [1.7.0]

[Loaded devices.json - 34 devices]

Scanning on UDP ports 6666 and 6667 for devices for 18 seconds...

    Option: Network force scanning requested.

    Running Scan...
  Starting Scan for network <IP_ADDRESS>/24
Connecting to debug ip <IP_ADDRESS>
Debug sock <IP_ADDRESS> failed!
<socket.socket fd=18, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('<IP_ADDRESS>', 57812)>
Traceback (most recent call last):
  File "/home/pi/sandbox/tinytuya/tinytuya/scanner.py", line 219, in get_peer
    addr = self.sock.getpeername()[0]
OSError: [Errno 107] Transport endpoint is not connected

ForceScannedDevice: Retrying connect <IP_ADDRESS>
Connecting to debug ip <IP_ADDRESS>
 ForceScannedDevice: Force-Scan Found Device <IP_ADDRESS>

... [clip] ...

Kitchen Light   Product ID = ?  [Failed to Force-Scan]:
    Address = 10.0.1.x,  Device ID = x, Local Key = x,  Version = 3.3, MAC = 60:01:94:c5:18:86
    Status: {'1': True, '9': 0}
Guest Room Lamp   Product ID = ?  [Failed to Force-Scan]:
    Address = 10.0.1.x,  Device ID = x, Local Key = x,  Version = 3.3, MAC = c4:4f:33:b4:c0:32
    Status: {'1': False, '11': 0}
Chandelier Upstairs   Product ID = ?  [Failed to Force-Scan]:
    Address = 10.0.1.x,  Device ID = x, Local Key = x,  Version = 3.3, MAC = bc:dd:c2:59:44:ea
    Status: {'1': False, '101': 0}

... [clip] ...

Connecting to debug ip <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.0529058
v3.4 trying key <IP_ADDRESS> b'9541e603af7266a4'
v3.4 session key neg start, debug ip <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Unknown v? Device   Product ID = ?  [Failed to Force-Scan]:
    Address = <IP_ADDRESS>,  Device ID = , Local Key = ,  Version = ?, MAC =
    Polling <IP_ADDRESS> Failed: Device stopped responding before key was found
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.1535273
Scanned in 18.706149578094482

Scan Complete!  Found 24 devices.
Broadcasted: 0
Force-Scanned: 24  - Matched GWID: 0 Matched Key: 22 Unmatched: 2
Force-Scan Errors: Connection Errors: 1 Version Detect Failed: 1

>> Saving device snapshot data to snapshot.json

We are close!  I like the "a" addition.  Thanks @uzlonewolf - you're amazing.

:facepalm: I was reopening the connection but not re-sending the payload, so of course none of the retries ever succeeded.  It also seems your device requires some payload data or it'll ignore the request instead of rejecting it with an error.  Both of those should be fixed with this.

Success! 🚀   That did it!!!
ForceScannedDevice: Brute force <IP_ADDRESS> decrypted: data format error
ForceScannedDevice: v3.3 brute forced key b'377d01b7c86e970c' for <IP_ADDRESS>
Resetting device scan end time due to debug ip <IP_ADDRESS> 0<PHONE_NUMBER>.8605416
ForceScannedDevice: <IP_ADDRESS> got step 100 data: b'\x00\x00U\xaa\x00\x00\x00\x02\x00\x00\x00\n\x00\x00\x00\\\x00\x00\x00\x00g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]rMk\xfcN\xad\xdcFS\xb0\xa6\xf3|=\xac\xd3\xee\xdf\xd8\xa7\x84kQ\xc6\x84O\xe2w\xbb\xe8\xa3\xee\x8b\x898\x89\xdf\xba\xef*\x9dE9\x9c\xb4H\xed! /-\xdf\x00\x00\xaaU'
ForceScannedDevice: Final Poll <IP_ADDRESS> 100 b'g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]rMk\xfcN\xad\xdcFS\xb0\xa6\xf3|=\xac\xd3\xee\xdf\xd8\xa7\x84kQ\xc6\x84O\xe2w\xbb\xe8\xa3\xee\x8b\x898\x89\xdf\xba\xef*\x9dE9\x9c\xb4H\xed!'
{'devId': '00712485d8f15bc9d55b', 'dps': {'1': True, '3': 71, '101': False}}
Chandelier   Product ID = ?  [Force-Scanned]:
    Address = <IP_ADDRESS>,  Device ID = 00712485d8f15bc9d55b, Local Key = 377d01b7c86e970c,  Version = 3.3, MAC = d8:f1:5b:c9:d5:5b
    Status: {'1': True, '3': 71, '101': False}

I notice this OSError always happens at the top - doesn't seem to matter but is curious:
    Running Scan...
  Starting Scan for network <IP_ADDRESS>/24
Connecting to debug ip <IP_ADDRESS>
Debug sock <IP_ADDRESS> failed!
<socket.socket fd=28, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('<IP_ADDRESS>', 36660)>
Traceback (most recent call last):
  File "/home/pi/sandbox/tinytuya/tinytuya/scanner.py", line 219, in get_peer
    addr = self.sock.getpeername()[0]
OSError: [Errno 107] Transport endpoint is not connected

And the thread:
$ python3 -m tinytuya scan -force <IP_ADDRESS>/24 -no-broadcasts -nocolor | egrep "<IP_ADDRESS>|Chandelier" | egrep -v "Resetting|Upstairs"
Connecting to debug ip <IP_ADDRESS>
 ForceScannedDevice: Force-Scan Found Device <IP_ADDRESS>
ForceScannedDevice: Debug sock <IP_ADDRESS> socket send failed, receive timed out
Connecting to debug ip <IP_ADDRESS>
ForceScannedDevice: <IP_ADDRESS> got step 1 data: b'\x00\x00U\xaa\x00\x00\x00\x01\x00\x00\x00\n\x00\x00\x00,\x00\x00\x00\x01\xa5\x87\x08\xf8\x1a\x16q\xcfW\x92B\xc9\xe3s\xaa\xa2\xfdhe\xe26\x9e\xb6\xf7\x1d\x82t\x7f\x03\xcat\xf0\xe5\xcf\xcbv\x00\x00\xaaU'
ForceScannedDevice: Brute force <IP_ADDRESS> decrypted: data format error
ForceScannedDevice: v3.3 brute forced key b'377d01b7c86e970c' for <IP_ADDRESS>
ForceScannedDevice: <IP_ADDRESS> got step 100 data: b'\x00\x00U\xaa\x00\x00\x00\x02\x00\x00\x00\n\x00\x00\x00\\\x00\x00\x00\x00g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]rMk\xfcN\xad\xdcFS\xb0\xa6\xf3|=\xac\xd3\xee\xdf\xd8\xa7\x84kQ\xc6\x84O\xe2w\xbb\xe8\xa3\xee\x8b\x898\x89\xdf\xba\xef*\x9dE9\x9c\xb4H\xed! /-\xdf\x00\x00\xaaU'
ForceScannedDevice: Final Poll <IP_ADDRESS> 100 b'g\xe1\r\xcf=\xcdvc\\\xd9\n\xec\x07tU%+\xbb\xb8P\xe4!d\x95\x7f&\t\x14\xaa~Z]rMk\xfcN\xad\xdcFS\xb0\xa6\xf3|=\xac\xd3\xee\xdf\xd8\xa7\x84kQ\xc6\x84O\xe2w\xbb\xe8\xa3\xee\x8b\x898\x89\xdf\xba\xef*\x9dE9\x9c\xb4H\xed!'
Chandelier   Product ID = ?  [Force-Scanned]:
    Address = <IP_ADDRESS>,  Device ID = 00712485d8f15bc9d55b, Local Key = 377d01b7c86e970c,  Version = 3.3, MAC = d8:f1:5b:c9:d5:5b

Great!  I just need to get v3.1 working now.
That "OSError: [Errno 107] Transport endpoint is not connected" just means the device accepted the connection but then immediately closed it.  I have no idea why they do that, it seems pretty random with some devices never doing it but others doing it on occasion.  Everything seems to work normally on the 2nd try for whatever reason.  I should probably clean up that output a bit to make it a bit more obvious.

Hi @uzlonewolf - remind me, what is our next step for this draft PR?

I've been delaying finishing this as I really, really, really wanted a v3.2 device to test against.  With how it needs to brute-force the version (since that information isn't available anywhere) I have no idea how it'll act when encountering one.  I have a switch on order that was reported to have v3.2, however I'm not holding my breath as I have no idea how long ago the reporter bought it.  It should arrive tomorrow, so we'll see.
Beyond that I think it just needs some clean-up and documentation.  I'll try to get the master-branch changes merged in later today.  Once it's ready I think I'm going to close this PR and open a new one so we don't get this giant list of commits in the release notes.

Oh yeah, as for v3.1 devices, I've only tried with 1 so far (and promptly forgot where I put the box with the rest of them) but it worked fine.

Wait, how did I close this?  I didn't mean to do that...

decrypt_udp() has been modified; instead of trimming with result[20:-8], just pass the entire result.

Finalized in #252


Feature Request: Also publish Sparky as a standalone module

I have a project that is using Sparky as a task runner, because it is simply awesome.  However, we cannot leverage fuse-box as a bundler at the moment, so we are bringing in those dependencies needlessly in our project.  It would be a huge help if Sparky could additionally be published as a standalone npm module.
Cheers 👍

@booninite hi
Do you think you could help making a gulp task? Similar to this one?
https://github.com/fuse-box/fuse-box/blob/c8958a7b23178e71b1057cca0555cb70cbe0b67c/gulpfile.js#L216-L227
I would accept a PR for that. It's just currently the process is manual, however, linking it to the release of @next would automate things. I don't mind that.


Use memory store since loading rails 5 fails to connect to memcached.

We don't need a session store in anaconda so force Rails to use memory store
since rails 5 mem_cache_store tries and fails to connect to a not running
memcached.
Depends on: https://github.com/ManageIQ/manageiq/pull/6751

Replaces #90

:+1: yay

:( I hate more hacks...would love a to find a way to not have this at all


MouseWheelZoomModifier feature is not working properly

When I try to zoom in/out using mouse, Chart not zoom in/out Y-axis even I set
XyDirection = XyDirection.XyDirection
and X-Axis is working properly.
Can you guide is there anything wrong or miss to set anything?

Hi @dhruvindudhat,
Thanks for reaching out, and welcome!
I threw together some quick code to test this out and it looks like it works as expected. XyDirection.XyDirection is the default behavior. I suspect the issue you're having is that in the Y axis AutoRange = AutoRange.Always   (this is the case in the example wasm LineRenderableSeriesChartExample test). This will effectively override the XyDirection, and it will resize the Y axis every time it changes. Try changing this to Once or Never.
Hope this helps!

@mikehollas123 Thanks for helping, Looks like it is working fine! You may close this issue.


2FA Required, Click Here, Wait a Few Seconds And Try Again

Unable to login to MiCloud to discover new devices, existing BLE devices that require forced micloud connection has also broken.
After clicking and logging in to micloud account authentication, the webpage just displays ok in text. nothing else happens

I think you have to disable 2FA for your account.

I think you have to disable 2FA for your account.

Theres no option to disable of mi account.

You could also try to authenticate in a different browser window

I'm having the same problem, have you found out how to solve it?

I would need to try create an account with 2FA enabled as my primary account does not have it. Could tell where did you create the account and how you enabled 2FA?

2FA was enabled by default. My workaround was to log into my mi account via https://account.xiaomi.com/fe/service/login/phone?_locale=en on another tab while homebridge tries to authenticate at the same time on another browser tab.
On 14 Aug 2024, at 1:59 PM, Marcin @.***> wrote:

I would need to try create an account with 2FA enabled as my primary account does not have it. Could tell where did you create the account and how you enabled 2FA?
—
Reply to this email directly, view it on GitHubhttps://github.com/merdok/homebridge-miot/issues/618#issuecomment-2287916066, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYWLJPZPTLHEMBUSQWB3WKDZRLW5XAVCNFSM6AAAAABL4VWHESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBXHEYTMMBWGY.
You are receiving this because you authored the thread.Message ID: @.***>


Add ignoreAtLeastPermissionLevel Property In Inhibitors

Description of the PR
This PR adds the ignoreAtLeastPermissionLevel to inhibitors so certain inhibitors do not get run on certain users. For example, you can prevent inhibitors from running on the bot owner, bot developers, server admins, server owners, server mods etc... Adds a huge layer of flexibility to inhibitors without needing to add it inside the run.
Changes Proposed in this Pull Request (List new items in CHANGELOG.MD)

Add ignoreAtLeastPermissionLevel to inhibitors.

Semver Classification

[ ] This PR only includes documentation or non-code changes.
[ ] This PR fixes a bug and does not change the (intended) framework interface.
[x] This PR adds methods or properties to the framework interface.
[ ] This PR removes or renames methods or properties in the framework interface.

PermissionLevels are expensive and shouldn't be run for every inhibitor.

What method would you suggest for this functionality if not by PermissionLevels?

doing it on an as-needed basis. For instance, there is no need to check permissionLevels for the permissions inhibitor, the bot permissions inhibitor, and many more... It's a niche idea


Update requirements.txt to skip uvloop on Windows machines

Skips installing uvloop on Windows machines.
Purpose

uvloop doesn't support Windows
Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type
What kind of change does this Pull Request introduce?

[x ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test
pip install -r src/requirements.txt on a windows machine

Test the code

What to Check
Verify that the following are valid

...

Other Information

See https://github.com/Azure-Samples/openai-chat-app-quickstart/pull/193 for a PR for the issue and a discussion of the best approach. The author hasn't updated it per our discussed changes, so you could make the changes there, or I can make them when I have the chance.

I just merged a fix for this, thanks for reminding me about it!


Problems in Terminal Python 3 and 2

When working in the terminal with the following code, an error occurs:
from pscript import py2js
def foo(a, b=2):
   print(a - b)
print(py2js(foo))

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pscript/functions.py", line 77, in py2js_
    lines, linenr = inspect.getsourcelines(ob)
  File "/usr/lib/python3.6/inspect.py", line 955, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/lib/python3.6/inspect.py", line 786, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/pscript/functions.py", line 145, in py2js
    return py2js_(ob)
  File "/usr/local/lib/python3.6/dist-packages/pscript/functions.py", line 80, in py2js_
    (ob, err))
ValueError: Could not get source code for object <function foo at 0x7fc97016aea0>: could not get source code

When studying, it becomes clear that inspect.getsource(foo) gives an error when working in the terminal. But if you use the module dill then everything is ok.
from dill.source import getsource
print(getsource(foo))

When working in the terminal

Do you mean that you type the lines in right at the terminal? In that cases, indeed, Python is not able to get the source code for these lines. Nothing we can do about that, unfortunately.

Everything works well when using dill
from dill.source import getsource
print(getsource(foo))

Ah, I read over that bit. That's interesting. Seems to be due to this. I'm not very enthusiastic about adding dill (or readline) as a dependency. Though maybe we can add terminal support (using readline as an optional dependency) using a littlebit of custom code that we extract from that dill module...

Made #28 to describe and track the potential enhancement. Thanks!


[FEA] [Audit] -  [SPARK-40389][SQL] Decimals can't upcast as integral types if the cast can overflow

Is your feature request related to a problem? Please describe.
This is a bug fix where the condition is checked if it is safe to cast to Integral type. I think this change doesn't affect spark-rapids but I am not certain. It would be good if someone familiar with this code take a look. We could add a test if required.
Apache Spark commit: https://github.com/apache/spark/commit/17982519a7

You are correct that this should not really impact us. The check is mostly for automatic casting done in the logical planning stages.  The only possible impact would be that some queries/tests might need to have some explicit casts added that were not needed before, but I don't think that is very likely.

Closing based on the above comment.


Connect with Replicate API?

Thanks for sharing this great repo!
I am trying to give it a test drive and connect with replicate stable diffusion API:
https://replicate.com/stability-ai/stable-diffusion/api
But I am a bit lost, and they don’t seem to have an IP.
Is it possible to connect to replicate? If not, what would be another option?
Thanks
Alex

Update...
I deployed SD WEBUI now to banana.dev (as recommended by the SD WEBUI community, but neither Banana nor Replicate give me a http API address...
https://github.com/patienceai/stable-diffusion-automatic1111-banana
Does it work to host on those servers, or I would need some other setup?

Hi Alex, this app is for the original webui server, and you have to deploy it to your own server to use it.
replicate use a different api from automatic111, and banana seems like a modified version of the original, so I don’t think it’s compatible.

Thanks a lot for the help!
What server do you recommend? One that only charges when you run inference (ex, don’t need to have the machine on and pay by the hour whether you are using or not).
If you could put some very basic tutorial here in how to deploy alto a server and get the IP address to use in the app, I think would help a lot to promote and use your app. 🙂👍
Cheers from Stockholm!
Alex

Also tried now using the local  URL generated in google colab, but doesn’t work…

to use handydiffu, you should do the followings

follow the instructions of the official automatic111 webui to install webui server on your device
enable api access

thanks for the instructions  @BIGPPWONG!
If I want to use without Tailscale on a server, I guess I need to :
APIUrl: the Api generated by WEBUI
Username: login via SSH
password: pw via SSH
is that correct?


[TEST] test case test_system_backup_and_restore_volume_with_data always failed

What's the test to develop? Please describe
The test case test_system_backup_and_restore_volume_with_data always failed since it's been created on Jan 6, 2023.
Both on master and v1.4.x, this test case never passed. But if you run it separately (only run this single test case), it will pass. It only failed in a full regression test.
It expect volume state becomes detached, but it's empty:
AssertionError:  value=detached .             volume[key]= . volume={
	'accessMode': 'rwo',
	'backingImage': '',
	'backupStatus': [],
	'cloneStatus': {
		'snapshot': '',
		'sourceVolume': '',
		'state': ''
	},
	'conditions': {},
	'controllers': [],
	'created': '2023-02-23 00:28:15 +0000 UTC',
	'currentImage': 'longhornio/longhorn-engine:v1.4.x-head',
	'dataLocality': 'disabled',
	'dataSource': '',
	'disableFrontend': False,
	'diskSelector': [],
	'encrypted': False,
	'engineImage': 'longhornio/longhorn-engine:v1.4.x-head',
	'fromBackup': 's3://backupbucket@us-east-1/backupstore?backup=backup-a3a975678042427e&volume=longhorn-testvol-8wgnwb',
	'frontend': 'blockdev',
	'kubernetesStatus': {
		'lastPVCRefAt': '',
		'lastPodRefAt': '',
		'namespace': '',
		'pvName': '',
		'pvStatus': '',
		'pvcName': '',
		'workloadsStatus': None
	},
	'lastAttachedBy': '',
	'lastBackup': '',
	'lastBackupAt': '',
	'migratable': False,
	'name': 'longhorn-testvol-8wgnwb',
	'nodeSelector': [],
	'numberOfReplicas': 3,
	'purgeStatus': None,
	'ready': False,
	'rebuildStatus': [],
	'recurringJobSelector': None,
	'replicaAutoBalance': 'ignored',
	'replicas': [],
	'restoreRequired': False,
	'restoreStatus': [],
	'restoreVolumeRecurringJob': 'ignored',
	'revisionCounterDisabled': False,
	'robustness': '',
	'shareEndpoint': '',
	'shareState': '',
	'size': '16777216',
	'snapshotDataIntegrity': 'ignored',
	'staleReplicaTimeout': 0,
	'standby': False,
	'state': '',
	'unmapMarkSnapChainRemoved': 'ignored'
}

Describe the items of the test development (DoD, definition of done) you'd like

Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists

[ ] item 1

Additional context
Add any other context or screenshots about the test request here.

I feel some test dependencies are in between. @yangchiu please identify the cause. Thanks.

From @yangchiu info, it seems the test never succeeds. @c3y1huang Please take a look first. Thanks.

As @yangchiu described, based on the test results here and here the test fails only in full regression. Need to check if there are some dependencies.

Found in support bundle:
2023-03-02T01:09:11.996703176Z time="2023-03-02T01:09:11Z" level=warning msg="Error syncing Longhorn volume longhorn-system/longhorn-testvol-xm4uyl" controller=longhorn-volume error="failed to sync longhorn-system/longhorn-testvol-xm4uyl: failed to reconcile volume state for longhorn-testvol-xm4uyl: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-422da1ea33ab4eb2&volume=longhorn-testvol-xm4uyl: backup.longhorn.io \"backup-422da1ea33ab4eb2\" not found" node=ip-10-0-2-222
2023-03-02T01:09:12.002255813Z time="2023-03-02T01:09:12Z" level=warning msg="Error syncing Longhorn volume longhorn-system/longhorn-testvol-xm4uyl" controller=longhorn-volume error="failed to sync longhorn-system/longhorn-testvol-xm4uyl: failed to reconcile volume state for longhorn-testvol-xm4uyl: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-422da1ea33ab4eb2&volume=longhorn-testvol-xm4uyl: backup.longhorn.io \"backup-422da1ea33ab4eb2\" not found" node=ip-10-0-2-222
2023-03-02T01:09:12.012897386Z time="2023-03-02T01:09:12Z" level=warning msg="Error syncing Longhorn volume longhorn-system/longhorn-testvol-xm4uyl" controller=longhorn-volume error="failed to sync longhorn-system/longhorn-testvol-xm4uyl: failed to reconcile volume state for longhorn-testvol-xm4uyl: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-422da1ea33ab4eb2&volume=longhorn-testvol-xm4uyl: backup.longhorn.io \"backup-422da1ea33ab4eb2\" not found" node=ip-10-0-2-222
2023-03-02T01:09:12.033562946Z E0302 01:09:12.033451       1 volume_controller.go:226] failed to sync longhorn-system/longhorn-testvol-xm4uyl: failed to reconcile volume state for longhorn-testvol-xm4uyl: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-422da1ea33ab4eb2&volume=longhorn-testvol-xm4uyl: backup.longhorn.io "backup-422da1ea33ab4eb2" not found
2023-03-02T01:09:12.033579757Z time="2023-03-02T01:09:12Z" level=warning msg="Dropping Longhorn volume longhorn-system/longhorn-testvol-xm4uyl out of the queue" controller=longhorn-volume error="failed to sync longhorn-system/longhorn-testvol-xm4uyl: failed to reconcile volume state for longhorn-testvol-xm4uyl: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-422da1ea33ab4eb2&volume=longhorn-testvol-xm4uyl: backup.longhorn.io \"backup-422da1ea33ab4eb2\" not found" node=ip-10-0-2-222

Randomly looking through the support bundles. This seems to be caused by the overwriting of longhorn-system/longhorn-default-setting resulting empty backup target. However, I am not sure why this is only an issue with full regression.
2023-02-27T01:21:44.931269393Z time="2023-02-27T01:21:44Z" level=warning msg="Error syncing ConfigMap longhorn-system/longhorn-default-setting" controller=longhorn-kubernetes-configmap-controller error="longhorn-kubernetes-configmap-controller: failed to sync longhorn-system/longhorn-default-setting: failed to update built-in settings with customized values: Operation cannot be fulfilled on settings.longhorn.io \"backup-target\": the object has been modified; please apply your changes to the latest version and try again" node=ip-10-0-2-36
2023-02-27T01:21:45.000381618Z time="2023-02-27T01:21:45Z" level=info msg="Deleting system backup in backup target" controller=longhorn-system-backup node=ip-10-0-2-36 systemBackup=test-system-backup-0rvy1b
2023-02-27T01:21:45.000441174Z time="2023-02-27T01:21:45Z" level=info msg="Requeue test-system-backup-0rvy1b for next phase: Deleting" controller=longhorn-system-backup node=ip-10-0-2-36 systemBackup=test-system-backup-0rvy1b
2023-02-27T01:21:45.000650106Z time="2023-02-27T01:21:45Z" level=info msg="Event(v1.ObjectReference{Kind:\"SystemBackup\", Namespace:\"longhorn-system\", Name:\"test-system-backup-0rvy1b\", UID:\"412dc792-aeea-482c-b501-86cfe2e12068\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"180329\", FieldPath:\"\"}): type: 'Normal' reason: 'Deleting' Deleting system backup in backup target"
2023-02-27T01:21:45.004021129Z time="2023-02-27T01:21:45Z" level=info msg="Cannot find a reusable failed replicas for volume longhorn-testvol-4cfj02"
2023-02-27T01:21:45.009591871Z time="2023-02-27T01:21:45Z" level=warning msg="Failed to list system backups in backup target" controller=longhorn-system-backup error="error listing system backup in : failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-master-head/longhorn [system-backup list ], output , stderr, time=\"2023-02-27T01:21:45Z\" level=fatal msg=\"Failed to run list system backup command\" error=\"destination URL hasn't been specified\"\n, error exit status 1" node=ip-10-0-2-36 systemBackup=test-system-backup-0rvy1b
2023-02-27T01:21:45.118191840Z time="2023-02-27T01:21:45Z" level=info msg="Cannot find a reusable failed replicas for volume longhorn-testvol-4cfj02"
2023-02-27T01:21:45.225214549Z time="2023-02-27T01:21:45Z" level=info msg="Cannot find a reusable failed replicas for volume longhorn-testvol-4cfj02"
2023-02-27T01:21:45.333991572Z time="2023-02-27T01:21:45Z" level=debug msg="Schedule replica to node ip-10-0-2-36" dataDirectoryName=longhorn-testvol-4cfj02-09d71a9c disk=100189fe-a3d8-410b-9fba-839acf38fc34 diskPath=/var/lib/longhorn/ replica=longhorn-testvol-4cfj02-r-1690e77f
2023-02-27T01:21:45.334034367Z time="2023-02-27T01:21:45Z" level=debug msg="Schedule replica to node ip-10-0-2-179" dataDirectoryName=longhorn-testvol-4cfj02-d1fb8805 disk=c2d5e363-2844-4414-9f6f-5cc45580be16 diskPath=/var/lib/longhorn/ replica=longhorn-testvol-4cfj02-r-002ba31f
2023-02-27T01:21:45.334127115Z time="2023-02-27T01:21:45Z" level=debug msg="Schedule replica to node ip-10-0-2-242" dataDirectoryName=longhorn-testvol-4cfj02-80e03e5f disk=7b6d23d1-5966-4338-aff2-1affc10b5555 diskPath=/var/lib/longhorn/ replica=longhorn-testvol-4cfj02-r-75e62294
2023-02-27T01:21:45.344030359Z time="2023-02-27T01:21:45Z" level=debug msg="Replica controller picked up" controller=longhorn-replica controllerID=ip-10-0-2-36 dataPath= node=ip-10-0-2-36 nodeID= ownerID= replica=longhorn-testvol-4cfj02-r-75e62294
2023-02-27T01:21:45.344120851Z time="2023-02-27T01:21:45Z" level=debug msg="Instance handler updated instance longhorn-testvol-4cfj02-r-75e62294 state, old state , new state stopped"
2023-02-27T01:21:45.453965890Z time="2023-02-27T01:21:45Z" level=debug msg="Replica controller picked up" controller=longhorn-replica controllerID=ip-10-0-2-36 dataPath= node=ip-10-0-2-36 nodeID=ip-10-0-2-36 ownerID=ip-10-0-2-179 replica=longhorn-testvol-4cfj02-r-1690e77f
2023-02-27T01:21:45.461255034Z time="2023-02-27T01:21:45Z" level=debug msg="Requeue volume due to error failed to reconcile volume state for longhorn-testvol-4cfj02: failed to get backup volume: longhorn-testvol-4cfj02: backupvolume.longhorn.io \"longhorn-testvol-4cfj02\" not found or Operation cannot be fulfilled on replicas.longhorn.io \"longhorn-testvol-4cfj02-r-75e62294\": the object has been modified; please apply your changes to the latest version and try again" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=ip-10-0-2-36 owner= state= volume=longhorn-testvol-4cfj02
2023-02-27T01:21:45.569949000Z time="2023-02-27T01:21:45Z" level=warning msg="Error syncing Longhorn volume longhorn-system/longhorn-testvol-4cfj02" controller=longhorn-volume error="failed to sync longhorn-system/longhorn-testvol-4cfj02: failed to reconcile volume state for longhorn-testvol-4cfj02: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-d1e6f67d09d7425b&volume=longhorn-testvol-4cfj02: backup.longhorn.io \"backup-d1e6f67d09d7425b\" not found" node=ip-10-0-2-36
2023-02-27T01:21:45.575672737Z time="2023-02-27T01:21:45Z" level=warning msg="Error syncing Longhorn volume longhorn-system/longhorn-testvol-4cfj02" controller=longhorn-volume error="failed to sync longhorn-system/longhorn-testvol-4cfj02: failed to reconcile volume state for longhorn-testvol-4cfj02: cannot inspect the backup config s3://backupbucket@us-east-1/backupstore?backup=backup-d1e6f67d09d7425b&volume=longhorn-testvol-4cfj02: backup.longhorn.io \"backup-d1e6f67d09d7425b\" not found" node=ip-10-0-2-36

To fix this, we should skip restoring the longhorn-default-setting configmap as well as the backup target and secret settings to avoid an invalid backup target overwritten by the restore.

To fix this, we should skip restoring the longhorn-default-setting configmap as well as the backup target and secret settings to avoid an invalid backup target overwritten by the restore.

Test results: https://github.com/longhorn/longhorn-manager/pull/1728#issuecomment-1452786362

After the fix merged on 3/3, this test case always passes in master branch's daily regressions since then.


Ctrl+F support, finding text on a page (even when scrolled off screen)

Current status: https://github.com/flutter/flutter/issues/65504#issuecomment-1475059683

cannot browser search (CTRL +F) to find text on a page

Some ideas from @sidrao2006

Use case
Add support for browser search
Proposal

Let browser handle the search

Provide a widget specifically to support static text features

Let html handle semantics and rendering for such elements

Use the Intersection Observer API (https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API) to detect searches. Also add style change listeners to the text, such that it won't be triggered when the framework changes the style of the text but will be triggered when the browser changes the text style (since the browser changes the text style to highlight the searched words)

Once the framework knows that the browser is navigating to a certain piece of text, render the relevant elements/widgets which are supposed to be in the surroundings of the text

The static text rendering widget should be low level and abstracted under the Text widget such that static text mide will be enabled automatically for texts that do not have dynamic dependencies.

Just some brainstorming:
Browser native search may not be the best fit for Flutter due to lazy rendering, and because it's unclear how the same functionality can be achieved on mobile and desktop. A custom search might be a better fit because it would work on all platforms. We should make sure that developers can easily implement text search in their apps. Perhaps the framework could assist somehow, I'm not sure. A quick survey of non-Flutter web apps shows that many developers prefer custom search bars, e.g. Twitter, Gmail, Discord, Docs, Calendar, Slack, Chat, Maps, Facebook. Many actively redirect the Ctrl + F shortcut to their custom search bars. In some text heavy cases both can be used (and are useful): Github, Amazon.com.
Mobile users frequently don't have keyboards, so there's no way to intercept Ctrl + F. If there's no app-specific search bar, they would invoke "Find in page..." (also available on desktop browsers, if you know where to find it in the menus). I imagine, however, that users would prefer the custom search bar, especially if it's sufficiently prominent on the UI.

is it possible to propose changes to the html to add hooks for this? lack of native browser "Find in page..." on mobiles cannot be really solved by embedding some custom search. users don't know what we used to build our website and won't understand why this feature is suprisingly not working

This is also a frequent customer complaint in our use-cases. As a user myself I hate websites that override CTRL+F functionality (which is considered a bad practise). It's really convenient and it's also a quick way to navigate around websites.
However, I believe that with canvaskit rendering it is impossible to implement native text search, as the text isn't (and won't be) rendered by the browser natively. Maybe the next best thing would be to have an easy to use implementation of a custom search bar which would:

imitate standard search behaviour (highlight any matching text that was rendered - if we have lazy loaded lists etc.. it is perfectly fine and expected to have no matches
after exiting the search, focus should move to the last highlighted widget / text (if there is any focusable parent)

For mobile devices there is one way to at least show a button for the in-app / emulated search bar when user enters a query into the default "find on page" bar:
https://user-images.githubusercontent.com/8578790/120169535-5b26f380-c200-11eb-8624-9e03642ecae1.mov
If we add all letters of the alphabet to a hidden section at the bottom, the browser will scroll to it when any letter is typed into the search box. It is a bit hackish, but I don't see any other way to achieve this.

It might be even possible to capture the native searches by observing scroll events
https://user-images.githubusercontent.com/8578790/120180225-cb874200-c20b-11eb-9449-44c01b1fdc4c.mov
https://gist.github.com/vidklopcic/766951868068d666fd3f0e916d9bf602

any updates...?

Will this be feature be added?

I've been trying to implement a custom search bar, but even this workaround currently seems to be impossible for focus reasons:
https://github.com/flutter/flutter/issues/111470
Does anyone know how to implement a custom search field that can be focused when CTRL + F is pressed anywhere on the page?

Hi, any plans for the flutter team to tackle this? We use flutter web in production and our users are asking. Thanks.

I've changed the default renderer to HTML and now the search with CTRL + F is working.
https://docs.flutter.dev/development/platform-integration/web/renderers
I know it's a workaround and not a perfect solution, but it's working and my clientes are happy with it.

I think that to fix this, they could go with two approaches:
A) Overriding the native Ctrl+F functionality of the browser (something that looks like a Flutter way of doing it, so it would work cross-platform)
B) Actually exporting the text of the app to "HTML readable" code. This would probably help with the SEO issue, but I really don't know how you could link that search results to a visual highlight (and scrolling to that location) in the rendered Flutter code (JS canvas).

Based on popular demand, I'm putting this back into triage queue for reprioritization. I'm also forwarding this to the framework team due to the following:

Due to lazy rendering, the engine only sees a subset of the UI that's currently visible. For proper text search functionality we should have the ability to scroll to the desired content, but scrolling is a framework-level concept.
There's no link from layers and pictures back to widgets, so even if we did search them it's not clear what to do next. We should provide some level of interaction with the found content such as highlighting it, selecting it, etc.
Searching UI should not be limited to web. It should work on AOT desktop and mobile too.
The search should follow some logical widget order, similar to cross-widget text selection. This information lives in the framework.

In AOT desktop and mobile, and on desktop web, this function can be implemented via a keyboard shortcut. Mobile web will be tricky though due to reasons outlined in https://github.com/flutter/flutter/issues/65504#issuecomment-790820972. Here's an idea though:

The framework sends a list of text strings that can be searched with unique widget IDs to the engine.
The engine creates a hidden scrollable area and puts the text strings in it, such that "Find in page" can find them.
When the user finds a string using the browser's built-in search function, the browser scrolls to the string.
The engine listens to the scroll offset and maps it back to the original text string and the corresponding widget ID. This mapping can be made precise if the strings are spaced apart more than the height of the scroll viewport. This way a scroll offset value uniquely identifies the string that's found. The engine then sends a message back to the framework so that the framework scrolls the desired widget into view.

A problem with this approach is when the search text spans more than one string, we can only find the first string.
As an MVP we could start with a pure framework implementation that uses a keyboard shortcut. Integration with browser search can attempted as a follow-up.

Adding to this that building/running with --web-renderer html does make this issue a little better -- it still won't find something that's been scrolled off-screen, but it can find and highlight text that is on-screen.

Even with --web-renderer html it can't seem to find special characters. So if you're search string has for instance ' ', ':', '_', etc. it won't find it. Even if it's on-screen.

We've had issues with getting google ads approved for our website because the verifier could not find our given contact information on the website. He was doing ctrl+f for that info. We really need this, we are too far gone into development to re-build our website using another framework.

any updates on this guys, pretty critical feature

Do they realize that this is the single biggest downside of using Flutter? If it were resolved, I'd use it for most projects. It's crazy 😕.

Dear @yjbanov,
The ability to search within a page using browser commands, such as Ctrl+F, is a fundamental feature that operates seamlessly across most websites. This issue has been open for four years, gathered significant community support with over 100 likes, and has multiple duplicate threads.
Could you please provide an update on whether we can anticipate a resolution to this problem in the coming months?
Thank you for your attention to this matter.

Is there a chance if someone we're to step forward and financially sponsor the feature (e.g. $1000), we could get it in?

Is there a chance if someone we're to step forward and financially sponsor the feature (e.g. $1000), we could get it in?

+1, I would definitely contribute to this financially.

Any updates? It's a huge problem on a huge platform like Flutter, CTRL + F is fundamental for every website. @yjbanov

My team at the Flutter Bounty Hunters is always available if there are financial backers.
I should point out that the amount of labor needed to actually propose, build, and land this in Flutter would be well beyond $1,000. This is largely due to Flutter's review process - both for design docs and for PRs. I would estimate something like $5,000 => 50 hours.
I'll also offer to record and publish a video of the work along the way, including drafting the design doc.
PS - For those focused on web, this might not be possible to implement due to weird browser controls over IME capabilities.

@matthew-carroll

PS - For those focused on web, this might not be possible to implement due to weird browser controls over IME capabilities.

This issue seems primarily focused on web, and it even has label:platform-web. It seems like its doable, but with quite a bit of work.

@dsyrstad - I think it depends on the specific requirements. There's definitely some level of support that could be brought to web. For example, we can definitely catch the shortcut for ctrl+f and cmd+f. From there, we can definitely query a content index that we assemble within Flutter.
I was referring more to any browser-level behaviors that might happen without any interaction with JS APIs. I'm not sure all the things browsers do for "Find", but think about browsers where you right-click on a text field or something like that. You get a "native" browser popup. That behavior isn't exposed to any JS APIs. It's done automatically by the browser, completely outside of the developer context. Things like that are simply outside the scope of what Flutter, or any other UI toolkit, can interact with. You either use those DOM elements or you're SOL.

@matthew-carroll

I was referring more to any browser-level behaviors ... all the things browsers do for "Find", but think about browsers where you right-click on a text field or something like that...

Agreed. This issue seems to focus on Ctrl+F, so it'd be great to have something for that - the 80/20 rule.

In that case, happy to do it with financial backing. The shortcut is easy. We'll need a design doc to cover the intricacies of assembling the content/text index from the widget tree.

How to korean text??
i can't search korean text like that (renderer option is html)

Triage: acknowledged

The fact that this isn't clearly stated in the documentation makes it feel like Google is trying to quietly sweep it under the rug to avoid making Flutter Web look bad. Standard web functionality like Ctrl+F can even be an absolute dealbreaker for apps where this is essential. Clear documentation on such limitations is crucial for setting the right expectations. It would take literally just a sentence in the documentation to save developers hours of frustration and set clear expectations.

I don't know how Flutter triage works but this is absolutely a must have for a web framework to be viable. Unfortunately this one is what made our team vote against Flutter and go back to Angular.

This is insane. https://github.com/input-output-hk/catalyst-voices/issues/888#issuecomment-2385079269

@omar-selo is this still valid? flutter build web --web-renderer html and search for special symbols/emojis works perfect.

@omar-selo is this still valid? flutter build web --web-renderer html and search for special symbols/emojis works perfect.

Now try to add space to your search. Or search for something that is off screen and not currently visible.

@omar-selo is this still valid? flutter build web --web-renderer html and search for special symbols/emojis works perfect.

@cristaloleg: The html renderer is going to be removed: #145954,  and it doesn't work at all with canvaskit.

 CTRL F ussue together with SEO is the biggest problems in Flutter Web

I imagine the importance of this feature is already clear. Hundreds of people have taken part in this issue.
However, the importance of this feature is doubly significant for those already in production who forgot to take it into account because they assumed it was a given.
Over the years, this technology hasn't ceased to impress many of us. Its development has been meteoric, and we shouldn't let an issue like this hold it back.

I think, at least using the browser search the visible text only would be a helpful feature.
Of course the full version must be a complete search function..maybe a specific parent widget would help to collect all text and its position to make it visible to the browser search, although not yet fully rendered?


Explain what "forge remove" does more specifically

Removing a git submodule properly involves quite a few steps:
# Remove the submodule entry from .git/config
git submodule deinit -f path/to/submodule

# Remove the submodule directory from the superproject's .git/modules directory
rm -rf .git/modules/path/to/submodule

# Remove the entry in .gitmodules and remove the submodule directory located at path/to/submodule
git rm -f path/to/submodule

It would be helpful to mention whether forge remove performs all of the steps above, or just a combination of them, in the Removing dependencies section.

This is better suited for the command reference if anywhere


Second Contribution by Treasure Okafor

Name: Treasure Okafor
Projects Analyzed: BQplot | PySAL
BQplot
PySAL
Comparative Analysis of the Governance Models
-- BQplot and PySAL are both open source projects that develop software for data science, though BQplot is majorly for interactive and explanatory data visualization while PySAL focuses on spatial analysis with python.
The governance models of these two projects however have notable similarities and differences in administration, decision making and overall governance.
BQplot's governance model is based on The Steering Council consisting of significant project contributors which make decisions about the project's direction, collaboration with other organizations, financial sources and major structural and systemic changes to the project. Nominating potential Steering council members is based on successful votes by existing council members. The council reserves full authority to eject current member if need be.
PySAL's governance model follows a benevolent dictator for life (BDFL) model of governance who has the authority to make all final decisions for the project. Subsequentially, a subset of individuals called the Steering council, (usually 3 members- package maintainers) make decisions about day-to-day projects. Majorly, the BDFL can defer or override that authority in certain cases.

Substantial Project Contributors are eligible to be members of The Steering Council in BQplot, the membership terms last for 2 years.
In PySAL, The Steering council membership lasts for 1 year and a new member is nominated and elected.
BQplot and PySAL have similar roles such as the steering council, institutional partners, project contributors and developers of course.
Openness & transparency, active contribution, institutional neutrality are key foundations of project governance.

Conclusion
In my own opinion, no governance model is better than the other, in the sense that each model has advantages and disadvantages.
The BQplot governance model has an open decision making process, this encourages more diverse perspectives, however this process could be slow if a decision point is not arrived at in time.
The PySAL governance model on the other hand, because of the BDFL enables a faster decision making in situations where there is a deadlock. However there isn't a diverse range of ideas or perspectives because of the small number of the steering council members (which are three).
In general, the mode of governance would depend on the type of project goals or needs.

Hello @arliss-NF , please kindly review my second contribution. Thank you😃

Hi @Teekafy - great work - your conclusion is solid - I like how you presented your arguments with pros and cons.  Good approach.  Your comment that each project and its community are different from another project is correct. It is important to choose the model that works best for them.

Thank you @arliss-NF ..I really appreciate your review and feedback. I'm glad you liked it😁


Update docs to clarify range of numeric, integer, etc. and meaning of unixTimestamp

fix #2892 Clarify numeric, integer, decimal, float
fix #2887 Clarify unixTimestamp encoding

:x: There is undocummented changes. Run chronus add to add a changeset or click here.
The following packages have changes but are not documented.

:x:@typespec/compiler

You can try these changes at https://cadlplayground.z22.web.core.windows.net/prs/2893/
Check the website changes at https://tspwebsitepr.z22.web.core.windows.net/prs/2893/


How to use multiple quill contexts in the same file

I'm writing ammonite scripts that query a Postgres database, then do analysis on them using Spark DataFrames. I would like to use Quill queries for both databases, but I'm finding it difficult to figure out how to use both contexts within the same script file. When I try to load the PostgresJdbcContext after importing the Quill SparkContext, I get error messages about missing Decoders, or complaints that the lift, quote, or run macros are already defined in the SparkContext. Is there any documentation on how to achieve this?
Thanks!
@getquill/maintainers

Yeah, this is probably the single most difficult thing about query contexts. They're really not into sharing functionality.  Also, when you start doing this, you can't always directly use JdbcContext or SparkContext when doing self types (deep groan) since they are classes and not traits. With out further ado, Let the mind bending begin!
For starters, create a common trait with a bunch of encoders that you can assume exist in both Postgres and Spark. These are the encoders that will be available to your common queries:
trait EncodersProvider {
  implicit val stringEncoder: Encoder[String]
  implicit val bigDecimalEncoder: Encoder[BigDecimal]
  implicit val booleanEncoder: Encoder[Boolean]
  ...
}

Then create the class where you are going to have queries common to Spark and Postegres, whose self type is a Quill Context as well the encoders provider you created earlier.
trait QuillExtensions {
  this: Context[_, _] with EncodersProvider =>

    def personGreaterThenAge(age:Int) =
      (query:Query[Person]) => query.filter(_.age > lift(age))
}

Once you've done that, throw together a bunch of traits in a haphazard way to get contexts. Since these things don't commute (oh Dotty, how we long for thee!) i.e. the order you do 'A with B with C' actually matters, you have to try a couple of ways to make it work properly.
The below recipe should do the trick:
import io.getquill.context.spark.{Decoders, Encoders}

// Have to use a 2-step process with Spark, otherwise it didn't work for me:
trait SparkEncodersProvider extends EncodersProvider with QuillSparkContext with Encoders with Decoders
object SparkQuillContext extends QuillSparkContext with QuillExtensions with SparkEncodersProvider with Encoders with Decoders {}

// Postgres was a bit simpler:
class PostgresQuillContext(dataSource: DataSource with Closeable) extends PostgresJdbcContext[Literal](Literal, dataSource)
  with QuillExtensions with EncodersProvider with io.getquill.context.jdbc.Encoders with io.getquill.context.jdbc.Decoders {}

... and that's all folks! Anything you put inside the QuillExtensions will be available in both the SparkQuillContext and the PostgresJdbcContext, then you pass these contexts down your stack and query stuff from them. Beware that working your way around date logic is tricky but with this pattern you can do it by placing MappedEncodings into your PostgresJdbcContext and SparkQuillContext. Let me know if you want more help with that and I can put up a gist.
@maintainers. It really is a giant pain in the rear to share functionality across different kinds of Quill contexts. This would be greatly simplified by having some minimal trait containing Encoders/Decoders common to all contexts as well as turning things like JdbcContext into traits as opposed to classes. Perhaps even top level context like PostgresJdbcContext or QuillSparkContext should be traits as opposed to classes (that could easily have implementations DefaultPostgresJdbcContext etc...) because that would give us much nicer composition properties when writing custom contexts.

@deusaquilus Thanks for the comprehensive write-up! This seems like a real pain, and I'm not sure how well it would translate to my use-case of writing ad-hoc ammonite REPL scripts. So I'll probably put off trying this out until the maintainers can come up with a simpler way to achieve this. Thanks again!

Why was this issue closed? This is a serious usability issue and should be addressed.

Hi @benwbooth. I know it's been quite a long time but if this issue still interests you, please have a look at my example here. It shows an example of exactly how to do this.

FYI, I implemented a simple solution, I was able to use multiple Quill contexts within the same file like this:
// In any place other than MyRepo.scala
sealed trait DBContext
case object MySqlContext extends DBContext
case object H2Context extends DBContext

// MyRepo.scala
class UserRepo(dbContext: DBContext, config: Config) {

  private lazy val ctx = dbContext match {
    case MySqlContext => new MysqlJdbcContext(SnakeCase, config)
    case H2Context => new H2JdbcContext(SnakeCase, config)
  }

  import ctx._

  def cleanUpTable(): Unit = {
    ctx.run(query[User].delete)
  }

  def findAllUsers(): List[User] = {
    ctx.run(query[User]).map(toNews)
  }

}


[BUG] ERROR: Products do not overlap or were not provided in a contigious sorted list.

Bug
I am processing a timeseries for track 19.
I pruned and checked the data using the ariaPlot.py function, cropping to intersection.

When I apply the ariaTSsetup.py function, it runs for a while and then produces this error message.
NETCDF:"/u/sar-r2/rzinke/Tibet/T19_FullTrack_aria/19_S_Tibet/Sproducts/S1-GUNW-D-R-019-tops-20160517_20150405-001820-32430N_30444N-PP-ee95-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/Tibet/T19_FullTrack_aria/19_S_Tibet/Sproducts/S1-GUNW-D-R-019-tops-20160517_20150405-001845-30936N_28946N-PP-703d-v2_0_2.nc":/science/grids/data/unwrappedPhase is GDAL compatible
NETCDF:"/u/sar-r2/rzinke/Tibet/T19_FullTrack_aria/19_S_Tibet/Sproducts/S1-GUNW-D-R-019-tops-20160517_20150405-001845-30936N_28946N-PP-703d-v2_0_2.nc":/science/grids/data/connectedComponents is GDAL compatible
ERROR: Products do not overlap or were not provided in a contigious sorted list.
Traceback (most recent call last):
  File "/u/sar-r0/rzinke/python/miniconda3/envs/aria/bin/ariaTSsetup.py", line 15, in <module>
    main(inps)
  File "/u/sar-r0/rzinke/python/miniconda3/envs/aria/lib/python3.8/site-packages/ARIAtools/tsSetup.py", line 271, in main
    export_products(standardproduct_info.products[1], standardproduct_info.bbox_file, prods_TOTbbox, layers, dem=demfile, lat=Latitude, lon=Longitude, mask=inps.mask, outDir=inps.workdir, outputFormat=inps.outputFormat, stitchMethodType='overlap', verbose=inps.verbose, num_threads=inps.num_threads, multilooking=inps.multilooking)
  File "/u/sar-r0/rzinke/python/miniconda3/envs/aria/lib/python3.8/site-packages/ARIAtools/extractProduct.py", line 380, in export_products
    product_stitch_overlap(unw_files,conn_files,prod_bbox_files,bounds,prods_TOTbbox, outFileUnw=outFileUnw,outFileConnComp= outFileConnComp, mask=mask,outputFormat = outputFormat,verbose=verbose)
  File "/u/sar-r0/rzinke/python/miniconda3/envs/aria/lib/python3.8/site-packages/ARIAtools/unwrapStitching.py", line 1458, in product_stitch_overlap
    unw.UnwrapOverlap()
  File "/u/sar-r0/rzinke/python/miniconda3/envs/aria/lib/python3.8/site-packages/ARIAtools/unwrapStitching.py", line 340, in UnwrapOverlap
    self.__calculateCyclesOverlap__()
  File "/u/sar-r0/rzinke/python/miniconda3/envs/aria/lib/python3.8/site-packages/ARIAtools/unwrapStitching.py", line 369, in __calculateCyclesOverlap__
    raise Exception
Exception

To Reproduce
Steps to reproduce the behavior:

ariaDownload.py -t 19 -i 20160517_20150405 download products
ariaPlot.py -f "products/*.nc" -plottracks plot to confirm
ariaExtract.py -f "products/*.nc -ml 3 -w ML3" extract quickly
...ERROR: Products do not overlap or were not provided in a contigious sorted list....

Desktop (please complete the following information):

ARIA-tools git tag: b787e82
Linux CentOS

via @sssangha fix in #235


Tests: Fix bech32 test vectors

The bech32 test vectors are still using the bc1 prefix, which produces a test fail.  We should fix this.

Fixed by https://github.com/namecoin/electrum-nmc/pull/21


openssl not installed

Describe the bug
Steps to reproduce the behavior:
  run pkg_config fail: "`\"pkg-config\" \"--libs\" \"--cflags\" \"openssl\"` did not exit successfully: exit status: 1\nerror: could not find system library 'openssl' required by the 'openssl-sys' crate\n\n--- stderr\nPackage openssl was not found in the pkg-config search path.\nPerhaps you should add the directory containing `openssl.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'openssl' found\n"

  --- stderr
  thread 'main' panicked at '

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-gnu
  openssl-sys = 0.9.76

  ', /home/navy/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.76/build/find_normal.rs:191:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace```

**Expected behavior**
Install openssl automatically

**Screenshots or Video**
![image](https://user-images.githubusercontent.com/11570442/225255470-fb179d3f-918f-4eae-8161-fa9af9bb977c.png)

**Environments (please complete the following information):**
 - System info: Linux jia 5.15.0-67-generic #74~20.04.1-Ubuntu SMP Wed Feb 22 14:52:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
 - Dora version:dora-cli 0.2.0

**Additional context**
https://dora.carsmos.ai/dora/installation.html
This is triggered during the execution of the command `cargo build --all --release` abnormal

OK, thanks!


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_4eoj3ifaw6/clone/recipe/meta.yaml:25: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_4eoj3ifaw6/clone/recipe/meta.yaml:31: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_4eoj3ifaw6/clone/recipe/meta.yaml:35: incorrect_license: The recipe's about/license key is not an SPDX compliant license or license exception, reference https://spdx.org/licenses/
ERROR: /tmp/abs_4eoj3ifaw6/clone/recipe/meta.yaml:46: http_url: http://esprima.org/doc/ is not https
Errors were found


IncompletePayment exception doesn't return the subscription record

Cashier Version: 10.3.0
Laravel Version: 6.4
PHP Version: 7.3.9
Database Driver & Version: MySQL 5.7.23

Description:
When creating a subscription I want to continue to fill in custom data even if an IncompletePayment exception is thrown, but I can't because throwing the exception means the subscription record is not returned.
Steps To Reproduce:
try {
    $subscription = $user
        ->newSubscription($subscriptionName, $planId)
        ->withMetadata([
            'user_id' => $user->id,
            'venue_id' => $venue->id
        ])
        ->create($paymentMethod);
} catch (IncompletePayment $e) {
    $confirmPaymentId = $e->payment->id;
}

// The lines below won't work when IncompletePayment is thrown because
// $subscription is null
$subscription->venue_id = $venue->id;
$subscription->photo_upload_limit = 20;

This worked in previous versions, although I skipped 9.3, which is where incomplete payments started to be supported.

Yeah, this is something I thought about as well when I was working on v10. There's a couple of ways to tackle this.
First, you could derive the subscription from the payment object. You'll have an invoice id on the payment object with which in turn will link to a subscription id which you can retrieve the subscription from.
Secondly, you could overwrite the subscription webhook of Cashier where the subscription gets updated so you can sync the meta data with your db record.
But I do believe we could maybe improve the IncompletePayment exception to also contain the subscription itself. I'll leave this open as an enhancement for now because there's ways around it (as described above).

First, you could derive the subscription from the payment object. You'll have an invoice id on the payment object with which in turn will link to a subscription id which you can retrieve the subscription from.

This means doing two more requests, one for the invoice and one for the subscription, right?
I don't really know Stripe's api to be honest, I'll have a deeper look into it.

But I do believe we could maybe improve the IncompletePayment exception to also contain the subscription itself. I'll leave this open as an enhancement for now because there's ways around it (as described above).

This is what I had hoped for. I'll keep an eye here in case you decide to improve it.
Thanks

Actually it only needs one more request to get the invoice, which holds the subscription id.
Here's my solution, if anybody is stuck on the same issue:
use App\Models\Subscription;
use Laravel\Cashier\Cashier;
use Laravel\Cashier\Exceptions\IncompletePayment;
use Stripe\Invoice as StripeInvoice;

$user = auth()->user();

try {

	// Create the subscription
	$subscription = $user
		->newSubscription($subscriptionName, $planId)
		->withMetadata([
			'user_id' => $user->id,
			'venue_id' => $venue->id
		])
		->create($paymentMethod);

} catch (IncompletePayment $e) {

	// Since the exception does not return the subscription (but it is actually
	// created), retrieve it from the looking up the linked invoice
	$invoice = StripeInvoice::retrieve($e->payment->invoice, Cashier::stripeOptions());
	$subscription = Subscription::where('stripe_id', $invoice->subscription)->firstOrFail();

}

// Add custom data to the newly created subscription
$subscription->venue_id = $venue->id;
$subscription->photo_upload_limit = 20;
$subscription->save()

I've been looking into this further but unfortunately have come to the conclusion that there's too many unknowns to properly deal with this. For example, a payment intent doesn't always have an invoice and thus it can't always be linked back to a subscription.
If anyone wants to attempt to tackle this in a clean way feel free to send in a PR.


可以简要在文档中说明一下如何移植linux吗？

@KleinSteven The main thing is actually to write a device tree(.dts) and make rootfs. I will write this document in the near future. Thanks.


Updates for honeyguide

I am getting an error message when running this pull request.
To repeat this error:
first clone this pull request by doing:
git fetch origin +refs/pull/3/merge
git checkout FETCH_HEAD
Then run:

python setup.py install
create DANGER folder
quail redcap generate quail.conf.yaml hcvprod token https://redcap --intialize
quail redcap get_meta hcvprod

At step 4 I get the error:
Traceback (most recent call last):
File "/usr/local/bin/quail", line 11, in
load_entry_point('quail==0.1.1', 'console_scripts', 'quail')()
File "/usr/local/lib/python3.5/dist-packages/quail-0.1.1-py3.5.egg/quail/main.py", line 112, in cli_run
main(args)
File "/usr/local/lib/python3.5/dist-packages/quail-0.1.1-py3.5.egg/quail/main.py", line 94, in main
redcap.get_meta(**conf)
TypeError: get_meta() got an unexpected keyword argument 'batch'
I expected it to say
Pulling metadata for hcvprod
Done pulling metadata for hcvprod


fix: fix return type hint for dedupe()

Contributes to #87.
rapids-dependency-file-generator handles deduplicating the sets of dependencies provided to it, including pip:  lists for conda environments.
This PR modifies the logic the function does that in the following ways:

fixes return type hint
refactors it so that:

only a single pass is done over the inputs to split requirements in to pip vs non-pip
assignments are done in a way that's a bit easier for a type checker to understand

This fixes these errors found by mypy:
src/rapids_dependency_file_generator/_rapids_dependency_file_generator.py:65: error: Argument 1 to "append" of "list" has incompatible type "dict[str, list[str]]"; expected "str"  [arg-type]
src/rapids_dependency_file_generator/_rapids_dependency_file_generator.py:66: error: Incompatible return value type (got "list[str]", expected "list[str | dict[str, str]]")  [return-value]
src/rapids_dependency_file_generator/_rapids_dependency_file_generator.py:66: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
src/rapids_dependency_file_generator/_rapids_dependency_file_generator.py:66: note: Consider using "Sequence" instead, which is covariant
src/rapids_dependency_file_generator/_rapids_dependency_file_generator.py:66: note: Perhaps you need a type annotation for "deduped"? Suggestion: "list[str | dict[str, str]]"
src/rapids_dependency_file_generator/_rapids_dependency_file_generator.py:425: error: Argument "dependencies" to "make_dependency_file" has incompatible type "list[str | dict[str, str]]"; expected "list[str | dict[str, list[str]]]"  [arg-type]

How I tested this
Confirmed that this behavior is covered by existing unit tests (and added one more condition there for completeness).
Ran the following to confirm that some of the mypy issues had been resolved:
mypy \
    --ignore-missing-imports \
    --explicit-package-bases \
    ./src

:tada: This PR is included in version 1.13.6 :tada:
The release is available on:

GitHub release
PyPI release
Conda release - rapidsai
Conda release - rapidsai-nightly

Your semantic-release bot :package::rocket:


sita a  sdfl

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
** Browser (please complete the following information):**

OS / Browser [e.g. Mac / safari]

Additional context
Add any other context about the problem here.

@shadow198 what are you referring to? I'm not familiar.


Promote cloning instead of creating new releases

Cloning has some advantages over creating a new release from scratch.
The documentation for create is updated to link to the clone API.
Some information was missing in the clone doc, it is added now.
JIRA: PDC-355

Looks good to me.

Looks good.


Added support of last name

Added support of last name. If anyone wants to combine initials of first name and last name. Using lastName option (data-last-name) user can add First name and Last name initials

All works great!


TLS Support?

The push to encrypt all network traffic is making it increasingly difficult to operate unencrypted services. Adding TLS support would be very welcome.

TLS is on the horizon, likely part of an update to support SeedLink v4 (spec is currently in review) or shortly thereafter.  The v4 protocol includes authorization, which must be done in TLS.

Relatedly, the SeedLink client library has support for TLS now: https://github.com/EarthScope/libslink
As for ringserver, one option, perhaps even preferable for some, would be to use something like haproxy or stunnel in front of ringserver to handle the TLS termination.  On most linux distros installing and configuring haproxy is near trivial.