Downloads stop and get stuck

This is still happening in the latest version 1.8.0. (and it happened in the version previous to that one).
In most cases the course download speed drops to 0, the pause button becomes deactivated and the download doesn't continue. I have tried deleting the partial course folder and re-starting Udeler, and clicking it to start the download afresh, it resumes and then gets stuck again. I also tried to download one course at a time, and it didn't work either. Out of 50 courses, it happens to the same 9 courses anywhere and in different computers (both MacBooks with High Sierra) and networks (home and work).
In some other downloads, 1 out of 50 courses, a download error appears and an option to retry appears but in most cases it just fails again and again.
Thanks for your effort in creating such a useful tool and for any help to sort this issue please.

Me too I have this problem, Please could you fix it. Thanks

Same here.
Debian 10.2
Udeler 1.8.0
"Download failed. Click to try again."

Same here for a lot of courses.

It appears that udemy has added some kind of digital signing on the request.  Once in a while I will get a 1k "mp4" file.  If I open that with a text editor, I can see a message saying "The request signature we calculated does not match the signature you provided. Check your key and signing method.".  This is followed by a bunch of hex information.  This happens on some but not all courses.

Hi fellas, just to make it clear, I am having two different issues:
Issue 1, the download speed drops to 0 and silently stops without any error, it justs stops and stays stuck. The pause and play buttons become unresponsive.
Issue 2, the error "Download failed. Click to try again" appears and clicking it just restarts the download to thorw the error again.

Me too facing this issue now,inspite of deleting the mtd file, download stops within seconds, it seems Udemy has added some restriction or checks..... @FaisalUmair can you please look into this,thanks

Hi fellas, just to make it clear, I am having two different issues:
Issue 1, the download speed drops to 0 and silently stops without any error, it justs stops and stays stuck. The pause and play buttons become unresponsive.
Issue 2, the error "Download failed. Click to try again" appears and clicking it just restarts the download to thorw the error again.

Yes I am facing the same issue today.

It appears that udemy has added some kind of digital signing on the request. Once in a while I will get a 1k "mp4" file. If I open that with a text editor, I can see a message saying "The request signature we calculated does not match the signature you provided. Check your key and signing method.". This is followed by a bunch of hex information. This happens on some but not all courses.

You are right.
Some courses (not all) have some signature check before sending the video file from the server.

Use the new release of the Udeler which is 18.0.1 it worked for me
https://github.com/FaisalUmair/udemy-downloader-gui/releases

It appears that udemy has added some kind of digital signing on the request. Once in a while I will get a 1k "mp4" file. If I open that with a text editor, I can see a message saying "The request signature we calculated does not match the signature you provided. Check your key and signing method.". This is followed by a bunch of hex information. This happens on some but not all courses.

You are right.
Some courses (not all) have some signature check before sending the video file from the server.

What does that mean? Does that mean it won't work with Udeler?

Whatever the issue was, I believe it was resolved by 1.8.1 update in March.  I haven't had the problem since I updated to that version.  Udeler is working fine for me.

Whatever the issue was, I believe it was resolved by 1.8.1 update in March. I haven't had the problem since I updated to that version. Udeler is working fine for me.

I still get the 1k issue with 1.8.2.

I was only referring to the downloads getting stuck problem.
I have seen mp4s that are 1K.  If you open them in  a text editor, you'll probably see they're actually some kind of error message.  The only thing I've seen in this regard (since the 1.8.1 update) is that my login timed out.  If you have a slow internet and it takes a long time to download a class and it's the items at the end where you see the 1k mp4 files, then it's likely you're login is timing out.  You'll see that the 1k files are authentication errors.  If you are trying to download a lot of classes at once and (for example), let it run all night, then it's likely your login is timing out.  You need to do fewer classes at a time.
One solution for classes you didn't get all the files for is to delete the 1k files, restart udeler and have it redownload the class.  It won't redownload any videos that are already there and so should get to the later videos quicker.  Check other file types where you find the 1k mp4s.  When this happens, I usually find that attachments and subtitle files are also bad (delete them to so udeler will do a redownload).
Udeler does seem to get confused sometimes and redownload some attachments (even if they are already there) when you do a restart, but I just watch for that and delete any duplicate attachments (same name and size with a different number at the beginning).  I consider that a minor problem.
Otherwise, you should post what's in the 1k file (again, open it in a text editor like notepad -- it's not really an mp4 file).

I was only referring to the downloads getting stuck problem.
I have seen mp4s that are 1K. If you open them in a text editor, you'll probably see they're actually some kind of error message. The only thing I've seen in this regard (since the 1.8.1 update) is that my login timed out. If you have a slow internet and it takes a long time to download a class and it's the items at the end where you see the 1k mp4 files, then it's likely you're login is timing out. You'll see that the 1k files are authentication errors. If you are trying to download a lot of classes at once and (for example), let it run all night, then it's likely your login is timing out. You need to do fewer classes at a time.
One solution for classes you didn't get all the files for is to delete the 1k files, restart udeler and have it redownload the class. It won't redownload any videos that are already there and so should get to the later videos quicker. Check other file types where you find the 1k mp4s. When this happens, I usually find that attachments and subtitle files are also bad (delete them to so udeler will do a redownload).
Udeler does seem to get confused sometimes and redownload some attachments (even if they are already there) when you do a restart, but I just watch for that and delete any duplicate attachments (same name and size with a different number at the beginning). I consider that a minor problem.
Otherwise, you should post what's in the 1k file (again, open it in a text editor like notepad -- it's not really an mp4 file).

That's what I've found, but it took a while to figure out. Some classes I've had are really large, like over 30gb, so I've run into it. But as you said, sometimes it gets confused, so I've seen it redownload videos after resuming that are already downloaded, but with a different number. The video hashes are the exact same. When I encounter the 1k issue, I close Udeler, delete the entire chapter with the 1k files, and resume. It doesn't seem to get confused that way.

I've done that, but sometimes I still find duplicate attachments in earlier
chapters.
On Sat, Jun 20, 2020 at 10:34 PM bobbintb<EMAIL_ADDRESS>wrote:

I was only referring to the downloads getting stuck problem.
I have seen mp4s that are 1K. If you open them in a text editor, you'll
probably see they're actually some kind of error message. The only thing
I've seen in this regard (since the 1.8.1 update) is that my login timed
out. If you have a slow internet and it takes a long time to download a
class and it's the items at the end where you see the 1k mp4 files, then
it's likely you're login is timing out. You'll see that the 1k files are
authentication errors. If you are trying to download a lot of classes at
once and (for example), let it run all night, then it's likely your login
is timing out. You need to do fewer classes at a time.
One solution for classes you didn't get all the files for is to delete the
1k files, restart udeler and have it redownload the class. It won't
redownload any videos that are already there and so should get to the later
videos quicker. Check other file types where you find the 1k mp4s. When
this happens, I usually find that attachments and subtitle files are also
bad (delete them to so udeler will do a redownload).
Udeler does seem to get confused sometimes and redownload some attachments
(even if they are already there) when you do a restart, but I just watch
for that and delete any duplicate attachments (same name and size with a
different number at the beginning). I consider that a minor problem.
Otherwise, you should post what's in the 1k file (again, open it in a text
editor like notepad -- it's not really an mp4 file).
That's what I've found, but it took a while to figure out. Some classes
I've had are really large, like over 30gb, so I've run into it. But as you
said, sometimes it gets confused, so I've seen it redownload videos after
resuming that are already downloaded, but with a different number. The
video hashes are the exact same. When I encounter the 1k issue, I close
Udeler, delete the entire chapter with the 1k files, and resume. It doesn't
seem to get confused that way.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/FaisalUmair/udemy-downloader-gui/issues/336#issuecomment-647077700,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA4V7S4XKCY6AIS4SFMV4I3RXWEUVANCNFSM4JQA4AUA
.


get_mac_address from p2pd.netifaces_xtra fails on android.

Hi,
I'm trying to run your library on android, I found out that in this method I get a KeyError:
def get_mac_address(name, netifaces):
    return netifaces.ifaddresses(name)[netifaces.AF_LINK][0]["addr"]

I was able to try the netifaces library in a simple loop I made and I figured out that in android, the right index is not 17, the one coming from netifaces.AF_LINK but 2.
My simple code:
import netifaces
import pprint

for iface in netifaces.interfaces():
	print(iface)
	try:
		pprint.pprint(netifaces.ifaddresses(iface))
		print(netifaces.ifaddresses(iface)[netifaces.AF_LINK])
	except KeyError:
		print(iface, " error")
		continue

For every interface I get error, reading the output of pprint I can see that there is not 17 but typically in number 2 there is a ipv4 address.
I don't know if you are interested on making it work also in android, by the way I thought it could be a good thing to at least tell this to you.
Regards
magowiz

Hello, yes please do keep reporting bugs. My next sprint on the project I plan to try increase test coverage massively and fix more bugs that people report. Aiming for much better stability. I need to setup my test infrastructure better though because when you're trying to target multiple OSes and even Android it needs to be easy to run the tests.
I will get around to fixing your issue eventually. For now I can say: try to call init_p2pd() also before using netifaces stuff. My code isn't really designed to use netifaces directly and network interface stuff is done through the Interface class. To use the Interface class you can call await Interface() to load a default interface. Or for a specific one: await Interface("e.g.: wlan0"). This library is built to use coroutines so it also helps to know that.
Thanks for taking the effort to report this. I'll keep all issues open until they're fixed.

Sorry I realized that you are not searching for ip address in that case but for mac addresses, so the index 2 is wrong because contains IPs. By the way I tried to run this code, putting also init_p2pd() at the beginning, but with no luck:
the code I ran:
from p2pd import *

async def example():
    # Start the default interface.
    init_p2pd()
    i = await Interface()
    #
    # Load additional NAT details.
    # Restrict, random port NAT assumed by default.

    await i.load_nat()
    #
    # Show the interface details.

    print(i)

if __name__ == '__main__':
    async_test(example)

the error:
/storage/emulated/0/Code/1-network-interfaces.py:5: RuntimeWarning: coroutine 'init_p2pd' was never awaited
Coroutine created at (most recent call last)
  File "/storage/emulated/0/Code/1-network-interfaces.py", line 18, in <module>
    async_test(example)
  File "/data/data/com.termux/files/home/.virtualenvs/p2pd/lib/python3.11/site-packages/p2pd/utils.py", line 609, in async_test
    loop.run_until_complete(f())
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 1914, in _run_once
    handle._run()
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/storage/emulated/0/Code/1-network-interfaces.py", line 5, in example
    init_p2pd()
  init_p2pd()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "/storage/emulated/0/Code/1-network-interfaces.py", line 18, in <module>
    async_test(example)
  File "/data/data/com.termux/files/home/.virtualenvs/p2pd/lib/python3.11/site-packages/p2pd/utils.py", line 609, in async_test
    loop.run_until_complete(f())
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/storage/emulated/0/Code/1-network-interfaces.py", line 6, in example
    i = await Interface()
        ^^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/home/.virtualenvs/p2pd/lib/python3.11/site-packages/p2pd/interface.py", line 363, in start
    self.load_if_info()
  File "/data/data/com.termux/files/home/.virtualenvs/p2pd/lib/python3.11/site-packages/p2pd/interface.py", line 238, in load_if_info
    self.mac = get_mac_address(self.name, self.netifaces)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data/com.termux/files/home/.virtualenvs/p2pd/lib/python3.11/site-packages/p2pd/netiface_extra.py", line 30, in get_mac_address
    return netifaces.ifaddresses(name)[netifaces.AF_LINK][0]["addr"]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 17

I was able also to run 2 different native tools to get information about interfaces, ifconfig and iproute2, with ifconfig, in android, you cannot see mac addresses and for what I know it is intended like this by android developer for recent android versions, by the way running ip a the information is there, I don't know for sure which command/implementation netifaces uses in linux/android, but if it uses ifconfig under the hood or same library ifconfig uses, this can be the reason it cannot access mac addresses.
For completeness I provide you the output of both commands:
ifconfig
Warning: cannot open /proc/net/dev (Permission denied). Limited output.
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet <IP_ADDRESS>  netmask <IP_ADDRESS>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)

radio0: flags=579<UP,BROADCAST,RUNNING,ALLMULTI>  mtu 1500
        inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet <IP_ADDRESS>  netmask <IP_ADDRESS>  broadcast <IP_ADDRESS>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)

as you can see there is a warning that inform us about a permission denied and so a limited output
by the way ip a shows this:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 5a:3d:21:d3:d4:9b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::583d:21ff:fed3:d49b/64 scope link
       valid_lft forever preferred_lft forever
3: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip <IP_ADDRESS> brd <IP_ADDRESS>
4: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre <IP_ADDRESS> brd <IP_ADDRESS>
5: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip <IP_ADDRESS> brd <IP_ADDRESS>
8: ip6_vti0@NONE: <NOARP> mtu 1364 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
9: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit <IP_ADDRESS> brd <IP_ADDRESS>
10: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
11: ip6gre0@NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default qlen 1000
    link/gre6 :: brd ::
12: hwsim0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ieee802.11/radiotap 12:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
15: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:15:b2:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fec0::511b:6ec4:893d:7dc7/64 scope site temporary dynamic
       valid_lft 86093sec preferred_lft 14093sec
    inet6 fec0::d3fd:de08:e532:da1b/64 scope site dynamic mngtmpaddr stable-privacy
       valid_lft 86093sec preferred_lft 14093sec
    inet6 fe80::4325:4963:e6e9:9766/64 scope link stable-privacy
       valid_lft forever preferred_lft forever
17: radio0@if11: <BROADCAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 76:6a:b2:aa:57:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global radio0
       valid_lft forever preferred_lft forever
    inet6 fec0::8447:4a4d:3119:3789/64 scope site temporary dynamic
       valid_lft 86093sec preferred_lft 14093sec
    inet6 fec0::746a:b2ff:feaa:5701/64 scope site dynamic mngtmpaddr
       valid_lft 86093sec preferred_lft 14093sec
    inet6 fe80::746a:b2ff:feaa:5701/64 scope link
       valid_lft forever preferred_lft forever

it seems to me that for some interfaces the mac is shown, probably is the one after link/ether
tell me if you suggest something else to do, for what I know/can, I can provide you information about this android experiment.
Regards,
magowiz

Can you show the output of netifaces.ifaddresses("whatever interface you're checking") and also netifaces.IF_LINK? What version of android are you using?

ok,  selected the interface on mine android device that has an address on my wifi, wlan0, so the code I run was: (I think netifaces.IF_LINK was a typo so I used AF_LINK, if I am wrong tell me)
import netifaces
print('first command ====')
print(netifaces.ifadress("wlan0"))
print('second command ====')
print(netifaces.AF_LINK)

and here it is the output
first command ====
{10: [{'addr': 'fe80::c16d:5573:18d4:a244%wlan0', 'netmask': 'ffff:ffff:ffff:ffff::/64'}], 2: [{'addr': '<IP_ADDRESS>', 'netmask': '<IP_ADDRESS>', 'broadcast': '<IP_ADDRESS>'}]}
second command ====
17

I'm using android OS 11

Thanks, that's exactly what I need. I actually spent today working on adding better android support and I think I'm going to have to fork netifaces at this point because all the small patches I've made are adding up. I think this should be easy to fix. Will let you know when it's done.

I'm glad I was able to provide you useful information, thanks for your work, I saw that netifaces (vanilla one https://github.com/al45tair/netifaces , not forked) was archived, so probably if you need to introduce fixes it is a good idea the yours: forking and going on on a fork, so you don't get stuck.
Regards

I've added a fallback for the mac function now to use a different approach if the keys don't exist for netifaces. I need to do more testing on your version of android though.

I'm testing it now on an emulator in android studio and I can see that it may be harder to get the mac than I thought. They seem to lock down android hard. I'll see what I can do.

Hello, I was able to reproduce the bug in an emulator and wrote a fix using a library that uses the netlink API (which has root in the kernel.) Should work better on android. Let me know if it works for you. All tests are passing for me.

Hello, I was able to reproduce the bug in an emulator and wrote a fix using a library that uses the netlink API (which has root in the kernel.) Should work better on android. Let me know if it works for you. All tests are passing for me.

Of course I will give it a try and let you know, how can I try your version with the fix? Did you release it so it is currently available using pip install p2pd? Or do I need to install it from a particular branch in your repository? Maybe doing pip install git://.....?
Then I would try this code, right?
from p2pd import *

async def example():
    # Start the default interface.
    init_p2pd()
    i = await Interface()
    #
    # Load additional NAT details.
    # Restrict, random port NAT assumed by default.

    await i.load_nat()
    #
    # Show the interface details.

    print(i)

if __name__ == '__main__':
    async_test(example)

^ Yep, currently all the new fixes are on the 'test_sprint1' branch. Let me know if you check it out and the issue is fixed. I did try out the test suite on an emulator which showed the lack of mac address in the netiface fields. I think the issue was: in Android access to the MAC address for an interface is locked down and can't be done through standard means. So netifaces was doing that. But there's a kernel-level API that already has full permissions so it bypasses those issues. Hopefully it now works for you.

Hi @robertsdotpm ,
I tried this and for me, except some other errors, on mine android phone is working, I can see mac addresses:
/data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/src.py:5: RuntimeWarning: coroutine 'init_p2pd' was never awaited
Coroutine created at (most recent call last)
  File "/data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/src.py", line 18, in <module>
    async_test(example)
  File "/data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/lib/python3.11/site-packages/p2pd/utils.py", line 616, in async_test
    loop.run_until_complete(f())
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/base_events.py", line 1914, in _run_once
    handle._run()
  File "/data/data/com.termux/files/usr/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/src.py", line 5, in example
    init_p2pd()
  init_p2pd()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
{'id': 'wlan0',
 'is_default': {2: True, 10: False},
 'mac': 'e0-cc-f8-3c-a6-df',
 'name': 'wlan0',
 'nat': {'delta': {'type': 2, 'value': 0},
         'delta_info': 'equal delta (local port == mapped port)',
         'nat_info': 'restrict port',
         'type': 5},
 'nic_no': 0,
 'rp': {2: [{'af': 2,
             'ext_ips': [{'af': 2, 'cidr': 32, 'ip': '<IP_ADDRESS>'}],
             'nic_ips': [{'af': 2, 'cidr': 32, 'ip': '<IP_ADDRESS>'}]}],
        10: []}}
exception handler
{'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-106' coro=<nat_test_workers.<locals>.worker() done, defined at /data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/lib/python3.11/site-packages/p2pd/nat_test.py:154> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
exception handler
{'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-107' coro=<nat_test_workers.<locals>.worker() done, defined at /data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/lib/python3.11/site-packages/p2pd/nat_test.py:154> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
exception handler
{'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-110' coro=<nat_test_workers.<locals>.worker() done, defined at /data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/lib/python3.11/site-packages/p2pd/nat_test.py:154> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
exception handler
{'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-111' coro=<nat_test_workers.<locals>.worker() done, defined at /data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/lib/python3.11/site-packages/p2pd/nat_test.py:154> wait_for=<Future pending cb=[Task.task_wakeup()]>>}
exception handler
{'message': 'Task was destroyed but it is pending!', 'task': <Task pending name='Task-114' coro=<nat_test_workers.<locals>.worker() done, defined at /data/data/com.termux/files/home/.virtualenvs/tmp-ecb045798e88ca4/lib/python3.11/site-packages/p2pd/nat_test.py:154> wait_for=<Future pending cb=[Task.task_wakeup()]>>}

What I've done:

create a new temporary virtualenv using mktmpenv (virtualenvwrapper) to be sure to install all from scratch
install p2pd with pip install git+https://github.com/robertsdotpm/p2pd.git@test_sprint1
run the code I proposed before

Is it ok for you?

It looks like its working but try adding await before the init_p2pd func. That func is a coroutine so you need to await it.

It looks like its working but try adding await before the init_p2pd func. That func is a coroutine so you need to await it.
Edit: The NAT type you have on that device is the same one I have for one of my internet connections. The delta type + NAT type makes the NAT very workable with punching algorithms.

Ok so I changed source like this:
from p2pd import *

async def example():
    # Start the default interface.
    await init_p2pd()
    i = await Interface()
    #
    # Load additional NAT details.
    # Restrict, random port NAT assumed by default.

    await i.load_nat()
    #
    # Show the interface details.

    print(i)

if __name__ == '__main__':
    async_test(example)

and this is the new output (it seems to me similar to one before but without errors):
{'id': 'wlan0',
 'is_default': {2: True, 10: False},
 'mac': 'e0-cc-f8-3c-a6-df',
 'name': 'wlan0',
 'nat': {'delta': {'type': 6, 'value': 0},
         'delta_info': 'random delta (local port == rand port)',
         'nat_info': 'restrict port',
         'type': 5},
 'nic_no': 0,
 'rp': {2: [{'af': 2,
             'ext_ips': [{'af': 2, 'cidr': 32, 'ip': '<IP_ADDRESS>'}],
             'nic_ips': [{'af': 2, 'cidr': 32, 'ip': '<IP_ADDRESS>'}]}],
        10: []}}

also in this case I can see a mac address, so it is working, what do you think?
About nat type and punching algorithms I confess that I don't know so much about that stuff :-D , but I think it should be a positive thing that mine kind of NAT is working with p2pd.
Regards
magowiz

Hello @magowiz
I've edited all your posts to remove the sensitive information (and I didn't save any of your details.) Let me know if I've missed anything. I do agree with your concerns though -- IPs are a scary thing to have public and in the future I'll need a way to solve these issues so people don't have to expose so much information. That will be after I write a lot more tests though.
Thanks for helping me get to the bottom of this. I'll close the issue for now but feel free to re-open it should the need arise. If you find any other bugs let me know and I'll do my best to fix them.


2 Layer LSTM

In the article "LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection" from Malhotra are two LSTM layers (encoder and decoder) mentioned. You refered to that article and you wanted to implement that concept. But I can't see in your RNNPredictor-model that you implement two separate LSTM layer. Do you deviate from that concept intentionally or have I missed something?

Sorry to confuse you. I used only the part to calculate the anomaly score
of the paper.
2018년 12월 31일 (월) 오전 1:36, Tobias128128<EMAIL_ADDRESS>작성:

In the article "LSTM-based Encoder-Decoder for Multi-sensor Anomaly
Detection" from Malhotra are two LSTM layers (encoder and decoder)
mentioned. You refered to that article and you wanted to implement that
concept. But I can't see in your RNNPredictor-model that you implement two
separate LSTM layer. Do you deviate from that concept intentionally or have
I missed something?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/chickenbestlover/RNN-Time-series-Anomaly-Detection/issues/20,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJ2X1DmbCmM-vBPOOc1nWHpeUn2zqII4ks5u-Ot2gaJpZM4Zk_sg
.

Thank you for your response.
Do you think that the architecture in this paper from Malhotra is more powerful?

You mean enc-dec model is better than rnn prediction model? Yes, I assume
that it is because 'reconstruction' is easier than 'prediction'.
2018년 12월 31일 (월) 오후 10:37, Tobias128128<EMAIL_ADDRESS>작성:

Thank you for your response.
Do you think that the architecture in this paper from Malhotra is more
powerful?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/chickenbestlover/RNN-Time-series-Anomaly-Detection/issues/20#issuecomment-450646770,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJ2X1EWnIhnfuxieeLWeag42gEjHvVHFks5u-hMdgaJpZM4Zk_sg
.

I‘ve implemented the lstm autoencoder model in pytorch and it seems to work great with machine data. I‘m now validating the model...

I have read the article 'LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection'.
And interested in implementing the model to test also.
could you please share with me the information(code) in detail.
many thanks

I‘ve implemented the lstm autoencoder model in pytorch and it seems to work great with machine data. I‘m now validating the model...

Did you use a Sequence2Sequence like model for this?


Replace Strings with FastStr

Replaced Strings type with FastStr, removed a lot of to_owned(), some autoformatting.
Pull request checklist:

[x] CHANGELOG.md was updated, if applicable

I'm not convinced this is better, since it adds a dependency (code size, compile times) and there's no benchmark to compare before and after. The fast-str crate also says "What Is It For?
FastStr is better than String as long as it is not often necessary to modify strings and connect strings.", which is something that's done extensively in the parsing code. Unless we can get a benchmark in and it shows to create a huge plus in performance, I'm putting a big question mark on this change.

It's okay, I can benchmark this approach later on. I didn't get about breaking change, because public API is same. Can you suggest some scenarios to bench?

It's okay, I can benchmark this approach later on. I didn't get about breaking change, because public API is same. Can you suggest some scenarios to bench?

I've just not gotten around to it, but a large sample (maybe 10k-100k) actual messages logged from various twitch channels should give a realistic real-use scenario, run through the message parser (parsing to ServerMessage). I estimate the message parsing is the most CPU-intensive part of the library.

It's okay, I can benchmark this approach later on. I didn't get about breaking change, because public API is same. Can you suggest some scenarios to bench?

I've just not gotten around to it, but a large sample (maybe 10k-100k) actual messages logged from various twitch channels should give a realistic real-use scenario, run through the message parser (parsing to ServerMessage). I estimate the message parsing is the most CPU-intensive part of the library.

I am not sure, did I patch parser at all. I confirm that's a lot of to_owned() have been removed. I'll inform about bench results.

Thank you for looking into making a benchmark!


[NDTensors] Fix issue in Tensor constructor when storage involves wrapper types

This is the proposed solution from issue 1438 and should make it possible now to create a copy Tensor/ITensor from a view of another Tensor/ITensor.  I also fixed an additional constructor in UnallocatedArrays that should say UnallocatedX instead of new I am adding a test for that in the unittests now
Fixes #1438.

Can you split off the fixes to UnallocatedArrays into a separate PR?

Looks good, thanks. I'm going to double check that this fixes the issue I originally saw in https://github.com/JanReimers/ITensorMPOCompression.jl/pull/7 once I change that over to constructing the ITensor from the storage, but otherwise this looks good to go.
I see a failure in the Metal tests but maybe that is just a random failure unrelated to this PR?

Ok, I checked that this fixes the issue I was seeing in https://github.com/JanReimers/ITensorMPOCompression.jl/pull/7 so I'll merge once tests pass.


Basic endpoints and authentication for public API

Currently, public REST endpoints are defined in transition-backend in order to access basic transition objects (specifically: paths, nodes, scenarios and available routing modes). Here are a few things to note:

The API routes reuse logic that was originally defined in transitObjects.socketRoutes.ts, and as such they were moved to a new service called TransitObjectsDataHandler.ts. Comments from the original file were also copied over to the new service (and kept in the original file), so it may be worth checking if they're still relevant.
A lot of important logic was moved from the socket routes, it may be necessary to properly verify if everything still works as intended. No logic in itself should've been changed, and we have done some manual tests, but this is still something to keep in mind.
The endpoints for scenarios and available routing modes were mainly added for the QGIS plugin, as it needs that information in order to send a routing request.

Here is what still needs to be done:

The current authentication system is incomplete. As of now, we send the user's credentials in the body of every HTTP request and then verify them with the database before sending a response. This is why every endpoint in the public API is a post endpoint whereas some of them could be get endpoints. We are currently working on a system where the user can request a token from one endpoint (which is saved in the database), and then pass that token in the header of requests to the rest of the API.
There are currently few unit tests in place, and some tests may be broken. Any suggestions as for what should be tested would be appreciated.
We have yet to add OpenAPI documentation for the endpoints.

Côté logique et code, c'est bien beau, il y a quelques petits détails à fixer et des tests à ajouter.
Savez-vous comment je pourrais tester la fonctionnalité sans frontend, genre avec curl. J'ai essayé curl -d "username=u&password=p" -X POST http://localhost:8080/token, mais j'ai un bad request (quoique j'ai pas roulé les migrations, je vais attendre vos modifications, mais je me serais attendu à un 500 et non 400).

@tahini Il faut plutôt faire la commande curl -d "usernameOrEmail=u&password=p" -X POST http://localhost:8080/token, c'est de cette manière que le corps de la requête a été défini dans la stratégie local-login de passport.

En fait, avant de pouvoir merger, pourriez-vous faire un rebase de votre branche sur les changements les plus récents pour vous assurer qu'il n'y a pas de conflits?
Pour faire un rebase sur main:
git checkout main
git pull (en vous assurant que ça pull bien d'upstream et non de votre repo)
git checkout basic-endpoints
git rebase main
git push -f <remote> basic-endpoints


nixos/ksm: rewrite using systemd-tmpfiles

Motivation for this change
This is to get rid of the ugly systemd-udev-settle dependency.
See #73095 for more information.
Things done

[x] Tested via one or more NixOS test (nixosTests.ksm)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

How would moving this to tmpfiles fix the problem? This seems to be a script writing to /sys, possibly requiring these files to be present (and underlying hardware to be appropriately initialized), hence the dependency on udevadm-settle.
Nothing guarantees systemd-tmpfiles runs "late enough" for this to has happened.
Can this be some udev rule instead?

RedHat seems to have some daemon here, provided by qemu: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_tuning_and_optimization_guide/chap-ksm

How would moving this to tmpfiles fix the problem?

It changes nothing in this regard, but I don't think there is a problem to begin with, at least not anymore. I don't see why /sys/kernel/mm/ksm/ shouldn't be available as soon as the kernel is executed.
Try building a system with:
boot.initrd.preDeviceCommands = ''
 echo 1 > /sys/kernel/mm/ksm/run; cat /sys/kernel/mm/ksm/run
'';

this runs in stage 1, before udev is even started and succeeds.
Btw, I stole the tmpfiles rule from https://wiki.archlinux.org/index.php/QEMU#Enabling_KSM.

In that case, I'd prefer if this could stay a shell script, and we just drop the dependency. If we do that, worst case the script will fail (and people will realize). If this is switched to a tmpfiles rule, it might do the right thing every second time, and every other time it'll try to create the underlying file structure by itself ;-)

and every other time it'll try to create the underlying file structure by itself ;-)

According to tmpfiles.d(5) this should never happen: if the file does not exist it should simply fail, but ok I'll keep the script.


Last handler

A simple method do determine if the current handler is the last handler: check if next function exists.
TODO: we should add an example with more than 2 handlers, always using return next(), otherwise we might lose context

Maybe we could create a slightly more complicated app to an example branch. Doesn't make much sense to add it directly to the boilerplate as it just makes more things you need to remove to get started.


Server hangs and crashes when loading dimension

Preface: I am completely new to RF Tools dimension system so please excuse me if I mix up some of the terminology.
Basically I wanted to get started creating new dimensions so I put an Empty Dimension Tab into a Dimension Inscriber without adding any dimlets and put the resulting Realized Dimension Tab in to a Dimension builder. It built properly until 99% when it stopped and I pulled it out (thinking I had to put it in another machine to go there). It seems that at this point the server "crashed" because after about 10 seconds of randomly clicking things I got timed out. I do not have any logs of this happening.
I restarted my server and got back in to the game with a realized dimension tab that had a red tooltip saying somthing like "Dimension information not available". When I put it in the dimension builder it did go to 100% and it started charging my world to about 25 million RF at which point I got curious why my dimension would contain so much RF so I started googling. At this point I was also unable to dial my dimension and I stumbled upon an FAQ entry that suggested I should use 'Teleporter Probe' to access my dimension because the receiver could have been invalidated by a meteor. https://github.com/McJty/RFTools/wiki/RFTools-FAQ
I spawned the Teleporter Probe using OP and teleported myself to dimension 3 (which I'm quite sure is the dimension that I've just created).  The following pastebin shows me giving myself the probe and crashing the server http://pastebin.com/bYpTv9YA
It seems the probe did move me to the new dimension because now the server instantly crashes when I try to log in. The next  pastebin contains part of the console output I've captured during the crash http://pastebin.com/05Vd7BPh
Unfortunately my crash report directory is empty. I could provide you with a copy of my world if this issue is serious. Eitherway please let me know if I can provide anymore useful information.
UPDATE: I did found some crash reports in .json format. This is an example of one http://pastebin.com/tLLADxFz
UPDATE: Like I said in my comment, I fixed my world. I've now also generated a new dimension (plain with a different digit) and everything seems to work fine.

As an aside question is there any way to get my character back to the overworld because currently I cannot play the game anymore. I've tried moving my character in MCedit but I'm not sure I'm doing it correctly :cry:
This is part of the log in my client after I attempt to connect to my server.
http://pastebin.com/H1PdJFpq
I get timed out after a few seconds of 'Downloading terrain...'

Seems the original crash was caused by bluepower. Not sure exactly why though.

I presume it's related to the worldgen bluepower adds then. While I was moving my character out of the corrupt dimension I attempted to have a peek at what's inside but it seems non of the chunks were able to be saved.
I can consistently reproduce the issue in the backup I have with the dimlet that created the corrupted dimension. Outside of that like I mentioned other new dimensions seem to be fine. I could upload the backup of my server/world if you think it would help you track down the issue but I guess it's better to report the problem over at bluepower?

I would start by reporting to bluepower. Let's see how it goes from there

Not sure why [06/10/2015 23:37:34 PM] [Client thread/INFO] [mcjty.varia.Logging]: Dimension information for dimension 3 is missing! would be anything to do with BluePower. It seems clear that there was a crash partway through dimension creation. The appearance of Bluepower in a stacktrace is purely the result of normal world generation. I also note that so far there hasn't been a full log posted, this would be insanely useful, if only to rule out Cauldron as a factor.

Well the dimension information is missing because something went wrong during chunk generation (that was when the bluepower related crash occured) which causes rftools to fail proper creation of the dimension. I should note that bluepower and rftools have worked together in various modpacks already so I'm not sure why this would come up now. A full log would indeed be useful.

Copy of what I posted on the other issue. It seems only one of the logs mentions bluepower in the stacktrace.

Alright I've opened up my log directory and the relevant data seems to be in 3 different files:

fml-server2.log 49,877KB
fml-server3.log  2,744KB
fml-server-latest.log 2,743KB

The 2nd file seems to contain the first crash with no information related to bluepower. The 3rd file contains another crash also without information related to bluepower and the fml-server-latest.log file contains the stacktrace with bluepower in it.

Becase all of these files exceed the maximum filesize allowed on pastebin I've compressed my whole log directory and uploaded it here http://www.timfennis.com/logs.7z and here http://www.timfennis.com/logs.zip


sudo: 1.8.27 -> 1.8.28 (CVE-2019-14287)

Motivation for this change
https://www.sudo.ws/alerts/minus_1_uid.html
Latest sudo CVE allows trivial local privilege escalation by restricted sudoers. Tested that the vulnerability does not work anymore, and through nixos/tests/sudo.nix.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc @edolstra

@GrahamcOfBorg test sudo


G7910 reports Power Wash as ProgramID=44

The problem
Hey, first of all, thanks for all this work. It's amazing! On to the bug: I have the G7910 dishwasher, and it seems to report program id 44 when running the Power Wash program. The integration understands this as glasses_warm. Are you sure that's correct? Or is it maybe possible different dishwashers use different ids?
Once it completes, I'll try running the glasses program and see what it sends then.
Version of miele integration having the issue?
0.1.0
Version of Home Assistant Core having the issue?
2022.11.4
Anything in the logs that might be useful for us?
No response
Additional information
No response

cc @the-ress

@astrandb and I were under the impression that each type of device used a a list of IDs that sometimes contained duplicate entries but none that outright conflicted. That was what Miele integrations for other home automation systems used and we tried to expand the lists from available devices.
Now since his integration has found a wider audience, it appears that this theory is only holds true for older devices. I've been trying to solicit lists program IDs by device on the HA community forums, but with limited response so far. We might need to find another way to model program IDs (personally I'd hate to loose the abstraction level and just display the raw Miele API response, but without a diverse collection of devices with their specific program IDs, there might be no other way.

@the-ress had a great idea to look into the Android app, and we got the following data from it. I'm writing a script to automate reading all the translations.
ProgramID Translation            ShortName
--------- -----------            ---------
        1 Pots & Pans            PRG_INTENSIV
        2 Clean Machine          CAREFUL_CLEANING
        3 Economy                ECO
        5 Normal                 NORMAL
        6 Sensor Wash            AUTOMATIC_EU1_AUS_JAP
        7 Sensor Wash            AUTOMATIC_EU2
        8 Sensor Wash            AUTOMATIC_EU3
        9 EnergySaver            SOLARSPAR
       10 China&Crystal          MILD
       11 ExtraQuiet             EXTRA_QUIET
       12 SaniWash               HYGIENE
       13 QuickPowerWash         QUICK_POWER_WASH
       14 Starch & Cheese        PASTA_PAELLA
       17 Tall items             WITHOUT_BASKET
       19 Glasses - No Rinse Aid SCHOONER_WARM
       21 QuickIntenseWash       QUICK_INTENSE_WASH
       22 Economy                ECO_AUS
       23 Normal                 NORMAL_USA
       24 Rinse & Hold           RINSE_AND_HOLD
       25 Filters                GRATES_AND_FILTERS
       26 QuickIntenseWash       PL_SHORT
       27 Universal              PL_UNIVERSAL
       28 Pots & Pans            PL_INTENSE
       29 China & Crystal        PL_GLAS
       30 SaniWash               PL_HYGIENE
       31 ECO                    PL_ECO_EUR_BE_GB
       32 Plastics               PL_SYNTHETICS
       33 Rinse & Hold           PL_COLD_PRECLEANING
       34 ECO                    PL_ECO_USA_CDN
       35 ECO                    PL_ECO_AUS
       40 EcoCare                CN_ECOCARE
       41 Sensor Wash            CN_INTELLIGENT_WASH
       42 Rice Special           CN_RICE_SPECIAL
       43 Pots & Pans            CN_INTENSIVE_WASH
       44 PowerWash              POWER_WASH

Great! I really hope this initiative will be succesful. However, I am somewhat pessimistic since we have found many small (and bigger) differences between the public API and the internal API that Miele uses for the apps.

Yeah, it's probably not gonna match 100%, but it could fill some gaps.
https://github.com/juliekoubova/miele/tree/xamarin_extract/scripts/output

This is really helpful. I did a half-hearted attempt to poke into the Android code some time ago, but I could not get any useful data.
Do you think it would be possible to extract program strings for other appliances also, washing machines, dryers, etc ?

Interesting that they apparently have loads of country specific programs.

Do you think it would be possible to extract program strings for other appliances also, washing machines, dryers, etc ?

Yup, should be pretty easy! I'll try another one.
Question is how to implement this in a scalable way. I think the best long-term solution would be to have separate keys for program names of different appliances, so we don't accidentally introduce even more conflicts (the JSON suggests something like that, generating all the names with a dishwasher_program_ prefix). But that's obviously a breaking change for people depending on the current names.
I started working on my own integration this weekend (and then found yours and stopped, since you've fixed all the issues I have with the other integration). I was planning to have most of the smarts in the library layer (i.e. pymiele). That was before I fully realized how messy this stuff is :smile: But I still think it would be a good approach, maybe someone could even use it outside Home Assistant.
I'm thinking of an object model like this:
class MieleValue(Generic[T]):
  raw: T | None # converted into native format
  localized: str | None # the localized value from API
  unit: str | None

class MieleDeviceType(Enum):
  GENERIC = 0
  WASHING_MACHINE = 1
  TUMBLE_DRYER = 2
  DISHWASHER = 7
  # ...

class MieleDevice:
  """
  Allows clean access to all the data provided by the API,
  normalizing temperatures, replacing -32768 with None, etc.
  """
  def key(self) -> str
  def model(self) -> str
  def type(self) -> MieleDeviceType
  def temperatures(self) -> list[MieleValue[float]]
  def is_door_open(self) -> bool
  # etc...

and building upon that, there would be a layer of descriptors. Those would express what is currently expressed in the entity descriptors in the integration, only moved to pymiele and we'd have multiple layers of them.
class FridgeFreezerDescriptor(MieleDeviceDescriptor):
  def temperatures(self):
    """Described temperatures get to be enabled by default in HA""
    return [
      TemperatureDescriptor("Temperature"),
      TemperatureDescriptor("Freezer temperature", icon="mdi:freezer"),
    ]

  def DishwasherDescriptor(MieleDeviceDescriptor):
    # ...

  def DishwasherG9710Descriptor(MieleDeviceDescriptor):
    def program_name(self, program_id: DishwasherProgramID):
      if program_id == 44:
        return "dishwasher_program_power_wash"
      # delegate to a more generic implementation
      return self.generic_dishwasher.program_name(program_id)

We would have generic descriptors for each of the device types, and we could then simply add overrides for specific models, or even more granular.
What do you think?

I don't know much about Python idioms, but would it be possible to make the program ID handling more generic (i.e. letting it load some JSON data from description files instead of hardcoding program IDs)? That would make adding new devices much easier.
If you could retrieve program lists for other devices from the Android app, I'd like to make a per-device "database" via a spreadsheet to look for similarities/patterns. Right now, I've got a list compiled from various sources (other integrations, my own devices, ...) but that assumes that each device class has a non-conflicting list and currently does not identify the source for each program ID/name pair. I'll split that up into one spreadsheet for each device class, with columns by device. It can't be all random, can it?

Extracted the laundry resource keys, not the actual values yet... Does anything there look familiar? I don't have a Miele washer 😞  https://github.com/juliekoubova/miele/blob/xamarin_extract/scripts/output/laundry-programs.txt

I don't know much about Python idioms, but would it be possible to make the program ID handling more generic (i.e. letting it load some JSON data from description files instead of hardcoding program IDs)? That would make adding new devices much easier.

Yup, sure, using this approach, we can have data-driven descriptors, too, if it makes sense. Depends how many conflicts there are 🤷🏻♀

Extracted the laundry resource keys, not the actual values yet... Does anything there look familiar? I don't have a Miele washer 😞 https://github.com/juliekoubova/miele/blob/xamarin_extract/scripts/output/laundry-programs.txt

Dop2WmProgram would appear to be the washing machine programs (I have lot of empty slots in my current table, but what is there appears to match what I've pulled together so far.
So we probably have:

Dop2WmProgram: washing machines
Dop2TdProgram: tumble dryer

Not sure about the others. The SemiProfessional*Program lists probably refer to their "professional" line, but are to short to be the full lists. Maybe those are a kind of diffs already?
Interestingly, my washer-dryer seems to follow the washing machine program table, i.e. those programs that I've only found on my device and that are related to the dryer function appear in the Dop2WmProgram table. Some programs in my Tumble Dryer column do not appear in the extracted lists at all. Unfortunately, I have not recorded their origin :(

PS: Have you got any idea @juliekoubova why the extracted IDs are sometimes German and sometimes English within the same group?

why the extracted IDs are sometimes German and sometimes English within the same group?

No idea. Maybe they originally used German, and then decided to switch to English to more easily hire devs who don’t speak German? I don’t think there’s a deeper meaning in that

No idea. Maybe they originally used German, and then decided to switch to English to more easily hire devs who don’t speak German? I don’t think there’s a deeper meaning in that

Ah, OK. I was wondering whether they were an artifact of the extraction process.

Nah, it’s what the strings are referred to in code. I’ll get the actual localized values tomorrow, same as for the dishwasher

@juliekoubova

I started working on my own integration this weekend (and then found yours and stopped, since you've fixed all the issues I have with the other integration). I was planning to have most of the smarts in the library layer (i.e. pymiele). That was before I fully realized how messy this stuff is 😄 But I still think it would be a good approach, maybe someone could even use it outside Home Assistant.
I'm thinking of an object model like this:

I started to develop this integration for the same reasons as you stated above. My philosophy has been to try to keep it simple, to avoid complex interpretations of API data and to follow developer guidelines from the HA core team.
One sentence I chose to follow was API libraries should try to do as little as possible. I have experience from another integration for HA where we put some effort into data modelling and interpretation in the API lib. It works fine but it has required a new release of the library everytime a little change in the data model was made (which could easily been handled in the main integration).
So it has not been a priority to refactor the library or to develop and refine object models.
The main challenge has been to map the functions that Miele exposes through the 3rd party API to the entities and interface elements in Home Assistant - without having access to the different appliances. I own a connected  freezer and a refridgerator while the rest of my appliances are of an older generation - and they never break...
However, it is important to try to display correct programs, program phases, status messages and device types and to have an underlying structure that is easy to maintain when new appliance types or programs pop up. It would have been easier if Miele kept the API updated and used unique numeric codes and always included the clear text names, but that is not how they manage the API.

@astrandb

I did a half-hearted attempt to poke into the Android code some time ago, but I could not get any useful data.

The interesting stuff was hiding in the assemblies.blob file. Here's how to unpack it: https://www.thecobraden.com/posts/unpacking_xamarin_assembly_stores/
(Then you can look inside the assemblies with something like ILSpy.)

Hey, got back to this and added all the laundry program name translations. I also changed the output format of the scripts: there is a single strings_<culture>.json file with all the strings, and individual JSON files mapping the program numbers into keys to those strings. I hope this format will make it easier to use in the integration.
Washers and dryers are separated; there are three "generations" of each. dop1 (also called legacy in the DLLs), dop2, and semipro. We'll have to figure out how to detect the device type, there are conflicts between the numeric values.
 I agree that it can be a challenge to find out what "generation" a device belongs to.
BTW: Have you got an oven? In that case I am interested in a diagnistic dump. There is a new feature regarding food core temperature in API v 1.0.6 that was released last week. It is not documented and Miele doesn't answer any questions, at least not yet.

Nice, good to hear they are adding features. I ran a program that uses the food core temperature sensor, and it seems to be correctly reported:
        "state|targetTemperature|0|value_raw": 8000,
        "state|targetTemperature|0|value_localized": 80.0,
        "state|targetTemperature|0|unit": "Celsius",
        "state|targetTemperature|1|value_raw": -32768,
        "state|targetTemperature|1|value_localized": null,
        "state|targetTemperature|1|unit": "Celsius",
        "state|targetTemperature|2|value_raw": -32768,
        "state|targetTemperature|2|value_localized": null,
        "state|targetTemperature|2|unit": "Celsius",
        "state|coreTargetTemperature|0|value_raw": 5600,
        "state|coreTargetTemperature|0|value_localized": 56.0,
        "state|coreTargetTemperature|0|unit": "Celsius",
        "state|temperature|0|value_raw": 7018,
        "state|temperature|0|value_localized": 70.18,
        "state|temperature|0|unit": "Celsius",
        "state|temperature|1|value_raw": -32768,
        "state|temperature|1|value_localized": null,
        "state|temperature|1|unit": "Celsius",
        "state|temperature|2|value_raw": -32768,
        "state|temperature|2|value_localized": null,
        "state|temperature|2|unit": "Celsius",
        "state|coreTemperature|0|value_raw": 4425,
        "state|coreTemperature|0|value_localized": 44.25,
        "state|coreTemperature|0|unit": "Celsius",

config_entry-miele-7e2a4791b88e377cce5e6e5123804988.json (2).txt


Do collection hooks work with this?

Hello,
I'm thinking of switching to this db because mongodb has the surprising limitation that it cannot create a case-insensitive index.  For my use case, however, I need the collection-hooks package to work.  Will it work with this?

It wouldn't since in our case the it needs to extends CouchDB.Database, not Mongo.Collection. Porting it though should be straight-forward


Triggering OOB Document Location Creation On Record Creation

This is helpful, but some key info is missing.  Dynamics can automatically create both the document location records and the needed folders in Sharepoing using the OOB configuration. This occurs when the user clicks the "Documents" link from the target record.  How can the underlying code (plugin, organization svs request, action) be triggered either through a workflow, custom workflow activity, action, or plugin?  Rather than re-code what the system is doing under the hood, we just need a way to trigger it on record creation, rather than waiting for the user to click "Documents."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: eca590c3-e4d0-d68f-4881-f2d265a5c2b6
Version Independent ID: 41d7cf62-332f-4bbe-4892-a5a3054afa1d
Content: Actions on SharePoint location records (Developer Guide for Dynamics 365 for Customer Engagement)
Content Source: ce/developer/integration-dev/actions-on-sharepoint-location-records.md
Service: crm-online
GitHub Login: @KumarVivek
Microsoft Alias: kvivek

@ryanperrymba Did you find a way of doing what you have asked in your comment around kicking off that code in a simpler way?

Unfortunately, no.  Would love an update from MSFT.


Windows Filepaths With a Drive Specifier Incorrectly Identified as Urls

Hello, this is the first of the other issues that I mentioned in https://github.com/lycheeverse/lychee/issues/1454.
Many tests in the test suite are failing because the Url crate will erroneously parse a windows filebath with a drive specifier as a url. An example of the test results are included below. Note in particular the Network Errors caused by 'URL Scheme is not allowed'.
        FAIL [   0.565s] lychee::example_domains cli::test_do_not_exclude_false_positive_example_domains

--- STDOUT:              lychee::example_domains cli::test_do_not_exclude_false_positive_example_domains ---

running 1 test
test cli::test_do_not_exclude_false_positive_example_domains ... FAILED

failures:

failures:
    cli::test_do_not_exclude_false_positive_example_domains

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.55s

--- STDERR:              lychee::example_domains cli::test_do_not_exclude_false_positive_example_domains ---
thread 'cli::test_do_not_exclude_false_positive_example_domains' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\core\src\ops\function.rs:250:5:
Unexpected failure.
code=1
stderr=``````
Error: Network error

Caused by:
    0: builder error for url (c:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_EXAMPLE_DOMAINS_FALSE_POSITIVES.md)
    1: URL scheme is not allowed

command=`"C:\\Users\\bnhei\\source\\repos\\lychee\\target\\debug\\lychee.exe" "C:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_EXAMPLE_DOMAINS_FALSE_POSITIVES.md" "--include-mail" "--dump"`
code=1
stdout=""
stderr=```
Error: Network error

Caused by:
    0: builder error for url (c:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_EXAMPLE_DOMAINS_FALSE_POSITIVES.md)
    1: URL scheme is not allowed

note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Canceling due to test failure: 7 tests still running
FAIL [   0.574s] lychee::data_uris cli::test_dump_data_uris_in_verbose_mode
--- STDOUT:              lychee::data_uris cli::test_dump_data_uris_in_verbose_mode ---
running 1 test
test cli::test_dump_data_uris_in_verbose_mode ... FAILED
failures:
failures:
cli::test_dump_data_uris_in_verbose_mode
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.56s
--- STDERR:              lychee::data_uris cli::test_dump_data_uris_in_verbose_mode ---
thread 'cli::test_dump_data_uris_in_verbose_mode' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\core\src\ops\function.rs:250:5:
Unexpected failure.
code=1
stderr=``````
Error: Network error
Caused by:
0: builder error for url (c:\Users\bnhei\source\repos\lychee\fixtures\TEST_DATA_URIS.html)
1: URL scheme is not allowed

command="C:\\Users\\bnhei\\source\\repos\\lychee\\target\\debug\\lychee.exe" "C:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_DATA_URIS.html" "--dump" "--verbose"
code=1
stdout=""
stderr=```
Error: Network error
Caused by:
0: builder error for url (c:\Users\bnhei\source\repos\lychee\fixtures\TEST_DATA_URIS.html)
1: URL scheme is not allowed

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.560s] lychee::local_files cli::test_local_file

--- STDOUT:              lychee::local_files cli::test_local_file ---

running 1 test
test cli::test_local_file ... FAILED

failures:

failures:
    cli::test_local_file

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.54s

--- STDERR:              lychee::local_files cli::test_local_file ---
thread 'cli::test_local_file' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\core\src\ops\function.rs:250:5:
Unexpected failure.
code=1
stderr=``````
Error: Network error

Caused by:
    0: builder error for url (c:\\Users\\bnhei\\AppData\\Local\\Temp\\.tmpaYJnxn\\index.html)
    1: URL scheme is not allowed

command=`"C:\\Users\\bnhei\\source\\repos\\lychee\\target\\debug\\lychee.exe" "C:\\Users\\bnhei\\AppData\\Local\\Temp\\.tmpaYJnxn\\index.html" "--no-progress" "--verbose"`
code=1
stdout=""
stderr=```
Error: Network error

Caused by:
    0: builder error for url (c:\\Users\\bnhei\\AppData\\Local\\Temp\\.tmpaYJnxn\\index.html)
    1: URL scheme is not allowed

note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
    FAIL [   0.569s] lychee::local_files cli::test_local_dir

--- STDOUT:              lychee::local_files cli::test_local_dir ---
running 1 test
test cli::test_local_dir ... FAILED
failures:
failures:
cli::test_local_dir
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.55s
--- STDERR:              lychee::local_files cli::test_local_dir ---
thread 'cli::test_local_dir' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\core\src\ops\function.rs:250:5:
Unexpected failure.
code=1
stderr=``````
Error: Network error
Caused by:
0: builder error for url (c:\Users\bnhei\AppData\Local\Temp\.tmpxxgIJG)
1: URL scheme is not allowed

command="C:\\Users\\bnhei\\source\\repos\\lychee\\target\\debug\\lychee.exe" "C:\\Users\\bnhei\\AppData\\Local\\Temp\\.tmpxxgIJG" "--no-progress" "--verbose"
code=1
stdout=""
stderr=```
Error: Network error
Caused by:
0: builder error for url (c:\Users\bnhei\AppData\Local\Temp\.tmpxxgIJG)
1: URL scheme is not allowed

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.593s] lychee::data_uris cli::test_dont_dump_data_uris_by_default

--- STDOUT:              lychee::data_uris cli::test_dont_dump_data_uris_by_default ---

running 1 test
test cli::test_dont_dump_data_uris_by_default ... FAILED

failures:

failures:
    cli::test_dont_dump_data_uris_by_default

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.57s

--- STDERR:              lychee::data_uris cli::test_dont_dump_data_uris_by_default ---
thread 'cli::test_dont_dump_data_uris_by_default' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\core\src\ops\function.rs:250:5:
Unexpected failure.
code=1
stderr=``````
Error: Network error

Caused by:
    0: builder error for url (c:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_DATA_URIS.html)
    1: URL scheme is not allowed

command=`"C:\\Users\\bnhei\\source\\repos\\lychee\\target\\debug\\lychee.exe" "C:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_DATA_URIS.html" "--dump"`
code=1
stdout=""
stderr=```
Error: Network error

Caused by:
    0: builder error for url (c:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_DATA_URIS.html)
    1: URL scheme is not allowed

note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
    FAIL [   0.589s] lychee::example_domains cli::test_exclude_example_domains

--- STDOUT:              lychee::example_domains cli::test_exclude_example_domains ---
running 1 test
test cli::test_exclude_example_domains ... FAILED
failures:
failures:
cli::test_exclude_example_domains
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.56s
--- STDERR:              lychee::example_domains cli::test_exclude_example_domains ---
thread 'cli::test_exclude_example_domains' panicked at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081\library\core\src\ops\function.rs:250:5:
Unexpected failure.
code=1
stderr=``````
Error: Network error
Caused by:
0: builder error for url (c:\Users\bnhei\source\repos\lychee\fixtures\TEST_EXAMPLE_DOMAINS.md)
1: URL scheme is not allowed

command="C:\\Users\\bnhei\\source\\repos\\lychee\\target\\debug\\lychee.exe" "C:\\Users\\bnhei\\source\\repos\\lychee\\fixtures\\TEST_EXAMPLE_DOMAINS.md" "--include-mail" "--dump"
code=1
stdout=""
stderr=```
Error: Network error
Caused by:
0: builder error for url (c:\Users\bnhei\source\repos\lychee\fixtures\TEST_EXAMPLE_DOMAINS.md)
1: URL scheme is not allowed

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        PASS [   0.026s] lychee::bin/lychee commands::check::tests::test_show_cached_responses_in_progress_debug_output
        PASS [   0.021s] lychee::bin/lychee commands::check::tests::test_skip_cached_responses_in_progress_output
        PASS [   0.036s] lychee::bin/lychee commands::check::tests::test_invalid_url
        PASS [   0.025s] lychee::bin/lychee formatters::duration::tests::test_formatting
        PASS [   0.028s] lychee::bin/lychee formatters::response::color::tests::test_format_response_with_error_status
        PASS [   0.028s] lychee::bin/lychee formatters::response::color::tests::test_format_response_with_long_uri
        PASS [   0.023s] lychee::bin/lychee formatters::response::color::tests::test_format_response_with_ok_status
        PASS [   0.022s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_error_status
        PASS [   0.018s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_excluded_status
        PASS [   0.027s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_ok_status
        PASS [   0.023s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_redirect_status
        PASS [   0.022s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_unknown_status_code
        PASS [   0.018s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_error_status
        PASS [   0.018s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_excluded_status
        PASS [   0.017s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_unknown_status_code
        PASS [   0.026s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_ok_status
        PASS [   0.026s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_redirect_status
        PASS [   0.017s] lychee::bin/lychee formatters::stats::markdown::tests::test_markdown_response_cached_err
        PASS [   0.014s] lychee::bin/lychee formatters::stats::markdown::tests::test_markdown_response_cached_ok
        PASS [   0.014s] lychee::bin/lychee formatters::stats::markdown::tests::test_markdown_response_ok
        PASS [   0.015s] lychee::bin/lychee formatters::stats::markdown::tests::test_render_stats
        PASS [   1.078s] lychee::bin/lychee archive::wayback::tests::wayback_suggestion
        PASS [   4.059s] lychee::bin/lychee archive::wayback::tests::wayback_suggestion_unknown_url
------------
     Summary [   4.088s] 29/242 tests run: 23 passed, 6 failed, 0 skipped
        FAIL [   0.593s] lychee::data_uris cli::test_dont_dump_data_uris_by_default
        FAIL [   0.574s] lychee::data_uris cli::test_dump_data_uris_in_verbose_mode
        FAIL [   0.565s] lychee::example_domains cli::test_do_not_exclude_false_positive_example_domains
        FAIL [   0.589s] lychee::example_domains cli::test_exclude_example_domains
        FAIL [   0.569s] lychee::local_files cli::test_local_dir
        FAIL [   0.560s] lychee::local_files cli::test_local_file

I tracked this error to Input::new method.
    pub fn new(
        value: &str,
        file_type_hint: Option<FileType>,
        glob_ignore_case: bool,
        excluded_paths: Option<Vec<PathBuf>>,
    ) -> Result<Self> {
        let source = if value == STDIN {
            InputSource::Stdin
        } else if let Ok(url) = Url::parse(value) { //  <-- HERE
            InputSource::RemoteUrl(Box::new(url))
        } else {
       // file handling here

When I checked this in my debugger, I indeed found that Windows filepaths were being identified as Urls:
[lychee-lib\\src\\types\\input.rs:136:9] url = Url {
    scheme: \"c\", // <-- HERE, scheme is c. The file path that I provided was C://etc/etc
    cannot_be_a_base: true,
    username: \"\",
    password: None,
    host: None,
    port: None,
    path: \"\\\\Users\\\\{my-username}\\\\AppData\\\\Local\\\\Temp\\\\.tmpS5lXlI\",
    query: None,
    fragment: None,
}

Short Term Solution
In my investigation, I found that the reqwest crate's RequestBuilder struct will error if you try to construct a request using such a url, so I made a minor, if inelegant modification to the function:
    pub fn new(
        value: &str,
        file_type_hint: Option<FileType>,
        glob_ignore_case: bool,
        excluded_paths: Option<Vec<PathBuf>>,
    ) -> Result<Self> {
        let source = if value == STDIN {
            InputSource::Stdin
        } else if Request::builder().uri(value).body(()).is_ok() {
            let url = Url::parse(value)
                .map_err(|err| crate::ErrorKind::ParseUrl(err, value.to_owned()))?;
            InputSource::RemoteUrl(Box::new(url))
        } else {

Something better could be worked out in the future, but for now this does the trick. Will submit a PR for this soon.
In any case, with the above change, our test output on Windows changes to this:
    Starting 242 tests across 6 binaries (2 skipped; run ID: efc6a829-22fc-4822-8bc4-f04effc0b282, nextest profile: default)
        PASS [   0.558s] lychee::example_domains cli::test_do_not_exclude_false_positive_example_domains
        PASS [   0.569s] lychee::data_uris cli::test_dont_dump_data_uris_by_default
        PASS [   0.559s] lychee::example_domains cli::test_exclude_example_domains
        PASS [   0.562s] lychee::local_files cli::test_local_file
        PASS [   0.573s] lychee::local_files cli::test_local_dir
        PASS [   0.022s] lychee::bin/lychee commands::check::tests::test_show_cached_responses_in_progress_debug_output
        PASS [   0.595s] lychee::data_uris cli::test_dump_data_uris_in_verbose_mode
        PASS [   0.028s] lychee::bin/lychee commands::check::tests::test_skip_cached_responses_in_progress_output
        PASS [   0.027s] lychee::bin/lychee formatters::duration::tests::test_formatting
        PASS [   0.025s] lychee::bin/lychee formatters::response::color::tests::test_format_response_with_error_status
        PASS [   0.048s] lychee::bin/lychee commands::check::tests::test_invalid_url
        PASS [   0.029s] lychee::bin/lychee formatters::response::color::tests::test_format_response_with_long_uri
        PASS [   0.023s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_error_status
        PASS [   0.018s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_excluded_status
        PASS [   0.028s] lychee::bin/lychee formatters::response::color::tests::test_format_response_with_ok_status
        PASS [   0.029s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_ok_status
        PASS [   0.025s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_unknown_status_code
        PASS [   0.029s] lychee::bin/lychee formatters::response::emoji::emoji_tests::test_format_response_with_redirect_status
        PASS [   0.025s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_error_status
        PASS [   0.029s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_excluded_status
        PASS [   0.029s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_ok_status
        PASS [   0.024s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_redirect_status
        PASS [   0.019s] lychee::bin/lychee formatters::response::plain::plain_tests::test_format_response_with_unknown_status_code
        PASS [   0.029s] lychee::bin/lychee formatters::stats::markdown::tests::test_markdown_response_cached_err
        PASS [   0.029s] lychee::bin/lychee formatters::stats::markdown::tests::test_markdown_response_cached_ok
        PASS [   0.020s] lychee::bin/lychee formatters::stats::markdown::tests::test_render_stats
        PASS [   0.025s] lychee::bin/lychee formatters::stats::markdown::tests::test_markdown_response_ok
        PASS [   0.029s] lychee::bin/lychee formatters::stats::markdown::tests::test_render_summary
        PASS [   0.025s] lychee::bin/lychee parse::tests::test_parse_custom_headers
        PASS [   0.030s] lychee::bin/lychee options::tests::test_accept_status_codes
        PASS [   0.024s] lychee::bin/lychee stats::tests::test_detailed_stats
        PASS [   0.028s] lychee::bin/lychee parse::tests::test_parse_remap
        PASS [   0.021s] lychee::bin/lychee stats::tests::test_stats_is_empty
        PASS [   0.026s] lychee::bin/lychee stats::tests::test_stats
        PASS [   0.020s] lychee::bin/lychee verbosity::test::verify_app
        PASS [   0.025s] lychee::bin/lychee verbosity::test::test_default_log_level
        PASS [   0.027s] lychee-lib basic_auth::tests::test_basic_auth_extractor_matches
        PASS [   0.023s] lychee-lib basic_auth::tests::test_basic_auth_extractor_matches_multiple
        PASS [   0.028s] lychee-lib basic_auth::tests::test_basic_auth_extractor_no_match
        PASS [   0.032s] lychee-lib basic_auth::tests::test_basic_auth_extractor_new
        PASS [   0.034s] lychee-lib basic_auth::tests::test_basic_auth_first_match_wins
        PASS [   0.030s] lychee-lib basic_auth::tests::test_basic_auth_regex_match
        PASS [   0.031s] lychee-lib chain::test::early_exit_chain
        PASS [   0.031s] lychee-lib chain::test::simple_chain
        PASS [   0.030s] lychee-lib client::tests::test_avoid_reqwest_panic
        PASS [   0.026s] lychee-lib client::tests::test_exclude_mail_by_default
        PASS [   0.039s] lychee-lib client::tests::test_chain
        PASS [   0.032s] lychee-lib client::tests::test_file
        PASS [   1.021s] lychee::bin/lychee archive::wayback::tests::wayback_suggestion
        PASS [   0.032s] lychee-lib client::tests::test_include_mail
        PASS [   0.020s] lychee-lib client::tests::test_include_tel
        PASS [   1.194s] lychee::bin/lychee archive::wayback::tests::wayback_suggestion_unknown_url
        PASS [   0.025s] lychee-lib client::tests::test_limit_max_redirects
        PASS [   0.434s] lychee-lib client::tests::test_github_nonexistent_repo
        PASS [   0.470s] lychee-lib client::tests::test_github_nonexistent_file
        PASS [   0.031s] lychee-lib client::tests::test_max_redirects
        PASS [   0.032s] lychee-lib client::tests::test_non_github
        PASS [   0.516s] lychee-lib client::tests::test_exponential_backoff
        PASS [   0.034s] lychee-lib client::tests::test_nonexistent
        PASS [   0.023s] lychee-lib client::tests::test_unsupported_scheme
        PASS [   0.661s] lychee-lib client::tests::test_github
        PASS [   0.019s] lychee-lib collector::tests::test_collect_html_links
        PASS [   0.016s] lychee-lib collector::tests::test_collect_html_srcset
        PASS [   0.029s] lychee-lib collector::tests::test_collect_links
        PASS [   0.019s] lychee-lib collector::tests::test_collect_markdown_links
        PASS [   0.018s] lychee-lib collector::tests::test_email_with_query_params
        PASS [   0.237s] lychee-lib client::tests::test_youtube
        PASS [   0.021s] lychee-lib collector::tests::test_extract_html5_not_valid_xml_relative_links
        PASS [   0.019s] lychee-lib collector::tests::test_file_without_extension_is_plaintext
        PASS [   0.823s] lychee-lib client::tests::test_custom_headers
        PASS [   0.020s] lychee-lib collector::tests::test_markdown_internal_url
        PASS [   0.022s] lychee-lib collector::tests::test_relative_url_with_base_extracted_from_input
        PASS [   0.017s] lychee-lib extract::html::html5ever::tests::test_email_false_positive
        PASS [   0.020s] lychee-lib extract::html::html5ever::tests::test_exclude_email_without_mailto
        PASS [   0.017s] lychee-lib extract::html::html5ever::tests::test_exclude_script_tags
        PASS [   0.017s] lychee-lib extract::html::html5ever::tests::test_include_nofollow
        PASS [   0.016s] lychee-lib extract::html::html5ever::tests::test_include_verbatim
        PASS [   0.016s] lychee-lib extract::html::html5ever::tests::test_include_verbatim_recursive
        PASS [   0.019s] lychee-lib extract::html::html5ever::tests::test_skip_verbatim
        PASS [   0.017s] lychee-lib extract::html::html5ever::tests::test_valid_email
        PASS [   0.015s] lychee-lib extract::html::html5ever::tests::test_valid_tel
        PASS [   0.017s] lychee-lib extract::html::html5gum::tests::test_email_false_positive
        PASS [   0.017s] lychee-lib extract::html::html5gum::tests::test_exclude_email_without_mailto
        PASS [   0.015s] lychee-lib extract::html::html5gum::tests::test_exclude_script_tags
        PASS [   0.015s] lychee-lib extract::html::html5gum::tests::test_extract_fragments
        PASS [   0.016s] lychee-lib extract::html::html5gum::tests::test_extract_srcset
        PASS [   0.016s] lychee-lib extract::html::html5gum::tests::test_include_nofollow
        PASS [   0.014s] lychee-lib extract::html::html5gum::tests::test_include_verbatim
        PASS [   0.014s] lychee-lib extract::html::html5gum::tests::test_include_verbatim_nested
        PASS [   0.014s] lychee-lib extract::html::html5gum::tests::test_skip_verbatim
        PASS [   0.018s] lychee-lib extract::html::html5gum::tests::test_valid_email
        PASS [   0.016s] lychee-lib extract::html::html5gum::tests::test_valid_tel
        PASS [   0.017s] lychee-lib extract::html::srcset::tests::test_collect_sequence_characters_with_all_match
        PASS [   0.015s] lychee-lib extract::html::srcset::tests::test_collect_sequence_characters_with_alphabetic_predicate
        PASS [   0.016s] lychee-lib extract::html::srcset::tests::test_collect_sequence_characters_with_digit_predicate
        PASS [   0.016s] lychee-lib extract::html::srcset::tests::test_collect_sequence_characters_with_empty_string
        PASS [   0.017s] lychee-lib extract::html::srcset::tests::test_collect_sequence_characters_with_no_match
        PASS [   0.017s] lychee-lib extract::html::srcset::tests::test_parse_no_value
        PASS [   0.016s] lychee-lib extract::html::srcset::tests::test_parse_srcset_two_values
        PASS [   0.016s] lychee-lib extract::html::srcset::tests::test_parse_srcset_url
        PASS [   0.016s] lychee-lib extract::html::srcset::tests::test_parse_srcset_with_commas
        PASS [   0.018s] lychee-lib extract::html::srcset::tests::test_parse_srcset_with_unencoded_comma
        PASS [   0.018s] lychee-lib extract::html::srcset::tests::test_parse_url_one_value
        PASS [   0.015s] lychee-lib extract::html::tests::test_is_email_link
        PASS [   0.015s] lychee-lib extract::html::tests::test_verbatim_matching
        PASS [   0.016s] lychee-lib extract::markdown::tests::test_extract_fragments
        PASS [   0.014s] lychee-lib extract::markdown::tests::test_include_verbatim
        PASS [   0.014s] lychee-lib extract::markdown::tests::test_kebab_case
        PASS [   0.016s] lychee-lib extract::markdown::tests::test_skip_verbatim
        PASS [   0.016s] lychee-lib extract::plaintext::tests::test_extract_link_at_end_of_line
        PASS [   0.015s] lychee-lib extract::plaintext::tests::test_extract_local_links
        PASS [   0.015s] lychee-lib extract::tests::relative_urls
        PASS [   0.017s] lychee-lib extract::tests::test_extract_html5_custom_elements
        PASS [   0.017s] lychee-lib extract::tests::test_extract_html5_lowercase_doctype
        PASS [   0.015s] lychee-lib extract::tests::test_extract_html5_malformed
        PASS [   0.016s] lychee-lib extract::tests::test_extract_html5_minified
        PASS [   0.018s] lychee-lib extract::tests::test_extract_html5_not_valid_xml
        PASS [   0.018s] lychee-lib extract::tests::test_extract_link_at_end_of_line
        PASS [   0.016s] lychee-lib extract::tests::test_extract_relative_url
        PASS [   0.018s] lychee-lib extract::tests::test_extract_urls_with_at_sign_properly
        PASS [   0.018s] lychee-lib extract::tests::test_file_type
        PASS [   0.016s] lychee-lib extract::tests::test_md_escape
        PASS [   0.015s] lychee-lib extract::tests::test_non_markdown_links
        PASS [   0.015s] lychee-lib extract::tests::test_skip_markdown_anchors
        PASS [   0.017s] lychee-lib extract::tests::test_skip_markdown_email
        PASS [   0.015s] lychee-lib extract::tests::test_skip_markdown_internal_urls
        PASS [   0.015s] lychee-lib extract::tests::verbatim_elem
        PASS [   0.016s] lychee-lib filter::tests::test_const_sanity
        PASS [   0.021s] lychee-lib filter::tests::test_exclude_include_regex
        PASS [   0.020s] lychee-lib filter::tests::test_exclude_ip_v4_mapped_ip_v6_not_supported
        PASS [   0.014s] lychee-lib filter::tests::test_exclude_link_local
        PASS [   0.506s] lychee-lib collector::tests::test_url_without_extension_is_html
        PASS [   0.018s] lychee-lib filter::tests::test_exclude_loopback
        PASS [   0.016s] lychee-lib filter::tests::test_exclude_loopback_ips
        PASS [   0.016s] lychee-lib filter::tests::test_exclude_private_ips
        PASS [   0.022s] lychee-lib filter::tests::test_exclude_mail_by_default
        PASS [   0.024s] lychee-lib filter::tests::test_exclude_regex
        PASS [   0.021s] lychee-lib filter::tests::test_excludes_no_private_ips_by_default
        PASS [   0.021s] lychee-lib filter::tests::test_false_positives
        PASS [   0.020s] lychee-lib filter::tests::test_include_mail
        PASS [   0.024s] lychee-lib filter::tests::test_include_regex
        PASS [   0.025s] lychee-lib filter::tests::test_includes_and_excludes_empty
        PASS [   0.023s] lychee-lib filter::tests::test_overwrite_false_positives
        PASS [   0.020s] lychee-lib quirks::tests::test_cratesio_request
        PASS [   0.023s] lychee-lib quirks::tests::test_no_quirk_applied
        PASS [   0.911s] lychee-lib client::tests::test_require_https
        PASS [   0.023s] lychee-lib quirks::tests::test_non_video_youtube_url_untouched
        PASS [   0.026s] lychee-lib quirks::tests::test_youtube_video_request
        PASS [   0.029s] lychee-lib quirks::tests::test_youtube_video_shortlink_request
        PASS [   0.029s] lychee-lib remap::tests::test_remap
        PASS [   0.026s] lychee-lib remap::tests::test_remap_capture_group
        PASS [   0.025s] lychee-lib remap::tests::test_remap_named_capture
        PASS [   0.024s] lychee-lib remap::tests::test_remap_named_capture_shorthand
        PASS [   0.024s] lychee-lib remap::tests::test_remap_path
        PASS [   0.023s] lychee-lib remap::tests::test_remap_skip
        PASS [   0.022s] lychee-lib remap::tests::test_remap_url_to_file
        PASS [   0.023s] lychee-lib types::accept::range::test::test_from_str::case_1
        PASS [   0.022s] lychee-lib types::accept::range::test::test_from_str::case_2
        PASS [   0.023s] lychee-lib types::accept::range::test::test_from_str::case_3
        PASS [   0.022s] lychee-lib types::accept::range::test::test_from_str::case_4
        PASS [   0.018s] lychee-lib types::accept::range::test::test_from_str::case_5
        PASS [   0.019s] lychee-lib types::accept::range::test::test_from_str_invalid::case_1
        PASS [   0.020s] lychee-lib types::accept::range::test::test_from_str_invalid::case_2
        PASS [   0.024s] lychee-lib types::accept::range::test::test_from_str_invalid::case_3
        PASS [   0.024s] lychee-lib types::accept::range::test::test_from_str_invalid::case_4
        PASS [   0.026s] lychee-lib types::accept::range::test::test_from_str_invalid::case_6
        PASS [   1.559s] lychee-lib client::tests::test_basic_auth
        PASS [   0.031s] lychee-lib types::accept::range::test::test_from_str_invalid::case_5
        PASS [   0.026s] lychee-lib types::accept::range::test::test_from_str_invalid::case_8
        PASS [   0.031s] lychee-lib types::accept::range::test::test_from_str_invalid::case_7
        PASS [   0.022s] lychee-lib types::accept::range::test::test_from_str_invalid::case_9
        PASS [   0.022s] lychee-lib types::accept::range::test::test_merge::case_1
        PASS [   0.023s] lychee-lib types::accept::range::test::test_merge::case_2
        PASS [   0.025s] lychee-lib types::accept::range::test::test_merge::case_3
        PASS [   0.026s] lychee-lib types::accept::range::test::test_merge::case_4
        PASS [   0.026s] lychee-lib types::accept::selector::test::test_deserialize::case_1
        PASS [   0.026s] lychee-lib types::accept::selector::test::test_deserialize::case_2
        PASS [   0.028s] lychee-lib types::accept::selector::test::test_deserialize::case_3
        PASS [   0.026s] lychee-lib types::accept::selector::test::test_deserialize::case_4
        PASS [   0.025s] lychee-lib types::accept::selector::test::test_deserialize::case_5
        PASS [   0.021s] lychee-lib types::accept::selector::test::test_deserialize::case_7
        PASS [   0.030s] lychee-lib types::accept::selector::test::test_deserialize::case_6
        PASS [   0.025s] lychee-lib types::accept::selector::test::test_display::case_1
        PASS [   0.026s] lychee-lib types::accept::selector::test::test_display::case_2
        PASS [   0.021s] lychee-lib types::accept::selector::test::test_from_str::case_2
        PASS [   0.030s] lychee-lib types::accept::selector::test::test_from_str::case_1
        PASS [   0.025s] lychee-lib types::accept::selector::test::test_from_str::case_3
        PASS [   0.024s] lychee-lib types::accept::selector::test::test_from_str::case_4
        PASS [   0.023s] lychee-lib types::base::test_base::test_get_base_from_url
        PASS [   0.023s] lychee-lib types::base::test_base::test_invalid_url
        FAIL [   0.023s] lychee-lib types::base::test_base::test_valid_local

--- STDOUT:              lychee-lib types::base::test_base::test_valid_local ---

running 1 test
test types::base::test_base::test_valid_local ... FAILED

failures:

failures:
    types::base::test_base::test_valid_local

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 205 filtered out; finished in 0.00s

--- STDERR:              lychee-lib types::base::test_base::test_valid_local ---
Error: InvalidBase("C:\\Users\\bnhei\\AppData\\Local\\Temp\\.tmp96nQiF", "The given URL cannot be a base")

   Canceling due to test failure: 7 tests still running
        PASS [   0.018s] lychee-lib types::base::test_base::test_valid_remote
        PASS [   0.017s] lychee-lib types::basic_auth::selector::test::test_missing_credentials_basic_auth_selector
        PASS [   0.022s] lychee-lib types::basic_auth::selector::test::test_empty_basic_auth_selector
        PASS [   0.023s] lychee-lib types::basic_auth::selector::test::test_missing_uri_basic_auth_selector
        PASS [   0.024s] lychee-lib types::basic_auth::selector::test::test_too_many_parts_basic_auth_selector
        PASS [   0.025s] lychee-lib types::basic_auth::selector::test::test_valid_basic_auth_selector
        PASS [   0.026s] lychee-lib types::cache::tests::test_deserialize_cache_status_blank
        PASS [   1.378s] lychee-lib client::tests::test_invalid_ssl
        PASS [   0.025s] lychee-lib types::cache::tests::test_deserialize_cache_status_error_code
        PASS [   0.021s] lychee-lib types::cache::tests::test_deserialize_cache_status_excluded
        PASS [   0.021s] lychee-lib types::cache::tests::test_deserialize_cache_status_success_code
        PASS [   0.021s] lychee-lib types::cache::tests::test_deserialize_cache_status_unsupported
        PASS [   0.021s] lychee-lib types::file::tests::test_extension
        PASS [   2.070s] lychee-lib client::tests::test_nonexistent_with_path
        PASS [   7.119s] lychee-lib client::tests::test_timeout
------------
     Summary [   8.415s] 206/242 tests run: 205 passed, 1 failed, 2 skipped
        FAIL [   0.023s] lychee-lib types::base::test_base::test_valid_local
error: test run failed

As you can see, there still remain some issues, but this still gets us a long way.

Closing this as it should be fixed in master with #1460 by @toadslop.


Use passive event listeners

Use passive event listeners to improve perf
https://developers.google.com/web/updates/2016/06/passive-event-listeners

I wonder if that applies to this tho. I get Lighthouse warning about this, but they could be false positives 🤔


Some Menu changes and rename of modules

Rename Query as "Query & Discussions" and shift it to Project menu section
Custom field & Tax settings shift to Settings Menu section
Automation shifts to Other menu and Other menu Rename as "Automation & Settings"


[4º] - Release/v2.0.0 arc42 update: Updated the Arc42 and created the data retrieval guide

Why we create this PR?
The Arc42 was outdated and requires a new structure of archives.
Also several diagram are outdated and need to be fixed.
A Data retrieval guide needs to be provided to explain in detail how the DPP returns information.
What we want to achieve with this PR?
We want to update the Arc42 with the Authentication, new Workflow and create a new Data Retrieval Guide
What is new?
Added

New Data Retrieval Guide added with detailed information how to retrieve data in Catena-X like the DPP.
Added new structure of files for the Arc42
Added new Authentication & Authorization chapter in the Arc42
Added detailed description next to the API diagrams at the Arc42
Added IRS Integration documentation at the Arc42
Added diagrams in xml form for further edition

Updated

Updated the Install.md file with a section to install the prerequisites and configure values.yaml files
Updated the file structure from the Arc42
Updated data retrieval api diagrams with better quality
Updated context diagram with a new easy context
Updated Arc42 Descriptions in the Technical Integration chapter

The ammount of files is because documentation was update and images were deleted!


bug: Attempt to index local 'range' (a nil value)

I have been trying to sync/update LuaSnip now for quite some time with Lazy but every time I try to the checkout fails:

This is my setup in lazy:

Does anybody know what might be causing this, and how I can fix this.
Neovim version: 0.9.0

Yeah makes, sense thanks and sorry.


[QUESTION] Possible GC issues with HarfBuzzSharp.Blob.FromStream

Hi,
I have a question regarding the HarfBuzzSharp.Blob.FromStream implementation, which I pasted here for convenience:
isn't is necessary to pin the data for the whole Blob lifetime to avoid memory relocation by the GC?
public static unsafe Blob FromStream (Stream stream)
{
	// TODO: check to see if we can avoid the second copy (the ToArray)

	using (var ms = new MemoryStream ()) {
		stream.CopyTo (ms);
		var data = ms.ToArray ();

		fixed (byte* dataPtr = data) {
			return new Blob ((IntPtr)dataPtr, data.Length, MemoryMode.ReadOnly, () => ms.Dispose ());
		}
	}
}

Can confirm this is actually a problem. At best all codepoints in Buffer.GlyphInfos becomes 0, at worst AccessViolationException will be thrown.
For anyone else also encountering this nasty bug, a workaround is to allocate the blob's memory yourself with Marshal.AllocHGlobal, then instantiate the Blob with public Blob (IntPtr data, int length, MemoryMode mode, ReleaseDelegate releaseDelegate).


Correctly set autoplay/loop options in vimeo's embed URL when the source URL doesn't have a query string

Currently, if the source URL doesn't have a query string (https://vimeo.com/277464886 for example), and we set autoplay and/or loop to true, they are not being added to the iframe's URL.

Coverage remained the same at 74.487% when pulling a5b1a76a4e88964e6e3df45ebb71ee7ad9dbcc4b on andrefgneves:fix_vimeo-options-when-no-query-string into 6d665634da067ac075cd49631e646c7b16f25510 on mediaelement:master.

@andrefgneves Many thank for your PR, may I ask to remove build/renderers/vimeo.js and build/renderers/vimeo.min.js from the PR ?

Done. Sorry about that.


Navigation in TEST does not match TRAIN

Navigation bar in testing panel  not clickable.
Left and right arrows should be changed to PREVIOUS NEXT (to match navigation in training step)

Do we prefer the navigation arrows or text buttons?
Also, the bar isn't clickable in training either; only the information page has a clickable navigation bar. Do we want all bars to be clickable?

I'm a button fan. (I think the arrows will go wonky on mobile)
Yeah I see the the navigation buttons aren't clickable now...would be great if they were...(text and icon)

yes currently there are 3 different kinds of progression bars. let's just use one of them consistently (and yes text&icon should be clickable too)
no strong preference for arrows or prev/next buttons, but let's be consistent (and make them prominent enough so people never miss the ability)

forgot to make the icons clickable, reopening this accordingly