Object count in HASS integration disagrees with Frigate camera debug view

Version of the custom_component

Integration Version: 2.1.0
Configuration

{
  "birdseye": {
    "enabled": true,
    "height": 720,
    "mode": "objects",
    "quality": 8,
    "width": 1280
  },
  "cameras": {
    "den_camera": {
      "best_image_timeout": 60,
      "detect": {
        "enabled": true,
        "fps": 5,
        "height": 720,
        "max_disappeared": 25,
        "width": 1280
      },
      "ffmpeg": {
        "global_args": [
          "-hide_banner",
          "-loglevel",
          "warning"
        ],
        "hwaccel_args": [
          "-c:v",
          "h264_v4l2m2m"
        ],
        "input_args": [
          "-avoid_negative_ts",
          "make_zero",
          "-fflags",
          "+genpts+discardcorrupt",
          "-rtsp_transport",
          "tcp",
          "-stimeout",
          "5000000",
          "-use_wallclock_as_timestamps",
          "1"
        ],
        "inputs": [
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtsp://<redacted>:<redacted>@<IP_ADDRESS>:8080/h264_pcm.sdp",
            "roles": [
              "detect",
              "rtmp"
            ]
          }
        ],
        "output_args": {
          "detect": [
            "-f",
            "rawvideo",
            "-pix_fmt",
            "yuv420p"
          ],
          "record": [
            "-f",
            "segment",
            "-segment_time",
            "10",
            "-segment_format",
            "mp4",
            "-reset_timestamps",
            "1",
            "-strftime",
            "1",
            "-c",
            "copy",
            "-an"
          ],
          "rtmp": [
            "-c",
            "copy",
            "-f",
            "flv"
          ]
        }
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://<redacted>:<redacted>@<IP_ADDRESS>:8080/h264_pcm.sdp -c copy -f flv rtmp://<IP_ADDRESS>/live/den_camera -r 5 -s 1280x720 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "rtmp"
          ]
        }
      ],
      "live": {
        "height": 720,
        "quality": 8
      },
      "motion": {
        "contour_area": 99,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 180,
        "mask": "",
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "quality": 70,
        "required_zones": [],
        "timestamp": true
      },
      "name": "den_camera",
      "objects": {
        "filters": {
          "person": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": "",
        "track": [
          "person"
        ]
      },
      "record": {
        "enabled": false,
        "events": {
          "max_seconds": 300,
          "objects": null,
          "post_capture": 5,
          "pre_capture": 5,
          "required_zones": [],
          "retain": {
            "default": 10,
            "objects": {}
          }
        },
        "retain_days": 0
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": true,
        "clean_copy": true,
        "crop": false,
        "enabled": false,
        "height": null,
        "quality": 70,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {}
        },
        "timestamp": false
      },
      "timestamp_style": {
        "color": {
          "blue": 255,
          "green": 255,
          "red": 255
        },
        "effect": null,
        "format": "%m/%d/%Y %H:%M:%S",
        "position": "tl",
        "thickness": 2
      },
      "zones": {}
    },
    "front_door_camera": {
      "best_image_timeout": 60,
      "detect": {
        "enabled": true,
        "fps": 5,
        "height": 720,
        "max_disappeared": 25,
        "width": 1280
      },
      "ffmpeg": {
        "global_args": [
          "-hide_banner",
          "-loglevel",
          "warning"
        ],
        "hwaccel_args": [
          "-c:v",
          "h264_v4l2m2m"
        ],
        "input_args": [
          "-avoid_negative_ts",
          "make_zero",
          "-fflags",
          "+genpts+discardcorrupt",
          "-rtsp_transport",
          "tcp",
          "-stimeout",
          "5000000",
          "-use_wallclock_as_timestamps",
          "1"
        ],
        "inputs": [
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtsp://<redacted>:<redacted>@<IP_ADDRESS>:554/videoMain",
            "roles": [
              "detect",
              "rtmp",
              "record"
            ]
          }
        ],
        "output_args": {
          "detect": [
            "-f",
            "rawvideo",
            "-pix_fmt",
            "yuv420p"
          ],
          "record": [
            "-f",
            "segment",
            "-segment_time",
            "10",
            "-segment_format",
            "mp4",
            "-reset_timestamps",
            "1",
            "-strftime",
            "1",
            "-c",
            "copy",
            "-an"
          ],
          "rtmp": [
            "-c",
            "copy",
            "-f",
            "flv"
          ]
        }
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://<redacted>:<redacted>@<IP_ADDRESS>:554/videoMain -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/front_door_camera-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://<IP_ADDRESS>/live/front_door_camera -r 5 -s 1280x720 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "rtmp",
            "record"
          ]
        }
      ],
      "live": {
        "height": 720,
        "quality": 8
      },
      "motion": {
        "contour_area": 99,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 180,
        "mask": "",
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "quality": 70,
        "required_zones": [],
        "timestamp": true
      },
      "name": "front_door_camera",
      "objects": {
        "filters": {
          "airplane": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bicycle": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "car": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cat": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "dog": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "motorcycle": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "person": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 20000,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "truck": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": "",
        "track": [
          "person",
          "bicycle",
          "car",
          "truck",
          "motorcycle",
          "airplane",
          "cat",
          "dog"
        ]
      },
      "record": {
        "enabled": true,
        "events": {
          "max_seconds": 300,
          "objects": null,
          "post_capture": 5,
          "pre_capture": 10,
          "required_zones": [],
          "retain": {
            "default": 8,
            "objects": {
              "cat": 3,
              "dog": 3,
              "person": 15
            }
          }
        },
        "retain_days": 0
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": true,
        "clean_copy": true,
        "crop": false,
        "enabled": false,
        "height": null,
        "quality": 70,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {}
        },
        "timestamp": false
      },
      "timestamp_style": {
        "color": {
          "blue": 255,
          "green": 255,
          "red": 255
        },
        "effect": null,
        "format": "%m/%d/%Y %H:%M:%S",
        "position": "tl",
        "thickness": 2
      },
      "zones": {}
    }
  },
  "database": {
    "path": "/media/frigate/frigate.db"
  },
  "detect": {
    "enabled": true,
    "fps": 5,
    "height": 720,
    "max_disappeared": null,
    "width": 1280
  },
  "detectors": {
    "coral": {
      "device": "usb",
      "num_threads": 3,
      "type": "edgetpu"
    }
  },
  "environment_vars": {},
  "ffmpeg": {
    "global_args": [
      "-hide_banner",
      "-loglevel",
      "warning"
    ],
    "hwaccel_args": [
      "-c:v",
      "h264_v4l2m2m"
    ],
    "input_args": [
      "-avoid_negative_ts",
      "make_zero",
      "-fflags",
      "+genpts+discardcorrupt",
      "-rtsp_transport",
      "tcp",
      "-stimeout",
      "5000000",
      "-use_wallclock_as_timestamps",
      "1"
    ],
    "output_args": {
      "detect": [
        "-f",
        "rawvideo",
        "-pix_fmt",
        "yuv420p"
      ],
      "record": [
        "-f",
        "segment",
        "-segment_time",
        "10",
        "-segment_format",
        "mp4",
        "-reset_timestamps",
        "1",
        "-strftime",
        "1",
        "-c",
        "copy",
        "-an"
      ],
      "rtmp": [
        "-c",
        "copy",
        "-f",
        "flv"
      ]
    }
  },
  "live": {
    "height": 720,
    "quality": 8
  },
  "logger": {
    "default": "info",
    "logs": {}
  },
  "model": {
    "height": 320,
    "labelmap": {},
    "labelmap_path": null,
    "path": null,
    "width": 320
  },
  "motion": null,
  "mqtt": {
    "client_id": "<redacted>",
    "host": "core-mosquitto",
    "password": <redacted>,
    "port": 1883,
    "stats_interval": 60,
    "tls_ca_certs": null,
    "tls_client_cert": null,
    "tls_client_key": null,
    "tls_insecure": null,
    "topic_prefix": "frigate",
    "user": "mqtt"
  },
  "objects": {
    "filters": null,
    "mask": "",
    "track": [
      "person",
      "bicycle",
      "car",
      "truck",
      "motorcycle",
      "airplane",
      "cat",
      "dog"
    ]
  },
  "record": {
    "enabled": false,
    "events": {
      "max_seconds": 300,
      "objects": null,
      "post_capture": 5,
      "pre_capture": 5,
      "required_zones": [],
      "retain": {
        "default": 10,
        "objects": {}
      }
    },
    "retain_days": 0
  },
  "rtmp": {
    "enabled": true
  },
  "snapshots": {
    "bounding_box": true,
    "clean_copy": true,
    "crop": false,
    "enabled": false,
    "height": null,
    "quality": 70,
    "required_zones": [],
    "retain": {
      "default": 10,
      "objects": {}
    },
    "timestamp": false
  },
  "timestamp_style": {
    "color": {
      "blue": 255,
      "green": 255,
      "red": 255
    },
    "effect": null,
    "format": "%m/%d/%Y %H:%M:%S",
    "position": "tl",
    "thickness": 2
  }
}

Describe the bug
After upgrading to Frigate HASS Integration v2.1.0 (via HACS) and Frigate NVR Add-on 2.0 (Frigate 0.9.1-800f33e), I notice that the object count sensor for at least one of my cameras doesn't reflect what Frigate is detecting.  I have a camera called "den_camera" that's detecting people.  sensor.den_camera_person will often report 0 objects even though viewing the debug view of den_camera in the Frigate web UI does show an orange box around a detected person.
Note that I said "often report 0".  Sometimes the sensor does report correctly.
This was working pretty flawlessly before upgrading to the new version.

Blue boxes represent lost objects during tracking. They mark the last known bounding box for an object. You will see them stick around for max_disappeared frames.

Try watching the mqtt topic to see if it also seems off. If so, this would be a frigate issue rather than an integration issue. The sensor here is a fairly simple mqtt sensor, so I suspect this is not a bug in the integration.

Valid hypothesis.


One bad query in connect_to_postgres results in future failures

If one query has an error, the subsequent queries to postgres will fail. We either need to open a new connection for each query or we need to do a rollback on error.

this PR should fix it, #129 !


Make the on-screen keys optional

All functionality is available using the hardware keys, so they don't need to be displayed.
The only reason they are not optional is because Back does not work properly as Backspace in all applications, so there must be an alternative to the hardware key. However, after completing #35 and #29, it would be possible to customize the hardware key and the on-screen one will be completely obsolete.
For the reasons above:

Add a "Show on-screen controls" checkbox on the Settings screen. The option controls whether the soft keys are displayed or not.
Change the title of the option if it does not fit well on the screen.

After #35

Would it be possible to have the on-screen keys mapped to the builtin menu that some flip phones have on the bottom of every screen? I'm, not sure what these widget menu things are called in code, but here's an example image of how my default keyboard uses these menus for character set selection, settings, and emojis.

These are definitely not standard Android widgets. Probably your phone manifacturer has chosen to add these for convenience when you use its default IME.
I really doubt it is possible to wire Traditional T9 to them. And if it is, it would have to be a custom version that would work only on your phone.
Instead, I would recommend disabling the widgets, if possible. If not, just wait for the next release, where you would be able to disable the Traditional T9 on-screen buttons and use only your phone's native ones.

Hm, These widgets are not only for the keyboard, almost every app I open has them on the bottom of the screen, and usually, at least one of the spaces is filled with the entry 'Menu' which does nothing when pressed on most apps.
For some more context, the widget is not touch, it's mapped to the four 'function keys' on the left and right side of my dpad, and the centre button is my OK button on the dpad. The menu does work in the preinstalled apps on my phone so I have a hunch it's something similar to the (probably deprecated) options/menu button functionality that many phones had in the past. I also don't think it's possible to disable this widget :/
My phone is a Kyocera KYF31 on android 5.1, but I know another person who uses a Japanese flip-phone of a different brand and his phone uses these menus as well. Anyways, this isn't a priority- I'll look into it on my own time and if I find a way to use it maybe ill contribute! Thanks!

I see. It sounds they are just labels on the screen that indicate what the keys do. And if we are talking about hardware keys, it is not so difficult to map them to TT9 functions. Since the next release, all key shortcuts will be customizable. Currently, I have added support for CALL, BACK, MENU, # and *, but adding any other keys is trivial.
Here is how it will look:

If, even after that, you still find the settings are not flexible enough, just open a new issue and I should be able to improve them with a little help from your side (since the phone is in your hands).

I see. It sounds they are just labels on the screen that indicate what the keys do.
Yeah, that sounds about right. I found a way to hide them too.

That settings menu looks great! The keys that I described earlier aren't common, but here are keycodes that will probably be consistent across most Japanese phones and you could code them in and label them as Function 1 through 4:
Func 1: 131 (top left)
Func 2: 132 (top right)
Func 3: 133 (bottom left)
Func 4: 134 (bottom right)
Another option for the future could be to have a 'custom' option and prompt users to press a key to bind it, though that would take longer to implement.

If, even after that, you still find the settings are not flexible enough, just open a new issue and I should be able to improve them with a little help from your side (since the phone is in your hands).

Will do! My flipphone-owning friend and I are very grateful. I'll have him do tests on his phone as well so we can get a better idea about compatibility.

Another option for the future could be to have a 'custom' option and prompt users to press a key to bind it, though that would take longer to implement.

I considered this too, but it seemed a bit more challenging, so I decided not to go this way for now.

Func 1: 131 (top left)
Func 2: 132 (top right)
Func 3: 133 (bottom left)
Func 4: 134 (bottom right)

According to the official manual, these are recognized as F1, F2, F3, F4 (like on a computer keyboard). I have already added support for F1 and F2 and I can easily add F3 and F4, as well. However, I am unsure if I did it properly, because I don't have these keys on my phone. I will need you two to check if they appear as possible options when you go to the Settings screen.
I can build a beta verion .apk for you test. Or, if you prefer, build it on your own from the new-settings-screen branch. Have in mind you may need to uninstall and reinstall the app, if your phone sees the .apk as incompatible. This will erase all your settings and custom dictionary words (if any).
How would you like to proceed?

A beta apk build sounds great! I'll probably end up using the f3 and f4 keys, so if you add them to the beta I can test everything at once.
I don't have any custom words saved, so I'm fine to uninstall and reinstall. Let me know when the release is up and I'll try it out!

I came across this problem, but it doesn't happen every time or it happens under very specific circumstances. It may fail when you try to open the settings directly from the Android Settings, but I am not sure.
Could you please try again while you are in another application where you can type?

Ah oops I forgot that I could just hold star to access settings. The remapping seems to be working perfectly, aside from that settings bug it's all looking good. Thanks!

Ah oops I forgot that I could just hold star to access settings. The remapping seems to be working perfectly, aside from that settings bug it's all looking good. Thanks!

So, I take F1-F4 buttons are working fine and I don't need to work on them anymore. That's great!

One thing I am noticing is that my back button functions strangely when tt9 is enabled.

Now that you mentioned it, I think there may be some leftover code that could tamper with the button function. I'll double check.
Thank you very much for the feedback, it is most useful!

That seems to work! no more issues with the back button after a quick test.
I am having another problem though, with loading the English dictionary. (Since the last beta, I didn't test for very long and I figured i was cancelling the loading accidentally by letting my phone fall asleep or exiting the menu screen)
Loading a dictionary takes a while, but I can't tell if it's done or not after trying to load it. Prediction mode doesn't show any suggestions with real words, it just shows groups of letters based off the keys I've pressed. I think prediction is working because when I add a word manually it shows up in the prediction bar.
Update: while writing this my phone has been trying to load the dictionary for more than 20 minutes now, I think there may be a bug here too.

That seems to work! no more issues with the back button after a quick test.

Great! I'll give it a couple more days and if don't find anything that requires code changes, we can pronounce the next release ready!

I am having another problem though, with loading the English dictionary.

That's an unrelated problem. Let's continue the discussion here.


HA 0.96

Hi,
the component seems not working well with HA 0.96.
Any suggestions?
Thanks,
Manuel

Please, describe more information. Here is a template:
Version of the custom_component
If you are not using the newest version, download and try that before opening an issue
If you are unsure about the version check the const.py file.
Configuration

Add your configs here.

Describe the bug
A clear and concise description of what the bug is.
Debug log
To enable debug logs check this https://www.home-assistant.io/components/logger/

Add your logs here.


Mac M1 support

I was getting this error:
  • Installing wgpu (0.3.0): Failed

  EnvCommandError

  Command ['/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/bin/pip', 'install', '--no-deps', 'file:///Users/anentropic/Library/Caches/pypoetry/artifacts/aa/f3/b2/8caef6980405490715f8040d2349bddb320b3962d15c24f0525b66d749/wgpu-0.3.0.tar.gz'] errored with the following return code 1, and output:
  Processing /Users/anentropic/Library/Caches/pypoetry/artifacts/aa/f3/b2/8caef6980405490715f8040d2349bddb320b3962d15c24f0525b66d749/wgpu-0.3.0.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-req-build-99e_f9gh/setup.py'"'"'; __file__='"'"'/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-req-build-99e_f9gh/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-pip-egg-info-23tylv6i
           cwd: /private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-req-build-99e_f9gh/
      Complete output (5 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-req-build-99e_f9gh/setup.py", line 6, in <module>
          from wheel.pep425tags import get_platform
      ModuleNotFoundError: No module named 'wheel.pep425tags'

https://nomodulenamed.com/m/wheel.pep425tags says:

This is probably because you don't have package wheel installed.

But:
Python 3.9.2 (default, Mar 18 2021, 20:48:06)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from wheel.pep425tags import get_platform
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'wheel.pep425tags'
>>> import wheel
>>> wheel.__version__
'0.36.2'
>>> exit()

https://wheel.readthedocs.io/en/stable/news.html

0.35.0 (2020-08-13)
Switched to the packaging library for computing wheel tags

...sounded possibly relevant. The previous version to that is 0.34.2
which was also mentioned on the nomodulenamed page
And yes, this fixed it:
% pip install wheel==0.34.2
Collecting wheel==0.34.2
  Downloading wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Installing collected packages: wheel
  Attempting uninstall: wheel
    Found existing installation: wheel 0.36.2
    Uninstalling wheel-0.36.2:
      Successfully uninstalled wheel-0.36.2
Successfully installed wheel-0.34.2
% python
Python 3.9.2 (default, Mar 18 2021, 20:48:06)
[Clang 12.0.0 (clang-1<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import wheel
>>> wheel.__version__
'0.34.2'
>>> from wheel.pep425tags import get_platform
>>> exit()

and poetry install then succeeded.
I'm not totally sure where the problem originates, whether you need to pin wheel==0.34.2 in your setup.py or some other part of the build machinery is to blame.
(Pinning wheel==0.34.2 in my pyproject.toml didn't help, because Poetry didn't know about the dependency relationship and tries to install wgpu first... so for now it's fixed by manually installing old wheel version in my virtualenv)
Just posting this in case it helps someone else.

Following that:
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/backends/rs.py", line 108, in _get_wgpu_lib_path
    raise RuntimeError(f"Could not find WGPU library in {embedded_path}")
RuntimeError: Could not find WGPU library in /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/resources/libwgpu_native.dylib

Bypassing Poetry does not seem to help
% pip uninstall wgpu
Found existing installation: wgpu 0.3.0
Uninstalling wgpu-0.3.0:
  Would remove:
    /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/LICENSE
    /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu-0.3.0.dist-info/*
    /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/*
Proceed (y/n)? y
  Successfully uninstalled wgpu-0.3.0

% pip install wgpu
Collecting wgpu
  Downloading wgpu-0.3.0.tar.gz (65 kB)
     |████████████████████████████████| 65 kB 2.0 MB/s
Requirement already satisfied: cffi>=1.10 in /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages (from wgpu) (1.14.5)
Requirement already satisfied: pycparser in /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages (from cffi>=1.10->wgpu) (2.20)
Building wheels for collected packages: wgpu
  Building wheel for wgpu (setup.py) ... done
  Created wheel for wgpu: filename=wgpu-0.3.0-py3-none-macosx_11_2_arm64.whl size=72787 sha256=ef24b3c659660431881b441788b1205d7b90eec89f534f95a2f04ff468a0c040
  Stored in directory: /Users/anentropic/Library/Caches/pip/wheels/65/61/c8/553073b0633ba01220ede3798da3293ff8b054a5445ab2d218
Successfully built wgpu
Installing collected packages: wgpu
Successfully installed wgpu-0.3.0

% python experiments/pygfx_hexes.py
Traceback (most recent call last):
  File "/Users/anentropic/Documents/Dev/Personal/python-hexblade/experiments/pygfx_hexes.py", line 6, in <module>
    import wgpu.backends.rs  # noqa: F401, Select Rust backend
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/backends/rs.py", line 119, in <module>
    _lib = ffi.dlopen(_get_wgpu_lib_path())
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/backends/rs.py", line 108, in _get_wgpu_lib_path
    raise RuntimeError(f"Could not find WGPU library in {embedded_path}")
RuntimeError: Could not find WGPU library in /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/resources/libwgpu_native.dylib

% ls /Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/resources
__init__.py	__pycache__	webgpu.idl	wgpu.h

Looks like you installed the source distribution there which indeed does not contain prebuilt binaries:
Downloading wgpu-0.3.0.tar.gz (65 kB) (not a .whl file)
I'll investigate this further later on. The python packaging peeps have been making breaking changes to pip and wheel lately, we may have to adjust our setup.py file.

Perhaps it's because I'm on an M1 mac...
I thought I'd try installing more manually. The first step is python download-wgpu-native.py
Which downloads https://github.com/gfx-rs/wgpu-native/releases/download/v0.7.0/wgpu-macos-64-release.zip
But then I get:
no suitable image found.  Did find:
	/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/resources/libwgpu_native.dylib: mach-o, but wrong architecture
	/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/resources/libwgpu_native.dylib: mach-o, but wrong architecture.  Additionally, ctypes.util.find_library() did not manage to locate a library called '/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/resources/libwgpu_native.dylib'

I then followed the instructions to build gfx-rs/wgpu-native from source:
https://github.com/gfx-rs/wgpu/wiki/Getting-Started#getting-started
I copied the wgpu.h and libwgpu_native.dylib that I built into my virtualenv site-packages/wgpu/resources/ dir
But now I get:
Traceback (most recent call last):
  File "/Users/anentropic/Documents/Dev/Personal/python-hexblade/experiments/pygfx_hexes.py", line 6, in <module>
    import wgpu.backends.rs  # noqa: F401, Select Rust backend
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/backends/rs.py", line 117, in <module>
    ffi.cdef(_get_wgpu_h())
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/api.py", line 112, in cdef
    self._cdef(csource, override=override, packed=packed, pack=pack)
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/api.py", line 126, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/cparser.py", line 389, in parse
    self._internal_parse(csource)
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/cparser.py", line 396, in _internal_parse
    self._process_macros(macros)
  File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/cparser.py", line 479, in _process_macros
    raise CDefError(
cffi.CDefError: only supports one of the following syntax:
  #define WGPUFeatures_DEPTH_CLAMPING ...     (literally dot-dot-dot)
  #define WGPUFeatures_DEPTH_CLAMPING NUMBER  (with NUMBER an integer constant, decimal/hex/octal)
got:
  #define WGPUFeatures_DEPTH_CLAMPING (uint64_t)1

my mistake, I just downloaded latest gfx-rs/wgpu-native but I should have grabbed 0.5.2... I'll try that

Ok I built wgpu.h and libwgpu_native.dylib again from v0.5.2 tag fo wgpu-native
Different error this time:
Traceback (most recent call last):
  File "/Users/paul/Documents/Dev/Personal/python-hexblade/experiments/pygfx_hexes.py", line 6, in <module>
    import wgpu.backends.rs  # noqa: F401, Select Rust backend
  File "/Users/paul/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/wgpu/backends/rs.py", line 117, in <module>
    ffi.cdef(_get_wgpu_h())
  File "/Users/paul/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/api.py", line 112, in cdef
    self._cdef(csource, override=override, packed=packed, pack=pack)
  File "/Users/paul/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/api.py", line 126, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/Users/paul/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/cparser.py", line 389, in parse
    self._internal_parse(csource)
  File "/Users/paul/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/cparser.py", line 396, in _internal_parse
    self._process_macros(macros)
  File "/Users/paul/Library/Caches/pypoetry/virtualenvs/hexblade-gi2zk5D5-py3.9/lib/python3.9/site-packages/cffi/cparser.py", line 479, in _process_macros
    raise CDefError(
cffi.CDefError: only supports one of the following syntax:
  #define WGPUBufferUsage_MAP_READ ...     (literally dot-dot-dot)
  #define WGPUBufferUsage_MAP_READ NUMBER  (with NUMBER an integer constant, decimal/hex/octal)
got:
  #define WGPUBufferUsage_MAP_READ (uint32_t)1

Well there's multiple issues, but you trying this out on an M1 is definitely the biggest one :) it's a new build target and we have not looked into supporting it just yet. There are no CI runners available and I don't know anyone (except for you of course!) that owns one so it will be tricky indeed. If you manage to compile the right version of wgpu-native yourself it should all just work though

the last comment I posted was with 0.5.2 wgpu-native
wgpu-native itself works ok on the Rust side... I can run the make run-example-triangle test and a GLFW window opens up and draws a triangle

Did you also update the header file? The last error you posted is actually a failure to parse/load the wgpu.h header file from cffi.

🤔   I was sure I had but I will double-check that

my grep returns:
#define WGPUBufferUsage_MAP_READ (uint32_t)1

I have double-checked and this is what is built by wgpu-native 0.5.2 on my machine

There are no CI runners available

See also https://github.com/actions/virtual-environments/issues/2187

Different error this time

The only thing I can think of is that there is a mismatch between the header-file and the compiled lib.
Otherwise, with a bit of luck things will be better once we've moved to a newer version of wgpu-native ...

I'm a bit out of my depth here, but let me know if I can help test or check anything else

I can confirm that, with the latest wgpu-py and wgpu-native build from source everything is working on M1 with a couple of caveats:

wgpu-native asks for a version of bindgen that just plain doesn't work on M1. The latest bindgen works fine.
rs_helpers's get_surface_id_from_canvas doesn't seem to be able to recognize GLFWWindow. I replace this hacking-around-libobjc-with-ctypes code with my own hacking-around-libobjc-with-ctypes code and everything now works. I haven't figured out the differences yet.

Would you mind listing the specific versions and code adjustments you've used? It might be just what we need to propose the appropriate changes upstream.

brew install llvm is yielding an 12.0.0_1 llvm, which let's you set LLVM_CONFIG_PATH=/opt/homebrew/Cellar/llvm/12.0.0_1/bin/llvm-config and then
bindgen='0.58.1' inside wpgu-native's Cargo.toml gets it's cargo build to work.

@marcdownie would you be interested in submitting a PR to https://github.com/gfx-rs/wgpu-native for the required changes to get it compiling? Might also be the easiest way to discuss the get_surface_id_from_canvas.

Pull request for getting wgpu-native to build on M1 here: https://github.com/gfx-rs/wgpu-native/pull/114
My angry hacks to get get_surface_id_from_canvas working is harder to build a pull request from, not least of all because it's dependent on some random gist I found (https://gist.github.com/tlinnet/746a18788dd51f0827fb4840b9a8631c) which, at the very least, doesn't have a license.
I think the difference here is turning on calling methods (like contentView()). The existing get_surface_id_from_canvas is using a raw objc_msgSend while I'm using a method returned from class_getInstanceMethod(objc_class, someSelector) and calling that.
Specifically, I have:
cv = ObjCInstance(window).contentView()
cv.setWantsLayer(True)
metal_layer = ObjCClass("CAMetalLayer").layer()
cv.setLayer(metal_layer)

To replace the existing:
content_view = objc.objc_msgSend(window, content_view_sel)
...
objc.objc_msgSend(content_view, set_wants_layer_sel, True)
ca_metal_layer_class = objc.objc_getClass(b"CAMetalLayer")
metal_layer = objc.objc_msgSend(ca_metal_layer_class, layer_sel)
objc.objc_msgSend(content_view, set_layer_sel, ctypes.c_void_p(metal_layer))

My code works where get_surface_id_from_canvas fails because on M1 objc.objc_msgSend(window, responds_to_sel_sel, ctypes.c_void_p(content_view_sel)) isn't True when it clearly should be. objc.objc_msgSend is, famously, coupled directly to the ABI.
Meanwhile, I'm trying to inline everything from that gist so that I might actually have code you'd want in your repository, but it might take a few days for me to get to it.

Thanks for the effort! Would be great to get that code in, so others with an M1 can benefit as well :)

What is happening on this front? Im a Mac M1 user and would like to use wgpu-py. Any progress?

Looks like the mentioned https://github.com/gfx-rs/wgpu-native/pull/114 has been merged, so I imagine that it's at least possible to build for M1 now, but it doesn't look like there are prebuilt binaries yet: https://github.com/gfx-rs/wgpu-native/releases/tag/v<IP_ADDRESS>
So I guess the next step would be adjusting CI over there to provide those. Then we can easily add support here as a next step.

I just created https://github.com/gfx-rs/wgpu-native/issues/138 to track step 1.

Step 1 is done. Once there is a release of wgpu-native, we can do what's needed in wpu-py. @Korijn if you feel like starting with that, you could use the unofficial release on my fork.

What would need to be done in wgpu-py to add support? Just curious and eager to contribute

What would need to be done in wgpu-py to add support? Just curious and eager to contribute

Open a branch.
Run the script as documented at the end of the readme to update the binary pointer to Almar's custom build that he mentioned in his comment
Update CI to build the package for mac M1. I suppose that's maybe the non trivial part; you probably need to fiddle with the GitHub actions yml and possibly setup.py.
Push your changes and open a PR

I think CI and github yaml is a bit out of my depth. I updated the pointer to Almar´s custom build. I know the pointer is right because I got an error saying it could not find the lib when using the wrong path.
triangle_glfw.py gives:
Expected wgpu-native version (0, 9, 2, 2) but got (0, 0, 0)
Traceback (most recent call last):
  File "/Users/simon/Documents/Coding/wgpu-py/examples/triangle_glfw.py", line 20, in <module>
    main(canvas)
  File "/Users/simon/Documents/Coding/wgpu-py/examples/triangle.py", line 55, in main
    adapter = wgpu.request_adapter(canvas=canvas, power_preference="high-performance")
  File "/Users/simon/Documents/Coding/wgpu-py/wgpu/backends/rs.py", line 215, in request_adapter
    surface_id = get_surface_id_from_canvas(canvas)
  File "/Users/simon/Documents/Coding/wgpu-py/wgpu/backends/rs_helpers.py", line 107, in get_surface_id_from_canvas
    raise RuntimeError("Received unidentified objective-c object.")
RuntimeError: Received unidentified objective-c object.

I think the first line is just a warning. A few comments up @marcdownie discussed his hacks to get get_surface_id_from_canvas working. That to is abit out of my depth I´m afraid, but it looks like the solution is in the gist he referenced.

Expected wgpu-native version (0, 9, 2, 2) but got (0, 0, 0)

Yes, this is just a warning. Is this with the binary from the release of my fork? If so, it looks like the version is not baked-in correctly ... not something that affects anything directly, but we'd need to fix that eventually.

File "/Users/simon/Documents/Coding/wgpu-py/wgpu/backends/rs_helpers.py", line 107, in get_surface_id_from_canvas
raise RuntimeError("Received unidentified objective-c object.")
RuntimeError: Received unidentified objective-c object.

Why would this part be different for M1?

Is this with the binary from the release of my fork?

Yes I compiled the binary from your fork.

Why would this part be different for M1?

I´m not sure. In this comment @marcdownie discusses that "get_surface_id_from_canvas" fails on his M1 https://github.com/pygfx/wgpu-py/issues/136#issuecomment-861621333.
Looks like he solved the issue using code from the gist but had some issues making it into a pull request.

Is this with the binary from the release of my fork?

Yes I compiled the binary from your fork.

Ah, ok then it makes sense. I meant whether you downloaded it from the unofficial release of my fork :)

I´m not sure. In this comment @marcdownie discusses that "get_surface_id_from_canvas" fails on his M1 #136 (comment).

Right, I forgot how long this thread is :)
I understand that code like that feels overwhelming. I felt the same. I dug around in many (sometimes obscure) parts of the internet to piece everything together. But this is something that only someone with an M1 can do :)

I think CI and github yaml is a bit out of my depth.

I can handle that, and updating the pointer. I'll put it on a branch and open a draft PR.

Sorry for wall of text but I might have found a piece of the puzzle. Reading up on architectural differences between x86 and arm64 (m1) here: https://developer.apple.com/documentation/apple-silicon/addressing-architectural-differences-in-your-macos-code

The x86_64 and arm64 architectures have different calling conventions for variadic functions—functions with a variable number of parameters. On x86_64, the compiler treats fixed and variadic parameters the same, placing parameters in registers first and only using the stack when no more registers are available. On arm64, the compiler always places variadic parameters on the stack, regardless of whether registers are available. If you implement a function with fixed parameters, but redeclare it with variadic parameters, the mismatch causes unexpected behavior at runtime.

Bit further down on that page, discussing the function objc_msgSend that is being called by get_surface_id_from_canvas:

A function like objc_msgSend calls a method of an object, passing the parameters you supply to that method. Because objc_msgSend must support calls to any method, it accepts a variable list of parameters instead of fixed parameters. This usage of variable parameters changes how objc_msgSend calls your function, effectively redeclaring your method as a variadic function.

My guess is that the parameters for objc_msgSend is in the registers on x86 and in the stack on arm64. The article goes on to explain how to solve it. I will give that a go tomorrow.

Solved it by using the code in the gist mentioned by @marcdownie. The examples run. But as I don´t understand how the code in the gist actually works I can´t really make pull request out of it. Licensing is also an issue.

I've had my eye on the easily pip-installable https://github.com/beeware/rubicon-objc as an (almost) drop in replacement for that random gist we've both now used with success, but I haven't quite gotten around to it. How do we feel about adding a dependency?

Given a little:
from rubicon.objc.api import ObjCInstance, ObjCClass

Then I can successfully patch our get_surface_id_from_canvas(canvas) Darwin if clause with:
        cv = ObjCInstance(window).contentView
        cv.setWantsLayer(True)
        metal_layer = ObjCClass("CAMetalLayer").layer()
        cv.setLayer(metal_layer)

        struct = ffi.new("WGPUSurfaceDescriptorFromMetalLayer *")
        struct.layer = ffi.cast("void *", metal_layer.ptr.value)
        struct.chain.sType = lib.WGPUSType_SurfaceDescriptorFromMetalLayer

I only have access to M1 Macs, but this at the very least seems like much less code. Let me know if it's worth a pull request.

How do we feel about adding a dependency?

I was hoping we could solve this without adding a dependency. However, since rubicon-objc is pure Python and has no dependencies by itself (AFAIK), it's a relatively "safe" dependency (we would not depend on their maintainers to push new packages for new Python versions, etc).

Let me know if it's worth a pull request.

It is :)

I still think it'd be interesting to obtain the surface id without a dependency. But this is something that can be picked up later. What I'd try (if I had a mac M1) was to trace the code path through that gist (or through rubicon-objc) and check what (ctypes) API calls are performed. I suspect you'd end up with very similar code as what we have, and we can then see what we were missing. That said, I may be underestimating this  :)

I have added a list of things to fix at the top of this issue.
Another issue is the installation of cffi, this is where @anentropic got stuck on. @marcdownie, @SuperSimon81 how have you tackled that?

Short answer is that the cffi installed via conda install cffi installs the correct _cffi_backend.cpython-39-darwin.so (while pip install cffi, and thus python3 setup develop, gets you an x86_64 one).
There aren't any aarch64 releases of wgpu-native yet are there? Specifically: there's nothing I put in my pull request for download-wgpu-native.py to make it download the correct binaries?

Actually, we're not there yet; we have problems with the latest cffi pulled in from conda:
Traceback (most recent call last):
  File "/Users/marc/temp/wgpu_pull_req/wgpu-py/examples/triangle_glfw.py", line 11, in <module>
    import wgpu.backends.rs  # noqa: F401, Select Rust backend
  File "/Users/marc/temp/wgpu_pull_req/wgpu-py/wgpu/backends/rs.py", line 44, in <module>
    from .rs_ffi import ffi, lib, check_expected_version
  File "/Users/marc/temp/wgpu_pull_req/wgpu-py/wgpu/backends/rs_ffi.py", line 108, in <module>
    def _logger_callback(level, c_msg):
  File "/Users/marc/miniforge3/envs/wgpu_work/lib/python3.9/site-packages/cffi/api.py", line 396, in callback_decorator_wrap
    return self._backend.callback(cdecl, python_callable,
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

Another issue is the installation of cffi, this is where @anentropic got stuck on. @marcdownie, @SuperSimon81 how have you tackled that?

I used conda like @marcdownie.

It's fine if that works in anaconda land but we also need to provide a solution here in pypi/pip country.

There aren't any aarch64 releases of wgpu-native yet are there? Specifically: there's nothing I put in my pull request for download-wgpu-native.py to make it download the correct binaries?

Yes there are! And I just merged #185 that makes the necessary updates here. So if you pull the latest main and run the download script you should be up :)

MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this.

Also posting the error that @berendkleinhaneveld got, which is yet something different than @anentropic reported. This has been reported earlier but the proposed workaround does not seem to work:

E ImportError: dlopen(/Users/cg/Library/Caches/pypoetry/virtualenvs/wgpu-5bSf_T1V-py3.9/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 2): Symbol not found: _ffi_prep_closure

I'm back (sorry about the pause); and I'm up to date with main, out of conda, using the pre-release pip cffi, not having an x64 parallel brew installation any more, and ... I'm still on the cffi python MemoryError.

The MemoryError is the only remaining issue at this point, and so far only @marcdownie has reported it with a conda environment...

To be clear (because this is quite the issue!) I was only not getting the MemoryError in my base conda; I've never made it past the MemoryError outside of a conda environment. And, this morning, after much uninstalling bits of my system, now even conda is throwing the MemoryError.

Results! I walked away from my python superfund site onto a fresh machine. Either the prerelease cffi, or the lack of detritus left over from x64 brew, is in fact the missing ingredient.
WGPU_BACKEND_TYPE=Metal PYGLFW_LIBRARY=/opt/homebrew/Cellar/glfw/3.3.4/lib/libglfw.3.3.dylib python3.9 examples/triangle_glfw.py
Gives me a triangle!


improve memory efficiency of flac3d

The current FLAC3D is the worst of all when it comes to memory efficiency. https://github.com/nschloe/meshio#maximum-memory-usage
@keurfonluu You might want to look into this.

Don't know what else to do to improve the performances after PR #588.
By the way, aren't the I/O speed figures affected by tracemalloc?

By the way, aren't the I/O speed figures affected by tracemalloc?

Not really.


Avoid ethers code for possible infinite while loop

Closes https://github.com/api3dao/airseeker-v2/issues/210

It will be redundant because this.

That ensures that we've read the contract on the correct chain while the RPC call when creating provider ensures that the provider is working, so they serve different purposes.

Hi, that is the code we're still using and works for us. Not sure about the code snippet you've shared, but the provider inside the catch statement does not exist in scope - maybe that's the cause of your issue?

Your code is explicitly calling await provider._detectNetwork();. Please try with removing that as our code doesn't include it.

Thank you so much guys for your quickly reply
I adjusted my code above, please check it
But I still have this log
JsonRpcProvider failed to detect network and cannot start up; retry in 1s (perhaps the URL is wrong or the node is not started)
@bdrhn9
@Siegrift

Hello
Can you check above my code please?
Because I still have this log
@bdrhn9
@Siegrift

@yarikMyroniuk feel free to create a standalone issue that describes your problem. It's not clear what you're after here.


[BUG] LMSYS (claude-v1) 无法使用

LMSYS (claude-v1)，无法使用Error: THIS SESSION HAS BEEN INACTIVE FOR TOO LONG. PLEASE REFRESH THIS PAGE.

It happens with alpaca-13b also:

Go to settings, click the login/logout link of LMSYS can solve this problem. I'm also making a patch to friendly reminder this.


docs: youtube-dl → yt-dlp

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.591%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
74

Relevant Lines:
74

💛  - Coveralls


disable shared access key for cluster storage accounts

Which issue this PR addresses:
Testing cluster e2e in case of disabled shared access key for cluster storage accounts

Fixes
What this PR does / why we need it:

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

How do you know this will function as expected in production?

/azp run ci, e2e


[WIP] Proposal: Extract JobRunner interface

What this does?

Extracting run*Job to a JobRunner interface and adds structs to implement this interface.
Adds a JobRunner map to healer.

Why do this?

This extracts away methods and fields from healer - an already very heavy struct and creates JobRunners specific method and implementations which healer shouldn't be aware of.
This also, allows to extend JobRunners to any number and variety (If someday we would like to add more options more targets for jobs)
And most importantly; it allows me to stub for RunJob and by that write tests for alerts_worker and other parts of healer which currently not testable :cry: .

cc: @yaacov @moolitayer @jhernand

@moolitayer @jhernand @yaacov  Addressing you're comments I have done the following:

Extracted the JobRunners to pkg/job_runners
Extracted metrics_exporter.go from healer to a separate pkg so the runners can use it.
Fixed existing tests to use this Runner pattern.
Tested that healer works well with this new pattern.

Things to do in following PR's moving forward:

Move Init method outside the structs implementing Runner and into the Runner itself thus having a clean interface.
Other ideas are welcome here to enhance this patters :grinning: .

LGTM :+1:

@jhernand @moolitayer mind taking a look?

@jhernand Please take a look :+1:

@jhernand Done.


If you want to deploy to a production environment, what is the recommended way to deploy framework and tars-node?

Framework can be deployed with docker, tarsnode can be deployed with docker or physical machine.

framework可以用docker部署， tarsnode用docker或者物理机部署都可以。


Unable to set capabilities at run-time

I am trying to set capabilities at run time inside onPrepare function of protractor configuration file but it is not setting the value?
    browser.getProcessedConfig().then(function (conf) {
                     conf["capabilities"]["chromeOptions"]["prefs"]["download"]["default_directory"] = "C:/Automation/Drivers";

                });

For what reason do you need it to do it in Run Time?

Yes, You can't change it during run-time, this is as protractor designed.  Because protractor will launch an browser instance before any case beginning, And when code run into onPrepare function, the browser instance had launched with the capabilities.  Those changes of capabilities won't take effect on already launched browser instance.

Hi there!
Since it involves usage and debugging (and you haven't used the issue template to indicate otherwise), your question is better suited for StackOverflow or Gitter. Please ask a question there with the 'protractor' tag or post in the Gitter Channel to get help.
From the getting help section of the README.

Please ask usage and debugging questions on StackOverflow (use the "protractor" tag) or in the Angular discussion group. (Please do not ask support questions here on Github.)

Thanks!


Allow user to customize mouse wheel Zoom speed.

Purpose:
Allow user to setup mouse wheel zoom speed. #937
New Functionality:
Introduce new ScrollWheelZoomIncrement property for Control.Configuration.
Available range is (0, 1).
0.15 default value match to previous behaviour.
Control.Configuration.ScrollWheelZoomIncrement = 0.01; // very slow mouse wheel zoom
Control.Configuration.ScrollWheelZoomIncrement = 0.5 // fast mouuse wheel zoom
Control.Configuration.ScrollWheelZoomIncrement = 0.15 // default value

Just want to say thank you for this PR!

Thank you @StendProg, this looks great 👍
This PR reminds me that more control module fields could benefit from being converted to properties and adding error checking and testing. I noted this in #716 and hope to revisit this some time after 4.1 is released.


Organization billing section styles

[ ] Overview
[ ] Plan
[ ] Payment Method
[ ] Invoices

@sandersonet I think this is also not part of the immediate/launch organizations milestone, yes? We discussed that this is largely done manually today, and will simply continue to be manual after diesel becomes dashboard.
Dropping from the milestone but please let me know if I've missed something.

You are correct, thanks.

Closing in favor of separate Overview and Invoices screens.


Use ReadFormAsync instead Form on HttpRequest

Base on performance test, which showed hot path in using Form synch property (what in fact is sync-over-async) can You use async equivalent in this place - ReadFormAsync.
Check this comment on sync action: FormFeature.

look ok? https://github.com/IdentityServer/IdentityServer4/commit/43849dd94ba7dd29aa2c4840e1f6688e68f9ff37

Great job!

@leastprivilege any objections?

merged. thanks for the heads up!


double multicast causes skipping first event

Hey @briancavalier!

My old vague bug report:
 I’ve been chasing down a problem in our app, that seems to be an issue with `multicast`. I’m having a hard time reducing it down to a minimal case and thought I could give you at least a hint, that something is not working as it should.
The bug I see is that adding multicast to a stream subscribed by multiple consumers results in the source stream not emitting the first initial event for some of them. This leads to our app not initialising correctly. Removing multicast fixes the issue, but since it has multiple consumers it should use it, but not break.
I’ve tried replacing the producer with just and the issue still occurred. The minimal case I could nail it down to for now is having three consumers where the last one is not receiving the initial event. The one not receiving the event is a combination of combine and startWith. Removing startWith fixes the bug.
Removing one of the other two streams or adding another consumer fixes the bug as well. One of them samples the producer and some other streams. Removing a stream there also fixes the bug.
As you can see it’s hard to find the root cause, since it feels quite arbitrary what causes the issue.
Maybe you already have any idea what the problem might be. I’ll keep trying to reproduce it to a minimal example.

Update: I could boil it down to a minimum:
const most = require('most');

const producer = most.just().multicast();

const working = most.combine(() => 0, producer).multicast();
const broken = most.combine(() => 2, producer).startWith(1);

working.forEach(value => document.write('working: ' + value + '
'));

// expected: brokenConsumer should emit 1 and 2
broken.forEach(value => document.write('broken: ' + value + '
'));

// removing one of the multicasts fixes the bug
// removing startWith emits 2

I’ve created an ESNextbin too: https://esnextb.in/?gist=4e1952554ddb71f0e70ea4324f5f2cb1

My guess is that this is not a bug but a race condition. Multicast runs the producer and begins emitting to any attached observers the moment the first observer is added. What I suspect is happening your case is that working causes the upstream producer to be put into an active state first, resulting in a task being scheduled for the next tick following the producer's activation. The upstream producer schedules its one and only event at that time, which is received by the working observer before the broken observer has a chance to finish setting itself up. @briancavalier would need to confirm this.

I'm pretty sure what @axefrog has mentioned is correct, to it being a race condition.
This is precisely the kind of use case I've needed @most/hold for.
But to hopefully describe what is happening here, or at least very close to it:
First tick:

producer is created
working is created as a descendant of producer
broken is created as a descendant of producer
working is subscribed to
broken is subscribed to

Second tick:

producer "broadcasts" the value of undefined to it's listeners. Lets all it "A"

A is passed down to workings "combine" and then to the working forEach
A is passed down to brokens combine but
broken passes down its startWith value of 1 to it's associated forEach.

Last tick:
broken "continues" its sequence by "switching" to it's associated combine() which has already emitted  its events 2 ticks ago, so no value is seen further down the chain.

Hope this helps! I wanted to explain why some other ordering/changes will make this work but I need to run. I'll try to add them here if no one else has done so

Interesting, thanks for the description! I don’t understand yet why switching the order of the .forEachs does emit both events though. (emits 1 -> 0 -> 2)

At the risk of coming across as one of those smarmy "RTFM" types, I do suggest reading the architecture wiki page, as it's quite helpful in building an idea of how most works under the hood. I also suggest perusing some of the code; the design has an elegant but powerful simplicity, and once you understand how it works, you'll be able to troubleshoot most problems fairly quickly.

Ok thank, will take a look. 👍 Closing this one then.


Multipolygon processing errors on large extracts

Hi @ThomasEmge ,
This is the issue I promised to post here regarding some relatively rare errors with multipolygon processing that I see happening on large extracts with the latest build and the OSM File Loader (Load only) tool and that I did not see before the major refactoring related to the OSM File Loader addition in the toolbox.
First have a look at this image. It shows a multipolygon in the Vondelpark in Amsterdam representing a water body with two small, and one larger named island "Koeienweide" (although actually the "Koeienweide" meadow feature is NOT part of the multipolgyon, more on this later on...):
http://www.openstreetmap.org/relation/1141600

This is a classic "old style" multipolygon with no tags on the relation, and where tag transfer needs to take place to render it properly as a water body.
Now, when I create a small extract just covering the Vondelpark using the Download OSM Data (XAPI) tool of the Editor, and import it using the OSM File Loader tool and subsequently render it using my rendering tool, I get this image, which looks OK. The lake is nicely rendered as multipolygon, and if I look in the file geodatabase, it references nicely the multipolygon relation as mentioned also in the first image, so this seems fine.

Now, however, look at these images. They represent the features related to the multipolygons location (excluding the two small inners), and that I found in the file geodatabase created based on the entire country extract from Geofabrik. You verified the relation as present in the osm XML file, and the multipolygon should thus be created, yet it is not.
First, the water body that forms the outer way of the multipolygon. It should have been removed in case of tag transfer, yet it is not, it is still in the file geodatabase. It has just one tag (except source), which is the water tag that needs transferring to the multipolygon:

Next up the true inner representing the large island. It has no tags (which of course also should not be transferred to the multipolygon if there had been any, since only tags on the outer count for that process). This feature of course belongs in the file geodatabase, as inners should never be removed during tag transfer:

Lastly, an image of the Koeienweide "meadow". Actually, this has no direct relation to the multipolygon, as it is NOT part of it. I just show it here, since it shares most (but not all!) of its nodes with the untagged island inner feature mentioned above. Whether that fact has any relevance in the issue posted here, is questionable, but it is good to be aware of its presence:

Anyway, as you understood: why is the multipolygon representing the lake missing in the file geodabase for the large extract, while processing a small extract does not show this issue?
There is no way to create a small test case for this unfortunately, so I'll have to leave it to you to figure out a way to debug this...

Hi @ThomasEmge ,
I have now re-downloaded the Geofabrik Netherlands extract and reloaded it. The multipolygon now appeared good in the extract, see the image below the message.
Either two things can have happened:

There was an incomplete first import that did not finish relation processing completely. If this was the case, it may have been due to multiple ArcMap sessions running in my computer, even though by now I am pretty sure I am separating out things like Current and Scratch workspaces properly with dedicated, non shared, file geodatabaseses. I also loaded it twice if I remember well.

Secondly, I downloaded the previous file from Geofabrik around the time there were some heavy problems with OSM infrastructure, including overpass API. Maybe the Geofabrik extract of the previous file had minor issues for whatever reason that are now fixed.

Anyway, before I give this the full "clear", I am still waiting for my render session to finish. Seeing the symbolized and rendered data always proves very revealing in terms of data errors, and I really want to be sure the outer is also properly dropped and no other issues remain in the imported data.
I would also like you to confirm the presence of the multipolygon relation as well from a full Geofabrik Netherlands download and loading session with the OSM File Loader (Load only) tool. If that comes out OK at your side, I think I can close this issue.

@ThomasEmge ,
This is preliminary, but I do still see issues now the data for the Netherlands is rendered. There are missing multipolygons, and in the particular case shown here of the Vondelpark lake, the relation is there with the correct transfered "natural=water" tag, but the outer as well. The outer has not been dropped from the dataset, as it should. This causes an unwanted overlap and a false double representation of the feature.
Had my second hard drive crash as well now in the lifetime of this PC... rendering is intensive, and unfortunately, despite a respectable well known drive brand, I only discovered later on these particular drives aren't known for top ranking reliability. Luckily a full backup of this stuff, and this was only the secondary data drive not system, but it is time to get another SSD like the system drive for more rendering options.

If you can provide me some of the IDs of the relations I can investigate.
Yes, these operations are very input/output intensive.

Hi @ThomasEmge ,
You already have these, but just as a reminder:

https://www.openstreetmap.org/way/161587161 is the outer way feature tagged with natural=water that isn't being dropped.
https://www.openstreetmap.org/relation/1141600 is the relation that gets its tags (this happens properly, and the feature is in the resulting file geodatabase in the latest import) from the outer listed above

The above represents and "old style" multipolygon, with only tags on the outer way, not on the relation
The below listed is a "modern style" multipolygon with tags on the relation and not the outer. It is part of the "Het Ij" waterbody north of the Amsterdam city centre, essentially the historic harbour region. It is entirely missing in the Polygon feature class. The outer and inner ways are present:
https://www.openstreetmap.org/relation/1942596
This waterbody near it, is missing as well. The relation is not in the resulting feature class. Again, outer and inners are present:
https://www.openstreetmap.org/relation/1926875

Marco,
I think we start to mix different items. In your screen shot above you do show the waterbody from Koeienweide. Even with this correct geometry, as expressed in the relation, do you still get the inner and outer items?
I have tested the above listed relations (harbor and het ij) individually and they are loaded as expected.
So if you were to test the individual file, as attached, the features would load fine but when you attempt to load Netherlands as a country some relation features do not appear.
[hetij.zip](https://github.
harbor.zip
com/Esri/arcgis-osm-editor/files/458035/hetij.zip)

I have tested the above listed relations (harbor and het ij) individually and they are loaded as expected.
So if you were to test the individual file, as attached, the features would load fine but when you attempt to load Netherlands as a country some relation features do not appear.

Yes, exactly, that is what I am saying. I do see some multipolygons (or even the majority) being processed properly, just not all. Some features drop out when I process the entire Netherlands.
Here is a visual. Notice the Koeienweide gone blue and how, in the entire upper right, the Rijksmuseum building is gone missing:

The missing harbour. Do note that other multipolygon features like the Scheepvaartmuseum building in the left, have been properly processed:

"Het IJ" missing:

@ThomasEmge
I now loaded the features individually from the files you send, see the images below. I did make one other observation though: both features seem to suffer from "incorrect-ring-order" problems, that are being fixed in the Repair Geometry step. I wonder if this is related to the issues when loading at country scale?...
From the load session of the "Het IJ":

Correctly loaded harbour as individual feature:

Het IJ also correct:

Hi  @ThomasEmge,
This is preliminary, from a first attempt to load the Netherlands dataset again. That session ended with a COM related error again, see the image below. It failed just after the relations processing.

In the steps directly after the relations processing and just before the above error, the tool reports to have successfully deleted the intermediate osm datasets created in the Scratch workspace, yet, if I look in the Scratch workspace, the datasets are still there:

Although this session failed, I decided to have a closer look at the data created. As to some mildly good news, I can confirm that I now finally see the missing relations in the Polygon Feature Class. Both the IJ, harbour and Vondelpark lake relation polygon are there, see Het IJ example below and the identify results showing the relation OSMID.

However, for the Vondelpark lake example, I still see the outer way as well in the Polygon Feature Class, it hasn't been removed. I don't know if this is a result of the incomplete processing and failure, or still hints at an issue. The water tag did get properly transferred to the lake multipolygon during relations processing, so that part is OK.
Also, I now see another strange thing happening. When I looked at the File Geodatabase, I noticed the intermediate osmCountingTable that wasn't deleted:

That by itself is not a big surprise, because the processing failed. However, since I realized the osmSupportingElement nodes were most likely not deleted as well, I just kind of instinctively decided to open the Point Feature Class's attribute table:

And here the true problem becomes visible. Scrolling through the table, I of course saw the undeleted "supporting" nodes. Not surprising, since the processing never arrived at the stage of deleting them. However, more surprising, I noticed multiple nodes with apparently duplicated names. The most striking was the "Martinuskerk",  the name of a Dutch church in the city of Utrecht. It appeared on a dozen nodes or so. That felt uncomfortable. Look at the image above. The vertical arrow pointing upwards highlights the first "supporting" node with the Martinuskerk name attached to it. The dataframe shows its position near a canal. It turns out to be this node:
https://www.openstreetmap.org/node/21430322#map=19/52.08371/5.12271)
part of a road along the canal called "Twijnstraat aan de Werf". As clearly visible in the image below, that node doesn't have any tags associated, so why does the resulting supporting node in the file geodatabase? Of course, it would have been deleted if the option for deleting supporting nodes would have been chosen, but something is wrong here... It reminds me of the issue we had in processing a dataset in Berlin, where also tags appeared on the wrong object. I don't know the issue number right now.

Hmm, the error message is unexpected because after the temp file geodatabases are deleted there is no more new data created. Hence I am surprised that code fails with a CreateFeatureClass call.
The issue you are seeing with the points/nodes is an optimization step. When the user selects the parameter to delete the supporting nodes, the points are loaded with a buffer mechanism.  The side effects is that supporting points are getting a copy of the attributes from the last properly attributed node. This is acceptable, as these nodes are deleted at the end. On the other hand it also means if the parameter is not checked the loading of points will take longer as these attributes need to be properly initialized in the geodatabase.

Ok, good to hear the last issue is actually a known step in the code. It slightly made me worried as being something new.
That leaves two questions:

Why didn't the outer way get deleted of the Vondelpark lake? (also because of the incomplete processing?)
The COM related error.

Hence I am surprised that code fails with a CreateFeatureClass call.

You actually mean File Geodatabase creation / File Geodatabase workspacefactory, because I don't see a CreateFeatureClass call in the red error message text? Or am I interpreting this wrong?

I spoke too soon, it is in the loading step between relations and super-relation where the error occurs.  It looks like the delete of the netherlands-latest_yiy4aqrvvq_r0.gdb geodatabase failed, even though the previous tool reports that it succeeded. That is the COM error and it just goes downhill from there. However there is another check failing as well.
Any chance you were connected to the temp file geodatabase?

Any chance you were connected to the temp file geodatabase?

Not at the time of the exact failure, because I was out of the door. I do inevitably hit some geodatabases that are in some processing stage at times when I am multi-tasking with multiple sessions of ArcGIS, because simply opening the Add Data dialog or so, may take you to a File Geodatabase being used by another ArcMap session.
But I think it unlikely in this case, although admittedly I can not be absolutely sure there was no lock or something.
I do know I have seen this same error before, at the exact stage you are describing, and I am pretty sure I posted a similar screenshot before.

Hi @ThomasEmge ,
I have now attempted to load Poland. This time, the processing passed the relations - super-relations border. However, it failed with an unexpected invalid SQL statement that I haven't seen before (or at least can't remember seeing before):

When I subsequently attempted to add the Feature Classes - without closing the project in which I performed the processing - and attempted to open the referenced Polygon attribute table, I got this error:

Clearly a bogus error, and probably result of the inconsistent state in which the failed processing left the project, as I have plenty of disk space left on C-drive:

Closing and re-opening the project allowed me to add and view the three Feature Classes created. The screenshot below shows the object as referenced in the SQL statements, to be present in the Polygon Feature Class, the SQL thus should have been valid?... No idea what went wrong here. The object seems to be an outer way of a multipolygon, so should indeed be deleted, as the first error referred to in the code error messages (...IFeature.Delete):

Running the tool against a small section downloaded using the Download OSM Data (XAPI) tool, succeeds without errors...

My loading of Poland completed fine without any errors. Let's compare ArcGIS versions really quick: are you running 10.4 or 10.4.1? Are you using geoprocessing foreground or background processing?

@ThomasEmge:
10.4.1 on Windows 7, using foreground processing.

Hi @ThomasEmge,
I just had a look at the latest import results. As to the good news: the first run with the Netherlands was error free~! See the first image. This is of course all a bit preliminary as being based on a single run / observation, but this truly seems promising due to a number of other factors:

Output File Geodatabase size is now properly 10.1 GB as you also reported.
No left-over temporary osm files or temp file geodatabase in the Scratch workspace, everything is now properly deleted
All the problematic features I listed above (Het IJ, harbour, Vondelpark lake and Rijksmuseum building), now are present and properly resolved. I still need to render it, but simply identifying the data seemed to confirm this. The Vondelpark lake now also has the outer way properly dropped, only the relation is present.
The data was now processed in just under 10 1/2 hours (4 cores), which is pretty amazing considering older imports, as it cuts back 7 hours compared to previous runs! The latter may, though, also have to do with the fact that I bought a third SSD, meaning I could separate out the original osm file, the Scratch workspace and the final destination file geodatabase each on its own dedicated SSD for optimal processing.

There just seems one minor issue left, see below this image of the successful run:

As to the minor issue I am seeing, it seems an explicit area=no tag is being ignored? I noticed it on this feature, which represents a closed way of barrier=hedge immediately to the right of the Rijksmuseum building, tagged with explicit area=no, which should have forced it into the Polyline table, but it ended up in the Polygon table instead.
Feature in iD with visible tags:

Carto rendering: correctly displayed as line:

Incorrectly imported as closed way / polygon in ArcGIS:

Hi @ThomasEmge ,
Over the weekend, I have re-imported the Netherlands a second time, and also re-imported Poland without any issues. So that is good news.
As to the "number crunching", the Netherlands import finished in just over 10 1/2 hours again (using 4 cores), which is very consistent with the previous run. Poland, using 3 cores and using the same triple SSD setup, finished in 12 3/4 hour, which is also very consistent with the results for the Netherlands, as Poland and the Netherlands are almost the same size in terms of OSM XML file size, and number of nodes, ways and relations. So using 1 core less, the result seems utterly plausible. The resulting file geodatabase is also just under 10GB, which again seems normal.
As to the particular problem I reported above, regarding the ignoring of an area=no tag, a quick review using ArcMap Identify tool, indeed seems to confirm the issue is now fixed, the barrier=hedge feature is now in the Polyline, not the Polygon table.
There is just one - apparently completely benign - minor anomaly that I noticed during the processing and that I have seen before. Look at the attached image:

Notice the highlighted python.exe process. It has 0% activity. This process is left over after the node processing finishes and doesn't get removed after that step. However, it doesn't seem to have any detrimental effect on the processing or stability, and once the OSM File Loader (load only) tool finishes completely, it disappears. So although a minor anomaly, it doesn't  appear to have any consequences for the processing or use of the tool. I don't know if it actually happens always, but I do remember seeing it before.
Before I give the official "green light" on this one, I still would like to see the render results of the Netherlands. Both the Netherlands and Poland are currently being rendered, but due to the slow OSM Attribute Selector, it takes a couple of days to process them, mainly spend on extracting the attributes. I guess with the current implementation, rendering entire Europe could take a month... If anything, a faster OSM Attribute Selector is even higher on the priority list than a Pro version of the toolbox IMO.
Anyway, it all seems very hopeful, and I think we can close this issue in the coming days. I think many ArcGIS users will love the tool once the fixed version is released!

The shown python process might not even be originating from the geoprocessing tools but perhaps belong to the PyScripter process.
I haven't noticed any outstanding exes during the node, way, and relation loading process but I'll keep an eye on it.

Yes, you may be right it is from another process, although I still have the vague feeling it is related to the tool. Anyway, as I wrote before, if it is related, it seems inconsequential as far as I can tell up to now, so it is likely safe to be ignored.

Hi @ThomasEmge ,
As a final confirmation that things seem to be OK now, a few screenshots of the problematic features as now rendered (the Netherlands just finished). All look OK now:
Koeienweide:

Hedges and Rijksmuseum building:

IJ and harbour:

Based on the latest import and rendering results of my ArcGIS Renderer using the Netherlands and Poland extracts of Geofabrik, I am now pretty sure the multipolygon issues are fixed. I will therefore provisionally close this issue. As always, I will keep an eye out for any anomaly, but this now really looks fixed, I don't really expect any overseen issues with the latest version of the OSM File Loader (load only) tool.


[BUG] Unable to install if a wifi card is present

Describe the bug

If there's a wifi card present, harvester-configure-network will choke if a nearby wifi network contains spaces.
To Reproduce
Steps to reproduce the behavior:

Burn iso and boot on a machine with a wifi card
Ensure a local access point is broadcasting an SSID with spaces in it
Try to configure networking, cannot continue past the network configuration screen.

Due to being unable to continue past this point, it's probably safe to try on your personal laptops :)
Expected behavior

Able to configure networking.
Support bundle

Environment:

Harvester ISO version: v0.2.0
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): Bare Metal MBP 13,2 (needs separate patches currently in-progress)

Additional context
Add any other context about the problem here.

We change the os for the Harvester and use wicked which should support wifi cards to manage the network. I can not find a machine with a wifi card, @withinboredom  could you kindly help verify this issue?


Errors sometimes not rethrown

Been tracking down this tricky bug that was causing some real-world pain 😢
RxJS version: 5.3.1 & 5.4.3
Code to reproduce:
Runnable version here.
Observable.of(0, Scheduler.async)
  .mergeMap(() =>
    Observable.combineLatest(Observable.of(0))
  )
  .subscribe(() => { throw new Error('uh oh'); });

Expected behavior:
Error is re-thrown asynchronously and can be handled by normal uncaught exception means. (window.onerror, uncaughtException).
Actual behavior:
Error gets swallowed.
Additional information:
combineLatest seems to be involved here—if you get rid of it, the example behaves as expected.

Have to verify, but my current guess is this is related with https://github.com/ReactiveX/rxjs/pull/2313 - I've added few fixes around via https://github.com/ReactiveX/rxjs/pull/2626, https://github.com/ReactiveX/rxjs/pull/2796 but it seems not suffecient to cover all cases.

I run into the same problem. The interesting thing is that when I change it to:
Observable.of(0, Scheduler.async)
    .mergeMap(() => Observable.of(0))
    .combineLatest(Observable.of(0), Observable.of(0))
    .subscribe(() => { throw new Error('uh oh'); });

it throws the error.

We run into this issue in our production application, rxjs is swallowing errors in some cases which is very hard to debug and leads to strange behavior in our application.
The only way to work around this issue in our case is to async rethrow (this sounds as evil as it is) the error in the subscribe error handler.
This ensures that the global angular error handler receives the error and it does not disappear into the deeps of rxjs.
Observable.of(0)
    .subscribe(() => {}, (error) => { setTimeout(() => { throw error; }) });

Would anyone happen to know how this is not considered a show-stopper given it's so easy to reproduce?
https://jsbin.com/ribeyeyolo/edit?html,js,console

FYI: in order to be compliant with the tc39 proposal, error rethrowing is going to have to go away. This is for a variety of reasons. Probably the biggest reason is that if an error is synchronous the throne after a multicast, it causes producer interference. The error unwinds to callstack back to the loop that's notifying everyone in the multicast, and breaks it.
The plan is to remove the rethrowing behavior in 6.0.

FWIW I think we can provide some Rx specific mechanism of notification for unhandled errors.

@benlesh thanks!
Whilst I understand the need to remove it, it seems such a step backwards (from a safety POV) - the idea of not 'swallowing errors' was a major selling point and determining factor (especially when switching from promise-based flows).
Are you suggesting in the near future errors thrown in this way (from my link above) will just be swallowed regardless then?
I'm only curious on the path going forward, not trying to be awkward :)

@shakyShane it's a very complex topic when dealing with multicast subscriber interference and actually the same problem Promises have which solved it the same way; by not rethrowing aka swallowing.
If it means anything, I used to be stringently opposed until someone sat down and walked me through the problems of the current approach and how not rethrowing is the better tradeoff.
Although the backstory is many years old (with Promises), some of the more recent discussion: #2145, https://github.com/tc39/proposal-observable/issues/119
Since Promises do this too they later made a global unhandledrejection event you could listen for which will be called with all promise rejections that weren't handled, so they were "swallowed" by JS itself. In addition, modern browsers debug consoles automatically listen to this and log the errors in the console so that you still know they happened. It's highly likely Observables will get a very similar (or even the same) solutions.
You can reproduce this in the context of Promises with this:
https://jsbin.com/hifavaq/edit?js,output
console.log('first');

try {
  const promise = new Promise(() => {
    console.log('second');
    throw new Error('Silent, but deadly');
  });
} catch (e) {
  // this never runs, no error is caught
  // it is "swallowed" per spec
  console.log('NEVER RUNS AS IT DOESNT RETHROW');
}

console.log('third');

The TC39 observable discussions about errors are super interesting and important but can we keep this particular issue about RxJS 5? The current behavior is inconsistent, surprising, and a real source of pain. 😞

Fixed in 5.5.6 https://github.com/ReactiveX/rxjs/blob/stable/CHANGELOG.md#556-2017-12-21


Handle Homgears listMethods-implementation?

When calling ListMethod, Homegear supplies an additional paramter, which the official XML-API does not. Should we implement or ignore this? The CCU won't send that extra parameter, so supporting HGs implementation could break the CCU. And since CCU is the official, I'm in favour of the current approach.

Ignore it, because it will be fixed with the next version ;)
https://github.com/Homegear/Homegear/issues/256


Add script processing to seed dev

Currently, we only process the client.js library file. We need to also processs any inlined <script> blocks.
We should use esbuild, and treat them as individual modules.
A modulepreload should be added as well for each entry.
<script> blocks are not able to reference other <script> blocks within seedling or html files.
External *.js and *.ts file should be able to be referenced, and should be processed as a new indepdent module.

This works.
You can in theory also do the following -
/components/hi.ts
export default 'hi';

https:// some.path.com/goodbye.ts
export default 'goodbye';

/components/SayHello.html
import hi from './hi.ts';
import bye from 'https://some.path.com/goodbye.ts';
console.log(hi);
console.log(bye);