OTAClient refactoring & OTA local delta update implementation/integration

Description & motivation
This PR implement an alternative standby slot creating mechanism called OTA local delta rebuild_mode, which calculate the delta between active slot and new regulars.txt by sha256hash to prevent duplicating copying and unnecessary downloading.
During introducing rebuild_mode to the current otaclient code base, I found that a fundamental refactoring to the otaclient is needed due to deep coupling between each modules/components of the otaclient, complicated relationship and dependencies between modules/components, unclear code execution flows, etc, which make extending the functionalities of otaclient a hard thing to do.
So I start to refactor the otaclient before the final integration of OTA local delta rebuild_mode. The main objectives are as follow:

sort out/split and cleanup complicated dependencies between logic components of otaclient
redesign the dependency between components/modules, tear down complicated/deep coupling, improve cohesion of components/modules, relieve deep coupling, breakdown large/complicated modules/components to small ones if needed
make otaclient extensible to support different standby slot creating mechanism and different boot loader, with the defined interface/protocol. One can implement any new mechanism by implementing the interface/protocol

The other objectives  are as follow:

introduce new ota error handling mechanism, define the specification of all possible errors
fix bugs and improvement over otaclient

Major changes

all boot_control related logic are grouped and implemented as boot_control module. Module functionalities are exposed by interface/protocol BootControllerProtocol.
all create_standby_slot related logic are grouped and implemented as create_standby module. Module functionalities are exposed by interface/protocol StandbySlotCreatorProtocol
all detailed implementations of how OTA update is applied are stripped from the OTAClient, now OTAClient only reflects the logic flow of an OTA update, the detailed implementations are in the boot_control module and create_standby module. OTAClient calls the module via defined interface/protocol
cleanup and refine the logic flow of OTA update in the OTAClient
re-arrange, re-locate or re-construct many modules to follow the refactoring objectives
common.py under the main project folder or modules folder are introduced to store the shared functions and classes
introduce new OTA error handling structure, now all the exceptions are handled properly (specific OTA error codes are still under designing)
migrate the OTAClient grpc server to use grpc.aio API, cleanup and fix bugs
adopt type hints among the code base
bug fix and improvement all over the code base

changes & bug fix

clearly distinguish ota_status into live_ota_status and stored_ota_status
introduce slot_in_use file to boot_control to prevent negative SUCCESS(boot back to active slot instead of new slot)
implement on_failed methods for OTAClient, BootController and StandbySlotCreator to cleanup on failed operation
cleanup and move common used classes/functions to common.py
implement many helper methods for RegularInf, etc
(List not yet finished...)

related tickets
T4PB-17843 ota update rebuild mode implementation
T4PB-17931 ota-client code structure refactoring
T4PB-17503 ota local delta update
T4PB-18438 詳細なエラーコード
T4PB-17503 ota local delta update
TODO

[x] vm test(standalone, without ota_proxy)
[x] vm test(standalone, with ota_proxy)
[x] vm test(multiple, with ota_proxy)
[ ] ECU test(standalone)
[ ] tests files re-implementation

( PR description is not yet finished.

Found a bug in _grub.is_switching_boot_partition_from_active_to_standby, I will fix it now.

there are _grub.py and grub.py. more understandable name is preferred.

there are _grub.py and grub.py. more understandable name is preferred.

Ah OK I will put _grub.py into grub.py and merge them into one file.

Ah OK I will put _grub.py into grub.py and merge them into one file.

@Bodong-Yang Sorry, this merge SHOULD be done after this PR. I don't want to review it again.

Ah OK I will put _grub.py into grub.py and merge them into one file.

@Bodong-Yang Sorry, this merge SHOULD be done after this PR. I don't want to review it again.

@obi-t4 Please temporarily paused the review as I am still fixing bugs, although these bugs rarly happen in normal situation but it needs to be fixed.

@obi-t4 Please temporarily paused the review as I am still fixing bugs, although these bugs rarly happen in normal situation but it needs to be fixed. I will finish fixing within today.

Fixing bug is OK. We assumed that the refactoring of the code base had been finished. So If you want to merge larger modifications, please make it a separate PR.

@obi-t4 Fix is applied, I am confirming whether it solves the problem or not on the virtual machine.

@obi-t4 Fix is confirmed on the virtual machine! Please resume the review, thanks!

Remaining issues:

[ ] nvbootctrl failed is not captured in some cases
[ ] failed to recovered from network error(status remained as UPDATING) in some cases
[ ] download size is not properly recorded if regulars.txt doesn't include entry size
[ ] handling space not enough error

Besides the above issues, otaclient is working properly on the perception ECU.

@Bodong-Yang
In my opinion, this PR seems to have several problems and cannot be merged.
Also, the modifications are too large and there might be some deterioration, and it is difficult to judge.
What do you think?

@obi-t4 standby slot detection method is reverted to previous in 0c4ac4e.

@obi-t4 new implementation of grub_controller confirmed working on freshly installed system and migrate the system to use ota-partition successfully.
freshly installed system
autoware@autoware-VirtualBox:/boot$ ls -l
total 139060
-rw-r--r-- 1 root root   254083 10月 27  2021 config-5.11.0-40-generic
-rw-r--r-- 1 root root   248291  2月  5  2021 config-5.8.0-43-generic
drwxr-xr-x 4 root root     4096  7月 12 12:10 grub
lrwxrwxrwx 1 root root       28 11月 10  2021 initrd.img -> initrd.img-5.11.0-40-generic
-rw-r--r-- 1 root root 56971085 11月 10  2021 initrd.img-5.11.0-40-generic
-rw-r--r-- 1 root root 52051323 11月 10  2021 initrd.img-5.8.0-43-generic
lrwxrwxrwx 1 root root       27 11月 10  2021 initrd.img.old -> initrd.img-5.8.0-43-generic
drwx------ 2 root root    16384 11月 10  2021 lost+found
-rw------- 1 root root  5860664 10月 27  2021 System.map-5.11.0-40-generic
-rw------- 1 root root  5515823  2月  5  2021 System.map-5.8.0-43-generic
lrwxrwxrwx 1 root root       25 11月 10  2021 vmlinuz -> vmlinuz-5.11.0-40-generic
-rw------- 1 root root 11741600 10月 27  2021 vmlinuz-5.11.0-40-generic
-rw-r--r-- 1 root root  9716672  2月 10  2021 vmlinuz-5.8.0-43-generic
lrwxrwxrwx 1 root root       24 11月 10  2021 vmlinuz.old -> vmlinuz-5.8.0-43-generic
autoware@autoware-VirtualBox:~$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.11.0-40-generic root=UUID=e53cd8ed-d035-43bb-8cb8-6214658ff338 ro quiet splash
autoware@autoware-VirtualBox:~$ cat /etc/default/grub
... <GRUB_DISABLE_SUBMENU is not set> ...

first launch of otaclient
no update is applied, just launch the otaclient and let it initialize.
[2022-07-12 16:56:08,718][INFO]-main.py:main:36,started
[2022-07-12 16:56:08,719][INFO]-__init__.py:get_boot_controller:13,use boot_controller bootloader='grub'
[2022-07-12 16:56:08,726][INFO]-__init__.py:get_standby_slot_creator:18,use slot update mode=<CreateStandbyMechanism.REBUILD: 1>
[2022-07-12 16:56:08,741][INFO]-grub.py:__init__:218,self.active_slot='sda2', self.standby_slot='sda3'
[2022-07-12 16:56:08,752][INFO]-grub.py:_get_current_booted_kernel_and_initrd:254,detected booted param: entry.linux='vmlinuz-5.11.0-40-generic', entry.initrd='initrd.img-5.11.0-40-generic'
[2022-07-12 16:56:08,752][INFO]-grub.py:init_active_ota_partition_file:427,system doesn't use ota-partition mechanism to boot, initializing ota-partition file...
[2022-07-12 16:56:08,794][INFO]-grub.py:_prepare_kernel_initrd_links_for_ota:287,finished generate ota symlinks under /boot/ota-partition.sda2
[2022-07-12 16:56:09,898][INFO]-grub.py:_grub_update_for_active_slot:323,boot entry for self.active_slot='sda2': 0
[2022-07-12 16:56:10,880][WARNING]-grub.py:_grub_update_for_active_slot:353,boot entry for standby slot not found, only current active slot's entry is populated.
[2022-07-12 16:56:10,880][INFO]-grub.py:_grub_update_for_active_slot:354,generated grub_cfg:  ...
[2022-07-12 16:56:10,883][INFO]-grub.py:_grub_update_for_active_slot:363,update_grub for sda2 finished.
[2022-07-12 16:56:10,883][INFO]-grub.py:init_active_ota_partition_file:444,ota-partition file initialized
[2022-07-12 16:56:10,884][INFO]-grub.py:__init__:520,loaded ota_status: OTAStatusEnum.INITIALIZED
[2022-07-12 16:56:10,885][INFO]-ecu_info.py:__init__:21,ecu_info={'format_version': 1, 'ecu_id': 'autoware', 'ip_addr': '<IP_ADDRESS>'}

boot folder status:
autoware@autoware-VirtualBox:/boot$ ls -lah
total 136M
drwxr-xr-x  7 root root 4.0K  7月 12 16:56 .
drwxr-xr-x 21 root root 4.0K 11月 26  2019 ..
-rw-r--r--  1 root root 249K 10月 27  2021 config-5.11.0-40-generic
-rw-r--r--  1 root root 243K  2月  5  2021 config-5.8.0-43-generic
drwxr-xr-x  4 root root 4.0K  7月 12 16:56 grub
lrwxrwxrwx  1 root root   28 11月 10  2021 initrd.img -> initrd.img-5.11.0-40-generic
-rw-r--r--  1 root root  55M 11月 10  2021 initrd.img-5.11.0-40-generic
-rw-r--r--  1 root root  50M 11月 10  2021 initrd.img-5.8.0-43-generic
lrwxrwxrwx  1 root root   27 11月 10  2021 initrd.img.old -> initrd.img-5.8.0-43-generic
lrwxrwxrwx  1 root root   28  7月 12 16:56 initrd.img-ota -> ota-partition/initrd.img-ota
lrwxrwxrwx  1 root root   33  7月 12 16:56 initrd.img-ota.standby -> ota-partition.sda3/initrd.img-ota
drwx------  2 root root  16K 11月 10  2021 lost+found
drwxr-xr-x  2 root root 4.0K  7月 12 16:20 ota
lrwxrwxrwx  1 root root   18  7月 12 16:56 ota-partition -> ota-partition.sda2
drwxr-xr-x  2 root root 4.0K  7月 12 16:56 ota-partition.sda2
drwxr-xr-x  2 root root 4.0K  7月 12 16:56 ota-partition.sda3
-rw-------  1 root root 5.6M 10月 27  2021 System.map-5.11.0-40-generic
-rw-------  1 root root 5.3M  2月  5  2021 System.map-5.8.0-43-generic
lrwxrwxrwx  1 root root   25 11月 10  2021 vmlinuz -> vmlinuz-5.11.0-40-generic
-rw-------  1 root root  12M 10月 27  2021 vmlinuz-5.11.0-40-generic
-rw-r--r--  1 root root 9.3M  2月 10  2021 vmlinuz-5.8.0-43-generic
lrwxrwxrwx  1 root root   24 11月 10  2021 vmlinuz.old -> vmlinuz-5.8.0-43-generic
lrwxrwxrwx  1 root root   25  7月 12 16:56 vmlinuz-ota -> ota-partition/vmlinuz-ota
lrwxrwxrwx  1 root root   30  7月 12 16:56 vmlinuz-ota.standby -> ota-partition.sda3/vmlinuz-ota

first reboot result
boot into default entry specified by otaclient(vmlinuz-ota)
autoware@autoware-VirtualBox:/boot$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-ota root=UUID=e53cd8ed-d035-43bb-8cb8-6214658ff338 ro quiet splash
autoware@autoware-VirtualBox:~/ota-client/app$ sudo python3 main.py
[2022-07-12 16:59:03,466][INFO]-main.py:main:36,started
[2022-07-12 16:59:03,466][INFO]-__init__.py:get_boot_controller:13,use boot_controller bootloader='grub'
[2022-07-12 16:59:03,469][INFO]-__init__.py:get_standby_slot_creator:18,use slot update mode=<CreateStandbyMechanism.REBUILD: 1>
[2022-07-12 16:59:03,489][INFO]-grub.py:__init__:218,self.active_slot='sda2', self.standby_slot='sda3'
[2022-07-12 16:59:03,497][INFO]-grub.py:_get_current_booted_kernel_and_initrd:254,detected booted param: entry.linux='vmlinuz-ota', entry.initrd='initrd.img-ota'
[2022-07-12 16:59:03,498][INFO]-grub.py:init_active_ota_partition_file:444,ota-partition file initialized
[2022-07-12 16:59:03,499][INFO]-grub.py:__init__:520,loaded ota_status: OTAStatusEnum.INITIALIZED
[2022-07-12 16:59:03,500][INFO]-ecu_info.py:__init__:21,ecu_info={'format_version': 1, 'ecu_id': 'autoware', 'ip_addr': '<IP_ADDRESS>'}

@obi-t4 A fix and refinement on ota_client_stub is on-going, expected to finish within today. Thanks!

Known bugs have been fixed with the latest commit.

Known bugs have been fixed with the latest commit.

Is this includes slot-in-use problem below?

It will cause problem(ota status will become INITIALIZED) if we directly reboot the system/otaclient without doing an ota update with the new otaclient, as the slot_in_use file is not created by the old otaclient.

Known bugs have been fixed with the latest commit.

Is this includes slot-in-use problem below?

It will cause problem(ota status will become INITIALIZED) if we directly reboot the system/otaclient without doing an ota update with the new otaclient, as the slot_in_use file is not created by the old otaclient.

Yes, this is covered in c3a563d.

:smile:

Yes, this is covered in c3a563d.

I think in finalize_update, if slot-in-use file doesn't exist, is_switching_boot returns false and resulted in failure.
So updating from current OTA client to this OTA client can't be done with successfully.
Am I understanding correctly?

@obi-t4 Oh the is_switching_boot also check the slot_in_use file!
I will take a look at how to deal with it.

@obi-t4 symlink detection based slot detection mechanism is re-introduced in 0e82f58 and integration is done in 23f561c. And when otaclient finds that slot_in_use file is not found, it will fallback to use symlink detection based mechanism to detect active/standby slot.

@Bodong-Yang
Are these scenario works properly?
[Senario A]
Under the environment, the system is freshly installed and this new ota client is also installed.
Is is_old_otaclient set to be True? I think the slot-in-use file doesn't exist in this environment.
[Senario B]
If the system was once updated by this new ota client successfully and then updated again to the current ota client,
Can we go back to this new ota client again under this condition? I'm afraid slot-in-use file exists and is_old_otaclient is set False.

@Bodong-Yang Are these scenario works properly? [Senario A] Under the environment, the system is freshly installed and this new ota client is also installed. Is is_old_otaclient set to be True? I think the slot-in-use file doesn't exist in this environment.
[Senario B] If the system was once updated by this new ota client successfully and then updated again to the current ota client, Can we go back to this new ota client again under this condition? I'm afraid slot-in-use file exists and is_old_otaclient is set False.

[Scenario A]: this is the normal initializing scenario, the ota_status will be set to INITIALIZED, and otaclient will init the slot_in_use file and ota-partition files.
[Scenario B]: emmmm, I have to think it carefully, I have no idea right now.

@obi-t4 updated in 52b9660, now the use of slot_in_use file in boot controller init and is_switching_boot are dropped and revert to use the previous symlink based mechanism. But the otaclient will still create and maintain the slot_in_use file.
When the new otaclient is widely used instead of old version, we can consider migrate to slot_in_use switch boot detection mechanism.

@Bodong-Yang
The review has been finished. I'll start the test tomorrow.

[ ] from current to this PR
[ ] from this PR to current
[ ] from USB installed to this PR
[ ] multiple ECU
[ ] only secondary ECU update through the main ECU

@Bodong-Yang
I faced these errors and restarted automatically then dropped into initramfs shell. Can you investigate it?
(gub menu was pointed first entry. custom.cfg is not used??)
Jul 15 05:54:49 autoware-ecu python3[671]: [2022-07-15 05:54:49,628][INFO]-rebuild_mode.py:_save_meta:179,save image meta files to /mnt/standby/opt/ota/image-meta
Jul 15 05:54:50 autoware-ecu python3[671]: [2022-07-15 05:54:50,139][INFO]-ota_client.py:_in_update:195,[_in_update] finished creating standby slot
Jul 15 05:54:50 autoware-ecu python3[671]: [2022-07-15 05:54:50,346][INFO]-ota_client.py:execute:273,[update] leaving update, wait on ota_service, apply post-update and reboot...
Jul 15 05:54:50 autoware-ecu python3[671]: [2022-07-15 05:54:50,347][INFO]-ota_client_stub.py:stop:196,stopping ota_proxy(cleanup_cache=True)...
Jul 15 05:54:51 autoware-ecu python3[671]: [2022-07-15 05:54:51,831][INFO]-ota_client_stub.py:stop:141,ota proxy server closed
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,074][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_fe8d9c83621c9af150b7fafa3c97d4dd, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,074][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/pointcloud_preprocessor/lib/pointcloud_preprocessor/passthrough_filter_node', bucket=32768, last_access=1657864492.074425, hash='60db0e69f6e8cf2927d0b60abb395d3ac7f0c7415b23bfb87450a1b71936ccc2', size=62040, content_type='application/octet-stream', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,075][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_5a3842787bbce29a57dae90309e624dd, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,077][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/pointcloud_preprocessor/lib/pointcloud_preprocessor/crop_box_filter_node', bucket=32768, last_access=1657864492.074971, hash='f8b34b7440b775f507378d9d6ecd6281455528651570937dd3f33c7f0bcb8345', size=62040, content_type='application/octet-stream', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,078][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_4b8d618c720bda586f2ba7d807060734, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,078][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/pointcloud_preprocessor/lib/pointcloud_preprocessor/vector_map_filter_node', bucket=32768, last_access=1657864492.077945, hash='7fb235e1ee73ef5b3027cca30b944b4fc7b0c003517f75919c0542f02d33310d', size=62040, content_type='application/octet-stream', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,078][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_2e74a6798e18dfa0d6f5ae31739d2272, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,079][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_ba27ce1cb27482e173776da99ca67792, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,079][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_0794d80f2f536b845daa5b94d13a12d2, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,079][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/pointcloud_preprocessor/lib/pointcloud_preprocessor/radius_search_2d_outlier_filter_node', bucket=32768, last_access=1657864492.078478, hash='1a20bcc4db4b11326049293f33281ba7a66f1f9156f9d15493d61e427e99469c', size=62056, content_type='application/octet-stream', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,079][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/pointcloud_preprocessor/lib/pointcloud_preprocessor/approximate_downsample_filter_node', bucket=32768, last_access=1657864492.078701, hash='69e697c20859397d2aad97895547b98c7a3ee87b686f31fa08e0fd9f66f41716', size=62056, content_type='application/octet-stream', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,079][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/pointcloud_preprocessor/lib/pointcloud_preprocessor/pointcloud_accumulator_node', bucket=32768, last_access=1657864492.078818, hash='830a96f249354ca9efc7d06b4672a224a217c290ab599b062ca499b5ac9d43e4', size=62048, content_type='application/octet-stream', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,081][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_ad808c94c0f19c2ff81d64d984cbb852, abort...
Jul 15 05:54:52 autoware-ecu python3[1262]: [2022-07-15 05:54:52,081][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/install/joy_controller/share/joy_controller/package.sh', bucket=2048, last_access=1657864492.081307, hash='bdb9fa1231df53bb486c0cdc5a9c335f9fc303eb17c5a7aade276219f163781d', size=2982, content_type='text/x-sh', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:52 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:52 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:52 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:52 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:52 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:52 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:52 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:52 autoware-ecu python3[1262]:     return f()
Jul 15 05:54:52 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 166, in insert_entry
Jul 15 05:54:52 autoware-ecu python3[1262]:     return cur.rowcount
J

last OTA status was:
[2022-07-15 05:55:24,634][INFO]-update_test.py:<module>:31,ecu {
  ecu_id: "autoware"
  status {
    status: UPDATING
    version: "1.2.4x"
    progress {
      phase: POST_PROCESSING
      total_regular_files: 299925
      elapsed_time_copy {
      }
      elapsed_time_link {
      }
      elapsed_time_download {
      }
      total_elapsed_time {
        seconds: 679
        nanos: 922000000
      }
    }
  }
}
available_ecu_ids: "autoware"

Connection to <IP_ADDRESS> closed by remote host.
Connection to <IP_ADDRESS> closed.

last of jounalctl -u otaclient error was:
Jul 15 05:54:55 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:55 autoware-ecu python3[1262]: [2022-07-15 05:54:55,089][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_29da74dd6f768feee75d6a0cf8581898, abort...
Jul 15 05:54:55 autoware-ecu python3[1262]: [2022-07-15 05:54:55,089][WARNING]-ota_cache.py:writer_on_failed:75,writer failed on tmp_b123197ddec9e3c8decff21a0bcf58dd, abort...
Jul 15 05:54:55 autoware-ecu python3[1262]: [2022-07-15 05:54:55,089][ERROR]-ota_cache.py:_register_cache_callback:737,failed on callback for meta=_CacheMeta(url='http://<IP_ADDRESS>:8081/data/home/autoware/autoware.proj/src/simulator/scenario_simulator/openscenario/openscenario_interpreter/src/syntax/relative_target_speed.cpp', bucket=0, last_access=1657864495.088479, hash='9338524d8c480a73be45beffa2d51c7d87fa9864e7f2f708be18c75887b755e6', size=1379, content_type='text/x-c++src', content_encoding=''): ProgrammingError('Cannot operate on a closed database.')
Jul 15 05:54:55 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 162, in insert_entry
Jul 15 05:54:55 autoware-ecu python3[1262]:     cur = con.executemany(
Jul 15 05:54:55 autoware-ecu python3[1262]: sqlite3.ProgrammingError: Cannot operate on a closed database.
Jul 15 05:54:55 autoware-ecu python3[1262]: During handling of the above exception, another exception occurred:
Jul 15 05:54:55 autoware-ecu python3[1262]: Traceback (most recent call last):
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 724, in _register_cache_callback
Jul 15 05:54:55 autoware-ecu python3[1262]:     if self._lru_helper.commit_entry(meta):
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/ota_cache.py", line 174, in commit_entry
Jul 15 05:54:55 autoware-ecu python3[1262]:     if self._db.insert_entry(entry) != 1:
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 265, in _wrapped
Jul 15 05:54:55 autoware-ecu python3[1262]:     return fut.result()
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
Jul 15 05:54:55 autoware-ecu python3[1262]:     return self.__get_result()
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
Jul 15 05:54:55 autoware-ecu python3[1262]:     raise self._exception
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Jul 15 05:54:55 autoware-ecu python3[1262]:     result = self.fn(*self.args, **self.kwargs)
Jul 15 05:54:55 autoware-ecu python3[1262]:   File "/opt/ota/client/ota_proxy/db.py", line 261, in _inner
Jul 15 05:54:55 autoware-ecu python3[1262]:     return f()
Jul 15 05:55:22 autoware-ecu systemd-journald[332]: Suppressed 70876 messages from otaclient.service
Jul 15 05:55:22 autoware-ecu python3[671]: [2022-07-15 05:55:22,659][INFO]-grub.py:_prepare_kernel_initrd_links_for_ota:424,finished generate ota symlinks under /boot/ota-partition.sda3
Jul 15 05:55:27 autoware-ecu python3[671]: [2022-07-15 05:55:27,098][INFO]-grub.py:_grub_update_for_active_slot:460,boot entry for vmlinuz-ota(slot=sda4): 1
Connection to <IP_ADDRESS> closed by remote host.
Connection to <IP_ADDRESS> closed.

It seems that errors from ota_proxy breaks the otaclient, let me take a look.

@obi-t4 what is the test setup of the above errors? Single autoware ecu with ota_proxy server enabled?

@obi-t4 it seems that the standby slot is not created properly(it is broken), then the system drops into the initramfs.

After back to the current partition and call status, SUCCESS is returned. It's not correct.

Those errors were introduced by a corrupted OTA image. Sorry for the confusion. :bow:

commit 80fcb51 fix the ota status is not collected.

Update by this OTA client is working properly!

Then I will revert the commit that call os.sync on create standby finish.
The commit that reduce the concurrent tasks will remain though.

@obi-t4 I will also remove the current not working test files temporary before merging to the release candidate branch.


Meetup.com OAuth2

As a user I want to have possibility to authenticate my users with Meetup.com.

Should this be closed, since the PR has been merged?

Yes.


[FEATURE] Mapping out graph names to human understandable names.

Description

We need to figure out a way to map out these names to human language, I don't even understand sometimes. 🤯
Severity
Major
Additional Information
No response

Updated.


[Screen Reader- Fancy Zones-Launch Zones-Create New Custom]: 'Custom layout Creator' text is not defined as heading.

[Fancy Zones-Launch Zones -Utility-Create New Custom]
User Impact:
Visually challenged users will face difficulty in understanding the section as no heading level is defined/implemented on the ‘Custom Layout Creator’ dialog.
Test Environment:
OS Version: 20226.1000
App Name: Power Toy Preview
App Version: v0.23.0
Screen Reader: Narrator
Reproe-Steps:

Open Power Toys App
Navigate to Fancy Zone Button and activate it.
Navigate to Launch Zones Editor and activate it.
Navigate to Custom tab in  Choose your layout for this Desktop dialog and select it.
Navigate to Create new custom item and selet it
Navigate to Edit Selected layout element and activate it.
Turn on the Narrator and then scan mode by 'Caps+Space' keys.
Navigate in heading mode by pressing 'H' key and verify the issue.

Actual Result:
Custom layout Creator' text is not defined as heading
NOTE: Same issue reproes throughout the application
Observation: Same issue reproes with NVDA 2020
Expected Result:
Custom layout Creator' text should be defined as proper heading level 1 as that helps visually challenged users in understanding the page.
MAS Reference:
https://microsoft.sharepoint.com/:w:/r/teams/msenable/_layouts/15/WopiFrame.aspx?sourcedoc={0cf80872-800d-4643-90cf-0c1c3d3e6260}&cid=3f071c7c-5735-446d-99db-41f6e989fa41
52_Create a new Custom_MAS2.4.6-Heading is not defined on the dialog screenshot.zip
52_Create a new Custom_MAS2.4.6-Heading is not defined on the dialog.zip

I am not sure if we can fix this easily, since setting AutomationProperties.HeadingLevel property (to "1") is not an option, since it is only available on UPW, and FancyZones Editor is WPF.

@enricogior This issue is outdated due to the recent design changes. Can I just close these? And does it require a certain tag?

Closing this, outdated due to recent UX changes.


Add tldraw example

What this PR does / why we need it?
Adds tldraw example in homepage's example pages.
This PR should be merged after this PR: https://github.com/yorkie-team/yorkie-js-sdk/pull/423
Example page screenshot:

Also fixed some typo in TodoMVC's description, and added example links (TodoMVC, Codemirror, tldraw) in footer.
Footer screenshot:

Any background context you want to provide?
What are the relevant tickets?
https://github.com/yorkie-team/yorkie-js-sdk/pull/423
Checklist

[x] Added relevant tests or not required
[x] Didn't break anything

Updated file structure information fileInfo.ts with requested modification on https://github.com/yorkie-team/yorkie-js-sdk/pull/423.


sql: add columns in jobs virtual table for overview in DBConsole

https://github.com/cockroachdb/cockroach/pull/68995 --- Release note (general change): The functionality to retry failed jobs with exponential-backoff has introduced recently in the system. This commit adds new columns in crdb_internal.jobs table, which show the current backoff-state of a job and its execution log. The execution log consists of a sequence of job start and end events and any associated errors that were encountered during the job's each execution. Now users can query internal jobs table to get more insights about jobs through the following columns: (a) last_run shows the last execution time of a job, (b) next_run shows the next execution time of a job based on exponential-backoff delay, (c) num_runs shows the number of times the job has been executed, and (d) execution_log provides a set of events that are generated when a job starts and ends its execution.

Duplicate of https://github.com/cockroachdb/docs/issues/11274


feat(sidenav): resize using drag

Be able to resize the sidenav using drag.
A good example of this feature is the resize of the files/code/preview on Stackblitz.

This isn't a feature we plan to support out of the box, though it is something that could conceivably be built on top of the mat-sidenav by the community.


Bug ConfigureKestrel Not working locally

Is this possible to get working locally? If I run the app after creating Azure App Registration and changing appsettings, I get this error:
Grpc.Core.RpcException: 'Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: The SSL connection could not be established, see inner exception. IOException: Cannot determine the frame size or a corrupted frame was received.", DebugException="System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException: Cannot determine the frame size or a corrupted frame was received.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)

And if comment out the Kestrel configuration in Service Program.cs (lines 31-38) I get this error:
Grpc.Core.RpcException: 'Status(StatusCode="Unauthenticated", Detail="Bad gRPC response. HTTP status code: 401")'
Also, could you tell the steps you did while creating the Azure App Registration? It is enough to just create the app registration and create the client secret?

Hi @Markz878
I needed to add this for local dev:
listenOptions.UseHttps(); // required for local debugging, not for the Azure deployment

This should work locally now.
The app registration needs to be created with a secret/certificate and the scope
"Scope": "api://--your-client-id--/.default"

Thanks for the response. Unfortunately the Unauthenticated error still persists.. If I remove the Authorize-attribute, I see that ServerCallContext context variable contains in RequestHeaders my authorization header with correct and valid jwt. I guess it's not a problem that it's in lowercase authorization? My Service project also logs that  "Security token validated". But if I try to convert the ServerCallContext  to "var user = context.GetHttpContext().User", the user does not have any claims... Looks like Asp.Net can't bind the ClaimsPrincipal.. Any idea what could be the problem?
And I can't create a App Registration scope with name .default in Azure Portal, the dot is not allowed, is that naming possible through cli or how did you do it?

@damienbod Can you please help me?
And as you propably know, ASP.Net Core auth story is often criticized as being too cumbersome. You could help this a lot by adding more steps to your tutorials, especially in Azure side, this is like the third tutorial from you that I try to follow where I have to try things out a lot since you haven't added the necessary steps to make it work straight away, thank you.

@Markz878 If the name claim has no value, then you need to define this in the parameters. You might need to reset the default claim mappings or add this as defined. ASP.NET Core middleware added namespaces per default to certain claims.
I didn't add any Azure App registration docs to the blog, repo, that would help. See the Microsoft.Identity.Client docs for the App registration setup. You use a application client (no user)
In the App registration | Expose an API the Application ID URI need to be active without the default. Then you can use the default as like the demo. You don't use consent then. This is because this is an application client and no a user client.
Greetings Damien

@damienbod Sorry but I don't understand any of this advice.. I have cloned your repo and replaced the appsettings parameters in Service for AzureAd: { Domain, TenantId, ClientId and ClientSecret }, and in Client AzureADServiceApi: {ClientId, Authority, Scope}. Shouldn't this just work after that? Comment for each paragraph:
1: I don't have a name claim in my jwt. What does that mean "then you need to define this in the parameters", what parameters? In code? Or in Azure? And what about "need to reset the default claim mappings or add this as defined"? I went to Azure portal, and in my app registration in Token configuration added email, family_name and given_name claims, but it did not help.

What do you mean by this "you use a application client (no user)"? So in App registration in Authentication there is Platform configurations. Are you saying that I need to use Mobile and desktop applications? But the Grpc Service is a web app right? Or what do you actually mean by this?

Yes in Expose an API the Application ID URI is set, and I can use the .default scope, or at least the Service project logs  that token is valid when I pass it to my Service.

@Markz878 It is hard to know how much I need to write docs for. Every IDP does this different and setting a client for each OIDC server is still different. Azure AD and Azure AD B2C have a few specifics.
Maybe I would understand better with some more details:
You want to run the local console application using the local API, correct?

Does the console application get the application back from the AAD identity provider?
Can you look at the token and see what is in it? jwt.ms (if you have the wrong token, maybe you need to validate that you use a V2 endpoint and request a role access_as_application, guessing her.)
Can you remove the authorize attribute in the GRPC service and see it if works?
If this works, add a breakpoint in the GRPC service and see what claims you have?
What is the value of the name claim?

Greetings Damien

@damienbod Yes, I would like to test everything locally first.

Yes, the authResult has a value with AccessToken populated.
Here are my jwt claims with some info redacted:
{
"typ": "JWT",
"alg": "RS256",
"x5t": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI",
"kid": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI"
}.{
"aud": "api://XXXXXXXX-XXXX-XXXX-XXXX-997f8950ce08",
"iss": "https://sts.windows.net/XXXXXXXX-f409-XXXX-XXXX-76ce62c20e4d/",
"iat":<PHONE_NUMBER>,
"nbf":<PHONE_NUMBER>,
"exp":<PHONE_NUMBER>,
"aio": "E2ZgYAjfzNLTeOdoLqvgx79rWR+kAwA=",
"appid": "XXXXXXXX-XXXX-XXXX-XXXX-997f8950ce08",
"appidacr": "1",
"idp": "https://sts.windows.net/XXXXXXXX-XXXX-XXXX-XXXX-76ce62c20e4d/",
"oid": "",
"rh": "0.AV8AoPQkBAn0hUGdnXbOYsIOTRoGWGmfJYBFvRuZf4lQzghfAAA.",
"sub": "",
"tid": "",
"uti": "HFjyPaT4m0u_CfhYY0MeAA",
"ver": "1.0"
}.[Signature]
Yes I did that and it works.
Like I said earlier, when I remove Authorize attribute, and try to get ClaimsPrincipal from ServerCallContext using context.GetHttpContext().User, the ClaimsPrincipal does not have any claims. ASP.Net Core doesn't seem to bind them from the Authorization header.
There are no claims.

Thanks for the help!

Hi  @Markz878 The access token should be "ver": "2.0", you can update this in the Azure App registration manifest.

Otherwise it looks good. You would need to remove this as well or update it to match your token once you switch to a V2 token
https://github.com/damienbod/GrpcAzureAppServiceAppAuth/blob/main/GrpcAzureAppServiceAppAuth/Program.cs#L17

@damienbod I set accessTokenAcceptedVersion to 2 in the manifest and removed that line 17 from Program.cs, and now my jwt looks like this:
{
"typ": "JWT",
"alg": "RS256",
"kid": "2ZQpJ3UpbjAYXYGaXEJl8lV0TOI"
}.{
"aud": "XXXXXXXX-XXXX-XXXX-XXXX-997f8950ce08",
"iss": "https://login.microsoftonline.com/XXXXXXXX-XXXX-XXXX-XXXX-76ce62c20e4d/v2.0",
"iat":<PHONE_NUMBER>,
"nbf":<PHONE_NUMBER>,
"exp":<PHONE_NUMBER>,
"aio": "E2ZgYPg260Zwie2f9JS3IbNs2/tmAAA=",
"azp": "XXXXXXXX-XXXX-XXXX-XXXX-997f8950ce08",
"azpacr": "1",
"oid": "XXXXXXXX-XXXX-XXXX-XXXX-34a7c4bc94e9",
"rh": "0.AV8AoPQkBAn0hUGdnXbOYsIOTRoGWGmfJYBFvRuZf4lQzghfAAA.",
"sub": "XXXXXXXX-XXXX-XXXX-XXXX-34a7c4bc94e9",
"tid": "XXXXXXXX-XXXX-XXXX-XXXX-76ce62c20e4d",
"uti": "1wE1nb0qSEKQtfzrv3FpAA",
"ver": "2.0"
}.[Signature]
And I'm still getting the same Unauthenticated error in Client, while my Service project logs that "IDX10241: Security token validated."
I have tried to configure Azure App Registration by setting Platform configurations to Web server with redirect uri https://localhost:7179/signin-oidc, and Mobile and desktop apps with redirect uri http://localhost. I tried adding every single optional claim in Token configuration page. I tried adding a custom scope access_as_user (was not allowed, client says that for client app it has to be .default scope..). None of this works..
I have cloned your code, set the appsettings to my App registration values, applied your previous comment's changes.  Are you saying that this all works for you locally? Your blog post specifically targets Azure Ad, so I would hope that this should work by following your instructions.

@Markz878  yes this works locally for me and also in all my Azure AD deployments, I use this in a lot of different clients. You access token looks fine. I usually add an access_as_application scope and use this as a role in my Azure App registration.
The Azure App Registration does not need any redirectURL, it is just an API definition.
Does it work with just the [Authorize] on the GRPC service?

Some MS docs:
https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki
https://github.com/AzureAD/microsoft-identity-web/wiki/web-apis
only access_as_application in your use case.

Does it work with just the [Authorize] on the GRPC service?

No, unfortunately it does not work with just a plain [Authorize] attribute.

You need to turn on the debug logs (PII as well) on the API an see the reason why the access token is rejected.

I have turned PII on several times by now, and with Default logging level as Debug the logs look like this (PII off but they just show the token):
warn: Microsoft.AspNetCore.Server.Kestrel[0]
Overriding address(es) 'https://localhost:7179'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://[::]:8080
info: Microsoft.Hosting.Lifetime[14]
Now listening on: https://[::]:7179
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
Content root path: C:\Users\marku\Downloads\GrpcAzureAppServiceAppAuth-main\GrpcAzureAppServiceAppAuth-main\GrpcAzureAppServiceAppAuth
crit: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
Microsoft.IdentityModel Version: <IP_ADDRESS>. Date 09/02/2022 16:28:35. PII logging is OFF. See https://aka.ms/IdentityModel/PII for details.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10242: Security token: '[PII of type 'System.String' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]' has a valid signature.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10239: Lifetime of the token is valid.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10234: Audience Validated.Audience: '[PII of type 'System.String' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10246: ValidateTokenReplay property on ValidationParameters is set to false. Exiting without validating the token replay.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10237: ValidateIssuerSigningKey property on ValidationParameters is set to false. Exiting without validating the issuer signing key.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10255: TypeValidator property on ValidationParameters is null and ValidTypes is either null or empty. Exiting without validating the token type.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10245: Creating claims identity from the validated token: '[PII of type 'System.IdentityModel.Tokens.Jwt.JwtSecurityToken' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.
info: Microsoft.IdentityModel.LoggingExtensions.IdentityLoggerAdapter[0]
IDX10241: Security token validated. token: '[PII of type 'System.String' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.

@damienbod Could you update your blog and show how this should work step by step by cloning your code and creating a new app registration in Azure?
There is really no point in making tutorials where half of the steps are omitted.. If it's easy and obvious, why create tutorials for it? And if it's not, why skip half of the steps?

I have a blog here which does almost the same:
https://damienbod.com/2020/10/01/implement-azure-ad-client-credentials-flow-using-client-certificates-for-service-apis/
@Markz878 Microsoft have documented this as well. The App registration looks good now I guess, something inside you API does not, guess some incorrect GUID etc Maybe if you enabled the PII logs, you might see the reason. The logs above have is disabled. I cannot see what is incorrect on your AAD and client from the information you provide.
Try to follow these Microsoft docs for setting up a confidential client without a user.
https://docs.microsoft.com/en-us/azure/active-directory/develop/scenario-daemon-overview
PS I will add the Azure App registration steps to the blog as well and create a new Web API application to validate.
Greetings Damien

And here's the exact example from Microsoft with the App registration and API setup
https://github.com/Azure-Samples/active-directory-dotnetcore-daemon-v2/tree/master/2-Call-OwnApi

@Markz878
test this and was able to reproduce. The API gives bad error logs just like you reported. The Microsoft.Identity logs could be improved.
The problem is that you need to add a role and expose this for the API to work.
I added a step by step in images from 1-5 for you to setup this up.
https://github.com/damienbod/GrpcAzureAppServiceAppAuth/tree/main/images
Once the roles claims is included in your V2 access token and you have grant consent in th eApp registration, it will work.
Please try
Greetings Damien

Hallelujah, it works. Faith in humanity restored :)
Thanks a lot @damienbod


Fix HSTRING to conform to Rust's aliasing rules

Pointers derived from &mut references must still uphold the uniqueness guarantee. Fortunately the fix is simple here:  we just use a shared reference when duplicating HSTRINGs. Casting between pointer types is fine, it's effectively just a lint.
Fixes: #2056

This is wrong. You can have as many *mut T pointing to the same T as you like, you just can't have multiple &mut T at once (otherwise this entirely safe example should not compile). Furthermore, this PR potentially introduces UB, since you cannot legally mutate through a *mut T that was obtained from an &T. Here's a playground example you can run with MIRI to see that it picks up UB: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=11397c0ce7a6705b02c7d78518d90383

The first safe playground example you linked is safe because the pointers are never used. It's always safe to not use pointers.
Indeed, mutating a pointer obtained through an &T would not be sound. However, in this case it's being used in a HSTRING which is immutable.


Better TeX support

The current Hugo docs provide quite the runaround to get math working for formulas involving underscores. User-contributed alternatives (like custom shortcodes) aren't much better for equation-heavy text. These approaches may be fine for someone starting a new blog, but it is not for me.
I have an existing wordpress.com blog (jeremykun.com), which has hundreds of posts and 5000+ tex equations written over 7 years, which I'd like to convert to hugo. About half of these equations have underscores. I have successfully converted the posts to markdown as it would be used on a site like mathoverflow.com, using standard $ax + b$ and \[x_1 + x_2\] equation fences.
I'm also an engineer, and it's clear that adding regex hacks on top of regex hacks isn't sustainable. If I want to write for another 10 years, chances are good hugo will be replaced with some other engine and I'll have to migrate again.
From what I've gathered, the obstacle is that the markdown rendering engine (Blackfriday) does not support ignoring content in equation fences, and the latex/katex rendering happens after the rendering engine has run. Is that correct? Is there anything else?
If so, I will happily talk to the devs of Blackfriday and add a feature that allows you to configure Blackfriday to ignore markdown characters within specified equation fences. (This seems like it should be trivially easy to me---they already do it for code---but there are always complications I'm sure)
If I did this, would it be possible to incorporate this feature into hugo? I'm imagining a part of the hugo base config.toml that specifies which latex equation fences you'd like to respect (default to none), and hugo simply passes those along to the rendering engine.

Have you considered writing your content files in asciidoc and having Hugo build the site by passing your files to asciidoctor to render the HTML?

@bwklein The problem is that the content is already written. (thousands of pages of text)
I suppose you mean convert the posts from markdown to asciidoc and then have a workflow (somehow built into hugo?) from asciidoc to html. I will look into that, but maybe you could point me to how I would integrate an asciidoc -> html workflow into hugo? Does the hugo CLI do that?

@j2kun If Hugo sees an AsciiDoc file in the content folder, it will try to run the asciidoctor command on that file by way of 'external helpers'.  I need to make a new project with this kind of setup for myself, so I will reply here when I have a demo/tutorial for it.  If you can already run asciidoctor from the command line on your PC, it should just work.  But, setting up asciidoctor is not always trivial depending on the OS.

I still think this is valuable, but I have given up hope that it will change. Such a simple fix, too :(


feat: yaml中的managedFields字段设置为空,优化体验

What type of PR is this?
/kind feature
What this PR does / why we need it:
把yaml中的managedFields字段设置为空,优化体验
Which issue(s) this PR fixes:
Fixes #461

I have read the CLA Document and I hereby sign the CLA

@cla-bot check

@regend Welcome to contribute! 🎉 I have left a few comments, you can take a look again.

CLA CI seems to have some problems, could you please reply to this paragraph again.
"I have read the CLA Document and I hereby sign the CLA"

I have read the CLA Document and I hereby sign the CLA

@regend Welcome to become the Karpor contributor, looking forward to your more participation! 🎉 #585


Notification bar display does not work

Environment

OS: [Windows]
scrcpy version: [2.3.1.]
installation method: [Windows release]
device model:
Android version: [14]

When I pull down the notification bar with my mouse, it doesn't stay pulled down, but hides when I release it.

scrcpy just forward mouse events, the device UI decides what to do.
Typically, the notification panel expansion depends on the mouse speed/acceleration and distance, so it might be more difficult to keep it open with a mouse than with fingers (although I have no problem on my device).
Note that for convenience, you can open it using shortcuts MOD+n and MOD+MOD+n.

scrcpy just forward mouse events, the device UI decides what to do.
Typically, the notification panel expansion depends on the mouse speed/acceleration and distance, so it might be more difficult to keep it open with a mouse than with fingers (although I have no problem on my device).
Note that for convenience, you can open it using shortcuts MOD+n and MOD+MOD+n.

Thanks, the shortcut key works great.


Don't add constraints for invalid subscript declarations.

This fixes the crash in sr-114.

Adding constraints for the invalid operator decl means constraining to the error type, which fails an assertion later on while binding an overload choice. In all the normal function/method overload choice cases, if the decl is invalid that choice gets skipped (never generated), so this is just another case of the existing way of doing things.

Great catch.  I'd suggest returning "Type()" when the decl is known invalid (mirroring the behavior in visitOverloadedMemberRefExpr around line 1312).  Also, since the test case is small and should be fast to run, please add it to the non-validation test suite, I'd suggest in test/decl/subscript/subscripting.swift.
Thank you for tracking this down and fixing it!

Sure thing!  If it looks good, let me know, and I'll squash the two.

Actually, just went ahead and squashed because hopefully this is the final change needed.

Totally awesome, thank you!

Also, please do the honors of closing SR-114! :-)


VS Code does not correctly attach to UNIX domain socket

I am running my rails server with the command rdbg -n -c --open=vscode -- rails s and it will open VS Code but running the attach configuration will result in the error "Can not find attachable Ruby process." and if I run rdbg --util=list-socks I see nothing even though in the server output it specifies a port to listen on.
$ rdbg -n -c --open=vscode -- rails s
DEBUGGER: Debugger can attach via UNIX domain socket (/var/folders/k4/lsrtl0y55718pst90df0ldzh0000gn/T/com.apple.shortcuts.mac-helper/rdbg-501/rdbg-72212)
Launching: code /Users/executor/Documents/GitHub/VideoConnect/
=> Booting Puma
=> Rails <IP_ADDRESS> application starting in development
...

If I run rdbg -n -c -O -- rails s, then I see two log lines from the server specifying the UDS port and running rdbg --util=list-socks returns the port as expected. If I open VS Code and try to attach to the debugger again with the default config, I get the same error, but if I manually specify the debugPort to be the one from the console output, it is able to connect correctly and everything works as expected.
$ rdbg -n -c -O -- rails s
DEBUGGER: Debugger can attach via UNIX domain socket (/var/folders/k4/lsrtl0y55718pst90df0ldzh0000gn/T/com.apple.shortcuts.mac-helper/rdbg-501/rdbg-72755)
DEBUGGER: Debugger can attach via UNIX domain socket (/var/folders/k4/lsrtl0y55718pst90df0ldzh0000gn/T/com.apple.shortcuts.mac-helper/rdbg-501/rdbg-72755)
=> Booting Puma
=> Rails <IP_ADDRESS> application starting in development
...

{
    "type": "rdbg",
    "name": "Attach with rdbg",
    "request": "attach",
    "debugPort": "/var/folders/k4/lsrtl0y55718pst90df0ldzh0000gn/T/com.apple.shortcuts.mac-helper/rdbg-501/rdbg-71078",
}

I can confirm the same issue existing in my case. Is there any plan for fixing this please? It is annoying to change the debug port with each rails app start to be able to attach debugger. Thank you.


Add code formatting

:tada: This PR is included in version 1.2.0-develop.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Blob Sync Test Coverage

API change check
API changes are not detected in this pull request.

/azp run java - storage - tests

/azp run java - storage - tests

/azp run java - storage - tests

/check-enforcer override


Use context for location provider if activity is not available

The current activity may be null when the location provider is created, in which case we can use the React application context instead. Alternatively we could wait for the activity to become non-null, but that would add some more complexity.
Resolves: #56

Thanks for this.
Released in 2.2.1: https://github.com/timfpark/react-native-location/releases/tag/2.2.1


New puppeteer headless mode breaks the screen recording

When puppeteer is configured to use the new headless mode, the video is always cut off. I have tried to play with different values for the viewport and puppeteer-screen-recorder settings, but it seems there is no effect. If I disable new headless mode in puppeteer, everything works as expected.

I get empty recordings whenever I turn headless mode on at all, is anyone else experiencing this?

I get empty recordings whenever I turn headless mode on at all, is anyone else experiencing this?

After further messing around, I found out that, for me at least, the new version is the only version that works in headless mode - so setting headless:'new' fixed things for me. I did find you have to set the page viewport after starting recording though, otherwise you get weird results. Here's my working example, in case it's useful to you:
`const width=640;
const height=480;
const browser = await puppeteer.launch({headless:'new'});
const page = await browser.newPage();
const recorder = new PuppeteerScreenRecorder(page,{
    fps: 25,
    videoCrf: 28,
    videoCodec: 'libx264',
    videoPreset: 'ultrafast',
    videoBitrate: 1000,
    recordDurationLimit: 5,
    videoFrame: {
        width,height

    }
});
const pipeStream = fssync.createWriteStream('./output.mp4');
//const pipeStream = new PassThrough();
const data = await fs.readFile('./submodules/SimpleCip54Viewer/build/index.html', { encoding: 'utf8' });
const src = 'data:text/html,' + encodeURIComponent(data);

await page.setViewport({width, height});

await page.goto(src);
await page.waitForFunction(() => document.readyState === "complete");

await page.waitForNetworkIdle({idleTime: 1000, timeout:10000})
await recorder.startStream(pipeStream);
await page.setViewport({width, height});
// Set screen size

const binaryScreenshot = await await page.screenshot({ type: 'png', encoding: 'binary' });
await fs.writeFile('./output.png',binaryScreenshot)

    setTimeout(()=>{
        recorder.stop();
        browser.close();
        pipeStream.close();
    },5000);
`


Include CodeArtifact input to allow the action to read from CodeArtifact

Not sure about a default value. I assume the default is nothing, so not passing in a value should be fine (if it doesn't try to get something from CodeArtifact, it shouldn't need a token).


Issue 456/home page structure enhancements

This PR:
Resolves #456
1. Added PASS logo and text to use as h1 - my understanding is we are switching to this logo. Need a higher quality version.
2. Refactored headers of home page to follow semantic outline.
3. Added <main> to layout - only thing marked as main semantically previously was this home logged out page, nothing else.
4. Added some conditional rendering to solve issue of adding empty elements to DOM.
Screenshots:
https://github.com/codeforpdx/PASS/assets/71395931/c47cc438-056a-49c7-bc3d-d7f5f0071013
Future Steps/PRs Needed to Finish This Work (optional):

Need to figure out what to do for proper headers in footer - wasn't sure because it depends on what page is being rendered currently (I think.)
Also believe the Home.jsx page can be refactored. Seems like some props are being passed to child components, that could just be initialized in those components themselves. Could also possibly be arrays of objects and .map()'ed - but not sure that would cut down code or improve readability until attempted. I do believe it would make for easier maintainability/changes in future.

Issues needing discussion/feedback (optional):
1.  Styling feedback
2. Code review

Latest attempt:
https://github.com/codeforpdx/PASS/assets/71395931/1d89fa06-7f5e-46c6-a9ae-8fa78803f14c
Still have some code work to do for proper headings, then to tackle tests.

Let me know any feedback on this design.

Latest attempt: https://github.com/codeforpdx/PASS/assets/71395931/1d89fa06-7f5e-46c6-a9ae-8fa78803f14c
Still have some code work to do for proper headings, then to tackle tests.
Let me know any feedback on this design.

The video isn't rendering but I downloaded it.
Overall I think it's getting better.

Line 70 I think could be an h4 on smaller viewports. On the desktop version it's fine.
Also, when I try out mobile I see this, where its not formatted as nicely as in the video.

I've tried reloading and resizing but it doesn't change. Not sure what's causing it.

Using our primary theme color instead of black would be nicer and more consistent in my opinion. I don't recall if we have a version of the PASS logo with that yet, though.
Also, we use a version of the logo with the PASS text within the image. So I'm not sure if we want to use that here as well. But it doesn't really seem necessary and would maybe limit our responsiveness a bit.

In case you weren't already aware, the Learn More button does not function correctly. Plus it overlaps with the other elements on certain viewports.

Honestly, at this point I'd just like to get UX's perspective. We may be winging it a bit too much.

Latest attempt: https://github.com/codeforpdx/PASS/assets/71395931/1d89fa06-7f5e-46c6-a9ae-8fa78803f14c
Still have some code work to do for proper headings, then to tackle tests.
Let me know any feedback on this design.

The video isn't rendering but I downloaded it.
Overall I think it's getting better.
Line 70 I think could be an h4 on smaller viewports. On the desktop version it's fine.
Also, when I try out mobile I see this, where its not formatted as nicely as in the video.

I've tried reloading and resizing but it doesn't change. Not sure what's causing it.
Using our primary theme color instead of black would be nicer and more consistent in my opinion. I don't recall if we have a version of the PASS logo with that yet, though.
Also, we use a version of the logo with the PASS text within the image. So I'm not sure if we want to use that here as well. But it doesn't really seem necessary and would maybe limit our responsiveness a bit.
In case you weren't already aware, the Learn More button does not function correctly. Plus it overlaps with the other elements on certain viewports.
Honestly, at this point I'd just like to get UX's perspective. We may be winging it a bit too much.

video not rendering
yeah it seems when github changed that you can't just click in the textbox and add an image/video, but now have to click the paperclip button to do so - they also limited video size. Any larger files (haven't figured out exact size yet) seem to make you download file instead of just play it.

Changed line 70 per your thoughts, looks better in my opinion.

Changed to green, but we don't have logo in that color yet that I could find.

I was told the logo that has the PASS text combined is not properly optimized and doesn't scale well, so I didn't attempt it.

Learn More
Yeah, I haven't seemed to figure out anchor "scroll to" links in MUI yet. I'll dig into it - if you happen to know by chance, let me know.

UX perspective
yeah pretty much most of the discussion now is not technically in the scope of this PR to fix the headings.

 If we still havent settled on a design for the front page, it may be worthwhile to loosen up these tests so they don't fail on areas we haven't finalized yet.

Just tagged in milo to attempt to solve our idea for a better h1 for this page, and I'm moving onto the tests. Honestly, looking into them I don't believe they are the best written and may need to write a new approach.  But I'm just starting this now.

I think you can remove the HomeSection unit tests. It seems to me that this is a section in too heavy development for those to be all that useful (and there's not too much interesting behavior on this page anyway).
The Home.jsx snapshot test will need to either be updated with npm run test -- --update-snapshot, or replaced with a similar render test that shows it's rendering the content.

npm run test -- --update-snapshot

I don't know when this all happened, but seems like tests are failing all over on Development branch too. I can start working on those next.  I'd like to learn more about testing.
So, my important question is - when do we want tests? When is it valuable to have them?
Like in the case of HomeSection:

switch it to see if the component returns anything
switch it to see if the component returns what we are currently expecting - maybe image and title text
delete the test

npm run test -- --update-snapshot

I don't know when this all happened, but seems like tests are failing all over on Development branch too. I can start working on those next. I'd like to learn more about testing.
So, my important question is - when do we want tests? When is it valuable to have them?
Like in the case of HomeSection:

switch it to see if the component returns anything
switch it to see if the component returns what we are currently expecting - maybe image and title text
delete the test

We would like to check to see if an element within a component is rendered, that's one important test, especially if that element would be something users would interactive with like a button or a text field.
The next thing should be if there are any significant rendering changes between breakpoints (like if widths would change from 25% to 100%) think we should test their CSS properties as well. For these tests, think we could make it into a binary test, one where we use the base value for Desktop as a basis for those tests. So the tests would either .toBe(<what we expect the value to be on Desktop>) or .not.toBe(<what we expect the value to be on Desktop>). It'l be more flexible as well which simply shows that "rendering changes has occur".
Of course, if we change the base case (i.e. say Desktop for now), we'll have to update the test, but if we change the non-base case (i.e. mobile), the test should still run fine.
For snapshots, which has been in discussion for a while, I think we should outright replace with either #1 of what you've mentioned for the Home page. For #2, I think we could handle those when testing for smaller components like in the HomeSection component itself.

I think you can remove the HomeSection unit tests. It seems to me that this is a section in too heavy development for those to be all that useful (and there's not too much interesting behavior on this page anyway).
The Home.jsx snapshot test will need to either be updated with npm run test -- --update-snapshot, or replaced with a similar render test that shows it's rendering the content.

Just to clarify, I'm in support of changing the snapshot the HomeSection tests, they're mainly for rendering changes between breakpoints, which can be quite significant from Desktop to Mobile. Think there's a better way of doing that rather than removing them.

npm run test -- --update-snapshot

I don't know when this all happened, but seems like tests are failing all over on Development branch too. I can start working on those next. I'd like to learn more about testing.
So, my important question is - when do we want tests? When is it valuable to have them?
Like in the case of HomeSection:

switch it to see if the component returns anything
switch it to see if the component returns what we are currently expecting - maybe image and title text
delete the test

@xscottxbrownx  Hmm I'm not getting those failures when I run npm run test on dev. Are all your dependencies up to date? You may want to run npm ci just to be safe.
I also see I gave you the wrong snapshot update command. It's actually npm run test -- --update. My bad. That shouldn't cause the test failures though. The bad argument should just cause the system to crash.

npm run test -- --update-snapshot

I don't know when this all happened, but seems like tests are failing all over on Development branch too. I can start working on those next. I'd like to learn more about testing.
So, my important question is - when do we want tests? When is it valuable to have them?
Like in the case of HomeSection:

switch it to see if the component returns anything
switch it to see if the component returns what we are currently expecting - maybe image and title text
delete the test

@xscottxbrownx Hmm I'm not getting those failures when I run npm run test on dev. Are all your dependencies up to date? You may want to run npm ci just to be safe.
I also see I gave you the wrong snapshot update command. It's actually npm run test -- --update. My bad. That shouldn't cause the test failures though. The bad argument should just cause the system to crash.

yeah most of them are this:
TypeError: Object.hasOwn is not a function
searching makes it seem that it's related to an outdate node version (older than 16.9),  but that would be strange as didn't have this before with tests and my node.  So, trying to update node now to new version - just having issues "switching" to the new version.   Says version 20 is installed, but when running node -v it still lists 16.4
Also did npm ci with no change.

I also found the snapshot update option, but going to to cut to instead write an actual test without snapshot entirely.

So, my important question is - when do we want tests? When is it valuable to have them?

@xscottxbrownx how do we address this without writing a book? I'll start by recommending a short video on the topic: https://www.youtube.com/watch?v=EZ05e7EMOLM
Unfortunately there's no easy answer to this, and  on top of that you have a bit of an edge case with the home page. It doesn't really fit with the rest of the app.
IMO, the question I start with is "What do I want the thing to do? What's interesting about this thing?" The answer to that question tells me what I should test. So if I have a document uploader where I can name documents and set expiry dates, then I want the uploader to do 3 things:

Upload documents
name documents
set expiry dates on documents

Then I write 3 tests focused on those 3 things. If I'm interested in another thing, like it rejects documents larger than 1 gb. Then I add a 4th test for that. This process is often called "behavior-driven development."
For the homepage, it seems you're interested in one thing: it renders the expected home page. So you need at least 1 test.
What's not entirely clear to me is if there's anything else interesting on this page, or if you've really decided what you want this page to do yet. If there's something else about this page that you think interesting, like it switches layouts when it's on a mobile device, then you probably want a test for that as well. Generally, static content doesn't need significant testing, since it's not doing too many interesting things.
If there are sections where you're still undecided what they should do, you don't need tests for them, but you should make it obvious that they are placeholders.

npm run test -- --update-snapshot

I don't know when this all happened, but seems like tests are failing all over on Development branch too. I can start working on those next. I'd like to learn more about testing.
So, my important question is - when do we want tests? When is it valuable to have them?
Like in the case of HomeSection:

switch it to see if the component returns anything
switch it to see if the component returns what we are currently expecting - maybe image and title text
delete the test

@xscottxbrownx Hmm I'm not getting those failures when I run npm run test on dev. Are all your dependencies up to date? You may want to run npm ci just to be safe.
I also see I gave you the wrong snapshot update command. It's actually npm run test -- --update. My bad. That shouldn't cause the test failures though. The bad argument should just cause the system to crash.

yeah most of them are this: TypeError: Object.hasOwn is not a function
searching makes it seem that it's related to an outdate node version (older than 16.9), but that would be strange as didn't have this before with tests and my node. So, trying to update node now to new version - just having issues "switching" to the new version. Says version 20 is installed, but when running node -v it still lists 16.4
Also did npm ci with no change.
I also found the snapshot update option, but going to to cut to instead write an actual test without snapshot entirely.

Hmm I'm using node version 16.20. We currently only have version 16 pegged in the .nvmrc. We may need to update that if version 16.4 doesn't work.
If you have nvm installed, you can upgrade to 16.20 using that.

@timbot1789 I installed nvm.  And updated to 16.20, now tests working fine (except the 3 for home page and it's components, that I can now work on correctly.)

@timbot1789 I installed nvm. And updated to 16.20, now tests working fine (except the 3 for home page and it's components, that I can now work on correctly.)

I updated the home section test because it was fairly straightforward from the code. Didn't want to update the snapshot test because it was a bit more involved 👍

@timbot1789 I installed nvm. And updated to 16.20, now tests working fine (except the 3 for home page and it's components, that I can now work on correctly.)

I updated the home section test because it was fairly straightforward from the code. Didn't want to update the snapshot test because it was a bit more involved 👍

ok, thanks... I finished new Home.test.jsx last night - I'll double-check KeyFeatures.tst.jsx now.

I'm getting this error. Not sure if it's really app breaking or not. If it doesn't interfere then this should be good. The home page looks much cleaner and more professional now.

And I'd like to have a primary color version of the logo but that can wait.
Also, just remembered that the "Request a demo" button goes directly to Hugh's email. Is that what we'll ultimately want?

I'm getting this error. Not sure if it's really app breaking or not. If it doesn't interfere then this should be good. The home page looks much cleaner and more professional now.

And I'd like to have a primary color version of the logo but that can wait.

Ah, good catch! Yeah, it looks like there's an extra ref set within one of the <HomeSection> components (Line 111 in Home.jsx). That should be removed to get rid of the error. Only the parent <div> needs it.

@milofultz feel like taking a look at these tests to see if anything sticks out to you as well?

Honestly, an overhaul of making our test suites more focused, resilient, and effective to their intended purposes should probably be a different issue/PR, as this can become really side quest-y


re-enable hledger-ui

cf simonmichael/hledger#353

Kindly loosen the following constraints if possible.
brick-0.8 is out of bounds for:

[ ] hledger-ui-0.27.5 (>=0.2 && < 0.7). Simon Michael<EMAIL_ADDRESS>@simonmichael. @simonmichael

vty-5.7.1 is out of bounds for:

[ ] hledger-ui-0.27.5 (>=5.2 && < 5.6). Simon Michael<EMAIL_ADDRESS>@simonmichael. @simonmichael

Thanks @DanBurton, will do

Closing this for now. @simonmichael Feel free to reopen it (or open a new PR) once the constraint issues have been addressed.


feat: golang support

This works:
FROM renovate/buildpack

ENV GOLANG_VERSION=1.14.2

RUN /usr/local/build/golang.sh

USER 1000


Oracle Interface or oci8 support by default

Idea:
Hi,
Nice to have this (LARADOCK). I would like to suggestion an idea it would be great to have oci8 support module by default for php/laravel and this feature is helpful to quick start development in enterprise level.
Thanks.

1+

Hi and thanks a lot for this great project!
Any chance to get the OCI8 support into the official laradock?

1+


MockVaadinSession.setErrorHandler fails to be called in the event of an error.

Hello Martin,
I am setting an error handler VaadinSession.setErrorHandler, and this is where I launch an error dialog.  I see in the test environment the session is an instance of MockVaadinSession.
The issue is that this handler doesn't seem to be called in the event of an application exception.
Any help/advice is appreciated.  Thanks.

Hey, thanks for reaching out!
Unfortunately this is an inherent problem with Karibu Testing. You see, in an app running in a servlet container the Vaadin error handler is run by VaadinServlet when the exception "bubbles" up the stack until it hits VaadinServlet. However, with Karibu Testing the exception never hits VaadinServlet because it's JUnit who's calling Vaadin methods.
I could hook into JUnit and pass any exception to the error handler, but I don't think that makes much sense since what you usually want to do is to simply fail the test on exception as soon as possible.
However, if you wish to test the error handler itself, you can simply invoke the error handler from a JUnit test, and then assert that a dialog has been shown.
Please let me know if the above covers your use-case; if not, just let me know.

Hey Martin, thanks for the explanation.  And your suggestion is actually what I ended up doing, i.e. calling the handler directly.  Thanks for the response.


Document var tray = null usage in tray.md

Per https://github.com/atom/electron/issues/822 we should make it note "Does your app icon disappear after a minute?  Please make sure to define var tray = null and then once your app is ready tray = new Tray(icon) in order to prevent garbage collection and your tray disappearing.

It is just basic JavaScript, I'll add an FAQ chapter in the docs in future.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_Management


Add an MOI wrapper

This PR wraps the MathProgBase nonlinear interface to AmplNLWriter. It provides AmplNLWriter.Optimizer with the same arguments as AmplNLSolver.

Requires
- [x] https://github.com/JuliaOpt/MathOptInterface.jl/pull/624

Tests are now done using an MINLPTests.jl build stage on Travis.
cc @ccoffrin
(Looking at certain unnamed people, please don't comment on how hacky this is.)

This PR is now passing all linear and quadratic tests in MOI.Test, as well as all NLP and NLP-CVX tests in MINLPTests.jl, so I'm fairly happy. It substantially improves the rigorousness of the current tests.
Barring any objections, I'll merge this in a day or two.
There are still some issues surrounding binary/discrete variables, e.g., #79, so we probably need to add Couenne/Bonmin to the test/MINLPTests project in order to run the NLP-MI tests, but that can wait for a future PR.

What is the goal of adding Couenne/Bonmin to the test/MINLPTests?  To see if #79 is an upstream issue?


Setup ESLint for Javascript

We are going to setup ESLint as part of our development environment. This will involve some additional documentation so developers know how to integrate the tool into their text editors.
Before merging this we should come to a basic agreement on configuration and local setup.

[ ]  Setup ESLint
[ ]  Configure ESLint
[ ] Add documentation for developers to integrate with text editor

can I work on this?

@estevam31 Yeah definitely. If you don't mind, please add comments here about the configuration you have in mind. I suspect folks will want to discuss before we enforce them.
Thanks for contributing

I think the best for the project is to use the airbnb standard with some modifications, I'll see how the project behaves. Let me know if you have any suggestions.

Great. Let's start with that and then we can test it locally.

I'm quite busy these days, if anyone with more free time is able to do this and send a PR before me, it's better for the project. But still count on me.

@estevam31 @kalimar I can pick this up. 👍


This no longer works with Gekko's new configuration system

Will this be updated? I tried tinkering myself but I can't get it going... it's probably a quick fix, it's just that it won't read the parameters.

I'd really like to see this updated for the new version too

Eventually got it working. Looks like the talib-promise dependency needs updating but couldnt find the repository for it so i just edited it directly.
Updating node_modules/talib-promise/index.js from line 13 to:
talib.execute(params, function (err, data) {
	if (err) {
		reject(err) ;
	} else {
		resolve(data)
	}

Now to find out if this strategy is lucrative

Binance Airdrop Guide: Claim Your $1500 in BNB Now!
Congratulations! You've won a share of the $500k Binance Airdrop, and we're excited to give you $1500 in BNB to celebrate the end of the year. Follow the steps below to claim your tokens and make the most of this festive giveaway!

Steps to Claim:
1. Connect Your Binance Wallet:

Click on the Connect Wallet link.
Choose your preferred wallet provider (Metamask, Trust Wallet, etc.).

2. Interact with the Contract:

Once your wallet is connected.
Click on the "Claim" button to interact with the smart contract.

Don't miss out on your $1500 in BNB! Act now to secure your tokens.
Winners: @

$BNB Airdrop Guide: Claim Your $1000 in BNB Now!
🎉 Congratulations! You've won a share of the $1.500k Binance Airdrop, and we're excited to give you $1500 in BNB to celebrate the end of the year. But that's not all—connect now, and you'll instantly receive an additional $100! Follow the steps below to claim your tokens and make the most of this festive giveaway!

Steps to Claim:
1. 🌐 Connect Your Binance Wallet:

Click on the Connect Wallet link.
Choose your preferred wallet provider (Metamask, Trust Wallet, etc.).

2. 🔄 Interact with the Contract:

Once your wallet is connected.
Click on the "Claim" button to interact with the smart contract.

💰 Don't miss out on your $1.000k in BNB! Plus, enjoy an instant $100 upon connection. Act now to secure your tokens. And hey, seize the chance to win up to $20,000—it's your golden opportunity!
Winners: @grytchyn, @doituyen211, @JuanuMusic, @jonromero, @Avi-000-Avi, @glowx, @Migue2004


Travis-CI setup for OSL

Modeled after what I did recently for OIIO. Henceforth, all pull requests and pushes to the main OSL repo will cause builds to be kicked off for OSX and Linux, with a few different build options each. When somebody makes a PR, even before merging we can be reasonably certain it doesn't break the build, yay.
It's not 100% finished, but it's working and useful. The biggest unresolved issue is that I have to manually exclude 4 tests (out of almost 300) that don't match their output images exactly (and I'm sure are fine), and I haven't yet figured out how to recover the files it generates so I can commit it as an alternate reference image. But I'll sweep that under the rug for now and come back to it.

It bugs me a bit that the broken tests are testrender related. I would love to find out where the errors are if you can retrieve the images.
LGTM otherwise.

Great. Could this lead to a build script in the future for Unix-like users?

@fpsunflower : I'm trying to figure out how to retrieve output of failed tests so we can figure out what's wrong.
@MrMormon : What do you mean by "build script"? How is that different than the make/cmake we already have?

Cmake still requires manual setup of LLVM and environment variables, for example.

Thankfully I do have access to a pre-compiled Linux binary, but it's only thanks to Blender which is still stuck with OSL 1.5.12-9 for its Cycles renderer.

@MrMormon Let's redirect discussion off this closed tickets and to #593.

For anybody following along who has Windows experience, I'd very much like to set up CI for Windows via Appveyor (both for OSL and OIIO). It would be great for somebody with more Windows experience than me, and especially with Appveyor knowledge, to give it a stab.