New MCU showing "offline" while sleeping when on 4G

Describe the bug
After upgrading my car to MCU2, the sleep state is shown as "offline" when I'm not on WiFi.
When on WiFi it works fine and is correctly shown as "asleep".
Note that when I had MCU1 my car never fell asleep, so this issue may have been present before without me noticing.
Expected behavior
Should show as asleep regardless of being on WiFi or 4G
How to reproduce it (as minimally and precisely as possible):
For me, reproducing is as simple as leaving the car somewhere without WiFi for a couple of hours.
Relevant entries from the logs
Note that it did not start showing offline in states until 18:27:32
teslamate_1           | 2021-01-13 18:06:48.705 car_id=1 [info] Start / :online
teslamate_1           | 2021-01-13 18:06:48.724 car_id=1 [info] Connecting ...
teslamate_1           | 2021-01-13 18:09:54.658 car_id=1 [info] Suspending logging
teslamate_1           | 2021-01-13 18:11:32.326 [info] Connection was closed (a:1|t:1|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:11:43.221 [info] Connection was closed (a:1|t:2|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:16:31.374 [info] Connection was closed (a:1|t:1|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:16:42.283 [info] Connection was closed (a:1|t:2|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:21:31.266 [info] Connection was closed (a:1|t:1|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:21:42.231 [info] Connection was closed (a:1|t:2|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:21:53.125 [info] Connection was closed (a:1|t:3|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:22:04.014 [info] Connection was closed (a:1|t:4|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:22:20.731 car_id=1 [info] Fetching vehicle state ...
teslamate_1           | 2021-01-13 18:22:20.908 [info] Connection was closed (a:1|t:5|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:22:21.629 [info] control:hello – 0
teslamate_1           | 2021-01-13 18:22:51.630 [info] Stream.Timeout / 5
teslamate_1           | 2021-01-13 18:22:51.809 [info] Connection was closed (a:1|t:6|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:22:52.789 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:23:22.790 [info] Stream.Timeout / 6
teslamate_1           | 2021-01-13 18:23:22.969 [info] Connection was closed (a:1|t:7|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:23:23.964 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:23:53.965 [info] Stream.Timeout / 7
teslamate_1           | 2021-01-13 18:23:54.141 [info] Connection was closed (a:1|t:8|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:23:55.114 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:24:25.115 [info] Stream.Timeout / 8
teslamate_1           | 2021-01-13 18:24:25.293 [info] Connection was closed (a:1|t:9|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:24:26.257 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:24:56.258 [info] Stream.Timeout / 9
teslamate_1           | 2021-01-13 18:24:56.435 [info] Connection was closed (a:1|t:10|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:24:57.381 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:25:27.382 [info] Stream.Timeout / 10
teslamate_1           | 2021-01-13 18:25:27.559 [info] Connection was closed (a:1|t:11|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:25:28.486 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:25:58.487 [info] Stream.Timeout / 11
teslamate_1           | 2021-01-13 18:25:58.664 [info] Connection was closed (a:1|t:12|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:25:59.695 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:26:29.696 [info] Stream.Timeout / 12
teslamate_1           | 2021-01-13 18:26:29.873 [info] Connection was closed (a:1|t:13|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:26:30.809 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:27:00.809 [info] Stream.Timeout / 13
teslamate_1           | 2021-01-13 18:27:00.987 [info] Connection was closed (a:1|t:14|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:27:02.010 [error] Vehicle Error: Vehicle is offline
teslamate_1           | 2021-01-13 18:27:32.010 [info] Stream.Timeout / 14
teslamate_1           | 2021-01-13 18:27:32.010 car_id=1 [info] Fetching vehicle state ...
teslamate_1           | 2021-01-13 18:27:32.188 [info] Connection was closed (a:1|t:15|d:1). Reconnecting …
teslamate_1           | 2021-01-13 18:27:32.795 car_id=1 [info] Start / :offline
teslamate_1           | 2021-01-13 18:27:32.803 car_id=1 [info] Disconnecting ...

Screenshots
Screenshot showing part of my day today. sleeping while at home on WiFi, then going offline with the same intervals as when sleeping when I'm at work.

Data
States table from database for today:
1325,online,2021-01-13 03:00:30.809,2021-01-13 03:16:32.486323,1
1326,asleep,2021-01-13 03:16:32.486323,2021-01-13 04:15:38.369,1
1327,online,2021-01-13 04:15:38.369,2021-01-13 04:31:33.45213,1
1337,online,2021-01-13 11:13:11.793,2021-01-13 11:29:20.110467,1
1340,offline,2021-01-13 12:49:40.016239,2021-01-13 13:43:45.423,1
1342,offline,2021-01-13 15:08:15.805682,2021-01-13 15:51:45.856,1
1343,online,2021-01-13 15:51:45.856,2021-01-13 16:12:31.725538,1
1322,asleep,2021-01-13 00:46:10.716873,2021-01-13 01:45:18.443,1
1323,online,2021-01-13 01:45:18.443,2021-01-13 02:01:23.453075,1
1338,offline,2021-01-13 11:29:20.110467,2021-01-13 12:28:28.131,1
1345,online,2021-01-13 17:06:48.298,2021-01-13 17:27:32.795652,1
1332,offline,2021-01-13 07:44:02.116716,2021-01-13 08:43:08.958,1
1333,online,2021-01-13 08:43:08.958,2021-01-13 09:04:32.771329,1
1334,offline,2021-01-13 09:04:32.771329,2021-01-13 09:58:06.409,1
1341,online,2021-01-13 13:43:45.423,2021-01-13 15:08:15.805682,1
1346,offline,2021-01-13 17:27:32.795652,,1
1329,online,2021-01-13 05:30:39.126,2021-01-13 06:33:55.907706,1
1331,online,2021-01-13 07:28:00.659,2021-01-13 07:44:02.116716,1
1335,online,2021-01-13 09:58:06.409,2021-01-13 10:19:06.890721,1
1336,offline,2021-01-13 10:19:06.890721,2021-01-13 11:13:11.793,1
1324,asleep,2021-01-13 02:01:23.453075,2021-01-13 03:00:30.809,1
1328,asleep,2021-01-13 04:31:33.45213,2021-01-13 05:30:39.126,1
1330,offline,2021-01-13 06:33:55.907706,2021-01-13 07:28:00.659,1
1339,online,2021-01-13 12:28:28.131,2021-01-13 12:49:40.016239,1
1344,offline,2021-01-13 16:12:31.725538,2021-01-13 17:06:48.298,1
Environment

TeslaMate version: 1.21.1
Type of installation: Docker (with database server manualy installed)
OS TeslaMate is installed on: Ubuntu container
User OS & Browser: Windows10 & Manjaro + Chrome & Vivaldi

What Tesla software version are you on? I saw the same behavior randomly and then after an update it went back to ‘asleep’.

What Tesla software version are you on? I saw the same behavior randomly and then after an update it went back to ‘asleep’.

I got the same too - MCU2 upgrade with latest update 2020.48.30. I turned of Wifi to try and identify why my car sleeps for an hour and then wakes for 15 mins all the time. In the time that i was using LTE on, sleeping was reported as offline

I got the same too - MCU2 upgrade with latest update 2020.48.30. I turned of Wifi to try and identify why my car sleeps for an hour and then wakes for 15 mins all the time. In the time that i was using LTE on, sleeping was reported as offline

What Tesla software version are you on? I saw the same behavior randomly and then after an update it went back to ‘asleep’.

While on MCU 2 I've had 44.25, 48.10, 48.12.1, 48.26, and now 48.30. All of them have had the same issue.

What Tesla software version are you on? I saw the same behavior randomly and then after an update it went back to ‘asleep’.

While on MCU 2 I've had 44.25, 48.10, 48.12.1, 48.26, and now 48.30. All of them have had the same issue.

While on MCU 2 I've had 44.25, 48.10, 48.12.1, 48.26, and now 48.30. All of them have had the same issue.

I never got an upgrade, just had MCU 2 since I got the car. There might be something with that upgrade, but I know it is not TeslaMate related. It eventually goes back to "sleeping" as opposed to "offline"

I think the 15 min awake is to contact the mothership to check for updates and stuff. After a while it should go into a deeper sleep (at least that's what I've heard).

It does, after about 7 - 10, 15 - 20 minutes each cycle it will fall into a deep sleep. Used to be for almost 48 hours, now I am seeing it at 24 hours.

While on MCU 2 I've had 44.25, 48.10, 48.12.1, 48.26, and now 48.30. All of them have had the same issue.

I never got an upgrade, just had MCU 2 since I got the car. There might be something with that upgrade, but I know it is not TeslaMate related. It eventually goes back to "sleeping" as opposed to "offline"

I think the 15 min awake is to contact the mothership to check for updates and stuff. After a while it should go into a deeper sleep (at least that's what I've heard).

It does, after about 7 - 10, 15 - 20 minutes each cycle it will fall into a deep sleep. Used to be for almost 48 hours, now I am seeing it at 24 hours.

While on MCU 2 I've had 44.25, 48.10, 48.12.1, 48.26, and now 48.30. All of them have had the same issue.

I never got an upgrade, just had MCU 2 since I got the car. There might be something with that upgrade, but I know it is not TeslaMate related. It eventually goes back to "sleeping" as opposed to "offline"

I think the 15 min awake is to contact the mothership to check for updates and stuff. After a while it should go into a deeper sleep (at least that's what I've heard).

It does, after about 7 - 10, 15 - 20 minutes each cycle it will fall into a deep sleep. Used to be for almost 48 hours, now I am seeing it at 24 hours.

Hmm...mine has always done the sleep for an hour, wake for 15 mins...never has it slept for more then an hour before waking even if the car is sat on the drive for a couple days and not driven or checked with the official Tesla app.
Not sure if this could be done, but it would be useful if we could identify via Teslamate when it sleeps normally Vs deep sleep...I believe teslafi does this?

While on MCU 2 I've had 44.25, 48.10, 48.12.1, 48.26, and now 48.30. All of them have had the same issue.

I never got an upgrade, just had MCU 2 since I got the car. There might be something with that upgrade, but I know it is not TeslaMate related. It eventually goes back to "sleeping" as opposed to "offline"

I think the 15 min awake is to contact the mothership to check for updates and stuff. After a while it should go into a deeper sleep (at least that's what I've heard).

It does, after about 7 - 10, 15 - 20 minutes each cycle it will fall into a deep sleep. Used to be for almost 48 hours, now I am seeing it at 24 hours.

Hmm...mine has always done the sleep for an hour, wake for 15 mins...never has it slept for more then an hour before waking even if the car is sat on the drive for a couple days and not driven or checked with the official Tesla app.
Not sure if this could be done, but it would be useful if we could identify via Teslamate when it sleeps normally Vs deep sleep...I believe teslafi does this?

Not sure if this could be done, but it would be useful if we could identify via Teslamate when it sleeps normally Vs deep sleep...I believe teslafi does this?

Really? You have never had it go into a long stretch of sleep? Similar to this (this is a portion of a seven day graphic). Also pretty sure that the end of it, it is in a "deep sleep":

Not sure if there is something in the API that shows deep versus normal sleeping (@adriankumpf), but I know TeslaMate's dashboard seems to show varying amounts of data when it sleeps and when it shows the least amount of data, I know it will keep sleeping for hours.

Not sure if this could be done, but it would be useful if we could identify via Teslamate when it sleeps normally Vs deep sleep...I believe teslafi does this?

Really? You have never had it go into a long stretch of sleep? Similar to this (this is a portion of a seven day graphic). Also pretty sure that the end of it, it is in a "deep sleep":

Not sure if there is something in the API that shows deep versus normal sleeping (@adriankumpf), but I know TeslaMate's dashboard seems to show varying amounts of data when it sleeps and when it shows the least amount of data, I know it will keep sleeping for hours.

Not sure if this could be done, but it would be useful if we could identify via Teslamate when it sleeps normally Vs deep sleep...I believe teslafi does this?

Really? You have never had it go into a long stretch of sleep? Similar to this (this is a portion of a seven day graphic). Also pretty sure that the end of it, it is in a "deep sleep":

Not sure if there is something in the API that shows deep versus normal sleeping (@adriankumpf), but I know TeslaMate's dashboard seems to show varying amounts of data when it sleeps and when it shows the least amount of data, I know it will keep sleeping for hours.

Nope :( never...This is 7 days...

Not sure if this could be done, but it would be useful if we could identify via Teslamate when it sleeps normally Vs deep sleep...I believe teslafi does this?

Really? You have never had it go into a long stretch of sleep? Similar to this (this is a portion of a seven day graphic). Also pretty sure that the end of it, it is in a "deep sleep":

Not sure if there is something in the API that shows deep versus normal sleeping (@adriankumpf), but I know TeslaMate's dashboard seems to show varying amounts of data when it sleeps and when it shows the least amount of data, I know it will keep sleeping for hours.

Nope :( never...This is 7 days...

Nope :( never...This is 7 days...
You're doing daily drives though. I would try and leave it for more than a day without driving if possible to see if it goes into deep sleep then. Also, not sure if being connected to power has an effect since it would not be so critical to deep sleep when power is available. But this sleeping thing is new to me too, so I'm no expert.

Nope :( never...This is 7 days...
You're doing daily drives though. I would try and leave it for more than a day without driving if possible to see if it goes into deep sleep then. Also, not sure if being connected to power has an effect since it would not be so critical to deep sleep when power is available. But this sleeping thing is new to me too, so I'm no expert.

Nope :( never...This is 7 days...

You're doing daily drives though. I would try and leave it for more than a day without driving if possible to see if it goes into deep sleep then. Also, not sure if being connected to power has an effect since it would not be so critical to deep sleep when power is available. But this sleeping thing is new to me too, so I'm no expert.

I've done a few days with no driving and not plugged in before - still does the same 1 hour on then 15 mins awake - have changed tokens also to ensure no 3rd party apps are requesting access...still no effect...i guess ill have to keep experimenting! (or wait for the next vehicle update and hope it resolves it!)

Nope :( never...This is 7 days...

You're doing daily drives though. I would try and leave it for more than a day without driving if possible to see if it goes into deep sleep then. Also, not sure if being connected to power has an effect since it would not be so critical to deep sleep when power is available. But this sleeping thing is new to me too, so I'm no expert.

I've done a few days with no driving and not plugged in before - still does the same 1 hour on then 15 mins awake - have changed tokens also to ensure no 3rd party apps are requesting access...still no effect...i guess ill have to keep experimenting! (or wait for the next vehicle update and hope it resolves it!)

Going to ask the awful questions - have you tried a reboot? Have you also tried to power off the vehicle for three - five minutes?
You could ask Tesla to investigate, but the are going to adamantly say it’s the third-party app.
I will also be completely honest - never have I attempted to let my Tesla sleep on cellular because if it’s not home, Sentry Mode is on. Does it show offline at home if you kill WiFi on the Tesla?

Going to ask the awful questions - have you tried a reboot? Have you also tried to power off the vehicle for three - five minutes?
You could ask Tesla to investigate, but the are going to adamantly say it’s the third-party app.
I will also be completely honest - never have I attempted to let my Tesla sleep on cellular because if it’s not home, Sentry Mode is on. Does it show offline at home if you kill WiFi on the Tesla?

Hello,
During the last 7 days my MS75D was mostly asleep (good) but about every 2 days it wakes up and stays awake for about 12 hours:
Is this normal behavior? I notice a drop in the SoC (1% to 2% or so) every time it stays awake for such long periods. I would like to avoid this.
I would prefer it if the car would just keep sleeping or only wake up for 15 minutes and then go back to sleep. Why does it stay awake for 12 hours??
hh

Hello,
During the last 7 days my MS75D was mostly asleep (good) but about every 2 days it wakes up and stays awake for about 12 hours:
Is this normal behavior? I notice a drop in the SoC (1% to 2% or so) every time it stays awake for such long periods. I would like to avoid this.
I would prefer it if the car would just keep sleeping or only wake up for 15 minutes and then go back to sleep. Why does it stay awake for 12 hours??
hh

Hello, During the last 7 days my MS75D was mostly asleep (good) but about every 2 days it wakes up and stays awake for about 12 hours: Is this normal behavior? I notice a drop in the SoC (1% to 2% or so) every time it stays awake for such long periods. I would like to avoid this. I would prefer it if the car would just keep sleeping or only wake up for 15 minutes and then go back to sleep. Why does it stay awake for 12 hours?? hh

I think what you are seeing is normal software update behavior. Seen my 2019 Model S randomly stay awake for 12 hours after some updates and then sleep peacefully for days with some updates. It is then my belief that something in the software update throws off the coding for the sleep pattern and thus the inconsistent sleeping.
Maybe bias, but I would say that this seemingly effects S and X owners more than 3 and Y owners. Just by following comments here and on Reddit.

Hello, During the last 7 days my MS75D was mostly asleep (good) but about every 2 days it wakes up and stays awake for about 12 hours: Is this normal behavior? I notice a drop in the SoC (1% to 2% or so) every time it stays awake for such long periods. I would like to avoid this. I would prefer it if the car would just keep sleeping or only wake up for 15 minutes and then go back to sleep. Why does it stay awake for 12 hours?? hh

I think what you are seeing is normal software update behavior. Seen my 2019 Model S randomly stay awake for 12 hours after some updates and then sleep peacefully for days with some updates. It is then my belief that something in the software update throws off the coding for the sleep pattern and thus the inconsistent sleeping.
Maybe bias, but I would say that this seemingly effects S and X owners more than 3 and Y owners. Just by following comments here and on Reddit.


Core text vars

Core

New type vars for size, weight, and letter spacing

Text

Use new core type vars in text components

@jaketrent I've added new core type vars for size, weight, and letter spacing.

Sizes: the numeric scale now represents all font sizes across the scale
Weights: the new scale is numeric as opposed to being tied to the weight names which vary between fonts
Letterspacing: the new scale is more focused on letterspacing instead of using names that are tied to font sizes

I wasn't sure if it would be better to remove the old variables or leave them for now in order to not cause a breaking change.

@jaketrent I've added new core type vars for size, weight, and letter spacing.

Sizes: the numeric scale now represents all font sizes across the scale
Weights: the new scale is numeric as opposed to being tied to the weight names which vary between fonts
Letterspacing: the new scale is more focused on letterspacing instead of using names that are tied to font sizes

I wasn't sure if it would be better to remove the old variables or leave them for now in order to not cause a breaking change.


Re-render doc-site with latest documentation assets

On Thursday Apr 27 the entire doc-site (both readthedocs and pkgdown) needs to be re-render to reflect all the latest changes to the the documentation assets.

@pablo-gar I am closing this ticket since it's past Apr 27. Docs can be generated on demand now so I think small, pre-1.0 changes can go in without creating additional tickets.


Create sprint #1 demo

Film video
Edit video
Add video to slides
Submit video

Link to the presentation with demo:
https://docs.google.com/presentation/d/13cCb60PW9PQ572B05eJkWOyCbz9AZ2aGChI0wglltlw/edit?usp=sharing


use with DB::

Thanks for the great work.
Is there any way to change this package to work with the DB:: query builder? for example the query below:
DB::select("select name from users USE INDEX (names) limit 10");
being written like:
DB::table('users')->select('name')->useIndex('names')->limit(10);

@omidMolaverdi put this in your AppServiceProvider to use it anywhere 👍
Builder::macro('useIndex', function (string $indexName) {
    $tableName = $this->getQuery()->from;
    $this->getQuery()->from(\DB::raw("{$tableName} USE INDEX ({$indexName})"));

    return $this;
});


Support dynamic chunklists with pruned segments

This will allow the dynamically generated playlists to reference an endpoint that can dynamically generate the chunklists/renditions so that the chunklist can be pruned down in duration, either by pruning the beginning, the end, or both.
A chunklist can also be pruned in to a "preview" which will take segments from 3 different spots within the chunklist in order to form a stitched-together clip/preview of non-continuous segments.

Is it bad that I'm using the word chunklist everywhere instead of rendition or variant?

Nevermind. Just realized your console.log was on a demo. Not awake yet.  Merging now.

I only slept until 11am haha. Thanks for the merge!
I've got another small fix I'll open another PR for.


chore: support Python 3.12

/e2e --tests=substrafl,sdk,mnist,camelyon

End to end tests: :hourglass_flowing_sand: STARTING


not working on Syno (restart loop)

Funny thing is: I already used this container some months ago and it worked flawlessly, but ATM I can't get it to work, no matter where I set it up.
vpn    |
vpn    | ---- Running with the following variables ----
vpn    | Kill switch: on
vpn    | HTTP proxy: off
vpn    | SOCKS proxy: off
vpn    | Proxy username secret: none
vpn    | Proxy password secret: none
vpn    | Allowing subnets: none
vpn    | Using OpenVPN log level: 3
vpn    | Listening on: none
vpn    | Using configuration file: /data/vpn/nl24.ovpn
vpn    | Creating /data/vpn/nl24.ovpn.modified and making required changes to that file.
vpn    | Changes made.
vpn    |
vpn    | Creating VPN kill switch and local routes.
vpn    | Allowing established and related connections...
vpn    | Allowing loopback connections...
vpn    | Allowing Docker network connections...
vpn    | Allowing specified subnets...
vpn    | Allowing remote servers in configuration file...
vpn    |   Using:
vpn    |     nl-24.goosevpn.com (IP: <IP_ADDRESS> PORT: 443 PROTOCOL: tcp)
vpn    | Allowing connections over VPN interface...
vpn    | Preventing anything else...
vpn    | iptables rules created and routes configured.
vpn    |
vpn    | Running OpenVPN client.
vpn    |
vpn    | Options error: In [CMD-LINE]:1: Error opening configuration file: /data/vpn/nl24.ovpn.modified
vpn    | Use --help for more information.

Notice:
nl24.ovpn was taken and nl24.ovpn.modified was created, but suddenly it cant read it. If that is a problem the container could set it's rights automatically. Actually i don't know what exactly is wrong here, but seems like:

permission
config not working (but working on another machine)

I guess this issue is more relevant than the other :). What is the image ID of the VPN image being used? I wonder if an update messed something up for you.

sha256:2f0459a7a1468f593cffc35bb5eb24ad9a6d19445e532c70ed6cd19d32065fcb
Thanks

latest works on Syno for me, maybe do a pull for the latest image? Of remove the modified OVPN file as you will see if permissions are correct as it will create a new file


Clean up English messages.po

Cleaned up en/messages.po to reduce size of messages.po for new language translators

Removed old entries
Removed one "translated" entry
Added new messages since 1.92.1

This will certainly be helpful, especially for new translators to help them avoid wasting their effort translating obsolete strings, thanks @tkusano! 🙏


[feature] WiFi mesh integration #170

Closes #170

Coverage: 69.06% (-30.2%) from 99.289% when pulling beb02f66d17c876aba92c0be8a61ffb653f4bb0c on issues/170-wifi-mesh into 6c2d8a46f773fec6a45b497fa9a07c1e9568d4dd on master.


adding an advanced method for utilising cache with fetch

based off issue 7639

Will re-check build again


New RlpList type

Introduce new RlpList type using BoundedVec
Add bounded_vec_from_array function
Update project to use new structure

Looks quite good, several small things to improve and we can merge.


Bug with Mac

Hello,
I have this error when I try to run Anylogic with pyCom :
Error during model startup:
root.pyCom:
Problem with setting up Python Subprocess
Caused by: Python process failed to properly start. This process needs to be started by a 'live' agent. Error message from process: 'Traceback (most recent call last):
File "/var/folders/ln/hpcn96rx41j38zx70j7m_r0h0000gn/T/pypeline-server_9841271844594326379.py", line 5, in
from json.decoder import JSONDecodeError
ImportError: cannot import name JSONDecodeError
Can you help me ?
Thank you in advance,
Mickael

It seems like you're trying to programmatically create the Python Communicator agent? If you can share your model, it would help in troubleshooting.

Thank you for your quick reply.
The model I try to run is the Simple Hospital (AI Tested) from the exemples.

When I try to use python3, I get this error :
Error during model startup:
root.pyCom:
ModuleNotFoundError("No module named 'numpy'")java.lang.RuntimeException:

Ah right, I see what the problem is now. Macs have Python 2 installed by default, so if you initialize the PyCommunicator with the default python, that's what it uses, and Pypeline does not support Python 2.
Your update fixes to use Python 3 but as the error suggests, you're missing the numpy library, so you'll need to install that (and any other dependencies mentioned by the model). The simplest way that usually works is python3 -m pip install <library>

Thank you for your reply.
The thing is I already installed this package
(python3 -m pip install numpy
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (1.19.5)).
This is why I don't understand. Does it need a special env to run ?

Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (1.19.5))

There might be a weird path issue happening related to you being on a Mac (I admittedly haven't extensively tested Pypeline on it).
Try following the instructions listed on the Wiki's Setup page but configure it to use the "python3" command. Make sure the path it shows is similar to the mentioned one. If it's not, let me know what it is


Make playerData public to get rid of Unity's warning

testing

This was already merged from my PR.


Prevent "Voiceover" from appearing in the script

Prevent chatgpt to add "voiceover" at the beginning of a paragraph or line.

Added a more general instruction which should prevent any indicators.


fix lowercase/upercase empty string

Pull Request Test Coverage Report for Build 560

4 of 4 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.03%) to 52.474%

Totals

Change from base Build 555:
0.03%

Covered Lines:
3372

Relevant Lines:
6426

💛  - Coveralls


Fix header file import for latest ProtoBuf in grpcio-tools

Fixes https://github.com/grpc/grpc/issues/29338.

Triggered prod:grpc/core/master/linux/grpc_build_protobuf_at_head

http://sponge/64ab4e1f-68c9-4acf-8ecc-099b0440e224

Well, as is, this would break build against the current version of third_party/protobuf submodule.
grpc_tools/main.cc:27:10: fatal error: google/protobuf/compiler/python/generator.h: No such file or directory

Actually, the situation is quite bad, since it seems that protobuf has recently renamed lots of their public headers.
E.g. see https://github.com/protocolbuffers/protobuf/pull/9669/files#diff-0dd62c8d5a58dc167544764d463de9e6129568f09bbe20145f6ab7d4632f6608R111
And it's not just about python, other languages would likely suffer from a similar problem:
https://github.com/protocolbuffers/protobuf/pull/9669/files#diff-deef97e4c58bafd49f04825aa86dcc91ee390d5d23d041c3d7ccb9d4529ee72b

I'm planning to update the ProtoBuf submodule again, and try to fix breakages. I see ProtoBuf pushed packages to various package management plaforms (like PyPI), it might be harder to yank all that.
The ProtoBuf build at head job is failing constantly for at least 2 weeks (around the mentioned commit). We might want to dig into why this didn't raise an alarm for both teams. https://screenshot.googleplex.com/Ahybmk6DF5WkTtG

Updating ProtoBuf is blocked by another issue: https://github.com/grpc/grpc/issues/29297.

We probably need to figure out who should fix the issue first.


Student cannot view submission on OK

Student submitted hw04, can view submission on the ok course page, but after clicking on it, she receives an error: key<PHONE_NUMBER>123520 not found.
Piazza post @314. See followups.

Just to clarify, on the main student view, the assignment reports that a submission was made. However, when clicking the "View submission" button in the side bar, the error that @robhuang posted occurs.

We are able to search for the submission and view it, but the normal "View submission" button still breaks.

I have an idea why it might happen; I'll work on it in the morning.

I tried it locally and can't replicate this problem; I'm hoping it's because master simply hasn't been deployed yet. @Sumukh Could that be it? If the deployment is synced... I'll need to keep trying.
@robhuang Is this what you mean? (This is the latest version of master and may not be deployed yet.)

@albert12132 Can I have access to the Piazza?

Deployed.


⚠ Jenkins Plugins Artifactory has degraded performance

In 9a2b2e0, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1889 ms

Resolved: Jenkins Plugins Artifactory performance has improved in a5616bc.


Documentation dataset format

What does this PR do?

Fixes # (issue)
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a GitHub issue? Please add a link
to it if that's the case.
[x] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines.
[x] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
Help for review
New section in the doc

Added new section about dataset format in the doc, directly review in the online doc: https://moon-ci-docs.huggingface.co/docs/trl/pr_2020/en/dataset_formats

New data utils

Added new data utils: maybe_apply_chat_template, maybe_extract_prompt, maybe_unpair_preference_dataset, apply_chat_template, extract_prompt and unpair_preference_dataset.
Added tests for all of them

Update dataset example files
As explained in this section we provide script for converting dataset to TRL style.
The goal is, for every dataset in trl-lib to have its corresponding script in examples/datasets

Updated all files in examples/datasets to closely match the documented format.

Start to update some example scripts
We have to update all scripts to make sure they comply with the new style.

Updated DPO script

I prefer dedicating further PR to update them all

Updated the SIMPLE_QUERY_CHAT_TEMPLATE to include a prompt generation

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Everything LGTM, with one main question about what is meant by "standard dataset format".

Yes... "standard" is in fact "non-conversational". But it's weird to define something by something it's not.
It's not easy to find relevant, accurate and easy-to-understand names. If anyone have any suggestions for changes, I'd be happy to use them.

What I'm wondering in particular is whether we expect users to preformat their datasets for each trainer or whether we accept some formats like messages as column that are automatically formatted in our scripts.

Currently we expect users to preformat their datasets. But I think supporting everything in the trainers can make sense:
from datasets import Dataset

standard_dataset = Dataset.from_dict(
    {
        "prompt": ["The sky is", "The sun is"],
        "completion": [" blue.", " in the sky."],
    }
)
AnyTrlTrainer(..., train_dataset=standard_dataset)  # ok

conversational_dataset = Dataset.from_dict({
    "prompt": [
        [{"role": "user", "content": "What color is the sky?"}], [{"role": "user", "content": "Where is the sun?"}],
    ],
    "completion": [
        [{"role": "assistant", "content": "It is blue."}], [{"role": "assistant", "content": "In the sky."}],
    ],
})
AnyTrlTrainer(..., train_dataset=conversational_dataset)  # currently not ok, but we can support it in the future.

Yes... "standard" is in fact "non-conversational".

Ah, that makes sense - thanks for the clarification. One thing I'm wondering is whether there's any real advantage to support standard vs conversational, since technically all standard datasets can be converted to conversational by wrapping the prompt, completion etc in the messages format.
Given that most models are chat models, one approach would be to support conversational datasets natively in the trainers, but allow users to also provide a preprocessed / tokenized dataset if they wish more flexibility.
This also is related to #1646


Benchmark_Comparisons.ipynb crashes in halo_model cell with python2

I get a crash in the penultimate cell of Benchmark_Comparisons.ipynb (the one titled 'Halo Model Plots') when running the notebook with a python2 kernel. The crash is a severe kernel crash. The error is attached:
Crash_information.txt
Tilman says that the crash is in the spline being called by the growth factor within the Eisenstein & Hu P(k) approximation:
Thread 0 Crashed:: ZMQbg/1  Dispatch queue: com.apple.main-thread
0   libgsl.23.dylib                   0x0000000108ca3f60 gsl_spline_eval_e + 10
1   _ccllib.so                        0x0000000108999d44 ccl_growth_factor + 84
2   _ccllib.so                        0x00000001089a4643 ccl_cosmology_compute_power_eh + 4323
3   ???                               0x00007f87fdc7ad60 0 +<PHONE_NUMBER>34208

I am using gsl 2.4, but upgraded to 2.5 and the problem persists.

One thing to check is to set ccl.debug_mode(True) at the beginning of the halomodel cell and then check the output in the terminal the notebook is running in.

When I do this I get a halo-model specific error in the terminal from the C:
ERROR 1047: ccl_halomod.c: halo_concentration(): Duffy (2008) virial concentration only valid for virial Delta_v or 200

I'm worried that this might be because I have an IF statement check in the Duffy (2008) halo concentration for a number (odelta) to be exactly equal to the float 200. I guess it could be that numerical rounding changes this from 200 to 200+epsilon and then this causes an error.

If I check the halo concentration routine directly it crashes nicely in the python when I give it an unsupported odelta:
import numpy as np
import pyccl as ccl
import pylab as plt
%matplotlib inline

cosmo = ccl.Cosmology(Omega_c=0.27, Omega_b=0.045, h=0.67, sigma8=0.8, n_s=0.96, Neff=3.04,
                      transfer_function='class',
                      matter_power_spectrum='halofit',
                      mass_function='shethtormen',
                      halo_concentration='duffy2008'
                     )

mmin=1e10
mmax=1e16
nm=101

m=np.logspace(np.log10(mmin),np.log10(mmax),nm)
a=1.

c=ccl.halo_concentration(cosmo,m,a,odelta=201.)

This all works on Python3. Since we are depreciating Python 2.7.x support, this is now being closed.


Packages are still built against zfsStable when boot.zfs.enableUnstable

Describe the bug
A clear and concise description of what the bug is.
When you set boot.zfs.enableUnstable, the zfsUnstable module and userspace packages get installed. Packages that compile against ZFS like Docker and libvirt are still built against zfsStable however.
I am not sure if this would cause issues but it certainly sounds like something that could.
Expected behavior
A clear and concise description of what you expected to happen.
All ZFS-dependant packages should be built with zfs = zfsUnstable; when it is enabled.
Additional context
Add any other context about the problem here.
Came up while investigating https://github.com/NixOS/nixpkgs/issues/109001
Notify maintainers

@Mic92 @globin @wizeman @hmenke @jcumming @fpletz
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

I've noticed this as well and there is no solution unless nixpkgs adopts USE flags (cf. #56227) or the new module system for packages (cf. https://cfp.nixcon.org/nixcon2020/talk/K89WJY) is finalized.  The same issue applies to
{
  nix.package = pkgs.nixFlakes;
}

This will also not be respected if a package explicitly has nix as a dependency.  One possible solution is the steamroller approach to just globally override zfs with zfsUnstable in an overlay
{
  nixpkgs.overlays = [ (final: prev: {
    zfs = final.zfsUnstable;
  }) ];
}

This will of course start rebuilding everything that depends on zfs.

I've noticed this as well and there is no solution unless nixpkgs adopts USE flags (cf. #56227) or the new module system for packages (cf. https://cfp.nixcon.org/nixcon2020/talk/K89WJY) is finalized.  The same issue applies to
{
  nix.package = pkgs.nixFlakes;
}

This will also not be respected if a package explicitly has nix as a dependency.  One possible solution is the steamroller approach to just globally override zfs with zfsUnstable in an overlay
{
  nixpkgs.overlays = [ (final: prev: {
    zfs = final.zfsUnstable;
  }) ];
}

This will of course start rebuilding everything that depends on zfs.

But I agree that this is a big problem, especially for ZFS where you might accidentally corrupt your pool by acting with outdated user-space tool on an upgraded pool.

But I agree that this is a big problem, especially for ZFS where you might accidentally corrupt your pool by acting with outdated user-space tool on an upgraded pool.

Ideally ZFS tooling would to the same thing as modules using systemd and rely on the fact that the ZFS userspace utility is located at /run/current-system/sw/bin/zfs and not use zfs as an explicit dependency in the first place.  Unfortunately, I don't know whether that would work for Docker which presumably links against ZFS as a dynamic library.

Ideally ZFS tooling would to the same thing as modules using systemd and rely on the fact that the ZFS userspace utility is located at /run/current-system/sw/bin/zfs and not use zfs as an explicit dependency in the first place.  Unfortunately, I don't know whether that would work for Docker which presumably links against ZFS as a dynamic library.

Oh boy, GRUB depends on zfs, so you are lucky that you apparently do not boot from ZFS.

Oh boy, GRUB depends on zfs, so you are lucky that you apparently do not boot from ZFS.

We cannot/should not make the module add a full overlay, correct?

We cannot/should not make the module add a full overlay, correct?

Ideally ZFS tooling would to the same thing as modules using systemd and rely on the fact that the ZFS userspace utility is located at /run/current-system/sw/bin/zfs and not use zfs as an explicit dependency in the first place. Unfortunately, I don't know whether that would work for Docker which presumably links against ZFS as a dynamic library.

I implemented this bit. It uses the executable at runtime.

Ideally ZFS tooling would to the same thing as modules using systemd and rely on the fact that the ZFS userspace utility is located at /run/current-system/sw/bin/zfs and not use zfs as an explicit dependency in the first place. Unfortunately, I don't know whether that would work for Docker which presumably links against ZFS as a dynamic library.

I implemented this bit. It uses the executable at runtime.

I don't think it's that critical. I imagine most programs that have ZFS in their buildinputs only use features that have been stable in ZFS for ages and I'd expect ZFS itself to have sanity checks for at least the current two stable releases.
The worst thing that could happen would probably be the program itself not working correctly: GRUB not beeing able to read a pool because of incompatible features for example.
Not ideal of course but I don't think data loss is a possibility here.
I think the overlay would be a good solution. This way we can be sure the change propagates through to everything that uses it in a pure and well-defined manner.
If you explicitly need the stable ZFS somewhere, you can simply reference it directly via zfsStable but the meaning of the zfs package as a whole would simply be decided by boot.zfs.enableUnstable. Kinda intuitive actually.
Would it be feasible to make hydra build every ZFS dependent package with zfsUnstable too? That way we could prevent local builds.
Although I'm not sure how bad those would be anyways, how many ZFS-dependent packages do we have, how many of those would have to be built locally and how long would that take?

I don't think it's that critical. I imagine most programs that have ZFS in their buildinputs only use features that have been stable in ZFS for ages and I'd expect ZFS itself to have sanity checks for at least the current two stable releases.
The worst thing that could happen would probably be the program itself not working correctly: GRUB not beeing able to read a pool because of incompatible features for example.
Not ideal of course but I don't think data loss is a possibility here.
I think the overlay would be a good solution. This way we can be sure the change propagates through to everything that uses it in a pure and well-defined manner.
If you explicitly need the stable ZFS somewhere, you can simply reference it directly via zfsStable but the meaning of the zfs package as a whole would simply be decided by boot.zfs.enableUnstable. Kinda intuitive actually.
Would it be feasible to make hydra build every ZFS dependent package with zfsUnstable too? That way we could prevent local builds.
Although I'm not sure how bad those would be anyways, how many ZFS-dependent packages do we have, how many of those would have to be built locally and how long would that take?

I'd really like to get this tightened up so there is no question what ZFS versions are being used by different tools on my system. The overlay sounds like a good solution.
I'm trying to help track down a pretty severe bug with OpenZFS 2.0 (https://github.com/openzfs/zfs/issues/10697) as I am running with zfsUnstable in NixOS 20.09. Having additional uncertainty about ZFS versions feels like the exact thing I'm trying to sidestep by running NixOS in the first place.

Had a quick look at how much an overlay would cost us in rebuilds and it doesn't look like much. The only major dependent in my system closure is libvirt which takes a few minutes to build at worst.

All packages that depend on zfs

bareos
ceph
ceph
ceph-client
ceph-csi
collectd
collectd-data
diffoscope
docker-machine-kvm
docker-machine-kvm2
easysnap-unstable
gnome-boxes
libguestfs
libguestfs
libvirt
libvirt-glib
libvmi
minikube
minishift
nixops-with-plugins
ocaml-libvirt
partitionmanager
perl5.32.1-Sys-Virt
perl5.34.0-Sys-Virt
python3.8-guestfs
python3.8-libvirt
python3.9-guestfs
python3.9-libvirt
qemu
ruby2.6.8-ruby-libvirt
ruby2.7.4-ruby-libvirt
ruby3.0.2-ruby-libvirt
samba
sanoid
terraform-provider-libvirt
terraform-provider-libvirt
terraform-with-plugins
vagrant
virt-manager
virt-manager
virt-manager-qt
virt-manager-qt
virt-top
virt-viewer
virt-viewer
virtlyst
zfs
zfstools

I think that makes it a straightforward and easy solution without many downsides.


[vcpkg-cmake|cmake buildsystem] synchronize and sanitize scripts.

push vcpkg_cmake_ stuff to be more up to date to vcpkg_x_cmake  stuff.

add triplet variable VCPKG_CMAKE_GENERATOR for people who are brave enough to do that.

closes #23248 ?

tbb is a baseline regression, confirmed.
Can't repro draco[core]:x64-uwp regression with Visual Studio 2022:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2577,6): error MSB3795: There was a problem in the GetSDKReferenceFiles task. There are no more files. [D:\buildtrees\draco\x64-uwp-dbg\draco_metadata_dec.vcxproj]

Can't repro tidy-html5[*]:x64-linux regression with gcc 7.5.0:

cd /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg && /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy -xml-help > /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy-help.xml && cd /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg && /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy -xml-config > /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy-config.xml && cd /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg && xsltproc /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy1.xsl /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy-help.xml > /mnt/vcpkg-ci/buildtrees/tidy-html5/x64-linux-dbg/tidy.1
/bin/sh: 1: xsltproc: not found

file list:
libxslt:x64-linux:/tools/libxslt/bin/xslt-config
libxslt:x64-linux:/tools/libxslt/bin/xsltproc
libxslt:x64-linux:/tools/libxslt/debug/bin/xslt-config
libxslt:x64-linux:/tools/libxslt/debug/bin/xsltproc

tidy-html5 has been fixed in another PR.

@Neumann-A I don't like merging these two things together - generator and updating vcpkg_cmake_* scripts. If you only do one or the other (or both in separate PRs), I'd feel much more confident merging them.

Oh, the vcpkg_cmake part overlaps with #21507.

draft until https://github.com/microsoft/vcpkg/pull/23846 is merged

@JackBoosY: the qpid-proto failures look like a baseline regression. complaining that PYTHON_EXECUTABLE and Python_EXECUTABLE are defined.
The duckx regression is fixed by #24264

@JackBoosY the qpid-proto failure is also observed in https://github.com/microsoft/vcpkg/pull/24100

@JackBoosY Please remove tag requires:author-response. CI is green.

Will rerun the failed tests.

Thanks for the improvements!
I explicitly checked that there are only additions and things that shouldn't be backcompat issues in the scripts/ ones.


nixos/manpages: Explain nixos-rebuild -I option and how to build manpages

Description of changes
When running nixos-rebuild -I, the man page just said "-I path" which could be interpreted as
"just a path to nixpkgs", which in fact it actually has the same meaning as NIX_PATH.
This is now made clear in the manual, so that when grepping "-I" and "NIX_PATH" one quickly finds
the format of the option.
I don't know how to link to the "nix manual" as stated in the docbook, so I left that as it is.
Additionally, it wasn't clear to me how to actually build the man pages and view the changes I made. That's now in the contributing-to-this-manual.chapter.md.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result

I force-pushed a change. How does one restart the test?

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-can-i-quickly-test-nixpkg-modifications-in-a-container-vm/23797/7

Thank you for the review @ncfavier !


Readme: the Quarkus Apicurio Extension is no more.

The paragraph talking about the Quarkus Apicurio Extension should be removed.
Edit: after further investigation, it may still work with the generic maven plugin (OpenApi2Quarkus.java)

@ricardozanini any thoughts?

@carlesarnal do we have an alternative?

This is why I have created https://github.com/quarkiverse/quarkus-openapi-generator/pull/482.

Hi folks,
is it interesting to mention a workaround?
is the use of the apicurio maven plugin a valid workaround for that?
I am doing this:
  <build>
    <plugins>
      <plugin>
        <groupId>io.apicurio</groupId>
        <artifactId>apicurio-codegen-maven-plugin</artifactId>
        <version>1.1.1.Final</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>generate</goal>
            </goals>
            <configuration>
              <projectSettings>
                <javaPackage>com.example.api</javaPackage>
              </projectSettings>
              <inputSpec>src/main/resources/openapi.json</inputSpec>
            </configuration>
          </execution>
        </executions>
      </plugin>
  </plugins>
</build>

@carlesarnal is this solved by your PR?

Yes, the extension that existed in apicurio is now the one added to this project. In response to @akram yes, the maven plugin is a different wrapper around the same tool, so it's perfectly fine to use it (although I really recommend you using the quarkus extension if you're using Quarkus.)


Minnie needs to be show-ready!

Minnie currently has too much power and in order to have her family and people friendly, her power has to be reduced. Due date- 2/2.

updating this deadline to 2/22

I finish this :)


Extract write-side protocol classes to akka-ddd-protocol

May be DomainException must be moved to some shared module, which will be a dependency for contracts modules of application?

Currently write-front applications must depend on heavy akka-ddd-messaging. akka-ddd-protocol is needed.


Assignment 06 -- Mid semester course survey and random number

Fill out the mid semester course survey on Edunex.
Reread about descriptive statistics after delivered in this session and take some personal notes.
Study about random number and try all of the examples.
Report here with your ID and name that you have performed above tasks.

Total activity about 4 hours.

Ilham Ghufroni (10221012)
Intro into Random Number gist : https://gist.github.com/Frosty8171/62f001c16e52d377d0057d0db93e5e37

Mohammad Ibrahim Akhyari (10221036)
sudah membaca dan mencoba contoh pada slide.
Link Gist: https://gist.github.com/syzsxyera/9b149f3c173199ed15c98bd1756091f8

Ilham Prasetyo (10221074)
Sudah membaca dan mencoba contoh pada slide.
brief_into_random_number.ipynb: https://gist.github.com/ilhamprasetyooo/db8de34ee12725d07e6f2a7ff0c0fabc

Fadila Hasna Amalia (10221056)
Sudah membaca dan melakukan contoh pada slide random number
Link gist:  https://gist.github.com/fadilaha/df48e613a0c278a90031d2bee02510c0


Managing the Registry

https://ei.docs.wso2.com/en/7.1.0/micro-integrator/develop/creating-artifacts/registry/managing-the-Registry/

The former page on 'managing the Registry' is now redirected to [1] because the content is no longer applicable to MI.
[1] https://ei.docs.wso2.com/en/latest/micro-integrator/develop/creating-artifacts/creating-registry-resources/


GEODE-3397: Fixed issue with Tomcat locators in cache-client.xml file

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[x] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to<EMAIL_ADDRESS>

@ladyVader @nabarunnag @boglesby @jhuynh1 @upthewaterspout @gesterzhou

Looks ok to me...I'll merge this in.


Update deployment.yaml

Added nodeSelector: parameter
Issue number of the reported bug or feature request: #
Describe your changes
A clear and concise description of the changes you have made.
Testing performed
Describe the testing you have performed to ensure that the bug has been addressed, or that the new feature works as planned.
Additional context
Add any other context about your contribution here.

Could you add your changes in #164 to this PR, and add documentation around it?
Also it would be good to default to an empty node-selector, and just leave it for users to override at their will.

Could you add your changes in #164 to this PR, and add documentation around it?
Also it would be good to default to an empty node-selector, and just leave it for users to override at their will.

Replaced with #250


don't follow non-local redirects and elide lifecycle response body from logs

If you like this, it's probably best to fast forward your branch to it.

PTAL
I removed the commit that modify the probe logs and added a new commit that elides the body from lifecycle events


ServiceBus connection string does not require manage on the namespace any more.

This is no longer accurate. Creating a token specific to the queue is fine as long as it has manage on the queue.
To obtain a connection string, follow the steps shown at Get the management credentials. The connection string must be for a Service Bus namespace, not limited to a specific queue or topic.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1d50399c-0873-e903-35ca-1845c84b6c6c
Version Independent ID: 1a3150b4-5eb9-0ac5-ee2b-d1bb6b6fd63e
Content: Azure Service Bus trigger for Azure Functions
Content Source: articles/azure-functions/functions-bindings-service-bus-trigger.md
Service: azure-functions
GitHub Login: @craigshoemaker
Microsoft Alias: cshoe

Hello @wjdavis5 - Thank you for your feedback! We will review and update as appropriate.

Hi @wjdavis5, I tried using the connection string for the queue (instead of namespace) and it errors out with the following message:
Microsoft.Azure.ServiceBus: NamespaceConnectionString should not contain EntityPath. (Parameter 'namespaceConnectionString').
I might be missing something here. Can you please provide additional details regarding it?

.

Hi @wjdavis5, following up to check if you had a chance to review my comment above. Can you please share more details of the issue?

@wjdavis5, we will now proceed to close this thread as we have not heard back from you. If there are further questions regarding this matter, please tag me in your reply and reopen the issue. We will gladly continue the discussion.

@AnuragSingh-MSFT  - Apologies, I've been on holiday.
Your response:

Hi @wjdavis5, I tried using the connection string for the queue (instead of namespace) and it errors out with the following message:
Microsoft.Azure.ServiceBus: NamespaceConnectionString should not contain EntityPath. (Parameter 'namespaceConnectionString').
I might be missing something here. Can you please provide additional details regarding it?

Just as the error message suggests you should not be supplying the EntityPath in the connectionstring.

Hi @AnuragSingh-MSFT. Just wanted to check to see if you had any questions about my reply.

@wjdavis5, thank you for the reply. I am still unable to understand the issue with the following statement. It mentions that connection string to namespace is required (and not the queue or topic), which is correct. Am I missing something?
To obtain a connection string, follow the steps shown at Get the management credentials. The connection string must be for a Service Bus namespace, not limited to a specific queue or topic.

The connection string must be for a Service Bus namespace, not limited to a specific queue or topic.

This is the incorrect part. You can use a key generated specific to a queue or topic.

It isn't correct though. A root key from a queue has all the needed
permissions for the scale controller to enumerate the needed metrics.
On Wed, Sep 29, 2021 at 4:57 AM AnuragSingh-MSFT @.***>
wrote:

@wjdavis5 https://github.com/wjdavis5, thank you for the reply. I am
still unable to understand the issue with the following statement. It
mentions that connection string to namespace is required (and not the queue
or topic), which is correct. Am I missing something?
To obtain a connection string, follow the steps shown at Get the
management credentials. The connection string must be for a Service Bus
namespace, not limited to a specific queue or topic.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/80912#issuecomment-929977484,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AATWAXZMVOIRFLCED5YJQMLUELIGNANCNFSM5DZVUJ6A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@wjdavis5, thank you for the explanation. Here is what I did and can see the same behavior in my test environment.

Create a shared access policy for the queue with "Manage".
Use this queue's Connection string [minus] the EntityPath property as the connection string for Function App to the SB.

I will be forwarding this feedback to our content team for further review and modification as applicable.

Thank you!
On Thu, Sep 30, 2021 at 1:28 AM AnuragSingh-MSFT @.***>
wrote:

@wjdavis5 https://github.com/wjdavis5, thank you for the explanation.
Here is what I did and can see the same behavior in my test environment.

Create a shared access policy for the queue with "Manage".
Use this queue's Connection string [minus] the EntityPath property
as the connection string for Function App to the SB.

I will be forwarding this feedback to our content team for further review
and modification as applicable.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/80912#issuecomment-930796011,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AATWAX5CTMFUY3SPP2NF2FDUEPYORANCNFSM5DZVUJ6A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

bumping this thread for i have encountered "Microsoft.Azure.ServiceBus: NamespaceConnectionString should not contain EntityPath" when trying to connect to a queue
Code example:
public void Run([ServiceBusTrigger("%QueueName%",
Connection = "QueueSaSConnectionStringPk")]string command, ILogger log)
{
    my SAS policy was also constructed with the manage type.


[REQUEST] Libellum

Is your feature request related to a problem? Please describe.
I like the concept a lot, but I also like to have a clean lockscreen when I don't have a note, so...
Describe the solution you'd like
If I don't have a note at all, there is no point in showing a blank rectangle (even a small one). So just hide it. If I want to add a note (the first one), let me double-tap on lock screen and reveal the note rectangle for text input.
So if I have notes, but still want to hide them, provide two options in the top left and right corners of the note: a close icon (x) to close the notes and again reveal them by double-tap or a minimize icon (-) to leave a small icon on the lockscreen (e.g., a textfile icon, maybe with the number of notes in it) so that one keeps a reminder of existing notes on the lockscreen without the full note visible at all times.
Describe alternatives you've considered
N/A
Additional context
Nope, this tweak is next to perfect. Thanks for making it. It's a brilliant idea.

A form of the first solution will appear in the next update.

The minimizing like feature you described will not though.


How to get an entire array

Describe the bug
I have an array with 4 items that I want to display. Before using this package, which is awesome, by the way, I had the array defined like this: let arr = ['item 1', 'item 2', 'item 3', 'item 4']. This allowed me to use the {:each} block provided by svelte for easy output. Unfortunately, I have not found a way to get the entire array at once when using typesafe-i18n. Instead I have to use {$LL.arr["0"]()} to {$LL.arr["4"]()}, which get's annoying pretty quick.
Am I missing something or is there no way to get an entire array/object?
Reproduction
I adapted the sveltekit example.
Logs
No response
Config
{
   "$schema"<EMAIL_ADDRESS>   "adapter": "svelte"
}

Additional information
No response

Hi @GLAD-DE,
thanks for creating this issue. typesafe-i18n uses a Proxy to access the translations. By default it doesn't support iterating over it's contents.
I will think about a way to make you use-case possible :)

I may have found a way to enable this.
The downside of my approach is that there can't be more than one object with the same key in the same level since the first object would shaddow the following ones.
The links all lead to the same code

Link to jsfiddle
Link to OneComplier

iterating over arrays should now work with version 2.60.1.
const en: BaseTranslation = {
   arr: [
      'item 1',
      'item 2'
   ]
}

   {#each Object.values($LL.arr) as entry}

{entry()}

   {/each}

@GLAD-DE could you please confirm?

I can confirm that it is working as expected :)
Thank you very much for adding this feature so fast!


scaleio-gateway install hangs

Installing the gateway on ubuntu 14.04 hangs, seems to be related to strings /dev/urandom.
root@compute-0-2:/var/tmp# dpkg -i emc-scaleio-gateway_2.0-5014.0_amd64.deb
Nevertheless here are some tracing while hanging:
root@compute-0-2:/var/tmp# dpkg -s emc-scaleio-gateway
Package: emc-scaleio-gateway
Status: install ok half-configured
Priority: extra
Section: alien
Installed-Size: 152024
Maintainer: root root@VM-SIOBUILD-UBNT14
Architecture: amd64
Version: 2.0-5014.0
Conffiles:
/opt/emc/scaleio/gateway/conf/catalina.properties e2d932b561d0b1a4f63a169520bf1ab8
/opt/emc/scaleio/gateway/conf/server.xml 30e7d2053c288436572e1913e04112c7
/opt/emc/scaleio/gateway/conf/server_clientcert.xml 58fdbca2090abdf53a82a8fb173e172d
/opt/emc/scaleio/gateway/webapps/ROOT/WEB-INF/classes/gatewayInternal.properties aea282c598dcfc6207f5b104a858b190
/opt/emc/scaleio/gateway/webapps/ROOT/WEB-INF/classes/gatewayUser.properties 0d8a2feb2cc40be9a4ccd12cf262ecc8
Description: rpm
ScaleIO Gateway for ScaleIO ECS
.
(Converted from a rpm package by alien version 8.90.)
root@compute-0-2:/var/tmp# ps -ef | grep gateway
root     11456  7605  0 15:59 pts/10   00:00:00 grep --color=auto gateway
root     27231 27226  0 15:50 ?        00:00:00 /bin/sh -c dpkg -i /var/tmp/emc-scaleio-gateway_2.0-5014.0_amd64.deb
root     27232 27231  0 15:50 ?        00:00:01 dpkg -i /var/tmp/emc-scaleio-gateway_2.0-5014.0_amd64.deb
root     28114 27232  0 15:50 ?        00:00:00 /bin/bash /var/lib/dpkg/info/emc-scaleio-gateway.postinst configure
root     28264 28114  0 15:50 ?        00:00:00 /bin/bash /var/lib/dpkg/info/emc-scaleio-gateway.postinst configure
….
root@compute-0-2:/var/tmp# vi /var/log/dpkg.log
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 configure emc-scaleio-gateway:amd64 2.0-5014.0 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status unpacked emc-scaleio-gateway:amd64 2.0-5014.0
2016-06-21 15:50:17 status half-configured emc-scaleio-gateway:amd64 2.0-5014.0
root@compute-0-2:/var/tmp# dpkg --configure -a
dpkg: error: dpkg status database is locked by another process
root@compute-0-2:/var/tmp# dpkg -r emc-scaleio-gateway
dpkg: error: dpkg status database is locked by another process
root@compute-0-2:/var/tmp# ps -ef | grep dpkg
root     25353  7605  0 16:14 pts/10   00:00:00 grep --color=auto dpkg
root     27231 27226  0 15:50 ?        00:00:00 /bin/sh -c dpkg -i /var/tmp/emc-scaleio-gateway_2.0-5014.0_amd64.deb
root     27232 27231  0 15:50 ?        00:00:01 dpkg -i /var/tmp/emc-scaleio-gateway_2.0-5014.0_amd64.deb
root     28114 27232  0 15:50 ?        00:00:00 /bin/bash /var/lib/dpkg/info/emc-scaleio-gateway.postinst configure
root     28264 28114  0 15:50 ?        00:00:00 /bin/bash /var/lib/dpkg/info/emc-scaleio-gateway.postinst configure
root@compute-0-2:/var/tmp# top
top - 16:13:39 up 7 days, 20:55,  5 users,  load average: 3.53, 3.38, 3.15
Tasks: 335 total,   2 running, 333 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.2 us,  5.2 sy,  0.0 ni, 84.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65692052 total, 63029056 used,  2662996 free,   269156 buffers
KiB Swap:        0 total,        0 used,        0 free.  2527744 cached Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
28265 root      20   0   13376    992    776 R  99.9  0.0  23:22.14 strings
root@compute-0-2:/var/tmp# ps -ef | grep 28265
root     12220  7605  0 16:18 pts/10   00:00:00 grep --color=auto 28265
root     28265 28264 99 15:50 ?        00:28:16 strings /dev/urandom
root@compute-0-2:/var/tmp# ps -ef | grep dpk
root     28114     1  0 15:50 ?        00:00:00 /bin/bash /var/lib/dpkg/info/emc-scaleio-gateway.postinst configure
root     28170  7605  0 16:21 pts/10   00:00:00 grep --color=auto dpk
root     28264 28114  0 15:50 ?        00:00:00 /bin/bash /var/lib/dpkg/info/emc-scaleio-gateway.postinst configure
I’ve killed some dpkg proceses
root@compute-0-2:/var/tmp# dpkg --configure -a
Setting up emc-scaleio-gateway (2.0-5014.0) ...
Running post install operations (simulated rpm,install)
/usr/bin/java
keytool found based on java path : /opt/Oracle_Java/jdk1.8.0_92/bin /opt/Oracle_Java/jdk1.8.0_92/bin/keytool
Certificate stored in file </opt/emc/scaleio/gateway/temp/certificates/ScaleIO.cer>
Installing the web server service/process to run as root(default).
To run as a different user, define the variable GATEWAY_RUN_USER.
2
Creating service file /etc/init/scaleio-gateway.conf
Creating service file /opt/emc/scaleio/gateway/bin/scaleio-gateway-wd.bash
gateway user  : root
Creating service file /etc/init.d/scaleio-gateway
4
Adding system startup for /etc/init.d/scaleio-gateway ...
/etc/rc0.d/K20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/rc1.d/K20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/rc6.d/K20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/rc2.d/S20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/rc3.d/S20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/rc4.d/S20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/rc5.d/S20scaleio-gateway -> ../init.d/scaleio-gateway
/etc/init.d/scaleio-gateway: line 92: systemctl: command not found
The EMC ScaleIO Gateway is running. PID=31723.
root@compute-0-2:/var/tmp# top
top - 16:24:34 up 7 days, 21:06,  5 users,  load average: 3.76, 3.71, 3.43
Tasks: 334 total,   3 running, 331 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.9 us,  5.2 sy,  0.0 ni, 83.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65692052 total, 63614272 used,  2077780 free,   269328 buffers
KiB Swap:        0 total,        0 used,        0 free.  2528344 cached Mem
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
14567 root      10 -10 4050756 235360  13004 S 200.4  0.4  22526:52 ovs-vswitchd
28265 root      20   0   13376    992    776 R 100.0  0.0  34:17.76 strings
22076 nova      20   0 1860536  90312   9172 S   2.3  0.1 246:21.21 nova-compute
root@compute-0-2:/var/tmp# dpkg -s emc-scaleio-gateway
Package: emc-scaleio-gateway
Status: install ok installed
Priority: extra
Section: alien
Installed-Size: 152024
Maintainer: root root@VM-SIOBUILD-UBNT14
Architecture: amd64
Version: 2.0-5014.0
Conffiles:
/opt/emc/scaleio/gateway/conf/catalina.properties e2d932b561d0b1a4f63a169520bf1ab8
/opt/emc/scaleio/gateway/conf/server.xml 30e7d2053c288436572e1913e04112c7
/opt/emc/scaleio/gateway/conf/server_clientcert.xml 58fdbca2090abdf53a82a8fb173e172d
/opt/emc/scaleio/gateway/webapps/ROOT/WEB-INF/classes/gatewayInternal.properties aea282c598dcfc6207f5b104a858b190
/opt/emc/scaleio/gateway/webapps/ROOT/WEB-INF/classes/gatewayUser.properties 0d8a2feb2cc40be9a4ccd12cf262ecc8
Description: rpm
ScaleIO Gateway for ScaleIO ECS
.
(Converted from a rpm package by alien version 8.90.)
root@compute-0-2:/var/tmp# dpkg -l | grep emc
ii  emc-scaleio-gateway                 2.0-5014.0                                     amd64        rpm
ii  emc-scaleio-mdm                     2.0-5014.0.Ubuntu.14.04                        amd64        scaleio meta-data manager package
ii  python-memcache                     1.53-1~u14.04+mos2                             all          pure python memcached client
ii  python-pymemcache                   1.2.4-2~u14.04+mos2                            all          comprehensive, fast, pure Python memcached client - Python 2.x
root@compute-0-2:/var/tmp#

It seems to be the same problem as https://github.com/ansible/ansible/issues/12459 that was never fixed in Ansible as people are using workarounds.

Using python 3.4.3 on the target host (ubuntu) seems to solve the issue.
Will need amend the readme file to reflect this requirement.


Merge pull request #24 from dex-guru/main

Hello I need to add my token logo to your exchange. I appreciate your help.

Hello I need to add my token logo to your exchange. I appreciate your help. !

Please refer to the readme section of this repository, it contains steps on how to create a PR with asset: https://github.com/dex-guru/assets#readme


Encounter executors restart issue, when run Hibench Kmeans workload on AWS EKS environment, with Hibench small scale dataset

We are integrating OAP MLlib into Cloud. Encountered the executors restart issue,  although the result still outputted , when run Hibench Kmeans workload on AWS EKS environment, with Hibench small scale dataset.
Spark confs and command:
sh ./spark-submit-client.sh --conf spark.executor.instances=2 \
  --conf spark.executor.memoryOverhead=512m  \
  --conf spark.executor.cores=1  \
  --conf spark.default.parallelism=2  \
  --conf spark.sql.shuffle.partitions=2  \
  --conf spark.driver.extraClassPath=/opt/home/conda/envs/oap-1.2.0/oap_jars/oap-mllib-1.2.0.jar \
  --conf spark.executor.extraClassPath=/opt/home/conda/envs/oap-1.2.0/oap_jars/oap-mllib-1.2.0.jar \
  --class com.intel.hibench.sparkbench.ml.DenseKMeans /opt/home/sparkbench-assembly-8.0-SNAPSHOT-dist.jar -k 10 --numIterations 5 --storageLevel MEMORY_ONLY --initMode Random s3a://aws-emr-resources-348941870272-us-east-2/datagen/HiBench/Kmeans/Input/3000000/samples

data scale conf
hibench.kmeans.small.num_of_clusters		5
hibench.kmeans.small.dimensions			20
hibench.kmeans.small.num_of_samples		3000000
hibench.kmeans.small.samples_per_inputfile	600000
hibench.kmeans.small.max_iteration		5
hibench.kmeans.small.k				10

logs
21/07/30 13:01:25 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on <IP_ADDRESS>:46745 (size: 16.8 KiB, free: 107.0 MiB)
21/07/30 13:02:25 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Disabling executor 1.
21/07/30 13:02:25 INFO DAGScheduler: Executor lost: 1 (epoch 1)
21/07/30 13:02:25 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 107) in 59757 ms on <IP_ADDRESS> (executor 2) (1/2)
21/07/30 13:02:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_6 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_37_8 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_9 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_15 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_3 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_37_14 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_11 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_37_16 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_14 !
21/07/30 13:02:25 WARN BlockManagerMasterEndpoint: No more replicas available for rdd_6_1 !
...
21/07/30 06:37:18 INFO DAGScheduler: Job 10 finished: collect at ClusteringSummary.scala:49, took 124.838207 s
21/07/30 06:37:18 INFO CodeGenerator: Code generated in 17.830358 ms
21/07/30 06:37:18 INFO Instrumentation: [d644dcde] {"clusterSizes":"[210265,121453,427029,123869,684244,207123,389735,600544,118074,117664]"}
21/07/30 06:37:18 INFO Instrumentation: [d644dcde] training finished
Training time (ms) = 744550
Total cost = 1.3657241593861768E11.
21/07/30 06:37:18 INFO SparkUI: Stopped Spark web UI at http://spark-client-headless-service.default.svc.cluster.local:4040

But when vanilla spark ran this workload with the same confs,  everything is ok.
Vanilla spark command:
sh ./spark-submit-client.sh --conf spark.executor.instances=2 \
  --conf spark.executor.memoryOverhead=512m  \
  --conf spark.executor.cores=1  \
  --conf spark.default.parallelism=2  \
  --conf spark.sql.shuffle.partitions=2  \
  --class com.intel.hibench.sparkbench.ml.DenseKMeans /opt/home/sparkbench-assembly-8.0-SNAPSHOT-dist.jar -k 10 --numIterations 5 --storageLevel MEMORY_ONLY --initMode Random s3a://aws-emr-resources-348941870272-us-east-2/datagen/HiBench/Kmeans/Input/3000000/samples

Vanilla spark running log
21/08/02 07:18:45 INFO TaskSetManager: Finished task 16.0 in stage 14.0 (TID 187) in 190 ms on <IP_ADDRESS> (executor 1) (17/20)
21/08/02 07:18:45 INFO TaskSetManager: Finished task 18.0 in stage 14.0 (TID 188) in 138 ms on <IP_ADDRESS> (executor 2) (18/20)
21/08/02 07:18:45 INFO TaskSetManager: Starting task 19.0 in stage 14.0 (TID 190) (<IP_ADDRESS>, executor 1, partition 19, PROCESS_LOCAL, 4572 bytes) taskResourceAssignments Map()
21/08/02 07:18:45 INFO TaskSetManager: Finished task 17.0 in stage 14.0 (TID 189) in 211 ms on <IP_ADDRESS> (executor 1) (19/20)
21/08/02 07:18:45 INFO TaskSetManager: Finished task 19.0 in stage 14.0 (TID 190) in 105 ms on <IP_ADDRESS> (executor 1) (20/20)
21/08/02 07:18:45 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool
21/08/02 07:18:45 INFO DAGScheduler: ShuffleMapStage 14 (collect at ClusteringSummary.scala:49) finished in 2.552 s
21/08/02 07:18:45 INFO DAGScheduler: looking for newly runnable stages
21/08/02 07:18:45 INFO DAGScheduler: running: Set()
21/08/02 07:18:45 INFO DAGScheduler: waiting: Set(ResultStage 15)
21/08/02 07:18:45 INFO DAGScheduler: failed: Set()
21/08/02 07:18:45 INFO DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[50] at collect at ClusteringSummary.scala:49), which has no missing parents
21/08/02 07:18:45 INFO MemoryStore: Block broadcast_26 stored as values in memory (estimated size 50.5 KiB, free 413.4 MiB)
21/08/02 07:18:45 INFO MemoryStore: Block broadcast_26_piece0 stored as bytes in memory (estimated size 22.3 KiB, free 413.4 MiB)
21/08/02 07:18:45 INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on spark-client-headless-service.default.svc.cluster.local:33081 (size: 22.3 KiB, free: 413.8 MiB)
21/08/02 07:18:45 INFO SparkContext: Created broadcast 26 from broadcast at DAGScheduler.scala:1383
21/08/02 07:18:45 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 15 (MapPartitionsRDD[50] at collect at ClusteringSummary.scala:49) (first 15 tasks are for partitions Vector(0, 1))
21/08/02 07:18:45 INFO TaskSchedulerImpl: Adding task set 15.0 with 2 tasks resource profile 0
21/08/02 07:18:45 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 191) (<IP_ADDRESS>, executor 1, partition 0, NODE_LOCAL, 4465 bytes) taskResourceAssignments Map()
21/08/02 07:18:45 INFO TaskSetManager: Starting task 1.0 in stage 15.0 (TID 192) (<IP_ADDRESS>, executor 2, partition 1, NODE_LOCAL, 4465 bytes) taskResourceAssignments Map()
21/08/02 07:18:45 INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on <IP_ADDRESS>:36707 (size: 22.3 KiB, free: 91.2 MiB)
21/08/02 07:18:45 INFO BlockManagerInfo: Added broadcast_26_piece0 in memory on <IP_ADDRESS>:45589 (size: 22.3 KiB, free: 107.0 MiB)
21/08/02 07:18:45 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 6 to <IP_ADDRESS>:51316
21/08/02 07:18:45 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 6 to <IP_ADDRESS>:55576
21/08/02 07:18:46 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 191) in 76 ms on <IP_ADDRESS> (executor 1) (1/2)
21/08/02 07:18:46 INFO TaskSetManager: Finished task 1.0 in stage 15.0 (TID 192) in 87 ms on <IP_ADDRESS> (executor 2) (2/2)
21/08/02 07:18:46 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool
21/08/02 07:18:46 INFO DAGScheduler: ResultStage 15 (collect at ClusteringSummary.scala:49) finished in 0.140 s
21/08/02 07:18:46 INFO DAGScheduler: Job 8 is finished. Cancelling potential speculative or zombie tasks for this job
21/08/02 07:18:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 15: Stage finished
21/08/02 07:18:46 INFO DAGScheduler: Job 8 finished: collect at ClusteringSummary.scala:49, took 2.736976 s
21/08/02 07:18:46 INFO CodeGenerator: Code generated in 12.199455 ms
21/08/02 07:18:46 INFO Instrumentation: [825afca3] {"clusterSizes":"[286782,121453,119044,123869,600000,313218,600000,600000,118074,117560]"}
21/08/02 07:18:46 INFO Instrumentation: [825afca3] training finished
Training time (ms) = 22674
Total cost = 1.9109701621924158E11.
21/08/02 07:18:46 INFO SparkUI: Stopped Spark web UI at http://spark-client-headless-service.default.svc.cluster.local:4040

@xwu99 @carsonwang , please have a check when available.
Thanks

Are you using K8S? KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Disabling executor 1. We didn't test on K8S. I think there is different way to set classpath for K8S. Could you also send out stdout / stderr info?

@zhixingheyi-tian Pls try the latest master branch to see if everything ok. Also @haojinIntel

@xwu99
Now our team Cloud integration testings (include on EMR, Dataproc, EKS, etc.) are for branch-1.2.  Cloud integration is also the target of 1.2 release.
From @yao531441  compile script，have  included the " -Pspark-3.1.1"
oap-mllib )
    cd $OAP_HOME/oap-mllib/mllib-dal
    source /opt/intel/oneapi/setvars.sh
    source /tmp/oneCCL/build/_install/env/setvars.sh
    mvn clean package  -Dmaven.test.skip=true  -Pspark-3.1.1
    ;;

"EKS" just is Amazon Elastic Kubernetes Service. We used K8S scheduler to launch Spark Executor. From the "KMeansDAL" log, the oap-mllib jars have enabled in classpath.
The driver log is shown above.
The executor log before killed is as below:
21/08/02 07:37:27 INFO CoarseGrainedExecutorBackend: Got assigned task 106
21/08/02 07:37:27 INFO Executor: Running task 1.0 in stage 9.0 (TID 106)
21/08/02 07:37:27 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
21/08/02 07:37:27 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 35.9 MiB)
21/08/02 07:37:27 INFO TorrentBroadcast: Reading broadcast variable 10 took 20 ms
21/08/02 07:37:27 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 40.8 KiB, free 35.8 MiB)
21/08/02 07:37:27 INFO BlockManager: Found block rdd_39_1 locally
21/08/02 07:37:27 INFO OneCCL: Initializing with IP_PORT: <IP_ADDRESS>_3000
OneCCL (native): init
2021:08:02-07:37:27:(43) |WARN| hostname is quite long, len: 63, name: sinput3000000samples105memoryonlyrandom-9ae02e7b05c8a699-exec-1
KVS get error: timeout limit (60 > 60), prefix: CCL_POD_ADDR1, key atl-ofi-hostname-0

I doubt if there are some limitations for Spark configurations. For example: "executor.memory".
Now ,test dataset is small scale of HiBench, just less than 1G. Do OAP MLlib needs specific conf items for small datasets?

@zhixingheyi-tian From the log, it's not related to memory. oneCCL ranks are not about to connect to the rank 0 which is the executor listening in <IP_ADDRESS> port 3000. There is some network issue. Maybe the port is blocked or the IP is not reachable.
And there is a warning: |WARN| hostname is quite long, len: 63, name: sinput3000000samples105memoryonlyrandom-9ae02e7b05c8a699-exec-1. I am not sure if it's the cause of bad hostname resolve.

btw. It's encouraged to use build.sh instead of just using mvn to build. Some additional environments need to be set before calling mvn.

@xwu99
As baseline, we also started the vanilla Spark testing, with same configs and same workload. The running is smooth and quick. Can refer to previous description. So the network issue can be excluded.

Add "spark.shuffle.reduceLocality.enabled=false" conf item,
We verify two cases: two executors, and one executors, still exist  issues.
Two executors case, encounter hang issue, on both driver and executor sides.
sh ./spark-submit-client.sh --conf spark.executor.instances=2 \
  --conf spark.executor.memoryOverhead=512m  \
  --conf spark.executor.cores=1  \
  --conf spark.default.parallelism=2  \
  --conf spark.sql.shuffle.partitions=2  \
  --conf spark.shuffle.reduceLocality.enabled=false  \
  --conf spark.driver.extraClassPath=/opt/home/conda/envs/oap-1.2.0.emr630/oap_jars/oap-mllib-1.2.0.jar \
  --conf spark.executor.extraClassPath=/opt/home/conda/envs/oap-1.2.0.emr630/oap_jars/oap-mllib-1.2.0.jar \
  --class com.intel.hibench.sparkbench.ml.DenseKMeans /opt/home/tools/HiBench/sparkbench/assembly/target/sparkbench-assembly-8.0-SNAPSHOT-dist.jar -k 10 --numIterations 5 --storageLevel MEMORY_ONLY --initMode Random s3a://aws-emr-resources-348941870272-us-east-2/datagen/HiBench/Kmeans/Input/3000000/samples

driver hang log
21/08/10 08:06:16 INFO BlockManagerInfo: Removed broadcast_9_piece0 on spark-client-headless-service.default.svc.cluster.local:44815 in memory (size: 15.0 KiB, free: 413.9 MiB)
21/08/10 08:06:16 INFO BlockManagerInfo: Removed broadcast_8_piece0 on <IP_ADDRESS>:42623 in memory (size: 1278.0 B, free: 111.0 MiB)
21/08/10 08:06:16 INFO BlockManagerInfo: Removed broadcast_8_piece0 on spark-client-headless-service.default.svc.cluster.local:44815 in memory (size: 1278.0 B, free: 413.9 MiB)
21/08/10 08:06:16 INFO BlockManagerInfo: Removed broadcast_8_piece0 on <IP_ADDRESS>:41423 in memory (size: 1278.0 B, free: 87.3 MiB)
21/08/10 08:32:18 INFO BlockManagerInfo: Removed broadcast_1_piece0 on spark-client-headless-service.default.svc.cluster.local:44815 in memory (size: 11.4 KiB, free: 413.9 MiB)
21/08/10 08:32:18 INFO BlockManagerInfo: Removed broadcast_1_piece0 on <IP_ADDRESS>:42623 on disk (size: 11.4 KiB)
21/08/10 08:32:18 INFO BlockManagerInfo: Removed broadcast_1_piece0 on <IP_ADDRESS>:41423 on disk (size: 11.4 KiB)

executor hang log
21/08/10 08:05:50 INFO Executor: Running task 1.0 in stage 9.0 (TID 107)
21/08/10 08:05:50 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB)
21/08/10 08:05:50 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 110.5 MiB)
21/08/10 08:05:50 INFO TorrentBroadcast: Reading broadcast variable 10 took 13 ms
21/08/10 08:05:50 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 40.9 KiB, free 110.5 MiB)
21/08/10 08:05:50 INFO BlockManager: Found block rdd_39_1 locally
21/08/10 08:05:50 INFO OneCCL: Initializing with IP_PORT: <IP_ADDRESS>_3000
OneCCL (native): init

One executor case, encounter executor restart issue without error log. And only restarted once. And the job completed with the second exector
sh ./spark-submit-client.sh --conf spark.executor.instances=1 \
  --conf spark.executor.memoryOverhead=512m  \
  --conf spark.executor.cores=1  \
  --conf spark.default.parallelism=1  \
  --conf spark.sql.shuffle.partitions=1  \
  --conf spark.shuffle.reduceLocality.enabled=false  \
  --conf spark.driver.extraClassPath=/opt/home/conda/envs/oap-1.2.0.emr630/oap_jars/oap-mllib-1.2.0.jar \
  --conf spark.executor.extraClassPath=/opt/home/conda/envs/oap-1.2.0.emr630/oap_jars/oap-mllib-1.2.0.jar \
  --class com.intel.hibench.sparkbench.ml.DenseKMeans /opt/home/tools/HiBench/sparkbench/assembly/target/sparkbench-assembly-8.0-SNAPSHOT-dist.jar -k 10 --numIterations 5 --storageLevel MEMORY_ONLY --initMode Random s3a://aws-emr-resources-348941870272-us-east-2/datagen/HiBench/Kmeans/Input/3000000/samples

Driver log
21/08/10 07:31:38 INFO TaskSetManager: Finished task 11.0 in stage 6.0 (TID 100) in 5095 ms on <IP_ADDRESS> (executor 2) (18/20)
21/08/10 07:31:43 INFO BlockManagerInfo: Added rdd_6_15 in memory on <IP_ADDRESS>:43073 (size: 19.7 MiB, free: 142.4 MiB)
21/08/10 07:31:44 INFO BlockManagerInfo: Added rdd_37_15 in memory on <IP_ADDRESS>:43073 (size: 24.0 B, free: 142.4 MiB)
21/08/10 07:31:44 INFO TaskSetManager: Finished task 15.0 in stage 6.0 (TID 101) in 5313 ms on <IP_ADDRESS> (executor 2) (19/20)
21/08/10 07:32:44 WARN HeartbeatReceiver: Removing executor 1 with no recent heartbeats: 137201 ms exceeds timeout 120000 ms
**_21/08/10 07:32:44 INFO KubernetesClusterSchedulerBackend: Requesting to kill executor(s) 1
21/08/10 07:32:44 INFO KubernetesClusterSchedulerBackend: Actual list of executor(s) to be killed is 1_**
21/08/10 07:32:44 INFO TaskSchedulerImpl: Executor 1 on <IP_ADDRESS> killed by driver.
21/08/10 07:32:44 INFO TaskSetManager: task 10.0 in stage 6.0 (TID 92) failed because while it was being computed, its executor exited for a reason unrelated to the task. Not counting this failure towards the maximum number of failures for the task.
21/08/10 07:32:44 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
21/08/10 07:32:44 INFO BlockManagerMaster: Removal of executor 1 requested
21/08/10 07:32:44 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 1
21/08/10 07:32:44 INFO TaskSetManager: Starting task 10.1 in stage 6.0 (TID 102) (<IP_ADDRESS>, executor 2, partition 10, ANY, 4583 bytes) taskResourceAssignments Map()

Executor exited without error.
21/08/10 07:30:32 INFO MemoryStore: Block rdd_37_8 stored as values in memory (estimated size 24.0 B, free 16.1 MiB)
21/08/10 07:30:32 INFO Executor: Finished task 8.0 in stage 6.0 (TID 90). 1552 bytes result sent to driver
21/08/10 07:30:32 INFO CoarseGrainedExecutorBackend: Got assigned task 91
21/08/10 07:30:32 INFO Executor: Running task 9.0 in stage 6.0 (TID 91)
21/08/10 07:30:32 INFO BlockManager: Found block rdd_6_9 locally
21/08/10 07:30:32 INFO OneDAL: Partition index: 9, numCols: 20, numRows: 168405
21/08/10 07:30:32 INFO OneDAL: Loading native libraries
21/08/10 07:30:32 INFO MemoryStore: Block rdd_37_9 stored as values in memory (estimated size 24.0 B, free 16.1 MiB)
21/08/10 07:30:32 INFO Executor: Finished task 9.0 in stage 6.0 (TID 91). 1552 bytes result sent to driver
21/08/10 07:30:32 INFO CoarseGrainedExecutorBackend: Got assigned task 92
21/08/10 07:30:32 INFO Executor: Running task 10.0 in stage 6.0 (TID 92)
21/08/10 07:30:32 INFO BlockManager: Found block rdd_6_10 locally
21/08/10 07:30:32 INFO OneDAL: Partition index: 10, numCols: 20, numRows: 168920
21/08/10 07:30:32 INFO OneDAL: Loading native libraries

cc @xwu99 @carsonwang

Add another test to make the domain name shorter with "spark.kubernetes.executor.podNamePrefix=oapmllib"
sh ./spark-submit-client.sh --conf spark.executor.instances=2 \
  --name oap-mllib  \
  --conf spark.kubernetes.executor.podNamePrefix=oapmllib  \
  --conf spark.executor.memoryOverhead=512m  \
  --conf spark.executor.cores=1  \
  --conf spark.default.parallelism=2  \
  --conf spark.sql.shuffle.partitions=2  \
  --conf spark.shuffle.reduceLocality.enabled=false  \
  --conf spark.driver.extraClassPath=/opt/home/conda/envs/oap-1.2.0.emr630/oap_jars/oap-mllib-1.2.0.jar \
  --conf spark.executor.extraClassPath=/opt/home/conda/envs/oap-1.2.0.emr630/oap_jars/oap-mllib-1.2.0.jar \
  --class com.intel.hibench.sparkbench.ml.DenseKMeans /opt/home/tools/HiBench/sparkbench/assembly/target/sparkbench-assembly-8.0-SNAPSHOT-dist.jar -k 10 --numIterations 5 --storageLevel MEMORY_ONLY --initMode Random s3a://aws-emr-resources-348941870272-us-east-2/datagen/HiBench/Kmeans/Input/3000000/samples

And the executor also restarted.
21/08/11 06:45:28 INFO TorrentBroadcast: Reading broadcast variable 10 took 26 ms
21/08/11 06:45:28 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 40.8 KiB, free 161.5 MiB)
21/08/11 06:45:28 INFO BlockManager: Found block rdd_39_0 locally
21/08/11 06:45:28 INFO OneCCL: Initializing with IP_PORT: <IP_ADDRESS>_3000
OneCCL (native): init
"client: barrier read data": read: can not process all data, size 4, shift 0

Maybe related to #122


We should immediately open a blob after downloading it

From #88

Similar to #49

If we do this, users will expect to be able to save/update.  In which case there would be no difference between edit and download.  Should there be?  Other than save/update, the only current difference is that you can specify where it gets saved to for download.

I forgot about that - we already do this.  I agree this ought to be enough for now, closing.


Panic on episode fetch for patreon feed

Describe the bug
Trying to add a patreon private feed I get the following error:
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', src/service/podcast_episode_service.rs:234:41
stack backtrace:
   0: rust_begin_unwind
             at /rustc/8460ca823e8367a30dda430efda790588b8c84d3/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/8460ca823e8367a30dda430efda790588b8c84d3/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/8460ca823e8367a30dda430efda790588b8c84d3/library/core/src/panicking.rs:114:5
   3: podfetch::service::podcast_episode_service::PodcastEpisodeService::get_url_file_suffix
   4: podfetch::service::rust_service::PodcastService::handle_insert_of_podcast::{{closure}}
   5: podfetch::controllers::podcast_controller::insert_outline::{{closure}}
   6: futures_executor::local_pool::block_on
   7: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
   8: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
   9: tokio::runtime::task::core::Core<T,S>::poll
  10: tokio::runtime::task::harness::Harness<T,S>::poll
  11: tokio::runtime::blocking::pool::Inner::run

Looks like patreon file URLS aren't conforming to your regex capture structure correctly.
Reproduction
add a private patreon feed it will produce this stack trace.
System Info
Not sure how to get this via my docker setup.

Used Package Manager
n/a
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

Thanks for creating a bug report. Do you have a private patreon feed so I can reproduce the issue? I have never tried adding this via mime type but that could facilitate the process. I'm reading indeed the file type from the url.

Any updates on this one?

I'll close it in 1 week as this issue seems to be irrelevant. If you really experience this issue again please open a new one with a better description and a way that I can reproduce this issue.

Closed. Please create a new one with better reproduction.

sorry I'll have to give some example data with a new ticket and better reproducibility. I might have to setup a similar hosting setup because I don't really want to give a private patreon feed out since that is something the authors of the feed are offering for pay.

No problem. You can also send it to this email address if you want to keep the url secret.


Public Opensearch clusters -> Finegrained access control

Public Opensearch clusters -> Finegrained access control
Description

Cleaned up the code for opensearch pulumi builds and made things a little more consistent
Added a bootstrapping script to configure users and roles in public+secured opensearch clusters.
Enabled encryption at rest on secured opensearch clusters.
Removed most of the IAM config from opensearch cluster buildouts.
Changed the public opensearch clusters to use finegrained security rather than resource or user based policies.
Created documentation on the bootstrap script.

Motivation and Context
Role-based or user-based access control on opensearch clusters would not be a drop-in replacement for the elasticsearch clusters we are currently using. Finegrained security is a drop-in replacement that we currently need.
How Has This Been Tested?

Destroyed and built new apps clusters for CI/QA/Prod.
Verified expected changes on existing not-apps-clusters (should just be a minor tweak to the security policy of existant, non-apps-clusters)

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[X] Enhancement (improves on existing behavior)
[X] Breaking change (fix or feature that would cause existing functionality to change)

Maybe, but I don't think so. I destroyed all the user/role based clusters. Those would definitely be broken if not.

Checklist:

[ ] My code follows the code style of this project. (Did you install and run the pre-commit hooks?)
[ ] My change requires a change to the documentation.
[X] I have updated the documentation accordingly.

One follow up is to revisit the role definitions and define indices and make them more restrictive to just the permissions the two users need. They are pretty broad at the moment.

:+1: