TP Integration Tests GHA: SE_EVENT_BUS_HOST not set, exiting!

This Bug Report affects these Traffic Control components:

Automation
unknown

Current behavior:

The chromium service fails to start in the Traffic Portal Integration Tests GitHub Action:
2021-10-14 18:41:20,050 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
SE_EVENT_BUS_HOST not set, exiting!
2021-10-14 18:41:20,381 INFO exited: selenium-node (exit status 1; not expected)

Expected behavior:

The chromium service should start successfully.
Steps to reproduce:

Using the environment variables set in .github/workflows/tp.integration.tests.yml:
<<'DOCKER_COMPOSE' docker-compose -f- up
version: '3.8'
services:
  hub:
    image: selenium/hub
  chromium:
    image: selenium/node-chrome
    environment:
      HUB_HOST: hub
      HUB_PORT: 4444
    volumes:
      - /dev/shm:/dev/shm
DOCKER_COMPOSE

However, the TP Integration Tests will fail, even if all of the services are started correctly. Changes:
diff --git a/.github/workflows/tp.integration.tests.yml b/.github/workflows/tp.integration.tests.yml
index 03caaaf282..5d7cf96a7b 100644
--- a/.github/workflows/tp.integration.tests.yml
+++ b/.github/workflows/tp.integration.tests.yml
@@ -57,13 +57,17 @@ jobs:
       hub:
         image: selenium/hub
         ports:
-          - "4444:4444"
+          - 4442:4442
+          - 4443:4443
+          - 4444:4444
         options: --health-cmd=/opt/bin/check-grid.sh --health-interval=5s --health-timeout=15s --health-retries=5
       chromium:
         image: ossalternatives/selenium-node-chromium
         env:
-          HUB_HOST: hub
-          HUB_PORT: 4444
+          SE_EVENT_BUS_HOST: hub
+          SE_EVENT_BUS_PUBLISH_PORT: 4442
+          SE_EVENT_BUS_SUBSCRIBE_PORT: 4443
+          SE_NODE_GRID_URL: http://localhost:4444
         volumes:
           - /dev/shm:/dev/shm
       postgres:

Log of the end of the Run TP step with the above change committed:
><EMAIL_ADDRESS>build /home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration
> tsc

[19:42:40] I/launcher - Running 1 instances of WebDriver
[19:42:40] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
[19:42:46] E/launcher - Unknown error: {"additionalInformation":"Driver info: driver.version: unknown","localizedMessage":"Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: unknown error: Chrome failed to start: exited abnormally.\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429}),platform=Linux 5.8.0-1042-azure x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 117 milliseconds\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: 'f38215bb26d8', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown","systemInformation":"System info: host: 'f38215bb26d8', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'","supportUrl":null,"cause":null,"suppressed":[],"stackTrace":[{"fileName":"RemoteNode.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"newSession","className":"org.openqa.selenium.grid.node.remote.RemoteNode","lineNumber":149,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"startSession","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor","lineNumber":513,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"newSession","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor","lineNumber":440,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"handleNewSessionRequest","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable","lineNumber":648,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"lambda$run$1","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable","lineNumber":612,"classLoaderName":"app"},{"fileName":"ThreadPoolExecutor.java","moduleVersion":"11.0.11","moduleName":"java.base","nativeMethod":false,"methodName":"runWorker","className":"java.util.concurrent.ThreadPoolExecutor","lineNumber":1128,"classLoaderName":null},{"fileName":"ThreadPoolExecutor.java","moduleVersion":"11.0.11","moduleName":"java.base","nativeMethod":false,"methodName":"run","className":"java.util.concurrent.ThreadPoolExecutor$Worker","lineNumber":628,"classLoaderName":null},{"fileName":"Thread.java","moduleVersion":"11.0.11","moduleName":"java.base","nativeMethod":false,"methodName":"run","className":"java.lang.Thread","lineNumber":829,"classLoaderName":null}],"message":"Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: unknown error: Chrome failed to start: exited abnormally.\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429}),platform=Linux 5.8.0-1042-azure x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 117 milliseconds\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: 'f38215bb26d8', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown","rawMessage":"Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: unknown error: Chrome failed to start: exited abnormally.\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429}),platform=Linux 5.8.0-1042-azure x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 117 milliseconds\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown","buildInformation":{"buildRevision":"3a21814679","releaseLabel":"4.0.0"}}
[19:42:46] E/launcher - WebDriverError: Unknown error: {"additionalInformation":"Driver info: driver.version: unknown","localizedMessage":"Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: unknown error: Chrome failed to start: exited abnormally.\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429}),platform=Linux 5.8.0-1042-azure x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 117 milliseconds\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: 'f38215bb26d8', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown","systemInformation":"System info: host: 'f38215bb26d8', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'","supportUrl":null,"cause":null,"suppressed":[],"stackTrace":[{"fileName":"RemoteNode.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"newSession","className":"org.openqa.selenium.grid.node.remote.RemoteNode","lineNumber":149,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"startSession","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor","lineNumber":513,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"newSession","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor","lineNumber":440,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"handleNewSessionRequest","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable","lineNumber":648,"classLoaderName":"app"},{"fileName":"LocalDistributor.java","moduleVersion":null,"moduleName":null,"nativeMethod":false,"methodName":"lambda$run$1","className":"org.openqa.selenium.grid.distributor.local.LocalDistributor$NewSessionRunnable","lineNumber":612,"classLoaderName":"app"},{"fileName":"ThreadPoolExecutor.java","moduleVersion":"11.0.11","moduleName":"java.base","nativeMethod":false,"methodName":"runWorker","className":"java.util.concurrent.ThreadPoolExecutor","lineNumber":1128,"classLoaderName":null},{"fileName":"ThreadPoolExecutor.java","moduleVersion":"11.0.11","moduleName":"java.base","nativeMethod":false,"methodName":"run","className":"java.util.concurrent.ThreadPoolExecutor$Worker","lineNumber":628,"classLoaderName":null},{"fileName":"Thread.java","moduleVersion":"11.0.11","moduleName":"java.base","nativeMethod":false,"methodName":"run","className":"java.lang.Thread","lineNumber":829,"classLoaderName":null}],"message":"Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: unknown error: Chrome failed to start: exited abnormally.\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429}),platform=Linux 5.8.0-1042-azure x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 117 milliseconds\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: 'f38215bb26d8', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown","rawMessage":"Could not start a new session. Could not start a new session. Error while creating session with the driver service. Stopping driver service: unknown error: Chrome failed to start: exited abnormally.\n  (unknown error: DevToolsActivePort file doesn't exist)\n  (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\n  (Driver info: chromedriver=90.0.4430.212 (e3cd97fc771b893b7fd1879196d1215b622c2bed-refs/branch-heads/4430@{#1429}),platform=Linux 5.8.0-1042-azure x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 117 milliseconds\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown\nBuild info: version: '4.0.0', revision: '3a21814679'\nSystem info: host: '05d554c19192', ip: '<IP_ADDRESS>', os.name: 'Linux', os.arch: 'amd64', os.version: '5.8.0-1042-azure', java.version: '11.0.11'\nDriver info: driver.version: unknown","buildInformation":{"buildRevision":"3a21814679","releaseLabel":"4.0.0"}}
npm ERR! Test failed.  See above for more details.
Error on line 156 of /home/runner/work/trafficcontrol/trafficcontrol/.github/actions/tp-integration-tests/entrypoint.sh
    at Object.throwDecodedError (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/lib/error.js:520:9)
    at parseHttpResponse (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/lib/http.js:519:13)
    at /home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/lib/http.js:441:30
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
From: Task: WebDriver.createSession()
    at Function.createSession (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver.js:769:24)
    at Function.createSession (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/chrome.js:761:15)
    at createDriver (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/index.js:170:33)
    at Builder.build (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/selenium-webdriver/index.js:626:16)
    at Hosted.getNewDriver (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/built/driverProviders/driverProvider.js:53:33)
    at Runner.createBrowser (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/built/runner.js:195:43)
    at /home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/built/runner.js:339:29
    at _fulfilled (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/q/q.js:834:54)
    at /home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/q/q.js:863:30
    at Promise.promise.promiseDispatch (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/q/q.js:796:13)
    at /home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/q/q.js:556:49
    at runSingle (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/q/q.js:137:13)
    at flush (/home/runner/go/src/github.com/apache/trafficcontrol/traffic_portal/test/integration/node_modules/protractor/node_modules/q/q.js:125:13)
    at processTicksAndRejections (internal/process/task_queues.js:79:11)
[19:42:46] E/launcher - Process exited with error code 199
Detailed logs produced info Reports artifact
Child process " /home/runner/work/trafficcontrol/trafficcontrol/.github/actions/tp-integration-tests/entrypoint.sh " exited with status code 1 !
0s

Adding this text solved this issue for me:

     SE_EVENT_BUS_HOST: hub

     SE_EVENT_BUS_PUBLISH_PORT: 4442

     SE_EVENT_BUS_SUBSCRIBE_PORT: 4443

     SE_NODE_GRID_URL: http://localhost:4444


Missing tabular data

The Value section of ?Gest is missing the part describing the columns of the data frame.
https://www.rdocumentation.org/packages/spatstat/versions/1.49-0/topics/Gest
RDoc version:

Local RStudio version:

Not sure if this is related, but the documentation is also out of date. The current version of the package is 1.51-0, but I couldn't find documentation for that version of the function.

This is fixed and new version is there
https://www.rdocumentation.org/packages/spatstat/versions/1.51-0


'generate openapi' - CRD name and group are wrong

Bug Report
When running operator-sdk generate openapi against the Jaeger Operator, the CRD's name and group are overridden, from the original jaegers.jaegertracing.io / jaegertracing.io to jaegers.jaegertracing.jaegertracing / jaegertracing.jaegertracing.
What did you do?
Executed operator-sdk generate openapi against an existing operator (Jaeger's).

Additional context
This can be reproduced by running make generate in the following Jaeger Operator branch:
https://github.com/jpkrohling/jaeger-operator/tree/466-ValidateCR

/cc @estroz

@jpkrohling were these CRDs generated with the SDK? The behavior should have been the same when looking at the commits in the SDK, controller-tools, and jaeger-operator at the time jaegertracing_v1_jaeger_crd.yaml was generated; io_v1alpha1_jaeger_crd.yaml probably wasn’t generated with the SDK because we committed scaffolding code after its initial commit. The issue is in controller-tools’ API parsing, which we do not modify before scaffolding CRDs and in fact emulate when generating CRDs in non-Go projects.

I had the impression that the newest was generated via the SDK's generator, but I might be wrong. How is the name composed? Is there something I could change in the Jaeger Operator in order to get the SDK regenerate the proper CRD?

@jpkrohling when I delved into the issue further I figured out that the controller-tools CRD generator was getting the incorrect domain for jaegertracing_v1_jaeger_crd.yaml from io_v1alpha1_jaeger_crd.yaml. The gist of the issue is we only run the CRD generator once and assume the domain is the same for all API's, which is not necessarily correct. This is not an issue in controller-tools v0.2.0, which we'll be upgrading to after #1512 is merged, but I'll make a PR beforehand since this should be fixed in a backwards-compatible way.


youtube-cleanup: remove "New YT-posts" section on homepage

What filter does this issue target?
It doesn't quite belong in the youtube-cleanup template but there is no other good template
What changes should be made?
The following rule www.youtube.com##ytd-rich-item-renderer[is-post].ytd-rich-shelf-renderer:upward(ytd-rich-section-renderer) should be added to block the section on the homepage - after logging in - which show new posts from channels you may like and/or are subscribed to.

Did you write a uBlock filter solving this?
www.youtube.com##ytd-rich-item-renderer[is-post].ytd-rich-shelf-renderer:upward(ytd-rich-section-renderer)

The mobile version of YouTube doesn't have such a section.
Additional notes
No response

Time to start a youtube-homepage-cleanup template?

Yes, that would be a good idea.


Style/MethodName skip unary operator definition

class String
  def ~@
    self * 2
  end
end

p ~'hello'

the same issue happen on when define !@

Cool, will try later, thanks


Ensure passive veOCEAN holders get DF rewards

Current implementation of DF rewards only supports active users. They're getting 100% of funds. See calcrewards.py.
Here's the target breakdown: (From here)

What we care about for this issue is the second-last line: pro-rata to ve held.
What to do: edit calcrwards.py::_calcRewardsUsd(S_USD, P_USD, rewards_avail_USD: float) to allocate to passive users too.

What to do: edit calcrwards.py::_calcRewardsUsd(S_USD, P_USD, rewards_avail_USD: float) to allocate to passive users too.
Dev Ops:
There is no allocation to passive users that needs to happen in that function.
Just take the rewards_avail_usd, and send that amount to FeeDistributor as part of our weekly ops.
UX:
The work will be to move the FeeDistributor estimate rewards function to the FE so users know how much there is to claim. << this is where the majority of the work will be


Added badge and code snippet to the Readme

Developer Checklist

[X] Followed guidelines mentioned in the readme file.
[X] Followed directory structure. (e.g. ProjectName/{USERNAME}/...yourfiles)
[X] Starred ⭐ the Repo (Optional)

Summary
Added badge to the Readme
Screenshot

@harshhere905 You have label this as changeRequested, Can you please specify what are the changes that I need to do?

@SusheelThapa I'll review your changes sooner and will let you know of that.
Thank you so much for your invaluable contribution, folks like you are true hero of open source.

@SusheelThapa @PBJI  according to me this change is good. Just a little suggestion, can you make the badges curve from the side as that looks more good and also add some good colors to it , as these labels are present at the top of our readme file. So if a user came it should catch his eyes so that he can see what are repository has achieved yet.. Thanks and again sry @PBJI for late reply

@harshhere905 Yes you are write these changes would definitely add value, @SusheelThapa I'll merge your request for now and request you to fulfill the suggestions put forth by @harshhere905. Thank you.


chore: Set limit to number of permutations

Description
Issue: AWSUI-60140
When the number of permutations exceeds the established maximum, an error will be thrown, which will be shown instead of the page content:

How has this been tested?
In the first commit of this PR, the maximum was tentatively set to 200 and a few views showed the error which caused existing tests to fail: https://github.com/cloudscape-design/components/actions/runs/12263978366
Example dev page from the first deployment.
The highest number of permutations we currently have in a dev page is 276, so using that as the maximum.

Review checklist
The following items are to be evaluated by the author(s) and the reviewer(s).
Correctness

Changes include appropriate documentation updates.
Changes are backward-compatible if not indicated, see CONTRIBUTING.md.
Changes do not include unsupported browser features, see CONTRIBUTING.md.
Changes were manually tested for accessibility, see accessibility guidelines.

Security

If the code handles URLs: all URLs are validated through the checkSafeUrl function.

Testing

Changes are covered with new/existing unit tests?
Changes are covered with new/existing integration tests?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

but how this solves the issue with 337 permutations mentioned in the ticket? Did we split them somehow?

Yes, here: https://github.com/cloudscape-design/components/pull/3125


add example build with  PlatformIO

Add example build with  PlatformIO
Board : BOARD_ESP32_S3_LCD_EV_BOARD_V1_5

@Lzw655  Please check it out

Sorry, but I'm a bit busy at the moment. I will test it as soon as possible.

Hi @isthaison,
I tested this project and it worked very well. But, to make it more official, there are some suggestions:

Please rebase the commits on the master branch to avoid the conflicting changes caused by the merge from the master branch.
We should remove the large LVGL demo source files, as the "demos" folder contains numerous photos. It would be more beneficial to provide a simple example, such as porting.ino, and include comments to guide users in using LVGL demos.
It's necessary to update the README file. At present, the project is specifically designed for the BOARD_ESP32_S3_LCD_EV_BOARD_V1_5 development board. I recommend using the default ESP_Panel_Board_Supported.h and ESP_Panel_Board_Custom.h, while also providing instructions on how to replace the development board if needed.

It's totally fine if you are unfamiliar with the above process, I can help you with the modifications. Thank you once again for your contribution.

Thank you! I noticed a few errors in the pre-commit stage, so I took your commit and addressed them in pull request #42. Once I'm done with the documentation, I'll proceed to merge it.

I get it !!


Support Cassandra 4.x and add warning/error logs when using 3.x after year of 2033

About https://github.com/uber/cadence/blob/9b4aaa67a447e5158eb5a0c0a1dfdc9239352f68/common/persistence/cassandra/cassandraVisibilityPersistence.go#L45
(Yeah I know it's just 2021 today)
By the time of 2033, users should use Cassandra version 4.0+(Cadence should have support it), otherwise Cadence will not be able to clean up some records with TTL
This is because the max support time is 2038-01-19T03:14:06+00:00 in Cassandra 2.x/3.x using int32 presentation
See more details in https://issues.apache.org/jira/browse/CASSANDRA-14092 , https://en.wikipedia.org/wiki/Year_2038_problem  and
https://fossies.org/linux/apache-cassandra/CASSANDRA-14092.txt

i think given cassandra 3 versions are EOL and PaaS solutions are already stating to not support Cassandra 3 is probably more of a driver to support Cassandra 4. Are there any plans?
https://cassandra.apache.org/_/download.html
https://www.instaclustr.com/blog/instaclustrs-support-for-cassandra/

Cadence has been on v4 for a while and plan to switch to v5 whenever it's validated


Simpler docker image

A simpler docker image not based on Spilo, likely built on top of the postgres image, that uses UID/GID 70.
(Feedback from #21)

We've now shipped a postgres image! :tada:

@wuputah Awesome! Can you please send the link? I'll set it up on my side. We can create some Hydra connectors for Dataplane's Python library.


Table – 130 – Incoming Document

[External] CreateGeneralJournalLineWithDataExchange()
CreateGeneralJournalLineWithDataExchange is exposed as External and was available until the below version of Business Central.
bcinsider.azurecr.io/bcsandbox:dk-ltsc2016
Version: 13.4.27730.0-DK
Platform: 13.0.27722.0
Generic Tag: <IP_ADDRESS>
But in the Latest version of Business Central this function CreateGeneralJournalLineWithDataExchange is no longer available to be invoked externally from AL.
bcinsider.azurecr.io/bcsandbox:dk-ltsc2019
Version: 14.0.29523.0-DK
Platform: 14.0.29501.0
Generic Tag: <IP_ADDRESS>
Can you please check and make this function as external? Or Is there any alternative to this function?
Thanks

Hi clausmacali.
Unfortunately I'm going to return back to internal requested by you function. Looks like you should find another way of usage your functionality.


nixos/networkmanager: remove networking.networkmanager.dynamicHosts

This option was removed because allowing (multiple) regular users to
override host entries affecting the whole system opens up a huge attack
vector. There seem to be very rare cases where this might be useful.
Consider setting system-wide host entries using networking.hosts,
provide them via the DNS server in your network, or use
networking.networkmanager.appendNameservers to point your system to
another (local) nameserver to set those entries.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers

@flokli I commented on #71322 before seeing you opened this PR: https://github.com/NixOS/nixpkgs/pull/71322#issuecomment-543720804
I like to hear a little bit more about why you think this particular option is such a security risk that it needs to be removed from nixpkgs.
I agree with you on the flaws in its implementation (possibly misusing /run/NetworkManager), but my comment linked above offers a solution for that.
But security-wise: The option is disabled by default, and a user has to explicitly enable it and make a decision on which uid should have write permissions to the hosts dirs. I don't see a big security risk here. The documentation could be clearer on the security implications of enabling the option, though.

Moved from @rickynils https://github.com/NixOS/nixpkgs/pull/71322#issuecomment-543720804:

@flokli The dynamicHosts config was merged to master in d80292d, no PR created.
I'm fine with removing the nm-setup-hostsdirs service and the use of /run/NetworkManager/hostsdirs, and instead adding an option for configuring which host dirs NetworkManager/dnsmasq should look at. I guess a more sensible name for the option would then be networking.networkmanager.extraHostsDirs. It would then be up to the user to configure the directories and their permissions.

I like to hear a little bit more about why you think this particular option is such a security risk that it needs to be removed from nixpkgs.

The option is disabled by default, still it's advertised in the NixOS options documentation, so people might enable it because they think "it's a good idea".
Assume there's two users for whom this was configured , and one of this users gets compromised. It's possible to do system-wide DNS redirection attacks of the whole traffic. There's a reason on why /etc/hosts is only writable by root on regular distros, and why you need to use sudo to change most system-wide things.
On top of that, it's a feature that was implemented solely inside nixpkgs. There's no upstream NM "best practices", explaining that this is the recommended way for users to change /etc/hosts records, and I'm not aware of other distros providing something similar.

I guess a more sensible name for the option would then be networking.networkmanager.extraHostsDirs. It would then be up to the user to configure the directories and their permissions.

I'd assume it's also sufficient to do something like this:
environments.etc = {
  target = "NetworkManager/dnsmasq.d/dyndns.conf";
  text = ''
    hostsdir=/path/to/imperative-hostsdir
  '';
}

… and this is simply setting a setting documented in NetworkManager and dnsmasq.
I still consider imperatively adding entries in there as a regular user a hack, or what's the usecase?

My use case is for development/testing. It is a very simple way to dynamically update DNS entries to point to temporary VMs and containers.
I'm completely fine with removing it from nixpkgs, if it is not wanted there. It is simple enough for me to use a local NixOS module for this.
Maybe you should just mention the environment.etc = ... snippet in the release notes, instead of mentioning networking.networkmanager.appendNameservers, since piggy-backing the already running NM-controlled dnsmasq instance is much easier than configure an additional one.

You're right - just configuring the existing dnsmasq (if you happen to use the dnsmasq backend) is easier - I updated the PR and now point towards environment.etc. Does that look good to you now?
This might be a quick & dirty way around it, but at least for local containers this should be handled by NSS - and for anything remote I'd personally want HTTPS and a DNS record anyways.


Add es-switch-field

es-switch wrapped in a field, for use in forms
add field part to Field

fixes: UI-242

@Nurmaan test


error handler doesn't work if I remove next parameter

I'm learning Express.js and using the generator I generated a new app with:
npm install express-generator -g && express myapp

After that I saw in app.js that there is this code:
    // catch 404 and forward to error handler
    app.use(function(req, res, next) {
      next(createError(404));
    });

    // error handler
    app.use(function(err, req, res, next) {
      // set locals, only providing error in development
      res.locals.message = err.message;
      res.locals.error = req.app.get("env") === "development" ? err : {};

      // render the error page
      res.status(err.status || 500);
      res.render("error");
    });

As you can see in the image eslint is complaining about next parameter, declared and never used. I agree, but if I remove it express doesn't render the error page.
Why this behavior?
This is dangerous for me because I can't trust anymore eslint or my coding degree? I'm surely missing something. But what?

This is correct behavior. You may need to configure your eslint to work better with Express.js code. We recommend StandardJS or one of it's derritives.
You can read more about how error handling works in Express and why the next argument is important here: http://expressjs.com/en/guide/error-handling.html

@frederikhors you can add this line right above the line with the next argument - // eslint-disable-next-line no-unused-vars

@niftylettuce thanks. I dropped out eslint for StandardJS.

@niftylettuce but still I think something is not elegant in this way...

Yeah, I agree on it not being super elegant, and would personally like to see it changed in 5.x, there is an open issue about this here: #2896

Also I have a proposed fix for this in this comment: https://github.com/pillarjs/router/pull/59
If anyone has feedback I would appreciate it.  I am willing to put together the final PR for that if people are happy with the approach, but I have been waiting on feedback to that post.


Consider making Razor's File -> Syntax Tree cache a generalized library function

Razor recently got the ability to cache the syntax trees it generated from View files. For my own project, I could use that exact same functionality and I would guess that this is a common pattern for users of ICompileModule.
Please consider generalizing that into a library somewhere between .Mvc.Razor and .Framework.

We are not planning to provide a general capability here. We have a very specific implementation for are uses, and its super simple to write your own cache (where you will control the lifetime and eviction policies)


"git spr merge" ended up creating a new stack after renaming local branch

Case in point, this mess:

https://github.com/ejoffe/spr/pull/167
https://github.com/ejoffe/spr/pull/166
https://github.com/ejoffe/spr/pull/165
https://github.com/ejoffe/spr/pull/164
https://github.com/ejoffe/spr/pull/163
https://github.com/ejoffe/spr/pull/162
https://github.com/ejoffe/spr/pull/161
https://github.com/ejoffe/spr/pull/160
https://github.com/ejoffe/spr/pull/159

None of the commit-id tags changed.

@leoluk  : this sounds like a "you are holding it wrong" problem. what do you propose we do for this case?

Closing as this is inherit to the design of spr.


Fix any reference lookups on domain

This limits any reference lookups to only the main roles, designating roles with long directives
as alternate roles. The default implementation here is for the Sphinx env to lookup against all
roles on the domain, which will return duplicate objects, as both the short and long directives
point to the same object class. Ensure we have no warnings when using using any references that
shouldn't collide.
Fixes #40

:+1:


[APPSEC] Suspicious Request Blocking scenario

Description
This PR adds a new scenario for testing Suspicious Request blocking with ASM.
A new mocked Remote Config response was added to set the tracers in blocking mode.
The sequence of mocked Remote Config responses is as follow:

Distribute ASM-base and ASM-second. ASM-base enables blocking on all rules with confidence: 1. ASM-second disables blocking for crs-913-110.
Remove ASM-second. Blocking should be effective for all rules with confidence: 1.

Workflow

⚠⚠ Create your PR as draft (we're receiving lot of PR, it saves us lot of time) ⚠⚠
Follows the style guidelines of this project (See how to easily lint the code)
Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
Mark it as ready for review

NOTE:  By default in PR only default scenario tests will be launched. Please refer to the documentation to run all scenarios in your PR if needed.

Once your PR is reviewed, you can merge it ! :heart:

You just need to remove the label to run all scenario. I set back the PR as draft, you can reset it as normal once everything is fixed. And as always, please ping me if you need any help !

once the PR is ready, please ping me, I'll handle the conflict


Autofocus the room name field

Details

Adding auto focus functionality to the room name field
Fixed Issues

$ https://github.com/Expensify/App/issues/11507
Tests

At dev env go permissions.js file> return true at canUsePolicyRooms
Open local host > click on fab icon > new room
Verify room name field gets focused at opening

[x] Verify that no errors appear in the JS console

PR Review Checklist

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] iOS / native
[x] Android / native
[x] iOS / Safari
[x] Android / Chrome
[x] MacOS / Chrome
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product was added in all src/languages/* files
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] Any functional components have the displayName property
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

PR Reviewer Checklist
The reviewer will copy/paste it into a new comment and complete it after the author checklist is completed

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] iOS / native
[ ] Android / native
[ ] iOS / Safari
[ ] Android / Chrome
[ ] MacOS / Chrome
[ ] MacOS / Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product was added in all src/languages/* files
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by tagging the marketing team on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] Any functional components have the displayName property
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

QA Steps

Make sure you are inside policy rooms beta
Open new dot > click on fab icon > new room
Verify room name field gets focused at opening

[x] Verify that no errors appear in the JS console

Screenshots

Web

https://user-images.githubusercontent.com/101883770/193874501-f22e7e1e-b044-41e5-8776-73c96561ad20.mov
Mobile Web - Chrome

https://user-images.githubusercontent.com/101883770/193874261-96d51d70-38aa-48ea-945f-42a838f4a266.mov
Mobile Web - Safari

https://user-images.githubusercontent.com/101883770/193874324-fc91599d-2491-4561-b533-828722590650.mov
Desktop

https://user-images.githubusercontent.com/101883770/193874193-85496475-377b-49ef-8da2-e97c871875eb.mov
iOS

https://user-images.githubusercontent.com/101883770/193874352-d7d3cde2-efce-4ec0-ad67-22936f48aa6b.mov
Android

https://user-images.githubusercontent.com/101883770/193874419-ffda6262-684f-4545-a907-d8d0a8483940.mov

Tests are failing but not due to this PR.

@parasharrajat, I have made the change.

🚀 Deployed to staging by @Beamanator in version: 1.2.13-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by @yuwenmemon in version: 1.2.13-5 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Key Trust or Cert Trust

Hello , please specify if the feature is available for WHFB in Key Trust or Cert Trust architecture.
It seems it works only in the Cert Trust WHFB model .
Thanks in advance
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f50a8599-301a-5993-1a86-e722f66ffcf2
Version Independent ID: fffd0fc9-97f6-cfa9-70fb-7f1308804de0
Content: Dual Enrollment - Microsoft 365 Security
Content Source: windows/security/identity-protection/hello-for-business/hello-feature-dual-enrollment.md
Product: w10
GitHub Login: @mapalko
Microsoft Alias: mapalko

Closing via #9277
Thank you


add ability to configure service monitor and prometheus rules

Add Service Monitor
Add Prometheus Rules
Add configurable values
Bump chart version

Thank you @alinjie for the contribution, sorry for the late reply, we missed this PR before.
Could you please file PR to this here? Because this repo is mirrored from the https://github.com/longhorn/longhorn/tree/master/chart, thank you.
Besides that, you need to add the signed-off-by in your git commit.

We have an PR to address this issue already.

Closing but follow up with https://github.com/longhorn/longhorn/pull/3580. @alinjie thanks for the contribution.


Motion detection stop recording only 90+ sec after it should have stopped recording

Tested on 2 Raspberry Pi 2, RPi Cam Control v6.0.12, with updated Raspbian Jessie:
Linux raspberrypi 4.1.12-v7+ #824 SMP PREEMPT Wed Oct 28 16:46:35 GMT 2015 armv7l GNU/Linux
Motion detection stop recording only 90+ sec after it should have stopped recording. Tested by pointing on a static room, and second test by covering with dark box (without red led).
Here are my configuration file (user name and password removed):
# Rename this distribution example file to motion.conf
#
# This config file was generated by motion 3.2.12

############################################################
# Daemon
############################################################

# Start in daemon (background) mode and release terminal (default: off)
daemon on

# File to store the process ID, also called pid file. (default: not defined)
; process_id_file /var/run/motion/motion.pid

############################################################
# Basic Setup Mode
############################################################

# Start in Setup-Mode, daemon disabled. (default: off)
setup_mode off

###########################################################
# Capture device options
############################################################

# Videodevice to be used for capturing  (default /dev/video0)
# for FreeBSD default is /dev/bktr0
; videodevice /dev/video0

# v4l2_palette allows to choose preferable palette to be use by motion
# to capture from those supported by your videodevice. (default: 8)
# E.g. if your videodevice supports both V4L2_PIX_FMT_SBGGR8 and
# V4L2_PIX_FMT_MJPEG then motion will by default use V4L2_PIX_FMT_MJPEG.
# Setting v4l2_palette to 1 forces motion to use V4L2_PIX_FMT_SBGGR8
# instead.
#
# Values :
# V4L2_PIX_FMT_SN9C10X : 0  'S910'
# V4L2_PIX_FMT_SBGGR8  : 1  'BA81'
# V4L2_PIX_FMT_MJPEG   : 2  'MJPEG'
# V4L2_PIX_FMT_JPEG    : 3  'JPEG'
# V4L2_PIX_FMT_RGB24   : 4  'RGB3'
# V4L2_PIX_FMT_UYVY    : 5  'UYVY'
# V4L2_PIX_FMT_YUYV    : 6  'YUYV'
# V4L2_PIX_FMT_YUV422P : 7  '422P'
# V4L2_PIX_FMT_YUV420  : 8  'YU12'
v4l2_palette 8

# Tuner device to be used for capturing using tuner as source (default /dev/tuner0)
# This is ONLY used for FreeBSD. Leave it commented out for Linux
; tunerdevice /dev/tuner0

# The video input to be used (default: 8)
# Should normally be set to 0 or 1 for video/TV cards, and 8 for USB cameras
input 8

# The video norm to use (only for video capture and TV tuner cards)
# Values: 0 (PAL), 1 (NTSC), 2 (SECAM), 3 (PAL NC no colour). Default: 0 (PAL)
norm 0

# The frequency to set the tuner to (kHz) (only for TV tuner cards) (default: 0)
frequency 0

# Rotate image this number of degrees. The rotation affects all saved images as
# well as mpeg movies. Valid values: 0 (default = no rotation), 90, 180 and 270.
rotate 0

# Image width (pixels). Valid range: Camera dependent, default: 352
width 352

# Image height (pixels). Valid range: Camera dependent, default: 288
height 288

# Maximum number of frames to be captured per second.
# Valid range: 2-100. Default: 100 (almost no limit).
framerate 4

# Minimum time in seconds between capturing picture frames from the camera.
# Default: 0 = disabled - the capture rate is given by the camera framerate.
# This option is used when you want to capture images at a rate lower than 2 per second.
minimum_frame_time 0

# URL to use if you are using a network camera, size will be autodetected (incl http:// ftp:// or file:///)
# Must be a URL that returns single jpeg pictures or a raw mjpeg stream. Default: Not defined
netcam_url http://localhost/cam_pic.php

# Username and password for network camera (only if required). Default: not defined
# Syntax is user:password
netcam_userpass ***MY_USER_AND_PASSWORD_IS_HERE***

# The setting for keep-alive of network socket, should improve performance on compatible net cameras.
# 1.0:         The historical implementation using HTTP/1.0, closing the socket after each http request.
# keep_alive:  Use HTTP/1.0 requests with keep alive header to reuse the same connection.
# 1.1:         Use HTTP/1.1 requests that support keep alive as default.
# Default: 1.0
; netcam_http 1.0

# URL to use for a netcam proxy server, if required, e.g. "http://myproxy".
# If a port number other than 80 is needed, use "http://myproxy:1234".
# Default: not defined
; netcam_proxy value

# Set less strict jpeg checks for network cameras with a poor/buggy firmware.
# Default: off
netcam_tolerant_check off

# Let motion regulate the brightness of a video device (default: off).
# The auto_brightness feature uses the brightness option as its target value.
# If brightness is zero auto_brightness will adjust to average brightness value 128.
# Only recommended for cameras without auto brightness
auto_brightness off

# Set the initial brightness of a video device.
# If auto_brightness is enabled, this value defines the average brightness level
# which Motion will try and adjust to.
# Valid range 0-255, default 0 = disabled
brightness 0

# Set the contrast of a video device.
# Valid range 0-255, default 0 = disabled
contrast 0

# Set the saturation of a video device.
# Valid range 0-255, default 0 = disabled
saturation 0

# Set the hue of a video device (NTSC feature).
# Valid range 0-255, default 0 = disabled
hue 0

############################################################
# Round Robin (multiple inputs on same video device name)
############################################################

# Number of frames to capture in each roundrobin step (default: 1)
roundrobin_frames 1

# Number of frames to skip before each roundrobin step (default: 1)
roundrobin_skip 1

# Try to filter out noise generated by roundrobin (default: off)
switchfilter off

############################################################
# Motion Detection Settings:
############################################################

# Threshold for number of changed pixels in an image that
# triggers motion detection (default: 1500)
threshold 1500

# Automatically tune the threshold down if possible (default: off)
threshold_tune off

# Noise threshold for the motion detection (default: 32)
noise_level 32

# Automatically tune the noise threshold (default: on)
noise_tune on

# Despeckle motion image using (e)rode or (d)ilate or (l)abel (Default: not defined)
# Recommended value is EedDl. Any combination (and number of) of E, e, d, and D is valid.
# (l)abeling must only be used once and the 'l' must be the last letter.
# Comment out to disable
despeckle EedDl

# Detect motion in predefined areas (1 - 9). Areas are numbered like that:  1 2 3
# A script (on_area_detected) is started immediately when motion is         4 5 6
# detected in one of the given areas, but only once during an event.        7 8 9
# One or more areas can be specified with this option. (Default: not defined)
; area_detect value

# PGM file to use as a sensitivity mask.
# Full path name to. (Default: not defined)
; mask_file value

# Dynamically create a mask file during operation (default: 0)
# Adjust speed of mask changes from 0 (off) to 10 (fast)
smart_mask_speed 0

# Ignore sudden massive light intensity changes given as a percentage of the picture
# area that changed intensity. Valid range: 0 - 100 , default: 0 = disabled
lightswitch 0

# Picture frames must contain motion at least the specified number of frames
# in a row before they are detected as true motion. At the default of 1, all
# motion is detected. Valid range: 1 to thousands, recommended 1-5
minimum_motion_frames 2

# Specifies the number of pre-captured (buffered) pictures from before motion
# was detected that will be output at motion detection.
# Recommended range: 0 to 5 (default: 0)
# Do not use large values! Large values will cause Motion to skip video frames and
# cause unsmooth mpegs. To smooth mpegs use larger values of post_capture instead.
pre_capture 0

# Number of frames to capture after motion is no longer detected (default: 0)
post_capture 0

# Gap is the seconds of no motion detection that triggers the end of an event
# An event is defined as a series of motion images taken within a short timeframe.
# Recommended value is 60 seconds (Default). The value 0 is allowed and disables
# events causing all Motion to be written to one single mpeg file and no pre_capture.
gap 3

# Maximum length in seconds of an mpeg movie
# When value is exceeded a new mpeg file is created. (Default: 0 = infinite)
max_mpeg_time 0

# Always save images even if there was no motion (default: off)
output_all off

############################################################
# Image File Output
############################################################

# Output 'normal' pictures when motion is detected (default: on)
# Valid values: on, off, first, best, center
# When set to 'first', only the first picture of an event is saved.
# Picture with most motion of an event is saved when set to 'best'.
# Picture with motion nearest center of picture is saved when set to 'center'.
# Can be used as preview shot for the corresponding movie.
output_normal off

# Output pictures with only the pixels moving object (ghost images) (default: off)
output_motion off

# The quality (in percent) to be used by the jpeg compression (default: 75)
quality 75

# Output ppm images instead of jpeg (default: off)
ppm off

############################################################
# FFMPEG related options
# Film (mpeg) file output, and deinterlacing of the video input
# The options movie_filename and timelapse_filename are also used
# by the ffmpeg feature
############################################################

# Use ffmpeg to encode mpeg movies in realtime (default: off)
ffmpeg_cap_new off

# Use ffmpeg to make movies with only the pixels moving
# object (ghost images) (default: off)
ffmpeg_cap_motion off

# Use ffmpeg to encode a timelapse movie
# Default value 0 = off - else save frame every Nth second
ffmpeg_timelapse 0

# The file rollover mode of the timelapse video
# Valid values: hourly, daily (default), weekly-sunday, weekly-monday, monthly, manual
ffmpeg_timelapse_mode daily

# Bitrate to be used by the ffmpeg encoder (default: 400000)
# This option is ignored if ffmpeg_variable_bitrate is not 0 (disabled)
ffmpeg_bps 500000

# Enables and defines variable bitrate for the ffmpeg encoder.
# ffmpeg_bps is ignored if variable bitrate is enabled.
# Valid values: 0 (default) = fixed bitrate defined by ffmpeg_bps,
# or the range 2 - 31 where 2 means best quality and 31 is worst.
ffmpeg_variable_bitrate 0

# Codec to used by ffmpeg for the video compression.
# Timelapse mpegs are always made in mpeg1 format independent from this option.
# Supported formats are: mpeg1 (ffmpeg-0.4.8 only), mpeg4 (default), and msmpeg4.
# mpeg1 - gives you files with extension .mpg
# mpeg4 or msmpeg4 - gives you files with extension .avi
# msmpeg4 is recommended for use with Windows Media Player because
# it requires no installation of codec on the Windows client.
# swf - gives you a flash film with extension .swf
# flv - gives you a flash video with extension .flv
# ffv1 - FF video codec 1 for Lossless Encoding ( experimental )
# mov - QuickTime ( testing )
ffmpeg_video_codec swf

# Use ffmpeg to deinterlace video. Necessary if you use an analog camera
# and see horizontal combing on moving objects in video or pictures.
# (default: off)
ffmpeg_deinterlace off

############################################################
# Snapshots (Traditional Periodic Webcam File Output)
############################################################

# Make automated snapshot every N seconds (default: 0 = disabled)
snapshot_interval 0

############################################################
# Text Display
# %Y = year, %m = month, %d = date,
# %H = hour, %M = minute, %S = second, %T = HH:MM:SS,
# %v = event, %q = frame number, %t = thread (camera) number,
# %D = changed pixels, %N = noise level, \n = new line,
# %i and %J = width and height of motion area,
# %K and %L = X and Y coordinates of motion center
# %C = value defined by text_event - do not use with text_event!
# You can put quotation marks around the text to allow
# leading spaces
############################################################

# Locate and draw a box around the moving object.
# Valid values: on, off and preview (default: off)
# Set to 'preview' will only draw a box in preview_shot pictures.
locate off

# Draws the timestamp using same options as C function strftime(3)
# Default: %Y-%m-%d\n%T = date in ISO format and time in 24 hour clock
# Text is placed in lower right corner
text_right %Y-%m-%d\n%T-%q

# Draw a user defined text on the images using same options as C function strftime(3)
# Default: Not defined = no text
# Text is placed in lower left corner
; text_left CAMERA %t

# Draw the number of changed pixed on the images (default: off)
# Will normally be set to off except when you setup and adjust the motion settings
# Text is placed in upper right corner
text_changes off

# This option defines the value of the special event conversion specifier %C
# You can use any conversion specifier in this option except %C. Date and time
# values are from the timestamp of the first image in the current event.
# Default: %Y%m%d%H%M%S
# The idea is that %C can be used filenames and text_left/right for creating
# a unique identifier for each event.
text_event %Y%m%d%H%M%S

# Draw characters at twice normal size on images. (default: off)
text_double off

############################################################
# Target Directories and filenames For Images And Films
# For the options snapshot_, jpeg_, mpeg_ and timelapse_filename
# you can use conversion specifiers
# %Y = year, %m = month, %d = date,
# %H = hour, %M = minute, %S = second,
# %v = event, %q = frame number, %t = thread (camera) number,
# %D = changed pixels, %N = noise level,
# %i and %J = width and height of motion area,
# %K and %L = X and Y coordinates of motion center
# %C = value defined by text_event
# Quotation marks round string are allowed.
############################################################

# Target base directory for pictures and films
# Recommended to use absolute path. (Default: current working directory)
target_dir /var/www/html/media

# File path for snapshots (jpeg or ppm) relative to target_dir
# Default: %v-%Y%m%d%H%M%S-snapshot
# Default value is equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d/%H/%M/%S-snapshot
# File extension .jpg or .ppm is automatically added so do not include this.
# Note: A symbolic link called lastsnap.jpg created in the target_dir will always
# point to the latest snapshot, unless snapshot_filename is exactly 'lastsnap'
snapshot_filename %v-%Y%m%d%H%M%S-snapshot

# File path for motion triggered images (jpeg or ppm) relative to target_dir
# Default: %v-%Y%m%d%H%M%S-%q
# Default value is equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d/%H/%M/%S-%q
# File extension .jpg or .ppm is automatically added so do not include this
# Set to 'preview' together with best-preview feature enables special naming
# convention for preview shots. See motion guide for details
jpeg_filename vthumb_%Y%m%d_%H%M%S

# File path for motion triggered ffmpeg films (mpeg) relative to target_dir
# Default: %v-%Y%m%d%H%M%S
# Default value is equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d/%H%M%S
# File extension .mpg or .avi is automatically added so do not include this
# This option was previously called ffmpeg_filename
movie_filename %v-%Y%m%d%H%M%S

# File path for timelapse mpegs relative to target_dir
# Default: %Y%m%d-timelapse
# Default value is near equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d-timelapse
# File extension .mpg is automatically added so do not include this
timelapse_filename %Y%m%d-timelapse

############################################################
# Live Webcam Server
############################################################

# The mini-http server listens to this port for requests (default: 0 = disabled)
webcam_port 0

# Quality of the jpeg (in percent) images produced (default: 50)
webcam_quality 50

# Output frames at 1 fps when no motion is detected and increase to the
# rate given by webcam_maxrate when motion is detected (default: off)
webcam_motion off

# Maximum framerate for webcam streams (default: 1)
webcam_maxrate 1

# Restrict webcam connections to localhost only (default: on)
webcam_localhost on

# Limits the number of images per connection (default: 0 = unlimited)
# Number can be defined by multiplying actual webcam rate by desired number of seconds
# Actual webcam rate is the smallest of the numbers framerate and webcam_maxrate
webcam_limit 0

############################################################
# HTTP Based Control
############################################################

# TCP/IP port for the http server to listen on (default: 0 = disabled)
control_port 6642

# Restrict control connections to localhost only (default: on)
control_localhost on

# Output for http server, select off to choose raw text plain (default: on)
control_html_output off

# Authentication for the http based control. Syntax username:password
# Default: not defined (Disabled)
; control_authentication username:password

############################################################
# Tracking (Pan/Tilt)
############################################################

# Type of tracker (0=none (default), 1=stepper, 2=iomojo, 3=pwc, 4=generic, 5=uvcvideo)
# The generic type enables the definition of motion center and motion size to
# be used with the conversion specifiers for options like on_motion_detected
track_type 0

# Enable auto tracking (default: off)
track_auto off

# Serial port of motor (default: none)
; track_port value

# Motor number for x-axis (default: 0)
track_motorx 0

# Motor number for y-axis (default: 0)
track_motory 0

# Maximum value on x-axis (default: 0)
track_maxx 0

# Maximum value on y-axis (default: 0)
track_maxy 0

# ID of an iomojo camera if used (default: 0)
track_iomojo_id 0

# Angle in degrees the camera moves per step on the X-axis
# with auto-track (default: 10)
# Currently only used with pwc type cameras
track_step_angle_x 10

# Angle in degrees the camera moves per step on the Y-axis
# with auto-track (default: 10)
# Currently only used with pwc type cameras
track_step_angle_y 10

# Delay to wait for after tracking movement as number
# of picture frames (default: 10)
track_move_wait 10

# Speed to set the motor to (stepper motor option) (default: 255)
track_speed 255

# Number of steps to make (stepper motor option) (default: 40)
track_stepsize 40

############################################################
# External Commands, Warnings and Logging:
# You can use conversion specifiers for the on_xxxx commands
# %Y = year, %m = month, %d = date,
# %H = hour, %M = minute, %S = second,
# %v = event, %q = frame number, %t = thread (camera) number,
# %D = changed pixels, %N = noise level,
# %i and %J = width and height of motion area,
# %K and %L = X and Y coordinates of motion center
# %C = value defined by text_event
# %f = filename with full path
# %n = number indicating filetype
# Both %f and %n are only defined for on_picture_save,
# on_movie_start and on_movie_end
# Quotation marks round string are allowed.
############################################################

# Do not sound beeps when detecting motion (default: on)
# Note: Motion never beeps when running in daemon mode.
quiet on

# Command to be executed when an event starts. (default: none)
# An event starts at first motion detected after a period of no motion defined by gap
on_event_start echo -n '1' > /var/www/html/FIFO1

# Command to be executed when an event ends after a period of no motion
# (default: none). The period of no motion is defined by option gap.
on_event_end echo -n '0' > /var/www/html/FIFO1

# Command to be executed when a picture (.ppm|.jpg) is saved (default: none)
# To give the filename as an argument to a command append it with %f
; on_picture_save value

# Command to be executed when a motion frame is detected (default: none)
; on_motion_detected value

# Command to be executed when motion in a predefined area is detected
# Check option 'area_detect'.   (default: none)
; on_area_detected value

# Command to be executed when a movie file (.mpg|.avi) is created. (default: none)
# To give the filename as an argument to a command append it with %f
; on_movie_start value

# Command to be executed when a movie file (.mpg|.avi) is closed. (default: none)
# To give the filename as an argument to a command append it with %f
; on_movie_end value

# Command to be executed when a camera can't be opened or if it is lost
# NOTE: There is situations when motion doesn't detect a lost camera!
# It depends on the driver, some drivers don't detect a lost camera at all
# Some hang the motion thread. Some even hang the PC! (default: none)
; on_camera_lost value

############################################################
# Common Options For MySQL and PostgreSQL database features.
# Options require the MySQL/PostgreSQL options to be active also.
############################################################

# Log to the database when creating motion triggered image file  (default: on)
sql_log_image on

# Log to the database when creating a snapshot image file (default: on)
sql_log_snapshot on

# Log to the database when creating motion triggered mpeg file (default: off)
sql_log_mpeg off

# Log to the database when creating timelapse mpeg file (default: off)
sql_log_timelapse off

# SQL query string that is sent to the database
# Use same conversion specifiers has for text features
# Additional special conversion specifiers are
# %n = the number representing the file_type
# %f = filename with full path
# Default value:
# insert into security(camera, filename, frame, file_type, time_stamp, text_event) values('%t', '%f', '%q', '%n', '%Y-%m-%d %T', '%C')
sql_query insert into security(camera, filename, frame, file_type, time_stamp, event_time_stamp) values('%t', '%f', '%q', '%n', '%Y-%m-%d %T', '%C')

############################################################
# Database Options For MySQL
############################################################

# Mysql database to log to (default: not defined)
; mysql_db value

# The host on which the database is located (default: localhost)
; mysql_host value

# User account name for MySQL database (default: not defined)
; mysql_user value

# User password for MySQL database (default: not defined)
; mysql_password value

############################################################
# Database Options For PostgreSQL
############################################################

# PostgreSQL database to log to (default: not defined)
; pgsql_db value

# The host on which the database is located (default: localhost)
; pgsql_host value

# User account name for PostgreSQL database (default: not defined)
; pgsql_user value

# User password for PostgreSQL database (default: not defined)
; pgsql_password value

# Port on which the PostgreSQL database is located (default: 5432)
; pgsql_port 5432

############################################################
# Video Loopback Device (vloopback project)
############################################################

# Output images to a video4linux loopback device
# The value '-' means next available (default: not defined)
; video_pipe value

# Output motion images to a video4linux loopback device
# The value '-' means next available (default: not defined)
; motion_video_pipe value

##############################################################
# Thread config files - One for each camera.
# Except if only one camera - You only need this config file.
# If you have more than one camera you MUST define one thread
# config file for each camera in addition to this config file.
##############################################################

# Remember: If you have more than one camera you must have one
# thread file for each camera. E.g. 2 cameras requires 3 files:
# This motion.conf file AND thread1.conf and thread2.conf.
# Only put the options that are unique to each camera in the
# thread config files.
; thread /usr/local/etc/thread1.conf
; thread /usr/local/etc/thread2.conf
; thread /usr/local/etc/thread3.conf
; thread /usr/local/etc/thread4.conf

This conf file is the one that has been put on by the installer which is one designed for the wheezy version of motion and is almost certainly confusing the motion programme.
We are making good progress on supporting Jessie including using external motion detection, but at the moment it is better to use the internal motion detection selected under camera settings. I use the internal method exclusively now as it has quite a few advantages over the original external motion detection method.


cosign/tuf: init

Summary

Towards #280, adapt Cosign and TUF towards protobuf schemas.
TODO

[ ] test regressions now that validation catches test helpers using expired certificates
[ ] https://github.com/sigstore/sigstore-rs/pull/305/files#diff-26cf51a00e2cededd5aa7f74218d4a0798eb29488a837a0341760fee609ba832R79-R84
[ ] https://github.com/sigstore/sigstore-rs/pull/305/files#diff-26cf51a00e2cededd5aa7f74218d4a0798eb29488a837a0341760fee609ba832R88-R89

Release Note

Documentation

@woodruffw and @tnytown mentioned some ownership design considerations that I'm not sure I fully understood. A closer look at some lifetime choices e.g.

@tnytown can confirm, but I believe the question was whether it makes sense to allow the interior 'a lifetime on CertificateDer<'a> to "cascade" throughout the refactor. The alternative would be to create an OwnedCertificateDer wrapper that maintains a self-reference, using a crate like self_cell.
This is exactly what PyCA Cryptography does, turning a Certificate<'a> into an OwnedCertificate:
https://github.com/pyca/cryptography/blob/4c07d8eb289aaa0fbcdbcf370724122905fade02/src/rust/src/x509/certificate.rs#L29C1-L36
Whether or not this actually makes sense to do, however, is ultimately a design question. If the sigstore-rs maintainers are okay with the lifetime, then it probably doesn't make sense to do 🙂


ADD: Travis CI and Codecov

Overview
Adds .travis.yml and coverage script for codecov
Review Checklist

[ ] Merge destination is correct
[ ] Tests cases have been added where appropriate (Jest, Loki, Cypress)
[ ] Code is correct as understood and conforms to quality standards

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@c5347d2). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             develop     #12   +/-   ##
=========================================
  Coverage           ?   9.09%
=========================================
  Files              ?       6
  Lines              ?      55
  Branches           ?      12
=========================================
  Hits               ?       5
  Misses             ?      50
  Partials           ?       0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c5347d2...4c0d207. Read the comment docs.


Add files via upload

Adds Virginia Communities Colleges to system  http://www.vccs.edu

@jessicamendonza I am sorry, the domain vccs.edu is blacklisted because the school allows to register an email address there even if you are not a student of the school. If you are the student of this school, please apply with official document (OFFICIAL DOCUMENT tab at https://www.jetbrains.com/shop/eform/students).


Nbhoski patch 1

Description
https://github.com/jenkinsci/matlab-plugin
https://issues.jenkins-ci.org/browse/HOSTING-738
Committers List
@nbhoski
@mcafaro
@acampbel
@mw-akumar
Submitter checklist for changing permissions

Always

[x] Add link to plugin/component Git repository in description above

For a newly hosted plugin only

[x] Add link to resolved HOSTING issue in description above

For a new permissions file only

[x] Make sure the file is created in permissions/ directory
[x] artifactId (pom.xml) is used for name (permissions YAML file).
[x] groupId / artifactId (pom.xml) are correctly represented in path (permissions YAML file)
[x] Check that the file is named plugin-${artifactId}.yml for plugins

Request to approve the pull request and merge it.


Resetting current_random_doc and current_doc

In the class BERTDataset the two variables self.current_random_doc and self.current_doc are never reset to 0, even when the corpus is closed and reopened. Is it supposed to work this way? I'd think it would run into issues on a small corpus where one counter gets to the same document but the counter is different because it was opened a second time.
https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ae8c8be1e3fc770968cd3fdb3b643e0b166e540/examples/lm_finetuning/simple_lm_finetuning.py#L42-L231

Hmm maybe @Rocketknight1 have an insight on this?


Reference index

This PR implments a new ReferenceIndex which records all references between models. This includes references from ForeignKeys, embeds/links in rich text and chooser blocks. The index is automatically updated on post_save and a rebuild_references_index management command is provided to rebuild it from scratch. The usage reports have been updated to use this index.
I have also removed the WAGTAIL_USAGE_COUNT_ENABLED setting, which was originally included to allow people to opt-in to usage count for performance reasons. Performance should be much better with the new approach so I think it should always be enabled.

Thanks for adding the permission check @kaedroho!
Looks like we still need to address the other side - ensuring that the object listing doesn't show anything they're not supposed to see - but I'd be happy to merge this as is, and create a follow-up PR for that.

Afraid I've found some new issues on final testing:

The usage count on snippets is unstyled and missing margins:

It probably makes sense for this to appear in the Status side panel (which currently only appears for snippets with revisioning or draft status enabled, but could be made always-on)
On Bakerydemo, after running rebuild_references_index, the "Japan" snippet is still showing 0 uses despite being referenced from the Anpan breads page - investigating now.

On Bakerydemo, after running rebuild_references_index, the "Japan" snippet is still showing 0 uses despite being referenced from the Anpan breads page - investigating now.

Got it - it's because rebuild_references_index is scanning the wagtailcore.Page model after breads.BreadPage. When retrieving the Anpan page as a plain Page, it naturally finds no references, and that empty list overwrites the real reference list that was stored under the same ID and content type.

Got it - it's because rebuild_references_index is scanning the wagtailcore.Page model after breads.BreadPage. When retrieving the Anpan page as a plain Page, it naturally finds no references, and that empty list overwrites the real reference list that was stored under the same ID and content type.

This could be fixed for pages in particular by ensuring that we only ever index the specific instance. However, we'd strongly prefer to avoid special-casing the Page model, since in all other respects this implementation is agnostic of any Wagtail models. Since it acts on all Django models unless explicitly opted-out, we ought to ensure it works for all models using multi-table inheritance, and in the general case there may not be a way of obtaining a specific instance from a basic one (other than by scanning all installed models to find subclasses and doing a database lookup for each one).
The bug could also be handled within the rebuild_references_index command by keeping track of the already-indexed model / pk pairs, and skipping the indexing of any objects that we've previously indexed under a more specific subclass. However, this bug could also surface elsewhere any time we trigger a reindex on a model instance that isn't the most specific subclass (for example, by retrieving a Page instance, updating its title, and saving).
I haven't fully figured out a comprehensive fix yet, but I suspect it might require us to store the content type of the source object as two distinct fields, one for the base model and one for the specific model that we actually ran ReferenceIndex.create_or_update_for_object against. Then, on any subsequent call to ReferenceIndex.create_or_update_for_object, potentially with an object in its less specific form, it's only allowed to overwrite records where the specific_content_type field corresponds to that object or a more general subclass. That way, any references relating to a field on a more specific subclass will be left alone. (Alternatively, create_or_update_for_object could be made to take into account the model_path field of ReferenceIndex, and skip over any records that refer to an unrecognised field on the assumption that it belongs to a more specific subclass - however, this risks leaving behind stale entries if the field in question has actually been dropped in a migration.)

Another issue - it doesn't seem to be picking up document links in rich text. Tested by editing the FooterText snippet on bakerydemo to add a page link, document link and image, then running the following from the shell:
>>> from wagtail.models import ReferenceIndex
>>> from bakerydemo.base.models import FooterText
>>> ft = FooterText.objects.first()
>>> list(ReferenceIndex._extract_references_from_object(ft))
[(1, '61', 'body.', 'body.'), (2, '52', 'body.', 'body.')]

The two returned references correspond to the page and image - there is no reference returned for the document link.

Superseded by #9279


New update URL?

Thank you for this writeup!
I checked the WU_URL
https://weatherstation.wunderground.com/weatherstation
And got a 404 error. Any chance you know the new url to send the updates?

Checked their PWS upload protocol. The link with /weatherstation.php is correct. I was mistaken.