[BUG] Unable to join new v1.2.0 node to v1.2.0 cluster

Describe the bug
Provision fresh install v1.2.0 and join to existing v1.2.0 cluster
To Reproduce
Steps to reproduce the behavior:

Upgrade existing cluster to v1.2.0
Install from ISO a new v1.2.0 node and configure to join cluster.

Expected behavior
Joins cluster without error.
Support bundle
It does not make it past rancherd bootstrap, no bundle to provide
Environment

Harvester ISO version:
v1.2.0
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):
Dell Optiplex 3080
i5-12500t
nvme 2TB
let's encrypt wildcard cert without IPss in SAN

Additional context
It spams the logs with:
Sep 10 14:33:28 hv03 systemd[1]: Starting Rancher Bootstrap...
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Loading config file [/usr/share/rancher/rancherd/config.yaml.d/50-defaults.yaml]"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Loading config file [/usr/share/rancher/rancherd/config.yaml.d/91-harvester-bootstrap-repo.yaml]"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Loading config file [/etc/rancher/rancherd/config.yaml]"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Bootstrapping Rancher (v2.7.5/v1.25.9+rke2r1)"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Writing plan file to /var/lib/rancher/rancherd/plan/plan.json"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Applying plan with checksum "
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="No image provided, creating empty working directory /var/lib/rancher/rancherd/plan/work/20230910-143328-applied.plan/_0"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="Running command: /usr/bin/env [sh /var/lib/rancher/rancherd/install.sh]"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="[stdout]: [INFO]  Using default agent configuration directory /etc/rancher/agent"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="[stdout]: [INFO]  Using default agent var directory /var/lib/rancher/agent"
Sep 10 14:33:28 hv03 rancherd[2050]: time="2023-09-10T14:33:28Z" level=info msg="[stderr]: [WARN]  /usr/local is read-only or a mount point; installing to /opt/rancher-system-agent"
Sep 10 14:33:29 hv03 rancherd[2050]: time="2023-09-10T14:33:29Z" level=info msg="[stdout]: [INFO]  Determined CA is necessary to connect to Rancher"
Sep 10 14:33:29 hv03 rancherd[2050]: time="2023-09-10T14:33:29Z" level=info msg="[stdout]: [INFO]  Successfully downloaded CA certificate"
Sep 10 14:33:29 hv03 rancherd[2050]: time="2023-09-10T14:33:29Z" level=info msg="[stdout]: [INFO]  Value from https://<IP_ADDRESS>/cacerts is an x509 certificate"
Sep 10 14:33:29 hv03 rancherd[2050]: time="2023-09-10T14:33:29Z" level=info msg="[stderr]: curl: (60) SSL: no alternative certificate subject name matches target host name '<IP_ADDRESS>'"
Sep 10 14:33:29 hv03 rancherd[2050]: time="2023-09-10T14:33:29Z" level=info msg="[stderr]: More details here: https://curl.se/docs/sslcerts.html"
Sep 10 14:33:29 hv03 rancherd[2050]: time="2023-09-10T14:33:29Z" level=info msg="[stderr]: "

Checking install.sh I see:
hv03:/oem # head -n10 /var/lib/rancher/rancherd/install.sh
#!/usr/bin/env sh

CATTLE_AGENT_BINARY_BASE_URL="https://<IP_ADDRESS>/assets"
CATTLE_SERVER=https://<IP_ADDRESS>
CATTLE_CA_CHECKSUM="798db3fbaa5fe211159fa1cddb8e1e2cdbca12e3d355a491e6bf8134d2f14272"

This is the IP of the cluster VIP - harvester.slack.house. The IP was never entered during installation. Only 'harvester.slack.house' for vip.
No proxy was configured, or remote harvester config.
I do not understand why rancherd is placing these two variables at the top of install.sh, or why they are converting DNS to an IP :(
Checking a v1.1.2 file only the first variable is set, and using the same value from the installer  (dns).
I have not found a work around.

Hi @slackspace-io, I'm not sure I understand it correctly. Did you provide harvester.slack.house to the VIP field when installing the new node? And is harvester.slack.house resolved to <IP_ADDRESS>?
I can join a new node with the VIP (not the domain name) configured. While joining a new node with the domain name, it complains about the certificate issue because the original self-signed certificate does not signed with the domain name but the VIP.

I entered 'harvester.slack.house' as the VIP during the install for joining the new node.
I ended up changing it to an IP within the /oem/ config files, but at the same time did stop rancher addon, and removed the SSL settings on the harvester cluster.
I then installed my last node, used an IP during the install screens, and it joined without any issue.
After both new nodes were in, I restored the SSL settings and enabled rancher again.
Are you saying had  I only used IP (insetad of domain name) for the VIP it of probably just worked ? :)
(i spent allot of time because I noticed in the agent-install.sh which is fetched from harvester contained CATTLE_SERVER=https://$IP:443 even tho I had used a domain name in the install. Checking an older v1.1.2 node I installed, it's existing 'install.sh' that was still present on the machine contained a domain name -- not an IP. Which is why I focused trying to understand why that was different-- until finally going nuclear and removing all the SSL and addons + setting IP within the config file.
I can try to reproduce this with a final node I have now if there is any interest, or if i'm just an idiot happy to close this :)
Thanks for reply!

cURL error 60 usually means that there is a problem with the configuration of the SSL on the server you are trying to connect to. One issue that might be occurring is that the host name used in the request does not match any host name shown on the SSL.

I have some findings:
The following two lines of environment variable definition are generated by Rancher in runtime when Rancherd requests for the install.sh script:
CATTLE_AGENT_BINARY_BASE_URL="https://<IP_ADDRESS>/assets"
CATTLE_SERVER=https://<IP_ADDRESS>

The relevant code snippet is at:
https://github.com/rancher/rancher/blob/b39ba33fda7609291fb5d29ffeddff2798169d5f/pkg/capr/installer/installer.go#L58-L103
The VIP <IP_ADDRESS> was got from the Setting CR named server-url of the embedded Rancher in Harvester.
As a workaround, would you please try to modify the server-url Setting CR in the embedded Rancher with your FQDN instead of the IP address? In that case, the request host name matches the name from the certificate.
And I'm still curious about this:

Checking an older v1.1.2 node I installed, it's existing 'install.sh' that was still present on the machine contained a domain name -- not an IP.
Could you paste the same part (head -n 10) here?

Hi,
Here is the install.sh from a node which was installed with v1.1.2 to a v1.1.2 cluster. Also showed first lines of the 90_custom.yaml
hv02:/home/rancher # head -n10 /var/lib/rancher/rancherd/install.sh
#!/usr/bin/env sh

CATTLE_AGENT_BINARY_BASE_URL="https://harvester.slack.house:443/assets"

CATTLE_CA_CHECKSUM="798db3fbaa5fe211159fa1cddb8e1e2cdbca12e3d355a491e6bf8134d2f14272"

#!/bin/sh

if [ "${DEBUG}" = 1 ]; then
hv02:/home/rancher # head -n20  /oem/90_custom.yaml
name: Harvester Configuration
stages:
  initramfs:
  - commands:
    - sed -i 's/^NETCONFIG_DNS_STATIC_SERVERS.*/NETCONFIG_DNS_STATIC_SERVERS="<IP_ADDRESS>"/'
      /etc/sysconfig/network/config
    - modprobe kvm
    - modprobe vhost_net
    files:
    - path: /etc/rancher/rancherd/config.yaml
      permissions: 384
      owner: 0
      group: 0
      content: |
        server: https://harvester.slack.house:443
        role: agent
        token: "somethingverysecureipromise"
        kubernetesVersion: v1.24.11+rke2r1
        rancherVersion: v2.6.11
        rancherInstallerImage: rancher/system-agent-installer-rancher:v2.6.11
hv02:/home/rancher #

I am prettty sure there was a change in behavior of what is being added to the install.sh by rancher runtime.

I followed the following steps to replicate the issue:

create a private CA and sign a cert for harvester.$VIP.sslip.io
update existing harvester v1.2.0 node to use the new certificate.
join a new node v1.2.0 to existing cluster. I used the harvester.$VIP.sslip.io address for VIP.
rancherd tries to use the server-url from embedded rancher to join the node, which is currently pointing to the VIP, and results in the SAN error.

Sep 18 04:52:58 upgrade-path-2 rancherd[2569]: time="2023-09-18T04:52:58Z" level=info msg="[stderr]: establish a secure connection to it. To learn more about this situation and"
Sep 18 04:52:58 upgrade-path-2 rancherd[2569]: time="2023-09-18T04:52:58Z" level=info msg="[stderr]: how to fix it, please visit the web page mentioned above."
Sep 18 04:52:58 upgrade-path-2 rancherd[2569]: time="2023-09-18T04:52:58Z" level=info msg="[stderr]: [ERROR]  000 received while testing Rancher connection. Sleeping for 5 seconds and trying again"
Sep 18 04:53:03 upgrade-path-2 rancherd[2569]: time="2023-09-18T04:53:03Z" level=info msg="[stderr]: curl: (60) SSL: no alternative certificate subject name matches target host name '<IP_ADDRESS>'"
Sep 18 04:53:03 upgrade-path-2 rancherd[2569]: time="2023-09-18T04:53:03Z" level=info msg="[stderr]: More details here: https://curl.se/docs/sslcerts.html"
Sep 18 04:53:03 upgrade-path-2 rancherd[2569]: time="2023-09-18T04:53:03Z" level=info msg="[stderr]: "

To work around this, update the setting for server-url, as follows kubectl edit setting.managenent server-url to the DNS record associated with the new certificate.
Restart rancherd on the joining node systemctl restart rancherd
Now rancher will use the join script with correct DNS record, for which a certificate already exists on ingress.

upgrade-path-2:~ # journalctl -fu rancherd
Sep 18 04:55:07 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:07Z" level=info msg="[stdout]: [INFO]  Starting/restarting rancher-system-agent.service"
Sep 18 04:55:08 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:08Z" level=info msg="No image provided, creating empty working directory /var/lib/rancher/rancherd/plan/work/20230918-045504-applied.plan/_1"
Sep 18 04:55:08 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:08Z" level=info msg="Running command: /usr/bin/rancherd [probe]"
Sep 18 04:55:08 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:08Z" level=info msg="[stderr]: time=\"2023-09-18T04:55:08Z\" level=info msg=\"Running probes defined in /var/lib/rancher/rancherd/plan/plan.json\""
Sep 18 04:55:09 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:09Z" level=info msg="[stderr]: time=\"2023-09-18T04:55:09Z\" level=info msg=\"Probe [kubelet] is unhealthy\""
Sep 18 04:55:15 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:15Z" level=info msg="[stderr]: time=\"2023-09-18T04:55:15Z\" level=info msg=\"Probe [kubelet] is healthy\""
Sep 18 04:55:15 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:15Z" level=info msg="[stderr]: time=\"2023-09-18T04:55:15Z\" level=info msg=\"All probes are healthy\""
Sep 18 04:55:15 upgrade-path-2 rancherd[3163]: time="2023-09-18T04:55:15Z" level=info msg="Successfully Bootstrapped Rancher (v2.7.5/v1.25.9+rke2r1)"
Sep 18 04:55:15 upgrade-path-2 systemd[1]: rancherd.service: Deactivated successfully.
Sep 18 04:55:15 upgrade-path-2 systemd[1]: Finished Rancher Bootstrap.

@bk201 we should may be explore having a harvester dns record setting which syncs to the embedded rancher server-url. This should avoid having to disable/enable custom ssl certs when adding new nodes.

@bk201 @ibrokethecloud @starbops
this issue seems also existing in v1.1.2, please double confirm, when true, we need to update Harvester doc asap, thanks.

Yes, it happens to v1.1.2 too. We've also documented the workaround in the harvester/docs#462, which is pretty much the same as v1.2.0.

@starbops For this milestone, we'll add a doc for the issue. Please help create a new enhancement issue to support adding the certs in the installation stage. Thanks.

I had this problem in version 1.1.2.
I installed my cluster with 3 nodes, 2 using join mode, I put a certificate with a valid CA in my VIP's DNS, and everything worked.
When I went to insert a 4 node into the cluster, I had this same problem.
The solution was to enter the url of my cluster through kubectl edit setting.management server-url, where the server-url parameter returned empty.
After restarting the rancherd service, the node joined the cluster perfectly.
I try to install 2 clusters in production with more than 3 nodes each, with the same bootstrap procedure and certificate in the VIP domain, for me this error is very strange.

@mauriiciodias Thanks for reporting the issue. What value did you fill in the management address during the 4th node installation? Was it the VIP address or the DNS name? Thanks!

@starbops VIP address. I change from DNS and had sucess.
I install other new cluster, but 1.2x version and this problem also repeat.
This workround this top of mind! 😉

Test Plan
Basically, the following steps are distilled from harvester/docs#462. The issue happens on v1.1.2, too. So, ideally, we'd like to verify the documented workarounds work on v1.2 and v1.1 branches respectively.

Prepare a v1.2.0/v1.1.2 Harvester cluster (single node is good enough)
Prepare a set of private key, public certificate, and CA, and set up the ssl-certificates setting with them (for the commands to generate those files, you can refer to the step 2)
Join a new node with the same version, finish the installation
After the node rebooted, get access to the node via console or SSH and run the following command to add the CA to the node

# prepare the CA as additional-ca.pem on the nodes
$ sudo cp additional-ca.pem /etc/pki/trust/anchors/
$ sudo update-ca-certificates

The node should join the cluster successfully

After syncing with @starbops - the work-around looks great with v1.2.0.
One thing noticed is that rancherd.service did succeed in coming up on join, and the node was able to join the cluster as an additional host:
Oct 25 03:22:02 thinkcentre-test systemd[1]: Starting Rancher Bootstrap...
Oct 25 03:22:02 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:02Z" level=info msg="Loading config file [/usr/share/rancher/rancherd/config.yaml.d/50-defaults.yaml]"
Oct 25 03:22:02 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:02Z" level=info msg="Loading config file [/usr/share/rancher/rancherd/config.yaml.d/91-harvester-bootstrap-repo.yaml]"
Oct 25 03:22:02 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:02Z" level=info msg="Loading config file [/etc/rancher/rancherd/config.yaml]"
Oct 25 03:22:02 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:02Z" level=info msg="Bootstrapping Rancher (v2.7.5/v1.25.9+rke2r1)"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="Writing plan file to /var/lib/rancher/rancherd/plan/plan.json"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="Applying plan with checksum "
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="No image provided, creating empty working directory /var/lib/rancher/rancherd/plan/work/20231025-032203-applied.plan/_0"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="Running command: /usr/bin/env [sh /var/lib/rancher/rancherd/install.sh]"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Using default agent configuration directory /etc/rancher/agent"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Using default agent var directory /var/lib/rancher/agent"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stderr]: [WARN]  /usr/local is read-only or a mount point; installing to /opt/rancher-system-agent"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Determined CA is necessary to connect to Rancher"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Successfully downloaded CA certificate"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Value from https://<IP_ADDRESS>/cacerts is an x509 certificate"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Successfully tested Rancher connection"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Downloading rancher-system-agent binary from https://<IP_ADDRESS>/assets/rancher-system-agent-amd64"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Successfully downloaded the rancher-system-agent binary."
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Downloading rancher-system-agent-uninstall.sh script from https://<IP_ADDRESS>/assets/system-agent-uninstall.sh"
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script."
Oct 25 03:22:03 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:03Z" level=info msg="[stdout]: [INFO]  Generating Cattle ID"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stdout]: [INFO]  Successfully downloaded Rancher connection information"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stdout]: [INFO]  systemd: Creating service file"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stdout]: [INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stdout]: [INFO]  Enabling rancher-system-agent.service"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.s>
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stdout]: [INFO]  Starting/restarting rancher-system-agent.service"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="No image provided, creating empty working directory /var/lib/rancher/rancherd/plan/work/20231025-032203-applied.plan/_1"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="Running command: /usr/bin/rancherd [probe]"
Oct 25 03:22:04 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:04Z" level=info msg="[stderr]: time=\"2023-10-25T03:22:04Z\" level=info msg=\"Running probes defined in /var/lib/rancher/rancherd/plan/plan.json\""
Oct 25 03:22:06 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:06Z" level=info msg="[stderr]: time=\"2023-10-25T03:22:06Z\" level=info msg=\"Probe [kubelet] is unhealthy\""
Oct 25 03:22:10 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:10Z" level=info msg="[stderr]: time=\"2023-10-25T03:22:10Z\" level=info msg=\"Probe [kubelet] is healthy\""
Oct 25 03:22:10 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:10Z" level=info msg="[stderr]: time=\"2023-10-25T03:22:10Z\" level=info msg=\"All probes are healthy\""
Oct 25 03:22:10 thinkcentre-test rancherd[2022]: time="2023-10-25T03:22:10Z" level=info msg="Successfully Bootstrapped Rancher (v2.7.5/v1.25.9+rke2r1)"
Oct 25 03:22:10 thinkcentre-test systemd[1]: rancherd.service: Deactivated successfully.
Oct 25 03:22:10 thinkcentre-test systemd[1]: Finished Rancher Bootstrap.

But still rancher-system-agent.service did see the similar behavior where the workaround would need to be implemented to succeed:
Oct 25 03:32:51 thinkcentre-test systemd[1]: rancher-system-agent.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 03:32:51 thinkcentre-test systemd[1]: rancher-system-agent.service: Failed with result 'exit-code'.
Oct 25 03:32:56 thinkcentre-test systemd[1]: rancher-system-agent.service: Scheduled restart job, restart counter is at 116.
Oct 25 03:32:56 thinkcentre-test systemd[1]: Stopped Rancher System Agent.
Oct 25 03:32:56 thinkcentre-test systemd[1]: Started Rancher System Agent.
Oct 25 03:32:56 thinkcentre-test rancher-system-agent[21595]: time="2023-10-25T03:32:56Z" level=info msg="Rancher System Agent version v0.3.3 (9e827a5) is starting"
Oct 25 03:32:56 thinkcentre-test rancher-system-agent[21595]: time="2023-10-25T03:32:56Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Oct 25 03:32:56 thinkcentre-test rancher-system-agent[21595]: time="2023-10-25T03:32:56Z" level=info msg="Starting remote watch of plans"
Oct 25 03:32:56 thinkcentre-test rancher-system-agent[21595]: time="2023-10-25T03:32:56Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://<IP_ADDRESS>/version\": x509: certificate signed by unknown authority, removing CA data and trying again"
Oct 25 03:32:56 thinkcentre-test rancher-system-agent[21595]: time="2023-10-25T03:32:56Z" level=fatal msg="error while connecting to Kubernetes cluster with nullified CA data: Get \"https://<IP_ADDRESS>/version\": x509: certificate signed by unknown authority"
Oct 25 03:32:56 thinkcentre-test systemd[1]: rancher-system-agent.service: Main process exited, code=exited, status=1/FAILURE
Oct 25 03:32:56 thinkcentre-test systemd[1]: rancher-system-agent.service: Failed with result 'exit-code'.
Oct 25 03:33:01 thinkcentre-test systemd[1]: rancher-system-agent.service: Scheduled restart job, restart counter is at 117.
Oct 25 03:33:01 thinkcentre-test systemd[1]: Stopped Rancher System Agent.
Oct 25 03:33:01 thinkcentre-test systemd[1]: Started Rancher System Agent.
Oct 25 03:33:01 thinkcentre-test rancher-system-agent[21674]: time="2023-10-25T03:33:01Z" level=info msg="Rancher System Agent version v0.3.3 (9e827a5) is starting"
Oct 25 03:33:01 thinkcentre-test rancher-system-agent[21674]: time="2023-10-25T03:33:01Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Oct 25 03:33:01 thinkcentre-test rancher-system-agent[21674]: time="2023-10-25T03:33:01Z" level=info msg="Starting remote watch of plans"
Oct 25 03:33:01 thinkcentre-test rancher-system-agent[21674]: time="2023-10-25T03:33:01Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://<IP_ADDRESS>/version\": x509: certificate signed by unknown authority, removing CA data and trying again"
Oct 25 03:33:01 thinkcentre-test rancher-system-agent[21674]: time="2023-10-25T03:33:01Z" level=fatal msg="error while connecting to Kubernetes cluster with nullified CA data: Get \"https://<IP_ADDRESS>/version\": x509: certificate signed by unknown authority"

Attaching the logs from both systemd services.
Still need to verify with v1.1.2 - so will be doing that now :smile: :+1:
rancher-system-agent.service.log
rancherd.service.log

Testing with v1.1.2, this looks good as well :smile:  - I'll go ahead and close this out :smile: :+1:  - thanks @starbops !


AI responds multiple times But only the last reply will be displayed and recorded.

Describe the bug
When AI responds multiple times at once,All replies except for the first reply's first character will disappear, and only the last reply will be displayed and recorded.
This is what I am looking for.But only for a moment

And then it became like this, and all the previous content disappeared.The cache file is the same.

To Reproduce
Steps to reproduce the behavior:
1.Set some special system messages
2.Let AI reply multiple times at once
Expected behavior
All AI replies are displayed and stored
Screenshots

Node.js version (please complete the following information):

OS:Windows 11
Version 19

Package version (please complete the following information):

1.35.0

Additional context
Allowing AI to respond multiple times at once with no more than 30 words can effectively bypass censorship.Maybe you can add this feature.

I tried to solve it myself.Thank you for your project.
Here is the result.

And here is my modification.

Perhaps this will be useful for your project.
Anyway, thanks!

Feel free to submit a pull request, but otherwise I'll keep this issue open until it's fixed.

I am not a professional, and there are still many bugs in the code I modified.So let's hope someone can fix it.

A PR would still be valuable.

Yes, PR is still valuable, but the main bug is not this. I tried to fix some on my own, but I am a novice on GitHub, so I am not very good at using it like pull request.I can only paste the code I have modified, sorry.
Here.


No viable head due to optimistic sync

This test covers scenario outlined in https://github.com/ethereum/consensus-specs/pull/2955#issuecomment-1198282331
The outlined scenario checks only that CL stays optimistic if there are no viable tips.
Ideally, we should also check that CL may recover from the optimistic state without viable tips by importing a viable chain. But creating this scenario seems non-trivial as there should be CL validator that produces a malicious fork of a few blocks length. It can be implemented as follows:

Builder produces blocks up to slot 0 block into epoch N+1
Importer turns optimistic on 20th block and invalidates all blocks starting from 21st by receiving corresponding LVH value on newPayload in 0th slot of epoch N+1
Builder realizes that blocks from 21st of epoch N till 0th of epoch N+1 are invalid and continues to build a chain from the 1st slot of epoch N+1 with parent block from 20th slot of epoch N and eventually justifies N+1. Importer gets VALID status on each of these new blocks

"Builder realizes that blocks from 21st of epoch N till 0th of epoch N+1 are invalid" is something that I am not sure could be implemented with existing tools cc @marioevz
cc @potuz @paulhauner @ajsutton @tersec

In the scenario described in 2 Prysm (and I assume all clients) will not be affected because they will not import block 0 in N+1. You need to either return INVALID from FCU or from block 1 in N+1.
The unit tests we have for this scenario cover the unibranch cases (as in this proposal) for errors coming from NewPayload and FCU and two kinds of LVH: one such that importing a new block will still result in invalid heads (as in this description, if importing a block in N+1 based on 20 does not justify immediately N) and it may take up to Epoch N+3 to actually recover. Other situation is if the LVH was slot 30, in which case importing slot 2 in N+1 based on 30 may immediately justify N and become valid.

I have updated the test case. @potuz could you please check if it does make sense to you?

I have updated the test case. @potuz could you please check if it does make sense to you?

This test looks good to me. Just notice that at least for prysm the test for the case when 32 returns SYNCING and 33 returns INVALID from NewPayload would test a different path with the same expected behavior


WIP: Atomic multi-tx

This is the basic primitive needed to implement atomic schema migration: the ability to establish a transaction and run queries and transact within it, rolling back or committing in its entirety when done.
This is a long way from being ready — I need to introduce some kind of equivalent of Conn that I can query within this, and perhaps maintains mutable state rather than invoking transact directly — but I wanted to get to a POC first.

@ncalexan, @fluffyemily: could you take a look at this? We'll need this at some point for syncing, I expect.

And you might find this interesting, @grigoryk.

Everything done apart from the Vec of reports, which we can do later if we need it. @ncalexan ?

@ncalexan to explain your transaction question:

IMMEDIATE means 'start the transaction now, but don't exclude readers'. It prevents other connections from taking immediate or exclusive transactions. This is appropriate for our writes and InProgress: it means we are ready to write whenever we want to, and nobody else can start a transaction that's not DEFERRED, but we don't need exclusivity yet.
EXCLUSIVE means just that. This is appropriate when bootstrapping: if anything will go wrong with preparing to write, it will happen right here, and we don't expect other writers or readers yet.

There is an argument for always using EXCLUSIVE, but I think it's a little cheaper for read-only transactions and some amount of read-concurrency to stick with IMMEDIATE.

@ncalexan to explain your transaction question:
IMMEDIATE means 'start the transaction now, but don't exclude readers'. It prevents other connections from taking immediate or exclusive transactions. This is appropriate for our writes and InProgress: it means we are ready to write whenever we want to, and nobody else can start a transaction that's not DEFERRED, but we don't need exclusivity yet.
EXCLUSIVE means just that. This is appropriate when bootstrapping: if anything will go wrong with preparing to write, it will happen right here, and we don't expect other writers or readers yet.

There is an argument for always using EXCLUSIVE, but I think it's a little cheaper for read-only transactions and some amount of read-concurrency to stick with IMMEDIATE.

OK, this is helpful.  Fold this into the code in some way, please.

Everything done apart from the Vec of reports, which we can do later if we need it. @ncalexan ?

I see this landed, which is fine.  But I still think that there's a weirdness with this API.  How do you extract all of the IDs allocated during a sequence of InProgress transactions?  Right now, you can't (and I mean you really can't, even in consumer code, since you can't even know the first transaction ID allocated by an InProgress transaction).  I think that's a bad design.

How do you extract all of the IDs allocated during a sequence of InProgress transactions?

There are several responses to this.

You don't, just as in SQLite you can't do

BEGIN TRANSACTION;
INSERT INTO …;
UPDATE …;
INSERT INTO …;
END TRANSACTION;

and get a single count of inserted and updated rows, or the lastUpdateRowid. The same applies here: within the in-progress transaction, each time you act you find out what happened, but you don't get some kind of aggregate report at the end. It would be fair for InProgress to return nothing at all, but we return the last thing for convenience, because it includes the final-state schema etc.

The output includes a final parts table, so you know what happened — look at the previous parts table vs the output if you want to see how many identifiers were added.

Perhaps you'd like some kind of InProgressResult type that aggregates some data from each write?

Calling code should be looking at the result of each internal-transact that they invoke. The results (in terms of ID allocations and tx IDs) are stable.

It doesn't make sense to aggregate tempID mappings, because they can overlap.

In short, though: InProgress is exactly like just doing transact and q_once on the raw connection, except that your operations all succeed or fail or rollback together.
Does that change anything for you, @ncalexan?

Added a comment in b7fb44a.


Problem on NB 13 and Windows 11.

Every try to New from Json.. on Windows 11 returns Exception
java.util.regex.PatternSyntaxException: Unexpected internal error near index 1
..
at io.github.ranSprd.pojomaker.actions.NewFromJsonSourceFolderAction.extractPackageName(NewFromJsonSourceFolderAction.java:87)
...
Temporary I changed the code in extractPackageName method to:
String os = System.getProperty("os.name");
if(os.startsWith("Windows 11")) return Optional.of( relativePath.replaceAll("/", "."));
then Plugin is working perfect.

I'm wondering because I assumed that the separator on windows systems is '' and not '/'. Is there any special with your configuration? Are you using WSL1/2?
Please don't get me wrong. I want to avoid other Windows 11 users having problems afterwards

I checked the problem on the three different desktops with Windows 11. I have always the same error with regex. I'm not using any WSL but only one specific settings is a polish os language.


module 'keras.backend' has no attribute 'control_flow_ops'

I get this problem when run code on google colab. Please show me how to fix it? Thanks!

The code  is old. It would not work with Keras 2.2.5. You can switch to
Keras 2.2.4 or lower, or change K.control_flow_ops.while_loop to
tf.while_loop. I havent tried this, though.
On Sat, Aug 31, 2019, 5:05 AM smalldroid<EMAIL_ADDRESS>wrote:

I get this problem when run code on google colab. Please show me how to
fix it? Thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/qqwweee/keras-yolo3/issues/544?email_source=notifications&email_token=AFSNTUOYUPH6ZRL2TC27GJDQHIX6PA5CNFSM4ISTCQM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HISOZOA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFSNTUPXSQM6N5WV2R6UK5LQHIX6PANCNFSM4ISTCQMQ
.

Thanks for you answer! I tried to change to tf.while_loop but still not get luck.

tf.while_loop working here, using tensorflow 1.14.0

I also met this problem on google colab. Here's a solution:
add
from .load_backend import control_flow_ops
from .load_backend import set_image_dim_ordering
to /usr/local/lib/python3.6/dist-packages/keras/backend/__init__.py
code:
os.chdir('/usr/local/lib/python3.6/dist-packages/keras/backend')
!cat __init__.py
then copy and overwrite it:
%%writefile __init__.py

Run the following Bash script:
echo "from .load_backend import control_flow_ops
from .load_backend import set_image_dim_ordering
$(cat /usr/local/lib/python3.6/dist-packages/keras/backend/__init__.py)" > /usr/local/lib/python3.6/dist-packages/keras/backend/__init__.py

Following Jarvis Peng to solve this problem. Thanks all!

I also get this problem on google colab.how to overwrite file on google colab.It seems that it doesn't really change the init.py .Because when I see this file again,it remains the same.And I try to change the version of tensorflow and keras,but I don't know why it remain prints the original version.When I using pip list,it show the version I try to install

I even changed  the __init.py. But the error still persist.
Any suggestion????

I think that control_flow_ops belongs to tensorflow, thus I added following to init.py in backend folder in keras folder. Then it worked.
elif backend() == 'tensorflow':
from tensorflow.python.ops import control_flow_ops
Im not sure about set_image_dim_ordering, though.

I think that control_flow_ops belongs to tensorflow, thus I added following to init.py in backend folder in keras folder. Then it worked.
elif backend() == 'tensorflow':
from tensorflow.python.ops import control_flow_ops
Im not sure about set_image_dim_ordering, though.

thx, it works on tf==1.15.2 keras==2.3.0.

add
from tensorflow.python.ops import control_flow_ops
to model.py
code:
control_flow_ops.while_loop(lambda b,*args: b<m, loop_body, [0, ignore_mask])

i used this
%tensorflow_version 2.x !pip uninstall -y tensorflow !pip install tensorflow-gpu==1.14.0
and it worked

The code is old. It would not work with Keras 2.2.5. You can switch to Keras 2.2.4 or lower, or change K.control_flow_ops.while_loop to tf.while_loop. I havent tried this, though.
…
On Sat, Aug 31, 2019, 5:05 AM smalldroid @.***> wrote: I get this problem when run code on google colab. Please show me how to fix it? Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#544?email_source=notifications&email_token=AFSNTUOYUPH6ZRL2TC27GJDQHIX6PA5CNFSM4ISTCQM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HISOZOA>, or mute the thread https://github.com/notifications/unsubscribe-auth/AFSNTUPXSQM6N5WV2R6UK5LQHIX6PANCNFSM4ISTCQMQ .

it worked perfectly.


switch to alamo-common and use base settings

Coverage decreased (-0.5%) to 95.652% when pulling 4eeb87db49356ae916b7bb64dd7582a4771ab8bf on feature/use-alamo-common into 33298bcc9306f73141a4181194e05a9d0a1bb6b3 on develop.

⚡


Scrape United States Supreme Court (SCOTUS)

New electronic docketing system was introduced at the United States Supreme Court this week. Unfortunately it didn't seem to change much of their existing public-facing system, but we should scrape the system for filings and index them.
Some structural notes:

There is no way to get a list of recent filings across all cases, such as a global RSS feed.

Each case has its own RSS fee, e.g. https://www.supremecourt.gov/rss/cases/17-74.xml. For some reason the timestamp seems to bump daily, even though there is no visible docket activity. Also, the timestamp is marked as EDT but really is in EST.

Each case has a JSON and XML report:
http://www.supremecourt.gov/RSS/Cases/JSON/17-74.json
http://www.supremecourt.gov/RSS/Cases/XML/17-74.xml

Although the above don't have proper timestamps for linke PDFs, the name of the link contains a timestamp in "YYYYMMDDhhmmss{milliseconds}" format.
e.g.: http://www.supremecourt.gov/DocketPDF/17/17-74/19830/20171113205323142_17-71 17-74 Weyerhaeuser Co. Markle Interests LLC.pdf

@johnhawkinson I assume this is stale and no longer and issue.

I assume this is stale and no longer an[] issue.

Err…why would you assume that?
I think everything in the original PR remains true, and with appellate RECAP on the horizon, setting sights on the Supreme Court seems more achievable.

This is about filings too, @flooie.

I have a working implementation that downloads the SCOTUS docket JSON feeds and stores them in Sqlite. It uses a subset of Juriscraper's dependencies. Happy to work on adapting this if you'd like a PR, but I will need design guidance on where to fit this into the package structure and workflows.

Ooooh, that's fun @ralexx. I think we'd want it to have a directory next to the pacer directory.
With that in mind, do you want to suggest a class hierarchy, and perhaps @flooie or @grossir can help guide your work so it fits with the rest of our style/approach/architecture?

Sounds good, @mlissner.
Is this issue the best place to ask further questions?

Yep, it's a great place to discuss things.

I think we'd want it to have a directory next to the pacer directory.

I will work in juriscraper/juriscraper/scotus_docket and juriscraper/tests/examples/scotus_docket. You've used underscore naming for the oral_args/ directory so I'm following that, but I also noticed that pacerdocket.py doesn't use an underscore so happy to fit whichever style.
My first design question is about how you (@flooie and @grossir) would prefer to handle obtaining the Supreme Court docket numbers with which to scrape dockets. The only two sources of truth I've found are PDF documents:

The Journal of the Supreme Court, which appears to be a comprehensive listing of cases regardless of disposition.
The Supreme Court "Granted & Noted List", which looks like a subset of cases where certiorari has been granted.

The PDFs are pretty clean and I've had reasonable success extracting docket numbers and even some unique metadata from the Granted & Noted List using pypdf and regex. However, I'm not sure if you want to add pypdf as a package dependency. And the more prose-like format of the Journal may be a bit more tricky to parse; I haven't attempted this as yet.
Instead I resorted to brute force and am just trying docket numbers sequentially (I am rate-limiting my requests to 1/sec). It's hacky but it works. The challenge from a resource-usage point of view is that there are discontinuities in the docket numbering, e.g. YY-2000 to YY-4999 seem largely unused but from YY-5000 to YY-12000 about 10% of docket numbers are used. I tried using search algorithms to reduce the search space of docket numbers but it wasn't worth the effort.

Are you aware of any other comprehensive list of Supreme Court dockets?
Absent that, do you prefer the PDF parsing approach or the brute force approach to identifying active dockets?
If you prefer brute force, an easy way to shrink the docket number search space over time is excluding previously identified docket numbers. That would require persistent data storage that doesn't appear to be part of Juriscraper. Is this an approach you would like to investigate?

I have been looking into this and I need some clarification on the scope and integration into CL of this source @mlissner @flooie
What DB tables are we going to fill with this data?
It seems to me that this source contains mainly Docket and DocketEntry objects, and some related objects such as OriginatingCourtInformation, Party and some PDF Documents that belong to the DocketEntry object. About the documents, an example highlighted in blue from here:

However, I think we do not create DocketEntry from any source except RECAP. Should we change this here?
How are users interacting with this data?
I am guessing this is for setting up Docket Alerts, as in RECAP?  If that's the case, the scraper should have 2 starting points:

"discovery" of previously unknown dockets (which ties into @ralexx analysis)
re-scraping known dockets (to fulfill alerts)

About the re-scraping, the RSS endpoint data is not static. If you check the example from the original comment, it has docket entries which are more recent than the comment itself.

Anyhow, I think we would need we will need to create a new caller on courtlistener to call the new juriscraper scraper and ingest this data, in the folder courtlistener/cl/scrapers/management/commands/, something like cl_scrape_dockets.py

hi @ralexx thanks for looking into this.
About folder structure

I will work in juriscraper/juriscraper/scotus_docket

I think it should be juriscraper/dockets/united_states/federal_appellate/scotus.py since we may want to support other docket scrapers in the future.
Also, this structure mirrors the one on juriscraper/opinions and juriscraper/oral_args

I will work in  ... juriscraper/tests/examples/scotus_docket

similarly, I think it would be better to mirror examples/opinions/united_states and examples/oral_args/united_states and use examples/dockets/united_states/
About getting the docket list...
I don't think we should brute force the search, we try to be as gentle as possible with the server, since we use a user agent that identifies us (except for a couple of sources).
I am going to limit myself to checking the case of "discovery" of new dockets, as mentioned in the previous comment.
... from Docket Search page
There is a Docket Search page. The search string is used in a full text search, so if you query for February 2024, it returns dockets that contain such string. For example:
"SET FOR ARGUMENT on Wednesday, February 21, 2024",
"February 14, 2024 United States Court of Appeals for the Eighth Circuit Feb 01 2024 Motion to direct the Clerk"

The search supports exact results operator "February 21, 2024", which returns fewer results. However, the search is limited to the last 5 years, and to 500 results, and has a page size of 5, which could make us do several requests.
... from PDFs
About using the PDFs as source of docket numbers, I think it is a valid idea. About adding the pypdf dependency I don't know if @flooie is against it design wise, since we use doctor to extract text from PDFs. Maybe we could send requests to doctor from juriscraper?
About the Journals, they seem to be published / updated once each year, and the end of the term, so we wouldn't get fresh data from them (which is important if we are implementing this for alerting). They seem great for backscraping old cases.
The Granted and Noted List documents are updated more frequently, and we could indeed get the list from there. However, the "most recent" document October Term 2024 has an updated dated older than the one on the second document October Term 2023
(January 22, 2024 vs February 8, 2024), so we would have to check both

I think that if one of the use cases of this data is to alert users (previous comment), we should use the HTML Docket Search to collect the docket numbers, since by using a date-like query we will get recent and active cases, and do not limit ourselves only to the "Granted and Noted" list.  What do you think @flooie?

About using the PDFs as source of docket numbers, I think it is a valid idea. About adding the pypdf dependency I don't know if @flooie is against it design wise, since we use doctor to extract text from PDFs. Maybe we could send requests to doctor from juriscraper?

I will work with whatever PDF extractor you prefer; I’m not familiar with doctor so I simply reached for something I have used.

I assume we should build this to begin adding dockets directly into CL @mlissner?
@ralexx - doctor  is our open source micro service tool we use to process documents for Courtlistener.

Using full text search
It sounds as if belt-and-suspenders search for docket numbers is the way to go. @grossir you make a good point about the full-text search feature: I didn’t play with it much because it doesn’t expose the underlying database API, so one more layer of abstraction.
The full-text search definitely returns spurious matches for our purpose. We can reject them but we may still have to request the dockets and parse them to validate the docket entries. For example, on result from searching on “Feb|February 16, 2024” is

“Motion to extend the time to file a response is granted and the time is further extended to and including February 16, 2024, for all respondents.”
https://www.supremecourt.gov/search.aspx?filename=/docket/docketfiles/html/public/23-402.html was updated on 2024-02-02 but the search is matching the entry’s text.
The SCOTUS server supports the If-Modified-Since HTML header so we may be able to avoid the parsing step.
I’m on vacation this week, when I get back I will test full text search against a sample of dockets in the Journal. If that search returns a superset of the dockets with new entries on a given date then I will work on filtering out spurious matches before I report back.

@ralexx @grossir
I think the Orders Content is a superset of all docket numbers.  I checked this evening and the order list - and misc. orders which come out a regular frequency should be enough to both identify all docket numbers and JSON endpoints.
I wrote a simple script this evening and pulled out 3,199 docket JSON endpoints for the 2023 Term.
SCOTUS Orders

@flooie, take for example 23A745, Trump v. US (the immunity appeal from DC circuit). There hasn’t been any order on the case so I don’t think it appears on the Orders: there was an amicus brief filed on 2024-02-20 that I did not find in the Orders PDF for the same date.
Particularly in higher profile cases there are amicus briefs on the docket that may be of general interest; these tend to appear prior to an order (one exception appears to be an order denying amicus curiae leave to file). Relying on orders for docket numbers could leave a substantial lag before such dockets were scraped. I can test this next week.

Got it.

Sorry to be a bit slow chiming in. I've been utterly swamped with meetings this week. A few things come to my mind as the boss man 'round these parts:
Goals
It sounds like we're on a path to scrape all SCOTUS content. That would be phenomenal, and it's something we should have been doing since the day they stood up their open docket website.
If we are to do this, speed and completeness will be essential, because SCOTUS content constantly goes viral among journalists. It's embarrassing for something with a gazillion re-posts not to be in our system at all.
Architecture

Do we want to use our current tables in CL?
I'm always on the fence about this, and it's a huge question: If we're going to add filings to CL from dozens or hundreds of courts, do we do it in one schema or in dozens? When we did Los Angeles Superior Court, it got its own schema and its own django app, and that seemed good. I think it's generally the right way to go.

How aggressively do we scrape?
Normally we try to be gentle on court server, but this is SCOTUS and from what I've gathered they're the one court in the land that actually has some sort of scalable architecture. We should still be smart and kind, but I think we can crawl aggressively if needed to hit completeness and speed goals. I would advocate for the least aggressive approach that is guaranteed to work within a few minutes of content being posted.

Do we store state in Juriscraper?
No. Juriscraper is supposed to be a library that other systems can build on top of. It is usually a wrapper for downloading and parsing particular pages on court web pages. For example you tell it:

search for this query
download this docket
grab that PDF

And it provides you with JSON, response codes, and PDF binaries. If you need to do something like crawl the docket number space (except for certain numbers you already know about), the storage for that should be up a level in CourtListener or whatever other system is calling Juriscraper.

What about folder structure in Juriscraper?
I'm sorry to ask for another tweak here, but I would suggest one tweak with what Gianfranco said above. He suggested juriscraper/dockets/united_states/federal_appellate/scotus.py. I'd suggest going one step further with something like: juriscraper/dockets/united_states/federal_appellate/scotus/dockets.py. That will give you a module to add files for lots of different SCOTUS-related tools.

What about crawling/parsing PDFs?
This is usually my last resort. I'd rather download 100 URLs around the clock in a brute-force manner than rely on some slow, fragile, and unreliable PDF to do the job. But I haven't looked at the possibilities here. If it's the only way, so be it. I leave choice of library up to y'all, but as Bill knows, I usually prefer PyMuPDF. I think the API is a bit worse for scraping in particular, but the speed is (much) better and it's well maintained.

Putting this all together...
What all this means when you put it together is that we'll need the following components:

Juriscraper to scrape content and make it digestible
A daemon of some kind to call Juriscraper and scrape content around the clock.
Some database models, probably new ones, to store what we scrape.
HTML and views to display the content.
Integration into our APIs, bulk data, and search engines (somehow!).

I think that's it for my comments. Sorry this is a lot, but adding a new court is a fairly big task. I think it's worth doing though and I'm excited to have some momentum here.

How do you (all) feel about the SCOTUS scraper having access to persistent state, whether an underlying docket database or its own database?
As far as I have seen, this doesn't happen elsewhere in Juriscraper: scraped data is presented to the caller for it to handle. I presume this statelessness is a design feature, but I'd like to clarify that.
Based on what I've found so far, there appear to be efficiency gains in downloading dockets when it's possible to discriminate between docket numbers that are known good and the rest. I will elaborate once I have some data to show you, but if you want SCOTUS scrapers to be stateless as a design principle then I won't work on/include that part of my prototype code.

Yeah, I think it's best to keep Juriscraper stateless, but we'll certainly need that analysis when it comes to the calling code, so it can do that.

I've looked into the different sources and types of information available at <supremecourt.gov> to guide my planning. Please let me know if you see that I'm missing something.
Published docket information
These are parts of supremecourt.gov most likely to be useful here. There are some additional sources such as the Orders of the Court by Circuit that I have omitted because they are merely different presentation of information found elsewhere.

Click for information sources summary table

Source
Format
Completeness
Timeliness

Journal of the SCOTUS
PDF
Supposedly, every disposition at the Court. Includes Bar admissions and other cruft.
"New Journal entries are posted on this website about two weeks after the event." That is being optimistic: as of this writing on March 1, 2024, the last Journal entry for the 2023 term is dated January 10, 2024.

Orders of the Court
PDF
Unsigned orders, i.e. not including Opinions
"Regularly scheduled lists of orders are issued on each Monday that the Court sits, but 'miscellaneous' orders may be issued in individual cases at any time. Scheduled order lists are posted on this Website on the day of their issuance, while miscellaneous orders are posted on the day of issuance or the next day."

Granted/Noted Cases List
PDF
Mostly grants of certiorari, including decided cases (for which there are opinions). Very limited subset of cases.
Less than a week of lag; as of March 1, 2024 the document is dated as of February 28, 2024.

Opinions Relating to Orders
PDF
Only opinions that accompany select summary dispositions (typically dissents). These also appear at the back of the Orders documents.
"Any opinions...will be posted here on the day they are issued. "

Opinions of the Court
PDF
All cases decided by the full court have their opinions published here.
These appear to be posted the day the case is decided.

Docket Search
HTML, JSON, XML
As far as I can tell, it's all here.
Reflects the timeliness of the underlying dockets.

Calendars and Lists: Argument Calendar
PDF
Contains docket numbers of cases scheduled for oral arguments.
Published 2-3 months prior to session (within a term) in which the arguments will be heard.

Calendars and Lists: Court Calendar
PDF
Waste of time unless you want to fine tune image recognition ML models. Easier to eyeball and transcribe; but can be useful for timing scrapes of Orders Lists, which appear "at 9:30 a.m. on the Monday following a Court conference, usually held three times a month when the Court is sitting".
Has all the key dates for a term.

Calendars and Lists: Hearing Lists
PDF
Published (late in the) prior week.
Somehow different from the Argument Calendar, I'm not clear on the distinction. Contains docket numbers of the cases to be heard on an upcoming day over the following week.

Calendars and Lists: Day Call
PDF
Apparently a daily update to the Argument Calendar, published the morning (ET) of the prior day.
Contains docket numbers of the cases to be heard that day.

 There is a graphical captcha interstitial to inhibit simply signing up for all dockets and receiving push notifications by email. Becase heaven forbid the unwashed masses might want what's theirs?
HTML pages
As returned by search queries: https://www.supremecourt.gov/search.aspx?filename=/docket/docketfiles/html/public/23-175.html
Also available directly: https://www.supremecourt.gov/docket/docketfiles/html/public/23-175.html
These are the pages linked to by the full-test docket search feature. They further contain links to JSON and XML representations of what I believe is the identical data. Not surprisingly I have not found systematic (or any, so far) discrepancies between HTML and JSON representations of dockets.
These pages are updated intra-day.
JSON pages
Found after navigating to RSS feeds of dockets: http://www.supremecourt.gov/RSS/Cases/JSON/23-175.json
XML representations are found by substituting ../XML/.xml in the above URL.
Contains all docket information. Updated intra-day.
Example: docket 23-939
{"CaseNumber":"23-939 ","bCapitalCase":false,"sJsonCreationDate":"03/01/2024","sJsonTerm":"2023","sJsonCaseNumber":"00939","sJsonCaseType":"Paid","RelatedCaseNumber":[],"PetitionerTitle":"Donald J. Trump, Petitioner","RespondentTitle":"United States","DocketedDate":"February 28, 2024","Links":"Linked with 23A745","LowerCourt":"United States Court of Appeals for the District of Columbia Circuit","LowerCourtCaseNumbers":"(23-3228)","LowerCourtDecision":"February 6, 2024","QPLink":"../qp/23-00939qp.pdf","ProceedingsandOrder":[{"Date":"Feb 12 2024","Text":"Application (23A745) for a stay, submitted to The Chief Justice.","Links":[{"Description":"Main Document","File":"2024-02-12 - US v. Trump - Application to S. Ct. for Stay of D.C. Circuit Mandate - Final With Tables and Appendix.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300410/20240212154110541_2024-02-12%20-%20US%20v.%20Trump%20-%20Application%20to%20S.%20Ct.%20for%20Stay%20of%20D.C.%20Circuit%20Mandate%20-%20Final%20With%20Tables%20and%20Appendix.pdf"},{"Description":"Proof of Service","File":"2024-02-12 - Certificate of Service for Stay Application.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300410/20240212154123465_2024-02-12%20-%20Certificate%20of%20Service%20for%20Stay%20Application.pdf"}]},{"Date":"Feb 12 2024","Text":"Petition for a writ of certiorari filed."},{"Date":"Feb 13 2024","Text":"Response to application (23A745) requested by The Chief Justice, due February 20, 2024, by 4pm (EST)."},{"Date":"Feb 13 2024","Text":"Brief amicus curiae of Jon Danforth, J. Michael Luttig, Carter Phillips, Peter Keisler, Larry Thompson, Stuart Gerson, et al. filed.","Links":[{"Description":"Main Document","File":"2024-2-13 Amici Curiae Brief Opposing Application for Stay.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300472/20240213120356911_2024-2-13%20Amici%20Curiae%20Brief%20Opposing%20Application%20for%20Stay.pdf"},{"Description":"Proof of Service","File":"Proof of Service.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300472/20240213120405955_Proof%20of%20Service.pdf"}]},{"Date":"Feb 13 2024","Text":"Brief amicus curiae of Constitutional Law Scholars filed.","Links":[{"Description":"Main Document","File":"Trump v. US CAC Scholars Brief.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300499/20240213140115053_Trump%20v.%20US%20CAC%20Scholars%20Brief.pdf"},{"Description":"Certificate of Word Count","File":"Trump v. US CAC Cert Compliance.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300499/20240213140124180_Trump%20v.%20US%20CAC%20Cert%20Compliance.pdf"},{"Description":"Proof of Service","File":"Trump v. US CAC Cert of Service.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300499/20240213140133124_Trump%20v.%20US%20CAC%20Cert%20of%20Service.pdf"}]},{"Date":"Feb 14 2024","Text":"Response to application from respondent United States filed.","Links":[{"Description":"Main Document","File":"23A745_Trump v. United States_Gov. stay resp_FINAL.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300627/20240214180323991_23A745_Trump%20v.%20United%20States_Gov.%20stay%20resp_FINAL.pdf"},{"Description":"Proof of Service","File":"23A745 - Trump v USA Certificate.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300627/20240214180338307_23A745%20-%20Trump%20v%20USA%20Certificate.pdf"}]},{"Date":"Feb 14 2024","Text":"Brief amicus curiae of Protect Democracy Project filed.","Links":[{"Description":"Main Document","File":"23A745 Trump v. USA_Amicus Brief.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300593/20240214142544324_23A745%20Trump%20v.%20USA_Amicus%20Brief.pdf"},{"Description":"Proof of Service","File":"23A745 Trump v. USA_Proof of Service.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300593/20240214142520222_23A745%20Trump%20v.%20USA_Proof%20of%20Service.pdf"}]},{"Date":"Feb 15 2024","Text":"Reply of applicant Donald J. Trump filed.","Links":[{"Description":"Reply","File":"2024-02-15 - 23A745 - Reply iso Application to S. Ct. for Stay of D.C. Circuit Mandate.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300749/20240215174027604_2024-02-15%20-%2023A745%20-%20Reply%20iso%20Application%20to%20S.%20Ct.%20for%20Stay%20of%20D.C.%20Circuit%20Mandate.pdf"},{"Description":"Proof of Service","File":"2024-02-15 - Certificate of Service for Reply iso Stay Application.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300749/20240215174038799_2024-02-15%20-%20Certificate%20of%20Service%20for%20Reply%20iso%20Stay%20Application.pdf"}]},{"Date":"Feb 15 2024","Text":"Brief amicus curiae of David Boyle filed.","Links":[{"Description":"Main Document","File":"23A745_tsac_DavidBoyle.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300655/20240215114857700_23A745_tsac_DavidBoyle.pdf"}]},{"Date":"Feb 16 2024","Text":"Brief amicus curiae of Alabama and 21 Other States filed.","Links":[{"Description":"Main Document","File":"States Brief in Trump v US FINAL 2.16.24.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300793/20240216132806756_States%20Brief%20in%20Trump%20v%20US%20FINAL%202.16.24.pdf"},{"Description":"Proof of Service","File":"Certificate of Service for States Br. FINAL.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300793/20240216132818877_Certificate%20of%20Service%20for%20States%20Br.%20FINAL.pdf"}]},{"Date":"Feb 19 2024","Text":"Brief amicus curiae of Christian Family Coalition filed.","Links":[{"Description":"Main Document","File":"23A745 Amicus CFC.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300875/20240219151217308_23A745%20Amicus%20CFC.pdf"},{"Description":"Certificate of Word Count","File":"CERTIFICATE OF COMPLIANCE.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300875/20240219151223585_CERTIFICATE%20OF%20COMPLIANCE.pdf"},{"Description":"Proof of Service","File":"CERTIFICATE OF SERVICE.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300875/20240219151229160_CERTIFICATE%20OF%20SERVICE.pdf"}]},{"Date":"Feb 19 2024","Text":"Brief amicus curiae of Jeremy Bates filed.","Links":[{"Description":"Main Document","File":"amicus brief oppn to stay 2 19 2024.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300877/20240219152614536_amicus%20brief%20oppn%20to%20stay%202%2019%202024.pdf"},{"Description":"Proof of Service","File":"COS amicus brief oppn to stay 2 19 2024 .pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300877/20240219152631553_COS%20amicus%20brief%20oppn%20to%20stay%202%2019%202024%20.pdf"}]},{"Date":"Feb 20 2024","Text":"Brief amicus curiae of Former Attorney General Edwin Meese III, Law Professors Steven Calabresi and Gary Lawson, and Citizens United filed.","Links":[{"Description":"Main Document","File":"Trump v US Stay Amicus Final.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300972/20240220173530766_Trump%20v%20US%20Stay%20Amicus%20Final.pdf"},{"Description":"Certificate of Word Count","File":"Certificate Word Count.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300972/20240220173536501_Certificate%20Word%20Count.pdf"},{"Description":"Proof of Service","File":"Proof of Service.pdf","DocumentUrl":"http://www.supremecourt.gov/DocketPDF/23/23-939/300972/20240220173542733_Proof%20of%20Service.pdf"}]},{"Date":"Feb 28 2024","Text":"Application (23A745) referred to the Court."},{"Date":"Feb 28 2024","Text":"Petition GRANTED."},{"Date":"Feb 28 2024","Text":"The application for a stay presented to The Chief Justice is referred by him to the Court.  The Special Counsel’s request to treat the stay application as a petition for a writ of certiorari s granted (23-939), and that petition is granted limited to the following question:  Whether and if so to what extent does a former President enjoy presidential immunity from criminal prosecution for conduct alleged to involve official acts during his tenure in office.  Without expressing a view on the merits, this Court directs the Court of Appeals to continue withholding issuance of the mandate until the sending down of the judgment of this Court.  The application for stay is dismissed as moot.  \r\n     The case will be set for oral argument during the week of April 22, 2024.  Petitioner’s brief on the merits, and any amicus curiae briefs in support or in support of neither party, are to be filed on or before Tuesday, March 19, 2024.  Respondent’s brief on the merits, and any amicus curiae briefs in support, are to be filed on or before April 8, 2024.  The reply brief, if any, is to be filed on or before 5 p.m., Monday, April 15, 2024."},{"Date":"Feb 29 2024","Text":"Record requested from the United States Court of Appeals for the District of Columbia Circuit."},{"Date":"Mar 01 2024","Text":"Record received electronically from the United States Court of Appeals for the District of Columbia Circuit and available with the Clerk."},{"Date":"Mar 01 2024","Text":"Record received from the United States District Court for the District of Columbia. The record is electronic and is available on PACER."}],"AttorneyHeaderPetitioner":"Attorneys for Petitioner","Petitioner":[{"Attorney":"D. John Sauer","IsCounselofRecord":true,"Title":"James Otis Law Group, LLC","PrisonerId":null,"Phone":"314-562-0031","Address":"13321 North Outer Forty Road\r\nSuite 300","City":"St. Louis","State":"MO","Zip":"63017","Email":"John.Sauer@james-otis.com","PartyName":"Donald J. Trump"},{"Attorney":"D. John Sauer","IsCounselofRecord":true,"Title":"James Otis Law Group, LLC","PrisonerId":null,"Phone":"314-562-0031","Address":"13321 North Outer Forty Road\r\nSuite 300","City":"St. Louis","State":"MO","Zip":"63017","Email":"John.Sauer@james-otis.com","PartyName":"President Donald J. Trump"}],"AttorneyHeaderRespondent":"Attorneys for Respondent","Respondent":[{"Attorney":"Michael R. Dreeben","IsCounselofRecord":true,"Title":"Counselor to the Special Counsel","PrisonerId":null,"Phone":"202-305-9654","Address":"Department of Justice\r\n950 Pennsylvania Ave, NW","City":"Washington","State":"DC","Zip":"20530","Email":"SCO_JLS_SupremeCtBriefs@usdoj.gov","PartyName":"United States"}]}

Docket numbering
There are four docket types but all use a consistent numbering format: \d\d[-AMO]\d{1,5}. I have found four types of dockets based on their identifier symbol:

Petition (regular) dockets: '-'
Application dockets: 'A'
Motion dockets: 'M'
Original(?) dockets: 'O'

In the JSON presentations of dockets, the 'sJsonCaseNumber' field is zero-padded to the full five digits, e.g. for docket 23-2 that field contains '00002'. I believe this is the only use of zero padding in the docket numbers found on the SCOTUS site, probably to avoid lexical sorting in SCOTUS's docket database.
Petition (regular) dockets
Petitions for writ of certiorari are given these docket numbers, e.g. 23-175.
The SCOTUS Public Information Office offers some guidance here:

All cases receive a docket number upon filing in the Clerk's Office, ranging from 3 to 7
digits (e.g., 21–1, 21–2000).

The term In Forma Pauperis (IFP) describes permission given to an indigent to proceed
without liability for Court fees or costs.* "Pauper" cases are always given up to a 7–digit
number with the last digits up to the 10,000 or 11,000 series (e.g., 21–5661, 21-10269).

IFP cases seem to number consistently starting at <YY>-5001.
Docket numbering between 2000-5000 is not consistent; I don't have a sense of any pattern here.
Dockets 5001 and up appear most likely to have few docket entries, and seem most likely to be summarily denied writs of certiorari.

Application dockets
Applications to extend the time to file petitions for certiorari; or for stays pending disposition of petitions for certiorari; possibly other types of actions that I haven't found yet.
When successful, these cases become petition dockets with regular docket numbers. However, there does not appear to be any link from the A docket to the - docket; only in the other direction. Thus petition docket 23-232 has a reference back to the application 23A2 that preceded it,
{...,
Links: "Linked with 23A2",
...}

but there is no reference in 23A2 to 23-232.
Motion dockets
Movants often ask for leave to proceed with their applications as veterans. I'm not clear on what this status accords movants, but it seems to offer some distinction from Applications.
"Orig." dockets
I have found one case, 22O141 (Texas v. New Mexico and Colorado) that is listed in the Granted/Noted List as "Orig. 141". Maybe there are more? But I've included this identifier in the regex patterns.

Progress update
Constrained by lack of a central index
The lack of a single source of truth about SCOTUS dockets looks like it will be the major determinant of how scraping can proceed.
Unfortunately, the simplest solution has been intentionally placed out of reach. SCOTUS's own docket activity email notification system is protected with the ominous-sounding BotDetectCaptcha scripts.
That leaves us needing two dimensions of information: where to look for docket information, and when/how often to look there.
When to look
If handling state will be up to the scrapers' caller, as @mlissner suggested above, I would like to leave this part for later, or for others.
Where to look
So far I have tried three approaches:

Brute-force queries of docket number ranges.
Extracting docket numbers from the Granted/Noted List and from Orders of the Court, both of which come as PDFs.
The Docket Search page, as pointed out by @grossir. It returns URLs to docket HTML pages; I then regex match the docket numbers in the URLs and scrape the corresponding JSON dockets.

My sense is that some combination of Docket Search query results and brute force will be closest to optimal in terms of server resource use, completeness, and timeliness.
Docket Search
Because the date strings for docket entries in the JSON presentation use a consistent format (strftime "%b %d, %Y"), search results on those date strings are generally fairly accurare (true positives / total).

Click for search accuracy table

date_string
spurious
all
accuracy

0
2024-03-01
130
165
0.212121

1
2024-02-29
38
147
0.741497

2
2024-02-28
28
127
0.779528

3
2024-02-27
26
96
0.729167

4
2024-02-26
31
203
0.847291

5
2024-02-25
2
5
0.6

6
2024-02-24
0
6
1

7
2024-02-23
30
200
0.85

8
2024-02-22
15
141
0.893617

9
2024-02-21
16
130
0.876923

10
2024-02-20
15
450
0.966667

11
2024-02-19
6
19
0.684211

12
2024-02-18
2
4
0.5

13
2024-02-17
3
6
0.5

14
2024-02-16
33
486
0.932099

15
2024-02-15
18
171
0.894737

16
2024-02-14
13
112
0.883929

17
2024-02-13
12
84
0.857143

18
2024-02-12
22
124
0.822581

19
2024-02-11
8
12
0.333333

20
2024-02-10
2
10
0.8

21
2024-02-09
16
130
0.876923

22
2024-02-08
12
173
0.930636

23
2024-02-07
9
135
0.933333

24
2024-02-06
11
83
0.86747

25
2024-02-05
12
100
0.88

26
2024-02-04
2
3
0.333333

27
2024-02-03
2
7
0.714286

28
2024-02-02
10
115
0.913043

29
2024-02-01
9
176
0.948864

Here I classified as spurious any docket in the search results whose whose 'Last-Modified' header or whose last docket entry date was less than the date given by the search string.
Other benefits from using Docket Search:

Dockets are updated intraday, and I believe (more testing needed) that these updates are captured by searches in real time.
The docket numbers used in the URLs returned by the Search don't have text artifacts like PDFs do. I believe that regex matching on those URLs is faithfully capturing all docket numbers returned by each search.

PDF extraction
What @mlissner said. I used pymupdf and it seems to give decent results, but the text artifacts it returns (e.g. atypical Unicode dashes U+2010,...,U+2014) means a fair bit of trial and error on regex patterns just for docket numbers.
Combined with the idiosyncrasies of publication times for the various documents, I think sources like the Orders pages (as @flooie pointed out) can be useful for backscraping and making sure that higher-profile dockets have been updated. But they don't update frequently enough to be useful for real-time scraping.
Brute force
Assume -- in the absence of testing -- that Docket Search results are precise and false negatives (dockets updated but not reflected in the search results) are low. That would leave brute force searches largely for discovery of new dockets, particularly the A/M/O types.
The good support for 'If-Modified-Since' request headers on <supremecourt.gov> is encouraging. I've found the status code 304 behavior of not downloading assets speeds up both Docket Searches and downloads of known-good docket numbers. More of a "light touch" than brute force.
Upstream questions
I have the core of @mlissner's first action item,

Juriscraper to scrape content and make it digestible

and I'm working on making the code more robust to network and parsing errors.
One area where I need your input is with the database model(s) that @mlissner mentioned. I haven't tried to write a full docket parser since I don't know what the caller's data requirements will be. But I will need to turn to that.
Another question I have is about object APIs. Should I assume that the SCOTUS docket parser will called from the command line as python -m, such that object interfaces matter less than what's in argparse and main()? Or should I be trying to make objects conform to existing API patterns e.g. AppellateDocketReport?

Thanks for all this work and detail!
A couple thoughts:

I think we should just assume we can sign up for email updates. I just wrote a note to SCOTUS asking for help doing this. If they help us, great. If not, let's bust captchas and sign up for email alerts.

There are APIs for busting captchas now, and I think we should put them on the table, if we need them. If SCOTUS doesn't reply to my message, I think an analysis of available captcha-busting APIs would be really useful as a first step here (maybe in a separate issue?).

If we can get email updates, does that help us learn about new cases or does that only help with cases we already know exist?

I love that you're optimizing things by using their search tool, but I feel like brute force is a more reliable method, no? Could we imagine an algorithm that explores the docket number space to identify which dockets exist? Then, once we know a docket exists, we could subscribe to it for email updates?

If we do the email update route, the architecture changes a bit. We'll want:

An email address that can be used for this. (Maybe we use recap.email for this, and set up scotus@recap.email?)

When that email comes into AWS, it'll send an HTTP post to our API, so we'll need an API endpoint to handle that.

Then, once the API is hit, we can respond by merging whatever data is in the email (or just scraping it and merging that way).
Does that sound right?

Email notifications

There are APIs for busting captchas now, and I think we should put them on the table, if we need them.

I would not want to open up that can of worms myself. Leaving aside the poetic irony of doing that to the SCOTUS, you will want/need an obfuscation layer for IP addresses, user agent string, etc; and if their admins decide that<EMAIL_ADDRESS>suddenly appearing on 40K-ish dockets is not consistent with human effort, you could be back to square one. That could devolve into Mutually Assured Whac-A-Mole.
Other approaches, including what I've described, are going to be sub-optimal but they can be complete. I can continue to work on those but I'm not going to mess with captcha defeats.
Docket search

I love that you're optimizing things by using their search tool, but I feel like brute force is a more reliable method, no?

Agreed. But after looking into it, I think @grossir was right to point out the search interface as a viable tool. In a nutshell, search interface for fast but possibly incomplete results, combined with brute force sequential downloads for slow but complete results.
As mentioned I think issues of search timing aren't blockers for scraping at this point. But if we take email notifications as the gold standard, there's already a modest lag between content arriving at supremecourt.gov and the notifications going out. See for example this notification I received:

"Amicus brief of United States..."
Return-path<EMAIL_ADDRESS>Received: from [IPA] (helo=mailfront20.[server])
	by delivery05.[server] with esmtp  (Exim 4.86_2)
	id 1rhI2u-0006v5-EI
	for<EMAIL_ADDRESS>[timestamp]
Received: from exim by mailfront20.[server] with sa-scanned  (Exim 4.93)
	id 1rhI2t-008ah8-RM
	for<EMAIL_ADDRESS>[timestamp]
[...snip...]<EMAIL_ADDRESS>helo=a65-150.smtp-out.amazonses.com
Received: from a65-150.smtp-out.amazonses.com
	by mailfront20.[server] with esmtps  (TLS1.2:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_CBC__SHA1:128)
	(Exim 4.93)
	id 1rhI2k-008ae1-5C
	for<EMAIL_ADDRESS>[timestamp]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=sglce367a3eekz5cgo2jpvwdiom4ooya; d=sc-us.gov; t=1709596104;
	h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID:Date;
	bh=28APxLDdZGKYN8cytr+CtYFVI2hMZwtFrwWODzxt9Ug=;
	b=XKzxz0kZ9d6WlgHjyJLkc7DBW9V5daLi8yTCiItcWh28DrC1ywbbTAmWpLqQ56pV
	8EOOk2itFhQujXkdijeMql/eM3GUQRSjnDJtfhKluWmoW2xupKRRbYq7/mw7otNSHg0
	IknCxfbKCXI8ENOchtz7DiDnana1QvPbligBLP24=
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
	s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1709596104;
	h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID:Date:Feedback-ID;
	bh=28APxLDdZGKYN8cytr+CtYFVI2hMZwtFrwWODzxt9Ug=;
	b=SJWIgiOt/V8jsRjxpv50sS9YGlkCZTLjUlkAZxnb0S+mi9XmvFDjSl72VTekjf45
	pcc8t/PCC27u847OPbrF/EmNlkQysSxbMTjOl8q/s5DJF9Jslhm80PrOHGR/uv4AYd6
	oV8S2BGXitN/SrkbmdpTopTq1GDicECgd8pyqyfY=
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>Subject: Supreme Court Electronic Filing System
MIME-Version: 1.0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit
Message-ID<EMAIL_ADDRESS>Date: Mon, 4 Mar 2024 23:48:24 +0000
Feedback-ID: 1.us-east-1.+5GeZMB3eXeyv3WY8brP46tghxJpXFIF9yDDvLuTQrk=:AmazonSES
X-SES-Outgoing: 2024.03.04-<IP_ADDRESS>

A new docket entry, "Amicus brief of United States submitted." has been added for <a href='https://www.supremecourt.gov/search.aspx?filename=/docket/DocketFiles/html/Public/23-175.html'>City of Grants Pass, Oregon, Petitioner v. Gloria Johnson, et al., on Behalf of Themselves and All Others Similarly Situated</a>. You have been signed up to receive email notifications for No. 23-175. <br><br> If you no longer wish to receive email notifications on this case, please

<a href='https://file.supremecourt.gov/Request/NotificationOptOutGet?id=CfDJ8LWjh78o-U5EigyPTWy9BmcYx3Mv5n9kVSbUZvEjTlHeuXmoUJm4CgkH4w-SoXRjva81LlpVk5vw_tiAJt5Gcd2uexHzkPCAYwm-ByGsJxc4lBW3vbKTJljYr1BXC-6TnboxUcviCs05dpHQOBBmkT0'>click here</a>.

Look at the notification lag by comparing the email time stamp header (Date: Mon, 4 Mar 2024 23:48:24 +0000) with the JSON docket header (last-modified: Mon, 04 Mar 2024 23:42:46 GMT), and lastly with the time stamp portion of the docket filing URL http://www.supremecourt.gov/DocketPDF/23/23-175/302264/20240304183726571_23-175npUnitedStates.pdf (i.e. 2024-03-04-T18:37:26, which should be EST so 23:37 GMT). Around five minutes between the docket item hitting the web site and the notification email arriving.
Compare that with scraping results from the docket search interface. I just ran a test using the string 'Feb 3, 2024', including sending the 'If-Modified-Since' filter for 2024-03-04T00:00:00, and I was able to download the 174 valid dockets (i.e. dockets actually updated on/after 2024-03-04) out of 198 search results in one minute and 37 seconds. Multi-threaded or asyncio performance would obviously be even better although I believe httpx integration is a separate issue in your project. Running this job every, say, five minutes wouldn't be a huge tradeoff to avoid messing with captchas, I hope.

Could we imagine an algorithm that explores the docket number space to identify which dockets exist? Then, once we know a docket exists, we could subscribe to it for email updates?

This is what the petition (i.e. regular) docket number search space looks like after a few days' worth of brute force attempts (X axis is SCOTUS term, Y axis is the sequential integer portion of the docket number):

There is just a single Y axis in the graph; what appears to be the bottom series is actually the "pauper" case numbers that seem to begin strictly at 5000, as described in my earlier note. For years 2018+ the numbering is not perfectly contiguous but close.
Unfortunately, that 'close' is too sparse for e.g. bisection or ternary searches; I tried. Instead for docket number discovery I have been doing sequential searches on the 1-2500 and 5000+ ranges, excluding known good docket numbers, and limiting the number of "Not Found" page results before the search quits. Slow, but effective. And embarrassingly parallel. It just requires known good docket numbers as state.

This is great, thank you. Two questions come to mind if we don't want to do the Capcha thing and the court doesn't get back to us (I just emailed again).

How often would it be practical to update the dockets if we have to crawl them all and we do it in parallel?

Am I gathering that doing the brute force approach without doing search is viable? It seems more reliable to me.

Is there a way to know that we can stop searching a particular case? Something to indicate that it's completed?

Sorry for the naive questions. I'm really leaning on your help, but I really appreciate the research you're doing!

I mean - we could just manually subscribe to each one - or all the important cases?  its not ideal but considering the source would be appropriate I think.

You mean with a human instead of a captcha buster?

I am

I think I'd rather automate it, even if that involves busting captchas or scanning their website every few minutes for updates. Let's see what @ralexx thinks about the scanning idea (they said it was "embarrassingly parallel," which is promising), but my general thought is we shouldn't set ourselves up to have to do things as humans, because that scales poorly and we're bad at it.

@mlissner  you reminded me I didn't have my pronouns set on my profile; thanks. He/him.

How often would it be practical to update the dockets if we have to crawl them all and we do it in parallel?

I can see three independent scraping routines with different speeds:

Objective
Routine
Uses 'If-Modified-Since'
Relative speed

Docket discovery: find cases newly docketed since last run
Query URLs in sequential order of docket numbers
No
Slow

Reliable sources update: Find dockets guaranteed to have been updated
Scrape docket numbers from Docket Search, Orders, etc. then update those dockets
Yes
Faster

Recent update check: find dockets modified since last run
Similar to docket discovery but only operate on known docket numbers and use header filtering
Yes
Fastest

Docket discovery with no header filtering is 2-3 seconds per request for me, of which only 1s is me rate-limiting so they don't ban my IPA. Also, the 1.7 kBps download speeds I'm seeing on those requests are unlikely to be slow from my end, hence my comment about the possible gains from parallelization.
For reliable sources updates that do use header filtering, my guess is <= 10 minutes. Since yesterday I have been re-crawling dockets to save their 'Last-Modified' headers. When that's done I can estimate the number of dockets being updated on a given day. So far I think the largest number of dockets I have seen updated on the Orders List is ~400, plus there would also be some other dockets that had non-dispositive entries. I'm not fully parsing the dockets yet but that should be fast. So unless you're looking at scraping >1000 dockets per run, I think sub-10 minutes should be feasible.
The use of header filtering on recent update checks makes it subjectively faster, as does avoiding the delay resulting from being served the 'Not Found' page on unused docket numbers. I can also see further optimization from, say, prioritizing update requests by descending order of docket count, so that more active dockets are checked first in a given run.

Am I gathering that doing the brute force approach without doing search is viable? It seems more reliable to me.

I do think brute force is viable by itself. Not only that, I think it's a requirement for discovering when new dockets have been created, because of its reliability as you say.
The impact of brute force docket discovery can also be broken into smaller chunks. Once we have a sense of how many dockets might be populated on a given day, we know roughly how far ahead to run docket discovery given the sequential numbering patterns and knowledge of existing dockets. And once per day, say, you can run docket discovery on gaps in the numbering just as a precaution.

Is there a way to know that we can stop searching a particular case? Something to indicate that it's completed?

By scraping docket entries and finding a case disposition (e.g. from 22-175, {"Date":"Dec 12 2022","Text":"Petition DENIED."}). There does not appear to be any data item in dockets to indicate case status.

A few data points on download speed using brute force.
With a no-frills ThreadPoolExecutor and five workers I can scrape (but not write to disk) about 6 dockets per second in docket discovery. And that's including a loop of three download retries when a response is missing the 'Last-Modified' header. This network traffic graph shows the same 250-docket search space being scraped; the first two bursts of activity use the thread pool, the third one is single-threaded.

Nice. That sounds fast enough to be able to be as aggressive as we need to be.

Questions about docket entries:

Do you want to scrape PDFs?
If so, does that include filing boilerplate such as the 'Certificate of Word Count' and 'Proof of Service' as in the example below?

[
	{
		"Description": "Petition",
		"File": "Petition Trevino v. Palmer.pdf",
		"DocumentUrl": "http://www.supremecourt.gov/DocketPDF/23/23-484/288797/20231103144956195_Petition%20Trevino%20v.%20Palmer.pdf"
	},
	{
		"Description": "Appendix",
		"File": "Appendix Trevino v. Palmer.pdf",
		"DocumentUrl": "http://www.supremecourt.gov/DocketPDF/23/23-484/288797/20231103145014794_Appendix%20Trevino%20v.%20Palmer.pdf"
	},
	{
		"Description": "Certificate of Word Count",
		"File": "Word Count Trevino v. Palmer.pdf",
		"DocumentUrl": "http://www.supremecourt.gov/DocketPDF/23/23-484/288797/20231103145051994_Word%20Count%20Trevino%20v.%20Palmer.pdf"
	},
	{
		"Description": "Proof of Service",
		"File": "Service Trevino v. Palmer.pdf",
		"DocumentUrl": "http://www.supremecourt.gov/DocketPDF/23/23-484/288797/20231103145104461_Service%20Trevino%20v.%20Palmer.pdf"
	}
]

Do you want to scrape PDFs?

Emphatically yes!

If so, does that include

Yes. We want to have everything that's possible on the docket. You just never know when some scholar is going to get interested in a topic and it's usually easier to just get everything anyway!

Two areas that I need guidance from others, please:
Data model for parsed docket output
@mlissner  you asked for

Some database models, probably new ones, to store what we scrape.

When those data models are available, I need mappings from the SCOTUS docket back to the models' data items so I can finish the docket parser.
For reference, I merged the dict keys from 39,464 dockets that I've scraped so far; see below.

Merged dict keys from dockets
{'CaseNumber': None,
 'bCapitalCase': None,
 'RelatedCaseNumber': [],
 'PetitionerTitle': None,
 'RespondentTitle': None,
 'DocketedDate': None,
 'Links': None,
 'LowerCourt': None,
 'LowerCourtCaseNumbers': None,
 'LowerCourtDecision': None,
 'LowerCourtRehearingDenied': None,
 'QPLink': None,
 'ProceedingsandOrder': [{'Date': None,
   'Text': None,
   'Links': [{'Description': None, 'File': None, 'DocumentUrl': None}]},
  {'Date': None, 'Text': None}],
 'AttorneyHeaderPetitioner': None,
 'Petitioner': [{'Attorney': None,
   'IsCounselofRecord': None,
   'Title': None,
   'PrisonerId': None,
   'Phone': None,
   'Address': None,
   'City': None,
   'State': None,
   'Zip': None,
   'Email': None,
   'PartyName': None}],
 'AttorneyHeaderRespondent': None,
 'Respondent': [{'Attorney': None,
   'IsCounselofRecord': None,
   'Title': None,
   'PrisonerId': None,
   'Phone': None,
   'Address': None,
   'City': None,
   'State': None,
   'Zip': None,
   'Email': None,
   'PartyName': None}],
 'AttorneyHeaderOther': None,
 'Other': [{'Attorney': None,
   'IsCounselofRecord': None,
   'Title': None,
   'PrisonerId': None,
   'Phone': None,
   'Address': None,
   'City': None,
   'State': None,
   'Zip': None,
   'Email': None,
   'PartyName': None},
  {'Attorney': None,
   'IsCounselofRecord': None,
   'Title': None,
   'PrisonerId': None,
   'Phone': None,
   'Address': None,
   'City': None,
   'State': None,
   'Zip': None,
   'Email': None,
   'PartyName': None}],
 'DiscretionaryCourtDecision': None,
 'sJsonCreationDate': None,
 'sJsonTerm': None,
 'sJsonCaseNumber': None,
 'sJsonCaseType': None,
 'DiscretionaryCourtRehearing': None}

Parser object inheritance or conformance
SCOTUS dockets are sufficiently different from Juriscraper's other sources that I have used pacer.appellate_docket.AppellateDocketReport as a guide but not as a superclass. Do you prefer that I inherit from your existing objects, and if so, which one(s)? If not inheritance, is there an existing caller for the parser to conform to?

I have used pacer.appellate_docket.AppellateDocketReport as a guide but not as a superclass.

Yeah, that's how I would do it too.

I need mappings from the SCOTUS docket back to the models' data items so I can finish the docket parser.

I think it might be good to see some code. The models are often pretty tricky, but maybe if we had your PR for scraping, we could use that to build the models. I'm also worried we've had so much conversation without a code check-in, which can sometimes be a mistake.
Do you want to try to get your Juriscraper PR at least drafted, and then we can go from there?

OK

I heard back from SCOTUS, finally. They don't want to talk, so it's up to us to figure out how we want to approach things (no surprise here, I guess).


Adding a query param to limit the /cookie_sync pixel count

Fixes #44.
This also replaces gulp-connect with gulp-webserver because of that same flatmap-stream vulnerability... I had to do this to be able to test it.

nice thanks for the PR! Does it make sense to randomize the order of the syncs before we truncate it for MAX? Thinking here that the same pixels will be fired over and over and some bidders may never see the pixels fired because they are later in the list...

PBS has a related PR: https://github.com/prebid/prebid-server/pull/759... we should probably wait until that's released and then use the server-side limit.
Server-side limiting is better for a few reasons (less bandwidth, easier to roll out bugfixes + smarter sync logic, etc)