golang client net/http: timeout awaiting response headers

Required information

Distribution: Ubuntu
Distribution version: 20.04
The output of "lxc info" or if that fails:

Kernel version: 5.4.0-109-generic
LXC version:
LXD version: 5.1
Storage backend in use: ZFS

Issue description
A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.
I developend an application with pylxd before and did not run into any issues with it. Now I am migrating my codebase to go and using the lxd client library.
I ran into in issue with timeouts
Post "https://<IP_ADDRESS>:8443/1.0/networks?project=sample-project": net/http: timeout awaiting response headers

In this case, the python code uses a timeout of None which means that it will wait indefinitely until it gets a response back.
https://github.com/lxc/pylxd/blob/a44deea30cb2dce3b1fd34240155c5af6479f1a2/pylxd/client.py#L333
On network creation, the function is blocked until it was created
https://github.com/lxc/pylxd/blob/a44deea30cb2dce3b1fd34240155c5af6479f1a2/pylxd/models/network.py#L112
In the go codebase, there is a timeout of 30 seconds for response headers
https://github.com/lxc/lxd/blob/2ed60cd252b2d3daf64e8ad0413d5097568f2a1e/client/util.go#L29
That is the problem, my network creation request takes more than 30 seconds to complete.
Possible fixes are to increase the timeout or provide a way for the programmer to increase it manually.
Note: The network gets created in the background even after the timeout, but I want a way to block my application until the request is completed successfully. I do not want to write a loop checking the state of the network until it gets created :(
Steps to reproduce

Step one
Deploy a slow LXD server
Step two
Write a go application to create a new OVN network
Step three
After 30 seconds, the CreateNetwork function thorws a timeout error

Information to attach

[ ] Any relevant kernel output (dmesg)
[ ] Container log (lxc info NAME --show-log)
[ ] Container configuration (lxc config show NAME --expanded)
[ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

What is the contents and url of your request?

We added timeouts recently (as all network programming should have timeouts) but we may need to increase it to cover all common workload types (although our tests are completing in under that time) so would be good to understand why its taking so long in your case.

How long does it take with your pylxd app?

import time
from pylxd import Client

client = Client(
    verify=False,
    cert=("lxd.crt", "lxd.key"),
    endpoint="https://remote-ip:8443",
    project="sample-project",
)

tic = time.perf_counter()
client.networks.create(
    "ovn-network",
    type="ovn",
    config={
        "ipv4.nat": "true",
        "ipv6.address": "none",
        "network": "parent-network",
    },
)
toc = time.perf_counter()

print(f"Request took {toc - tic:0.4f} seconds")

Request took 103.0310 seconds

My LXD server is running dozens of containers/vms/networks, that might be why it takes almost 2 seconds to create that network

2s?

Please can you run 'lxc monitor --type=logging --pretty' on the server while trying to create the network so we can see which steps take time.

I recently starting getting this timeout error too copying an image from a local to a remote (my ISP upload is around 18-20mbps & the lxd nodes are connected with wireguard) - the image size is 179mb
lxc image copy -v --copy-aliases --mode=push local:ubuntu-minio-22.04 myremote:
the image copies but the alias does not get applied

Please can you run the command with --debug output as I'm surprised that an operation like copying an image isn't done with an async operation.

Please can you also provide the exact error. We did not limit request timeout. But rather just the initial header response timeout. So I suspect something else aside from bandwidth limitations is at play here (it may be firewall or multi ip issue).

lxc image copy --debug -v --copy-aliases --mode=push local:ubuntu-minio-22.04 iacimg:
DEBUG  [2022-05-08T06:46:57+01:00] Connecting to a local LXD over a Unix socket
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "config": {
                        "core.https_address": "<IP_ADDRESS>"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "<IP_ADDRESS>:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIICCzCCAZKgAwIBAgIQUI0swBDs6vJwp2PlwqkJazAKBggqhkjOPQQDAzA3MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QGVuZGVh\ndm91cjAeFw0yMjA0MjUxNTU0NDlaFw0zMjA0MjIxNTU0NDlaMDcxHDAaBgNVBAoT\nE2xpbnV4Y29udGFpbmVycy5vcmcxFzAVBgNVBAMMDnJvb3RAZW5kZWF2b3VyMHYw\nEAYHKoZIzj0CAQYFK4EEACIDYgAEgjrXWznR94gEKBONGs4smRBLo7qqyaMwkiTq\n/2J7iMKF5nsZgqwNv1UJnw1AlJnLbx+M5bkwA0AtCTyeSbtNRnIgFgs8pjmR6n7q\nHuV428RLPl4JMlrdsQ9SIdQx3dSVo2MwYTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\nBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAsBgNVHREEJTAjggllbmRlYXZv\ndXKHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwQvRC\nwVy+aWeg0rUYSOboOVy7rYzQEdDNlfgJWe2HNwBHhywtpab7+ooZgZfU4mXMAjBW\n8w9kCHwLh3/bBwO7TNmV7t/k2ZHB+GpBZiRCNpEHUpdfio20Wjm+7954uyd6g6Y=\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "0532a6edbf8bdd51b97faeb9b2669191ed7f5aa1ece9234a4d1e4c7abf3b3c34",
                        "driver": "lxc | qemu",
                        "driver_version": "4.0.12 | 6.1.1",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.17.5-hardened1-1-hardened",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "EndeavourOS",
                        "os_version": "",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "endeavour",
                        "server_pid": 13740,
                        "server_version": "5.1",
                        "storage": "zfs",
                        "storage_version": "2.1.4-1",
                        "storage_supported_drivers": [
                                {
                                        "Name": "zfs",
                                        "Version": "2.1.4-1",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "btrfs",
                                        "Version": "5.4.1",
                                        "Remote": false
                                },
                                {
                                        "Name": "cephfs",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                                        "Remote": false
                                }
                        ]
                }
        }
DEBUG  [2022-05-08T06:46:57+01:00] Connecting to a remote LXD over HTTPS
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="https://<IP_ADDRESS>:8443/1.0"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "config": {
                        "core.https_address": "<IP_ADDRESS>"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "<IP_ADDRESS>:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIICAjCCAYmgAwIBAgIQN4N01EkRTWt3KmQdy46muDAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGJhY2t1\ncDAeFw0yMjA1MDcyMjI2MzNaFw0zMjA1MDQyMjI2MzNaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAYmFja3VwMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAERizSl/gpUEExmNc27xXKpHIlqQOOWNh0oNhj+4Bgf0Sf\nHV6/ggxBIKGxzZvwGp/shDU2PONlwnATOt2OUUGVVVlKcZLleP5DV9Ku82ty1Y8k\ntbRYE4nGq55NRLBBXMvno2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZiYWNrdXCHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwUZ1NZCLyKl6mfWjg\nZdRl6Ga7UwXVxzaw+YVmHFvZdDDslJWDODpRfHktYOcwmarWAjAmHKRMdv6Z2Ro3\nQgYeHZ1DkVsK7XX9HKHmHuUNmmafeeVmyyuKiztkKSl1h/8ty5U=\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "2aca9ad8acf0160e3bc16302ae31515d1aeff4a6d3ad030bff619a49c58562ac",
                        "driver": "qemu | lxc",
                        "driver_version": "6.1.1 | 4.0.12",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.15.0-27-generic",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Ubuntu",
                        "os_version": "22.04",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "backup",
                        "server_pid": 1154,
                        "server_version": "5.1",
                        "storage": "zfs",
                        "storage_version": "2.1.2-1ubuntu3",
                        "storage_supported_drivers": [
                                {
                                        "Name": "btrfs",
                                        "Version": "5.4.1",
                                        "Remote": false
                                },
                                {
                                        "Name": "cephfs",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "zfs",
                                        "Version": "2.1.2-1ubuntu3",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "15.2.14",
                                        "Remote": true
                                }
                        ]
                }
        }
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/images/aliases/ubuntu-minio-22.04"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "description": "",
                "target": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
                "name": "ubuntu-minio-22.04",
                "type": "container"
        }
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "auto_update": false,
                "properties": {
                        "description": "IAC image: ubuntu-minio-22.04"
                },
                "public": false,
                "expires_at": "2022-05-26T08:51:17+01:00",
                "profiles": [
                        "default"
                ],
                "aliases": [
                        {
                                "name": "ubuntu-minio-22.04",
                                "description": ""
                        }
                ],
                "architecture": "x86_64",
                "cached": false,
                "filename": "",
                "fingerprint": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
                "size": 188141771,
                "type": "container",
                "created_at": "2022-05-07T17:34:21.514701392Z",
                "last_used_at": "2022-05-07T17:34:45.49451231Z",
                "uploaded_at": "2022-05-07T17:34:21.516895766Z"
        }
DEBUG  [2022-05-08T06:46:57+01:00] Connected to the websocket: wss://<IP_ADDRESS>:8443/1.0/events
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=POST url="https://<IP_ADDRESS>:8443/1.0/images"
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "auto_update": false,
                "properties": {
                        "description": "IAC image: ubuntu-minio-22.04"
                },
                "public": false,
                "expires_at": "2022-05-26T08:51:17+01:00",
                "profiles": null,
                "filename": "",
                "source": {
                        "alias": "",
                        "certificate": "",
                        "protocol": "",
                        "server": "",
                        "image_type": "",
                        "mode": "push",
                        "type": "",
                        "url": "",
                        "name": "",
                        "fingerprint": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
                        "secret": "",
                        "project": ""
                },
                "compression_algorithm": "",
                "aliases": null
        }
DEBUG  [2022-05-08T06:46:57+01:00] Got operation from LXD
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "id": "452b81cf-ec74-4d9c-883e-29e299ac4fac",
                "class": "token",
                "description": "Image download token",
                "created_at": "2022-05-08T06:46:57.896860241+01:00",
                "updated_at": "2022-05-08T06:46:57.896860241+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "images": [
                                "/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0"
                        ]
                },
                "metadata": {
                        "aliases": null,
                        "expires_at": "2022-05-26T08:51:17+01:00",
                        "properties": {
                                "description": "IAC image: ubuntu-minio-22.04"
                        },
                        "public": false,
                        "secret": "fbc575da2c7bc2c17eb6fa742796997998146a9500d7db1c3699775bc681cbe8"
                },
                "may_cancel": true,
                "err": "",
                "location": "none"
        }
DEBUG  [2022-05-08T06:46:57+01:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0/export"
DEBUG  [2022-05-08T06:46:57+01:00]
        {
                "target": "https://<IP_ADDRESS>:8443",
                "secret": "fbc575da2c7bc2c17eb6fa742796997998146a9500d7db1c3699775bc681cbe8",
                "certificate": "-----BEGIN CERTIFICATE-----\nMIICAjCCAYmgAwIBAgIQN4N01EkRTWt3KmQdy46muDAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGJhY2t1\ncDAeFw0yMjA1MDcyMjI2MzNaFw0zMjA1MDQyMjI2MzNaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAYmFja3VwMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAERizSl/gpUEExmNc27xXKpHIlqQOOWNh0oNhj+4Bgf0Sf\nHV6/ggxBIKGxzZvwGp/shDU2PONlwnATOt2OUUGVVVlKcZLleP5DV9Ku82ty1Y8k\ntbRYE4nGq55NRLBBXMvno2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZiYWNrdXCHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwUZ1NZCLyKl6mfWjg\nZdRl6Ga7UwXVxzaw+YVmHFvZdDDslJWDODpRfHktYOcwmarWAjAmHKRMdv6Z2Ro3\nQgYeHZ1DkVsK7XX9HKHmHuUNmmafeeVmyyuKiztkKSl1h/8ty5U=\n-----END CERTIFICATE-----\n",
                "aliases": [
                        {
                                "name": "ubuntu-minio-22.04",
                                "description": ""
                        }
                ],
                "project": "default",
                "profiles": null
        }
DEBUG  [2022-05-08T06:46:58+01:00] Got operation from LXD
DEBUG  [2022-05-08T06:46:58+01:00]
        {
                "id": "f3472697-cdb2-4489-9761-3f9ae2ca8d1f",
                "class": "task",
                "description": "Downloading image",
                "created_at": "2022-05-08T06:46:58.185805989+01:00",
                "updated_at": "2022-05-08T06:46:58.185805989+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": null,
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DEBUG  [2022-05-08T06:46:58+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/operations/f3472697-cdb2-4489-9761-3f9ae2ca8d1f"
DEBUG  [2022-05-08T06:46:58+01:00] Got response struct from LXD
DEBUG  [2022-05-08T06:46:58+01:00]
        {
                "id": "f3472697-cdb2-4489-9761-3f9ae2ca8d1f",
                "class": "task",
                "description": "Downloading image",
                "created_at": "2022-05-08T06:46:58.185805989+01:00",
                "updated_at": "2022-05-08T06:46:58.185805989+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": null,
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DEBUG  [2022-05-08T06:48:19+01:00] Sending request to LXD                        etag= method=DELETE url="https://<IP_ADDRESS>:8443/1.0/operations/452b81cf-ec74-4d9c-883e-29e299ac4fac"
Error: Post "https://<IP_ADDRESS>:8443/1.0/images?project=default": net/http: timeout awaiting response headers

Does using relay mode help by the way? Ive tried reproducing using tc to simulate a slower network so the transfer takes over 30s and have been able to reproduce.

yes relay mode completes without errors

Is it possible to get the output of 'lxc monitor --type=logging --pretty' on source and target servers when you run the command. I'd like to see where the delay is occurring.

SOURCE MONITOR:
DEBUG  [2022-05-09T13:39:21+01:00] Event listener server handler started         listener=6facecee-8fc1-42d7-9f39-0f7e4d3c15c9 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=stuart
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/images/aliases/archlinux-base-current username=stuart
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/images/4d142148795f0eb41fd6f638c613e12ca5914c189127f918b763ab381947960b username=stuart
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=stuart
DEBUG  [2022-05-09T13:39:32+01:00] Event listener server handler started         listener=9bfd2080-d2a0-4f0b-981c-e4490723c241 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/images/4d142148795f0eb41fd6f638c613e12ca5914c189127f918b763ab381947960b/export username=stuart
DEBUG  [2022-05-09T13:39:32+01:00] Connecting to a remote LXD over HTTPS
DEBUG  [2022-05-09T13:39:32+01:00] Sending request to LXD                        etag= method=GET url="https://<IP_ADDRESS>:8443/1.0"
DEBUG  [2022-05-09T13:39:32+01:00] Got response struct from LXD
DEBUG  [2022-05-09T13:39:32+01:00]
        {
                "config": null,
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "untrusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": null,
                        "architectures": null,
                        "certificate": "",
                        "certificate_fingerprint": "",
                        "driver": "",
                        "driver_version": "",
                        "firewall": "",
                        "kernel": "",
                        "kernel_architecture": "",
                        "kernel_features": null,
                        "kernel_version": "",
                        "lxc_features": null,
                        "os_name": "",
                        "os_version": "",
                        "project": "",
                        "server": "",
                        "server_clustered": false,
                        "server_event_mode": "",
                        "server_name": "",
                        "server_pid": 0,
                        "server_version": "",
                        "storage": "",
                        "storage_version": "",
                        "storage_supported_drivers": null
                }
        }
DEBUG  [2022-05-09T13:39:32+01:00] New task Operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e
DEBUG  [2022-05-09T13:39:32+01:00] Started task operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/2313a3e6-77a1-4270-95c2-4483f85caa4e username=stuart
DEBUG  [2022-05-09T13:43:12+01:00] Failure for task operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e: Post "https://<IP_ADDRESS>:8443/1.0/images?project=default": net/http: timeout awaiting response headers
DEBUG  [2022-05-09T13:43:13+01:00] Event listener server handler stopped         listener=9bfd2080-d2a0-4f0b-981c-e4490723c241 local=/var/snap/lxd/common/lxd/unix.socket remote=@

DESTINATION MONITOR:
DEBUG  [2022-05-09T13:39:24+01:00] Event listener server handler started         listener=06f44bbc-1d78-46f0-af18-20369e26bf10 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-09T13:39:31+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:31+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip="<IP_ADDRESS>:37056" method=GET protocol=tls url=/1.0 username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:39:31+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip="<IP_ADDRESS>:37058" method=GET protocol=tls url=/1.0/events username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:39:31+01:00] Event listener server handler started         listener=4b20a751-b8fa-4b9e-b721-295a62d577d8 local="<IP_ADDRESS>:8443" remote="<IP_ADDRESS>:37058"
DEBUG  [2022-05-09T13:39:32+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip="<IP_ADDRESS>:37060" method=POST protocol=tls url=/1.0/images username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:39:32+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:32+01:00] New token Operation: 61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG  [2022-05-09T13:39:32+01:00] Started token operation: 61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG  [2022-05-09T13:39:32+01:00] Allowing untrusted GET                        ip="<IP_ADDRESS>:37062" url=/1.0
DEBUG  [2022-05-09T13:39:32+01:00] Allowing untrusted POST                       ip="<IP_ADDRESS>:37064" url="/1.0/images?project=default"
DEBUG  [2022-05-09T13:39:32+01:00] Cancelling operation                          class=token operation=61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG  [2022-05-09T13:39:32+01:00] Cancelled operation                           class=token operation=61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG  [2022-05-09T13:43:13+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:43:13+01:00] Handling API request                          ip="<IP_ADDRESS>:37066" method=DELETE protocol=tls url=/1.0/operations/61a3897d-c899-476c-85db-3ceee2c14e15 username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:43:13+01:00] Database error                                err="Operation not found"
DEBUG  [2022-05-09T13:43:13+01:00] Event listener server handler stopped         listener=4b20a751-b8fa-4b9e-b721-295a62d577d8 local="<IP_ADDRESS>:8443" remote="<IP_ADDRESS>:37058"

<IP_ADDRESS> is the source wireguard interface (with routing rules in /etc/systemd/network/99-wg0.network)
<IP_ADDRESS>/24 is the destination LXD subnet
<IP_ADDRESS>/24 is the source LXD subnet
Only trusted ip's are allowed to connect to the remote wireguard (allowed at the  ingress hook level in nftables so there shouldn't be any mangling by the firewall)

Hi, I'm seeing some issues with similar errors that I've never had before when creating lxd container on Fedora 34.  Started in the last few days:  "timeout awaiting response headers"
Creating oel84d10
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate "local:oel84d10": Instance not found
Creating oel84d10
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate "local:oel84d10": Instance not found
Creating oel84d10
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate "local:oel84d10": Instance not found
Creating oel84d10
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed to validate "local:oel84d10": Instance not found
Creating oel84d10

@gstanden can you rerun command with --debug and paste output please.

[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug
DEBUG  [2022-05-09T20:03:51-05:00] Connecting to a local LXD over a Unix socket
DEBUG  [2022-05-09T20:03:51-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-09T20:03:51-05:00] Got response struct from LXD
DEBUG  [2022-05-09T20:03:51-05:00]
	{
		"config": {
			"cluster.https_address": "<IP_ADDRESS>:8443",
			"core.https_address": "<IP_ADDRESS>:8443",
			"core.trust_password": true
		},
		"api_extensions": [
			"storage_zfs_remove_snapshots",
			"container_host_shutdown_timeout",
			"container_stop_priority",
			"container_syscall_filtering",
			"auth_pki",
			"container_last_used_at",
			"etag",
			"patch",
			"usb_devices",
			"https_allowed_credentials",
			"image_compression_algorithm",
			"directory_manipulation",
			"container_cpu_time",
			"storage_zfs_use_refquota",
			"storage_lvm_mount_options",
			"network",
			"profile_usedby",
			"container_push",
			"container_exec_recording",
			"certificate_update",
			"container_exec_signal_handling",
			"gpu_devices",
			"container_image_properties",
			"migration_progress",
			"id_map",
			"network_firewall_filtering",
			"network_routes",
			"storage",
			"file_delete",
			"file_append",
			"network_dhcp_expiry",
			"storage_lvm_vg_rename",
			"storage_lvm_thinpool_rename",
			"network_vlan",
			"image_create_aliases",
			"container_stateless_copy",
			"container_only_migration",
			"storage_zfs_clone_copy",
			"unix_device_rename",
			"storage_lvm_use_thinpool",
			"storage_rsync_bwlimit",
			"network_vxlan_interface",
			"storage_btrfs_mount_options",
			"entity_description",
			"image_force_refresh",
			"storage_lvm_lv_resizing",
			"id_map_base",
			"file_symlinks",
			"container_push_target",
			"network_vlan_physical",
			"storage_images_delete",
			"container_edit_metadata",
			"container_snapshot_stateful_migration",
			"storage_driver_ceph",
			"storage_ceph_user_name",
			"resource_limits",
			"storage_volatile_initial_source",
			"storage_ceph_force_osd_reuse",
			"storage_block_filesystem_btrfs",
			"resources",
			"kernel_limits",
			"storage_api_volume_rename",
			"macaroon_authentication",
			"network_sriov",
			"console",
			"restrict_devlxd",
			"migration_pre_copy",
			"infiniband",
			"maas_network",
			"devlxd_events",
			"proxy",
			"network_dhcp_gateway",
			"file_get_symlink",
			"network_leases",
			"unix_device_hotplug",
			"storage_api_local_volume_handling",
			"operation_description",
			"clustering",
			"event_lifecycle",
			"storage_api_remote_volume_handling",
			"nvidia_runtime",
			"container_mount_propagation",
			"container_backup",
			"devlxd_images",
			"container_local_cross_pool_handling",
			"proxy_unix",
			"proxy_udp",
			"clustering_join",
			"proxy_tcp_udp_multi_port_handling",
			"network_state",
			"proxy_unix_dac_properties",
			"container_protection_delete",
			"unix_priv_drop",
			"pprof_http",
			"proxy_haproxy_protocol",
			"network_hwaddr",
			"proxy_nat",
			"network_nat_order",
			"container_full",
			"candid_authentication",
			"backup_compression",
			"candid_config",
			"nvidia_runtime_config",
			"storage_api_volume_snapshots",
			"storage_unmapped",
			"projects",
			"candid_config_key",
			"network_vxlan_ttl",
			"container_incremental_copy",
			"usb_optional_vendorid",
			"snapshot_scheduling",
			"snapshot_schedule_aliases",
			"container_copy_project",
			"clustering_server_address",
			"clustering_image_replication",
			"container_protection_shift",
			"snapshot_expiry",
			"container_backup_override_pool",
			"snapshot_expiry_creation",
			"network_leases_location",
			"resources_cpu_socket",
			"resources_gpu",
			"resources_numa",
			"kernel_features",
			"id_map_current",
			"event_location",
			"storage_api_remote_volume_snapshots",
			"network_nat_address",
			"container_nic_routes",
			"rbac",
			"cluster_internal_copy",
			"seccomp_notify",
			"lxc_features",
			"container_nic_ipvlan",
			"network_vlan_sriov",
			"storage_cephfs",
			"container_nic_ipfilter",
			"resources_v2",
			"container_exec_user_group_cwd",
			"container_syscall_intercept",
			"container_disk_shift",
			"storage_shifted",
			"resources_infiniband",
			"daemon_storage",
			"instances",
			"image_types",
			"resources_disk_sata",
			"clustering_roles",
			"images_expiry",
			"resources_network_firmware",
			"backup_compression_algorithm",
			"ceph_data_pool_name",
			"container_syscall_intercept_mount",
			"compression_squashfs",
			"container_raw_mount",
			"container_nic_routed",
			"container_syscall_intercept_mount_fuse",
			"container_disk_ceph",
			"virtual-machines",
			"image_profiles",
			"clustering_architecture",
			"resources_disk_id",
			"storage_lvm_stripes",
			"vm_boot_priority",
			"unix_hotplug_devices",
			"api_filtering",
			"instance_nic_network",
			"clustering_sizing",
			"firewall_driver",
			"projects_limits",
			"container_syscall_intercept_hugetlbfs",
			"limits_hugepages",
			"container_nic_routed_gateway",
			"projects_restrictions",
			"custom_volume_snapshot_expiry",
			"volume_snapshot_scheduling",
			"trust_ca_certificates",
			"snapshot_disk_usage",
			"clustering_edit_roles",
			"container_nic_routed_host_address",
			"container_nic_ipvlan_gateway",
			"resources_usb_pci",
			"resources_cpu_threads_numa",
			"resources_cpu_core_die",
			"api_os",
			"container_nic_routed_host_table",
			"container_nic_ipvlan_host_table",
			"container_nic_ipvlan_mode",
			"resources_system",
			"images_push_relay",
			"network_dns_search",
			"container_nic_routed_limits",
			"instance_nic_bridged_vlan",
			"network_state_bond_bridge",
			"usedby_consistency",
			"custom_block_volumes",
			"clustering_failure_domains",
			"resources_gpu_mdev",
			"console_vga_type",
			"projects_limits_disk",
			"network_type_macvlan",
			"network_type_sriov",
			"container_syscall_intercept_bpf_devices",
			"network_type_ovn",
			"projects_networks",
			"projects_networks_restricted_uplinks",
			"custom_volume_backup",
			"backup_override_name",
			"storage_rsync_compression",
			"network_type_physical",
			"network_ovn_external_subnets",
			"network_ovn_nat",
			"network_ovn_external_routes_remove",
			"tpm_device_type",
			"storage_zfs_clone_copy_rebase",
			"gpu_mdev",
			"resources_pci_iommu",
			"resources_network_usb",
			"resources_disk_address",
			"network_physical_ovn_ingress_mode",
			"network_ovn_dhcp",
			"network_physical_routes_anycast",
			"projects_limits_instances",
			"network_state_vlan",
			"instance_nic_bridged_port_isolation",
			"instance_bulk_state_change",
			"network_gvrp",
			"instance_pool_move",
			"gpu_sriov",
			"pci_device_type",
			"storage_volume_state",
			"network_acl",
			"migration_stateful",
			"disk_state_quota",
			"storage_ceph_features",
			"projects_compression",
			"projects_images_remote_cache_expiry",
			"certificate_project",
			"network_ovn_acl",
			"projects_images_auto_update",
			"projects_restricted_cluster_target",
			"images_default_architecture",
			"network_ovn_acl_defaults",
			"gpu_mig",
			"project_usage",
			"network_bridge_acl",
			"warnings",
			"projects_restricted_backups_and_snapshots",
			"clustering_join_token",
			"clustering_description",
			"server_trusted_proxy",
			"clustering_update_cert",
			"storage_api_project",
			"server_instance_driver_operational",
			"server_supported_storage_drivers",
			"event_lifecycle_requestor_address",
			"resources_gpu_usb",
			"clustering_evacuation",
			"network_ovn_nat_address",
			"network_bgp",
			"network_forward",
			"custom_volume_refresh",
			"network_counters_errors_dropped",
			"metrics",
			"image_source_project",
			"clustering_config",
			"network_peer",
			"linux_sysctl",
			"network_dns",
			"ovn_nic_acceleration",
			"certificate_self_renewal",
			"instance_project_move",
			"storage_volume_project_move",
			"cloud_init",
			"network_dns_nat",
			"database_leader",
			"instance_all_projects",
			"clustering_groups",
			"ceph_rbd_du",
			"instance_get_full",
			"qemu_metrics",
			"gpu_mig_uuid",
			"event_project",
			"clustering_evacuation_live",
			"instance_allow_inconsistent_copy",
			"network_state_ovn",
			"storage_volume_api_filtering",
			"image_restrictions",
			"storage_zfs_export",
			"network_dns_records",
			"storage_zfs_reserve_space",
			"network_acl_log",
			"storage_zfs_blocksize",
			"metrics_cpu_seconds",
			"instance_snapshot_never",
			"certificate_token",
			"instance_nic_routed_neighbor_probe",
			"event_hub",
			"agent_nic_config",
			"projects_restricted_intercept",
			"metrics_authentication",
			"images_target_project",
			"cluster_migration_inconsistent_copy",
			"cluster_ovn_chassis",
			"container_syscall_intercept_sched_setscheduler",
			"storage_lvm_thinpool_metadata_size",
			"storage_volume_state_total",
			"instance_file_head",
			"instances_nic_host_name",
			"image_copy_profile",
			"container_syscall_intercept_sysinfo",
			"clustering_evacuation_mode"
		],
		"api_status": "stable",
		"api_version": "1.0",
		"auth": "trusted",
		"public": false,
		"auth_methods": [
			"tls"
		],
		"environment": {
			"addresses": [
				"<IP_ADDRESS>:8443"
			],
			"architectures": [
				"x86_64",
				"i686"
			],
			"certificate": "-----BEGIN CERTIFICATE-----\nMIICAzCCAYmgAwIBAgIQW1WMY5wWjGGh84OVrZUNxTAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2\nMTAeFw0yMjA1MDkyMjI0NDhaFw0zMjA1MDYyMjI0NDhaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAE4p63DAbe7w7iED5wf+Ra54b9WEStQbvB/bwnSq9tXOiz\nqIVoqWxxxAho8HHASCVAVMdkY0vVGtffqiFAe7de+zH+FFGKZysqHNXoigDOtTVO\nHY3E5IMAMCfG4YNesffao2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIwErX8iCV4YL6/gLyA\nSCDaVqTkge0PPsvlHVCgrzes569GZlPDndA9W9X3yDItHcAiAjEAkbMZOaB61PfR\nwAbUZ7YbNbVw8FSFGXMfVNdowhwyTod+oz+CstDbGd5QHnDHgHW/\n-----END CERTIFICATE-----\n",
			"certificate_fingerprint": "4797e3981caf2829d8c7be9fc1768f5ebc6d7bfc49e6bacd3fe20fcc678c07f3",
			"driver": "lxc",
			"driver_version": "4.0.12",
			"firewall": "nftables",
			"kernel": "Linux",
			"kernel_architecture": "x86_64",
			"kernel_features": {
				"idmapped_mounts": "true",
				"netnsid_getifaddrs": "true",
				"seccomp_listener": "true",
				"seccomp_listener_continue": "true",
				"shiftfs": "false",
				"uevent_injection": "true",
				"unpriv_fscaps": "true"
			},
			"kernel_version": "5.17.5-100.fc34.x86_64",
			"lxc_features": {
				"cgroup2": "true",
				"core_scheduling": "true",
				"devpts_fd": "true",
				"idmapped_mounts_v2": "true",
				"mount_injection_file": "true",
				"network_gateway_device_route": "true",
				"network_ipvlan": "true",
				"network_l2proxy": "true",
				"network_phys_macvlan_mtu": "true",
				"network_veth_router": "true",
				"pidfd": "true",
				"seccomp_allow_deny_syntax": "true",
				"seccomp_notify": "true",
				"seccomp_proxy_send_notify_fd": "true"
			},
			"os_name": "Fedora",
			"os_version": "34",
			"project": "default",
			"server": "lxd",
			"server_clustered": true,
			"server_event_mode": "full-mesh",
			"server_name": "f34sv1",
			"server_pid": 160714,
			"server_version": "5.1",
			"storage": "zfs | dir",
			"storage_version": "2.1.4-1 | 1",
			"storage_supported_drivers": [
				{
					"Name": "zfs",
					"Version": "2.1.4-1",
					"Remote": false
				},
				{
					"Name": "ceph",
					"Version": "15.2.14",
					"Remote": true
				},
				{
					"Name": "btrfs",
					"Version": "5.4.1",
					"Remote": false
				},
				{
					"Name": "cephfs",
					"Version": "15.2.14",
					"Remote": true
				},
				{
					"Name": "dir",
					"Version": "1",
					"Remote": false
				},
				{
					"Name": "lvm",
					"Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
					"Remote": false
				}
			]
		}
	}
Creating oel
DEBUG  [2022-05-09T20:03:51-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-09T20:03:51-05:00] Connected to the websocket: ws://unix.socket/1.0/events
DEBUG  [2022-05-09T20:03:51-05:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2022-05-09T20:03:51-05:00]
	{
		"architecture": "",
		"config": {},
		"devices": {},
		"ephemeral": false,
		"profiles": [
			"olxc_sx1a"
		],
		"stateful": false,
		"description": "",
		"name": "oel",
		"source": {
			"type": "image",
			"certificate": "",
			"alias": "oracle//amd64",
			"server": "https://images.linuxcontainers.org",
			"protocol": "simplestreams",
			"mode": "pull",
			"allow_inconsistent": false
		},
		"instance_type": "",
		"type": "container"
	}
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
[orabuntu@f34sv1 ~]$

Thanks @gstanden can you also provide output of lxc monitor --type=logging --pretty on the server when you're running that command. I'm trying to see where the hold up in response headers are. Thanks

Trying to reproduce now.

@gstanden is it possible to get access to this server?

@itoffshore is it possible to get access to this server for testing?

@tomponline here you go ... also, it worked fine this time!  of course - it always does when it "knows" we are watching :) ... did you make any changes ... if so they seem to be working.
[orabuntu@f34sv1 ~]$ lxc monitor --type=logging --pretty
DEBUG  [2022-05-10T08:41:23-05:00] Event listener server handler started         listener=28f116df-c67a-48a6-bd00-b940aeb74256 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:41:30-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:41:30-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:41:30-05:00] Completed heartbeat round                     duration=28.118492ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:41:33-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:41:33-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:41:33-05:00] Event listener server handler started         listener=ed32291c-37e0-45de-bac2-83eebdccb9da local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:41:33-05:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/instances username=orabuntu
DEBUG  [2022-05-10T08:41:33-05:00] Responding to instance create
DEBUG  [2022-05-10T08:41:33-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-10T08:41:35-05:00] New task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:35-05:00] Started task operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:35-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b username=orabuntu
DEBUG  [2022-05-10T08:41:35-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-10T08:41:35-05:00] Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG  [2022-05-10T08:41:35-05:00] Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG  [2022-05-10T08:41:35-05:00] Database error                                err="Image not found"
DEBUG  [2022-05-10T08:41:35-05:00] Database error                                err="Image not found"
INFO   [2022-05-10T08:41:35-05:00] Downloading image                             alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=baa997f7-c06a-40e6-a27f-01396220ba6b server="https://images.linuxcontainers.org" trigger=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:35-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:36-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:36-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:37-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:37-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:39-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:39-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:40-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:40-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:41:40-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:41:40-05:00] Completed heartbeat round                     duration=5.473403ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:41:40-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:43-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:43-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:44-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:44-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:45-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:45-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:46-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:46-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:47-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:47-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:48-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:48-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:49-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:49-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:50-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:50-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:41:50-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:41:50-05:00] Completed heartbeat round                     duration=21.990183ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:41:50-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:53-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:53-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:54-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:54-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:56-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:56-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:58-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:59-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:59-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:00-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:00-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:42:00-05:00] Completed heartbeat round                     duration=22.543699ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:01-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:01-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:03-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:03-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:04-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:04-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:06-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:06-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:07-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:07-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:08-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:08-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:09-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:09-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:10-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:42:10-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:10-05:00] Completed heartbeat round                     duration=23.069247ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:11-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:11-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:12-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:12-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:15-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:16-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:16-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:18-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:18-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:20-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:20-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:42:20-05:00] Completed heartbeat round                     duration=10.655645ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:42:20-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:20-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:21-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:21-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
INFO   [2022-05-10T08:42:22-05:00] Image downloaded                              alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=baa997f7-c06a-40e6-a27f-01396220ba6b server="https://images.linuxcontainers.org" trigger=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:22-05:00] Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG  [2022-05-10T08:42:22-05:00] Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066"
DEBUG  [2022-05-10T08:42:22-05:00] Instance operation lock created               action=create instance=oel84d10 project=default reusable=false
INFO   [2022-05-10T08:42:22-05:00] Creating container                            ephemeral=false instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:22-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:22-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:22-05:00] Adding device                                 device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:22-05:00] Adding device                                 device=root instance=oel84d10 instanceType=container project=default type=disk
INFO   [2022-05-10T08:42:22-05:00] Created container                             ephemeral=false instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:22-05:00] CreateInstanceFromImage started               instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:22-05:00] EnsureImage started                           fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:22-05:00] Database error                                err="Storage pool volume not found"
INFO   [2022-05-10T08:42:22-05:00] Image unpack started                          imageFile=/var/snap/lxd/common/lxd/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:22-05:00] Mounted ZFS dataset                           dev=olxd-0933/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local
DEBUG  [2022-05-10T08:42:22-05:00] Running filler function                       dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local
DEBUG  [2022-05-10T08:42:22-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b
INFO   [2022-05-10T08:42:25-05:00] Image unpack stopped                          imageFile=/var/snap/lxd/common/lxd/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:30-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:30-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:42:30-05:00] Completed heartbeat round                     duration=16.878987ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:42:31-05:00] Unmounted ZFS dataset                         dev=olxd-0933/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:31-05:00] EnsureImage finished                          fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:31-05:00] Checking volume size                          instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:31-05:00] Set new volume size                           instance=oel84d10 project=default size=
DEBUG  [2022-05-10T08:42:31-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:31-05:00] Unmounted ZFS dataset                         dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG  [2022-05-10T08:42:31-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:31-05:00] CreateInstanceFromImage finished              instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:31-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:32-05:00] Instance operation lock finished              action=create err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:32-05:00] Unmounted ZFS dataset                         dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG  [2022-05-10T08:42:32-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:32-05:00] Success for task operation: baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:32-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:32-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG  [2022-05-10T08:42:32-05:00] New task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG  [2022-05-10T08:42:32-05:00] Instance operation lock created               action=start instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:32-05:00] Start started                                 instance=oel84d10 instanceType=container project=default stateful=false
DEBUG  [2022-05-10T08:42:32-05:00] Started task operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG  [2022-05-10T08:42:32-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/3b33fee1-63ac-4fa8-893f-d2ce36852ae6 username=orabuntu
DEBUG  [2022-05-10T08:42:32-05:00] MountInstance started                         instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:32-05:00] MountInstance finished                        instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:32-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:32-05:00] Updated metadata for task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG  [2022-05-10T08:42:32-05:00] Container idmap changed, remapping            instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:33-05:00] Updated metadata for task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
DEBUG  [2022-05-10T08:42:33-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:33-05:00] Starting device                               device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:33-05:00] Starting device                               device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG  [2022-05-10T08:42:33-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
INFO   [2022-05-10T08:42:33-05:00] Starting container                            action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG  [2022-05-10T08:42:33-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:33-05:00] Skipping unmount as in use                    driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG  [2022-05-10T08:42:33-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstart?project=default" username=root
DEBUG  [2022-05-10T08:42:33-05:00] Scheduler: container oel84d10 started: re-balancing
DEBUG  [2022-05-10T08:42:33-05:00] Success for task operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6
INFO   [2022-05-10T08:42:33-05:00] Started container                             action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG  [2022-05-10T08:42:33-05:00] Start finished                                instance=oel84d10 instanceType=container project=default stateful=false
DEBUG  [2022-05-10T08:42:33-05:00] Instance operation lock finished              action=start err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:33-05:00] Event listener server handler stopped         listener=ed32291c-37e0-45de-bac2-83eebdccb9da local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:40-05:00] Starting heartbeat round                      local="<IP_ADDRESS>:8443" mode=normal
DEBUG  [2022-05-10T08:42:40-05:00] Heartbeat updating local raft members         members="[{{1 <IP_ADDRESS>:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:40-05:00] Completed heartbeat round                     duration=46.256365ms local="<IP_ADDRESS>:8443"
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Event listener server handler started         listener=f94d1b77-620e-4b9e-bd7a-309f7cafbfcc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/instances/oel84d10/exec username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] New websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Waiting for exec websockets to connect
DEBUG  [2022-05-10T08:42:48-05:00] Started websocket operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=46adcf39146218420742abbaeec8c24cd50f3ec5aa1884dc4094620474ea653b" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=ca9356d7c0ea567b1a811df478974372876d21629215f0491112e824e84419f0" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=89b3ab2d10db416808cf3d7f2f952be88d061b1a546ff278e86a6eaf7c2e167e" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=3925d5e121c6dbd82d84cc4f6d2ef318bc7f16936b84a39d17374ab86942db07" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Retrieved PID of executing child process      attachedPid=110357 instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket started                 PID=110357 instance=oel84d10 interactive=false number=1 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec control handler started                  PID=110357 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket started                 PID=110357 instance=oel84d10 interactive=false number=0 project=default
DEBUG  [2022-05-10T08:42:49-05:00] WebsocketRecvStream got message barrier
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket finished                PID=110357 instance=oel84d10 interactive=false number=0 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Instance process started                      PID=110357 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket started                 PID=110357 instance=oel84d10 interactive=false number=2 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec control handler finished                 PID=110357 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Success for websocket operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:49-05:00] Instance process stopped                      PID=110357 exitStatus=0 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket finished                PID=110357 instance=oel84d10 interactive=false number=1 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket finished                PID=110357 instance=oel84d10 interactive=false number=2 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Updated metadata for websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler stopped         listener=f94d1b77-620e-4b9e-bd7a-309f7cafbfcc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler started         listener=40b2f8fd-916b-41f7-985a-31f7e94f2cec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] New task Operation: f30078f7-3d05-4f79-9312-5e6fc9889839
DEBUG  [2022-05-10T08:42:49-05:00] Instance operation lock created               action=update instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:49-05:00] Started task operation: f30078f7-3d05-4f79-9312-5e6fc9889839
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/f30078f7-3d05-4f79-9312-5e6fc9889839 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:49-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:49-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Skipping unmount as in use                    driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG  [2022-05-10T08:42:49-05:00] Success for task operation: f30078f7-3d05-4f79-9312-5e6fc9889839
DEBUG  [2022-05-10T08:42:49-05:00] Instance operation lock finished              action=update err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler stopped         listener=40b2f8fd-916b-41f7-985a-31f7e94f2cec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler started         listener=d170a28f-c83a-4d6d-a7b7-adbf52695cdc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Started task operation: c7e0508c-3d04-4351-8b79-78b744f9fad7
DEBUG  [2022-05-10T08:42:49-05:00] Instance operation lock created               action=stop instance=oel84d10 project=default reusable=true
DEBUG  [2022-05-10T08:42:49-05:00] Shutdown started                              instance=oel84d10 instanceType=container project=default timeout=-1s
DEBUG  [2022-05-10T08:42:49-05:00] New task Operation: c7e0508c-3d04-4351-8b79-78b744f9fad7
DEBUG  [2022-05-10T08:42:49-05:00] Shutdown request sent to instance             instance=oel84d10 instanceType=container project=default
INFO   [2022-05-10T08:42:49-05:00] Shutting down container                       action=shutdown created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default timeout=-1s used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/c7e0508c-3d04-4351-8b79-78b744f9fad7 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstopns?netns=%2Fproc%2F109702%2Ffd%2F4&project=default&target=stop" username=root
DEBUG  [2022-05-10T08:42:49-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:49-05:00] Stopping device                               device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:50-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstop?project=default&target=stop" username=root
DEBUG  [2022-05-10T08:42:50-05:00] Container stopped, cleaning up                instance=oel84d10 instanceType=container project=default