Example 258 fortigate perimeter package deploy procedure/verify for core lz unmanaged client

The client requires deployment of the #258 perimeter on top of the core lz with additional DNS zones TBD
Document and reuse on top of #420 and and #421
gcloud deployment testing later 2022 - #158
See pre-kcc deployment run in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/158
gcloud reference install: https://github.com/fortinet/fortigate-tutorial-gcp/issues/1
See

#78
#168
#166
#177
#207
#573
todo:
add client-landing-zone on top of existing core-landing-zone before hub-env

Package Inventory

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-landing-zone
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/project/hub-env/fortigate
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone

edit
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/setters.yaml
via
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/README.md

know oci and gitops are the core - and I agree having the code in github/ado/gitlab/csr is preferred - but some clients have requested the easier kpt option and it should be there as a base deployment option - since mid 2022
To be fair the base case deployment option is actually pure kubernetes krm yaml like in https://cloud.google.com/config-connector/docs/how-to/getting-started
see original gitops docs in https://cloud.google.com/anthos-config-management/docs/concepts/config-controller-overview and https://cloud.google.com/anthos-config-management/docs/how-to/unstructured-repo
see original kpt docs in https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt
20230814: revisit kls
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ export PROJECT_ID=kcc-kls-cluster3
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ ls
core-landing-zone  setters.yaml
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ rm -rf core-landing-zone/
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@main
Package "core-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 + 52f93a3...ea2e57f main       -> origin/main  (forced update)
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ cp setters.yaml core-landing-zone/

re-add kpt documentation at the end of section 2
see #409
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/docs/landing-zone-v2#2-create-your-landing-zone
needs
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/solutions/core-landing-zone/0.3.0/docs/landing-zone-v2#kpt
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$   kpt live init core-landing-zone --namespace config-control
initializing "resourcegroup.yaml" data (namespace: config-control)...success

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render core-landing-zone
Package "core-landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[FAIL] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.1s
  Results:
    [error]: failed to apply setters: values for setters [${platform-and-component-log-bucket}] must be provided
  Stderr:
    "values for setters [${platform-and-component-log-bucket}] must be providedvalues for setters [${platform-and-component-log-bucket}] must be provided"
  Exit code: 1

  fix: did not have the latest version of setters.yaml - updated

  mirroring changes to my local repo from the core-landing-zone kpt folder download

root_@cloudshell:~/kcc-kls/lz-20230803-gh/pubsec-declarative-toolkit (kcc-kls-cluster3)$ git diff
diff --git a/solutions/core-landing-zone/setters.yaml b/solutions/core-landing-zone/setters.yaml
index f3168d3..ca53ae4 100644
--- a/solutions/core-landing-zone/setters.yaml
+++ b/solutions/core-landing-zone/setters.yaml
@@ -14,10 +14,11 @@
 #########
 apiVersion: v1
 kind: ConfigMap
-metadata:
+metadata: # kpt-merge: /setters
   name: setters
   annotations:
     config.kubernetes.io/local-config: "true"
+    internal.kpt.dev/upstream-identifier: '|ConfigMap|default|setters'
 data:
   ##########################
   # Instructions
@@ -38,9 +39,9 @@ data:
   # General Settings Values
   ##########################
   #
-  org-id: "0000000000"
-  lz-folder-id: '0000000000'
-  billing-id: "AAAAAA-BBBBBB-CCCCCC"
+  org-id: "15....993"
+  lz-folder-id: '444....332'
+  billing-id: "01....833"
   #
   ##########################
   # Management Project
@@ -48,8 +49,8 @@ data:
   #
   # This is the project where the config controller instance is running
   # Values can be viewed in the Project Dashboard
-  management-project-id: management-project-12345
-  management-project-number: "0000000000"
+  management-project-id: kcc-kls-cluster3
+  management-project-number: "53....547"
   management-namespace: config-control
   #
   ##########################
@@ -68,14 +69,14 @@ data:
   # org/org-policies/essentialcontacts-allowed-contact-domains.yaml
   # this setting MUST be changed
   allowed-contact-domains: |
-    -<EMAIL_ADDRESS>+    -<EMAIL_ADDRESS>   #
   # a list of directory customer IDs from which users can be added to IAM policies, see YAML file for more info:
   # org/org-policies/iam-allowed-policy-member-domains.yaml
   # this setting MUST be changed to include the GCP org's directory ID and any other directory containing users that will need IAM roles assigned
   # run 'gcloud organizations list' as described in https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains#retrieving_customer_id
   allowed-policy-domain-members: |
-    - "DIRECTORY_CUSTOMER_ID"
+    - "C0....m1"
   #
   # a list of allowed projects, folders, networks for VPC peering, see YAML file for more info:
   # org/org-policies/compute-restrict-vpc-peering.yaml
@@ -87,13 +88,13 @@ data:
   # Logging
   ##########################
   #
-  logging-project-id: logging-project-12345
+  logging-project-id: logging-project-kls
   #
   # Log Buckets
   # Security Logs Bucket
-  security-log-bucket: security-log-bucket-12345
+  security-log-bucket: security-log-bucket-kls
   # Platform and Component Log Bucket
-  platform-and-component-log-bucket: platform-and-component-log-bucket-12345
+  platform-and-component-log-bucket: platform-and-component-log-bucket-kls
   #
   # Retention settings
   # Set the number of days to retain logs in Cloud Logging buckets
@@ -110,8 +111,9 @@ data:
   # DNS
   ##########################
   #
-  dns-project-id: dns-project-12345
-  dns-name: "example.com."
+  dns-project-id: dns-project-kls
+  # the appended . is required by google cloud domain zones
+  dns-name: "kcc.landing.systems."

Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.BadRequest",
    "fieldViolations": [
      {
        "description": "Invalid value: [under:organizations/ORGANIZATION_ID]",
        "field": "policy.list_policy.allowed_values[0]"
      }
    ]
  }
]
, badRequest
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-skip-default-network-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-trusted-image-projects is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-can-ip-forward is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/compute-vm-external-ip-access is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/essentialcontacts-allowed-contact-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/gcp-restrict-resource-locations is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-allowed-policy-member-domains is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/iam-disable-service-account-key-creation is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/sql-restrict-public-ip is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-public-access-prevention is Current: Resource is Current
inventory-85852139/resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/policies/storage-uniform-bucket-level-access is Current: Resource is Current

working additions
add
organizations/123456789012/roles/FortigateSdnViewer

add packages
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/client-landing-zone@main
Package "client-landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 + ea2e57f...10ca23d main       -> origin/main  (forced update)
Adding package "solutions/client-landing-zone".

Fetched 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/client-setup@main
Package "client-setup":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
Adding package "solutions/client-setup".

Fetched 1 package(s).

Current status (deploying hub) - move from https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/445
kcc.landing.zone
root_@cloudshell:~/kcc-kls/lz-20230803$ ls
client-landing-zone  client-setup  core-landing-zone  hub-env  setters.yaml
root_@cloudshell:~/kcc-kls/lz-20230803$

Known Issues Workarounds

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/fortigates.md#known-issues
turn down security on the management VPC - turn back on public access for googleapis.com public access without putting in a private service connect endpoint like
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/468

restarting hub-env adjustment
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kpt fn render hub-env
Package "hub-env":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.1s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dmu-admin1-hub-kls"
    ...(102 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/search-replace:v0.2.0"
[PASS] "gcr.io/kpt-fn/search-replace:v0.2.0" in 1.7s
  Results:
    [info]: no matches

Successfully executed 2 function(s) in 1 package(s).
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$  kpt live apply hub-env --reconcile-timeout=2m --output=table

Restarting clean org fortigate install for monday
obrien.industries
    1  export<EMAIL_ADDRESS>    2  gcloud organizations get-iam-policy roles/resourcemanager.organizationAdmin --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
    3  gcloud organizations list
    4  gcloud organizations list | grep ID
    5  gcloud organizations list --format="get(name)"
    6  export DOMAIN=obrien.industries
    7  ORG_ID=$(gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN)
    8  echo $ORG_ID
    9  gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN
   10  export ORG_ID=459065442144
   11  gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$EMAIL" --flatten="bindings[].members" --format="table(bindings.role)"
   12  gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$EMAIL --role=iam.serviceAccountTokenCreator
   13  gcloud organizations add-iam-policy-binding $ORG_ID  --member=serviceAccount:$EMAIL --role=roles/iam.serviceAccountTokenCreator
   14  gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/iam.serviceAccountTokenCreator
   15  gcloud organizations add-iam-policy-binding $ORG_ID  --member=user:$EMAIL --role=roles/billing.projectManager
   16  mkdir kcc-oi
   17  cd kcc-oi
   18  mkdir github
   19  mkdir kpt
   20  cd github
   21  git clone https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git
   22  git clone https://github.com/ssc-spc-ccoe-cei/gcp-tools.git
   23  cd gcp-tools/scripts/bootstrap/
   24  cp .env.sample kcc.env
   25  export PROJECT_ID=kcc-oi
   26  gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default
   27  gcloud config set project "${PROJECT_ID}"
   28  echo $ORG_ID
   29  export ROOT_FOLDER=kcc
   30  gcloud resource-manager folders create --display-name=$ROOT_FOLDER --organization=$ORG_ID
   31  export BILLING_ID=$(gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)' | sed 's/.*\///')
   32  echo $BILLING_ID
   33  gcloud alpha billing projects describe $PROJECT_ID '--format=value(billingAccountName)'
   34  echo $PROJECT_ID
   35  export BILLING_ID=014479-806359-2F5F85
   36  gcloud beta billing projects link "$PROJECT_ID" --billing-account "$BILLING_ID"
   37  ls
   38  chmod 777 setup-kcc.sh
   39  ./setup-kcc.sh -af kcc.env
   40  history
   41  gcloud config set project kcc-oi
   42  cd kcc-oi/
   43  ls
   44  cd kpt/
   45  PACKAGE="solutions/gatekeeper-policies"
   46  VERSION=$(curl -s $URL | jq -r ".\"$PACKAGE\"")
   47  URL=https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   48  VERSION=$(curl -s $URL | jq -r ".\"$PACKAGE\"")
   49  curl -s $URL
   50  https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
   51  curl -s $URL | jq -r ".\"$PACKAGE\""
   52  VERSION=main
   53  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
   54  PACKAGE="solutions/core-landing-zone"
   55  kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/${PACKAGE}@${VERSION}
   56  gcloud organizations list
   57  kpt live init core-landing-zone --namespace config-control
   58  kpt fn render core-landing-zone
   59  gcloud config set project kcc-oi
   60  cd kcc-oi/
   61  cd kpt/
   62  ls
   63  kpt fn render core-landing-zone
   64  gcloud config set project kcc-oi
   65  gcloud config set project kcc-oi-cluster
   66  kubectl edit validatingwebhookconfiguration/gatekeeper-validating-webhook-configuration
   67  kubectl get nodes
   68  gcloud config set project kcc-oi
   69  kubectl get nodes
   70  kubectl get pods --all-namespaces
   71  gcloud anthos config controller get-credentials krmapihost-kcc-oi  --location northamerica-northeast1
   72  gcloud config set project kcc-oi-cluster
   73  gcloud anthos config controller get-credentials krmapihost-kcc-oi  --location northamerica-northeast1
   74  gcloud anthos config controller get-credentials kcc-oi  --location northamerica-northeast1
   75  kubens config-control
   76  kubectl get pods --all-namespaces
   77  cd kcc-oi/kpt/
   78  ls
   79  kpt live apply core-landing-zone --reconcile-timeout=2m --output=table
   80  gcloud config set project kcc-oi-cluster
   81  kubectl get pods
   82  ls
   83  ls -la
   84  ls -la .kube/
   85  cat .kube/config
   86  kubectl get nodes
   87  history
   88  gcloud config set project kcc-oi
   89  cd kcc-oi/github/
   90  cd gcp-tools/scripts/bootstrap/
   91  ./setup-kcc.sh -afp kcc.env
   92  kubectl get nodes
   93  history

https://github.com/ssc-spc-ccoe-cei/gcp-tools/pull/53
and
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/568
editupdate: found them in the new 2nd script
https://github.com/ssc-spc-ccoe-cei/gcp-tools/blob/main/scripts/bootstrap/configure-kcc-access.sh#L35
Issue is that the access script assumes rootsync usage - it leaves out the kpt optionI recommend we put the yakima service account role additions back to the generic setup script.

automation test target env root at landing.systems
dev target obrien.enginnering
partial fortigate kcc.landing.systems
pull/run
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/gh446-hub

add fix for hub-env setters.yaml missing org-id in #573
should unblock anything under the SDN custom role

next fix
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions
   Message:               Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1<EMAIL_ADDRESS>for project "dmu-admin1-hub-kls"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1<EMAIL_ADDRESS>for project \"dmu-admin1-hub-kls\"" both failed. Final error: Error applying IAM policy for project "dmu-admin1-hub-kls": Error setting IAM policy for project "dmu-admin1-hub-kls": googleapi: Error 400: Service account<EMAIL_ADDRESS>does not exist., badRequest

obrien.industries via full script

clean cluster
clean core-landing-zone

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa

 Warning  UpdateFailed  36s (x9 over 6m44s)  iamserviceaccount-controller  Update call failed: error applying desired state: summary: Error creating service account: googleapi: Error 403: Permission 'iam.serviceAccounts.create' denied on resource (or it may not exist).

checking sa permissions for
iam.serviceAccounts.create

wrong project number

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$  gcloud projects list --filter="kcc-oi-629" '--format=value(PROJECT_NUMBER)'
1020702930278

adding single service account admin role to the yakima gke account to test the reconcile

re-kpt
UpdateFailed: 8
UpToDate: 36
Context "gke_kcc-oi-629_northamerica-northeast1_krmapihost-kcc-oi3" modified.
Active namespace is "config-control".
NAME                                                                AGE   READY   STATUS         STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin          25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin       25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin      25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin       25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin    25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin      25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin           25m   False   UpdateFailed   25m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin   25m   False   UpdateFailed   25m

NAME                                                                                AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   25m   True    UpToDate   5m28s
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          25m   True    UpToDate   5m28s
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            25m   True    UpToDate   5m27s
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         25m   True    UpToDate   4m21s
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           25m   True    UpToDate   4m15s
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           25m   True    UpToDate   5m26s

NAME                                                                                AGE   READY   STATUS     STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             25m   True    UpToDate   25m
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          25m   True    UpToDate   4m21s
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   25m   True    UpToDate   5m3s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   25m   True    UpToDate   4m43s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        25m   True    UpToDate   5m9s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          25m   True    UpToDate   4m43s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 25m   True    UpToDate   4m42s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     25m   True    UpToDate   4m38s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          25m   True    UpToDate   4m54s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       25m   True    UpToDate   4m37s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     25m   True    UpToDate   4m54s
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 25m   True    UpToDate   2m54s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    25m   True    UpToDate   4m54s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 25m   True    UpToDate   4m15s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 25m   True    UpToDate   4m14s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                25m   True    UpToDate   4m14s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   25m   True    UpToDate   4m14s
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              25m   True    UpToDate   4m14s

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   25m   True    UpToDate   5m29s
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          25m   True    UpToDate   5m29s
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            25m   True    UpToDate   5m28s
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         25m   True    UpToDate   5m28s
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           25m   True    UpToDate   4m15s
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           25m   True    UpToDate   5m27s

NAME                                                                         AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-accesscontextmanager   25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudbilling           25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-cloudresourcemanager   25m   True    UpToDate   25m
service.serviceusage.cnrm.cloud.google.com/kcc-oi-629-serviceusage           25m   True    UpToDate   25m

Added fix for missing yakima roles
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/578
Checking folders in clz
https://cloud.google.com/config-connector/docs/reference/resource-docs/resourcemanager/folder
see
kubectl get crds --selector cnrm.cloud.google.com/managed-by-kcc=true

getting status via
https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe folder audits
Error from server (NotFound): folders.resourcemanager.cnrm.cloud.google.com "audits" not found

working for known up services
ichael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi-629)$ kubectl describe IAMServiceAccount logging-sa
Name:         logging-sa

rerun
onfig-con  IAMServiceAccount/networking-sa           Skipped       Current                 Ready                                     2h      Resource is Current
config-con  IAMServiceAccount/policies-sa             Skipped       Current                 Ready                                     2h      Resource is Current
config-con  IAMServiceAccount/projects-sa             Skipped       Current                 Ready                                     2h      Resource is Current
config-con  Service/kcc-oi-629-accesscontextmanager   Skipped       Current                 Ready                                     2h      Resource is Current
config-con  Service/kcc-oi-629-cloudbilling           Skipped       Current                 Ready                                     2h      Resource is Current
config-con  Service/kcc-oi-629-cloudresourcemanager   Skipped       Current                 Ready                                     2h      Resource is Current
config-con  Service/kcc-oi-629-serviceusage           Skipped       Current                 Ready                                     2h      Resource is Current
gatekeeper  ConfigConnectorContext/configconnectorco  Skipped       Current                 <None>                                    2h      status.healthy is true
hierarchy   ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -
hierarchy   RoleBinding/allow-folders-resource-refer  Skipped       Unknown                 -                                         -
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Skipped       Unknown                 -                                         -
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Skipped       Unknown                 -                                         -
hierarchy   RoleBinding/allow-hierarchy-resource-ref  Skipped       Unknown                 -                                         -
hierarchy   Folder/audits                             Skipped       Unknown                 -                                         -
hierarchy   Folder/clients                            Skipped       Unknown                 -                                         -
hierarchy   Folder/services                           Skipped       Unknown                 -                                         -
hierarchy   Folder/services-infrastructure            Skipped       Unknown                 -                                         -
logging     ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -
logging     LoggingLogBucket/platform-and-component-  Skipped       Unknown                 -                                         -
logging     LoggingLogBucket/security-log-bucket-oi   Skipped       Unknown                 -                                         -
logging     LoggingLogSink/logging-project-oi-securi  Skipped       Unknown                 -                                         -
logging     LoggingLogSink/mgmt-project-cluster-disa  Skipped       Unknown                 -                                         -
logging     LoggingLogSink/mgmt-project-cluster-plat  Skipped       Unknown                 -                                         -
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -
logging     LoggingLogSink/platform-and-component-se  Skipped       Unknown                 -                                         -
logging     RoleBinding/allow-logging-resource-refer  Skipped       Unknown                 -                                         -
networking  ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -
networking  DNSManagedZone/dns-project-oi-standard-c  Skipped       Unknown                 -                                         -
policies    ConfigConnectorContext/configconnectorco  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-disable-gu  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-disable-ne  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-disable-se  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-disable-vp  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-require-os  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-require-sh  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-require-sh  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-restrict-l  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-restrict-s  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-restrict-v  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-skip-defau  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-trusted-im  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-vm-can-ip-  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/compute-vm-externa  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/essentialcontacts-  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/gcp-restrict-resou  Skipped       Unknown                 -                                         -
policies    ResourceManagerPolicy/iam-allowed-policy  Skipped       Unknown                 -                                         -

update folder id (was older one)
lz-folder-id: '871256537757'
deleted lz, reran kpt
GKE cluster crashed
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/580
projects    Project/logging-project-oi                Skipped       Unknown                 -                                         -
projects    Service/dns-project-oi-dns                Skipped       Unknown                 -                                         -
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1821bb2]

goroutine 757 [running]:
k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.(*Unstructured).GetResourceVersion(...)
        /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.9/pkg/apis/meta/v1/unstructured/unstructured.go:282
github.com/GoogleContainerTools/kpt/pkg/live.(*InventoryResourceGroup).ApplyWithPrune(0xc001692ea0?, {0x221cea0?, 0xc002758530?}, {0x223fd10?, 0xc001fa02a0?}, 0x1, {0x68?, 0x92?, 0x1ca6d60?})
        /home/runner/work/kpt/kpt/pkg/live/inventoryrg.go:299 +0x112
sigs.k8s.io/cli-utils/pkg/inventory.(*ClusterClient).Replace(0xc00118c980, {0x22398e8, 0xc000e89040}, {0xc00072b800?, 0x2c, 0x2c}, {0xc000e16000, 0x68, 0x92}, 0x0)
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/inventory/inventory-client.go:202 +0x830
sigs.k8s.io/cli-utils/pkg/apply/task.(*DeleteOrUpdateInvTask).updateInventory(0xc000cba300, 0xc000dd8400)
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/apply/task/inv_set_task.go:161 +0x294a
sigs.k8s.io/cli-utils/pkg/apply/task.(*DeleteOrUpdateInvTask).Start.func1()
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/apply/task/inv_set_task.go:54 +0x3f
created by sigs.k8s.io/cli-utils/pkg/apply/task.(*DeleteOrUpdateInvTask).Start in goroutine 181
        /home/runner/go/pkg/mod/sigs.k8s.io/cli-utils@v0.35.0/pkg/apply/task/inv_set_task.go:49 +0x67

recycling cluster

CC get resource status https://cloud.google.com/config-connector/docs/how-to/monitoring-your-resources#listing_all_resources
CFT https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/tree/master/config-connector/solutions
KCC https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions
Simple PubSub example to get familiar with https://cloud.google.com/config-connector/docs/how-to/getting-started
KPT reference https://kpt.dev/reference/schema/plan/
Google Architecture Docs on config controller and config sync https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt

LZ inventory - kls - folders ok
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt pkg tree core-landing-zone
Package "core-landing-zone"
├── [Kptfile]  Kptfile core-landing-zone
├── [resourcegroup.yaml]  ResourceGroup config-control/inventory-49021548
├── [setters.yaml]  ConfigMap setters
├── audits
│   ├── [folder.yaml]  Folder hierarchy/audits
│   └── logging-project
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/platform-and-component-log-bucket-oi
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/security-log-bucket-oi
│       ├── [project-iam.yaml]  IAMAuditConfig projects/logging-project-data-access-log-config
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-infra-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/security-log-bucket-writer-permissions
│       └── [project.yaml]  Project projects/logging-project-oi
├── clients
│   └── [folder.yaml]  Folder hierarchy/clients
├── services
│   ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-log-sink
│   ├── [folder.yaml]  Folder hierarchy/services
│   └── services-infrastructure
│       ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-infra-log-sink
│       ├── [folder.yaml]  Folder hierarchy/services-infrastructure
│       └── dns-project
│           ├── [dns.yaml]  DNSManagedZone networking/dns-project-oi-standard-core-public-dns
│           ├── [project.yaml]  Project projects/dns-project-oi
│           └── [services.yaml]  Service projects/dns-project-oi-dns
├── mgmt-project
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-disable-default-bucket
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink
│   ├── [services.yaml]  Service config-control/kcc-oi-629-accesscontextmanager
│   ├── [services.yaml]  Service config-control/kcc-oi-629-cloudbilling
│   ├── [services.yaml]  Service config-control/kcc-oi-629-cloudresourcemanager
│   ├── [services.yaml]  Service config-control/kcc-oi-629-serviceusage
│   └── org-policies
│       └── [compute-require-shielded-vm-except-mgmt-project.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm-except-mgt-project
├── namespaces
│   ├── [gatekeeper-system.yaml]  IAMServiceAccount config-control/gatekeeper-admin-sa
│   ├── [gatekeeper-system.yaml]  IAMPolicyMember config-control/gatekeeper-admin-sa-metric-writer-permissions
│   ├── [gatekeeper-system.yaml]  IAMPartialPolicy config-control/gatekeeper-admin-sa-workload-identity-binding
│   ├── [gatekeeper-system.yaml]  ConfigConnectorContext gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [hierarchy.yaml]  Namespace hierarchy
│   ├── [hierarchy.yaml]  IAMServiceAccount config-control/hierarchy-sa
│   ├── [hierarchy.yaml]  IAMPolicyMember config-control/hierarchy-sa-folderadmin-permissions
│   ├── [hierarchy.yaml]  IAMPartialPolicy config-control/hierarchy-sa-workload-identity-binding
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-folders-resource-reference-to-logging
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-config-control
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-policies
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-projects
│   ├── [hierarchy.yaml]  ConfigConnectorContext hierarchy/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  Namespace logging
│   ├── [logging.yaml]  IAMServiceAccount config-control/logging-sa
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-bigqueryadmin-permissions
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-logadmin-permissions
│   ├── [logging.yaml]  IAMPartialPolicy config-control/logging-sa-workload-identity-binding
│   ├── [logging.yaml]  RoleBinding logging/allow-logging-resource-reference-from-projects
│   ├── [logging.yaml]  ConfigConnectorContext logging/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-editor-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-serviceaccountadmin-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-orgroleadmin-permissions
│   ├── [networking.yaml]  Namespace networking
│   ├── [networking.yaml]  IAMServiceAccount config-control/networking-sa
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-dns-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-networkadmin-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-security-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-service-control-org-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-servicedirectoryeditor-permissions
│   ├── [networking.yaml]  IAMPartialPolicy config-control/networking-sa-workload-identity-binding
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-xpnadmin-permissions
│   ├── [networking.yaml]  ConfigConnectorContext networking/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [policies.yaml]  Namespace policies
│   ├── [policies.yaml]  IAMServiceAccount config-control/policies-sa
│   ├── [policies.yaml]  IAMPolicyMember config-control/policies-sa-orgpolicyadmin-permissions
│   ├── [policies.yaml]  IAMPartialPolicy config-control/policies-sa-workload-identity-binding
│   ├── [policies.yaml]  ConfigConnectorContext policies/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [projects.yaml]  Namespace projects
│   ├── [projects.yaml]  IAMServiceAccount config-control/projects-sa
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-billinguser-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectcreator-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectdeleter-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectiamadmin-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectmover-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-serviceusageadmin-permissions
│   ├── [projects.yaml]  IAMPartialPolicy config-control/projects-sa-workload-identity-binding
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-logging
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-networking
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-policies
│   └── [projects.yaml]  ConfigConnectorContext projects/configconnectorcontext.core.cnrm.cloud.google.com
└── org
    ├── [org-sink.yaml]  LoggingLogSink logging/logging-project-oi-security-sink
    ├── custom-roles
    │   ├── [gke-firewall-admin.yaml]  IAMCustomRole config-control/gke-firewall-admin
    │   ├── [tier2-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier2-dnsrecord-admin
    │   ├── [tier2-vpcpeering-admin.yaml]  IAMCustomRole config-control/tier2-vpcpeering-admin
    │   ├── [tier3-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier3-dnsrecord-admin
    │   ├── [tier3-firewallrule-admin.yaml]  IAMCustomRole config-control/tier3-firewallrule-admin
    │   ├── [tier3-subnetwork-admin.yaml]  IAMCustomRole config-control/tier3-subnetwork-admin
    │   ├── [tier3-vpcsc-admin.yaml]  IAMCustomRole config-control/tier3-vpcsc-admin
    │   └── [tier4-secretmanager-admin.yaml]  IAMCustomRole config-control/tier4-secretmanager-admin
    └── org-policies
        ├── [compute-disable-guest-attribute-access.yaml]  ResourceManagerPolicy policies/compute-disable-guest-attribute-access
        ├── [compute-disable-nested-virtualization.yaml]  ResourceManagerPolicy policies/compute-disable-nested-virtualization
        ├── [compute-disable-serial-port-access.yaml]  ResourceManagerPolicy policies/compute-disable-serial-port-access
        ├── [compute-disable-vpc-external-ipv6.yaml]  ResourceManagerPolicy policies/compute-disable-vpc-external-ipv6
        ├── [compute-require-os-login.yaml]  ResourceManagerPolicy policies/compute-require-os-login
        ├── [compute-require-shielded-vm.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm
        ├── [compute-restrict-load-balancer-creation-for-types.yaml]  ResourceManagerPolicy policies/compute-restrict-load-balancer-creation-for-types
        ├── [compute-restrict-shared-vpc-lien-removal.yaml]  ResourceManagerPolicy policies/compute-restrict-shared-vpc-lien-removal
        ├── [compute-restrict-vpc-peering.yaml]  ResourceManagerPolicy policies/compute-restrict-vpc-peering
        ├── [compute-skip-default-network-creation.yaml]  ResourceManagerPolicy policies/compute-skip-default-network-creation
        ├── [compute-trusted-image-projects.yaml]  ResourceManagerPolicy policies/compute-trusted-image-projects
        ├── [compute-vm-can-ip-forward.yaml]  ResourceManagerPolicy policies/compute-vm-can-ip-forward
        ├── [compute-vm-external-ip-access.yaml]  ResourceManagerPolicy policies/compute-vm-external-ip-access
        ├── [essentialcontacts-allowed-contact-domains.yaml]  ResourceManagerPolicy policies/essentialcontacts-allowed-contact-domains
        ├── [gcp-resource-locations.yaml]  ResourceManagerPolicy policies/gcp-restrict-resource-locations
        ├── [iam-allowed-policy-member-domains.yaml]  ResourceManagerPolicy policies/iam-allowed-policy-member-domains
        ├── [iam-disable-service-account-key-creation.yaml]  ResourceManagerPolicy policies/iam-disable-service-account-key-creation
        ├── [sql-restrict-public-ip.yaml]  ResourceManagerPolicy policies/sql-restrict-public-ip
        ├── [storage-public-access-prevention.yaml]  ResourceManagerPolicy policies/storage-public-access-prevention
        └── [storage-uniform-bucket-level-access.yaml]  ResourceManagerPolicy policies/storage-uniform-bucket-level-access

LZ inventory - oi - no folders
michael@cloudshell:~/kcc-oi/kpt (kcc-oi)$ kpt pkg tree core-landing-zone
Package "core-landing-zone"
├── [Kptfile]  Kptfile core-landing-zone
├── [resourcegroup.yaml]  ResourceGroup config-control/inventory-49021548
├── [setters.yaml]  ConfigMap setters
├── audits
│   ├── [folder.yaml]  Folder hierarchy/audits
│   └── logging-project
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/platform-and-component-log-bucket-oi
│       ├── [cloud-logging-buckets.yaml]  LoggingLogBucket logging/security-log-bucket-oi
│       ├── [project-iam.yaml]  IAMAuditConfig projects/logging-project-data-access-log-config
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-infra-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/platform-and-component-services-log-bucket-writer-permissions
│       ├── [project-iam.yaml]  IAMPartialPolicy projects/security-log-bucket-writer-permissions
│       └── [project.yaml]  Project projects/logging-project-oi
├── clients
│   └── [folder.yaml]  Folder hierarchy/clients
├── services
│   ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-log-sink
│   ├── [folder.yaml]  Folder hierarchy/services
│   └── services-infrastructure
│       ├── [folder-sink.yaml]  LoggingLogSink logging/platform-and-component-services-infra-log-sink
│       ├── [folder.yaml]  Folder hierarchy/services-infrastructure
│       └── dns-project
│           ├── [dns.yaml]  DNSManagedZone networking/dns-project-oi-standard-core-public-dns
│           ├── [project.yaml]  Project projects/dns-project-oi
│           └── [services.yaml]  Service projects/dns-project-oi-dns
├── mgmt-project
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-disable-default-bucket
│   ├── [project-sink.yaml]  LoggingLogSink logging/mgmt-project-cluster-platform-and-component-log-sink
│   ├── [services.yaml]  Service config-control/kcc-oi-629-accesscontextmanager
│   ├── [services.yaml]  Service config-control/kcc-oi-629-cloudbilling
│   ├── [services.yaml]  Service config-control/kcc-oi-629-cloudresourcemanager
│   ├── [services.yaml]  Service config-control/kcc-oi-629-serviceusage
│   └── org-policies
│       └── [compute-require-shielded-vm-except-mgmt-project.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm-except-mgt-project
├── namespaces
│   ├── [gatekeeper-system.yaml]  IAMServiceAccount config-control/gatekeeper-admin-sa
│   ├── [gatekeeper-system.yaml]  IAMPolicyMember config-control/gatekeeper-admin-sa-metric-writer-permissions
│   ├── [gatekeeper-system.yaml]  IAMPartialPolicy config-control/gatekeeper-admin-sa-workload-identity-binding
│   ├── [gatekeeper-system.yaml]  ConfigConnectorContext gatekeeper-system/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [hierarchy.yaml]  Namespace hierarchy
│   ├── [hierarchy.yaml]  IAMServiceAccount config-control/hierarchy-sa
│   ├── [hierarchy.yaml]  IAMPolicyMember config-control/hierarchy-sa-folderadmin-permissions
│   ├── [hierarchy.yaml]  IAMPartialPolicy config-control/hierarchy-sa-workload-identity-binding
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-folders-resource-reference-to-logging
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-config-control
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-policies
│   ├── [hierarchy.yaml]  RoleBinding hierarchy/allow-hierarchy-resource-reference-from-projects
│   ├── [hierarchy.yaml]  ConfigConnectorContext hierarchy/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [logging.yaml]  Namespace logging
│   ├── [logging.yaml]  IAMServiceAccount config-control/logging-sa
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-bigqueryadmin-permissions
│   ├── [logging.yaml]  IAMPolicyMember config-control/logging-sa-logadmin-permissions
│   ├── [logging.yaml]  IAMPartialPolicy config-control/logging-sa-workload-identity-binding
│   ├── [logging.yaml]  RoleBinding logging/allow-logging-resource-reference-from-projects
│   ├── [logging.yaml]  ConfigConnectorContext logging/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-editor-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-management-project-serviceaccountadmin-permissions
│   ├── [management-namespace.yaml]  IAMPolicyMember config-control/config-control-sa-orgroleadmin-permissions
│   ├── [networking.yaml]  Namespace networking
│   ├── [networking.yaml]  IAMServiceAccount config-control/networking-sa
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-dns-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-networkadmin-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-security-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-service-control-org-permissions
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-servicedirectoryeditor-permissions
│   ├── [networking.yaml]  IAMPartialPolicy config-control/networking-sa-workload-identity-binding
│   ├── [networking.yaml]  IAMPolicyMember config-control/networking-sa-xpnadmin-permissions
│   ├── [networking.yaml]  ConfigConnectorContext networking/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [policies.yaml]  Namespace policies
│   ├── [policies.yaml]  IAMServiceAccount config-control/policies-sa
│   ├── [policies.yaml]  IAMPolicyMember config-control/policies-sa-orgpolicyadmin-permissions
│   ├── [policies.yaml]  IAMPartialPolicy config-control/policies-sa-workload-identity-binding
│   ├── [policies.yaml]  ConfigConnectorContext policies/configconnectorcontext.core.cnrm.cloud.google.com
│   ├── [projects.yaml]  Namespace projects
│   ├── [projects.yaml]  IAMServiceAccount config-control/projects-sa
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-billinguser-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectcreator-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectdeleter-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectiamadmin-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-projectmover-permissions
│   ├── [projects.yaml]  IAMPolicyMember config-control/projects-sa-serviceusageadmin-permissions
│   ├── [projects.yaml]  IAMPartialPolicy config-control/projects-sa-workload-identity-binding
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-logging
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-networking
│   ├── [projects.yaml]  RoleBinding projects/allow-projects-resource-reference-from-policies
│   └── [projects.yaml]  ConfigConnectorContext projects/configconnectorcontext.core.cnrm.cloud.google.com
└── org
    ├── [org-sink.yaml]  LoggingLogSink logging/logging-project-oi-security-sink
    ├── custom-roles
    │   ├── [gke-firewall-admin.yaml]  IAMCustomRole config-control/gke-firewall-admin
    │   ├── [tier2-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier2-dnsrecord-admin
    │   ├── [tier2-vpcpeering-admin.yaml]  IAMCustomRole config-control/tier2-vpcpeering-admin
    │   ├── [tier3-dnsrecord-admin.yaml]  IAMCustomRole config-control/tier3-dnsrecord-admin
    │   ├── [tier3-firewallrule-admin.yaml]  IAMCustomRole config-control/tier3-firewallrule-admin
    │   ├── [tier3-subnetwork-admin.yaml]  IAMCustomRole config-control/tier3-subnetwork-admin
    │   ├── [tier3-vpcsc-admin.yaml]  IAMCustomRole config-control/tier3-vpcsc-admin
    │   └── [tier4-secretmanager-admin.yaml]  IAMCustomRole config-control/tier4-secretmanager-admin
    └── org-policies
        ├── [compute-disable-guest-attribute-access.yaml]  ResourceManagerPolicy policies/compute-disable-guest-attribute-access
        ├── [compute-disable-nested-virtualization.yaml]  ResourceManagerPolicy policies/compute-disable-nested-virtualization
        ├── [compute-disable-serial-port-access.yaml]  ResourceManagerPolicy policies/compute-disable-serial-port-access
        ├── [compute-disable-vpc-external-ipv6.yaml]  ResourceManagerPolicy policies/compute-disable-vpc-external-ipv6
        ├── [compute-require-os-login.yaml]  ResourceManagerPolicy policies/compute-require-os-login
        ├── [compute-require-shielded-vm.yaml]  ResourceManagerPolicy policies/compute-require-shielded-vm
        ├── [compute-restrict-load-balancer-creation-for-types.yaml]  ResourceManagerPolicy policies/compute-restrict-load-balancer-creation-for-types
        ├── [compute-restrict-shared-vpc-lien-removal.yaml]  ResourceManagerPolicy policies/compute-restrict-shared-vpc-lien-removal
        ├── [compute-restrict-vpc-peering.yaml]  ResourceManagerPolicy policies/compute-restrict-vpc-peering
        ├── [compute-skip-default-network-creation.yaml]  ResourceManagerPolicy policies/compute-skip-default-network-creation
        ├── [compute-trusted-image-projects.yaml]  ResourceManagerPolicy policies/compute-trusted-image-projects
        ├── [compute-vm-can-ip-forward.yaml]  ResourceManagerPolicy policies/compute-vm-can-ip-forward
        ├── [compute-vm-external-ip-access.yaml]  ResourceManagerPolicy policies/compute-vm-external-ip-access
        ├── [essentialcontacts-allowed-contact-domains.yaml]  ResourceManagerPolicy policies/essentialcontacts-allowed-contact-domains
        ├── [gcp-resource-locations.yaml]  ResourceManagerPolicy policies/gcp-restrict-resource-locations
        ├── [iam-allowed-policy-member-domains.yaml]  ResourceManagerPolicy policies/iam-allowed-policy-member-domains
        ├── [iam-disable-service-account-key-creation.yaml]  ResourceManagerPolicy policies/iam-disable-service-account-key-creation
        ├── [sql-restrict-public-ip.yaml]  ResourceManagerPolicy policies/sql-restrict-public-ip
        ├── [storage-public-access-prevention.yaml]  ResourceManagerPolicy policies/storage-public-access-prevention
        └── [storage-uniform-bucket-level-access.yaml]  ResourceManagerPolicy policies/storage-uniform-bucket-level-access

Additions to triage since core-landing-zone was last working

Hold: remaining services on
kcc.landing.systems
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d

root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies | grep false
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         67d   False   DependencyNotFound   67d
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-primary-instance     67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-fgt-secondary-instance   67d   False   DependencyNotFound   67d
computeinstance.compute.cnrm.cloud.google.com/hub-management-instance      67d   False   DependencyNotFound   67d
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n projects | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n policies | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n hierarchy | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$ kubectl get gcp -n logging | grep False
root_@cloudshell:~/kcc-kls/lz-20230803 (kcc-kls-cluster3)$

obrien.industries

still in clz

landing.systems

creating

Triage a way to managed a full lz delete and redeploy - with cycled project names (will need random ids in the future)
Issue here is a previously unmet iam role on the log sink is causing issues with deletion
https://console.cloud.google.com/kubernetes/object/logging.cnrm.cloud.google.com/logginglogsinks/northamerica-northeast1/krmapihost-kcc-oi3/logging/mgmt-project-cluster-disable-default-bucket?apiVersion=v1beta1&project=kcc-oi-629&orgonly=true&supportedpurview=project
see #446
status:
  conditions:
  - lastTransitionTime: "2023-10-21T21:13:45Z"
    message: |
      Delete call failed: error fetching live state: error reading underlying resource: summary: Error when reading or editing Project Logging Sink _Default: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

Switched back to boot project kcc-oi
Start:<PHONE_NUMBER>
unique string: oi
REGION: northamerica-northeast1
NETWORK: kcc-ls-vpc
SUBNET: kcc-ls-sn
CLUSTER: kcc-oi4
Reusing project: kcc-oi-9428
CC_PROJECT_ID: kcc-oi-9428
BOOT_PROJECT_ID: kcc-oi
BILLING_ID:<PHONE_NUMBER>59-2F5F85
ORG_ID:<PHONE_NUMBER>44
Switching to KCC project kcc-oi-9428
Updated property [core/project].
wait 60 sec to let the GKE cluster stabilize 15 workloads
KCC_PROJECT_NUMBER:<PHONE_NUMBER>737
Context "gke_kcc-oi-9428_northamerica-northeast1_krmapihost-kcc-oi4" modified.
Active namespace is "config-control".
deploying core-landing-zone
get kpt release package solutions/core-landing-zone version 0.3.2
Package "core-landing-zone":
Fetching<EMAIL_ADDRESS>From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * tag               solutions/core-landing-zone/0.3.2 -> FETCH_HEAD
Adding package "solutions/core-landing-zone".

Fetched 1 package(s).
removing org/org-policies folder
kpt live init
initializing "resourcegroup.yaml" data (namespace: config-control)...success
kpt fn render
Package "core-landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 1.3s
  Results:
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] metadata.name: set field value to "security-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.projectRef.name: set field value to "logging-project2-oi"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.name: set field value to "platform-and-component-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.projectRef.name: set field value to "logging-project2-oi"
    [info] spec.locked: set field value to "false"
    [info] spec.retentionDays: set field value to "1"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] spec.bindings[0].members[0].memberFrom.logSinkRef.name: set field value to "logging-project2-oi-security-sink"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.resourceRef.name: set field value to "logging-project2-oi"
    [info] metadata.name: set field value to "logging-project2-oi"
    [info] spec.name: set field value to "logging-project2-oi"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] metadata.name: set field value to "dns-project2-oi-standard-core-public-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project2-oi"
    [info] spec.dnsName: set field value to "obrien.industries."
    [info] metadata.name: set field value to "dns-project2-oi"
    [info] spec.name: set field value to "dns-project2-oi"
    [info] spec.billingAccountRef.external: set field value to "014479-806359-2F5F85"
    [info] metadata.name: set field value to "dns-project2-oi-dns"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "dns-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/dns-project2-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] spec.folderRef.external: set field value to "716446322787"
    [info] spec.projectRef.external: set field value to "kcc-oi-9428"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/platform-and-component-log-bucket-oi"
    [info] spec.projectRef.external: set field value to "kcc-oi-9428"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/platform-and-component-log-bucket-oi"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/logging-project2-oi"
    [info] spec.projectRef.external: set field value to "kcc-oi-9428"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/_Default"
    [info] metadata.name: set field value to "kcc-oi-9428-cloudbilling"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.name: set field value to "kcc-oi-9428-cloudresourcemanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.name: set field value to "kcc-oi-9428-serviceusage"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.name: set field value to "kcc-oi-9428-accesscontextmanager"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "kcc-oi-9428"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[gatekeeper-system/gatekeeper-admin]"
    [info] spec.googleServiceAccount: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-hierarchy]"
    [info] spec.googleServiceAccount: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-logging]"
    [info] spec.googleServiceAccount: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "kcc-oi-9428"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "kcc-oi-9428"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-networking]"
    [info] spec.googleServiceAccount: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-policies]"
    [info] spec.googleServiceAccount: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "716446322787"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.resourceRef.external: set field value to "459065442144"
    [info] spec.member: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/project-id: set field value to "kcc-oi-9428"
    [info] spec.bindings[0].members[0].member: set field value to "serviceAccount:kcc-oi-9428.svc.id.goog[cnrm-system/cnrm-controller-manager-projects]"
    [info] spec.googleServiceAccount: set field value to<EMAIL_ADDRESS>    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.namespace: set field value to "config-control"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "459065442144"
    [info] metadata.name: set field value to "logging-project2-oi-security-sink"
    [info] metadata.annotations.config.kubernetes.io/depends-on: set field value to "logging.cnrm.cloud.google.com/namespaces/logging/LoggingLogBucket/security-log-bucket-oi"
    [info] spec.organizationRef.external: set field value to "459065442144"
    [info] spec.destination.loggingLogBucketRef.external: set field value to "logging.googleapis.com/projects/logging-project2-oi/locations/northamerica-northeast1/buckets/security-log-bucket-oi"

Successfully executed 1 function(s) in 1 package(s).
kpt live apply
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
namespace/hierarchy apply successful
namespace/logging apply successful
namespace/networking apply successful
namespace/policies apply successful
namespace/projects apply successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com apply successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin apply successful
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding apply successful
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager apply successful
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage apply successful
apply phase finished
reconcile phase started
namespace/hierarchy reconcile successful
namespace/logging reconcile successful
namespace/networking reconcile successful
namespace/policies reconcile successful
namespace/projects reconcile successful
configconnectorcontext.core.cnrm.cloud.google.com/configconnectorcontext.core.cnrm.cloud.google.com reconcile successful
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin reconcile pending
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding reconcile pending
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager reconcile pending
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-accesscontextmanager reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudbilling reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-cloudresourcemanager reconcile failed
service.serviceusage.cnrm.cloud.google.com/kcc-oi-9428-serviceusage reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa reconcile failed

Testing setters.yaml automation in 3rd env<EMAIL_ADDRESS>see #591
michael@cloudshell:~ (kcc-oi)$ gcloud organizations list --filter="${DIRECTORY_CUSTOMER_ID}" '--format=value(DIRECTORY_CUSTOMER_ID)'
C0...kc

Working hub-env automation
added ambiguity on the project-parent-folder
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/596

cat << EOF > ./${REL_SUB_PACKAGE}/setters-${REL_SUB_PACKAGE}.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: setters
  annotations:
    config.kubernetes.io/local-config: "true"
data:
  org-id: "${ORG_ID}"
  project-billing-id: "${BILLING_ID}"
  project-parent-folder: project-parent-folder
  hub-project-id: ${HUB_PROJECT_ID_PREFIX}-${PREFIX}
  hub-admin: ${HUB_ADMIN_GROUP_EMAIL}
  project-allowed-restrict-vpc-peering: |
    - under:organizations/${ORG_ID}
  project-allowed-vm-external-ip-access: |
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-a/instances/fgt-primary-instance"
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-b/instances/fgt-secondary-instance"
  project-allowed-vm-can-ip-forward: |
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-a/instances/fgt-primary-instance"
    - "projects/${HUB_PROJECT_ID_PREFIX}-${PREFIX}/zones/${REGION}-b/instances/fgt-secondary-instance"
  fgt-primary-image: ${FORTIGATE_PRIMARY_IMAGE}
  fgt-primary-license: |
    LICENSE
  fgt-secondary-image: ${FORTIGATE_SECONDARY_IMAGE}
  fgt-secondary-license: |
    LICENSE
EOF

triaging
error: invalid object: "projects_xxdmu-admin1-hub-oi5_resourcemanager.cnrm.cloud.google.com_Project": invalid "config.kubernetes.io/depends-on" annotation: external dependency: resourcemanager.cnrm.cloud.google.com/namespaces/projects/Project/xxdmu-admin1-hub-oi5 -> resourcemanager.cnrm.cloud.google.com/namespaces/hierarchy/Folder/services-infrastructure

missing parent folder
data:
  org-id: "459065442144"
  project-billing-id: "014479-806359-2F5F85"
  project-parent-folder: project-parent-folder

  set
    project-parent-folder: #{HUB_PROJECT_PARENT_FOLDER}
    via
  HUB_PROJECT_PARENT_FOLDER=services-infrastructure

seeing a hang now on "kpt live apply"
https://github.com/kptdev/kpt/issues/825
however a 2nd run restarted after 5 min
ichael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
inventory update started
inventory update finished
apply phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 apply successful
apply phase finished
reconcile phase started
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5 reconcile successful
reconcile phase finished
apply phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions apply successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project apply successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute apply successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns apply successful
apply phase finished
reconcile phase started
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile pending
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project reconcile successful
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project reconcile successful
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute reconcile successful

iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions reconcile successful
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions reconcile failed
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns reconcile successful
reconcile phase finished
apply phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc apply skipped: dependency apply reconcile failed: config-control_networking-sa-computeinstanceadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc apply successful
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc apply successful
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa apply skipped: dependency apply reconcile failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa apply skipped: dependency apply reconcile failed: config-control_networking-sa-serviceaccountadmin-permissions_iam.cnrm.cloud.google.com_IAMPolicyMember
apply phase finished
reconcile phase started
computedisk.compute.cnrm.cloud.google.com/hub-fgt-primary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-fgt-secondary-log-disk reconcile skipped
computedisk.compute.cnrm.cloud.google.com/hub-mgmt-data-disk reconcile skipped
computehttphealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-httphc reconcile skipped
computehealthcheck.compute.cnrm.cloud.google.com/hub-http-8008-hc reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc reconcile pending
iamserviceaccount.iam.cnrm.cloud.google.com/hub-fortigatesdn-sa reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/hub-managementvm-sa reconcile skipped

starting to come in but no hub project yet

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp
NAME                                                                  AGE     READY   STATUS     STATUS AGE
iamcustomrole.iam.cnrm.cloud.google.com/gke-firewall-admin            27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/hub-fortigatesdnreader-role   4m21s   True    UpToDate   4m20s
iamcustomrole.iam.cnrm.cloud.google.com/tier2-dnsrecord-admin         27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier2-vpcpeering-admin        27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-dnsrecord-admin         27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-firewallrule-admin      27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-subnetwork-admin        27h     True    UpToDate   26h
iamcustomrole.iam.cnrm.cloud.google.com/tier3-vpcsc-admin             27h     True    UpToDate   140m
iamcustomrole.iam.cnrm.cloud.google.com/tier4-secretmanager-admin     27h     True    UpToDate   26h

NAME                                                                                AGE   READY   STATUS     STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-workload-identity-binding   27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/hierarchy-sa-workload-identity-binding          27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/logging-sa-workload-identity-binding            27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/networking-sa-workload-identity-binding         27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/policies-sa-workload-identity-binding           27h   True    UpToDate   26h
iampartialpolicy.iam.cnrm.cloud.google.com/projects-sa-workload-identity-binding           27h   True    UpToDate   26h

NAME                                                                                AGE     READY   STATUS         STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-editor-permissions                27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-management-project-serviceaccountadmin-permissions   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/config-control-sa-orgroleadmin-permissions                             27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/gatekeeper-admin-sa-metric-writer-permissions                          27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/hierarchy-sa-folderadmin-permissions                                   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-computeinstanceadmin-permissions                             4m21s   True    UpToDate       4m7s
iampolicymember.iam.cnrm.cloud.google.com/hub-admin-iaptunnelresourceaccessor-permissions                        4m21s   True    UpToDate       4m6s
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-bigqueryadmin-permissions                                   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/logging-sa-logadmin-permissions                                        27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         4m20s   False   UpdateFailed   4m7s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-dns-permissions                                          27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-networkadmin-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-security-permissions                                     27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-service-control-org-permissions                          27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          4m20s   False   UpdateFailed   4m6s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           4m20s   False   UpdateFailed   4m5s
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-servicedirectoryeditor-permissions                       27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-xpnadmin-permissions                                     27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/policies-sa-orgpolicyadmin-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-billinguser-permissions                                    27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectcreator-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectdeleter-permissions                                 27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectiamadmin-permissions                                27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-projectmover-permissions                                   27h     True    UpToDate       26h
iampolicymember.iam.cnrm.cloud.google.com/projects-sa-serviceusageadmin-permissions                              27h     True    UpToDate       26h

NAME                                                              AGE   READY   STATUS     STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/gatekeeper-admin-sa   27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/hierarchy-sa          27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/logging-sa            27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/networking-sa         27h   True    UpToDate   26h
iamserviceaccount.iam.cnrm.cloud.google.com/policies-sa           27h   True    UpToDate   25h
iamserviceaccount.iam.cnrm.cloud.google.com/projects-sa           27h   True    UpToDate   26h

NAME                                                                          AGE   READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-accesscontextmanager   27h   True    UpToDate   26h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudbilling           27h   True    UpToDate   26h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-cloudresourcemanager   27h   True    UpToDate   26h
service.serviceusage.cnrm.cloud.google.com/kcc-oi-3552-serviceusage           27h   True    UpToDate   26h

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n projects
NAME                                                                              AGE   READY   STATUS     STATUS AGE
iamauditconfig.iam.cnrm.cloud.google.com/logging-project-data-access-log-config   26h   True    UpToDate   26h

NAME                                                                                AGE   READY   STATUS               STATUS AGE
iampartialpolicy.iam.cnrm.cloud.google.com/mgmt-project-cluster-platform-and-component-log-bucket-writer-permissions   26h   False   DependencyNotFound   26h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-infra-log-bucket-writer-permissions         26h   False   DependencyNotFound   26h
iampartialpolicy.iam.cnrm.cloud.google.com/platform-and-component-services-log-bucket-writer-permissions               26h   False   DependencyNotFound   26h
iampartialpolicy.iam.cnrm.cloud.google.com/security-log-bucket-writer-permissions                                      26h   False   DependencyNotFound   26h

NAME                                                                 AGE   READY   STATUS     STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/dns-project-oi5        26h   True    UpToDate   26h
project.resourcemanager.cnrm.cloud.google.com/logging-project-oi5    26h   True    UpToDate   26h
project.resourcemanager.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5   16m   True    UpToDate   5m10s

NAME                                                                      AGE    READY   STATUS     STATUS AGE
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-compute   5m7s   True    UpToDate   5m4s
service.serviceusage.cnrm.cloud.google.com/xxdmu-admin1-hub-oi5-dns       5m7s   True    UpToDate   4m43s

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n policies
NAME                                                                                AGE     READY   STATUS     STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-disable-serial-port-access-except-hub-project                  5m47s   True    UpToDate   5m46s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-hub-project                         5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-require-shielded-vm-except-mgt-project                         26h     True    UpToDate   26h
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-load-balancer-creation-for-types-except-hub-project   5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-restrict-vpc-peering-except-hub-project                        5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-trusted-image-projects-except-hub-project                      5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-can-ip-forward-except-hub-project                           5m46s   True    UpToDate   5m45s
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/compute-vm-external-ip-access-except-hub-project                       5m45s   True    UpToDate   5m44s

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n hierarchy
NAME                                                                   AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audits                    26h   True    UpToDate   26h
folder.resourcemanager.cnrm.cloud.google.com/clients                   26h   True    UpToDate   26h
folder.resourcemanager.cnrm.cloud.google.com/services                  26h   True    UpToDate   26h
folder.resourcemanager.cnrm.cloud.google.com/services-infrastructure   26h   True    UpToDate   26h

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                                AGE     READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                               6m11s   True    UpToDate             5m59s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   6m9s    False   DependencyNotFound   6m9s

NAME                                                                   AGE     READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   6m56s   True    UpToDate   6m22s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   6m56s   True    UpToDate   54s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       6m56s   True    UpToDate   3s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    6m56s   True    UpToDate   6m22s

NAME                                                                    AGE     READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   6m10s   True    UpToDate   5m57s

NAME                                                                                AGE     READY   STATUS     STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   6m10s   True    UpToDate   5m59s

NAME                                                                          AGE     READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        6m10s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    6m10s   True    UpToDate   5m57s

NAME                                                                 AGE     READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    6m10s   True    UpToDate   6m9s

status
michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kubectl get gcp -n networking
NAME                                                                                AGE     READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/hub-allow-external-fwr                               6m11s   True    UpToDate             5m59s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         6m10s   False   DependencyNotFound   6m10s
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   6m9s    False   DependencyNotFound   6m9s

NAME                                                                   AGE     READY   STATUS     STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/hub-global-external-vpc   6m56s   True    UpToDate   6m22s
computenetwork.compute.cnrm.cloud.google.com/hub-global-internal-vpc   6m56s   True    UpToDate   54s
computenetwork.compute.cnrm.cloud.google.com/hub-global-mgmt-vpc       6m56s   True    UpToDate   3s
computenetwork.compute.cnrm.cloud.google.com/hub-global-transit-vpc    6m56s   True    UpToDate   6m22s

NAME                                                                    AGE     READY   STATUS     STATUS AGE
computerouter.compute.cnrm.cloud.google.com/hub-nane1-external-router   6m10s   True    UpToDate   5m57s

NAME                                                                                AGE     READY   STATUS     STATUS AGE
computeroute.compute.cnrm.cloud.google.com/hub-external-vpc-internet-egress-route   6m10s   True    UpToDate   5m59s

NAME                                                                          AGE     READY   STATUS     STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-external-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-internal-paz-snet   6m11s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-mgmt-rz-snet        6m10s   True    UpToDate   5m57s
computesubnetwork.compute.cnrm.cloud.google.com/hub-nane1-transit-paz-snet    6m10s   True    UpToDate   5m57s

NAME                                                                 AGE     READY   STATUS     STATUS AGE
dnspolicy.dns.cnrm.cloud.google.com/hub-external-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-internal-logging-dnspolicy   6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-mgmt-logging-dnspolicy       6m11s   True    UpToDate   6m9s
dnspolicy.dns.cnrm.cloud.google.com/hub-transit-logging-dnspolicy    6m10s   True    UpToDate   6m9s

Reproducing error Chris C. found where we are missing the management-project-id: from setters.yaml
Thank you Chris for this 3rd fix for the hub-env
need to look for it first so I can be sure to fix it
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp | grep False
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions                         174m   False   UpdateFailed   173m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountadmin-permissions                          174m   False   UpdateFailed   173m
iampolicymember.iam.cnrm.cloud.google.com/networking-sa-serviceaccountuser-permissions                           174m   False   UpdateFailed   173m

all 3 are
michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl describe iampolicymember.iam.cnrm.cloud.google.com/networking-sa-computeinstanceadmin-permissions

  Warning  UpdateFailed  72s (x88 over 172m)  iampolicymember-controller  Update call failed: error setting policy member: error applying changes: summary: Request `Create IAM Members roles/compute.instanceAdmin.v1<EMAIL_ADDRESS>for project "xxdmu-admin1-hub-oi5"` returned error: Batch request and retried single request "Create IAM Members roles/compute.instanceAdmin.v1<EMAIL_ADDRESS>for project \"xxdmu-admin1-hub-oi5\"" both failed. Final error: Error applying IAM policy for project "xxdmu-admin1-hub-oi5": Error setting IAM policy for project "xxdmu-admin1-hub-oi5": googleapi: Error 400: Service account<EMAIL_ADDRESS>does not exist., badRequest

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl get gcp -n networking | grep False
computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr                          174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-allow-spokes-to-fortigates-fwr                   174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-elb-allow-health-checks-to-fortigate-fwr         174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr                174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-ilb-allow-health-checks-to-fortigate-fwr         174m   False   DependencyNotFound   174m
computefirewall.compute.cnrm.cloud.google.com/hub-managementvm-allow-ssh-https-to-fortigates-fwr   174m   False   DependencyNotFound   174m

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl describe computefirewall.compute.cnrm.cloud.google.com/hub-allow-fortigates-ha-fwr  -n networking
Status:
  Conditions:
    Last Transition Time:  2023-10-23T17:33:52Z
    Message:               reference IAMServiceAccount networking/hub-fortigatesdn-sa is not found

    all above except below

michael@cloudshell:~/kcc-oi/github/pubsec-declarative-toolkit/solutions (kcc-oi)$ kubectl describe computefirewall.compute.cnrm.cloud.google.com/hub-iap-allow-rdp-to-managementvm-fwr   -n networking

  Warning  DependencyNotReady  32m                    computefirewall-controller  reference ComputeNetwork networking/hub-global-mgmt-vpc is not ready
  Warning  DependencyNotFound  4m10s (x17 over 179m)  computefirewall-controller  reference IAMServiceAccount networking/hub-managementvm-sa is not found

SAs for hub are in the networking namespace instead of the config-control one
metadata: # kpt-merge: networking/hub-managementvm-sa
  name: hub-managementvm-sa
  namespace: networking

metadata: # kpt-merge: networking/hub-fortigatesdn-sa
  name: hub-fortigatesdn-sa
  namespace: networking

found the issue
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}

should be networking namespace

this would fix the fact that the IAMPolicyMember is in the config-control namespace but it's service account is in the networking namespace

to summarize
hub-env/fortigate/service-account.yaml
- sa in networking
- policy in config-control

hub-env/fortigate/management-vm/service-account.yaml
- sa in networking
- policy in networking

fix is to move the
fortigatesdn-sa-fortigatesdnviewer-role-permissions
to the network namespace

metadata: # kpt-merge: config-control/fortigatesdn-sa-fortigatesdnviewer-role-permissions
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}

to
metadata: # kpt-merge: networking/fortigatesdn-sa-fortigatesdnviewer-role-permissions
  name: fortigatesdn-sa-fortigatesdnviewer-role-permissions
  namespace: config-control # kpt-set: ${management-namespace}

to match
metadata: # kpt-merge: networking/hub-admin-serviceaccountuser-permissions
  name: hub-admin-serviceaccountuser-permissions
  namespace: networking

and add
 management-namespace: networking
to setters.yaml

add to setters
  management-namespace: networking

fixes
  namespace: networking # kpt-set: ${management-namespace}

but breaks the VM's - we will hardcode instead like the management-vm/service-account.yaml

michael@cloudshell:~/kcc-oi/kpt (kcc-oi-3552)$ kpt live apply hub-env
installing inventory ResourceGroup CRD.
error: 5 errors: