Speeds up additional role creation and label addition on accessible namespaces

fixes: https://github.com/kiali/kiali/issues/3892
fixes: https://github.com/kiali/kiali/issues/3928
Instead of individually (separately) creating the role and rolebinding on each individual namespace, do a bulk create of every role and every role binding in every namespace in one single, big k8s task invocation. This should speed things up considerable.
There are a couple concerns I have. Is there a limit to either the full size of the yaml string or the number of resources that the k8s task or the kubernetes master API itself will accept? Can Ansible build an arbitrarily long (theoretically unlimited) yaml string? And will the ansible k8s task or the kubernetes master API accept an arbitrary (theorteically unlimited) number of resources to create? Hopefully the answer is yes for both, or if not yet, hopefully the limit is so large so as to not concern us. But we should test this out - for example, if I put 1,000 namespaces in the spec.deployment.accessible_namespaces setting, will that be OK? How about 2,000? What's the theoretical maximum number of namespaces that we should support?

This PR also should speed up adding the additional kiali.io/member-of label on all the namespaces too (that probably was part of the problem, too).
I added a molecule test to ensure the new way of the label addition doesn't affect existing labels on the namespace.
@FilipB please try this out and see how it goes.

Theoretical maximum  number of namespaces is 10 000 - https://docs.openshift.com/container-platform/4.7/scalability_and_performance/planning-your-environment-according-to-object-maximums.html#cluster-maximums-major-releases_object-limits

Now it shows this:
TASK [Log reconciliation processing time] **************************************
Tuesday 20 April 2021  10:36:36 +0000 (0:00:00.029)       0:02:01.189 *********
ok: [localhost] => {
    "msg": "Processing time: [121] seconds"
}

TASK [debug] *******************************************************************
Tuesday 20 April 2021  10:36:36 +0000 (0:00:00.034)       0:02:01.224 *********
ok: [localhost] => {
    "msg": "KIALI RECONCILIATION IS DONE."
}

PLAY RECAP *********************************************************************
localhost                  : ok=984  changed=19   unreachable=0    failed=0    skipped=70   rescued=0    ignored=0

Tuesday 20 April 2021  10:36:36 +0000 (0:00:00.024)       0:02:01.248 *********
===============================================================================
default/kiali-deploy : Create additional Kiali roles on OpenShift ------- 9.40s
default/kiali-deploy : Process OpenShift Console Links ------------------ 4.93s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 4.49s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 4.04s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.40s
default/kiali-deploy : Build additional Kiali roles yaml for OpenShift --- 2.34s
default/kiali-deploy : Build yaml for additional Kiali label on accessible namespaces --- 1.88s
default/kiali-deploy : Get information about the operator --------------- 1.81s
default/kiali-deploy : Find current configmap, if it exists ------------- 1.56s
default/kiali-deploy : Delete all Kiali roles from namespaces if view_only_mode is changing since role bindings are immutable --- 1.53s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 1.33s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.07s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.03s
default/kiali-deploy : Create resource [rolebinding] on [openshift] ----- 0.98s
default/kiali-deploy : Wait for Monitoring Dashboards CRD to be ready --- 0.97s
default/kiali-deploy : Build additional Kiali roles yaml for Kubernetes --- 0.93s
default/kiali-deploy : Confirm the cluster can access github.com when it needs to determine the last release of Kiali --- 0.92s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 0.91s
default/kiali-deploy : Process OpenShift OAuth client ------------------- 0.89s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 0.88s

It says it's successful but kiali UI does not show any newly added namespace. Kiali CR is updated but Kiali config map is not and it still contains old accessible_namespaces.
Another weird thing is that kiali log is reset after reconciliation and it only shows new lines added after that.

I forgot to remove existing namespaces from previous test so those were labeled already and probably skipped because of that?
Running again with clean env.

I forgot to remove existing namespaces from previous test so those were labeled already and probably skipped because of that?

I can't think why that would have an effect. I'll think about this - and this might be something else to test - because that sounds weird. It should be OK that you already have namespaces that the operator already processed - there is lots of code in the operator to handle namespaces that were already configured and accessible to Kiali.

Tried again using our job for bookinfo and it behaves the same. Kiali CR is udpated but kiali config map is not. Is this working for you (basic scenario of adding a namespace to accessible_namespaces)?

Is this working for you (basic scenario of adding a namespace to accessible_namespaces)?

All molecule tests passed, including the accessible_namespaces one. Let me run some manual tests to see if something else is happening that the tests don't catch.

Here's the tests that I did, and all is working as expected. The config map is always updated with the expected values.

Install Kiali CR with accessible_namespaces = "[**]"
Edit Kiali CR with accessible_namespaces changed to "[test1]"
Edit Kiali CR with accessible_namespaces changed to "[test1, test2]"
Edit Kiali CR with accessible_namespaces changed to "[test2]"

So adding and deleting namespaces works. Note - I also checked the label on the namespaces to ensure they had "kiali.io/member-of=istio-system" as appropriate.

Sorry for delay. We have problems with our infra all day.
Just adding stats from maistra team to be able to compare. Reconciliation of SMCP takes < 30s for 200 members.

Adding profiler outputs for following scenarios:

Add 200 namespaces to accessible_namespaces

TASK [Log reconciliation processing time] **************************************
Wednesday 21 April 2021  16:47:43 +0000 (0:00:00.042)       0:02:35.076 *******
ok: [localhost] => {
    "msg": "Processing time: [154] seconds"
}

TASK [debug] *******************************************************************
Wednesday 21 April 2021  16:47:43 +0000 (0:00:00.048)       0:02:35.125 *******
ok: [localhost] => {
    "msg": "KIALI RECONCILIATION IS DONE."
}

PLAY RECAP *********************************************************************
localhost                  : ok=988  changed=27   unreachable=0    failed=0    skipped=73   rescued=0    ignored=0

Wednesday 21 April 2021  16:47:43 +0000 (0:00:00.032)       0:02:35.157 *******
===============================================================================
default/kiali-deploy : Create additional Kiali roles on OpenShift ------ 33.83s
default/kiali-deploy : Create additional Kiali label on accessible namespaces --- 6.28s
default/kiali-deploy : Process OpenShift Console Links ------------------ 5.22s
default/kiali-deploy : Build additional Kiali roles yaml for OpenShift --- 2.52s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.25s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.15s
default/kiali-deploy : Build yaml for additional Kiali label on accessible namespaces --- 2.11s
default/kiali-deploy : Delete all Kiali roles from namespaces if view_only_mode is changing since role bindings are immutable --- 1.77s
default/kiali-deploy : Get information about the operator --------------- 1.52s
default/kiali-deploy : Force the Kiali pod to restart if necessary ------ 1.19s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.12s
default/kiali-deploy : Get namespace [default] whose label is to be removed --- 1.08s
default/kiali-deploy : Build additional Kiali roles yaml for Kubernetes --- 1.06s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.05s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.03s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 1.02s
default/kiali-deploy : Saving the label removal from namespace [default] --- 1.00s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 0.99s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 0.98s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 0.97s

remove one namespace

TASK [Log reconciliation processing time] **************************************
Wednesday 21 April 2021  15:55:47 +0000 (0:00:00.029)       0:02:09.546 *******
ok: [localhost] => {
    "msg": "Processing time: [129] seconds"
}

TASK [debug] *******************************************************************
Wednesday 21 April 2021  15:55:47 +0000 (0:00:00.034)       0:02:09.580 *******
ok: [localhost] => {
    "msg": "KIALI RECONCILIATION IS DONE."
}

PLAY RECAP *********************************************************************
localhost                  : ok=984  changed=24   unreachable=0    failed=0    skipped=73   rescued=0    ignored=0

Wednesday 21 April 2021  15:55:47 +0000 (0:00:00.022)       0:02:09.603 *******
===============================================================================
default/kiali-deploy : Create additional Kiali roles on OpenShift ------ 10.22s
default/kiali-deploy : Process OpenShift Console Links ------------------ 5.19s
default/kiali-deploy : Create additional Kiali label on accessible namespaces --- 5.03s
default/kiali-deploy : Build additional Kiali roles yaml for OpenShift --- 2.54s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.43s
default/kiali-deploy : Build yaml for additional Kiali label on accessible namespaces --- 2.39s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.18s
default/kiali-deploy : Delete all Kiali roles from namespaces if view_only_mode is changing since role bindings are immutable --- 1.82s
default/kiali-deploy : Get information about the operator --------------- 1.47s
default/kiali-deploy : Force the Kiali pod to restart if necessary ------ 1.26s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.09s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.07s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.05s
default/kiali-deploy : Build additional Kiali roles yaml for Kubernetes --- 1.02s
default/kiali-deploy : Get namespace [test200] whose label is to be removed --- 1.01s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 0.99s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 0.98s
default/kiali-deploy : Saving the label removal from namespace [test200] --- 0.98s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 0.96s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 0.94s

remove all 200

TASK [Log reconciliation processing time] **************************************
Wednesday 21 April 2021  16:17:32 +0000 (0:00:00.033)       0:10:53.523 *******
ok: [localhost] => {
    "msg": "Processing time: [653] seconds"
}

TASK [debug] *******************************************************************
Wednesday 21 April 2021  16:17:32 +0000 (0:00:00.040)       0:10:53.563 *******
ok: [localhost] => {
    "msg": "KIALI RECONCILIATION IS DONE."
}

PLAY RECAP *********************************************************************
localhost                  : ok=1451 changed=85   unreachable=0    failed=0    skipped=596  rescued=0    ignored=0

Wednesday 21 April 2021  16:17:32 +0000 (0:00:00.031)       0:10:53.595 *******
===============================================================================
default/kiali-deploy : Delete all ConsoleLinks for namespaces that are no longer accessible - 155.89s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 8.61s
default/kiali-deploy : Remove Kiali label from namespaces that Kiali no longer has access to --- 2.94s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.15s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.14s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.14s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.13s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.13s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.12s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.11s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.11s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.10s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.10s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.09s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.09s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.08s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.06s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.05s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.05s
default/kiali-deploy : Delete unused Kiali roles from previously accessible namespaces --- 2.04s

Improvement is significant. It was 1763s, now it is 154s.
It looks like there is still some space for improvement for removal which took 653s
@jmazzitelli  @lucasponce @jshaughn Question is what is acceptable time. Maistra operator finishes the reconciliation in 30s for the same scenario. Is 154s sufficient for Kiali or we want to improve it even more?

I expect removal to be longer. I didn't spend alot of time on that. I figured that was less important.
I'll tell you right now that there is no more room to help this because this task is a single, bulk request to the cluster - this is the actual k8s cluster taking 30+ seconds to process our really long list of roles that we need created (it is essentially the same as if you built a really big .yaml file with 200 roles and 200 role bindings and did a "oc apply -f the-big.yaml"):
default/kiali-deploy : Create additional Kiali roles on OpenShift ------ 33.83s

Same for this - this is a single request:
default/kiali-deploy : Create additional Kiali label on accessible namespaces --- 6.28s

@FilipB can you write up another github issue on the delete issue? Removing 1 namespace, removing 200 namespaces? We will want to work on that, too. But I do not want to make that stop this PR from going in - so let's have a separate issue and work on that after this PR is done.

Considering that theoretical maximum number of namespaces in OCP 4.7 is 10 000, I started with 1000. Reconciliation was successful but there were two peaks in memory consumption by kiali operator. One peak at 7GiB and second peak at 10 GiB. I also tried 2000 namespaces but the kiali operator pod was evicted as it was consuming over 13GiB of memory. It seems it will be necessary to change the bulk operations to use smaller chunks.

OK, then since this PR is not ready, we'll also attempt to implement the fix for this, too, here in the same PR.

Latest commit adds batching. Default batch size is 100 but an admin can configure the operator to use a different batch size by changing the NAMESPACE_BATCH_SIZE env var on the operator deployment (similar to how you can change  other env vars like ALLOW_AD_HOC_KIALI_IMAGE).
(Do not merge until https://github.com/kiali/kiali-operator/pull/296 is merged and then this PR needs to update the community and upstream OLM metadata to add the new env var to the operator deployment)

Confirming that NAMESPACE_BATCH_SIZE setting is working fine. It resolves the problem with too high memory consumption. With 2000 namespaces and NAMESPACE_BATCH_SIZE=100, kiali operator pod consumed maximally 3GiB.
There is still one problem which is not visible from the profiler output. With 2000 namespaces it took 1155s to reconcile. Longest tasks are:
default/kiali-deploy : Determine all accessible namespaces, expanding regex expressions to matched namespaces -- 72.53s
default/kiali-deploy : Process OpenShift Console Links ----------------- 43.33s
default/kiali-deploy : Create additional Kiali roles on batch of namespaces -- 20.62s

But the longest general part (message: 12. Creating additional roles) which took ~ 14 minutes is not visible in the profiler output.
"Build additional Kiali label yaml for batch of [100] namespaces]" task is adding the labels to the yaml one by one ("Build additional Kiali label yaml for a namespace" task). Building the yaml is actually taking much longer than applying it ("Create additional Kiali label on all accessible namespaces" task).
Is it possible to build the yaml more efficiently as it's the part which is taking the most time?

Building the yaml is actually taking much longer than applying it ("Create additional Kiali label on all accessible namespaces" task).

hmm.. it wasn't doing that before. This is basically the same code as last time before the batching. Let me see if I did something to break that.

I'm going to add some config to the profile so it lists 100 items in the profile report (right now it defaults to 20 which I don't think is enough - I'm not sure if there is a way to get around how it puts duplicates in the list because of the way ansible handles the looping).

@FilipB if you rebuild the operator with the latest - you can now set the PROFILE_TASKS_TASK_OUTPUT_LIMIT env var (like you do with the other env vars) and that will set the length of the profiler report size - so you can set that to, say, 2,000 and you should get a list of everything... I'm trying to find out an easy way to parse those reports.. must be some tool or "sed" command or something to help combine duplicate entries.

Here's a script to use to get better profiler numbers. Copy the profile report to a file (just the lines with the task name/seconds time). Pipe that file to this script and you'll get a report with the duplicate lines combined to see the full times per task.
parse-profiler-report.sh.txt
Example:
$ cat /tmp/my-profiler-report.txt | ./parse-profiler-report.sh
Create additional Kiali roles on batch of namespaces                        1.1
Create additional Kiali roles on all accessible namespaces on Kubernetes    0.09
debug                                                                       0.04
Build additional Kiali roles yaml for batch of [3] namespaces               0.03
Build additional Kiali roles yaml for batch of [2] namespaces               0.02
Build additional Kiali roles yaml for a namespace                           0.1
Erase potentially large fact                                                0.02
set_fact                                                                    0.01

This python function is slow - anyone know how to speed it up? :)
https://github.com/kiali/kiali-operator/blob/master/roles/default/kiali-deploy/filter_plugins/only_accessible_namespaces.py
With 1,000 items in a list and passing that list as both arguments, that takes 10 seconds to run. With 2,000 items, takes 50 seconds to run. There has to be a better way. I need a python guru :)

I'm trying a completely, simplified, implementation. An epiphany hit me that I've been thinking about this all wrong. I'm not sure if this will also cause memory bloat, but I'm going to rely more on the k8s collection to do this rather than our code build up the yaml. Stay tuned for a new push of code with this new stuff.

@FilipB the current code here is super-fast :) But I don't know how memory-intensive it is. This is utilizing mostly on the k8s ansible collection task and templating. I'm impressed at how much faster this is. I hope the memory requirements aren't outrageous because this is probably the largest improvement in speed that is possible.
Note the NAMESPACE_BATCH_SIZE is gone - that has been removed and is not used anymore.

all molecule tests pass on minikube:
              accessible-namespaces-test... success [4m 16s]
     affinity-tolerations-resources-test... success [3m 38s]
                                api-test... success [3m 33s]
                      config-values-test... success [3m 41s]
                  default-namespace-test... success [2m 2s]
                        header-auth-test... success [2m 0s]
                            metrics-test... success [4m 11s]
                     null-cr-values-test... success [1m 54s]
                only-view-only-mode-test... success [3m 8s]
                             openid-test... success [2m 23s]
                     openshift-auth-test... skipped
                   os-console-links-test... skipped
                              roles-test... success [5m 53s]
                    rolling-restart-test... success [2m 59s]
                              token-test... success [1m 55s]

the OpenShift-specific tests pass on my OpenShift cluster:
                     openshift-auth-test... success [3m 41s]
                   os-console-links-test... success [9m 47s]

First test with 50 namespaces (named "testN" where N=1..50). Kiali CR has accessible_namespaces set to [test.*].
Saturday 24 April 2021  03:46:04 +0000 (0:00:00.026)       0:01:30.779 ********
===============================================================================
default/kiali-deploy : Create additional Kiali roles on all accessible namespaces on OpenShift --- 6.63s
default/kiali-deploy : Create additional Kiali role bindings on all accessible namespaces on OpenShift --- 5.96s
default/kiali-deploy : Create additional Kiali label on all accessible namespaces --- 4.14s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 3.90s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 3.49s
default/kiali-deploy : Process OpenShift Console Links ------------------ 2.88s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.72s
default/kiali-deploy : Get information about the operator --------------- 2.01s
default/kiali-deploy : Create resource [rolebinding] on [openshift] ----- 1.69s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.58s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 1.53s
default/kiali-deploy : Create secret to store a random signing key if a secret does not already exist and we need one --- 1.48s
default/kiali-deploy : Find current configmap, if it exists ------------- 1.42s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.37s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 1.36s
default/kiali-deploy : Create resource [role] on [openshift] ------------ 1.33s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.30s
default/kiali-deploy : Remove HPA if disabled on OpenShift -------------- 1.26s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 1.20s
default/kiali-deploy : Create resource [deployment] on [openshift] ------ 1.16s
default/kiali-deploy : Create resource [role-controlplane] on [openshift] --- 1.14s
default/kiali-deploy : Wait for Monitoring Dashboards CRD to be ready --- 1.12s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.11s
default/kiali-deploy : Process OpenShift OAuth client ------------------- 1.07s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.05s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.03s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.00s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.94s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.94s
default/kiali-deploy : Create resource [rolebinding-controlplane] on [openshift] --- 0.94s
default/kiali-deploy : Detect Kiali route on OpenShift ------------------ 0.93s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.90s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.90s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.89s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.88s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.88s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.87s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.86s
default/kiali-deploy : set_fact ----------------------------------------- 0.85s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.84s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.79s
default/kiali-deploy : Find all namespaces (this is limited to what the operator has permission to see) --- 0.70s
default/kiali-deploy : Find all Monitoring Dashboards packaged with the operator --- 0.67s
default/kiali-deploy : Determine the Kubernetes version ----------------- 0.63s
default/kiali-deploy : Determine the OpenShift version ------------------ 0.63s
default/kiali-deploy : set_fact ----------------------------------------- 0.61s
Run the version-specific deploy role ------------------------------------ 0.58s
default/kiali-deploy : Find current deployment, if it exists ------------ 0.53s
default/kiali-deploy : Get information about the cluster ---------------- 0.47s
default/kiali-deploy : Determine which Monitoring Dashboards are to be created --- 0.42s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod --- 0.40s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod_anti --- 0.29s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.node --- 0.27s
Determine the version that is to be installed --------------------------- 0.25s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_labels --- 0.24s
default/kiali-deploy : Replace snake_case with camelCase in deployment.hpa.spec --- 0.24s
default/kiali-deploy : Create Kiali objects on OpenShift ---------------- 0.23s
default/kiali-deploy : Replace snake_case with camelCase in deployment.service_annotations --- 0.22s
default/kiali-deploy : include_tasks ------------------------------------ 0.22s
default/kiali-deploy : Set default istio component namespace: grafana --- 0.21s
Playbook start time ----------------------------------------------------- 0.20s
default/kiali-deploy : include_tasks ------------------------------------ 0.19s
default/kiali-deploy : Create Kiali objects on Kubernetes --------------- 0.19s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_annotations --- 0.19s
default/kiali-deploy : include_tasks ------------------------------------ 0.19s
default/kiali-deploy : Replace snake_case with camelCase in deployment.tolerations --- 0.19s
Make sure a default supported image or an override image is known ------- 0.18s
Override the supported images if found in the environment --------------- 0.18s
default/kiali-deploy : Print some debug information --------------------- 0.17s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
debug ------------------------------------------------------------------- 0.17s
default/kiali-deploy : Replace snake_case with camelCase in deployment.node_selector --- 0.17s
default/kiali-deploy : Replace snake_case with camelCase in deployment.resources --- 0.17s
default/kiali-deploy : Only allow ad-hoc kiali namespace when appropriate --- 0.16s
default/kiali-deploy : include_tasks ------------------------------------ 0.15s
Examine environment and determine if supported image for the requested version is overridden even if no default is known --- 0.15s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : Set default istio namespace ---------------------- 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : Create Route on OpenShift if enabled ------------- 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : Replace snake_case with camelCase in deployment.override_ingress_yaml --- 0.12s
default/kiali-deploy : Replace snake_case with camelCase in deployment.additional_service_yaml --- 0.12s
default/kiali-deploy : Set default deployment namespace to the same namespace where the CR lives --- 0.11s
default/kiali-deploy : Set default Grafana in_cluster_url --------------- 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : Point signing key to the generated secret -------- 0.11s
default/kiali-deploy : Delete all Kiali roles from namespaces if view_only_mode is changing since role bindings are immutable --- 0.10s
default/kiali-deploy : Set Kiali TLS Termination from OpenShift route --- 0.10s
default/kiali-deploy : Delete Monitoring Dashboards that should no longer be deployed --- 0.10s

Next test - deleting just a couple of namespaces from accessible_namespaces:
Saturday 24 April 2021  03:54:44 +0000 (0:00:00.047)       0:01:20.032 ********
===============================================================================
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.93s
default/kiali-deploy : Create additional Kiali roles on all accessible namespaces on OpenShift --- 2.81s
default/kiali-deploy : Delete all ConsoleLinks for namespaces that are no longer accessible --- 2.76s
default/kiali-deploy : Create additional Kiali label on all accessible namespaces --- 2.40s
default/kiali-deploy : Create additional Kiali role bindings on all accessible namespaces on OpenShift --- 2.04s
default/kiali-deploy : Process OpenShift Console Links ------------------ 2.02s
default/kiali-deploy : Force the Kiali pod to restart if necessary ------ 1.52s
default/kiali-deploy : Get information about the operator --------------- 1.46s
Run the version-specific deploy role ------------------------------------ 1.38s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.36s
default/kiali-deploy : Remove Kiali label from namespaces that Kiali no longer has access to --- 1.33s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.29s
default/kiali-deploy : Delete Kiali roles from previously accessible namespaces --- 1.29s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.28s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.21s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 1.18s
default/kiali-deploy : Create resource [deployment] on [openshift] ------ 1.15s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.13s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.13s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.11s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 1.10s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 1.10s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.10s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 1.06s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.05s
default/kiali-deploy : Create resource [role] on [openshift] ------------ 1.05s
default/kiali-deploy : Wait for Monitoring Dashboards CRD to be ready --- 1.05s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.02s
default/kiali-deploy : Remove HPA if disabled on OpenShift -------------- 1.00s
default/kiali-deploy : Create resource [role-controlplane] on [openshift] --- 1.00s
default/kiali-deploy : Create resource [rolebinding] on [openshift] ----- 0.99s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.98s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.98s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.98s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.97s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.96s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.95s
default/kiali-deploy : Create resource [rolebinding-controlplane] on [openshift] --- 0.95s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.94s
default/kiali-deploy : Detect Kiali route on OpenShift ------------------ 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.91s
default/kiali-deploy : Process OpenShift OAuth client ------------------- 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.89s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.89s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.88s
default/kiali-deploy : Find all namespaces (this is limited to what the operator has permission to see) --- 0.86s
default/kiali-deploy : Find all Monitoring Dashboards packaged with the operator --- 0.77s
default/kiali-deploy : set_fact ----------------------------------------- 0.64s
default/kiali-deploy : Find current configmap, if it exists ------------- 0.64s
default/kiali-deploy : set_fact ----------------------------------------- 0.59s
Examine environment and determine if supported image for the requested version is overridden even if no default is known --- 0.58s
default/kiali-deploy : Find current deployment, if it exists ------------ 0.55s
default/kiali-deploy : Determine version_label based on image_version --- 0.53s
default/kiali-deploy : Get information about the cluster ---------------- 0.43s
default/kiali-deploy : Determine the OpenShift version ------------------ 0.42s
default/kiali-deploy : Find some current configuration settings --------- 0.39s
default/kiali-deploy : Determine the Kubernetes version ----------------- 0.37s
Override the supported images if found in the environment --------------- 0.36s
default/kiali-deploy : Determine which Monitoring Dashboards are to be created --- 0.36s
Make sure a default supported image or an override image is known ------- 0.29s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.node --- 0.29s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod_anti --- 0.28s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_annotations --- 0.28s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod --- 0.27s
default/kiali-deploy : Replace snake_case with camelCase in deployment.hpa.spec --- 0.26s
default/kiali-deploy : Replace snake_case with camelCase in deployment.service_annotations --- 0.24s
default/kiali-deploy : Replace snake_case with camelCase in deployment.resources --- 0.24s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_labels --- 0.23s
default/kiali-deploy : Replace snake_case with camelCase in deployment.node_selector --- 0.23s
Playbook start time ----------------------------------------------------- 0.23s
default/kiali-deploy : Set default istio component namespace: grafana --- 0.22s
Determine the version that is to be installed --------------------------- 0.21s
default/kiali-deploy : Create Kiali objects on OpenShift ---------------- 0.21s
default/kiali-deploy : include_tasks ------------------------------------ 0.21s
default/kiali-deploy : set_fact ----------------------------------------- 0.21s
default/kiali-deploy : Replace snake_case with camelCase in deployment.tolerations --- 0.20s
debug ------------------------------------------------------------------- 0.20s
default/kiali-deploy : include_tasks ------------------------------------ 0.18s
default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field --- 0.18s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
default/kiali-deploy : include_tasks ------------------------------------ 0.16s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : Set default istio namespace ---------------------- 0.13s
default/kiali-deploy : Print some debug information --------------------- 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : Set default deployment namespace to the same namespace where the CR lives --- 0.13s
default/kiali-deploy : Replace snake_case with camelCase in deployment.additional_service_yaml --- 0.13s
default/kiali-deploy : Replace snake_case with camelCase in deployment.override_ingress_yaml --- 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : Delete all additional Kiali roles from namespaces that Kiali no longer has access to --- 0.12s
default/kiali-deploy : Only allow ad-hoc kiali namespace when appropriate --- 0.12s
default/kiali-deploy : Create Route on OpenShift if enabled ------------- 0.12s
default/kiali-deploy : Use OpenShift service CA file for Prometheus ----- 0.11s
default/kiali-deploy : Ensure Ingress is Enabled if Auth Strategy is openshift --- 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s

Next test - I deleted the actual namespaces for test38, test39, test48, test49 and I removed all namespaces except for test1 and test2 from accessible_namespaces:
Saturday 24 April 2021  04:00:06 +0000 (0:00:00.042)       0:01:54.752 ********
===============================================================================
default/kiali-deploy : Delete all ConsoleLinks for namespaces that are no longer accessible -- 41.81s
default/kiali-deploy : Delete Kiali roles from previously accessible namespaces --- 3.30s
default/kiali-deploy : Remove Kiali label from namespaces that Kiali no longer has access to --- 3.16s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.51s
default/kiali-deploy : Get information about the operator --------------- 1.64s
default/kiali-deploy : Force the Kiali pod to restart if necessary ------ 1.58s
default/kiali-deploy : Create resource [deployment] on [openshift] ------ 1.51s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 1.40s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 1.22s
default/kiali-deploy : Create additional Kiali label on all accessible namespaces --- 1.21s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.20s
default/kiali-deploy : Find current configmap, if it exists ------------- 1.20s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 1.18s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.16s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.14s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.13s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.12s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.11s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.10s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 1.07s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.07s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.07s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.06s
default/kiali-deploy : Process OpenShift OAuth client ------------------- 1.05s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.02s
default/kiali-deploy : Process OpenShift Console Links ------------------ 1.00s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.98s
default/kiali-deploy : Create resource [role] on [openshift] ------------ 0.98s
default/kiali-deploy : Create resource [rolebinding-controlplane] on [openshift] --- 0.96s
default/kiali-deploy : Detect Kiali route on OpenShift ------------------ 0.95s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.95s
default/kiali-deploy : Create additional Kiali roles on all accessible namespaces on OpenShift --- 0.94s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.93s
default/kiali-deploy : Find all namespaces (this is limited to what the operator has permission to see) --- 0.93s
default/kiali-deploy : Create resource [rolebinding] on [openshift] ----- 0.92s
default/kiali-deploy : Create resource [role-controlplane] on [openshift] --- 0.92s
default/kiali-deploy : Remove HPA if disabled on OpenShift -------------- 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.90s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.89s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.88s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.86s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.85s
default/kiali-deploy : Wait for Monitoring Dashboards CRD to be ready --- 0.85s
default/kiali-deploy : Create additional Kiali role bindings on all accessible namespaces on OpenShift --- 0.84s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.82s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.81s
default/kiali-deploy : Find all Monitoring Dashboards packaged with the operator --- 0.74s
default/kiali-deploy : set_fact ----------------------------------------- 0.65s
default/kiali-deploy : set_fact ----------------------------------------- 0.58s
Run the version-specific deploy role ------------------------------------ 0.55s
default/kiali-deploy : Determine the OpenShift version ------------------ 0.45s
default/kiali-deploy : Get information about the cluster ---------------- 0.40s
default/kiali-deploy : Determine which Monitoring Dashboards are to be created --- 0.40s
default/kiali-deploy : Determine the Kubernetes version ----------------- 0.36s
default/kiali-deploy : include_tasks ------------------------------------ 0.35s
default/kiali-deploy : Find current deployment, if it exists ------------ 0.35s
default/kiali-deploy : Find some current configuration settings --------- 0.35s
default/kiali-deploy : Replace snake_case with camelCase in deployment.hpa.spec --- 0.32s
default/kiali-deploy : Print some debug information --------------------- 0.32s
default/kiali-deploy : Replace snake_case with camelCase in deployment.service_annotations --- 0.31s
default/kiali-deploy : Replace snake_case with camelCase in deployment.resources --- 0.27s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod --- 0.26s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.node --- 0.26s
default/kiali-deploy : Replace snake_case with camelCase in deployment.node_selector --- 0.24s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_labels --- 0.24s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod_anti --- 0.24s
default/kiali-deploy : Create Kiali objects on OpenShift ---------------- 0.23s
default/kiali-deploy : Delete all additional Kiali roles from namespaces that Kiali no longer has access to --- 0.22s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_annotations --- 0.22s
debug ------------------------------------------------------------------- 0.21s
Determine the version that is to be installed --------------------------- 0.20s
Playbook start time ----------------------------------------------------- 0.18s
Examine environment and determine if supported image for the requested version is overridden even if no default is known --- 0.18s
default/kiali-deploy : Set default istio component namespace: grafana --- 0.18s
Make sure a default supported image or an override image is known ------- 0.17s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
default/kiali-deploy : Replace snake_case with camelCase in deployment.tolerations --- 0.16s
default/kiali-deploy : include_tasks ------------------------------------ 0.16s
default/kiali-deploy : include_tasks ------------------------------------ 0.16s
Override the supported images if found in the environment --------------- 0.16s
default/kiali-deploy : include_tasks ------------------------------------ 0.15s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.14s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : Replace snake_case with camelCase in deployment.override_ingress_yaml --- 0.12s
default/kiali-deploy : Create Route on OpenShift if enabled ------------- 0.12s
default/kiali-deploy : Set default istio namespace ---------------------- 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : Replace snake_case with camelCase in deployment.additional_service_yaml --- 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : Only allow ad-hoc kiali namespace when appropriate --- 0.10s
default/kiali-deploy : Set default deployment namespace to the same namespace where the CR lives --- 0.10s
default/kiali-deploy : Set default web root based on cluster type ------- 0.09s

All these tests show the operator is much, much faster than before. The only thing left to try to speed up is default/kiali-deploy : Delete all ConsoleLinks for namespaces that are no longer accessible but as it stands now, I think this PR is ready for merge once QE runs its final tests. If I can find an improvement to the ConsoleLinks task, I'll add that. But that should not hold up this PR if everything else is OK.

Oh, one final test - delete the CR entirely to uninstall Kiali:
Saturday 24 April 2021  04:05:17 +0000 (0:00:00.027)       0:00:52.871 ********
===============================================================================
default/kiali-remove : Delete Kiali resources -------------------------- 39.42s
default/kiali-remove : Delete OpenShift-specific Kiali resources -------- 7.52s
default/kiali-remove : Delete all additional Kiali roles in current accessible namespaces --- 1.38s
default/kiali-remove : Remove Kiali label from namespaces found in current accessible namespaces --- 1.32s
default/kiali-remove : Find all namespaces (this is limited to what the operator has permission to see) --- 0.91s
default/kiali-remove : Find current configmap, if it exists ------------- 0.54s
default/kiali-remove : Get information about the cluster ---------------- 0.47s
include_role : {{ version | default(default_playbook.playbook) }}/kiali-remove --- 0.33s
Playbook start time ----------------------------------------------------- 0.20s
debug ------------------------------------------------------------------- 0.18s
default/kiali-remove : Find currently configured label selector --------- 0.09s
default/kiali-remove : Find currently accessible namespaces ------------- 0.07s
default/kiali-remove : Set default istio namespace ---------------------- 0.06s
default/kiali-remove : Set default deployment namespace to the same namespace where the CR lives --- 0.05s
default/kiali-remove : Print some debug information --------------------- 0.05s
default/kiali-remove : Determine the cluster type ----------------------- 0.05s
default/kiali-remove : debug -------------------------------------------- 0.04s
Log removal processing time --------------------------------------------- 0.04s
default/kiali-remove : Delete Kiali cluster roles ----------------------- 0.03s
Playbook end time ------------------------------------------------------- 0.03s
debug ------------------------------------------------------------------- 0.03s
debug ------------------------------------------------------------------- 0.03s
Determine the default playbook ------------------------------------------ 0.03s

Speed up of console link deletion is now in, also. I again created a 50-namespace mesh, then removed all but 1 of the namespaces from accessible_namespaces - the task for deletion of the console links isn't even in the top of the profiler report anymore (in the previous test it took over 40 seconds; in this test it took just over 2 seconds):
===============================================================================
default/kiali-deploy : Delete Kiali roles from previously accessible namespaces --- 3.57s
default/kiali-deploy : Remove Kiali label from namespaces that Kiali no longer has access to --- 2.79s
default/kiali-deploy : Delete all ConsoleLinks for namespaces that are no longer accessible --- 2.21s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.11s
default/kiali-deploy : Get information about the operator --------------- 1.77s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.73s
default/kiali-deploy : Force the Kiali pod to restart if necessary ------ 1.71s
...

I know of nothing else that needs to be fixed - everything that was really slow has now been sped up signficantly.

Just ran another test because I'm curious - I created 50 namespaces and put them in my accessible_namespaces. I then created another 150 namespaces and those went into accessible namespaces (so this second reconciliation had 200 namespaces total).
Here's the profilter report of that second reconciliation (processing 150 new namespaces, 200 total) and the metrics page for the memory consumption of the kiali operator pod (taken from the OpenShift UI Console):
Saturday 24 April 2021  14:50:13 +0000 (0:00:00.032)       0:01:45.351 ********
===============================================================================
default/kiali-deploy : Create additional Kiali roles on all accessible namespaces on OpenShift -- 15.44s
default/kiali-deploy : Create additional Kiali role bindings on all accessible namespaces on OpenShift -- 14.59s
default/kiali-deploy : Create additional Kiali label on all accessible namespaces --- 7.55s
default/kiali-deploy : Process OpenShift Console Links ------------------ 6.01s
default/kiali-deploy : Create the Monitoring Dashboards ----------------- 2.26s
default/kiali-deploy : Force the Kiali pod to restart if necessary ------ 1.70s
default/kiali-deploy : Get information about the operator --------------- 1.68s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.44s
default/kiali-deploy : Create resource [configmap] on [openshift] ------- 1.31s
default/kiali-deploy : Create resource [service] on [openshift] --------- 1.24s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.14s
default/kiali-deploy : Delete Kiali roles from previously accessible namespaces --- 1.12s
default/kiali-deploy : Create resource [serviceaccount] on [openshift] --- 1.10s
default/kiali-deploy : Create resource [cabundle] on [openshift] -------- 1.08s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.07s
default/kiali-deploy : Create resource [route] on [openshift] ----------- 1.06s
default/kiali-deploy : Create resource [deployment] on [openshift] ------ 1.03s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 1.03s
default/kiali-deploy : Get information about any existing signing key secret if we need to know if it exists or not --- 1.02s
default/kiali-deploy : Create resource [role-controlplane] on [openshift] --- 1.00s
default/kiali-deploy : Remove HPA if disabled on OpenShift -------------- 1.00s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.98s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.94s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.93s
default/kiali-deploy : Create resource [rolebinding] on [openshift] ----- 0.93s
default/kiali-deploy : Detect Kiali route on OpenShift ------------------ 0.92s
default/kiali-deploy : Process OpenShift OAuth client ------------------- 0.92s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.92s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.91s
default/kiali-deploy : Create resource [rolebinding-controlplane] on [openshift] --- 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.91s
default/kiali-deploy : Create resource [role] on [openshift] ------------ 0.91s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.88s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.88s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.84s
default/kiali-deploy : Wait for Monitoring Dashboards CRD to be ready --- 0.83s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.83s
default/kiali-deploy : Delete all ConsoleLinks for namespaces that are no longer accessible --- 0.81s
default/kiali-deploy : Find all namespaces (this is limited to what the operator has permission to see) --- 0.81s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.80s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.80s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.79s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.76s
default/kiali-deploy : operator_sdk.util.k8s_status --------------------- 0.74s
default/kiali-deploy : Remove Kiali label from namespaces that Kiali no longer has access to --- 0.69s
default/kiali-deploy : Find current configmap, if it exists ------------- 0.64s
default/kiali-deploy : Find all Monitoring Dashboards packaged with the operator --- 0.60s
Run the version-specific deploy role ------------------------------------ 0.60s
default/kiali-deploy : set_fact ----------------------------------------- 0.55s
default/kiali-deploy : set_fact ----------------------------------------- 0.50s
default/kiali-deploy : Find current deployment, if it exists ------------ 0.48s
default/kiali-deploy : Get information about the cluster ---------------- 0.41s
default/kiali-deploy : Determine the OpenShift version ------------------ 0.39s
default/kiali-deploy : Determine the Kubernetes version ----------------- 0.36s
default/kiali-deploy : Determine which Monitoring Dashboards are to be created --- 0.33s
default/kiali-deploy : Find some current configuration settings --------- 0.30s
default/kiali-deploy : Replace snake_case with camelCase in deployment.hpa.spec --- 0.28s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.node --- 0.27s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod --- 0.26s
default/kiali-deploy : Replace snake_case with camelCase in deployment.resources --- 0.26s
default/kiali-deploy : Set default istio component namespace: grafana --- 0.25s
default/kiali-deploy : Create Kiali objects on OpenShift ---------------- 0.25s
default/kiali-deploy : Replace snake_case with camelCase in deployment.service_annotations --- 0.23s
Playbook start time ----------------------------------------------------- 0.23s
default/kiali-deploy : Replace snake_case with camelCase in deployment.tolerations --- 0.22s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_labels --- 0.22s
default/kiali-deploy : Replace snake_case with camelCase in deployment.affinity.pod_anti --- 0.22s
default/kiali-deploy : Replace snake_case with camelCase in deployment.node_selector --- 0.21s
Determine the version that is to be installed --------------------------- 0.21s
debug ------------------------------------------------------------------- 0.21s
Override the supported images if found in the environment --------------- 0.21s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
default/kiali-deploy : Delete all additional Kiali roles from namespaces that Kiali no longer has access to --- 0.17s
default/kiali-deploy : include_tasks ------------------------------------ 0.17s
default/kiali-deploy : Replace snake_case with camelCase in deployment.pod_annotations --- 0.16s
default/kiali-deploy : include_tasks ------------------------------------ 0.16s
default/kiali-deploy : Only allow ad-hoc kiali namespace when appropriate --- 0.15s
default/kiali-deploy : Set default istio namespace ---------------------- 0.15s
Examine environment and determine if supported image for the requested version is overridden even if no default is known --- 0.15s
default/kiali-deploy : Print some debug information --------------------- 0.13s
Make sure a default supported image or an override image is known ------- 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.13s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : Create Kiali objects on Kubernetes --------------- 0.12s
default/kiali-deploy : Set default deployment namespace to the same namespace where the CR lives --- 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.12s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : Create Route on OpenShift if enabled ------------- 0.11s
default/kiali-deploy : Get current deployment last-updated annotation timestamp from existing deployment --- 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : include_tasks ------------------------------------ 0.11s
default/kiali-deploy : Replace snake_case with camelCase in deployment.additional_service_yaml --- 0.11s

Memory usage:

Resource utilization:

Interestingly enough, by using my parse-profiler script, I can see now that the top-item that takes the most time now is the status update! This is the task that updates the status field of the Kiali CR so the user can track the progress of the reconciliation:
default/kiali-deploy : operator_sdk.util.k8s_status                                                                       17.59
default/kiali-deploy : Create additional Kiali roles on all accessible namespaces on OpenShift                            15.44
default/kiali-deploy : Create additional Kiali role bindings on all accessible namespaces on OpenShift                    14.59
default/kiali-deploy : Create additional Kiali label on all accessible namespaces                                         7.55
default/kiali-deploy : Process OpenShift Console Links                                                                    6.01
default/kiali-deploy : Create the Monitoring Dashboards                                                                   2.26
default/kiali-deploy : include_tasks                                                                                2.16
default/kiali-deploy : Force the Kiali pod to restart if necessary                                                        1.7
default/kiali-deploy : Get information about the operator                                                                 1.68
default/kiali-deploy : Create resource [configmap] on [openshift]                                                         1.31
default/kiali-deploy : Create resource [service] on [openshift]                                                           1.24
default/kiali-deploy : Delete Kiali roles from previously accessible namespaces                                           1.12
...

This is an unexpected result (but also shows how fast I made everything else) - I will look to see what, if anything, can be done to speed that up. Each invocation is relatively fast (about between sub-second and 2 seconds) but the status is updated frequently (to continually update it so we can track progress) so cumulatively it takes a long time. Perhaps I should limit the number of times status is updated during progress, or perhaps there is a way to speed up the update of the status field.

Deleting the CR (i.e. uninstall Kiali and all its resources) is slower than installation - looks like its the removal of the ConsoleLinks that is slow:
===============================================================================
default/kiali-remove : Delete OpenShift-specific Kiali resources ------ 183.00s
default/kiali-remove : Delete Kiali resources -------------------------- 38.75s
default/kiali-remove : Delete all additional Kiali roles in current accessible namespaces -- 10.66s
default/kiali-remove : Remove Kiali label from namespaces found in current accessible namespaces --- 7.79s
default/kiali-remove : Find all namespaces (this is limited to what the operator has permission to see) --- 0.80s
default/kiali-remove : Find currently configured label selector --------- 0.57s
default/kiali-remove : Find current configmap, if it exists ------------- 0.55s
default/kiali-remove : Get information about the cluster ---------------- 0.48s

sped up the removal of consolelinks (this was with 200 namespaces)
Saturday 24 April 2021  16:02:15 +0000 (0:00:00.028)       0:01:17.911 ********
===============================================================================
default/kiali-remove : Delete Kiali resources -------------------------- 45.75s
default/kiali-remove : Delete all additional Kiali roles in current accessible namespaces -- 11.67s
default/kiali-remove : Remove Kiali label from namespaces found in current accessible namespaces --- 7.43s
default/kiali-remove : Delete OpenShift-specific Kiali ConsoleLinks ----- 6.08s
default/kiali-remove : Delete OpenShift-specific Kiali resources -------- 3.61s
default/kiali-remove : Find all namespaces (this is limited to what the operator has permission to see) --- 0.98s
default/kiali-remove : Find current configmap, if it exists ------------- 0.61s
default/kiali-remove : Get information about the cluster ---------------- 0.41s
include_role : {{ version | default(default_playbook.playbook) }}/kiali-remove --- 0.29s
Playbook start time ----------------------------------------------------- 0.20s
debug ------------------------------------------------------------------- 0.19s
default/kiali-remove : Find currently configured label selector --------- 0.13s
default/kiali-remove : Find currently accessible namespaces ------------- 0.11s
default/kiali-remove : Determine the cluster type ----------------------- 0.05s
Log removal processing time --------------------------------------------- 0.04s
default/kiali-remove : Delete Kiali cluster roles ----------------------- 0.04s
Determine the default playbook ------------------------------------------ 0.04s
default/kiali-remove : Print some debug information --------------------- 0.04s
default/kiali-remove : Set default istio namespace ---------------------- 0.03s
default/kiali-remove : Set default deployment namespace to the same namespace where the CR lives --- 0.03s
Playbook end time ------------------------------------------------------- 0.03s
default/kiali-remove : debug -------------------------------------------- 0.03s
debug ------------------------------------------------------------------- 0.03s
debug ------------------------------------------------------------------- 0.03s

Removed some of the status progress updates - now only updates on major milestones. Speeds it up by just not updating the status frequently. Not much we can do per-invocation - it's a direct call to the collection task k8s_status. It is just a fact that it takes about a second on average to update the status field of a resource.

OK, NOW I think I'm done. In this PR we have done several major improvements - all improved the speed of the operator when:

Creating and deleting roles and rolebindings
Creating and deleting OpenShift ConsoleLinks
Creating and deleting the namespace labels
Running the py filter that performs the namespace regex matching to find all accessible namespaces
Updating the status field in the CR
Uninstalling Kiali entirely

Also, as far as my testing shows, there is no memory bloat here.

Great improvement.

Adding 200 new namespaces: 85s
Adding 2000 new namespaces: 584s
Remove 2000 namespaces: 246s
Max kiali operator pod memory usage: 3GiB

There is still one more thing to be discussed. When a namespace is removed from the cluster immediately after removing it from accessible_namespaces (while the reconciliation is still running), the reconciliation might fail with:
TASK [default/kiali-deploy : Remove Kiali label from namespaces that Kiali no longer has access to] ***
Monday 26 April 2021  13:00:51 +0000 (0:00:00.648)       0:00:56.297 **********
fatal: [localhost]: FAILED! => {"changed": false, "error": 403, "msg": "Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"namespaces is forbidden: User \\\\\"system:serviceaccount:kiali-operator:kiali-operator\\\\\" cannot create resource \\\\\"namespaces\\\\\" in API group \\\\\"\\\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"namespaces\"},\"code\":403}\\n'", "reason": "Forbidden", "status": 403}

Next reconciliation will be successful.
Do we want to document this as known issue or do we want to document that it's necessary to wait for reconciliation to finish before removing a namespace from the cluster?

Next reconciliation will be successful.

Oh, that's good! That makes me feel better. At least it doesn't completely break. The operator can eventually reconcile successfully.

Do we want to document this as known issue or do we want to document that it's necessary to wait for reconciliation to finish before removing a namespace from the cluster?

I think we should do both. I think we should document this as a known issue - that this is what will happen if you delete a namespace while reconciliation is ongoing; and that a workaround would be to just wait until after reconciliation is complete before deleting namespaces.
I discussed this with the Ansible Operator SDK folks, and it appears this is a limitation of the SDK and k8s ansible collection task. I was told we have a good use-case but there currently does not seem to be any support for something like this right now. If something is found in the ansible operator SDK that can support this (or one is implemented in the future), we will incorporate it in our operator code - but for now, it looks like we need to document this as a known issue as explained above.

as a result of a conversation I had with the ansible collections/k8s folks: https://github.com/ansible-collections/kubernetes.core/issues/84

@FilipB if we are good with the performance and memory utilization, let's approve this so it can be merged. We can work on the documentation of that namespace-deletion thing in another PR (probably need to document it in kiali.io)


Improve selection handling

This pull request modifies the way ASCollectionView handles selection.

Rename selectedItems to selectedIndexes to reflect the fact the indexes are binded to, not the items themselves.
Allow manual configuration of selection and multiple selection instead of being dependent on the whether or not the view is currently in editing mode.
Add onUpdateSelection callback to ASSection to allow triggering state changes when a section's selection changes.

These changes were merged to master in https://github.com/shareup/ASCollectionView/commit/a5ce0f57ae923ce10c3a8f9e24392d70a0f3986f


feat(nova): add laravel nova docs extension

Description

Adds a new extension to search through Laravel Nova documentation directly in raycast.
Relies on a mix of static sections seen for each version of nova on the site as well as searching against their algolia index.
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

Hey @a-drew! I've moved your screenshot to the metadata folder so it could be used for the Store extension's screenshots.
The only thing is that it has the wrong sizing, could you please update the size? Or just move it back to your /media folder if you don't want to use this image as the Store screenshot.

Cool, thank you!

@mattisssa good to go!


Introduction should clarify that hashes are used in paths to software

This section:
https://github.com/divnix/nix-book/blob/f6448f447d5584854803ac7829e3dd1c00686a32/src/ch02-00-introduction.md?plain=1#L12-L18
is difficult to follow because no mention has yet been made that hashes are used to determine the paths at which software is installed. Many of the claims in this paragraph don't make sense without that context.

;( Fair, I'll think about how to word this in a more clear way.

The first few paragraphs are the hardest to get right, but also the most important. I'm sure it'll take some iterating.
Is it enough to just make it:

This information is aggregated through hashing, and used to determine the software's installation path, allowing nix to describe...


Elm 19 upgrade

@architectcodes okay looks like I am not picking up on the projects prettier settings. I didn't see a config for that so let me know what we need to do for the style changes that I think were changed on save in my editor.

Okay there are too many style changes going on here. Maybe we can coordinate tomorrow and I can get the style rules from you and make sure only the real changes show!


Change observer.stub to include all possible event methods + sort methods alphabetically

Just a short patch for observer.stub to

Include all possible methods
Sort methods alphabetically

I recently generated an observer and was surprised that it did not contain all possible methods. This patch fixes this.

See #32267, #29489

Please see the PR's from above.


Frontend-Project "Youtube_clone" Search Section Design Improvement

Title and Issue number
This PR contains the modified code for frontend project "Youtube_clone" search section design improvement. This PR is the result of following issue:
Frontend-Project "Youtube_clone" Search Section Design Improvement #2458
Close #2458
Checklist:

[x] I have mentioned the issue number in my Pull Request.
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have created a helpful and easy to understand README.md

@Saurabh-kumar09 @pranjay-poddar Kindly review this PR so that I can publish other issues PR.

@MohitGupta121 @Saurabh-kumar09 @pranjay-poddar  Kindly review this PR


Implement clean up strategies

This PR seeks to implement some of the repo clean up strategies discussed in issue #105. The major change in this PR is the integration of the sheet2linkml package as a dependency in the project, and the consequent removal of the vendor directory which previously housed sheet2linkml.

Might overlap with #125


[AutoRelease] t2-containerservice-2024-01-18-75421(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/4847
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=982
BuildTargetingString
azure-mgmt-containerservice
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/4847


I use mixmatch to train my own data set, which is simpler than cifar10 image. After running several epoch, the loss function of training and trainstats will become larger, and the accuracy of test set will be reduced

from utils import Bar, Logger, AverageMeter, accuracy, mkdir_p, savefig，I can‘t find the Bar in utils

I face the same question : the loss of train becomes smaller , but the loss of val becomes larger.
My own dataset has 200 classes

I face the same question : the loss of train becomes smaller , but the loss of val becomes larger.
My own dataset has 200 classes

I face the same question : the loss of train becomes smaller , but the loss of val becomes larger. My own dataset has 200 classes

Hey, do you solve this problem? I meet the same question, is it realted to dataset imbalance?

我也遇见了这个问题，我是用这个方法做分割


devise/sessions/create

I hit an error where the code wanted a devise/sessions/create view. So I added it. But it breaks the after_sign_in_path_for.
If I add the create view, it redirects 302 to sign_in path.
I step traced the stack. The after_sign_path_for does fire. But the path gets set to sign_in if that create view is in the devise/sessions directory.

Hi @starchild33, thanks for the issue report.
Could you please provide us a sample application that reproduces the issue in isolation?
That would help us find the issue.

I'm closing this issue because it had no recent activity.
If you're still facing this on the latest version, please open a new one with all the information requested in the template.
Thank you!


Typo "Azure function"→"Azure Function"

https://docs.microsoft.com/en-us/azure/logic-apps/logic-apps-add-run-inline-code

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi @hyoshioka0128, did you have questions about my requested changes to your PR? If not, please go ahead and accept the changes. I can then sign off. Thank you!

Hi @hyoshioka0128, did you have questions about my requested changes to your PR? If not, please go ahead and accept the changes. I can then sign off. Thank you!

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

#sign-off


GraphQL errors in docker environment

Hi there! I'm attempting to use the docker container development environment so that I can set up a demo of how OpenCollective can be used to support a local community.
I'm currently running into a few errors. I mitigated one with https://github.com/opencollective/opencollective-api/pull/1311, but I'm now running into a GraphQL error  when I visit a collective page.
Screenshot

Logs
frontend_1  | info: GET /opensource 200 61ms
frontend_1  | error: >>> Error calling API http://localhost:13060/graphql?api_key=dvl-1510egmf4a23d80342403fb599qd {"meta":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect","address":"<IP_ADDRESS>","port":13060}}
frontend_1  | info: POST /api/graphql 500 12ms

Any thoughts? Thanks again!

Got it sorted: The INTERNAL_API_URL envvar must be added and set to http://api:3060 so that the frontend server can find it:
https://github.com/opencollective/opencollective-api/blob/master/docker/docker-compose.yml#L32
https://github.com/opencollective/frontend/blob/master/src/lib/utils.js#L211
Also, as it turns out, the database dump has some old image sources in it, that the frontend fails on when it finds (this only affects the sample opensource collective in the sample dev database, but apex works fine)
For example, this image doesn't exist anymore, but is in the dump:
https://images.githubusercontent.com/farhd
Any thoughts on how to improve this second hitch?

@patcon maybe the database dumps needs to be refreshed. Moving the discussion to: https://github.com/opencollective/opencollective-api/issues/1314


IMPROVE FEEDBACK BUTTON STYLING AND INCREASE MESSAGE AREA

Describe the feature
Change CSS properties to style the feedback button
Expanded the message area
Add ScreenShots
before:

after:

Please Assign it to me
Thank you @Suchitra-Sahoo
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

I want to work on this project. Please assign it to me.


Nix: gBar build failure

Please fill out this form and delete the defaults(everything not in bold font), so I can help you better
Describe the bug
Build Fails with the flake
When revertnig the flake to the old commit some days ago but the rest of the system the same it works.
Steps to Reproduce
Build latest gBar
Expected behavior
Builds successfully
Screenshots/Error logs
last 10 log lines:
> /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: :(.text.startup+0x127): undefined reference to Window::Run(int, char**)'        > /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: <artificial>:(.text.startup+0x12c): undefined reference to System::FreeResources()'
> /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: :(.text.startup+0x13d): undefined reference to Window::~Window()'        > /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: <artificial>:(.text.startup+0x1ac): undefined reference to Bar::Create(Window&, int)'
> /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: :(.text.startup+0x238): undefined reference to Plugin::LoadWidgetFromPlugin(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Window&, int)'        > /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: <artificial>:(.text.startup+0x314): undefined reference to Logging::Log(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
> /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: :(.text.startup+0x391): undefined reference to Logging::Log(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'        > /nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ld: <artificial>:(.text.startup+0x3d2): undefined reference to Window::~Window()'
> collect2: error: ld returned 1 exit status
> ninja: build stopped: subcommand failed.
For full logs, run 'nix log /nix/store/k7bh10a2akn4rc0ybcp5axilny8152jg-gbar.drv'.
Information about your system and gBar

OS: [NixOS]
Desktop environment [Hyprland]
commit sha256 if possible [c536e43cd095db536b5bcb5a5c6f727723e60a50

Should now be fixed. Issue was missing gcc-ar script in path required by gcc for static-library LTO (Link-time-Optimization). LTO is now disabled fixing the issue.
Please reopen/ping if the issue persists.


K8S 1.18 cass-operator unable to pull image.

What Happened
Following the installation guides deployment of the cass-operator using cass-operator-manifests-v1.18.yaml fails to pull a valid image from the repository.   Causing the operator to fail to start.
pullerror.log
(K8S version 1.18.6 )

We've observed this before, and we've been trying to isolate what creates the problem. This is the issue...
  Warning  Failed     3m1s (x4 over 4m35s)   kubelet, ip-10-0-0-80.eu-west-2.compute.internal  Failed to pull image "datastax/cass-operator:1.4.1": rpc error: code = FailedPrecondition desc = failed to pull and unpack image "docker.io/datastax/cass-operator:1.4.1": failed commit on ref "layer-sha256:391dba5ec43113396bb1a701c2b3c8a160c560076e13a1c51759d35b4a6a64ae": "layer-sha256:391dba5ec43113396bb1a701c2b3c8a160c560076e13a1c51759d35b4a6a64ae" failed size validation: 146 != 147: failed precondition

Was this running with docker? or containerd?

containerd

@RobbieJVMW can you try again? I fixed some images that containerd checks wouldn't allow. It seems like we were effected by the same bug reported here - https://github.com/containers/skopeo/issues/1054

Looks like the operator has successfully deployed.  I'll run some full tests to confirm.

@RobbieJVMW let me know if we should reopen this.

nope need to re-open looks good.


Unhandled error during execution of scheduler flush.

Version
3.2.33
Reproduction link
https://oneconnect.it/
Steps to reproduce
sign up and go to payroll and then earnings and then try to add an earning
What is expected?
i have no clue
What is actually happening?
runtime-core.esm-bundler.js?9e79:6568 [Vue warn]: Unhandled error during execution of scheduler flush. This is likely a Vue internals bug. Please open an issue at https://new-issue.vuejs.org/?repo=vuejs/vue-next
at
at <QCard style= {max-width: '500px', min-height: '100vh'} flat="" square="" >
at <QLayout style= {width: '500px !important'} onCloseDialogue=fn type="earning"  ... >
at <AddEditEarning onCloseDialogue=fn type="earning" currentUser= {id: 2, username: 'devtest', email<EMAIL_ADDRESS>name: 'Demo', phoneNumber: '+9647517695800', …}  ... >
at <BaseTransition appear=true persisted=false mode=undefined  ... >
at
at <QDialog seamless="" position="right" modelValue=true  ... >
at <QScrollArea style= {max-height: 'auto !important', min-height: 'auto !important', height: 'calc(100vh - 131px)'} class="q-px-md" >
at
at <Earnings class="full-height" onVnodeUnmounted=fn ref=Ref< Proxy {$i18n: {…}, $t: ƒ, $rt: ƒ, $tc: ƒ, …} > >
at
at <QPageContainer class="bg-silk" style= {margin: '0px', width: 'calc(100% - 245px) !important', transform: 'translateX(245px)'} >
at
at <MainLayout onVnodeUnmounted=fn ref=Ref< Proxy {$i18n: {…}, $t: ƒ, …} > >
at
at

We can't debug live sites.
Please follow the issue guidelines.


Make it impossible to pass as input some type which was generated only to preserve forwards compatibility but was not meant to be used as input

Use case
For a schema which contains these
type Mutation {
    someMutation(input: SomeInput!): Irrelevant!
}

input SomeInput {
    otherField: ID!
    enumInput: SomeEnum
}

enum SomeEnum {
    FOO,
    BAR,
}

Apollo-kotlin generates this code for SomeEnum:
public enum class SomeEnum(
  public val rawValue: String,
) {
  FOO("FOO"),
  BAR("BAR"),
  /**
   * Auto generated constant for unknown enum values
   */
  UNKNOWN__("UNKNOWN__"),
  ;

  public companion object {
    public val type: EnumType = EnumType("SomeEnum", listOf("FOO", "BAR"))

    public fun safeValueOf(rawValue: String): ChatMessageContext =
        values().find { it.rawValue == rawValue } ?: UNKNOWN__

    /**
     * Returns all [SomeEnum] known at compile time
     */
    public fun knownValues(): Array<SomeEnum> = arrayOf(
      FOO,
      BAR,
    )
  }
}

This makes it possible in user code to try and pass to a mutation this:
apolloClient.mutation(SomeMutation(input = SomeInput(otherField = ..., enumInput = SomeEnum.UNKNOWN__))

This however will always fail on the backend validation since this entry is not part of the real schema, and there is nothing to map it to.
Describe the solution you'd like
The problem here is just that this is possible and the compiler is not warning us about trying to run something that would never work.
Optimally, this code should not compile in the first place.
If this turns out to be way too complicated, even a lint warning or something similar might be a good first step. But as of right now making this mistake is unfortunately something that can happen silently.

For the record, a possible solution is to use sealedClassesForEnumsMatching.set(listOf(".*")) (probably make it the default in v4) and have the UNKNOWN__(rawValue) constructor be annotated with an @OptIn annotation  so that it's harder to call from user code.

Heads up that we added the @OptIn annotation to the unknown constructor in #5813. This should be available in the v4 snapshots shortly!
For now we haven't decided on making sealed classes for enums the default in v4.

After wome back and forth, in #5904 we ended up removing the @OptIn annotation, and instead removed the UNKNOWN__ constructor. Now safeValueOf should be used in the rare case where one needs to construct unknown values.

Closing as this is available to try with the v4 betas.


Propagate stablehlo.reshape and stablehlo.transpose through unary operations

Some operations do not manipulate shape at all (specifically unary operations). In these cases we should push shape manipulations through the unary op. E.g.
func.func @main(%arg0 : tensor<4x4xf32>) -> tensor<4x4xf32> {
  %0 = stablehlo.reshape(%arg0) : (tensor<4x4xf32>) -> tensor<16xf32>
  %1 = stablehlo.cos(%0) : (tensor<16xf32>) -> tensor<16xf32>
  %2 = stablehlo.reshape(%1) : (tensor<16xf32>) -> tensor<4x4xf32>
  return %2 : tensor<4x4xf32>
}

Has shape manipulations that are unnecessary. These can be corrected by pushing the shape manipulation the elementwise operations:
func.func @main(%arg0 : tensor<4x4xf32>) -> tensor<4x4xf32> {
  %0 = stablehlo.cos(%arg0) : (tensor<4x4xf32>) -> tensor<4x4xf32>
  %1 = stablehlo.reshape(%0) : (tensor<4x4xf32>) -> tensor<16xf32>
  %2 = stablehlo.reshape(%1) : (tensor<16xf32>) -> tensor<4x4xf32>
  return %2 : tensor<4x4xf32>
}

And ultimately simplified to:
func.func @main(%arg0 : tensor<4x4xf32>) -> tensor<4x4xf32> {
  %0 = stablehlo.cos(%arg0) : (tensor<4x4xf32>) -> tensor<4x4xf32>
  return %0 : tensor<4x4xf32>
}

This is also true with stablehlo.transpose.

(+1 this can really help clean up IR and makes further stages easier to read as less of this happens in the more verbose forms!)

@NatashaKnk Marking as a P1 for you - please edit as needed.

First part covered in 14494
Is there anybody working on the second part? (i.e. merging the shape manipulation ops together)
I can also pick it up in a bit but as we've discussed it might be a good first bug :)

I'd be happy to pick up the second part
P.S. if one can flesh out a bit more details - this would be very helpful


Failed to resolve component: vuedatepicker

Hi guys,
Since I need to build a new page using Vue3 that include a new datapicker component, I have a issue unable to perform normally.
the information shown in the console.
[Vue warn]: Failed to resolve component: vuedatepicker

<!DOCTYPE html>

    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
    <link rel="stylesheet" href="https://unpkg.com/<EMAIL_ADDRESS>

  {{ message }}
  <VueDatePicker ></VueDatePicker>

The documentation has the wrong example, In the HTML part, instead VueDatePicker use Datepicker, or just replace
components: { Datepicker: VueDatePicker },
with components: {  VueDatePicker },


How to build, distribute and run clients

Try to script and add instructions how to build your own Anytype client from source to connect to your selfhosted node

Just a note, on a future release the clients built and distributed by AnyType will support changing the network to a local one, so this is not really required.
See this.

You are right, but I'm not sure what will be there first (docker image vs custom storage location). As I want to become familiar with docker image building instead of only leeching of the work of others, I want to give it a try with the current Anytype version.
Thanks for thinking along!


add workflow for editing mouse acceleration

Discord username
mikikiv#7340
Description of changes
Added new workflow for MacOS 13+. Since there is no UI method of fully disabling mouse acceleration a terminal command is the best alternative.
Echo out their old scaling amount before and after, in case the user decides they don't like their new setting, or if the command doesn't work (because of using wrong OS version, for example)

Restarting the device resets this value from -1 to a default, which is why this command is often used

@mikikiv thanks for opening this. I fixed the broken CI so will merge.


Add a free-text "Usage Terms" property?

Annette Feldman of AP pointed out that we don't seem to have a generic free-text property for describing usage rights a la the Rights Usage Terms property in IPTC Photo Metadata.
Is this deliberate or should we think about adding it?

We discussed this week - we agree that it's useful and should add in the next version.


Does not get crash report file when crash happened in simulator.

Hello,
I am not able to get the crash report for the crash that was happened on simulator. I tried this on the advanced-example that was given with the SDK.
Please help..

Make sure you're running the simulator by itself, not via an Xcode debug session. When it detects a debug session, it disables the crash handlers so that it doesn't interfere with the debugger.

Okay.
This is applicable for device also? Means, when I run the app in debug mode on device, it will not capture the crash logs?
Because I want to do debug after I get the crash log, I need to some process which I need to debug. Please suggest alternative if any..

You must run outside of the debugger to get a crash report. After the cash you can run it under a debugger again because the report has been written.

Great.
Thanks for quick response. It works...


fix: Prefetch apis: Include only branchname header in request key

Description
The request key used to store the mutex for prefetch request was including all header keys. The prefetch request created by the service worker only had one key (branchname) but the request initiated by the client had more headers. Because of this mismatch in keys the request was missing the cache.
Fixes #Issue Number
or
Fixes Issue URL

[!WARNING]
If no issue exists, please create an issue first, and check with the maintainers if the issue is valid.

Automation
/ok-to-test<EMAIL_ADDRESS>:mag: Cypress test results

[!IMPORTANT]
🟣 🟣 🟣 Your tests are running.
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/9612628976
Commit: e9c4a982eddded5dce31005365978b4729dadba2
Workflow: PR Automation test suite
Tags: ``

Communication
Should the DevRel and Marketing teams inform users about this change?

[ ] Yes
[ ] No

/build-deploy-preview skip-tests=true


Dang/add tokenfactory module

add module tokenfactory

Thank you for your PR!
This code looks like a version that you copied from osmosis.
Although I do understand the demand to have this code available, we should work on a process that includes the Osmosis team as code owners and maintainers.
Please contribute to #911 to discuss and follow up on this.

Thank you for your PR! This code looks like a version that you copied from osmosis. Although I do understand the demand to have this code available, we should work on a process that includes the Osmosis team as code owners and maintainers. Please contribute to #911 to discuss and follow up on this.

Thank you for your reviews! @alpe

Hi guys, the osmosis team is well aware that we are using and promoting the token factory code in the ecosystem.
In fact just about everybody that we work with is really keen on this particular pull request

@faddat thanks for the additional context! I appreciate the effort and like us to continue the discussion on the issue.
A PR is often pushing a concrete solution while the problem description and motivation can be more inspiring and helpful to come up with different solutions. In this case though it is more about go.mod vs embeding and prioritization of course.


Allow icon to be customised through config

This change keeps the icon as it is by default, but would optionally allow it to be customised in the config file.
My reason for suggesting this change is that currently the icon is the same as that used by the nova settings package.

Would it be acceptable to provide it during registration?
    public function tools()
    {
        return [
            MenuBuilder::make()
                ->title('Custom Title'),
                ->icon('another-icon'),
        ];
    }

Keeps the config file less bloated and the tool is initiated only once anyways.

Merged via #161


Serialize collection/model before passing it to views

The collection or model was not serialized into json before passed to a
view, this made it a little cumbersome accessing model fields in the
view template.
Example in handlebars accessing field of first model in collection
before this commit:
collection.models.0.attributes.name
After the commit fields can be accessed "normally":
items.1.name

+1

Seems reasonable. Care to update the merge conflicts?


Add imaging timestamps using new conversion option always_write_timestamps

To be reviewed after merging #15
Add imaging timestamps using new conversion option always_write_timestamps

The other PRs can be merged now. Let me know when to take a look at this.

You can review this already, it is pretty straight forward

I am confused because I see a lot of code that I think we already merged? Let me check

I am confused because I see a lot of code that I think we already merged? Let me check

Sometimes it does not update right away. Now it should be ok


[#125] - Enhancing clustering functionality

Transfering rabbitmq-cluster cookbook into rabbitmq cookbook

Changing attribute prefix : [:rabbitmq-cluster] to [:rabbitmq][:clustering]
Enhancement on regex : user ruby matcher
Removing unnecessary resource : node_type on change_cluster_node_type action
Some other enhancement on provider

Enhancing from rabbitmq-cluster as the manual clustering

Removing master/slave concept
Removing cluster node type
Introducing new attribute that defines all the cluster nodes and its cluster node type : [:rabbitmq][:clustering][:cluster_nodes]
Adding set_cluster_name action

Applying new features and attributes into auto clustring

Using default[:rabbitmq][:clustering][:cluster_nodes] attribute instead of ['rabbitmq']['cluster_disk_nodes']
Applying set_cluster_name and change_cluster_node_type for auto clustering
Long story short, All the nodes will be set as disc node in the rabbitmq.config even though you have some ram nodes. however, change_cluster_node_type action will fix the node type as you expected.

Additional changes on the others

Fixing typo in default receipe for stopping service to set erlang cookie

Attributes example
default[:rabbitmq][:clustering][:use_auto_clustering] = false
default[:rabbitmq][:clustering][:cluster_name] = 'seoul_tokyo_newyork'
default[:rabbitmq][:clustering][:cluster_nodes] = [
    {
        :name => 'rabbit@rabbit1',
        :type => 'disc'
    },
    {
        :name => 'rabbit@rabbit2',
        :type => 'ram'
    },
    {
        :name => 'rabbit@rabbit3',
        :type => 'disc'
    }
]

Thank you for taking the time to work on this. I am taking a look through it now.
Can you fix the style issues that travis is complaining about?
Also I think we might want to a section to the README describing how to use these changes.

@cmluciano
Sure, I will.

@cmluciano I fixed style issue and rubocop issue. and Travis has passed. please review it.
Thank you.

@jjasghar @cmluciano
seems like,,,, Travis has been stopped by connection timed out...
https://travis-ci.org/jjasghar/rabbitmq/builds/53843704

I restarted it, lets see if it passes.

@jjasghar Looks good!

:+1:

Can we please get this reviewed once more and merged? Thanks!