schedule job in pipeline does not start after upgrade to 3.14.2 with enterprisegateway 3.2.0 and elyra/kernel-py:3.2.0

After upgrade of below components, kubernetes pods stay in hanged state.

Component
Current Version
Target version

IBM Elyra
elyra/elyra:3.1.1
elyra/elyra:3.14.2

Jupyter enterprise gateway
elyra/enterprise-gateway:2.5.1
elyra/enterprise-gateway:3.2.0

python kubernetes kernel
elyra/kernel-py:2.5.1
elyra/kernel-py:3.2.0

In the upgraded environment pipeline that uses kernel 2.5.1 works as before.
However, same python notebook and pipeline when used with kernel 3.2.0 hangs at below state.
Executing:   0%|          | 0/34 [00:00<?, ?cell/s][I 14:45:16.137] Executing notebook with kernel: python_kubernetes [2023-03-21, 14:49:35 UTC] {pod_manager.py:228} INFO
To Reproduce
not sure, if this is reproducable. But, in this environment, I just need to change the version number of the elyra/kernel-py and the same notebook stops working.
Screenshots or log output
If applicable, add screenshots or log output to help explain your problem.
Log Output

It seems to successfully initial task_executor pod, which then starts the next pod to run this notebook. it downloads all relevant files and then just when starting the notebook it hangs.
-------------------------------------------------------------------------------
[2023-03-21, 14:45:06 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-03-21, 14:45:06 UTC] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-03-21, 14:45:06 UTC] {taskinstance.py:1383} INFO - Executing <Task(KubernetesPodOperator): test_script_v1> on 2023-03-21 14:44:48.247613+00:00
[2023-03-21, 14:45:06 UTC] {standard_task_runner.py:55} INFO - Started process 98 to run task
[2023-03-21, 14:45:06 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test_script_v1.1.2_testimage_upgrade-0315223443', 'test_script_v1', 'manual__2023-03-21T14:44:48.247613+00:00', '--job-id', '8029', '--raw', '--subdir', 'DAGS_FOLDER/test_script_v1.1.2_testimage_upgrade-0315223443.py', '--cfg-path', '/tmp/tmpb28djnk9']
[2023-03-21, 14:45:06 UTC] {standard_task_runner.py:83} INFO - Job 8029: Subtask test_script_v1
[2023-03-21, 14:45:07 UTC] {task_command.py:376} INFO - Running <TaskInstance: test_script_v1.1.2_testimage_upgrade-0315223443.test_script_v1 manual__2023-03-21T14:44:48.247613+00:00 [running]> on host testscriptv112testimageupgrade-6aa1460b24ce40b88edb1878f993e827
[2023-03-21, 14:45:08 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test_script_v1.1.2_testimage_upgrade-0315223443
AIRFLOW_CTX_TASK_ID=test_script_v1
AIRFLOW_CTX_EXECUTION_DATE=2023-03-21T14:44:48.247613+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-03-21T14:44:48.247613+00:00
[2023-03-21, 14:45:08 UTC] {kubernetes_pod.py:587} INFO - Creating pod test-script-v1-0b5591e1b8a64a828f7b77c6133846d9 with labels: {'dag_id': 'test_script_v1.1.2_testimage_upgrade-0315223443', 'task_id': 'test_script_v1', 'run_id': 'manual__2023-03-21T144448.2476130000-cdb0af080', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-03-21, 14:45:08 UTC] {kubernetes_pod.py:380} INFO - Found matching pod test-script-v1-0b5591e1b8a64a828f7b77c6133846d9 with labels {'airflow_kpo_in_cluster': 'True', 'airflow_version': '2.4.3', 'dag_id': 'test_script_v1.1.2_testimage_upgrade-0315223443', 'kubernetes_pod_operator': 'True', 'run_id': 'manual__2023-03-21T144448.2476130000-cdb0af080', 'task_id': 'test_script_v1', 'try_number': '1'}
[2023-03-21, 14:45:08 UTC] {kubernetes_pod.py:381} INFO - `try_number` of task_instance: 1
[2023-03-21, 14:45:08 UTC] {kubernetes_pod.py:382} INFO - `try_number` of pod: 1
[2023-03-21, 14:45:08 UTC] {pod_manager.py:180} WARNING - Pod not yet started: test-script-v1-0b5591e1b8a64a828f7b77c6133846d9
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO - Downloading https://raw.githubusercontent.com/elyra-ai/elyra/v3.14.2/elyra/airflow/bootstrapper.py
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -                                  Dload  Upload   Total   Spent    Left  Speed
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 21173  100 21173    0     0  89716      0 --:--:-- --:--:-- --:--:-- 89337
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO - Downloading https://raw.githubusercontent.com/elyra-ai/elyra/v3.14.2/etc/generic/requirements-elyra.txt
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO - Downloading https://raw.githubusercontent.com/elyra-ai/elyra/v3.14.2/etc/generic/requirements-elyra-py37.txt
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -                                  Dload  Upload   Total   Spent    Left  Speed
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1320  100  1320    0     0   7292      0 --:--:-- --:--:-- --:--:--  7292
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2023-03-21, 14:45:09 UTC] {pod_manager.py:228} INFO -                                  Dload  Upload   Total   Spent    Left  Speed
[2023-03-21, 14:45:10 UTC] {pod_manager.py:228} INFO -
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1363  100  1363    0     0   5951      0 --:--:-- --:--:-- --:--:--  5951

[2023-03-21, 14:45:14 UTC] {pod_manager.py:228} INFO - test_script_v1.1.2.ipynb
[2023-03-21, 14:45:14 UTC] {pod_manager.py:228} INFO - tar: Removing leading `/' from member names
[2023-03-21, 14:45:14 UTC] {pod_manager.py:228} INFO - [I 14:45:14.609] 'test_script_v1.1.2_testimage_upgrade':'test_script_v1.1.2' - dependencies processed (0.023 secs)
[2023-03-21, 14:45:14 UTC] {pod_manager.py:228} INFO - [I 14:45:14.609] 'test_script_v1.1.2_testimage_upgrade':'test_script_v1.1.2' - executing notebook using 'papermill test_script_v1.1.2.ipynb test_script_v1.1.2-output.ipynb'
[2023-03-21, 14:45:14 UTC] {pod_manager.py:228} INFO - [I 14:45:14.804] Input Notebook:  test_script_v1.1.2.ipynb
[2023-03-21, 14:45:14 UTC] {pod_manager.py:228} INFO - [I 14:45:14.804] Output Notebook: test_script_v1.1.2-output.ipynb
[2023-03-21, 14:45:16 UTC] {pod_manager.py:228} INFO -
Executing:   0%|          | 0/34 [00:00<?, ?cell/s][I 14:45:16.137] Executing notebook with kernel: python_kubernetes
[2023-03-21, 14:49:35 UTC] {pod_manager.py:228} INFO -

Expected behavior
pod_manager.py should execute the notebook as the one with older kernel and should show progress as below.
[2023-03-21, 13:51:00 UTC] {pod_manager.py:228} INFO -
Executing:   3%|▎         | 1/34 [00:01<00:48,  1.47s/cell]
Executing:   9%|▉         | 3/34 [00:24<04:36,  8.92s/cell]
Executing:  15%|█▍        | 5/34 [00:24<02:06,  4.36s/cell]
Executing:  21%|██        | 7/34 [00:24<01:08,  2.54s/cell]
Executing:  26%|██▋       | 9/34 [00:24<00:40,  1.60s/cell]
Executing:  32%|███▏      | 11/34 [00:25<00:26,  1.17s/cell]
Executing:  38%|███▊      | 13/34 [00:28<00:27,  1.31s/cell]
Executing:  44%|████▍     | 15/34 [00:29<00:18,  1.01cell/s]
Executing:  47%|████▋     | 16/34 [00:29<00:14,  1.21cell/s]
Executing:  56%|█████▌    | 19/34 [00:29<00:07,  2.09cell/s]
Executing:  62%|██████▏   | 21/34 [00:29<00:04,  2.81cell/s]
Executing:  68%|██████▊   | 23/34 [00:29<00:03,  3.65cell/s]
Executing:  74%|███████▎  | 25/34 [00:30<00:01,  4.74cell/s]
Executing:  79%|███████▉  | 27/34 [00:35<00:06,  1.08cell/s]
Executing:  79%|███████▉  | 27/34 [00:36<00:09,  1.36s/cell]

Deployment information
Describe what you've deployed and how:

Elyra version: provided above
Operating system: MS AKS with kubernetes 1.25
Installation source:  docker images provided above
Deployment type: Kubernetes , Elyra configured with enterprise-gateway to run with elyra/kernel-py

Pipeline runtime environment

Apache Airflow (2.4.3). I know this is not official supported. But same setup works fine to schedule jobs as far as airflow is concerned. Initial pod and then notebook pods get scheduled. it gets stuck when pod_manager.py works with new version of elyra/kernel-py image

Runtime configuration settings
If the issue is related to pipeline execution, document the runtime configuration settings from the Elyra UI, omitting confidential information.
Right now there are no logs, or error messages to check. It is just stuck. Even any hint on how to get any info to investigate would be helpful.

Thanks for the additional information.  So, based on that response and given your issue only occurs when you use the Airflow runtime (and not the local runtime via EG), I think we can eliminate EG from the equation and this boils down to why using elyra/kernel-py:2.5.1 works and elyra/kernel-py:3.2.0 doesn't.  Would you agree that this is a correct summary?
Based on your screenshots, it doesn't look like you use the "elrya/kernel-py" images directly when configuring your runtime image metadata, but instead, extend those images with something else.  Is the Dockerfile used to extend the "elyra/kernel-py" images identical relative to the 2.5.1 and 3.2.0 base images?
Per this information,
I 14:45:16.137] Executing notebook with kernel: python_kubernetes

your notebook file (e.g., test_script_v1.1.2.ipynb) references a kernel named python_kubernetes.  Does that kernelspec exist in the runtime-image and, if so, can you provide its content?
If it is the same content as what is defined on your EG server this will not work.  The reason is because that particular kernelspec is going to try to launch the kernel as a k8s pod and, because EG is not available (and papermill is unaware that it should use a custom engine to talk to EG), the interaction with the kernel will be lost.  I'm actually surprised papermill is hanging and not erroring out.
((Hmm, can you check for orphaned pods?  Their names would likely be odd placeholders like "None" or strange things like that.  I just wonder if the launch-kubenetes.py script that is part of the python_kubernetes kernelspec machinery is able to get a pod off the ground or not.))
I also can't explain why doing the same thing with elyra/kernel-py:2.5.1 even works since it would essentially be the same issue.  You're trying to use a kernel image as a notebook node image when all you really need in this case is the default python3 kernel.  Might you NOT have transferred python_kubernetes to the notebook node image when elyra/kernel-py:2.5.1 is used as the base image?
Along those lines, you might try removing python_kubernetes from your runtime image.  Then, when the elyra bootstrapper goes invokes papermill, it will determine the referenced kernel (python_kubernetes) does not exist and will try to run the notebook cells using python3.  This kernel should load and run locally (and not try to run the kernel in pod) and since you will have the same packages that you had when running via EG, your notebook cells should have the packages they need.
Looking at the bootstrapper code used in Airflow, we're not seeing the log_operation_info messages, which are enabled by default.  Are you disabling that logging by setting env ELYRA_ENABLE_PIPELINE_INFO to something other than "true" for the container?  This would need to be specified in the notebook node's properties to disable it, so I doubt that's happening and it's another mystery why we aren't seeing the logging the bootstrapper performs outside of papermill.  If properly enabled (which is the default), we should see this just prior to invoking papermill:
executing notebook using 'papermill test_script_v1.1.2.ipynb test_script_v1.1.2-output.ipynb'

I am not sure what you mean when you say "kernelSpec that prefers elyra/kernel-py2.5.1"

Sorry about that.  I had originally stated "kernelSpec that preferences elyra/kernel-py2.5.1", but edited it to say "kernelSpec that references elyra/kernel-py2.5.1".  I hope that clarifies things a bit.

Ok, that gave me lot to read and think and also meant that I understand this whole setup a bit more 😄
yes, I agree that EG is not in scope and issue is only about elyra/kernel-py versions.
You are right, we do not directly use elyra/kernel-py image. Main reasons to use the image were to
a) avoid matching all library versions when using other python images
b) we set an environment variable in the image, which helps identify if the notebook is running as a background job or in an interactive environment. Accordingly we have some filters inside notebook.
c) we needed to install ipykernel with name python_kubernetes as notebook needed this named version to run.
Other than that those custom images  2.5.1  and  3.2.0 follow the same structure.
Regarding python_kubernetes, I am not sure about the exact flow and reason for this. But, one or the other way this is linked to EG python kernel as it has name python_kubernetes which it pulls as one of the standard kernel. When running notebook interactively, this kernel is selected ; Image 1 below. In the custom elyra/kernel-py image we install ipykernel by giving it name python_kubernetes.  RUN python -m ipykernel install --user --name python_kubernetes  .What I do not know is if it is needed because notebook at the time of submitting pipieline stores that information somewhere or is comes from requirement of elyra/kernel-py image that it needs a name matching with as defined in EG. This naming is same for the notebook
Image 1: python_kubernetes kernel selection is notebook in Elyra

Regarding to be able to run pods, that functionality is working, especially in case of old image once pod is running it actually runs the notebook. Below pods get created in k8s cluster.

Here test-script-v1 is the actual pod (started by the other pod) trying to run the notebook, which then hangs with last log message as given above and repeated here
tar: Removing leading `/' from member names
 21:34:58.368] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - dependencies proessed (0.023 secs)                                                                       [I 21:34:58.369] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - executing notebook using 'papermill test_script_v1.1.2.ipynb test_script_v1.1.2-output.ipynb'
 [I 21:34:58.581] Input Notebook:  test_script_v1.1.2.ipynb
 [I 21:34:58.581] Output Notebook: test_script_v1.1.2-output.ipynb
Executing:   0% 0/34 [00:00<?, ?cell/s][I 21:34:59.761] Executing notebook with kernel: python_kubernetes

and btw, this ipykernel name exists inside this running pod. (And I get to see that, as it just hangs unlike the working image 2.5.11 which finishes its work, before I can login
(base) jovyan@test-script-v1-3d20b63e5c98491592fe52c914025046:~$ jupyter kernelspec list
Available kernels:
  python_kubernetes    /home/jovyan/.local/share/jupyter/kernels/python_kubernetes
  python3              /opt/conda/share/jupyter/kernels/python3

Along those lines, you might try removing python_kubernetes from your runtime image. Then, when the elyra bootstrapper invokes papermill, it will determine the referenced kernel (python_kubernetes) does not exist and will try to run the notebook cells using python3. This kernel should load and run locally (and not try to run the kernel in pod) and since you will have the same packages that you had when running via EG (since it's the same image), your notebook cells should have the packages they need.

As I see it, once this test-script-v1 pod is running it is not trying to start the kernel as one more pod. it runs inside this pod, but simply uses that name python_kubernetes to select the kernel (probably that name gets stored somewhere with notebook config) . Same can be seen with elyra/kernel-py:2.5.1 image.
Executing:   0%|          | 0/34 [00:00<?, ?cell/s][I 21:52:30.587] Executing notebook with kernel: python_kubernetes
[2023-03-21, 21:52:35 UTC] {pod_manager.py:228} INFO -
Executing:   3%|▎         | 1/34 [00:01<00:50,  1.53s/cell]
Executing:   9%|▉         | 3/34 [00:02<00:23,  1.31cell/s]
Executing:  24%|██▎       | 8/34 [00:02<00:05,  4.43cell/s]
Executing:  32%|███▏      | 11/34 [00:03<00:05,  3.99cell/s]
Executing:  38%|███▊      | 13/34 [00:04<00:07,  2.63cell/s]
Executing:  53%|█████▎    | 18/34 [00:05<00:03,  4.88cell/s]
Executing:  68%|██████▊   | 23/34 [00:05<00:01,  6.98cell/s]
Executing:  79%|███████▉  | 27/34 [00:05<00:00,  9.17cell/s]
Executing:  88%|████████▊ | 30/34 [00:05<00:00, 11.08cell/s]
Executing:  97%|█████████▋| 33/34 [00:05<00:00, 13.05cell/s]
Executing: 100%|██████████| 34/34 [00:06<00:00,  5.40cell/s]
[2023-03-21, 21:52:35 UTC] {pod_manager.py:228} INFO - [I 21:52:35.386] 'test_script_v1.1.2_old':'test_script_v1.1.2' - notebook execution completed (6.640 secs)
[2023-03-21, 21:52:35 UTC] {pod_manager.py:228} INFO - [I 21:52:35.588] 'test_script_v1.1.2_old':'test_script_v1.1.2' - converting from test_script_v1.1.2-output.ipynb to test_script_v1.1.2.html
[2023-03-21, 21:52:36 UTC] {pod_manager.py:228} INFO - [I 21:52:36.312] 'test_script_v1.1.2_old':'test_script_v1.1.2' - test_script_v1.1.2-output.ipynb converted to test_script_v1.1.2.html (0.723 secs)

Looking at the bootstrapper code used in Airflow, we're not seeing the log_operation_info messages, which are enabled by default. Are you disabling that logging by setting env ELYRA_ENABLE_PIPELINE_INFO to something other than "true" for the container? This would need to be specified in the notebook node's properties to disable it, so I doubt that's happening and it's another mystery why we aren't seeing the logging the bootstrapper performs outside of papermill. If properly enabled (which is the default), we should see this just prior to invoking papermill:

Sorry about that, I trimmed the logs above, just keeping top and bottom. Full log below.  The last error message at the bottom is reported when I manually kill the pod.
  Full logs

*** Reading remote log from s3://airflow-logs/dag_id=test_script_v1.1.2_testimage-0321153312/run_id=manual__2023-03-21T21:34:35.872533+00:00/task_id=test_script_v1/attempt=1.log.
[2023-03-21, 21:34:50 UTC] {taskinstance.py:1165} INFO - Dependencies all met for
[2023-03-21, 21:34:50 UTC] {taskinstance.py:1165} INFO - Dependencies all met for
[2023-03-21, 21:34:50 UTC] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2023-03-21, 21:34:50 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2023-03-21, 21:34:50 UTC] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2023-03-21, 21:34:50 UTC] {taskinstance.py:1383} INFO - Executing  on 2023-03-21 21:34:35.872533+00:00
[2023-03-21, 21:34:50 UTC] {standard_task_runner.py:55} INFO - Started process 97 to run task
[2023-03-21, 21:34:50 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test_script_v1.1.2_testimage-0321153312', 'test_script_v1', 'manual__2023-03-21T21:34:35.872533+00:00', '--job-id', '8035', '--raw', '--subdir', 'DAGS_FOLDER/test_script_v1.1.2_testimage-0321153312.py', '--cfg-path', '/tmp/tmpge5lcx9c']
[2023-03-21, 21:34:50 UTC] {standard_task_runner.py:83} INFO - Job 8035: Subtask test_script_v1
[2023-03-21, 21:34:51 UTC] {task_command.py:376} INFO - Running  on host testscriptv112testimage0321153-49c35bc0a2e748138ff8e1d071d58f3d
[2023-03-21, 21:34:52 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test_script_v1.1.2_testimage-0321153312
AIRFLOW_CTX_TASK_ID=test_script_v1
AIRFLOW_CTX_EXECUTION_DATE=2023-03-21T21:34:35.872533+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-03-21T21:34:35.872533+00:00
[2023-03-21, 21:34:52 UTC] {kubernetes_pod.py:587} INFO - Creating pod test-script-v1-3d20b63e5c98491592fe52c914025046 with labels: {'dag_id': 'test_script_v1.1.2_testimage-0321153312', 'task_id': 'test_script_v1', 'run_id': 'manual__2023-03-21T213435.8725330000-9cea982bd', 'kubernetes_pod_operator': 'True', 'try_number': '1'}
[2023-03-21, 21:34:52 UTC] {kubernetes_pod.py:380} INFO - Found matching pod test-script-v1-3d20b63e5c98491592fe52c914025046 with labels {'airflow_kpo_in_cluster': 'True', 'airflow_version': '2.4.3', 'dag_id': 'test_script_v1.1.2_testimage-0321153312', 'kubernetes_pod_operator': 'True', 'run_id': 'manual__2023-03-21T213435.8725330000-9cea982bd', 'task_id': 'test_script_v1', 'try_number': '1'}
[2023-03-21, 21:34:52 UTC] {kubernetes_pod.py:381} INFO - `try_number` of task_instance: 1
[2023-03-21, 21:34:52 UTC] {kubernetes_pod.py:382} INFO - `try_number` of pod: 1
[2023-03-21, 21:34:52 UTC] {pod_manager.py:180} WARNING - Pod not yet started: test-script-v1-3d20b63e5c98491592fe52c914025046
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO - Downloading https://raw.githubusercontent.com/elyra-ai/elyra/v3.14.2/elyra/airflow/bootstrapper.py
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -                                  Dload  Upload   Total   Spent    Left  Speed
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 21173  100 21173    0     0   101k      0 --:--:-- --:--:-- --:--:--  101k
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO - Downloading https://raw.githubusercontent.com/elyra-ai/elyra/v3.14.2/etc/generic/requirements-elyra.txt
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO - Downloading https://raw.githubusercontent.com/elyra-ai/elyra/v3.14.2/etc/generic/requirements-elyra-py37.txt
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -                                  Dload  Upload   Total   Spent    Left  Speed
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1320  100  1320    0     0   7542      0 --:--:-- --:--:-- --:--:--  7500
100  1320  100  1320    0     0   7542      0 --:--:-- --:--:-- --:--:--  7500
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -                                  Dload  Upload   Total   Spent    Left  Speed
[2023-03-21, 21:34:53 UTC] {pod_manager.py:228} INFO -
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1363  100  1363    0     0   6648      0 --:--:-- --:--:-- --:--:--  6648
[2023-03-21, 21:34:54 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (21.3)
[2023-03-21, 21:34:54 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.9/site-packages (from packaging) (3.0.7)
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.793] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - starting operation
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.793] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - Installing packages
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.793] Newer ipykernel package with version 6.20.2 already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.793] WARNING: Package 'ipython' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/ipython_1642613619922/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.793] Package not found. Installing jinja2 package with version 3.0.3...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.793] WARNING: Package 'jupyter-client' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1642858610849/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'jupyter-core' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1636814260563/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'MarkupSafe' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/markupsafe_1635833544680/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.794] Package not found. Installing minio package with version 7.1.8...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'nbclient' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/nbclient_1642144614450/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'nbconvert' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/nbconvert_1641352009342/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'nbformat' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/nbformat_1617383142101/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.794] Package not found. Installing papermill package with version 2.3.4...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'pyzmq' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/pyzmq_1635877397296/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'prompt-toolkit' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1639065841292/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'requests' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/requests_1641580202195/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'tornado' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/tornado_1635819584296/work already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [I 21:34:55.794] Newer traitlets package with version 5.8.1 already installed. Skipping...
[2023-03-21, 21:34:55 UTC] {pod_manager.py:228} INFO - [W 21:34:55.794] WARNING: Package 'urllib3' found with unsupported Legacy version scheme file:///home/conda/feedstock_root/build_artifacts/urllib3_1641584929973/work already installed. Skipping...
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: jinja2==3.0.3 in /opt/conda/lib/python3.9/site-packages (3.0.3)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Collecting minio==7.1.8
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO -   Downloading minio-7.1.8-py3-none-any.whl (75 kB)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Collecting papermill==2.3.4
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO -   Downloading papermill-2.3.4-py3-none-any.whl (37 kB)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.9/site-packages (from jinja2==3.0.3) (2.0.1)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: certifi in /opt/conda/lib/python3.9/site-packages (from minio==7.1.8) (2022.12.7)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: urllib3 in /opt/conda/lib/python3.9/site-packages (from minio==7.1.8) (1.26.8)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: click in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (8.0.3)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: requests in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (2.27.1)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: tqdm>=4.32.2 in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (4.62.3)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: nbformat>=5.1.2 in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (5.1.3)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Collecting tenacity
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO -   Downloading tenacity-8.2.2-py3-none-any.whl (24 kB)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Collecting ansiwrap
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO -   Downloading ansiwrap-0.8.4-py2.py3-none-any.whl (8.5 kB)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: nbclient>=0.2.0 in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (0.5.10)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: entrypoints in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (0.3)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: pyyaml in /opt/conda/lib/python3.9/site-packages (from papermill==2.3.4) (6.0)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: nest-asyncio in /opt/conda/lib/python3.9/site-packages (from nbclient>=0.2.0->papermill==2.3.4) (1.5.4)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: jupyter-client>=6.1.5 in /opt/conda/lib/python3.9/site-packages (from nbclient>=0.2.0->papermill==2.3.4) (7.1.2)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.9/site-packages (from nbclient>=0.2.0->papermill==2.3.4) (5.8.1)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.9/site-packages (from nbformat>=5.1.2->papermill==2.3.4) (4.4.0)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.9/site-packages (from nbformat>=5.1.2->papermill==2.3.4) (0.2.0)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.9/site-packages (from nbformat>=5.1.2->papermill==2.3.4) (4.9.1)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Collecting textwrap3>=0.9.2
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO -   Downloading textwrap3-0.9.2-py2.py3-none-any.whl (12 kB)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.9/site-packages (from requests->papermill==2.3.4) (3.3)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.9/site-packages (from requests->papermill==2.3.4) (2.0.10)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.9/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=5.1.2->papermill==2.3.4) (21.4.0)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /opt/conda/lib/python3.9/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=5.1.2->papermill==2.3.4) (0.18.1)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.5->nbclient>=0.2.0->papermill==2.3.4) (22.3.0)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.5->nbclient>=0.2.0->papermill==2.3.4) (2.8.2)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: tornado>=4.1 in /opt/conda/lib/python3.9/site-packages (from jupyter-client>=6.1.5->nbclient>=0.2.0->papermill==2.3.4) (6.1)
[2023-03-21, 21:34:56 UTC] {pod_manager.py:228} INFO - Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.9/site-packages (from python-dateutil>=2.1->jupyter-client>=6.1.5->nbclient>=0.2.0->papermill==2.3.4) (1.16.0)
[2023-03-21, 21:34:57 UTC] {pod_manager.py:228} INFO - Installing collected packages: textwrap3, tenacity, ansiwrap, papermill, minio
[2023-03-21, 21:34:57 UTC] {pod_manager.py:228} INFO - Successfully installed ansiwrap-0.8.4 minio-7.1.8 papermill-2.3.4 tenacity-8.2.2 textwrap3-0.9.2
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - alembic @ file:///home/conda/feedstock_root/build_artifacts/alembic_1636656371607/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - altair @ file:///home/conda/feedstock_root/build_artifacts/altair_1640799865332/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ansiwrap==0.8.4
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1641898043316/work/dist
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1640855140139/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1618968359944/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - async-generator==1.10
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1640799537051/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1619719576210/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1631087867185/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - black @ file:///home/conda/feedstock_root/build_artifacts/black-recipe_1639405010350/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1629908509068/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - blinker==1.4
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - bokeh @ file:///home/conda/feedstock_root/build_artifacts/bokeh_1637615122801/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Bottleneck @ file:///home/conda/feedstock_root/build_artifacts/bottleneck_1636024229570/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - brotlipy @ file:///home/conda/feedstock_root/build_artifacts/brotlipy_1636012194889/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - certifi==2022.12.7
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - certipy==0.1.3
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1671179360775/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1641584610847/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - click @ file:///home/conda/feedstock_root/build_artifacts/click_1635822573388/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1631273254894/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - comm==0.1.2
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - conda==4.11.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1636021710069/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1639699280527/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Cython @ file:///home/conda/feedstock_root/build_artifacts/cython_1639702830128/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - cytoolz==0.11.2
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1642200691755/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - dataclasses @ file:///home/conda/feedstock_root/build_artifacts/dataclasses_1628958434797/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1636043249011/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - dill @ file:///home/conda/feedstock_root/build_artifacts/dill_1623610058511/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - distributed @ file:///home/conda/feedstock_root/build_artifacts/distributed_1642210134503/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1605121927639/work/dist/entrypoints-0.3-py2.py3-none-any.whl
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1633213722787/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1640697294996/work/source/flit_core
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1639926289397/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1641933807587/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - future @ file:///home/conda/feedstock_root/build_artifacts/future_1673596611778/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1641732907770/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - greenlet @ file:///home/conda/feedstock_root/build_artifacts/greenlet_1635836790477/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1637963928098/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - HeapDict==1.0.1
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1642433548627/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - imagecodecs @ file:///home/conda/feedstock_root/build_artifacts/imagecodecs_1637750400866/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1640778033138/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1642513903188/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1635615662634/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ipykernel==6.20.2
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ipympl @ file:///home/conda/feedstock_root/build_artifacts/ipympl_1642683202097/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1642613619922/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ipython-genutils==0.2.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1631590360471/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1637175083648/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1636510082894/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1633637554808/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1642000296051/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1642858610849/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1636814260563/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1642861369800/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyter-telemetry @ file:///home/conda/feedstock_root/build_artifacts/jupyter_telemetry_1605173804246/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyterhub @ file:///home/conda/feedstock_root/build_artifacts/jupyterhub-feedstock_1642765651112/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1642103664616/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyterlab-server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1641592475363/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1631590465624/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1635836719639/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - libmambapy @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1638990061747/work/libmambapy
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - llvmlite==0.38.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - locket==0.2.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1637170539581/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - mamba @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1638990061747/work/mamba
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1635833544680/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1639359646028/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1631080358261/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - minio==7.1.8
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1635844679781/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - mock @ file:///home/conda/feedstock_root/build_artifacts/mock_1635819531655/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1612895720168/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - msgpack @ file:///home/conda/feedstock_root/build_artifacts/msgpack-python_1637771281198/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - munkres==1.1.4
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - mypy-extensions @ file:///home/conda/feedstock_root/build_artifacts/mypy_extensions_1635839744187/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1641934401341/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1642144614450/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert_1641352009342/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1617383142101/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1638419302549/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1635253012265/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1642016744468/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - numba @ file:///home/conda/feedstock_root/build_artifacts/numba_1642189988883/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1639069486777/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1640083068831/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1622563202229/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - olefile @ file:///home/conda/feedstock_root/build_artifacts/olefile_1602866521163/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pamela==1.0.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pandas==1.4.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - papermill==2.3.4
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1617910651905/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pathspec @ file:///home/conda/feedstock_root/build_artifacts/pathspec_1626613672358/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1632667180946/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1636558800840/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1630400214373/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1635538335951/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1639065841292/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - protobuf==3.19.3
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1640887138049/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1636020362356/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pycryptodomex @ file:///home/conda/feedstock_root/build_artifacts/pycryptodomex_1669581254197/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pycurl==7.44.1
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1641580240686/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1638819640841/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1633192417276/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1642753572664/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1642534390768/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1635862404942/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1602545356084/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1633452062248/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - PyWavelets @ file:///home/conda/feedstock_root/build_artifacts/pywavelets_1638209721487/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1636139833192/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1635877397296/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1641580202195/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ruamel-yaml-conda @ file:///home/conda/feedstock_root/build_artifacts/ruamel_yaml_1636009144459/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1640809661358/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1636815619750/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - scikit-image @ file:///home/conda/feedstock_root/build_artifacts/scikit-image_1639751509164/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1640464150285/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1637806669496/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1629095986539/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1628511208346/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1635844660462/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - sortedcontainers @ file:///home/conda/feedstock_root/build_artifacts/sortedcontainers_1621217038088/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1638550740809/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1642761938807/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1642255706390/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1636765196796/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1636237085010/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tables @ file:///home/conda/feedstock_root/build_artifacts/pytables_1638208841175/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tblib @ file:///home/conda/feedstock_root/build_artifacts/tblib_1616261298899/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tenacity==8.2.2
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1636052205136/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - testpath @ file:///home/conda/feedstock_root/build_artifacts/testpath_1621261527237/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - textwrap3==0.9.2
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1633102299089/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1635944860688/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1635181214134/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1636177962100/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1635819584296/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1632160078689/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - traitlets==5.8.1
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - typed-ast @ file:///home/conda/feedstock_root/build_artifacts/typed-ast_1638670855266/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1638334978229/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1641584929973/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - webencodings==0.5.1
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1638949723239/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1637174147962/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - xlrd @ file:///home/conda/feedstock_root/build_artifacts/xlrd_1610224409810/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - zict==2.0.0
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1641731510635/work
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.279] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - Packages installed (2.486 secs)
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.346] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - processing dependencies
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.358] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - downloaded test_script_v1.1.2-123c4d16-7a7c-4be7-8adc-8061a7b3f5eb.tar.gz from bucket: airflow, object: test_script_v1.1.2_testimage-0321153312/test_script_v1.1.2-123c4d16-7a7c-4be7-8adc-8061a7b3f5eb.tar.gz (0.012 secs)
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - /
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - test_script_v1.1.2.ipynb
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - tar: Removing leading `/' from member names
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.368] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - dependencies processed (0.023 secs)
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.369] 'test_script_v1.1.2_testimage':'test_script_v1.1.2' - executing notebook using 'papermill test_script_v1.1.2.ipynb test_script_v1.1.2-output.ipynb'
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.581] Input Notebook:  test_script_v1.1.2.ipynb
[2023-03-21, 21:34:58 UTC] {pod_manager.py:228} INFO - [I 21:34:58.581] Output Notebook: test_script_v1.1.2-output.ipynb
[2023-03-21, 21:34:59 UTC] {pod_manager.py:228} INFO -
Executing:   0%|          | 0/34 [00:00

Regarding using python3 direct image rather than elyra/kernel-py, I think I can try that, but as mentioned above, we went this way as that way it is much easier to match runtime environment in Elyra using python_kubernetes names kernel from EG and using the same image as base for Runtime Image for Airflow scheduled job.
I would also try to setup a new system tomorrow and see, if I can find any difference.

Thanks again - the full log is helpful.

What I do not know is if it is needed because notebook at the time of submitting pipieline stores that information somewhere or is comes from requirement of elyra/kernel-py image that it needs a name matching with as defined in EG.

The name of the kernel used to run the notebook cells is stored in the notebook's metadata and becomes the "preferred" name.  The Elyra bootstrapper, however, extracts that metadata and determines if a kernelspec by that name exists.  If it does not, it falls back to using the default name of 'python3'.  Since you are installing a new (and vanilla) ipykernel with a name of python_kubernetes, that should be fine.

Regarding to be able to run pods, that functionality is working,...

The pods you list, as you say, are relative to the notebook operators (nodes) and not a kernel pod.  Since you're simply installing the default python kernel as python_kubernetes there's nothing additional that would trigger the creation of a kernel pod - thanks for checking.
I'm guessing there's some kind of incompatibility getting introduced but, realistically speaking, there shouldn't be anything about that image that shouldn't work from a kernel operation perspective.
Are you able to share the portion of the Dockerfile used to extend the two elyra/kernel-py images?  Is that SAME Dockerfile used for each of the two kernel-py images?
I'm trying to take the kernel-py images, add papermill and then run a command like the following....
docker run -it --rm -v `pwd`:/tmp my-modified-image papermill /tmp/k1.ipynb /tmp/k1-out.ipynb

where k1.ipynb exists in pwd.
Are you able to do something similar with each of your images?  I essentially run the target image with bash, then pip install papermill==2.3.4, then docker commit image-id-from-bash-run my-modified-image to capture the update into a new image.

The name of the kernel used to run the notebook cells is stored in the notebook's metadata and becomes the "preferred" name. The Elyra bootstrapper, however, extracts that metadata and determines if a kernelspec by that name exists. If it does not, it falls back to using the default name of 'python3'. Since you are installing a new (and vanilla) ipykernel with a name of python_kubernetes, that should be fine.

Just reporting that this fallback behavior does seems to be working (if I understood that properly). If the image does not have names ipykernel then it just fails.
 pod fails with no named ipykernel found error"

Executing:   0%|          | 0/34 [00:00
[2023-03-22, 09:51:59 UTC] {pod_manager.py:228} INFO - [I 09:51:59.712] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - converting from test_script_v1.1.2-output.ipynb to test_script_v1.1.2.html
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO - [I 09:52:00.351] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - test_script_v1.1.2-output.ipynb converted to test_script_v1.1.2.html (0.640 secs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO - [I 09:52:00.362] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - uploaded test_script_v1.1.2-output.ipynb to bucket: airflow object: test_script_v1.1.2_pd141-0322093805/test_script_v1.1.2.ipynb (0.010 secs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO - [I 09:52:00.384] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - uploaded test_script_v1.1.2.html to bucket: airflow object: test_script_v1.1.2_pd141-0322093805/test_script_v1.1.2.html (0.022 secs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO - Traceback (most recent call last):
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 506, in
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     main()
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 499, in main
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     file_op.execute()
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 239, in execute
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     raise ex
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 224, in execute
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     papermill.execute_notebook(notebook, notebook_output)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/execute.py", line 107, in execute_notebook
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     nb = papermill_engines.execute_notebook_with_engine(
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/engines.py", line 359, in execute_notebook
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/engines.py", line 418, in execute_managed_notebook
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return PapermillNotebookClient(nb_man, **final_kwargs).execute()
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/clientwrap.py", line 43, in execute
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     with self.setup_kernel(**kwargs):
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return next(self.gen)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/client.py", line 592, in setup_kernel
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     self.start_new_kernel(**kwargs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/util.py", line 87, in wrapped
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return just_run(coro(*args, **kwargs))
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return loop.run_until_complete(coro)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return future.result()
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/client.py", line 547, in async_start_new_kernel
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/util.py", line 99, in ensure_async
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     result = await obj
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 83, in wrapper
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     raise e
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 75, in wrapper
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     out = await method(self, *args, **kwargs)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 383, in _async_start_kernel
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     kernel_cmd, kw = await ensure_async(self.pre_start_kernel(**kw))
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/utils.py", line 38, in ensure_async
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     return await obj
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 346, in _async_pre_start_kernel
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     self.kernel_spec,
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 176, in kernel_spec
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/kernelspec.py", line 294, in get_kernel_spec
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO -     raise NoSuchKernel(kernel_name)
[2023-03-22, 09:52:00 UTC] {pod_manager.py:228} INFO - jupyter_client.kernelspec.NoSuchKernel: No such kernel named python_kubernetes
[2023-03-22, 09:52:01 UTC] {pod_manager.py:253} WARNING - Pod test-script-v1-2e2963683e854432a49d2ff20c30382f log read interrupted but container base still running
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -
Executing:   0%|          | 0/34 [00:00
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO - [I 09:51:59.712] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - converting from test_script_v1.1.2-output.ipynb to test_script_v1.1.2.html
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO - [I 09:52:00.351] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - test_script_v1.1.2-output.ipynb converted to test_script_v1.1.2.html (0.640 secs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO - [I 09:52:00.362] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - uploaded test_script_v1.1.2-output.ipynb to bucket: airflow object: test_script_v1.1.2_pd141-0322093805/test_script_v1.1.2.ipynb (0.010 secs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO - [I 09:52:00.384] 'test_script_v1.1.2_pd141':'test_script_v1.1.2' - uploaded test_script_v1.1.2.html to bucket: airflow object: test_script_v1.1.2_pd141-0322093805/test_script_v1.1.2.html (0.022 secs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO - Traceback (most recent call last):
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 506, in
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     main()
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 499, in main
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     file_op.execute()
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 239, in execute
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     raise ex
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/var/lib/pandas/jupyter-work-dir/bootstrapper.py", line 224, in execute
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     papermill.execute_notebook(notebook, notebook_output)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/execute.py", line 107, in execute_notebook
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     nb = papermill_engines.execute_notebook_with_engine(
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/engines.py", line 359, in execute_notebook
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/engines.py", line 418, in execute_managed_notebook
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return PapermillNotebookClient(nb_man, **final_kwargs).execute()
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/papermill/clientwrap.py", line 43, in execute
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     with self.setup_kernel(**kwargs):
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return next(self.gen)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/client.py", line 592, in setup_kernel
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     self.start_new_kernel(**kwargs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/util.py", line 87, in wrapped
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return just_run(coro(*args, **kwargs))
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return loop.run_until_complete(coro)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return future.result()
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/client.py", line 547, in async_start_new_kernel
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     await ensure_async(self.km.start_kernel(extra_arguments=self.extra_arguments, **kwargs))
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/nbclient/util.py", line 99, in ensure_async
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     result = await obj
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 83, in wrapper
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     raise e
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 75, in wrapper
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     out = await method(self, *args, **kwargs)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 383, in _async_start_kernel
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     kernel_cmd, kw = await ensure_async(self.pre_start_kernel(**kw))
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/utils.py", line 38, in ensure_async
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     return await obj
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 346, in _async_pre_start_kernel
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     self.kernel_spec,
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/manager.py", line 176, in kernel_spec
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -   File "/usr/local/lib/python3.9/site-packages/jupyter_client/kernelspec.py", line 294, in get_kernel_spec
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO -     raise NoSuchKernel(kernel_name)
[2023-03-22, 09:52:02 UTC] {pod_manager.py:228} INFO - jupyter_client.kernelspec.NoSuchKernel: No such kernel named python_kubernetes
[2023-03-22, 09:52:02 UTC] {pod_manager.py:275} INFO - Pod test-script-v1-2e2963683e854432a49d2ff20c30382f has phase Running
[2023-03-22, 09:52:04 UTC] {kubernetes_pod.py:475} INFO - Deleting pod: test-script-v1-2e2963683e854432a49d2ff20c30382f
[2023-03-22, 09:52:04 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):

Dockerfiles are quite similar.  there is a difference in user statement as with 3.2.0 the build image was failing with permission errors. elyraflownetes is just a local package which has our custom libraries for application.
2.5.1
FROM elyra/kernel-py:2.5.1
USER root
RUN apt-get update && apt-get  install --yes \
    apt-utils \
    curl \
    git
# set the user back to original setting.
USER $NB_UID
RUN python -m ipykernel install --user --name python_kubernetes
ENV THIS_IS_ELYRAFLOW=1
ADD . $HOME/elyraflownetes

USER root
RUN pip install --user $HOME/elyraflownetes/
RUN rm -r $HOME/elyraflownetes
USER $NB_UID

3.2.0
FROM elyra/kernel-py:3.2.0
USER root
RUN apt-get update && apt-get  install --yes \
    apt-utils \
    curl \
    git
# set the user back to original setting.
USER $NB_UID
RUN python -m ipykernel install --user --name python_kubernetes
ENV THIS_IS_ELYRAFLOW=1
ADD . $HOME/elyraflownetes

USER root
RUN pip install --user $HOME/elyraflownetes/
RUN rm -r $HOME/elyraflownetes
USER $NB_UID

I have some error with Docker on my local laptop as build images are failing with network error reaching to any server e.g. ubuntu for apt-get or pypi for pip install. Will update about running those image on local Docker setup as soon as I have network issues resolved.

@kevin-bates it seems I have found the issue, but still do not know the solution.
Reason seems to be the change in sequence of USER command in Dockerfile.
in 3.2.0 it does install the custom package and build succeeds, but it is not available to pip inside the running container. I am not expert on pip, but I am assuming this is because of some reason root install package is not available to other user i.e. joyvan in this case.
Why notebook hangs is because it has try-catch block at import of that package and if fails, then it tries to install from github repo. Now that pip install inside notebook was hanging as the github token used was expired and command was waiting to enter new password manually... Wow, atleast I know where the trouble is.
However, now when I change the sequence of USER command in Dockerfile to align as it is in 2.5.1, the build process fails with permissions issue. I tried pip install --user  $HOME/elyraflownetes  as well as  pip install $HOME/elyraflownetes. I am assuming something has changed in pip version from 2.5.1 and 3.2.0 that it behaves differently.
#0 24.94     Running setup.py install for elyraflownetes: started
#0 25.12     Running setup.py install for elyraflownetes: finished with status 'error'
#0 25.12     ERROR: Command errored out with exit status 1:
#0 25.12      command: /opt/conda/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/jovyan/elyraflownetes/setup.py'"'"'; __file__='"'"'/home/jovyan/elyraflownetes/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-t9oyij5p/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/jovyan/.local/include/python3.9/elyraflownetes
#0 25.12          cwd: /home/jovyan/elyraflownetes/
#0 25.12     Complete output (7 lines):
#0 25.12     running install
#0 25.12     /opt/conda/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
#0 25.12       warnings.warn(
#0 25.12     running build
#0 25.12     running build_py
#0 25.12     creating build
#0 25.12     error: could not create 'build': Permission denied
#0 25.12     ----------------------------------------
#0 25.12 ERROR: Command errored out with exit status 1: /opt/conda/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/jovyan/elyraflownetes/setup.py'"'"'; __file__='"'"'/home/jovyan/elyraflownetes/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-t9oyij5p/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/jovyan/.local/include/python3.9/elyraflownetes Check the logs for full command output.

I updated the notebook with new github token and scheduled it again. Now when it runs, it does not find that custom package and installs from github successfully and rest of the notebook also gets executed.
I also changed below lines in Dockerfile and then build works and also finally notebook runs.
- ADD . $HOME/elyraflownetes
- RUN pip install $HOME/elyraflownetes/
+ RUN  pip install git+https://token@github.com/repo/elyraflownetes@dev > ./tmp.tmp

i would rather like that it can work standard ADD, RUN commands from local repo rather than pulling the same repo from github. Looking for what can I change to make it work. Any suggestions?

I have updated Dockerfile now to below and it builds successfully. So mainly pip install  is executed by root but without --user which makes sense as then it installs and makes it available for all users. Not sure, what changed between these versions as earlier pip install --user was working with $NB_UID. I do not if that was a bug which was fixed or something changed between Docker/pip versions. But it seems at the moment, I am making crazy workaround to make it work.
FROM elyra/kernel-py:3.2.0
USER root
RUN apt-get update && apt-get  install --yes \
    apt-utils \
    curl \
    git
# set the user back to original setting.
USER $NB_UID
RUN python -m ipykernel install --user --name python_kubernetes
ENV THIS_IS_ELYRAFLOW=1
ADD . $HOME/elyraflownetes

USER root

RUN pip install $HOME/elyraflownetes/

RUN rm -rf $HOME/elyraflownetes
USER $NB_UID

Considering this is solved, may I ask what is the usual practice to ensure notebook runs in "same/similar" environment when executed by EG kernel and when scheduled as a job?
If I have to avoid using elyra/kernel-py images for runtime Image for the scheduled pod for this notebook, it means that I somehow need to get requirements.txt from that kernel matched to another custom image based on e.g. "python3.x". We use elyra/kernel-py as it helps to easily keep the environments in sync.

Glad to hear you're moving forward.  Got a few comments....

Considering this is solved, may I ask what is the usual practice to ensure notebook runs in "same/similar" environment when executed by EG kernel and when scheduled as a job?

I'm not sure it's common for users to use EG for their "development" of pipelines destined for Airflow or Kubeflow.  That said, what you're doing, by using the EG kernel image as the node runtime image, seems perfectly reasonable.  However, we don't use --user when installing things into the kernel images because they are natively single-user anyway and it creates confusion about where things are located and discovered.
Another thing I notice, which isn't necessarily an issue but could make things more clear, is that you're using $HOME in two different user blocks.  In the first, $HOME may equate to /home/jovian, while $HOME in USER root should be referring to /root.  Since elyraflownetes is ephemeral, it's probably best to "ADD" it to a user-agnostic directory (e.g., /tmp) and avoid any ambiguities altogether.

Just reporting that this fallback behavior does seems to be working (if I understood that properly). If the image does not have names ipykernel then it just fails.

I'm sorry.  I didn't realize that we have separate bootstrapper code for Airflow and Kubeflow and, it turns out that only the KFP bootstrapper contains the find_best_kernel logic!  I have opened #3134 for this and marked it as a "good first issue" if you're interested :wink:.

Sure, many thanks for your inputs. Those helped to investigate and narrow down the issue. Regarding $HOME, yes did not realize that. Will fix/improve that as well. I hope no one else is doing such mistakes as I am, but maybe this thread will help someone, if they do 😄 .  But, it seems like my mistake helped identify a gap that can be improved.


Also Use USE_BIOMETRIC Permission

Can you transform the library to also utilize androidx.biometric:biometric?

I started a PR with this change but I closed it because it would require devices with Pie to use a hacky workaround to determine if a fingerprint is registered. This may cause even more compatibility issues. I'll keep the branch around if anyone wants to look into it more.

Echoing @benwicks comment, it looks like using BiometricPrompt has no advantages currently (since it just forwards calls to  FingerprintManager), and does have some drawbacks (the workaround). So I'm going to keep using FingerprintManager for now, but we can revisit this when future versions of Android come out.
Thanks again to @benwicks for working on this!


respeakerd stream over the network

Is there any way to expose respeakerd as a network service. I am looking at ways to try using Respeaker Core v2.0 as an external microphone. Any ideas?

@plushpluto I was looking for something similar. Unfortunately, there's no support from maintainers. So I came up with the following solution so far. It streams pre-processed audio chunks (librespeaker) to a custom ASR server via websockets.


Karate 1.20 move to graalvm breaks usage of variables in multi-value parameters

Code of the type
@name=testScenario
Scenario: example scenario

* def foo = 'foo'
* def bar = 'bar'

* url 'example.com'

Given param test = [#(foo), #(bar)]

fails with the error message
org.graalvm.polyglot.PolyglotException: SyntaxError: Unnamed:1:1 Expected an operand but found error
[#(foo), #(bar)]
 ^

This used to work fine with Nashorn, and using variables in json objects (so using { ... } is still working fine. I don't know if it's an issue with the param keyword, or with the fact that we use [ ... ] here - this is breaking our tests.

@LanDinh
Enclosing your array in parenthesis should work:
Given param test = ([foo, bar])

This approach is also more concise, since you don't need to use the #() syntax.

agree with @nathanchilton
also we recommend that embedded expressions always have string quotes around them. yes the json parser is lenient but it causes trouble like this. the design of embedded expressions is so that JSON remains well-formed: https://github.com/karatelabs/karate#embedded-expressions

ignore my above comment, there indeed seems to be a regression in latest versions. @LanDinh do note that we spend a LOT of time creating RC versions, so if you try those earlier and provide feedback, that would save all of us a lot of trouble

is fixed in develop no ETA for release

@LanDinh 1.4.0.RC3 has been released, please try it and let us know


Have multiple pages visible at once

Is this possible? Must we use reusable GBInfiniteScrollViewPages??

Hi @mbcoder17,
I'm afraid not. :cry:  It is necessary use reusable pages in order to reduce memory usage.


Moving into @rose-pine/palette

Appreciate you making this! I must've manually typed our palette as css variables half a dozen times.
I would love to get this into https://github.com/rose-pine/palette but am curious of your thoughts on the best way to do this as I am unsure how to maintain your commit history and such (to give you full credit for this addition).

there's various ways

Patch file
Squashed commit reference

the easiest one is for me to just make the change in your repo as a fork and raise a PR, i'll get that done soon,
Should've actually done that originally 😂 , I was too lazy to ask and hear and no so I just made this.
I'll raise a PR soon


GitHub PR Validations are failing in ADO GitLeaks task.

Hi - we are trying to integrate this tool.
Manual build triggers for detecting secrets work fine, but PR validation is failing with the below error.
##[error]Cannot read property 'length' of null

Here is the code:
          - task: Gitleaks@2
            inputs:
              scanlocation: '$(Build.SourcesDirectory)'
              configtype: 'predefined'
              predefinedconfigfile: 'GitleaksUdmCombo.toml'
              reportformat: 'sarif'
              verbose: true
            displayName: "Gitleaks Credential Scanner"

Happy to provide any other required information.
Thank you in advance. Have a nice day!

Hi @Raag007, while this error should not be there, GItHub PR's are not supported. The extension fetches the PR commits from the Azure DevOps API and that API does not look for changes in GitHub.
There is now a detection for pre-validation/smart scan options when the repo is on GitHub. The extension should print a warning now.
Sorry, but this won't be fixed.


Updated homework_4.sql for completion of homework 4

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)
Completed homework_4.sql as instructed.
What did you learn from the changes you have made?
More SQL commands.
Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?
No.
Were there any challenges? If so, what issue(s) did you face? How did you overcome it?
No.
How were these changes tested?
Verified by compare with raw data from original database.
A reference to a related issue in your repository (if applicable)
Not applicable.
Checklist

[x] I can confirm that my changes are working as intended

HW4 completed.


Bug: contractmanager.json wiped?

BUG REPORT
Stack Trace or error message
My storage folders  IN SIA were removed.
contractmanager.json no longer was connecting my data to my HDDs, All the data is still on the HDDs
Expected Behavior
For it to still be there.
How to reproduce it (as minimally and precisely as possible)
Possibly happened after a restart, not sure.
Environment
GUI

Sia version:
1.3.1

OS:
Windows 7

Is there no way to reconnect my HDD which still has all the data from contracts that are still intact?  Lost thousands of SIA because of this bug.

A 12GB log file is certainly not normal.  I rotate my logs (via logrotate on Linux), but I've never had my logs get close to that size.  What is in there?
Side note:  here's my section from my logrotate script.
tbenz9@spartan1:~/go/src/github.com/NebulousLabs/Sia$ cat /etc/logrotate.d/sia
/home/tbenz9/Documents/Sia/consensus/consensus.log {
        size 10M
        copytruncate
        create 660 tbenz9 tbenz9
        rotate 10
        compress
}

I tried to open it but none of my note programs would open a file that big

less can handle huge files: https://linux.die.net/man/1/less

So would head or tail but alas, OP is on Windows.  :)

When i get a chance I will do a little research and find a way.

Based on the information you've provided there is no action for the dev team to take.  I'm going to close this issue, feel free to reopen it if you are able to access your log files or have additional information.


handle RN 0.47 breaking change

RN 0.47 removed createJSModules from ReactPackage commit

This is not a good fix, this package needs to work with earlier versions of RN. What did they replace it with?

Removing just the @Override line should work yes?  Others have said they tested it with 0.46.x, and @Override is an optional declaration in Java, e.g. https://news.ycombinator.com/item?id=7866204


Where is the directory /fsdet/data/？

hi, I don't find the directory /fsdet/data/. I try using the files in directory /fsdet/data  of https://github.com/ucbdrive/few-shot-object-detection, but it occurs error.

the missing fsdet/data has been added. please try again. let me know if you still have issues.


Error in prune_ancestors: ValueError "Need at least one array to stack"

I encountered a ValueError when trying to fit a model using the GraphDiscovery class from the ComputationalHypergraphDiscovery library. This error occurred when calling the fit() method. The error message indicates that there is an issue with the np.stack function, which expects at least one array to stack but received none.
Steps to Reproduce
Here is a minimal example to reproduce the error:
import ComputationalHypergraphDiscovery as chd
import numpy as np
import pandas as pd

num_rows = 100  # Adjust number of rows
rand_arr0 = np.random.rand(num_rows)
rand_arr1 = np.random.rand(num_rows)

# Example 1 (works fine)
df = pd.DataFrame({
    'Column_1': rand_arr0,
    'Column_2': rand_arr1,
})
df = df.sample(n=df.shape[0]//2, random_state=1)

graph_discovery = chd.GraphDiscovery.from_dataframe(df)
graph_discovery.fit()
graph_discovery.plot_graph()

# Example 2 (almost identical to example 1 but errors out)
df = pd.DataFrame({
    'Column_1': rand_arr0,
    'Column_2': 5*rand_arr0+1,
})
df = df.sample(n=df.shape[0]//2, random_state=1)

graph_discovery = chd.GraphDiscovery.from_dataframe(df)
graph_discovery.fit()
graph_discovery.plot_graph()

Expected Behavior
I expected the GraphDiscovery.fit() function to process the DataFrame without throwing an error.
Observed Behavior
The execution failed on second example with the following traceback:
Finding ancestors with kernel [linear]: 0it [00:00, ?it/s]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[10], line 23
     19 df = df.sample(n=df.shape[0]//2,random_state=1).copy()
     22 graph_discovery=chd.GraphDiscovery.from_dataframe(df)
---> 23 graph_discovery.fit()
     24 graph_discovery.plot_graph()

File ~/repos/test/lib/python3.12/site-packages/ComputationalHypergraphDiscovery/_GraphDiscoveryMain.py:298, in GraphDiscovery.fit(self, targets, key, kernel_chooser, mode_chooser, message)
    287 if not np.any(mask_kernel):
    288     continue
    290 (
    291     active_modess_kernel,
    292     noisess_kernel,
    293     Z_lows_kernel,
    294     Z_highs_kernel,
    295     activationss_kernel,
    296     gammas_kernel,
    297     ybs_kernel,
--> 298 ) = self.prune_ancestors(
    299     kernel,
    300     self.ancestor_finding_step_funcs,
    301     X=self.X,
    302     active_modess_kernel=active_modess[mask_kernel],
    303     gas_kernel=gas[mask_kernel],
    304     ybs_kernel=ybs[mask_kernel],
    305     gammas_kernel=gammas[mask_kernel],
    306     subkeys_kernel=subkeys[mask_kernel],
    307     noise_kernel=noisess[mask_kernel],
    308     Z_low_kernel=Z_lows[mask_kernel],
    309     Z_high_kernel=Z_highs[mask_kernel],
    310     loop_number=len(self.modes.clusters) - 2,
    311     message=message,
    312 )
    313 anomaly_noise = np.logical_or(noisess_kernel > 1.05, noisess_kernel < -0.05)
    314 anomaly_Z_low = np.logical_or(Z_lows_kernel > 1.05, Z_lows_kernel < -0.05)

File ~/repos/test/lib/python3.12/site-packages/ComputationalHypergraphDiscovery/_GraphDiscoveryMain.py:631, in GraphDiscovery.prune_ancestors(self, kernel, ancestor_finding_step_funcs, X, active_modess_kernel, gas_kernel, ybs_kernel, gammas_kernel, subkeys_kernel, noise_kernel, Z_low_kernel, Z_high_kernel, loop_number, message)
    624         break
    625 pbar.close()
    626 return (
    627     np.stack(ancestor_modess, axis=1),
    628     np.stack(noisess_kernel, axis=1),
    629     np.stack(Z_lows_kernel, axis=1),
    630     np.stack(Z_highs_kernel, axis=1),
--> 631     np.stack(activationss_kernel, axis=1),
    632     np.stack(gammas, axis=1),
    633     np.stack(ybs, axis=1),
    634 )

File ~/repos/test/lib/python3.12/site-packages/jax/_src/numpy/lax_numpy.py:2828, in stack(arrays, axis, out, dtype)
   2824 @util.implements(np.stack, skip_params=['out'])
   2825 def stack(arrays: np.ndarray | Array | Sequence[ArrayLike],
   2826           axis: int = 0, out: None = None, dtype: DTypeLike | None = None) -> Array:
   2827   if not len(arrays):
-> 2828     raise ValueError("Need at least one array to stack.")
   2829   if out is not None:
   2830     raise NotImplementedError("The 'out' argument to jnp.stack is not supported.")

ValueError: Need at least one array to stack.

Environment

ComputationalHypergraphDiscovery version: latest commit
Python version: 3.12.4
Operating System: MacOS

Additional Context
I am investigating the Computational Hypergraph Discovery (CHD) mechanisms by applying it to small graphs. There is a noticeable difference in execution paths between Example 1 and Example 2. Specifically, Example 1 does not trigger the prune_ancestors method, unlike Example 2. This divergence stems from the kernel selection process in GraphDiscovery.choose_kernel: Example 1 results in chosen_kernels=[-1,-1], suggesting no kernel was chosen, while Example 2 consistently selects a kernel with chosen_kernels=[0,0], which initiates the call to prune_ancestors.
The error within prune_ancestors is linked to the activationss_kernel list, which is empty (len(activationss_kernel)=0), in contrast to other lists in the return statement, each containing one element. This inconsistency causes a ValueError because the np.stack function requires at least one non-empty array to function correctly.
I would appreciate some guidance on getting these issues resolved for small examples.

Hi Alexey,
Thank you for catching this issue! You tried an edge case I had never tried before, two nodes in the graph. The activation computation for one step is done during the next pruning. Since there is only one ancestor here, there was no pruning to do!
I added the edge cases in prune_ancestors in the commit c7748f7.
Let me know if the error is still there.

The fix works. Thank you!


make entire wallet option row clickable

Resolves https://github.com/planetdecred/godcr/issues/327 & https://github.com/planetdecred/godcr/issues/332
This PR;

makes the entire rows in the wallet options drop down clickable and not just the texts
separates the wallet menu into sections using line separators
changes "Privacy to "StakeShuffle"
increases the font size of the wallet menu option titles
fixes the right margin alignment of the wallet menu

The update to the menu seems to work fine on my 1080p display. Howerver, it doesn't display quite the same on my 1440 display. Below is a screenshot of how it looks.

Other thing to note:

Remove the top shadow on the menu and put it below the menu. You can have a look at the mockups for an accurate represatation.
Increase the margin between the two bordered options (view property and StakeShuffle). Also have a look at the mockups for an accurate represenation.