[BUG] Workflow does not work correctly in example notebook "Serving an XGBoost Model with Merlin Systems"

Bug description

The official example for Serving XGBoost Model with Merlin System does not work correctly
Steps/Code to reproduce bug

Start container with GPU resource allocated from nvcr.io/nvidia/merlin/merlin-tensorflow:22.07
Execute official notebook

Expected behavior

Environment details

Merlin version: As is in official image nvcr.io/nvidia/merlin/merlin-tensorflow:22.07
Platform:
Python version: Python3
PyTorch version (GPU?):
Tensorflow version (GPU?): As is in official image nvcr.io/nvidia/merlin/merlin-tensorflow:22.07

Additional context

ValueError                                Traceback (most recent call last)
/home/polonsky/Documents/merlin-test/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb Cell 10 in <cell line: 5>()
      [2](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f6d65726c696e227d/home/polonsky/Documents/merlin-test/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1) from merlin.systems.dag.ensemble import Ensemble
      [3](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f6d65726c696e227d/home/polonsky/Documents/merlin-test/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2) from merlin.systems.dag.ops.workflow import TransformWorkflow
      [5](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f6d65726c696e227d/home/polonsky/Documents/merlin-test/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4) inf_ops = inf_workflow.input_schema.column_names >> TransformWorkflow(inf_workflow) \
----> [6](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f6d65726c696e227d/home/polonsky/Documents/merlin-test/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb#X12sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5)                     >> PredictForest(model, inf_workflow.output_schema)

File /usr/local/lib/python3.8/dist-packages/merlin/systems/dag/ops/fil.py:66, in PredictForest.__init__(self, model, input_schema, backend, **fil_params)
     52 """Instantiate a FIL inference operator.
     53
     54 Parameters
   (...)
     63     The parameters to pass to the FIL operator.
     64 """
     65 if model is not None:
---> 66     self.fil_op = FIL(model, **fil_params)
     67 self.backend = backend
     68 self.input_schema = input_schema

File /usr/local/lib/python3.8/dist-packages/merlin/systems/dag/ops/fil.py:256, in FIL.__init__(self, model, max_batch_size, predict_proba, output_class, threshold, algo, storage_type, threads_per_tree, blocks_per_sm, transfer_threshold, instance_group)
    245 self.max_batch_size = max_batch_size
    246 self.parameters = dict(
    247     predict_proba=predict_proba,
    248     output_class=output_class,
...
    399         f"Model type not supported. {type(model)} " f"Must be one of: {supported_model_types}"
    400     )
    401 return fil_model

ValueError: Model type not supported. <class 'merlin.models.xgb.XGBoost'> Must be one of: {'xgboost.XGBModel', 'lightgbm.Booster', 'cuml.ensemble.RandomForestClassifier', 'xgboost.Booster', 'cuml.ensemble.RandomForestRegressor', 'sklearn.ensemble.RandomForestRegressor', 'lightgbm.LGBMModel', 'sklearn.ensemble.RandomForestClassifier'}

Hello @BorisPolonsky,
Can you pull the latest version of Merlin Systems, Merlin Core, Merlin Models, etc? We added the example after the 22.07 release

@BorisPolonsky You can try something like this to run start a jupyter notebook in the 22.07 container from a local checkout of this systems repo.
docker run --gpus=all --rm -it -p 8888:8888 --ipc=host -v $(pwd):/systems --workdir /systems nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 bash -c 'export PYTHONPATH="/systems:${PYTHONPATH}"; jupyter notebook --ip <IP_ADDRESS> --allow-root'

The main trick here is to ensure the library code used by the notebook is the same version as the code where the notebook is running. In this example this is done by setting the env var PYTHONPATH="/systems:${PYTHONPATH} so that python will first look in the directory where the code is checked out before looking at the already installed version inside the docker image.

Hello @BorisPolonsky,
Can you pull the latest version of Merlin Systems, Merlin Core, Merlin Models, etc? We added the example after the 22.07 release

@BorisPolonsky You can try something like this to run start a jupyter notebook in the 22.07 container from a local checkout of this systems repo.
docker run --gpus=all --rm -it -p 8888:8888 --ipc=host -v $(pwd):/systems --workdir /systems nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 bash -c 'export PYTHONPATH="/systems:${PYTHONPATH}"; jupyter notebook --ip <IP_ADDRESS> --allow-root'

The main trick here is to ensure the library code used by the notebook is the same version as the code where the notebook is running. In this example this is done by setting the env var PYTHONPATH="/systems:${PYTHONPATH} so that python will first look in the directory where the code is checked out before looking at the already installed version inside the docker image.

Thanks for clarifying. It looks like that I'm running the examples while using docker image that are not built for the current version as I've just found out that the 22.07 image ships with these example (not up to date as in the current version of this repo though.
I would like to update docker image to solve this problem. However, 22.07 is already the latest version I found on NGC. Any suggestion?

Use 22.07 image with current version of the repo, with env var PYTHONPATH="/{REPO_PATH}:${PYTHONPATH}",

I try downloading release 0.5.0 from this repo and did the following
docker run -it --name merlin -v /home:/home -e LANG=C.UTF-8 -v /home:/home --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -p 6006:6006 nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 bash
jupyter notebook --allow-root --ip <IP_ADDRESS> --port 6006
export PYTHONPATH="${REPO_PATH}:${PYTHONPATH}"

I got the current problem solved. However the python part of the ensembled xgboost-based model failed to initialize in triton server.
+---------------------+---------+------------------------------------------------------------------------------------+
| Model               | Version | Status                                                                             |
+---------------------+---------+------------------------------------------------------------------------------------+
| 0_transformworkflow | 1       | READY                                                                              |
| 1_fil               | 1       | READY                                                                              |
| 1_predictforest     | 1       | UNAVAILABLE: Internal: TypeError: __init__() got an unexpected keyword argument 'm |
|                     |         | odel_repository'                                                                   |
|                     |         |                                                                                |
|                     |         |   /home/polonsky/Documents/merlin-systems-0.5.0/examples/ensemble/1_predictforest/ |
|                     |         | 1/model.py(61): initialize                                                         |
+---------------------+---------+------------------------------------------------------------------------------------+

I would recommend to get the container of 22.07 and pull the latest branch of merlin models, merlin systems and merlin core in the container.
cd /models/ && git pull origin main && pip install .
cd /nvtabular && git pull origin main && pip install .
cd /core/ && git pull origin main && pip install .
cd /systems/ && git pull origin main && pip install .

We run the unittest for every PR and integration test nightly.

I would recommend to get the container of 22.07 and pull the latest branch of merlin models, merlin systems and merlin core in the container.
cd /models/ && git pull origin main && pip install .
cd /nvtabular && git pull origin main && pip install .
cd /core/ && git pull origin main && pip install .
cd /systems/ && git pull origin main && pip install .

We run the unittest for every PR and integration test nightly.

Thanks for this clarification. All the problem above were solved. Except that
assert np.allclose(predictions_from_triton, local_predictions)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/polonsky/Documents/merlin-systems/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb Cell 24 in <cell line: 1>()
----> [1](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f6d65726c696e2d66617374666f7277617264227d/home/polonsky/Documents/merlin-systems/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb#X32sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) assert np.allclose(predictions_from_triton, local_predictions)

File <__array_function__ internals>:5, in allclose(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2249, in allclose(a, b, rtol, atol, equal_nan)
   2179 @array_function_dispatch(_allclose_dispatcher)
   2180 def allclose(a, b, rtol=1.e-5, atol=1.e-8, equal_nan=False):
   2181     """
   2182     Returns True if two arrays are element-wise equal within a tolerance.
   2183
   (...)
   2247
   2248     """
-> 2249     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2250     return bool(res)

File <__array_function__ internals>:5, in isclose(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2358, in isclose(a, b, rtol, atol, equal_nan)
   2356 yfin = isfinite(y)
   2357 if all(xfin) and all(yfin):
-> 2358     return within_tol(x, y, atol, rtol)
...
   2337 def within_tol(x, y, atol, rtol):
   2338     with errstate(invalid='ignore'):
-> 2339         return less_equal(abs(x-y), atol + rtol * abs(y))

ValueError: operands could not be broadcast together with shapes (10,10) (9,10)

Failed. It's wierd that given ten input samples. I got 9 outputs from local prediction, while triton returns 10 outputs as expected

I would recommend to get the container of 22.07 and pull the latest branch of merlin models, merlin systems and merlin core in the container.
cd /models/ && git pull origin main && pip install .
cd /nvtabular && git pull origin main && pip install .
cd /core/ && git pull origin main && pip install .
cd /systems/ && git pull origin main && pip install .

We run the unittest for every PR and integration test nightly.

Thanks for this clarification. All the problem above were solved. Except that
assert np.allclose(predictions_from_triton, local_predictions)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/polonsky/Documents/merlin-systems/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb Cell 24 in <cell line: 1>()
----> [1](vscode-notebook-cell://attached-container%2B7b22636f6e7461696e65724e616d65223a222f6d65726c696e2d66617374666f7277617264227d/home/polonsky/Documents/merlin-systems/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb#X32sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) assert np.allclose(predictions_from_triton, local_predictions)

File <__array_function__ internals>:5, in allclose(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2249, in allclose(a, b, rtol, atol, equal_nan)
   2179 @array_function_dispatch(_allclose_dispatcher)
   2180 def allclose(a, b, rtol=1.e-5, atol=1.e-8, equal_nan=False):
   2181     """
   2182     Returns True if two arrays are element-wise equal within a tolerance.
   2183
   (...)
   2247
   2248     """
-> 2249     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2250     return bool(res)

File <__array_function__ internals>:5, in isclose(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/numpy/core/numeric.py:2358, in isclose(a, b, rtol, atol, equal_nan)
   2356 yfin = isfinite(y)
   2357 if all(xfin) and all(yfin):
-> 2358     return within_tol(x, y, atol, rtol)
...
   2337 def within_tol(x, y, atol, rtol):
   2338     with errstate(invalid='ignore'):
-> 2339         return less_equal(abs(x-y), atol + rtol * abs(y))

ValueError: operands could not be broadcast together with shapes (10,10) (9,10)

Failed. It's wierd that given ten input samples. I got 9 outputs from local prediction, while triton returns 10 outputs as expected

I just ran the unittest and it ran successful. I pulled the merlin-tensorflow:22.07 container, updated the libraries and exucted the unittest
cd /models/ && git pull origin main && pip install .
cd /nvtabular && git pull origin main && pip install .
cd /core/ && git pull origin main && pip install .
cd /systems/ && git pull origin main && pip install .
pytest tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py

I cannot reproduce the bug

I just ran the unittest and it ran successful. I pulled the merlin-tensorflow:22.07 container, updated the libraries and exucted the unittest
cd /models/ && git pull origin main && pip install .
cd /nvtabular && git pull origin main && pip install .
cd /core/ && git pull origin main && pip install .
cd /systems/ && git pull origin main && pip install .
pytest tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py

I cannot reproduce the bug

I did the same in the container again and unfortunately I did NOT make it to the final assert np.allclose(predictions_from_triton, local_predictions) this time. The previously reported error from triton-inference-server is back.
This is how I prepare my image
FROM nvcr.io/nvidia/merlin/merlin-tensorflow:22.07
WORKDIR /models
RUN git pull origin main && pip install .
WORKDIR /nvtabular
RUN git pull origin main && pip install .
WORKDIR /core
RUN git pull origin main && pip install .
WORKDIR /systems
RUN git pull origin main && pip install .
WORKDIR /opt/tritonserver

And this how I conducted pytest and the result
 docker run --rm --workdir /systems -e LANG=C.UTF-8 -v /etc/localtime:/etc/localtime:ro --gpus all
--ipc=host --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/merlin/merlin-tensorflow:22.07-fastforward pytest tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py

==================================
== Triton Inference Server Base ==
==================================

NVIDIA Release 22.06 (build 39726160)

Copyright (c) 2018-2022, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.

        If the argument is a coroutine, it is wrapped in a Task.

        WARNING: It would be disastrous to call run_until_complete()
        with the same coroutine twice -- it would wrap it in two
        different Tasks and that can't be good.

        Return the Future's result, or raise its exception.
        """
        self._check_closed()
        self._check_running()

        new_task = not futures.isfuture(future)
        future = tasks.ensure_future(future, loop=self)
        if new_task:
            # An exception is raised if the future didn't complete, so there
            # is no need to log the "destroy pending task" message
            future._log_destroy_pending = False

        future.add_done_callback(_run_until_complete_cb)
        try:
            self.run_forever()
        except:
            if new_task and future.done() and not future.cancelled():
                # The coroutine raised a BaseException.

        To execute all cells see :meth:`execute`.

        Parameters
        ----------
        cell : nbformat.NotebookNode
            The cell which is currently being processed.
        cell_index : int
            The position of the cell within the notebook object.
        execution_count : int
            The execution count to be assigned to the cell (default: Use kernel response)
        store_history : bool
            Determines if history should be stored in the kernel (default: False).
            Specific to ipython kernels, which can store command histories.

        Returns
        -------
        output : dict
            The execution output payload (or None for no output).

        Raises
        ------
        CellExecutionError
            If execution failed and should raise an exception, this will be raised
            with defaults about the failure.

        Returns
        -------
        cell : NotebookNode
            The cell which was just processed.
        """
        assert self.kc is not None

        await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

        if cell.cell_type != 'code' or not cell.source.strip():
            self.log.debug("Skipping non-executing cell %s", cell_index)
            return cell

        if self.skip_cells_with_tag in cell.metadata.get("tags", []):
            self.log.debug("Skipping tagged cell %s", cell_index)
            return cell

        if self.record_timing:  # clear execution metadata prior to execution
            cell['metadata']['execution'] = {}

        self.log.debug("Executing cell:\n%s", cell.source)

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
        )

        await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
        parent_msg_id = await ensure_async(
            self.kc.execute(
                cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
            )
        )
        await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
        # We launched a code cell to execute
        self.code_cells_executed += 1
        exec_timeout = self._get_timeout(cell)

        cell.outputs = []
        self.clear_before_next_output = False

        task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
        task_poll_output_msg = asyncio.ensure_future(
            self._async_poll_output_msg(parent_msg_id, cell, cell_index)
        )
        self.task_poll_for_reply = asyncio.ensure_future(
            self._async_poll_for_reply(
                parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
            )
        )
        try:
            exec_reply = await self.task_poll_for_reply
        except asyncio.CancelledError:
            # can only be cancelled by task_poll_kernel_alive when the kernel is dead
            task_poll_output_msg.cancel()
            raise DeadKernelError("Kernel died")
        except Exception as e:
            # Best effort to cancel request if it hasn't been resolved
            try:
                # Check if the task_poll_output is doing the raising for us
                if not isinstance(e, CellControlSignal):
                    task_poll_output_msg.cancel()
            finally:
                raise

        if execution_count:
            cell['execution_count'] = execution_count
        await run_hook(
            self.on_cell_executed, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
>       await self._check_raise_for_error(cell, cell_index, exec_reply)

/usr/local/lib/python3.8/dist-packages/nbclient/client.py:1025:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <testbook.client.TestbookNotebookClient object at 0x7fe52211ba60>
cell = {'cell_type': 'code', 'execution_count': 8, 'id': '2fefd5b8', 'metadata': {'execution': {'iopub.status.busy': '2022-09...enceServerClient("localhost:8001") as client:\n    response = client.infer("ensemble_model", inputs, outputs=outputs)'}
cell_index = 17
exec_reply = {'buffers': [], 'content': {'ename': 'InferenceServerException', 'engine_info': {'engine_id': -1, 'engine_uuid': '4abc...e, 'engine': '4abc3f4b-b233-4593-9841-b36d5c06d84e', 'started': '2022-09-27T02:54:36.895621Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
    ) -> None:

        if exec_reply is None:
            return None

        exec_reply_content = exec_reply['content']
        if exec_reply_content['status'] != 'error':
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get('ename') in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           from merlin.systems.triton import convert_df_to_triton_input
E           import tritonclient.grpc as grpcclient
E
E           ten_examples = train.compute().drop(columns=['rating', 'title', 'rating_binary'])[:10]
E           inputs = convert_df_to_triton_input(inf_workflow.input_schema, ten_examples, grpcclient.InferInput)
E
E           outputs = [
E               grpcclient.InferRequestedOutput(col)
E               for col in inf_ops.output_schema.column_names
E           ]
E           # send request to tritonserver
E           with grpcclient.InferenceServerClient("localhost:8001") as client:
E               response = client.infer("ensemble_model", inputs, outputs=outputs)
E           ------------------
E
E           ---------------------------------------------------------------------------
E           InferenceServerException                  Traceback (most recent call last)
E           Input In [8], in <cell line: 12>()
E                11 # send request to tritonserver
E                12 with grpcclient.InferenceServerClient("localhost:8001") as client:
E           ---> 13     response = client.infer("ensemble_model", inputs, outputs=outputs)
E
E           File /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py:1322, in InferenceServerClient.infer(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)
E              1320     return result
E              1321 except grpc.RpcError as rpc_error:
E           -> 1322     raise_error_grpc(rpc_error)
E
E           File /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py:62, in raise_error_grpc(rpc_error)
E                61 def raise_error_grpc(rpc_error):
E           ---> 62     raise get_error_grpc(rpc_error) from None
E
E           InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '1_predictforest', message: TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
E               1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)
E
E           Invoked with: kwargs: tensors=[], error="<class 'c_python_backend_utils.TritonModelException'>, Failed to open the cudaIpcHandle. error: unknown error, [<FrameSummary file /systems/ensemble/1_predictforest/1/model.py, line 105 in execute>, <FrameSummary file /systems/merlin/systems/dag/op_runner.py, line 54 in execute>, <FrameSummary file /systems/merlin/systems/dag/ops/fil.py, line 161 in transform>]"
E
E           At:
E             /systems/ensemble/1_predictforest/1/model.py(122): execute
E
E           InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '1_predictforest', message: TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
E               1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)
E
E           Invoked with: kwargs: tensors=[], error="<class 'c_python_backend_utils.TritonModelException'>, Failed to open the cudaIpcHandle. error: unknown error, [<FrameSummary file /systems/ensemble/1_predictforest/1/model.py, line 105 in execute>, <FrameSummary file /systems/merlin/systems/dag/op_runner.py, line 54 in execute>, <FrameSummary file /systems/merlin/systems/dag/ops/fil.py, line 161 in transform>]"
E
E           At:
E             /systems/ensemble/1_predictforest/1/model.py(122): execute

/usr/local/lib/python3.8/dist-packages/nbclient/client.py:919: CellExecutionError

During handling of the above exception, another exception occurred:

tb = <testbook.client.TestbookNotebookClient object at 0x7fe52211ba60>

    @pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
    @pytest.mark.notebook
    @testbook(REPO_ROOT / "examples/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb", execute=False)
    def test_example_serving_xgboost(tb):
        tb.inject(
            """
            from unittest.mock import patch
            from merlin.datasets.synthetic import generate_data
            mock_train, mock_valid = generate_data(
                input="movielens-100k",
                num_rows=1000,
                set_sizes=(0.8, 0.2)
            )
            p1 = patch(
                "merlin.datasets.entertainment.get_movielens",
                return_value=[mock_train, mock_valid]
            )
            p1.start()
            """
        )
        NUM_OF_CELLS = len(tb.cells)
        # TODO: the following line is a hack -- remove when merlin-models#624 gets fixed
        tb.cells[4].source = tb.cells[4].source.replace(
            "without(['rating_binary', 'title'])", "without(['rating_binary', 'title', 'userId_count'])"
        )
        tb.execute_cell(list(range(0, 14)))

        with run_triton_server("ensemble", grpc_port=8001):
>           tb.execute_cell(list(range(14, NUM_OF_CELLS)))

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <testbook.client.TestbookNotebookClient object at 0x7fe52211ba60>
cell = {'cell_type': 'markdown', 'id': '7808bc12', 'metadata': {}, 'source': "Let's now package the information up as inputs and send it to Triton for inference."}
kwargs = {}, cell_indexes = [14, 15, 16, 17, 18, 19, ...]
executed_cells = [{'cell_type': 'markdown', 'id': '65b7e4e8', 'metadata': {}, 'source': '## Retrieving Recommendations from Triton Infe...c12', 'metadata': {}, 'source': "Let's now package the information up as inputs and send it to Triton for inference."}]
idx = 17

    def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
        """
        Executes a cell or list of cells
        """
        if isinstance(cell, slice):
            start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
            if cell.step is not None:
                raise TestbookError('testbook does not support step argument')

            cell = range(start, stop + 1)
        elif isinstance(cell, str) or isinstance(cell, int):
            cell = [cell]

        cell_indexes = cell

        if all(isinstance(x, str) for x in cell):
            cell_indexes = [self._cell_index(tag) for tag in cell]

        executed_cells = []
        for idx in cell_indexes:
            try:
                cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
            except CellExecutionError as ce:
>               raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))
E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:
E               ------------------
E               from merlin.systems.triton import convert_df_to_triton_input
E               import tritonclient.grpc as grpcclient
E
E               ten_examples = train.compute().drop(columns=['rating', 'title', 'rating_binary'])[:10]
E               inputs = convert_df_to_triton_input(inf_workflow.input_schema, ten_examples, grpcclient.InferInput)
E
E               outputs = [
E                   grpcclient.InferRequestedOutput(col)
E                   for col in inf_ops.output_schema.column_names
E               ]
E               # send request to tritonserver
E               with grpcclient.InferenceServerClient("localhost:8001") as client:
E                   response = client.infer("ensemble_model", inputs, outputs=outputs)
E               ------------------
E
E               ---------------------------------------------------------------------------
E               InferenceServerException                  Traceback (most recent call last)
E               Input In [8], in <cell line: 12>()
E                    11 # send request to tritonserver
E                    12 with grpcclient.InferenceServerClient("localhost:8001") as client:
E               ---> 13     response = client.infer("ensemble_model", inputs, outputs=outputs)
E
E               File /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py:1322, in InferenceServerClient.infer(self, model_name, inputs, model_version, outputs, request_id, sequence_id, sequence_start, sequence_end, priority, timeout, client_timeout, headers, compression_algorithm)
E                  1320     return result
E                  1321 except grpc.RpcError as rpc_error:
E               -> 1322     raise_error_grpc(rpc_error)
E
E               File /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/__init__.py:62, in raise_error_grpc(rpc_error)
E                    61 def raise_error_grpc(rpc_error):
E               ---> 62     raise get_error_grpc(rpc_error) from None
E
E               InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '1_predictforest', message: TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
E                   1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)
E
E               Invoked with: kwargs: tensors=[], error="<class 'c_python_backend_utils.TritonModelException'>, Failed to open the cudaIpcHandle. error: unknown error, [<FrameSummary file /systems/ensemble/1_predictforest/1/model.py, line 105 in execute>, <FrameSummary file /systems/merlin/systems/dag/op_runner.py, line 54 in execute>, <FrameSummary file /systems/merlin/systems/dag/ops/fil.py, line 161 in transform>]"
E
E               At:
E                 /systems/ensemble/1_predictforest/1/model.py(122): execute
E
E               InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '1_predictforest', message: TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
E                   1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)
E
E               Invoked with: kwargs: tensors=[], error="<class 'c_python_backend_utils.TritonModelException'>, Failed to open the cudaIpcHandle. error: unknown error, [<FrameSummary file /systems/ensemble/1_predictforest/1/model.py, line 105 in execute>, <FrameSummary file /systems/merlin/systems/dag/op_runner.py, line 54 in execute>, <FrameSummary file /systems/merlin/systems/dag/ops/fil.py, line 161 in transform>]"
E
E               At:
E                 /systems/ensemble/1_predictforest/1/model.py(122): execute

/usr/local/lib/python3.8/dist-packages/testbook/client.py:135: TestbookRuntimeError
----------------------------- Captured stdout call -----------------------------
Signal (2) received.
----------------------------- Captured stderr call -----------------------------
2022-09-27 10:54:28,648 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
[10:54:30] task [xgboost.dask]:tcp://<IP_ADDRESS>:34303 got new rank 0
I0927 02:54:31.052199 257 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x204e00000' with size 268435456
I0927 02:54:31.052354 257 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I0927 02:54:31.053728 257 model_repository_manager.cc:1191] loading: 1_predictforest:1
I0927 02:54:31.154411 257 model_repository_manager.cc:1191] loading: 0_transformworkflow:1
I0927 02:54:31.157013 257 python_be.cc:1774] TRITONBACKEND_ModelInstanceInitialize: 1_predictforest (GPU device 0)
I0927 02:54:31.255020 257 model_repository_manager.cc:1191] loading: 1_fil:1
I0927 02:54:33.225941 257 model_repository_manager.cc:1345] successfully loaded '1_predictforest' version 1
I0927 02:54:33.234426 257 initialize.hpp:43] TRITONBACKEND_Initialize: fil
I0927 02:54:33.234461 257 backend.hpp:47] Triton TRITONBACKEND API version: 1.10
I0927 02:54:33.234466 257 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10
I0927 02:54:33.234637 257 python_be.cc:1774] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)
I0927 02:54:34.749860 257 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_fil (version 1)
I0927 02:54:34.750320 257 model_repository_manager.cc:1345] successfully loaded '0_transformworkflow' version 1
I0927 02:54:34.750652 257 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_fil_0 (GPU device 0)
I0927 02:54:34.832266 257 model_repository_manager.cc:1345] successfully loaded '1_fil' version 1
I0927 02:54:34.832870 257 model_repository_manager.cc:1191] loading: ensemble_model:1
I0927 02:54:34.933322 257 model_repository_manager.cc:1345] successfully loaded 'ensemble_model' version 1
I0927 02:54:34.933439 257 server.cc:556]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0927 02:54:34.933481 257 server.cc:583]
+---------+-------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Backend | Path                                                  | Config
                                                                                |
+---------+-------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
| fil     | /opt/tritonserver/backends/fil/libtriton_fil.so       | {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |
+---------+-------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0927 02:54:34.933508 257 server.cc:626]
+---------------------+---------+--------+
| Model               | Version | Status |
+---------------------+---------+--------+
| 0_transformworkflow | 1       | READY  |
| 1_fil               | 1       | READY  |
| 1_predictforest     | 1       | READY  |
| ensemble_model      | 1       | READY  |
+---------------------+---------+--------+

I0927 02:54:34.961167 257 metrics.cc:650] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU
I0927 02:54:34.962185 257 tritonserver.cc:2159]
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value
                                                                                |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton
                                                                                |
| server_version                   | 2.23.0
                                                                                |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0]         | ensemble
                                                                                |
| model_control_mode               | MODE_NONE
                                                                                |
| strict_model_config              | 1
                                                                                |
| rate_limit                       | OFF
                                                                                |
| pinned_memory_pool_byte_size     | 268435456
                                                                                |
| cuda_memory_pool_byte_size{0}    | 67108864
                                                                                |
| response_cache_byte_size         | 0
                                                                                |
| min_supported_compute_capability | 6.0
                                                                                |
| strict_readiness                 | 1
                                                                                |
| exit_timeout                     | 30
                                                                                |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

E0927 10:54:34.963027993     257 server_chttp2.cc:40]        {"created":"@1664247274.963015133","description":"Only 1 addresses added out of total 2 resolved","file":"/tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":403,"referenced_errors":[{"created":"@1664247274.963013601","description":"Address family not supported by protocol","errno":97,"file":"/tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/grpc-repo/src/grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":395,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:8001"}]}
I0927 02:54:34.963723 257 grpc_server.cc:4587] Started GRPCInferenceService at localhost:8001
I0927 02:54:34.963919 257 http_server.cc:3303] Started HTTPService at <IP_ADDRESS>:8000
I0927 02:54:35.006502 257 http_server.cc:178] Started Metrics Service at <IP_ADDRESS>:8002
W0927 02:54:35.965807 257 metrics.cc:426] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0927 02:54:36.966135 257 metrics.cc:426] Unable to get power limit for GPU 0. Status:Success, value:0.000000
0927 02:54:37.553849 266 pb_stub.cc:776] Failed to process the request(s) for model '1_predictforest', message: TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)

Invoked with: kwargs: tensors=[], error="<class 'c_python_backend_utils.TritonModelException'>, Failed to open the cudaIpcHandle. error: unknown error, [<FrameSummary file /systems/ensemble/1_predictforest/1/model.py, line 105 in execute>, <FrameSummary file /systems/merlin/systems/dag/op_runner.py, line 54 in execute>, <FrameSummary file /systems/merlin/systems/dag/ops/fil.py, line 161 in transform>]"

At:
  /systems/ensemble/1_predictforest/1/model.py(122): execute

W0927 02:54:37.970011 257 metrics.cc:426] Unable to get power limit for GPU 0. Status:Success, value:0.000000
I0927 02:54:38.182747 257 server.cc:257] Waiting for in-flight requests to complete.
I0927 02:54:38.182770 257 server.cc:273] Timeout 30: Found 0 model versions that have in-flight inferences
I0927 02:54:38.182776 257 model_repository_manager.cc:1223] unloading: ensemble_model:1
I0927 02:54:38.182846 257 model_repository_manager.cc:1223] unloading: 1_fil:1
I0927 02:54:38.182906 257 model_repository_manager.cc:1223] unloading: 1_predictforest:1
I0927 02:54:38.183108 257 model_repository_manager.cc:1223] unloading: 0_transformworkflow:1
I0927 02:54:38.183116 257 model_repository_manager.cc:1328] successfully unloaded 'ensemble_model' version 1
I0927 02:54:38.183365 257 instance_finalize.hpp:36] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0927 02:54:38.183563 257 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state
I0927 02:54:38.183591 257 model_repository_manager.cc:1328] successfully unloaded '1_fil' version 1
I0927 02:54:38.183598 257 server.cc:288] All models are stopped, unloading models
I0927 02:54:38.183615 257 server.cc:295] Timeout 30: Found 2 live models and 0 in-flight non-inference requests
I0927 02:54:39.183726 257 server.cc:295] Timeout 29: Found 2 live models and 0 in-flight non-inference requests
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
  warnings.warn(
I0927 02:54:39.338012 257 model_repository_manager.cc:1328] successfully unloaded '1_predictforest' version 1
I0927 02:54:39.553948 257 model_repository_manager.cc:1328] successfully unloaded '0_transformworkflow' version 1
I0927 02:54:40.183876 257 server.cc:295] Timeout 28: Found 0 live models and 0 in-flight non-inference requests
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
  warnings.warn(
/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 15 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
=============================== warnings summary ===============================
../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:19
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:19: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    DESCRIPTOR = _descriptor.FileDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:37
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:37: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:44
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:44: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:51
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:51: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:30
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:30: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _INTEGERSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:82
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:82: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:89
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:89: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:96
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:96: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:75
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:75: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _DOUBLESTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:127
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:127: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:134
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:134: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:141
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:141: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:120
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:120: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _STRINGSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:172
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:172: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:165
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:165: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _BUCKETSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:203
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:203: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:210
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:210: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:217
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:217: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:196
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:196: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _DECIMALSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:248
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:248: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:255
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:255: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:241
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:241: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _DATESTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:286
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:286: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:293
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:293: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:300
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:300: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:307
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:307: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:279
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:279: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _TIMESTAMPSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:338
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:338: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:331
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:331: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _BINARYSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:369
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:369: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:376
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:376: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:383
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:383: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:390
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:390: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:397
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:397: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:404
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:404: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:411
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:411: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:418
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:418: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:425
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:425: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:432
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:432: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:362
  /usr/local/lib/python3.8/dist-packages/cudf/utils/metadata/orc_column_statistics_pb2.py:362: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _COLUMNSTATISTICS = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:19
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:19: DeprecationWarning: Call to deprecated create function FileDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    DESCRIPTOR = _descriptor.FileDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:33
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:33: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:37
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:37: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:41
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:41: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:45
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:45: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:49
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:49: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:53
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:53: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:57
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:57: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:61
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:61: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:65
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:65: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:69
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:69: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:73
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:73: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:77
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:77: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:81
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:81: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:85
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:85: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:89
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:89: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:27
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:27: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _DATATYPE = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:125
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:125: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:119
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:119: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELINSTANCEGROUP_SECONDARYDEVICE_SECONDARYDEVICEKIND = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:143
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:143: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:147
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:147: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:151
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:151: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:155
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:155: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:137
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:137: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELINSTANCEGROUP_KIND = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:173
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:173: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:177
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:177: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:181
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:181: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:167
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:167: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELINPUT_FORMAT = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:199
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:199: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:203
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:203: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:207
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:207: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:211
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:211: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:215
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:215: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:219
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:219: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:193
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:193: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _BATCHINPUT_KIND = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:237
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:237: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:231
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:231: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _BATCHOUTPUT_KIND = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:255
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:255: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:259
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:259: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:263
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:263: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:249
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:249: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELOPTIMIZATIONPOLICY_MODELPRIORITY = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:281
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:281: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:285
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:285: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:275
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:275: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELQUEUEPOLICY_TIMEOUTACTION = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:303
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:303: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:307
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:307: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:311
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:311: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:315
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:315: DeprecationWarning: Call to deprecated create function EnumValueDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.EnumValueDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:297
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:297: DeprecationWarning: Call to deprecated create function EnumDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELSEQUENCEBATCHING_CONTROL_KIND = _descriptor.EnumDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:335
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:335: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:342
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:342: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:349
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:349: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:328
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:328: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELRATELIMITER_RESOURCE = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:379
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:379: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:386
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:386: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:372
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:372: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELRATELIMITER = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:417
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:417: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:424
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:424: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:410
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:410: DeprecationWarning: Call to deprecated create function Descriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _MODELINSTANCEGROUP_SECONDARYDEVICE = _descriptor.Descriptor(

../usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:455
  /usr/local/lib/python3.8/dist-packages/tritonclient/grpc/model_config_pb2.py:455: DeprecationWarning: Call to deprecated create function FieldDescriptor(). Note: Create unlinked descriptors is going to go away. Please use get/find descriptors from generated code or query the descriptor_pool.
    _descriptor.FieldDescriptor(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py::test_example_serving_xgboost
======================= 1 failed, 100 warnings in 23.85s =======================

I use the nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 image.
I start the container and update the repositories using the following commands:
cd /models && git pull && pip install .
cd /nvtabular && git pull && pip install .
cd /core && git pull && pip install .
cd /systems && git pull && pip install .

I ensure I am on main in systems and that I have pulled before running the tests inside the container.
From /systems I run the test using the following command:
pytest tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py

And the test passes without errors.
root@50acea2c936f:/systems# pytest tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py                                                                                ================================================================================================================================================================================== test session starts ===================================================================================================================================================================================platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /systems, configfile: pytest.ini
plugins: anyio-3.6.1
collected 1 item
tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py .

This is nothing more than a wild guess, but looking at this error and doing a bit of search online:
Invoked with: kwargs: tensors=[], error="<class 'c_python_backend_utils.TritonModelException'>, Failed to open the cudaIpcHandle. error: unknown error, [<FrameSummary file /systems/ensemble/1_predictforest/1/model.py, line 105 in execute>, <FrameSummary file /systems/merlin/systems/dag/op_runner.py, line 54 in execute>, <FrameSummary file /systems/merlin/systems/dag/ops/fil.py, line 161 in transform>]"
it seems there is a chance this could be related to CUDA version on the host machine. This is the version that I am running: Driver Version: 515.65.01    CUDA Version: 11.7 (output obtained via nvidia-smi). Could you please verify the CUDA version you are running?

I'm running docker with the latest driver installed on my host machine that supports CUDA 11.7. With driver untouched I did got triton worked once in the previous comment, yet I can't replicate it.
This is what I got from nvidia-smi
nvidia-smi
Tue Sep 27 16:42:46 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 516.94       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A   51C    P0    17W /  N/A |      0MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

@BorisPolonsky I have also been getting the error Failed to open the cudaIpcHandle running this on WSL 2 Ubuntu with the nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 image (and earlier ones too).
The good news is that I have tried this on the new pre-release 22.09 image and I don't get that error anymore. This new image should be available later today. It includes an updated version of triton server, so it seems like whatever it was that caused this cuda error in WSL 2 environments may have been resolved.

@BorisPolonsky I have also been getting the error Failed to open the cudaIpcHandle running this on WSL 2 Ubuntu with the nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 image (and earlier ones too).
The good news is that I have tried this on the new pre-release 22.09 image and I don't get that error anymore. This new image should be available later today. It includes an updated version of triton server, so it seems like whatever it was that caused this cuda error in WSL 2 environments may have been resolved.

I'll test it later. If things got solved I'll close this issue and the one I submitted separately in nvidia/triton-inference-server.

@BorisPolonsky I have also been getting the error Failed to open the cudaIpcHandle running this on WSL 2 Ubuntu with the nvcr.io/nvidia/merlin/merlin-tensorflow:22.07 image (and earlier ones too).
The good news is that I have tried this on the new pre-release 22.09 image and I don't get that error anymore. This new image should be available later today. It includes an updated version of triton server, so it seems like whatever it was that caused this cuda error in WSL 2 environments may have been resolved.

The 22.09 version of the image is not released yet on NGC. The latest version is still 22.07.

@BorisPolonsky thank you for your patient. The 22.09 container is released, now: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow/tags
Have you had a chance to test it?

@BorisPolonsky thank you for your patient. The 22.09 container is released, now: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow/tags
Have you had a chance to see if you still get the error?

I have trouble pulling the image due to network condition. I'll try it later.

@BorisPolonsky thank you for your patient. The 22.09 container is released, now: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/merlin/containers/merlin-tensorflow/tags
Have you had a chance to see if you still get the error?

Dockerfile
FROM nvcr.io/nvidia/merlin/merlin-tensorflow:22.09
WORKDIR /models
RUN git pull origin main && pip install .
WORKDIR /nvtabular
RUN git pull origin main && pip install .
WORKDIR /core
RUN git pull origin main && pip install .
WORKDIR /systems
RUN git pull origin main && pip install .
WORKDIR /opt/tritonserver

pytest
docker run --rm --workdir /systems -e LANG=C.UTF-8 -v /etc/localtime:/etc/localtime:ro --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/merlin/merlin-tensorflow:22.09-fastforward pytest tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py

==================================
== Triton Inference Server Base ==
==================================

NVIDIA Release 22.08 (build 42766143)

Copyright (c) 2018-2022, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
   Use the NVIDIA Container Toolkit to start this container with GPU support; see
   https://docs.nvidia.com/datacenter/cloud-native/ .

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /systems, configfile: pytest.ini
plugins: anyio-3.6.1
collected 1 item

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py F [100%]

=================================== FAILURES ===================================
_________________________ test_example_serving_xgboost _________________________

tb = <testbook.client.TestbookNotebookClient object at 0x7fc3e582d2b0>

    @pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
    @pytest.mark.notebook
    @testbook(REPO_ROOT / "examples/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb", execute=False)
    def test_example_serving_xgboost(tb):
>       tb.inject(
            """
            from unittest.mock import patch
            from merlin.datasets.synthetic import generate_data
            mock_train, mock_valid = generate_data(
                input="movielens-100k",
                num_rows=1000,
                set_sizes=(0.8, 0.2)
            )
            p1 = patch(
                "merlin.datasets.entertainment.get_movielens",
                return_value=[mock_train, mock_valid]
            )
            p1.start()
            """
        )

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py:20:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <testbook.client.TestbookNotebookClient object at 0x7fc3e582d2b0>
code = '\n        from unittest.mock import patch\n        from merlin.datasets.synthetic import generate_data\n        mock_...ertainment.get_movielens",\n            return_value=[mock_train, mock_valid]\n        )\n        p1.start()\n        '
args = None, kwargs = None, run = True, before = None, after = None, pop = False

    def inject(
        self,
        code: str,
        args: List = None,
        kwargs: Dict = None,
        run: bool = True,
        before: Optional[Union[str, int]] = None,
        after: Optional[Union[str, int]] = None,
        pop: bool = False,
    ) -> TestbookNode:
        """Injects and executes given code block

        Parameters
        ----------
        code : str
            Code or function to be injected
        args : iterable, optional
            tuple of arguments to be passed to the function
        kwargs : dict, optional
            dict of keyword arguments to be passed to the function
        run : bool, optional
            Control immediate execution after injection (default is True)
        before, after : int, str, optional
            Inject code before or after cell
        pop : bool
            Pop cell after execution (default is False)

        Returns
        -------
        TestbookNode
            Injected cell
        """

        if isinstance(code, str):
            lines = dedent(code)
        elif callable(code):
            lines = getsource(code) + (
                dedent(self._construct_call_code(code.__name__, args, kwargs)) if run else ''
            )
        else:
            raise TypeError('can only inject function or code block as str')

        inject_idx = len(self.cells)

        if after is not None and before is not None:
            raise ValueError("pass either before or after as kwargs")
        elif before is not None:
            inject_idx = self._cell_index(before)
        elif after is not None:
            inject_idx = self._cell_index(after) + 1

        code_cell = new_code_cell(lines)
        self.cells.insert(inject_idx, code_cell)

        cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)

        if self._contains_error(cell):
            eclass = self._get_error_class(cell.get('outputs')[0]['ename'])
            evalue = cell.get('outputs')[0]['evalue']
>           raise TestbookRuntimeError(evalue, evalue, eclass)
E           testbook.exceptions.TestbookRuntimeError: Failed to dlopen libcuda.so

/usr/local/lib/python3.8/dist-packages/testbook/client.py:242: TestbookRuntimeError
=============================== warnings summary ===============================
../usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py:62
  /usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py:62: UserWarning: Function "cuDeviceGetCount" not found
    warnings.warn(str(e))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py::test_example_serving_xgboost

I was able to replicate your error and then fix it by adding this to the execution command: --shm-size 8g (you may not need a full 8g, but the flag should be set for triton)