Failure to query data from the core

Issue description
It is not possible to operate with data in the orchestrator (e.g., on processes). An Internal Server Error is returned, indicating the "processes" table is not available in the DB.
Steps to reproduce the issue
Instructions from the STF slides (pp. 14-17) were used (just changing the main.py name).

virtualenv venv --python=python3.9.5

source venv/bin/activate

(venv) pip install orchestrator-core

docker run --name postgres-orchestrator -p 5432:5432 -e POSTGRES_PASSWORD=nwa -d postgres

# set any password (here, "nwa")
docker exec -it postgres-orchestrator createuser -sP nwa -p 5432 -h <IP_ADDRESS> -U postgres

docker exec -it postgres-orchestrator createdb orchestrator-core -O nwa -p 5432 -h <IP_ADDRESS> -U postgres

docker run --name redis-orchestrator -p 6379:6379 -d redis

cat<<EOF>main_cli.py
from orchestrator import OrchestratorCore
from orchestrator.cli.main import app as core_cli
from orchestrator.settings import AppSettings

app = OrchestratorCore(base_settings=AppSettings())

if __name__ == "__main__":
    core_cli()
EOF

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db init

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db upgrade

(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages uvicorn --reload --host <IP_ADDRESS> --port 8080 main_cli:app

 curl -X 'GET' http://<IP_ADDRESS>:8080/api/processes/ -H 'accept: application/json'

What's the expected result?
Not sure. Although I would guess an empty list of products at this stage.
What's the actual result?
When initialising the DB:
# (Command previously run in this environment, backing up and removing "migrations" folder)
mv migrations migrations.bak

# Initialising DB
(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db init
ujson module not found, using json
2022-03-29 20:09.13 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-03-29 20:09.13 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-03-29 18:09:13 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-03-29 18:09:13 [info     ] Creating directory             [orchestrator.cli.database] directory=./surfnet/orchestrator-core/migrations
2022-03-29 18:09:13 [info     ] Creating directory             [orchestrator.cli.database] directory=./surfnet/orchestrator-core/migrations/versions
2022-03-29 18:09:13 [info     ] Creating directory             [orchestrator.cli.database] directory=./surfnet/orchestrator-core/migrations/versions/schema
2022-03-29 18:09:13 [info     ] Creating file                  [orchestrator.cli.database] file=./surfnet/orchestrator-core/migrations/env.py
2022-03-29 18:09:13 [info     ] Creating file                  [orchestrator.cli.database] file=./surfnet/orchestrator-core/migrations/script.py.mako
2022-03-29 18:09:13 [info     ] Creating file                  [orchestrator.cli.database] file=./surfnet/orchestrator-core/migrations/helpers.py
2022-03-29 18:09:13 [info     ] Skipping Alembic.ini file. It already exists [orchestrator.cli.database]

# Checking generated directory
(venv) $ tree migrations
migrations
├── env.py
├── helpers.py
├── script.py.mako
└── versions
    └── schema

When performing a DB upgrade:
(venv) $ PYTHONPATH=./venv/lib/python3.9/site-packages python main_cli.py db upgrade
(...)
2022-03-29 18:10:55 [info     ] Version Locations              [orchestrator.cli.database] locations=migrations/versions/general ./surfnet/orchestrator-core/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "./surfnet/orchestrator-core/main_cli.py", line 10, in <module>
    core_cli()
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "./surfnet/orchestrator-core/orchestrator/cli/database.py", line 139, in upgrade
    command.upgrade(alembic_cfg(), revision)
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/alembic/command.py", line 277, in upgrade
    if ":" in revision:
TypeError: argument of type 'NoneType' is not iterable

When querying the API, it returns an Internal Server Error (see details from the API logs):
curl -X 'GET' http://<IP_ADDRESS>:8080/api/processes/ -H 'accept: application/json'

>> Details:
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "./surfnet/orchestrator-core/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "processes" does not exist
LINE 2: FROM processes LEFT OUTER JOIN processes_subscriptions AS pr…
[SQL: SELECT processes.pid AS processes_pid, processes.workflow AS processes_workflow, processes.assignee AS processes_assignee, processes.last_status AS processes_last_status, processes.last_step AS processes_last_step, processes.started_at AS processes_started_at, processes.last_modified_at AS processes_last_modified_at, processes.failed_reason AS processes_failed_reason, processes.created_by AS processes_created_by, processes.is_task AS processes_is_task, products_1.product_id AS products_1_product_id, products_1.name AS products_1_name, products_1.description AS products_1_description, products_1.product_type AS products_1_product_type, products_1.tag AS products_1_tag, products_1.status AS products_1_status, products_1.created_at AS products_1_created_at, products_1.end_date AS products_1_end_date, subscriptions_1.subscription_id AS subscriptions_1_subscription_id, subscriptions_1.description AS subscriptions_1_description, subscriptions_1.status AS subscriptions_1_status, subscriptions_1.product_id AS subscriptions_1_product_id, subscriptions_1.customer_id AS subscriptions_1_customer_id, subscriptions_1.insync AS subscriptions_1_insync, subscriptions_1.start_date AS subscriptions_1_start_date, subscriptions_1.end_date AS subscriptions_1_end_date, subscriptions_1.note AS subscriptions_1_note, processes_subscriptions_1.id AS processes_subscriptions_1_id, processes_subscriptions_1.pid AS processes_subscriptions_1_pid, processes_subscriptions_1.subscription_id AS processes_subscriptions_1_subscription_id, processes_subscriptions_1.created_at AS processes_subscriptions_1_created_at, processes_subscriptions_1.workflow_target AS processes_subscriptions_1_workflow_target
FROM processes LEFT OUTER JOIN processes_subscriptions AS processes_subscriptions_1 ON processes.pid = processes_subscriptions_1.pid LEFT OUTER JOIN subscriptions AS subscriptions_1 ON subscriptions_1.subscription_id = processes_subscriptions_1.subscription_id LEFT OUTER JOIN products AS products_1 ON products_1.product_id = subscriptions_1.product_id]

Database content shows just the DB, not the tables:
$ docker exec -it postgres-orchestrator /bin/bash
root@ac30141c3cc1:/# su postgres
postgres@ac30141c3cc1:/$ psql
psql (14.2 (Debian 14.2-1.pgdg110+1))
Type "help" for help.

postgres=# \l
                                     List of databases
       Name        |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-------------------+----------+----------+------------+------------+-----------------------
 orchestrator-core | nwa      | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres          | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                   |          |          |            |            | postgres=CTc/postgres
 template1         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                   |          |          |            |            | postgres=CTc/postgres
(4 rows)

postgres=# \dt
Did not find any relations.

Used environment

OS: Ubuntu 20.04.4 LTS
Docker: 20.10.14, build a224086
Python: 3.8.10

Additional details / screenshot
N/A

Seems like the migrations are not executed. Could you try just starting the webserver: that would normally trigger the migrations. And you should be able to see that in the logs.
side note: python 3.8 or 3.9: the docker setup is 3.8, the rest of your description 3.9?

Seems like the migrations are not executed. Could you try just starting the webserver: that would normally trigger the migrations. And you should be able to see that in the logs.

I understand that was step 11, prior to the cURL call. If that worked, should the Postgre DB have some relations defined at that stage? It does not have any.

side note: python 3.8 or 3.9: the docker setup is 3.8, the rest of your description 3.9?

Started with Python 3.8 as system default. After the issues to deploy, I installed python 3.9 in my system and set it as default via alternatives. However, it breaks some binaries, so I am back to python 3.8.
In summary: currently it is python 3.8.10 system-wide and python 3.9.5 in venv (first time I try a venv with a different version than the one in the system, and it fails with missing libraries, i.e., pyscopg2). When trying the same version in venv, it complains about missing modules under sqlalchemy.exc

It seems that you are running the db init from inside the orchestrator-core folder. Did you start you own project and included orchestrator-core in the pip requirements or are you running the command in a clone of this repo?

The latter. When first starting with workflow-orchestrator, started with a clone of this project and used the development instructions (using flit, etc). In the last tests, used the pip requirement on orchestrator-core, as well as executing all commands under this same orchestrator-core repository folder.
I have tried now in a new folder. No repository of any kind there, just a new folder with the main_cli.py from above and a venv with the same version as Python system-wide, i.e., 3.8.10:
# Initialise DB
(venv) $ PYTHONPATH=. python main_cli.py db init
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:45.08 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:45.09 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:45:09 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations/versions
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations/versions/schema
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/env.py
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/script.py.mako
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/helpers.py
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/alembic.ini

# Run webserver
(venv) $ PYTHONPATH=. uvicorn --reload --host <IP_ADDRESS> --port 8080 main_cli:app
INFO:     Uvicorn running on http://<IP_ADDRESS>:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [39098] using watchgod
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:46.45 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:46.46 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:46:46 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
INFO:     Started server process [39100]
2022-03-30 13:46:46 [info     ] Started server process [39100] [uvicorn.error]
INFO:     Waiting for application startup.
2022-03-30 13:46:46 [info     ] Waiting for application startup. [uvicorn.error]
INFO:     Application startup complete.
2022-03-30 13:46:46 [info     ] Application startup complete.  [uvicorn.error]

No changes in the Postgre container w.r.t. the relations or the databases. Should this container be recreated, so no "orchestrator-core" database is present when running the init command (or the webserver or migrate steps)?
Also, when trying the upgrade command in this folder the same error from last time is produced:
(venv) $ PYTHONPATH=. python main_cli.py db upgrade
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:48.22 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:48.23 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:48:23 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-03-30 13:48:23 [info     ] Version Locations              [orchestrator.cli.database] locations=migrations/versions/general /home/carolina/.local/lib/python3.8/site-packages/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "main_cli.py", line 10, in <module>
    core_cli()
  File "/home/carolina/.local/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/cli/database.py", line 126, in upgrade
    command.upgrade(alembic_cfg(), revision)
  File "/home/carolina/.local/lib/python3.8/site-packages/alembic/command.py", line 277, in upgrade
    if ":" in revision:
TypeError: argument of type 'NoneType' is not iterable

Seems like the migrations are not executed. Could you try just starting the webserver: that would normally trigger the migrations. And you should be able to see that in the logs.

I understand that was step 11, prior to the cURL call. If that worked, should the Postgre DB have some relations defined at that stage? It does not have any.

side note: python 3.8 or 3.9: the docker setup is 3.8, the rest of your description 3.9?

Started with Python 3.8 as system default. After the issues to deploy, I installed python 3.9 in my system and set it as default via alternatives. However, it breaks some binaries, so I am back to python 3.8.
In summary: currently it is python 3.8.10 system-wide and python 3.9.5 in venv (first time I try a venv with a different version than the one in the system, and it fails with missing libraries, i.e., pyscopg2). When trying the same version in venv, it complains about missing modules under sqlalchemy.exc

It seems that you are running the db init from inside the orchestrator-core folder. Did you start you own project and included orchestrator-core in the pip requirements or are you running the command in a clone of this repo?

The latter. When first starting with workflow-orchestrator, started with a clone of this project and used the development instructions (using flit, etc). In the last tests, used the pip requirement on orchestrator-core, as well as executing all commands under this same orchestrator-core repository folder.
I have tried now in a new folder. No repository of any kind there, just a new folder with the main_cli.py from above and a venv with the same version as Python system-wide, i.e., 3.8.10:
# Initialise DB
(venv) $ PYTHONPATH=. python main_cli.py db init
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:45.08 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:45.09 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:45:09 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations/versions
2022-03-30 13:45:09 [info     ] Creating directory             [orchestrator.cli.database] directory=./orchestrator-core__mine/migrations/versions/schema
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/env.py
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/script.py.mako
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/migrations/helpers.py
2022-03-30 13:45:09 [info     ] Creating file                  [orchestrator.cli.database] file=./orchestrator-core__mine/alembic.ini

# Run webserver
(venv) $ PYTHONPATH=. uvicorn --reload --host <IP_ADDRESS> --port 8080 main_cli:app
INFO:     Uvicorn running on http://<IP_ADDRESS>:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [39098] using watchgod
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:46.45 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:46.46 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:46:46 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
INFO:     Started server process [39100]
2022-03-30 13:46:46 [info     ] Started server process [39100] [uvicorn.error]
INFO:     Waiting for application startup.
2022-03-30 13:46:46 [info     ] Waiting for application startup. [uvicorn.error]
INFO:     Application startup complete.
2022-03-30 13:46:46 [info     ] Application startup complete.  [uvicorn.error]

No changes in the Postgre container w.r.t. the relations or the databases. Should this container be recreated, so no "orchestrator-core" database is present when running the init command (or the webserver or migrate steps)?
Also, when trying the upgrade command in this folder the same error from last time is produced:
(venv) $ PYTHONPATH=. python main_cli.py db upgrade
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:48.22 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
ujson module not found, using json
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2022-03-30 15:48.23 [error    ] Could not get git commit hash
Traceback (most recent call last):
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/version.py", line 39, in __getattr__
    return check_output(["/usr/bin/env", "git", "rev-parse", "HEAD"]).decode().strip()  # noqa: S603
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/env', 'git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
2022-03-30 13:48:23 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-03-30 13:48:23 [info     ] Version Locations              [orchestrator.cli.database] locations=migrations/versions/general /home/carolina/.local/lib/python3.8/site-packages/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "main_cli.py", line 10, in <module>
    core_cli()
  File "/home/carolina/.local/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/carolina/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/carolina/.local/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/carolina/.local/lib/python3.8/site-packages/orchestrator/cli/database.py", line 126, in upgrade
    command.upgrade(alembic_cfg(), revision)
  File "/home/carolina/.local/lib/python3.8/site-packages/alembic/command.py", line 277, in upgrade
    if ":" in revision:
TypeError: argument of type 'NoneType' is not iterable

I've discussed this during our standup. Tomorrow I will have some time to start a small example project; so I can try to reproduce your problem.

I created an repo that shows how you can use the orchestrator from inside your own project:
https://github.com/acidjunk/orchestrator-demo
When you follow the README, setup a postgres user orchestrator-demo with password orchestrator-demo and installed all python deps + loaded the needed ENV vars, this should yield you a working DB:
Note: redis should be running...
dropdb orchestrator-demo
createdb orchestrator-demo
./start.sh

You can see that it runs the migrations from Core 0.4.0-RC6.

I also added an initial migration in the project itself to demonstrate how you can add your own tables.
Tested API endpoint (returns empty list, which is ok):

whilst working in this I think I encountered the same bug: I needed to create an init migration for my own orchestrator project so I did what the README mentioned:
PYTHONPATH=. python main.py db init -> that created a migrations folder with some files.
Then I wanted to create my own migration for the UserPreference model in my own project but the README doesn't cover it. I tried a couple of approaches but when a migration was created it would be inside the venvs site_packages folder instead of migrations/versions.
PYTHONPATH=. python main.py db revision --autogenerate -m "New schema" --head=schema@head (I know that the --head` parameter should probably contain something else; but not sure what....)
So I copied it myself to the correct place, to get it working.
Curious if this also happened with your setup.

@CarolinaFernandez did you manage to get your own project working?

Hi Rene,
I tried following the instructions from the new repository you created (https://github.com/acidjunk/orchestrator-demo.git) and then, the instructions from your message of 2nd April 2022. To that end, this time I ditched Docker and created a new VM in Vagrant (with Virtualbox), using Ubuntu 22.04 LTS, with Python 3.10.4, PostgreSQL 14 and Redis 5:6.0.16-1ubuntu1.
Summing up, I am now able to query the API (e.g., http://localhost:8080/api/products) without errors, so that it returns the expected empty list and where the server returns a 307 (temporary redirect). However, I can see no migrations (I guess it should be the file "migrations/versions/general/2022-04-02_022505714cf8_add_user_preference.py"?) when executing the command PYTHONPATH=. python main.py db init (I get "OSError: Directory migrations already exists and is not empty" and nothing new is prompted at the server side), so no chance of defining tables atm. I am also trying to setup the Vagrant environment properly so to access the Swagger UI from outside and try to force the creation of some resources to see which errors result from the persisting attempt.
Commands used below:
# Install Python3.10-dev (for venv)
sudo apt install -y python3.10-venv
# Install PostgreSQL
sudo apt install -y postgresql

#
# 1 . Perform steps from README.md
#

# Download specific repository
mkdir orchestrator-demo
cd orchestrator-demo
# git clone git@github.com:acidjunk/orchestrator-demo.git
git clone https://github.com/acidjunk/orchestrator-demo.git
cd orchestrator-demo
# Create venv and install all dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r ./requirements/all.txt

# Create PostgreSQL user
# IMPORTANT: add "postgresql" to the "vagrant" group in /etc/group,
# so that PostgreSQL commands can access the provided migrations
# and also sudo as user before running these commands
sudo -u postgres -i
createuser -s -P orchestrator-demo
createdb orchestrator-demo -O orchestrator-demo
createdb orchestrator-demo-test -O orchestrator-demo
exit

sudo systemctl restart postgresql
sudo systemctl status postgresql
# May 15 22:50:49 ubuntu-jammy systemd[1]: Starting PostgreSQL RDBMS...
# May 15 22:50:49 ubuntu-jammy systemd[1]: Finished PostgreSQL RDBMS.

# May 15 22:55:27 ubuntu-jammy systemd[1]: Started Advanced key-value store.

# Run in development mode
cat<<EOF>>.env
export HTTP_PORT=8080
export MAIL_PORT=1025
export NSO_ENABLED=True
export OAUTH2_ACTIVE=False
export STACK_LIMIT=""
export SERVER_THREADED=1
export TESTING=False
export DATABASE_URI="postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo"
export DATABASE_URI_TEST="postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo-test"
export WEBSOCKET_BROADCASTER_URL="redis://localhost:6379"

# Populator config
export POPULATOR_ORGANISATION="GEANT"
export POPULATOR_CONTACT_NAME="Jane Doe"
export<EMAIL_ADDRESS>export POPULATOR_CONTACT_PHONE="0099700900500" # optional
EOF

#
# 2. Perform steps from the GitHub issue tracker
#

# IMPORTANT: add "postgresql" to the "vagrant" group in /etc/group,
# so that PostgreSQL commands can access the provided migrations
# and also sudo as user before running these commands
sudo -u postgres -i
dropdb orchestrator-demo
createdb orchestrator-demo
exit

## You can also ensure that you loaded the needed env vars and run a hot reloading webserver
DATABASE_URI=postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo PYTHONPATH=. uvicorn main:app --reload --port 8080
# INFO:     Will watch for changes in these directories: ['/home/vagrant/orchestrator-demo/orchestrator-demo']
# INFO:     Uvicorn running on http://<IP_ADDRESS>:8080 (Press CTRL+C to quit)
# INFO:     Started reloader process [5916] using watchgod
# ujson module not found, using json
# msgpack not installed, MsgPackSerializer unavailable
# 2022-05-15 23:22.30 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='fgdJSOCPKKiJqWuZ' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='<IP_ADDRESS>' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://<IP_ADDRESS>:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
# 2022-05-15 23:22.30 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
# 2022-05-15 23:22.30 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
# 2022-05-15 23:22:30 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
# INFO:     Application startup complete.
# 2022-05-15 23:22:30 [info     ] Application startup complete.  [uvicorn.error]
# INFO:     Started server process [6713]
# 2022-05-15 23:22:30 [info     ] Started server process [6713]  [uvicorn.error]
# INFO:     Waiting for application startup.
# 2022-05-15 23:22:30 [info     ] Waiting for application startup. [uvicorn.error]
# 2022-05-15 23:22:30 [info     ] Connecting to redis            [asyncio_redis]
# 2022-05-15 23:22:30 [info     ] Redis connection made          [asyncio_redis]
# 2022-05-15 23:22:30 [info     ] Connecting to redis            [asyncio_redis]
# 2022-05-15 23:22:30 [info     ] Redis connection made          [asyncio_redis]
# INFO:     Application startup complete.

# Perform some cURL tests
curl http://localhost:8080/api/products
curl http://localhost:8080/api/products2
# {"detail":"Not Found"}

# INFO:     <IP_ADDRESS>:57672 - "GET /api/products HTTP/1.1" 307 Temporary Redirect
# INFO:     <IP_ADDRESS>:57674 - "GET /api/products2 HTTP/1.1" 404 Not Found

Hi @CarolinaFernandez If I understand you correctly at the moment the tables aren't created at the moment? What is the output of the following call. I expect a stacktrace when the database is not initialised:
curl http://localhost:8080/api/products/

Note the trailing slash. As you can see in the logs the first call returned a Temp Redirect that was not followed. (You need to use curl -L) to follow redirects on the CLI.
Another question I have is as follows:
What is the out put of the following command:
PYTHONPATH=. python main.py db upgrade heads

Does that yield any results. If that commands runs correctly I curious to know what the results of products curl call is.

Hi @pboers1988. Indeed, tables are not being created in the DB.
1. Call to products endpoint
Reduced log output for curl -L http://localhost:8080/api/products/:
(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ DATABASE_URI=postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo PYTHONPATH=. uvicorn main:app --reload --port 8080
INFO:     Will watch for changes in these directories: ['/home/vagrant/orchestrator-demo/orchestrator-demo']
INFO:     Uvicorn running on http://<IP_ADDRESS>:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [4001] using watchgod
ujson module not found, using json
msgpack not installed, MsgPackSerializer unavailable
2022-05-19 18:32.04 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='iVXSBPhlOIpZcoln' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='<IP_ADDRESS>' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://<IP_ADDRESS>:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
2022-05-19 18:32.04 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-05-19 18:32.04 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-05-19 18:32:04 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
INFO:     Started server process [4003]
2022-05-19 18:32:04 [info     ] Started server process [4003]  [uvicorn.error]
INFO:     Waiting for application startup.
2022-05-19 18:32:04 [info     ] Waiting for application startup. [uvicorn.error]
2022-05-19 18:32:04 [info     ] Connecting to redis            [asyncio_redis]
2022-05-19 18:32:04 [info     ] Redis connection made          [asyncio_redis]
2022-05-19 18:32:04 [info     ] Connecting to redis            [asyncio_redis]
2022-05-19 18:32:04 [info     ] Redis connection made          [asyncio_redis]
INFO:     Application startup complete.
2022-05-19 18:32:04 [info     ] Application startup complete.  [uvicorn.error]
INFO:     <IP_ADDRESS>:55202 - "GET /api/products/ HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
    self.dialect.do_execute(
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "products" does not exist
LINE 2: FROM products
             ^
(...)

[SQL: SELECT products.product_id AS products_product_id, products.name AS products_name, products.description AS products_description, products.product_type AS products_product_type, products.tag AS products_tag, products.status AS products_status, products.created_at AS products_created_at, products.end_date AS products_end_date
FROM products]
(Background on this error at: https://sqlalche.me/e/14/f405)

Output from Postgres (NOTE: I realised later that this connected to some database, I guess "postgres" that is not the orchestrator one. I cannot ensure the tables were not created at this moment):
vagrant@ubuntu-jammy:~$ sudo -u postgres -i
postgres@ubuntu-jammy:~$ psql
psql (14.2 (Ubuntu 14.2-1ubuntu1))
Type "help" for help.

postgres=# \dt
Did not find any relations.
postgres=# \l
                                         List of databases
          Name          |       Owner       | Encoding | Collate |  Ctype  |   Access privileges
------------------------+-------------------+----------+---------+---------+-----------------------
 orchestrator-demo      | postgres          | UTF8     | C.UTF-8 | C.UTF-8 |
 orchestrator-demo-test | orchestrator-demo | UTF8     | C.UTF-8 | C.UTF-8 |
 postgres               | postgres          | UTF8     | C.UTF-8 | C.UTF-8 |
 template0              | postgres          | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
                        |                   |          |         |         | postgres=CTc/postgres
 template1              | postgres          | UTF8     | C.UTF-8 | C.UTF-8 | =c/postgres          +
                        |                   |          |         |         | postgres=CTc/postgres
(5 rows)
postgres=# \du
                                       List of roles
     Role name     |                         Attributes                         | Member of
-------------------+------------------------------------------------------------+-----------
 orchestrator-demo | Superuser, Create role, Create DB                          | {}
 postgres          | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

2. Upgrade heads on DB
The command fails. Logs below:
vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ source .venv/bin/activate
(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ PYTHONPATH=. python main.py db upgrade heads
ujson module not found, using json
msgpack not installed, MsgPackSerializer unavailable
2022-05-19 18:39.19 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='HFAsBeKyOkZBdOAy' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://nwa:nwa@localhost/orchestrator-core' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='<IP_ADDRESS>' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://<IP_ADDRESS>:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
2022-05-19 18:39.19 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-05-19 18:39.19 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-05-19 18:39:19 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-05-19 18:39:19 [info     ] Version Locations              [orchestrator.cli.database] locations=/home/vagrant/orchestrator-demo/orchestrator-demo/migrations/versions/general /home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/orchestrator/migrations/versions/schema
Traceback (most recent call last):
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3250, in _wrap_pool_connect
    return fn()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 665, in __connect
    with util.safe_reraise():
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 590, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 597, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user "nwa"
FATAL:  password authentication failed for user "nwa"
(...)

(Background on this error at: https://sqlalche.me/e/14/e3q8)

Although the list of databases above seems to show that "postgres" is the owner, I get an error regarding password auth error on user "nwa". In this environment (a clean VM) I have not configured anything with that user. Unsure on why this is happening, as "DATABASE_URI" is passed explicitly when running uvicorn (although the connection string to the DB might suggest that it is connecting to "orchestrator-demo" database with the same user and password, whereas the database is owned by postgres).
Also, when grepping for the "nwa" term in the orchestrator-demo repo I see the METADATA file of the orchestrator_core package details the following commands when installed via pip : createuser -sP nwa, createdb orchestrator-core -O nwa, createuser -sP nwa, createdb orchestrator-core-test -O nwa. Just noticed this (nwa-stdlib) is another automation package from the workfloworchestrator repo and installed as dependency of the core.

Update 1: the command for the "upgrade heads" did not force the proper connection string to the DB, and was taking "nwa" (DATABASE_URI='postgresql://nwa:nwa@localhost/orchestrator-core') by default, thus failing.
New log output:
(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ DATABASE_URI=postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo PYTHONPATH=. python main.py db upgrade heads
ujson module not found, using json
msgpack not installed, MsgPackSerializer unavailable
2022-05-19 19:17.42 [debug    ] Starting the app with the following settings app_settings=TESTING=True SESSION_SECRET='CFaDpwmdRJlchkMv' CORS_ORIGINS='*' CORS_ALLOW_METHODS=['GET', 'PUT', 'PATCH', 'POST', 'DELETE', 'OPTIONS', 'HEAD'] CORS_ALLOW_HEADERS=['If-None-Match', 'Authorization', 'If-Match', 'Content-Type'] CORS_EXPOSE_HEADERS=['Cache-Control', 'Content-Language', 'Content-Length', 'Content-Type', 'Expires', 'Last-Modified', 'Pragma', 'Content-Range', 'ETag'] ENVIRONMENT='local' WORKFLOWS_SWAGGER_HOST='localhost' WORKFLOWS_GUI_URI='http://localhost:3000' DATABASE_URI='postgresql://orchestrator-demo:orchestrator-demo@localhost/orchestrator-demo' MAX_WORKERS=5 MAIL_SERVER='localhost' MAIL_PORT=25 MAIL_STARTTLS=False CACHE_HOST='<IP_ADDRESS>' CACHE_PORT=6379 ENABLE_DISTLOCK_MANAGER=True DISTLOCK_BACKEND='redis' CC_NOC=0 SERVICE_NAME='orchestrator-core' LOGGING_HOST='localhost' LOG_LEVEL='DEBUG' SLACK_ENGINE_SETTINGS_HOOK_ENABLED=False SLACK_ENGINE_SETTINGS_HOOK_URL='' TRACING_ENABLED=False TRANSLATIONS_DIR=PosixPath('/home/vagrant/orchestrator-demo/orchestrator-demo/company/translations') WEBSOCKET_BROADCASTER_URL='redis://<IP_ADDRESS>:6379' ENABLE_WEBSOCKETS=True DISABLE_INSYNC_CHECK=False
2022-05-19 19:17.42 [warning  ] WebSocketManager object configured, all methods referencing `websocket_manager` should work.
2022-05-19 19:17.42 [warning  ] DistLockManager object configured, all methods referencing `distlock_manager` should work.
2022-05-19 19:17:42 [warning  ] Database object configured, all methods referencing `db` should work. [orchestrator.db]
2022-05-19 19:17:42 [info     ] Version Locations              [orchestrator.cli.database] locations=/home/vagrant/orchestrator-demo/orchestrator-demo/migrations/versions/general /home/vagrant/orchestrator-demo/orchestrator-demo/.venv/lib/python3.10/site-packages/orchestrator/migrations/versions/schema
2022-05-19 19:17:42 [info     ] Context impl PostgresqlImpl.   [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Will assume transactional DDL. [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade  -> c112305b07d3, Initial schema migration. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] new branch insert c112305b07d3 [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade c112305b07d3 -> a76b9185b334, Add generic workflows to core. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update c112305b07d3 to a76b9185b334 [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade a76b9185b334 -> 3323bcb934e7, Fix tsv triggers. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update a76b9185b334 to 3323bcb934e7 [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade 3323bcb934e7 -> 3c8b9185c221, Add task_validate_products. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update 3323bcb934e7 to 3c8b9185c221 [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade 3c8b9185c221 -> 6896a54e9483, Add product_block_relations. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update 3c8b9185c221 to 6896a54e9483 [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade 6896a54e9483 -> 19cdd3ab86f6, fix_parse_websearch. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update 6896a54e9483 to 19cdd3ab86f6 [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade 19cdd3ab86f6 -> bed6bc0b197a, rename_parent_and_child_block_relations. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update 19cdd3ab86f6 to bed6bc0b197a [alembic.runtime.migration]
2022-05-19 19:17:42 [info     ] Running upgrade bed6bc0b197a -> 022505714cf8, Add user preference. [alembic.runtime.migration]
2022-05-19 19:17:42 [debug    ] update bed6bc0b197a to 022505714cf8 [alembic.runtime.migration]

New call to products:
(.venv) vagrant@ubuntu-jammy:~/orchestrator-demo/orchestrator-demo$ curl -L http://localhost:8080/api/products/
[]

Now it is indeed able to query the service w/o server errors.
Update 2: also realised that my last dt was carried out on an unknown database. If forcing the change to the orchestrator-demo DB:
postgres=# \c orchestrator-demo
You are now connected to database "orchestrator-demo" as user "postgres".
orchestrator-demo=# \dt
                            List of relations
 Schema |                Name                | Type  |       Owner
--------+------------------------------------+-------+-------------------
 public | alembic_version                    | table | orchestrator-demo
 public | engine_settings                    | table | orchestrator-demo
 public | fixed_inputs                       | table | orchestrator-demo
 public | process_steps                      | table | orchestrator-demo
 public | processes                          | table | orchestrator-demo
 public | processes_subscriptions            | table | orchestrator-demo
 public | product_block_relations            | table | orchestrator-demo
 public | product_block_resource_types       | table | orchestrator-demo
 public | product_blocks                     | table | orchestrator-demo
 public | product_product_blocks             | table | orchestrator-demo
 public | products                           | table | orchestrator-demo
 public | products_workflows                 | table | orchestrator-demo
 public | resource_types                     | table | orchestrator-demo
 public | subscription_customer_descriptions | table | orchestrator-demo
 public | subscription_instance_relations    | table | orchestrator-demo
 public | subscription_instance_values       | table | orchestrator-demo
 public | subscription_instances             | table | orchestrator-demo
 public | subscriptions                      | table | orchestrator-demo
 public | user_preference                    | table | orchestrator-demo
 public | workflows                          | table | orchestrator-demo
(20 rows)

As a final comment/question (not to be answered if the assumption is correct), I see the "user_preference" table has a match under "migrations/versions/general/2022-04-02_022505714cf8_add_user_preference.py" . I assume that migrations are then used either to create new custom tables or to insert entries in the already existing/default tables, like in this example).

Given that there are no longer server errors and the products and workflows show now an empty and a filled list, respectively; it is time to close the ticket and find out how to populate other structures. Thanks all who worked on this and replied here, and special thanks on the updated instructions and repo.

🎉
Migrations can indeed be used to create tables AND to insert/populate data.
We use them also to add a new workflow.
Reading the last comments I think it would be nice to provide a docker/vagrant based environment to make it easier; to play with the demo project in a "known environment". 10 million ways to run python.
There seem to be problems regarding setting up stuff to correctly run the initial migration...
Thanks for reporting back!

Thanks for the clarification on the migrations and the support before, @acidjunk !
There is much material to read and test when continuing with this analysis in the meantime.

Reading the last comments I think it would be nice to provide a docker/vagrant based environment to make it easier; to play with the demo project in a "known environment". 10 million ways to run python.

Definitely seconding this. I think that a Docker environment would be ideal, otherwise a Vagrant one. In any case, whatever allows replicability and provides a base for developers to (hopefully) also packet their extended logic. As these are both self-contained I foresee no help needed there, but otherwise drop me a line.


Update README.md

Coverage remained the same at 85.562% when pulling 5159a21c838179a575a02410ab4aff1b5451dcc3 on pavelmaliy-patch-1 into e250784c76503ef66a75778a2b7e22270a68d1e0 on master.


OrderBy string column name

Is there any easy way to pass a string column name to OrderBy or OrderByDescending, or provide an extension method to do so?
Expected:
Query.OrderBy("Name");
Would effectively be the same as calling:
Query.OrderBy(x => x.Name);
This would allow for passing in different column names into the specification and not having to modify, if/else, or switch on columns.

Hi @ovation22
That would be handy actually, but we're constrained by the functionalities that are provided from LINQ. There is no method that accepts string. All methods stricly accept keySelectors.
public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(this IEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer);
public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

In order this to be implemented through this package, we effectively have to find the property by the string that is given. The other way is easy, but this one is a tedious task which will incorporate reflection, generating the predicate and a lot of other details.

Are you OK with the answer, should we close the issue here?
Indeed, it would be useful, but if we want the functionality, it would be better if we make a PR to .NET, not here. They might elaborate even further and analyze all edge cases, and they might have other reasons why this was not implemented in the first place.

Some options here that might be helpful:
https://stackoverflow.com/questions/31955025/generate-ef-orderby-expression-by-string

Indeed, we find the property by reflection, and then dynamically generate the expression.
You want to include this as part of the package here? Or you meant as an extension that users can use on their own?

Thanks, I've got it working with a switch/Expression<Func for now. I'll explore the expression trees some more.
Thanks for the help!
Feel free to close.

I think users can create their own extensions. Thanks all!

Do we have any examples of how you implement an extension like this for ISpecificationBuilder<T>? Understand how it works for an IQueryable<T> but Query isn't an IQueryable<T> or am I missing something obvious?

Do we have any examples of how you implement an extension like this for ISpecificationBuilder<T>? Understand how it works for an IQueryable<T> but Query isn't an IQueryable<T> or am I missing something obvious?

Yea you're right, we have no examples of that. Before answering that, we should clarify few topics first.
Specification class is nothing else than just a bag of properties. It doesn't have any behavior by itself, the main purpose is to just hold some information. You may consider it as DTO. Then, during the evaluation, we utilize this saved data to construct the query for the ORM (in our case EF). So, in this context, the builder has no importance, we could simply set all those properties directly and populate the specification in that way.
In order to enforce some rules and to avoid wrong usages, we thought it's wise to create a mechanism for populating the specification in a more safe manner. And that's what the builder is. It's just a helper to correctly populate the specification, and it has no significance other than that. So, the specification is a data object which additionally provides a special mechanism how that data should be populated.
The other benefits of the builder, as implemented in our case

Collects/groups all available assignment operations under one single construct.
It has a fluent design, and it resembles LINQ. The users are quite familiar with the syntax.
Since we used the same LINQ method names, you can just copy/paste the expressions. This is valuable during the transition phase if you want to apply this pattern in an existing project. For example, let's assume this EF expression

var customers = await dbContext.Customers
                            .Where(x => x.IsActive)
                            .Include(x => x.Stores)
				.ThenInclude(x=>x.Addresses)
                            .OrderBy(x => x.DisplayName)
                            .Skip(20)
                            .Take(5)
                            .ToListAsync();

Now, you can literally copy the expressions and define them in the specification, and it will work as it is.
public CustomerSpec()
{
	Query.Where(x => x.IsActive)
		.Include(x => x.Stores)
                    .ThenInclude(x=>x.Addresses)
		.OrderBy(x => x.DisplayName)
		.Skip(20)
		.Take(5);
}

How to write extensions
For demonstration purposes, let's add support for AsSplitQuery feature (recently requested).
First of all, we need the base specification to be able to hold information whether this is enabled for a particular specification or not. So, inherit the Specification class and create your own base specification. Then, use the newly created one as a base for all the defined specifications in your project.
public class MyBaseSpecification<T> : Specification<T>
{
	public bool AsSplitQuery { get; set; }
}

Basically, that's all you need, you don't even have to modify the builder, just set this property directly if you wish.
public CustomerSpec()
{
	Query.Where(x => x.IsActive);

	AsSplitQuery = true;
}

Now, that you have this extra information, use it during the evaluation to add this extra expression in the EF query.
Probably, soon we can add step by step tutorial on how to write extensions, including how to extend the builder. We recently refactored the internal infrastructure, and all you need is to write an extension method to the builder interface.

@ardalis probably we might add this to the Readme as well.

@fiseni This is all really good information, however I’m still unclear how you would use this to pass an OrderBy via field name instead of Lambda. Steve’s answer was just a bit vague;

I think users can create their own extensions. Thanks all!

Regarding this issue, as I stated in this first reply it's not a quite straightforward implementation. The LINQ itself doesn't have support for ordering by string. And that's by a design and for a good reason. It's not a strongly typed definition, and you have no way how to get to the second level (ThenBy definitions).
Anyhow, here is the extension for that. You can use it in your projects but we don't see it as part of the package.
LINQ extensions
First of all, let's write extensions for LINQ
public static class LinqExtensions
{
    private static PropertyInfo GetPropertyInfo(Type objType, string name)
    {
        var properties = objType.GetProperties();
        var matchedProperty = properties.FirstOrDefault(p => p.Name == name);
        if (matchedProperty == null)
            throw new ArgumentException("name");

        return matchedProperty;
    }
    private static LambdaExpression GetOrderExpression(Type objType, PropertyInfo pi)
    {
        var paramExpr = Expression.Parameter(objType);
        var propAccess = Expression.PropertyOrField(paramExpr, pi.Name);
        var expr = Expression.Lambda(propAccess, paramExpr);
        return expr;
    }

    public static IQueryable<T> OrderBy<T>(this IQueryable<T> query, string name)
    {
        var propInfo = GetPropertyInfo(typeof(T), name);
        var expr = GetOrderExpression(typeof(T), propInfo);

        var method = typeof(Queryable).GetMethods().FirstOrDefault(m => m.Name == "OrderBy" && m.GetParameters().Length == 2);
        var genericMethod = method.MakeGenericMethod(typeof(T), propInfo.PropertyType);
        return (IQueryable<T>)genericMethod.Invoke(null, new object[] { query, expr });
    }
}

Specification extensions
Now that we have the appropriate LINQ method, lets create a new base specification class and write the extensions. Use this base class as a base for all your specifications in your project.
namespace Ardalis.Specification
{
    public class MySpecification<T> : Specification<T>
    {
        public string OrderByStringExpression { get; set; }
    }

    public static class MyExtensions
    {
        public static ISpecificationBuilder<T> OrderByColumnName<T>(
            this ISpecificationBuilder<T> specificationBuilder,
            string columnName)
        {
            ((MySpecification<T>)specificationBuilder.Specification).OrderByStringExpression = columnName;

            return specificationBuilder;
        }
    }
}

namespace Ardalis.Specification.EntityFrameworkCore
{
    public class MySpecificationEvaluator<T> : SpecificationEvaluator<T> where T : class
    {
        public IQueryable<T> GetQuery(IQueryable<T> inputQuery, MySpecification<T> specification)
        {
            var query = base.GetQuery(inputQuery, specification);

            if (!string.IsNullOrEmpty(specification.OrderByStringExpression))
            {
                query = query.OrderBy(specification.OrderByStringExpression);
            }

            return query;
        }
    }
}

Note: Ordering after pagination may lead to wrong results, so instead of inheriting from the base evaluator, you may want to fully implement your own evaluator with the appropriate evaluation order. Here is the evaluator. Copy the content and add the above logic accordingly.

@fiseni This is great thank you very much, wasn't expecting such a detailed response but it's greatly appreciated and I'm sure it will help others. Would even go as far as suggesting it might be worth adding to the documentation in relation to extending Specification perhaps?

Sure, soon we'll improve the documentation and add information regarding these tasks.

@fiseni  Just looking at this and the evaluation order is going to be important for pagination, would it not make more sense if implementing your own evaluator instead of inheriting the base evaluator to convert the orderby string into the existing OrderExpressions property instead of adding a new one? That way we don't have to adjust the existing evaluation order or mess with the base class.

Well, yea you can do it, but in that case, instead of building upon the IQueryable, you should construct Expression<Func<T, object>>. All you need in that case is just writing an extension to ISpecificationBuilder<T>.
Here you go, this is all you need. But you should test this thoroughly (I didn't test it).
public static class MyExtensions
{
    public static IOrderedSpecificationBuilder<T> OrderByColumnName<T>(
        this ISpecificationBuilder<T> specificationBuilder,
        string columnName)
    {
        var matchedProperty = typeof(T).GetProperties().FirstOrDefault(p => p.Name == columnName);
        if (matchedProperty == null)
            throw new ArgumentException("name");

        var paramExpr = Expression.Parameter(typeof(T));
        var propAccess = Expression.PropertyOrField(paramExpr, matchedProperty.Name);
        var expr = Expression.Lambda<Func<T, object?>>(propAccess, paramExpr);

        ((List<(Expression<Func<T, object?>> OrderExpression, OrderTypeEnum OrderType)>)specificationBuilder.Specification.OrderExpressions)
            .Add((expr, OrderTypeEnum.OrderBy));

        var orderedSpecificationBuilder = new OrderedSpecificationBuilder<T>(specificationBuilder.Specification);

        return orderedSpecificationBuilder;
    }
}

@fiseni This is exactly the approach I'm taking, the only difference being I'm allowing a comma separated list of column names something like "field1,-field2" etc *(the - denoting descending OrderTypeEnum to be applied. This is very helpful thank you.

@fiseni  Just wanted to say have implemented this and I'm testing it thoroughly, works really well. Thanks again for pointing me in the right direction.

Great, I'm glad it worked out!
Cheers!

Code from Fiseni is helpfull (Thanks!) but when I try order by decimal column I get excepiotn "Expression of type 'System.Decimal' cannot be used for return type 'System.Object'". I was able to fix this code and I also optimized using reflection in loop.
public static class SpecificationBuilderExtensions
    {
        public static IOrderedSpecificationBuilder<T> OrderBy<T>(this ISpecificationBuilder<T> specificationBuilder, string orderByFields)
        {
            var fields = ParseOrderBy(orderByFields);
            if (fields != null)
            {
                foreach (var field in fields)
                {
                    //var matchedProperty = typeof(T).GetProperties().FirstOrDefault(p => p.Name.ToLower() == field.Key.ToLower());
                    var matchedProperty = typeof(T).GetProperty(field.Key, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
                    if (matchedProperty == null)
                        throw new ArgumentNullException(field.Key);

                    //Expression conversion = Expression.Convert(expression, typeof(object));
                    //func = Expression.Lambda<Func<T, Object>>(conversion, parameterExpression).Compile();

                    var paramExpr = Expression.Parameter(typeof(T));
                    var propAccess = Expression.PropertyOrField(paramExpr, matchedProperty.Name);
                    var propAccess2 = Expression.Convert(propAccess, typeof(object));
                    var expr = Expression.Lambda<Func<T, object>>(propAccess2, paramExpr);
                    ((List<(Expression<Func<T, object>> OrderExpression, OrderTypeEnum OrderType)>)specificationBuilder.Specification.OrderExpressions).Add((expr, field.Value));
                }
            }
            var orderedSpecificationBuilder = new OrderedSpecificationBuilder<T>(specificationBuilder.Specification);

            return orderedSpecificationBuilder;
        }

        private static IDictionary<string, OrderTypeEnum> ParseOrderBy(string orderByFields)
        {
            //Write your own implementation
        }
}

@ardalis @fiseni
Could we add this to the package?

I think it would also help to accept an Ascending/Descending enum in there as well so we can one line these based on ordering.

@lkroliko How did you fix the System.ArgumentException: Expression of type 'System.Int32' cannot be used for return type 'System.Object' error?

You need the conversion part to be able to use int (also for decimal, guid, etc.)
If you're trying to order by only one field, then this might do the trick for you. You can pass a newly created enum in your app, or just "asc", "desc" string parameters. On the other hand, if you wanna have a list of fields as parameter, then try one of the above answers.
public enum OrderByEnum
{
    Ascending = 1,
    Descending = 2
}

public static class MyExtensions
{
    public static IOrderedSpecificationBuilder<T> OrderBy<T>(
        this ISpecificationBuilder<T> specificationBuilder,
        string fieldName,
        string orderBy)
    {
        if (orderBy.Equals("desc", StringComparison.InvariantCultureIgnoreCase))
        {
            return OrderBy(specificationBuilder, fieldName, OrderByEnum.Descending);
        }
        else
        {
            return OrderBy(specificationBuilder, fieldName, OrderByEnum.Ascending);
        }
    }

    public static IOrderedSpecificationBuilder<T> OrderBy<T>(
        this ISpecificationBuilder<T> specificationBuilder,
        string fieldName,
        OrderByEnum orderByEnum)
    {
        var matchedProperty = typeof(T).GetProperty(fieldName, BindingFlags.IgnoreCase | BindingFlags.Public | BindingFlags.Instance);
        if (matchedProperty == null)
        {
            //Actually you can decide no to throw here. Ordering is not a crucial task.
            throw new ArgumentException("name");
        }

        var paramExpr = Expression.Parameter(typeof(T));
        var propAccess = Expression.PropertyOrField(paramExpr, matchedProperty.Name);
        var propAccessConverted = Expression.Convert(propAccess, typeof(object));
        var expr = Expression.Lambda<Func<T, object?>>(propAccessConverted, paramExpr);

        if (orderByEnum == OrderByEnum.Descending)
        {
            ((List<(Expression<Func<T, object?>> OrderExpression, OrderTypeEnum OrderType)>)specificationBuilder.Specification.OrderExpressions)
                .Add((expr, OrderTypeEnum.OrderByDescending));
        }
        else
        {
            // Ascending would be default.
            ((List<(Expression<Func<T, object?>> OrderExpression, OrderTypeEnum OrderType)>)specificationBuilder.Specification.OrderExpressions)
                .Add((expr, OrderTypeEnum.OrderBy));
        }

        var orderedSpecificationBuilder = new OrderedSpecificationBuilder<T>(specificationBuilder.Specification);

        return orderedSpecificationBuilder;
    }
}

You are a godsend @fiseni thank you!

@fiseni how would you implement a lambda instead of a fieldname in this extension?
OrderBy(x => x.Address.AddressLine1, OrderByEnum.ASC);

Hmm, the lambda is supported out of the box. You just use Query.OrderBy or Query.OrderByDescending.
Did I misunderstand your question? The whole point with the fieldname is that you don't know the field beforehand.

I was meaning just so I can write it with the OrderByEnum as the second parameter, so I didn't have to write 2 queries based on the ordering so I could write it like this:
OrderBy(x => x.Address.AddressLine1, OrderByEnum.Ascending);
So I just need to create another extension that passes the OrderByEnum down and orders ascending or descending?

Would this work?
public static IOrderedSpecificationBuilder<T> OrderBy<T>(
            this ISpecificationBuilder<T> specificationBuilder,
            Expression<Func<T, object?>> orderExpression,
            OrderByEnum orderBy)
{
  if (orderBy == OrderByEnum.Ascending)
    OrderBy<T>(specificationBuilder, orderExpression);
  else
    OrderByDescending<T>(specificationBuilder, orderExpression);
}

Oh ok. Btw, I have used the newly created enum OrderByEnum just as an example. You don't need that one and you can use the one you have SortOrder. We have decided to go with the OrderBy, OrderByDescending, ThenBy, ThenByDescending approach, since that's the official LINQ syntax. In that way, users can just copy-paste the code from their existing EF queries and create specifications. But, surely having additional extensions is not a deal breaker.
public enum OrderByEnum
{
    Ascending = 1,
    Descending = 2
}

public static class MyExtensions
{
    public static IOrderedSpecificationBuilder<T> OrderBy<T>(
        this ISpecificationBuilder<T> specificationBuilder,
        Expression<Func<T, object?>> orderExpression,
        OrderByEnum orderBy)
    {
        if (orderBy == OrderByEnum.Descending)
        {
            ((List<(Expression<Func<T, object?>> OrderExpression, OrderTypeEnum OrderType)>)specificationBuilder.Specification.OrderExpressions)
                .Add((orderExpression, OrderTypeEnum.OrderByDescending));
        }
        else
        {
            // Ascending is default.
            ((List<(Expression<Func<T, object?>> OrderExpression, OrderTypeEnum OrderType)>)specificationBuilder.Specification.OrderExpressions)
                .Add((orderExpression, OrderTypeEnum.OrderBy));
        }

        var orderedSpecificationBuilder = new OrderedSpecificationBuilder<T>(specificationBuilder.Specification);

        return orderedSpecificationBuilder;
    }
}

Awesome. Thank you!

@davidhenley You are kind of re-hashing what myself and @fiseni already did, but each to their own.

@lankymart I know I am really just learning and seeing what's available. Did you implement a lambda version that takes in a sort order enum as well?

@davidhenley Didn't see the point as that's what the Query.OrderBy() and Query.OrderByDescending() do by default. It was more the dynamic element of the order by that was a concern and ISpecificationBuilder extensions solved that.

@lankymart thanks! Yeah I can see the use case for several different ways. I'm still learning expressions and your implementation has helped a lot!

I get the following error on the line and I could sure use some insight...
        ((List<(Expression<Func<T, object?>> OrderExpression, OrderTypeEnum OrderType)>)
                specificationBuilder.Specification.OrderExpressions)
            .Add((expr, OrderTypeEnum.OrderBy));

System.InvalidCastException: Unable to cast object of type 'System.Collections.Generic.List1[Ardalis.Specification.OrderExpressionInfo1[Overcast.Core.ProjectAggregates.OvercastUser.OvercastUser]]' to type 'System.Collections.Generic.List1[System.ValueTuple2[System.Linq.Expressions.Expression1[System.Func2[Overcast.Core.ProjectAggregates.OvercastUser.OvercastUser,System.Object]],Ardalis.Specification.OrderTypeEnum]]'.
at Overcast.Core.ProjectAggregates.Extensions.SpecificationOrderByColumnNameExtension.OrderBy[T](ISpecificationBuilder`1 specificationBuilder, String orderByFields)

I'm getting the same @mjanofsky
@fiseni Any idea why version 6 broke this?

Hi @mjanofsky @davidhenley,
In version 6, we have a breaking change regarding the specification state (we have published this info in the release notes. It is on the Readme page as well).
The state for Order, Where, Search is no longer kept as a list of tuples, but there are separate specific types. This is also visible in the ISpecification contract. So, your code now would look something like:
public static IOrderedSpecificationBuilder<T> OrderBy<T>(
this ISpecificationBuilder<T> specificationBuilder,
Expression<Func<T, object?>> orderExpression,
SortOrder orderBy)
{
    ((List<OrderExpressionInfo<T>>)specificationBuilder.Specification.OrderExpressions)
        .Add(orderBy == SortOrder.DESC
            ? new OrderExpressionInfo<T>(orderExpression, OrderTypeEnum.OrderByDescending)
            : new OrderExpressionInfo<T>(orderExpression, OrderTypeEnum.OrderBy));

    var orderedSpecificationBuilder = new OrderedSpecificationBuilder<T>(specificationBuilder.Specification);
    return orderedSpecificationBuilder;
}

Ah, I see! Makes sense. Thank you so much! Will close separate issue

Add
using Microsoft.EntityFrameworkCore;
in the code:
var sortOrder = "Name";
var descending = true;

if (descending)
     {
         Query.OrderByDescending(e => EF.Property<object>(e, sortOrder));
     }
     else
     {
         Query.OrderBy(e => EF.Property<object>(e, sortOrder));
     }

Add
using Microsoft.EntityFrameworkCore;
in the code:
var sortOrder = "Name";
var descending = true;

if (descending)
     {
         Query.OrderByDescending(e => EF.Property<object>(e, sortOrder));
     }
     else
     {
         Query.OrderBy(e => EF.Property<object>(e, sortOrder));
     }

public static class IQueryableExtensions
{
    public static void ApplyOrder<T>(ISpecificationBuilder<T> query, string propertyName, bool ascendingOrder)
    {
        if (ascendingOrder)
            query.OrderBy(T => EF.Property<object>(T!, propertyName));
        else
            query.OrderByDescending(T => EF.Property<object>(T!, propertyName));
    }
}


[PL-47565] PL SMP 0.14.0 Release Notes

PL-47565
0.14.0 - https://harness.atlassian.net/wiki/spaces/PD/pages/21654175763/Release+Notes+tickets+for+0.14.0+release
What Type of PR is This?

[ ] Issue
[ ] Feature
[ ] Maintenance/Chore

If tied to an Issue, list the Issue(s) here:

Issue(s)

House Keeping
Some items to keep track of. Screen shots of changes are optional but would help the maintainers review quicker.

[x] Tested Locally
[ ] Optional Screenshot.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65ce25b7438afd1e957be51a--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65ce2f8b81086c009fb0caad--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d470ee50dc11af6cf5805e--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d47aea4fa67db7cef566e0--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d4c999472a40f4e10aec23--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d4da5ddede2d037b27ac14--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d5bdf656c7db785b4722e4--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d5db4a04d0dc8ac4bf1bf7--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d5dbdab1e8ef8f5a8093ff--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65e0a53f4ee1031a03cf4004--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65e0a8c4c7f5621e77ba3ed3--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65e1acb79ba445b6995229f0--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65e1acf1a881b5b6419534ff--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65e5bf6a578de018a5e12d93--harness-developer.netlify.app

@brian-f-harness @pratmit @nataraja2018 @gn-harness
I have updated all the versions with respect to the newly released 0.14.1, which was previously with respect to 0.14.0
Can you please review it and approve the changes? I believe SMP 0.14.1 has already been released. Thanks

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65e60aeca4630a1590c239dc--harness-developer.netlify.app


Some work related to the Test Case's full external id and exposed filter options.

Some work related to the Test Case's full external id and exposed filter options.

Added support for the full external ID in 'getTestCase' and implemented the new 'updateTestCase' service.


Feature Request: Upload / Create Folder button

Hi @joni2back, while you design is awesome (after i spent several hours looking something useful for remote file management over web), i'm personally not really like when buttons upload/Make folder are hidden far right side under drop-down menu.
I think for High resolutions screens it nicer to have it much closer to area where usually you work with files, while for mobile/tablet view most probably way how those button located now is the best.
It is possible to place buttons other way for High-res screens while keep current style for low-res screens ?

Hi @md2k yes, I will take into account this suggestion for next release!!

Thanks.
Mean while, i did required changes to Go Provider and now it able to work with version 1.5 of your interface and higher, not all API functions a ready yet, but most of them fixed version
I did pull request to original owner of that project, but no idea if he still interested in support of that provider, so if any one want, they can use it from my repository.

@joni2back one more request/suggestion is to add api and its button representation to interface for LogOut if backend supports authorization , while login page can be done separately from main interface, logout button should be injected to interface , and i think it will be nicer if you will do it similar to all your features which can be enabled or disabled if needed.
Thanks in advance :)

Here is Ready for Use compiled version for Linux: https://github.com/md2k/gofe/releases/tag/0.1


Fix notation for sequences, etc.

When I was working on my project, I wasn't sure about how to do this either lol - I would recommend looking at this resource from Dr. Smith's undergrad course he used to teach and follow this format.
For example, in the Temperature ODEs Module, I would represent the following as $\textit{tuple of }(T1 : \mathbb{R}, T2 : \mathbb{R})$ - there will only be two elements here, so it makes more sense to enforce this.

Thanks for this resource. It's helpful.


Engage with communities and promote PortalJS on Stack Overflow

Description
This issue revolves around improving our presence on Stack Overflow, promoting PortalJS as a solution in relevant discussions (incl data visualization, geospatial, etc.), and fostering a more interactive community around PortalJS.
Our goal is to seek out new and existing threads that relate to topics in which PortalJS could be promoted as a viable solution. By proactively participating in these discussions, we can help solve problems while showcasing the strengths of PortalJS.

(please see Action Plan: PortalJS on Stack Overflow

Acceptance

[ ] Contribute to at least 7 relevant Stack Overflow discussions that relate to PortalJS by July 31st
[ ] Establish a consistent, weekly process for monitoring new discussions related to PortalJS on Stack Overflow by July 31st.
[ ] Generate and share a list of potential PortalJS-related questions among at least 5 Datopians, and ensure they ask these questions on Stack Overflow by July 31st.

### Tasks
- [x] Create an action plan **done: [Action Plan: PortalJS on Stack Overflow](https://docs.google.com/document/d/1HBXnvAKCoohWv1J-kCh0foaHZgI7CWpsYIz19dHufVM/edit?usp=sharing)**
- [ ] Search Stack Overflow for new and existing discussions related to PortalJS or similar topics
- [ ] Categorize and prioritize these discussions based on relevance and the level of engagement they've already received
- [ ] Draft responses for these discussions, ensuring that they are clear, comprehensive, and helpful
- [ ] Participate in these discussions by posting our drafted responses + engage further as necessary based on the response
- [ ] Create a list of potential questions that users might have related to PortalJS
  - [x] started in the [Action Plan: PortalJS on Stack Overflow](https://docs.google.com/document/d/1HBXnvAKCoohWv1J-kCh0foaHZgI7CWpsYIz19dHufVM/edit?usp=sharing)
  - [ ] refine the questions, pick several
- [ ] (just an idea) Share this list with Datopians, encouraging them to ask these questions on Stack Overflow
- [ ] Establish a process to regularly review Stack Overflow (e.g. weekly) to identify any new discussions related to PortalJS or similar topics

@anuveyatsu we agreed with @demenech and @luccasmmg that they will do initial search, feel free to give advice or provide input if the action plan makes sense (i know you have experience with Stack Overflow)

DEP
moved to monorepo https://github.com/datopian/portals-monorepo/issues/397


Upgrade upstream to 3.5 beta2

Coverage remained the same at 94.488% when pulling 1941f4be8ef056b58f98c88b7a93ec8e1f87aff5 on 3_5_beta2 into f8cd532b1e0b8525be8885f55da56aee6ca2e546 on master.

Coverage remained the same at 94.488% when pulling 1941f4be8ef056b58f98c88b7a93ec8e1f87aff5 on 3_5_beta2 into f8cd532b1e0b8525be8885f55da56aee6ca2e546 on master.

Coverage remained the same at 94.488% when pulling 1941f4be8ef056b58f98c88b7a93ec8e1f87aff5 on 3_5_beta2 into f8cd532b1e0b8525be8885f55da56aee6ca2e546 on master.

Coverage remained the same at 94.488% when pulling 1941f4be8ef056b58f98c88b7a93ec8e1f87aff5 on 3_5_beta2 into f8cd532b1e0b8525be8885f55da56aee6ca2e546 on master.

Coverage remained the same at 94.488% when pulling 1941f4be8ef056b58f98c88b7a93ec8e1f87aff5 on 3_5_beta2 into f8cd532b1e0b8525be8885f55da56aee6ca2e546 on master.

Coverage remained the same at 94.488% when pulling 1941f4be8ef056b58f98c88b7a93ec8e1f87aff5 on 3_5_beta2 into f8cd532b1e0b8525be8885f55da56aee6ca2e546 on master.


Test invalid JSON at POST method

In gitlab by @sylvaingaudan on Aug 8, 2017, 10:22
Send invalid JSON to a REST POST method and check the returned status.

In gitlab by @sylvaingaudan on Aug 10, 2017, 08:29
changed milestone to %10

In gitlab by @sfalquier on Aug 18, 2017, 09:11
added ~2424198 label

In gitlab by @sfalquier on Aug 25, 2017, 10:35
added ~2424199 and removed ~2424198 labels

In gitlab by @sfalquier on Aug 25, 2017, 10:53
mentioned in commit 95007fa36a0f0153cafbddee0e97b96b9b1372ca

In gitlab by @sfalquier on Aug 25, 2017, 10:55
mentioned in merge request !85

In gitlab by @sfalquier on Aug 25, 2017, 10:57
assigned to @sfalquier

In gitlab by @MohamedHamouGisaia on Aug 25, 2017, 11:07
closed via merge request !85

In gitlab by @MohamedHamouGisaia on Aug 25, 2017, 11:07
closed via commit 2e824aac52f52659d5f4af9b4fd7f8da7f4fb79f

In gitlab by @MohamedHamouGisaia on Aug 25, 2017, 11:48
mentioned in commit 2e824aac52f52659d5f4af9b4fd7f8da7f4fb79f


[BUG]: The Baklava logo does not appear clearly in the dark theme.

Issue description
The Baklava logo in the README file does not appear clearly in dark mode
The text color may be light.
Media & Screenshots

Baklava Version
No response
Operating system
No response
Priority this issue should have
Low (slightly annoying)
Please review the checkboxes that are applicable.

[ ] This issue is directly related to React

this code solves the issue

  <picture>
    <source width="200" media="(prefers-color-scheme: dark)" srcset="https://github.com/Trendyol/baklava/assets/40598819/5a2d9304-8422-46f5-8d70-948e24100fb5">

  </picture>

The "BAKLAVA" of Dark mode may be  text color white. @buseselvi

I agree, here I added the dark mode bl-logo. You can use it 🙏

links not working :/ @buseselvi

I share the design file link, can you export from there? I couldn't share it as a PNG file, I don't know why 😔

Yes, I exported it from figma and opened a PR #790, thank you for your support.


Corrected wrong role for issuing TAP code

Authentication Policy Admin cannot issue a TAP code, it can only enable/manage the TAP method policy (it even says so in the linked role doc). Authentication Admin is the correct and lowest privileged role for issuing TAP for standard users.

@tilarso

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Should we get this checked and merged soon? The docs are wrong AFAIK ;)

@tilarso, @justinha
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.