⬆ Upgrade to the latest everything

right now, we are using some funky image with hacked code from AFS + 5 patches
roll forward to end of september release as the base for the image
test managed charging scenarios with SIL
see if we need any other patches (hopefully small)

@the-bay-kay decided to continue tracking the roll forward in #74 since she was going to make the simulator improvements after the roll forward. Since that is issue is super long now, I am using this to track my testing and cleanup (if needed) of the roll forward. ETA tomorrow.

Actually, the JsCarSimulator already supported the payment method, so we didn't really need the change.
However, trying to plug the car in stuck at "PrepareCharging"
Adding a bunch of additional logs to figure out what is going on, I first figured out that:

If we are not compiling, we need to edit build/dist/libexec/everest/modules/JsCarSimulator/index.js, not modules/simulation/JsCarSimulator/index.js
the iso_start_v2g_session is apparently not getting registered at all, because I have a syntax error in it, which is not getting triggered,

  if (mod.uses_list.ev.length > 0) registerCmd(mod, 'iso_start_v2g_session', 2, (mod, c) => {
    evlog.info('SHANKARI: Invoked v2g session start' + c.args[0] + c.args[1]);

although when I copy-pasted the offending log to the previous command
  registerCmd(mod, 'iso_wait_slac_matched', 0, (mod, c) => {
    evlog.info('SHANKARI: Invoked iso_wait_slac_matched with slac_state " + mod.slac_state);

I did get an error when starting
/ext/source/build/dist/libexec/everest/modules/JsCarSimulator/index.js:463
    evlog.info('SHANKARI: Invoked iso_wait_slac_matched with slac_state " + mod.slac_state);
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

SyntaxError: Invalid or unexpected token
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1178:20)
    at Module._compile (node:internal/modules/cjs/loader:1220:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
    at node:internal/main/run_main_module:23:47

Node.js v18.18.2

As we can see from the code, this must be because mod.uses_list.ev.length  is zero. I also notice that, in a successful session from march, https://github.com/EVerest/everest-demo/issues/35#issuecomment-2027675314
I should expect to see HLC enabled, and I am not. Let's check the config that I'm trying to copy over with the old config...
2024-03-29 19:48:28.327823 [INFO] evse_manager_1:  :: Max AC hardware capabilities: 32A/3ph
2024-03-29 19:48:28.419715 [INFO] evse_manager_1:  :: AC HLC mode enabled.

2024-03-29 19:49:28.197119 [INFO] car_simulator_1  :: {
  cmd: 'iso_wait_slac_matched',
  args: [],
  exec: [Function (anonymous)]
}

2024-03-29 19:49:31.201259 [INFO] car_simulator_1  :: {
  cmd: 'iso_start_v2g_session',
  args: [ 'contract', 'ac_three_phase_core' ],
  exec: [Function (anonymous)]
}

Argh, no!     evlog.info('SHANKARI: Invoked v2g session start' + c.args[0] + c.args[1]); is in fact valid.
2024-11-11 07:31:33.293053 [INFO] car_simulator_1  :: SHANKARI: mod.uses_list.ev = [{"call":{}}] of length 1
2024-11-11 07:31:33.293078 [INFO] car_simulator_1  :: SHANKARI_REGISTER: Found more than one ev, registering... 1
2024-11-11 07:31:33.293102 [INFO] car_simulator_1  :: SHANKARI: Finished registering start_v2g_session

Ok, so there were actually a couple of errors:

I was launching /ext/source/build/run-scripts/run-sil-ocpp201.sh instead of /ext/source/build/run-scripts/run-sil-ocpp201-pnc.sh
the non-PnC version has a config with hlc turned off
because there was no hlc, we didn't try to start the ISO session!

The challenge here, of course, was that there was no error reported here. Something around "Looking for HLC, not found", would be helpful!

SP1 fails because, although we are using an unencrypted connection between the station and the CSMS, we are still trying to use an encrypted connection between the station and the car, which apparently fails. Does SP1 imply TLS between station and car? It seems like that is a separate flag.
If we are in SP1, we don't copy certs around, but that should be OK because the self-signed certs are used by both the car and manager modules in EVerest. So I am still not sure why this fails.
2024-11-11 08:03:04.244681 [DEBG] iso15118_car    Everest::Everest::provide_cmd(std::string, std::string, JsonCommand)::<lambda(Everest::json)> :: Incoming iso15118_car:PyEvJosev->ev:ISO15118_ev->start_charging(EnergyTransferMode,PaymentOption) for <handler>
2024-11-11 08:03:04.245015 [DEBG] iso15118_car    Everest::Everest::provide_cmd(std::string, std::string, JsonCommand)::<lambda(Everest::json)> :: RETVAL: true
2024-11-11 08:03:04.245374 [DEBG] iso15118_car    void Everest::MQTTAbstractionImpl::publish(const std::string&, const std::string&, QOS) :: publishing to everest/iso15118_car/ev/cmd
2024-11-11 08:03:04.246043 [DEBG] iso15118_car    void Everest::MQTTAbstractionImpl::on_mqtt_message(std::shared_ptr<Everest::Message>) :: topic everest/iso15118_car/ev/cmd starts with everest/
2024-11-11 08:03:04.285009 [INFO] car_simulator_1  :: { cmd: 'iso_wait_pwr_ready', args: [], exec: [Function (anonymous)] }
2024-11-11 08:03:07.049808 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: EXI Codec version: 1.55
2024-11-11 08:03:07.050050 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Starting 15118 version: 0.25.2
2024-11-11 08:03:07.050289 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Communication session handler started
2024-11-11 08:03:07.051540 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Remaining SDP requests: 50
2024-11-11 08:03:07.153425 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Sending SDPRequest: [Security: TLS, Protocol: TCP]

2024-11-11 08:03:07.153724 [INFO] iso15118_charge  :: Received packet from [fe80::42:c0ff:fea8:6003]:34929 with security 0x00 and protocol 0x00

2024-11-11 08:03:07.153786 [INFO] iso15118_charge  :: SDP requested TLS, announcing TLS

2024-11-11 08:03:07.153929 [INFO] iso15118_charge  :: sendto([fe80::42:c0ff:fea8:6003]:34929) succeeded

2024-11-11 08:03:07.154006 [INFO] iso15118_charge  :: Received packet from [fe80::42:c0ff:fea8:6003]:34929 with security 0x00 and protocol 0x00

2024-11-11 08:03:07.154077 [INFO] iso15118_charge  :: SDP requested TLS, announcing TLS

2024-11-11 08:03:07.154114 [INFO] iso15118_charge  :: sendto([fe80::42:c0ff:fea8:6003]:34929) succeeded
2024-11-11 08:03:07.154202 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Received datagram from UDP server at address ('fe80::42:c0ff:fea8:6003', 15118, 0, 165)
2024-11-11 08:03:07.154484 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Received datagram from UDP server at address ('fe80::42:c0ff:fea8:6003', 15118, 0, 165)
2024-11-11 08:03:07.154779 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: SDPResponse received: [ IP address: fe80::42:c0ff:fea8:6003, Port: 64109 , Security: TLS , Transport: TCP ]
2024-11-11 08:03:07.154913 [DEBG] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: Starting TLS client, trying to connect to fe80::42:c0ff:fea8:6003 at port 64109 ...

2024-11-11 08:03:07.158575 [INFO] iso15118_charge  :: Incoming connection on eth0 from [a00:dbba:0:0:fe80::]:56250

2024-11-11 08:03:07.158843 [INFO] iso15118_charge  :: Started new TLS connection thread
2024-11-11 08:03:07.161137 [INFO] evse_security:E  :: Requesting certificate file: [V2G] file:"/ext/source/build/dist/etc/everest/certs/ca/v2g/V2G_ROOT_CA.pem"
2024-11-11 08:03:07.205291 [INFO] evse_security:E  :: Requesting key/pair: V2G
2024-11-11 08:03:07.212482 [INFO] evse_security:E  :: TPM Key: false
2024-11-11 08:03:07.214356 [WARN] evse_security:E evse_security::GetKeyPairResult evse_security::EvseSecurity::get_key_pair_internal(evse_security::LeafCertificateType, evse_security::EncodingFormat) :: Could not find private key for the selected certificate: (Could not find private key for given certificate: /ext/source/build/dist/etc/everest/certs/client/cso/CPO_CERT_CHAIN.pem key path: /ext/source/build/dist/etc/everest/certs/client/cso)

2024-11-11 08:03:07.246226 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: Failed to read key/pair!

2024-11-11 08:03:07.246330 [INFO] iso15118_charge  :: Closing TLS connection thread

2024-11-11 08:03:07.246397 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: Last TLS error was: -0x0001 - ERROR - Generic error
2024-11-11 08:03:07.248693 [ERRO] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: ConnectionResetError when trying to connect to host fe80::42:c0ff:fea8:6003 and port 64109
Traceback (most recent call last):
  File "/ext/source/build/dist/libexec/everest/modules/PyEvJosev/../../3rd_party/josev/iso15118/evcc/comm_session_handler.py", line 420, in start_comm_session
    self.tcp_client = await TCPClient.create(
  File "/ext/source/build/dist/libexec/everest/modules/PyEvJosev/../../3rd_party/josev/iso15118/evcc/transport/tcp_client.py", line 62, in create
    raise exc
  File "/ext/source/build/dist/libexec/everest/modules/PyEvJosev/../../3rd_party/josev/iso15118/evcc/transport/tcp_client.py", line 53, in create
    self.reader, self.writer = await asyncio.open_connection(
  File "/usr/lib/python3.10/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1103, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.10/asyncio/base_events.py", line 1133, in _create_connection_transport
    await waiter
ConnectionResetError

Turning off TLS in the config
--- a/manager/config-sil-ocpp201-pnc.yaml
+++ b/manager/config-sil-ocpp201-pnc.yaml
@@ -14,7 +14,7 @@ active_modules:
     config_module:
       device: auto
       supported_ISO15118_2: true
-      tls_active: true
+      tls_active: false
       is_cert_install_needed: false
   evse_manager_1:
     module: EvseManager

We can SP1 to work. I believe this is the current state after rolling forward as well.
https://github.com/EVerest/everest-demo/issues/74#issuecomment-2454812248
So I am not going to debug this now since there is a workaround, and we can just debug it in the Sept release.
Moving on to cleaning up the demo script a bit before rolling forward.

I checked the current demo script on the march release and it is broken for SP1 and SP2 (with ISO 15118-2) because of the cert/TLS issue. I am going to disable them temporarily while cleaning up the demo script, because there is no point in having a broken demo.
2024-11-11 15:57:06.007087 [WARN] evse_security:E evse_security::GetKeyPairResult evse_security::EvseSecurity::get_key_pair_internal(evse_security::LeafCertificateType, evse_security::EncodingFormat) :: Could not find private key for the selected certificate: (Could not find private key for given certificate: /ext/source/build/dist/etc/everest/certs/client/cso/CPO_CERT_CHAIN.pem key path: /ext/source/build/dist/etc/everest/certs/client/cso)

2024-11-11 15:57:06.042828 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: Failed to read key/pair!

2024-11-11 15:57:06.042886 [INFO] iso15118_charge  :: Closing TLS connection thread

2024-11-11 15:57:06.042905 [ERRO] iso15118_charge void dlog_func(dloglevel_t, const char*, int, const char*, const char*, ...) :: Last TLS error was: -0x0001 - ERROR - Generic error
2024-11-11 15:57:06.045178 [ERRO] iso15118_car    pybind11_init_everestpy(pybind11::module_&)::<lambda(const std::string&)> :: ConnectionResetError when trying to connect to host fe80::42:c0ff:fea8:c003 and port 64109

SP1 and SP2 do work without ISO, but it literally says ISO in the demo name. We should not be promising things we can't deliver.

Now that we have cleared the decks with https://github.com/EVerest/everest-demo/pull/82, we are ready to roll everything forward. I will base my changes and testing on https://github.com/the-bay-kay/everest-demo/tree/rollforward-demo
Onward and upward!

The final step is to upload the test script so that we can run it.
Running the automated test script fails because everest.framework
/usr/lib/python3.10/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
===================================================== test session starts =====================================================
platform linux -- Python 3.10.15, pytest-8.3.3, pluggy-1.5.0
rootdir: /ext/source/tests
configfile: pytest.ini
plugins: asyncio-0.24.0
asyncio: mode=strict, default_loop_scope=None
collected 0 items / 1 error

=========================================================== ERRORS ============================================================
________________________________________ ERROR collecting core_tests/startup_tests.py _________________________________________
ImportError while importing test module '/ext/source/tests/core_tests/startup_tests.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
core_tests/startup_tests.py:12: in <module>
    from everest.testing.core_utils.fixtures import *
/usr/lib/python3.10/site-packages/everest/testing/core_utils/fixtures.py:12: in <module>
    from ._configuration.everest_environment_setup import \
/usr/lib/python3.10/site-packages/everest/testing/core_utils/_configuration/everest_environment_setup.py:14: in <module>
    from everest.testing.core_utils.everest_core import EverestCore, Requirement
/usr/lib/python3.10/site-packages/everest/testing/core_utils/everest_core.py:19: in <module>
    from everest.framework import RuntimeSession
E   ModuleNotFoundError: No module named 'everest.framework'
=================================================== short test summary info ===================================================
ERROR core_tests/startup_tests.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================== 1 error in 0.18s =======================================================

I double checked the docs, and the instructions are the same as the run_test.sh
cd ${SOURCE_DIR}/tests
pytest --everest-prefix ${INSTALL_PREFIX} core_tests/*.py framework_tests/*.py

Same error.
Searched to see where the framework is so that we can put it into the PYTHONPATH - could not find it
33b7d4f75906:/ext/source# find . -name \*framework.py\*
33b7d4f75906:/ext/source# find /usr/lib/python3.10/ -name \*framework.py\*
/usr/lib/python3.10/site-packages/pygments/lexers/robotframework.py
/usr/lib/python3.10/ctypes/macholib/framework.py

I do notice that the framework is used in the pyjosev code
https://github.com/EVerest/everest-core/blob/5e967358322fabd34e66a34f2919588df3ebb33f/modules/PyEvJosev/utilities.py#L6
or
https://github.com/EVerest/everest-core/blob/5e967358322fabd34e66a34f2919588df3ebb33f/modules/PyEvJosev/module.py#L8
and pyjosev does run, so we should be able to figure out where it is running from. But this is not required for the interactive demo, so let's ask a question in the zulip chat and move on.

Tracking the automated test python dependency issue in https://lfenergy.zulipchat.com/#narrow/channel/417679-EVerest.3A-Testing-.26-CI/topic/.60everest.2Eframework.60.20not.20found.20when.20trying.20to.20run.20tests/near/482047129

For the record, the list of existing PYTHONPATH references is
./third-party/bazel/edm-wrapper.sh:export PYTHONPATH=$TARGET_PATH
./build/_deps/everest-framework-build/src/manager:PYTHONPATH
./build/_deps/everest-framework-build/src/CMakeFiles/manager.dir/manager.cpp.o:PYTHONPATH
 [ a bunch of files in the venv directory ]
./build/dist/bin/manager:PYTHONPATH

This makes me think that maybe the issue is that I have to activate a virtual environment, but it is not clear which one, and why it wasn't activated during build or while following the outlined instructions.

Returning to https://github.com/EVerest/everest-demo/issues/78#issuecomment-2471439936 and

dist/share/everest/modules/OCPP201/config.json: this is not created any more so does not need to be copied over. We do need to figure out what the changes were here and where they have moved to now.

Per @the-bay-kay and @catarial we should just edit the config directory, and that will be use to generate the device model when the manager is launched.
Verifying this by checking that the device model DB does not exist
# ls /ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db
ls: /ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db: No such file or directory

Starting up the manager requires an MQTT server, so we install and run one on the manager
# apk install mosquitto
# mosquitto &
[1] 12562
33b7d4f75906:/ext/source#<PHONE_NUMBER>: mosquitto version 2.0.15 starting
1731460630: Using default config.
1731460630: Starting in local only mode. Connections will only be possible from clients running on this machine.
1731460630: Create a configuration file which defines a listener to allow remote access.
1731460630: For more details see https://mosquitto.org/documentation/authentication-methods/
1731460630: Opening ipv4 listen socket on port 1883.
1731460630: Opening ipv6 listen socket on port 1883.
1731460630: mosquitto version 2.0.15 running

This now starts up everything
# /ext/source/build/run-scripts/run-sil-ocpp201.sh
...
2024-11-13 01:17:40.305161 [INFO] manager          ::   ________      __                _
2024-11-13 01:17:40.305253 [INFO] manager          ::  |  ____\ \    / /               | |
2024-11-13 01:17:40.305391 [INFO] manager          ::  | |__   \ \  / /__ _ __ ___  ___| |_
2024-11-13 01:17:40.305419 [INFO] manager          ::  |  __|   \ \/ / _ \ '__/ _ \/ __| __|
2024-11-13 01:17:40.306071 [INFO] manager          ::  | |____   \  /  __/ | |  __/\__ \ |_
2024-11-13 01:17:40.306278 [INFO] manager          ::  |______|   \/ \___|_|  \___||___/\__|
2024-11-13 01:17:40.306317 [INFO] manager          ::
2024-11-13 01:17:40.306389 [INFO] manager          :: everest-framework 0.17.2 @v0.17.2
2024-11-13 01:17:40.306421 [INFO] manager          :: everest-core 2024.9.0 @2024.9.0
2024-11-13 01:17:40.306507 [INFO] manager          ::
2024-11-13 01:17:40.306539 [INFO] manager          :: Using MQTT broker localhost:1883
2024-11-13 01:17:40.357532 [INFO] everest_ctrl     :: Launching controller service on port 8849
2024-11-13 01:17:40.405279 [INFO] manager          :: Loading config file at: /ext/source/config/config-sil-ocpp201.yaml
2024-11-13 01:17:41.074668 [INFO] manager          :: Config loading completed in 725ms
...
2024-11-13 01:17:45.411420 [INFO] error_history:E  :: Resetting database
1731460665: New connection from ::1:50510 on port 1883.
1731460665: New connection from ::1:50518 on port 1883.
...
2024-11-13 01:17:51.333637 [INFO] manager          :: 🚙🚙🚙 All modules are initialized. EVerest up and running [11033ms] 🚙🚙🚙

2024-11-13 01:17:51.352417 [INFO] iso15118_charge  :: TCP server on eth0 is listening on port [fe80::42:acff:fe11:2%122]:61341

2024-11-13 01:17:51.353179 [INFO] iso15118_charge  :: TLS server on eth0 is listening on port [fe80::42:acff:fe11:2%122]:64109

2024-11-13 01:17:51.496545 [INFO] ocpp:OCPP201     :: Established connection to database: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db"
2024-11-13 01:17:51.506374 [INFO] ocpp:OCPP201     :: Target version: 1, current version: 0

2024-11-13 01:17:51.511952 [INFO] iso15118_charge  :: SDP socket setup succeeded
2024-11-13 01:17:51.512929 [INFO] ocpp:OCPP201     :: Migration files to apply:
2024-11-13 01:17:51.513047 [INFO] ocpp:OCPP201     :: Migration file [up] version 1, path: /ext/source/build/dist/share/everest/modules/OCPP201/device_model_migrations/1_up-initial.sql
2024-11-13 01:17:51.527992 [INFO] ocpp:OCPP201     :: Successfully closed database: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db"
2024-11-13 01:17:51.528198 [INFO] ocpp:OCPP201     :: Established connection to database: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db"
2024-11-13 01:17:51.607005 [INFO] ocpp:OCPP201     :: Successfully closed database: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db"
2024-11-13 01:17:51.607144 [INFO] ocpp:OCPP201     :: Database file "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db" is already closed
2024-11-13 01:17:51.607354 [INFO] ocpp:OCPP201     :: Established connection to database: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db"
2024-11-13 01:17:51.607441 [INFO] ocpp:OCPP201     :: Established connection to device model database successfully: "/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db"

2024-11-13 01:17:51.629857 [INFO] ocpp:OCPP201     :: Successfully retrieved Device Model from DeviceModelStorage

And after killing the entry, we do see the database
# ls /ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db
/ext/source/build/dist/share/everest/modules/OCPP201/device_model_storage.db

So we can change the config (if required) in the demo scripts and create the database that way.

Error while building in the CI/CD
#0 building with "builder-dde454bd-d070-48d9-92a0-7b09810cb1df" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 849B done
#1 WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/x86_64" (line 1)
#1 DONE 0.0s

#2 [internal] load metadata for ghcr.io/everest/build-kit-alpine:1.4.2
#2 ERROR: ghcr.io/everest/build-kit-alpine:1.4.2: not found
------
 > [internal] load metadata for ghcr.io/everest/build-kit-alpine:1.4.2:
------

 1 warning found (use docker --debug to expand):
 - FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/x86_64" (line 1)
Dockerfile:1
--------------------
   1 | >>> FROM --platform=linux/x86_64 ghcr.io/everest/build-kit-alpine:1.4.2
   2 |
   3 |     ARG EVEREST_VERSION=2024.9.0
--------------------
ERROR: failed to solve: ghcr.io/everest/build-kit-alpine:1.4.2: failed to resolve source metadata for ghcr.io/everest/build-kit-alpine:1.4.2: ghcr.io/everest/build-kit-alpine:1.4.2: not found
Reference
Check build summary support
Error: buildx failed with: ERROR: failed to solve: ghcr.io/everest/build-kit-alpine:1.4.2: failed to resolve source metadata for ghcr.io/everest/build-kit-alpine:1.4.2: ghcr.io/everest/build-kit-alpine:1.4.2: not found
```

But the 1.4.2 version is clearly present
https://github.com/EVerest/everest-ci/pkgs/container/build-kit-alpine/297600528?tag=v1.4.2

I wonder if it is the `x86` versus `arm` that is the problem.

Bingo! <img width="888" alt="Screenshot 2024-11-12 at 5 45 53 PM" src="https://github.com/user-attachments/assets/d2648e4a-56be-498a-b315-2911427b549b">

Ok, so I can confirm that I get the same error while trying to build locally even when I am on an x86 mac
 > [manager internal] load metadata for ghcr.io/everest/build-kit-alpine:1.4.2:
------
failed to solve: ghcr.io/everest/build-kit-alpine:1.4.2: failed to resolve source metadata for ghcr.io/everest/build-kit-alpine:1.4.2: ghcr.io/everest/build-kit-alpine:1.4.2: not found

Removing the --platform=linux/x86_64 still fails, but removing the version number and using "latest" (=> [manager 1/4] FROM ghcr.io/everest/build-kit-alpine:latest@sha256:273c75b2986435daa5b49cd8eea736e0a75b2d0f03955a815a  0.2s) allows it to work.
But the SHA of the latest is the same as the SHA for 1.4.2

I guess it pulls the "unknown" image above.
Inspecting the image, we get
$ docker inspect e4a2478d4d54
    {
        "Id": "sha256:e4a2478d4d545b461ab2242bc9988e2ed539d67905196a11dbc348dd8c387a6b",
        "RepoTags": [
            "ghcr.io/everest/build-kit-alpine:v1.4.2"
        ],
        "RepoDigests": [
            "ghcr.io/everest/build-kit-alpine@sha256:273c75b2986435daa5b49cd8eea736e0a75b2d0f03955a815a6cd9ee9258a91d"
        ],
....
        "Architecture": "amd64",
        "Os": "linux",
        "Size":<PHONE_NUMBER>,

But I am still able to run it on my x86
$ docker run -it e4a2478d4d54 /bin/bash
---------------------------------------------
WARNING
This docker image is depreacted.
Please use the debian based build-kit 'ghcr.io/everest/everest-ci/build-kit-base' instead.
Fore more information see https://github.com/EVerest/everest-ci/pull/22
---------------------------------------------

d352e4885032:/workspace#

Ah, it looks like amd64 and x86_64 are compatible (jeez, it has been a really long time since I had to look at processor architectures and instruction sets!) - I now have to use wikipedia and don't know this stuff cold
https://en.wikipedia.org/wiki/X86-64 ( x64, x86_64, AMD64, and Intel 64)
So:

It looks like the architecture used to be called x86_64
was added in https://github.com/EVerest/everest-demo/commit/b0dfa4de9d86e143638c42c22e76b6539cbe705c to reduce the size of the image
then the architecture name changed to arm64 (but it is still compatible)

So we can just remove the image architecture. But this doesn't solve our arm problems (at least not yet).

build-kit-base has the same version https://github.com/orgs/EVerest/packages/container/everest-ci%2Fbuild-kit-base/297603441?tag=v1.4.2 and significantly more downloads (it looks like most people have switched over). Let's switch over as well now and fall back to the alpine version if we run into issues.

Fixed this, but then running out of disk space while building...
ERROR: failed to solve: rpc error: code = Unknown desc = write /ext/source/build/modules/CMakeFiles/YetiDriver.dir/__/generated/modules/YetiDriver/ld-ev.cpp.o: no space left on device
Reference
Check build summary support
Error: buildx failed with: ERROR: failed to solve: rpc error: code = Unknown desc = write /ext/source/build/modules/CMakeFiles/YetiDriver.dir/__/generated/modules/YetiDriver/ld-ev.cpp.o: no space left on device

This is going to be super complicated to fix since we can't really control the everest build or the space available on the github action workers, so I am going to just build locally and push.

Building locally,
145.7 DIR /ext/source/build/_deps/libwebsockets-build CMP /ext/cache/cpm/libwebsockets/3866e4bc96015c77bf61889acd9dd6f4642df6fe/libwebsockets/cmake
145.9 -- CPM: everest-framework: adding package<EMAIL_ADDRESS>(v3.11.2 to /ext/cache/cpm/nlohmann_json/b3708972f6694fe462e4112e47aa04f10d2390b4/nlohmann_json)
256.1 [ 11%] Creating directories for 'nlohmann_json-populate'
256.1 [ 22%] Performing download step (git clone) for 'nlohmann_json-populate'
256.1 Cloning into 'nlohmann_json'...
256.1 error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
256.1 error: 1073 bytes of body are still expected
256.1 fetch-pack: unexpected disconnect while reading sideband packet
256.1 fatal: early EOF
256.1 fatal: fetch-pack: invalid index-pack output
256.1 Cloning into 'nlohmann_json'...
256.1 error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
256.1 error: 1629 bytes of body are still expected
256.1 fetch-pack: unexpected disconnect while reading sideband packet
256.1 fatal: early EOF
256.1 fatal: fetch-pack: invalid index-pack output
256.1 Cloning into 'nlohmann_json'...
256.1 error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
256.1 error: 408 bytes of body are still expected
256.1 fetch-pack: unexpected disconnect while reading sideband packet
256.1 fatal: early EOF
256.1 fatal: fetch-pack: invalid index-pack output
256.1 -- Had to git clone more than once: 3 times.
256.1 CMake Error at /ext/source/build/_deps/nlohmann_json-subbuild/nlohmann_json-populate-prefix/tmp/nlohmann_json-populate-gitclone.cmake:39 (message):
256.1   Failed to clone repository: 'https://github.com/nlohmann/json'
256.1
256.1
256.1 make[2]: *** [CMakeFiles/nlohmann_json-populate.dir/build.make:102: nlohmann_json-populate-prefix/src/nlohmann_json-populate-stamp/nlohmann_json-populate-download] Error 1
256.1 make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/nlohmann_json-populate.dir/all] Error 2
256.1 make: *** [Makefile:91: all] Error 2
256.1
256.1 CMake Error at /usr/share/cmake/Modules/FetchContent.cmake:1604 (message):
256.1   Build step for nlohmann_json failed: 2
256.1 Call Stack (most recent call first):
256.1   /usr/share/cmake/Modules/FetchContent.cmake:1744:EVAL:2 (__FetchContent_directPopulate)
256.1   /usr/share/cmake/Modules/FetchContent.cmake:1744 (cmake_language)
256.1   /usr/lib/cmake/everest-cmake/3rd_party/CPM.cmake:885 (FetchContent_Populate)
256.1   /usr/lib/cmake/everest-cmake/3rd_party/CPM.cmake:685 (cpm_fetch_package)
256.1   build/_deps/everest-framework-build/dependencies.cmake:30 (CPMAddPackage)
256.1   /usr/lib/cmake/everest-cmake/edm.cmake:31 (include)
256.1   /ext/cache/cpm/everest-framework/d01be681d203f70289900147e87a8f4c43ff47f0/everest-framework/CMakeLists.txt:58 (evc_setup_edm)
256.1
256.1
256.1 -- Configuring incomplete, errors occurred!
256.1 See also "/ext/source/build/CMakeFiles/CMakeOutput.log".
256.1 See also "/ext/source/build/CMakeFiles/CMakeError.log".
------
failed to solve: process "/bin/sh -c git clone https://github.com/EVerest/everest-core.git         && cd everest-core         && git checkout ${EVEREST_VERSION}         && cd ..         && mkdir -p /ext/scripts         && mv install.sh /ext/scripts/install.sh         && mv everest-core /ext/source         && /entrypoint.sh run-script install" did not complete successfully: exit code: 1

I then tried to run the image and ran into a similar issue - it looks like there are issues with SSL periodically.
In this case, I was able to restart the build. Committing it so we can start testing; will try to use ethernet to build multi-platform tomorrow.
[100%] Installing everestpy from distribution
Processing /ext/cache/cpm/everest-framework/d01be681d203f70289900147e87a8f4c43ff47f0/everest-framework/everestpy
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/setuptools/
      ERROR: Could not find a version that satisfies the requirement setuptools>=42 (from versions: none)
      ERROR: No matching distribution found for setuptools>=42

      [notice] A new release of pip is available: 23.0.1 -> 24.3.1
      [notice] To update, run: python3.10 -m pip install --upgrade pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Using the committed image, the basic AC charging demo works.
DC ISO 15118-2 fails with
manager-1      | 2024-11-13 07:22:15.911271 [ERRO] ev_manager:EvMa bool module::main::car_simulatorImpl::run_simulation_loop() :: Invalid number of arguments for: iso_start_v2g_session

And I haven't even tried the OCPP integration.
But now that the basic AC demo is working, let's push the changes, then switch to the recommended base image and then get the rest of it working one-by-one.

Switching to the recommended base image, and building directly using docker compose, we get an error with system packages
1434.5 Found everest-utils at /ext/cache/cpm/everest-utils/0f1ca67a16722003325fca81f993f47e24e10efb/everest-utils
1434.9 error: externally-managed-environment
1434.9
1434.9 × This environment is externally managed
1434.9 ╰─> To install Python packages system-wide, try apt install
1434.9     python3-xyz, where xyz is the package you are trying to
1434.9     install.
1434.9
1434.9     If you wish to install a non-Debian-packaged Python package,
1434.9     create a virtual environment using python3 -m venv path/to/venv.
1434.9     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
1434.9     sure you have python3-full installed.
1434.9
1434.9     If you wish to install a non-Debian packaged Python application,
1434.9     it may be easiest to use pipx install xyz, which will manage a
1434.9     virtual environment for you. Make sure you have pipx installed.
1434.9
1434.9     See /usr/share/doc/python3.11/README.venv for more information.
1434.9
1434.9 note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
1434.9 hint: See PEP 668 for the detailed specification.
1434.9 make[3]: *** [CMakeFiles/install_everest_testing.dir/build.make:71: CMakeFiles/install_everest_testing] Error 1
1434.9 make[2]: *** [CMakeFiles/Makefile2:4664: CMakeFiles/install_everest_testing.dir/all] Error 2
1434.9 make[1]: *** [CMakeFiles/Makefile2:4671: CMakeFiles/install_everest_testing.dir/rule] Error 2
1434.9 make: *** [Makefile:667: install_everest_testing] Error 2

Checking the changes to the CI since I last set this up, to see how the build process has changed....

The CI workflow which this was based on is https://github.com/EVerest/everest-ci/blob/17cc37fdad6e7fc92390294f9153e58e80e767fc/.github/workflows/continuous_integration.yml#L14
It looks like it is set up to run as part of a reusable workflow, so we can include it directly without defining our own set of steps. This is also consistent with https://github.com/EVerest/everest-demo/issues/65
We should switch over to including it directly, after I get it to work for now since I am not super familiar with using reusable workflows. This is also intended to build a single image for everything, while we were hoping to split out the node-red since it does not need to run on the container in a production build. (@catarial or @Abby-Wheelis, maybe you can take this on as a cleanup task). We should also revisit
The scripts are in .ci/build-kit/scripts. The two scripts that are run are build-kit run-script compile and build-kit run-script run_unit_tests
Here are the new scripts:
https://github.com/EVerest/everest-core/tree/5e967358322fabd34e66a34f2919588df3ebb33f/.ci/build-kit/scripts
Copying them over and using compile, install and potentially, install_wheel

Ok, so /entrypoint.sh run-script compile, followed by /entrypoint.sh run-script install worked!
I noticed that we built and installed everestpy so maybe things will work now!

After switching to the build-kit approach, I can successfully build and run the basic AC charging and ISO 15118-2 DC demos. While trying to run the ISO + OCPP demo, however, the maeve build consistently fails due to
 => ERROR [manager builder 6/8] RUN go mod download                                                                       2.5s
------
 > [manager builder 6/8] RUN go mod download:
2.424 go<EMAIL_ADDRESS>Get<EMAIL_ADDRESS>tls: failed to verify certificate: x509: certificate signed by unknown authority
------
failed to solve: process "/bin/sh -c go mod download" did not complete successfully: exit code: 1

This is almost certainly due to VPN/netscope issues. The solution seems to be to use go mod with the insecure flag. I am not sure we can install the root CA in this environment, since we don't really have the netscope root CA.
However, maeve does not have any changes in the past 6 months

And the repo with the setChargingProfile change (from @louisg1337) is ahead of it.

So we can just use the older built images. Argh! But the older built images were on the old (non-containerd) image store.
Let's poke around to see how we can fix this. Alternatively, we can also change the CI to build and push the CSMS images as well, which will avoid the CSMS issue.

There is no option to turn off https in the go module (issue 27332 in https://github.com/golang/go).
I can figure out how to add it to the CI, but for now, I plan to just save the image after turning off the containerd store and restoring it to make progress.

build is still too large to fit in Github actions, so I will need to build and push. Even more argument for just saving + load to make progress.

#2 [internal] load metadata for ghcr.io/everest/everest-ci/build-kit-base:v1.4.2
#2 DONE 0.9s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [internal] load build context
#4 DONE 0.0s

#5 [1/3] FROM ghcr.io/everest/everest-ci/build-kit-base:v1.4.2@sha256:8b2caf2baafb843244680b1de5ebaf745d09d81e20079e99e9707d5a3165b2ee
#5 resolve ghcr.io/everest/everest-ci/build-kit-base:v1.4.2@sha256:8b2caf2baafb843244680b1de5ebaf745d09d81e20079e99e9707d5a3165b2ee done
#5 DONE 0.0s

#10 loading layer 7b98422d8569 4.79kB / 4.79kB 20.6s done
#10 loading layer c43cb7699bfd 108.07MB / 127.77MB 20.6s done
#10 loading layer 3d1f8d91b65a 388B / 388B 12.1s done
#10 loading layer 325a5bb3367f 131.07kB / 11.11MB 12.1s done
#10 loading layer 81202e26e552 32.77kB / 1.50MB 11.8s done
#10 loading layer 7695c6b1d2b7 24.67kB / 24.67kB 11.8s done
#10 loading layer d21295d7d6fd 448B / 448B 11.8s done
#10 loading layer 16313a1b9b23 158B / 158B 11.8s done
#10 loading layer 45d619634592 99B / 99B 11.8s done
#10 ERROR: write /ext/build/_deps/libocpp-build/lib/CMakeFiles/ocpp.dir/ocpp/v16/profile.cpp.gcno: no space left on device

#9 exporting to docker image format
#9 sending tarball 58.0s done
#9 ERROR: rpc error: code = Unknown desc = write /ext/build/_deps/libocpp-build/lib/CMakeFiles/ocpp.dir/ocpp/v16/profile.cpp.gcno: no space left on device
------
 > exporting to docker image format:
------
------
 > importing to docker:
------
ERROR: failed to solve: rpc error: code = Unknown desc = write /ext/build/_deps/libocpp-build/lib/CMakeFiles/ocpp.dir/ocpp/v16/profile.cpp.gcno: no space left on device
Reference
Check build summary support
Error: buildx failed with: ERROR: failed to solve: rpc error: code = Unknown desc = write /ext/build/_deps/libocpp-build/lib/CMakeFiles/ocpp.dir/ocpp/v16/profile.cpp.gcno: no space left on device

Switched back to the old container store, and I don't see any images. So it must have been building every time. I am not sure how it was building until yesterday and is not building today - unless it is some issue with netskope. But regardless, I cannot build now, and I can't figure out how to get the netskope root CA to install in the container.

Ok so now I have a couple of options:

install the root CA in the maeve container - this will work for now, but is not a long-term solution
change the script so that it uses a pre-built image, but this will be complex to restructure

image can be built using either the CI or a personal linux machine and pushed

we can just remove the go mod download. it looks like that is a performance optimization; go get will also pull the modules if needed. And at least go get does support --insecure (https://stackoverflow.com/questions/66356034/what-is-the-difference-between-go-get-command-and-go-mod-download-command). That may be the way to go for now.

Let's pull and edit the Dockerfile and then try to build locally.

Trying to use the --insecure flag, we get
 > [manager builder 7/8] RUN go get --insecure ./...:
0.168 go: -insecure flag is no longer supported; use GOINSECURE instead

Trying to use GOINSECURE does not work
/src # GOINSECURE=proxy.golang.org
/src # export GOINSECURE=proxy.golang.org
/src # go get<EMAIL_ADDRESS>go<EMAIL_ADDRESS>Get<EMAIL_ADDRESS>tls: failed to verify certificate: x509: certificate signed by unknown authority

Just to clarify, the issue is that the alpine image only has a single root CA, which is the one from netskope. The problem is that netskope doesn't seem to support the letsencrypt root CA
So
echo | openssl s_client -showcerts -servername k8s.io -connect k8s.io:443 2>/dev/null | openssl x509 -inform pem -noout
-text
/src # echo | openssl s_client -showcerts -servername k8s.io -connect k8s.io:443 2>/dev/null | openssl x509 -inform pem -noout
-text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            80:ba:10:02:09:78:44:07:96:6a:7e:f2:c6:d1:2c:75
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = Colorado, L = Golden, O = National Renewable Energy Laboratory, OU = Information Technology Services, CN = netskope.nrel.gov
        Validity
            Not Before: Oct 15 06:15:04 2024 GMT
            Not After : Nov 14 06:15:04 2025 GMT
        Subject: CN = k8s.io

So we can access non letencrypt locations
# curl https://github.com | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

<!DOCTYPE html>
<html
  lang="en"
100  5471    0  5471    0     0  16539      0 --:--:-- --:--:-- --:--:-- 16578

or
/src # curl https://google.com | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100   220    0   <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
 <TITLE>301 Moved</TITLE></HEAD><BODY>
 <H1>301 Moved</H1>
0The document has moved
 <A HREF="https://www.google.com/">here</A>.
 </BODY></HTML>
  663      0 --:--:-- --:--:-- --:--:--   664

but anything with letsencrypt fails
/src # curl https://letencrypt.com
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

If we can install the letsencrypt root CA and chain into the image, we should be fine.

Tried to switch to a debian build instead of alpine. same error
 => [manager builder 4/8] RUN if [ "amd64" = "arm64" ]; then         TARGETARCH=aarch64 ;         fi;     wget -O /usr/  10.1s
 => [manager builder 5/8] COPY ./go.mod ./go.sum ./                                                                       0.0s
 => ERROR [manager builder 6/8] RUN go mod download                                                                       2.2s
------
 > [manager builder 6/8] RUN go mod download:
2.169 go<EMAIL_ADDRESS>Get<EMAIL_ADDRESS>tls: failed to verify certificate: x509: certificate signed by unknown authority
------
failed to solve: process "/bin/sh -c go mod download" did not complete successfully: exit code: 1

I tried both bookworm and bullseye. No luck.

In the argument for not including the insecure flag, the golang commenters say:

If you can't set up a verifiable TLS cert chain, you should be using a non-HTTPS URL, to make it clear that this is not a secure connection.

But the automatic redirect to TLS defeats that
/src # go get<EMAIL_ADDRESS>go<EMAIL_ADDRESS>Get<EMAIL_ADDRESS>tls: failed to verify certificate: x509: certificate signed by unknown authority

Tried to build with docker-compose and push, but the push is running into issues.
8fa10c0194df: Waiting
03af25190641: Waiting
denied: permission_denied: The requested installation does not exist.
Error: Process completed with exit code 1.

This seems to be because we don't have a package listed

Ah no it is because we are testing this in US-JOET. Let's temporarily switch to the US-JOET namespace...
Doh!

And now the push works
  Process "buildx dial-stdio" was killed with signal SIGKILL
  Beginning upload of artifact content to blob storage
  Uploaded bytes 61599
  Finished uploading artifact content to blob storage!
  SHA256 hash of uploaded artifact is ded295fe8492880639e6e460842fe581db9b7cafc6274971fac35967d851a384
  Finalizing artifact upload
  Artifact successfully finalized<PHONE_NUMBER>)
  Artifact download URL: https://github.com/US-JOET/everest-demo/actions/runs/11843846643/artifacts/2189482814
  Writing summary

It would have likely worked with the old version where we used docker-compose as well, but let's take the win and move on for now.
https://github.com/EVerest/everest-demo/issues/78#issuecomment-2477073447

During cleanup, we could also build from docker-compose and then push using https://github.com/akhilerm/tag-push-action

At long last, the build werks ❗❗❗
Now to go ahead and run
$ bash demo-iso15118-2-ocpp-201.sh -r $(pwd) -1 -m

 ✔ Network maeve-csms                Created                                                                              0.1s
 ✔ Container maeve-csms-mqtt-1       Healthy                                                                              0.2s
 ✔ Container maeve-csms-firestore-1  Healthy                                                                              0.2s
 ✔ Container maeve-csms-manager-1    Healthy                                                                              0.2s
 ✔ Container maeve-csms-gateway-1    Healthy                                                                              0.1s

 ✔ Network everest-ac-demo_default          Created                                                                       0.0s
 ✔ Container everest-ac-demo-mqtt-server-1  Healthy                                                                       0.1s
 ✔ Container everest-ac-demo-nodered-1      Healthy                                                                       0.2s
 ✔ Container everest-ac-demo-manager-1      Healthy                                                                       0.1s

2024-11-14 21:28:46.844762 [INFO] manager          ::   ________      __                _
2024-11-14 21:28:46.845188 [INFO] manager          ::  |  ____\ \    / /               | |
2024-11-14 21:28:46.845257 [INFO] manager          ::  | |__   \ \  / /__ _ __ ___  ___| |_
2024-11-14 21:28:46.845568 [INFO] manager          ::  |  __|   \ \/ / _ \ '__/ _ \/ __| __|
2024-11-14 21:28:46.845723 [INFO] manager          ::  | |____   \  /  __/ | |  __/\__ \ |_
2024-11-14 21:28:46.845788 [INFO] manager          ::  |______|   \/ \___|_|  \___||___/\__|
2024-11-14 21:28:46.845820 [INFO] manager          ::
2024-11-14 21:28:46.845968 [INFO] manager          :: everest-framework 0.17.2 @v0.17.2
2024-11-14 21:28:46.846116 [INFO] manager          :: everest-core 2024.9.0 @2024.9.0
2024-11-14 21:28:46.846261 [INFO] manager          ::
2024-11-14 21:28:46.846467 [INFO] manager          :: Using MQTT broker mqtt-server:1883
2024-11-14 21:28:47.298496 [INFO] everest_ctrl     :: Launching controller service on port 8849
2024-11-14 21:28:47.358023 [INFO] manager          :: Loading config file at: /ext/source/config/config-sil-ocpp201-pnc.yaml

Runs!! But then immediately crashes with
/ext/cache/cpm/everest-framework/d01be681d203f70289900147e87a8f4c43ff47f0/everest-framework/lib/config.cpp(262): Throw in function void Everest::Config::load_and_validate_manifest(const std::string&, const Everest::json&)
Dynamic exception type: boost::wrapexcept<Everest::EverestConfigError>
std::exception::what: Failed to load and parse manifest file /ext/dist/libexec/everest/modules/JsCarSimulator/manifest.yaml: File 'manifest.(yaml|json)' does not exist
[boost::log::v2_mt_posix::current_scope_info_tag*] = Everest::Config::Config(std::shared_ptr<Everest::RuntimeSettings>, bool)

Time to start tweaking the config - fortunately, @the-bay-kay has gone through this and figured it out (mostly!)

After changing a bunch of config, EVerest started up and was able to finish a charge session with -1

/ext/source/config/config-sil-ocpp201-pnc.yaml
/ext/dist/share/everest/modules/OCPP201/component_config/standardized/InternalCtrlr.json
/ext/dist/share/everest/modules/OCPP201/component_config/custom
the node-red UI

Now applying those changes and then I will merge the current open PR https://github.com/EVerest/everest-demo/pull/83/
Note that these are similar to the changes that @the-bay-kay already made, but cleaned up a bit; for example, instead of adding a connector to the config, we just remove the existing connector from the component_config so that they are compatible. Note that the uMWC has only one port.

Incorporated changes (with copious patches) into the current demo. This is the stock charging profile, without any departure time. Committed as https://github.com/EVerest/everest-demo/pull/83/commits/b613ba6c49f00339a090abe033aecfb45101dbad

Renamed the US-JOET images to
sdesk-35737s:everest-demo kshankar$ docker tag ghcr.io/us-joet/everest-demo/manager:0.0.16 ghcr.io/everest/everest-demo/maeve-manager:0.0.17-localbuild
sdesk-35737s:everest-demo kshankar$ docker tag ghcr.io/us-joet/everest-demo/gateway:0.0.16 ghcr.io/everest/everest-demo/maeve-gateway:0.0.17-localbuild

And changed the tag to 0.0.17-localbuild
SP1 works without any changes!!
Almost done with cleanup and about ready to merge.

Now that I have merged https://github.com/EVerest/everest-demo/pull/83
this is largely done so I am closing this.
I am now going to pivot to #79, so I will close this.
But for the record, other "nice to have" changes would be to:

get SP2 and SP3 to work
support multi-platform images (#63)

@abby-wheelis, you should be able to test the AC, ISO 15118-2 DC and the ISO 15118 + OCPP 201 demos now.
@Catarial needs to wait until #63 is done


Could not find type id 12 isObject=false

What is the output link of /viaversion dump?
https://dump.viaversion.com/78f0336cd6f189375e396ba71319b42ce2089c726c1b321d55309dca15d47318
Are you using any additional software like BungeeCord? If so, what software and version? (Please list your plugins as well)
Waterfall 1.13.1 build 219
How does this error happen? login? Using an item?
Write console. I don't know.
Is there an error in the console? Use pastebin.com. Is there a kick message?
02.09 15:46:34 [Server] ERROR [ViaVersion] Could not find type id 12 isObject=false

Could you test with the latest build and see if it still happens?
It might be worth trying to narrow this down and just go around your server by yourself. It might be a mob etc related.

See above,


[WIP] Live reload

For visibility on current work on issue #11:

[x] Rebuild on file changes
[x] Reload page on rebuild
[x] Allow customisation of watched directories*
[ ] (Stretch) Only rebuild files that have changed
[ ] (Stretch) Pick up data sources changes

Note this is currently done through a config object passed to startAppServer I'm not sure if this is the nicest API but happy to have input.

Note this requires following script tag in src/template.html of JungleJS template to enable page reload:
<script src="/reload/reload.js"></script>

@kiraarghy  I think we both kind of went in different directions here, cause I've managed to get some stuff working like only rebuilding files that changed, but have been unable to get browser refresh working... take a look at my branch here, I've kind of entirely changed the build process but rebuild of a page is now almost instant https://github.com/junglejs/junglejs/tree/feature-live-reloading


Weird text rendering with SkewTransform

Describe the bug
So, I updated to version 11.1.1 of Avalonia, and my text (that was working just fine previously) renders like this:

If I remove the <SkewTransform>, I get this:

Has something changed between versions?

Expected behavior
It should look properly skewed, like this (old 11.0.x version):

Avalonia version
11.1.1
OS
Windows
Additional context
No response

This is seems to be fixed in master

This problem is still present in 11.1.2.


hot-fix

Description
Quick fix to delete app.js.

https://github.com/BuildForSDG/team-010-frontend/tree/hot-fix/src please could you confirm that you can see the appropriate files on the remote version? I think you should clone the repo afresh

@sudo-kaizen https://github.com/BuildForSDG/team-010-frontend/tree/hot-fix/src please could you confirm that you can see the appropriate files on the remote version? I think you should clone the repo afresh

Please find attached a screenshot of the ls src/ command for the PR commit. I cant find both App.js and app.js

@sudo-kaizen , could you check now, I have followed procedure

The latest commit works on my local


Bump forms dependencies

Closes #3529

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

I played around with it and found no issues. Two things I stumbled upon though:

Moving form fields still feel clunky:

Resize handles are gone. Where did they go?

/cc @marcellobarile @CatalinaMoisuc I think it makes sense to bump the same versions in the Web Modeler. I'm not familiar with the setup, so I can't do that on my own, unfortunately 😅

@pinussilvestrus thanks for the heads up, we should get the bump from the Renovate bot along with other bpmnio libraries updates


Update api.py

added content-type to requests.

GET requests don't need content-type header. json=data already sets the content type.


Added Guide to library

Guide (sans types)

@jasonzoladz Looking good!
@cryogenian Please do the final review & merge when you're happy.

Awesome!
@jasonzoladz Thank you very much!


Alternate cmd style for session.run and additional session.run_cmds to run several commands from a multiline string or text file

How would this feature be useful?
One feature from tox that I am missing in nox is the ability to "see" my command as a simple text line, and also to pass several commands in a row. The most pythonic equivalent seems to be a multiline string containing the commands:
cmds = """
dir
echo "hello_world"
"""

Even without going for multiline, I find it a bit unelegant having to write this:
session.run(*'echo "hello_world"'.split(' '))
instead of this:
session.run('echo "hello_world"')
Describe the solution you'd like

For one-line commands, maybe we could modify session.run so that it accepts a single string containing space-separated command and arguments ?

For multiline commands (several commands, one per line), a new session.run_cmds() method able to run commands either from a multiline string or a multiline text file.

session.run_cmds(cmds_str=None, cmdsfile_path=None,
                       env: Mapping[str, str] = None,
                       silent: bool = False, success_codes: Sequence[int] = (0, ), external: bool = False
)

Behind the scenes, after splitting lines, stripping them and keeping only the non-empty, it would call session.run(*line.split(' ')), **kwargs
(note: it should not split the spaces sitting between double quotes)

Afterthought: actually we could support all of these features in the same run method, without ambiguity. The rare case where the newline symbol would be in the command could be resolved by splitting it manually. I'll push a PR.

Nox requiring commands to be specified as a list of strings is intentional to help avoid common errors with specifying arguments to subprocesses.
I'm inclined not to add this, but I'm curious what the other maintainers think.

Which common errors do you have in mind ?
At least for my usage of nox, all cases that I encountered so far were cases where stripping/splitting the command according to spaces (with the subtelty of not splitting a space that is inside double quotes) would provide the exact same result than having me split the command into a tuple of strings.
With one major difference though: readability. Indeed, it is much easier to read my own code when the commands are full strings rather than tuples of strings. This is especially true when I execute several commands in a nox session.
ok let's wait for other maintainers' feedback!

I think for now I'm -1 on including this, and no other maintainers have chimed in. Happy to change my mind if other folks have thoughts, but for now I'm going to close this.


BFT Block Puller: test byzantine behavior of the orderer during delivery against the peer

When running a BFT ordering service the peer is pulling blocks from the ordering service using the BFT Deliverer that protects against censorship attacks of the orderer that is asked to deliver blocks.
In this task we have to create an integration test environment that emulates the behavior of a malicious orderer. We that have to emulate malicious + faulty behavior against a peer pulling blocks.
Scenario 1 - censorship detection

an orderer O1 that is asked to deliver blocks, but delays delivering a block n
other orderers (e.g. O2,3,4) deliver block attestation n, it is picked up by the header receivers and the censorship monitor
after a BFTCensorshipTimeout expires, the monitor declares a censorship alert on O1, disconnects from it, and chooses a new orderer from (e.g. O2,3,4) to ask blocks from.

Scenario 2 - censorship suspicion that resets

an orderer O1 that is asked to deliver blocks delays delivering a block n
other orderers (e.g. O2,3,4) deliver block attestation n, it is picked up by the header receivers and the censorship monitor
just before BFTCensorshipTimeout expires, O1 delivers the block n. The monitor resets the timeout.

Scenario 3 - bad blocks and comm problems

an orderer O1 that is asked to deliver blocks
the orderer sends a bad block: bad signature | bad content
the BFTDeliverer disconnects from O1, and chooses a new orderer from (e.g. O2,3,4) to ask blocks from.

Scenario 4 - bad attestations

an orderer O1 that is asked to deliver blocks, works ok
other orderers (e.g. O2,3,4) deliver block attestation n, but are causing problems: some disconnect, some bad blocks, some bad signatures, etc.
errors in header receivers connected orderers (e.g. O2,3,4) delivering block attestation are logged but ignored, and the receivers are relaunched (with a backoff interval).

These scenarios are tested in the unit tests of the BFTDeliverer and the CensorshipMonitor, now we want to test some of these basic scenarios in the integration tests.
The challenge here is to emulate the behavior of a byzantine orderer.

Start a network with 4 BFT orderers  2 orgs and 4 peers
In the genesis block, the orderer endpoints (delivery service address as the peer sees it) is set to different addresses then what the orderers actually bind too.
the peers cannot connect to orderers.
feed the orderers with TXs to create soem blocks (~100)
the TXs can be junk, that will cause them to be invalid when committed by the peers, but we don't care.
stop one orderer and copy its ledger to a side location.
write a new test service that mimics the orderer's side of the delivery service - "orderer facade".
the facade access the copied ledger.
the facade accepts the SeekInfo request and responds as the orderer does, delivering the blocks that were created earlier.
the facade has a test API that allows the test to control the behavior - start/ stop/ pause/ deliver until /etc
use these controls to emulate the scenarios above
start a facade for each orderer O1-4

Stages:

[ ] Create the ledger: start an ordering service, submit TXs to make blocks, shut down, copy the ledger
[ ] Mock process: start N mock processes that load the ledger
[ ] Test the mocks can access the ledger
[ ] Delivery service server side: add the delivery gRPC service to the mock, delivers block correctly
[ ] Connect peer to mock, consume (pull) blocks correctly
[ ] Add byzantine behavior to f  mocks, check peer pulls blocks correctly

@semil and @arkadiPiven

The delivery service server handlers are here:
common/deliver/deliver.go

Continue from PR https://github.com/hyperledger/fabric/pull/4402

Added the test PR: https://github.com/hyperledger/fabric/pull/4526

Added the test PR: https://github.com/hyperledger/fabric/pull/4542

See also #4724


Adds the check-required javascript action

This PR adds the check-required javascript action to be used instead of the code in the workflow

@jgiannuzzi PR update. Please re-review


Extra slash in "Edit on GitHub" link

Details

Read the Docs project URL: http://glom.readthedocs.io/
GitHub URL: https://github.com/mahmoud/glom
Build URL (if applicable): N/A
Read the Docs username (if applicable): mahmoud

Expected Result
"Edit on GitHub" link at the top of the page should link to doc on GitHub.
Actual Result
"Edit on GitHub" link at the top of the page links to 404. There is an extra slash in the URL: https://github.com/mahmoud/glom//master/docs/index.rst (after glom, before master)
Interestingly, the "Edit" link in the bottom left menu seems to work fine.

Hi magnunleno,
I don't think it happens because of the extra slash but because of the missing word blob. The url should be https://github.com/mahmoud/glom/blob/master/docs/index.rst. Isn't it?

Can you try updating the rtd-theme? https://github.com/mahmoud/glom/blob/5a10e66c2f1204988f97e6545055bfc5b36e052f/requirements-rtd.txt#L36 Current one is 0.4.3 https://pypi.python.org/pypi/sphinx_rtd_theme

@rexzing you're absolutely right!
@stsewd trying this now, thanks!

@stsewd Hmm, no luck I'm afraid. In fact, I see 3 slashes now: https://github.com/mahmoud/glom///master/docs/index.rst.

Thanks, I'll try to replicate this locally and dig a little more

Whoops, I lied, did stuff on the wrong local branch. Updating the theme got me a working link, with blob, but it does have an extra slash fwiw: https://github.com/mahmoud/glom//blob/master/docs/index.rst
It still gets me to GitHub though, so I'm happy enough! :) Thanks for your help!

The extra slash is from the repo url, we could strip it, tho

I'll leave it up to you, but I think an rstrip and join of a single slash should yield a nicely normalized URL. :)

@ststewd Thank you!


Enums not used when creating events in meshery handlers

Current Behavior
Events are generated every time we run a handler to keep record of the activity of a user. These events currently are not using Enums and directly passing string.

Expected Behavior
We should define all the type of actions for events and then use it when building a event.
Screenshots/Logs

Environment

Host OS: Mac Linux Windows
Platform: Docker or Kubernetes
Meshery Server Version: stable-v
Meshery Client Version: stable-v

Contributor Guides and Handbook

🛠 Meshery Build & Release Strategy
📚 Instructions for contributing to documentation

Meshery documentation site and source

🎨 Wireframes and designs for Meshery UI in Figma (open invite)
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

I want to work on this issue.
Propose solution is:
use constant:

https://github.com/meshery/meshkit/pull/619
Added constant to the model.

Events and their types will need to be schema-driven. See the meshery/schemas repo…


chore: Upgrade Golang version from 1.18 to 1.19

Fixed lint issues:
Can't run linter goanalysis_metalinter: goanalysis_metalinter: buildir: package "netip" (isInitialPkg: false, needAnalyzeSource: true): in net/netip.AddrFromSlice: cannot convert Load <[]byte> t0 ([]byte) to [4]byte

Warnings:
WARN [runner] The linter 'structcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter.  Replaced by unused.
WARN [runner] The linter 'deadcode' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter.  Replaced by unused.
WARN [runner] The linter 'varcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter.  Replaced by unused.

Hi @spolti -- could you separate the changes into 2 PRs:

upgrade Go from 1.18 to 1.19
upgrade golang.org/x/net to v0.17.0

Hi @ckadner, sure.

Done, dependencies upgrade: https://github.com/kserve/rest-proxy/pull/30
@ckadner I don't have permissions to see the dependabot issues, are they still related only with the Go lang update? Or should we update the above PR's descrioption?

the CI issue will be fixed by https://github.com/kserve/rest-proxy/pull/30

Hi @spolti -- I added another comment on a thread from my past review

https://github.com/kserve/rest-proxy/pull/28#discussion_r1402800637

Thanks @ckadner, sry for the late reply, busy day.


edge case splitting from first post of discussion

Let's validate that splitting from post 1 in a dicussion does not break our functionality. It should set the first not-split post as OP in this case.

Wow, there are old issues here.
At the moment the extension does not allow to select the first post. Should we allow it ?

not right now, this request is very low priority ;)

Will be nice split a first post to another topic in the future.
Maybe a user create a duplicate of a locked topic so if you can split the first post you can "delete" the duplicate topic and add as post of the other discussion. Will be nice have this.