Create profiling recipe for DeepProfiler output

@Arkkienkeli has produced Cell Painting CNN v1 embeddings for all of JUMP. Currently these are at the single-cell level.
We now need a workflow that does everything downstream.
We will use this issue to discuss and plan and likely move this to a new repo.
@jccaicedo and @shntnu will create a Snakemake workflow similar to https://github.com/broadinstitute/jump-profiling-recipe, but will start at the single cell level (Level 2) not at the replicate level (Level 3)

Source:https://github.com/cytomining/pycytominer/blob/main/media/pipeline.png
Here are the steps based on this notebook recommended by @Arkkienkeli

Site-level median
Well-level mean
Experiment-level Sphering
Treatment-level mean
No feature selection, no outlier removal, and no per-feature normalization (i.e. no standardization, only Sphering)

We will need input from

@Zitong-Chen-16 for Snakemake
@ashah03 for figuring out how to profitably use SkyPilot if we can and maybe figuring out how to get write access to the staging bucket

Also from

@niranjchandrasekaran for general pointers about JUMP
@ErinWeisbart for general pointers about workflow, because this workflow will eventually end up in Cimini lab's hands

Location of data

we will need to rename the folder Cell_Patining_CNN_conv6a_b3667957 to fix the typo, when we sync it over to s3://cellpainting-gallery
@Arkkienkeli seems to have used parquets instead of npz, and this works well!

Notes

Slack discussions are here https://broadinstitute.slack.com/archives/C06CZUGR1C5/p1709341592744799
The artefacts should be structured like described here https://broadinstitute.github.io/cellpainting-gallery/data_structure.html#workspace-dl-folder-structure. We can propose changes to the structure if it doesn't make sense.
Mike Ando is planning to produce well level profiles for EffNet embeddings created using a Terra workflow https://github.com/jump-cellpainting/datasets-private/issues/50#issuecomment-1982217038. However, if our workflow is general enough, we can use our workflow to process it instead (and we will keep him posted in which case)
Whatever we produce through this effort will likely become a de facto standard – or at least a baseline – in the field :) So while we don't want to overthink this, we should certainly justify each step to some reasonable extent.

@viditagr and I have started with level-2 to level-3 aggregation and want to implement the output in the correct format.
@shntnu can you point us to how the aggregated files are supposed to be structured, named and organized in the directories?

What we are doing is to create a parallel directory called profiles and follow the specified directory structure with batches and plates. Inside we save a single file with the well-level aggregated data in parquet format.

can you point us to how the aggregated files are supposed to be structured, named and organized in the directories?

I missed replying to this
Please see below
https://broadinstitute.github.io/cellpainting-gallery/data_structure.html#workspace-dl-folder-structure

Update from Caicedo lab, which I am posting here for @jessica-ewald's visibility:

We are making progress with the aggregation, but we found a few corrupted parquet files. Nikita is helping fix this, and we will also take the opportunity to fix the typo in the Cell Painting directory names. After this is complete, the aggregated data will be available together with the fixed data.

@Arkkienkeli can you upload the fixed embeddings to the staging bucket? We should be able to take it from there

We are almost ready!

Thanks @Arkkienkeli
Do you have an ETA?

@shntnu I have messaged you and Juan on Slack. Let's finalize details there.


[BUG] ImageGenerationOptions missing DeploymentName property

Library name and version
Azure.AI.OpenAI 1.0.0-beta.9
Describe the bug
GetImageGenerationsAsync function is now completely broken because of missing DeploymentName property in ImageGenerationOptions model. I can't produce Dalle request
Expected behavior
ImageGenerationOptions has DeploymentName property like ChatCompletionsOptions
Actual behavior
Only Prompt, ImageCount, Size and User are supported now.
Reproduction Steps
var responseWithoutStream = await client.GetImageGenerationsAsync(
          new ImageGenerationOptions
           {
                    Prompt = text,
                    ImageCount = 1,
                    Size = ImageSize.Size1024x1024,
            }
     );

I this goes to 404 address.
Environment
No response

Thanks for getting in touch! Prior to yesterday's beta.10 release, the Azure.AI.OpenAI library used the older "no-deployment" endpoint that only worked with legacy dall-e-2 image generation models. With yesterday's change, GetImageGenerations() is now fully aligned with OpenAI's dall-e-3 calls and should be able to use your dall-e-3 resources as expected.


#8704: Program _HI registers in BH noc nonblocking apis and zero out _MID registers

Ticket

8704
Closed because this commit has been previously pushed/validated on BH bringup branch abhullar/bh-bringup

Problem description

NOC command buffer programming sequence has changed for BH to support wider addresses (3 registers need to be programmed rather than 2)

What's changed

*_HI register is programmed rather than *_MID (BH noc apis accept 36 bit addresses)
*_MID registers are zeroed out in noc_init since we don't address over 4GB

Checklist

[ ] Post commit CI passes
[ ] Model regression CI testing passes (if applicable)
[ ] New/Existing tests provide coverage for changes. (No new tests added for this change, incorrectly programming these registers would leave to mismatches / hangs in all BH tests)

future work, fwiw: probably no point in initializing some of those regs to "my" at init, just takes code space (not a big deal). eventually we should move all the reg writes out of the dataflow api into target specific files, that would also move the ifdefs

Yes I didn't love the ifdefs in dataflow api but since we discussed updating the noc api structure, I figured the clean up could roll into that
Opened up

https://github.com/tenstorrent/tt-metal/issues/9446 to explore removing my_x / my_y in noc_init
https://github.com/tenstorrent/tt-metal/issues/9445 to remove reg writes out of dataflow


RsXslt fails to resolve files in classpath

@rultor release, tag is 1.24.3

@rultor release, tag is 1.24.3

@yegor256 OK, I will release it now. Please check the progress here

@rultor release, tag is 1.24.3

@yegor256 Done! FYI, the full log is here (took me 10min)

Job gh:yegor256/takes#1145 is not assigned, can't get performer


tox-factor does not work with tox 4

Any tox command immediately fails after installing tox-factor alongside tox 4:
tox --version
ROOT: No tox.ini or setup.cfg or pyproject.toml found, assuming empty tox.ini at /tmp/tmp.ZZGIilhvQf
Traceback (most recent call last):
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox_factor/compat.py", line 2, in <module>
    from tox.config.parallel import ENV_VAR_KEY_PUBLIC as TOX_PARALLEL_ENV
ModuleNotFoundError: No module named 'tox.config.parallel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmp.ZZGIilhvQf/venv/bin/tox", line 8, in <module>
    sys.exit(run())
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/run.py", line 19, in run
    result = main(sys.argv[1:] if args is None else args)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/run.py", line 38, in main
    state = setup_state(args)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/run.py", line 53, in setup_state
    options = get_options(*args)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/config/cli/parse.py", line 38, in get_options
    guess_verbosity, log_handler, source = _get_base(args)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/config/cli/parse.py", line 61, in _get_base
    MANAGER.load_plugins(source.path)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/plugin/manager.py", line 90, in load_plugins
    self._register_plugins(inline)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox/plugin/manager.py", line 38, in _register_plugins
    self.manager.load_setuptools_entrypoints(NAME)
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox_factor/hooks.py", line 5, in <module>
    from .compat import TOX_PARALLEL_ENV
  File "/tmp/tmp.ZZGIilhvQf/venv/lib/python3.10/site-packages/tox_factor/compat.py", line 4, in <module>
    from tox.config.parallel import ENV_VAR_KEY as TOX_PARALLEL_ENV
ModuleNotFoundError: No module named 'tox.config.parallel'

tested in a fresh python 3.10 virtualenv, pip freeze output
cachetools==5.2.0
chardet==5.1.0
colorama==0.4.6
distlib==0.3.6
filelock==3.9.0
packaging==22.0
platformdirs==2.6.2
pluggy==1.0.0
pyproject_api==1.4.0
tomli==2.0.1
tox==4.2.5
tox-factor==0.1.2
virtualenv==20.17.

I added a note in PR #19.


Recent troubles: expl3, fixltx2e, etc.

The following MCE:
\RequirePackage{pkgloader}
\documentclass{article}
\usepackage{cleveref}
\usepackage{hyperref}
\LoadPackagesNow
\begin{document}
Foo.
\end{document}

points out some (recent, I guess) troubles:

Package l3regex is obsolete and has been removed!
LaTeX warning: "kernel/invalid-variant"

Variant form nncn invalid for base form \int_step_inline:nnnn.
Variant form cn invalid for base form \withargs:nnn.
Variant form cn invalid for base form \withargs:nnnn.

Package fixltx2e is not required with releases after 2015.
LaTeX Warnings:

You have requested document class article, but the document class provides
article.
You have requested package hyperref, but the package provides hyperref.
You have requested package cleveref, but the package provides cleveref.

Many thanks for reporting this, and my apologies for taking so long to respond. The fact is that I haven't used LaTeX for a number of years, and I'm too busy to actively maintain this package.
Short of finding someone who'd like to take over maintenance of the package, I'm happy to look at any PR you submit. I'll have a look at it, make sure it doesn't actually break stuff, and release it. But I'm afraid I won't be coding on this any time soon.

I think, this is fixed by https://github.com/mhelvens/latex-pkgloader/pull/25.
Think, one can just merge the PR and just release a new version.


Permission question for deploying v5

Customer is trying to deploy v5 in QA, ran into an issue from v4 with roles that may delay production upgrade, why is this role needed?  Is it necessary? Is there a workaround?
Basically, we are not having the access to assign roles in Azure RBAC as mentioned below which is a prerequisite.
•	An role to assign roles in Azure RBAC. To check if you have permission to do this,
o	Goto the subscription page in Azure portal. Then, goto Access Control(IAM) and click on View my access button.
o	Click on your role and in search permissions text box, search for Microsoft.Authorization/roleAssignments/Write.
o	If your current role does not have the permission, then you can grant yourself the built in role User Access Administrator or create a custom role.
o	Please follow this link. Use this action Microsoft.Authorization/roleAssignments/Write in the custom role to assign roles in Azure RBAC.
Azure custom roles - Azure RBAC
Learn how to create Azure custom roles with Azure role-based access control (Azure RBAC) for fine-grained access management of Azure resources.

@v-royavinash the customer is trying to get approval from their InfoSec team to provide this access, they have asked "What would happen if they tried to deploy v5.0 without this role and how would it break the deployment?"

Users who’ve User Access Administrator permissions only can deploy the v5.0 app. This is the pre-requisite for the app.
The below error will be encountered if you try deploying the app without the required privilege.

Error Details :
It will throw an Authorization failed for type 'Microsoft.Authorization/roleAssignments. The generic error message is mentioned below,
{"code":"InvalidTemplateDeployment","message":"Deployment failed with multiple errors: 'Authorization failed for template resource ****** of type 'Microsoft.Authorization/roleAssignments'. The client **' with object id ' does not have permission to perform action 'Microsoft.Authorization/roleAssignments/write' at scope '/subscriptions//resourceGroups/rg-cc-version-5/providers/Microsoft.ServiceBus/namespaces/ccwithoutaccess/providers/Microsoft.Authorization/roleAssignments/.:

Hi @scjohnmsft , Do you have any further query related to the permissions or are we good to close this request?


Add basic options selection to demo

This will make it easier to test options without needing to modify the demo.

Great. Hope that this will help out.


hax: IndexError: list index out of range when Preparing the reply for HaNvecGetEvent

On SAGE cluster when starting any client process (like c0cp or mcp) it gets stuck forever with the following stack:
#3  0x00007fab6d941014 in m0_semaphore_down (semaphore=0x7ffd9ffe3158) at lib/user_space/semaphore.c:62
#4  0x00007fab6d933b15 in m0_chan_wait (link=link@entry=0x7ffd9ffe3130) at lib/chan.c:341
#5  0x00007fab6d8c2994 in m0_conf_objs_ha_update (nvec=nvec@entry=0x7ffd9ffe3270) at conf/helpers.c:576
#6  0x00007fab6d8c2cc8 in m0_conf_confc_ha_update_async (confc=<optimized out>, nvec=nvec@entry=0x7ffd9ffe3270, chan=chan@entry=0x0) at conf/helpers.c:642
#7  0x00007fab6d8c2d62 in m0_conf_confc_ha_update (confc=<optimized out>) at conf/helpers.c:663
#8  0x00007fab6d95baba in confc_init (m0c=0x2107d00) at motr/client_init.c:939
#9  initlift_confc (mach=0x2111138) at motr/client_init.c:1008
#10 0x00007fab6d9ce41c in state_set (mach=0x2111138, state=5, rc=<optimized out>) at sm/sm.c:457
#11 0x00007fab6d95c98c in initlift_move_next_floor (m0c=m0c@entry=0x2107d00) at motr/client_init.c:364
#12 0x00007fab6d95d0c0 in m0_client_init (m0c_p=0x801468 <instance>, conf=0x801420 <conf>, init_m0=<optimized out>) at motr/client_init.c:1568

In the system log there is following error from hax:
Jan 11 16:04:26 client-25 hare-hax[16976]: 2021-01-11 16:04:26,545 [DEBUG] {qconsumer} Preparing the reply for HaNvecGetEvent (nvec size = 976)
Jan 11 16:04:26 client-25 hare-hax[16976]: 2021-01-11 16:04:26,596 [ERROR] {qconsumer} **ERROR**
Jan 11 16:04:26 client-25 hare-hax[16976]: Traceback (most recent call last):
Jan 11 16:04:26 client-25 hare-hax[16976]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 42, in wrapper
Jan 11 16:04:26 client-25 hare-hax[16976]: return fn(*args, **kwargs)
Jan 11 16:04:26 client-25 hare-hax[16976]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 278, in ha_nvec_get_reply
Jan 11 16:04:26 client-25 hare-hax[16976]: n.note.no_id.f_key) !=
Jan 11 16:04:26 client-25 hare-hax[16976]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 414, in get_conf_obj_status
Jan 11 16:04:26 client-25 hare-hax[16976]: return self.get_node_health(node_name)
Jan 11 16:04:26 client-25 hare-hax[16976]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 435, in get_node_health
Jan 11 16:04:26 client-25 hare-hax[16976]: return str(node_data[0]['Status'])
Jan 11 16:04:26 client-25 hare-hax[16976]: IndexError: list index out of range

(Full log -
hax-IndexError-on-HaNvecGetEvent-syslog.txt.)
It was working fine yesterday, but not today. Maybe it has something to do with the fact that a few client nodes were shutdown:
$ consul members
Node          Address           Status  Type    Build  Protocol  DC   Segment
sage-tier1-1  <IP_ADDRESS>:8301   alive   server  1.7.8  2         dc1  <all>
client-21     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-22     <IP_ADDRESS>:8301  left    client  1.7.8  2         dc1  <default>
client-23     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-24     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-25     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-26     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-27     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-28     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-01   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-02   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-03   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-04   <IP_ADDRESS>:8301  left    client  1.7.8  2         dc1  <default>
sage-tier1-2  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-1  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-2  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-4  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier3-1  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
visnode-01    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-02    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-03    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-04    <IP_ADDRESS>:8301  left    client  1.7.8  2         dc1  <default>

But such things should not affect the cluster or the other client nodes which are alive.
Versions:
Hare: cortx-hare-1.0.0-1_git28f3372.el7.x86_64
Motr: cortx-motr-1.0.0-1_git89f7737_3.10.0_1127.19.1.el7.x86_64 (from https://github.com/Seagate/cortx-motr/pull/304)

Checking..

Checking..

@andriytk, can you provide the output of,
curl http://<IP_ADDRESS>:8500/v1/health/node/<node-name>

for the nodes that are shutdown? Do you see the problem if all the nodes are up and running?
Probably node_data is empty, we might need to take care of it. I will try to check this locally as well.

@andriytk, can you provide the output of,
curl http://<IP_ADDRESS>:8500/v1/health/node/<node-name>

for the nodes that are shutdown? Do you see the problem if all the nodes are up and running?
Probably node_data is empty, we might need to take care of it. I will try to check this locally as well.

$ for n in client-22 datawarp-04 visnode-04; do curl http://<IP_ADDRESS>:8500/v1/health/node/client-22; done
[][][]

Yesterday all nodes where up and running, there was no problem with running mcp client utility then.

$ for n in client-22 datawarp-04 visnode-04; do curl http://<IP_ADDRESS>:8500/v1/health/node/client-22; done
[][][]

Yesterday all nodes where up and running, there was no problem with running mcp client utility then.

@knekrasov, I am trying to build this patch,
diff --git a/hax/hax/util.py b/hax/hax/util.py
index 4473917..382fe54 100644
--- a/hax/hax/util.py
+++ b/hax/hax/util.py
@@ -446,9 +446,23 @@ class ConsulUtil:
             if f'{fid}' == x['Key'].split('/')[-1]
         ]

+    def is_node_alive(self, node: str) -> bool:
+        try:
+            members_data = self.cns.agent.members()
+            LOG.debug("members: %s", members_data)
+            for member in members_data:
+                if member['Name'] == node:
+                    return int(member['Status']) == 1
+            return True
+        except (ConsulException, HTTPError, RequestException) as e:
+            raise HAConsistencyException(
+                'Failed to members data from Consul') from e
+
     def get_node_health(self, node: str) -> str:
         try:
             node_data = self.cns.health.node(node)[1]
+            if not node_data and (not self.is_node_alive(node)):
+                return 'failed'
             return str(node_data[0]['Status'])
         except (ConsulException, HTTPError, RequestException) as e:
             raise HAConsistencyException(

but failing with below build error,
WARNING: flake8 setuptools integration is deprecated and scheduled for removal in 4.x.  For more information, see https://gitlab.com/pycqa/flake8/issues/544
running mypy
hax/util.py:451: error: "Agent" has no attribute "members"
Found 1 error in 1 file (checked 18 source files)

error: Mypy returned 1. Exiting with FAILURE status
make[1]: *** [check-hax] Error 1
make: *** [build] Error 2

but I can see the function part of Agent class in the python-consul sources also able to invoke it explicitly in python3 interactive shell.
Tried multiple things but somehow none of the Agent routines are available other than self(), I suspect some kind of build problem but not able to figure it out yet. Kindly check the same. Thank you.

@knekrasov, I am trying to build this patch,
diff --git a/hax/hax/util.py b/hax/hax/util.py
index 4473917..382fe54 100644
--- a/hax/hax/util.py
+++ b/hax/hax/util.py
@@ -446,9 +446,23 @@ class ConsulUtil:
             if f'{fid}' == x['Key'].split('/')[-1]
         ]

+    def is_node_alive(self, node: str) -> bool:
+        try:
+            members_data = self.cns.agent.members()
+            LOG.debug("members: %s", members_data)
+            for member in members_data:
+                if member['Name'] == node:
+                    return int(member['Status']) == 1
+            return True
+        except (ConsulException, HTTPError, RequestException) as e:
+            raise HAConsistencyException(
+                'Failed to members data from Consul') from e
+
     def get_node_health(self, node: str) -> str:
         try:
             node_data = self.cns.health.node(node)[1]
+            if not node_data and (not self.is_node_alive(node)):
+                return 'failed'
             return str(node_data[0]['Status'])
         except (ConsulException, HTTPError, RequestException) as e:
             raise HAConsistencyException(

but failing with below build error,
WARNING: flake8 setuptools integration is deprecated and scheduled for removal in 4.x.  For more information, see https://gitlab.com/pycqa/flake8/issues/544
running mypy
hax/util.py:451: error: "Agent" has no attribute "members"
Found 1 error in 1 file (checked 18 source files)

error: Mypy returned 1. Exiting with FAILURE status
make[1]: *** [check-hax] Error 1
make: *** [build] Error 2

but I can see the function part of Agent class in the python-consul sources also able to invoke it explicitly in python3 interactive shell.
Tried multiple things but somehow none of the Agent routines are available other than self(), I suspect some kind of build problem but not able to figure it out yet. Kindly check the same. Thank you.

@mssawant The problem is quite clear - we need to improve our mypy stubs for python-consul. I'll do that today.
In the meantime, can you explain why m0conf/nodes subtree can have not only node names (which are valid to query about from Consul Health) but plain FIDs as well?
Quote from KV/4 RFC:

Key
Value
Description

m0conf/nodes/<name>/processes/<process_fidk>/endpoint
endpoint address
Endpoint address of the Motr process (Consul service) with fid key <process_fidk>.  Example: <IP_ADDRESS>@tcp:12345:44:101.

m0conf/nodes/<name>/processes/<process_fidk>/meta_data
path to meta-data disk
m0mkfs uses this value to create meta-data pool.

m0conf/nodes/<name>/processes/<process_fidk>/services/<svc_type>
Fid key
Fid key of the Motr service, specified by its type, parent process, and node.

m0conf/nodes/<node_fid>
{ "name": "<node name>", "state": "<HA state>" }

m0conf/nodes/<node_fid>/processes/<process_fid>
{ "name": "<process name>", "state": "<HA state>" }

m0conf/nodes/<node_fid>/processes/<process_fid>/services/<svc_fid>
{ "name": "<service name>", "state": "<HA state>" }

m0conf/nodes/<node_fid>/processes/<process_fid>/services/<svc_fid>/sdevs/<sdev_fid>
{ "path": "<sdev path>", "state": "<HA state>" }

I guess that is the reason why IndexError appears.

@mssawant The problem is quite clear - we need to improve our mypy stubs for python-consul. I'll do that today.
In the meantime, can you explain why m0conf/nodes subtree can have not only node names (which are valid to query about from Consul Health) but plain FIDs as well?
Quote from KV/4 RFC:

Key
Value
Description

m0conf/nodes/<name>/processes/<process_fidk>/endpoint
endpoint address
Endpoint address of the Motr process (Consul service) with fid key <process_fidk>.  Example: <IP_ADDRESS>@tcp:12345:44:101.

m0conf/nodes/<name>/processes/<process_fidk>/meta_data
path to meta-data disk
m0mkfs uses this value to create meta-data pool.

m0conf/nodes/<name>/processes/<process_fidk>/services/<svc_type>
Fid key
Fid key of the Motr service, specified by its type, parent process, and node.

m0conf/nodes/<node_fid>
{ "name": "<node name>", "state": "<HA state>" }

m0conf/nodes/<node_fid>/processes/<process_fid>
{ "name": "<process name>", "state": "<HA state>" }

m0conf/nodes/<node_fid>/processes/<process_fid>/services/<svc_fid>
{ "name": "<service name>", "state": "<HA state>" }

m0conf/nodes/<node_fid>/processes/<process_fid>/services/<svc_fid>/sdevs/<sdev_fid>
{ "path": "<sdev path>", "state": "<HA state>" }

I guess that is the reason why IndexError appears.

Now all the nodes are alive, but the issue is still there:
$ consul members
Node          Address           Status  Type    Build  Protocol  DC   Segment
sage-tier1-1  <IP_ADDRESS>:8301   alive   server  1.7.8  2         dc1  <all>
client-21     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-22     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-23     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-24     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-25     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-26     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-27     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-28     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-01   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-02   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-03   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
sage-tier1-2  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-1  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-2  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-4  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier3-1  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
visnode-01    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-02    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-03    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>

The same error from hax when running mcp:
Jan 12 10:33:41 client-22 hare-hax[20538]: 2021-01-12 10:33:41,887 [DEBUG] {qconsumer} Preparing the reply for HaNvecGetEvent (nvec size = 976)
Jan 12 10:33:41 client-22 hare-hax[20538]: 2021-01-12 10:33:41,888 [DEBUG] {qconsumer} Attempting to invoke the repeatable call: get_conf_obj_status
Jan 12 10:33:41 client-22 hare-hax[20538]: 2021-01-12 10:33:41,936 [ERROR] {qconsumer} **ERROR**
Jan 12 10:33:41 client-22 hare-hax[20538]: Traceback (most recent call last):
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 40, in wrapper
Jan 12 10:33:41 client-22 hare-hax[20538]: return fn(*args, **kwargs)
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 230, in ha_nvec_get_reply
Jan 12 10:33:41 client-22 hare-hax[20538]: n.note.no_id.f_key) !=
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 117, in wrapper
Jan 12 10:33:41 client-22 hare-hax[20538]: result = f(*args, **kwds)
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 389, in get_conf_obj_status
Jan 12 10:33:41 client-22 hare-hax[20538]: return self.get_node_health(node_name)
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 410, in get_node_health
Jan 12 10:33:41 client-22 hare-hax[20538]: return str(node_data[0]['Status'])
Jan 12 10:33:41 client-22 hare-hax[20538]: IndexError: list index out of range

Now all the nodes are alive, but the issue is still there:
$ consul members
Node          Address           Status  Type    Build  Protocol  DC   Segment
sage-tier1-1  <IP_ADDRESS>:8301   alive   server  1.7.8  2         dc1  <all>
client-21     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-22     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-23     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-24     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-25     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-26     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-27     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
client-28     <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-01   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-02   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
datawarp-03   <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
sage-tier1-2  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-1  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-2  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier2-4  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
sage-tier3-1  <IP_ADDRESS>:8301   alive   client  1.7.8  2         dc1  <default>
visnode-01    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-02    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>
visnode-03    <IP_ADDRESS>:8301  alive   client  1.7.8  2         dc1  <default>

The same error from hax when running mcp:
Jan 12 10:33:41 client-22 hare-hax[20538]: 2021-01-12 10:33:41,887 [DEBUG] {qconsumer} Preparing the reply for HaNvecGetEvent (nvec size = 976)
Jan 12 10:33:41 client-22 hare-hax[20538]: 2021-01-12 10:33:41,888 [DEBUG] {qconsumer} Attempting to invoke the repeatable call: get_conf_obj_status
Jan 12 10:33:41 client-22 hare-hax[20538]: 2021-01-12 10:33:41,936 [ERROR] {qconsumer} **ERROR**
Jan 12 10:33:41 client-22 hare-hax[20538]: Traceback (most recent call last):
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 40, in wrapper
Jan 12 10:33:41 client-22 hare-hax[20538]: return fn(*args, **kwargs)
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 230, in ha_nvec_get_reply
Jan 12 10:33:41 client-22 hare-hax[20538]: n.note.no_id.f_key) !=
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 117, in wrapper
Jan 12 10:33:41 client-22 hare-hax[20538]: result = f(*args, **kwds)
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 389, in get_conf_obj_status
Jan 12 10:33:41 client-22 hare-hax[20538]: return self.get_node_health(node_name)
Jan 12 10:33:41 client-22 hare-hax[20538]: File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 410, in get_node_health
Jan 12 10:33:41 client-22 hare-hax[20538]: return str(node_data[0]['Status'])
Jan 12 10:33:41 client-22 hare-hax[20538]: IndexError: list index out of range

After all the client nodes which were offline joined back to the cluster - it started working fine!

After all the client nodes which were offline joined back to the cluster - it started working fine!

@mssawant I've put your patch and added the required typing information to #1455, please have a look.
BTW, I have a couple of concerns regarding the proposed patch.
Perhaps "members" is not safe to use in this context. Here is the snippet from python-consule module, please pay attention to the docstring:
        def members(self, wan=False):
            """
            Returns all the members that this agent currently sees. This may
            vary by agent, use the nodes api of Catalog to retrieve a cluster
            wide consistent view of members.

            For agents running in server mode, setting *wan* to *True* returns
            the list of WAN members instead of the LAN members which is
            default.
            """
            params = []
            if wan:
                params.append(('wan', 1))
            return self.agent.http.get(
                CB.json(), '/v1/agent/members', params=params)

Can we use consul.catalog.nodes() instead? Here is what I see in my singlenode cluster:
Python 3.6.8 (default, Aug  7 2019, 17:28:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from consul import Consul
>>> c = Consul()
>>> c.agent.members()
[{'Name': 'localhost', 'Addr': '<IP_ADDRESS>', 'Port': 8301, 'Tags': {'acls': '0', 'bootstrap': '1', 'build': '1.7.8:9a5a1218', 'dc': 'dc1', 'id': 'dd8a91f6-ca32-30e0-983c-8f309d653045', 'port': '8300', 'raft_vsn': '3', 'role': 'consul', 'segment': '', 'vsn': '2', 'vsn_max': '3', 'vsn_min': '2', 'wan_join_port': '8302'}, 'Status': 1, 'ProtocolMin': 1, 'ProtocolMax': 5, 'ProtocolCur': 2, 'DelegateMin': 2, 'DelegateMax': 5, 'DelegateCur': 4}]
>>> c.catalog.nodes()
('6', [{'ID': 'dd8a91f6-ca32-30e0-983c-8f309d653045', 'Node': 'localhost', 'Address': '<IP_ADDRESS>', 'Datacenter': 'dc1', 'TaggedAddresses': {'lan': '<IP_ADDRESS>', 'lan_ipv4': '<IP_ADDRESS>', 'wan': '<IP_ADDRESS>', 'wan_ipv4': '<IP_ADDRESS>'}, 'Meta': {'consul-network-segment': ''}, 'CreateIndex': 5, 'ModifyIndex': 6}])
>>>

@mssawant I've put your patch and added the required typing information to #1455, please have a look.
BTW, I have a couple of concerns regarding the proposed patch.
Perhaps "members" is not safe to use in this context. Here is the snippet from python-consule module, please pay attention to the docstring:
        def members(self, wan=False):
            """
            Returns all the members that this agent currently sees. This may
            vary by agent, use the nodes api of Catalog to retrieve a cluster
            wide consistent view of members.

            For agents running in server mode, setting *wan* to *True* returns
            the list of WAN members instead of the LAN members which is
            default.
            """
            params = []
            if wan:
                params.append(('wan', 1))
            return self.agent.http.get(
                CB.json(), '/v1/agent/members', params=params)

Can we use consul.catalog.nodes() instead? Here is what I see in my singlenode cluster:
Python 3.6.8 (default, Aug  7 2019, 17:28:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from consul import Consul
>>> c = Consul()
>>> c.agent.members()
[{'Name': 'localhost', 'Addr': '<IP_ADDRESS>', 'Port': 8301, 'Tags': {'acls': '0', 'bootstrap': '1', 'build': '1.7.8:9a5a1218', 'dc': 'dc1', 'id': 'dd8a91f6-ca32-30e0-983c-8f309d653045', 'port': '8300', 'raft_vsn': '3', 'role': 'consul', 'segment': '', 'vsn': '2', 'vsn_max': '3', 'vsn_min': '2', 'wan_join_port': '8302'}, 'Status': 1, 'ProtocolMin': 1, 'ProtocolMax': 5, 'ProtocolCur': 2, 'DelegateMin': 2, 'DelegateMax': 5, 'DelegateCur': 4}]
>>> c.catalog.nodes()
('6', [{'ID': 'dd8a91f6-ca32-30e0-983c-8f309d653045', 'Node': 'localhost', 'Address': '<IP_ADDRESS>', 'Datacenter': 'dc1', 'TaggedAddresses': {'lan': '<IP_ADDRESS>', 'lan_ipv4': '<IP_ADDRESS>', 'wan': '<IP_ADDRESS>', 'wan_ipv4': '<IP_ADDRESS>'}, 'Meta': {'consul-network-segment': ''}, 'CreateIndex': 5, 'ModifyIndex': 6}])
>>>

(answering to myself) c.catalog.nodes() doesn't seem to provide Status or anything similar. So the question remains - how safe can we be if we rely on consul.agent.members()?

(answering to myself) c.catalog.nodes() doesn't seem to provide Status or anything similar. So the question remains - how safe can we be if we rely on consul.agent.members()?

@knekrasov,

In the meantime, can you explain why m0conf/nodes subtree can have not only node names (which are valid to query about from Consul Health) but plain FIDs as well?

Inorder to map incoming requests with FIDs from motr to their corresponding node names.

The problem is quite clear - we need to improve our mypy stubs for python-consul. I'll do that today.

Ah, I did not realize the stubs to restrict the library usage, thank you.
Regarding members(),

Returns all the members that this agent currently sees. This may
vary by agent, use the nodes api of Catalog to retrieve a cluster
wide consistent view of members.

I don't think we have a case where an agent in the cluster is not able to see rest of the agents can happen in case we divide the Consul cluster in data centers but in that case we have wan=True parameter.
Let me try Catalog's nodes api as well with the patch.

@knekrasov,

In the meantime, can you explain why m0conf/nodes subtree can have not only node names (which are valid to query about from Consul Health) but plain FIDs as well?

Inorder to map incoming requests with FIDs from motr to their corresponding node names.

The problem is quite clear - we need to improve our mypy stubs for python-consul. I'll do that today.

Ah, I did not realize the stubs to restrict the library usage, thank you.
Regarding members(),

Returns all the members that this agent currently sees. This may
vary by agent, use the nodes api of Catalog to retrieve a cluster
wide consistent view of members.

I don't think we have a case where an agent in the cluster is not able to see rest of the agents can happen in case we divide the Consul cluster in data centers but in that case we have wan=True parameter.
Let me try Catalog's nodes api as well with the patch.

@knekrasov, @mssawant, I don't quite understand - why do we need to call the members in order to get the node's health? Why not just report that the health is in bad shape (Status is critical) if self.cns.health.node() returns the empty array?

@knekrasov, @mssawant, I don't quite understand - why do we need to call the members in order to get the node's health? Why not just report that the health is in bad shape (Status is critical) if self.cns.health.node() returns the empty array?

Very good question. Perhaps it is safe to consider any node as failed by default. So if Consul doesn't provide anything about the node - then consider it as failing.
@mssawant Do you have objections?

Very good question. Perhaps it is safe to consider any node as failed by default. So if Consul doesn't provide anything about the node - then consider it as failing.
@mssawant Do you have objections?

Did think of it but I am not sure if its safe to assume that the node has failed on getting node status as empty because we don't really know the status of that node which we get from the members data. Actually I think we can do the other way and just invoke is_node_alive() that gives us exact status of the node which can also be logged.
Ideally there also needs to be a watcher for the client nodes as well.

Did think of it but I am not sure if its safe to assume that the node has failed on getting node status as empty because we don't really know the status of that node which we get from the members data. Actually I think we can do the other way and just invoke is_node_alive() that gives us exact status of the node which can also be logged.
Ideally there also needs to be a watcher for the client nodes as well.

@mssawant such check for the nodes is built-in into Consul - it's called serfHealth. That's, actually, what curl http://<IP_ADDRESS>:8500/v1/health/node/<node> returns:
$ curl -s http://<IP_ADDRESS>:8500/v1/health/node/client-21 | jq
[
  {
    "Node": "client-21",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "passing",
...

If node is absent and the array returns [] - I think it would be quite safe and sane to assume that the node is not in a good shape.

@mssawant such check for the nodes is built-in into Consul - it's called serfHealth. That's, actually, what curl http://<IP_ADDRESS>:8500/v1/health/node/<node> returns:
$ curl -s http://<IP_ADDRESS>:8500/v1/health/node/client-21 | jq
[
  {
    "Node": "client-21",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "passing",
...

If node is absent and the array returns [] - I think it would be quite safe and sane to assume that the node is not in a good shape.

@andriytk,
I would actually prefer to get an explicit status (which is available) rather than taking a decision on an empty string. It would have been good if  http://<IP_ADDRESS>:8500/v1/health/node/<node> would have returned something like,
[
  {
    "Node": "client-21",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "critical",
...

for a failed node instead of returning nothing, which is not very intuitive.

@andriytk,
I would actually prefer to get an explicit status (which is available) rather than taking a decision on an empty string. It would have been good if  http://<IP_ADDRESS>:8500/v1/health/node/<node> would have returned something like,
[
  {
    "Node": "client-21",
    "CheckID": "serfHealth",
    "Name": "Serf Health Status",
    "Status": "critical",
...

for a failed node instead of returning nothing, which is not very intuitive.

OK, so what do you suggest doing if the node is not alive in members or is not present in the list of members at all (after some time)? Do you think there should be some special logic to handle this case?

OK, so what do you suggest doing if the node is not alive in members or is not present in the list of members at all (after some time)? Do you think there should be some special logic to handle this case?

Well, couple of cases when a node is not part of members data (I am ignoring the fact that this function is just invoked from Motr),

node name is wrong - log the message and/or return some appropriate error (e.g. EINVAL in c)
node is part of a different data center, in this case the wan=True parameter must be set, I think it is anyway good to use wan=True  - again if node is not present in the members_data, return appropriate error code (e.g. ENOENT in c).

So I think we can extend the logic to return the appropriate error code, caller can handle it accordingly, for Motr if the node is part of configuration and the function returns something equivalent to ENOENT, then instead of FAILED hax can return OFFLINE, I know there's no special handling for OFFLINE as of now but this is what we can do.

Well, couple of cases when a node is not part of members data (I am ignoring the fact that this function is just invoked from Motr),

node name is wrong - log the message and/or return some appropriate error (e.g. EINVAL in c)
node is part of a different data center, in this case the wan=True parameter must be set, I think it is anyway good to use wan=True  - again if node is not present in the members_data, return appropriate error code (e.g. ENOENT in c).

So I think we can extend the logic to return the appropriate error code, caller can handle it accordingly, for Motr if the node is part of configuration and the function returns something equivalent to ENOENT, then instead of FAILED hax can return OFFLINE, I know there's no special handling for OFFLINE as of now but this is what we can do.

@knekrasov , Consul.catalog.nodes() has the same problems as earlier, the result does not include the node details that is left.
[root@ssc-vm-1623:cortx-hare-1] consul members
Node                          Address             Status  Type    Build  Protocol  DC   Segment
ssc-vm-1623.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1624.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1625.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1626.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1627.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1628.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
[root@ssc-vm-1623:cortx-hare-1] consul members
Node                          Address             Status  Type    Build  Protocol  DC   Segment
ssc-vm-1623.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1624.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1625.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1626.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1627.colo.seagate.com  <IP_ADDRESS>:8301  left    server  1.7.8  2         dc1  <all>
ssc-vm-1628.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
[root@ssc-vm-1623:cortx-hare-1] curl http://<IP_ADDRESS>:8500/v1/catalog/nodes
[{"ID":"5b871cb1-78f0-3652-b0e7-08f2770aecc2","Node":"ssc-vm-1623.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":5,"ModifyIndex":6},{"ID":"2354e7f2-ff72-33d9-a9ba-2819eba4fc17","Node":"ssc-vm-1624.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":490,"ModifyIndex":497},{"ID":"30df6a5f-7e08-3b7b-b37d-22964771141e","Node":"ssc-vm-1625.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":484,"ModifyIndex":496},{"ID":"53e641f2-0bde-3d80-a939-36a87ec7d02a","Node":"ssc-vm-1626.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":492,"ModifyIndex":495},{"ID":"ad6c185c-f685-30f6-8b3f-b7c499d81b71","Node":"ssc-vm-1628.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""}

I think for now let's continue with Consul.agent.members().

@knekrasov , Consul.catalog.nodes() has the same problems as earlier, the result does not include the node details that is left.
[root@ssc-vm-1623:cortx-hare-1] consul members
Node                          Address             Status  Type    Build  Protocol  DC   Segment
ssc-vm-1623.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1624.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1625.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1626.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1627.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1628.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
[root@ssc-vm-1623:cortx-hare-1] consul members
Node                          Address             Status  Type    Build  Protocol  DC   Segment
ssc-vm-1623.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1624.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1625.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1626.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
ssc-vm-1627.colo.seagate.com  <IP_ADDRESS>:8301  left    server  1.7.8  2         dc1  <all>
ssc-vm-1628.colo.seagate.com  <IP_ADDRESS>:8301  alive   server  1.7.8  2         dc1  <all>
[root@ssc-vm-1623:cortx-hare-1] curl http://<IP_ADDRESS>:8500/v1/catalog/nodes
[{"ID":"5b871cb1-78f0-3652-b0e7-08f2770aecc2","Node":"ssc-vm-1623.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":5,"ModifyIndex":6},{"ID":"2354e7f2-ff72-33d9-a9ba-2819eba4fc17","Node":"ssc-vm-1624.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":490,"ModifyIndex":497},{"ID":"30df6a5f-7e08-3b7b-b37d-22964771141e","Node":"ssc-vm-1625.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":484,"ModifyIndex":496},{"ID":"53e641f2-0bde-3d80-a939-36a87ec7d02a","Node":"ssc-vm-1626.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""},"CreateIndex":492,"ModifyIndex":495},{"ID":"ad6c185c-f685-30f6-8b3f-b7c499d81b71","Node":"ssc-vm-1628.colo.seagate.com","Address":"<IP_ADDRESS>","Datacenter":"dc1","TaggedAddresses":{"lan":"<IP_ADDRESS>","lan_ipv4":"<IP_ADDRESS>","wan":"<IP_ADDRESS>","wan_ipv4":"<IP_ADDRESS>"},"Meta":{"consul-network-segment":""}

I think for now let's continue with Consul.agent.members().


fix: November phpoffice/phpspreadsheet vulnerabilities

Some additional security issues in phpoffice/phpspreadsheet were fixed in November.
+-------------------+----------------------------------------------------------------------------------+
| Package           | phpoffice/phpspreadsheet                                                         |
| Severity          | high                                                                             |
| CVE               | CVE-2024-48917                                                                   |
| Title             | XXE in PHPSpreadsheet's XLSX reader                                              |
| URL               | https://github.com/advisories/GHSA-7cc9-j4mv-vcjp                                |
| Affected versions | >=3.3.0,<3.4.0|>=2.2.0,<2.3.2|>=2.0.0,<2.1.3|<1.29.4                             |
| Reported at       | 2024-11-18T20:01:46+00:00                                                        |
+-------------------+----------------------------------------------------------------------------------+
+-------------------+----------------------------------------------------------------------------------+
| Package           | phpoffice/phpspreadsheet                                                         |
| Severity          | high                                                                             |
| CVE               | CVE-2024-47873                                                                   |
| Title             | XmlScanner bypass leads to XXE                                                   |
| URL               | https://github.com/advisories/GHSA-jw4x-v69f-hh5w                                |
| Affected versions | >=3.3.0,<3.4.0|>=2.2.0,<2.3.2|>=2.0.0,<2.1.3|<1.29.4                             |
| Reported at       | 2024-11-18T20:01:20+00:00                                                        |
+-------------------+----------------------------------------------------------------------------------+

2.3.3 also incorporates this PR https://github.com/PHPOffice/PhpSpreadsheet/pull/4233
fix #66

Thank you for your contribution, appreciate it. Release on v11.3.1 🚀


Latin-prompt

One of the innovations brought by your work is to have structure-preserving OCR (implemented as whitespacing and newlines) for the documents as part of the prompts.
From the first Figure in the paper I reckoned you gave literally the string "5_" to indicate 5 whitespaces. What is now the most correct way? Doesn't the tokenizer of the LLM just filter away multiple whitespaces?

That's a good question. In fact, the tokenizer is aware of the number of spaces. Here is a simple example.

I hope this is helpful for you.


tlp: 1.0 -> 1.1

https://github.com/linrunner/TLP/blob/1.1/changelog
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build tlp

Failure on aarch64-linux (full log)
Partial log (click to expand)
Package ‘x86_energy_perf_policy-4.9.79’ in /var/lib/gc-of-borg/nix-test-rs-3/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-3/pkgs/os-specific/linux/x86_energy_perf_policy/default.nix:20 is not supported on ‘aarch64-linux’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Partial log (click to expand)
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/bin/run-on-bat: interpreter directive changed from "/bin/sh" to "/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin/sh"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/bin/tlp: interpreter directive changed from "/bin/sh" to "/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin/sh"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/bin/tlp-stat: interpreter directive changed from "/bin/sh" to "/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin/sh"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/bin/.tlp-pcilist-wrapped: interpreter directive changed from "/usr/bin/perl" to "/nix/store/93865mz7pp3r3vjwqwlpr9wjknybj946-perl-5.24.3/bin/perl"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/lib/udev/tlp-usb-udev: interpreter directive changed from "/bin/sh" to "/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin/sh"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/share/tlp-pm/tlp-rf-func: interpreter directive changed from "/bin/sh" to "/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin/sh"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/share/tlp-pm/tlp-functions: interpreter directive changed from "/bin/sh" to "/nix/store/i0ay05pqkbnvpfijm52mmlrp6kmkl80c-bash-4.4-p12/bin/sh"
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1/share/tlp-pm/tpacpi-bat: interpreter directive changed from "/usr/bin/perl" to "/nix/store/93865mz7pp3r3vjwqwlpr9wjknybj946-perl-5.24.3/bin/perl"
checking for references to /tmp/nix-build-tlp-1.1.drv-0 in /nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1...
/nix/store/n7af3jhvd8m1gphf7crmkpdicjcz2hx9-tlp-1.1

Failure on x86_64-darwin (full log)
Partial log (click to expand)
Package ‘tlp-1.1’ in /Users/graham/nix-borg/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-ndnd/pkgs/tools/misc/tlp/default.nix:63 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

Thanks!


Use json string instead

The value expected is a string representation of a json, during load the string is interpreted as json, we need to convert it back to a string

This is the job for the validation: https://www.distributed-ci.io/jobs/60f37b72-a270-4831-8fab-1921987d5309


⚠ Bofrost.at has degraded performance

In 53f0919, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 5861 ms

Resolved: Bofrost.at performance has improved in 2fefe5f after 17 minutes.


Anthenea Stable Release v1.2.1

Release for version vVERSION

[x] edit HISTORY.rst and ensure a proper section is added

[x] edit admin_upgrade.rst to reflect the upgrade instructions

[x] create a git tag v.MAJOR.MINOR.PATCH and git push --tags

[x] create a "Github release` based on the tag with the text
```
All details can be found in the `HISTORY.rst` file.
```

Released

Validation Results
GRCh37
chrmt_singleton: OK
comphet_singleton: OK
comphet_trio: OK
denovo_trio: OK
het_singleton: OK
het_trio: OK
hom_singleton: OK
hom_trio: OK

GRCh38
chrmt_singleton: OK
comphet_singleton: OK
comphet_trio: OK
denovo_trio: OK
het_singleton: OK
het_trio: OK
hom_singleton: OK
hom_trio: OK


[WIP]Remove support for nvdxml

Related to: #139 Remove NVD XML support

It seems to be good to delete this part too...
https://github.com/kotakanbe/go-cve-dictionary/blob/6f05125ce74c6be8b1246873172a273e878480be/db/rdb.go#L86

plz check it :)
https://github.com/segatomo/go-cve-dictionary/pull/1

@segatomo
A list sub command also has xml related code.
https://github.com/segatomo/go-cve-dictionary/blob/remove_xml/commands/list.go#L99
Please remove it.


Update MapOptions.java

It should be possible to create MapOptions in type-safe mode, not only as MapOptions<Object, Object>(); which provided by the default method

You can do it like MapOptions<K, V> mo = MapOptions.defaults();

you are right


Loading app in welcome viewmodel fixes #622

fix(starterkit): Issue 622
Not sure there is anything to test, and can't define the problem better than "down the track it breaks RequireJS, perhaps when relying on Almond". It could probably be duplicated, but is it worth the effort?

@EisenbergEffect this PR can be merged


Workaround for pip-tools issue upstream

First guess was to pin pip to 22.0.4, but I wondered: What exactly is
pip-sync doing here? There's no virtualenv to synchronize since we get a
fresh environment in each workflow run.

@jstvz Are you sure? Even with the pip cache from actions/setup-python?

Even with the pip cache from actions/setup-python?

Oh, I suppose that's a cache of downloaded distributions, not an installed virtualenv

@Br4nd0R Would you sanity check this tomorrow?  (The release will be blocked otherwise)
If we do need to run pip-sync in the runner after all we can just pin pip:
diff --git a/Makefile b/Makefile
index e3981f0f0..9b6dcb7bc 100644
--- a/Makefile
+++ b/Makefile
@@ -109,7 +109,7 @@ update-deps:
 	pip-compile --upgrade requirements/dev.in

 dev-install:
-	python -m pip install --upgrade pip pip-tools setuptools
+	python -m pip install --upgrade pip==22.0.4 pip-tools setuptools
 	pip-sync requirements/*.txt
 	python -m pip install -e .


Upgrade SDK, update diamond address

Type of change
Fix
Does it contain breaking changes?
No
Description
Upgrades the SDK. Since it now returns the transaction, we can wait for the transaction to be mined.
Also updates the Diamond address.

need to use toast.contractTransaction


Add default TZ to parse datetime for age and expiration value_type

Fix for #1019

the tools can have different dependencies, without installing their respective reqs its unclear if their tests will run. we can handle those in future case by case. lgtm

looks good to me, thanks
per our readme, would you mind signing our contributor license agreement.
https://docs.google.com/forms/d/e/1FAIpQLSfwtl1s6KmpLhCY6CjiY8nFZshDwf_wrmNYx1ahpsNFXXmHKw/viewform

@kapilt I've completed the license agreement


AMD code generator setting exports.prop1 = exports.prop2 = [...] void 0 blows stack

We are upgrading from TypeScript to 3.1 to 4.4.
We have a source file that itself is an auto-generated file that relates to our data-model and hence it contains approximately 3,700 classes at present. Each class basically just contains a bunch of constant strings for various purposes.
Our compilerOptions object defines a target of es5 and a module of amd.
The compiled/transpiled code (JS output file) generates (amongst other things) one line of code that is 70,000 bytes long that basically reads like this:
exports.prop1 = exports.prop2 = exports.prop3 [....] = void 0;
Of course this continues for 3,700 properties.
As soon as we try load this code (just that line with a bit of scaffolding for testing), even via a base

Seems like a V8 bug.

This is legal JS and doesn't have any intrinsic reason to crash. We could consider a workaround here but I'd prefer this be fixed in the runtime instead of us needing to implement some special behavior.


Fix deprecated warnings in themes

This involves some PR openings ...

See Deprecated errors when running this site
Fix theme, bump min_version => 0.14 and create PR

See https://github.com/spf13/hugo/issues/1172

I have created a pull request for all deprecation errors in the theme site script.


MAT-7725 stratification display issue after STU4 change

MADiE PR
Jira Ticket: MAT-0000
(Optional) Related Tickets:
Summary
All Submissions

[ ] This PR has the JIRA linked.
[ ] Required tests are included
[ ] No extemporaneous files are included (i.e Complied files or testing results)
[ ] This PR is into the correct branch.
[ ] All Documentation as needed for this PR is Complete (or noted in a TODO or other Ticket)
[ ] Any breaking changes or failing automation are noted by placing a comment on this PR.

DevSecOps
If there is a question if this PR has a security or infrastructure impact, please contact the Security or DevOps engineer assigned to this project to discuss it further.

[ ] This PR has NO significant security impact (i.e Changing auth methods, Adding a new user type, Adding a required but vulnerable package)
[ ] All CDN/Web dependencies are hosted internally (i.e MADiE-Root Repo)

Reviewers
By Approving this PR you are attesting to the following:

Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
The tests appropriately test the new code, including edge cases
If you have any concerns they are brought up either to the developer assigned, security engineer, or leads

@adongare I added the fix for the duplicates/test.
I didn't add the add the additional fix to simplify state from that useEffect since it broke more than expected.


expose api metadata handlers

Codecov Report

Merging #1494 (b28e78b) into master (8804241) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1494      +/-   ##
==========================================
- Coverage   38.52%   38.50%   -0.03%
==========================================
  Files         164      164
  Lines       13242    13244       +2
==========================================
- Hits         5101     5099       -2
- Misses       7405     7406       +1
- Partials      736      739       +3

Impacted Files
Coverage Δ

api/controller.go
32.94% <100.00%> (+0.07%)
:arrow_up:

pyramid/tier_fs.go
56.76% <0.00%> (-1.75%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8804241...b28e78b. Read the comment docs.


Login error

Stuck with this

and your problem is not login u didnt updated ip

<IP_ADDRESS>

fixed in #72


Add support for k8s 1.11

[x] add support for k8s 1.11
[x] update k8s-1.8, 1.9 and 1.11 to newer versions
[x] fix k8s master build

/lgtm

/approve

/assign jellonek

/lgtm

/lgtm


Graceful error handling

At the moment it's using a fail-fast model. (Failing at the first sight of any issues, even recoverable ones.)

Handles and reports skipped over files.
Reports crashes in PR status check.
Fails are contained to single PRs.
Restarts on hard failure.


trait for new_ads1015() structure?

I am trying to define a function with code for reading a couple of ads1015s  so the loop is easier to read. The difficulty is that this requires the type for the structure returned by Ads1x1x::new_ads1015() which is used to read channels. Since the structure type is messy I would like to use a trait, but can only find traits for channels, not for the object doing the reading. The code below is a simplified extract which illustrates the problem.

Click to expand
// read_adc for blue pill stm32f103

#![deny(unsafe_code)]
#![no_std]
#![no_main]

#[cfg(debug_assertions)]
use panic_semihosting as _;

#[cfg(not(debug_assertions))]
use panic_halt as _;

use cortex_m_rt::entry;

use ads1x1x::{channel as AdcChannel, Ads1x1x, FullScaleRange, SlaveAddr};

use nb::block;

use stm32f1xx_hal::{
    delay::Delay,
    device::I2C2,
    i2c::{BlockingI2c, DutyCycle, Mode, Pins},
    pac::{CorePeripherals, Peripherals},
    prelude::*,
};

fn setup() -> (BlockingI2c<I2C2, impl Pins<I2C2>>, Delay) {
    let cp = CorePeripherals::take().unwrap();
    let p = Peripherals::take().unwrap();

    let mut rcc = p.RCC.constrain();
    let clocks = rcc.cfgr.freeze(&mut p.FLASH.constrain().acr);

    let mut gpiob = p.GPIOB.split(&mut rcc.apb2); // for i2c scl and sda

    // can have (scl, sda) using I2C1  on (PB8, PB9 ) or on  (PB6, PB7)
    //     or   (scl, sda) using I2C2  on (PB10, PB11)

    let i2c = BlockingI2c::i2c2(
        p.I2C2,
        (
            gpiob.pb10.into_alternate_open_drain(&mut gpiob.crh), // scl on PB10
            gpiob.pb11.into_alternate_open_drain(&mut gpiob.crh), // sda on PB11
        ),
        //&mut afio.mapr,  need this for i2c1 but not i2c2
        Mode::Fast {
            frequency: 100_000.hz(),
            duty_cycle: DutyCycle::Ratio2to1,
        },
        clocks,
        &mut rcc.apb1,
        1000,
        10,
        1000,
        1000,
    );

    let delay = Delay::new(cp.SYST, clocks);

    (i2c, delay) // return tuple (i2c, delay)
}

fn read_adc(adc_a: impl Ads1x1x, adc_b: impl Ads1x1x) -> (i16, i16, i16, i16, [i16; 3]) {

    // signature needs type of Ads1x1x::new_ads1015() or trait to use impl

    let bat_ma =
        block!(adc_a.read(&mut AdcChannel::DifferentialA1A3)).unwrap_or(8091);
    let load_ma =
        block!(adc_a.read(&mut AdcChannel::DifferentialA2A3)).unwrap_or(8091);

    // toggle FullScaleRange to measure battery voltage, not just diff across shunt resistor

    adc_a.set_full_scale_range(FullScaleRange::Within4_096V).unwrap();
    let bat_mv = block!(adc_a.read(&mut AdcChannel::SingleA0)).unwrap_or(8091);
    adc_a.set_full_scale_range(FullScaleRange::Within0_256V).unwrap();

    // second adc
    let values_b = [
        block!(adc_b.read(&mut AdcChannel::SingleA0)).unwrap_or(8091),
        block!(adc_b.read(&mut AdcChannel::SingleA1)).unwrap_or(8091),
        block!(adc_b.read(&mut AdcChannel::SingleA2)).unwrap_or(8091),
    ];

    let temp_c = block!(adc_b.read(&mut AdcChannel::SingleA3)).unwrap_or(8091);

    (bat_mv, bat_ma, load_ma, temp_c, values_b)
}

#[entry]
fn main() -> ! {

    let (i2c, mut delay) = setup();

    let manager = shared_bus::BusManager::<cortex_m::interrupt::Mutex<_>, _>::new(i2c);

    let mut adc_a = Ads1x1x::new_ads1015(manager.acquire(), SlaveAddr::Alternative(false, false)); //addr = GND
    let mut adc_b = Ads1x1x::new_ads1015(manager.acquire(), SlaveAddr::Alternative(false, true)); //addr =  V

    adc_a
        .set_full_scale_range(FullScaleRange::Within0_256V)
        .unwrap();
    adc_b
        .set_full_scale_range(FullScaleRange::Within4_096V)
        .unwrap();

    loop {
        let (_bat_mv, _bat_ma, _load_ma, _temp_c, _values_b) = read_adc(adc_a, adc_b);

        // do other things
        delay.delay_ms(2000_u16); // sleep for 2s
    }
}

Since I don't have the trait right it gives
fn read_adc(adc_a: impl Ads1x1x, adc_b: impl Ads1x1x) -> (i16, i16, i16, i16, [i16; 3]) {
                          ^^^^^^^ not a trait

Is there a way to specify the trait to do this, or just a better way to approach the whole thing?

If you split read_adc() into two functions so that you can change the FSR outside of it, you could have two functions that simply take the ADC as an embedded_hal::adc::OneShot trait.
Would that be a solution for you?

Yes, this would work. But now I'm stuck trying to figure out the type arguments
error[E0107]: wrong number of type arguments: expected 3, found 0
  --> examples/zz.rs:64:27
   |
64 | fn read_adc_a(adc_a: impl embedded_hal::adc::OneShot) -> (i16, i16, i16) {
   |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 3 type arguments

(I expect there should be an easy way to find  type arguments in documentation or compiler error messages, but I have not figure it out.)

Ah, that solution I proposed cannot work because the OneShot trait is channel-specific and you want to pass the whole ADC and measure several channels inside the function.
You can use your function as initially proposed if you define the concrete types of the ADC.
An example of doing this is in this example.
You will only need to replace I2cdev with something like BlockingI2c<I2C2, ...> although putting the pin types there will bloat the type up a bit.
I have a look at adding a trait to simplify this.

I have added the DynamicOneShot trait to ease the usage in functions in release 0.2.2. See this example.
Closing.

Thanks for adding this trait. It works well except when  I try to use the method .set_full_scale_range in the read function.  (I do this because I am measuring a voltage across a shunt resistor and also to ground.)  I'm not sure if I just don't have the proper syntax figured out, I am a newbie still, or if there is a problem de-referencing the method. I have tried various syntax without luck:
113 |     adc_a.set_full_scale_range(FullScaleRange::Within4_096V).unwrap();
    |           ^^^^^^^^^^^^^^^^^^^^ method not found in `&mut A`

113 |     *adc_a.set_full_scale_range(FullScaleRange::Within4_096V).unwrap();
    |            ^^^^^^^^^^^^^^^^^^^^ method not found in `&mut A`

113 |     (*adc_a).set_full_scale_range(FullScaleRange::Within4_096V).unwrap();
    |              ^^^^^^^^^^^^^^^^^^^^ method not found in `A`

A stripped down version of the code follows.

Click to expand
//  setup for bluepill   stm32f1xx_hal

#![deny(unsafe_code)]
#![no_std]
#![no_main]

#[cfg(debug_assertions)]
use panic_semihosting as _;

#[cfg(not(debug_assertions))]
use panic_halt as _;

use cortex_m_rt::entry;

use ads1x1x::{Ads1x1x, ChannelSelection, DynamicOneShot, FullScaleRange, SlaveAddr};

use core::fmt::Write;

use embedded_graphics::{
    mono_font::{ascii::FONT_8X13, MonoTextStyle, MonoTextStyleBuilder}, //FONT_6X10  FONT_8X13
    pixelcolor::BinaryColor,
    prelude::*,
    text::{Baseline, Text},
};

use ssd1306::{mode::BufferedGraphicsMode, prelude::*, I2CDisplayInterface, Ssd1306};

use nb::block;

pub trait LED {
    fn on(&mut self) -> ();
    fn off(&mut self) -> ();

    // default methods
    fn blink(&mut self, time: u16, delay: &mut Delay) -> () {
        self.on();
        delay.delay_ms(time);
        self.off()
    }
}

use stm32f1xx_hal::{
    delay::Delay,
    device::I2C2,
    gpio::{gpioc::PC13, Output, PushPull},
    i2c::{BlockingI2c, DutyCycle, Mode, Pins},
    pac::{CorePeripherals, Peripherals},
    prelude::*,
};

fn setup() -> (BlockingI2c<I2C2, impl Pins<I2C2>>, impl LED, Delay) {
    let cp = CorePeripherals::take().unwrap();
    let p = Peripherals::take().unwrap();

    let rcc = p.RCC.constrain();
    let clocks = rcc.cfgr.freeze(&mut p.FLASH.constrain().acr);

    let mut gpiob = p.GPIOB.split(); // for i2c scl and sda

    let i2c = BlockingI2c::i2c2(
        p.I2C2,
        (
            gpiob.pb10.into_alternate_open_drain(&mut gpiob.crh), // scl on PB10
            gpiob.pb11.into_alternate_open_drain(&mut gpiob.crh), // sda on PB11
        ),
        Mode::Fast {
            frequency: 100_000.hz(),
            duty_cycle: DutyCycle::Ratio2to1,
        },
        clocks,
        1000,
        10,
        1000,
        1000,
    );

    let delay = Delay::new(cp.SYST, clocks);

    // led
    let mut gpioc = p.GPIOC.split();
    let led = gpioc.pc13.into_push_pull_output(&mut gpioc.crh); // led on pc13

    impl LED for PC13<Output<PushPull>> {
        fn on(&mut self) -> () {
            self.set_low()
        }
        fn off(&mut self) -> () {
            self.set_high()
        }
    }

    (i2c, led, delay)
}

pub fn read_all<E, A: DynamicOneShot<Error = E>>(adc_a: &mut A, adc_b: &mut A
      ) -> (i16, i16, i16, i16, [i16; 3]) {
    let scale_cur = 10; // calibrated to get mA/mV depends on FullScaleRange above and values of shunt resistors
    let scale_a = 2; // calibrated to get mV    depends on FullScaleRange
    let scale_b = 2; // calibrated to get mV    depends on FullScaleRange

    let scale_temp = 5; //divides
    let offset_temp = 50;

    let bat_ma  = block!(adc_a.read(ChannelSelection::DifferentialA1A3)).unwrap_or(8091) / scale_cur;
    let load_ma = block!(adc_a.read(ChannelSelection::DifferentialA2A3)).unwrap_or(8091) / scale_cur;

    // toggle FullScaleRange to measure battery voltage, not just diff across shunt resistor
    // set FullScaleRange to measure expected max voltage.
    // This is very small for diff across low value shunt resistors
    //   but up to 5v for single pin with usb power.
    // +- 6.144v , 4.096v, 2.048v, 1.024v, 0.512v, 0.256v

    adc_a.set_full_scale_range(FullScaleRange::Within4_096V).unwrap();
    let bat_mv = block!(adc_a.read(ChannelSelection::SingleA0)).unwrap_or(8091) * scale_a;
    adc_a.set_full_scale_range(FullScaleRange::Within0_256V).unwrap();

    // second adc
    let values_b = [
        block!(adc_b.read(ChannelSelection::SingleA0)).unwrap_or(8091) * scale_b,
        block!(adc_b.read(ChannelSelection::SingleA1)).unwrap_or(8091) * scale_b,
        block!(adc_b.read(ChannelSelection::SingleA2)).unwrap_or(8091) * scale_b,
    ];

    let temp_c = block!(adc_b.read(ChannelSelection::SingleA3)).unwrap_or(8091) / scale_temp
        - offset_temp;

    (bat_mv, bat_ma, load_ma, temp_c, values_b )
}

fn display<S>(
    bat_mv: i16,
    bat_ma: i16,
    load_ma: i16,
    temp_c: i16,
    values_b: [i16; 3],
    text_style: MonoTextStyle<BinaryColor>,
    disp: &mut Ssd1306<impl WriteOnlyDataCommand, S, BufferedGraphicsMode<S>>,
) -> ()
where
    S: DisplaySize,
{
    let mut lines: [heapless::String<32>; 4] = [
        heapless::String::new(),
        heapless::String::new(),
        heapless::String::new(),
        heapless::String::new(),
    ];

    // Many SSD1306 modules have a yellow strip at the top of the display, so first line may be yellow.
    // it is now possible to use \n in place of separate writes, with one line rather than vector.
    write!(lines[0], "bat:{:4}mV{:4}mA", bat_mv, bat_ma).unwrap();
    write!(lines[1], "load:    {:5}mA", load_ma).unwrap();
    write!(
        lines[2],
        "B:{:4} {:4} {:4}",
        values_b[0], values_b[1], values_b[2]
    )
    .unwrap();
    write!(lines[3], "temperature{:3} C", temp_c).unwrap();

    disp.clear();
    for i in 0..lines.len() {
        // start from 0 requires that the top is used for font baseline
        Text::with_baseline(
            &lines[i],
            Point::new(0, i as i32 * 16),
            text_style,
            Baseline::Top,
        )
        .draw(&mut *disp)
        .unwrap();
    }
    disp.flush().unwrap();
    ()
}

#[entry]
fn main() -> ! {
    let (i2c, mut led, mut delay) = setup();

    let manager = shared_bus::BusManager::<cortex_m::interrupt::Mutex<_>, _>::new(i2c);
    let interface = I2CDisplayInterface::new(manager.acquire());

    let mut disp = Ssd1306::new(interface, DisplaySize128x64, DisplayRotation::Rotate0)
        .into_buffered_graphics_mode();
    disp.init().unwrap();

    let text_style = MonoTextStyleBuilder::new()
        .font(&FONT_8X13) //.&FONT_6X10  &FONT_8X13
        .text_color(BinaryColor::On)
        .build();

    Text::with_baseline("Display initialized ...", Point::zero(), text_style, Baseline::Top)
        .draw(&mut disp)
        .unwrap();

    let mut adc_a = Ads1x1x::new_ads1015(manager.acquire(), SlaveAddr::Alternative(false, false)); //addr = GND
    let mut adc_b = Ads1x1x::new_ads1015(manager.acquire(), SlaveAddr::Alternative(false, true)); //addr =  V

    adc_a.set_full_scale_range(FullScaleRange::Within4_096V).unwrap();
    adc_b.set_full_scale_range(FullScaleRange::Within4_096V).unwrap();

    loop {
        // Blink LED to check that everything is actually running.
        led.blink(10_u16, &mut delay);

        let (bat_mv, bat_ma, load_ma, temp_c, values_b ) = read_all(&mut adc_a, &mut adc_b);

        display(bat_mv, bat_ma, load_ma, temp_c, values_b, text_style, &mut disp);

        delay.delay_ms(2000_u16); // sleep for 2s
    }
}

Ah, I forgot about that. Could you just use the actual driver type then, instead of this trait?
You just need to use the concrete type (Depending on your IDE, it might help you there) like here
Then you can use it easily in a function like here
I would start with adc_a only.

I tried using the actual driver type, switching from linux to no std, etc.  I have
type Adc = Ads1x1x<I2cInterface<BlockingI2c<I2C2, impl Pins<I2C2>>>, Ads1015, Resolution16Bit, ads1x1x::mode::OneShot>;

but then I get
`impl Trait` in type aliases is unstable

and it looks downhill from there. I guess I have to spell out the whole concrete type for every hal. I have a reasonable work around just leaving that one read in the main loop ( I just have to disambiguate), so I think I will leave it like that until there is an easy way to make it clean.

Ok, I removed DynamicOneShot and `ChannelSelection' from all my examples and I think everything is working. (Not yet tested on hardware but at least compiling.)


The RendezvousInformation flag generated in manual code and QR code is invalid.

Problem

The 'RendezvousInformation' flag generated in the manual code  is  '0'. But according to the spec 0 means device can be discovered via softAP, since SoftAP is not in V1.0 scope, so this value is invalid.

The 'RendezvousInformation' flag generated in the QR code  is  '2' when we launch all-clusters-app using --wifi option which is invalid, as according to the spec the expected value should be  1 while launching the all-clusters-app using --wifi option.

Please find the attached DUT log file
DUT_log.txt

@PSONALl PTAL

@sowmyassp As per the specs RendezvousInformation flag is set for modes:
1  -> SoftAp
2  -> BLE
4  -> on/network

@sowmyassp I think there is quite a confusion here, the values of Rendezvous Information flags are correct in the code as per the specs, and 0 in case of manual pairing represents that the Device does not support any method for rendezvous.
and you can send --capabilities argument while running the application set to the above-mentioned values of Rendezvous flags.  In the second case If the Rendezvous Information flag is set to 2 then the device must be using BLE for discovering

can send --capabilities argument while running the application set to the above-mentioned values of Rendezvous flags

e.g. for setting SoftAp-mode you need to send the command:
sudo ./chip-all-clusters-app --capabilities 1

@sowmyassp Closing this issue wrt Sonali's comment. Please feel free to re-open in case of anything.

@PSONALl @dhrishi per @sowmyassp 's comment, here is the excerpt from spec, 0 means device supports Soft AP, which is not a POR in V1.0.

Also for the 2nd scenario, when an all cluster app on raspi is launched with '--wifi' flag, the DUT does BLE advertisement and commissioner is expected to look for BLE advertisement. Since the QR code generated shows Rendezvous flag to 2, it misleads commissioner to look for the device on the IP network. So when the app is launched with this flag , The Rendezvous flag should be 1.

@cjandhyala  in the table above, first column, bits indicate which bit is set and not the value of flag and the description explains the meaning when the bit is 0 and 1
Thus,

if bit '0'  is set to 1 (i.e, value of flag = 001 = 1) then the device supports SoftAp
if bit '0'  is set to 0 (i.e, value of flag = 000 = 0) means the device does not support any rendezvous mode
Similarly in the case of --wifi flag when DUT does BLE advertisement when bit 1 is set i.e, the value of Rendezvous flag is = 010 = 2
So the value generated for the Rendezvous flag in manual code and QR code is correct


Backup function will not work on Windows due to format of the backup filename

I have been unable to use the SDK to download a backup of the KEMP as it throws an exception. The function joins the folder location to a filename which the function automatically generates. The function uses a time function and format to name the log file and uses a colon in the log file name. This will not work on Windows systems as the colon is not allowed in file names or folders.
In 'python_kemptech_api.models'
def backup(self, location='backups'):
    if not os.path.exists(location):
        os.makedirs(location)
    file_name = os.path.join(location, "{}_{}.backup".format(
        self.ip_address, time.strftime("%Y-%m-%d_%H:%M:%S")))

This needs to be modified to not include colons.

Hey @matthew-hickok thanks for letting us know. We'll take a look at this soon and get back to you. In the interim, feel free to hack it out, simply by changing
        self.ip_address, time.strftime("%Y-%m-%d_%H:%M:%S")))

to something like
        self.ip_address, time.strftime("%Y-%m-%d_%H-%M-%S")))

@matthew-hickok we did some research and it looks like there are some other bits in the lib that are linux specific. We'll look to this issue as a feature request to support windows.

@Community if you'd like Windows support please +1 this thread so that we can identify interested parties.


chore: update package.json

For some reason, the CI updates the package version to 1.0.0-alpha.4.
This PR attempts to address it by updating the version to 1.0.0-alpha.5, but this doesn't solve the underlying issue this only attempts to allow the release to go through.

Pull request title looks good 👍!
If this pull request gets merged, it will not cause a new release of the software. Example: If this project's latest release version is 1.0.0. If this pull request gets merged in, the next release of this project will be 1.0.0. This pull request is not a breaking change.
All merged pull requests will eventually get deployed. But some types of pull requests will trigger a deployment (such as features and bug fixes) while some pull requests will wait to get deployed until a later time.
To merge this pull request, add the label Ready to merge to this pull request and I'll merge it for you.

This project uses a special format for pull requests titles. Expand this section to learn more (expand by clicking the ᐅ symbol on the left side of this sentence)...

This project uses a special format for pull requests titles. Don't worry, it's easy!
This pull request title should be in this format:
<type>: short description of change being made

If your pull request introduces breaking changes to the code, use this format:
<type>!: short description of breaking change

where <type> is one of the following:

feat: - A feature is being added or modified by this pull request. Use this if you made any changes to any of the features of the project.

fix: - A bug is being fixed by this pull request. Use this if you made any fixes to bugs in the project.

docs: - This pull request is making documentation changes, only.

refactor: - A change was made that doesn't fix a bug or add a feature.

test: - Adds missing tests or fixes broken tests.

style: - Changes that do not effect the code (whitespace, linting, formatting, semi-colons, etc)

perf: - Changes improve performance of the code.

build: - Changes to the build system (maven, npm, gulp, etc)

ci: - Changes to the CI build system (Travis, GitHub Actions, Circle, etc)

chore: - Other changes to project that don't modify source code or test files.

revert: - Reverts a previous commit that was made.

Examples:
feat: edit profile photo
refactor!: remove deprecated v1 endpoints
build: update npm dependencies
style: run formatter

Need more examples? Want to learn more about this format? Check out the official docs.
Note: If your pull request does multiple things such as adding a feature and makes changes to the CI server and fixes some bugs then you might want to consider splitting this pull request up into multiple smaller pull requests.

General feedback  - It is good to add a description in a PR. Helps the reviewer get a gist of the updates.

:tada: This PR is included in version 1.0.0-alpha.5 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0-beta.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


build navigation UX bug

Describe the bug
If you're on the latest build, and you are navigating to previous builds. it's way too easy to accidentally click "redeploy this version". It's really annoying -- I've done it too many times to count now.
To Reproduce
Steps to reproduce the behavior:

Go to latest build of any project that has current builds
Click on the < button to go to previous build
The cursor is now on the "Redeploy this version" button

Expected behavior
The mouse cursor should stay on the back arrow. maybe move "Redploy this version" somewhere else.
Additional context
See this screen recording:

@stefangomez Thanks for the feedback

This is fixed. @stefangomez


input with content

When the page loads with the input is already filled (edit form) does not display maxlength, it works with "alwaysShow" however I would like it to appear only in the focus event and every field, is there any way to do this?

Maybe you could initialize maxlength on the first focus of the input?
Let me know if that works, I'm going to close this issue for now as it's not related to the plugin itself.


Revert "delete fast reassemble code"

Reverts flutter/flutter#102842

Could you please leave a quick note about WHY something was reverted in the PR description of a revert for people who later look at these and wonder why this failed? Thanks!

There were test failures on CI, for example: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux tool_integration_tests_4_4/4243/overview.
The rest are flagged on https://flutter-dashboard.appspot.com/#/build?taskFilter&authorFilter&messageFilter&hashFilter=ec8693e80bf2c9f31c06d9832ebb4bccc5274440&showMac=true&showWindows=true&showiOS=true&showLinux=true&showAndroid=true&showStaging=false&repo=flutter&branch=master


Vyperdatum 0.1.8 - Add extra regions

I'll look into this a bit more tomorrow.  Only thing I see missing currently is a DatumData.set_custom_directory similar to set_vdatum_directory but for the custom path.

Thanks.  My thought was to leave that alone (so just editing the config directly) until we had decided which path to use for incorporating multiple version of VDatum.

It's funny, I was sure that I added tif support as a viable option for region grids, but I didn't I guess.  I only added it as a discoverable grid type.  Not added in an operational sense.  I'll work on that.

I think this is probably fine.  it ensures that the return from get_transformation_pipeline will always have the grids with the correct file extensions.  It is annoying that the pipeline starts out all .gtx, but I don't think it hurts anything.
Your custom region lacks a tss, so I can't test everything.  But it appears to work otherwise.


Expand RestElement to support all patterns

This is a followup to #236 that brings RestElement into full spec compliance.

This is ready to merge and hopefully uncontroversial. :smile:


Accept List.Nonempty inputs so you don't have to return Maybes everywhere

That's true! The dependency-free way to do nonempty lists is:
(Type, List Type) where the first item is in the tuple and the list can be empty as usual.
Lots of us prefer the syntax sugar in the Nonempty package though (look at the code and you'll see it's just a wrapper over exactly that - not much to the "library"), and with Elm 0.19's forced semantic versioning, tree shaking and inlining, there's not much point in "avoiding dependencies". (None of the wrappers make it into the compiled production code)

I finally got around to this one.
In tag 3.0.0 there is a separate module for nonempty lists: List.Nonempty.Statistics.
Functions for normal lists are now in List.Statistics.


[FEATURE] Evers Single Remote availa

?

Sorry pressed enter by accident
Make it possible to make every single remote show up. Means that in apple remotes you can control every single devices singularly without action and that all of them show up at one.
I know this doesn't work yet because it doesn't work with only one homebridge server but In a long run it would make the Harmony hub even better.

like exposed in #53 you mean ?

Yes didnt See it before


Update layers/meta-balena digest to 4ec7add

This PR contains the following updates:

Package
Update
Change

layers/meta-balena
digest
80ca81f -> 4ec7add

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Can one of the admins verify this patch?


add example using Create React App

Reference [issue](https://github.com/facebookincubator/create-react-app/issues/737

Current coverage is 93.04% (diff: 100%)

Merging #593 into master will not change coverage

@@             master       #593   diff @@
==========================================
  Files            11         11
  Lines           503        503
  Methods          15         15
  Messages          0          0
  Branches         90         90
==========================================
  Hits            468        468
  Misses           12         12
  Partials         23         23

Powered by Codecov. Last update 8a775a3...753e877


Crash on database server down

Environment
Knex version: 2.5.1
Database + version: 10.2.14-MariaDB
OS: CentOS Linux 7 (Core)
Bug
When DB server is down I get this error in my node.js (v18) code and app crashes:
  errno: -111,   code: 'ECONNREFUSED',   syscall: 'connect',   address: '::1',   port: 3306,   fatal: true
I guess there should be callback to handle this situation. I don't want my app to crash. I want it to wait for DB server to come up.

For now we can handle this by using a catch all unhandled promises handler.


Selecting seconds

Currently your time only allows for the use of hours and minutes and I was hoping there was an easy way to add seconds to the selection/display?

@symposiumjim that's some deep detail, I will try to add support to it in the ClockView.

Thanks! I am writing a reporting interface for firefighters/ems/first responders and the gov likes their detail :) I'll send some Paypal beer money


add init file to openedx dir

Add init.py to openedx directory
I ran into an ImportError when testing the site configuration client on devstack and realized it's because of the missing init file.
After adding the init file, I pip installed this branch from devstack, ran ./manage.py lms shell and confirmed that we are no longer getting the error below:
ImportError: No module named 'site_config_client.openedx'

@OmarIthawi
The shell no longer errors but emits the following messages:
root@lms:/edx/app/edxapp/edx-platform# ./manage.py lms shell
Not initializing Google storage bucket due to missing settings parameter.
Not initializing SITE_CONFIG_CLIENT due to missing parameter.

☝ this is after I added settings.SITE_CONFIG_BASE_URL and settings.SITE_CONFIG_API_TOKEN to edx-platform/openedx/core/djangoapps/appsembler/settings/settings/devstack_lms.py. Is there a missing step to initialize SITE_CONFIG_CLIENT?
Thank you!

Thank you so much @estherjsuh! I've had this bug in my devstack yesterday!


Documentation outdated?

I tried using this package with Xamarin.Froms but it feels like the documentation is outdated. I just wanted to ask if im correct.

We've never really documented XNFC because we only used it for one internal project. Soon, we should hopefully have time to rewrite XNFC with other users in mind.

Hello, so say I want to use this on a Xamarin.Forms project.
Is there a sample guide you can provide? Or you don't recommend using it at this stage now?

I don’t currently recommend using it, though we’re going to be rewriting it at some point so I’ll update this issue when that’s happened


how to run it without caddy or nginx?

on doc https://piped-docs.kavin.rocks/docs/self-hosting/ there is only example to run it with reverse proxy.
configure-instance will raise error if i enter empty string for reverseproxy
or is it out of this project scope?

What's the rationale behind this? This has been done specifically for your convenience, I don't understand why you would want to do this.

What's the rationale behind this? ...I don't understand why you would want to do this.

easier installation. as comparison with invidious you can just copy the compose file and run docker compose with the file
i'm beginner in docker and have no experience with nginx or caddy. i just want to try to run piped locally

This has been done specifically for your convenience

if  it is special case and out of scope from this project, this issue can be closed

i just want to try to run piped locally

You shouldn't be doing this, Piped wasn't intended for that!
The reason being pubsub wouldn't work, and managing 3 different services/ports will be a pain.
Unlike Invidious which is monolithic, Piped requires 3-5 containers for everything, which makes things significantly difficult to run without hostnames, this is hence why there is a script for normal server installations!
I'll close this as you requested.


Local jobs don't start when xenon library is stopped

Use case: starting a local job with a command line tool that uses Xenon internally. Once the tool stops, the job is also not run. In Xenon this is encoded with the Scheduler.isOnline() function that returns true for the local adaptor: it only runs jobs online. Proposal: create an alternative local adaptor that executes a new independent job (like nohup), or schedules a job using for example the at tool.

Alternative: just write nohup in the job you're going to run. In that case, you still have to ask the tool to wait until the job has finished, otherwise, the local queue may just disappear without starting anything.

This is also an issue with the Xenon-CLI. The SSH adaptor has a similar problem. In general, with interactive schedulers, the process disappears when Xenon disappears.
Not sure if we can fix this is in a portable way. Or if we want to ? Also, how are you going to find the process again later on ? Any ideas, comments, suggestions are welcome.

We should add a separate at scheduler for this

The at adaptor has been added and will be part of the 3.0.0 release.