Fix use of WorkQueue executor in g3wfpipe

I saw errors earlier when using the WorkQueue executor from the g3wfpipe application. I would like to resolve those and ensure that executor is adequately supported.

I tried running a job with WorkQueue:

config: w2321-visit:277-pipe:isr-init-proc
howfig: shifter-wq:20-tmax:10m
Result: Done. Workflow complete: 87/87 tasks.

Despite the latter, the short run time and the following log snippet indicate the job was not successful.

Submit dir: /pscratch/sd/d/dladams/descprod-jobs/job000437/submit/u/dladams/isr/20230621T161828Z
Run Name: u/dladams/isr/20230621T161828Z
2023/06/21 09:19:13.34 work_queue_python[1060860] notice: Could not create work_queue on port 9000.
Process WorkQueue-Submit-Process:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/work_queue.py", line 1973, in __init__
    raise Exception('Could not create work_queue on port %d' % port)
Exception: Could not create work_queue on port 9000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/parsl/process_loggers.py", line 27, in wrapped
    r = func(*args, **kwargs)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/parsl/executors/workqueue/executor.py", line 867, in _work_queue_submit_wait
    raise e
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/parsl/executors/workqueue/executor.py", line 861, in _work_queue_submit_wait
    q = WorkQueue(port, debug_log=wq_debug_log)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/work_queue.py", line 1984, in __init__
    raise Exception('Unable to create internal Work Queue structure: %s' % e)
Exception: Unable to create internal Work Queue structure: Could not create work_queue on port 9000
2023-06-21 09:19:13: Check of quantum graph raised an exception.
2023-06-21 09:19:13: Quantum graph was created.
2023-06-21 09:19:13:
2023-06-21 09:19:13: Starting workflow
Exception in thread WorkQueue-collector-thread:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/parsl/process_loggers.py", line 27, in wrapped
    r = func(*args, **kwargs)
  File "/opt/lsst/software/stack/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/parsl/executors/workqueue/executor.py", line 781, in _collect_work_queue_results
    raise ExecutorError(self, "Workqueue Submit Process is not alive")
parsl.executors.errors.ExecutorError: Executor work_queue failed due to: Workqueue Submit Process is not alive
2023-06-21 09:19:24: Workflow task count: 87
2023-06-21 09:19:24: Finished 87 of 87 tasks.
2023-06-21 09:19:34: Workflow complete: 87/87 tasks.
2023-06-21 09:19:34: All steps completed.

Here is the yaml config for the above job:

includeConfigs:
  - ${GEN3_WORKFLOW_DIR}/python/desc/gen3_workflow/etc/bps_drp_baseline.yaml
  - ${GEN3_WORKFLOW_DIR}/examples/bps_DC2-3828-y1_resources.yaml

pipelineYaml: "${DRP_PIPE_DIR}/pipelines/_ingredients/LSSTCam-imSim/DRP.yaml#isr"

payload:
  inCollection: LSSTCam-imSim/defaults
  payloadName: isr
  butlerConfig: /global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo
  dataQuery: "visit>0 and skymap='DC2' and visit=277"

# Default task properties.
requestCpus: 1
requestMemory: 1000

monitoring: MonitoringHub(hub_address=address_by_hostname(),
                   hub_port=None,
                   monitoring_debug=False,
                   resource_monitoring_interval=60)

parsl_config:
  retries: 1
  executor: WorkQueue
  provider: Local
  nodes_per_block: 1
  worker_options: "--memory=20000"

I ran a status job for the above:

config: job:437-status
howfig: tmax:1m
Result: Done. Finished 0 of 87 tasks. (87 failed.)

So the incorrect success reported by the original job is likely something for me to fix rather than a problem in gen3_workflow or parsl.

I verified that port 9000 was busy on the (perlmutter login) node where the above job was run:

 login13> checkport 9000 x
Port 9000 is in use: tcp LISTEN 0 100 <IP_ADDRESS>:9000 <IP_ADDRESS>:*

This is the code for the checkport command:

#!/bin/bash

PORT=${1:-9000}
VERBOSE=$2
LINE=$(ss -tualn | grep ":$PORT " 2>/dev/null)
STAT=$?
if test -n "$VERBOSE"; then
  if test -n "$LINE"; then
    echo Port $PORT is in use: $LINE
  else
    echo Port $PORT is not in use
  fi
fi
exit $STAT

I had to log in and out four times to find a machine where the port was not busy. Once I did , I resubmitted the above job. It ran a little slower spending about 2 min in the processing. But the result was the same: all tasks failed.
The console log now looks fine. The parsl log shows this for each task:

1687367401.943255 2023-06-21 10:10:01 MainProcess-1664149 WorkQueue-collector-thread-139936724911872 parsl.dataflow.dflow:351 handle_exec_update ERROR: Task 8 failed after 1 retry attempts. Last exception was: TypeError: Cannot subclass special typing classes

My run directory is /pscratch/sd/d/dladams/descprod-jobs/job000439 and is (now) world readable.

For the type error, this is the relevant parsl PR https://github.com/Parsl/parsl/pull/2678
Basic fix to try: checking typing-extensions package mentioned at the end of that PR comment thread is at least 4.6.0

To make parsl+wq choose a port automatically, specify port number 0 as the work queue port - this was introduced in PR #2602.
so for example this WorkQueueExecutor(label='work_queue', port=9000, ... should become WorkQueueExecutor(label='work_queue', port=0, ...
I think (but I'm not absolutely sure) that in gen3_workflow this happens in python/desc/gen3_workflow/config/parsl_configs.py in the workqueue_config function: https://github.com/LSSTDESC/gen3_workflow/blob/70e5b925f74f2192d7df9e8bddc7c070113fd81b/python/desc/gen3_workflow/config/parsl_configs.py#L142
which looks like it can't be explicitly configured - but could be modified in the gen3_workflow source.

@benclifford : I added a dependency on typing_extensions to my package and the typing errors have gone away but I get a crash when I check the futures:

2023-07-03 10:52:22: Starting workflow
Traceback (most recent call last):
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/./local/desc-gen3-prod/bin/g3wfpipe-run.py", line 191, in
    futures = [job.get_future() for job in pg.values() if not job.dependencies]
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/./local/desc-gen3-prod/bin/g3wfpipe-run.py", line 191, in
    futures = [job.get_future() for job in pg.values() if not job.dependencies]
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 354, in get_future
    if self.done:
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 305, in done
    elif self.status == _SUCCEEDED:
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 331, in status
    if self.have_outputs():
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 378, in have_outputs
    for node in self.qgraph_nodes:
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 391, in qgraph_nodes
    qgraph = self.parent_graph.qgraph
  File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 517, in qgraph
    self._qgraph = QuantumGraph.loadUri(qgraph_file, DimensionUniverse())
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/graph.py", line 969, in loadUri
    qgraph = loader.load(universe, nodes, graphID)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/_loadHelpers.py", line 220, in load
    return self.deserializer.constructGraph(nodeSet, _readBytes, universe)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/_versionDeserializers.py", line 617, in constructGraph
    qnode = QuantumNode.from_simple(nodeDeserialized, loadedTaskDef, universe, recontitutedDimensions)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/quantumNode.py", line 139, in from_simple
    quantum=Quantum.from_simple(
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/daf_butler/g98bc2251b0+353af21826/python/lsst/daf/butler/core/quantum.py", line 449, in from_simple
    rebuiltDatasetRef = _reconstructDatasetRef(
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/daf_butler/g98bc2251b0+353af21826/python/lsst/daf/butler/core/quantum.py", line 64, in _reconstructDatasetRef
    reconstructedDim = DimensionRecord.from_simple(tmpSerialized, universe=universe)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/daf_butler/g98bc2251b0+353af21826/python/lsst/daf/butler/core/dimensions/_records.py", line 378, in from_simple
    record_model = record_model_cls(**simple.record)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SpecificSerializedDimensionRecordInstrument
visit_system
  field required (type=value_error.missing)
Terminated
Handling SIGTERM (e.g. from timeout)

On closer inspection I see some log errors:

Installing collected packages: typing-extensions, desc-gen3-prod
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-paramiko, which is not installed.
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-requests, which is not installed.
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-six, which is not installed.
parsl 2023.5.29.dev0+desc.2023.6.02a requires typeguard<3,>=2.10, but you have typeguard 3.0.2 which is incompatible.
Successfully installed desc-gen3-prod-0.0.16.dev2 typing-extensions-4.7.1

Should I be installing those as well?

@benclifford I have not been able to get past the typing_extensions version problem. Are you able to run LSST jobs with the current parsl? The latest LSST weekly uses typing_extension 4.4.0.
Is there an older version of DESC parsl that I should use?

I'm unclear if that pydantic stack trace is directly related to typing extensions: it seems to be reporting that a particular field (visit_system?) is missing constructing SpecificSerializedDimensionRecordInstrument
Thats seems far away from the parsl code - even though it's happening in the parsl part of gen3_workflow.
(eg this call in the middle of the stack trace that you pasted)
File "/pscratch/sd/d/dladams/descprod-jobs/job000447/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 517, in qgraph
    self._qgraph = QuantumGraph.loadUri(qgraph_file, DimensionUniverse())

I thought that the parsl code in gen3_workflow was meant to go away and be replaced by ctrl_bps_parsl? (maybe Jim, who I can't tag here, can clarify that?)
I'd rather not go debugging gen3_workflow if it really is unsupported now, but otherwise I guess we need to debug what's happening with the graph file being loaded here:      self._qgraph = QuantumGraph.loadUri(qgraph_file, DimensionUniverse())

Jumping back to the port issue, Jim sent me this:

I was in the process of implementing and testing this, but then I saw that you can set the port yourself from the config like this:
parsl_config:
  retries: 1
  monitoring: true
  executor: WorkQueue
  provider: Local
  port:	0
This seems better than changing the default in code, especially since a different version of parsl is needed for it to work.

I make this mod in 0.0.16.dev8. Note this includes attempts to reolve the typing_extensions problem.

Jobs with the above mod seem to run fine on a machine where port 9000 was busy. An WorkQueue is now working though there are some build errors:

Successfully built desc-gen3-prod
Installing collected packages: typing-extensions, desc-gen3-prod
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-paramiko, which is not installed.
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-six, which is not installed.
parsl 2023.5.29.dev0+desc.2023.6.02a requires typeguard<3,>=2.10, but you have typeguard 4.0.0 which is incompatible.
Successfully installed desc-gen3-prod-0.0.16.dev8 typing-extensions-4.6.0

Here I explicitly ask for typing_extension 4.6.0. I ran with apparent success using this config:

  "jobtype": "g3wfpipe",
  "config": "w2321-visit:277-pipe:isr-init-proc",
  "howfig": "shifter-wq:20-tmax:10m",

I set the version to 0.0.16.

a quick review of those dependency errors:
These two are used if you want to do type checking as part of hacking on the parsl source code. You aren't doing that so it should not be a problem:
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-paramiko, which is not installed.
parsl 2023.5.29.dev0+desc.2023.6.02a requires types-six, which is not installed

I'm fairly certain that parsl will be compatible with typeguard 4.0.0 - I'll try that out and change the requirements file if so:
parsl 2023.5.29.dev0+desc.2023.6.02a requires typeguard<3,>=2.10, but you have typeguard 4.0.0 which is incompatible.

@benclifford thanks for you comments here. Sorry I didn't see this before restarting the discussion on slack.
As reported there, I can now run successfully if I install parsl with dependencies (i.e. without the --no-deps flag) and don't install explicitly all the other packages. I will use this for my studies.
I did not succeed installing parsl without dependencies and explicitly installing the other package including typeguard<3 and typing_extensions==4.6.3. Instead I find these versions (using install from job 482):

 login24> g3wf-run-cvmfs w2321:tyx
g3wf-setup-cvmfs: Setting up LSST cvmfs distrib w_2023_21
g3wf-setup-cvmfs: Activating conda env /pscratch/sd/d/dladams/descprod-out/installs/g3wf-w2321-tyx
 cvmfs g3wf-w2321-tyx> pypkg_version typeguard
Package typeguard has version 4.0.0
 cvmfs g3wf-w2321-tyx> pypkg_version typing_extensions
Package typing_extensions has version 4.4.0

I will give up on this option for now and use parsl with dependencies.

The cvmfs setup/install script g3wf-setup-cvmfs is modified to install parsl with dependencies if no arguments are supplie with the release tag of if the argument "parsl" is used, e.g. w2321 or w2321:parsl. A test job with the former appeared to run fine. Changes are in 0.0.17.

The above success was with w2321 (LSST weekly w_2023_21). When I try with w2328, my job crashes because desc_wfmon is missing. I don't know why this wasn't an issue for the previous release.
I add install of desc_wfmon to local dir in runapp_wfpipe. Previously this was only done for shifter. With this change test job for w2328 runs ok.

The above change is commited in version 0.0.18.

I had a job fail with this error:

2023-08-01 12:16:34: Running workflow
2023-08-01 12:16:34: Total task count: 463
Traceback (most recent call last):
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/./local/desc-gen3-prod/bin/g3wfpipe-run.py", line 206, in
    futures = [job.get_future() for job in pg.values() if not job.dependencies]
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/./local/desc-gen3-prod/bin/g3wfpipe-run.py", line 206, in
    futures = [job.get_future() for job in pg.values() if not job.dependencies]
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 354, in get_future
    if self.done:
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 305, in done
    elif self.status == _SUCCEEDED:
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 331, in status
    if self.have_outputs():
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 378, in have_outputs
    for node in self.qgraph_nodes:
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 391, in qgraph_nodes
    qgraph = self.parent_graph.qgraph
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000533/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 517, in qgraph
    self._qgraph = QuantumGraph.loadUri(qgraph_file, DimensionUniverse())
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/graph.py", line 969, in loadUri
    qgraph = loader.load(universe, nodes, graphID)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/_loadHelpers.py", line 220, in load
    return self.deserializer.constructGraph(nodeSet, _readBytes, universe)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/_versionDeserializers.py", line 617, in constructGraph
    qnode = QuantumNode.from_simple(nodeDeserialized, loadedTaskDef, universe, recontitutedDimensions)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/pipe_base/g807f6e7cd4+93168c461e/python/lsst/pipe/base/graph/quantumNode.py", line 139, in from_simple
    quantum=Quantum.from_simple(
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/daf_butler/g98bc2251b0+353af21826/python/lsst/daf/butler/core/quantum.py", line 449, in from_simple
    rebuiltDatasetRef = _reconstructDatasetRef(
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/daf_butler/g98bc2251b0+353af21826/python/lsst/daf/butler/core/quantum.py", line 64, in _reconstructDatasetRef
    reconstructedDim = DimensionRecord.from_simple(tmpSerialized, universe=universe)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_21/conda/envs/lsst-scipipe-6.0.0-exact-ext/share/eups/Linux64/daf_butler/g98bc2251b0+353af21826/python/lsst/daf/butler/core/dimensions/_records.py", line 378, in from_simple
    record_model = record_model_cls(**simple.record)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for SpecificSerializedDimensionRecordInstrument
visit_system
  field required (type=value_error.missing)

In 0.0.20, I add a try block to catch this exception up to 10 times before giving up. When I resubmitted the job, the exception was not raised and the job finished without incident.

I notioced an exception was raised if user asked for status in g3wfpipe before the pipeline had started running. I fixed this to instead return a message "Workflow has not started.". Change is in 0.0.21.

For 0.0.22, add application g3wfenv that installs and checks, or removes g3wf environments. The check returns the parsl version.

For 0.0.23, add script that was inadvertently omitted from git in 0.0.22.

For 0.0.25, update config templates so each job has a perfstat report in its error log.

For 0.0.26, add first pass at performance plots including monexp.ipynb copied from desc-wfmon. Navigate to the run directory in Jupyter and execute the notebook to generate performance plots.

For 0.0.28, remove the reinstallation of local packages in continuation jobs.
A bug in the PATH for the sysmon was introduced and then fixed in 0.0.29.

For 0.0.30:

Add monitoring interval of 5 sec to g3wfpipe config. But gen3_workflow does not yet support this.
After g3wfpipe proc step, get the parsl run_id from the monitoring DB and set the value in monexp.py. This way the monexp notebook runs out of the box.

For 0.0.31, add howfig option pmon:DT to set the parsl monitoring interval. This was added and may different values tested successfully.

I noticed I was getting slow parsl launch (2-3 sec/task) for a w2330 request. On investigation, I found the parsl install had failed and I was picking up parsl from the LSST release.
For 0.0.32:

Add switch to gen3_workflow branch for w2331 and later. Getting ahead of myself here.
Add g3wf-check-cvmfs to check if an env (e.g. w2321:parsl) exists.
Add quotes when writing CON_QUERY.
Add g3wf-install-cvmfs to git.
Exit runapp-g3wfpipe with error for w_2023_30 and later. I don't yet have prescription to install DESC parsl for these.

For 0.0.33:

Change list of bad releases to just be 2330 and 2331 so I can test 2332 that Ben reports is working.
Fix a couple small bugs giving log errors in runapp_g3wfpipe
in g3wfpipe_run.py, update monexp.py early so user can use the notebook during processing.

For 0.0.34, I want to understand which files are accessed in pipeline tasks. I couldn't get iotrace to work as a command prepend and so went to strace with a file filter. Right now, all tasks write to strace.log in the run dir. The trace is only run for user dladams.

For 0.0.35, add howfig option strace to run strace with each task. Als stime produces strime timing report.

For 0.0.36: First part a bproc job failed because sysmon.pid was not present. I modified g3wfproc-run to only kill monitor if that file is present.

For 0.0.37.

Stop checking LSST release for continuation jobs in runapp-g3wfpipe. The check was done in the first job and the variable IVER used for the check is not defined in the continuation job.
Change shifter image version to 08, 09, 10, 11 in g3wf-run-shifter w/o success.
The docker files for those versions are in the repository.

Versions 08 and 09 give this error for each task:

2023-09-13 17:06:43: Try 8 raised exception: 1 validation error for SpecificSerializedDimensionRecordInstrument
visit_system
  field required (type=value_error.missing)

Versions 10 and 11 give this error in a parsl import:

Traceback (most recent call last):
  File "./local/desc-gen3-prod/bin/g3wfpipe-run.py", line 178, in
    from desc.gen3_workflow import start_pipeline
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000910/gen3_workflow/python/desc/gen3_workflow/__init__.py", line 1, in
    from .parsl_service import *
  File "/pscratch/sd/d/dladams/descprod-out/jobs/job000910/gen3_workflow/python/desc/gen3_workflow/parsl_service.py", line 13, in
    import lsst.utils
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/utils/ga34f01d9d8+f0a1800e64/python/lsst/utils/__init__.py", line 14, in
    from .deprecated import *
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-6.0.0/Linux64/utils/ga34f01d9d8+f0a1800e64/python/lsst/utils/deprecated.py", line 20, in
    import deprecated.sphinx
ModuleNotFoundError: No module named 'deprecated'

For 0.0.38 I am adding a prescription based on the dockerfile Jim Chiang uses to run with w_2023_32: https://lsstc.slack.com/archives/C038ZBE4QJJ/p1692824749070059
I drop the Rubin packages and make equivalent installs in g3wf-install-parsl-w2332. I also fix one "=" --> "==" and drop the python spec for ndctools (commented out).
I add dockerfile-12 which uses this script to build its env. But it does not work for w_2023_21.

For 0.0.39, start working with LSST w_2023_32:

I rename my local install of gen3_workflow so we go back to use the git version.
Stop using special branch of gen3_workflow for IVER > 2331.
Add python 3.11 to python path extensions included in the cvmfs full deps parsl install (option parsl)
Modify g3wfpipe-run to return the return code from g3wfpipe-run.py.
Modify docker build script so output is plain and is logged to a file.
Remove spaces in the parsl spec in g3wf-install-parsl-w2332
Add all site-directories dirs in $wq_env to PYTHONPATH in g3wf-setup-cvmfs.
Add options to g3wf-setup-cvmfs:
** -v: OPT sets version OPT instead of taking it from LVER:OPT
** -x: call exit on error
Add -t -T -C to strace prepend

Future development is described in #5.

For 0.0.40, added field nNNN to the strace howfig (e.g. strace:n10:file1) so there is a 1/NNN chance of running strace.
I dropped time from the prepend. It was giving problems.
I added the -f (follow children) flag in strace to see the fits files.


chore: release 1.2.232

:robot: I have created a release *beep* *boop*
1.2.232 (2023-05-16)
Features

dev: tasks for local development (#5607) (5b96afc)
multiarch: support platform setting per image in werf.yaml configuration (39fd752)

Bug Fixes

harbor regular NOT_FOUND error treated as 'broken image' internal registry error (bc4ef3d)
multiarch: use correct multiarch manifests for werf-run and werf-kube-run commands (fca96f2)
rename ambiguous --skip-build to --require-built-images (#5619) (2a57b4b)
use 'built image' instead 'cache image' (fee0d67)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/werf/werf/releases/tag/v1.2.232 :sunflower:


Log Level is case sensitive

Please tell us about your environment:

winston version?

[ ] winston@2
[x] winston@3

_node -v outputs:_8.10.4

What is the problem?
No logs are generated when using a log level string that is not all lower case.
Example: DEBUG log level generates no logs.
What do you expect to happen instead?
Expect Log Level to be case insensitive.
Is the absence of this feature blocking you or your team? If so, how?
No. The environment variable is easily converted to lower case prior to creating the logger object.
Is this a feature you're prepared to implement, with support from us?
Yes.

If we changed winston to always do e.g. .toLower() on the level, that'd be a performance issue, so I would be hesitant to do that.  The best solution is probably to add custom levels https://github.com/winstonjs/winston#using-custom-logging-levels corresponding to you different casings, e.g. just have DEBUG behave the same way as debug.  That's the only way to avoid doing a toLower somewhere, hence the best performance option I think.

@DABH Fair enough. I wouldn't have thought that .toLower() would have been called multiple times. I had assumed that it would only happen on instantiation of the logger object which I would not have expected to make a noticeable impact on performance. That being said you know the code way better than I so I will trust your judgement. 😄


Add a method to resize a widget-backed surface

On some platforms (e.g. macos and wayland) widget-backed surfaces don't automatically resize when the widget is resized, so we need an API for that.

@bors-servo r=jdm

:pushpin: Commit 122eb77 has been approved by jdm

:hourglass: Testing commit 122eb77f17061afadf31f7ebe4551fbb66a5e979 with merge 1bb08b8db0bbbc43e9095a3aa084a79ae259d605...

:sunny: Test successful - checks-travis
Approved by: jdm
Pushing 1bb08b8db0bbbc43e9095a3aa084a79ae259d605 to master...


VSCode not outputting flake8 plugin output

Environment data

VS Code version: 1.41.0
Extension version (available under the Extensions sidebar): 2019.11.50794
OS and version: Windows 10 18362
Python version (& distribution if applicable, e.g. Anaconda): 3.8
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pipenv(virutalenv)
Relevant/affected Python packages and their versions: wemake-python-styleguide v0.13.2 (flake8 plugin)
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): No

Expected behaviour
VSCode fully lints a file using flake8 and the flake8_isort plugin and highlight errors, warnings and information as appropriate.
Actual behaviour
Presence of a WPS226 violation "Found string constant over-use" causes the error "Illegal argument: line must be non-negative" to be thrown by VSCode.
Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Install wemake-python-styleguide
Set VSCode to use flake8 as the linter
Run flake8 linting on any file through vscode, ensuring that a wps226 error is present

Logs
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
##########Linting Output - flake8##########
0,1,W,WPS202:Found too many module members: 15
0,1,W,WPS226:Found string constant over-use: id
1,1,C,C101:Coding magic comment not found
3,1,F,F401:'json' imported but unused
8,1,W,WPS111:Found too short name: T
8,13,Q,Q000:Remove bad quotes
9,17,Q,Q000:Remove bad quotes
12,14,W,WPS111:Found too short name: x
13,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
17,15,W,WPS111:Found too short name: x
18,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
22,20,W,WPS111:Found too short name: x
23,9,W,WPS111:Found too short name: f
26,9,W,WPS424:Found except `BaseException`
27,13,W,WPS420:Found wrong keyword: pass
28,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
28,5,B,B011:Do not call assert False since python -O removes these calls. Instead callers should raise AssertionError().
28,12,W,WPS444:Found wrong keyword condition
31,13,W,WPS111:Found too short name: c
31,29,W,WPS111:Found too short name: x
32,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
36,14,W,WPS111:Found too short name: x
37,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
41,13,W,WPS111:Found too short name: t
41,25,W,WPS111:Found too short name: x
42,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
46,15,W,WPS111:Found too short name: f
46,38,W,WPS111:Found too short name: x
47,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
48,22,W,WPS111:Found too short name: y
51,14,W,WPS111:Found too short name: c
51,26,W,WPS111:Found too short name: x
52,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
57,5,W,WPS115:Found upper-case constant in a class: OK
57,10,Q,Q000:Remove bad quotes
58,5,W,WPS115:Found upper-case constant in a class: TOO_LOW
58,15,Q,Q000:Remove bad quotes
62,1,W,WPS306:Found class without a base class: RowCount
68,5,W,WPS602:Found using `@staticmethod`
68,19,W,WPS110:Found wrong variable name: obj
68,32,Q,Q000:Remove bad quotes
69,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
70,62,Q,Q000:Remove bad quotes
71,63,Q,Q000:Remove bad quotes
72,60,Q,Q000:Remove bad quotes
76,9,W,WPS110:Found wrong variable name: result
77,16,Q,Q000:Remove bad quotes
78,16,Q,Q000:Remove bad quotes
79,21,W,WPS111:Found too short name: x
79,72,C,C812:missing trailing comma
81,16,Q,Q000:Remove bad quotes
86,1,W,WPS306:Found class without a base class: ReportStore
91,5,W,WPS602:Found using `@staticmethod`
91,19,W,WPS110:Found wrong variable name: obj
91,32,Q,Q000:Remove bad quotes
92,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
93,9,W,WPS221:Found line with high Jones Complexity: 17
93,47,W,WPS111:Found too short name: x
93,77,Q,Q000:Remove bad quotes
94,9,A,A001:"id" is a python builtin and is being shadowed, consider renaming the variable
94,56,Q,Q000:Remove bad quotes
98,9,W,WPS110:Found wrong variable name: result
99,16,Q,Q000:Remove bad quotes
101,24,W,WPS111:Found too short name: x
101,38,W,WPS430:Found nested function: lambda
101,45,W,WPS111:Found too short name: x
102,17,W,WPS221:Found line with high Jones Complexity: 18
102,24,W,WPS111:Found too short name: x
102,37,W,WPS430:Found nested function: lambda
102,44,W,WPS111:Found too short name: x
102,52,W,WPS430:Found nested function: lambda
102,59,W,WPS111:Found too short name: x
106,16,Q,Q000:Remove bad quotes
111,1,W,WPS306:Found class without a base class: Node
117,5,W,WPS602:Found using `@staticmethod`
117,19,W,WPS110:Found wrong variable name: obj
117,32,Q,Q000:Remove bad quotes
118,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
119,9,A,A001:"id" is a python builtin and is being shadowed, consider renaming the variable
119,56,Q,Q000:Remove bad quotes
120,79,Q,Q000:Remove bad quotes
122,21,W,WPS111:Found too short name: x
122,78,Q,Q000:Remove bad quotes
122,89,C,C812:missing trailing comma
127,9,W,WPS110:Found wrong variable name: result
128,16,Q,Q000:Remove bad quotes
129,16,Q,Q000:Remove bad quotes
130,21,W,WPS111:Found too short name: x
130,79,C,C812:missing trailing comma
132,16,Q,Q000:Remove bad quotes
133,21,W,WPS111:Found too short name: x
133,34,W,WPS430:Found nested function: lambda
133,41,W,WPS111:Found too short name: x
150,13,W,WPS110:Found wrong variable name: val
159,1,W,WPS306:Found class without a base class: Edge
169,5,W,WPS602:Found using `@staticmethod`
169,19,W,WPS110:Found wrong variable name: obj
169,32,Q,Q000:Remove bad quotes
170,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
171,64,Q,Q000:Remove bad quotes
175,9,W,WPS110:Found wrong variable name: result
176,9,W,WPS221:Found line with high Jones Complexity: 16
176,16,Q,Q000:Remove bad quotes
176,45,W,WPS111:Found too short name: x
181,1,W,WPS306:Found class without a base class: ReportStores
185,16,W,WPS219:Found too deep access level: 5
191,5,W,WPS602:Found using `@staticmethod`
191,28,W,WPS110:Found wrong variable name: obj
191,41,Q,Q000:Remove bad quotes
192,1,S,S101:Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
194,21,W,WPS111:Found too short name: x
195,13,W,WPS221:Found line with high Jones Complexity: 17
195,36,W,WPS111:Found too short name: x
195,45,Q,Q000:Remove bad quotes
195,53,Q,Q000:Remove bad quotes
195,62,Q,Q000:Remove bad quotes
200,9,W,WPS110:Found wrong variable name: result
201,16,Q,Q000:Remove bad quotes
202,13,W,WPS221:Found line with high Jones Complexity: 15
202,21,W,WPS111:Found too short name: x
202,34,W,WPS430:Found nested function: lambda
202,41,W,WPS111:Found too short name: x
202,89,C,C812:missing trailing comma
206,5,W,WPS210:Found too many local variables: 6
207,30,Q,Q000:Remove bad quotes
207,42,Q,Q000:Remove bad quotes
209,22,W,WPS110:Found wrong variable name: value
213,17,F,F841:local variable 'search_keyword' is assigned to but never used
217,30,Q,Q000:Remove bad quotes
219,9,W,WPS111:Found too short name: i
221,9,W,WPS337:Found multiline conditions
223,17,W,WPS219:Found too deep access level: 5
227,9,W,WPS324:Found inconsistent `return` statement
227,9,W,WPS324:Found inconsistent `return` statement
232,13,Q,Q000:Remove bad quotes
232,21,Q,Q000:Remove bad quotes
232,31,Q,Q000:Remove bad quotes
232,47,Q,Q000:Remove bad quotes
232,55,C,C812:missing trailing comma

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)
notificationsAlerts.ts:38 Error: Illegal argument: line must be non-negative
	at Object.t.illegalArgument (c:\Users\kazon\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:9:868)
	at new p (c:\Users\kazon\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:360:567)
	at new p (c:\Users\kazon\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:359:919)
	at S.createDiagnostics (c:\Users\kazon\.vscode\extensions\ms-python.python-2019.11.50794\out\client\extension.js:75:1185327)
	at S.lintDocument (c:\Users\kazon\.vscode\extensions\ms-python.python-2019.11.50794\out\client\extension.js:75:1184892)
	at async Promise.all (index 0)
	at async S.lintOpenPythonFiles (c:\Users\kazon\.vscode\extensions\ms-python.python-2019.11.50794\out\client\extension.js:75:1183938)

It's been fixed by https://github.com/microsoft/vscode-python/pull/8476 and will be in the next release. If you want it sooner then please feel free to use our insiders build of the extension.


feat: add the new ids that terraform has created.

Terraform Format and Style 🖌success
Terraform Initialization ⚙success
Terraform Validation 🤖success
Validation Output

Success! The configuration is valid.

Terraform Plan 📖success
Show Plan

terraform
Running plan in Terraform Cloud. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.

Preparing the remote plan...

To view this run in a browser, visit:
https://app.terraform.io/app/stackhpc/github/runs/run-AAMJzqPGoeuLprxg

Waiting for the plan to start...

Terraform v1.2.4
on linux_amd64
Initializing plugins and modules...
github_team.organisation_teams["ReleaseTrain"]: Refreshing state... [id=6372895]
github_team.organisation_teams["Ansible"]: Refreshing state... [id=2454000]
github_team.organisation_teams["Developers"]: Refreshing state... [id=6309608]
github_team.organisation_teams["Azimuth"]: Refreshing state... [id=6372898]
github_team.organisation_teams["OpenStack"]: Refreshing state... [id=6372899]
github_team.organisation_teams["Batch"]: Refreshing state... [id=6372897]
github_team.organisation_teams["SMSLab"]: Refreshing state... [id=6372896]
github_repository.repositories[".github"]: Refreshing state... [id=.github]
data.github_repository.repositories[".github"]: Reading...
github_team_repository.developers_repositories[".github"]: Refreshing state... [id=6309608:.github]
github_team_repository.release_train_repositories[".github"]: Refreshing state... [id=6372895:.github]
github_team_membership.team_membership["Ansible:jackhodgkiss"]: Refreshing state... [id=2454000:jackhodgkiss]
github_team_membership.team_membership["Ansible:GregWhiteyBialas"]: Refreshing state... [id=2454000:GregWhiteyBialas]
github_team_membership.team_membership["Ansible:grzegorzkoper"]: Refreshing state... [id=2454000:grzegorzkoper]
github_team_membership.team_membership["Ansible:m-bull"]: Refreshing state... [id=2454000:m-bull]
github_team_membership.team_membership["Developers:stackhpc-ci"]: Refreshing state... [id=6309608:stackhpc-ci]
github_team_membership.team_membership["Ansible:markgoddard"]: Refreshing state... [id=2454000:markgoddard]
github_team_membership.team_membership["Ansible:bbezak"]: Refreshing state... [id=2454000:bbezak]
github_team_membership.team_membership["Ansible:jovial"]: Refreshing state... [id=2454000:jovial]
github_team_membership.team_membership["Ansible:oneswig"]: Refreshing state... [id=2454000:oneswig]
github_team_membership.team_membership["Developers:jackhodgkiss"]: Refreshing state... [id=6309608:jackhodgkiss]
github_team_membership.team_membership["Ansible:JohnGarbutt"]: Refreshing state... [id=2454000:JohnGarbutt]
github_team_membership.team_membership["Ansible:cityofships"]: Refreshing state... [id=2454000:cityofships]
github_team_membership.team_membership["Ansible:dougszumski"]: Refreshing state... [id=2454000:dougszumski]
github_team_membership.team_membership["Ansible:mnasiadka"]: Refreshing state... [id=2454000:mnasiadka]
data.github_repository.repositories[".github"]: Read complete after 2s [id=.github]
github_team_membership.team_membership["Ansible:g0rgamesh"]: Refreshing state... [id=2454000:g0rgamesh]
github_team_membership.team_membership["Ansible:sjpb"]: Refreshing state... [id=2454000:sjpb]
github_team_membership.team_membership["Ansible:k-s-dean"]: Refreshing state... [id=2454000:k-s-dean]
github_branch_protection.release_train_branch_protection[".github"]: Refreshing state... [id=BPR_kwDOHK-Qcs4BodCX]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # github_team.organisation_teams["Ansible"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"79728b1cc3edefca9cf2d11a5d68ca20bb1d29a93ad26d8c228e6736a7d24ce0\"" -> "W/\"4977b0f32f2a321cac6549113155b6bc0795a1b1094e4a9320b7bff6ca0349a3\""
        id                        = "2454000"
        name                      = "Ansible"
        # (6 unchanged attributes hidden)
    }

  # github_team.organisation_teams["Azimuth"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"559b71a14a2c526dcce2066d1d24596226372c2de794dccd767ff489a5c603b8\"" -> "W/\"ae3647d2cb73b89cdd4233559944863de7d690add3e2e7a111a7d29c55039bd7\""
        id                        = "6372898"
        name                      = "Azimuth"
        # (6 unchanged attributes hidden)
    }

  # github_team.organisation_teams["Batch"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"b074e48e5cf49447f2645f6f4bb1d5fbfd1d5cb6b1904d162d4b16d6e929006b\"" -> "W/\"74d32a7f487bed4fc566d5255ecb045c8e6348770ccdee519a3edc1c4b150ea7\""
        id                        = "6372897"
        name                      = "Batch"
        # (6 unchanged attributes hidden)
    }

  # github_team.organisation_teams["Developers"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"e2f12bbb3e626262471b1f74fa9e01809cb460a2e51619cb82d950d55c2452ef\"" -> "W/\"a823c9bce0288f096c7e36b0fcce2b59398e3ab1d4710d86488739c1eafb1a88\""
        id                        = "6309608"
        name                      = "Developers"
        # (6 unchanged attributes hidden)
    }

  # github_team.organisation_teams["OpenStack"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"ec3d162281acf62eaf9af82e577b099e3a1b1d530a27a0f1d181708f5d157193\"" -> "W/\"115707b0185ad4207bd1678eeaab78bf0938e3241978c67cbaae304acb263c16\""
        id                        = "6372899"
        name                      = "OpenStack"
        # (6 unchanged attributes hidden)
    }

  # github_team.organisation_teams["ReleaseTrain"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"f7e8c6e662d0a4922158317fa157476060e6f8bcac029d32520f8d105d2a92b5\"" -> "W/\"6b0e57566d2e0d50a14c054e85895aaa06be96fc3884ee9e34ef026bae565b13\""
        id                        = "6372895"
        name                      = "ReleaseTrain"
        # (6 unchanged attributes hidden)
    }

  # github_team.organisation_teams["SMSLab"] has changed
  ~ resource "github_team" "organisation_teams" {
      ~ etag                      = "W/\"2e218e0c791ed5396ec97cda301dc75bc13fe1b8aadf6ad35512b1379b7e22fa\"" -> "W/\"4949793be32fcd543cf5bd8b421df159f4f16b707392a19904fc3ca9c27d1c9c\""
        id                        = "6372896"
        name                      = "SMSLab"
        # (6 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # github_issue_label.community_files_label[".github"] will be created
  + resource "github_issue_label" "community_files_label" {
      + color       = "3F84E5"
      + description = "Community files have been modified"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "community-files"
      + repository  = ".github"
      + url         = (known after apply)
    }

  # github_issue_label.stackhpc_ci_label[".github"] will be created
  + resource "github_issue_label" "stackhpc_ci_label" {
      + color       = "E6E2C0"
      + description = "Automated action performed by stackhpc-ci"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "stackhpc-ci"
      + repository  = ".github"
      + url         = (known after apply)
    }

  # github_issue_label.workflows_label[".github"] will be created
  + resource "github_issue_label" "workflows_label" {
      + color       = "638475"
      + description = "Workflow files have been modified"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "workflows"
      + repository  = ".github"
      + url         = (known after apply)
    }

  # github_repository.repositories[".github"] will be updated in-place
  ~ resource "github_repository" "repositories" {
      ~ delete_branch_on_merge = false -> true
        id                     = ".github"
        name                   = ".github"
        # (26 unchanged attributes hidden)
    }

  # github_team.organisation_teams["Kayobe"] will be created
  + resource "github_team" "organisation_teams" {
      + create_default_maintainer = false
      + description               = "Team responsible for Kayobe development"
      + etag                      = (known after apply)
      + id                        = (known after apply)
      + members_count             = (known after apply)
      + name                      = "Kayobe"
      + node_id                   = (known after apply)
      + privacy                   = "closed"
      + slug                      = (known after apply)
    }

  # github_team_membership.team_membership["Ansible:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "2454000"
      + username = "stackhpc-ci"
    }

  # github_team_membership.team_membership["Azimuth:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6372898"
      + username = "stackhpc-ci"
    }

  # github_team_membership.team_membership["Batch:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6372897"
      + username = "stackhpc-ci"
    }

  # github_team_membership.team_membership["Developers:GregWhiteyBialas"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "GregWhiteyBialas"
    }

  # github_team_membership.team_membership["Developers:JohnGarbutt"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "JohnGarbutt"
    }

  # github_team_membership.team_membership["Developers:bbezak"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "bbezak"
    }

  # github_team_membership.team_membership["Developers:cityofships"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "cityofships"
    }

  # github_team_membership.team_membership["Developers:darmach"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "darmach"
    }

  # github_team_membership.team_membership["Developers:dougszumski"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "dougszumski"
    }

  # github_team_membership.team_membership["Developers:g0rgamesh"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "g0rgamesh"
    }

  # github_team_membership.team_membership["Developers:grzegorzkoper"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "grzegorzkoper"
    }

  # github_team_membership.team_membership["Developers:jcoquillion"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "jcoquillion"
    }

  # github_team_membership.team_membership["Developers:johntaylorsc"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "johntaylorsc"
    }

  # github_team_membership.team_membership["Developers:jovial"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "jovial"
    }

  # github_team_membership.team_membership["Developers:k-s-dean"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "k-s-dean"
    }

  # github_team_membership.team_membership["Developers:m-dull"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "m-dull"
    }

  # github_team_membership.team_membership["Developers:markgoddard"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "markgoddard"
    }

  # github_team_membership.team_membership["Developers:mkjpryor"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "mkjpryor"
    }

  # github_team_membership.team_membership["Developers:mnasiadka"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "mnasiadka"
    }

  # github_team_membership.team_membership["Developers:onewsig"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "onewsig"
    }

  # github_team_membership.team_membership["Developers:priteau"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "priteau"
    }

  # github_team_membership.team_membership["Developers:sjpb"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6309608"
      + username = "sjpb"
    }

  # github_team_membership.team_membership["Developers:stackhpc-ci"] will be updated in-place
  ~ resource "github_team_membership" "team_membership" {
        id       = "6309608:stackhpc-ci"
      ~ role     = "maintainer" -> "member"
        # (3 unchanged attributes hidden)
    }

  # github_team_membership.team_membership["Kayobe:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = (known after apply)
      + username = "stackhpc-ci"
    }

  # github_team_membership.team_membership["OpenStack:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6372899"
      + username = "stackhpc-ci"
    }

  # github_team_membership.team_membership["ReleaseTrain:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6372895"
      + username = "stackhpc-ci"
    }

  # github_team_membership.team_membership["SMSLab:stackhpc-ci"] will be created
  + resource "github_team_membership" "team_membership" {
      + etag     = (known after apply)
      + id       = (known after apply)
      + role     = "member"
      + team_id  = "6372896"
      + username = "stackhpc-ci"
    }

Plan: 30 to add, 2 to change, 0 to destroy.

Pusher: @jackhodgkiss, Action: pull_request, Working Directory: ``, Workflow: Terraform GitHub


Broken for 0.40.0?

Does this work on latest RN release? Doesn't render for me

me using react-native-collapsible/Accordion, as soon as update to 0.40 I can see the titles of the Accordion. Clicking on the titles changes the icon for expansion/compression but no contents shows anymore.
Anything I can do to help fix this?

I upgraded from 0.22.0 to 41-rc and it is also broken for me (after fixing a lot of other things...)

Agreed that it's broken for latest release.

see https://github.com/oblador/react-native-collapsible/pull/69

AFAIK this should be fixed a few days ago in yoga master, hopefully that will make it into the next release of RN.

+1

@oblador  new infos that the fix will be definitively in the next react-native version ?
and do anyone know when the new react-native version will arrive....

@Kurt57 Didn't they say that they're slowing down the updates to once per month? I'm guessing it'll be sometime this week, considering 0.40.0 came out about 25 days ago.

with the fix for this module? @Naoto-Ida

@Kurt57 Not sure, I guess we'll have to wait for 0.41.0 and see.

I'm really desperate to get this working today. I've tried rolling back RN to 0.39.2, but still no expanding on iOS. Unfortunately, this is an urgent bug fix, so I can't wait for the next release of RN. Can anyone who has solved this by "building from the source" point me in the right direction for getting the collapsibles working on iOS?
Note, it works fine on Android (0.39.2 and 0.40.0)
So far, I've tried:

Upgrading to 0.40.0 and adding 'scroll' to Collapsible.js ln99.
Downgrading to 0.39.2. Also tried added 'scroll'
Downloading RN repo master branch and replacing 'react-native' folder in node_modules, then run npm install inside the repo. Gives a lot of error: 'React/RCTwhatever.h' file not found .

@davingreen Incredible how it is working on Android. Mine didn't work from RN 0.37 up. I needed to downgrade to RN 0.36.

@ammichael Thanks for the response. I tried a completely new RN project with:
react-native init Project --version<EMAIL_ADDRESS>But I'm still not getting the expansion of sections. Can you shed some more light on how you rolled back to 0.36.0 and if you did anything else to get it working?

@davingreen Actually I just downgrade using npm, both react and react-native. Something like:
npm install --save<EMAIL_ADDRESS>npm install --save<EMAIL_ADDRESS>
Then I restarted the packager and worked fine..

Reading through #51 , it seems like we keep waiting for a fix with each new RN release, but the last several have not had it.
@ammichael Downgrading didn't work. 😦  But I downgraded and then used this fork: git://github.com/relayfoods/react-native-collapsible.git#collapsible-not-expanding-temp-fix. Finally got it to work..
If anyone has an approach that doesn't require downgrading RN, please share!

Hey guys, try 0.8.0 together with the just released React Native 0.41.0

@oblador does it work? I'm on 0.41.0 so I can test it if you haven't yet.

I've tested the Example project and it works, but please do test it.

it works for me! 👍

Works for me too. Nice! (:


Add "back-to-top" Button

Summary
With the addition of this button, users will be able to navigate back to the top of the "Learn" index, glossary, and track pages during the middle of the content or if they reach the footer.

closes #387

Functionality

Button only displays if there's a hero banner on the page and the hero banner is outside the viewport
Clicking on the button navigates your back to the top

Options
I wanted to go with a fully rounded button, but I also wanted to utilize the x-button.primary component. Since the button has default padding that I couldn't workaround, I decided to go with what it gave me. If we decide to go a different route with its look, I have no problem making this a standalone element. Also, I'm okay with adjusting the position if it feels too close to the main content.
Screenshot

After speaking with Tammy a bit on this, we decided to leave it where it is.


Bugfix #229: Show Add/Update my profile based on user status

Fixes #229
Pull request checklist

[x] Your code contains tests relevant for code you modified
[x] You have linted and tested the project with bin/check

@joemasilotti  Done.
I agree with it. Should have thought of it.

Thanks! Approved - I'll merge this in tonight.


Fix PageCurl with key is missing config statement

Daamn, I had the feeling, that I forgot to add it somewhere 😄
Thx for the fix 👍

Released in 1.4.1 👍


Jira Document - environment of our docs

We'd like to have a single document, for now in Jira it seems, that would be a good starting point for someone who'd like to contribute or even maintain all aspects. So it should eventually get more details than for example https://github.com/crate/cloud-docs/blob/main/DEVELOP.rst

https://crateio.atlassian.net/wiki/spaces/TeamDBCloud/pages/818282503/Documentation+environment+overview

There is no JIRA document in Sphinx or in crate-docs-theme. Can you clarify?

Hi. Description updated.
It's something I'd like to discuss with you when you return, will expand on it then

Confluence is definitively the wrong place, only contains fragments of the whole story, and will easily get stale, because pages on Wikis will always go down that route, and stink.
Please use one of those spots to improve the documentation for technical writers, both internal or external, both newbies or oldfarts. Thanks!

https://crate-docs-theme.readthedocs.io/
https://github.com/crate/tech-writing

I'd agree with the Wiki assessment. We can discuss the best way to go about it later

Sure. Just submit patches about improvements to the two spots enumerate above. Thanks.


E2e test

Can you review and merge the test first, so that everybody can run it locally if need.
I will make it automatically run on Travis in next PR.

Hi @seabornlee, will do, once the tests pass.
It's going to be a great improvement once we have these auto-tests running. I can see these getting extended quite a bit too.

The app still can not get started in Travis emulator, I just undo my .travis.yml change.
As we can only have one travis job running at the same time, I will try to debug the build on my own fork so it won't stuck the other builds.

Hi @seabornlee the currently working setup; does this include the emulator performing tasks? I couldn't work out which part of the build feed is where this is happening.
Once we have this it's going to be really good - I would like to run through the tests not only on an API30/31 emulator but also an API24 emu.

I tried running on API 24 emulator, it works on my MacBook arm emulator, but it crashed on Travis CI, I need to get into debug mode to figure out the reason.
The current travis config not run e2e test, because it will fail.
I suggest we can separate the test and CI configuration, so every body can run or enhance the e2e test suite locally if they want.
I will continue trying to make it working on CI server.

Hi @seabornlee are the new tests ready to be merged into the main repo? It all looks good to me.

Not ready, very close. Now I am running it on API level 24, 30, and 31, randomly fail on one of them. Still trying to make test stable.

Close this and will create a new one later.


FINERACT-2079: Rectify query for cashier transactions

Description
Describe the changes made and why they were made.
Ignore if these details are present on the associated Apache Fineract JIRA ticket.
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[ ] Write the commit message as per https://github.com/apache/fineract/#pull-requests

[ ] Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.

[ ] Create/update unit or integration tests for verifying the changes made.

[ ] Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.

[ ] Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes

[ ] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the developer mailing list for guidance, if required.)

FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.

@wkigenyi Can you please write an integration test which calls the cashier transactions API and calls the cashier transactions with summary API? This way we can be sure, your fix is working and avoid regression in the future?

@adamsaghy this commit basically replaces the wrong table column in the sql query with the right one. Please guide me on the best way to write this this test.

@adamsaghy this commit basically replaces the wrong table column in the sql query with the right one. Please guide me on the best way to write this this test.

First of all thank you for the fix. Like I said in my earlier comment you should write an integration test which call the mentioned APIs and they are not failing (response is HTTP 200). This way we can make sure in the future if the query or the data model got changed we will know about it.
You may find plenty example in the "integration-test" directory.
Please let me know if you are facing any trouble!

Thanks Adam, I do understand the importance of the test. I will look into the mentioned directory for examples.

Thanks Adam, I do understand the importance of the test. I will look into the mentioned directory for examples.

Hi
I believe we dont really have too many test cases for cashier transactions, so i will try to give you some help here:

I would create a new Helper class which extends the IntegrationTest class which contains helper methods to call the "fineract-client" and via that we can call cashier transactions via type-safe way:
Example:
return ok(fineract().tellers.getTransactionsForCashier(tellerId, cashierId, currencyCode, ...))

You can see examples in the LoanTransactionHelper class, example in line 738

Thanks Alot Adam. I will at this in next few days. I have a commitment away from my machine.

Good Morning @adamsaghy. I tried to follow the example in LoanTransactionHelper, how to I run that test individually to see its results (either in intellij or gradlew)?

could you help me on how to locate this directory, I am using mifos 23.12 and I have failed to locate the suggested file inorder to make the changes (appuser_id)

@adamsaghy please have a look at this.

hey @wkigenyi , I am using the mifos 23 not docker version, the one downloaded from sourceforge.net/projects/mifos/
it runs in tomcat, is there any way of fixing that error from this kind of installation?
I noticed that the fixes apply to the cloned docker mifos version.

@bravosty you can locate the file fineract-provider/src/main/java/org/apache/fineract/organisation/teller/service/TellerManagementReadPlatformServiceImpl.java and make the change made in this commit.

[image: image.png]
this is the directory of the fineract-provider file in apache tomcat
directory
[image: image.png]
and this is what I find inside the directory, so I can not see the path you
told me to navigate to cause this is the tomcat, local machine installation
On Mon, May 20, 2024 at 10:24 AM Wilfred Kigenyi @.***>
wrote:

@bravosty https://github.com/bravosty you can locate the file
fineract-provider/src/main/java/org/apache/fineract/organisation/teller/service/TellerManagementReadPlatformServiceImpl.java
and make the change made in this commit.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/fineract/pull/3881#issuecomment-2119835440,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A3E5KA3GALQ4XMK7WFUFNGDZDGQJDAVCNFSM6AAAAABHIBJQ4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZHAZTKNBUGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@bravosty this has to be done in the source code. https://github.com/apache/fineract

how then will I add this change to the Mifos installation?
On Mon, May 20, 2024 at 10:43 AM Wilfred Kigenyi @.***>
wrote:

@bravosty https://github.com/bravosty this has to be done in the source
code. https://github.com/apache/fineract
—
Reply to this email directly, view it on GitHub
https://github.com/apache/fineract/pull/3881#issuecomment-2119864037,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A3E5KAZOBPAFWRXAMNKZM53ZDGSQ5AVCNFSM6AAAAABHIBJQ4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZHA3DIMBTG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

@bravosty you would have to build your own jar/war. The alternative is to wait for for this PR to be merged and Fineract 1.10 released.

@adamsaghy license added

@wkigenyi Please kindly check the failing test case.
If you want to test locally, you can run the fineract backend and then configure your intelliJ to run the test case with IntelliJ Runner:

@adamsaghy I improved the test

well done @wkigenyi, I am excited about the final fineract-provider.war file, I love the developments I am seeing, keep it up!

@adamsaghy I need some help here. getTransactionsForCashier(tellerId, cashierId, currencyCode, offset, limit, orderBy, sortOrder) returns an object (Page) and not a List. I have tried to change this but somewhere as the test is executed a List is expected. I am not sure where to fix.

Sorry for causing so much trouble for you. In the swagger file you can change it and then the fineract-client will generate the proper response object.
In the TellerApiResource line 305:
@ApiResponse(responseCode = "200", description = "OK", content = @Content(array = @ArraySchema(schema = @Schema(implementation = TellerApiResourceSwagger.GetTellersTellerIdCashiersCashiersIdTransactionsResponse.class)))) })
You can give a try to remove these annotations, so there will be no defined response object from the TellerApiResourceSwagger class and hopefully it will generate automatically the proper Response object the fineract-client. Alternatively you can change the annotations to point not as an ArraySchema, rather an (regular) schema (just remove the array parts) and change the TellerApiResourceSwagger.GetTellersTellerIdCashiersCashiersIdTransactionsResponse structure.
But i would say it is just easier to remove those annotations and let Fineract automatically generate the proper Response object ;)

@adamsaghy, done and with this PR I have discovered a lot about Fineract integration testing

@adamsaghy I think I am very close

@adamsaghy

@adamsaghy why are the last two checks here taking more than 24 hours ?
1.Fineract Build & Test - PostgreSQL / build
2.Fineract Docker build - PostgreSQL / build

@adamsaghy why are the last two checks here taking more than 24 hours ? 1.Fineract Build & Test - PostgreSQL / build 2.Fineract Docker build - PostgreSQL / build

Seems they stucked. I have restarted. I should be finished under 50 mins

@adamsaghy I have 1/8 failing
org.apache.fineract.client.util.CallFailedRuntimeException: HTTP failed: Request{method=GET, url=https://localhost:8443/fineract-provider/api/v1/tellers/1/cashiers/1/transactions?currencyCode=UGX&offset=0&limit=0, tags={class retrofit2.Invocation=org.apache.fineract.client.services.TellerCashManagementApi.getTransactionsForCashier() [1, 1, UGX, 0, 0, null, null]}}; Response{protocol=http/1.1, code=500, message=, url=https://localhost:8443/fineract-provider/api/v1/tellers/1/cashiers/1/transactions?currencyCode=UGX&offset=0&limit=0}; errorBody: <!doctype html>HTTP Status 500 – Internal Server Error

@wkigenyi Seems the cashier transaction API does not working with postgres database. Can you please take a look on it? It might requires different SQL for postgres database. You can find database dependent solution in the DatabaseSpecificSQLGenerator.
From the server logs it looks this exception:
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [select * from (select  txn.id as txn_id, txn.cashier_id as cashier_id,  txn.txn_type as txn_type,  txn.txn_amount as txn_amount, txn.txn_date as txn_date, txn.txn_note as txn_note,  txn.entity_type as entity_type, txn.entity_id as entity_id, txn.created_date as created_date,  o.id as office_id, o.name as office_name, t.id as teller_id, t.name as teller_name, s.display_name as cashier_name  from m_cashier_transactions txn  left join m_cashiers c on c.id = txn.cashier_id  left join m_tellers t on t.id = c.teller_id  left join m_office o on o.id = t.office_id  left join m_staff s on s.id = c.staff_id  where txn.cashier_id = ? and txn.currency_code = ? and o.hierarchy like ? AND ((case when c.full_day then Date(txn.created_date) between c.start_date AND c.end_date else ( Date(txn.created_date) between c.start_date AND c.end_date ) and ( TIME(txn.created_date) between TIME(c.start_time) AND TIME(c.end_time)) end) or txn.txn_type = 101))  cashier_txns  union (select  sav_txn.id as txn_id, null as cashier_id,  case      when renum.enum_value in ('deposit','withdrawal fee', 'Pay Charge', 'Annual Fee')          then 103      when renum.enum_value in ('withdrawal', 'Waive Charge', 'Interest Posting', 'Overdraft Interest', '')          then 104      else          105  end as txn_type,  sav_txn.amount as txn_amount, sav_txn.transaction_date as txn_date,  concat (renum.enum_value, ', Sav:', sav.id, '-', sav.account_no, ',Client:', cl.id, '-',cl.display_name) as txn_note,  'savings' as entity_type, sav.id as entity_id, sav_txn.created_date as created_date,  o.id as office_id, o.name as office_name, null as teller_id, null as teller_name, staff.display_name as cashier_name  from m_savings_account_transaction sav_txn  left join r_enum_value renum on sav_txn.transaction_type_enum = renum.enum_id and renum.enum_name = 'savings_transaction_type_enum'  left join m_savings_account sav on sav_txn.savings_account_id = sav.id  left join m_client cl on sav.client_id = cl.id  left join m_office o on cl.office_id = o.id  left join m_appuser user on sav_txn.created_by = user.id  left join m_staff staff on user.staff_id = staff.id  left join m_cashiers c on c.staff_id = staff.id  left join m_payment_detail payDetails on payDetails.id = sav_txn.payment_detail_id  left join m_payment_type payType on payType.id = payDetails.payment_type_id  left join m_account_transfer_transaction acnttrans  on (acnttrans.from_savings_transaction_id = sav_txn.id  or acnttrans.to_savings_transaction_id = sav_txn.id)  where sav_txn.is_reversed = false and c.id = ? and sav.currency_code = ? and o.hierarchy like ? and  sav_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day)  and renum.enum_value in ('deposit','withdrawal fee', 'Pay Charge', 'withdrawal', 'Annual Fee', 'Waive Charge', 'Interest Posting', 'Overdraft Interest')  and (sav_txn.payment_detail_id IS NULL OR payType.is_cash_payment = true)  AND acnttrans.id IS NULL )  union (select  loan_txn.id as txn_id, c.id as cashier_id,  case      when renum.enum_value in ('REPAYMENT_AT_DISBURSEMENT','REPAYMENT', 'RECOVERY_REPAYMENT', 'CHARGE_PAYMENT')          then 103      when renum.enum_value in ('DISBURSEMENT', 'WAIVE_INTEREST', 'WRITEOFF', 'WAIVE_CHARGES')          then 104      else          105  end as cash_txn_type,  loan_txn.amount as txn_amount, loan_txn.transaction_date as txn_date,  concat (renum.enum_value, ', Loan:', loan.id, '-', loan.account_no, ',Client:', cl.id, '-',cl.display_name) as txn_note,  'loans' as entity_type, loan.id as entity_id, loan_txn.created_date as created_date,  o.id as office_id, o.name as office_name, null as teller_id, null as teller_name, staff.display_name as cashier_name  from m_loan_transaction loan_txn  left join r_enum_value renum on loan_txn.transaction_type_enum = renum.enum_id and renum.enum_name = 'loan_transaction_type_enum'  left join m_loan loan on loan_txn.loan_id = loan.id  left join m_client cl on loan.client_id = cl.id  left join m_office o on cl.office_id = o.id  left join m_appuser user on loan_txn.created_by = user.id  left join m_staff staff on user.staff_id = staff.id  left join m_cashiers c on c.staff_id = staff.id  left join m_payment_detail payDetails on payDetails.id = loan_txn.payment_detail_id  left join m_payment_type payType on payType.id = payDetails.payment_type_id  left join m_account_transfer_transaction acnttrans  on (acnttrans.from_loan_transaction_id = loan_txn.id  or acnttrans.to_loan_transaction_id = loan_txn.id)  where loan_txn.is_reversed = false and c.id = ? and loan.currency_code = ? and o.hierarchy like ? and  loan_txn.transaction_date between c.start_date and date_add(c.end_date, interval 1 day)  and renum.enum_value in ('REPAYMENT_AT_DISBURSEMENT','REPAYMENT', 'RECOVERY_REPAYMENT','DISBURSEMENT', 'CHARGE_PAYMENT', 'WAIVE_CHARGES', 'WAIVE_INTEREST', 'WRITEOFF')  and (loan_txn.payment_detail_id IS NULL OR payType.is_cash_payment = true)  AND acnttrans.id IS NULL )  union (select  cli_txn.id AS txn_id, c.id AS cashier_id,  case  when renum.enum_value in ('PAY_CHARGE')  then 103  when renum.enum_value in ('WAIVE_CHARGE')  then 104  else  105  end as cash_txn_type,  cli_txn.amount as txn_amount, cli_txn.transaction_date as txn_date,  concat (renum.enum_value, ', Client:', cl.id, '-', cl.account_no, ',Client:', cl.id, '-',cl.display_name) as txn_note,  'client' as entity_type, cl.id as entity_id, cli_txn.created_date as created_date,  o.id as office_id, o.name as office_name, null as teller_id, null as teller_name, staff.display_name as cashier_name  from m_client_transaction cli_txn  left join r_enum_value renum on cli_txn.transaction_type_enum = renum.enum_id AND renum.enum_name = 'client_transaction_type_enum'  left join m_client cl on cli_txn.client_id = cl.id  left join m_office o on cl.office_id = o.id  left join m_appuser user on cli_txn.created_by = user.id  left join m_staff staff on user.staff_id = staff.id  left join m_cashiers c on c.staff_id = staff.id  left join m_payment_detail payDetails on payDetails.id = cli_txn.payment_detail_id  left join m_payment_type payType on payType.id = payDetails.payment_type_id  where cli_txn.is_reversed = false and c.id = ? and cli_txn.currency_code = ? and o.hierarchy like ? and cli_txn.transaction_date  between c.start_date and date_add(c.end_date, interval 1 day)  and renum.enum_value in ('PAY_CHARGE', 'WAIVE_CHARGE')  and (cli_txn.payment_detail_id IS NULL OR payType.is_cash_payment = true) )  order by created_date ]
	at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:112)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:107)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:116)
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1548)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:677)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:723)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:754)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:767)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:825)
	at org.apache.fineract.infrastructure.core.service.PaginationHelper.fetchPage(PaginationHelper.java:44)
	at org.apache.fineract.organisation.teller.service.TellerManagementReadPlatformServiceImpl.retrieveCashierTransactions(TellerManagementReadPlatformServiceImpl.java:504)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:713)
	at org.apache.fineract.organisation.teller.service.TellerManagementReadPlatformServiceImpl$$SpringCGLIB$$0.retrieveCashierTransactions(<generated>)
	at org.apache.fineract.organisation.teller.api.TellerApiResource.getTransactionsForCashier(TellerApiResource.java:324)

@wkigenyi are you planning to fix the PostgreSQL version of the query?

Yes, though I would have preferred to write something that is dB independent. I am just a bit stuck but will see it through.

@wkigenyi I agree but it's a good strategy to first get it working and merged. Then as a step 2, you could try to change the query so it's DB independent.

Okay. Will find time for this and resolve it.

@wkigenyi any update here?

@wkigenyi any update here?

I have been a bit swamped. I have time this weekend to work on it.

@wkigenyi superb, as soon as you're ready, I can review it.

@wkigenyi superb, as soon as you're ready, I can review it.

There are two ways to create a cashier

Full Day
Not full day (you have to specify the startTime and endTime)

The second option does not seem to be supported anymore because when you provide values for fields when creating the cashier you get an error that they are not supported, when you leave them out in the request without setting isFullDay to true you will get an error message saying startTime is required.
I have noticed that the web-app UI does not have provision for startTime and endTime.
Should I therefore assume that these fields are no longer used when creating the cashier and rewrite the query without considering them?

@wkigenyi superb, as soon as you're ready, I can review it.
I have made some improvements but I am not sure if I squashed this correctly.

@wkigenyi formatting is off, please run a ./gradlew spotlessApply on the code.

It fails with

Task :fineract-tax:spotlessMiscApply FAILED


network: add a zebra_network::connect_isolated() method.

The peer set provides an automatically managed connection pool, abstracting
away all the details of handling individual peer connections.  However, it's
also useful to be able to create completely isolated and
minimally-distinguishable connections to individual peers, in order to be able
to send specific messages over Tor, or to implement some custom network crawler
logic.
Extracted from #1008.

@dconnolly @teor2345 I think that the best way to test this code is to have a test that attempts to use connect_isolated to connect to a local instance of the listener used by the peer set.  I'll implement that this week.

Added a test that the handshake's version message matches specified fields, but the test does not compile, because rustc doesn't believe that the Box<dyn std::error::Error + Send + Sync + 'static> is 'static, and therefore isn't a Box<dyn std::error::Error + Send + Sync + 'static>.  This manifests as being unable to spawn the connect_isolated task.  From digging through Tokio issues I believe that this is an instance of https://github.com/rust-lang/rust/issues/64552 .
I'm not sure how to move this forward in the presence of that bug.

@yaahc awesome! Looks like that change fixed the problem, and the test runs and passes now.


Fix crash

sudden crash caused by volume pull request
Updated clientVersion & fixed Android input error that caused 403 Error pull request

Both errors checked
Thanks to Rubén Ortiz

There are a lot of commits but they are only to synchronize the changes, only the last 2 are the fixes

Hmm... I've never experienced a crash from the normalize_volume option. Is this from a custom-implemented track? Because iirc (and even in index.d.ts), the volume property is non-nullable.
Other than that, LGTM

Would you mind rebasing for me and using cherry-pick rather than merging?

I saw it too late, sorry


Undo move of customisation code

Codecov Report

Merging #85 into v2-master will decrease coverage by <.01%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##           v2-master      #85      +/-   ##
=============================================
- Coverage      71.95%   71.94%   -0.01%
=============================================
  Files            685      685
  Lines          29565    29565
  Branches        6783     6783
=============================================
- Hits           21274    21272       -2
- Misses          8291     8293       +2


rework package structure and bump version, fixes #12

bumps version to 0.2
moves all code into sureal package
fixes installation via setup.py
CLI entry point now is sureal.main.main()
program can be run via "sureal"
add developer documentation
slightly rework README for installation
add requirements.txt file

@li-zhi Please test whether this is suitable for you. It all ran fine here and should be more in line with Python packaging guidelines. I don't know why there was a hack with config.py but it does not seem to be needed.


How to generate a .pem/.pfx and .key files from Azure Key Vault certificate in Linux

Troubleshooting
What would be the equivalent process to generate PEM/PFX and key files from Azure Key Vault in Linux? I know you can do this in Powershell, but I'm using a Linux hosted agent.
Thanks in advance.

Nevermind, I've solved it:
#!/bin/bash
echo ${1} > cert.encoded.pfx
base64 --decode cert.encoded.pfx >> cert.pfx
openssl pkcs12 -in cert.pfx -out cert.pem -nodes -nokeys -password pass: >/dev/null 2>&1
openssl pkcs12 -in cert.pfx -out cert.key -nodes -nocerts -password pass: >/dev/null 2>&1


Improvement: Value Edit value restoring

Hello!
Would it be possible to add extra icon to already existing (copy, edit) icons, which would restore edit to previous step?
Example: edit value of field amount from 320 -> 10. New icon would appear on hovering the field, which by pressing, we would revert the value 1 step back - from 10 to 320. If the value is back to original one, the icon disappears.
Can this be a new improvement, perhaps? In our line of work, we would appreciate this update. Thanks!

I think it is difficult to support this feature.

There is no easy way for users to customize the action buttons.
Reverting value can be troublesome, considering providing a universal solution for all users.

I can leave this issue open as a reminder to add the ability to customize the buttons.

I tried to implement the feature in a different way.

https://stackblitz.com/edit/textea-json-viewer-v3-b4wgxq-xa7j2z?file=pages%2Findex.js

This actually looks good. Is this available feature already? I would like to add it to our project.

It's available for V3 V4, simply copy over the example into your project, and it should work without problems.
However, this implementation is quite limited. Your data source needs to remain stable since it will restore the whole object, not just the "field". You can improve it by storing the changed path and its previous and new values and then applying the reverted changes back into the object.

Let me know if you need the improved version or if you need help understanding my rough explanation.

Considered this as completed


jetson agx xavier with latest OS isn't supported

could you extend on how to setup the deployment on Jetson with tegra 35.2.1 with gmsl camera, please?
given the objective is to get somehow a third party camera outputs within iven package using nvidia developers kit AGX Xavier with Tegra OS version 35.2.1
Thank you / arigato
er.log

folks from nvidia ask if you could update the code for compatibility with Jetpack 5
https://forums.developer.nvidia.com/t/tieriv-perception-build-error-tensorrt/247308/5

Hi, @AndreV84 . Thank you for your suggestion and sorry for my late reply.
We are currently working to support L4T 35.2 but it will take for a while. I appreciate your understanding.

@manato Thank you for your reply.
If there is anything that could be crowdsourced at nvidia devtalk forum do not hesitate to let me know, please.
We can try  to engage nvidia support possibly if there are some specific complications related to their technology maybe they will be able to add on resolution. However, it seems mostly the patch should address TensorRT version.
I will be looking forward to hear news from you.
Arigato


Make space after "FAIL:" optional

CMock may fail without a preceding space:
[...]\CMock\examples\temp_sensor\test\TestMain.c:14:testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted:FAIL:Function Executor_Run.  Called fewer times than expected.
[...]\CMock\examples\temp_sensor\test\TestTemperatureFilter.c:37:testShouldResetAverageIfPassedInfinityOrInvalidValue:FAIL: Expected -inf Was nan

Unity tools deal with this just fine, but here the regex was to strict.
(Above output is reproducible by duplicating line 21 here: https://github.com/ThrowTheSwitch/CMock/blob/master/examples/temp_sensor/test/TestMain.c#L21 and running rake in the temp_sensor folder)

Hi again, are you waiting on more changes before pushing an update to the vscode marketplace?

Hi, terribly sorry. Meant to include this in 0.0.6 in the marketplace but forgot to pull this commit before publishing.
Published 0.0.7 containing this fix now.


Switched to using the uWebSockets library for http.

We are currently using a custom http implementation on top of native sockets to provide the http server. This pr instead uses
the uWebSockets library to provide the http and routing implementation. The library was chosen for its advertised speed, being maintained and having a small list of dependencies (currently only zlib. We also need openssl for it, if we want ssl support).
I've currently done the bare minimum to get the library working, as it wasn't quite clear yet what library we want to use, and I was mainly curious about the usability of the library. There is a lot of old code that can be removed, and the integration into cmake should probably be changed (e.g. by using ExternalProject_Add).
@joka921 @hannahbast  What do you think about the library? Should I finish this pr, or do we want to use something else?

Thank you very much. I really like this so far.
I think for the Server business we need external dependencies, since the current implementation by POSIX-Sockets
is very hard to maintain and hacky.
The library you found also seems to be reasonable (small, fast, compatible licensing and well maintained).
Also This would help very much by indeed adding a Websocket API which I have somewhere on my list.
This would allow tracking the progress of queries in the UI or canceling queries etc.
@hannahbast Could you also comment on what you think about external HTTP libraries in general and this one in particular?

I also think this is a very good idea, I originally planned to use Beast HTTP that's currently being incubated for the boost project. Used that on another test, but yeah anything decently maintained will be leaps better than what's being used now, so I'm all for it!

I have also looked a tiny little bit into BEAST but I prefer this solution for the following reasons:

its lighter to integrate (BEAST basically requires all of Boost)
it seems much easier to use (the BEAST examples seemed to me as if you still had to do a lot of low-level things manually).

Ok, when I tried it it was still possible to use beast without the rest of boost and even a modified (i.e. don't spawn a thread per request) classic threaded model. IMHO that still fits QLever well since result generation does need CPU and so one thread per CPU with a shared accept() is both simple and also makes sure the system doesn't get overloaded (just last week spawning a thread per connection killed almost the enitre AWS us-east-1).
There should be a version of a wrapper for that still in the AD Gitlab but yeah I think Beast is definitely adopting the whole Boost stuff. So I think uWebSockets makes sense and if I skimmed the code right it would even still use the same threading model, though it's a bit hard to tell what that listen call does.

Thank your very much for this work.
In the meantime we have decided to not use uWebSockets for several reasons, but are using a server based on Boost::Beast.
Note that this does not speak against your code (which is of high quality, as usual), but against the underlying library.


Add support for reply lib to messagegui app

Adds support for the canned responses/reply lib to the default messagegui app.
By clicking on a notification to head to the notification settings, the user will see a "reply" menu item if the message can be replied to, which will trigger the reply library flow

@soneret was eager to try this in https://github.com/espruino/BangleApps/pull/3473#issuecomment-2227525727.
If I'm correct this update can be installed from https://whelanb.github.io/BangleApps/?id=messagegui , right @WhelanB ? :)
I want to try it some myself and we should wait for @gfwilliams to be back so he can chime in as well before we merge.

@soneret was eager to try this in #3473 (comment).
If I'm correct this update can be installed from https://whelanb.github.io/BangleApps/?id=messagegui , right @WhelanB ? :)
I want to try it some myself and we should wait for @gfwilliams to be back so he can chime in as well before we merge.

Yes! it's hosted there now if you want to point Gadgetbridge to that link :) Let me know if you hit any issues running it!

This looks great, thanks! Just the 2 comments above. It'll be good to have the ability to do proper replies.
I guess maybe later we might even want to make swipe right reply to a message if it supports replies?

I've just pushed an update to address the PR comments, and add swipe right to reply! spotted some issues around empty replies as well in the reply lib that are now handled

Looks great to me - thanks!

tested out the new version and found a bug, once you receive a message and reply to it, the message popup again and can not be dismissed (as in it remains in the messages app and can not be removed without disconnecting from the phone). This may just be an android issue, im not sure

Update: it seems to be an issue with how google messages handles replies via notifications, signal works fine the notification appears twice but its dismissed once I swipe left

Sorry for the slow reply on this - @RelapsingCertainly is it that the notification isn't actually dismissed on your phone after replying and so stays present on the Bangle? I've mostly testing on apps like Whatsapp, where the notification is dismissed by Whatsapp after replying and so gets deleted from the Bangle, but I know some apps (like Discord, which doesn't support reply over Bangle atm) keeps the notification around, so maybe I need to handle that case

When I get a text message on Google Messages, I can reply to it as usual and it sends successfully. However, the message notification repeats on my watch (and not on the phone), allowing me to reply again. Even after dismissing it on the watch, the notification remains in the Messages app.
I don’t think this is related to the recent pull since a similar issue occurred with music notifications not dismissing.

@RelapsingCertainly I'm not sure this is something others are seeing - but perhaps if you could post up a Gadgetbridge log file then we can see what's actually going on.

Oh that's weird. When I tried to recreate the bug it has seemingly fixed itself. I'm so confused but looks like no fix is needed anymore :/

Also thing to note: the bug also fixed itself on my "dynamic island" app so it might have been a bug from google and now its fixed

@WhelanB question: are there any plans re add the ability to dismiss the notification without replying? What I mean is sending an affermative action to open the app on my phone. Kind of how its like it is when you cant reply.
This is mostly useful when getting a notification but I dont have the time to respond but also dont want to clear the notification or have the flashing icon.

In future we could add a toggle for whether you want to be able to reply or ack the message from swipe right, but I think for this PR I'll keep the behaviour limited to swipe right to reply for messages that support it

email  and other apps reply possible e.g aliexpress
Android için BlueMail uygulamasını edinin
On 13 Ağu 2024 22:40 , at 22:40 , Brian Whelan @.***> yazdı:

In future we could add a toggle for whether you want to be able to
reply or ack the message from swipe right, but I think for this PR I'll
keep the behaviour limited to swipe right to reply for messages that
support it
--
Reply to this email directly or view it on GitHub:
https://github.com/espruino/BangleApps/pull/3513#issuecomment-2286997152
You are receiving this because you were mentioned.
Message ID: @.***>

@gfwilliams is there anything stoping this pull from being merged? I have been testing for a while and it seems to be ready for stable.

email  and other apps reply possible e.g aliexpress Android için BlueMail uygulamasını edinin  On 13 Ağu 2024 22:40 , at 22:40 , Brian Whelan @.> yazdı:
…
In future we could add a toggle for whether you want to be able to reply or ack the message from swipe right, but I think for this PR I'll keep the behaviour limited to swipe right to reply for messages that support it -- Reply to this email directly or view it on GitHub: #3513 (comment) You are receiving this because you were mentioned. Message ID: @.>

I think this would need to be handled on the Gadgetbridge side, rather than here. I'd like to take a look at at some point since it has some support for canned responses already, and doesn't support some apps like Discord currently.

@gfwilliams is there anything stoping this pull from being merged? I have been testing for a while and it seems to be ready for stable.

Yeah, I don't think there's much more on my side for this PR, so if it's looking good would be great to get this merged!

Thanks - yes, this looks good to me. Merging now!
Sorry for the delay, I've been on/off holiday a bit the last few weeks

is it just me or is the updated version not showing up on app loader?

Check the Development App Loader - accessible from the "More..." tab of the official App Loader. Apps/updates show up there virtually as soon as they are merged here in the repo.
Gordon updates the official App Loader manually with some delay so we get a chance to fix instabilities if they appear on the development one.


merge dynamic_link into feat/1.5.0_dynamiclink (copy of feat/bump_1.5.0)

Description

Closes #0
Types of changes

[ ] Bug fix (changes which fixes an issue)
[ ] New feature (changes which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] ETC (build, ci, docs, perf, refactor, style, test)

Checklist

[ ] I followed the contributing guidelines.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] The PR title and commits are followed conventional commit form.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.5 out of 6 committers have signed the CLA.:white_check_mark: loloicci:white_check_mark: da1suk8:white_check_mark: dudong2:white_check_mark: kokeshiM0chi:white_check_mark: 170210:x: Jiyong HaJiyong Ha seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.