Add hyperspectral data extractor to pipeline

The steps for converting the sensor binary output to a netcdf data product are documented (https://github.com/terraref/documentation/blob/master/hyperspectral_data_pipeline.md).
Next steps:

[ ] Create extractor (docker image?) contact @czender w/ questions (#86)

requires development version of nco installed, on Roger or in Docker container (?)

git clone git@github.com:/nco/nco.git
cd nco
configure --prefix=/usr/local
make install.

[ ] add converter to Clowder
[ ] run on files on Roger
[ ] document

@yanliu-chn what is the status of this? I know the extractor isn't finished, but can it be added as an extractor and tested out on data so that we have some examples to show?

Is this related to https://github.com/terraref/computing-pipeline/issues/86 ?

@yanliu-chn yes this issue follows #86 - that is to write the converter, this issue is to add it to the pipeline.

@yanliu-chn - what is the status on this?

Now that #86 is in good shape, our team will work with Clowder team and Charlie/Jerome to develop the extractor at dataset level.
The extractor will watch for the readiness of a couple files in a dataset; call Charlie's script to create netcdf files and write them back to the dataset.

@yanliu-chn - please have your team update this issue?

@rachelshekar Thanks for you reminder, I'm assigning this task to Xingchen @zodiase

@yanliu-chn I did not realize this was already assigned to you when I spoke with @czender yesterday and offered to look into this.
https://github.com/terraref/computing-pipeline/tree/hyperspectral_extractor/scripts/hyperspectral/extractor
I have committed an initial draft of an extractor for the terraref.sh script, but it doesn't currently work. I haven't tried to really dig into charlie's script, but I have the extractor downloading a _raw and raw.hdr file and invoking the script to encounter this error:
found raw file: 73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
found hdr file: 73d03a9d-a7e0-4147-864d-801f4e5d0083_raw.hdr
invoking terraref.sh for: 73d03a9d-a7e0-4147-864d-801f4e5d0083.nc4
Input #00: /var/folders/1r/nh9w8jks7rx3t8k9jvlffg4r0000gn/T/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(in)  : /var/folders/1r/nh9w8jks7rx3t8k9jvlffg4r0000gn/T/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(out) : /var/folders/1r/nh9w8jks7rx3t8k9jvlffg4r0000gn/T/terraref_tmp_trn.nc.pid93170.fl00.tmp
./terraref.sh: line 480: ncks: command not found
terraref.sh: ERROR Failed to translate raw data. Debug this:
ncks -O --trr_wxy=955,1600,0 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/var/folders/1r/nh9w8jks7rx3t8k9jvlffg4r0000gn/T/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw ~/nco/data/in.nc /var/folders/1r/nh9w8jks7rx3t8k9jvlffg4r0000gn/T/terraref_tmp_trn.nc.pid93170.fl00.tmp

It looks like maybe a hard-coded path is causing problems? In any case, I will stop working on this branch if @Zodiase is going to look into it.

This is a $PATH issue. terraref.sh does this on roger:
module add gdal-stack-2.7.10
module add netcdf nco
Basically it needs a current (4.6.0+) version of NCO.

@czender I'm a bit confused by the two parallel threads between #86 and this. I'd like to sort this out: is the steps:

git clone git@github.com:/nco/nco.git
cd nco
configure --prefix=/usr/local
make install.

mentioned in OP still required, given

I rebuilt nco 4.6.0 with ncap2 and antlr-2.7.7

from @yanliu-chn in #86?

No. The OP shows how to build/install NCO yourself. #86 shows that @yanliu-chn did this on roger, and packaged it appropriately. terraref.sh needs NCO (and thus ncks) on its path. It doesn't matter from where. "module add nco" is one option (on roger).

https://github.com/terraref/computing-pipeline/tree/hyperspectral_extractor/scripts/hyperspectral/extractor

Archive.zip

@max-zilla
I was following the Brown Dog tutorial and encountered this error when trying the text "wc" example. Any ideas?
parallels@ubuntu:~/bd-extractors-template$ docker run --rm -i -t --link bdextractorstemplate_rabbitmq_1:rabbitmq example-extractor
INFO    : pyclowder.extractors -  Waiting for messages. To exit press CTRL+C
INFO    : pyclowder.extractors -  Registering extractor...
ERROR   : pyclowder.extractors -  Error in registering extractor: [Errno 2] No such file or directory: '/home/clowder/extractor_info.json'
INFO    : pyclowder.extractors -  Starting a New Thread for Process File
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyclowder/extractors.py", line 150, in thread_function
    inputfile = download_file(channel, header, host, secretKey, fileid, intermediatefileid,ext)
  File "/usr/local/lib/python2.7/dist-packages/pyclowder/extractors.py", line 483, in download_file
    stream=True, verify=_sslVerify)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 467, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='localhost', port=9000): Max retries exceeded with url: /api/files/57869efee4b0049e260eb3e0?key=r1ek3rs (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f31c970de50>: Failed to establish a new connection: [Errno 111] Connection refused',))

@max-zilla Thanks for creating a sample hyperspectral extractor. It appears you have copied terraref.sh, but none of the other hyperspectral scripts, into this new extractor directory. We (@FlyingWithJerome  and I) want all the hyperspectral scripts to run from the same directory in order to make it easier to install and to find all required scripts. Please either delete terraref.sh from the extractor directory, or add the extractor files to the directory that contains the terraref.sh source. terraref.sh will not run unless the *.py and *.nco scripts are in the same directory.

@Zodiase problem is that the docker container tries to connect to localhost (which is the container itself). When you connect to clowder you need to connect to http://:9000/ not localhost.

@czender thanks, meant to fix this earlier. Removed the duplicate of terraref.sh and added a config option to config.py where you can point to the correct directory. The default is "../" so it will look one level above the /extractor directory by default now.

@robkooper I'm following the BD tutorial here to the step Build and Start Example Extractor and so far I couldn't find anywhere to specify a different host option. Also from the log ConnectionError: HTTPConnectionPool(host='localhost', port=9000) it appears the extractor is connecting to port 9000. Are you saying it should connect to (host='', port=9000)?

@robkooper Also I'm wondering what Error in registering extractor: [Errno 2] No such file or directory: '/home/clowder/extractor_info.json' is about.

@robkooper
So after some digging in https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse/pyclowder/extractors.py I found the host option used for the HTTPConnection that is causing errors is passed in all the way from the message (https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/pyclowder/browse/pyclowder/extractors.py#91). How am I supposed to fix this?

@Zodiase did you see my comment about not using http://localhost:9000/ in your browser? You will need to use http://ipaddress:9000/ where ipaddress is the ipaddress of either your macine, or of your virtual machine where docker runs.

@robkooper Sorry I didn't realize you were talking about accessing clowder in browser with ip address. I took it as I should configure the extractor to connect to clowder with ip address.

@robkooper Using a real IP address the example extractor can finish its task. Although I still get an error:
parallels@ubuntu:~/bd-extractors-template$ docker run --rm -i -t --link bdextractorstemplate_rabbitmq_1:rabbitmq example-extractor
INFO    : pyclowder.extractors -  Waiting for messages. To exit press CTRL+C
INFO    : pyclowder.extractors -  Registering extractor...
ERROR   : pyclowder.extractors -  Error in registering extractor: [Errno 2] No such file or directory: '/home/clowder/extractor_info.json'
INFO    : pyclowder.extractors -  Starting a New Thread for Process File
{'channel': <pika.adapters.blocking_connection.BlockingChannel object at 0x7f49460d8790>, u'filename': u'foo.txt', u'secretKey': u'r1ek3rs', 'header': <BasicProperties(['content_type=application\\json', 'correlation_id=047480bc-4026-40dc-9537-81d5770c88d5', 'reply_to=amq.gen-zFP9E0oHDjP0fuzjCwhwlg'])>, u'host': u'http://<IP_ADDRESS>:9000', u'flags': u'', u'fileSize': u'9', u'intermediateId': u'57891a81e4b0049e1326d3cd', 'inputfile': u'/tmp/tmp42RUGx.txt', u'id': u'57891a81e4b0049e1326d3cd', u'datasetId': u'57868281e4b0049e260eb382', 'fileid': u'57891a81e4b0049e1326d3cd'}

Do you know the cause of this?

You can ignore the ERROR, that is bascially some code we have to register the extractors with a centralized service. In this case it could not find the extractor definition file, for example see: https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/extractors-dbpedia/browse/extractor_info.json

@robkooper Do you know where all the message types are documented?

@Zodiase they are not documented anywhere yet (I pointed this out recently and intend to create such a list at some point). One thing you can do is go to the RabbitMQ Management Console > Queues and add a new Queue to the Clowder exchange that listens on #.* (aka it will receive every message). Then you can click stuff in Clowder and see when messages arrive on that queue. I believe Yaping has set this up before.

In general the messages are send to you as needed. The general format is:
*.X.Y.Z...
X == file/dataset/collection/space/metadata
if X == file the Y = mime.major & Z = mime.minor
if X == dataset then Y.Z is add.file and remove.file
if X == metadata then Y == add or remove

@robkooper @max-zilla Thanks for the info.
I'm also getting a new error that I did not see in the beginning. It started showing up after I made some changes to config.py and extractor.py in the template. But the problem is, the error is still there even after I have reset all my changes. Although the error doesn't seem to affect the extractor as the metadata is being written just fine.
ERROR   : pyclowder.extractors -  [57896073e4b0049e1326d54e] error processing
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pyclowder/extractors.py", line 93, in on_message
    filename=jbody['filename']
KeyError: 'filename'

which version of clowder do you have, make sure it is at least 0.9.3.

@robkooper From the web UI it says 0.9.x#274 branch:develop sha1:1c1a81c

@robkooper Also there's something strange: the extractor still responds to uploading .txt files when its messageType has been set to *.file.image.#. Is that supposed to happen? I thought in extractors.connect_message_bus(extractorName=extractorName, messageType=messageType,                                    processFileFunction=process_file, rabbitmqExchange=rabbitmqExchange,                                    rabbitmqURL=rabbitmqURL) the point for passing in the message type is to filter the messages?

If you use the same queue name as before then yes. Once a binding is made between a queue and a message type that binding is there forever until it is forcefully removed. You can do this using the rabbitmq management interface (http://localhost:15672/ and u/p = guest/guest)

@Zodiase do you have my branch of pyclowder installed? Check out my last post here:
https://github.com/terraref/computing-pipeline/issues/64#issuecomment-232795385
You need a version of pyclower installed that supports dataset extractors - that error looks like you have the master branch installed. I mentioned it earlier in this topic but I think I edited it in later.

@max-zilla I followed the steps in that comment to install pyclowder on my testing environment, but I still get the same error. How do I confirm that the correct version is installed?

@Zodiase make a little python script with this (or open IDLE and type this in):
from pyclowder import extractors
extractors.get_dataset_info()

if you have the right version, you should see:
Traceback (most recent call last):
  File "<pyshell#9>", line 1, in <module>
    extractors.get_dataset_info()
TypeError: get_dataset_info() takes exactly 2 arguments (0 given)

If you have wrong version, you should see:
Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    extractors.get_dataset_infoz()
AttributeError: 'module' object has no attribute 'get_dataset_info'

...if you get the 2nd message, it could be that the extractor is somehow calling a different installation of Python than the one you installed pyclowder into.

@max-zilla And I get neither:
parallels@ubuntu:~$ python test-pyclowder.py
Traceback (most recent call last):
  File "test-pyclowder.py", line 1, in <module>
    from pyclowder import extractors
  File "/usr/local/lib/python2.7/dist-packages/pyclowder/__init__.py", line 4, in <module>
    from .extractors import *
  File "/usr/local/lib/python2.7/dist-packages/pyclowder/extractors.py", line 1, in <module>
    import pika
ImportError: No module named pika

@Zodiase ok, that's interesting because in the error you posted above (KeyError: filename) it would've had to import pika before getting that far.
I suspect that when you run python from command line and when you are running extractor, there are 2 different pythons. Looks like you have pika installed but the old pyclowder for extractor, and on "main" installation (that's being used to run test-pyclowder.py) it doesn't have pika installed.
Looks like you used the BrownDog tutorial to run your extractor in Docker, which I have not done. I think you should be able to change this step:
docker build -t example-extractor .

...specifically the Dockerfile you're using to build that extractor. You can include the code about git clone, git checkout in the Dockerfile to install the correct pyclowder there. If you can't get it to work, let's meet in my office tomorrow morning to get it figured out.

@max-zilla With the updated DockerFile I passed that error. Thanks!

@Zodiase can you give an update, or point me to your code? You can create a new branch on computing-pipeline and push it there if you haven't put it somewhere else.
@czender no updates since 3 days ago, but I will work with him to get something posted next week (I'm out of the office until Tues morning).

@max-zilla @czender My last comment was the point where I finally finished setting up the Docker environment correctly and received a dataset message with the sample wordcount extractor. After that I had to work on another project for the CyberGIS 16 conference so no update so far.

Test run on ROGER:
[yanliu@cg-cmp13 hyperspectral] module purge
[yanliu@cg-cmp13 hyperspectral] module load gdal-stack-2.7.10 nco
[yanliu@cg-cmp13 hyperspectral] cd /projects/arpae/sw/computing-pipeline/scripts/hyperspectral/
[yanliu@cg-cmp13 hyperspectral] /projects/arpae/sw/computing-pipeline/scripts/hyperspectral/terraref.sh -d 1 -O /gpfs_scratch/arpae/tmp/hyperspectral/output -I /gpfs_scratch/arpae/tmp/hyperspectral/input

Terraref hyperspectral data pipeline invoked with:
terraref.sh -d 1 -O /gpfs_scratch/arpae/tmp/hyperspectral/output -I /gpfs_scratch/arpae/tmp/hyperspectral/input
NCO version "4.6.0" from directory /gpfs/smallblockFS/sw/nco-4.6.0/bin
Pipeline scripts in directory /gpfs/largeblockFS/projects/arpae/sw/computing-pipeline/scripts/hyperspectral
Output stored in directory /gpfs_scratch/arpae/tmp/hyperspectral/output
Temporary files written to directory /gpfs_scratch/arpae/imaging_spectrometer
Input #00: /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(in)  : /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid86252.fl00.tmp
ncks -O --trr_wxy=955,1600,468 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw ~zender/nco/data/in.nc /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid86252.fl00.tmp
att(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid86252.fl00.tmp
att(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp
ncatted -O --gaa terraref_script=terraref.sh --gaa terraref_hostname=cg-cmp13 --gaa terraref_version="4.6.0" -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Wed Jul 27 14:18:38 CDT 2016: terraref.sh -d 1 -O /gpfs_scratch/arpae/tmp/hyperspectral/output -I /gpfs_scratch/arpae/tmp/hyperspectral/input" /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_trn.nc.pid86252.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp
jsn(in)  : /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
jsn(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252
python /gpfs/largeblockFS/projects/arpae/sw/computing-pipeline/scripts/hyperspectral/JsonDealer.py /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp
mrg(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp
mrg(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp
ncks -A /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp
clb(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp
clb(out) : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid86252.fl00.tmp
ncap2 -O -S /gpfs/largeblockFS/projects/arpae/sw/computing-pipeline/scripts/hyperspectral/terraref.nco /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid86252.fl00.tmp

rip(in)  : /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid86252.fl00.tmp
rip(out) : /gpfs_scratch/arpae/tmp/hyperspectral/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc
mv /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid86252.fl00.tmp /gpfs_scratch/arpae/tmp/hyperspectral/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc
Cleaning-up intermediate files...
Quick views of last processed data file and its original image (if any):
ncview  /gpfs_scratch/arpae/tmp/hyperspectral/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc &
panoply /gpfs_scratch/arpae/tmp/hyperspectral/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc &
open /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_image.jpg
Elapsed time 1m3s

[yanliu@cg-gpu01 hyperspectral]$ **gdalinfo output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc**

Driver: netCDF/Network Common Data Format
Files: output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc
Size is 512, 512
Coordinate System is `'
Metadata:
  NC_GLOBAL#Conventions=CF-1.5
  NC_GLOBAL#created_by=yanliu
  NC_GLOBAL#history=Wed Jul 27 14:18:51 2016: ncap2 -O -S /gpfs/largeblockFS/projects/arpae/sw/computing-pipeline/scripts/hyperspectral/terraref.nco /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_clb.nc.pid86252.fl00.tmp
Wed Jul 27 14:18:50 2016: ncks -A /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_att.nc.pid86252.fl00.tmp
Wed Jul 27 14:18:48 2016: python /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp
  NC_GLOBAL#history_of_appended_files=Wed Jul 27 14:18:50 2016: Appended file /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp had following "history" attribute:
Wed Jul 27 14:18:48 2016: python /gpfs_scratch/arpae/tmp/hyperspectral/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /gpfs_scratch/arpae/imaging_spectrometer/terraref_tmp_jsn.nc.pid86252.fl00.tmp
NC_GLOBAL#NCO="4.6.0"
NC_GLOBAL#nco_openmp_thread_number=1
NC_GLOBAL#Project=TERRAREF
NC_GLOBAL#terraref_hostname=cg-cmp13
NC_GLOBAL#terraref_script=terraref.sh
NC_GLOBAL#terraref_version=4.6.0
NC_GLOBAL#title=None given (supply with --trr ttl="Title")
Subdatasets:
SUBDATASET_1_NAME=NETCDF:"output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc":rfl_img
SUBDATASET_1_DESC=[955x468x1600] rfl_img (32-bit floating-point)
SUBDATASET_2_NAME=NETCDF:"output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc":xps_drk
SUBDATASET_2_DESC=[955x1600] xps_drk (16-bit unsigned integer)
SUBDATASET_3_NAME=NETCDF:"output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc":xps_wht
SUBDATASET_3_DESC=[955x1600] xps_wht (16-bit unsigned integer)
SUBDATASET_4_NAME=NETCDF:"output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc":xps_img
SUBDATASET_4_DESC=[955x468x1600] xps_img (16-bit unsigned integer)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

A couple of new issues:
I'm mostly adopting @max-zilla 's extractor code.
subprocess.call(["bash", os.path.join(scriptDirectory, "terraref.sh"), "-i", rawFilePath, "-o", outFilePath])
After invoking terraref.sh, I get this:
invoking terraref.sh to create: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083.nc4
dirname: missing operand
Try 'dirname --help' for more information.
ERROR: Unable to find NCO, nco_exe =

which has two errors:

dirname is missing an operand.
nco is not installed.

I have no idea what to do with the first error. For the second, I tried to install nco in the Dockerfile with:
RUN NCODIR=/home/nco && \
    git clone https://github.com/nco/nco.git $NCODIR && \
    cd $NCODIR && \
    configure --prefix=/usr/local && \
    make install

and then I get:
Step 5 : RUN NCODIR=/home/nco &&     git clone https://github.com/nco/nco.git $NCODIR &&     cd $NCODIR &&     configure --prefix=/usr/local &&     make install
 ---> Running in 3c89052ab307
Cloning into '/home/nco'...
/bin/sh: 1: configure: not found
The command '/bin/sh -c NCODIR=/home/nco &&     git clone https://github.com/nco/nco.git $NCODIR &&     cd $NCODIR &&     configure --prefix=/usr/local &&     make install' returned a non-zero code: 127

What does configure: not found mean?

New error:
Step 5 : RUN NCODIR=/home/nco &&     git clone https://github.com/nco/nco.git $NCODIR &&     cd $NCODIR &&     sh configure --prefix=/usr/local &&     make install
 ---> Running in 7544e0084e67
Cloning into '/home/nco'...
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking how to run the C preprocessor... gcc -E
checking how to run the C++ preprocessor... g++ -E
checking for gcc option to accept ISO C99... none needed
checking for gcc option to accept ISO Standard C... (cached) none needed
checking for flex... no
checking for lex... no
checking for bison... no
checking for byacc... no
checking whether ln -s works... yes
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
configure: line 8270: /usr/bin/file: No such file or directory
checking for mt... no
checking if : is a manifest tool... no
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether to enable maintainer-specific portions of Makefiles... no
checking for working memcmp... yes
checking whether lstat correctly handles trailing slash... yes
checking whether stat accepts an empty string... no
checking for working strtod... yes
checking for canonicalize_file_name... yes
checking for floor... no
checking for gethostname... yes
checking for getpagesize... yes
checking for getrusage... yes
checking for memchr... yes
checking for memmove... yes
checking for memset... yes
checking for mkstemp... yes
checking for pow... no
checking for readlink... yes
checking for realpath... yes
checking for regcomp... yes
checking for setlocale... yes
checking for sqrt... no
checking for strchr... yes
checking for strerror... yes
checking for strpbrk... yes
checking for strrchr... yes
checking for strstr... yes
checking for strtol... yes
checking for ANSI C header files... (cached) yes
checking getopt.h usability... yes
checking getopt.h presence... yes
checking for getopt.h... yes
checking libintl.h usability... yes
checking libintl.h presence... yes
checking for libintl.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for strings.h... (cached) yes
checking for unistd.h... (cached) yes
checking cstdlib usability... yes
checking cstdlib presence... yes
checking for cstdlib... yes
checking for an ANSI C-conforming const... yes
checking for inline... inline
checking for pid_t... yes
checking for size_t... yes
checking whether struct tm is in sys/time.h or time.h... time.h
checking for uid_t in sys/types.h... yes
checking for ptrdiff_t... yes
checking size of int*... 8
checking whether C++ compiler implements namespaces... yes
checking whether C++ compiler has working valarray<T>... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
#################################
#  Test for netCDF             #
################################
checking for /opt/local/include/netcdf.h... no
checking for /opt/local/lib/libnetcdf.a... no
checking netcdf.h usability... no
checking netcdf.h presence... no
checking for netcdf.h... no
configure: error: in `/home/nco':
configure: error: cannot find netCDF header
See `config.log' for more details
The command '/bin/sh -c NCODIR=/home/nco &&     git clone https://github.com/nco/nco.git $NCODIR &&     cd $NCODIR &&     sh configure --prefix=/usr/local &&     make install' returned a non-zero code: 1

@yanliu-chn that's good news that the hyperspectral extractor works from system directories.
@Zodiase while it's good to know how to build NCO, it's unclear to me why you need to do it. Presumably installation of NCO in system directories should be independent of the extractor. Or am I missing something?

@czender I think the NCO build is taking place inside Docker - Xingchen is creating a Docker container that will contain the extractor + all dependencies as a single VM essentially so it's easier to port. @Zodiase I can take a look with you tomorrow if you have time.

I can also find a time to create the Dockerfile for running the extractor. I built all the needed software on ROGER and can rebuild in a Ubuntu environment.

OK, that makes good sense. I can see how it would be useful to have it all in a Docker container, even though I personally have never used them.

@Zodiase your last message indicates that the NCO build fails because netcdf is not installed. That is a pre-requisite. See the NCO home page for a list of all prerequisites.

@czender I changed to using apt-get install nco instead of building it myself and passed that error. Now here comes a new one:
invoking terraref.sh to create: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083.nc4
dirname: missing operand
Try 'dirname --help' for more information.
Input #00: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(in)  : /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp
ncks: unrecognized option '--trr_wxy=955,1600,0'
ncks Command line options cheatsheet (full details at http://nco.sf.net/nco.html#ncks):
ncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [-b fl_bnr] [--bfr sz] [-C] [-c] [--cdl] [--cnk_byt sz] [--cnk_dmn nm,sz] [--cnk_map map] [--cnk_min min] [--cnk_plc plc] [--cnk_scl sz] [-D nco_dbg_lvl] [-d ...] [-F] [--fix_rec_dmn dim] [--fl_fmt fmt] [-G grp:lvl] [-g ...] [--glb ...] [--grp_xtr_var_xcl] [-H] [-h] [--hdn] [--hdr_pad nbr] [-L lvl] [-l path] [-M] [-m] [--map map.nc] [--md5_dgs] [--md5_wrt] [--mk_rec_dmn dim] [--msa] [--no_blank] [--no_tmp_fl] [-O] [-o out.nc] [-P] [-p path] [--ppc ...] [-Q] [-q] [-R] [-r] [--rad] [--rgr] [--ram_all] [--rnr=wgt] [-s format] [-t thr_nbr] [-u] [--unn] [-V] [-v ...] [-X box] [-x] [--xml] [--xml_no_loc] [--xml_spr_chr sng] [--xml_spr_nmr sng] [--xtn_var ...] in.nc [[out.nc]]

-3, --3, --fl_fmt=classic	Output file in netCDF3 CLASSIC (32-bit offset) storage format
-4, --4, --netcdf4		Output file in netCDF4 (HDF5) storage format
-5, --5, 		Print alphabetically by group then variable
-6, --6, --64, --fl_fmt=64bit	Output file in netCDF3 64-bit offset storage format
-7, --7, --fl_fmt=netcdf4_classic	Output file in netCDF4 CLASSIC format (3+4=7)
-A, --apn, --append	Append to existing output file, if any
-a, --abc, --alphabetize	Disable alphabetization of extracted variables
-b, --fl_bnr, --binary-file fl_bnr	Unformatted binary file to write
    --bfr_sz, --buffer_size sz	Buffer size to open files with
-C, --nocoords		Associated coordinate variables should not be processed
-c, --crd, --coords	Coordinate variables will all be processed
    --cdl		Print CDL (netCDF lingua franca used by ncdump/ncgen)
    --cnk_dmn, --chunk_dimension nm,sz	Chunksize of dimension nm is sz
    --cnk_map, --chunk_map map		Chunking map [dmn,lfp,prd,rd1,rew,scl,xpl,xst]
    --cnk_min, --chunk_min min		Minimum size [B] of variable to chunk
    --cnk_plc, --chunk_policy plc	Chunking policy [all,g2d,g3d,xpl,xst,uck]
    --cnk_scl, --chunk_scalar sz	Chunksize scalar (for all dimensions)
-D, --dbg_lvl, --debug-level lvl	Debug-level is lvl
-d, --dmn, --dimension dim,[min][,[max]][,[stride]] Dimension's limits and stride in hyperslab
-F, --ftn, --fortran	Fortran indexing conventions (1-based) for I/O
    --fix_rec_dmn dim	Change dimension dim (or all) to fixed dimension in output file
    --fl_fmt, --file_format fmt	File format [classic,64bit,netcdf4,netcdf4_classic]
-G, --gpe [grp_nm][:[lvl]]	Group Path Editing path, levels to replace
-g, --grp grp1[,grp2[...]] Group(s) to process (regular expressions supported)
    --glb_att_add nm=val	Global attribute to add
-H, --data, --hieronymus	Toggle printing data
-h, --hst, --history	Do not append to "history" global attribute
    --hdn, --hidden	Print hidden (aka special) attributes
    --hdr_pad, --header_pad	Pad output header with nbr bytes
-L, --dfl_lvl, --deflate lvl	Lempel-Ziv deflation/compression (lvl=0..9) for netCDF4 output
-l, --lcl, --local path	Local storage path for remotely-retrieved files
-M, --Mtd, --Metadata	Toggle printing global metadata
-m, --mtd, --metadata	Toggle printing variable metadata
    --map, --rgr_map map.nc	File containing (ESMF- or SCRIP-format) weights to regrid input to output grid
    --mk_rec_dmn dim	Define dim as record dimension in output file
    --no_blank		Print numeric missing values instead of blanks (underscores)
    --no_tmp_fl		Do not write output to temporary file
-o, --output, --fl_out out.nc	Output file name (or use last positional argument)
-O, --ovr, --overwrite	Overwrite existing output file, if any
-P, --prn, --print	Print data, metadata, and units. Abbreviation for -C -H -M -m -u.
-p, --pth, --path path	Path prefix for all input filenames
    --ppc v1[,v2[,...]]=prc	Precision-Preserving Compression: Number of total or decimal significant digits
-Q, --quiet		Quiet printing of dimension indices and coordinate values
-q, --quench		Quench (turn-off) all printing to screen
-R, --rtn, --retain	Retain remotely-retrieved files after use
-r, --revision, --version	Compile-time configuration and/or program version
    --orphan, --rph_dmn	Retain or print all (including orphaned) dimensions
    --ram_all, --diskless_all	Open netCDF3 files and create output files in RAM
    --rgr key=value		See http://nco.sf.net/nco.html#regrid for valid keys
    --rnr=wgt, --renormalize		Weight threshold for valid destination value
-s, --sng_fmt, --string format	String format for text output
-t, --thr_nbr, --threads, --omp_num_threads thr_nbr	Thread number for OpenMP
-u, --units		Toggle printing units of variables, if any
    --unn, --union	Select union of specified groups and variables
-V, --var_val		Print variable values only
-v, --variable var1[,var2[...]] Variable(s) to process (regular expressions supported)
-X, --auxiliary lon_min,lon_max,lat_min,lat_max	Auxiliary coordinate bounding box
-x, --xcl, --exclude	Extract all variables EXCEPT those specified with -v
    --xml		Print XML (NcML, netCDF Markup Language)
    --xml_no_location	Omit NcML location element
    --xml_spr_chr sng	Separator for NcML character types
    --xml_spr_nmr sng	Separator for NcML numeric types
    --xtn_var, --extensive var	Extensive variables for regridding (summed not averaged)
in.nc			Input file name
out.nc			Output file name (or use -o switch)

Eight ways to find more help on ncks and/or NCO:
1. Examples:     http://nco.sf.net/nco.html#xmp_ncks
2. Ref. manual:  http://nco.sf.net/nco.html#ncks
3. User Guide:   http://nco.sf.net#RTFM
4. Manual pages: 'man ncks', 'man nco', ...
5. Homepage:     http://nco.sf.net
6. FAQ:          http://nco.sf.net#FAQ
7. Help Forum:   http://sf.net/p/nco/discussion/9830
8. Publications: http://nco.sf.net#pub
Post questions, suggestions, patches at http://sf.net/projects/nco
terraref.sh: ERROR Failed to translate raw data. Debug this:
ncks -O --trr_wxy=955,1600,0 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw ~zender/nco/data/in.nc /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp

I'm still very concerned about the first error dirname: missing operand which seems to be from this line: https://github.com/terraref/computing-pipeline/blob/master/scripts/hyperspectral/terraref.sh#L65.

This error is related to #134 .

@Zodiase terraref.sh requires NCO 4.6.0 or newer and the "apt-get" retrieved an earlier NCO version. So you may need to build from source. Yes, the dirname issue is a consequence of #134 .

@czender I followed the instructions on the NCO website (http://nco.sourceforge.net/) for the prerequisites:
sudo aptitude install antlr libantlr-dev # ANTLR
sudo aptitude install libcurl4-gnutls-dev libexpat1-dev libxml2-dev # DAP-prereqs (curl, expat XML parser)
sudo aptitude install bison flex gcc g++ # GNU toolchain
sudo aptitude install gsl-bin libgsl0-dev # GSL
sudo aptitude install libnetcdf6 libnetcdf-dev netcdf-bin # netCDF and DAP
sudo aptitude install libhdf5-serial-dev # HDF5
sudo aptitude install ncl-ncarg # ESMF_RegridWeightGen (for ncremap)
sudo aptitude install udunits-bin libudunits2-0 libudunits2-dev # UDUnits

and hit an error:
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Reading package lists...
Building dependency tree...
Reading state information...
Package libnetcdf6 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  libnetcdf11

E: Package 'libnetcdf6' has no installation candidate
The command '/bin/sh -c apt-get install -y antlr libantlr-dev &&     apt-get install -y libcurl4-gnutls-dev libexpat1-dev libxml2-dev &&     apt-get install -y bison flex gcc g++ &&     apt-get install -y gsl-bin libgsl0-dev &&     apt-get install -y libnetcdf6 libnetcdf-dev netcdf-bin &&     apt-get install -y libhdf5-serial-dev &&     apt-get install -y ncl-ncarg &&     apt-get install -y udunits-bin libudunits2-0 libudunits2-dev' returned a non-zero code: 100

Then I tried:
parallels@ubuntu:~/bd-extractors-template$ apt-cache show libnetcdf6
N: Can't select versions from package 'libnetcdf6' as it is purely virtual
N: No packages found
parallels@ubuntu:~/bd-extractors-template$ apt-cache show libnetcdf11
N: Unable to locate package libnetcdf11
E: No packages found

Seems the instructions are outdated?

@Zodiase yes, these packages change names every few years so an update is needed. The offender appears to be libnetcdf6. Try libnetcdf or libnetcdf11 instead. I don't know what version of Devian/Ubuntu you're using, so I cannot be more definitive. When you find the right libnetcdf package, please let me know and I'll update the NCO homepage.

@czender I could find neither libnetcdf nor libnetcdf11 with aptitude show. As far as I understand, if aptitude show can't find the package info, it doesn't exist. However what confuses me is the error log However the following packages replace it:  libnetcdf11.

@czender After successfully installed NetCDF and NCO, I get this new error while running the extractor:
invoking terraref.sh to create: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083.nc4
dirname: missing operand
Try 'dirname --help' for more information.
Input #00: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(in)  : /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp
ncks: error while loading shared libraries: libnco-4.6.1-alpha02.so: cannot open shared object file: No such file or directory
terraref.sh: ERROR Failed to translate raw data. Debug this:
ncks -O --trr_wxy=955,1600,0 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw ~zender/nco/data/in.nc /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp

I assume that's the readlink issue?

@Zodiase the first problem is readlink. the second is due to a mismatch between the ncks executable and the shared nco library. either PATH or LD_LIBRARY_PATH must be changed to resolve this issue. also, please post the name of the ubuntu netcdf package you ended up using.

@czender I'm using libnetcdf11.

@czender Also could you explain the ~zender/nco/data/in.nc at https://github.com/terraref/computing-pipeline/blob/master/scripts/hyperspectral/terraref.sh#L491 ?

 Please pull latest hyperspectral pipeline from github and test and report any problems.

@czender The readlink issue seems to be gone. But I'm still solving the second one, the shared object file.

the problem may be related to these lines:
case "${HOSTNAME}" in
    cg-gpu* )
# 20160422: /usr/bin/python is version 2.6.6. Must load Python 2.7+
	module add gdal-stack-2.7.10
	module add netcdf nco

they are there to ensure NCO is found. but if you are building it yourself from scratch maybe the paths conflict...

@yanliu-chn @Zodiase any updates here?

@max-zilla
I think @yanliu-chn is helping me install the correct prerequisites in the Docker container. He's on vacation today though.

BTW the extractor code I've been playing with is here, along with some other scripts and notes I put down: https://gist.github.com/Zodiase/ddcbe628af93401260ef7b9d7e92b400

Dependencies seem to have been installed correctly.
New error:
invoking terraref.sh to create: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083.nc4
Input #00: /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(in)  : /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid10.fl00.tmp
jsn(in)  : /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid10
One or more important file(s) is(are) missing. Program terminated:
73d03a9d-a7e0-4147-864d-801f4e5d0083_raw.hdr is missing
73d03a9d-a7e0-4147-864d-801f4e5d0083_metadata.json is missing
73d03a9d-a7e0-4147-864d-801f4e5d0083_frameIndex.txt is missing
terraref.sh: ERROR Failed to parse JSON metadata. Debug this:
python /home/computing-pipeline/scripts/hyperspectral/JsonDealer.py /tmp/73d03a9d-a7e0-4147-864d-801f4e5d0083_raw /tmp/terraref_tmp_jsn.nc.pid10.fl00.tmp

Seems to be an issue with the way terraref.sh is invoked. @max-zilla Do you know what is the correct way of calling terraref.sh?

73d03a9d-a7e0-4147-864d-801f4e5d0083_raw.hdr doesn't seem to be missing to me. But the script still complains about it.

@czender any ideas? it looks like the files are indeed in /tmp but I'm guessing there's some kind of path confusion going on here.

there seems to be a combination of two problems.
the *.hdr file is clearly present.
in fact terraref.sh has actually read and used it before exiting.
the JsonDealer.py script is what emits the error message and terminates the program.
@FlyingWithJerome what triggers this message?
it looks like the test returns a false positive, at least for *.hdr.
the screenshot above shows that *.json and *.txt ARE missing, but the diagnostic says all three are missing. So, Jerome, please find/fix the diagnostic if possible. And @max-zilla and @Zodiase please place the *.json and *.txt files are are in the input directory before running terraref.sh.

@czender @max-zilla @Zodiase Investigating. Plan to use regex to match and capture the file names.

@FlyingWithJerome Sorry that seems to be caused by a typo.
@czender I'm trying with the complete set of input files that @yanliu-chn used above and got this error reporting insufficient memory. I'll try again after allocating more memory to the testing VM.
found _metadata.json file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_metadata.json
found _raw.hdr file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw.hdr
found _frameIndex.txt file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_frameIndex.txt
found _raw file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
invoking terraref.sh to create: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc4
Input #00: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(in)  : /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid15.fl00.tmp
ncks: ERROR nco_malloc() unable to allocate<PHONE_NUMBER> B = 1396687 kB = 1363 MB = 1 GB
ncks: INFO NCO has reported a malloc() failure. malloc() failures usually indicate that your machine does not have enough free memory (RAM+swap) to perform the requested operation. As such, malloc() failures result from the physical limitations imposed by your hardware. Read http://nco.sf.net/nco.html#mmr for a description of NCO memory usage. The likiest case is that this problem is caused by inadequate RAM on your system, and is not an NCO bug. If so, there are two potential workarounds: First is to process your data in smaller chunks, e.g., smaller or more hyperslabs. The second is to use a machine with more free memory, so that malloc() succeeds.

Large tasks may uncover memory leaks in NCO. This is likeliest to occur with ncap2. ncap2 scripts are completely dynamic and may be of arbitrary length and complexity. A script that contains many thousands of operations may uncover a slow memory leak even though each single operation consumes little additional memory. Memory leaks are usually identifiable by their memory usage signature. Leaks cause peak memory usage to increase monotonically with time regardless of script complexity. Slow leaks are very difficult to find.  If you have good reasons to believe that your malloc() failure is ultimately due to an NCO memory leak (rather than inadequate RAM on your system), then we would like to receive a detailed bug report.
terraref.sh: ERROR Failed to translate raw data. Debug this:
ncks -O --trr_wxy=955,1600,468 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /home/computing-pipeline/scripts/hyperspectral/dummy.nc /tmp/terraref_tmp_trn.nc.pid15.fl00.tmp

New Error:
found _frameIndex.txt file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_frameIndex.txt
found _image.jpg file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_image.jpg
found _raw.hdr file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw.hdr
found _raw file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
found _metadata.json file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_metadata.json
found _settings.txt file: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_settings.txt
invoking terraref.sh to create: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc4
Input #00: /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(in)  : /tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp
../computing-pipeline/scripts/hyperspectral/terraref.sh: line 506:    53 Killed                  ncks -O --trr_wxy=955,1600,468 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /home/computing-pipeline/scripts/hyperspectral/dummy.nc /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp
terraref.sh: ERROR Failed to translate raw data. Debug this:
ncks -O --trr_wxy=955,1600,468 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/tmp/extractor/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /home/computing-pipeline/scripts/hyperspectral/dummy.nc /tmp/terraref_tmp_trn.nc.pid10.fl00.tmp

@Zodiase I'm close to get a working Dockerfile file set up. Could you wait a little bit before diving into more bugs.

@Zodiase @yanliu-chn Is there still an issue? as you suggest, both failures above look like they are due to inadequate allocation of system resources.

@FlyingWithJerome does JsonDealer still produce false positives when looking for required input files?

@max-zilla @Zodiase @robkooper @czender I have successfully created the docker container of the hyperspectral image conversion. @Zodiase , please use this docker file as your development environment for extractor development. Let me know if you have any questions.
I created a branch to add the docker files:
https://github.com/terraref/computing-pipeline/tree/hyperspectral-extractor-dockerfile/scripts/hyperspectral/docker

@czender I'm debugging it. I tested it locally and it's all fine, and it will not warn on ROGER if all the files are there. I'm debugging it on ROGER and a possible reason would be Python OS module since the file checker largely depends on it. I will figure it out

@FlyingWithJerome that makes sense to me. do you suspect the python used in the extractor is earlier than version 2.7? or something else?

@czender In my expectation, any version of Python would fine. Since 2.7 is the most common one and numpy is not compatible 3.X version, 2.7 would be the best choice.

@yanliu-chn Could you rebase your Dockerfile on ncsa/clowder-extractors-python-base  as what I have in here? https://gist.github.com/Zodiase/ddcbe628af93401260ef7b9d7e92b400#file-dockerfile

there is no need to. i have included all the stuff the python based did in the current Dockerfile: https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/dockerfiles/browse/clowder/python-base/Dockerfile
You can modify my Dockerfile to include rabbitmq config. other than that, it's already in good shape. for the rabbitmq part, please refer to the plantcv extractor https://opensource.ncsa.illinois.edu/bitbucket/projects/BD/repos/dockerfiles/browse/clowder/plantcv/Dockerfile

please use my Dockerfile for extractor development, as it's been tested to work with terraref.sh.

@yanliu-chn So there are a few differences that I'm not sure how to handle.

USER clowder v.s. USER ubuntu. Is it fine if I replace all appearances of ubuntu with clowder?
entrypoint.sh. I believe that was in the extractor template.
CMD ["extractor"]. What does that do and what would the absence of that do?

i suggest to keep user 'ubuntu' because that is consistent with Kilo VM default user name. this way, we can use the Dockerfile as a user guide to create extractor VM, too.
be it entrypoint.sh or start.sh or anything else, it is just a script to be invoked by Docker at the end and keep it running
CMD ["script"] calls the script within the container after the container is up. RUN is used to build the container.

@yanliu-chn
For 2. I meant what do I do if the line ENTRYPOINT ["/home/clowder/entrypoint.sh"] is not in the Dockerfile? What difference would it make? I also could not find where I got that file. It's not in any of the extractor code directories. Should I add it to your Dockerfile?
For 3. Is there actually a extractor command in ncsa/clowder-extractors-python-base or does it just gets interpreted as python extractor.py?

@yanliu-chn
In other words and in short, the issue here is your Dockerfile seems to miss things that weren't written by me and I don't know what they do and I'm not sure how to modify your Dockerfile to make sure I don't either miss anything important or conflict with your logic.

for 2. you don't need to care about entrypoint. we don't need that for the extractor.
for 3. the CMD ["script.sh"] part in Dockerfile executes the extractor. So in start.sh or any script name you want, you should invoke the extractor code. For this extractor, like the following pseudo code:
cd /home/clower && . .softenv && . pyenv/bin/activate && python <your extractor.py>
To understand the Dockerfile, https://docs.docker.com/engine/reference/builder/ is a good reference.
All you need to do now is:

add rabbitmq Dockerfile config based on the plantcv Docerfile
develop the extractor python code in the container. see that now I only run the sleep command. so once the container is up, connect to it using a terminal and put your extractor code there. you can then manually run the extractor code to test.
after your extractor code runs well, slightly modify Dockerfile to run the extractor by default

were you able to start my docker container in your Docker environment and test the terraref.sh sample run?

The Dockerfile and extractor code temporarily stored in my notes were updated and now the extractor can finish running terraref.sh. Except the result (.nc) file is wrong due to a malloc error during a ncap2 process due to insufficient memory on my testing host machine.
INFO    : pyclowder.extractors -  Waiting for messages. To exit press CTRL+C
INFO    : pyclowder.extractors -  Registering extractor...
ERROR   : pyclowder.extractors -  Error in registering extractor: [Errno 2] No such file or directory: '/home/ubuntu/extractor_info.json'
INFO    : pyclowder.extractors -  Starting a New Thread for Process Dataset
{'files': [], 'download_bypassed': True, 'channel': <pika.adapters.blocking_connection.BlockingChannel object at 0x7f06af08e110>, 'filelist': [{u'filename': u'0596c17f-2e4c-4d43-9d77-cde8ffbde663_frameIndex.txt', u'date-created': u'Wed Aug 17 22:00:37 UTC 2016', u'contentType': u'text/plain', u'id': u'57b4de85e4b0049e7845295c', u'size': u'10603'}, {u'filename': u'0596c17f-2e4c-4d43-9d77-cde8ffbde663_image.jpg', u'date-created': u'Wed Aug 17 22:00:38 UTC 2016', u'contentType': u'image/jpeg', u'id': u'57b4de86e4b0049e78452963', u'size': u'40117'}, {u'filename': u'0596c17f-2e4c-4d43-9d77-cde8ffbde663_metadata.json', u'date-created': u'Wed Aug 17 22:00:44 UTC 2016', u'contentType': u'application/json', u'id': u'57b4de8ce4b0049e7845296a', u'size': u'1533'}, {u'filename': u'0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw.hdr', u'date-created': u'Wed Aug 17 22:00:52 UTC 2016', u'contentType': u'application/octet-stream', u'id': u'57b4de94e4b0049e78452971', u'size': u'10197'}, {u'filename': u'0596c17f-2e4c-4d43-9d77-cde8ffbde663_settings.txt', u'date-created': u'Wed Aug 17 22:00:54 UTC 2016', u'contentType': u'text/plain', u'id': u'57b4de96e4b0049e78452978', u'size': u'817'}, {u'filename': u'0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw', u'date-created': u'Wed Aug 17 22:01:14 UTC 2016', u'contentType': u'application/octet-stream', u'id': u'57b4deaae4b0049e7845297f', u'size': u'1430208000'}, {u'filename': u'foo.txt', u'date-created': u'Wed Aug 24 21:30:30 UTC 2016', u'contentType': u'text/plain', u'id': u'57be11f6e4b0049e15fe81ef', u'size': u'9'}], 'method': <Basic.Deliver(['consumer_tag=ctag1.58348bf89a444ad3a9a23e78f2fe58e2', 'delivery_tag=1', 'exchange=clowder', 'redelivered=False', 'routing_key=clowder.dataset.file.added'])>, u'secretKey': u'r1ek3rs', 'header': <BasicProperties(['content_type=application\\json', 'correlation_id=b23ae144-f1d1-46d0-a308-bf0b0011a816', 'reply_to=amq.gen-9SGIHuqFCaYqtFhQk89VCg'])>, u'host': u'http://<IP_ADDRESS>:9000', u'flags': u'', u'fileSize': u'9', u'intermediateId': u'57be11f6e4b0049e15fe81ef', 'datasetInfo': {u'description': u'', u'created': u'Wed Jul 13 18:03:45 UTC 2016', u'id': u'57868281e4b0049e260eb382', u'authorId': u'57866660292acbb6539f5e85', u'thumbnail': u'None', u'name': u'Hello World'}, 'filename': u'foo.txt', u'id': u'57be11f6e4b0049e15fe81ef', u'datasetId': u'57868281e4b0049e260eb382', 'fileid': u'57be11f6e4b0049e15fe81ef'}
found _frameIndex.txt file: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_frameIndex.txt
found _image.jpg file: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_image.jpg
found _raw.hdr file: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw.hdr
found _raw file: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
found _metadata.json file: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_metadata.json
found _settings.txt file: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_settings.txt
invoking terraref.sh to create: /home/ubuntu/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc4
Terraref hyperspectral data workflow invoked with:
terraref.sh -d 1 -I /home/ubuntu/input -O /home/ubuntu/output
Hyperspectral workflow scripts in directory /home/ubuntu/computing-pipeline/scripts/hyperspectral
NCO version "4.6.1" from directory /srv/sw/nco-4.6.1/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/ubuntu/output
Input #00: /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(in)  : /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid156.fl00.tmp
ncks -O --trr_wxy=955,1600,468 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /home/ubuntu/computing-pipeline/scripts/hyperspectral/dummy.nc /tmp/terraref_tmp_trn.nc.pid156.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid156.fl00.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid156.fl00.tmp
ncatted -O --gaa terraref_script=terraref.sh --gaa terraref_hostname=eb4936de98e3 --gaa terraref_version="4.6.1" -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Wed Aug 24 21:31:06 UTC 2016: terraref.sh -d 1 -I /home/ubuntu/input -O /home/ubuntu/output" /tmp/terraref_tmp_trn.nc.pid156.fl00.tmp /tmp/terraref_tmp_att.nc.pid156.fl00.tmp
jsn(in)  : /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid156
python /home/ubuntu/computing-pipeline/scripts/hyperspectral/JsonDealer.py /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_raw /tmp/terraref_tmp_jsn.nc.pid156.fl00.tmp
Processing ...
Done.
mrg(in)  : /tmp/terraref_tmp_jsn.nc.pid156.fl00.tmp
mrg(out) : /tmp/terraref_tmp_att.nc.pid156.fl00.tmp
ncks -A /tmp/terraref_tmp_jsn.nc.pid156.fl00.tmp /tmp/terraref_tmp_att.nc.pid156.fl00.tmp
clb(in)  : /tmp/terraref_tmp_att.nc.pid156.fl00.tmp
clb(out) : /tmp/terraref_tmp_clb.nc.pid156.fl00.tmp
ncap2 -A -S /home/ubuntu/computing-pipeline/scripts/hyperspectral/terraref.nco /tmp/terraref_tmp_att.nc.pid156.fl00.tmp /tmp/terraref_tmp_att.nc.pid156.fl00.tmp;/bin/mv -f /tmp/terraref_tmp_att.nc.pid156.fl00.tmp /tmp/terraref_tmp_clb.nc.pid156.fl00.tmp
ncap2: ERROR nco_malloc() unable to allocate<PHONE_NUMBER> B = 5586750 kB = 5455 MB = 5 GB
ncap2: INFO NCO has reported a malloc() failure. malloc() failures usually indicate that your machine does not have enough free memory (RAM+swap) to perform the requested operation. As such, malloc() failures result from the physical limitations imposed by your hardware. Read http://nco.sf.net/nco.html#mmr for a description of NCO memory usage. The likiest case is that this problem is caused by inadequate RAM on your system, and is not an NCO bug. If so, there are two potential workarounds: First is to process your data in smaller chunks, e.g., smaller or more hyperslabs. The second is to use a machine with more free memory, so that malloc() succeeds.

Large tasks may uncover memory leaks in NCO. This is likeliest to occur with ncap2. ncap2 scripts are completely dynamic and may be of arbitrary length and complexity. A script that contains many thousands of operations may uncover a slow memory leak even though each single operation consumes little additional memory. Memory leaks are usually identifiable by their memory usage signature. Leaks cause peak memory usage to increase monotonically with time regardless of script complexity. Slow leaks are very difficult to find.  If you have good reasons to believe that your malloc() failure is ultimately due to an NCO memory leak (rather than inadequate RAM on your system), then we would like to receive a detailed bug report.
rip(in)  : /tmp/terraref_tmp_clb.nc.pid156.fl00.tmp
rip(out) : /home/ubuntu/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc
/bin/mv -f /tmp/terraref_tmp_clb.nc.pid156.fl00.tmp /home/ubuntu/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc
Cleaning-up intermediate files...
Quick views of last processed data file and its original image (if any):
ncview  /home/ubuntu/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc &
panoply /home/ubuntu/output/0596c17f-2e4c-4d43-9d77-cde8ffbde663.nc &
open /home/ubuntu/input/0596c17f-2e4c-4d43-9d77-cde8ffbde663_image.jpg
Elapsed time 1m13s

ncap2 exhausted all available memory out of 8GB on the host machine.

I think this memory issue raises a good question: exactly how much memory is needed, when given a certain size of input? Say for example, given a x Bytes _raw file?

@Zodiase @czender
I did a simple memory profiling on Roger for Hyperspectral workflow, and here are some data might be useful for you.
I run the terraref.sh with a raw file around 1.6 GB (to be precise, 1.647GB, /projects/arpae/terraref/sites/ua-mac/raw_data/SWIR/2016-04-06/2016-04-06__18-18-30-855/*_raw) and monitored the memory with my Python script concurrently; the script gathers memory usage per .5 seconds.
The peak usage for the whole process is 6.3GB (6294MB @ ncap2). So obviously 4.5GB free memory is not enough to handle a 1.5GB raw file.
I would recommend you to test with raw files around 800MB; they are in the folders in May and June. The peak usage would be around 3GB, which should be affordable to your system.

@zodiase @FlyingWithJerome @yanliu-chn @dlebauer  I reduced the memory required by ncap2 from 5x sizeof(raw) to 4x sizeof(raw). best to allot 4.1x so as not to cut it too close. the footprint cannot be further reduced without significantly slowing the processing. half the memory holds the original image (promoted from short to float) and the other half holds the computed reflectance (float). so 4x raw image is the smallest natural size of the computation without using loops.
Note that required memory will increase if final form of hyperspectral calibration includes new (wavelength,x,y) arrays.

I think I've finished the extractor. See the above PR. However I did find something strange. The file uploaded by the extractor can not be deleted on the Clowder Web UI. It says no permission to delete.

What is the use case for manually deleting something generated by the
extractor?
On Wed, Aug 31, 2016 at 8:35 PM Xingchen Hong<EMAIL_ADDRESS>wrote:

I think I've finished the extractor. See the above PR. However I did find
something strange. The file uploaded by the extractor can not be deleted on
the Clowder Web UI. It says no permission to delete.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/terraref/computing-pipeline/issues/81#issuecomment-243951725,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcX59qxs5VcYBXVaqu1F6-Aabl__Pq-ks5qliu-gaJpZM4H6H2W
.

@dlebauer Perhaps someone would like to delete an incorrect result (for example, resulted from malloc failures) and rerun the extractor for that dataset?

Write access to directories used by extractors is very restricted. Only the
user who runs the extractor should be able to delete the output, and the
user accounts that run the extractors and create data are not the same as
the user accounts that humans will use. Probably would be a good idea if
extractors could only read inputs and could write outputs (preferably only
if there is no error).
How would the error be caught? If such an error occurs, can the extractor
clean up after itself and try again?
On Wed, Aug 31, 2016 at 10:01 PM Xingchen Hong<EMAIL_ADDRESS>wrote:

@dlebauer https://github.com/dlebauer Perhaps someone would like to
delete an incorrect result (for example, resulted from malloc failures) and
rerun the extractor for that dataset?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/terraref/computing-pipeline/issues/81#issuecomment-243962960,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcX55828kJ0uLBV9r-6ceEZBpoQH1s2ks5qlkAIgaJpZM4H6H2W
.

@Zodiase @dlebauer the reason for this is that the extractors currently upload to Clowder using the API key (not user/password), which is associated with an Anonymous User. @robkooper mentioned that Luigi is working on user-specific keys so we can have the extractor upload files as the Maricopa Site user, for instance (like the raw data) and that user would be able to delete and rerun if necessary.


Image alt text not persisted between 'Upload' and 'Choose format' dialogs.

When inserting an image into a RichTextField the alt text is not persisted between the 'Upload' tab on the 'Choose Image' dialog and the subsequent 'Choose a format', rather the default, image title is shown regardless of the alt text entered in the 'Upload' tab.
I think the fix is relatively simple:
initial={"alt_text": image.default_alt_text}, here
would become:
initial={"alt_text": image.alt_text if image.alt_text else image.default_alt_text},
I am happy to raise a PR for the fix but would like someone to confirm that the proposed solution above is appropriate.
Steps to Reproduce

Insert an image into a RichTextField using the default Wagtail RichText Editor

Choose the 'Upload' tab and enter details including the alt text, then press 'Upload'

After successfully uploading the image, the 'Choose a format' dialog is shown with the alt text set to the image's title.

Expected behaviour
The alt text entered in the 'Upload' tab (step 2) is shown in the alt text field of the 'Choose a format' dialog (step 3).
Technical details

Python version: 3.9.13
Django version: 4.1.10
Wagtail version: 4.2.0
Browser version: Chrome 114

The Upload form on a new Wagtail installation does not have an alt text field - I think this must be a customisation you've made with a custom image model:

Wagtail does not provide an alt text field on images as standard because we want to encourage editors to specify alt text that's appropriate for the context where it's used at the point of inserting it on the page, rather than globally per image.

No probs, thanks for looking and replying.


Add functionality to open the Settings UI tab through openSettings

This PR's goal was to add an option to the OpenSettings keybinding to open the Settings UI in a tab. In order to implement that, a couple of changes had to be made to Tab, specifically:

Introduce a tab interface named ITab
Create/Rename two new Tab classes that implement ITab called SettingsTab and TerminalTab

I've also put some TODOs that I wanted to get some thoughts on - I'll make a follow up PR but perhaps they can be revisited when we flesh out the Settings UI more.

Does a Settings UI tab shutdown need to do anything special for cleanup?
Does a Settings UI tab need to have GetActiveTitle? Maybe depending on which page in the UI is open?
Technically, I can't focus a Grid control, so I'll need to figure out what to Focus when the tab is selected.
The Settings UI tab doesn't have a TermControl, so once focus is moved to that tab, users won't be able to nextTab/prevTab out of it (along with all other keybindings).

References #1564, #5915

/azp run

Ignore me if this was on purpose … The PR has not been merged into master.

Hey uh, was this supposed to get merged? since it wasn't going into master, the bot wasn't going to enforce the two signoff requirement...

Oh yeah this was 100% intended to go into the feature branch for the SUI, not master. I got so scared waking up to these two comments, I thought it accidentally went into master instead 😆

To ruin your morning was 100% unintended 😦

We're just doing one sign-off needed for feature/settings-ui. We'll do a full review when that merges into master in preparation for release. So occasionally we'll tag team members for a closer look or some immediate feedback. We were all surprised that AutoMerge worked hahaha.

Ah okay. I was planning on giving feedback on this part, but nbd. It would have probably boiled down to "I have concerns, but we need to ship the SUI so :shipit:". The concerns aren't that important now TBH

@zadjii-msft they’re still worth tracking, so do write them up! Since this will merge to main eventually


Added Example to Upload Images using Ruby SDK

Fix #68
Similar to node sdk example to upload images to cloudinary. An example has been made for uploading images using Cloudinary Ruby SDK.
README file describes simple steps to get started on adding cloudinary to your Gemfile

hey @aazam-gh got it running and worked great! have a few comments here first before we merge it in. let me know what you think

@colbyfayock I have made some changes based on the reviews :)

looks great, thanks @aazam-gh

@aazam-gh congrats on the merged PR! since it was merged during Hacktoberfest, you're eligible for some free swag. Please email<EMAIL_ADDRESS>with your Github username and a link to this PR where we'll follow up with you there.
https://cloudinary.com/blog/hacktoberfest-celebrate-open-source-sdks

@all-contributors please add @aazam-gh for code

@colbyfayock Thanks for merging. I have sent an email :D


⚠ Spryr has degraded performance

In 3e60e8e, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 8616 ms

Resolved: Spryr performance has improved in dacc98f after 20 minutes.


Preferences window

Currently there is an implementation of the layout of a Preferences window, but it is still not implemented.
Here is a screenshot:

Really interesting!!! Usually I use Jdownloader when I want to extract the music track, cause JDownloader let me choose between ogg (opus), m4a (mp4) and other containers and the quality. It would be very interesting if your implementation let user not only select "Quality" but container and quality as kbps not as generic like "Medium", "Low" ... etc.
Thanks!!!

 Based on that information, it is not very easy to find the right format and bitrate pair as you are suggesting. To be able to choose the quality, I created a simple rating system that categorizes each stream and then it picks one for each quality level. It can happen for example to use the same stream in high and low quality. It will depend in how Youtube had store the video.
It is possible (and probably is useful) to implement what you suggested with the help of a transcoder, but I have no plans to that. It will add extra complexity with little in return.
But unfortunately I'm not been able to work on this project at all... :(

You must understand that for users looking for not proprietary format, to have the ogg format option (using opus codec) is very important. Really I don't know if JDownloader uses FFmpeg in background to make the conversion or is YouTube who make all possible format/bitrates available for download  :(


Refresh branch: faq folder files converted from .rst to .md

Converted all faq rst files into md (English).
Checked via local docker Jekyll renderer.

Please augment the commit messages following these suggestions as a guide - https://chris.beams.io/posts/git-commit/

Please augment the commit messages following these suggestions as a guide - https://chris.beams.io/posts/git-commit/

Please move the internationalization to a separate PR. For reference, here is a possible plugin we can use to help with this piece in jekyll - https://github.com/kurtsson/jekyll-multiple-languages-plugin.

Please move the internationalization to a separate PR. For reference, here is a possible plugin we can use to help with this piece in jekyll - https://github.com/kurtsson/jekyll-multiple-languages-plugin.

@rbuysse @rberg2 The PR build fails because @kentydotcom is not part of approved list.

@rbuysse @rberg2 The PR build fails because @kentydotcom is not part of approved list.

👍 Kenty's been added to the authorized builders list.

That should have read "There is a comment".

This work has been completed and merged into main.


Anti-raid features

The DMA page on auto-moderation has some notes on how major bots do anti-raid functionality. Some of it doesn't sound very hard to implement, so we should have that here.
https://discord.com/moderation/1500000178701-321:-Auto-Moderation-in-Discord#Bot-Controlled-Auto-Moderation

Marking as independent because CEM uses KupoBot for this.


Capture stdout on successful tests

This small change allows the user to capture stdout on successful tests to the comment field in testrails. This can be useful to track results that go beyond a binary PASS/FAIL.
In order to use it in testrails, simply run your pytest with the --capture=sys flag to enable capture (see here)

@nicovanduijn  Can you merge master here please to retrigger the build. There was an issue with the gitflows.

Closing in favor of #136


How to sendMessage to user

How to use sendMessage method.. ?
I've tried but always failed..
Please help

Just saw your message here too :smiley:
This project is using the php-telegram-api too, so I guess you can close this issue.
(answered in akalongman/php-telegram-api#91)


Update Set-AzureADTrustedCertificateAuthority.md

Get-AzureADTrustedCertificateAuthority should be used, not Set

Docs Build status updates of commit e2b4066:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

azureadps-2.0/AzureAD/Set-AzureADTrustedCertificateAuthority.md
:white_check_mark:Succeeded
View (azureadps-2.0)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit e2b4066:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

azureadps-2.0/AzureAD/Set-AzureADTrustedCertificateAuthority.md
:white_check_mark:Succeeded
View (azureadps-2.0)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Hello @rafalfitt-mptv - thanks for your contribution. Please redo this PR with main as the base branch instead of live.

Docs Build status updates of commit e2b4066:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

azureadps-2.0/AzureAD/Set-AzureADTrustedCertificateAuthority.md
:white_check_mark:Succeeded
View (azureadps-2.0)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@CelesteDG, this is good to go.


Implement OAuth 2.0 authentication

Tell us about the task you want to perform and are unable to do so because the feature is not available
I want to send requests to services that use OAuth 2.0. Right now I have to manually acquire an authorization token and copy/paste it into APIDash. The tokens expire usually in an hour, so I have to do this multiple times a day, not only separately for every service, but even for every individual request I have in the library.
Describe the solution/feature you'd like us to add
APIDash should automatically open a browser session/popup and let me sign in using OAuth 2.0's "hybrid", "implicit" flows. It should also allow me to obtain a token using the "client credentials" flow, which doesn't require a browser session (this is way easier to implement and would already be a huge step forward).

Thanks for opening this issue @paulius-valiunas
Tagged it as a good first issue so that contributors can also take a look into it and send across a PR

Can you please assign this to me

@beingPro007 We do not assign any issues. Anyone can work on the issue and submit a PR.

Hi, I am Abhinav. I was working on this issue for the past 3-4 days. Currently, I have implemented the core functionalities of the OAuth2 feature. I am attaching the images of the UI and the flow. Please help me by providing iterations to make it better.
OAuth2 Feature Implementation
Current State

OAuth2 Package: Utilizes the official Dart OAuth2 package.
OAuth Provider: Implemented using GitHub as the OAuth provider for illustration purposes.
Flow Type: Currently, only the Authorization Code flow type is implemented.

Images and Flow

@abhinavs1920 You can send across a PR for review


add ctx keys to debug, make debug a11y help open for more specific when

addresses https://github.com/microsoft/vscode/issues/224638 in insider's

Note that the editor's dialog will open if focus is on these buttons, but I feel it's overkill to add a context key just for those.


Dashboards don't load when using the nginx ssl proxy addon

Install and configure the nginx ssl proxy addon.  Now ha-fusion dashboards will not load.

Same here with Traefik. It looks like the hassurl is set incorrectly when doing this. I guess it should point to the Home Assistant server, but it gets set to http://fusion.mydomain.se (not https).

Duplicate of https://github.com/matt8707/ha-fusion/issues/344


Correction to my GitHub repo link

Sorry to report the link you posted to my GitHub TM1637 stuff should read as below, the emailed reply got mangled in the upload and gives a 404. Hence your .md needs a small edit. Should be:
https://github.com/SteveMicroCode/PIC-TM1637-Library-Code
Steve

oops missed that one

fixed it in master, will automatically be in next release

Email to GitHub seems to mangle stuff!  Frustrating.SteveSent from my iPadOn 23 Nov 2023, at 13:31, Rob Tillaart @.***> wrote:
oops missed that one
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

Yes, a well known "feature"

As it is fixed in master I will close the issue.


Sending JSON objects

I want to send the following code
RequestParams params = new RequestParams();
        params.add("action", "login");
        params.add("user_name", username);
        params.add("user_pwd", password);
        params.add("gcm_id", sharedPreferences.getString(TwofeetConstants.DEVICE_ID,null));

as a JSON string to the server
like this
			JSONObject	 obj = new JSONObject();
			 try
			 {
				obj.put("action", "login");
				obj.put("user_name", etUserName.getText().toString());
				obj.put("user_pwd", etPassword.getText().toString());
				obj.put("gcm_id", urls[0]);

			} catch (JSONException e) {
				e.printStackTrace();
			}
			return lk.infoseek.twofeet.JSONParser.POSTDATA(Constants.POST_USERDATA_URL, obj);

how to do this.. thanks in advance

Try simply calling params.setUseJsonStreamer(true); to send the data as Json string

I am totally new to this library.. so bare with me if i ask anythig stupid
following is the updated code
RequestParams params = new RequestParams();
params.setUseJsonStreamer(true);
params.add("action", "login");
params.add("user_name", username);
params.add("user_pwd", password);
params.add("gcm_id", sharedPreferences.getString(TwofeetConstants.DEVICE_ID,null));
try {
api.apiCall(TwofeetConstants.POST_USERDATA_URL, params, new OnApiFinishedListener() {
but i keep getting this error
onFailure(int, Header[], String, Throwable) was not overriden, but callback was received
org.json.JSONException: Response cannot be parsed as JSON data
at com.loopj.android.http.JsonHttpResponseHandler$1$1.run(JsonHttpResponseHandler.java:157)
i printed the params and i got
action=login&gcm_id=358639060235903&user_name=Hasunika&user_pwd=123
this doesn't seem like a JSON String..

Two things, you printed params, but they are sent in JSON, this should be checked on server side, or post the data to https://httpbin.org/post which will return them back to you, so you can check the request is fine.
Second, response you've got from server is apparently not a JSON data, there is no need to use JsonHttpResponseHandler, if you're not expecting JSON data to come back as response.

I have similiar problem and I can't figure out how to solve it. I need to send this JSON to API.
{
    "firstName":"frwferfwg",
    "lastName":"wergwerg",
    "address":{
            "address1":"wergwerg",
            "zipCode":"ergee",
            "city":"wgergwerg",
            "country":"SI"},
    "gsm":"+38631123123",
    "email"<EMAIL_ADDRESS>    "password" : "12345678"
}

I've been trying to post it like this:
RequestParams requestParams = new RequestParams();
requestParams.put("firstName", newUser.getFirstName());
requestParams.put("lastName", newUser.getLastName());
requestParams.put("email", newUser.getEmail());
requestParams.put("password", newUser.getPassword());
requestParams.put("address.address1", newUser.getAcAddress().getAddress1());
requestParams.put("address.zipCode", newUser.getAcAddress().getZipCode());
requestParams.put("address.city", newUser.getAcAddress().getCity());
requestParams.put("address.country", newUser.getAcAddress().getCountry());
requestParams.put("gsm", newUser.getGsm());

Which used to work but doesn't anymore. It doesn't recognize the address.address1, address.zipCode, address.city, address.country on server side. So I need to find a different way to do it.
I've also tried creating JSONObject with "address1", "zipCode", "city", "country" and putting it to RequestParams under key "address" but that doesn't work either.
Also tried using params.setUseJsonStreamer(true); but that also returns error. I've Googled other ways to post JSON using this library but all I could find was StringEntity which is deprecated.
Any help would be greatly appreciated.

http://stackoverflow.com/questions/13052036/posting-json-xml-using-android-async-http-loopj
that one worked for me.. give it a try

This is where the problem is for me. StringEntity entity = new StringEntity(jsonParams.toString());.
StringEntity is now deprecated. How do I do it now?

@matejhacin StringEntity is not deprecated, only one of the constructors seems deprecated to me from documentation. In both cases, deprecated doesn't mean it won't work, so I wouldn't worry about it now.

@smarek I just realized where the problem was. I was using older version of async http library (1.4.5) which still relied on Apache library, which is no longer used in Android.
I now updated async http to the newest version 1.4.9 and now it works with StringEntity from cz.msebera.android.httpclient package.

:)) Cool, @matejhacin glad to know you've found solution


TMPE tool should behave when returning from free camera view

User presses Escape to return from the free camera view. This switches current tool is set to whatever it was before entering the free camera view.
Problem:
This can re-enable TMPE tool without displaying the TMPE panel. This behaviour confuses Biffa:
https://cdn.discordapp.com/attachments/545066674605391882/712986161936728124/CS_Fixit_-_TMPE2a.mp4
Reason:
Threading extension hides TMPE menu if TMPE is not the tool.
We do not view the Panel when TMPE tool is enabled.
Need:
TMPE must act like other tools and TMPE panel should be visible when returning from free camera view.
Solution:
Threading extension is the wrong place to show/hide TMPE panel.
this must be done from inside TMPE tool.
Problem2:
TMPE panel disappears when user presses escape in free camera view.
Reason:
Threading extension captures the Escape key and closes TMPE.
Need:
TMPE must ignore Escape key when in free camera mod.
Solution:
Handle Escape the same way other tools handle it: from TrafficManagerTool.OnToolGUI()
PS: I deleted ThreadingExtension.OnUpdate() because it does not do anything anymore.

Code looks good, I'll need to test in game.

There was a problem that when I switched to another tool TMPE panel did not disappeared. I fixed that problem.

@aubergine10 Can you please approve the new code I put in? I can merge this ...  but ...

I'll test in-game today.

Found some possible issues when testing in-game.
Note: I'm using First-person Auto-Walking mod, rather than the more commonly used First Person Camera (Updated) mod.
First, if a tool is active, the overlays are still rendering in first person cam. At the very least this is bad for fps.

Second, I press Esc to exit the first person cam, and TM:PE toolbar still visible etc. (this is correct).
However, when I press Esc to exit the toolbar, the toolbar does get closed but the Pause Menu is also displayed. It's like the UI fsm somehow got confused.

Actually, it doesn't look like the TM:PE toolbar is getting closed; I suspect it thinks it's already closed (due to Esc press while exiting the first person cam):

Steps:

Open TMPE toolbar, select lane connector tool (for example)
Tab key= go in to first person autowalking cam
Esc key = exit first person cam
Esc key = should only close TMPE toolbar, but it doesn't; instead it shows Pause Menu

@aubergine10 I am unable to use your first-person auto walk mod. When I press that nothing happens. No exceptions either???? (when I pressed tab to bulldoze it throws some exception. I did sub out of that though and still, I had the problem).
In any case, those lane connections will not get rendered if the Traffic manager tool is not the active tool.

Can you reproduce this problem with TMPE labs as well?
If so let's fix this in another pull request. I think the fix would be for first-person auto walk mod to change tool to default. Can't say for sure if I cannot use the mod.

@aubergine10 The problem you raised is due to an exception preventing the rest of the code from running:
NullReferenceException: Object reference not set to an instance of an object
  at ColossalFramework.UI.UIComponent.OnResolutionChanged (Vector2 previousResolution, Vector2 currentResolution) [0x00000] in <filename unknown>:0
  at ColossalFramework.UI.UIView.OnResolutionChanged (Vector2 oldSize, Vector2 currentSize) [0x00000] in <filename unknown>:0
  at ColossalFramework.UI.UIView.OnResolutionChanged (Int32 oldSize, Int32 currentSize) [0x00000] in <filename unknown>:0
  at ColossalFramework.UI.UIView.OnResolutionChanged () [0x00000] in <filename unknown>:0
  at ColossalFramework.UI.UIView.OnEnable () [0x00000] in <filename unknown>:0
UnityEngine.Behaviour:set_enabled(Boolean)
ColossalFramework.UI.UIView:ShowView(Boolean)
ColossalFramework.UI.UIView:Show(Boolean)
IINS.AutoWalking.MonoCamera:UpdateFreeCamera()
IINS.AutoWalking.MonoCamera:OnDestroy()
IINS.AutoWalking.WalkingCamera:OnDestroy()

(Filename:  Line: -1)

NullReferenceException: Object reference not set to an instance of an object
  at ColossalFramework.UI.UIComponent.OnResolutionChanged (Vector2 previousResolution, Vector2 currentResolution) [0x00000] in <filename unknown>:0
  at ColossalFramework.UI.UIView.OnResolutionChanged (Vector2 oldSize, Vector2 currentSize) [0x00000] in <filename unknown>:0
  at ColossalFramework.UI.UIView.Update () [0x00000] in <filename unknown>:0

To prevent scope creep can you please ignore this issue. If there are no other issues please approve this PR.

@aub I created #958 to track the issue you raised

Also, I tested with current LABS version and there are similar (but slightly less bad) issues with that too.

@aubergine10 I think the issue is with TMPE.
CS Camera view changes the tool so it disables TMPE and avoids the problem.
auto-walk mod does not change the tool (as it should) and TMPE also has null reference exception. both of these together leads to #958


nsh_vars: Remove rogue sched_unlock()

Summary
There is no corresponding sched_lock() for this and therefore this causes random crashes.
Found when running a long init script which utilizes shell variables in SMP mode.
Impact
Remove sched_unlock() from a place that does not use sched_lock()
Testing
Downstream MPFS target which utilizes nsh_vars.

@patacongo is it safe to remove this abandoned shed_unlock() or the sched_lock() is missing? From a history of file I can see it is here from the beginning of time :-)

There should be no reason to do mutual exclusion (sched_lock) here as pstate is allocated per shell and only the shell itself can add / remove vars.

@patacongo is it safe to remove this abandoned shed_unlock() or the sched_lock() is missing?

Ville has it right.  It might have had meaning at one time.  Originally, NSH was part of the nuttx repository and was moved out, losing the history that far back.  The apps were separated at NuttX-5.19 since they are not architecturally a part of the OS.
NOTE also:  sched_lock() does not work does not provide mutual exclusion in the case of SMP; the locked code section can still be re-entered from other CPUs.  So it is wrong as is on many levels.

Thank you @patacongo and @pussuw this PR can be safely merged :-)


ONNX Parser support for SqueezeNet ?

Hi,
I'm trying to make armnn's ONNX Parser work with an open source version of squeezenet, however, the parser is failing to load the model throwing the following error:
terminate called after throwing an instance of 'armnn::ParseException'
  what():  Unsupported operation Concat for node 'squeezenet0_concat0'  at function LoadGraph [/home/sahib/ARMNN/armnn/src/armnnOnnxParser/OnnxParser.cpp:680]
Aborted (core dumped)

From the documentation on supported layers, I'm able to see that the concat layer is not supported for the time-being, so I was wondering if anyone has been able to make the same work, or pointers to how the support can be added.
Thanks!

Hi @sahibdhanjal
Onnx parser support for Concat is on the roadmap for an upcoming release but there is no ETA at the moment.
If you wish to add support for a layer to the Onnx parser then the Concat operator is described at https://github.com/onnx/onnx/blob/master/docs/Operators.md#Concat and a good example of the way that an Onnx operator is added would be: https://review.mlplatform.org/c/ml/armnn/+/2978
If you get it working then we are open to contributions. You can find more information here:
https://mlplatform.org/contributing/
Our gerrit server: https://review.mlplatform.org/#/dashboard/self
The first thing to do is to get it working with the latest master branch of Arm NN.
Next make sure you can log in to gerrit at https://review.mlplatform.org/#/dashboard/self using your Github account. Add your ssh key under "settings -> SSH public keys".
Then submit it to a gerrit review with a command like this, replacing with your Github username:
git push ssh://@review.mlplatform.org:29418/ml/armnn HEAD:refs/for/master
Best regards,
Mike

I see. Thanks for the quick response! If I'm able to make it work on my side I'll create a PR for the same. Closing this ticket for now


Project doesn't compile on Deno 1.20.3

How to reproduce:
deno upgrade
deno cache<EMAIL_ADDRESS>
Version:
deno --version
deno 1.20.3 (release, x86_64-unknown-linux-gnu)
v8 <IP_ADDRESS>
typescript 4.6.2

Logs:
Check<EMAIL_ADDRESS>error: TS2769 [ERROR]: No overload matches this call.
  Overload 1 of 2, '(input: string | Request, init?: RequestInit | undefined): Promise<Response>', gave the following error.
    Argument of type 'string | URL' is not assignable to parameter of type 'string | Request'.
      Type 'URL' is not assignable to type 'string | Request'.  Overload 2 of 2, '(input: URL, init?: RequestInit | undefined): Promise<Response>', gave the following error.
    Argument of type 'string | URL' is not assignable to parameter of type 'URL'.
      Type 'string' is not assignable to type 'URL'.
    const { body } = await fetch(url);
                                 ~~~
    at https://deno.land/x/grammy@v1.7.0/platform.deno.ts:134:34

TS2322 [ERROR]: Type '((root: string, token: string, method: string) => URL) | ((root: string, token: string, method: string) => string)' is not assignable to type '(root: string, token: string, method: string) => URL'.
  Type '(root: string, token: string, method: string) => string' is not assignable to type '(root: string, token: string, method: string) => URL'.
    Type 'string' is not assignable to type 'URL'.
            buildUrl: options.buildUrl ??
            ~~~~~~~~
    at https://deno.land/x/grammy@v1.7.0/core/client.ts:207:13

    The expected type comes from property 'buildUrl' which is declared here on type 'Required<ApiClientOptions>'
        buildUrl?: (
        ~~~~~~~~
        at https://deno.land/x/grammy@v1.7.0/core/client.ts:131:5

Found 2 errors.

Same issue

It works on 1.20.2 as stated in https://t.me/grammyjs/41202

@KnorpelSenf we can close this issue now?

Yes!

It seems like Deno will fix this on their side soon: https://github.com/denoland/deno/pull/14113#issuecomment-1080750014


Explore performance / scalability improvement of stub storage

Investigate use of an external component (like Redis) for the storage of the OMRS stub information, to both improve overall performance of the event mapper as well as (probably) increase its scalability.

Such an improvement could also be made optional, based on whether the repository connector is configured with a property to refer to such an external component or not.  (If not, it can simply fall-back to the default of using the existing OpenIGC stub mechanism.)

Initial look at Redis seems promising: memory footprint appears negligible (single-digit MB in official Docker container at https://hub.docker.com/_/redis), performance is blazing fast, and there appear to be mature clients for various languages:

https://github.com/redisson/redisson
https://github.com/lettuce-io/lettuce-core
https://github.com/xetorthio/jedis
https://github.com/luin/ioredis

Downside to Redis is that it does not appear to be something that can be easily run in an embedded way; alternative may be to investigate H2, which we're already using for some of the UI demos.

For redis there have.been a few attempts. https://www.baeldung.com/spring-embedded-redis may help?

Indeed, I saw that, but still feels like it's geared towards more of a unit-test-specific scenario...  But maybe that's still sufficient nonetheless.
I'm trying to keep in mind a broader vision here as well that such a "cache" might be a useful, more broadly configurable option to include in the core of Egeria for anything to make use of that needs to store state in an efficient and scalable way: whether that be repository connectors or other servers / daemons.  (That way we don't end up with lots of one-off persistence mechanisms that are all managed differently using different tools, but could leverage the more general connector framework of Egeria itself to define which (configurable) "cache" to use for various scenarios: including whether embedded, external, persistent vs. non-persistent, etc.
In that sense, it probably makes sense to first come up with the interface(s) and configurable options, and then which connectors we build to particular back-ends just gives us a variety of options for backing those interface(s) (?)

Will leave this open, but as the event mapper (only piece that requires the stubs) is now considered experimental, this is unlikely to be implemented.


WIP: Onboarding form

OnboardingForm wraps all other steps
Custom styles in SCSS (using tailwind config and utilities)
Conditional paths can be determined within the getNextStep method for each subclass of OnboardingFormStepComponent
Not ready for releases!

I started integrating it with tipoff/addresses, but there are many database fields in the package that is not required here. What would you suggest? Here are the options I thought of:

Use a lookup service to get full address details based on the zip code
Make some of those fields nullable in the package

Don't worry about a full implementation with tipoff/addresses yet. We'll refactor and use that later once that addresses package is tagged with a more final release.

This is great! I'm going to merge it and then do another PR changing things around. I want to rename the package and the tables so it is not specific to workers' compensation, but that is an add-on for the Lead.