antsApplyTransforms: incorrect return codes

$ antsApplyTransforms -e 3 -d 3 -i foo.nii -o bar.nii -r baz.nii -t foobar.mat
Transform file does not exist: foobar.mat
$ echo $?
0

It would be nice if antsApplyTransforms and potentially other binaries could return proper failure status codes as it is otherwise rather hard to react on errors. Maybe I was just unlucky in this case as I've seen that return EXIT_FAILURE is used throughout the code. I unfortunately don't know the code base well enough to supply a patch for this specific issue, so it would be nice if someone could look into this.
Thank you!

There are many functions that return EXIT_FAILURE, but this is often ignored at the class / executable level, and EXIT_SUCCESS, or literal 0, is returned.

I ran into a similar issue with N4BiasFieldCorrection, some of its failures are completely silent without the verbose flag, and the return code is success.

Pull requests are always welcome.

Also, pointing to specific cases (data and usage) is much easier to fix than non-specific claims of something not working.  I've actually gone through the executables to address this precise issue but I could've easily missed things.

Consistent exit codes are helpful for debugging, especially now that the error messages have been curtailed.
What should be done if the command is invoked with no args? I've seen that classified as success (return  0) and failure (return 1) in various unix contexts. My personal policy is to call this a failure, because it's almost always a sign of failure if that happens in a script, which is where 99% of exit code checks will live.
If "-h" or "--help" is passed, I call that a success. That's unlikely to happen by accident, whereas
antsRegistration $args
with $args being undefined could happen.
So in summary, I propose
EXIT_SUCCESS (not literal 0) on success (EXIT_SUCCESS returned internally to main)
EXIT_FAILURE (not literal 1) on failure (EXIT_FAILURE returned internally to main)
EXIT_SUCCESS if command is run with -h or --help
EXIT_FAILURE if command is run with no args or insufficient args
If that sounds good, I will make a branch to get started on this.

Sounds good to me.  I think your breakdown is what I maintained in my first pass of this issue.

Re opening this now I've made a branch for fixing exit codes:
https://github.com/stnava/ANTs/tree/returnExitCodes
I made one commit, changing antsApplyTransforms. As I thought, the success / failure return values are mostly correctly coded, and it was just a matter of returning the value from the call to antsApplyTransforms() rather than returning EXIT_SUCCESS after the switch statement.

Hi Phil,  Are you going to make a pull request since everything is passing?

Will do - I was working through the other programs alphabetically but I only got as far as A. I'll leave the branch open so I (or anyone else who feels like it) can come back to it later.

Sounds good.

The above mentioned branch has been merged into master. Most programs, including antsApplyTransforms and N4BiasFieldCorrection, should return non-zero exit codes where appropriate.


Using ruby 2.4 fails

I tried installing the most modern ruby available in Homebrew (ruby 2.4), and kept running into a problem compiling json (1.8.3):
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/lib/ruby/gems/2.4.0/gems/json-1.8.3/ext/json/ext/generator
/usr/local/opt/ruby/bin/ruby -r ./siteconf20170624-88439-qnl4kq.rb extconf.rb
creating Makefile

current directory: /usr/local/lib/ruby/gems/2.4.0/gems/json-1.8.3/ext/json/ext/generator
make "DESTDIR=" clean

current directory: /usr/local/lib/ruby/gems/2.4.0/gems/json-1.8.3/ext/json/ext/generator
make "DESTDIR="
compiling generator.c
generator.c:861:25: error: use of undeclared identifier 'rb_cFixnum'
    } else if (klass == rb_cFixnum) {

Easily fixed by editing Gemfile.lock to use the latest json (2.1.0):
diff --git a/Gemfile.lock b/Gemfile.lock
index d1313a4..81b7532 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -42,7 +42,7 @@ GEM
       rake
       rdoc
       semver
-    json (1.8.3)
+    json (2.1.0)
     jwt (1.5.6)
     memoizer (1.0.1)
     mini_portile2 (2.1.0)

I got rid of the jeweler gem as a development dependency.  The jeweler gem is what was pulling in the json gem.  That should take care of the problem you were having running bundle against this library in Ruby 2.4.  I'll close this issue, then, but I can't test on homebrew so please let me know if I've made a mistake.
Please note that the tests currently do not pass in Ruby 2.3, because the double-bag-ftps gem that ftpd's tests use is currently broken for Ruby 2.4 (see https://github.com/bnix/double-bag-ftps/issues/17).  You'll need to use ruby-2.3 to develop ftpd for the mean time.  I believe ftpd works in ruby-2.4; I just don't have running tests to prove it.

The tests now pass in Ruby 2.4.  Well, sort of pass.  I marked those tests that are broken as "pending"; that doesn't make me very nervous because those few tests do run (and pass) in Rubies 1.9.3 through 2.3.4, and it's pretty unlikely that there'll be a Ruby 2.4 specific bug in that part of the code.
Eventually I may be able to get the double-bag-ftps gem fully operational in Ruby 2.4, and then the tests I marked "pending" for Ruby 2.4 can be put back into full service.

@wconrad, any chance support for Ruby 2.4.2 will be added in the next month or two? Our project has a dependency on this gem, but we don't have a good test FTPS environment where we can validate work on fixing ruby 2.4.2 support.

Update: I don't have time to work on double-bag-ftps or ftpd right now, so sadly this issue will have to remain unsolved.


Improvements to POC tools

Make the following improvements to the Proof Of Concept tools.

Eliminate the duplicate output of movies for situations where a movie exists on multiple media, like both a DVD and a Blu-ray Disc.
Implement a --random command line flag to output a small subset of movies, for the purpose of spot checking and quality control.

Depends on completion of https://github.com/cjcodeproj/medialibrary/issues/17

I'm trimming the requirements for this ticket right now.  Right now the only actionable improvement should be:

Eliminate the duplicate output of movies for situations where a movie exists on multiple media, like both a DVD and a Blu-ray Disc.

Other Improvements will be coming down the line, in smaller iterations.

Commit completed.
https://github.com/cjcodeproj/medialibrary/commit/234b368b1c3ee22db71abd17bd5e5d1d62b5dd71
Works effectively with a sample library of over 150 movies.


feat: introduce github actions to generate HTML and PDF automatially

Motivation
Sometimes, it's a bother for developers to install docker to local env.
@VFR-maniac is one of such developer.
https://twitter.com/ChaosSoulLover/status/1240244707197734912
Why don't you use CI? Let's use Github Actions!
Now, you don't need to install docker to local.
What changes did you make

Write Github Actions config file to generate HTML and PDF and deploy to gh-pages branch
fix build/gen.sh: some hack is now not required!

When CI will work?
only when you push to master branch.
What is required after you merge this PR?
You need to enable gh-pages.
see peaceiris/actions-gh-pages: GitHub Actions for GitHub Pages 🚀 Deploy static files and publish your site easily. Static-Site-Generators-friendly. #First Deployment with GITHUB_TOKEN
example deployed gh-pages

https://yumetodo.github.io/opus-dash/opus-iso-binding.html
https://yumetodo.github.io/opus-dash/opus-iso-binding.pdf

@googlebot I signed it!


Only add new SonarQube issues when using re-import

I have coded a webhook that as soon as a sonarqube scan is finished, I call Dojo API to re-import the SonarQube API import test. This creates tones of duplicate findings and the dashboard also loses count of Open findings and Dashboard shows 0 open findings.
Or please suggest any more workflow.

Do you have deduplication enabled? It should deduplicate after the import in the background celery process. Not ideal, but should work.

yes, deduplication is selected,

Hi @valentijnscholten , had you a chance to look into it?


[Revised] Support Plus(+) and Minus(-) signs in WHERE

This pull request resolves issue #101 by supporting unary operations of Value. Specific modifications are the following.

value.rs & evaluated.rs: Functions to evaluate unary operations of the expression were added.
mod.rs: Expressions now handle unary operations.
test.rs: Two test cases to check if WHERE supports plus(+) and minus(-) signs were added.

@panarch I really appreciate your thorough reviews! It was my first time making a pull request to an open source project. I learning so much. :)
I finished changing function names, simplifying match cases, and adding codes for a float type. I'm currently working on errors regarding SELECT id FROM Hunter WHERE +2.0 > 1. It seems like I need to add more cases inside fn partial_cmp(&self, other: &Evaluated<'a>) -> Option<Ordering> { which is in src/data/evaluated.rs. It should be done by tonight. Again, thanks for your review!

I found out that the error comes from unnecessary reverse operations. Thank you for pointing out! :) I changed the code and added some additional test cases in src/tests/filter.rs.


Add checkstyle to the java code

Add a checkstyle to the project

Do you have a style format you like? sun vs. google? tabs vs. spaces? I don't care much-- but code with tabs.. but don't mind running Code-Format for all source files.
BTW, I'm running FindBugs on the code and fixing a few of the warnings.

I think we just need to choose one. I'd go with google just because it has the largest width (100) and it's very easy to set up. Then we could set up also a Travis job to verify the PRs

Google is fine. Made a PR but not sure exactly how checkstyle works. Checkstyle now builds a report when you run maven site.. but I did not bother to correct all of the warnings.
Also, I think checkstyle/code formatting should ignore or not modify the generated source code. Or at least not require a "format source" on generated code to match what is in the repository. Perhaps a "generated source" folder is a good idea.
Noticed on one camera that getting the list of audioSources fails if a camera doesn't support audio by throwing a soapfaultexception. Not sure how to avoid without wrapping the call with a catch throwable. Is there a check to see if audio is supported? Not very important, but want the "checkDevice" test to complete if the url, user and pass are correct.

Merged the PR but we still needs to exclude generated code from checks and fix the distributionManagmenet in the pom.xml so I keep this issue open just as a reminder

Sounds good. I've been busy on other projects.. but using this internally in my software. Thanks!


[README] replace pod line

to be much easier to copy/paste the Pod, please replace
pod "FDTake"
with:
pod 'FDTake'
thanks ;)``

👍🏻💪🏻


systeminfo: add windows implementation

systeminfo package is implemented in all major platforms except Windows. This PR fixes this.

@ppknap lgtm 👍 please rebase with #161

@rjeczalik rebased with current master

lgtm 👍

Thanks @ppknap
👍


not support table render

I'm not sure i understand the issue, I can easily render table using this plugin....


Fbo media change

To change dev_config run:
set attribute dev_size=661370880 dev_config=fbo//home/neo/ubuntu-16.10-server-ppc64el1.iso

@mikechristie So far this testing has been approved by our internal IBMi OS team in terms of our use case and how we want to use it, let me know if you have any other questions or comments.

I have not forgot this patchset.
I have some patches to make reopening a device safer. Currently, for some handlers it might cause problems if commands are in flight when we ere changing internal structs like changing the fd being used. I am going to adapt your patches to mine. Will cc you on the PR.

Okay thanks! I figured you were reworking something

I am making a 1.3.0-rc1 build tonight and I was not able to get this in.
I am still hoping to get it into the final. I have added __tcmu_reopen_dev which can be used to reopen a device with the new settings. I am just fixing up the UA code. I am going to make it work under the state_lock added.
I am just trying to make the update nicer in that it is kind of klunky to have to do multiple update calls. I am debating about just passing all the needed info in the cfg string then libtcmu parsing out values like the size and then passing the handler just the handler info like the file path. I am still testing some things though.

Yeah, it is kind of klunky... But I had to do it incase IO comes in during different states of the reconfiguration...
I guess if you can make size and string handled in one go then you wouldn't need to have the two states for a media change.

Added UA support here:
https://github.com/open-iscsi/tcmu-runner/pull/309
Going forward for the multiple settings (size and dev_config string) type of operation we need for fbo backing change, I am thinking maybe we just want to pass all the new values in the dev_config or add a new configfs file that can be used to update multiple settings.
I think we want to add a new file for this, so dev_config can be used for that specific string and there will be no compat issues.

Hi Mike,
I have some free time to get this patch updated, since your last comment do you have a specific way you want this implemented?

@mikechristie

Sorry for the late reply. I have been debating with myself on this the last couple of days.
I think just adding a new configfs attr that is for updating multiple options will be best. It can be a string of multiple options similar to what is used for the initial setup.
What do you think?

@mikechristie I think that is probably best so that a media change can be processed all at once and not one option at a time

@mikechristie How do you want the new attribute to be delimited? I was thinking a ":" ?
set attribute dev_reconfig=dev_config:blahblah:dev_size=2222:etc?

":" is probably best looking but some tools/libs like rtslib does not allow ":" in some inputs.
We have been using ";".


Plugins don't seem to work on a clean Windows Install (VM)

I tried using plugins on a clean install of Grimlite on a clean install of Windows, and it seems that there is a problem.
When i try to use plugins, i got the error message that no plugins are currently loaded, so i go in plugin manager, click on the right dll and get the following error message.

The fix for me was pretty straight forward, i followed the fix suggested by msdn which is to add the following line under <runtime> in Grimoire.exe.config.
<loadFromRemoteSources enabled="true"/>

You might want to add this line by default in the release or add a disclaimer on the readme page about this fix. It would be of great use for people with little to no programming/tech knowledge.

Btw it might just be my VM setup, i'm playing AQW on my malware analysis machine. Anyway of any other people run into this problem... here's the fix.


Time Sync from the phone fails

@LIFsCode @OOHehir
@josephx86  Might that be a app issue? Not sure.
Sorry for another issue :/
But sometimes i get recordings from 1970 because they don't get the time from the phone. So i was hunting the time-sync issue and was "successful" :)
How it happens:
Running firmware version: Release/V1.0-16-gd1e8350-dirty (But this issue also happened with commits from weeks before)

Boot the ELOC
Connect with the app
Disconnect before the time sync happens
Connect again and do a few fast refreshs (swipe down in the app = reconnect to ELOC = refresh status)

That's what i did. But maybe some random quick reconnects will do the job too.
Result
This should give you some BT errors and sometimes also the timesync problem.
Starts with this:
E (93606) BtCmds: Parsing time config failed with IncompleteInput
And then this, which i then often also get after restarting the ELOC:
[E][BluetoothSerial.cpp:342] esp_spp_cb(): RX Full! Discarding 20 bytes
E (51332) BluetoothServer: Invalid Command gesetTime, Received gesetTime
The [E][BluetoothSerial.cpp error does not appear in the ELOC log file. Not sure why. That's why i have uploaded two logs:
This is the log file from the ELOC (problem starts @ line 4482): Time_Sync_Failed.log
And this a logfile from VScode including the [E][BluetoothSerial.cpp Errors (Not the same time as ELOC log): Time Sync fail with BT_Serial_Error.txt
Running with this Configuration:
{
  "device": {
    "fileHeader": "34",
    "locationCode": "UNKNOWN",
    "locationAccuracy": 100,
    "nodeName": "ELOC_344"
  },
  "config": {
    "secondsPerFile": 3600,
    "cpuMaxFrequencyMHZ": 80,
    "cpuMinFrequencyMHZ": 10,
    "cpuEnableLightSleep": true,
    "bluetoothEnableAtStart": true,
    "bluetoothEnableOnTapping": true,
    "bluetoothEnableDuringRecord": true,
    "bluetoothOffTimeoutSeconds": 120,
    "logConfig": {
      "logToSdCard": true,
      "filename": "/sdcard/log/eloc.log",
      "maxFiles": 10,
      "maxFileSize": 52100000
    },
    "intruderCfg": {
      "enable": false,
      "threshold": 5,
      "windowsMs": 2000
    },
    "battery": {
      "updateIntervalMs": 600000,
      "avgSamples": 10,
      "avgIntervalMs": 0,
      "noBatteryMode": false
    }
  },
  "mic": {
    "MicType": "ns",
    "MicBitShift": 11,
    "MicSampleRate": 16000,
    "MicUseAPLL": true,
    "MicUseTimingFix": true,
    "MicChannel": "Left"
  }

In case it helps i have a collection of BT errors which i accumulated over the last weeks. Every error belongs to it's own log which you can find here: BT errors collection.txt
E (35985) BT_BTC: btc_spp_cb_handler unable to find RFCOMM slot!, handle:129

E (177458) BluetoothServer: Invalid Command getStatggetStatusgetStatus, Received getStatggetStatusgetStatus

E (44463) BT_HCI: command_timed_out hci layer timeout waiting for response to a command. opcode: 0x40f

E (40746) BT_BTC: btc_spp_write unable to find RFCOMM slot!

[ 40864][E][BluetoothSerial.cpp:327] esp_spp_cb(): ESP_SPP_WRITE_EVT failed!, status:7

Getting the date set to 1970 means some recording are being started before the time sync has completed. The app runs the time sync command in the background and I would guess if recording is started before syncing is completed, it makes sense the date/time would be 1970. I will re-implement the "handshake" between phone and board to block further commands being sent to the board until the time sync is completed successfully. In the app this will mean seeing a connecting progressbar until the time sync command succeeds. This will likely be less than 2 seconds.

@EDsteve Great find.
Maybe too many commands or too long commands are sent at once. There is a limitation in the Bluetooth stack which limits the Rx buffer size.
This may also be affected by the total CPU load when the Rx queue is not emptied fast enough.

@josephx86 @LIFsCode

I would guess if recording is started before syncing is completed,

I don't think it is possible to start recording before a time sync has happened (or at least i never observed that). Only when a time sync fails due to BT errors, a recording from 1970 can happen. And even that suprises me because i thought the build time is used if there is no time from the phone or is there something i am missing? @LIFsCode
The connection progress bar is a good idea. The problem is, that it takes 16 second after a BT connection for the time-sync to happen. And that is a too long time to wait.
I think the reason for that is because it seems the app asks 16 times getStatus every second after a BT connection. Only after that the time sync happens. Is that correct? @josephx86
When i send the command getStatus. I see this line in the log:
I (18741) Battery: Calibrate voltage: read 3.301V, offset 0.000, scale 1.000000, result 3.301V
I (18753) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (18761) SDC: SD card free space: 118.825714 GB
I (18768) SDC: SD card free space: 124597792 KiB
So i am assuming that the app sends the command getStatus 16 times after a BT connection.
This is the log after a BT connection till the time sync (I removed the task-table from the log to make it more easy to read):
I (10522) BluetoothServer: SerialBT is ready ------------------------------------------------------
I (10602) main: Battery: Voltage: 3.319V, 83% SoC, Temp 31 °C
I (10610) main: Heap: Min=94652, free=96040 (49%), largestFreeBlock=90112, fragmentation=7%
I (10620) main: PSRAM Heap: Min=4164071, free=4165187 (1%), largestFreeBlock=4128768, fragmentation=1%
I (10984) gpio: GPIO[13]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
W (17092) BT_HCI: hcif conn complete: hdl 0x80, st 0x0
W (17109) BT_HCI: hcif link supv_to changed: hdl 0x80, supv_to 8000
W (18020) BT_APPL: new conn_srvc id:26, app_id:255
I (18308) gpio: GPIO[13]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
I (18741) Battery: Calibrate voltage: read 3.301V, offset 0.000, scale 1.000000, result 3.301V
I (18753) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (18761) SDC: SD card free space: 118.825714 GB
I (18768) SDC: SD card free space: 124597792 KiB
I (19684) Battery: Calibrate voltage: read 3.301V, offset 0.000, scale 1.000000, result 3.301V
I (19709) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (19718) SDC: SD card free space: 118.825714 GB
I (19724) SDC: SD card free space: 124597792 KiB
I (20616) Battery: Calibrate voltage: read 3.301V, offset 0.000, scale 1.000000, result 3.301V
I (20625) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (20648) SDC: SD card free space: 118.825714 GB
I (20654) SDC: SD card free space: 124597792 KiB
I (20721) main: Battery: Voltage: 3.301V, 70% SoC, Temp 31 °C
I (20729) main: Heap: Min=85716, free=88276 (53%), largestFreeBlock=81920, fragmentation=8%
I (20741) main: PSRAM Heap: Min=4164071, free=4165187 (1%), largestFreeBlock=4128768, fragmentation=1%
I (21565) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (21572) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (21579) SDC: SD card free space: 118.825714 GB
I (21583) SDC: SD card free space: 124597792 KiB
I (22483) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (22516) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (22524) SDC: SD card free space: 118.825714 GB
I (22530) SDC: SD card free space: 124597792 KiB
W (23186) BT_HCI: hcif mode change: hdl 0x80, mode 2, intv 798, status 0x0
I (23433) Battery: Calibrate voltage: read 3.322V, offset 0.000, scale 1.000000, result 3.322V
I (23452) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (23458) SDC: SD card free space: 118.825714 GB
I (23462) SDC: SD card free space: 124597792 KiB
W (23471) BT_HCI: hci cmd send: unsniff: hdl 0x80
I (24288) Battery: Calibrate voltage: read 3.322V, offset 0.000, scale 1.000000, result 3.322V
I (24300) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (24305) SDC: SD card free space: 118.825714 GB
I (24309) SDC: SD card free space: 124597792 KiB
W (24665) BT_HCI: hcif mode change: hdl 0x80, mode 0, intv 0, status 0x0
I (25127) Battery: Calibrate voltage: read 3.301V, offset 0.000, scale 1.000000, result 3.301V
I (25146) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (25154) SDC: SD card free space: 118.825714 GB
I (25161) SDC: SD card free space: 124597792 KiB
I (26055) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (26067) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (26085) SDC: SD card free space: 118.825714 GB
I (26092) SDC: SD card free space: 124597792 KiB
I (26971) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (26981) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (26985) SDC: SD card free space: 118.825714 GB
I (26997) SDC: SD card free space: 124597792 KiB
I (27906) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (27918) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (27925) SDC: SD card free space: 118.825714 GB
I (27932) SDC: SD card free space: 124597792 KiB
I (28869) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (28881) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (28889) SDC: SD card free space: 118.825714 GB
I (28895) SDC: SD card free space: 124597792 KiB
W (29721) BT_HCI: hcif mode change: hdl 0x80, mode 2, intv 798, status 0x0
I (29853) Battery: Calibrate voltage: read 3.320V, offset 0.000, scale 1.000000, result 3.320V
I (29860) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (29865) SDC: SD card free space: 118.825714 GB
I (29869) SDC: SD card free space: 124597792 KiB
W (29878) BT_HCI: hci cmd send: unsniff: hdl 0x80
I (30688) Battery: Calibrate voltage: read 3.318V, offset 0.000, scale 1.000000, result 3.318V
I (30695) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (30700) SDC: SD card free space: 118.825714 GB
I (30704) SDC: SD card free space: 124597792 KiB
W (30773) BT_HCI: hcif mode change: hdl 0x80, mode 0, intv 0, status 0x0
I (30800) main: Battery: Voltage: 3.318V, 82% SoC, Temp 31 °C
I (30809) main: Heap: Min=83624, free=84968 (55%), largestFreeBlock=77824, fragmentation=9%
I (30820) main: PSRAM Heap: Min=4164071, free=4165187 (1%), largestFreeBlock=4128768, fragmentation=1%
I (31574) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (31585) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (31604) SDC: SD card free space: 118.825714 GB
I (31611) SDC: SD card free space: 124597792 KiB
I (32546) Battery: Calibrate voltage: read 3.300V, offset 0.000, scale 1.000000, result 3.300V
I (32566) COMMANDS: WavWriterMode = disabled(0), AI = OFF
I (32574) SDC: SD card free space: 118.825714 GB
I (32587) SDC: SD card free space: 124597792 KiB
I (34980) BtCmds: updating time with {"seconds":1709530008, "ms" : -564, "timezone" : 7}
I (34990) BtCmds: timestamp in from type ? Time Zone: 7 sec:<PHONE_NUMBER> millisec: -564
I (34998) BtCmds: new timestamp from new sys time (local time)<PHONE_NUMBER>436
I (35004) BtCmds: new timestamp from timeobJect (local time) 158223634
I (35019) BtCmds: new time set to (local time) Mon, Mar 04 2024 12:26:47
W (35811) BT_HCI: hcif mode change: hdl 0x80, mode 2, intv 798, status 0x0
W (37456) BT_HCI: hcif ssr evt: st 0x0, hdl 0x80, tx_lat 798 rx_lat 798

I (40943) main: Battery: Voltage: 3.300V, 70% SoC, Temp 32 °C

Sending getStatus 16 times in a short span of time is definitely inefficient. The app requests info from the board depending on UI events, so I will refactor the code and reduce the number of requests.
Another thing, that I was hoping to bring up later after I had written a patch with the feature is can we add a field in the response from a board that will have an ID that matches an ID provided as an integer parameter in the request? This will help improve 2 issues. The first is that the app can better keep track of which requests have been completed. Second, for uploading data from the board, the app needs to match a getStatus response with a getConfig response. So far, there is no reliable way to match the 2 responses. Some times it is necessary to call getStatus twice within 10 seconds. If some requests are indeed getting dropped, the files to be uploaded to the database might have incorrect state information.

@josephx86 currently the interface was intended to serve only a single command at a time, i.e. all commands are responded and the next command is not expected to be issued before the first commands response is received.
If that's an issue and the app requires multiple commands at once in the flight, this can be easily changed from a device perspective, but then it definitely requires a command ID for command and responses to match.
This could be easily identified via the cmdVersion sent during the initial connection message.
@josephx86 do you require this? Should I make a proposal for interface version 2?

I did not understand the part about using the cmdVersion, isn't that attribute always a constant value?

do you require this? Should I make a proposal for interface version 2?

I need a way to match the response of getStatus and getConfig at a given time. This is because the information in from both command responses will be combined into a single database record. So I get either make 2 separate requests for each command, but the app will keep track of the IDs so that it knows which getStatus response and getConfig response should be matched.
Alternatively, we could introduce a new command, like say "getInfo" for example, that will return a combination of information from getStatus and getConfig.
I do not yet fully understand the intricacies of the firmware code but I would imagine it would be easier for the board to read the ID parameter  supplied with a command, and then include the same ID back in the response. Something like:
getStatus#id=56\n
and the respnse would be something like:
{     "id": 56,     "ecode" : 0,      "cmd" : "getStatus",      "payload" : {         .         .         . }<EOT>

@josephx86 @LIFsCode @OOHehir
We still need to get to the bottom of the time synch errors.

Sending getStatus 16 times in a short span of time is definitely inefficient. The app requests info from the board depending on > UI events, so I will refactor the code and reduce the number of requests.

When i connect the app with the ELOC. I don't request anything. But in the firmware log i can see that requests are sent (or at least it looks like it). Sometimes 16 times (as experienced above), sometimes 4 times and sometimes 6 times. It's kind of random. @josephx86 Which UI events could request getStatus automatically without interaction?
Firmware synchs 1970 time from phone without error:
(Not sure if related to above but i post it here for now.)
I (34891) BtCmds: timestamp in from type ? Time Zone: 7 sec: 0 millisec: 0
I (34895) BtCmds: new timestamp from new sys time (local time) 25200000
I (34902) BtCmds: new timestamp from timeobJect (local time) 25200007
W (35577) BT_HCI: hcif mode change: hdl 0x80, mode 2, intv 798, status 0x0
W (37218) BT_HCI: hcif ssr evt: st 0x0, hdl 0x80, tx_lat 798 rx_lat 798
I (34909) BtCmds: new time set to (local time) Thu, Jan 01 1970 07:00:04

Here is the complete log: 1970 time and too many getStatus.txt
There is no error, but phone synchs time from 1970? Not sure how that is possible. That seems to happen only on one of my phones (which usually worked fine before, using the app from Playstore). For me also not clear if that is a problem from the app or from the FW.

@LIFsCode @OOHehir
While @josephx86 has made changes to make the timy synch more fail-safe, he found out that the response from the Firmware returns a different time, which seems strange.
The App sends seconds:1710682419 but the firmware returns:<PHONE_NUMBER> (which is a date from 2011)
I (898220) BtCmds: updating time with {"seconds":1710682419, "ms":0 ,"timezone" : 7}
I (898232) BtCmds: timestamp in from type ? Time Zone: 7 sec:<PHONE_NUMBER> millisec: 0
I (898239) BtCmds: new timestamp from new sys time (local time)<PHONE_NUMBER>000
I (898245) BtCmds: new timestamp from timeobJect (local time)<PHONE_NUMBER>
I (898252) BtCmds: new time set to (local time) Sun, Mar 17 2024 20:33:39

The ELOC does have the correct time after that. But maybe good to have the correct time in the response.

@edsteve That is a integer resolution problem. I w'll fix this.

@EDsteve I was trying to test the functionality of setting the ELOC from the Android app but couldn't find it. Am I missing something?

@OOHehir
You mean you couldn't find the app in the Playstore? Searching for ELOC doesn't bring up the correct app? Will need to do some SEO i guess. Haha
Here is the link to the ELOC app

@EDsteve That is a integer resolution problem. I w'll fix this.

@LIFsCode Just did a quick test of your commit regarding that issue. Suprisingly now the date in the response is a few months in the future. The date which gets returned in response after the "setTime" command is the "timeobJect (local time)<PHONE_NUMBER>".
I (3478494) BtCmds: updating time with {"seconds":1711103199, "ms":0 ,"timezone" : 8}
I (3478501) BtCmds: timestamp in from type ? Time Zone: 8 sec:<PHONE_NUMBER> millisec: 0
I (3478515) BtCmds: new timestamp from new sys time (local time)<PHONE_NUMBER>000
I (3478521) BtCmds: new timestamp from timeobJect (local time)<PHONE_NUMBER>
I (3478527) BtCmds: new time set to (local time) Fri, Mar 22 2024 18:26:39

@OOHehir
I just uploaded the new app which should fix the "too many request problem". Google needs a few hours to release an update though.
Also the Google Sign in worked with the last update on my phone. But please try it on your end if you want.

@josephx86 has fixed all issues regarding time synch from app

the 1970 issue happened only from specific phones which can't connect to time servers -> Only local phone time is synched now to the ELOC
The issues with too many requests after connectin to the ELOC via BT is resolved
Time synch procedure is more rigid

@LIFsCode has fixed the time-response time with 81eff2b9b891d80a43b3f239c870f6d4e7e1fdb9
I have tested all and woks fine so far -> Closing this issue
@OOHehir the new app should work fine. Also the Google Sign in worked on my side with the updated version.

@EDsteve OK, I see there was a new version of the app. I'll try that


chore(main): release 1.15.1

:robot: I have created a release beep boop
1.15.1 (2023-10-19)
Bug Fixes

scraper: ignore tests on scrape (c474936)
vite version (e7e5cc1)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/devoteam-se/devolunch/releases/tag/v1.15.1 :sunflower:


Remove extra m in edit comment Spanish translation

@eVoloshchak @johnmlee101   PR is ready for review.
Details
There is an extra ‘m’ in the edit comment translated into Spanish. i.e. It was 'Editar commentario' in Spanish. So we removed extra m from that So correct spelling will be "Editar comentario" in Spanish.
Fixed Issues
$ https://github.com/Expensify/App/issues/16053
PROPOSAL:  https://github.com/Expensify/App/issues/16053#issuecomment-1478745293
Tests

Open App in Web
Change language to Spanish
Go to any chat
Send a message
Hover or right click on the message.
Verify that tooltip for edit comment in Spanish shows Editar comentario previously it was 'Editar commentario' so we corrected it in this pr. (i.e. removed extra m)

[x] Verify that no errors appear in the JS console

Offline tests

Open App in Web
Change language to Spanish
Go to any chat
Send a message
Hover or right click on the message.
Verify that tooltip for edit comment in Spanish shows Editar comentario previously it was 'Editar commentario' so we corrected it in this pr. (i.e. removed extra m)

QA Steps

Open App in Web
Change language to Spanish
Go to any chat
Send a message
Hover or right click on the message.
Verify that tooltip for edit comment in Spanish shows Editar comentario previously it was 'Editar commentario' so we corrected it in this pr. (i.e. removed extra m)

Native App Test

Open App in iOS/Android/mWeb
Change language to Spanish
Go to any chat
Send a message
Long Press  on the message. It will show popover
Verify that text for edit comment in Spanish shows Editar comentario previously it was 'Editar commentario' so we corrected it in this pr (i.e. removed extra m)

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web
https://user-images.githubusercontent.com/7823358/226983456-28871878-10fe-444b-9924-810ff0dbb544.mov

Mobile Web - Chrome
https://user-images.githubusercontent.com/7823358/226983377-60b1d780-e8a9-44b1-9516-3f369e19a6e5.mov

Mobile Web - Safari
https://user-images.githubusercontent.com/7823358/226983591-36f76370-89dd-4da0-80ad-6f92e4c21e28.mov

Desktop
https://user-images.githubusercontent.com/7823358/226983656-4ae25c59-9914-409c-aaf0-d7b427ba18a7.mov

iOS
https://user-images.githubusercontent.com/7823358/226983724-dee2b940-115e-4993-9922-c1f8666641ee.mov

Android
https://user-images.githubusercontent.com/7823358/226983843-4d14cd23-2d0e-4f39-898e-39bdd955cadb.mov

@johnmlee101 @eVoloshchak One of you needs to copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

Reviewer Checklist

[ ] I have verified the author checklist is complete (all boxes are checked off).
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the steps for local testing are in the Tests section
[ ] I verified the steps for Staging and/or Production testing are in the QA steps section
[ ] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[ ] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I included screenshots or videos for tests on all platforms
[ ] I verified tests pass on all platforms & I tested again on:

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[ ] MacOS / Chrome / Safari
[ ] MacOS / Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[ ] I verified that comments were added to code that is not self explanatory
[ ] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[ ] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[ ] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[ ] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[ ] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[ ] I verified the JSDocs style guidelines (in STYLE.md) were followed

[ ] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines
[ ] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[ ] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[ ] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[ ] If a new component is created I verified that:

[ ] A similar component doesn't exist in the codebase
[ ] All props are defined accurately and each prop has a /** comment above it */
[ ] The file is named correctly
[ ] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[ ] The only data being stored in the state is data necessary for rendering and nothing else
[ ] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[ ] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[ ] All JSX used for rendering exists in the render method
[ ] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[ ] If any new file was added I verified that:

[ ] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[ ] If a new CSS style is added I verified that:

[ ] A similar style doesn't already exist
[ ] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[ ] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[ ] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[ ] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[ ] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/johnmlee101 in version: 1.2.89-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/luacmartins in version: 1.2.89-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


Not selecting the right element (use XPath or give a more specific element identifier)

I've been doing some tests with TagUI but the first one I did simply didn't work. The tests consist only on opening packet pub web page and logging in but TagUI simply doesn't type the password on the right element. Here is the script:
https://www.packtpub.com/packt/offers/free-learning/
click .login-popup .float-left
type email as<EMAIL_ADDRESS>type password as passwordhere
click edit-submit-1

The password is always typed on the email input element, right after the email. Using the XPath as the selector results on the same behavior. Am I doing something wrong or is it a bug?

Probably password as the selector is not specific enough. Below shows the order of selectors TagUI uses - https://github.com/kelaberetiv/TagUI#steps-description In general use something as specific as possible while balancing with natural language readability.
It depends on the user's comfort level to give something very specific that breaks easily if something changes, or something more generic which may still work with minor website changes.
You can try running in live mode to test and debug which selector works, right-click element in Chrome DevTools to copy XPath (https://github.com/kelaberetiv/TagUI#find-xpath-of-web-element), or search in Chrome DevTools interface to make sure it's not a duplicate.
When I copy the XPath returned is //*[@id="password"] and it works.

Adding on, the live mode can be started with the live step.

Hi @kensoh, I am scraping an html table that is spanning multiple pages. I want to extract the table into a datatable and then load this datatable in a csv file. Please guide how can I do this in tagui??
Thanks again and please reply as soon as possible

Please reply to my post as soon as possible! I need help in it

You can try the table step and provide the XPath of the table. This only works for simple HTML tables, some tables have merged cells which will not work with this step. If multiple pages, you have to google and write your own JavaScript code to append the different CSV files.

Thank you so much. I shall try this. Please try to reply the query as soon as possible. Thank you
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

From: Ken Soh<EMAIL_ADDRESS>Sent: Thursday, May 21, 2020 9:26:36 PM
To: kelaberetiv/TagUI<EMAIL_ADDRESS>Cc: Hassan Ali<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [kelaberetiv/TagUI] Not selecting the right element (use XPath or give a more specific element identifier) (#415)
You can try the table step and provide the XPath of the table. This only works for simple HTML tables, some tables have merged cells which will not work with this step. If multiple pages, you have to google and write your own JavaScript code to append the different CSV files.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/kelaberetiv/TagUI/issues/415#issuecomment-632190344, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJGETE5LNT3QSMJTC4465XLRSVI3ZANCNFSM4HJ26DGQ.

I'm away from work and no estimate of when I will be back - https://github.com/kelaberetiv/TagUI/issues/804
You can try raise a new issue with your sample script and website so other users can advise -

Ok no issue. Actually I asked this because I was in an issue last month so needed your help. But next time I shall create new issue
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10
From: Ken<EMAIL_ADDRESS>Sent: Thursday, 21 May 2020 21:35
To<EMAIL_ADDRESS>Cc: Hassan<EMAIL_ADDRESS><EMAIL_ADDRESS>Subject: Re: [kelaberetiv/TagUI] Not selecting the right element (use XPath or give a more specific element identifier) (#415)
I'm away from work and no estimate of when I will be back - #804https://github.com/kelaberetiv/TagUI/issues/804
You can try raise a new issue with your sample script and website so other users can advise -
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/kelaberetiv/TagUI/issues/415#issuecomment-632198724, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJGETEYEQJS4GTRKPZA6DHDRSVJ6FANCNFSM4HJ26DGQ.


LingoVaporLeaf as an available product

I forgot this change. Without this, the LingoVaporLeaf target cannot be used when importing the package in the Package.swift. See this example:

Released in 4.3.1

Thanks. I started implementing it in my web app, and it works like a charm 😍


Code Structure Suggestion: Namespaces and Modules

I suggest that we divide any new functionality into modules.
Such a module should have it's own namespace and it's own subdirectory, so for instance the module Gui would be in the namespace inexor::gui and would reside in src/gui.
These modules should provide a header with the name of the module (src/gui/gui.h) that servers as a convenience header to pull the entire functionality of that module.
Each of these modules should also provide a description of what the module does inside a readme file (src/gui/readme.md).
What do you think of these suggestions?

Sounds great so far.
Please that namespaces could also be used to make our code backward-compatible (see, #1 ) by outsourcing existing modules to (eg) sauer::
Anyhow I don't see the point for very small modules (such as jSON-parser) who can ship everything and once and don't need a seperate folder.

Yup, outsourcing to sauer:: would be cool!
For very small modules I would suggest, that we either keep them in util/ or in the root source folder src/.
I would like to avoid storing modules inside engine/, fpsgame/ or shared/, because I consider those to contain all the legacy code.

Accepted and moved to the wiki.


get illegal token error

I suddenly get this error when opening jingo:
SyntaxError: Unexpected token ILLEGAL
    at Function (native)
    at wrap (/home/user/jingo/node_modules/pug/node_modules/pug-runtime/wrap.js:8:30)
    at Object.exports.compile (/home/user/jingo/node_modules/pug/lib/index.js:263:7)
    at handleTemplateCache (/home/user/jingo/node_modules/pug/lib/index.js:216:25)
    at Object.exports.renderFile (/home/user/jingo/node_modules/pug/lib/index.js:418:10)
    at Object.exports.renderFile (/home/user/jingo/node_modules/pug/lib/index.js:408:21)
    at View.exports.__express [as engine] (/home/user/jingo/node_modules/pug/lib/index.js:455:11)
    at View.render (/home/user/jingo/node_modules/express/lib/view.js:128:8)
    at tryRender (/home/user/jingo/node_modules/express/lib/application.js:640:10)
    at EventEmitter.render (/home/user/jingo/node_modules/express/lib/application.js:592:3)
    at ServerResponse.render (/home/user/jingo/node_modules/express/lib/response.js:966:7)
    at ServerResponse.res.render (/home/user/jingo/node_modules/express-flash/lib/express-flash.js:29:16)
    at /home/user/jingo/lib/app.js:199:9
    at Layer.handle_error (/home/user/jingo/node_modules/express/lib/router/layer.js:71:5)
    at trim_prefix (/home/user/jingo/node_modules/express/lib/router/index.js:315:13)
    at /home/user/jingo/node_modules/express/lib/router/index.js:284:7

I already completely reinstalled jingo and created a new repo, but the error is still there. I cant imagine why?! As far as I know, that can happen when there is an invisible character in JS code, but how suddenly and why its not working after reinstallation?

@hajika ouch, this sounds bad. Can you give me a bit more details? When is that you get that? On starting jingo, on loading the first page or somewhere else?
thanks!

I get this on opening any page. I checked the wrap.js in the meantime, and it actually complains about the last + (that is the sign in wrap.js:8:30). When I remove the last two +, it will complain about the other plus. Strange.
var runtime = require('./');

module.exports = wrap;
function wrap(template, templateName) {
  templateName = templateName || 'template';
  return Function('pug',
    template + '\n' +
    'return ' + templateName + ';'
  )(runtime);
}

It might be because of the template variable, I put it in a console.log() and got a very long output with actual code, I guess thats not intended? What should be in the template var? I will attach the output here to give you a hint (everything between the xxx):
jingolog.txt

Thanks, but this is still not enough: can you please send me also your config.yaml? What is your server (or workstation) configuration? Windows maybe?

CentOs 6
As I said, I completely reinstalled the whole thing, in the config.yaml I only changed the port and the repository path, anything else is like in the default. I also initiated a new git repo.

So I still encounter that problem with the master branch, but not with the v1.7.3, so I hope this will vanish with the next release :)

It must have something to do with the 1.8.0 release, then (there were big changes there), but I cannot reproduce it on OSX and Ubuntu at the moment.
One of the templates is probably broken somehow but I can't understand where the problem is.

In the meanwhile try the current version (1.8.1)

Can you also please tell me you node js version?

I just tried the v1.8.2 and the error appears again. Node version is v0.12.9, but I can change I easily if there is any benefit in it..

I am pretty sure Node 0.x is not supported anymore, or not supported very well at least.
Please use any of the LTS (since you are updating, go for 6.x)
Thanks


Update swearwords.go

don't pr to master
legacy won't receive updates
use automod v2
how is this beneficial?


LCFS - Add UCI column to "Fuel supply" and "Export" sections of the compliance report

Describe the task
Add a new input column titled UCI to the Fuel Supply and Export sections of the compliance report. This column should be located immediately after the CI of Fuel column. When an option is selected in the End Use column, the UCI column should automatically populate with the numerical values specified in card #574.
Purpose
To enhance the Fuel Supply and Export sections by including a UCI column that provides relevant numerical values based on the End Use selection, improving data accuracy and streamlining the compliance reporting process.
Acceptance Criteria

[ ] Add a new column titled UCI in the Fuel Supply and Export sections after the CI of Fuel column.
[ ]  The UCI column should populate automatically with the correct numerical values from card #574 based on the End Use selection.
[ ]  Ensure that the populated values in the UCI column are accurate and consistent with the values specified in card #574.
[ ]  Confirm that this feature works in both the Fuel Supply and Export sections and does not interfere with other fields or calculations.

Additional context

related to card #1175

@StuartGALL hi! This is the design ask I had related to adding a UCI column to the compliance report Fuel supply and Export sections. Please add a link to a wireframe once you have one ready to go. Thank you!

We are currently not clear on exactly how Marine use will apply to LNG, so ticket is blocked

For QA:
There are currently no fuel codes in the system setup to use UCI, so you may need to make a new one for testing.


Don't 500 on invalid dates for PPU

URLs like /ccg/07H/price_per_unit/?date=123 are being spidered, and are producing a lot of error alert noise.
These should return a 200 to the user, with an error message that the date is not valid.

👍 for 400 from me.

Yes, makes sense -- though shouldn't we use rest_framework exceptions like the 400s we already raise e.g. here?

We should indeed!

Deployed to the staging server:
http://staging.openprescribing.net/ccg/07H/price_per_unit/?date=not-really-a-date


Updates to Notebook Cell output causes abrupt flashing/flickering in notebooks and interactive window

python -m pip install livelossplot
Run this code https://github.com/microsoft/vscode-jupyter/blob/b4a3e1d13446c27b0bedf66272b08f5e13ace57f/src/test/datascience/liveloss.py
Observe some jarring flashing behavior as the plot updates

The visual effect is exacerbated for interactive window because auto scrolling is done in VS Code core, but the issue is also present for notebooks (abrupt flashing as the plot disappears and gets recreated)

In classic Jupyter there's no such flashing. The output updates are smooth:

Also reported here https://github.com/microsoft/vscode/issues/132143
Old behavior (no flashes, in webview interactive & webview notebooks)

New behavior
(Note: as this is a gif, the flickering isn't as noticeable).

@DonJayamanne @joyceerhl when you transfer issues, would you mind assigning them to the right person, if you know who that is?

I can still repro this with the same code for the progress bar.

The progress bar might be a different case, @DonJayamanne can you share the code?

from fastprogress.fastprogress import master_bar, progress_bar
from time import sleep
mb = master_bar(range(10))
for i in mb:
    for j in progress_bar(range(100), parent=mb):
        sleep(0.01)
        mb.child.comment = f'second bar stat'
    mb.main_bar.comment = f'first bar stat'
    mb.write(f'Finished loop {i}.')

Spent the whole afternoon digging into this and finally found the root cause: the flash is caused by overflow: hidden. Right now there is no primitive for "replace output" in the webview layer so we will always perform a remove and then add DOM operation. The remove will make the height of the container to be zero and since it has overflow: hidden, we are not going to see anything inside the output container. The next add enlarges the container and becomes visible. Somehow the browser doesn't optimize for this case and always "flashes".
The overflow: hidden was added to ensure that we see minimal overlap of outputs and code editors. Apparently it comes with a cost ;)(

Have the same problem here when visualising gradient descent with Matplotlib on VSCode notebooks :(

@rebornix Any progress on this issue? Progress bars like that of @DonJayamanne cause a strobe-like flickering due to constant redraws.

Issue mentioned on stackoverflow: https://stackoverflow.com/questions/69037124/vscode-version-1-60-0-changed-interactive-view-to-epilepsy-old-or-web-view-av

Affects me as well.

Verifier, please test when you are rerunning cells with outputs, the cell doesn't shrink too early, you can test this with following code
print('hello')
print('world')


Create test environment for Rex Engine

a PR has been made https://github.com/New-Creta/Rex/pull/177


Placement modes: calculate required variables in specific placement mode only

PlacementMode.gd generates random coordinates for the MultiMesh instances. Each specific placement modes then generates the height of the spawned of the instance and place it to the new height.
However, some placement modes got more complex algorithms and require additional information which are passed to all placement modes.
This should be cleaned up, so that each placement mode only gets the information which are required for this placement mode.
This also includes that the calculation of the random position should take place in the specific placement mode.
This would make it easier to add new placement modes which use different algorithms.

This requires a lot of rework and refactoring. The way to calculate the position of the spawned instance is different for each Placment mode. So there should be a more object based implementation of the PlacementModes so that each placment mode takes control of it's own algorithm.


IndexError: index 8 is out of bounds for axis 0 with size 7

Code:
#Specify path to the coco.json file
path_to_annotations = "data/coco.json"
#Specify the path to the images (if they are in a different folder than the annotations)
path_to_images = "data/image_patches/"

#Import the dataset into the pylable schema
dataset = importer.ImportCoco(path_to_annotations, path_to_images=path_to_images, name="tanks_coco")
dataset.df.head(5)

Error:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb Cell 7' in <cell line: 7>()
      [4](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=3)[ path_to_images = "data/image_patches/"
      ]()[6](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=5)[ #Import the dataset into the pylable schema
----> ]()[7](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=6)[ dataset = importer.ImportCoco(path_to_annotations, path_to_images=path_to_images, name="tanks_coco")
      ]()[8](vscode-notebook-cell:/Users/robin/Documents/GitHub/oil-storage-tank/pre-processing.ipynb#ch0000010?line=7)[ dataset.df.head(5)

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py:94, in ImportCoco(path, path_to_images, name)
     ]()[89](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=88)[ df.ann_category_id = df.ann_category_id.astype(str)
     ]()[91](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=90)[ df[
     ]()[92](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=91)[     ["ann_bbox_xmin", "ann_bbox_ymin", "ann_bbox_width", "ann_bbox_height"]
     ]()[93](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=92)[ ] = pd.DataFrame(df.ann_bbox.tolist(), index=df.index)
---> ]()[94](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=93)[ df.insert(8, "ann_bbox_xmax", df["ann_bbox_xmin"] + df["ann_bbox_width"])
     ]()[95](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=94)[ df.insert(10, "ann_bbox_ymax", df["ann_bbox_ymin"] + df["ann_bbox_height"])
     ]()[97](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=96)[ # debug print(df.info())
     ]()[98](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=97)[
     ]()[99](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pylabel/importer.py?line=98)[ # Join the annotions with the information about the image to add the image columns to the dataframe

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py:4439, in DataFrame.insert(self, loc, column, value, allow_duplicates)
   ]()[4436](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py?line=4435)[     raise TypeError("loc must be int")
   ]()[4438](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py?line=4437)[ value = self._sanitize_column(value)
-> ]()[4439](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/frame.py?line=4438)[ self._mgr.insert(loc, column, value)]()
File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:1230, in BlockManager.insert(self, loc, item, value)
   [1220](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1219)[ """
   ]()[1221](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1220)[ Insert item at selected position.
   ]()[1222](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1221)[
   (...)
   ]()[1227](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1226)[ value : np.ndarray or ExtensionArray
   ]()[1228](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1227)[ """
   ]()[1229](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1228)[ # insert to the axis; this could possibly raise a TypeError
-> ]()[1230](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1229)[ new_axis = self.items.insert(loc, item)
   ]()[1232](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1231)[ if value.ndim == 2:
   ]()[1233](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py?line=1232)[     value = value.T

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py:6602, in Index.insert(self, loc, item)
   ]()[6595](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6594)[ if arr.dtype != object or not isinstance(
   ]()[6596](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6595)[     item, (tuple, np.datetime64, np.timedelta64)
   ]()[6597](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6596)[ ):
   ]()[6598](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6597)[     # with object-dtype we need to worry about numpy incorrectly casting
   ]()[6599](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6598)[     # dt64/td64 to integer, also about treating tuples as sequences
   ]()[6600](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6599)[     # special-casing dt64/td64 https://github.com/numpy/numpy/issues/12550
   ]()[6601](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6600)[     casted = arr.dtype.type(item)
-> ]()[6602](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6601)[     new_values = np.insert(arr, loc, casted)
   ]()[6604](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6603)[ else:
   ]()[6605](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6604)[     # No overload variant of "insert" matches argument types
   ]()[6606](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6605)[     # "ndarray[Any, Any]", "int", "None"  [call-overload]
   ]()[6607](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/pandas/core/indexes/base.py?line=6606)[     new_values = np.insert(arr, loc, None)  # type: ignore[call-overload]

File <__array_function__ internals>:180, in insert(*args, **kwargs)

File ~/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py:5280, in insert(arr, obj, values, axis)
   ]()[5278](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5277)[ index = indices.item()
   ]()[5279](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5278)[ if index < -N or index > N:
-> ]()[5280](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5279)[     raise IndexError(f"index {obj} is out of bounds for axis {axis} "
   ]()[5281](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5280)[                      f"with size {N}")
   ]()[5282](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5281)[ if (index < 0):
   ]()[5283](file:///Users/robin/Documents/GitHub/oil-storage-tank/venv/lib/python3.9/site-packages/numpy/lib/function_base.py?line=5282)[     index += N

IndexError: index 8 is out of bounds for axis 0 with size 7]()

Hello @robmarkcole I figured out what the problem is. The .json file for that dataset is missing some of the fields that are usually in a COCO annotation file. For example it doesn't have the height or width of the images:
   {
        "id": 5142,
        "file_name": "52_4_2.jpg"
    },

    It also doesn't have some of the usual metadata about the annotations.

   Unfortunately this library won't work with that dataset.

@alexheat do you know of any validators that could be run against the file? Did a brief search but did not find one

I am not aware of any validation tool or even an official place where the schema is documented. But the image height and width are included in the original COCO dataset. The height and width are required information to convert to other formats like YOLO because it requires calculating the size of the bounding box relative to the size of the image.

this looks like the spec https://cocodataset.org/#format-data
for this issue, I suggest raising an exception if the keys are missing.

I have updated the json file but still get the error... file on https://gist.github.com/robmarkcole/286951efe0830d3e205666f68c2b490e

OK the issue was missing area in the annotations

this looks like the spec https://cocodataset.org/#format-data
for this issue, I suggest raising an exception if the keys are missing.

Thank you. Glad you got it working. I will keep the issue open as a reminder to add the better error handling.


Unexpected termination on USBTransferFragment summary

Sometimes ViewSB will terminate unexpectedly with the following error in the presence of a packet fragment:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/code/src/ViewSB/viewsb/ipc.py", line 124, in _subordinate_process_entry
    task.run()
  File "/code/src/ViewSB/viewsb/frontend.py", line 226, in run
    self.handle_communications()
  File "/code/src/ViewSB/viewsb/frontend.py", line 218, in handle_communications
    self.handle_incoming_packet(packet)
  File "/code/src/ViewSB/viewsb/frontends/cli.py", line 21, in handle_incoming_packet
    print(repr(packet))
  File "/code/src/ViewSB/viewsb/packet.py", line 304, in __repr__
    self.summarize(), data_summary, self.summarize_status())
  File "/code/src/ViewSB/viewsb/packet.py", line 588, in summarize
    return "ORPHANED {}B {}-{} transfer".format(len(self.data), self.direction.name, self.pid.name)
AttributeError: 'NoneType' object has no attribute 'name'

As a quick workaround to test ViewSB for the first time, I made the following change to check that self.direction and self.pid are defined first.
index b1145eb..e527704 100644
--- a/viewsb/packet.py
+++ b/viewsb/packet.py
@@ -585,7 +585,8 @@ class USBTransferFragment(USBTransfer):
     FIELDS = {'transfer_type'}

     def summarize(self):
-        return "ORPHANED {}B {}-{} transfer".format(len(self.data), self.direction.name, self.pid.name)
+        if self.direction and self.pid:
+            return "ORPHANED {}B {}-{} transfer".format(len(self.data), self.direction.name, self.pid.name)

 class USBBulkTransfer(USBDataTransfer):

Closed in preparation for archiving the repository. We have replaced ViewSB with Packetry.