Fix for missing ShardReplicationTasks on new nodes

Signed-off-by: Ankit Kala<EMAIL_ADDRESS>Description
In current implementation of IndexReplicationTask, if the task gets killed and spawned up on a new node, it tries to figure out if ShardReplicationTask for all the shards are running or not. If not, it tries to create those again. This logic is broken as of now as instead of checking for ShardReplicationTask for current index, it check for all the ShardReplicationTask on the cluster. This change fixes this by filtering the tasks specific to the current index.
Ideally we should be adding IT for this case but haven't as simulating the usecase would be very difficult.
Testing done
Simulating this code path would be tricky locally as we need to stop the ShardReplicationTasks and then immediately kill the IndexReplicationTask.
For local testing, I've verified that the filtering logic that is added here works as expected and only filters tasks related to the current index.
Issues Resolved
482
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #497 (8654467) into main (a859596) will decrease coverage by 2.22%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main     #497      +/-   ##
============================================
- Coverage     75.16%   72.94%   -2.23%
+ Complexity     1007      988      -19
============================================
  Files           141      141
  Lines          4579     4583       +4
  Branches        506      506
============================================
- Hits           3442     3343      -99
- Misses          823      922      +99
- Partials        314      318       +4

Impacted Files
Coverage Δ

...rch/replication/task/index/IndexReplicationTask.kt
67.20% <0.00%> (-3.87%)
:arrow_down:

...cation/action/changes/TransportGetChangesAction.kt
63.49% <0.00%> (-25.40%)
:arrow_down:

...ch/replication/task/index/IndexReplicationState.kt
51.66% <0.00%> (-15.00%)
:arrow_down:

...ication/action/setup/TransportSetupChecksAction.kt
59.52% <0.00%> (-14.29%)
:arrow_down:

...ication/seqno/RemoteClusterRetentionLeaseHelper.kt
80.00% <0.00%> (-11.67%)
:arrow_down:

...tion/repository/RemoteClusterMultiChunkTransfer.kt
90.24% <0.00%> (-9.76%)
:arrow_down:

.../replication/metadata/store/ReplicationMetadata.kt
62.96% <0.00%> (-8.65%)
:arrow_down:

...action/stop/TransportStopIndexReplicationAction.kt
68.00% <0.00%> (-6.67%)
:arrow_down:

...earch/replication/metadata/UpdateIndexBlockTask.kt
74.19% <0.00%> (-6.46%)
:arrow_down:

...ication/action/stop/StopIndexReplicationRequest.kt
50.00% <0.00%> (-5.00%)
:arrow_down:

... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

One thing I am not clear is why the missing shard tasks where not spawned . We do keep calling startMissingShardTasks in the MONITORING state  : https://github.com/opensearch-project/cross-cluster-replication/blob/main/src/main/kotlin/org/opensearch/replication/task/index/IndexReplicationTask.kt#L210
This fix is needed for sure, but it won't fix the current problem

One thing I am not clear is why the missing shard tasks where not spawned . We do keep calling startMissingShardTasks in the MONITORING state : https://github.com/opensearch-project/cross-cluster-replication/blob/main/src/main/kotlin/org/opensearch/replication/task/index/IndexReplicationTask.kt#L210
This fix is needed for sure, but it won't fix the current problem

That is because these ShardReplicationTasks were not in failed state. cc: @soosinha

This fix is needed for sure, but it won't fix the current problem

Yes, there are two places, where we need to make this fix.
For both startShardTaks and MissingShardTasks (and consolidate logic).
For startShardTasks, the issue is seen when index replication task get initialises on the new node and If the replication tasks are not executing, It won't start again.

The changes to add IT for this case is not added in this PR. In the meanwhile, I'll explore on how we can add test to simulate this scenario.

LGTM . can we write UTs for same, since integ tests are not possible ?

My bad. DIdn't realise that we have configured the unit test for IndexReplicationTask. Added the unit test to cover the new method.


Is it possible to share clipboard with host?

Currently I am trying to share the system clipboard between host and vm to allow for easier copy and pasting.

Clipboard of what?

An example would be if I copied something from vim (in vm) then wanted to paste into safari (on my Mac).

Are you using something other than the regular terminal application, to access your limactl shell ?

Nope just iterm2 and then directly into lima shell

I was thinking that it would have the usual "select text" and copy/paste functions, but am missing something...

I was thinking that it would have the usual "select text" and copy/paste functions, but am missing something...

It does, and it works fine if you use iTerm copy/paste functionality.
@euforic Are you using vim commands to copy into the clipboard? That will not work because the Linux vim is not connected to the macOS clipboard. But if you select the text using iTerm and copy via ⌘C, then it works fine.

This plugin looked interesting: https://github.com/ojroques/vim-oscyank

I will dig a little deeper into my config make sure its not something on my end. Thank you for the help 🤙🏽. Also thank you for this project it is awesome. All the work is truly appreciated 🙏🏼

I was thinking that it would have the usual "select text" and copy/paste functions, but am missing something...

It does, and it works fine if you use iTerm copy/paste functionality.
@euforic Are you using vim commands to copy into the clipboard? That will not work because the Linux vim is not connected to the macOS clipboard. But if you select the text using iTerm and copy via ⌘C, then it works fine.

Yeah I am using the system clipboard and was trying to bridge the linux system clipboard with then Mac OS system clip board similar to how UTM does it with the Spice tools

https://iterm2.com/documentation-utilities.html - see it2copy
Copies text to the pasteboard. Works over ssh. Accepts either standard input or a named file.

It turns out (and TIL), sharing clipboard isn't a straight forward thing to make it happen (at least via SSH, X11 forwarding seems to be the promising way to do so).
So I was able to make it work via that way.
This requires installing XQuartz and X11 forwarding (which is described in the gist) and I chose to SSH to lima guest via ssh (but this requires additional authentication setup - like adding public key to authorized_keys on lima)  instead of lima since that's the way I know at the moment. But lima configuration yaml also seems to support ssh configuration there, so you might get lucky with that.
Although it's not an auto magic way that lima just comes with, but I'm quite satisfied with the way it works at the moment.


Ch 4 - Redesign Online DFS Agent diagram

The existing diagram is an implementation of the algorithm but we want diagrams that show the concepts and how the algorithm works. For example, the diagram should show untried and unbacktracked from the algorithm.

@redblobgames Hey, i wish to work on this.I want to propose a few representations and ask for improvements.

In the same grid, at every current state changing the colors of the surrounding cells based on whether they are unbacktracked or not and also showing arrows for the all the directions of untried.
Showing three grids, one for actual movement, one for the untried and one for the unbacktracked.

Apart from these i also propose to add a few more things like
1.pausing
2.going back to previous state
3.Initial state and goal given by the user.
4.Cost function calculation.
so that anyone who is looking at the simulation can understand whether it is truly online dfs or not.
Can you  please suggest me improvements  so that i can start working on it.
P.S:- I am very new to designing but i look forward to learn it by contributing and brainstorming in a more known field i.e AI.

@kaushikgunda If you are interested in GSoC 2018, then be sure to read https://github.com/aimacode/aima-gsoc and contact the mentor. If you are NOT interested in GSoC, then yes, feel free to work on this issue, but I am no longer on this project so you may have to wait until there are new people who start working on it.


Add a section for search/ in the docs

As the endpoint is public, we should have a section in https://huggingface.co/docs/datasets-server.
For https://huggingface.co/docs/hub/datasets-viewer, let's wait to have the search integrated into the Hub dataset viewer.

Also: add the field search in https://huggingface.co/docs/datasets-server/valid

Done


[ENH] missing data input check in forecasters

This PR introduces missing data input checks to forecasters and fixes #3402.
Changes:

_check_X_y in forecasters now raises an informative exception if input contains missing data and the forecaster does not support it
the forecaster extension template is updated accordingly
reducers are updated so missing value handling tags mirror the wrapped sklearn estimator's tag
switched the tag to True in some estimators causing issues: AutoETS

Hm, this seems to work as expected, but the WindowSummarizer in one example, in the hierarchical forecasting notebook seems to produce X/y with missing values? Which previously did not break the reducer? Odd.
Are the sklearn tags perhaps not reliable?


Addon will not start with shae.sh RPi4 HASS

Add-on version 1.17.1
Running HassOS op RPi 4
Shae.sh CC2652RB flashed with CC2652RB_coordinator_20210120.hex (tested with znp_CC2652RB_20200715.bin and that worked)
Antenna plugged in to USB3 port (tried 2 as well)
Running Mosquito broker as add on (I tested the connection with my HASS user / pass)

Description of problem:
The add on cannot start. I get the gateway error if I go to the UI.
I tried different ports, flashing it again, reinstalling the add-on. When I flash the debugging firmware, both LEDs go on. So flashing works.
I see the number it often ttyAM*. With my device, I get ttyUSB0. I use it in combination with a P1 meter. I tried using the long name as well with the same result: /dev/serial/by-id/usb-Silicon_Labs_slae.sh_cc2652rb_stick_-_slaesh_s_iot_stuff_00_12_4B_00_21_CC_4B_CE-if00-port0
Neither button on the device changes the LEDs. I think I read that the LED should go off when resetting, I don't see that happening.
Did I just miss something very silly?
USB number
~ $ ls -lt /dev/serial/by-id/
total 0
lrwxrwxrwx    1 root     root            13 Feb 10 17:26 usb-Silicon_Labs_slae.sh_cc2652rb_stick_-_slaesh_s_iot_stuff_00_12_4B_00_21_CC_4B_CE-if00-port0 -> ../../ttyUSB0

~ $ ls -lt /dev/ttyUSB0
crw-rw----    1 root     audio     188,   0 Feb 10 17:42 /dev/ttyUSB0

Your entire configuration from the frontend (with sensitive fields redacted):
data_path: /share/zigbee2mqtt
external_converters: []
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: true
mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://localhost:1883'
  user: redacted
  password: redacted
serial:
  port: /dev/ttyUSB0
  disable_led: true
advanced:
  log_level: debug
  pan_id: 6754
  channel: 11
  network_key:
    redacted, unchanged
  availability_blocklist: []
  availability_passlist: []
device_options: {}
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental:
  new_api: true
socat:
  enabled: false
  master: 'pty,raw,echo=0,link=/dev/ttyZ2M,mode=777'
  slave: 'tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5'
  restartdelay: 1
  initialdelay: 1
  options: '-d -d'
  log: false

Your logs from Home Assistant
<EMAIL_ADDRESS>start: `node index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-02-10T16_41_47_759Z-debug.log
2021-02-10T17:41:48: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
2021-02-10T17:41:48: PM2 log: App [npm:0] starting in -fork mode-
2021-02-10T17:41:48: PM2 log: App [npm:0] online
><EMAIL_ADDRESS>start /zigbee2mqtt-1.17.1
> node index.js
Zigbee2MQTT:debug 2021-02-10 17:41:52: Can't load state from file /share/zigbee2mqtt/state.json (doesn't exist)
Zigbee2MQTT:info  2021-02-10 17:41:52: Logging to console and directory: '/share/zigbee2mqtt/log/2021-02-10.17-41-50' filename: log.txt
Zigbee2MQTT:debug 2021-02-10 17:41:52: Removing old log directory '/share/zigbee2mqtt/log/2021-02-10.17-37-19'
Zigbee2MQTT:info  2021-02-10 17:41:52: Starting Zigbee2MQTT version 1.17.1 (commit #1.17.1)
Zigbee2MQTT:info  2021-02-10 17:41:52: Starting zigbee-herdsman (0.13.59)
Zigbee2MQTT:debug 2021-02-10 17:41:52: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null},"backupPath":"/share/zigbee2mqtt/coordinator_backup.json","databaseBackupPath":"/share/zigbee2mqtt/database.db.backup","databasePath":"/share/zigbee2mqtt/database.db","network":{"channelList":[11],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":6754},"serialPort":{"path":"/dev/ttyUSB0"}}'
Zigbee2MQTT:error 2021-02-10 17:42:12: Error while starting zigbee-herdsman
Zigbee2MQTT:error 2021-02-10 17:42:12: Failed to start zigbee
Zigbee2MQTT:error 2021-02-10 17:42:12: Exiting...
Zigbee2MQTT:error 2021-02-10 17:42:12: Error: Failed to connect to the adapter (Error: SRSP - SYS - ping after 6000ms)
    at ZStackAdapter.<anonymous> (/zigbee2mqtt-1.17.1/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/zStackAdapter.js:94:31)
    at Generator.throw (<anonymous>)
    at rejected (/zigbee2mqtt-1.17.1/node_modules/zigbee-herdsman/dist/adapter/z-stack/adapter/zStackAdapter.js:25:65)
npm
 ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR!
<EMAIL_ADDRESS>start: `node index.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2021-02-10T16_42_12_222Z-debug.log
2021-02-10T17:42:12: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
2021-02-10T17:42:12: PM2 log: App [npm:0] starting in -fork mode-
2021-02-10T17:42:12: PM2 log: App [npm:0] online
><EMAIL_ADDRESS>start /zigbee2mqtt-1.17.1
> node index.js

dmesg output
[ 1603.323625] cp210x ttyUSB0: failed set request 0x7 status: -19
[ 1603.323662] cp210x ttyUSB0: failed set request 0x12 status: -19
[ 1603.323678] cp210x ttyUSB0: failed set request 0x0 status: -19
[ 1603.326259] cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0
[ 1603.326389] cp210x 1-1.1:1.0: device disconnected
[ 1621.586763] usb 1-1.1: new full-speed USB device number 7 using xhci_hcd
[ 1621.732186] usb 1-1.1: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[ 1621.732196] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1621.732203] usb 1-1.1: Product: slae.sh cc2652rb stick - slaesh's iot stuff
[ 1621.732210] usb 1-1.1: Manufacturer: Silicon Labs
[ 1621.732217] usb 1-1.1: SerialNumber: 00_12_4B_00_21_CC_4B_CE
[ 1621.736517] cp210x 1-1.1:1.0: cp210x converter detected
[ 1621.744358] usb 1-1.1: cp210x converter now attached to ttyUSB0
[ 2326.874549] kauditd_printk_skb: 8 callbacks suppressed
[ 2326.874553] audit: type=1325 audit(1612975068.051:284): table=nat family=2 entries=29
[ 2326.911835] audit: type=1325 audit(1612975068.091:285): table=filter family=2 entries=37
[ 2326.945928] audit: type=1325 audit(1612975068.121:286): table=nat family=2 entries=28
[ 2326.963893] hassio: port 11(veth044b14c) entered disabled state
[ 2326.964226] veth3031e82: renamed from eth0
[ 2335.985588] hassio: port 11(veth044b14c) entered disabled state
[ 2336.003816] device veth044b14c left promiscuous mode
[ 2336.003843] hassio: port 11(veth044b14c) entered disabled state
[ 2336.004139] audit: type=1700 audit(1612975077.131:287): dev=veth044b14c prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 2337.658477] hassio: port 11(veth646f999) entered blocking state
[ 2337.658486] hassio: port 11(veth646f999) entered disabled state
[ 2337.666545] device veth646f999 entered promiscuous mode
[ 2337.666603] audit: type=1700 audit(1612975078.841:288): dev=veth646f999 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 2337.797369] audit: type=1325 audit(1612975078.981:289): table=nat family=2 entries=27
[ 2337.849438] audit: type=1325 audit(1612975079.031:290): table=filter family=2 entries=36
[ 2337.879083] audit: type=1325 audit(1612975079.061:291): table=nat family=2 entries=28
[ 2338.673070] audit: type=1325 audit(1612975079.850:292): table=nat family=2 entries=0
[ 2338.697582] audit: type=1325 audit(1612975079.880:293): table=filter family=2 entries=0
[ 2338.756795] audit: type=1325 audit(1612975079.940:294): table=nat family=2 entries=5
[ 2338.769226] audit: type=1325 audit(1612975079.950:295): table=nat family=2 entries=7
[ 2338.789017] audit: type=1325 audit(1612975079.970:296): table=nat family=2 entries=8
[ 2338.991374] eth0: renamed from veth76de7e1
[ 2339.028481] IPv6: ADDRCONF(NETDEV_CHANGE): veth646f999: link becomes ready
[ 2339.028612] hassio: port 11(veth646f999) entered blocking state
[ 2339.028619] hassio: port 11(veth646f999) entered forwarding state

I fixed it.
I suspect the problem was related with another usb device (DSMR smart meter) that used to have USB0 mapping. Perhaps that integration was keeping the antenna occupied. I changed that to USB1 and set the zigbee2mqtt to serial by id and now it works like a charm.


Title in Scaffold AppBar does not show head characters correctly in RTL

In RTL TextDirection when use Scaffold widget, its AppBar does not show head characters correctly.
I use Flutter (Channel beta, v0.1.5)

Can you please also try in dev channel. There were quite a few changes since beta was released

In dev channel TextOverflow.ellipsis worked fine but TextOverflow.clip and TextOverflow.fade did not work correctly.

master channel might be worth a try as well for this topic, just to figure out if it's already fixed.

master channel was like dev channel.

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.


removed redundant transition declarations

transition property declarations should only need to be on the base element instead of the modifiers unless the transition-duration or transition-timing-function
http://codepen.io/davidpett/pen/EgbLQP

if this looks good, i will go ahead and make the changes in the other modules as well.

Thanks for the PR but going to punt on this for now. Set this up so that the values were explicit per state and could easily be changed.

That makes sense to me. My thinking was just using default tachyons it would be unnecessary, if a user wants to customize, they would be editing anyway.
No worries.

Yeah - totally get the reasoning. If you roll through the commit history I'm quite sure the module looks like the patch you proposed :)
What I found was, by not exposing it a lot of people didn't know you could edit it / customize it. And I found the extra lines didn't add file weight for me to care about. That being said I don't have the strongest opinion on this so I'll keep it in mind for v5.


Webworker

Hi there! Great work on this plugin!
Has anybody managed to run this in a webworker?

I haven't tried it but it should work. I am assuming you want to use a webworker so that the ui doesn't get blocked during the face recognition? Do keep in mind that you won't be able to access dom elements from a webworker.

Wanted to use a tensor as input in the worker, to avoid DOM elements, would just transfer the image data to the worker and then transfer the results back m.

I did it and it works, but at the moment tfjs is not compatible with offscreen canvas (https://github.com/tensorflow/tfjs/issues/102) so you don't have access to the GPU from a webworker and the result is more than slow actually ...
If your purpose is to get better performances I advice you to have a try with the mtcnn model which is really faster, see the 0.9.0 version of this project.

Hi All,
Is there a a sample code how tsjf could be used inside a web worker.
I am using something like this.
let window = self;
importScripts("https://cdn.jsdelivr.net/npm/@tensorflow/tfjs");
It does not work.

Out of curiosity, how did you make it work ?

I had same experience with tracking.js library. I managed to to put its part to web worker https://github.com/eduardolundgren/tracking.js/issues/99 and completely get rid of the UI lags but code was just for POC

I did it and it works, but at the moment tfjs is not compatible with offscreen canvas (tensorflow/tfjs#102) so you don't have access to the GPU from a webworker and the result is more than slow actually ...
If your purpose is to get better performances I advice you to have a try with the mtcnn model which is really faster, see the 0.9.0 version of this project.

@akofman I also would be interested in how you got it working in a worker, if you're willing to share!

Bumping this for relevance because i'm now working on doing the same thing and with offscreen canvas. My project is here if anyone wants to check out why. I'm rendering a threejs scene and the face detection allows me to use face detection to control the perspective of the 3d scene. Each time the face detection runs at 100ms, the scene janks out briefly because the 3d part runs in about 1ms cpu time over 16ms budget for RAF calls, but the face detection part goes 40ms-100ms so you lose 3 frames at a time, making it look like the rendering is broken. Now I have to find a way to get it under budget for the reasons @thexiroy mentioned

It's probably better to ask for help at tfjs regarding how to get this running in a webworker

For those interested, there is a pull request open here which was a bit dated, but I've been working on. The OffscreenCanvas support doesn't appear that involved, but there doesn't appear to be any special consideration for web workers or transferable objects, and those may take longer to integrate in. I did see there is a branch for web workers open, but I haven't been through it.
https://github.com/tensorflow/tfjs-core/pull/1221

@justadudewhohacks tfjs updates is a non-starter. Typescript wont have support for OffscreenCanvas until 3.5.0 and that's not officially released. Even if they do release it, it's currently buggy and TensorFlow wont build against 3.5.0 without ts-ignore hacks. Even if you do the ts-ignore hacks, the resulting build of tensorflow running in face-api.js barf due to those ignored incompatibilities. Models don't work. flattened maps don't work. Whole thing barfs.
So, hacking created and updated tickets for those findings. The tfjs-core update thread above was updated. I also created a typescript issue that can be tracked here: https://github.com/Microsoft/TypeScript/issues/30998
It looks like things on both those projects move pretty quickly, so hopefully this wont fall to the bottom of the thousands of filed issues on the pile and actually get some updates. For now, I will be attempting to create a fake interface in the worker thread which proxies back to the main js with updates and commands. The approach is similar to one that @mizchi took here: https://github.com/tensorflow/tfjs/issues/102#issuecomment-462167706
The difference between @mizchi 's approach and my approach will be that i am attempting to fool the tensorflow library into believing it is running under non-worker thread conditions using my new found knowledge of how it works (gained by trying to fix their code). The plan is to build a faux document object and window object, complete with interfaces and values the library checks for when creating a canvas. Instead, I'll return wrapped instances of OffscreenCanvas, maybe with Proxy Traps, and catch any calls by the library to APIs I haven't stubbed out and build adaptors for those to canvas. Because TensorFlow does not ever return canvas elements to be drawn to the screen, the only overhead I'll have to worry about is sending the data from video into the worker to be processed. Because I'll be doing this with ImageBitmap, those updates will be zero-copy transferable objects (low latency). I suppose this is somewhat of a "shim" pattern and could be added to face-api.js as an adaptor or different API call if it works.
 You'll end up having to update core, then update all the other modules the depend on core, then rebuild the whole tensorflow project with the same version of typescript, which in the case of this feature set, would be "next" or 3.5.0+, all of which aren't compatible with that version out of the box (at this time). Typescript appears to be driven primarily by features in the IE/Edge browser suite because Microsoft owns that project, and TensorFlow being Google but subject to limitations in Typescript means they A) have their own blessed version of Typescript, B) this is older than whatever the most recent release is, and C) is not as up to date with the features of the web as Google's Chrome browser team support. Maybe, eventually, once MS moves Edge to the Chromium/Blink/Whatever engine, possibly Typescript becomes one with the universe and offers support for these DOM features in sync with minimally Chrome and Edge, but ideally all major browsers. That would be awesome! But, back to the topic, the issues I ended up seeing were related to compilation errors stemmed from code that was doing type conversion like " float32ToTypedArray ", including some map functions/loops. Those were throwing errors at compile time, but unit tests ran fine. I was never able to get browserstack tests to run correctly, so I'm not sure if it really did work in the browser or not. Best of luck! @ me if you want to chat!

    function isBrowser() {         return typeof window === 'object'             && typeof document !== 'undefined'             && typeof HTMLImageElement !== 'undefined'             && typeof HTMLCanvasElement !== 'undefined'             && typeof HTMLVideoElement !== 'undefined'             && typeof ImageData !== 'undefined';     }
This is a horrible function. If someone (me) wants to fake out a library into thinking it's in a browser, don't stifle that person by doing some oddball browser check (this is not how you detect if you're in a browser) and then be really silent and confusing when the library errors. I've been working against an error i thought was in tensorflow for hours only to realize it came from face-api.js code:
Error: getEnv - environment is not defined, check isNodejs() and isBrowser()
Tensorflow already has browser and node checks and it's own idea of environment. Why did you guys reinvent the wheel? :|

I've been working against an error i thought was in tensorflow for hours only to realize it came from face-api.js code

I agree, that the error message might not be the best, but by looking at the stack trace one could have figured where the error message comes from.
The browser check is that complex, because we want to only initialize the corresponding environment of the library in case we are in a valid browser or nodejs environment to avoid errors at runtime. In any other case, it is up to the user to initialize the environment manually. All environment specifics can be monkey patched using faceapi.env.monkeyPatch.

Ok, so I'm posting this update to let everyone in this thread know that it is possible today to fool both tensorflow and face-api.js into running in a web worker and that they will run GPU accelerated, however you shouldn't get your hopes way up for perfectly jank free UX.
In my app, face detection takes approximately 60ms on a MacBook Pro (Retina, 15-inch, Mid 2015), which is only processing 640x480 stills. The stills are transferred to the worker using zero copy, so they avoid the serialize/deserialize and structured copy performance hits.
The app itself is only taking 1-2ms for any given RAF cycle, but visual jank is still occurring on Chrome when the worker thread takes longer than expected. I'm not even seeing any GC issues. The jank appears to happen while processing microtasks. I see bunches of timers being set. I'd have to look further into the face-api.js source to see if it's breaking apart workloads into chunks using 0ms setTimeout calls. If it is, those should be converted to Promises. Allowing the browser to handle batch processing stacks of timeouts will definitely result in slower performance if that's what's happening. Timeouts can take 2-4ms to resolve and Promises are almost immediate. I believe the details of how promise scheduling is done is still on a per browser basis, but if you're in this thread, you're today only interested in the ones that support OffscreenCanvas, and that's Chrome. Chrome handles them async.
Here's the admittedly over engineered code for creating a worker environment that tensorflow and face-api.js will run in:
Parent
`
	var screenCopy = {};
	for(let key in screen){
		screenCopy[key] = +screen[key];
	}
	screenCopy.orientation = {};
	for(let key in screen.orientation){
		if (typeof screen.orientation[key] !== 'function') {
			screenCopy.orientation[key] = screen.orientation[key];
		}
	}

	var visualViewportCopy = {};
	if (typeof window['visualViewport'] !== 'undefined') {
		for(let key in visualViewport){
			if(typeof visualViewport[key] !== 'function') {
				visualViewportCopy[key] = +visualViewport[key];
			}
		}
	}

	var styleMediaCopy = {};
	if (typeof window['styleMedia'] !== 'undefined') {
		for(let key in styleMedia){
			if(typeof styleMedia[key] !== 'function') {
				styleMediaCopy[key] = styleMedia[key];
			}
		}
	}

	let fakeWindow = {};
	Object.getOwnPropertyNames(window).forEach(name => {
		try {
			if (typeof window[name] !== 'function'){
				if (typeof window[name] !== 'object' &&
					name !== 'undefined' &&
					name !== 'NaN' &&
					name !== 'Infinity' &&
					name !== 'event' &&
					name !== 'name'
				) {
					fakeWindow[name] = window[name];
				} else if (name === 'visualViewport') {
					console.log('want this?', name, JSON.parse(JSON.stringify(window[name])));
				} else if (name === 'styleMedia') {
					console.log('want this?', name, JSON.parse(JSON.stringify(window[name])));
				}
			}
		} catch (ex){
			console.log('Access denied for a window property');
		}
	});

	fakeWindow.screen = screenCopy;
	fakeWindow.visualViewport = visualViewportCopy;
	fakeWindow.styleMedia = styleMediaCopy;
	console.log(fakeWindow);

	let fakeDocument = {};
	for(let name in document){
		try {
			if(name === 'all') {
				// o_O
			} else if (typeof document[name] !== 'function' && typeof document[name] !== 'object') {
					fakeDocument[name] = document[name];
			} else if (typeof document[name] === 'object') {
				fakeDocument[name] = null;
			} else if(typeof document[name] === 'function') {
				fakeDocument[name] = { type:'*function*', name: document[name].name };
			}
		} catch (ex){
			console.log('Access denied for a window property');
		}
	}

`
Worker
`
Canvas = HTMLCanvasElement = OffscreenCanvas;
HTMLCanvasElement.name = 'HTMLCanvasElement';
Canvas.name = 'Canvas';
function HTMLImageElement(){}
function HTMLVideoElement(){}
Image = HTMLImageElement;
Video = HTMLVideoElement;
// Canvas.prototype = Object.create(OffscreenCanvas.prototype);
function Storage () {
let _data = {};
this.clear = function(){ return _data = {}; };
this.getItem = function(id){ return _data.hasOwnProperty(id) ? _data[id] : undefined; };
this.removeItem = function(id){ return delete _data[id]; };
this.setItem = function(id, val){ return _data[id] = String(val); };
}
class Document extends EventTarget {}
let window, document = new Document();
		// do terrible things to the worker's global namespace to fool tensorflow
		for (let key in event.data.fakeWindow) {
			if (!self[key]) {
				self[key] = event.data.fakeWindow[key];
			}
		}
		window = Window = self;
		localStorage = new Storage();
		console.log('*faked* Window object for the worker', window);

		for (let key in event.data.fakeDocument) {
			if (document[key]) { continue; }

			let d = event.data.fakeDocument[key];
			// request to create a fake function (instead of doing a proxy trap, fake better)
			if (d && d.type && d.type === '*function*') {
				document[key] = function(){ console.log('FAKE instance', key, 'type', document[key].name, '(',document[key].arguments,')'); };
				document[key].name = d.name;
			} else {
				document[key] = d;
			}
		}
		console.log('*faked* Document object for the worker', document);

		function createElement(element) {
			// console.log('FAKE ELELEMT instance', createElement, 'type', createElement, '(', createElement.arguments, ')');
			switch(element) {
				case 'canvas':
					// console.log('creating canvas');
					let canvas = new Canvas(1,1);
					canvas.localName = 'canvas';
					canvas.nodeName = 'CANVAS';
					canvas.tagName = 'CANVAS';
					canvas.nodeType = 1;
					canvas.innerHTML = '';
					canvas.remove = () => { console.log('nope'); };
					// console.log('returning canvas', canvas);
					return canvas;
				default:
					console.log('arg', element);
					break;
			}
		}

		document.createElement = createElement;
		document.location = self.location;
		console.log('*faked* Document object for the worker', document);

`

Here's what I'm seeing btw. I'm going to try your suggestion of looking at using a different model that might process faster.

Check this out. This is what I'm talking about when I'm making this correlation. When timers are used in bulk, they appear to mess up the process scheduling by spamming the event loop. Promises don't appear to have the same problem. Chrome bundles them up nicely and still has the ability to handle requestAnimationFrame requests. I'd like to see if there's a way in face-api.js to fix the workload splitting so it doesn't rely on setTimeout

I'd like to see if there's a way in face-api.js to fix the workload splitting so it doesn't rely on setTimeout

Hmm, actually there are no calls to setTimeout, tf.nextFrame or requestAnimationFrame in face-api.js. Could it be, that the async behaviour you are encountering here is due to downloading data from the GPU via tf.data()?

ok, possibly disproved the timer spam theory. I'm now pointing to the GPU work. While I was overwriting everything sacred (window, document, etc) I rewrote the setTimeout function so it uses promises and request animation frame for 0 ms setTimeouts, and falls back to setInterval for actual timers. It worked exactly how I anticipated it would, except that the jank is still present and the only thing left to point a finger at is the GPU load that's 2 frames long. 👎
So, for everyone watching, probably keep your GPU load in mind. It can block things just like anything else.

Timeout replacement code
`
// More really bad practices to fix closed libraries. Here we overload setTimeout to replace it with a flawed promise implementation which sometimes cant be canceled.
let callStackCount = 0;
const maxiumCallStackSize = 750; // chrome specific 10402, of 774 in my tests
setTimeout = function (timerHandler, timeout) {
let args = Array.prototype.slice.call(arguments);
args = args.length <3 ? [] : args.slice(2, args.length);
if (timeout === 0) {
if (callStackCount < maxiumCallStackSize) {
var cancelator = {cancelable: false };
callStackCount++;
new Promise(resolve=>{
resolve(timerHandler.apply(self, args));
});
return cancelator;
} else {
requestAnimationFrame(()=>{
timerHandler.apply(self, args);
});
callStackCount = 0;
return;
}
}
const i = setInterval(()=>{
clearInterval(i);
timerHandler.apply(self, args);
}, timeout);
return i;
};
clearTimeout = (id)=>{ console.log(id); if(id && id.cancelable === false) { console.error('woops. cant cancel a 0ms timeout anymore! already ran it'); } else { clearInterval(id);} };
// var x = setTimeout((x,y,z)=>{console.log(x,y,z);}, 0, 'hello', 'im', 'cassius');
// var y = setTimeout((x,y,z)=>{console.log(x,y,z);}, 1000, 'hello', 'im', 'cassius');
// clearTimeout(x);
// clearTimeout(y);
`

Is there any other "cleaner" way to do this ?
@justadudewhohacks you mentioned the faceapi.env.monkeyPatch but does it work exactly ?
I mean lets say I have a main.js that only do this:
const worker = new Worker('worker.js');

worker.postMessage('foo');

and a worker where I want to be able to do this:
import * as faceapi from 'face-api.js';

faceapi.loadFaceExpressionMode('assets/models/');

onmessage = function(event) {
  console.log(event);
}

where and how should I use the .env.monkeyPatch ?
The error raised atm is the following:Uncaught (in promise) Error: getEnv - environment is not defined, check isNodejs() and isBrowser().

@maximeparisse you would monkey patch environment specific after importing the package. In the nodejs examples we monkey patch Canvas, Image and ImageData for example, as shown here.
Refer to the Environment type to see what can be overridden.

@justadudewhohacks : Thank you for your reply, i will give a shot and give my feedbacks here in case that can help others.

@justadudewhohacks : I've tried without success to do that in a web worker. I understand how you patched the env spec in nodejs but i can't see how i can reproduce it for a web worker.

@maximeparisse face-api.js only looks for those native methods as a node serve vs browser check, per my example above. There are also tfjs detections. You have to set those values inside the worker before loading the libraries in order to fool the libraries into believing they're in a browser. If they fall into the node detection block, they will fail that check too, then bail out to a null result rather than a default (browser). A good patch to apply to face-api.js would be to add worker cases and change the detection to if/elseif/elseif/else style blocks so there is always a default case and more reasonable fallbacks. This is doable, but the trick is in supporting workers for browsers other than Chrome or Firefox with the flag set to enable OffscreenCanvas.

Anyone manage to get this working?

Yes. Turned out the integrated gpu was the biggest insurmountable bottleneck to avoiding rendering pipeline. I’d be willing to revisit this once tensorflow and this lib have been updated to allow for offscreencanvas support, which is necessary to avoid excessive monkey patching and environmental fake outs to the two libs.

Also: https://caniuse.com/#feat=offscreencanvas

@jeffreytgilbert It appears TensorFlow.js now supports Offscreen Canvas... At least according to this article: https://medium.com/@wl1508/webworker-in-tensorflowjs-49a306ed60aa - does that jive with what you're seeing? Should face-api/tfjs "just work" in WebWorkers now...?

I can sadly confirm that face-api does not Just Work, even with monkeyPatch.
When I do the following in my worker:
import * as faceapi from 'face-api.js';

faceapi.env.monkeyPatch({ Canvas: OffscreenCanvas })

I get:
Uncaught Error: monkeyPatch - environment is not defined, check isNodejs() and isBrowser()
    at Object.monkeyPatch (index.ts:38)

I've checked the isBrowser module, and I've done a TON of monkey patching of my own BEFORE calling monekyPatch(), and got the following checks to pass in my own code:
// isBrowserCheck is true in my tests
const isBrowserCheck = typeof window === 'object'
&& typeof document !== 'undefined'
&& typeof HTMLImageElement !== 'undefined'
&& typeof HTMLCanvasElement !== 'undefined'
&& typeof HTMLVideoElement !== 'undefined'
&& typeof ImageData !== 'undefined'
&& typeof CanvasRenderingContext2D !== 'undefined';

My own monkey patching is based on @jeffreytgilbert 's example above,  with a few edits to make it compile, and I added CanvasRenderingContext2D = OffscreenCanvasRenderingContext2D;.
Bottom line: faceapi.env.monkeyPatch does not even try to monkey patch because of the error above.
Anyone have any suggestions on how to get this to even work? GPU or no GPU, I just want to try to get it to work. (Chrome 79 on a brand new MacBook Pro 15", so yes, OffscreenCanvas is supported.)

Update: Got it working.
How? Use this gist: https://gist.github.com/josiahbryan/770ca1a9d72f1b35c13219ba84dc0495
Import it into your worker. If you have a bundler setup for your worker, just do (assuming you put it in your utils/ folder):
import './utils/faceEnvWorkerPatch';

Don't need to call faceapi's monkeyPatch if you use that.
Fair warning: That gist is NOT pretty. It is a conglomeration of hacks and workarounds and whatever else. But it works. Face detection is working for me now in a web worker.
Ideally, face-api would support a WebWorker WITHOUT having to do that horrendous hack of a monkey patch I just uploaded, but, yeah. At least this works now.

I found my own way of monkey patching. Its pretty simple but only supports OffscreenCanvas,
faceapi.env.setEnv(faceapi.env.createNodejsEnv());

faceapi.env.monkeyPatch({
    Canvas: OffscreenCanvas,
    createCanvasElement: () => {
        return new OffscreenCanvas(480, 270);
    },
});

No need to import canvas, supports OffscreenCanvas rigidly, and seems this is the easiest valid way.

How would you monkeyPatch the HTMLImageElement in webworkers?
In the Node.js example we are using an external library.
// implements nodejs wrappers for HTMLCanvasElement, HTMLImageElement, ImageData
import * as canvas from 'canvas';

// patch nodejs environment, we need to provide an implementation of
// HTMLCanvasElement and HTMLImageElement, additionally an implementation
// of ImageData is required, in case you want to use the MTCNN
const { Canvas, Image, ImageData } = canvas;
faceapi.env.monkeyPatch({ Canvas, Image, ImageData});

Is there anything similar that can be used for webworkers.
E.g. these are the example steps that are called in the web worker

load the image using fetch -> returns a blob

function loadImage(url) {
    let p = new Promise( (resolve) => {

      fetch(url)
      .then( (response) => {
        if (!response.ok) {
          throw Error(response.statusText);
        }
        return response.blob();
      })
      .then( (blob) => {
        resolve(blob);
      })
    });

    return p;
  }

Create Image from blob

???

Use face-api to detect face/faces

let detections = await faceapi
          .detectSingleFace(image) // not sure how to get image in webworker
          .withFaceLandmarks()
          .withFaceDescriptor();

@ivanbacher I know I can use canvas in WebWorker, and faceapi can convert ImageData to a canvas for you in WebWorker - so what about rendering your Image to a Canvas in the main thread, then using the canvas getImageData function to grab the bytes, and then send it to the WebWorker in a message (transfer the buffer), then make ImageData out of the buffer, and use that ImageData with faceapi? I know that route does work, I do it even today in my own code (though the canvas gets video in the main thread instead of an image, but you can easily render an image to the main thread canvas and it will be the same.)
Let me know if you have questions on that route...

Thanks for getting back to me. The main point of using a frame image as an
htmlimageelement or blob is that I don't want to use the offscreen canvas.
I want to be able to keep render frames in real time on the main thread. I
do that with a context and you can't transfer control when your canvas has
a context.
On Tue, 25 Feb 2020, 11:06 Josiah Bryan<EMAIL_ADDRESS>wrote:

@ivanbacher https://github.com/ivanbacher I know I can use canvas in
WebWorker, and faceapi can convert ImageData to a canvas for you in
WebWorker - so what about rendering your Image to a Canvas in the main
thread, then using the canvas getImageData function to grab the bytes, and
then send it to the WebWorker in a message (transfer the buffer), then make
ImageData out of the buffer, and use that ImageData with faceapi? I know
that route does work, I do it even today in my own code (though the canvas
gets video in the main thread instead of an image, but you can easily
render an image to the main thread canvas and it will be the same.)
Let me know if you have questions on that route...
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/justadudewhohacks/face-api.js/issues/47?email_source=notifications&email_token=AIXHTMVWEHBFX7S5RELJA2DRETNQLA5CNFSM4FJTKRRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM3EMMI#issuecomment-590759473,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIXHTMQ2Y3GB4JFCPZSEDSDRETNQLANCNFSM4FJTKRRA
.

@ivanbacher do you have an example for that scenario (main->image->worker->face-api->json->main). I got stuck with face-api and WebWorker and there is simply no error and nothing I could find.
I think I mess up some part I simply don't understand in the first place.

@josiahbryan thanks for the pointer. Makes a lot of sense.
@muebau sure. Look at the code below. when implementing you have to take a couple of things into consideration: control flow, face-api setup, etc. But I think it gets the point across. Hope this helps
Main Thread
function getImage() {
    let p = new Promise( (resolve) => {
        let img = new Image();
        img.onload = ()=>{ resolve(img); }
        img.src = url
   });
  return p;
}

getImage().then( (img) => {
    let canvas = new OffscreenCanvas(img.width, img.height); //can also be normal canvas
    let context = canvas.getContext('2d');
    context.drawImage(img, 0, 0, img.width, img.height);

   let imgData = context.getImageData(0, 0, img.width, img.height);

    worker.postMessage( {
        type:'send-img-data',
        w: img.width,
        h: img.height,
        buffer: imgData.data.buffer
    }, [imgData.data.buffer] );
});

In Worker
onmessage = (e) => {
    let buffer = e.data.buffer;
    let w = e.data.w;
    let h = e.data.h;

    let canvas = new OffscreenCanvas(w, h);
    let context = canvas.getContext('2d');
    let imageData = context.createImageData(w, h);

    let arr = new Uint8ClampedArray(data.buffer);
    imageData.data.set(arr);

    context.putImageData(imageData, 0, 0);

     //now the image should be drawn to the offscreencanvas

    //you can use offscreen canvas as input to faceapi methods e.g.

    let detections = await faceapi.detectSingleFace(canvas)
            .withFaceLandmarks().withFaceDescriptor();
}

@ivanbacher When I pass an OffscreenCanvas instance to faceapi.detectSingleFace I get:
Uncaught (in promise) Error: toNetInput - expected media to be of type HTMLImageElement | HTMLVideoElement | HTMLCanvasElement | tf.Tensor3D, or to be an element id
    at toNetInput.js:36
    at Array.forEach (<anonymous>)
    at toNetInput.js:31
    at step (tslib.es6.js:99)
    at Object.next (tslib.es6.js:80)
    at tslib.es6.js:73
    at new Promise (<anonymous>)
    at __awaiter (tslib.es6.js:69)
    at toNetInput (toNetInput.js:15)
    at TinyFaceDetector.<anonymous> (TinyYolov2Base.js:123)

Is this a limitation with TinyFaceDetector specifically?

@aendrew could be, I haven't actually tried it with TinyFaceDetector

@ivanbacher Drat, you were too quick! 😅  I deleted my comment because I realised I forgot to monkeypatch. Your workflow seems to work fine. 👍

Update: Got it working.
How? Use this gist: https://gist.github.com/josiahbryan/770ca1a9d72f1b35c13219ba84dc0495
Import it into your worker. If you have a bundler setup for your worker, just do (assuming you put it in your utils/ folder):
import './utils/faceEnvWorkerPatch';

Don't need to call faceapi's monkeyPatch if you use that.
Fair warning: That gist is NOT pretty. It is a conglomeration of hacks and workarounds and whatever else. But it works. Face detection is working for me now in a web worker.
Ideally, face-api would support a WebWorker WITHOUT having to do that horrendous hack of a monkey patch I just uploaded, but, yeah. At least this works now.

@josiahbryan
I used your patch and it does work locally but when i build the project, the imported patch is not accepted and I get "Uncaught ReferenceError: HTMLCanvasElement is not defined"
I am using React.JS with react-rewired and worker-loader.
Can you or anyone else help me with this ?

Sounds like your prod build is running extra linter checks that your local dev build is not doing.
Without being able to recreate your compile error here (as it works fine for me right now...), I would start by trying to change lines 4 in the faceEnvWorkerPatch FROM:
Canvas = HTMLCanvasElement = OffscreenCanvas;

TO
self.Canvas = self.HTMLCanvasElement = OffscreenCanvas;

I'm guessing that's the line that it doesn't like, but you would have to say for sure since there was no line number in your message.
The idea here is that yeah, HTMLCanvasElement of COURSE is not defined - duh. We're trying to define it lol! Since the global context object in the worker is self, we (theoretically) can avoid your linter error by telling the compiler we're defining a new property on the self object instead of relying on the linter to realize that's what we're trying to do.
Anyway, yeah, see if that works. If not, looks at the line # where it's complaining about being undefined and see if you can fix it with those hints above, or ping back here with more details, or maybe somebody else knows better haha. Good luck!! 😄

Sounds like your prod build is running extra linter checks that your local dev build is not doing.
Without being able to recreate your compile error here (as it works fine for me right now...), I would start by trying to change lines 4 in the faceEnvWorkerPatch FROM:
Canvas = HTMLCanvasElement = OffscreenCanvas;

TO
self.Canvas = self.HTMLCanvasElement = OffscreenCanvas;

I'm guessing that's the line that it doesn't like, but you would have to say for sure since there was no line number in your message.
The idea here is that yeah, HTMLCanvasElement of COURSE is not defined - duh. We're trying to define it lol! Since the global context object in the worker is self, we (theoretically) can avoid your linter error by telling the compiler we're defining a new property on the self object instead of relying on the linter to realize that's what we're trying to do.
Anyway, yeah, see if that works. If not, looks at the line # where it's complaining about being undefined and see if you can fix it with those hints above, or ping back here with more details, or maybe somebody else knows better haha. Good luck!! 😄

Thank you for your answer ! it worked with adding "self." ! and yes it was a linting problem when building !

I found my own way of monkey patching. Pretty simple but only supports OffscreenCanvas,
faceapi.env.setEnv(faceapi.env.createNodejsEnv());

faceapi.env.monkeyPatch({
    Canvas: OffscreenCanvas,
    createCanvasElement: () => {
        return new OffscreenCanvas(480, 270);
    },
});

No need to import canvas, supports OffscreenCanvas rigidly, and seems this is the easiest valid way.

Thank you works flawlessly

@josiahbryan thanks for the pointer. Makes a lot of sense.
@muebau sure. Look at the code below. when implementing you have to take a couple of things into consideration: control flow, face-api setup, etc. But I think it gets the point across. Hope this helps
Main Thread
function getImage() {
    let p = new Promise( (resolve) => {
        let img = new Image();
        img.onload = ()=>{ resolve(img); }
        img.src = url
   });
  return p;
}

getImage().then( (img) => {
    let canvas = new OffscreenCanvas(img.width, img.height); //can also be normal canvas
    let context = canvas.getContext('2d');
    context.drawImage(img, 0, 0, img.width, img.height);

   let imgData = context.getImageData(0, 0, img.width, img.height);

    worker.postMessage( {
        type:'send-img-data',
        w: img.width,
        h: img.height,
        buffer: imgData.data.buffer
    }, [imgData.data.buffer] );
});

In Worker
onmessage = (e) => {
    let buffer = e.data.buffer;
    let w = e.data.w;
    let h = e.data.h;

    let canvas = new OffscreenCanvas(w, h);
    let context = canvas.getContext('2d');
    let imageData = context.createImageData(w, h);

    let arr = new Uint8ClampedArray(data.buffer);
    imageData.data.set(arr);

    context.putImageData(imageData, 0, 0);

     //now the image should be drawn to the offscreencanvas

    //you can use offscreen canvas as input to faceapi methods e.g.

    let detections = await faceapi.detectSingleFace(canvas)
            .withFaceLandmarks().withFaceDescriptor();
}

Thanks for this as well, I do have a couple of questions though.

After i get the detections how do i display it on a canvas?
I tried passing the detections obj via postmessage but after receiving it on the main thread, it's broken maybe because of the serialization process. So calling
faceapi.draw.drawDetections(canvas, detections);
won't work. How should i go about this?

Thank you!

Hey about your questions on displaying it on a canvas, you actually can
just render to a canvas from INSIDE the web worker and it will
AUTOMATICALLY update the canvas outside the webworker.
How?
Well, in my code, I passed in the canvas from the main thread like this:
const canvasFromMainThread = detectionResultsCanvas.
transferControlToOffscreen();
myWebWorker.postMessage({
type: "setCanvasFromMainThread",
canvasFromMainThread,
}, [ canvasFromMainThread ]);
Notice the array at the end of the "postMessage()" call - that transfers
the the canvas to the web worker thread instead of copying it. (See MDN on
the topic of postMessage and transfer lists:
https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)
In my web worker, I receive the message and store "canvasFromMainThread"
for later.
Then, in my web worker, after getting the detection results from the neural
net, I did the usual thing:
faceapi.draw.drawDetections(canvasFromMainThread, resizedResults.map(res =>
res.detection));
That's all that's needed - the canvas in the main thread will automatically
display the results drawn. You don't have to send the detection results or
the canvas back to the main thread. It all works magically.
On Wed, Apr 22, 2020 at 4:19 AM stephenjason89<EMAIL_ADDRESS>wrote:

I found my own way of monkey patching. Pretty simple but only supports
OffscreenCanvas,
faceapi.env.setEnv(faceapi.env.createNodejsEnv());
faceapi.env.monkeyPatch({
Canvas: OffscreenCanvas,
createCanvasElement: () => {
return new OffscreenCanvas(480, 270);
},
});
No need to import canvas, supports OffscreenCanvas rigidly, and seems this
is the easiest valid way.
Thank you works flawlessly
Main Thread
function getImage() {
let p = new Promise( (resolve) => {
let img = new Image();
img.onload = ()=>{ resolve(img); }
img.src = url
});
return p;
}
getImage().then( (img) => {
let canvas = new OffscreenCanvas(img.width, img.height); //can also be normal canvas
let context = canvas.getContext('2d');
context.drawImage(img, 0, 0, img.width, img.height);
let imgData = context.getImageData(0, 0, img.width, img.height);
worker.postMessage( {
    type:'send-img-data',
    w: img.width,
    h: img.height,
    buffer: imgData.data.buffer
}, [imgData.data.buffer] );

});
In Worker
onmessage = (e) => {
let buffer = e.data.buffer;
let w = e.data.w;
let h = e.data.h;
let canvas = new OffscreenCanvas(w, h);
let context = canvas.getContext('2d');
let imageData = context.createImageData(w, h);

let arr = new Uint8ClampedArray(data.buffer);
imageData.data.set(arr);

context.putImageData(imageData, 0, 0);

 //now the image should be drawn to the offscreencanvas

//you can use offscreen canvas as input to faceapi methods e.g.

let detections = await faceapi.detectSingleFace(canvas)
        .withFaceLandmarks().withFaceDescriptor();

}
Thanks for this as well, I do have a couple of questions though.

After i get the detections how do i display it on a canvas?
I tried passing the detections obj via postmessage but after
receiving it on the main thread, it's broken maybe because of the
serialization process. So calling
faceapi.draw.drawDetections(canvas, detections);
won't work. How should i go about this?

Thank you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-617477547,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEZELHWW2GMB6MRRND7IYTRNZA3XANCNFSM4FJTKRRA
.

--
Josiah Bryan
+972-050-794-0566
www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig
<EMAIL_ADDRESS>

Hey about your questions on displaying it on a canvas, you actually can
just render to a canvas from INSIDE the web worker and it will
AUTOMATICALLY update the canvas outside the webworker.
How?
Well, in my code, I passed in the canvas from the main thread like this:
const canvasFromMainThread = detectionResultsCanvas.transferControlToOffscreen();

myWebWorker.postMessage({
    type: "setCanvasFromMainThread",
    canvasFromMainThread,
}, [ canvasFromMainThread ]);

Notice the array at the end of the postMessage() call - that transfers
the the canvas to the web worker thread instead of copying it. (See MDN on
the topic of postMessage and transfer lists:
https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)
In my web worker, I receive the message and store canvasFromMainThread
for later.
Then, in my web worker, after getting the detection results from the neural
net, I did the usual thing:
const ctx = canvasFromMainThread.getContext('2d');
// Have to clear it because everytime before we render results
ctx.clearRect(0, 0, canvasFromMainThread.width, canvasFromMainThread.height);

faceapi.draw.drawDetections(canvasFromMainThread, resizedResults.map(res =>
res.detection));

That's all that's needed - the canvas in the main thread will automatically
display the results drawn. You don't have to send the detection results or
the canvas back to the main thread. It all works magically.
One tip:
The detectionResultsCanvas that I pass IN to the web worker will NOT have the video/image on it that the web worker is using - it is only for OUTPUT of the results.
So, in my main thread, what I really have is something like this:

CanvasContainer

Canvas 1 - LiveVideoCanvas
Canvas 2 - OverlayCanvas

I use simple CSS to make the LiveVideoCanvas be position: absolute; top: 0; left: 0; z-index: 0, and then the OverlayCanvas is position: absolute; top: 0; left: 0; z-index: 1
Since the OverlayCanvas is positioned on top of the LiveVideoCanvas and it is being cleared every time before detection results and it is transparent, what we end up having is the live video showing underneath the overlay with the live results from the web worker being rendered on top of the live video.
Make sense? Feel free to ping with questions!

Hey about your questions on displaying it on a canvas, you actually can
just render to a canvas from INSIDE the web worker and it will
AUTOMATICALLY update the canvas outside the webworker.
How?
Well, in my code, I passed in the canvas from the main thread like this:
const canvasFromMainThread = detectionResultsCanvas.transferControlToOffscreen();

myWebWorker.postMessage({
    type: "setCanvasFromMainThread",
    canvasFromMainThread,
}, [ canvasFromMainThread ]);

Notice the array at the end of the postMessage() call - that transfers
the the canvas to the web worker thread instead of copying it. (See MDN on
the topic of postMessage and transfer lists:
https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)
In my web worker, I receive the message and store canvasFromMainThread
for later.
Then, in my web worker, after getting the detection results from the neural
net, I did the usual thing:
const ctx = canvasFromMainThread.getContext('2d');
// Have to clear it because everytime before we render results
ctx.clearRect(0, 0, canvasFromMainThread.width, canvasFromMainThread.height);

faceapi.draw.drawDetections(canvasFromMainThread, resizedResults.map(res =>
res.detection));

That's all that's needed - the canvas in the main thread will automatically
display the results drawn. You don't have to send the detection results or
the canvas back to the main thread. It all works magically.
One tip:
The detectionResultsCanvas that I pass IN to the web worker will NOT have the video/image on it that the web worker is using - it is only for OUTPUT of the results.
So, in my main thread, what I really have is something like this:

CanvasContainer

Canvas 1 - LiveVideoCanvas
Canvas 2 - OverlayCanvas

I use simple CSS to make the LiveVideoCanvas be position: absolute; top: 0; left: 0; z-index: 0, and then the OverlayCanvas is position: absolute; top: 0; left: 0; z-index: 1
Since the OverlayCanvas is positioned on top of the LiveVideoCanvas and it is being cleared every time before detection results and it is transparent, what we end up having is the live video showing underneath the overlay with the live results from the web worker being rendered on top of the live video.
Make sense? Feel free to ping with questions!

Thank you for the detailed reply!
I just finished doing everything before i saw your reply and ended up exactly as what you said.
My last question though would be, how did you implement passing the live feed images to your webworker?
What i did was create an ImageBitmap from the live feed and pass it to the worker as a transferable.
on the worker, i ended up creating a new offscreen canvas and using .drawImage(imageBitmap,0 ,0)
Would like to know if there's a better way :)
Thank you!

Not really. I ended up passing individual frames into the worker at about 12fps (I set a timeout).
The core of it looks like in the main thread:
// Use this to grab video frame for sending to web worker
		const faceCaptureCtx = this.faceCaptureCanvas.getContext('2d');
		faceCaptureCtx.drawImage(this.localVideo, 0, 0);
		const imgData = faceCaptureCtx.getImageData(0, 0, this.faceCaptureCanvas.width, this.faceCaptureCanvas.height);

		const { height, width, data } = imgData;
		// Transfer the buffer via a transferlist
		// https://stackoverflow.com/questions/41497124/allowable-format-for-webworker-data-transfer
		ProtoCamCore.worker.postMessage({
			type: "frame",
			height,
			width,
			data,
		}, [ data.buffer ]);

And in the web worker:
// props is the message from the main thread
const imgData = new ImageData(
   new Uint8ClampedArray(props.data),
    props.width,
    props.height
);

// Create a canvas from our rgbaBuffer
const img = faceapi.createCanvasFromMedia(imgData);

const results = await faceapi.detectAllFaces(img, this.faceDetectorOptions)

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps was good enough for my purposes and still real-time enough - and most importantly, it didn't cause the main thread to lag.

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps was good enough for my purposes and still real-time enough - and most importantly, it didn't cause the main thread to lag.

Thank you!  I don't really know if this will improve anything but based on this thread
https://stackoverflow.com/questions/60031536/difference-between-imagebitmap-and-imagedata
imageBitmap is stored directly to the GPU and imgData in the CPU
so this might help a bit in terms of performance
let bmp = offlineCanvas.transferToImageBitmap();
worker.postMessage({
            image: bmp
        }, [bmp]);

then you can get width and height from worker by the image.width and image.height props.
let me know if it improves the performance.
I don't know how to measure it. I just use chrome dev tools light house but can't get an accurate measurement.
Thank you

Appreciate that!
I don't have that project running right now so I can't really measure
performance either - I'm knee deep in another code base on some stuff haha!
But, when I do return to that project, I'll try that out! Good find!
On Fri, Apr 24, 2020 at 7:07 AM stephenjason89<EMAIL_ADDRESS>wrote:

Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps
was good enough for my purposes and still real-time enough - and most
importantly, it didn't cause the main thread to lag.
Thank you! I don't really know if this will improve anything but based on
this thread
https://stackoverflow.com/questions/60031536/difference-between-imagebitmap-and-imagedata
imageBitmap is stored directly to the GPU and imgData in the CPU
so this might help a bit in terms of performance
let bmp = offlineCanvas.transferToImageBitmap();
worker.postMessage({
image: bmp
}, [bmp]);
then you can get width and height from worker by the image.width and
image.height props.
let me know if it improves the performance.
I don't know how to measure it. I just use chrome dev tools light house
but can't get an accurate measurement.
Thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-618790216,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEZELEZX47LU733GFTQNOTROEGB3ANCNFSM4FJTKRRA
.

--
Josiah Bryan
+972-050-794-0566
www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig
<EMAIL_ADDRESS>

Just a quick thank you to all who shared code in this thread.  I'm a beginner, but managed to get everything running in a webworker. Now the interface doesn't completely freeze, which is great. It does feel like it runs a but slower - especially the initial run - but it's a worthwhile tradeoff.

Just a quick thank you to all who shared code in this thread. I'm a beginner, but managed to get everything running in a webworker. Now the interface doesn't completely freeze, which is great. It does feel like it runs a but slower - especially the initial run - but it's a worthwhile tradeoff.

can you share the code? I can't load the models

Calling the following line recursively causes a memory leak and eventually
results in a crash:
const imgData = faceCaptureCtx.getImageData(0, 0,
this.faceCaptureCanvas.width, this.faceCaptureCanvas.height);
It seems that there is a known memory leak issue in calling getImageData on
the context object of a canvas repeatedly in short intervals.
Has anyone experienced this? Any suggestions?
On Fri, Apr 24, 2020 at 3:20 AM Josiah Bryan<EMAIL_ADDRESS>wrote:

Not really. I ended up passing individual frames into the worker at about
12fps (I set a timeout).
The core of it looks like in the main thread:
// Use this to grab video frame for sending to web worker
const faceCaptureCtx = this.faceCaptureCanvas.getContext('2d');
faceCaptureCtx.drawImage(this.localVideo, 0, 0);
const imgData = faceCaptureCtx.getImageData(0, 0, this.faceCaptureCanvas.width, this.faceCaptureCanvas.height);
  const { height, width, data } = imgData;
  // Transfer the buffer via a transferlist
  // https://stackoverflow.com/questions/41497124/allowable-format-for-webworker-data-transfer
  ProtoCamCore.worker.postMessage({
  	type: "frame",
  	height,
  	width,
  	data,
  }, [ data.buffer ]);

And in the web worker:
// props is the message from the main thread
const imgData = new ImageData(
new Uint8ClampedArray(props.data),
props.width,
props.height
);
// Create a canvas from our rgbaBuffer
const img = faceapi.createCanvasFromMedia(imgData);
const results = await faceapi.detectAllFaces(img, this.faceDetectorOptions)
Obviously, I'm leaving out a lot, but hopefully you get the idea. 12fps
was good enough for my purposes and still real-time enough - and most
importantly, it didn't cause the main thread to lag.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-618690317,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFZFNZRVOOQKRE6LF7LOKZTROCZ3XANCNFSM4FJTKRRA
.

Here is my webworker:
importScripts('faceEnvWorkerPatch.js');
importScripts('js/face-api.min.js');

let model;
var detector_options;

let selected_face_detector = 'tiny_face_detector'; //'ssd_mobilenetv1';
let inputSize = 512
let scoreThreshold = 0.5
const minConfidence = 0.05; // expression

var loaded_models = [];
var selected_use = 'face'; // can be changed with a command from the main script.

faceapi_worker_loaded = false;
beauty_worker_loaded = false;
ethnicity_worker_loaded = false;

async function load_mobile_net() {
    try{
        await faceapi.nets.ssdMobilenetv1.load('models/');
        console.log("loaded mobilenet face detection");
		detector_options = getFaceDetectorOptions();
    }
    catch(e){console.log(e);}
}

async function load_tiny_net() {
    try{
        await faceapi.nets.tinyFaceDetector.load('models/');
        console.log("loaded tiny face detection");
		detector_options = getFaceDetectorOptions();
    }
    catch(e){console.log(e);}
	postMessage({'loaded':'tiny'});
}

async function load_age_and_gender() {
    try{
        await faceapi.nets.ageGenderNet.load('models/'); // gender
        console.log("loaded age and gender detection model");
    }
    catch(e){console.log(e);}
}

async function load_landmarks() {
    try{
        await faceapi.loadFaceLandmarkModel('models/'); // landmarks
        console.log("loaded landmark detection model");
    }
    catch(e){console.log(e);}
}

async function load_expression() {
    try{
        await faceapi.loadFaceExpressionModel('models/'); // expression
        console.log("loaded expression detection model");
    }
    catch(e){console.log(e);}
}

async function load_recognition() {
    try{
        await faceapi.loadFaceRecognitionModel('models/') // face vectors
        console.log("loaded face recognition detection model");
    }
    catch(e){console.log(e);}
}

function getCurrentFaceDetectionNet() {
  if (selected_face_detector === SSD_MOBILENETV1) {
    return faceapi.nets.ssdMobilenetv1
  }
  if (selected_face_detector === TINY_FACE_DETECTOR) {
    return faceapi.nets.tinyFaceDetector
  }
}

function getFaceDetectorOptions() {
    //new faceapi.SsdMobilenetv1Options({ minConfidence });
	if(selected_face_detector == 'tiny_face_detector'){
    	return new faceapi.TinyFaceDetectorOptions({ inputSize, scoreThreshold });
	}
	else{
		return new faceapi.SsdMobilenetv1Options({ minConfidence });
	}
}

onmessage = function(incoming) {
 	//console.log('Message received from main script:');
  	//console.log(incoming.data);

  	var return_message = '';

	if(incoming.data.type == 'frame'){
		console.log("worker: incoming frame");
		use(selected_use, incoming.data)
	}
	//else if(Array.isArray(incoming.data)){
	else {

		console.log("worker: incoming init message");

		if(incoming.data.net !== undefined){
			if(incoming.data.net == 'tiny'){
				selected_face_detector = 'tiny_face_detector';
				load_tiny_net();
				console.log("worker: loading tiny face net");
			}
			else if(incoming.data.net == 'mobilenet'){
				selected_face_detector = 'ssd_mobilenetv1';
				load_mobile_net();
				console.log("worker: loading mobile face net");
			}
			//postMessage('net loaded');
		}

		if(incoming.data.load !== undefined){
			incoming.data.load.forEach(function (item, index) {
			  //console.log("loading model:");
			  //console.log(index, item);
			  if(item == 'age_and_gender'){
				  load_age_and_gender();
				  console.log('loading age and gender model');
			  }
			  if(item == 'expression'){
				  load_expression();
				  console.log('loading expression model');
			  }
			  if(item == 'recognition'){
				  load_recognition();
				  console.log('loading face recognition model');
			  }
			  if(item == 'landmarks'){
				  load_landmarks();
				  console.log('loading landmarks model');
			  }
			  //console.log('loading ' + item);
			  //postMessage("loaded");
			});
		}

		if(incoming.data.use !== undefined){
			selected_use = incoming.data.use;
			console.log("selected_use is now set to " + selected_use);

		}

	}
}

async function use(use_case, image_data){

	console.log("in worker USE. use_case = " + use_case + ", imagedata:");
	console.log(image_data);
	// props is the message from the main thread
	const imgData = new ImageData(
	   new Uint8ClampedArray(image_data.data),
	    image_data.width,
	    image_data.height
	);

	console.log(imgData);

	// Create a canvas from our rgbaBuffer
	const img = faceapi.createCanvasFromMedia(imgData);

	//const results = await faceapi.detectAllFaces(img, this.faceDetectorOptions)

	var result = {};
	if(use_case == 'face'){
		result = await faceapi.detectSingleFace(img, detector_options);
	}
	else if(use_case == 'landmarks'){
		result = await faceapi.detectSingleFace(img, detector_options).withFaceLandmarks();

		try {
			const jawOutline = result.landmarks.getJawOutline();
			const nose = result.landmarks.getNose();
			const mouth = result.landmarks.getMouth();
			const leftEye = result.landmarks.getLeftEye();
			const rightEye = result.landmarks.getRightEye();
			const leftEyeBrow = result.landmarks.getLeftEyeBrow();
			const rightEyeBrow = result.landmarks.getRightEyeBrow();
			postMessage({
				"landmarks":result.landmarks.positions,
				"jawOutline":jawOutline,
				"nose":nose,
				"mouth":mouth,
				"leftEye":leftEye,
				"rightEye":rightEye,
				"leftEyeBrow":leftEyeBrow,
				"rightEyeBrow":rightEyeBrow,
				"faceparts":true
			});
		}
		catch (e) {
			postMessage({"faceparts":false});
		}

	}
	else if(use_case == 'age_and_gender'){
		result = await faceapi.detectSingleFace(img, detector_options).withFaceLandmarks().withAgeAndGender();
	}
	else if(use_case == 'expression'){
		result = await faceapi.detectSingleFace(img, detector_options).withFaceLandmarks().withFaceExpressions()
	}
	else if(use_case == 'recognition'){
		result = await faceapi.detectSingleFace(img, detector_options).withFaceLandmarks().withFaceDescriptor();
	}
	else if(use_case == 'all'){
		result = await faceapi.detectSingleFace(img, detector_options).withFaceLandmarks().withFaceExpressions().withAgeAndGender().withFaceDescriptor();
		console.log("use case all result was: ");
		console.log(result);
	}
	else{
		console.log("Error: no fitting use case provided");
		postMessage({"error":"selected use does not exist"});
	}

	if( result === undefined){
		result = {'error':'result undefined'};
	}
	postMessage(result);
}

That file also loads second patch file called faceEnvWorkerPatch.js to make faceapijs work as a webworker. It contains:
// From: https://github.com/justadudewhohacks/face-api.js/issues/47

self.Canvas = self.HTMLCanvasElement = OffscreenCanvas;
self.HTMLCanvasElement.name = 'HTMLCanvasElement';
self.Canvas.name = 'Canvas';

self.CanvasRenderingContext2D = OffscreenCanvasRenderingContext2D;

function HTMLImageElement(){}
function HTMLVideoElement(){}

self.Image = HTMLImageElement;
self.Video = HTMLVideoElement;

// Canvas.prototype = Object.create(OffscreenCanvas.prototype);

function Storage () {
	let _data = {};
	this.clear = function(){ return _data = {}; };
	this.getItem = function(id){ return _data.hasOwnProperty(id) ? _data[id] : undefined; };
	this.removeItem = function(id){ return delete _data[id]; };
	this.setItem = function(id, val){ return _data[id] = String(val); };
}
class Document extends EventTarget {}

let window, document = new Document();

window = self.Window = self;
self.localStorage = new Storage();

function createElement(element) {
	// console.log('FAKE ELELEMT instance', createElement, 'type', createElement, '(', createElement.arguments, ')');
	switch(element) {
		case 'canvas':
			// console.log('creating canvas');
			let canvas = new Canvas(1,1);
			canvas.localName = 'canvas';
			canvas.nodeName = 'CANVAS';
			canvas.tagName = 'CANVAS';
			canvas.nodeType = 1;
			canvas.innerHTML = '';
			canvas.remove = () => { console.log('nope'); };
			// console.log('returning canvas', canvas);
			return canvas;
		default:
			console.log('arg', element);
			break;
	}
}

document.createElement = createElement;
document.location = self.location;
//console.log('*faked* Document object for the worker', document);

if(!typeof window == 'object') {
	console.warn("Check failed: window");
}
if(typeof document === 'undefined') {
	console.warn("Check failed: document");
}
if(typeof HTMLImageElement === 'undefined') {
	console.warn("Check failed: HTMLImageElement");
}
if(typeof HTMLCanvasElement === 'undefined') {
	console.warn("Check failed: HTMLCanvasElement");
}
if(typeof HTMLVideoElement === 'undefined') {
	console.warn("Check failed: HTMLVideoElement");
}
if(typeof ImageData === 'undefined') {
	console.warn("Check failed: ImageData");
}
if(typeof CanvasRenderingContext2D === 'undefined') {
	console.warn("Check failed: CanvasRenderingContext2D");
}

self.window = window;
self.document = document;
self.HTMLImageElement = HTMLImageElement;
self.HTMLVideoElement = HTMLVideoElement;

const isBrowserCheck = typeof window === 'object'
&& typeof document !== 'undefined'
&& typeof HTMLImageElement !== 'undefined'
&& typeof HTMLCanvasElement !== 'undefined'
&& typeof HTMLVideoElement !== 'undefined'
&& typeof ImageData !== 'undefined'
&& typeof CanvasRenderingContext2D !== 'undefined';
;
// console.warn("++ Check passed locally:", isBrowserCheck);
if(!isBrowserCheck) {
	throw new Error("Failed to monkey patch for face-api, face-api will fail");
}

The webworker is interacted with from the main script. Here is some of that code:
// If the browser supports web workers...
if (typeof(Worker) !== "undefined") {

    window.faceapi_worker = new Worker('faceapi-worker.js');
    window.faceapi_worker.onmessage = function(incoming) {

        console.log('Message received from faceapi worker .data: ' + JSON.stringify(incoming.data,null,2));

        if( 'data' in incoming ){

            if(incoming.data === undefined){
                console.log("incoming.data was undefined?")
            }
            else if('loaded' in incoming.data ){
                console.log("incoming message that ML was loaded");
            }
            else if('detection' in incoming.data ){
                console.log(incoming.data); // Do something with the data here.
                const face_box = incoming.data.detection['_box']; // example
            }
            else{
                console.log("No detection data in incoming data..");
            }

        }
        else{
            console.log("undefined result from worker.");
        }

    }
    window.faceapi_worker.postMessage({'net':'tiny','load':['landmarks','expression','age_and_gender','recognition'],'use':['all']});

}

And here's the part of the main code that sends a webcam frame to the webworker:
    canvy_context.drawImage(webcam_video,0,0,640,480);
    var imgData2 = canvy_context.getImageData(0, 0, 640, 480);

    console.log(imgData2);

    const { height, width, data } = imgData2;
    console.log("height:" + height);
    // Transfer the buffer via a transferlist. See https://stackoverflow.com/questions/41497124/allowable-format-for-webworker-data-transfer

    window.faceapi_worker.postMessage({
        type: "frame",
        height,
        width,
        data,
    }, [ data.buffer ]);

Hi Guys,
I able to make faceapi working with web worker using offscreencanvas. My issue now is android webview doesn't support offscreen canvas, is there any other workaround ?
Thanks

Hi @rlaurente could you please add a Gist how you did it?
Best,
Stefan

@sprengerst using @flatsiedatsie code above. It works perfectly in browser now but android webview doesn't support OffscreenCanvas. Still finding a way to make it work in capacitor / cordova without switching to crosswalk library.

@rlaurente Please let us know if you find a way. I'm also running into the OffScreenCanvas not working on mobile browsers issue.

@flatsiedatsie
I think the only way for now is using this plugin https://github.com/JaneaSystems/nodejs-mobile lol. Haven't tried yet though. I hope there's a simpler workaround.

Hi !
These lines of monkey patch cause errors on the latest version of Chrome, when used elsewhere than localhost:
self.HTMLCanvasElement.name = 'HTMLCanvasElement';
self.Canvas.name = 'Canvas';

Cannot assign to read only property name of Canvas

As face-api.js works well with OffscreenCanvas,  shouldn't this type be added as a possible source ? (for detectSingleFace() per example)

If you are interested in a PR let me know.

Anyone seeing massive memory leaks in WebWorkers in the latest Chrome using detectAllFaces? I've found detectAllFaces to be leaking memory like a siv - https://github.com/justadudewhohacks/face-api.js/issues/732

Hey about your questions on displaying it on a canvas, you actually can
just render to a canvas from INSIDE the web worker and it will
AUTOMATICALLY update the canvas outside the webworker.
How?
Well, in my code, I passed in the canvas from the main thread like this:
const canvasFromMainThread = detectionResultsCanvas.transferControlToOffscreen();

myWebWorker.postMessage({
    type: "setCanvasFromMainThread",
    canvasFromMainThread,
}, [ canvasFromMainThread ]);

Notice the array at the end of the postMessage() call - that transfers
the the canvas to the web worker thread instead of copying it. (See MDN on
the topic of postMessage and transfer lists:
https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)
In my web worker, I receive the message and store canvasFromMainThread
for later.
Then, in my web worker, after getting the detection results from the neural
net, I did the usual thing:
const ctx = canvasFromMainThread.getContext('2d');
// Have to clear it because everytime before we render results
ctx.clearRect(0, 0, canvasFromMainThread.width, canvasFromMainThread.height);

faceapi.draw.drawDetections(canvasFromMainThread, resizedResults.map(res =>
res.detection));

That's all that's needed - the canvas in the main thread will automatically
display the results drawn. You don't have to send the detection results or
the canvas back to the main thread. It all works magically.
One tip:
The detectionResultsCanvas that I pass IN to the web worker will NOT have the video/image on it that the web worker is using - it is only for OUTPUT of the results.
So, in my main thread, what I really have is something like this:

CanvasContainer

Canvas 1 - LiveVideoCanvas
Canvas 2 - OverlayCanvas

I use simple CSS to make the LiveVideoCanvas be position: absolute; top: 0; left: 0; z-index: 0, and then the OverlayCanvas is position: absolute; top: 0; left: 0; z-index: 1
Since the OverlayCanvas is positioned on top of the LiveVideoCanvas and it is being cleared every time before detection results and it is transparent, what we end up having is the live video showing underneath the overlay with the live results from the web worker being rendered on top of the live video.
Make sense? Feel free to ping with questions!

I am pulling frames off the live video element, drawing them into a canvasVideoBuffer, then passing the imageData via postMessage to my worker. this worker draws the face detection box into self.offScreenCanvasCtx which was set up at the loading of the worker and the models. It's super slow however - the canvasVideoBuffer doesn't seem to keep up with the video feed (both canvas elements are visible/overlaid over the source video). Should I be setting the canvasVideoBuffer display to none or change the frame intervals or something to speed it up? thanks!

Hey @cindyloo - yeah the canvas I use for capture-and-transfer is invisible / hidden. The user's see the live <video> element. I do draw that video element to a canvas then do .getImageData to xfer it to the worker, but the users don't see the canvas, so they don't know how fast/slow I'm drawing. They just see the raw <video> feed, so they feel like it's real time.
The only thing they see is the "overlay" canvas from the worker - and yeah, that is a bit laggy at 12fps (I actually wrote a tuner to automatically move FPS up/down as needed, can share if desired) - but yeah, the overlay can be slow, but IMHO that's okay as long as the video behind it is buttery smooth.
Ping with specific questions for more details, happy to share!

@josiahbryan so your face detection overlay is a bit laggy too? I need it to be as fast to real-time as possible, looking at setting up an interval instead of sending the image to the worker at every render.. any suggestions are welcome!

@cindyloo and anyone else who is interested - I know you didn't ask, but here's the FPS Tuner I use in some of my projects as needed. Using it right now in a bespoke project with face-api and a webworker to tune the FPS up/down as performance allows: https://gist.github.com/josiahbryan/c4716f7c9f051d7c084b1536bc8240a0

I mean, I don't need the faces 100% real time - as long as within a few frames of the actual real time, I'm happy.
But yeah, the bottle neck is the detection, not the sending of frames. I can get 16-20fps when I force it on a MacBook pro with tons of ram and cpu.
I haven't tried this yet with faceapi, but I do it with OpenCV in another worker: Specifically, to improve performance, I resize the video WHEN I DRAW IT into the canvas down to a smaller size:
faceCaptureCtx.drawImage(videoEl, 0, 0, smallWidth, smallHeight);
The small size is calculated from the aspect ratio of the video and downsized to something like 420px x whatever for best results.
I imagine doing something like that before sending the frame to the worker for face-api would improve both the transfer and the detection, but haven't tried it yet myself @cindyloo
Just fair warning tho - might not matter for your application - but the smaller your source frame (e.g. the smaller you resize the video), the worse the detector will be at finding small faces. Will still work fine with faces that fill "a lot" of the frame, but will fail to find smaller faces as you resize smaller. Just FYI

These lines of monkey patch cause errors on the latest version of Chrome, when used elsewhere than localhost:

Hey @remipassmoilesel , you're right, when I revisited this project this year, I did have to update the monkeypatch I wrote to work with lates chrome. I've updated the gist above, but here is the updated monkeypatch script:
https://gist.github.com/josiahbryan/770ca1a9d72f1b35c13219ba84dc0495
Also, unrelated, but for others (@cindyloo or whoever), here's the writeup on FpsTuner I did a while ago on how to use it:
https://github.com/josiahbryan/fps-auto-tuner

Hey @josiahbryan ! What do you think about just using @OffscreenCanvas as an input source ? It works well for me

@remipassmoilesel not sure what you mean...can you be more specific? :-D Like pseudo code example or bulletpoints? Afaik you can't render into the canvas from outside the worker once you xfer a canvas to the worker? Or maybe I'm not understanding what you're saying?

Hi @josiahbryan,
I mean presently this sample code needs a type assertion on input (as TNetInput):
  public async faceDetection(input: HTMLVideoElement | OffscreenCanvas): Promise<FaceDetection | undefined> {
    return faceApi
      .detectSingleFace(input as TNetInput, new faceApi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor()
      .withAgeAndGender();
  }

OffscreenCanvas cannot be used as input according to type definitions in face-api.js/build/commonjs/dom/types.d.ts. But I tried it and it work. I don't know if it is a desirable practice but if it is, it can simplify the use of face-api.js in a worker.

The bottlenecks to making this work without jank are not simply cpu bound. On a machine without a discrete or very new mobile GPU, the GPU will lock up and cause jank on the main thread. See, the GPU is also used to render the main thread, and you can pin it with ease on intel integrated graphics chips that only have like 40ish cores. All the discrete GPUs these days start in the 1000+ cores range, so i think it just gets geeked when it cant handle the same quantity of tasks. I was never able to get an exact answer for just why that is the case, but it's the case. So in your dev tools, you can monitor GPU too. I would watch that one if you see issues on your target browser/feature support level.

I couldn't get close to the speed of this example and as such implemented a webworker. Combined with an upgrade to tfjs and a throttle of the frames sent to the worker, I have a fairly responsive detector w landmarks. Now that I'm realizing we can't use offscreenCanvases on mobile browsers, I'm questioning my decision. Any thoughts/comments?

It's not mobile browsers. It's embedded webviews in apps. Mobile browsers probably support what you need unless the android version is too old.
As some have suggested above, if offscreen canvas is not supported for web workers and gpu accelerated detection, you can alternatively use a more cpu bound brute force technique with opencv. It's a different library built for cpu not gpu acceleration, so it should work. The web supported portions of this back in 2010s when a team at opera was leading the charge on perspective driven face detection and 3d, mimicking the nintendo wii functionality using your face, and orienting the 3d camera and depth based on where you are perceived to be standing from the display. Worked well back then on just cpu, but with much simpler models. Tensorflow is a great library, but if you cant use it because of offscreen canvas support, browser support, gpu support, etc, the thing to do would be go down the cpu accelerated route and simpler models like opencv and webassembly and stuff like that which might be good enough for your use case.

thanks @jeffreytgilbert. The example cited above doesn't appear to work on iphone X chrome (error attached). I have used opencv but not opencv.js. I'll have a look.

thanks @jeffreytgilbert. The example cited above doesn't appear to work on iphone X chrome (error attached). I have used opencv but not opencv.js. I'll have a look.

That's unfortunate. Dumb question, but did you grant access to the microphone and camera for the site? That looks like either the support for getUserMedia isn't there which would be very strange since chrome were the ones to introduce the webrtc apis i believe, or potentially an access restriction issue. You wouldnt be able yo access this from within the webworker, but you should be able to get to it from the main window.

Trying to use anything other than Safari on an iPhone or iPad is not going to work well, in my experience. Apple doesn't give full WebRTC support to those browsers the way it does to Safari.

Trying to use anything other than Safari on an iPhone or iPad is not going to work well, in my experience. Apple doesn't give full WebRTC support to those browsers the way it does to Safari.

I believe they recently discontinued their policy around locking down other browser apps so they have to use the webkit engine. Chrome may still use it. Havent looked to see who swapped out webkit for their own engine.

So, I did this for a project and had it working fairly well in a worker with OffscreenCanvas, but honestly wasn't getting any better perf from it than having it on the main thread, ended up removing it. I think the data transfer between working and main thread might be the bottleneck?

Entirely possible that it is the bottleneck. The only reason I ever did
things in a worker thread was so that video playing in the main UI thread
didn't stutter or be affected. When I tried doing face detection in the
main thread, the video display from the camera was seriously affected, but
when I do face detection in a worker thread, the video display in the main
thread is butter smooth
On Thu, Mar 18, 2021, 9:10 AM Ændrew Rininsland @.***>
wrote:

So, I did this for a project and had it working fairly well in a worker
with OffscreenCanvas, but honestly wasn't getting any better perf from it
than having it on the main thread, ended up removing it. I think the data
transfer between working and main thread might be the bottleneck?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-801959968,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEZELFQ5XZCCRNKFGUK4KTTEICWFANCNFSM4FJTKRRA
.

@josiahbryan Ohhh okay that totally makes sense. I was rendering the video at the same speed as the face-api.js processing so that there wouldn't be lag when drawing stuff on top of the canvas element, but if you're wanting the video to play in realtime that would be beneficial. Cool! 👍
I'm still a bit of a ways off from publishing my project but will post a working example repo once I do, have learned a lot getting this working!

Yeah that makes total sense, I see what you're saying then in that case. In
my case I was capturing from a WebCam and showing it on the screen so
people had to see themselves moving in real time, but then the face
detection would be just at 12 FPS so yeah the face detection would lag a
bit but as long as people could see themselves move in real time early in a
matter in my case.
Good work on getting this far! It is so much fun learning things as we go
isn't it? Good job!
On Thu, Mar 18, 2021 at 9:16 AM Ændrew Rininsland @.***>
wrote:

@josiahbryan https://github.com/josiahbryan Ohhh okay that totally
makes sense. I was rendering the video at the same speed as the face-api.js
processing so that there wouldn't be lag when drawing stuff on top of the
canvas element, but if you're wanting the video to play in realtime that
would be beneficial. Cool! 👍
I'm still a bit of a ways off from publishing my project but will post a
working example repo once I do, have learned a lot getting this working!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-801964027,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEZELFIUKOUCUOCC3AMEWDTEIDKLANCNFSM4FJTKRRA
.

--
Josiah Bryan
Phone/SMS: +1-765-215-0511
WhatsApp: +972-050-794-0566
www.josiahbryan.com https://www.josiahbryan.com/?utm_source=sig
@.***

You definitely don't need to process on every video frame. If you do that, it will choke. Rather than making it arbitrary (like say running it at 12FPS capture) you can have it async so the capture only happens when the detection has finished and the background thread is ready to process a new frame. If this causes too many hiccups in performance on the main thread, it's likely due to the video card not being fast enough to run the ML and also handle the Main UI thread. In Chrome, last I checked, even from a worker thread, the GPU is not isolated from the Main UI thread, so your GPU ends up being a bottleneck. If this is the case you are seeing where GPU (which can be evaluated in dev tools under the performance recorder) is taking too long, you'll need to get a better GPU orrrrr you can simplify your detection model orrrrr pick a simpler one. OffscreenCanvas uses zero copy to move data between parent and worker thread, however I did also notice the capture of the image data from the frame is a non-trivial hit to performance.
So, quick recap of steps to take:

check your GPU performance when it runs that detection to see if it takes longer than 16ms as referenced here: https://github.com/justadudewhohacks/face-api.js/issues/47#issuecomment-486106670
only do detection work when you're not currently already doing detection work (promises help)
don't capture too much data when reading from video if you can avoid it. Instead, wait until your promise resolves from the previous detection and then attempt to read more

If that doesn't work, let us know and I'll see if there's something I might have missed.

Also some food for thought, but if you can't buy a new GPU to run the ML without delays, you can fall back to CPU processing which would be isolated in a web worker on a background thread and won't cause visual jank. The downside on that approach is you will have fewer updates because each check will take longer and you're going to spin the fans on the CPU while it brute forces its way through the work. I don't have an exact answer on how you would force TensorFlow to CPU, but I seem to remember it doing that when it thought the runtime environment was node.js originally before I added the container hacks to fool it into thinking it was in the main UI thread context. I bet it could be done pretty trivially.

Why BodyPix supports ImageData

Since the OverlayCanvas is positioned on top of the LiveVideoCanvas and it is being cleared every time before detection results and it is transparent, what we end up having is the live video showing underneath the overlay with the live results from the web worker being rendered on top of the live video.

is this solution work for 30 FPS video. Currently in my application, I've everything rendering video/canvas, and face detection in main thread. and on video play I'm trying to process each frame. But what end up happening is that I'm only able to process the about 1/10th of all video frames. Which is not I want to implement
so my question is, for 1 minute 30 FPS video (i.e 1800 frames). can I detect all the faces from each and every frames using the Web worker solution?