Support Ruby 3.1

The current stable version of Ruby is 3.1.*
It might be a good idea to create a base image for it

Are there any update for this issue?

I posted a question in re:Post to see if there was any insight on that front.  Adding your vote might help push this issue along.  🤷
https://repost.aws/questions/QUa131-hEISH6QvT7aUMeYtA/when-will-lambda-support-ruby-3-x

Now the current version of Ruby is 3.2.0.
I still don't see any update, AWS docs still refer to 2.7 only, which is now in "security maintenance" only (see https://www.ruby-lang.org/en/downloads/branches/).
AFAIK Ruby 2.7.7 end of life has not been announced yet (https://en.wikipedia.org/wiki/History_of_Ruby#Ruby_2.7), but I found a reference that it will be in 3 months, around March 31, 2023 (https://endoflife.date/ruby)

@awood45 - Curious about your opinion of the future of first party Ruby/Lambda support?
Sorry to ping you, but from the post here it seems like Ruby's got a special place in your heart.
Will AWS continue to support Ruby as a runtime for Lambda? The future doesn't seem very bright in the community. Thanks!

@jessedoyle , I see some recent activity in the web assembly front for Ruby, like here https://rubykaigi.org/2022/presentations/kateinoigakukun.html (the talk includes some lambda examples, but not using AWS).
Right now the only options I see are:

Build your own Ruby runtime (AWS supports it)
Use another lambda provider
Abandon serverless and use a server approach
Please let me know if you know of another option. My company has an app running in AWS lambda 2.7 and I will probably be tasked to do something about it. My inclination is for getting out of lambda, specially as we expect a lot of more traffic hitting that app… I prefer not to rewrite it in another language. Ruby is still an awesome way to go

In case it helps anyone else, I found https://github.com/aws-samples/rails-lambda-handler extremely helpful in setting up a custom Ruby 3.1 container image via CDK.
But yes, the native Ruby support here is really disappointing.

Ruby 2.7 is approaching "End of security maintenance phase" (2023-03-31). Any update on support for Ruby > 3 please?

Hello Everyone!
Recently in my team we were planning to upgrade all our serverless app from ruby 2.7 (native runtime) to 3.2 (using docker container), however we recently received an email from AWS Support saying that they will add native runtime support for Ruby 3.2 at the end of Q2 2023.
In the same way, I want to share with you a simple HelloWorld project using AWS SAM, Rbenv + Ruby 3.2.1 + aws_lambda_ric and Docker, hope it helps to someone that will decide to still use this approach instead to wait the native support
https://github.com/wildomonges/lambda-ruby3.2-container-image
Kind Regards!

@jtuliani - can you confirm what @wildodecisiv is reporting?
Is 3.x coming soon?

We are aware of the up-coming end-of-life of Ruby 2.7 and are working to release a Ruby 3.2 runtime. We expect to publish a container base image before end of March 2023, and the managed runtime within 90 days of the container image release.
To provide existing Ruby customers with time to transition from Ruby 2.7 to Ruby 3.2, we will extend support for the Ruby 2.7 runtime for at least 6 months past the General Availability release of the Ruby 3.2 runtime. For example, if Lambda launches the Ruby 3.2 runtime on May 15, 2023, then we will deprecate the Ruby 2.7 runtime on or after November 15, 2023. During this time, the Ruby 2.7 runtime will continue to receive OS patches and functions using this runtime remain eligible for technical support. For further information, see Lambda runtimes.

Hi @jtuliani, thanks, that's great news! Will the aws_lambda_ric gem get Ruby 3 support as part of this work too, or should those of us using it consider switching to the base image instead?

Update: We have today released a container base image for Ruby 3.2 support in Lambda. This container image is fully supported and suitable for production applications. We will release the corresponding managed runtime within 90 days.
@dancorne Great question. We will update the aws_lambda_ric gem either with or very shortly after the managed runtime release. Until then, we recommend using the newly published container image to build Ruby 3.2 functions for Lambda.

@jtuliani https://gallery.ecr.aws/lambda/ruby Usage doc still refers to the old (2.7) image
not sure if everything else is up to date there

I just noticed Ruby 3.2 images show up here. https://gallery.ecr.aws/lambda/ruby

@ababich Ack, thanks for flagging, we'll address that as we continue to work on the Ruby 3.2 release.


another crash 2

stack-trace.txt
config.txt
log.txt
It's funny that you can see me creating the other two issues I made in the log.
Also, I have no idea what I did what could cause the crash.

Completely broken coredump. Try providing a debug one. (See issue guidelines)

I don't think I will be able to reproduce the bug, since I don't know what could have caused it, should I close this issue then?

yep. Reopen it if it happens again.


Bounty Proposal: Logging Improvements

Describe the background or context
Logging can help us diagnose issues more effectively.
Describe the solution

We should move everything to Cloudflare Logs from Mezmo.
Every log should associate a comment (or event handler) ID. Using the IDs we should aim to be able to:
1. see every log, and
2. backtrace the exact conversation/context from the GitHub conversation
3. compare what happened on the GitHub to the logs.

@0xcodercrane can you please set the time label and approve or adjust this proposal?

Right now we're using mezmo for logging service but how do we do that with Cloudflare logs? Means cloudflare is providing three different ways which one we're going with? LogPush or Log Engine?

If you want to get assigned to the issue, please make a detailed plan and post it here @b4s36t4

@0xcodercrane I wanted to understand which one we're going with LogPush Or LogEngine so that I could spring out my proposal here.

I think instant logs are fine

I don't think this persists the logs.. and integrating cloudflare logs into this application is gonna be a hassle.
It handles logs from websites linked to your account, doesn't really accept a HTTP logs push, retention is bad and it requires business or enterprise account

Since we have a supabase, we can create a logger handler to persist and delete stale logs (if needed) and the duration is under our control, we don't have to pay if we do not have to many logs.
We do not perform read requests, just writes and we have full control to send and persist what we like.. at this point we can easily create something to consume logs from the DB and make it more easier to read and track down problems

@0xcodercrane

/assign

/wallet 0xeFeF11577Db8f882F6Bf92F46008170Ba3602D40

@ByteBallet This is still a proposal, you are supposed to propose a solution not assign yourself

The original spec was to move to Cloudflare logs. but if you think the supabase is better, would you give us pros/cons?

Yes, of course
First of all, Supabase is based on PostgreSQL and Cloudflare is based on SQLite
Supabase provides higher security than Cloudflare by supporting TLS encryption while Cloudflare doesn't have encryption
Also supabase provides stronger monitoring and analytics features like log retention and statistics
Cloudflare provides replication and partitioning
Cloudflare automatically backups the data to R2
Supabase is mostly used for North America and Cloudflare is widely used in Europe
And finally, I am much more familiar with Supabase

I am leaning toward @whilefoo 's solution because the basic features of a monitoring system are search, filter, sort, etc.
we still need to consider if its worth paying right now, Logdna is also good but under free plan, we can only see real-time logs and couldn't invite members to share them.

I looked into Google Cloud Log and it seemed better.

I just dislike the idea of breaking apart our tech stack across more services (more credentials, and more cognitive overhead for documentation around different services)
If complex queries are the hangup for using supabase, why not just generate a few useful queries using ChatGPT and make an API to call those useful ones?

I just removed the bounty for now until we figure out where to store the logging data. I'm really leaning towards using existing infrastructure (we currently do use Cloudflare, and Supabase)

If the only issues for supabase is complexity of SQL queries, I don't think it's problem.
Generating sql queries is not so difficult if logic is clear.

If using existing infra is a must then storing logs in the database and creating a web app to view them is the best way. Even better if such open source solution already exists.

I surfed on some open source projects but couldn't find perfect match. I think the best way is to start with Supabase right now. Here, what I am not sure is "3. compare what happened on the GitHub to the logs". Isn't is same with "2. backtracing"?

Yes seems that I was redundant about 2 and 3!

Yes seems that I was redundant about 2 and 3!
So it seems that the consensus is to go with Supabase. I'm okay with this for now!
However I want to propose a last, but perhaps naive idea: what if we use Cloudflare KV for logs? It's just a key value pair storage mechanism which seems pretty simple to work with conceptually. Maybe with a Unix time stamp as the key and then we dump any logs at that time stamp in the value encoded in an array? It would also be handy to post a link back to the comment on GitHub that the log is associated with (or better if we could generate the URL based on the comment "node id" etc)

I will take a look on it. I once used Cloudflare to save timestamp of uploading files to S3 bucket. But as I know, Cloudflare KV has limitations on the number of requests and storage size.

I don't think using Cloudflare KV makes sense. It's very limited in its design - we can't do filtering, searching, sorting or any sort of complex operations. It seems like you can't even fetch many keys (logs) in one API call but have to do it for every timestamp which means we can't list logs efficiently.
I don't see any real advantage using KV, because we can reference the original comment with Supabase too.

I can't easily turn the checklist into sub tasks from the GitHub iOS app so I will do that shortly

/unassign


add a flag to disable io uring

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (b3e6e7e) 45.26% compared to head (ee48b78) 45.31%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #62      +/-   ##
==========================================
+ Coverage   45.26%   45.31%   +0.05%
==========================================
  Files          54       54
  Lines        3999     3999
  Branches      478      478
==========================================
+ Hits         1810     1812       +2
  Misses       2058     2058
+ Partials      131      129       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add UK L1 hospitalization data source

Small update to add UK L1 hospitalization data source from official UK authority.
Also related to #232.

Thanks @themonk911 for pointing me to the API!
Made the changes and pulled additional data on current_hospitalized and current_ventilator as well.
Tested using python update.py --only hospitalizations --verify simple locally (per the epidemiology README) and the correct output was generated.
The unit tests have been running for quite some time (almost ~5 hours now) however. Hope I didn't break anything...

The unit tests have now run through successfully (I triggered a re-run).

@owahltinez unrelated to this commit, the unit tests are now failing because it looks like the package is failing to build.
  Downloading lxml-4.5.2.tar.gz (4.5 MB)
    ERROR: Command errored out with exit status 1:
     command: /opt/hostedtoolcache/Python/3.9.0/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vd0h4e0b/lxml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vd0h4e0b/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-w7dtmxg1
         cwd: /tmp/pip-install-vd0h4e0b/lxml/
    Complete output (3 lines):
    Building lxml version 4.5.2.
    Building without Cython.
    Error: Please make sure the libxml2 and libxslt development packages are installed.

It sounds like the base Ubuntu image used by GitHub Actions changed. Let's merge this and I'll fix the dependencies in the base image in another commit.
Thanks for this PR!


📘 EPIC: Profile page design

UI/UX design
Implementation

I close as https://github.com/DPGAlliance/digitalpublicgoods-nextjs/issues/9 duplicates


Fix/move group by column to left

Fixes #291

@limonte - doesn't work when selection column is present

@limonte - Breaks sorting by the left most column, as it's applied to group text now?

@limonte - if first column is hidden in grouped by mode it breaks, we need to have logic to update where grouped by value is drawn if first column is hidden

@limonte - the group by value is still inside what ever rendered the first column has, this will cause issue. Just go to custom rendered example, make the icon column the left most, and then try to group by. We need to find way to render these outside the rendered. Could we do this with dom-if restamp that could check if item has _groupId defined? If it is, then normal renderer would be not be used?

I'm closing this in favor of https://github.com/vaadin/px-data-grid/pull/658


ENH: Default section for add_* methods

Description
The add_* methods for Card now have default sections (the one used in the skops template) and no longer set descriptions by default. None is no longer a valid argument for section.
This is useful because it allows us to completely remove all the template checks in the diverse add_* methods. Before this change, many add_* methods had to check the template and if it was a custom template, would raise an error when the section is not indicated.
All methods have been homogenized in that they no longer add a description by default and put a placeholder there if empty. This means that after this PR is merged, the same code will sometimes produce a different looking model card.
Comment
For the Card class, this is a nice simplification of the code. The main change affects the unit tests. All tests that used to raise when using a custom template without a section argument no longer raise but use the default template. Furthermore, since some default descriptions have been removed, other tests had to be touched too. Apart from that, some minor cleanups in the tests.

@skops-dev/maintainers ready for review


Change author of last commit with --amend-author

Hi,
I often find myself forgetting to change the git pair when making commits on a new workstation.  It would be nice to be able to easily change the author pair of the last commit.
This allows you to amend the last commit by using the same initials used to set git pair.
Thanks,
Sai To

Hello @styeung thanks for opening this PR. I'd love to merge this PR, but I come bearing both terrible and good news.
The good news is that this PR is really good! You wrote tests, and the code you wrote is very clear and concise. Thank you for that.
The bad news is that this feature already exists in the git commit command, which you used to implement your feature. You should really just be using git pair-commit --amend --reset-author when you want to do this. I'm not sure that adding a new flag that combines those is a net win, since it increases the surface area of that feature, requires additional documentation, and results in there being TWO ways of doing the same thing.
You should be able to do this today with git pair-commit. I certainly do it all the time and also find it very helpful.
Hope this helps! If you have any questions, I'd be more than happy to answer them.


SUCCESSFULLY ENHANCED THE RATING FORM OF THE SITE

[ ]

[ ] # Related Issue #469 #55

[Cite any related issue(s) this pull request addresses. If none, simply state “None”]
Fixes:  #(#469 .)
Description
UI Overhaul: Redesigned the user interface of the rating form to enhance both functionality and aesthetics.
Input Section Revamp: Modified the input sections, including fields and buttons, for improved usability and a more refined design.
Interactive Elements: Incorporated interactive and visually appealing effects across all sections of the form to elevate the user experience.
Visual Showcase: Added a relevant image to the left side of the rating section to clearly depict the purpose of the form, creating a visually engaging layout.
Button Redesign: Enhanced the "Send" button with a modern, interactive design that makes it more inviting to users.
Star Rating Enhancement: Refined the star rating component, making it more visually attractive and easier to use.
Next-Level User Experience: Overall, these enhancements have taken the user interface of the rating form to a significantly higher level, offering a seamless and enjoyable experience.
SOLVED ISSUE: #469
Type of PR

[  ] Feature enhancement.

Screenshots / videos (if applicable)
### THE UPDATION:
https://github.com/user-attachments/assets/c05c05ac-d0e0-4c90-9d5c-8b0b7e034b96

Checklist:

[ *] I have made this change from my own.

[ * ] I have taken help from some online resources.

[ * ] My code follows the style guidelines of this project.

[ * ] I have performed a self-review of my own code.

[ * ] I have commented my code, particularly in hard-to-understand areas.

[ * ] I have made corresponding changes to the documentation.

[ * ] My changes generate no new warnings.

[ * ] I have tested the changes thoroughly before submitting this pull request.

[ * ] I have provided relevant issue numbers and screenshots after making the changes.

HELLO MAM, I HAD SUCCESSFULLY ENHANCED THE RATING FORM OF THE SITE AND MADE IT VERY INERACTIVE AND ATTRACTIVE .  AND BY MERGIG THIS PR PLEASE CO-OPERATE AND ENCOURAGE ME. ~Thankyou. @PriyaGhosal @PriyaGhosal #merge pr

@dnyanesh99borse Please see the conflicts . Resolve the conflicts . Ensure you pull all the latest changes before pushing your feature/bug fixes to the main branch. After resolving the conflicts, please open a new PR for review.
Thank you!


Helm charts under Apps and MarketPlace section are not showing all versions of the charts which are available in the private chart repository

Rancher is not showing all the helm charts which is available in repository...
Rancher UI charts version drop down screen shot

Private helm repo tied with rancher containing helm charts

Anything with a dash in it is a prerelease and is hidden by default (unless it's the current version).
You can enable seeing them in the user menu (top-right) -> Preferences


Strange identify file format failures

Seeing a couple of strange file format identification failures.
First, claiming a PRONOM id doesn't exist (when it does):
Job: SAACT_2433
Task: 67dbbe03-7f00-444d-a036-5f868bcb44a3

IDCommand UUID: 75290b14-2931-455f-bdde-3b4b3f8b7f15
File: (0b245643-c3d9-4649-8ef2-71218ac2a93c) /var/archivematica/sharedDirectory/watchedDirectories/workFlowDecisions/selectFormatIDToolTransfer/SAACT_2433-c4d39e9e-9d9f-4bb4-98a7-1ef6b153f5bc/objects/Thumbs.db
Command output: fmt/111

Error: No FPR format record found for PUID fmt/111

Second, no idea:
Job: SANCT_2367
Task: 00293fdd-3014-4cad-9451-1b68ac84300a

IDCommand UUID: 75290b14-2931-455f-bdde-3b4b3f8b7f15
File: (fdc7fc77-cf68-4ab8-ae1d-75fb9c100f7b) /var/archivematica/sharedDirectory/watchedDirectories/workFlowDecisions/selectFormatIDToolTransfer/SANCT_2367-06a28446-7530-4f85-b41e-6aa9402aa679/objects/[REDACTED]/conversation_1_17.1.12-edit-copy_data/e00/d00/e0000cea.au

Error: IDCommand with UUID 75290b14-2931-455f-bdde-3b4b3f8b7f15 exited non-zero.

Thumbs.db is being identified by siegfried as PRONOM format fmt/111 which is "OLE2 Compound Document Format"
In the import_pronom_ids command this is intentionally excluded from the fpr database tables with the following comment:

Many Windows file formats that aren't otherwise IDed will be identified as this. We choose not to import this one ID in order to allow Archivematica to recognize those files as unidentified instead of returning something misleading.

It might be worth noting that by default Archivematica ignores/removes Thumbs.db for a non-bagged transfer.

For the second one, running siegfried manually against that file:
matches  :
  - ns      : 'pronom'
    id      : 'UNKNOWN'
    format  :
    version :
    mime    :
    basis   :
    warning : 'no match; possibilities based on extension are x-fmt/139'

The wrapper script around siegfried reports this as a return status of 1. So it's saying that it can't identify that file type.

I've made a fix so that the stderr output from the identification command makes it into the job output.

@jtweed I've identified what's going on here and made the error output visible in the second case. Are you happy with this, or do you think there is work to be done (either now or later on) around the actual identification of those files?

Thanks, happy to close this. Archivematica removing fmt/111 explains it.


Bug in pipe length by year

It appears only "20" without actual value in popup.
https://rural.water-gis.com/#11.89/-1.9535/30.36818

I just checked vectortiles by QGIS, and data is no problem. All of pipe length data is inside vectortiles.

The result of the following SQL is also OK.
https://github.com/WASAC/vt/blob/53d37bac2e4851de588eaabe22deea75a3b1566e/config.js#L491-L527
I am not very sure why all of text can't appear in popup. But I suspect it is matter of mapbox-gl-popup plugin.


Precompilation fails when more than one _ViewImports has the same @namespace directive

From @pranavkm on July 18, 2017 18:13
Consider,
/Views
    /_ViewImports.cshtml (`@namespace WebApp`)
    /_ViewStart.cshtml
/Pages
    /_ViewImports.cshtml (`@namespace WebApp`)
    /_ViewStart.cshtml

Both _ViewStarts would be generated with the same fully qualified type name. This works at runtime because the two types are in distinct assemblies but fails for precompilation because we compile them in to one assembly.
Copied from original issue: aspnet/MvcPrecompilation#174

From @rynowak on August 23, 2017 23:23
Mangling the names would be a good solution, at least for viewstarts.

From @pranavkm on August 23, 2017 21:12
@rynowak do you recall why we chose not to mangle the type name when we added the @namespace feature? Is mangling the type name a possible solution here? The alternative to this would be to mirror what we do at runtime and produce an assembly per view, but that doesn't seem like a very good solution.

From @pranavkm on August 29, 2017 16:3
@ctolkien until we fix this, picking a different namespace is the way to go.

From @ctolkien on August 29, 2017 13:8
Just hit this with a MVC Area with it's own _ViewImports.

I don't think mangling just _ViewStarts.cshtml would help here. Consider this, now that we have areas support:
/Areas/MyArea/Pages/_ViewImports.cshtml @namespace MyApp
/Areas/MyArea/Pages/Index.cshtml @page

/Pages/_ViewImports.cshtml @namespace MyApp
/Pages/Index.cshtml @page

The fix isn't obvious (change one of the namespaces) and introduces friction between runtime and build-time compilation. Not using prettified names seems like the correct fix here.

You might be right, we might have to mangle/generate names across the board.


Increase width of circles by one

This makes the width match the height (and also matches what the original unfilled algorithm would do). It does mean that the circle size is actually r * 2 + 1 though...
Result:

Should fix #31

Oooh excellent stuff, thank you!


Missing data file in lattice vibration example

Hi there, I'm reviewing this repository for JOSS. Everything else looks great to me, but I am unable to run examples/lattice_vibration.py as the file phonopy_mp-2998.yaml.xz is missing, and I couldn't find it in the raw data from http://phonondb.mtl.kyoto-u.ac.jp/ph20180417/d002/mp-2998.html. Is this an oversight, or am I missing something?

@ml-evs Thank you for examining the example script. I have fixed the location for phonopy data phonopy_mp-2998.yaml.xz. Can you check if the issue is solved at the HEAD of develop branch (https://github.com/spglib/spgrep/commit/161fd31ea469702083934cecbcf50fb662d934ab)?
Actually, xz-format phonopy datum are stored in http://phonondb.mtl.kyoto-u.ac.jp/xz-files/ in phonondb.

Thanks, that works for me now! Indeed, I was running the script rather than the Jupyter notebook, which probably always worked.


linkprop + assert_exists in QB-style query gives "more than one row"

On the cards test database:
SELECT User {
  cards := assert_exists((
    WITH
      Q := (
        SELECT User.deck {
          __count := User.deck@count
        }
      )
    SELECT Q {
      single @count := Q.__count
    }
  ))
};

As is typical for most of our "more than one row" type bugs, if you change the single to a multi, the query runs but gives back all of the counts for each user.
This might be (probably is?) the same bug as https://github.com/edgedb/edgedb/issues/3706

Writing this in a non-query builder style also breaks though!
This gives a compiler ISE:
SELECT User {
  cards := assert_exists(.deck {@count})
};

This fails in the same was as the QB query:
SELECT User {
  cards := assert_exists(User.deck {x := User.deck@count})
};

This tells me "unexpected reference to link property 'count' outside of a path expression" (as it tries to do late shape recompilation):
SELECT User {
  cards := assert_exists(User.deck {x := @count})
};


Facebook will not be available on this browser starting October 28, 2024

This is displayed on the mbaisc website of Facebook. Should I be worried?

I think its just some browsers.
https://forums.opera.com/topic/74331/facebook-is-now-blocking-the-opera-browser-from-28-oct-2024

@UlyssesZh this shouldn't be an issue as no browser is specifically used by this library, you can set a different userAgent from the header's dict (check the README)


Ghost of Tsushima - MOD

Game Name
Ghost of Tsushima
Issue Description
I am encountering an error when trying to unpack .psarc files using the UnPSARC tool for the game Ghost of Tsushima.
Data Files List
The list of data files can be found in the following directory structure:

\cache_pc\psarc

Root Folder Files List
The list of files in the root folder is as follows:

Root Folder Files Structure

Error Encountered
The following error occurs during the unpacking process:
UnPSARC (PSARC Archive Tool For Uncharted4 PC)
By NoobInCoding
https://github.com/rm-NoobInCoding/UnPSARC
Unpacking gapack_misc_a.psarc...
Unhandled Exception: System.Exception: Not valid PSARC file! Magic:DSAR
at UnPSARC.PSARC.Read()
at UnPSARC.Archive.Unpack(Stream ArchiveRaw, String Folder)
at UnPSARC.Program.UnpackArchiveFile(String inputPath, String outputDirectory)
at UnPSARC.Program.Main(String[] args)
Additional Information
I am unsure which .dll file is required for using the uncompressed data of this game. Any assistance in resolving this issue would be greatly appreciated.

thanks ❤


Merge pull request #2 from conda-forge/main

Merge pull request #34 from melissaebrooks/vn0.7.11

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

Whenever possible python packages should use pip. See https://conda-forge.org/docs/maintainer/adding_pkgs.html#use-pip


This pull request replaces all instances of java standard lib reflection usages with mirror reflections where applicable. It also provides minor tweaks to code readability by renaming some variables from single letter names to proper words.

You can ignore that if you want. Looks great!


Converting C++ code with MediaPipe library into Web Assembly

I organized my project in the same workspace as MediaPipe and build my project with Bazel. Now I am looking into converting my project into a wasm Web Assembly file so that I can import it on a static website.
I thought about directly converting using Ecmscript but not sure how to efficiently include all the dependencies of MediaPipe present in the MediaPipe's WORKSPACE file.
I am also wondering how to use Ecmscript when building the MediaPipe project using Bazel.
Also, is it better to include MediaPipe as a separate dependency and organize my project from scratch with a separate WORKSPACE file? If so then how to include MediaPipe as a dependency?
Is there any code demo where MediaPipe example is converted to Web Assembly?
I am new to Bazel and Ecmscript so the questions might seem obvious ^_^

I tried to it by getting the .bazelrc and WORKSPACE from this blog: https://hackernoon.com/c-to-webassembly-using-bazel-and-emscripten-4him3ymc. I can produce Web Assembly code for simple cc code with no dependencies.
However, I am stuck with 2 types of errors based on what MediaPipe dependency I include in the BUILD file.
Here is my .bazelrc:
# The bazelrc file for MediaPipe OSS.

# Tensorflow needs remote repo
common --experimental_repo_remote_exec

# Basic build settings
build --jobs 128
build --define='absl=1'
build --enable_platform_specific_config

# Linux
build:linux --cxxopt=-std=c++14
build:linux --host_cxxopt=-std=c++14
build:linux --copt=-w

# emscripten

##### WASM #####
# Use our custom-configured c++ toolchain.
build:wasm --crosstool_top=//javascript/toolchain:emscripten

# Use --cpu as a differentiator.
build:wasm --cpu=wasm

# Use the default C++ toolchain to build the tools used during the build.
build:wasm --host_crosstool_top=@bazel_tools//tools/cpp:toolchain

# These compile flags are active no matter which build mode we are in
# (dbg vs opt). For flags specific to build mode, see cc_toolchain_config.bzl.
#build:wasm --cxxopt="-flto" --copt=-flto
#build:wasm --host_cxxopt="-fno-rtti" # disable generation of info of every class for runtime access
build:wasm --host_cxxopt="-fno-exceptions"
#build:wasm --host_cxxopt="-fomit-frame-pointer"

build:wasm --cxxopt=-std=c++14
build:wasm --host_cxxopt=-std=c++14
#build:wasm --copt=-w

# Disable sandbox environment because emsdk caches files by writing to
# home directory.
build:wasm --spawn_strategy=local

I turned off the optimization flags --cxxopt="-flto", --host_cxxopt="-fomit-frame-pointer", --host_cxxopt="-fno-rtti" on purpose. In fact, --host_cxxopt="-fno-rtti" prevented dynamic type inferencing.
Here is the WORKSPACE:
workspace(name = "practice")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

skylib_version = "0.9.0"
http_archive(
    name = "bazel_skylib",
    type = "tar.gz",
    url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel_skylib-{}.tar.gz".format (skylib_version, skylib_version),
    sha256 = "1dde365491125a3db70731e25658dfdd3bc5dbdfd11b840b3e987ecf043c7ca0",
)
load("@bazel_skylib//lib:versions.bzl", "versions")
versions.check(minimum_bazel_version = "2.0.0")

# ABSL cpp library lts_2020_02_25
http_archive(
    name = "com_google_absl",
    urls = [
        "https://github.com/abseil/abseil-cpp/archive/20200225.tar.gz",
    ],
    # Remove after https://github.com/abseil/abseil-cpp/issues/326 is solved.
    patches = [
        "@//third_party:com_google_absl_f863b622fe13612433fdf43f76547d5edda0c93001.diff"
    ],
    patch_args = [
        "-p1",
    ],
    strip_prefix = "abseil-cpp-20200225",
    sha256 = "728a813291bdec2aa46eab8356ace9f75ac2ed9dfe2df5ab603c4e6c09f1c353"
)

http_archive(
    name = "rules_cc",
    strip_prefix = "rules_cc-master",
    urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
)

# GoogleTest/GoogleMock framework. Used by most unit-tests.
http_archive(
     name = "com_google_googletest",
     urls = ["https://github.com/google/googletest/archive/master.zip"],
     strip_prefix = "googletest-master",
)

# Google Benchmark library.
http_archive(
    name = "com_google_benchmark",
    urls = ["https://github.com/google/benchmark/archive/master.zip"],
    strip_prefix = "benchmark-master",
    build_file = "@//third_party:benchmark.BUILD",
)

# gflags needed by glog
http_archive(
    name = "com_github_gflags_gflags",
    strip_prefix = "gflags-2.2.2",
    sha256 = "19713a36c9f32b33df59d1c79b4958434cb005b5b47dc5400a7a4b078111d9b5",
    url = "https://github.com/gflags/gflags/archive/v2.2.2.zip",
)

# glog v0.3.5
# TODO: Migrate MediaPipe to use com_github_glog_glog on all platforms.
http_archive(
    name = "com_github_glog_glog_v_0_3_5",
    url = "https://github.com/google/glog/archive/v0.3.5.zip",
    sha256 = "267103f8a1e9578978aa1dc256001e6529ef593e5aea38193d31c2872ee025e8",
    strip_prefix = "glog-0.3.5",
    build_file = "@//third_party:glog.BUILD",
    patches = [
        "@//third_party:com_github_glog_glog_9779e5ea6ef59562b030248947f787d1256132ae.diff"
    ],
    patch_args = [
        "-p1",
    ],
)

# 2020-02-16
http_archive(
    name = "com_github_glog_glog",
    strip_prefix = "glog-3ba8976592274bc1f907c402ce22558011d6fc5e",
    sha256 = "feca3c7e29a693cab7887409756d89d342d4a992d54d7c5599bebeae8f7b50be",
    urls = [
        "https://github.com/google/glog/archive/3ba8976592274bc1f907c402ce22558011d6fc5e.zip",
    ],
)

# easyexif
http_archive(
    name = "easyexif",
    url = "https://github.com/mayanklahiri/easyexif/archive/master.zip",
    strip_prefix = "easyexif-master",
    build_file = "@//third_party:easyexif.BUILD",
)

# libyuv
http_archive(
    name = "libyuv",
    urls = ["https://chromium.googlesource.com/libyuv/libyuv/+archive/refs/heads/master.tar.gz"],
    build_file = "@//third_party:libyuv.BUILD",
)

# Note: protobuf-javalite is no longer released as a separate download, it's included in the main Java download.
# ...but the Java download is currently broken, so we use the "source" download.
http_archive(
    name = "com_google_protobuf_javalite",
    sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288ed4380426f9",
    strip_prefix = "protobuf-3.11.4",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz"],
)

http_archive(
    name = "com_google_protobuf",
    sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288ed4380426f9",
    strip_prefix = "protobuf-3.11.4",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz"],
    patches = [
        "@//third_party:com_google_protobuf_fixes.diff"
    ],
    patch_args = [
        "-p1",
    ],
)

http_archive(
    name = "com_google_audio_tools",
    strip_prefix = "multichannel-audio-tools-master",
    urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
)

http_archive(
    name = "ceres_solver",
    url = "https://github.com/ceres-solver/ceres-solver/archive/1.14.0.zip",
    patches = [
        "@//third_party:ceres_solver_compatibility_fixes.diff"
    ],
    patch_args = [
        "-p1",
    ],
    strip_prefix = "ceres-solver-1.14.0",
    sha256 = "5ba6d0db4e784621fda44a50c58bb23b0892684692f0c623e2063f9c19f192f1"
)

new_local_repository(
    name = "linux_opencv",
    build_file = "@//third_party:opencv_linux.BUILD",
    path = "/usr/local",
)

new_local_repository(
    name = "linux_ffmpeg",
    build_file = "@//third_party:ffmpeg_linux.BUILD",
    path = "/usr"
)

# Maven dependencies.

RULES_JVM_EXTERNAL_TAG = "3.2"
RULES_JVM_EXTERNAL_SHA = "82262ff4223c5fda6fb7ff8bd63db8131b51b413d26eb49e3131037e79e324af"

http_archive(
    name = "rules_jvm_external",
    strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
    sha256 = RULES_JVM_EXTERNAL_SHA,
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)

load("@rules_jvm_external//:defs.bzl", "maven_install")

# Needed by TensorFlow
http_archive(
    name = "io_bazel_rules_closure",
    sha256 = "e0a111000aeed2051f29fcc7a3f83be3ad8c6c93c186e64beb1ad313f0c7f9f9",
    strip_prefix = "rules_closure-cf1e44edb908e9616030cc83d085989b8e6cd6df",
    urls = [
        "http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz",
        "https://github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz",  # 2019-04-04
    ],
)

#Tensorflow repo should always go after the other external dependencies.
# 2020-05-11
_TENSORFLOW_GIT_COMMIT = "7c09d15f9fcc14343343c247ebf5b8e0afe3e4aa"
_TENSORFLOW_SHA256= "673d00cbd2676ae43df1993e0d28c10b5ffbe96d9e2ab29f88a77b43c0211299"
http_archive(
    name = "org_tensorflow",
    urls = [
      "https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
      "https://github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
    ],
    patches = [
        "@//third_party:org_tensorflow_compatibility_fixes.diff",
    ],
    patch_args = [
        "-p1",
    ],
    strip_prefix = "tensorflow-%s" % _TENSORFLOW_GIT_COMMIT,
    sha256 = _TENSORFLOW_SHA256,
)

load("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")
tf_workspace(tf_repo_name = "org_tensorflow")

load("//javascript/toolchain:cc_toolchain_config.bzl", "emsdk_configure")
emsdk_configure(name = "emsdk")

In particular, I added the lines in the end:
load("//javascript/toolchain:cc_toolchain_config.bzl", "emsdk_configure")
emsdk_configure(name = "emsdk")

The contents of //javascript/toolchain:cc_toolchain_config.bzl:
load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES")
load(
    "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl",
    "feature",
    "flag_group",
    "flag_set",
    "tool_path",
    "with_feature_set",
)

def _impl(ctx):
    tool_paths = [
        tool_path(
            name = "gcc",
            path = "emcc.sh",
        ),
        tool_path(
            name = "ld",
            path = "emcc.sh",
        ),
        tool_path(
            name = "ar",
            path = "emar.sh",
        ),
        tool_path(
            name = "cpp",
            path = "false.sh",
        ),
        tool_path(
            name = "gcov",
            path = "false.sh",
        ),
        tool_path(
            name = "nm",
            path = "NOT_USED",
        ),
        tool_path(
            name = "objdump",
            path = "false.sh",
        ),
        tool_path(
            name = "strip",
            path = "NOT_USED",
        ),
    ]
    preprocessor_compile_actions = [
        ACTION_NAMES.c_compile,
        ACTION_NAMES.cpp_compile,
        ACTION_NAMES.linkstamp_compile,
        ACTION_NAMES.preprocess_assemble,
        ACTION_NAMES.cpp_header_parsing,
        ACTION_NAMES.cpp_module_compile,
        ACTION_NAMES.clif_match,
    ]

    all_link_actions = [
        ACTION_NAMES.cpp_link_executable,
        ACTION_NAMES.cpp_link_dynamic_library,
        ACTION_NAMES.cpp_link_nodeps_dynamic_library,
    ]

    all_compile_actions = [
        ACTION_NAMES.c_compile,
        ACTION_NAMES.cpp_compile,
        ACTION_NAMES.linkstamp_compile,
        ACTION_NAMES.assemble,
        ACTION_NAMES.preprocess_assemble,
        ACTION_NAMES.cpp_header_parsing,
        ACTION_NAMES.cpp_module_compile,
        ACTION_NAMES.cpp_module_codegen,
        ACTION_NAMES.clif_match,
        ACTION_NAMES.lto_backend,
    ]
    toolchain_include_directories_feature = feature(
        name = "toolchain_include_directories",
        enabled = True,
        flag_sets = [
            flag_set(
                actions = all_compile_actions,
                flag_groups = [
                    flag_group(
                        flags = [
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include/libcxx",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/lib/libcxxabi/include",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include/compat",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include/libc",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/lib/libc/musl/arch/emscripten",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/local/include",
                        ],
                    ),
                ],
            ),
        ],
    )

    crosstool_default_flag_sets = [
        # Optimized (opt)
        flag_set(
            actions = preprocessor_compile_actions,
            flag_groups = [flag_group(flags = ["-DNDEBUG"])],
            with_features = [with_feature_set(features = ["opt"])],
        ),
        # In advanced C++ apps, -O3 can break asmjs.
        flag_set(
            actions = all_compile_actions + all_link_actions,
            flag_groups = [flag_group(flags = ["-g0", "-O3"])],
            with_features = [with_feature_set(features = ["opt"])],
        ),
        # Fastbuild (fastbuild)
        flag_set(
            actions = all_compile_actions + all_link_actions,
            flag_groups = [flag_group(flags = ["-O2"])],
            with_features = [with_feature_set(features = ["fastbuild"])],
        ),
        # Debug (dbg)
        flag_set(
            actions = all_compile_actions + all_link_actions,
            flag_groups = [flag_group(flags = ["-g2", "-O0"])],
            with_features = [with_feature_set(features = ["dbg"])],
        ),
    ]

    features = [
        toolchain_include_directories_feature,
        # These 3 features will be automatically enabled by blaze in the
        # corresponding build mode.
        feature(
            name = "opt",
            provides = ["variant:crosstool_build_mode"],
        ),
        feature(
            name = "dbg",
            provides = ["variant:crosstool_build_mode"],
        ),
        feature(
            name = "fastbuild",
            provides = ["variant:crosstool_build_mode"],
        ),
        feature(
            name = "crosstool_default_flags",
            enabled = True,
            flag_sets = crosstool_default_flag_sets,
        ),
    ]

    return cc_common.create_cc_toolchain_config_info(
        ctx = ctx,
        toolchain_identifier = "wasm-toolchain",
        host_system_name = "i686-unknown-linux-gnu",
        target_system_name = "wasm-unknown-emscripten",
        target_cpu = "wasm",
        target_libc = "unknown",
        compiler = "emscripten",
        abi_version = "unknown",
        abi_libc_version = "unknown",
        tool_paths = tool_paths,
        features = features,
    )

cc_toolchain_config = rule(
    implementation = _impl,
    attrs = {},
    provides = [CcToolchainConfigInfo],
)

def _emsdk_impl(ctx):
    if "EMSDK" not in ctx.os.environ or ctx.os.environ["EMSDK"].strip() == "":
        fail("The environment variable EMSDK is not found. " +
             "Did you run source ./emsdk_env.sh ?")
    path = ctx.os.environ["EMSDK"]
    ctx.symlink(path, "emsdk")
    ctx.file("BUILD", """
filegroup(
    name = "all",
    srcs = glob(["emsdk/**"]),
    visibility = ["//visibility:public"],
)
""")

emsdk_configure = repository_rule(
    implementation = _emsdk_impl,
    local = True,
)

And the dummy cc file:
#include <emscripten/bind.h>
#include <iostream>
#include <sys/syscall.h>

int main() {

    std::cout << "empty inside\n";
}

And the BUILD file used for bazel build:
package(default_visibility = ["//visibility:public"])

DEFAULT_EMSCRIPTEN_LINKOPTS = [
    # "-flto",                        # Specify lto (has to be set on for compiler as well)
    "-O0",
    "--bind",                       # Compiles the source code using the Embind bindings to connect C/C++ and JavaScript
    "--closure 1",                  # Run the closure compiler
    "-s MALLOC=emmalloc",           # Switch to using the much smaller implementation
    "-s ALLOW_MEMORY_GROWTH=1",     # Our example doesn't need memory growth
    "-s USE_PTHREADS=1",            # Disable pthreads
    "-s ASSERTIONS=0",              # Turn off assertions
    "-s EXPORT_ES6=1",              # Export as es6 module, used for rollup
    "-s MODULARIZE=1",              # Allows us to manually invoke the initializatio of wasm
    "-s EXPORT_NAME=createModule",  # Not used, but good to specify
    "-s USE_ES6_IMPORT_META=0",     # Disable loading from import meta since we use rollup
    "-s SINGLE_FILE=1"              # Pack all webassembly into base64
]

WASM_LINKOPTS = [
    "-s WASM=1",                    # Specify wasm output
]

linux_or_darwin_copts = [
    # Disable warnings that exists in glog.
    "-Wno-sign-compare",
    "-Wno-unused-function",
    "-Wno-unused-local-typedefs",
    "-Wno-unused-variable",
    # # Allows src/base/mutex.h to include pthread.h.
    "-DHAVE_PTHREAD",
    # # Allows src/logging.cc to determine the host name.
    "-DHAVE_SYS_UTSNAME_H",
    # For src/utilities.cc.
    "-DHAVE_SYS_SYSCALL_H",
    "-DHAVE_SYS_TIME_H",
    # # Enable dumping stacktrace upon sigaction.
    # "-DHAVE_SIGACTION",
    # # For logging.cc.
    "-DHAVE_PREAD",
    "-DHAVE___ATTRIBUTE__",
]

cc_binary(
    name = "khaliwasm",
    srcs = ["bindings/khali.cpp"],
    linkopts = DEFAULT_EMSCRIPTEN_LINKOPTS + WASM_LINKOPTS + linux_or_darwin_copts,
    deps = [
        "//desktop:main_gpu",
        # "//mediapipe/graphs/hand_tracking:multi_hand_mobile_calculators",
    ],
)

The desktop:main_gpu contains the BUILD contents of the Two Hands GPU example.
The command I used for bazel build:
 bazel build -c opt //javascript:khaliwasm --config=wasm --sandbox_debug --verbose_failures

Now, if I do not include the calculators dependency then I get this main error related to SYS_gettid:
Execution platform: @local_execution_config_platform//:platform
mediapipe/framework/deps/threadpool_pthread_impl.cc:63:64: error: use of undeclared identifier 'SYS_gettid'
      internal::CreateThreadName(thread->name_prefix_, syscall(SYS_gettid));

If I include the calculators dependency then I get the following error:
ERROR: /home/prantoran/.cache/bazel/_bazel_prantoran/53d0985b61cdf3fef140f3aded9a4395/external/cpuinfo/BUILD.bazel:96:11: Configurable attribute "srcs" doesn't match this configuration (would a default condition help?).
Conditions checked:
 @cpuinfo//:linux_x86_64
 @cpuinfo//:linux_arm
 @cpuinfo//:linux_armhf
 @cpuinfo//:linux_aarch64
 @cpuinfo//:macos_x86_64
 @cpuinfo//:windows_x86_64
 @cpuinfo//:android_armv7
 @cpuinfo//:android_arm64
 @cpuinfo//:android_x86
 @cpuinfo//:android_x86_64
 @cpuinfo//:ios_x86_64
 @cpuinfo//:ios_x86
 @cpuinfo//:ios_armv7
 @cpuinfo//:ios_arm64
 @cpuinfo//:ios_arm64e
 @cpuinfo//:watchos_x86_64
 @cpuinfo//:watchos_x86
 @cpuinfo//:watchos_armv7k
 @cpuinfo//:watchos_arm64_32
 @cpuinfo//:tvos_x86_64
 @cpuinfo//:tvos_arm64
ERROR: Analysis of target '//javascript:khaliwasm' failed; build aborted: Analysis failed
INFO: Elapsed time: 1.252s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 3563 targets configured)

I do not have any idea about how to solve the errors

@prantoran Were you able to fix this? Also the web version seems to be loading a .data binary file.
Do you think it means that we can't deploy our own tflite Model and will have to stick to the model inside that .data file.

@droidlabour I have not made any new attempts. Last time, I was stuck with GLog version in build files which was later updated in MediaPipe. Currently, I am learning about wasm and pthreads, after which I will attempt again. Not sure about .data binary file though.

I was experimenting this again and stumbled onto the following error:
mediapipe/mediapipe/framework/deps/BUILD:314:11: C++ compilation of rule '//mediapipe/framework/deps:threadpool' failed (Exit 1)
mediapipe/framework/deps/threadpool_pthread_impl.cc:63:64: error: use of undeclared identifier 'SYS_gettid'
      internal::CreateThreadName(thread->name_prefix_, syscall(SYS_gettid));
                                                               ^
mediapipe/framework/deps/threadpool_pthread_impl.cc:101:15: error: use of undeclared identifier 'pthread_setname_np'
  int error = pthread_setname_np(name.c_str());

and found a prospective solution which requires using a specific commit of GLog. I was thinking how to add the commit update locally.

@prantoran , I am also stuck at the same, I guess you should try again as MediaPipe launched a release and you already explored a lot. best of luck in advance.

@vKrypto It is harder than it seems to be, currently I am doing the desktop version but trying to port to web-assembly. I faced the pthread issue, as far as I know the current web-assembly version do not support multithreading natively and I am not sure whether XNNPACK web-assembly version or TensorflowJS was used for web version. There is a MediaPipe pre-trained handpose model available as npm package that uses TensorflowJS so I guess TensorflowJS is compatible.

Hi all-- this is quite a tricky task, so expect to run into a lot of strange build errors along the way.  However, a few tips that might be helpful:
(1) I'd recommend (even if building on Desktop) that you use mostly mobile versions of the different MediaPipe components (so mostly iOS/Android C++ code), since mobile code is generally more portable.
(2) I'd also recommend starting simpler with the options, and then once you have things building, you can increase the specificity/difficulty.  So for example, leave most things as default (e.g. don't change malloc), and then once things work tweak for optimizations/improvements.
(3) Disable pthreads, and consider enabling assertions.
(4) Make sure you're using the official emscripten bazel rules (I think the TF.js project you indirectly copied from initially wasn't, but a few days ago was updated to do so?  Not sure on the details there).
(5) When I tried a build this way, glog was the library I ran into the first BUILD incompatibility with-- so if you're reaching that, then you're off to a good start.
Also, yes, the .data file is for packaging up assets to load at runtime, and is tied to the Emscripten virtual FileSystem.  So don't worry too much about that, since it won't really be relevant until you have things building.

Hi, has anyone made progress building Mediapipe for Javascript?
I am trying to use Objectron

@sgowroji I saw the comment, I am working on it. I did make some progress by gradually adding dependencies and fixing bugs.

Reopening issue. Any progress on this from anyone?

I did setup WebAssembly compilation with the guidance from @tyrmullen.
I made changes all over the MediaPipe codebase, but the changes themselves are very simple. From the top of my head, the changes I remember:

add C++ macros to use Android specific code when I compiler is Emscripten while excluding anything does not work for android
setup config_settings when the cpu target is emscripten_wasm: https://github.com/prantoran/mediapipe/blob/master/mediapipe/BUILD#L20-L40
setup select statements in various BUILD files to include headers that work with android (and exclude headers that does not work with android).
Use mediapipe/gpu/gl_context_webgl.cc where ever there were errors due to ..gl_context_.. files. This file handles the code for converting OpenGL to WebGL.
Exclude any code that uses OpenCV since converting to WebAssembly is complicated.

Here is the cloned repo where I did the experimentation: https://github.com/prantoran/mediapipe. My codes are in hello-world and hello-server folders.
The changes seem a bit of mouthful but they are very simple. I am relatively free after 9th September and I will attempt to start fresh and document all the changes using another clone of MediaPipe.

Hello, may I ask if the running environment of your project seems to be under Linux? What other software, such as Bazel, Emscritpen，need to be installed?  What versions are they?

I ran in Ubuntu 20.04, Bazel version 4.2.0, and installed OpenCV using the MediaPipe scripts.
I setup Emscripten in Bazel using the instructions here: https://github.com/emscripten-core/emsdk/tree/main/bazel

As an aside, @evanbrown3000: We recently released a MediaPipe JS Solutions API for Objectron.  See codepen demo and reference documentation.

@prantoran  Hello, appreciate for the solutions you shared which really helps a lot. Otherwise, I've tried the live demo present in your own MediaPipe repo but I'm wondering that why the performance seems much worse than the official web demo, is there any known cause with your custom code ?

@zwaittt The face mesh is slow because my demo code was drawing the filled circles for the landmarks using '2d' context, instead of using 'webgl2' context. I tried drawing with 'webgl2' context but code was pretty complicated. I think reference for how to draw fast can be found in draw_utils.

I think the CodePen demos (although not the Visualizer ones) actually use '2d' context rendering as well, but they're fairly well-optimized for that case (so draw_utils might be a good reference still).  Although as prantoran@ mentions, 'webgl2' rendering would be faster.  Probably the easiest way to do that would be to build that into the MediaPipe graph itself, like any normal OpenGL-based (gpu/) calculator.
Also, currently the initial video input uses an ImageFrame format rather than a GpuBuffer-- this will cause extra slowdown as well (more GPU<-->CPU round-trips).
Unfortunately, one other large delta is likely because the ML inference (traditionally the most expensive part of the pipeline) is done CPU-side here, but we don't have a choice in the matter.  The GPU ML inference engine, which we're able to use for our CodePen demos, is not open-sourced.

@tyrmullen could you share some details about "The GPU ML inference engine" ?

test@test-virtual-machine:~/Desktop/bazel-emscripten_1/mediapipe$  bazel build -c opt //javascript:khaliwasm --config=wasm --sandbox_debug --verbose_failures
Starting local Bazel server and connecting to it...
INFO: Repository org_tensorflow instantiated at:
/home/test/Desktop/bazel-emscripten_1/mediapipe/WORKSPACE:194:13: in
Repository rule http_archive defined at:
/home/test/.cache/bazel/_bazel_test/7c967cf77477bcf20c2d149c1dd1891c/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in
INFO: Repository 'org_tensorflow' used the following cache hits instead of downloading the corresponding file.

Hash '673d00cbd2676ae43df1993e0d28c10b5ffbe96d9e2ab29f88a77b43c0211299' for https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/7c09d15f9fcc14343343c247ebf5b8e0afe3e4aa.tar.gz
If the definition of 'org_tensorflow' was updated, verify that the hashes were also updated.
ERROR: An error occurred during the fetch of repository 'org_tensorflow':
Traceback (most recent call last):
File "/home/test/.cache/bazel/_bazel_test/7c967cf77477bcf20c2d149c1dd1891c/external/bazel_tools/tools/build_defs/repo/http.bzl", line 121, column 10, in _http_archive_impl
patch(ctx)
File "/home/test/.cache/bazel/_bazel_test/7c967cf77477bcf20c2d149c1dd1891c/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 135, column 22, in patch
ctx.patch(patchfile, strip)
Error in patch: Error applying patch /home/test/Desktop/bazel-emscripten_1/mediapipe/third_party/org_tensorflow_compatibility_fixes.diff: Incorrect Chunk: the chunk content doesn't match the target
Original Position: 40

Original Content:
// The advantages of using gmock matchers instead of self defined matchers are
// better error messages, more maintainable tests and more test coverage.
#if !defined(PLATFORM_GOOGLE) && !defined(PLATFORM_GOOGLE_ANDROID)
#include <gmock/gmock-generated-matchers.h>
#include <gmock/gmock-matchers.h>
#include <gmock/gmock-more-matchers.h>
#endif
Revised Content:
// The advantages of using gmock matchers instead of self defined matchers are
// better error messages, more maintainable tests and more test coverage.
#if !defined(PLATFORM_GOOGLE) && !defined(PLATFORM_GOOGLE_ANDROID)
#include <gmock/gmock-matchers.h>
#include <gmock/gmock-more-matchers.h>
#endif
ERROR: no such package '@org_tensorflow//tensorflow': Error applying patch /home/test/Desktop/bazel-emscripten_1/mediapipe/third_party/org_tensorflow_compatibility_fixes.diff: Incorrect Chunk: the chunk content doesn't match the target
Original Position: 40
Original Content:
// The advantages of using gmock matchers instead of self defined matchers are
// better error messages, more maintainable tests and more test coverage.
#if !defined(PLATFORM_GOOGLE) && !defined(PLATFORM_GOOGLE_ANDROID)
#include <gmock/gmock-generated-matchers.h>
#include <gmock/gmock-matchers.h>
#include <gmock/gmock-more-matchers.h>
#endif

I encountered the above problem when running, may I ask if you have any solution?
And any progress on this from anyone?

@DLGL99GL I assume that you can run the C++ examples in Linux and followed the emsdk github to setup emsdk in Bazel. Tensorflow supports WASM so I guess the error is in the WorkSpace file or in the files for org_tensorflow  in the third_party folder.

Also maybe Bazel is accessing old cached binaries. To clean Bazel cache: bazel clean.

@DLGL99GL I assume that you can run the C++ examples in Linux and followed the emsdk github to setup emsdk in Bazel. Tensorflow supports WASM so I guess the error is in the WorkSpace file or in the files for org_tensorflow in the third_party folder.

no, i setuped emsdk in Bazel  according to  https://hackernoon.com/c-to-webassembly-using-bazel-and-emscripten-4him3ymc

@AlexShafir  Sure-- for some of our web demos we compile into our WebAssembly binaries special GPU-based ML inference code which is not yet open-sourced.  Non-web platforms have their own GPU-based ML backends, but these do not run on web.  When building from open-source, the only option we offer for web is therefore CPU-based ML (e.g. TfLiteInferenceCalculator or InferenceCalculator); using XNNPack delegates for these will ensure the best performance/behavior.

hello， I found that the version of MediaPipe I used was somewhat different from yours.
I only added the part you added to my WORKPACE, which solved the errors about “org_tensorflow”；

load("//javascript/toolchain:cc_toolchain_config.bzl", "emsdk_configure")
emsdk_configure(name = "emsdk")

but there were still errors about “//javascript/toolchain:emscripten”

test@test-virtual-machine:~/Desktop/mediapipe-master$  bazel build -c opt //javascript:khaliwasm --config=wasm --sandbox_debug --verbose_failures
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'com_google_absl' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'com_google_protobuf' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'com_google_googletest' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'com_github_gflags_gflags' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'build_bazel_rules_apple' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'build_bazel_rules_swift' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'build_bazel_apple_support' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'bazel_skylib' because it already exists.
DEBUG: /home/test/.cache/bazel/_bazel_test/be32fe62fbd0fe95a3b09acff98abd5c/external/org_tensorflow/third_party/repo.bzl:109:14:
Warning: skipping import of repository 'pybind11' because it already exists.
ERROR: /home/test/Desktop/mediapipe-master/javascript/BUILD:44:10: every rule of type cc_binary implicitly depends upon the target '//javascript/toolchain:emscripten', but this target could not be found because of: no such target '//javascript/toolchain:emscripten': target 'emscripten' not declared in package 'javascript/toolchain'; however, a source directory of this name exists.  (Perhaps add 'exports_files(["emscripten"])' to javascript/toolchain/BUILD, or define a filegroup?) defined by /home/test/Desktop/mediapipe-master/javascript/toolchain/BUILD
ERROR: Analysis of target '//javascript:khaliwasm' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.146s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets conf
igured)

I didn't find the folder named javascript in emsdk github，so i add some but i don't know how to add files about  "//javascript/toolchain:emscripten"; Could you tell me what I should do  next，thanks； @prantoran

@tyrmullen
Thank you for your thoughtful response!

I merged all the changes that seemed essential to me in a single commit on the latest version of MediaPipe: https://github.com/prantoran/mediapipe/tree/latest_wasm.
The latest_wasm has all the changes in a single commit while the default branch refers to the original project.

@prantoran @tyrmullen  could you tell :
Background of my question:

goal to use: mediapipe face mesh solution in js.
my environment : PC windows10，CPU:  intel-i7-8700, GPU: intel UHD Graphics 630
already run:

official mediapipe js face mesh solution and tf-js face mesh model. The performance is  15fps but with 65% GPU Utilization. This high Utlization of gpu leads to few gpus left for other programs to use.
c++ CPU version mediapipe face mesh solution, i can compile it to a dll, and it runs fast in about 3-5ms( only inference part without rendering), and it can meet my demand. so i want  converting this c++ CPU mediapipe library into wasm.

MY question:

is it possible to  converting this c++ CPU version mediapipe library into wasm?
if it possible , does the speed of wasm version can be as fast as the c++ cpu version?
how can i learn from your project?

Thanks a lot!

@LebronJames0423 In the latest branch of the project, all the changes in the MediaPipe's internal files are shown in the commits.
One caveat is that I developed using Ubuntu so there might be additional requirements for Windows.

@LebronJames0423 For any of our APIs running on GPU, you can instead force CPU ML inference by setting the useCpuInference option to true in the configuration options. This should work for our FaceMesh API as well. So to answer your first two questions: (1) Yes-- just use the above advice. (2) C++ CPU should always be faster, all other things being equal. But the gap between them used to be huge, and now is much smaller, and continuing to narrow (with features like Wasm-SIMD helping tremendously).
@tyrmullen @prantoran  OK，I will try your suggestions，thanks!

@prantoran By refering to your repo, I was able to compile the hello world desktop into webassembly. I was able to load the wasm file in chrome and the cout commands in the cpp file were appearing in the browser console as well. However, the wasm hangs at poller.next(&packet), as is evident from observing the debug flags I had put in the cpp code from the console. I did some digging in the internet and found out that it is hanging because poller.next() waits until the next packet is available. The same code is working properly when run from the terminal as a cpp file but upon converting to wasm something is not working. It would be great if someone could provide more information on why the code behaves as such upon being converted to wasm.
build file for cpp helloworld
cc_binary(
    name = "hello_world",
    srcs = ["hello_world.cc"],
    visibility = ["//visibility:public"],
    deps = [
        "//mediapipe/calculators/core:pass_through_calculator",
        "//mediapipe/framework:calculator_graph",
        "//mediapipe/framework/port:logging",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
    ],
)

cpp helloworld code that runs properly
#include "mediapipe/framework/calculator_graph.h"
#include "mediapipe/framework/port/logging.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

//namespace mediapipe {

int PrintHelloWorld() {
  // Configures a simple graph, which concatenates 2 PassThroughCalculators.
  mediapipe::CalculatorGraphConfig config =
      mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(R"pb(
        input_stream: "in"
        output_stream: "out"
        node {
          calculator: "PassThroughCalculator"
          input_stream: "in"
          output_stream: "out1"
        }
        node {
          calculator: "PassThroughCalculator"
          input_stream: "out1"
          output_stream: "out"
        }
      )pb");

  mediapipe::CalculatorGraph graph;
  graph.Initialize(config);
  auto status_or_poller = graph.AddOutputStreamPoller("out");
  mediapipe::OutputStreamPoller poller = std::move(status_or_poller.value());
  graph.StartRun({});
  // Give 10 input packets that contains the same std::string "Hello World!".
  for (int i = 0; i < 10; ++i) {
    graph.AddPacketToInputStream(
        "in", mediapipe::MakePacket<std::string>("Hello World!").At(mediapipe::Timestamp(i)));
  }
  // Close the input stream "in".
  graph.CloseInputStream("in");
  mediapipe::Packet packet;
  // Get the output packets std::string.
  while (poller.Next(&packet)) {
    std::cout << poller.QueueSize()<< std::endl;
    std::cout << packet.Get<std::string>()<<std::endl;
  }
  // return graph.WaitUntilDone();
}
//}  // namespace mediapipe

int main(int argc, char** argv) {
  google::InitGoogleLogging(argv[0]);
  PrintHelloWorld();
  //std::cout << "hello world!!!!!!!!!" << std::endl;
  return 0;
}

output as expected in the terminal

build file for the wasm helloworld:

cc_binary(
    name = "hello-world-web",
    srcs = ["helloWorldWasm.cc"],
    deps = [
        "//mediapipe/calculators/core:pass_through_calculator",
        "//mediapipe/framework:calculator_graph",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "//third_party:glog",
        "//mediapipe/framework/port:logging",
    ],
    linkopts = [
        "-s USE_PTHREADS=0",
        "-s ALLOW_MEMORY_GROWTH=1",
        "-s ASSERTIONS=1",
        "-s USE_WEBGL2=1",
        "-s ERROR_ON_UNDEFINED_SYMBOLS=0",
        "--bind",
    ]
)

wasm_cc_binary(
    name = "hello-world-wasm",
    cc_target = ":hello-world-web",
)

cpp code used to compile the wasm file:
#include "mediapipe/framework/calculator_graph.h"
#include "mediapipe/framework/port/logging.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

#include <emscripten/bind.h>
//#include <emscripten.h>

    int PrintHelloWorld() {
                std::cout << "flag 1"<< std::endl;
                // Configures a simple graph, which concatenates 2 PassThroughCalculators.
                mediapipe::CalculatorGraphConfig config =
        mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(R"pb(
            input_stream: "in"
            output_stream: "out"
            node {
            calculator: "PassThroughCalculator"
            input_stream: "in"
            output_stream: "out1"
            }
            node {
            calculator: "PassThroughCalculator"
            input_stream: "out1"
            output_stream: "out"
            }
        )pb");

            mediapipe::CalculatorGraph graph;
            graph.Initialize(config);
            auto status_or_poller = graph.AddOutputStreamPoller("out");
            mediapipe::OutputStreamPoller poller = std::move(status_or_poller.value());
            graph.StartRun({});
            std::cout << "flag 2"<< std::endl;
            // Give 10 input packets that contains the same std::string "Hello World!".
            for (int i = 0; i < 10; ++i) {
                graph.AddPacketToInputStream(
                    "in", mediapipe::MakePacket<std::string>("Hello World!").At(mediapipe::Timestamp(i)));
            }
            // Close the input stream "in".
            graph.CloseInputStream("in");
            std::cout << "flag 3"<< std::endl;
            mediapipe::Packet packet;
            // Get the output packets std::string.
            while (poller.Next(&packet)) {
                std::cout << "flag 4"<< std::endl;
                std::cout << poller.QueueSize()<< std::endl;
                std::cout << packet.Get<std::string>()<<std::endl;
            }
            // return graph.WaitUntilDone();
            std::cout << "flag 5"<< std::endl;
            return 1;
    }
        int main(int argc, char** argv) {
            google::InitGoogleLogging(argv[0]);
            std::cout << "hello world wasm file loaded." << std::endl;
            PrintHelloWorld();
            return 0;
          }

     EMSCRIPTEN_BINDINGS(hello_world_module) {
             emscripten::function("PrintHelloWorld", &PrintHelloWorld);

     }

the output from chrome console:

The chrome tab hangs and will need multiple clicks to close upon loading the wasm file. I guess it is not throwing any errors since packet.next() function is behaving as it is intended to behave, however, I would like to understand why the packets aren't in the queue like it is in the cpp example when run in the terminal.
The machine I used to compile the code to wasm is mac m1 running MacOS Montenery.
Any help is much appreciated.

WebAssembly by default is single-threaded, while in your terminal cpp example, it is almost certainly multi-threaded.  Therefore, any statements which "wait" for something to happen from another thread will not work.  Instead of using OutputStreamPoller, try using ObserveOutputStream instead.

@tyrmullen  Hi, thank you for the suggestion. Following your advice I tried to modify the code and compiled it to wasm. This time it is not hanging when it is loaded in chrome. However I'm not getting the "hello world" strings either. I might be doing something silly here.
cpp code compiled to wasm:
#include "mediapipe/framework/calculator_graph.h"
#include "mediapipe/framework/port/logging.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

#include <emscripten/bind.h>
//#include <emscripten.h>

//extern "C"{

          //EMSCRIPTEN_KEEPALIVE
    int PrintHelloWorld() {
                std::cout << "flag 1"<< std::endl;
                // Configures a simple graph, which concatenates 2 PassThroughCalculators.
                mediapipe::CalculatorGraphConfig config =
        mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(R"pb(
            input_stream: "in"
            output_stream: "out"
            node {
            calculator: "PassThroughCalculator"
            input_stream: "in"
            output_stream: "out1"
            }
            node {
            calculator: "PassThroughCalculator"
            input_stream: "out1"
            output_stream: "out"
            }
        )pb");

            mediapipe::CalculatorGraph graph;
            graph.Initialize(config);
            // auto status_or_poller = graph.AddOutputStreamPoller("out");
            // mediapipe::OutputStreamPoller poller = std::move(status_or_poller.value());
            graph.StartRun({});
            std::cout << "flag 2"<< std::endl;
            // Give 10 input packets that contains the same std::string "Hello World!".
            for (int i = 0; i < 10; ++i) {
                graph.AddPacketToInputStream(
                    "in", mediapipe::MakePacket<std::string>("Hello World!").At(mediapipe::Timestamp(i)));
            }
            // Close the input stream "in".
            graph.CloseInputStream("in");
            std::cout << "flag 3"<< std::endl;
            //mediapipe::Packet packet;
            // Get the output packets std::string.
            // while (poller.Next(&packet)) {
            //     std::cout << "flag 4"<< std::endl;
            //     std::cout << poller.QueueSize()<< std::endl;
            //     std::cout << packet.Get<std::string>()<<std::endl;
            // }
            std::string a;
            graph.ObserveOutputStream("out", [&](const mediapipe::Packet& p) {
                    a = p.Get<std::string>();
                    std::cout << "flag 4"<< std::endl;
                    std::cout << p.Get<std::string>()<<std::endl;
                    // return 80;
                    return absl::OkStatus();
                });
            std::cout << a << std::endl;
            // return graph.WaitUntilDone();
            std::cout << "flag 5"<< std::endl;
            return 1;
    }
        int main(int argc, char** argv) {
            google::InitGoogleLogging(argv[0]);
            std::cout << "hello world wasm file loaded." << std::endl;
            PrintHelloWorld();
            return 0;
          }

     EMSCRIPTEN_BINDINGS(tracking_module) {
             emscripten::function("PrintHelloWorld", &PrintHelloWorld);

     }

console output in chrome when the wasm is loaded:

Try moving ObserveOutputStream to before GraphStart to make sure it's connecting, and add a WaitUntilIdle where you used to have WaitUntilDone.  WaitUntilIdle will not actually wait by default on web (despite the name), but rather will force all processing to occur inline; otherwise, you're just queueing up input packets without actually running them through the graph.  You may also want to move the std::cout << a << std::endl bit to after the WaitUntilIdle, or else it will still be blank.

@tyrmullen That seems to be what was causing the problems. It is showing the outputs correctly in chrome console now. Thanks again.

@Hrituraj202 I was saying that non-web C++ CPU should be faster than WebAssembly CPU.
If you're looking just at web apps, then GPU will usually be faster than CPU (although CPU lends itself better to multithreading, so it's not as straightforward a comparison), so most users will not want to use the useCpuInference flag.  Also, that specific question was in reference to our JavaScript FaceMesh Solution API, so it doesn't really apply to the rest of this thread.

@JonathanGoorBlink I don't know about all the modules that use OpenCV, but I was able to compile the box tracking cpu module which uses OpenCV to WebAssembly following the information in this thread. My fork is heavily modified but I think it might be useful.
iirc one of the things I did to get it to compile was to use the static OpenCV libraries. By default, the OpenCV BUILD file specified the .dylib files.
Note: I built it on an M1 mac so some of the changes that I made to the various BUILD files in the project adds mac specific changes to the default parameter because Bazel was not identifying my mac as a mac os device and was choosing the default parameter.

@JonathanGoorBlink I don't know about all the modules that use OpenCV, but I was able to compile the box tracking cpu module which uses OpenCV to WebAssembly following the information in this thread. My cloned repo is heavily modified but I think it might be useful. iirc one of the things I did to get it to compile was to use the static OpenCV libraries. By default, the OpenCV BUILD file specified the .dylib files. I built it on an M1 mac so some of the changes that I made to the various BUILD files in the project adds mac specific changes to the default parameter because Bazel was not identifying my mac as a mac os device and was choosing the default parameter.

Can u do a bootstrap code to use box tracking cpu js ?
Thank u in advance.


[FLINK-15220][Connector/Kafka][Table] Add startFromTimestamp in KafkaTableSource

What is the purpose of the change
KafkaTableSource supports all startup modes in DataStream API except startFromTimestamp, but startFromTimestamp is a common and valid use case in Table/SQL API as well.
Brief change log

Add startFromTimestamp(long) to Kafka descriptor API.
Add new keys connector.startup-timestamp-millis and connector.startup-timestamp to Kafka connector properties.
Add new value timestamp to connector.startup-mode in Kafka connector properties.
Add startup timestamp parameter to KafkaTableSource and KafkaTableSourceSinkFactory.

Verifying this change
This change added tests and can be verified as follows:

Added tests to verify Kafka descriptor API generated properties.
Added tests to verify the validation of new properties of timestamp startup mode in KafkaValidator.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs

cc @wuchong

The changes look mostly good. I have some concerns regarding using a LocalDateTime as an Instant.
To be honest, I don't like that connector.startup-timestamp assumes UTC as the time zone. I think this will be troublesome. I understand this is a helpful feature so I would suggest to expect the user to provide the time-zone explicitly. That said rather than parsing a LocalDateTime from the properties. I would parse an OffsetDateTime from the properties. For UTC users can pass a date as e.g. `2011-12-03T10:15:30Z'

@dawidwys I agree that OffsetDateTime is more deterministic and reliable. I didn't give much thought to it, because I saw many temporal functions are using UTC time zone.
Ping Jark @wuchong who would possibly be in favor of LocalDateTime for his opinion.

I agree with @dawidwys , parsing from "2011-12-03 10:15:30" to a time second since epoch should involve time zone infromation. I'm fine to parse as OffsetDateTime.
However, passing an OffsetDateTime is a little unfriendly to users. IMO, a better solution might be to use the time zone of TableConfig, but TableConfig is not exposed to TableFactory yet. So What about only support connector.startup-timestamp-millis in this version, and postpone to support connector.startup-timestamp when TableFactory can access the time zone information?

Sounds good. I'll remove connector.startup-timestamp from this PR and file a new Jira ticket to track the further support for it.

@wuchong I am not sure if an additional Z is such a hassle for users. I really think specifying instants should be as explicit as possible. That said I think splitting that to a separate issue is a good idea.

@link3280 Generally looks really good. Could you just update the remaining languages in the docs? (DDL, YAML, Python)

@dawidwys Rebased on the current master with the docs updated. Please kindly have a look.

Thank you for the update @link3280 . Will merge it later today.


How is questData.json generated?

Hi, would you consider open sourcing the tool you use to generate your data JSON files?

Hey tha23rd!
I actually do not have a tool that scrapes the wiki or anything. I manually created that json by going through each quest.
Do you have any ideas of a tool that should be able to do this? possibly a python script to iterate through the wiki pages?
Feel free to reach out to me anytime and If there is any data that is incorrect please let me know.

Created Quest Data Generation project to resolve this.
Python script that gets all current quests from url and will parse each individually. It's a WIP.

Hi, is the tool working? If so, would you mind sharing the link?

@Evaldash Hey sorry for the late response. I just published a branch 'dev', inside I updated the python file used to grab all the quest data.
Let me know if this answers your question, I have to make a couple of parsing updates


Add options for external use.

logstuff must have the ability to change the file extension. This is already implemented when using as import.
Create a way to change the file extension when saved(default must remain .txt). Maybe create a --ext flag.

Would like to add --append and --extension for now. This can be extended later.
Also need to identify the flag deliminators: '-', '--'
-> slice and parse


Pair Programming redesign of game V1

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

51 of 51 (100.0%)  changed or added relevant lines in 3 files are covered.
60 unchanged lines in 3 files lost coverage.
Overall coverage decreased (-29.0%) to 69.268%

Files with Coverage Reduction
New Missed Lines
%

repository/IngSoft2-Model/Board.class.st
6
73.91%

repository/IngSoft2-Model/Player.class.st
20
39.39%

repository/IngSoft2-Model/GameMaster.class.st
34
29.17%

Totals

Change from base Build<PHONE_NUMBER>:
-29.0%

Covered Lines:
142

Relevant Lines:
205

💛  - Coveralls

todo joya


Fix accel and steer ignore to not disable the system on overrides

This is a pull request from !important Safety Technologies USA Inc.

Please run uncrustify on your code changes:
ament_uncrustify --reformat

Looks good so far! I'm going to test it out & get back to you.

Testing went perfectly; code is merged. Thanks!


'DistributionalThesaurus' object has no attribute 'keys': is everything ok with the rdt model?

Today I have tried to run the code situated at _https://github.com/nlpub/rdt/blob/master/dt.py_ and, unfortunately, this attempt was not successful.
I manually downloaded the rdt pickle model, installed all the dependencies and adapted some code from the file in order to make it work for python 3.
Here is what happened during running: after 25 seconds of running the code crashed saying that "'DistributionalThesaurus' object has no attribute 'keys'". This error message might mean that there is something wrong with rdt model:

either the DistributionalThesaurus object is uninitialized (the absense of the attribute 'keys' means that the init function has not been run which is strange)
or there is something wrong with loading on python 3 the model which was pickled using python 2. For loading I used python 3 library called pickle. The line responsible for loading is pickle.load(open(dt_pkl_fpath, "rb"), encoding='bytes').

Hello and thank you for your interest in this repository. I am pretty sure that your problem is caused to a too new version of Gensim that has its Word2Vec/KeyedVectors API changed. Please try using an older version of Gensim.
@alexanderpanchenko could you please remind us when have you created these pickled RDTs?

Hello and thank you for your interest in this repository. I am pretty sure that your problem is caused by a too recent version of Gensim that has its Word2Vec/KeyedVectors API changed. Please try using a much older version of Gensim.
@alexanderpanchenko could you please remind us when have you created these pickled RDTs?

Hello @dustalov ,
Thank you for the answer but I have not understood what Gensim has to do with rdt. As far as I know, rdt does not use Gensim (it was not mentioned in the requirements.txt file and it was not imported anywhere in the code). Am I missing somehing?

Usually, you get quick answers here, but we are very busy these days. I am afraid I cannot help you until mid-February 2019.

@dustalov I will be glad to receive an answer if it is possible.

@sadov-m thank you for your patience. I can return to this issue only on Monday.

@sadov-m thank you for your patience. I can return to this issue only on Monday.
Just in case you need a word embeddings model based on RDT, you can download it on ZENODO: https://zenodo.org/record/400631.

I added a model converter script ./convert_python3.py rdt.pkl that outputs a file named rdt.pkl.3. You can use it instead of the original one. I do not like how the model fetching code is written, so currently make sure that the file name is still rdt.pkl.

Thank you for help! I will try to use it.


code-sum of digits of a number

I have changed the code. This code is beginner friendly and more modular. Let me know if any improvements are needed.

No issues great work but cleaning few lines might make it a bit more user friendly
Thank u @OfficialAhmed


Can I save pictures to my PC?

What is the principle of this project? Did you forward the request to another Dicord channel? Can I access MidJourney through discord's API to retrieve images and save them locally? Thank you

Yes you can, as you download any attachment in discord message.


How to make a screen work on specific GPU?

Hi, thank you for you work. I have a issue about multi GPUs. I have four GPUs and I add four screen. I run some andriod simulators on the screen. I found total four screen load on one single GPU. Is there a way to make a screen work on specific GPU? I want the four screen load average on each GPU.
Thank you. Wish you a good day.

As I said in #46 about selecting parent GPU, you should find the Vendor ID of your GPU. But I'm not sure VDD can support a specific GPU per virtual display.
Run this program to enumerate your GPUs: enum_dxgi.zip
Adapter (1): AMD Radeon(TM) Graphics
  LUID: 0x20F88
  Vendor ID: 0x1002
  Device ID: 0x1638

I don't have any device that has multiple GPUs to support your case, so do it yourself. Note that my app does not support multiple VDD adapters, you should modify the source code to run multiple instances and install multiple VDD adapters via devcon/nefcon. Then follow this guide to set Vendor ID before pluggin a virtual display for each VDD.
In case of multiple GPUs have the same Vendor ID (AMD - 0x1002, NVIDIA - 0x10DE), the issue cant be solved with VDD. I see you have opened the same issue in https://github.com/itsmikethetech/Virtual-Display-Driver/issues/110, that's open source project and easy to solve your case.

Latest HDR beta has this implemented, but may be buggy. Please let us know how it works. https://github.com/itsmikethetech/Virtual-Display-Driver/releases/tag/24.7.18HDR

I have two 3090, but, don't know why there are 3, and it seems to use the LUID, it's unique.
D:>enum_dxgi.exe
Adapter (0): NVIDIA GeForce RTX 3090
LUID: 0x1524C
Vendor ID: 0x10DE
Device ID: 0x2204
Adapter (1): NVIDIA GeForce RTX 3090
LUID: 0x141B5
Vendor ID: 0x10DE
Device ID: 0x2204
Adapter (2): NVIDIA GeForce RTX 3090
LUID: 0xA5C85
Vendor ID: 0x10DE
Device ID: 0x2204

seems i now need to rely on anyone else that can give a pull request for this function.
really needed an item in per screen right mouse button menu that can switch the VD between detected GPUs
btw, I'm using R5 7600x with RX560D and seems these can work together. And RX560D is really sucks and when I swinging my mouse in VD which connected by sunshine, the local machine processing latency went up and so did the GPU usage

@fr1g that's impossible with Parsec VDD, it allows parent GPU selection in v0.45 but for AMD/NVIDIA, and cannot switch immediately without restarting device/adapter. Note that the driver is closed source, you should look at itsmikethetech's virtual display.

@fr1g that's impossible with Parsec VDD, it allows parent GPU selection in v0.45 but for AMD/NVIDIA, and cannot switch immediately without restarting device/adapter. Note that the driver is closed source, you should look at itsmikethetech's virtual display.

usable and fulfilled my needs (though they seems not providing an independent manager GUI)


CLI Coexistence Tests -  Unify the installation of new tanzu cli

What this PR does / why we need it
The purpose of this testing is to analyze the implications of unification of Tanzu CLI to be able to use both new and legacy Tanzu CLI versions.

Unifying the installation of Tanzu - All "_" prefixed directories used by tanzu are changed to use the existing CLI installations directories without "_" prefix .
Added cli coexistence go tests using e2e framework
Expanded the CLI lifecycle operations methods in E2E to install, uninstall tanzu cli
Implemented docker to run cli coexistence tests
Added test results summary

E2E framework changes for coexistence

made tanzu prefix dynamic for commands.
Added new CLI lifecycle methods to install, uninstall, Reinstall and Rollback of CLI
Update the method definitions of various helpers like feature flags, context lifecycle, config lifecycle, plugin lifecycle to accept Tanzu prefix

Which issue(s) this PR fixes

Fixes #
Describe testing done for PR

Did run the CLI Coexistence tests GitHub workflow and local testing using
make build-cli-coexistence

make cli-coexistence-tests

This will run various tests testing plugin lifecycle and core cli commands using legacy and new tanzu cli on a docker env.

GitHub Workflow to run the tests
GitHub badge for the status of the tests

Test Result Summary

Release note

 Added CLI Coexistence Tests

Additional information
Special notes for your reviewer

Follow up PR changes

Refine Test results summary with separate test suite.
Additional tests around context.

Please update the e2e documentation (https://github.com/vmware-tanzu/tanzu-cli/tree/main/test/e2e) with new methods signature and coexisting functionality.

Thanks for the review. Since this PR has been growing I have divided these changes into multiple PRs
PRs to be reviewed in order
https://github.com/vmware-tanzu/tanzu-cli/pull/220
https://github.com/vmware-tanzu/tanzu-cli/pull/222
https://github.com/vmware-tanzu/tanzu-cli/pull/124


Ability to redirect the execute_test output to file

Is your feature request related to a problem? Please describe.
Currently, because of the sys.stdout.isatty() check in https://github.com/opensearch-project/opensearch-benchmark/blob/main/osbenchmark/utils/console.py#L157, there's no way to do redirect for the execution output.
Describe the solution you'd like
We want to support opensearch-benchmark to output to stdout even when it is not connected to a terminal, such as when a shell redirection is employed.
Describe alternatives you've considered
Additional context

@ansjcy Have you tried appending the parameter --results-file? This parameter tells OSB to write the command line output to the specified file.

Closing issue as this feature is supported already. Please feel free to reopen if needed.


[SPARK-42669][CONNECT] Short circuit local relation RPCs

What changes were proposed in this pull request?
Operations on LocalRelation can mostly be done locally (without sending RPCs).
We should leverage this.
Why are the changes needed?
Avoid sending RPCs for LocalRelation.
Does this PR introduce any user-facing change?
'No'.
New feature.
How was this patch tested?
Exists test cases.

The failure GA is unrelated to this PR>
ping @hvanhovell @zhengruifeng cc @dongjoon-hyun @amaliujia

sorry, test_parity_torch_distributor became flaky, I am going to disable related cases for now

So this happens when
val df = createDataFrame(...)
df.collect()

?

So this happens when

It happens when the root plan is LocalRelation.

@hvanhovell The failed GA is unrelated to this PR.

Because there are difference suggestion from @hvanhovell and @ueshin, I don't know how to continue this job.

cc @HyukjinKwon

ping @hvanhovell cc @ueshin

@hvanhovell Do we need this change?


Team comments showing up in docs

For example: https://angular.io/docs/ts/latest/guide/template-syntax.html
`Ward: firstName appears here (below) for the first time without any mention it was coming and isn't in the
code sample. Do we mean name instead?
`
I assume it's not intended for these to be incorporated in the docs, and figure pull requests to remove them aren't the correct solution.
Thanks

That was fixed.


Multi - Module

Reactor Project with multi-module.
clearer structure and reusable

This project has been refactored! And this should keep the style for a long time!


[Key Manager] Adds an end-to-end smoke test for the key manager.

Motivation
This PR adds a single smoke test for the key manager. It does so in 3 commits:

First, it modifies the key manager execution loop to handle liveness errors internally (i.e., log the errors, but not kill the process). This is useful as liveness errors can be caused due to human error (e.g., if the operator calls the key manager too often), so it's safer to log than to kill the process.

We fix a renaming issue with secure storage configurations.

We add the smoke test that: (i) spawns a local validator swarm; (ii) modifies a local validator config to initialize the key manager; (iii) bootstraps secure storage with the required keys; and (iv) checks a rotation occurs on chain and in storage.

Have you read the Contributing Guidelines on pull requests?
Yes.
Test Plan
All local tests pass (including the new smoke test).
Related PRs
None.

:umbrella: The latest upstream changes (presumably 83878247a01b4765a0b55ce07656200cb32290c3) made this pull request unmergeable. Please resolve the merge conflicts.

:umbrella: The latest upstream changes (presumably 544be842a5fdf25ac9985d534b32df5380b617e7) made this pull request unmergeable. Please resolve the merge conflicts.

Thanks, David! 😄 @bors-libra=davidiw

:pushpin: Commit 75b314e has been approved by davidiw

:hourglass: Testing commit 75b314ed2dfd044be811b2bf62f06b327192272a with merge 95eb5d5c142baa6b6f92dad5ecb446fd25dc013f...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: davidiw
Pushing 95eb5d5c142baa6b6f92dad5ecb446fd25dc013f to master...


[Request] Add line number next to each result in search results

Hi,
Please add line number and column number next to each result in the search results.
For example as it done in "Problems" tab in the bottom panel and also in the "Breakpoints" list.
Current vscode version: 1.77.1 (Windows)
Thanks.

Enabling search.showLineNumbers shows the line number, but not the column number(s). Does this suffice, or do you want to put in a feature request for column also?

Thanks @andreamah,
I wasn't familiar with that setting :)
And regarding your question. so yes.
I think it could be nice to add this option as a separate setting like "search.showColumnNumbers"

This feature request has not yet received the 20 community upvotes it takes to make to our backlog. 10 days to go. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

:slightly_frowning_face: In the last 60 days, this feature request has received less than 20 community upvotes and we closed it. Still a big Thank You to you for taking the time to create this issue! To learn more about how we handle feature requests, please see our documentation.
Happy Coding!


Using behavior to test BigtableBufferedMutator

Removing use of internal heapSizeManager methods to test BigtableBufferedMutator in favor of mimicking a completion of the ListenableFuture.

LGTM


⬆ Updated floating-ui to latest version

resolves #2703

For some unexplained reason two sidebar tests fail after upgrading floating-ui. These tests does not fail in develop. Floating-ui is not used by sidebar and I can not see floating-ui installing any different versions of jest/testing-library in package-lock 🤔
Continuing the investigation...

I suspect the problem with sidebar was related to Tooltip and how the FloatingPortal is injected since v(0.13.0) combined with the flimsy selector we had in the test (expect(screen.getAllByRole('generic')[2]).toHaveStyle({) So I think its fine 🤞