Configurable preloads

Remove loadStyles() and copy loadStyle() and loadLink() from Milo.
Authors or developers can configure the preloads metadata for an individual or a group of pages to improve the loading performance.
The preloads metadata is a string of comma-separated links. The variable $MILOLIBS is for the Milo location.

Resolve: MWPW-154349
Test URLs:
Before: https://main--dc--adobecom.hlx.live/acrobat/resources/how-to-create-fillable-pdf
After: https://MWPW-154349--dc--adobecom.hlx.live/acrobat/resources/how-to-create-fillable-pdf

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.31%. Comparing base (91aa506) to head (967d8fa).

Additional details and impacted files
@@            Coverage Diff             @@
##            stage     #713      +/-   ##
==========================================
+ Coverage   90.33%   91.31%   +0.98%
==========================================
  Files          39       39
  Lines        4718     4746      +28
==========================================
+ Hits         4262     4334      +72
+ Misses        456      412      -44

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Python exercises

Python Code Demos for I2ML
=============================
demo-cart

cell 8: Error because X_test (and y_test) is not defined
and after that there is missing a lot (like 80% or so) - why?

=============================
demo-kaggle

there are 2 versions - which one should I look at?

=============================
demo-cmpclass

cell 10: In the R solution there is no train test split (because that is the default of plot_learner_prediction, see https://mlr3viz.mlr-org.com/reference/plot_learner_prediction.html), but perhaps it is better the way you do it (with train test split), so keep it that way :)
but explain a bit what you are doing here (most of all the np.meshgrid)
cell 12: comment a bit more (in general). Here you are plotting train and test data, that is good idea, but say that you do is. When I read that as a student, perhaps I don't have to understand the code in the cell, because it is "just" plotting the data. But without a comment, I don't know that until I read the code line by line, and then I see "ah, ok, that's a plot of the data". And a legend would be nice in the plot.
cell 13 and following: Could you have all the plots in one cell as in R solution? Now some are missing and since they do not fit on one page I have to scroll up and down to compare them visually. And some are missing, right? (Perhaps, if they do not fill all, you can just use the 5 from the R solution)
cassini: why are the decision boundaries of the multinomial regression and the LDA not linear? They should be from a theoretical perspective. Is that a bug in the plot_decision_boundaries() function?
overall: the interpretations are missing
the last part is missing: why?

=============================
demo-genclass

You changed a lot of the descriptive text, why?
nice idea with the 3D plotly plot! :)
naive Bayes: The "normal" version is missing, correct?
naive Bayes: The model fit and results of the version wirh categorical predictors is missing, correct?
=> The rest I did not look into that thoroughly yet, but seems pretty nice

=============================
demo-knn

You changed a lot of the descriptive text, why?
And there is missing like a lot, isn't it?

=============================
demo-limo

that one looks really nice. Only in the end, where you have the comparison with scikit-learn and the exact solution: where is the manually computed solution as comparison?

=============================
demo-logreg

here is missing like the second half, why?

=============================
demo-nested

here is missing a lot as well?
plus: undefined "i" in cell 6

=============================
demo-overfitting

mmce is quite far away from the R solution, why?
seems pretty complete
could you add - here and in all other chunks - labels for the confusion matrices? sklearn has truth in rows and predictions in columns, but in I2ML we usually have it the other way around. That's ok as long as the confusion matrices have labels. Thanks :)

=============================
demo-randomforests

seems pretty complete
more comments would be nice

=============================
demo-resampling

seems pretty complete
why are numbers in cell 14 quite off those from the R solution?

=============================
demo-roc

seems pretty complete

=============================
demo-splines

pretty much work in progress?

@ludwigbothmann is this still relevant?

yes

We have now new, proper python exercises, I close this now and delete the branch.


Can't get GTA val set

When I'm in Data Preparation step, after I run the following code:
python loader/download.py mini
I got a result:
Connecting to dl.yf.io (dl.yf.io)|<IP_ADDRESS>|:80... --2022-10-27 17:46:01--  (try: 3)  http://dl.yf.io/bdd-data/3d-vehicle-tracking/label/gta_3d_tracking_val_label_0001.zip
Connecting to dl.yf.io (dl.yf.io)|<IP_ADDRESS>|:80... failed: Unknown error.
Retrying.

failed: Unknown error.
Retrying.

failed: Unknown error.
Retrying.

Is there something wrong with my Internet or VPN? How can I get this GTA dataset?

Hi, I've met the same problem. Have u download successfully ever? Would u like try to share them throw Netdisk or other ways.
I found it was sooo slowly download through http://dl.yf.io/bdd-data/3d-vehicle-tracking/

Sry, I think you just have to wait for some time. I also found it downloading to slow so I interrupted it and delete all of them. 😕


minor Add sql manual for stats

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run feut


Linting with the new version of Black 23.1

This PR is solely to fix linting errors raised by the CI due to updates to black. No other changes have been made herein.
My other PR (#934) has been rebased on this branch according to a suggestion by @mashehu on Slack.

I have no issues with the python code new lint, but concerned by the new json lint

Fair enough. Linting the JSON seems to have crashed the whole thing... :-(


Add tx decode/encode subcommands

What
Add tx decode and encode subcommands.
Why
To make decoding tx's to JSON more discoverable and convenient. It's the most common XDR to want to decode.
Including encode for symmetry.
Dependent on:

https://github.com/stellar/rs-stellar-xdr/pull/403

I just had this same idea. This could be very useful for the tx edit functionality. E.g.
{
    "tx": {
      "tx": {
        "source_account":  "alice",
        "fee": 5000,
        "seq_num":<PHONE_NUMBER>1945,
        "cond": {
          "time": {
            "min_time": 0,
            "max_time":<PHONE_NUMBER>
          }
        },
        "memo": {
          "text": "Load lumens"
        },
        "operations": [
          {
            "source_account": null,
            "body": {
              "payment": {
                "destination": "bob",
                "asset": "native", // could also handle `usdc:cicle` here
                "amount": 50000000
              }
            }
          }
        ],
        "ext": "v0"
      },
      "signatures": [
        {
          "hint": "26f802ad",
          "signature": "b26cde9fc068173241013279207343d17124488de0d44e43858faab093aca30efee4dbbce8a9fd96b11b7e36bd21db355c77fbe9bc2c493545b1bf19e31f5109"
        }
      ]
    }
  }

Perhaps we could add a tx update, which would then resolve all of the accounts and also maybe update the seq_num and fees?

Perhaps we could add a tx update, which would then resolve all of the accounts and also maybe update the seq_num and fees?

Replying to this over here:

https://github.com/stellar/stellar-cli/issues/1643#issuecomment-2533725947


Unset livecd root password prevents automatic deployments

To ssh to nixos machine i need physical access(monitor/console attached) to setup password so i am able to ssh(most of the times i'm just blindly typing to do that).
This problem is bothering me for quite some time. I could create my livecd variation, but wouldn't be it easier to just set some default root password?

There is an issue upstream, closing.


Insert contained subroutine bodies

Here's the routine I use to insert contained subroutine bodies.
Should I clean up the documentation, add a test and make a pull request?
BTW, is there any way to squash commits on github?

Thanks! I believe @mlange05 has looked into this some more and may have come up with a similar utility himself - so I suggest to wait for his comments before spending time on clean-up and tests.

PS: I don't know if squashing is available via a Github functionality but you can do this in the regular Git way, e.g. by doing a git rebase -i origin/main where you can select the commits to squash, and then force-push the branch.


Customizable powerups

Similarly to spheres, powerups should also be defined.
A given powerup entry should hold the following:

Name when picked up
Particle/sprite/path (the way powerups are rendered will change in Beta 1.1 and a few times in future versions too)
Sound when picked up
Effect (example effects below)
Weight (chance of dropping)

The entry index should be an internal powerup name, like it occurs in main.lua header.
Example effects:

replace current sphere (wild, lightning, fireball) - parameters: sphere color
apply a certain speed for a certain amount of time (slow, stop, reverse) - parameters: speed, time
destroy a certain color (color bomb) - parameters: color
apply speed shot (speed shot) - parameters: time, speed(?)
Future versions:
summon a scorpion - no parameters(?)

Implemented: https://github.com/jakubg1/OpenSMCE/commit/a56c8387800de688648832cc70b25d9a8429117e

Implemented: https://github.com/jakubg1/OpenSMCE/commit/a56c8387800de688648832cc70b25d9a8429117e


Version 0.9.7

After usate at versioni 0.9.7 integration stop completly to work. I downgraded to 0.9.6 and work correcly

Same here. I cant see Goodwe in Settings - Integrations.
Thx for great job.

After usate at versioni 0.9.7 integration stop completly to work. I downgraded to 0.9.6 and work correcly

I have the same issue. Trying to downgrade to prevous version.

I have the same issue. Trying to downgrade to trhe previous version,

The new version is not the culprit of the issue, it is the installation from Hacs that caused the issue.
After the upgrade of the new version I noticed this folder tree:
custom_components\godwee\goodwe
This is not correct because all the files must be in goodwe folder under custom_component, that is custom_components\godwee
I moved all the files to custom_components\godwee\ folder and the integration was correctly loaded.
So the issue is not version 0.9.7 but the way Hacs install the integration.

This is weird, it works without any issues, here, also other people reported upgrade to 0.9.7 without any issues ?
What inverter ? What HA version ? Are there any errors in the log files ?

I had a similar problem w 0.9.7. Then downgraded to 0.9.6
which worked. Then installed 0.9.7 again and it also worked. Yesterday when installing node red Hacs also created two nested folders: custom components/node red/node red

HACS 1.19.0 and then 1.19.1 and 1.19.2 came out pretty much at the same time - might be related?

I would suspect some HACS issue, since there were not changes to integration between 0.9.6 and 0.9.7, only the underlying goodwe library was updated.

Its a HACS issue. I have same problem with a different integration.
There is no need to install 0.9.6.
Just reinstall 0.9.7. with HACS and restart HA


Subnetwork searching for car takes a lot longer if using multiple vehicles

As you can see: subnetwork searching takes roughly 3h if done with 3 flag encoders, which is 9 (!) times longer than if car search is done alone (20min). Memory shouldn't be an issue, but maybe GC or some tarjan implementation limits for the traversal etc?
2015-09-16 22:45:15,965 [main] INFO  com.graphhopper.GraphHopper - start finding subnetworks, totalMB:73393, usedMB:25026
2015-09-16 23:16:48,988 [main] INFO  com.graphhopper.routing.util.PrepareRoutingSubnetworks - 5241731 subnetworks found for bike2, totalMB:74569, usedMB:36508
2015-09-16 23:42:44,763 [main] INFO  com.graphhopper.routing.util.PrepareRoutingSubnetworks - 4475144 subnetworks found for foot, totalMB:74469, usedMB:45832
2015-09-17 02:39:53,375 [main] INFO  com.graphhopper.routing.util.PrepareRoutingSubnetworks - 16834684 subnetworks found for car, totalMB:74021, usedMB:44128

Numbers: there are roughly 10 times more subnetworks than for foot or bike, and there are 5 times more subnetworks compared to the single flag encoder case with car.
This happens only if mixing with outdoor profiles. If e.g. only car and motorcycle all is fine (~30min)

This is now a lot improved (<15min each) but still graph.optimize() takes ~50min

This gave a hugh speed up if using multiple vehicles (up to 3h per vehicle). Nice!


Build version stack for library

Build a file structure for housing multiple versions of same library.

Old - not needed


fix: blur on popover

What kind of change does this PR introduce?
Fix #286
Does this PR introduce a breaking change?
No

Codecov Report

Merging #287 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #287   +/-   ##
=======================================
  Coverage   67.18%   67.18%
=======================================
  Files          85       85
  Lines         649      649
  Branches      149      149
=======================================
  Hits          436      436
  Misses        213      213

Impacted Files
Coverage Δ

packages/reakit/src/Popover/Popover.tsx
5.35% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b3a78e...076f009. Read the comment docs.


Getting 3D shap_values for an autoencoder model using Kernel Explainer

@slundberg
Hello,
I am using an autoencoder-based unsupervised model for anomaly detection. Input is the 2D vector comprising 22 features at different time stamps (Input shape: for e.g (100, 22)). The autoencoder model summary is attached below.
. In the model, the input shape is reshaped before passing it through the convolution layer. The output of the model thus is (100, 22, 1) if an input vector of (100, 22) is given.
My question is: When I am using Kernel explainer to interpret this autoencoder model, I am getting shap_values shape as (22, 100, 22) for an input of (100, 22)... which I am not understanding.
I mean if I provide 100-row data of 22 features, shouldn't I expect the generated shap values to be of the size (100, 22) instead of (22, 100, 22)? Why are the shape values in 3D?

Hi, did you find any solution?


Refactored logic to allow Sink errors to be reported

Fixes #10269
Motivation
When an exception is thrown inside the sendOutputMessage method of the JavaInstanceRunnable class, since it is inside a CompletableFuture, it gets "swallowed up" and not properly reported. This can result in a scenario in which the Pulsar Function is continuing to run and process messages, but no output is getting generated.
Modifications
Move the call to sendOutputMessage method outside of the CompletableFuture stages, and into the normal execution flow, which allows us to catch any exceptions thrown by that method.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)
If a feature is not applicable for documentation, explain why?  Giving up after multiple attempts to get this to build properly w/o any test failures. FWIW, these same tests NEVER fail when run locally.....not once in 10+ attempts, but here in OSS-land, the opposite is true....0 for 10+.  Can someone please fix this issue, so that the builds are consistent and repeatable across the developer environment and this build environment?


WIP: Tanaya/umbrella

Pull Request Description
What issue does this change request address? (Use "#" before the issue to link it, i.e., #42.)
WIP: UMBRELLA SAMPLING
What are the significant changes in functionality due to this change request?

For Change Control Board: Change Request Review
The following review must be completed by an authorized member of the Change Control Board.

[ ] 1. Review all computer code.
[ ] 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
[ ] 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
[ ] 4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
[ ] 5. If significant functionality is added, there must  be tests added to check this. Tests should cover all possible options.  Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in  XML block, the node <internalParallel> to True.
[ ] 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change.  The requirements and the requirements test shall be in sync.
[ ] 7. The merge request must reference an issue.  If the issue is closed, the issue close checklist shall be done.
[ ] 8. If an analytic test is changed/added is the the analytic documentation updated/added?
[ ] 9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.

@wangcj05 @PaulTalbot-INL do you want me to try to address the comments or you guys are taking care of this (since part of the IES collaboration with GWU)?

Hi @aalfonsi,  @nickblackmore is the student at GWU working on this contract activity, so I think he'll be working on addressing comments. Certainly if there's something of particular note that you're interested in feel free to drop a note!

Hi @aalfonsi, @nickblackmore is the student at GWU working on this contract activity, so I think he'll be working on addressing comments. Certainly if there's something of particular note that you're interested in feel free to drop a note!
@PaulTalbot-INL great.... I am just trying to get rid of all my old branches and realized this PR is still pending... @nickblackmore let me know when and if I can delete the branch.

@aalfonsi Hi, I think that I just need to resolve the conflicts-- I addressed the other requested changes. I've been working on another aspect of the project lately, but I can clean this up in the next couple of days.

Hi @PaulTalbot-INL @nickblackmore, do you have plan to finish this PR?

@PaulTalbot-INL @nickblackmore @wangcj05 Is this going to be finalized? As mentioned in January 2022 I am trying to delete/clean up all my old branch. Thanks.

@PaulTalbot-INL @nickblackmore @wangcj05 Closing this. It is clearly inactive since Jan 2021


Argument list too long for yesod-job-queue Haddocks

runghc: /opt/ghc/7.10.3/bin/ghc: rawSystem: runInteractiveProcess: exec: resource exhausted (Argument list too long)

This is marked in build-constraints so lets close this.


Modify pipeline to handle strandseq files to potentially handle multiple strandseq suffixes

potential that when combining data from different sequencing runs, that they could have different file extensions, eg .txt.gz vs .fastq.gz. and the pipeline cannot currently handle that.
The whole handling of the strandseq suffixes is really bad in general, need a better way of doing that.

conceptually simple solution: modify the pipeline to take a file where every file pair is specified directly. This simplifies the {lib} and {pair} wildcards in the pipeline and shifts some of the work to the config specification.

Current process: Sort files then pair them by order, with a check that they differ only by one character, and that those characters are 1 or 2


Make build workflows fail if test reports not found

This PR updates build workflows to fail if test reports not found.
This insures us against the case when tests don't start, but build passes successfully.
Also, pull script is updated to clean up old files that can shadow newly created types from testing package. Newer version of registerTestTasks() method contains include directives:
filter {
    includeTestsMatching("*Test")
    includeTestsMatching("*Spec")
}

They can also prevent a build from passing successfully without tests, but only when not shadowed by old files.
Thus, it's better to have this protection in two places simultaneously.

@alexander-yevsyukov @armiol Please take a look.


[Enhancement] Print remaining tablets for decommission BE

What type of PR is this：

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr will affect users' behaviors
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto backported to target branch

[x] 2.5
[x] 2.4
[x] 2.3
[x] 2.2

[FE PR Coverage Check]
:heart_eyes: pass : 2 / 2 (100.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/alter/SystemHandler.java
2
2
100.00%
[]

run starrocks_admit_test

@mergify backport branch-2.2 branch-2.3 branch-2.4 branch-2.5


[CP] (beta) have Java.version return null if java --version fails or cannot be run

Issue Link
https://github.com/flutter/flutter/issues/139180
Target
beta
PR Link
https://github.com/flutter/flutter/pull/139838
Changelog Description
Fixes flutter tool crash that can occur during flutter create when Flutter is unable to run Java.
Impacted Users
All Flutter devs that create Android-enabled projects
Impact Description
flutter create crashes with a vague error message stating that <some path here>/java could not be found.
Workaround
Users will have to troubleshoot their Java installation. They can start with flutter doctor -v, which will print a validation failure when java cannot be run.
Risk
low
Test Coverage
yes
Validation Steps
No response

cc @christopherfujino

Released in 3.18.0-0.2.pre


Dataset v2.0

What this does
This PR introduces a new format for LeRobotDataset, which is accompanied by a new file structure. As these changes are not backward compatible, we increase CODEBASE_VERSION from v1.6 to v2.0.
What do I need to do?
If you already pushed a dataset using v1.6 of our codebase, you can use the conversion script lerobot/common/datasets/v2/convert_dataset_v1_to_v2.py to convert it to the new format.
You will be asked to enter a prompt describing the task performed in the dataset.
Examples for single-task dataset:
python lerobot/common/datasets/v2/convert_dataset_v1_to_v2.py \
    --repo-id lerobot/aloha_sim_insertion_human_image \
    --task "Insert the peg into the socket."

If you recorded your dataset with one of the manipulator robots currently supported in LeRobot (or your own implementation), you can provide its configuration path to add the motor names and robot type to the dataset info using the --robot-config option:
python lerobot/common/datasets/v2/convert_dataset_v1_to_v2.py \
    --repo-id aliberts/koch_tutorial \
    --task "Pick the Lego block and drop it in the box on the right." \
    --robot-config lerobot/configs/robot/koch.yaml

For the more complicated cases of one task per episode or multiple tasks per episodes, please refer to the documentation in that script.
Motivation
Current implementation of our LeRobotDataset suffers from a few shortcomings which make it not easy to use on some aspects. Specifically:

The structure of the files does not accurately reflect the data structure. Our datasets are structured by episodes, which contrasts with a typical ML scenarios with train/val/test splits (although these concepts can still be relevant here). This makes it hard to easily select a subset of episodes from a dataset since the whole dataset has to be downloaded/loaded. Related: #440
Due to the hub's limitations, one can not push a dataset with — at most - more than 10k episodes (less if there are multiple cameras).
The format is not transparent to the user: in order to get information about the content of a dataset, current options are limited to download the entire dataset and inspect it with a custom script, or try to visualize it using our visualization tool. Related: #383
The default file cache system used by datasets and huggingface_hub makes it not convenient to create datasets locally (with recording). In order to use the newly created files on disk, these libraries check if those files are present in the cache (which they won't) and if not, will download them even though they may already be on disk.
Some file format used are too framework specific for this format to be more universal (e.g. .safetensors)
The dataset viewer on the hub is not compatible with our datasets due to VideoFrame not yet being integrated into datasets.
The current implementation lacks support for future features that we may want to add such as:

Task-tokens-conditioned training
Multirobot policies
Depth images (Related: #435)

Changes
Some of the biggest change come from the new file structure and their content:
  .
  ├── data
- │   ├── train-00000-of-0001.parquet
+ │   ├── chunk-000
+ │   │   ├── episode_000000.parquet
+ │   │   ├── episode_000001.parquet
+ │   │   ├── episode_000002.parquet
+ │   │   └── ...
+ │   ├── chunk-001
+ │   │   ├── episode_001000.parquet
+ │   │   ├── episode_001001.parquet
+ │   │   ├── episode_001002.parquet
+ │   │   └── ...
+ │   └── ...
- ├── meta_data
+ ├── meta
- │   ├── episode_data_index.safetensors
+ │   ├── episodes.jsonl
  │   ├── info.json
+ │   ├── stats.json
- │   ├── stats.safetensors
+ │   └── tasks.jsonl
  └── videos
+     ├── chunk-000
+     │   ├── observation.images.laptop
      │   │   ├── episode_000000.mp4
      │   │   ├── episode_000001.mp4
      │   │   ├── episode_000002.mp4
      │   │   └── ...
+     │   ├── observation.images.phone
      │   │   ├── episode_000000.mp4
      │   │   ├── episode_000001.mp4
      │   │   ├── episode_000002.mp4
      │   │   └── ...
+     ├── chunk-001
      └── ...

Note that this file-based structure is designed to be as versatile as possible. The parquet files are split by episodes (this was already the case for videos) which allows a much more granular control over which episodes one wants to use and download. The structure of the dataset is entirely described in the info.json file, which can be easily downloaded or viewed directly on the hub before downloading any actual data. The type of files used are very simple and do not need complex tools to be read, it only uses .parquet, .json, .jsonl and .mp4 files (.md for the README).
Added

A LeRobotDataset can now be called with an episodes argument (e.g. episodes=[1, 10, 12, 40]) to select a specific subset of episodes by their episode_index. By doing so, only the files corresponding to these episodes will be downloaded (if they're not already on disk). In that case, the hf_dataset attribute will only contain data from these episodes, as well as the episode_data_index.
Dataset metadata logic is now handled by the LeRobotDatasetMetadata class. This allows to get info about a dataset before loading the data. For example, you could do this:

# Fetch metadata from the hub
metadata = LeRobotDatasetMetadata("lerobot/pusht")

# Calculate train and val episodes
total_episodes = metadata.total_episodes
episodes = list(range(metadata.total_episodes))
num_train_episodes = math.floor(total_episodes * 90 / 100)
train_episodes = episodes[:num_train_episodes]
val_episodes = episodes[num_train_episodes:]

# Load train an val datasets
train_dataset = LeRobotDataset("lerobot/pusht", episodes=train_episodes)
val_dataset = LeRobotDataset("lerobot/pusht", episodes=val_episodes)

Tasks as natural language prompts are now in every datasets and is needed to create one. Every single task of a dataset is listed in the tasks.json mapped to its task_index which is what's actually stored in parquet files. Using the api, they can be accessed either with dataset.tasks to get that mapping or through dataset.episode_dict[episode_index]["tasks"] if you're only interested in a particular episode.
Various information about the structure of the dataset have been added and is now centralized in the info.json (keys, shapes, number of episodes, etc.). It serves as a source of truth for what's inside the dataset.
episodes.jsonl contains per-episode information (episode_index, tasks in natural language and episode lengths). This is accessed through the episode_dict attribute in the api.
LeRobotDataset.create() allows to create a new dataset from scratch, either for recording data or for porting an existing dataset to the LeRobotDataset format. To that end, new methods are added:

start_image_writter(): This instantiates an ImageWriter in the image_writer attribute to write images asynchonously during data recording. This is automatically called during LeRobotDataset.create() if specified in the arguments.
stop_image_writter(): This is to properly stop and remove the ImageWriter from the dataset's attributes. Importantly: if the image_writer has been set to a multiprocess ImageWriter, this needs to be called first if you want to pass this dataset into a parallelized DataLoader as the ImageWriter class is not pickleable (required for objects to be transfered between processes). This is not needed when instantiating a dataset with __init__ as the image_writer then is not created.
add_frame(): Adds a single timestamp data frame to the episode_buffer, which keep data in memory temporarily. Note: this will be merged with the DataBuffer from #445 in a subsequent PR.
add_episode(): Saves the content of the episode_buffer to disk and updates metadata for them to be in sync with the contents of the files. This method expects a task argument as a string prompt in natural language describing the task performed in the episode. Videos from that episode can optionally be encoded during this phase but it's not mandatory and can be done later in order to give more flexibility on when to do that.
consolidate(): This will encode videos that have not yet been encoded, clean up the temporary image files, compute dataset statistics, check timestamps are in sync with the fps and perform additional sanity checks in the dataset. It needs to be done before uploading the dataset to the hub with push_to_hub().
clear_episode_buffer(): This can be used to reset the episode_buffer (e.g. to discard data from a current recording).

Changed

The logic for checking timestamps and delta_timestamps sync is taken outside of the __get_item__() and is now done during __init__ or consolidate. This has the benefit of both saving computation during the __get_item__() as well as knowing immediately if there are sync issues with the timestamps.
The paths for the parquet and video files are now embedded in the info.json to allow flexibility and to easily split chunks of files between directories to avoid the hub's limit of files (10k) per folder.
We now store every datasets (created or downloaded) in ~/.cache/huggingface/lerobot by default. Changing root or setting the LEROBOT_HOME env variable allows to change that location. Every call to the huggingface_hub download functions like snapshot_download or hf_hub_download use the local_dir argument to that location so that files are not duplicated in cache and to solve the issue of having to download again files already present on disk.
Refactored the image writing code from populate_dataset.py into an ImageWriter class.
stats.safetensors is now stats.json (the content remains the same but it's unflattened).
episode_data_index.safetensors is removed but the episode_data_index is still in the api to map episode_index to indices.

Performance
In the nominal case (no delta_timestamp), LeRobotDataset.__get_item__() is on par with the previous version, sometimes slightly improved but still in the same ballpark generally.
__get_item__() call time in seconds (average on 10k iterations):
repo_id                                 | v1.6   | v2.0
--------------------------------------- | ------ | ------
lerobot/aloha_sim_insertion_human_image | 0.0036 | 0.0037
lerobot/aloha_sim_insertion_human       | 0.0029 | 0.0027
lerobot/pusht_image                     | 0.0003 | 0.0003
lerobot/pusht                           | 0.0011 | 0.0009
aliberts/koch_tutorial                  | 0.0111 | 0.0106
lerobot/aloha_mobile_cabinet            | 0.0104 | 0.0101

Benchmarking code
from pathlib import Path
import time
import torch
from lerobot.common.datasets.lerobot_dataset import CODEBASE_VERSION, LeRobotDataset

repo_ids = [
    "lerobot/aloha_sim_insertion_human_image",
    "lerobot/aloha_sim_insertion_human",
    "lerobot/pusht_image",
    "lerobot/pusht",
    "aliberts/koch_tutorial",
    "lerobot/aloha_mobile_cabinet",
]
num_iterations = 10000
logfile = Path(f"perf_log_{CODEBASE_VERSION}_{num_iterations}.txt")
with open(logfile, "a") as file:
    file.write(f"__get_item__() call time in seconds (average on {num_iterations} iterations)\n\n")
    file.write(f"repo_id                                 | {CODEBASE_VERSION}  \n")
    file.write("--------------------------------------- | ------\n")

for repo_id in repo_ids:
    dataset = LeRobotDataset(repo_id=repo_id)
    durations = []
    for i in range(num_iterations):
        start = time.perf_counter()
        item = dataset[i]
        duration = time.perf_counter() - start
        durations.append(duration)

    avg_duration = torch.Tensor(durations).mean()
    results = f"{repo_id} | {avg_duration:.4f}s"
    print(results)
    with open(logfile, "a") as file:
        file.write(results + "\n")

Using delta_timestamps, results are more diverse depending on the dataset but still remain in the same ballpark.
__get_item__() call time in seconds (average on 10k iterations), delta_timestamps=[-1/fps, 0, 1/fps]:
repo_id                                 | v1.6   | v2.0
--------------------------------------- | ------ | ------
lerobot/aloha_sim_insertion_human_image | 0.0176 | 0.0160
lerobot/aloha_sim_insertion_human       | 0.0073 | 0.0068
lerobot/pusht_image                     | 0.0024 | 0.0032
lerobot/pusht                           | 0.0028 | 0.0043
aliberts/koch_tutorial                  | 0.0200 | 0.0184
lerobot/aloha_mobile_cabinet            | 0.0224 | 0.0181

Benchmarking code (delta_timestamps)
from pathlib import Path
import time
import torch
from lerobot.common.datasets.lerobot_dataset import CODEBASE_VERSION, LeRobotDataset

repo_ids = [
    "lerobot/aloha_sim_insertion_human_image",
    "lerobot/aloha_sim_insertion_human",
    "lerobot/pusht_image",
    "lerobot/pusht",
    "aliberts/koch_tutorial",
    "lerobot/aloha_mobile_cabinet",
]
num_iterations = 10000
logfile = Path(f"perf_log_{CODEBASE_VERSION}_{num_iterations}.txt")
with open(logfile, "a") as file:
    file.write(f"__get_item__() call time in seconds (average on {num_iterations} iterations)\n\n")
    file.write(f"repo_id                                 | {CODEBASE_VERSION}  \n")
    file.write("--------------------------------------- | ------\n")

for repo_id in repo_ids:
    dataset = LeRobotDataset(repo_id=repo_id)
    fps = dataset.fps
    keys = ["observation.state", *dataset.camera_keys]
    delta_timestamps = {key: [-1/fps, 0, 1/fps] for key in keys}
    dataset = LeRobotDataset(repo_id=repo_id, delta_timestamps=delta_timestamps)
    durations = []
    for i in range(num_iterations):
        start = time.perf_counter()
        item = dataset[i]
        duration = time.perf_counter() - start
        durations.append(duration)

    del dataset
    avg_duration = torch.Tensor(durations).mean()
    results = f"{repo_id} | {avg_duration:.4f}s"
    print(results)
    with open(logfile, "a") as file:
        file.write(results + "\n")

Fixes

Fix a bug in load_previous_and_future_frames which didn't actually raise an error when the requested timestamps from delta_timestamps did not correspond to actual timestamps in the dataset.
Various fixes on the datasets have been made:

Some tasks already present in some datasets contained strings which were not part of the task (e.g. "tf.Tensor(b'Do something', shape=(), dtype=string)")
Some video files were not properly tracked by git lfs
Some datasets present a mismatch between the number of episodes in their parquet and the number of video files. This is being investigated [TODO]

lerobot/aloha_mobile_shrimp
lerobot/aloha_static_battery
lerobot/aloha_static_fork_pick_up
lerobot/aloha_static_thread_velcro
lerobot/uiuc_d3field

lerobot/viola is missing video keys [TODO]

How it was tested

Adds tests/fixtures/ in which fixtures and fixtures factories have been added to simplify writing/adding tests. These factories allow the flexibility to create partially mocked objects on the fly to be used in tests, while not relying on other components of the codebase that are not meant to be tested in a particular test (e.g. initializing a dataset using hydra).
Adds tests/test_image_writer.py
Adds tests/test_delta_timestamps.py
Deactivates a bunch of tests which will need to be redesigned and simplified in further PRs.

How to checkout & try? (for the reviewer)
Use an existing dataset:
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

REPO_ID = "lerobot/aloha_sim_insertion_human"  # try with '_image' as well

delta_timestamps = {
    "observation.images.top": [-1, -1/50, 0, 25/50],
    "observation.state": [-1, -1/50, 0, 25/50],
}
dataset = LeRobotDataset(repo_id=REPO_ID, delta_timestamps=delta_timestamps)

Try out the new feature to select / download specific episodes:
dataset = LeRobotDataset(repo_id=REPO_ID, episodes=[1, 10, 12, 40])

You can also create a new dataset:
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

REPO_ID = "your_hf_username/test_v2"

new_dataset = LeRobotDataset.create(
    repo_id=REPO_ID,
    fps=30,
    robot=robot,
    image_writer_threads_per_camera=1,
)

# TODO
frame = {
    ...
}
new_dataset.add_frame(frame)
new_dataset.add_episode(task="Do something")
new_dataset.consolidate()

Hey folks! Awesome work here. I was wondering what the timeline is for merging Dataset 2.0?
The reason I ask is I'm about to start working on adding support for Elephant Robotics MyArm M&C, and I'm not sure if I should target dataset 1.0 or 2.0 initially.

Hey folks! Awesome work here. I was wondering what the timeline is for merging Dataset 2.0?

@apockill Thank you for your support! The team took some time off and we're just getting back. Hopefully this will be merged very soon (we mainly need to update some more tests now).

The reason I ask is I'm about to start working on adding support for Elephant Robotics MyArm M&C, and I'm not sure if I should target dataset 1.0 or 2.0 initially.

We will refactor how robot classes are structured soon but for now this PR shouldn't have a big impact on adding support for a new robot. The only thing that's being added to robot classes are the motor_features and camera_features for now (this interface might change in the next PR).

TODO after merging: https://github.com/huggingface/lerobot/pull/485


create subquery always add at the end

to reproduce:

create two top queries (1 and 2)
create a subquery of the first one
it will appear as a sub query of the second one (at the bottom of the list)

The sub query should be inserted as a child of the query that created it

Yes you are right. The query it will be shown at the bottom of the list but is not a child of the second one. Is a child of the first one. This a problem only of the visualisation because the created queries always go to the bottom of the list.
We discussed this when I was in Bolivia and you told me to do it like that: to put the queries always to the bottom of the list. But I agree that this is confusing and that we have to change it


Lambdaworks pow method

This method was created while developing but was not used in the actual feature. We decided to create a new pr with this code if at some point is needed. depends on #69

Closes as it was implemented in #126


Create Teams High Contrast Theme Variables

We need to create high contrast theme variables for all components.  We will first do this explicitly in siteVariables and each component's variables file.  Afterward, we can consider an abstraction for applying "color schemes" to components.  Let's wait on that until we can see the pattern that emerges.
Please link you PR after the item you are completing below.
Doc Site

[ ] ComponentExample.tsx background should change on theme switch
[ ] (Low priority) theme switcher for each Component Example

Components

[ ] Accordion
[ ] Attachment
[ ] Avatar
[ ] Button #336 @bcalvery
[ ] Chat
[x] Divider @codepretty #338
[ ] Grid
[ ] Header
[ ] Icon
[ ] Image
[ ] Input
[ ] ItemLayout
[ ] Label
[ ] Layout
[ ] List
[ ] Menu
[ ] Popup
[ ] Portal
[ ] RadioGroup
[ ] Segment
[ ] Status
[ ] Text

fyi @levithomason , @jurokapsiar , @hughreeling , @codepretty , @notandrew

Pretty much done.


remove extra.ops and BufferOps from test_linearizer.py

BufferOps was present in a test tagged with expectedFailure and slipped through

cool, can you also delete extra/ops and the 3 lines in kernel in this diff?

fyi external/external_test_nv.py is really out of date -- they were broken on multiple levels before I even attempted to port them to UOp AST.

@nimlgen fyi, external_test_nv.py doesn't run on CI, is this expected?

@greg-niemeyer as long as the rest of CI is green we can move forward for now.

chenyu also moved the sops.gz dataset to UOp ast #6238,
wanna see if extra/optimization/helpers.py can be cleaned up?

👍 ill take a look


Add tests that ensure all struct members fully initialize to zero

Not sure the best way to do this since sometimes we can just "get lucky" and struct members will initialize to zero even though they do not enforce that this must occur.

Replaced by https://github.com/AztecProtocol/aztec-packages/issues/1046


editorconfig, prettier, eslint

adds editorconfig, eslint, and prettier
disables no-undefined and default-case eslint rules, as they are not compatible with our typescript conventions
ignores react/no-string-refs as that will need to be fixed properly at a later date

No idea why the build is still failing for https://github.com/textileio/photos/pull/1092/commits/8bf0f072dd5bfd5d2884e27309b8180774b507c7

argh, out of date again

remerged, hopefully build passses

no idea why the build keeps failing

Looks like that call to requireNativeComponent that we looked at yesterday:
yarn run v1.13.0
$ tsc -p . --noEmit
App/Components/TextileImage.tsx:59:26 - error TS2554: Expected 1 arguments, but got 2.

 59 const TextileImageView = requireNativeComponent(
                             ~~~~~~~~~~~~~~~~~~~~~~~
 60   'TextileImageView',
    ~~~~~~~~~~~~~~~~~~~~~
...
 62   TextileImage
    ~~~~~~~~~~~~~~
 63 )

Didn't we put a @ts-ignore in there?

@balupton I just tested that method with one argument and it works fine:
const TextileImageView = requireNativeComponent('TextileImageView')

Just update it and remove the @ts-ignore

Didn't we put a @ts-ignore in there?

yup


[HOLD for payment 2023-01-05] [$4000] Emoji Picker list jumps while scrolling through arrow keys reported by @parasharrajat

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Login to the app on web.
Navigate to any report.
Open the Emoji Picker.
User down arrow key to scroll the list until you reach the section Travel & Places.
Now keep on scrolling down via arrow key .
Notice that list jumps and highlighted emoji is out of focus.

Expected result:
Highlighted emoji should remain in focus and list does not jump positions.
Actual:
Emoji List jump positions and highlighted emoji goes out of focus.
Workaround:
unknown
Platform:

Where is this issue occurring?

Web
Desktop App

Version Number:  1.2.28-0
Reproducible in staging?: y
Reproducible in production?: y
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:
https://user-images.githubusercontent.com/43996225/202214414-eb1f460c-a2bb-4a90-b5fb-9eeac0a76383.mp4
https://user-images.githubusercontent.com/43996225/202214442-562dd0c3-0523-4b0f-94a6-acbc040e3391.mp4
Expensify/Expensify Issue URL:
Issue reported by: @parasharrajat
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1668550863631129
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~011a004a1208ec7fec
Upwork Job ID:<PHONE_NUMBER>393093632
Last Price Increase: 2022-12-15

Taking this one given that I'm on the emoji tracking issue. Not treating this as an accessibility issue.

I'm wavering a bit based on my last comment. This bug seems to only affect that one section right? If this is a clear and quick fix, then I'm more amenable to addressing it now, even though we're not focused on keyboard/accessibility at the moment. Getting another opinion from engineering.

@Gonals Please see above for a second opinion. Thanks!

This issue does not come under A11y/navigation issues. This is a bug with emoji list viewing. This does not affect only one section. It might affect more sections we add them. It is just a coincidence that the section lies where the bug occurs. The main issue is with the Flat list scrolling/rendering. Due to virtual scrolling, something bad is happening with our scrolling logic.

Ok well I think that's also my point here. Is this the only place this issue is happening with FlatList. I'm less and less interested is solving issues like this one as if this is the only place this is happening, and I think whatever solution we come up with, let's make sure that we aren't working on this same thing somewhere else in the repo in a silo.

@Gonals Please see above for a second opinion. Thanks!

It took me quite a bit to actually see the issue 😅.
If the behavior is only happening in this specific section of the emoji list, I honestly don't think it is a bit deal. If we have a quick fix, great, otherwise... 🤷.
If we start seeing scrolling issues with our flatlists elsewhere, then we'd certainly need to tackle it with more urgency

Why do we think that this is not a  big deal? IMO, It is affecting user experience, and thus should be fixed.

Still feel the same as last week, though open to differing opinions.

Make sense https://github.com/Expensify/App/issues/12772#issuecomment-1320579574 :+1:

Given that we've decided that WishList is quite far away, I'm open to re-evaluating this one. @stitesExpensify curious to get your take on this issue since you're also working on the larger emoji initiative. Maybe we even take this internal via either you or @Gonals to see if a quick fix exists?

I am not able to reproduce this issue:
https://user-images.githubusercontent.com/42391420/205135213-e2ace602-ee8e-448f-bf69-3eb17927e79a.mp4

I think you have to scroll a little further in the travel & places section

🤦 yep you're right

Yeah it's real tough to reproduce. It's a bug, though I do think it's pretty low value. Removing the FlatList hold, though I concern I have is whether the solution is straightforward around the scrolling logic. @stitesExpensify @Gonals shall we open up to External in any case?

Let me look into it a little deeper and see if it's easy. If I don't figure it out quickly I'll open it up

I think that the problem is in this method, but i'm not positive. I'm going to make this external since I've already got 2 bugs I'm working on personally.

Sorry @shawnborton misclick. Unassigning you

Looking for proposals

Doubling the price of this issue to try and get more interest.

Proposal
Issue: I am seeing that on scroll the FlatList initially removes the items from the dom and applies that height to the div and after some more scrolling, it removes the height also from the div which seems to be causing this inconsistent scroll issue. This video helps to understand.
https://user-images.githubusercontent.com/14358475/206183860-8ba79691-04e7-477b-a188-d530efc69b0c.mp4
To solve the specific issue, we can pass the max size to maxToRenderPerBatch within VirtualizedList/FlatList which won't take advantage of the virtualization and won't remove the div from the dom.
https://github.com/Expensify/App/blob/cb9a54138bafac9d39598e492669cd01a9b31af5/src/components/EmojiPicker/EmojiPickerMenu/index.js#L496
maxToRenderPerBatch={this.emojis.length}
https://user-images.githubusercontent.com/14358475/206185097-63693ffd-d46e-4354-8e0e-4637bb69fefd.mp4

@Ollyws
Your getItemLayout solution not working as expected when using an upward Arrow after scrolling down.
windowSize seems to work well, not sure about the impact we might have on performance/storage based on the con here @Gonals Any thought on this?
@Pujan92

To solve the specific issue, we can pass the max size to maxToRenderPerBatch within VirtualizedList/FlatList which won't take advantage of the virtualization and won't remove the div from the dom.

We don't want to disable virtualization here.
So no.
Also I am also thinking solution should be fix this method mostly this method but sure either. B

I think for the getItemLayout all the items need to be of the same size to make it work, how about making the header height 32px too?
https://github.com/Expensify/App/blob/699d46986253282403ecb4a0b74a82643a57c8be/src/styles/styles.js#L1437
Update to
height: 32,
display: 'flex',
alignItems: 'center',

No, we won't be doing that mostly as it is a design requirement! cc: @Gonals

Thanks for the great discussion! Still working through proposals.

Proposal
Solution 1
We can add the getItemLayout prop to correctly calculate the offset by calculating the exact height of the elements that are before the current index.
This solution also takes care of this issue

Sometimes it partially is not visible. Look at this case, scroll to the end of the list, and use the up arrow mostly it's hidden.

getItemLayout={(list, index) => {
  const passedHeaders = _.filter(this.state.headerIndices, i => i < index).length;
  const passedNonHeaderRows = index - passedHeaders;
  const offset = (CONST.EMOJI_PICKER_ITEM_HEIGHT * passedNonHeaderRows) + (passedHeaders * CONST.EMOJI_PICKER_HEADER_HEIGHT);
  return (
      {length: CONST.EMOJI_PICKER_ITEM_HEIGHT, offset, index}
  );
}}

Solution 2
We just add the initialNumToRender={this.state.filteredEmojis.length} to the flatlist, this fixes the issue as it does not unmount any items even after we have gone past the windowSize, thus our contentSize & offset don't change unexpectedly.
Con: Not super efficient to render all items at once.
Solution 3
Combine both solution 1 & 2 to be at least somewhat efficient than solution 2 by providing getItemLayout
@Santhosh-Sellavel @Gonals @JmillsExpensify

Solution 1
We can add the getItemLayout prop to correctly calculate the offset by calculating the exact height of the elements that are before the current index.
getItemLayout={(list, index) => {
  const passedHeaders = _.filter(this.state.headerIndices, i => i < index).length;
  const passedNonHeaderRows = index - passedHeaders;
  const offset = (CONST.EMOJI_PICKER_ITEM_HEIGHT * passedNonHeaderRows) + (passedHeaders * CONST.EMOJI_PICKER_HEADER_HEIGHT);
  return (
      {length: CONST.EMOJI_PICKER_ITEM_HEIGHT, offset, index}
  );
}}

with this, the issue partially is not visible seems to still persist.

with this, the issue partially is not visible seems to still persist.

Attached the video above, I cannot seem to reproduce

@Pujan92 Attached the video above, I cannot seem to reproduce

Plz try to go top from bottom, you will see that for multiple sections it happens.

@Pujan92 Attached the video above, I cannot seem to reproduce

Plz try to go top from bottom, you will see that for multiple sections it happens.

Ohh yes found it, that's because the CONST.EMOJI_PICKER_HEADER_HEIGHT is incorrect, in the code it is defined as 38 but in reality it is 37, so we should change that as that's causing incorrect calculations in getItemLayout

Ohh yes found it, that's because the CONST.EMOJI_PICKER_HEADER_HEIGHT value is incorrect, in the code it is defined as 38 but in reality it is 37, so we should change that as that's causing incorrect calculations in getItemLayout

Yes, now while going bottom the issue occurs after 'Travel & Places' section. As per my understanding, the issue exists when FlatList starts creating dynamic height div onscroll.

@Santhosh-Sellavel Thoughts on the more recent convo?

Proposal 4 - Further Optimisations
@Santhosh-Sellavel @JmillsExpensify This fixes all the above mentioned bugs & adds performance improvements.
Solution
We calculate the row sizes for the full list only once at the start of the component & cache that in a map on which we can do lookups by row index in getItemLayout. Then we just return the normal getItemLayout values when the list is in search mode as we do not show the headers while searching which makes all row items the same height so we don't have to do any complex computations.
Add this in the constructor before this.currentScrollOffset = 0;
this.getEmojiListItemLayout = this.getEmojiListItemLayout.bind(this);
this.rowSizes = this.calculateRowSizes(this.emojis);

Add this after setupEventHandlers
/**
 * Returns the length, offset and index for a renderItem row.
 * Uses the cached size if the list is not in search mode.
 *
 * @param {Array} data
 * @param {Number} index
 *
 * @return {Object}
 */
getEmojiListItemLayout(data, index) {
    if (this.state.filteredEmojis.length !== this.emojis.length) {
        // just return the item height because headers are not shown during search
        return {
            length: CONST.EMOJI_PICKER_ITEM_HEIGHT,
            offset: CONST.EMOJI_PICKER_ITEM_HEIGHT * index,
            index,
        };
    }
    return this.rowSizes[index];
}

/**
 * Calculates all emoji row sizes and returns a map of index to row sizes (length, offset & index).
 *
 * @param {Array} emojiList
 * @return {Object}
 */
calculateRowSizes(emojiList) {
    let offset = 0;
    const sizesByIndex = {};
    _.each(emojiList, (emoji, emojiIndex) => {
        const columnIndex = emojiIndex % CONST.EMOJI_NUM_PER_ROW;

        // Calculate size only once for each row so ignore every column after first emoji in the row.
        if (columnIndex > 0) { return; }
        const index = (emojiIndex - columnIndex) / CONST.EMOJI_NUM_PER_ROW;
        const length = emoji.header ? CONST.EMOJI_PICKER_HEADER_HEIGHT : CONST.EMOJI_PICKER_ITEM_HEIGHT;
        sizesByIndex[index] = {length, offset, index};
        offset += length;
    });
    return sizesByIndex;
}

Add this in the FlatList props getItemLayout={this.getEmojiListItemLayout}
Also I don't think we need to change the header size to 37 anymore as it now comes up as 38, tested with the above changes.

@JmillsExpensify Will review this tonight, but both look little over-engineered

Proposal
diff --git a/src/CONST.js b/src/CONST.js
index 935ce523f7..809c1ef3d7 100755
--- a/src/CONST.js
+++ b/src/CONST.js
@@ -497,7 +497,7 @@ const CONST = {
     },
     NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT: 298,
     EMOJI_PICKER_ITEM_HEIGHT: 32,
-    EMOJI_PICKER_HEADER_HEIGHT: 38,
+    EMOJI_PICKER_HEADER_HEIGHT: 37,
     COMPOSER_MAX_HEIGHT: 125,
     CHAT_FOOTER_MIN_HEIGHT: 65,
     CHAT_SKELETON_VIEW: {
diff --git a/src/components/EmojiPicker/EmojiPickerMenu/index.js b/src/components/EmojiPicker/EmojiPickerMenu/index.js
index 2d97a24439..6073e17aab 100755
--- a/src/components/EmojiPicker/EmojiPickerMenu/index.js
+++ b/src/components/EmojiPicker/EmojiPickerMenu/index.js
@@ -333,21 +333,19 @@ class EmojiPickerMenu extends Component {
         }

         // Calculate the scroll offset at the bottom of the currently highlighted emoji
-        // (subtract numHeaders because the highlightedIndex includes them, and add 1 to include the current row)
-        const numEmojiRows = (Math.floor(this.state.highlightedIndex / this.numColumns) - numHeaders) + 1;
+        const numEmojiRows = Math.floor(this.state.highlightedIndex / this.numColumns) - numHeaders;

         // The scroll offsets at the top and bottom of the highlighted emoji
-        const offsetAtEmojiBottom = ((numHeaders) * CONST.EMOJI_PICKER_HEADER_HEIGHT)
+        const offsetAtEmojiTop = ((numHeaders) * CONST.EMOJI_PICKER_HEADER_HEIGHT)
             + (numEmojiRows * CONST.EMOJI_PICKER_ITEM_HEIGHT);
-        const offsetAtEmojiTop = offsetAtEmojiBottom - CONST.EMOJI_PICKER_ITEM_HEIGHT;
+        const offsetAtEmojiBottom = offsetAtEmojiTop + CONST.EMOJI_PICKER_ITEM_HEIGHT;

         // Scroll to fit the entire highlighted emoji into the window if we need to
         let targetOffset = this.currentScrollOffset;
         if (offsetAtEmojiBottom - this.currentScrollOffset >= CONST.NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT) {
             targetOffset = offsetAtEmojiBottom - CONST.NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT;
-        } else if (offsetAtEmojiTop - CONST.EMOJI_PICKER_HEADER_HEIGHT <= this.currentScrollOffset) {
-            // There is always a sticky header on the top, subtract the EMOJI_PICKER_HEADER_HEIGHT from offsetAtEmojiTop to get the correct scroll position.
-            targetOffset = offsetAtEmojiTop - CONST.EMOJI_PICKER_HEADER_HEIGHT;
+        } else if (offsetAtEmojiTop <= this.currentScrollOffset) {
+            targetOffset = offsetAtEmojiTop;
         }
         if (targetOffset !== this.currentScrollOffset) {
             // Disable pointer events so that onHover doesn't get triggered when the items move while we're scrolling
@@ -506,7 +504,6 @@ class EmojiPickerMenu extends Component {
                             extraData={
                               [this.state.filteredEmojis, this.state.highlightedIndex, this.props.preferredSkinTone]
                             }
-                            stickyHeaderIndices={this.state.headerIndices}
                             onScroll={e => this.currentScrollOffset = e.nativeEvent.contentOffset.y}
                         />
                     )}

Details
The root issue is caused by the use of sticky headers and the incorrect EMOJI_PICKER_HEADER_HEIGHT value

EMOJI_PICKER_HEADER_HEIGHT
Before the use of ExpensifyNeue-Regular font the height was actually 38. But now it's 37.
Sticky headers
Sticky headers make scrolling buggy.
In my proposal, I removed the use of sticky headers, so I'm not actually fixing the root issue but rather avoiding it. But still I think this is an acceptable approach in our case and here is why:

Sticky headers is a feature that nobody asked for. I have checked few social networks and they don't have sticky headers on their emoji picker, so we are not missing a feature that a user may be expecting.
No sticky headers means more space for the emoji, and from design/ux perspective this may be actually better
This is the cleanest solution so far

@s77rt Actually, removing the stickyHeaderIndices is enough solving this GH issue. But I'm not sure they want to remove a feature because of a bug.

@Santhosh-Sellavel @Gonals I wanted to give you both a heads up that I'm OOO this week. I intend to check in every couple of days, so if you need action on something faster, please either take it on or float the task in #expensify-open-source or #bug-zero. Thanks! 🙌🏼

@Gonals
I Like @Ollyws proposal here works well for me in all cases, which I feel is the first complete proposal here, all other proposal looks similar.
We have a different approach from @s77rt suggesting removing the sticky header here. I'm not sure we would do that, so let's confirm that with @Expensify/design here

I think we should keep the sticky headers personally.

@Santhosh-Sellavel @Gonals Does this solution not work? And isn't it optimised and better for performance than the other proposal?

Yes, @priyeshshah11!

I think we should keep the sticky headers personally.

Do we need to keep the height of the header to 37px only or are we ok to change it to 32px?

I think so, can you share screenshots though? Also, why is 32px better than 36px?

As the issue persists due to the dynamic height calculation(37px for header & 32px for emojis), so 32px makes it of the same height and removes the inconsistency which happens due to the diff heights.
https://user-images.githubusercontent.com/14358475/207641095-15818c17-04f5-4f08-b94f-74f8f7c1196c.mp4
diff --git a/src/CONST.js b/src/CONST.js
index ff7e71b49..522cfaca9 100755
--- a/src/CONST.js
+++ b/src/CONST.js
@@ -495,11 +495,11 @@ const CONST = {
     ADD_PAYMENT_MENU_POSITION_X: 356,
     EMOJI_PICKER_SIZE: {
         WIDTH: 320,
-        HEIGHT: 400,
+        HEIGHT: 390,
     },
-    NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT: 298,
+    NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT: 288,
     EMOJI_PICKER_ITEM_HEIGHT: 32,
-    EMOJI_PICKER_HEADER_HEIGHT: 38,
+    EMOJI_PICKER_HEADER_HEIGHT: 32,
     COMPOSER_MAX_HEIGHT: 125,
     CHAT_FOOTER_MIN_HEIGHT: 65,
     CHAT_SKELETON_VIEW: {
diff --git a/src/components/EmojiPicker/EmojiPickerMenu/index.js b/src/components/EmojiPicker/EmojiPickerMenu/index.js
index 2d97a2443..fa90bdd25 100755
--- a/src/components/EmojiPicker/EmojiPickerMenu/index.js
+++ b/src/components/EmojiPicker/EmojiPickerMenu/index.js
@@ -508,6 +508,9 @@ class EmojiPickerMenu extends Component {
                             }
                             stickyHeaderIndices={this.state.headerIndices}
                             onScroll={e => this.currentScrollOffset = e.nativeEvent.contentOffset.y}
+                            getItemLayout={(data, index) => (
+                                {length: CONST.EMOJI_PICKER_ITEM_HEIGHT, offset: CONST.EMOJI_PICKER_ITEM_HEIGHT * index, index}
+                            )}
                         />
                     )}
                 <EmojiSkinToneList

Got it, thanks for clarifying. That looks good and makes sense to me!

@Santhosh-Sellavel can you plz check and review my last comment? It seems @shawnborton is fine with the height change of header to 32px.

Thanks Shawn, @Pujan92 I'll check test the behaviour and let you know!

As the issue persists due to the dynamic height calculation(37px for header & 32px for emojis), so 32px makes it of the same height and removes the inconsistency which happens due to the diff heights.

@Pujan92 I don't think changing the heights of all these elements is really necessary here, this infact does more or same calculations than the above solutions which only calculates the heights once and then performs a map lookup which is just an O(1) operation

Sure @Santhosh-Sellavel , added comments on the changes line.
     EMOJI_PICKER_SIZE: {
         WIDTH: 320,
-        HEIGHT: 400,
+        HEIGHT: 390,  --    62px(TOP COMPOSER) + 228px(MIDDLE 9 ROWS WHICH IS 9*32px) + 40px(BOTTOM SKINTONE LIST)
     },
-    NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT: 298,
+    NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT: 288,   --   9 ROWS WHICH IS 9*32px
     EMOJI_PICKER_ITEM_HEIGHT: 32,
-    EMOJI_PICKER_HEADER_HEIGHT: 38,
+    EMOJI_PICKER_HEADER_HEIGHT: 32,  ---  NEW HEADER HEIGHT WHICH IS 32px

     emojiHeaderStyle: {
         backgroundColor: themeColors.componentBG,
         width: '100%',
-        ...spacing.pv3,
         fontFamily: fontFamily.EXP_NEUE_BOLD,
         fontWeight: fontWeightBold,
         color: themeColors.heading,
         fontSize: variables.fontSizeSmall,
+        height: 32,   ---  Giving fixed height to the header and aligning the item to the center vertically
+        display: 'flex',
+        alignItems: 'center',
     },

I am just confused why are we changing the HEIGHT of the picker 400 to 390 and 298 to 288?

I am just confused why are we changing the HEIGHT of the picker 400 to 390 and 298 to 288?

To adjust and show only 9 rows of the flatlist to the view. If we keep it to 400 then the extra part will be visible partially.
eg.

Thanks, @Pujan92!
@Gonals  @Pujan92's proposal Looks good let me know your thoughts!

@Santhosh-Sellavel @Gonals I still think the key solution here is to calculate the correct row height so actually there's no need to change all these sizes which is what @Ollyws & my proposals suggest.

@Santhosh-Sellavel @Gonals I still think the key solution here is to calculate the correct row height which is what @Ollyws & my proposals suggest, so actually there's no need to change all these sizes.

Yes definitely if we are not okay with changing height, but we are good with it. Now this is much faster, also never a fan storing the size map a fairly simple component which has fixed heights for rows unlike chat view's list where size varies.
I'll leave it for @Gonals to  make the final call on this one, thanks!

Yes definitely if we are not okay with changing height, but we are good with it. Now this is much faster, also never a fan storing the size map a fairly simple component which has fixed heights for rows unlike chat view's list where size varies.
I'll leave it for @Gonals to make the final call on this one, thanks!

I get that, if storing a size map is not ideal then Proposal 1 from here along with specifying the height in the style works too.

Add height: CONST.EMOJI_PICKER_HEADER_HEIGHT, in emojiHeaderStyle in src/styles/styles.js

Happy to hear @Gonals thoughts on it.

I get that, if storing a size map is not ideal then Proposal 1 from here along with specifying the height in the style works too.

Add height: CONST.EMOJI_PICKER_HEADER_HEIGHT, in emojiHeaderStyle in src/styles/styles.js

I tried it earlier & now didn't work as expected, look selected emoji is not visible clearly.

I tried it earlier & now didn't work as expected, look selected emoji is not visible clearly.

yes you're right, I take that back. Thus, I still think Proposal 4 is ideal as I don't see anything wrong/inefficient with storing a size map for variable heights but that's a decision for you & @Gonals to make. I'll leave it to the experts, Thanks! 👍

IMO @Pujan92 's proposal is the best solution so far. It works but we don't know why

Quick Update: I have found the root cause.
The root cause is in RN or at least in RNW
Will post more details in few minutes...

Proposal (2)
diff --git<EMAIL_ADDRESS><EMAIL_ADDRESS>index ac40acc..0b8c778 100644
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -1135,7 +1135,7 @@ class VirtualizedList extends React.PureComponent {
       if (!isVirtualizationDisabled && first > lastInitialIndex + 1) {
         var insertedStickySpacer = false;

-        if (stickyIndicesFromProps.size > 0) {
+        if (false && stickyIndicesFromProps.size > 0) {
           var stickyOffset = ListHeaderComponent ? 1 : 0; // See if there are any sticky headers in the virtualized space that we need to render.

           for (var ii = firstAfterInitial - 1; ii > lastInitialIndex; ii--) {

I know the fix looks silly but I will explain
RCA
I have been digging in the code and I found that the issue is in RN (or at least it was)
This is the offending code https://github.com/Expensify/react-native/blob/main/Libraries/Lists/VirtualizedList.js#L715-L759
My first thought was that the issue is a miscalculation of first and lastInitialIndex or maybe somehow due to not taking the number of columns into consideration.
The weird part is that this code does nothing at all, is it trying to render something that is already rendered? So I was just about to open a PR on RN to get info about that code and fix it, But RN has already refactored the VirtualizedList so there is a high chance that this has been fixed already.
However, RNW is still using the old version, I think we can open a PR / issue to "swap" the files, except that we have fork of RNW as well where we applied few changes on the VirtualizedList so switching to the new VirtualizedList won't be straightforward.

@huzaifa-99

I believe its with the FlatList for web

VirtualizedList on RN but I think it was fixed already but not for RNW. Check the proposal above

@s77rt I don't get it changes make if block unreachable and even the issue still exists for me after making the changes.

@Santhosh-Sellavel Sorry if it was unclear. As I stated in my first proposal this issue is related to sticky headers.
Apparently the issue is that RN/RNW is trying to render a header that is already rendered. In short, the block is useless. It was removed on the new refactored VirtualizedList.
Tests: Everything works fine and the tests passes.
After you apply the fix you have to clear the cache first: rm -rf node_modules/.cache && npm run web

After you apply the fix you have to clear the cache first: rm -rf node_modules/.cache && npm run web

I tried still the issue exists.
Uploading Screen Recording 2022-12-18 at 2.48.59 AM.mov…

@Santhosh-Sellavel

Apart from the RNW fix, we need to update the Header height in CONST to 37. Else we would get this partially hidden state.

Yes, that right.

@Gonals
@s77rt's Proposal here looks good, along with updating header height.

@s77rt Great to see that you find the root cause. 🎉
My only point is we are defining the height to the NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT by initially considering (2 header 2*37px = 64px and 7 emoji rows 7*32px = 224px which sums to 298px.
https://github.com/Expensify/App/blob/2483ad0a692b1392d7dc79f917edd44c5be46080/src/CONST.js#L501
Now when we scroll and on the view, if there is only 1 header in the view then in this case logically the height of the NON_NATIVE_EMOJI_PICKER_LIST_HEIGHT should be 293px(1 header 37px + 8 emoji rows 256px).
Changing the height of the list won't be a good option while scrolling and if we won't change that 5px might cause partial visibility of the next line.
https://user-images.githubusercontent.com/14358475/208281857-7f5e758d-6280-46ca-a9c3-99cd824d0441.mov

@Pujan92 I agree, however technically this is a different issue and should not be prioritised as it does not break the overall functionality. Fixing the EMOJI_PICKER_HEADER_HEIGHT should be enough. Besides there may be situations where dynamic height is required.
Ideally we should wait for RNW to update the VirtualizedList then update the Expensify's fork. Until then we can either apply a temporarily solution or disable the arrow navigation feature.
If the issue is on RN/RNW it must be fixed on RN/RNW

As we are mentioning the height/dynamic heights in the comments I thought it's better to point out the issue over here only. For fixed height, it seems to use getItemLayout as a better option for optimization purpose. I think for particular this use case this is a more permanent solution. As you stated for dynamic heights we should wait and depend on the sticky header fix from the RN.
getItemLayout is an optional optimization that allows skipping the measurement of dynamic content if you know the size (height or width) of items ahead of time. getItemLayout is efficient if you have fixed size items

@Luke9389 It looks like @Gonals is now on vacation, and we're stuck getting agreement on the path forward. Can you please review the latest comment from @Santhosh-Sellavel so we can unblock this issue and keep it moving forward please? Thank you!

We've discussed internally and decided on the following course of action:
This proposal best: https://github.com/Expensify/App/issues/12772#issuecomment-1351661286
Though it does use getItemLayout which was first mentioned in @Ollyws's proposal here: https://github.com/Expensify/App/issues/12772#issuecomment-1340926774
For that reason we will move forward with @Pujan92's proposal, but award @Ollyws an additional $1000.
I do appreciate that @s77rt found the root cause, but we do not have time to wait for RN to fix this on their end, and this never lead to a formal proposal to solve the issue.

@Luke9389 I respect your decision but it feels as I'm being punished for following the rules and for trying to fix the issue upstream

Thanks for expressing that @s77rt. Before we continue with the issue, I want to be sure to give you a fair chance as well.
So this is your proposal correct?

@Luke9389 Yes, that's it, it solves the issue but still I would prefer to actually go with refactoring the VirtualizedList i.e. wait for RNW to adapt the new VirtualizedList.
Usually on such situations I'd expect the issue to be put on hold but if we can't wait for that I'd appreciate a compensation

I see. There are a few things to discuss here.

Yes, that's it, it solves the issue

I'm curious to know why this is the case. After reading your explanation, I was still unsure.

but still I would prefer to actually go with refactoring the VirtualizedList i.e. wait for RNW to adapt the new VirtualizedList.

While it would be ideal for RNW to adapt the new VirtualizedList, it wouldn't be a guarantee that we use it. The problem with this approach is that we are waiting for them to make a change, and then we would need to enter a separate (and probably lengthy) dialogue to figure out if we should make the switch as well.

Usually on such situations I'd expect the issue to be put on hold

We would put this on hold if we agreed that we should wait for RNW to adapt VirtualizedList (and agreed on using it).

I'd appreciate a compensation

The reason we have elected to give compensation to @Ollyws, is because the proposal we are going with includes some of their work. If the proposal we selected was building off of your ideas, then we would also compensate you.

it feels as I'm being punished for following the rules

This is the most important thing for us to reach a resolution about, because we want to build a community that operates with fair rules. Would you be willing to point out which rule you feel you're being punished for following? A link to the rule in documentation would be very much appreciated. Also, if you'd be willing to suggest an improvement to our rules, that would also be very helpful.
I'll give you an opportunity to respond to these points before saying any more. Again, I just want to thank you for communicating with me about it. Our process is still undergoing change and we want to be sure everyone feels understood and listened to. 🙇

@Luke9389

I'm curious to know how it solves the issue. After reading your explanation, I was still unsure.

The issue is that code will try to re-render the sticky headers that are already rendered and add more spaces that will effect the overall contentSIze and reposition the scroller. This part of code has been greatly refactored on the new VirtualizedList.

While it would be ideal for RNW to adapt the new VirtualizedList, it wouldn't be a guarantee that we use it

I think we will have to use it the sooner or the latter.

We would put this on hold if we agreed that we should wait for RNW to adapt VirtualizedList.

Understood.

If the proposal we selected was building off of your ideas, then we would also compensate you

Understood

Would you be willing to point out which rule you feel you're being punished for following?

Actually I can't find an explicit rule for what I said, but fixing the issues upstream is something that goes without saying, although I'm sure we were encouraged on other issues (not the rules) to fix issues upstream if the root cause is not on Expensify/App

We want to be sure everyone feels understood and listened to

That's great, I appreciate your time for looking into my case.

Between my latest message and your response, I was made aware of this thread, which discusses our preference for updating packages upstream.
Given all of this new information, it may be a good idea to change course and use VirtualizedList after all. But before doing that, I think we should get more details.
Let's say that RNW adapted the refactored VirtualizedList. What changes would we then need to make? @s77rt, would you be willing to give a hypothetical proposal to help me determine what that would look like?

What changes would we then need to make?

To fix this issue?
Hopefully nothing, but if the issue still persist, we will have to create a PR in RN. Although the refactored VirtualizedList looks promising we can't be sure as we can't test it unless RNW adopts it.

In general
We will have to verify that those commits https://github.com/Expensify/react-native-web/commits/master/packages/react-native-web/src/vendor/react-native/VirtualizedList/index.js can still be applied elegantly to fix whatever they were designed to fix

The point 2 is unavoidable, if at any time RNW adopts the new VirtualizedList we will eventually do the same
Hypothetical proposal: (excluding point 2 because it's not directly linked to this issue and it's something we will do anyway, I suppose)

Best case scenario: do nothing
Worst case scenario: fix the persisting issue in RN

Is there any way for us to test whether using the updated VirtualizedList will work?

Maybe if we add the arrow navigation feature to native and update our forked RN to use the updated VirtualizedList, then yes it would be possible

Sorry which arrow navigation feature are you referring to?
Ultimately, what I'm really looking for is a clear plan for exactly how you think we should move forward. We could apply the above solution and close this issue soon, but if you can convince us that you have a better alternative, please spell it out in one detailed comment with a clear proposal and steps to follow.
This is your chance to change our course!

Sorry which arrow navigation feature are you referring to?

There is only one arrow navigation feature, it's what this issue is about. You can navigate through emojis using your keyboard, this is not available on native, but if we can replicate this behaviour on native then we can test whether the updated VirtualizedList will work.
Sometimes the ideal solution is too much time consuming that makes it feels less of an ideal.
Maybe it would be a good call to apply a temp solution or disable arrow navigation until:

RNW adapts the refactored VirtualizedList https://github.com/necolas/react-native-web/issues/2453
Expensify's RNW fork adapts the refactored VirtualizedList ( and re-apply https://github.com/Expensify/react-native-web/commits/master/packages/react-native-web/src/vendor/react-native/VirtualizedList/index.js )

Only after point 1 and point 2 are done, we can actually fix this issue properly.

Summary:
PLAN A:

Wait for RNW to adapt the refactored VirtualizedList
We should also adapt the refactored VirtualizedList
Issue is fixed.

PLAN B:

Wait for RNW to adapt the refactored VirtualizedList
We should also adapt the refactored VirtualizedList
Issue is still persist, re-open this GH issue for proposals that aim to fix RN's bug
Once the bug is identified, create a PR on RN and possibly on RNW too
Issue is fixed.

This proposal best: #12772 (comment) Though it does use getItemLayout which was first mentioned in @Ollyws's proposal here: #12772 (comment)

@Luke9389 Have you tried this https://github.com/Expensify/App/issues/12772#issuecomment-1344497243 solution? I feel this https://github.com/Expensify/App/issues/12772#issuecomment-1351661286 solution avoids the problem rather than fixing it.

In this case, where we have the fixed height, I think getItemLayout will be the preferable approach based on the react native doc too. @s77rt as per my understanding the RN fix which you have suggested helps for dynamic heights flatlist which doesn't seem the case here.

In this case, where we have the fixed height, I think getItemLayout will be the preferable approach based on the react native doc too. @s77rt as per my understanding the RN fix which you have suggested helps for dynamic heights flatlist which doesn't seem the case here.

@Luke9389 Have you tried this #12772 (comment) solution? I feel this #12772 (comment) solution avoids the problem rather than fixing it.

@priyeshshah11 I have explained there and here why the same height is required for good of the application perspective and for the same get approval from the design team(@shawnborton) too. So I don't think this solution is avoiding the problem from any direction.

@priyeshshah11 I have explained there and here why the same height is required for good of the application perspective and for the same get approval from the design team(@shawnborton) too. So I don't think this solution is avoiding the problem from any direction.

@Pujan92 I understand you're point but IMO changing the header height to be the same as the item height doesn't actually fix the actual bug, it avoids it. I understand that the design team is ok to change the height in this case but sometimes that might not be possible so IMO changing the design to fix the problem should be the absolute last resort where we can't find any good enough solutions.
Also in regards to the efficiency of the application, getItemLayout is equally efficient with variable heights when you know the exact heights in advance, which we do in this case.

Ok so I think we should go with the plan I posted here: https://github.com/Expensify/App/issues/12772#issuecomment-1360227084

I understand that the design team is ok to change the height in this case but sometimes that might not be possible so IMO changing the design to fix the problem should be the absolute last resort where we can't find any good enough solutions.

@priyeshshah11 I appreciate that you want to build a more general solution that could be used elsewhere, but the reality is that this is a small issue that we aren't seeing anywhere else on the app. It just doesn't need this level of complexity. I do think you've got great ideas, and you should continue to think this way for other issues, especially larger ones.
@s77rt, you did a good job of laying out your point, but at this time we don't want to wait for RNW and we don't want to disable arrow navigation on the emoji picker. You mentioned this

Maybe it would be a good call to apply a temp solution

and I think in this case, that's what we'll do. Thanks for all the time and energy you put toward this discussion.
With all that said, @Pujan92 you've got the 🟢 green light 🟢 to start your PR. @Santhosh-Sellavel and I will see you there.

Thanks, applied for the job and PR is ready for review.

It looks like only @Santhosh-Sellavel is assigned for review. @Luke9389 should you also be on the PR, or do we need to pull bear so that others are assigned? Let's help @Pujan92 get this merged in three business days!

Actually...something weird is going on? Looks like @Gonals was chosen but I'm pretty sure he's OOO now and also not assigned.

Actually...something weird is going on? Looks like @Gonals was chosen but I'm pretty sure he's OOO now and also not assigned.

I think it is my mistake and happens due to the incorrect way of linking the issue in the PR. Updated the PR.

Looks like the linked PR is ready to merge. Left a comment for @Gonals to confirm in the PR.

Today is payday on this issue, so I'm circling back to close the loop on a payment summary. Looking at the PR, I'm going to go ahead and award C/C+ the bonus for urgency. Any delay in merging the PR was caused on our side due to the holidays. That means that payouts will look like this:

Issue reporter: @parasharrajat - $250
Contributor: @Pujan92 - $6,000 (incl 50% bonus)
Contributor+: @Santhosh-Sellavel - $6,000 (incl 50% bonus).

Upwork job is here. An offer has been extended to @Pujan92, @Santhosh-Sellavel has been invited, as has @parasharrajat. Thanks!

Ah thank you! Can you please apply to the Upwork job? I'll issue an offer once you apply.

@Pujan92 and @Ollyws have been paid. @Santhosh-Sellavel was sent an offer (I'll pay out the bonus when payment is issued). Still waiting on @parasharrajat to accept the invite.

Then @Santhosh-Sellavel @Luke9389 Can either of you kick-off and complete your items on the BZ checklist? I've already created an issue for regression testing in the Expensify/Expensify repo.

I'm not sure if this is a regression. I think this behavior has always been here and we just recently noticed it. If that's the case, should we check the "The PR that introduced the bug has been identified." box?
I'll dig in to see if I can find an offending PR but I'd still like to know what to do in the case that there isn't one.

If there isn't a regression, then all good. We should be able to cover this moving forward with the regression test we'll add moving forward.

Ok, I didn't see anything that indicates this is a regression, so I checked us off.

@Santhosh-Sellavel has been paid out, and all items are successfully checked off in the BZ checklist. Closing this issue as a result.


Update build.zig to export zigfsm for other build.zigs

sources:

https://zig.news/liyu1981/how-to-use-your-fav-pkg-in-buildzig-3ni8
https://ziggit.dev/t/can-i-use-packages-inside-build-zig/2892/2

child build.zig export:
https://github.com/liyu1981/zcmd.zig/blob/ef24428acabe2c8b83252a011dc1e75f0d260c42/build.zig#L3
parent build.zig import:
https://github.com/liyu1981/kcov/blob/331d207908ff0518c18282d04ed029b35ff74bbc/build.zig#L2

Thanks!


Regression on main when allocating arrays of arrays

Minimal repro that fails to compile:
use spirv_std::spirv;

#[spirv(fragment)]
pub fn main() {
    let x = [[1; 2]; 1];
}

Bisects to https://github.com/Rust-GPU/rust-gpu/commit/02cd3248ad13df8e0b4fa2962ceb57ac089e2d49
It was found in https://github.com/Rust-GPU/rust-gpu/pull/21#issuecomment-2381578892 (assuming it is the same issue, but at least that is what I started with to get a repro)

@eddyb not sure if this will be fixed by your qptr stuff.

Yep, that's the error:
error: cannot cast between pointer types
       from `*i32`
         to `*[i32; 2]`

Also happens with arrays of vecs, which is more common: https://github.com/Rust-GPU/rust-gpu/issues/189


Add to useful links: https://www.universalminecrafteditor.com/

NBT Editor, under tools

Started and done


Create Posts “juish-nature”

Automatically generated by Netlify CMS

👷 Deploy Preview for benevolent-pika-f5fbe0 processing.

Name
Link

🔨 Latest commit
1cce5dfaed24bdff6770435f69db6386070da678

🔍 Latest deploy log
https://app.netlify.com/sites/benevolent-pika-f5fbe0/deploys/62e39e7a47b3340008b495c2


Adding support for loading GeoTIFF images

Adds a GeoTIFFDataset format with support for loading datasets of GeoTIFF images and storing their geolocation data in a GeoLocation field.
Since the GeoTIFF standard dictates that GeoTIFF images should be viewable as regular TIFF images, no image type conversion is performed during import. The example below demonstrates that, at least for the example images I found, this assumption is true; i.e., the dataset can be viewed in the App as expected (as long as you have an appropriate TIFF viewer browser extension installed, which I believe is required in browsers like Chrome to view any TIFF image in the first place!)
The rasterio package must be installed in order to work with GeoTIFF images, which seems to be a reasonably popular and stable dependency to use for this.
The geolocation information loaded when working with datasets of this type can be conveniently used to construct geolocation views and to generate location-based scatterplots.
Setup
Install new dependency:
pip install rasterio

Download some test data:
DATASET_DIR=/tmp/geotiff

mkdir -p $DATASET_DIR
wget -P $DATASET_DIR https://download.osgeo.org/geotiff/samples/GeogToWGS84GeoKey/GeogToWGS84GeoKey5.tif
wget -P $DATASET_DIR https://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
wget -P $DATASET_DIR https://download.osgeo.org/geotiff/samples/pci_eg/acea.tif
wget -P $DATASET_DIR https://download.osgeo.org/geotiff/samples/spot/chicago/SP27GTIF.TIF

Example usage
import fiftyone as fo

dataset = fo.Dataset.from_dir(
    dataset_dir="/tmp/geotiff",
    dataset_type=fo.types.GeoTIFFDataset,
    label_field="location",
)

print(dataset)
print(dataset.first())
print(dataset.values("location.point"))

session = fo.launch_app(dataset)

# Example geo-query: show samples within 5km of Chicago
session.view = dataset.geo_near([-87.6298, 41.8781], max_distance=5000)

I'll note that the Chrome TIFF extension link is dead in the FAQ. I'm unable to view these images in Chrome on my Linux machine and couldn't find a suitable extension.


Generate .dclasset files only for known asset types

#861m7jm20

Task linked: CU-861m7jm20 dclasset files are also generated for assets of type unknown


strange link error in Windows: xtensa-esp32-elf-g++.exe: error: CreateProcess: No such file or directory (IDFGH-4202)

Environment

Development Kit:      [none, using qemu, no hardware]
Kit version (for WroverKit/PicoKit/DevKitC): [none]
Module or chip used:  [ESP32]
IDF version (run git describe --tags to find it):
v4.3-dev-1561-g357a27760
Build System:         [CMake]
Compiler version (run xtensa-esp32-elf-gcc --version to find it):
xtensa-esp32-elf-gcc (crosstool-NG esp-2020r3) 8.4.0
Operating System:     [Windows]
(Windows only) environment type: [ESP Command Prompt&Plain Command Prompt].
Using an IDE?: [No]
Power Supply:         [None]

Problem Description
Compiles OK under Ubuntu(wsl2), but failed on windows command line.
below is the error log:
[1284/1288] Linking CXX executable hello-world.elf
FAILED: hello-world.elf
cmd.exe /C "cd . && C:\Users\SimonQian.espressif\tools\xtensa-esp32-elf\esp-2020r3-8.4.0\xtensa-esp32-elf\bin\xtensa-esp32-elf-g++.exe  -mlongcalls -Wno-frame-address   @CMakeFiles\hello-world.elf.rsp  -o hello-world.elf  && cd ."
xtensa-esp32-elf-g++.exe: error: CreateProcess: No such file or directory
if I remove some unused .o files in CMakeFiles\hello-world.elf.rsp, link will pass.
Expected Behavior
Link OK.
Actual Behavior
Link failed
Steps to reproduce

clone vsf(I'm porting vsf to esp32):
git clone https://github.com/vsfteam/vsf.git
set VSF_PATH
set VSF_PATH=PATH_TO_VSF_CLONED
extract esp_vsf_demo I provided
cd esp_vsf_demo
mkdir build
cd build
cmake -G Ninja ..
cmake --build .

Code to reproduce this issue
Debug Logs
Other items if possible

[ ] sdkconfig file (attach the sdkconfig file from your project folder)
[ ] elf file in the build folder (note this may contain all the code details and symbols of your project.)
[ ] coredump (This provides stacks of tasks.)

esp_vsf_demo.zip
This is the esp_vsf_demo I mentioned.

I switched to wsl2 Ubuntu, everything runs OK(and much more faster) now.

Hi @versaloon . Thank you for reporting the issue and we're sorry to responding so late to the issue with ESP-IDF 4.4 (master).
I was trying to replicate the behavior on Windows and Ubuntu with provided file, but it failed with error like:
CMake Error at C:/Users/georgik/Desktop/esp-idf/tools/cmake/component.cmake:305 (message):
  Include directory 'C:/projects/tmp/vsf/source/vsf' is not a directory.
Call Stack (most recent call first):
  C:/Users/georgik/Desktop/esp-idf/tools/cmake/component.cmake:485 (__component_add_include_dirs)
  C:/projects/tmp/vsf/CMakeLists.txt:1 (idf_component_register)

vsf source code in latest commit does not contain source/vsf.
It seems that project https://github.com/vsfteam/vsf moved quite a lot. Is the esp_vsf_demo source code still relevant?

yes, there are changes in vsf, and esp32 project is not updated.
esp32 support in vsf is not maintained now, so just ignore the issue.


bump milli

Fix https://github.com/meilisearch/MeiliSearch/issues/2082

bors try

Can you change the version in the Cargo.toml with v0.25.2 as well? 😇
I'll update the description of the PR to explain in detail what's inside

bors try

Let's try again!
bors merge

bors merge

bors merge

it is very awkward that these tests keep failing on windows


OneDrive /getpin error

BUG:
/getpin produce this error and the file status icon is not shown in Finder.
$ /Applications/OneDrive.app/Contents/MacOS/OneDrive /getpin /Users/username/OneDrive
2020-06-29 12:16:07.204 OneDrive[37720:1054071] invalid mode 'kCFRunLoopCommonModes' provided to CFRunLoopRunSpecific - break on _CFRunLoopError_RunCalledWithInvalidMode to debug. This message will only appear once per execution. 2020-06-29 12:16:07.246 OneDrive[37720:1054071] pin state=None

@lgimyuen Thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


GH-38614: [Java] Add VarBinary and VarCharWriter helper methods to more writers

Rationale for this change
Add the overrides for new convenience Writer methods added to VarCharWriter and VarBinaryWriter so that classes that
use composition such as UnionWriter and PromotableWriter can invoke them properly.
What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Closes: #38614

@lidavidm


[bgp] Add new CRD to manage FRRConfiguration

Secondary network interfaces on pods need be announced in a BGP environment. This can be done by creating FRRConfiguration, per default in the metallb namespace.
This PR introduce a new CRD which, if an instance got created, the controller watches pods

which have the NAD annotation on it
the NAD has an IPAM configured

For each of them a FRRConfiguration gets created. The metallbs k8s service FRRConfiguration of that worker node is taken as the base to create this configuration.
Known issue:
Since there are then two FRRConfiguration, which hold same configs, like timeouts. It is not possible to update thise. The FRRConfiguration webhook will block those. A possible way to change it, would be stop the infra-operator controller-manager, delete the pod FRRConfigurations, do the change that it gets reflected in the metallb LB FRRConfiguration, then enable the controller that the pod FRRConfiguration get re-created.
Depends-On: https://github.com/openstack-k8s-operators/lib-common/pull/588
Depends-On: https://github.com/openstack-k8s-operators/lib-common/pull/589
Jira: OSPRH-8680
Jira: OSPRH-12384

recheck


Align with best practices for production component

Vue community has implemented lower case props as the standard for props with dashes. It is enforced by eslint vue plugin. So I changed Highchart to highchart for the props

Remove sourcemaps for production builds to comply with CSP best practice of not using eval. Also, reduces size of included component  in others codebase.

Thanks~


chore(main): release 0.2.3

:robot: I have created a release beep boop
0.2.3 (2024-10-03)
Miscellaneous Chores

release 0.2.3 (cf4e0ca)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v0.2.3
:sunflower:


RTMPS Issue Facebook May 1 Help

Thank you @Lax for the solution! I tried it and it works perfectly and with little to none performance hiccups 🙌
I wrote instructions for those who are using this on Windows. We are all not doomed after all!
Originally posted by @JosueAChavez in https://github.com/arut/nginx-rtmp-module/issues/1397#issuecomment-480116923

@JosueAChavez, can you assist with how you got stunnel configured on nginx server from rtmps for Facebook work around? Newbie here. Thanks in advance for your assistance

For Linux server, check out setup rtmps relay post here: https://dev.to/lax/rtmps-relay-with-stunnel-12d3

thanks to akeems article i was able to set it up made a video about it check that for easy understanding https://www.youtube.com/watch?v=e6JeRVI0ioQ

Hi @Lax Thanks I saw the link.  But I am still confused on the command to insert the second step after I install stunnel. Then I don't know the command to enable the tunnel.
I am using debian
Any help will be appreciated.

Hi piyush008gupta I am using debian thanks for your windows tutorial though.

Please add RTMPS support

Related to #457 & #747

I'm getting two issues when trying to set this up and work on my mac:

My current stunnel.conf looks like this:
;setuid = stunnel4
;setgid = stunnel4
pid=/tmp/stunnel.pid
output = /usr/local/var/log/stunnel4/stunnel.log
include = /usr/local/etc/stunnel/conf.d

If I activate setuid and setgid, I'm getting error messages like this:
/usr/local/etc/stunnel/stunnel.conf:1: "setuid = stunnel4": Illegal UID

The connection with facebook does not work. I'm getting this error message:
Error resolving "live-api-s.facebook.com": Neither nodename nor servname known (EAI_NONAME)

Any suggestions?


live graphs / metrics (a la pgadmin)

description
periodically query some stats around connections and resource usage and visualize them in a graph
proposed solution

read throughput
wait groups
..?

on the fence about this one; maybe could create a more generic graphing functionality that can also apply to query results. not planned for now tho