[BUG] Throughput vs Gym AsyncVectorEnv

Describe the bug
Hello, I'm performing experiments that use a relatively small number of parallel environments (8-16). Using the PongNoFrameskip-v4 environment with no wrappers, it seems that TorchRL is 4-5x slower than Gym's AsyncVectorEnv (2600 vs 11000 FPS) with a random policy. Given the throughput results in Table 2 of the paper, I would expect comparable performance. Am I setting up the environments incorrectly?
To Reproduce
This is a very simple adaptation of the script in examples/distributed/single_machine/generic.py. Although it's not shown here, I observe similar performance with ParallelEnv and a synchronous collector.
import time
from argparse import ArgumentParser

import torch
import tqdm

from torchrl.collectors.collectors import (
    MultiaSyncDataCollector,
    MultiSyncDataCollector,
    RandomPolicy,
)
from torchrl.envs import EnvCreator
from torchrl.envs.libs.gym import GymEnv
import gymnasium as gym

parser = ArgumentParser()
parser.add_argument(
    "--num_workers", default=8, type=int, help="Number of workers in each node."
)
parser.add_argument(
    "--total_frames",
    default=500_000,
    type=int,
    help="Total number of frames collected by the collector. Must be "
    "divisible by the product of nodes and workers.",
)
parser.add_argument(
    "--env",
    default="PongNoFrameskip-v4",
    help="Gym environment to be run.",
)
if __name__ == "__main__":
    args = parser.parse_args()
    num_workers = args.num_workers
    frames_per_batch = 10*args.num_workers

    # Test asynchronous gym collector
    env = gym.vector.AsyncVectorEnv([lambda: gym.make(args.env) for _ in range(num_workers)])
    env.reset()
    global_step = 0
    start = time.time()
    for _ in range(args.total_frames//num_workers):
        global_step += num_workers
        env.step(env.action_space.sample())
        stop = time.time()
        if global_step % int(num_workers*1_000) == 0:
            print('FPS:', global_step / (stop - start))
    env.close()

    # Test multiprocess TorchRL collector
    device = 'cuda:0'
    make_env = EnvCreator(lambda: GymEnv(args.env, device=device))
    action_spec = make_env().action_spec
    collector = MultiaSyncDataCollector(
        [make_env] * num_workers,
        policy=RandomPolicy(action_spec),
        total_frames=args.total_frames,
        frames_per_batch=frames_per_batch,
        devices=device,
        storing_devices=device,
    )
    counter = 0
    for i, data in enumerate(collector):
        if i == 10:
            pbar = tqdm.tqdm(total=collector.total_frames)
            t0 = time.time()
        if i >= 10:
            counter += data.numel()
            pbar.update(data.numel())
            pbar.set_description(f"data shape: {data.shape}, data device: {data.device}")
    collector.shutdown()
    t1 = time.time()
    print(f"time elapsed: {t1-t0}s, rate: {counter/(t1-t0)} fps")
    exit()

System info
TorchRL installed via pip (v0.1.1)
import torchrl, numpy, sys
print(torchrl.__version__, numpy.__version__, sys.version, sys.platform)

None 1.22.0 3.10.11 (main, Apr 20 2023, 19:02:41) [GCC 11.2.0] linux

Checklist

[x] I have checked that there is no similar issue in the repo (required)
[x] I have read the documentation (required)
[x] I have provided a minimal working example to reproduce the bug (required)

Thanks for reporting this! We appreciate the feedback.
The results of the paper were obtained using collectors, not parallel envs. I will be posting the code shortly for reproducibility.
See this discussion for more context.
Also, I see that you're using the latest stable version of the library (which is good!). You'll be happy to know that we've speeded up a bunch of operations in tensordict and vectorized envs and the nightly releases of tensordict and torchrl should give you better results. There are other optimizations we can do so I'm confident we can accelerate things even more.
Executing a slightly modified version of the code above, on my McBook I get to a speed for TorchRL that is 2x slower than the gym one. The overhead is mainly caused by tensordict operations.
However, using the config we had in the paper (mainly 32 procs instead of 8 and more cuda devices used for passing data from one proc to another), I get to a speed of 8k fps as reported in the paper. When using 4 parallel envs / collector and 8 collectors, I get to a speed of 16k fps.
I will keep on updating this post and related as we optimize things further.

Hey!
I adapted the benchmark for a more straightforward comparison and ran on all the relevant parallel collectors (specs, code, and output below).
This is what I got with 8 envs in parallel and 2000 vs 80 frames per batch:
Summary:
# Output is for
# Running 8 envs with 2000 frames per batch (i.e. 250.0 frames per env).
# Running 8 envs with 80 frames per batch (i.e. 10.0 frames per env).

FPS Gym AsyncVectorEnv mean: 11587.749470459721
FPS Gym AsyncVectorEnv mean: 11317.07864610709

FPS TorchRL with MultiSyncDataCollector on cpu mean: 2866.4653702739765
FPS TorchRL with MultiSyncDataCollector on cpu mean:<PHONE_NUMBER>006343

FPS TorchRL with MultiSyncDataCollector on cuda:0 mean: 5108.929550860561
FPS TorchRL with MultiSyncDataCollector on cuda:0 mean: 3225.940279205892

FPS TorchRL with MultiaSyncDataCollector on cpu mean: 4349.725160643495
FPS TorchRL with MultiaSyncDataCollector on cpu mean: 1700.2908940649854

FPS TorchRL with MultiaSyncDataCollector on cuda:0 mean: 4745.486839045239
FPS TorchRL with MultiaSyncDataCollector on cuda:0 mean: 5223.255705987036

FPS TorchRL with SyncDataCollector on cpu mean:<PHONE_NUMBER>549519
FPS TorchRL with SyncDataCollector on cpu mean:<PHONE_NUMBER>961403

FPS TorchRL with SyncDataCollector on cuda:0 mean: 3171.8938244285127
FPS TorchRL with SyncDataCollector on cuda:0 mean: 2424.713380369402

(SyncDataCollector is running a ParallelEnv)
As @ShaneFlandermeyer got, Gym's AsyncVectorEnv runs both at 11k FPS.
For TorchRL, the highest FPS I got was 5k on CUDA with MultiaSyncDataCollector with both batch sizes or MultiSyncDataCollector with the larger batch size.
And in general, there's a big difference between running large batches and small batches with the torchrl collectors, and some numbers are a bit unexpected.

Running with CUDA is always better than running wth  CPU and to me this is quite unexpected: the environments are running on CPU and there is no image transformation happening that could benefit from CUDA, so why would storing the tensors on CUDA be faster? @vmoens is the TensorDict overhead much faster with CUDA?
Decreasing the batch size almost always decreases the FPS (up to 3x slower). In Gym the observations generated are directly thrown away, but with the collectors, the observations are stored and the storing TensorDict has to be initialized, etc so the benchmark is not really fair (The gym part doesn't include preallocating space and moving the tensors to CUDA, etc), but this tells a lot about the overhead incurred by the collector. Maybe something can be further improved here?
@vmoens in the complete output with detailed FPS at each step,  MultiaSyncDataCollector's FPS are varying so much (especially on CUDA). Anything I'm not doing correctly or does that come from the async nature of the collector?

Specs:
print(torchrl.__version__, numpy.__version__, sys.version, sys.platform)
2023.7.18 1.25.0 3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) [GCC 11.3.0] linux
# torchrl at commit @886635e

# Hardware
GPU 0: Tesla V100-SXM2-32GB
2x Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz

❯ mamba list | grep torch
pytorch                   2.0.1           py3.10_cuda11.7_cudnn8.5.0_0    pytorch
pytorch-cuda              11.7                 h778d358_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torchrl-nightly           2023.7.18                pypi_0    pypi
torchtriton               2.0.0                     py310    pytorch
torchvision               0.15.2              py310_cu117    pytorch

Code
import time
import warnings
from argparse import ArgumentParser

from torchrl.collectors.collectors import (
    MultiaSyncDataCollector,
    MultiSyncDataCollector,
    RandomPolicy,
    SyncDataCollector,
)
from torchrl.envs import EnvCreator, ParallelEnv
from torchrl.envs.libs.gym import GymEnv
import gymnasium as gym

warnings.filterwarnings("ignore", category=UserWarning)

parser = ArgumentParser()
parser.add_argument("--num_workers", default=8, type=int, help="Number of workers.")
parser.add_argument(
    "--frames_per_batch",
    default=10_000,
    type=int,
    help="Number of frames collected in a batch. Must be "
    "divisible by the number of workers.",
)
parser.add_argument(
    "--total_frames",
    default=100_000,
    type=int,
    help="Total number of frames collected by the collector. Must be "
    "divisible by the number of frames per batch.",
)
parser.add_argument(
    "--log_every",
    default=10_000,
    type=int,
    help="Number of frames between each log.",
)
parser.add_argument(
    "--env",
    default="PongNoFrameskip-v4",
    help="Gym environment to be run.",
)

if __name__ == "__main__":
    args = parser.parse_args()
    num_workers = args.num_workers
    frames_per_batch = args.frames_per_batch

    print(
        f"Running {num_workers} envs with {frames_per_batch} frames per batch"
        f" (i.e. {frames_per_batch / num_workers} frames per env)."
    )

    # Test asynchronous gym collector
    def test_gym():
        env = gym.vector.AsyncVectorEnv(
            [lambda: gym.make(args.env) for _ in range(num_workers)]
        )
        env.reset()
        global_step = 0
        times = []
        start = time.time()
        print("Timer started.")
        for _ in range(args.total_frames // num_workers):
            env.step(env.action_space.sample())
            global_step += num_workers
            if global_step % int(frames_per_batch) == 0:
                times.append(time.time() - start)
                fps = frames_per_batch / times[-1]
                if global_step % args.log_every == 0:
                    print(f"FPS Gym AsyncVectorEnv at step {global_step}:", fps)
                start = time.time()
        env.close()
        print("FPS Gym AsyncVectorEnv mean:", args.total_frames / sum(times))

    # Test multiprocess TorchRL collector
    def test_torch_rl(collector_class, device):
        make_env = EnvCreator(lambda: GymEnv(args.env, device=device))
        if collector_class in [MultiSyncDataCollector, MultiaSyncDataCollector]:
            mock_env = make_env()
            collector = collector_class(
                [make_env] * num_workers,
                policy=RandomPolicy(mock_env.action_spec),
                total_frames=args.total_frames,
                frames_per_batch=frames_per_batch,
                device=device,
                storing_device=device,
            )
        elif collector_class in [SyncDataCollector]:
            parallel_env = ParallelEnv(args.num_workers, make_env)
            collector = SyncDataCollector(
                parallel_env,
                policy=RandomPolicy(parallel_env.action_spec),
                total_frames=args.total_frames,
                frames_per_batch=frames_per_batch,
                device=device,
                storing_device=device,
            )
        global_step = 0
        times = []
        start = time.time()
        print("Timer started.")
        for i, data in enumerate(collector):
            global_step += data.numel()
            times.append(time.time() - start)
            fps = frames_per_batch / times[-1]
            if global_step % args.log_every == 0:
                print(
                    f"FPS TorchRL with {collector_class.__name__} on {device} at step {global_step}:",
                    fps,
                )
            start = time.time()
        collector.shutdown()
        print(
            "FPS TorchRL with",
            collector_class.__name__,
            "on",
            device,
            "mean:",
            args.total_frames / sum(times),
        )

    test_gym()
    for collector_class in [MultiSyncDataCollector, MultiaSyncDataCollector, SyncDataCollector]:
        for device in ["cpu", "cuda:0"]:
            test_torch_rl(collector_class, device)

    exit()

Output

❯ python benchmarks/test_torchrl_vs_gym.py --num_workers=8 --frames_per_batch=2_000

Running 8 envs with 2000 frames per batch (i.e. 250.0 frames per env).
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS Gym AsyncVectorEnv at step 10000: 11880.248747693306
FPS Gym AsyncVectorEnv at step 20000: 14411.756722164288
FPS Gym AsyncVectorEnv at step 30000: 11630.216962415272
FPS Gym AsyncVectorEnv at step 40000: 10516.125311837932
FPS Gym AsyncVectorEnv at step 50000: 9574.003693279737
FPS Gym AsyncVectorEnv at step 60000: 9945.684469115837
FPS Gym AsyncVectorEnv at step 70000: 11437.316106795084
FPS Gym AsyncVectorEnv at step 80000: 8848.28526795556
FPS Gym AsyncVectorEnv at step 90000: 11634.265615941727
FPS Gym AsyncVectorEnv at step 100000: 10506.484039768444
FPS Gym AsyncVectorEnv mean: 11587.749470459721
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiSyncDataCollector on cpu at step 10000: 2834.0370614368485
FPS TorchRL with MultiSyncDataCollector on cpu at step 20000: 3115.0810538222336
FPS TorchRL with MultiSyncDataCollector on cpu at step 30000: 2827.5250466248276
FPS TorchRL with MultiSyncDataCollector on cpu at step 40000: 3015.099926353219
FPS TorchRL with MultiSyncDataCollector on cpu at step 50000: 2788.602794517342
FPS TorchRL with MultiSyncDataCollector on cpu at step 60000: 3205.3090866295415
FPS TorchRL with MultiSyncDataCollector on cpu at step 70000: 2482.831288036687
FPS TorchRL with MultiSyncDataCollector on cpu at step 80000: 2844.3913975942423
FPS TorchRL with MultiSyncDataCollector on cpu at step 90000: 3150.4935152535104
FPS TorchRL with MultiSyncDataCollector on cpu at step 100000: 3045.399594411799
FPS TorchRL with MultiSyncDataCollector on cpu mean: 2866.4653702739765
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 10000: 5257.937418281604
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 20000: 5208.714091009286
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 30000: 5117.738499344772
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 40000: 5137.022029172594
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 50000: 5189.274791404789
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 60000: 5110.6078855326305
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 70000: 3927.3263204022173
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 80000: 5181.009037697941
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 90000: 5198.836110327817
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 100000: 5182.798409438583
FPS TorchRL with MultiSyncDataCollector on cuda:0 mean: 5108.929550860561
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiaSyncDataCollector on cpu at step 10000: 5337.133782600729
FPS TorchRL with MultiaSyncDataCollector on cpu at step 20000: 1227.4027505992447
FPS TorchRL with MultiaSyncDataCollector on cpu at step 30000: 4303.450257146888
FPS TorchRL with MultiaSyncDataCollector on cpu at step 40000: 14703.854837126471
FPS TorchRL with MultiaSyncDataCollector on cpu at step 50000: 1778.4022887743843
FPS TorchRL with MultiaSyncDataCollector on cpu at step 60000: 6623.503838558663
FPS TorchRL with MultiaSyncDataCollector on cpu at step 70000: 4115.28234834704
FPS TorchRL with MultiaSyncDataCollector on cpu at step 80000: 4658.867646209418
FPS TorchRL with MultiaSyncDataCollector on cpu at step 90000: 3634.1222290093533
FPS TorchRL with MultiaSyncDataCollector on cpu at step 100000: 6693.456638912273
FPS TorchRL with MultiaSyncDataCollector on cpu mean: 4349.725160643495
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 10000:<PHONE_NUMBER>529257
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 20000: 18005.632254962566
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 30000:<PHONE_NUMBER>7332327
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 40000:<PHONE_NUMBER>8714132
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 50000:<PHONE_NUMBER>266598
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 60000: 41273.37941892789
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 70000: 49679.94646230745
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 80000:<PHONE_NUMBER>5736885
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 90000:<PHONE_NUMBER>1847916
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 100000: 5840.644401694138
FPS TorchRL with MultiaSyncDataCollector on cuda:0 mean: 4745.486839045239
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with SyncDataCollector on cpu at step 10000:<PHONE_NUMBER>898969
FPS TorchRL with SyncDataCollector on cpu at step 20000:<PHONE_NUMBER>72972
FPS TorchRL with SyncDataCollector on cpu at step 30000:<PHONE_NUMBER>851568
FPS TorchRL with SyncDataCollector on cpu at step 40000:<PHONE_NUMBER>653071
FPS TorchRL with SyncDataCollector on cpu at step 50000:<PHONE_NUMBER>083208
FPS TorchRL with SyncDataCollector on cpu at step 60000:<PHONE_NUMBER>865648
FPS TorchRL with SyncDataCollector on cpu at step 70000:<PHONE_NUMBER>112759
FPS TorchRL with SyncDataCollector on cpu at step 80000:<PHONE_NUMBER>348757
FPS TorchRL with SyncDataCollector on cpu at step 90000:<PHONE_NUMBER>9324306
FPS TorchRL with SyncDataCollector on cpu at step 100000:<PHONE_NUMBER>134446
FPS TorchRL with SyncDataCollector on cpu mean:<PHONE_NUMBER>549519
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with SyncDataCollector on cuda:0 at step 10000: 3904.9437622718906
FPS TorchRL with SyncDataCollector on cuda:0 at step 20000: 3288.8724744697624
FPS TorchRL with SyncDataCollector on cuda:0 at step 30000: 3437.67936578839
FPS TorchRL with SyncDataCollector on cuda:0 at step 40000: 3400.23242154962
FPS TorchRL with SyncDataCollector on cuda:0 at step 50000: 3167.2741743898305
FPS TorchRL with SyncDataCollector on cuda:0 at step 60000: 3348.8633532834688
FPS TorchRL with SyncDataCollector on cuda:0 at step 70000: 3145.3018116045278
FPS TorchRL with SyncDataCollector on cuda:0 at step 80000: 3288.962738376645
FPS TorchRL with SyncDataCollector on cuda:0 at step 90000: 3215.286298039423
FPS TorchRL with SyncDataCollector on cuda:0 at step 100000: 1305.8973618519733
FPS TorchRL with SyncDataCollector on cuda:0 mean: 3171.8938244285127

------------------------------------------------------------------------------------------

❯ python benchmarks/test_torchrl_vs_gym.py --num_workers=8 --frames_per_batch=80

Running 8 envs with 80 frames per batch (i.e. 10.0 frames per env).

...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.

A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiSyncDataCollector on cpu at step 10000: 1075.6177012710166
FPS TorchRL with MultiSyncDataCollector on cpu at step 20000: 1057.8420350759939
FPS TorchRL with MultiSyncDataCollector on cpu at step 30000:<PHONE_NUMBER>134151
FPS TorchRL with MultiSyncDataCollector on cpu at step 40000: 1059.3114595731113
FPS TorchRL with MultiSyncDataCollector on cpu at step 50000:<PHONE_NUMBER>077409
FPS TorchRL with MultiSyncDataCollector on cpu at step 60000: 1175.6983882270497
FPS TorchRL with MultiSyncDataCollector on cpu at step 70000: 1007.8040282808623
FPS TorchRL with MultiSyncDataCollector on cpu at step 80000:<PHONE_NUMBER>645523
FPS TorchRL with MultiSyncDataCollector on cpu at step 90000: 1462.7104738904702
FPS TorchRL with MultiSyncDataCollector on cpu at step 100000: 1253.7666695313287
FPS TorchRL with MultiSyncDataCollector on cpu mean:<PHONE_NUMBER>006343
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 10000: 3048.4352826811787
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 20000: 3262.14582928252
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 30000: 3217.5086060582817
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 40000: 3344.70669152022
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 50000: 3387.694047330587
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 60000: 3303.544515658997
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 70000: 3303.2518212246505
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 80000: 3550.6959714712016
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 90000: 3221.864689954487
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 100000: 2741.64395202144
FPS TorchRL with MultiSyncDataCollector on cuda:0 mean: 3225.940279205892
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiaSyncDataCollector on cpu at step 10000: 5388.365878725591
FPS TorchRL with MultiaSyncDataCollector on cpu at step 20000: 27384.66661225822
FPS TorchRL with MultiaSyncDataCollector on cpu at step 30000: 2353.42530702708
FPS TorchRL with MultiaSyncDataCollector on cpu at step 40000:<PHONE_NUMBER>430935
FPS TorchRL with MultiaSyncDataCollector on cpu at step 50000: 43965.45073375262
FPS TorchRL with MultiaSyncDataCollector on cpu at step 60000: 2589.4961374914146
FPS TorchRL with MultiaSyncDataCollector on cpu at step 70000: 1340.4240054009356
FPS TorchRL with MultiaSyncDataCollector on cpu at step 80000: 47153.50196739742
FPS TorchRL with MultiaSyncDataCollector on cpu at step 90000: 40790.702650133724
FPS TorchRL with MultiaSyncDataCollector on cpu at step 100000: 9932.636315197442
FPS TorchRL with MultiaSyncDataCollector on cpu mean: 1700.2908940649854
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 10000: 27906.214238190285
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 20000: 5383.3518369966305
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 30000: 19511.793917543757
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 40000: 75863.51345240787
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 50000: 3048.1029768447443
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 60000: 73875.89608102158
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 70000: 13952.526924196432
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 80000: 3091.463160707211
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 90000:<PHONE_NUMBER>6916234
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 100000: 2116.4113432233326
FPS TorchRL with MultiaSyncDataCollector on cuda:0 mean: 5223.255705987036
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with SyncDataCollector on cpu at step 10000:<PHONE_NUMBER>260014
FPS TorchRL with SyncDataCollector on cpu at step 20000:<PHONE_NUMBER>980839
FPS TorchRL with SyncDataCollector on cpu at step 30000:<PHONE_NUMBER>225008
FPS TorchRL with SyncDataCollector on cpu at step 40000:<PHONE_NUMBER>571342
FPS TorchRL with SyncDataCollector on cpu at step 50000:<PHONE_NUMBER>98069
FPS TorchRL with SyncDataCollector on cpu at step 60000:<PHONE_NUMBER>924778
FPS TorchRL with SyncDataCollector on cpu at step 70000:<PHONE_NUMBER>2008165
FPS TorchRL with SyncDataCollector on cpu at step 80000:<PHONE_NUMBER>805746
FPS TorchRL with SyncDataCollector on cpu at step 90000:<PHONE_NUMBER>863564
FPS TorchRL with SyncDataCollector on cpu at step 100000: 81.004056910206
FPS TorchRL with SyncDataCollector on cpu mean:<PHONE_NUMBER>961403
...
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with SyncDataCollector on cuda:0 at step 10000: 3152.338058867187
FPS TorchRL with SyncDataCollector on cuda:0 at step 20000: 2331.9178272593335
FPS TorchRL with SyncDataCollector on cuda:0 at step 30000: 3397.9515741931564
FPS TorchRL with SyncDataCollector on cuda:0 at step 40000: 1760.971529035136
FPS TorchRL with SyncDataCollector on cuda:0 at step 50000: 2330.816337871631
FPS TorchRL with SyncDataCollector on cuda:0 at step 60000: 2617.146244442711
FPS TorchRL with SyncDataCollector on cuda:0 at step 70000: 2050.628368881012
FPS TorchRL with SyncDataCollector on cuda:0 at step 80000: 2597.574781693194
FPS TorchRL with SyncDataCollector on cuda:0 at step 90000: 2078.3301228251644
FPS TorchRL with SyncDataCollector on cuda:0 at step 100000: 90.8393065937377
FPS TorchRL with SyncDataCollector on cuda:0 mean: 2424.713380369402

Thanks guys for looking into this.
Here's a few things about the current state of collection speed in torchrl:

As noted by @skandermoalla the throughput with cuda is superior than with cpu. The explanation is simply that it's faster to write a tensor from ram to cuda than from ram to shared storage (on physical mem). Simply put, it's better to use cuda whenever you can, you can also benefit from the speedup of executing your model on device.
There is also some overhead caused by tensordict. We've been optimizing tensordict to the core so I'm not very optimistic regarding making the instantiation much faster but there are some tricks we can use. In a not so distant future we may look at a c++ backend for tensordict which would speed things up drastically, but the best we can do for now is to be patient regarding that.
When executing a rollout with a ParallelEnv with CartPole (which resets often) we spend as much time calling reset than step. In other words, we could somewhat make that env 2x faster if we could automatically reset the env locally on the remote process, rather than gathering data on the main process, reading the done state, sending a reset signal and waiting for it to happen. This would be a new feature that would not be immediate to code. I will have a look into this in the upcoming days, but it looks like it could speed things up drastically for both parallel envs and collectors.

Work plan
Here is what I'm envisioning for this:

Currently, when encountering a "done" state, what happens is that the ("next", "done") is set to True. This is read as a signal that reset should be called: when called, the root "obs" is rewritten with a new value.
I first thought about this: When creating an env, you have the option of saying Env(..., auto_reset=True). If so, when an env encounters a ("next", "done") == True (forgive me for the messy syntax), we deliver all the new data in the "next" key and update the root with the result of "reset". We must be careful when doing this because now the root of the tensordict is at episode e+1 and step 0 when the "next" nested tensordict is at step T and episode e. This is a bit annoying as we'll need to do some gymnastic to make sure that we still have data[... :1] == data["next"][..., :-1]. This will be a very heavy weight to carry in the code base on the long term.
My current preferred option (although it's gonna be a massive rework) is this: we change the signature of env.step to output 2 distinct get a rollout that works as this:

def rollout_autoreset(self):
    result = []
    next_result = []
    cur_data = env.reset()
    for i in range(T):
        _cur_data, next_data = env.step(cur_data)

        # cur_data and next_data are well synced
        result.append((cur_data, next_data))

        # now step_mdp chooses between _cur_data and next_data based on the done state.
        # with envs that have a non-empty batch size, it can mix them together
        cur_data = step_mdp(_cur_data, next_data)

    result, next_result = [torch.stack(r) for r in zip(*result)]
    result.set("next", next_result)
    return result

Why is this better?
For ParallelEnv, this would mean that we can just call env.step on each sub-env. The _cur_data buffer is synced between processes only if needed, and we never call _reset as the individual processes take care of that. Same for the collectors. Another plus side is that we have less creation of tensordicts during a rollout, which will further speed things up.
How do we get there?
This is going to be hugely bc-breaking so it will have to go through prototyping + deprecation message (0.2.0, a bit later this year) -> deprecation + possibility of using the old feature (0.3.0, early 2024) -> total deprecation (0.4.0, somewhere in 2024). I expect the speedup to bring the envs closer to par with gym in terms of rollout and bring the data collection using collector at a superior speed than all other regular loops when executed on device (across sync and async).
I will open a PoC soon, hoping to get some feedback!
cc @smorad @matteobettini @shagunsodhani

This sounds like a lot of work. Would it be easier to just wrap the gym env in the Autoreset wrapper?

It seems quite a complex solution.

the use of a tuple output in the step_and_reset_function kind of goes against the tensordict principle of signatures no? If I follow the tensordict philosophy, outputing one tensordict only with a reset subtensordict would be more aligned with our tenets.
I do not understand why we need this new fuse_tensordict function in my view when we want to get the new tensordict we just get next and call reset on it:

def step_and_maybe_reset(td):

    tensordict = self._step(tensordict) # this is the current step in main. aka tensordict will have both root and next

    reset_td = tensordict.get("next").clone()
    done = tensordict.get(("next", self.done_key))
    _reset = done.clone()
    reset_td.set("_reset", _reset)
    self.reset(reset_td)

    return reset_td, tensordict

Something really strange is happening with the current torchrl version since I ran this benchmark 3 months ago.
By running any of the TorchRL collectors with 8 envs I get 72 CPU maxed at 100% and bottlenecking everything else.
This does not happen with the Gymnasium "vectorized" envs.
@vmoens any quick intuition? I tried running as last time with a docker setup on the same cluster, but also natively on a GCP machine and the max-out CPU problem is still present. Can you reproduce this?
This does not happen on my M1 macOS.
Some logs of the new performance.
# python benchmark.py --num_workers=8 --frames_per_batch=2_000

FPS Gym AsyncVectorEnv mean: 5299.461355240903

FPS TorchRL with MultiSyncDataCollector on cpu mean: 1409.2643700762942
FPS TorchRL with MultiSyncDataCollector on cuda:0 mean: 1434.0322984191869

FPS TorchRL with MultiaSyncDataCollector on cpu mean: 1539.6185885727893
FPS TorchRL with MultiaSyncDataCollector on cuda:0 mean: 1473.2058280519127

FPS TorchRL with SyncDataCollector on cpu mean:<PHONE_NUMBER>4849865
FPS TorchRL with SyncDataCollector on cuda:0 mean:<PHONE_NUMBER>205446

/mloraw1/moalla/open-source/torchrl/rl/benchmarks main*
implicit-pg ❯ python skander.py --num_workers=8 --frames_per_batch=2_000
Running 8 envs with 2000 frames per batch (i.e. 250.0 frames per env).
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS Gym AsyncVectorEnv at step 10000: 4331.723599469573
FPS Gym AsyncVectorEnv at step 20000: 5204.412883276483
FPS Gym AsyncVectorEnv at step 30000: 4185.867384286969
FPS Gym AsyncVectorEnv at step 40000: 5084.152751110335
FPS Gym AsyncVectorEnv at step 50000: 4690.31147382411
FPS Gym AsyncVectorEnv at step 60000: 6040.902036892344
FPS Gym AsyncVectorEnv at step 70000: 6695.075310147556
FPS Gym AsyncVectorEnv at step 80000: 6469.561408176754
FPS Gym AsyncVectorEnv at step 90000: 7322.151306509504
FPS Gym AsyncVectorEnv at step 100000: 5166.639771866397
FPS Gym AsyncVectorEnv mean: 5299.461355240903
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiSyncDataCollector on cpu at step 10000: 1320.6028552739604
FPS TorchRL with MultiSyncDataCollector on cpu at step 20000: 1446.4029971546481
FPS TorchRL with MultiSyncDataCollector on cpu at step 30000: 1430.5873492922874
FPS TorchRL with MultiSyncDataCollector on cpu at step 40000: 1550.6232173654255
FPS TorchRL with MultiSyncDataCollector on cpu at step 50000: 1319.6976557084743
FPS TorchRL with MultiSyncDataCollector on cpu at step 60000: 1365.2251196885925
FPS TorchRL with MultiSyncDataCollector on cpu at step 70000: 1430.506355384124
FPS TorchRL with MultiSyncDataCollector on cpu at step 80000: 1391.8008827041454
FPS TorchRL with MultiSyncDataCollector on cpu at step 90000: 1450.1957484061622
FPS TorchRL with MultiSyncDataCollector on cpu at step 100000: 1506.2325100349867
FPS TorchRL with MultiSyncDataCollector on cpu mean: 1409.2643700762942
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 10000: 1511.1733695354542
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 20000: 1394.4858962330331
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 30000: 1495.3765765560202
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 40000: 1542.618458368687
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 50000: 1522.7536971932934
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 60000: 1519.255640425426
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 70000: 1313.5330130432453
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 80000: 1442.4469610732624
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 90000: 1369.7093138217786
FPS TorchRL with MultiSyncDataCollector on cuda:0 at step 100000: 1410.1916111111952
FPS TorchRL with MultiSyncDataCollector on cuda:0 mean: 1434.0322984191869
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]

A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiaSyncDataCollector on cpu at step 10000: 8251.768916064895
FPS TorchRL with MultiaSyncDataCollector on cpu at step 20000: 3297.1962499272845
FPS TorchRL with MultiaSyncDataCollector on cpu at step 30000: 2217.987399000709
FPS TorchRL with MultiaSyncDataCollector on cpu at step 40000: 8748.476584294975
FPS TorchRL with MultiaSyncDataCollector on cpu at step 50000:<PHONE_NUMBER>853168
FPS TorchRL with MultiaSyncDataCollector on cpu at step 60000: 2189.470475220034
FPS TorchRL with MultiaSyncDataCollector on cpu at step 70000:<PHONE_NUMBER>479872
FPS TorchRL with MultiaSyncDataCollector on cpu at step 80000: 3426.168919777176
FPS TorchRL with MultiaSyncDataCollector on cpu at step 90000: 4175.730351396986
FPS TorchRL with MultiaSyncDataCollector on cpu at step 100000: 2373.595130464702
FPS TorchRL with MultiaSyncDataCollector on cpu mean: 1539.6185885727893
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 10000: 16983.39440811451
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 20000: 8616.284707100422
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 30000: 10179.842993249122
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 40000: 30913.891079549223
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 50000:<PHONE_NUMBER>207772
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 60000: 3726.762953649857
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 70000: 3346.7536887226006
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 80000: 3940.057969326326
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 90000: 4978.213290415082
FPS TorchRL with MultiaSyncDataCollector on cuda:0 at step 100000: 14703.31361465317
FPS TorchRL with MultiaSyncDataCollector on cuda:0 mean: 1473.2058280519127
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with SyncDataCollector on cpu at step 10000:<PHONE_NUMBER>569255
FPS TorchRL with SyncDataCollector on cpu at step 20000:<PHONE_NUMBER>528101
FPS TorchRL with SyncDataCollector on cpu at step 30000:<PHONE_NUMBER>849032
FPS TorchRL with SyncDataCollector on cpu at step 40000:<PHONE_NUMBER>6803537
FPS TorchRL with SyncDataCollector on cpu at step 50000:<PHONE_NUMBER>3102915
FPS TorchRL with SyncDataCollector on cpu at step 60000:<PHONE_NUMBER>2362986
FPS TorchRL with SyncDataCollector on cpu at step 70000:<PHONE_NUMBER>032164
FPS TorchRL with SyncDataCollector on cpu at step 80000:<PHONE_NUMBER>517329
FPS TorchRL with SyncDataCollector on cpu at step 90000:<PHONE_NUMBER>6557294
FPS TorchRL with SyncDataCollector on cpu at step 100000:<PHONE_NUMBER>0006296
FPS TorchRL with SyncDataCollector on cpu mean:<PHONE_NUMBER>4849865
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
A.L.E: Arcade Learning Environment (version 0.8.1+53f58b7)
[Powered by Stella]
Timer started.
FPS TorchRL with SyncDataCollector on cuda:0 at step 10000:<PHONE_NUMBER>186096
FPS TorchRL with SyncDataCollector on cuda:0 at step 20000:<PHONE_NUMBER>727289
FPS TorchRL with SyncDataCollector on cuda:0 at step 30000:<PHONE_NUMBER>0608746
FPS TorchRL with SyncDataCollector on cuda:0 at step 40000:<PHONE_NUMBER>049812
FPS TorchRL with SyncDataCollector on cuda:0 at step 50000:<PHONE_NUMBER>430556
FPS TorchRL with SyncDataCollector on cuda:0 at step 60000:<PHONE_NUMBER>5558116
FPS TorchRL with SyncDataCollector on cuda:0 at step 70000:<PHONE_NUMBER>4769
FPS TorchRL with SyncDataCollector on cuda:0 at step 80000:<PHONE_NUMBER>452704
FPS TorchRL with SyncDataCollector on cuda:0 at step 90000:<PHONE_NUMBER>2415577
FPS TorchRL with SyncDataCollector on cuda:0 at step 100000:<PHONE_NUMBER>455133
FPS TorchRL with SyncDataCollector on cuda:0 mean:<PHONE_NUMBER>205446

Weird, let me look into it

Yes, the SyncDataCollector is using ParallelEnv inside.

I'm trying to revert to the previous version to see if it's my environment that changed or something in TorchRL.

Could be due to https://github.com/pytorch/rl/pull/1532 or something like that...

Oh have you tried torch.set_num_threads(1)?
Collection should be faster and you should also reduce the loading of your cpus

I think you saved me hours/days of debugging and research time 🥹🥹

Cool! So that corroborates what I thought:

By default, torch takes num_threads = num_cpus
This is (arguably) ok on one worker, but if you add another it's trickier: both "see" the same number of cpus, and both allocate a number of threads equal to the number of cpus.
That quickly scales up and you end up in the situation you were referring too. It also explain why perf does not plateau when you add workers but actually decreases!
Note that as pointed by @matteobettini this number of threads is only related to torch operations (read and write tensors etc) so that should not impact what is happening anywhere else
#1559 will hopefully cope with that at the cost of requiring users to tell how many threads they want at each level if it does not match the default. For instance, if you need 8 workers for the collector and 4 workers for something else, you need to set the number of threads to 12 and do so when you create the collector. Not sure it covers all use cases but at least it solves a big issue in torchrl!

Btw, I reran the benchmark with a 3-month old TorchRL/Tensordict versions and got the same issue when not specifying the number of threads. I think it's probably a change in the cluster I'm using.
With the num threads specified, I get a 2x improvement on the current version of T TorchRL/Tensordict. Good reminder of the improvements done since then!

Hi guys, #1602 contains a simple script we used for the paper to benchmark against gym async collector. Feel free to run it and give us feedback.
I merge it quickly to have it available to everyone, but it needs a bit of polishing!
We have quite good numbers on CPU when the env has few resets. If resets are frequent (eg, CartPole) we are still underperforming a bit. This is something we'll be tackling in v0.3!
Perf on cuda has deteriorated recently and I can't really figure out why atm, stay tune for more info!

Hi,
I was trying to find a way to maximize the throughput in TorchRL. After spending a lot of time trying many different configuration. For "cpu" only on a laptop with 8 physical cores (Intel i9-11900H), I had the best result using MultiSyncDataCollector with as many collecter as workers, each of them managing a single env. Here is the pseudo-code:
import os

os.environ['OPENBLAS_NUM_THREADS'] = '1'
os.environ['MKL_NUM_THREADS'] = '1'
os.environ['OMP_NUM_THREADS'] = '1'
os.environ['VECLIB_MAXIMUM_THREADS'] = '1'
os.environ['NUMEXPR_NUM_THREADS'] = '1'
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ':4096:8'

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

train_env_creator = EnvCreator(
    lambda: GymWrapper(gym_make(), device="cpu"))

collector = MultiSyncDataCollector(
    N_WORKERS * (train_env_creator,),
    RandomPolicy(env.action_spec),
    frames_per_batch=80000,
    total_frames=40000000,
    preemptive_threshold=0.8,
    device="cpu",
    storing_device="cpu",
    num_threads=N_WORKERS,
    num_sub_threads=1
)

It is running at about 80% of the maximum theoretical speed (8 completely independent processes collecting samples on their own). After doing some profiling with py-spy top, it is quite clear where the 20% of the resources are going:
  %Own   %Total  OwnTime  TotalTime  Function (filename)
 51.00%  75.00%   220.0s    302.2s   step (jiminy_py/simulator.py)
  8.00%  10.00%   18.86s    20.93s   compute_command (common/blocks/proportional_derivative_controller.py)
  1.00%   1.00%   13.40s    13.41s   refresh_observation (common/blocks/mahony_filter.py)
  4.00%   5.00%   12.70s    31.29s   refresh_observation (common/bases/pipeline_bases.py)
  5.00%  17.00%   11.73s    45.57s   compute_command (common/bases/pipeline_bases.py)
  3.00%   3.00%   11.15s    11.20s   encode (torchrl/data/tensor_specs.py)
  2.00%   2.00%   10.35s    12.31s   inner_terminated_or_truncated (torchrl/envs/utils.py)
  2.00%   7.00%    9.42s    40.71s   _observer_handle (common/bases/generic_bases.py)
  2.00%   2.00%    8.82s     9.33s   compute_command (common/blocks/motor_safety_limit.py)
  2.00%  78.00%    6.87s    321.8s   step (common/envs/env_generic.py)
  1.00%   1.00%    5.99s     5.99s   __new__ (tensordict/tensordict.py)
  3.00%   6.00%    4.66s    13.74s   __init__ (tensordict/tensordict.py)
  2.00%   2.00%    4.33s     4.35s   _convert_to_tensor (tensordict/tensordict.py)
  0.00%   0.00%    3.87s     3.87s   refresh_observation (common/envs/env_generic.py)
  0.00%   0.00%    3.54s     3.54s   _reduce (common/utils/spaces.py)
  0.00%   0.00%    3.43s     3.43s   _forward (common/utils/spaces.py)
  1.00%   3.00%    3.23s     8.48s   _validate_value (tensordict/tensordict.py)
  0.00%   1.00%    2.86s     7.94s   _set (torchrl/envs/utils.py)
  0.00%   0.00%    2.71s     2.73s   __contains__ (tensordict/utils.py)
  0.00%  92.00%    2.67s    374.1s   _step (torchrl/envs/gym_like.py)
  0.00%   4.00%    2.63s    11.59s   _set_str (tensordict/tensordict.py)
  1.00%   1.00%    2.56s     4.44s   get (tensordict/tensordict.py)
  1.00%   5.00%    2.51s    10.87s   to (tensordict/tensordict.py)
  2.00%  80.00%    2.32s    325.8s   step (common/bases/pipeline_bases.py)
  0.00%   0.00%    2.29s     3.63s   __iter__ (torchrl/data/tensor_specs.py)
  1.00%   1.00%    2.09s     2.09s   to_numpy (torchrl/data/tensor_specs.py)
  1.00%   1.00%    2.07s     2.07s   _numba_unpickle (numba/core/serialize.py)
  0.00%   0.00%    2.02s     2.02s   zero (torchrl/data/tensor_specs.py)
  1.00%  24.00%    1.88s    82.43s   _controller_handle (common/bases/generic_bases.py)
  0.00%   5.00%    1.82s     8.99s   apply (tensordict/tensordict.py)
  0.00%   0.00%    1.69s     1.69s   items (torchrl/data/tensor_specs.py)
  0.00%   0.00%    1.63s     1.63s   __getitem__ (torchrl/data/tensor_specs.py)
  0.00%   0.00%    1.49s     1.55s   _get_str (tensordict/tensordict.py)
  0.00% 100.00%    1.48s    433.9s   rollout (torchrl/collectors/collectors.py)
  1.00%   3.00%    1.45s    11.10s   set (tensordict/tensordict.py)
  2.00%   2.00%    1.42s     6.45s   clone (tensordict/tensordict.py)
  0.00%   2.00%    1.41s    12.43s   step_mdp (torchrl/envs/utils.py)
  0.00%   2.00%    1.29s     5.25s   _step_proc_data (torchrl/envs/common.py)
  0.00%   0.00%    1.26s     1.28s   is_shared (tensordict/tensordict.py)
  0.00%  94.00%    1.22s    382.1s   step (torchrl/envs/common.py)
  0.00%   1.00%    1.20s     2.25s   _complete_done (torchrl/envs/common.py)
  0.00%   0.00%    1.18s     5.69s   reward_keys (torchrl/envs/common.py)
  0.00%   2.00%    1.18s    18.28s   terminated_or_truncated (torchrl/envs/utils.py)
  0.00%   0.00%    1.15s     1.32s   start (jiminy_py/simulator.py)
  0.00%   6.00%    1.14s    44.11s   _step_and_maybe_reset (torchrl/collectors/collectors.py)

Is there anything I could do to spend less time doing processing in TorchRL ? Especially those bottlenecks:
  3.00%   3.00%   11.15s    11.20s   encode (torchrl/data/tensor_specs.py)
  2.00%   2.00%   10.35s    12.31s   inner_terminated_or_truncated (torchrl/envs/utils.py)
  1.00%   1.00%    5.99s     5.99s   __new__ (tensordict/tensordict.py)
  3.00%   6.00%    4.66s    13.74s   __init__ (tensordict/tensordict.py)
  2.00%   2.00%    4.33s     4.35s   _convert_to_tensor (tensordict/tensordict.py)

Since I have implemented the gym environment from scratch, I can do whatever modification is necessary. For instance, returning TensorDict objects directly if this would help.

It's a rather long and thorough thread (thanks for it!) so I'll answer piece by piece.

It is running at about 50% of the maximum theoretical speed (8 completely independent processes collecting samples on their own).

That doesn't surprise me (unfortunately).
The "bet" that TorchRL is implicitly making is that truly vectorized environments (Isaac, VMAS, Brax etc) will eventually be the real thing and overhead caused by the various methods you're pointing at here will become less relevant in the grand scheme of things. Besides, one day we could get torch.compile to reduce this even further.

Since I have implemented the gym environment from scratch, I can do whatever modification is necessary. For instance, returning TensorDict objects directly if this would help. I could also implement a dedicated interface with TorchRL to avoid copies and stuffs whenever possible. Typically, my env only allocates memory at init and then only set values later on.

In this case I think the best could be to simply code your environment from EnvBase. It could be simpler.
The interface with gym can be complex to handle. Here are some anecdotal examples:

(decode) During rendering, some np.ndarray objects have a negative stride and must be copied to be transformed into tensor. Hence we have to check the stride of each array to make sure that this does not happen.
(encode) Some actions are np.ndarray, some are simple integers... We have to do a bunch of checks to cover all use cases when we see an action coming, there's no way to tell what it's gonna be in advance.

There's a lot more there we need to account for and every time someone comes with an env-specific bug we need a patch that eats up a bit of compute time...
Something odd (IMO) is that many users, even close collaborators, only see our envs as "wrappers" when "wrappers" are only a fraction of what envs are. I'm super biased to take this with a massive pinch of salt but in some way I think EnvBase could serve as a base for pytorch-based env like gym.Env historically did with numpy-based envs.
Regarding inner_terminated_or_truncated, we can speed it up too for sure.

Still, there are about 30% of the computation time that is spent somewhere else, but I still need to understand where. Actually, it seems to be stuck waiting for something about half of the time, but it is not clear to me why. It seems to be related to the use of multiprocessing.Pipe for sending/receving messages between the main thread and subprocesses. So it is probably due to the time required to send the observation and stuff back to the main process:

That's communication overhead I think. Simply writing and reading tensors... We made good progress speeding this up (eg. using mp.Event in parallel processes) but there's always more to do!
If you don't need sync data (eg, off-policy) MultiaSync is usually faster than MultiSync.
Back in the days sharing tensors with CUDA devices was way faster than CPU (shared mem) but for some reason the balance has now shifted to cpu. I have no idea why!
Is there any way you could share the script and the method you're using for benchmarking, for reproducibility?
Thanks!

After looking more closely, I realised that most of the slowdown was coming from the generic wrapper around gymnasium environments instead of interprocess communication:

If I run the same benchmark with a single environment and a base SyncDataCollector, it is already running 25% slower than when dealing with play gymnasium environment using gymnasium.make. So it seems than implementing my own lightweight torchrl wrapper for gym_jiminy is the way to go.

Good to know thanks for investigating this!
We should defo make these wrappers more efficient!
To be open about it, even if I acknowledge that there is room for improvement, I wonder if our time would not be better spent making the next gen of simulators (that are truly vectorized, like mujoco-mjx and isaac) work without effort with torchrl rather than optimizing for envs that are slow by nature. Happy to hear what your take is on that!

In my way, there is no such thing as "next gen of simulators" in the real world. Both are relevant and will remain relevant for there own respective application. I don't think it is realistic to expect a simulator to support both "batched gpu" mode and classic cpu mode. Yet, only classic cpu mode is relevant for all but RL applications, since in the vast majority of use cases you are only willing to run a single simulation at a time, not to mention critical embedded software. Still, it is very important to use the same simulator over the whole pipeline, from RL training to classical offline planning algorithm and online model-based predictive control. Apart from it, in various real-world training scenarios, the actual simulated system may change internally between episodes, for instance to challenge the same policy on various model of the same physical platform as an advanced form of domain randomization. In such a case, batched gpu simulation in not applicable. Finally, parrallel simulation is already fast enough for real-world R&D on complex systems such as humanoid robotics. For instance, it takes only 1h to collect 100M timesteps on a Macbook Pro M3. No need to go faster if you cannot iterate faster anyway because analyzing the results is time consuming anway. Finally, many complex algorithm are not yet ready for batched gpu simulation such as complex mesh-mesh collision detection algorithms, so if you want a very realistic simulation you need to fallback to cpu-based simulator. Plus, Mujoco and Isaac are definitely not famous for how realistic they are. To wrap up, to me, CPU-based simulation is definitely nowhere near dead.

I second @duburcqa's comment. Simulators like Issac and the new mujoco are awesome if you're doing pure RL algorithm development, but seem less useful for applied RL research where you spend most of your time making custom environments. In my use case, CPU environments provide a reasonable trade-off between simulation speed and development time early on in the design process, which is an important first step that should not be overlooked IMO.
Just my perspective from the applications side of things.

I also agree. Especially in the field of robotics it is always important to remember that the real world is not vectorized and online learning in the real world is something that will gain increasing attention.

Thanks all for the valuable feedback!
Those a really valid points.
So what's your take on this topic for torchrl then? What's the best way forward?
The overhead observed by @duburcqa is hard to solve because gym is not very explicit about what it returns: unlike torchrl I can't tell in advance if I will have a info dict or not, if my obs dictionary is complete or not, I can't even tell if my reward is a float or a numpy.ndarray... For these reasons we have to do multiple checks.
It was once suggested to me that we could do these checks for the first iteration and then stick by it with some sort of compiled code but I don't really see how to make that happen in a simple way.
One option is to document how to write a custom gym wrapper with no checks to improve the runtime.

I also agree. Especially in the field of robotics it is always important to remember that the real world is not vectorized and online learning in the real world is something that will gain increasing attention.

For sure but I don't think that applies to this case, where you'd wrap a gym env in torchrl and do checks over the types and devices etc. If you're working with a robot you will most likely have your own environment tailored for that use case. In other words, I don't think that this impacts whether or not we should dedicate a lot of effort to bridge a potential 20% runtime gap compared to gym async envs.

@duburcqa I forgot to ask: are you using tensordict nightlies or the latest stable version?

It was once suggested to me that we could do these checks for the first iteration and then stick by it with some sort of compiled code but I don't really see how to make that happen in a simple way.

All of this could be done only once, at init, since it is reasonable to expect that types do not change over steps. This way, it would not add any runtime cost. Ideally, the whole computation path should be defined statically once and for all, then called whenever it is necessary. here is an example where I do this. I agree it is quite tricky to implement but the performance benefit can be very significant for the hot path. Still, maybe it is not necessary to go this far and there is a trade-off between full static computation path and full runtime path.

One option is to document how to write a custom gym wrapper with no checks to improve the runtime.

I'm clearly fine with it !

If you're working with a robot you will most likely have your own environment tailored for that use case. In other words, I don't think that this impacts whether or not we should dedicate a lot of effort to bridge a potential 20% runtime gap compared to gym async envs.

I agree this is not the best argument.

For the record, here are the result of my benchmark for latest release (0.3.x):
    %Own   %Total  OwnTime  TotalTime  Function (filename)
   71.00%  86.00%   176.6s    225.9s   step (jiminy_py/simulator.py)
    4.00%   5.00%   12.19s    13.27s   compute_command (common/blocks/proportional_derivative_controller.py)
    3.00%   9.00%    7.59s    30.21s   compute_command (common/bases/pipeline_bases.py)
    0.00%   0.00%    7.18s     7.60s   compute_command (common/blocks/motor_safety_limit.py)
    0.00%   4.00%    6.89s    17.33s   refresh_observation (common/bases/pipeline_bases.py)
    1.00%   1.00%    6.10s     6.12s   refresh_observation (common/blocks/mahony_filter.py)
->  0.00%   0.00%    5.10s     6.76s   _apply_nest (tensordict/_td.py)
    1.00%   5.00%    4.27s    21.60s   _observer_handle (common/bases/generic_bases.py)
    0.00%  87.00%    3.82s    237.2s   step (common/envs/env_generic.py)
->  0.00%   0.00%    3.75s     5.17s   inner_terminated_or_truncated (torchrl/envs/utils.py)
    2.00%   2.00%    3.41s     3.41s   refresh_observation (common/envs/env_generic.py)
->  0.00%   0.00%    2.82s     2.87s   encode (torchrl/data/tensor_specs.py)
    0.00%   0.00%    2.20s     2.97s   _reduce (common/utils/spaces.py)
->  0.00%   0.00%    2.00s     4.32s   _set (torchrl/envs/utils.py)
->  0.00%   1.00%    1.95s     3.09s   get (tensordict/base.py)
->  1.00%   1.00%    1.83s     1.83s   read_done (torchrl/envs/gym_like.py)
    0.00%  87.00%    1.68s    239.7s   step (common/bases/pipeline_bases.py)
    2.00%  15.00%    1.58s    49.60s   _controller_handle (common/bases/generic_bases.py)
    1.00%   1.00%    1.53s     1.56s   _forward (common/utils/spaces.py)
->  1.00%   1.00%    1.52s     1.52s   __getitem__ (torchrl/data/tensor_specs.py)
->  0.00%   1.00%    1.50s     2.48s   _validate_value (tensordict/base.py)
->  2.00%  91.00%    1.48s    258.9s   _step (torchrl/envs/gym_like.py)
->  0.00%   0.00%    1.19s     1.46s   _aggregate_end_of_traj (torchrl/envs/utils.py)
    2.00%   2.00%    1.16s     1.29s   start (jiminy_py/simulator.py)
->  1.00%   1.00%    1.03s     8.16s   to (tensordict/_td.py)
->  0.00%   1.00%   0.980s     3.46s   _set_str (tensordict/_td.py)
->  0.00%   0.00%   0.950s    0.950s   __init__ (tensordict/_td.py)
    1.00%   1.00%   0.950s    0.950s   _numba_unpickle (numba/core/serialize.py)
    1.00%   1.00%   0.940s    0.940s   __subclasscheck__ (<frozen abc>)
->  0.00%   0.00%   0.860s    0.860s   zero (torchrl/data/tensor_specs.py)
    0.00% 100.00%   0.860s    291.0s   decorate_context (torch/utils/_contextlib.py)
->  0.00%   0.00%   0.840s     1.77s   _complete_done (torchrl/envs/common.py)
->  0.00%   1.00%   0.830s     2.48s   _update_traj_ids (torchrl/collectors/collectors.py)
->  0.00%   0.00%   0.810s     5.93s   step_mdp (torchrl/envs/utils.py)
->  0.00%   0.00%   0.780s    0.780s   items (torchrl/data/tensor_specs.py)
->  1.00%   1.00%   0.780s    0.860s   _get_str (tensordict/_td.py)
->  0.00%   0.00%   0.740s     1.83s   _stack_onto_ (tensordict/_td.py)
->  0.00%   0.00%   0.670s    0.670s   _parse_to (tensordict/utils.py)
->  0.00% 100.00%   0.670s    290.2s   rollout (torchrl/collectors/collectors.py)
    0.00%   0.00%   0.610s    0.640s   __call__ (llvmlite/binding/ffi.py)
    0.00%   0.00%   0.600s     1.07s   deepcopy (copy.py)
->  0.00%   1.00%   0.590s     1.14s   _get_tuple (tensordict/_td.py)
->  0.00%   1.00%   0.570s     3.09s   _step_proc_data (torchrl/envs/common.py)
    0.00%   0.00%   0.560s     1.98s   _call_with_frames_removed (<frozen importlib._bootstrap>)
    1.00%   1.00%   0.530s    0.530s   reset (jiminy_py/simulator.py)
->  0.00%   0.00%   0.520s    0.520s   to_numpy (torchrl/data/tensor_specs.py)
->  0.00%   0.00%   0.480s     3.35s   read_obs (torchrl/envs/gym_like.py)
    1.00%   1.00%   0.470s     1.15s   _setup (common/envs/env_locomotion.py)
->  0.00%   0.00%   0.460s    0.500s   _clone_value (tensordict/utils.py)
->  0.00%  93.00%   0.450s    264.0s   step (torchrl/envs/common.py)
    0.00%   0.00%   0.450s    0.450s   __iter__ (<frozen _collections_abc>)
    0.00%   0.00%   0.410s    0.410s   unwrapped (common/bases/pipeline_bases.py)
->  0.00%   0.00%   0.410s    0.410s   _exclude (tensordict/_td.py)
->  0.00%   0.00%   0.390s    0.490s   is_tensor_collection (tensordict/base.py)
    0.00%   0.00%   0.380s    0.550s   is_training (common/bases/pipeline_bases.py)
->  0.00%  97.00%   0.380s    280.2s   step_and_maybe_reset (torchrl/envs/common.py)
    0.00%   0.00%   0.370s    0.460s   compute_reward (common/envs/env_locomotion.py)
    0.00%   0.00%   0.360s    0.370s   __instancecheck__ (<frozen abc>)
->  0.00%   1.00%   0.350s     3.28s   set (tensordict/base.py)
    0.00%   0.00%   0.340s    0.340s   compute_command (common/envs/env_generic.py)
->  0.00%   0.00%   0.340s    0.650s   <listcomp> (tensordict/_td.py)
    0.00%   0.00%   0.330s    0.430s   compute_transform_contact (jiminy_py/dynamics.py)
->  0.00%   0.00%   0.320s     5.98s   _terminated_or_truncated (torchrl/envs/utils.py)
->  0.00%   0.00%   0.320s    0.650s   _check_keys (tensordict/utils.py)
    0.00%   0.00%   0.310s    0.880s   _deepcopy_dict (copy.py)
    0.00%   1.00%   0.300s     2.11s   _call_impl (torch/nn/modules/module.py)

It shows that 38.66s is spent in torchrl/tensordict methods for a total running time of 291.0s, namely 13.3%. It was 43.26s (14.8%) on 0.2.x release. So it is slightly better than before but not a game changer.

What do you include in those 38s?
I see to is taking 8 secs but for instance that's not something TensorDict could really help with (except if we revert to non_blocking=True which will be part of 0.3.1)

What do you include in those 38s?

The OwnTime of all lines marked with ->, ie related to either torchrl or tensordict.

Got it
Unfortunately here I don't see a single bottleneck that would bring you a significant speed up. Even if we reduce _apply_nest to 0 you will only gain 5 secs. I don't mean the work is done but I wouldn't expect a couple of PRs to improve things by an order of magnitude. I'll keep an eye on that trace and make my best to improve things!
Besides, correct me if I'm wrong but I think that being around 10% of runtime for torchrl can be a hit that most people are ready to take, and this for two reasons:

runtime isn't everything, when it comes to dev time you also want to be faster in coming up with solutions to your problems
more importantly measuring the runtime of torchrl will always give you the impression that we incur some extra cost that you would not have had with anything else. What it does not show is all the bits and pieces of code where torchrl and TensorDict speed things up for you compared to a naive implementation. For instance, there are plenty of tricks we use (like the non_blocking above, using torch.where, vectorizing value functions etc) where another code would be slower but a profile would just not show any torchrl or tensordict overhead.

Unfortunately here I don't see a single bottleneck that would bring you a significant speed up.

Yes, indeed. I don't think it is worth the effort at this point. Yet, the issue with profiling is that it alters the original timing. I'm observing a slowdown due to torchrl data collection against running episodes without torch and throwing away all the samples that is twice larger if with profiling disabled than enabled.

runtime isn't everything, when it comes to dev time you also want to be faster in coming up with solutions to your problems

I completely agree. As I said, it was acceptable before and it still is. IMHO, the rational is whether or not torchrl is competitive against other RL libraries targeting similar problems. In practice, I was stuck with ray[rllib] because it was at least twice faster than all other RL libraries until recently. torchrl is still not faster on my use-cases, but it has become very competitive. I can accept 10-15% slowdown if it is superior on some other aspects (modularity, maintainability, easy to use...) but hardly more personally.


refactor(app): Remove recalibrate option from POC and TLC overflow menus [RAUT-93]

Overview
The "recalibrate" option has been removed from the pipette offset calibration and tip length calibration overflow menus on the robot settings page when the enableCalibrationWizards feature flag is enabled.
closes RAUT-93
Changelog
-- Look for 'enableCalibrationWizards' feature flag and do not display recalibration options for both the pipette offset calibration and tip length calibration overflow menus.
Review requests

POC and TLC overflow menus render correctly
POC and TLC overflow menu "delete" and "download" options function as expected

Risk assessment
Very low risk at this point since it is hidden behind the feature flag. Further calibration work will also be affecting these venues.

The code looks good but I am unclear, according to figma, if we want to completely remove the option to calibrate?

My apologies, @jerader ! I misunderstood your comment to be asking if we should also remove the calibration option, not pointing out that I had already (inadvertently) removed it.
We do not want to remove it, at least not yet. I've adjusted my code and added tests to ensure it does indeed remain. Thank you for catching this!


Add gameserver ghost logic

Allows servers to reregister with the same id without any change to launcher code

I have no idea wtf github did to my code


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/804 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target.git

Docs Build status updates of commit a1956e9:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_master_2021-03-19-07-56-14 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


How to integrate if use Android Studio?

I'm a beginner with android. How to integrate this library if you use android studio?

Android Studio is still in Beta for NDK Integration and I have not attempted to try it.

add these lines in build.gradle files:
android {
.
.
.
sourceSets.main {
jniLibs.srcDir 'src/main/jnLibs' //set libs as .so's location instead of jniLibs
jni.srcDirs = [] //disable automatic ndk-build call with auto-generated Android.mk
}
}


Optimize LineEdit and TextEdit menu item generation.

Prevents unnecessary popup menu items regeneration, and premature popup menu creation when certain LineEdit/TextEdit properties are set (e.g. editable in ColorPicker).
Fixes Display control characters not being set.

Couldn't _generate_context_menu() just be called in the of _ensure_menu()? They are always called together.

Couldn't _generate_context_menu() just be called in the of _ensure_menu()? They are always called together.

Indeed, they always called together, merged both to the one function.

Thanks!


Adds dnsPolicy as a controller option to values.yaml and places an ex…

Adds dnsPolicy as a controller option to values.yaml and places an explicit definition into the Pod templates for Deployments and DaemonSets.
Following up #3656 to propose the addition of dnsPolicy as an option in values.yaml. This should make installation of nginx-ingress-controller a bit more flexible in this matter.

/assign @sameersbn


It's been a while let's update!

Already manually pushed to nuget today. Lets merge to master at the end of next week


When execute some command, the font will changed.

Windows 10, Chinese language version.
Powershell 7 Preview 2.
Changed the font to other font. Jetbrains Mono.
When execute some command, the powershell font will changed to default font.
Explame:
ping command.
The command was not execute:

The command is running...

@onebits Please remove PSReadline and check again.

@onebits Please remove PSReadline and check again.

Thank u, But its not worked.

@daxian-dbw Could you please comment?

Maybe @Dhowett has any thoughts.

This is a side effect of https://github.com/PowerShell/PSReadLine/issues/964
When PSReadline changes the codepage, the display font is changed to match.

If PowerShell is doing the same codepage maintenance as PSReadline (it might be), this would happen with the module removed.


Update the Sway book

A few things I've noticed to be missing or out of date from the docs:

I can't find any mention of the if let syntax in the docs

@mohammadfawaz I updated my description as I wasn't on the latest Sway book version. I naievely thought that refreshing the browser would load the latest version, but I was waaay back on 16.x :)

@mohammadfawaz I updated my description as I wasn't on the latest Sway book version. I naively thought that refreshing the browser would load the latest version, but I was waaay back on 16.x :)
Interestingly, @mauricewbr told me that he faced the same issue actually. We do need to fix this so that latest stays in the web address and doesn't get resolved to a fixed version.


Added @Override annotation

What changes were proposed in this pull request?
Added @Override annotation on getTableBuilder in PostgreSqlTableOperations
Why are the changes needed?
Fix: #3250

I don't think teh test failure was caused by this change.

Thank you for giving me the opportunity.


Update api.config.js

Hi, why are you submitting this machine translated config file to the repo? You shouldn't send a pull request of your own changes to upstream, you only need to update files in your own repo to deploy new changes.
Closing this for now as it's probably a mistake.


[Wikipedia] Move igor9silva to past contributors

Description
Removing myself from contributors since I haven't contributed to this extension for a long time. AI made wikipedia obsolete for me.
Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

Maybe that shouldn't give me credit 😅


Fix appliance_console unable to join external db

Apparently Temp.new(path) no longer works.
This fix passes in the expected filename first, including the
extension for good measure:
Temp.new([prefix, postfix], path)

https://bugzilla.redhat.com/show_bug.cgi?id=1217818
/cc @jrafanie

:+1:

@Fryguy cool. yea, I had that before but wasn't sure

:+1: will merge when green.

Nice @kbrock


implement all destination methods, add topic id, update source metadata

Description
Please include a summary of the change and what type of change it is (new feature, bug fix, refactoring, documentation).
Please also include relevant motivation and context.
List any dependencies that are required for this change.
Fixes # (issue)
Quick checks:

[ ] I have followed the Code Guidelines.
[ ] There is no other pull request for the same update/change.
[ ] I have written unit tests.
[ ] I have made sure that the PR is of reasonable size and can be easily reviewed.

And one more little thing. It seems that your Makefile has an invalid path to main.go file. Could you please remove the extra dash?
build:
-	go build -o conduit-connector-gcp-pubsub cmd/gcp-pubsub/main.go
+	go build -o conduit-connector-gcp-pubsub cmd/gcppubsub/main.go


Support typeof x === "symbol"

not sure what flow's support for Symbol in general is, but typeof x === "symbol" is not currently supported so we should fix that and whatever else is necessary for that to work properly.

Support for well-known symbols would be really handy. For example, if we understood Symbol.iterator (and plugged that into dynamic object keys), it would be possible to define custom iterators for object types (Generators let us define iterables today, but in a more limited way.)

It would be a welcome improvement for Flow to properly support typeof x === "symbol" and related stuff for ES2015 symbols in general. TypeScript seems to be well-ahead in this regard.

I think there are two things,
One is proper support for the primitive data type of symbol.
  6: export type EventName = string | symbol;
                                      ^^^^^^ symbol. Could not resolve name

Second is support for common symbols:
@@iterator

Open since 2015... come on guys, I can't check for "symbol"? Really??
In https://github.com/facebook/flow/issues/712#issuecomment-152624916 @mroch wrote

I fixed the exception in d45c099 and filed #1015 to make typeof x === "symbol" actually work.

Well...

Any update here?

so when will this actually happen?

:(

https://github.com/facebook/flow/blob/master/lib/core.js#L84 ?

@kevinSuttle the Symbol primitive if supported, but specifically using typeof with "symbol" is not.

Ping! 2015, and this is simple! For once, not just when saying so but when actually implementing it: All it takes is add "symbol" so that there is no complaint
Cannot compare the result of typeof to string literal symbol because it is not a valid typeof return value.

No code checks needed, just stop that error. Actual full symbol support is of course a larger issues, but at least letting us CHECK without Flow complaining is as simple as it gets. I actually have such checks in a few places, such as values getting send over the network, to throw an error saying symbols are not supported, exactly in line with Flow, so why is Flow complaining??? I'm only doing its work here.

@lll000111 Is there a PR with this?

  I never meant to troll, just trying to stay with my point of view.
I'm literally asking, did you see a PR which we could force? Because it will help.
Here's a PR which is imported to their internal repo. So it will be merged. And it's not the first time.
https://github.com/facebook/flow/pull/7271
If you want to continue crying about flow team unresponsiveness go to TS like people hyping these days.

Just a side note: Importing pull request does not mean that it will be merged. AFAIK FB has their internal review tools that they use instead of github. If importing means something, it can be interpreted in a way that they are considering the proposed change. If you are more interested in that particular issue ( #7271 ), there is pretty interesting, even philosophical conversation about the topic: https://github.com/facebook/flow/issues/7128 If you think that my opinion is the correct one, you can show support by thumbing it up :)
If you are willing to contribute but don't now how to climb the OCaml mountain, this might help you forward: https://github.com/facebook/flow/issues/4181#issuecomment-447596374
They have some (pretty sparse though) documentation about flow internals in their wiki as well: https://github.com/facebook/flow/wiki/The-Flow-Codebase:-Introduction

Merged. Will come in upcoming release.

I mean hardly - it appears it just makes it empty if you do this check so it is uncovered.  Not really Symbol support.  Especially when it seems to have typings for things.
Flow Try
Do we have any sense of when symbol support might exist in Flow?

https://flow.org/en/docs/types/primitives/
I suppose someone needs to remove the mentioned ticket

agree with @a7madgamal, the doc seem has not been updated for a while!

I noticed that while:
function testSymbol(x: mixed) {
  if (typeof x === "symbol") { // ok
    (x: string); // error
  }
}

errors correctly, the following does not:
let x: mixed;
if (typeof x === "symbol") {
  (x: string);
}

Is this difference in behavior intentional?


Getting missing file from job returns 500 instead of 404

Describe the bug
When I have a completed job and try to access a file that does not exist I expected to get a 404 note found error, but I get a empty html page with 500 status code.
To Reproduce
Steps to reproduce the behavior:

Start dev server inside devcontainer
Submit any job
Goto a file that does exist
Change end of url so it does not exist

Expected behavior
Page with Page not found message.
Additional context
In the server log
8:22:04 AM [vite] Internal server error: This stream has already been locked for exclusive reading by another reader
      at new ReadableStreamDefaultReader (/workspace/node_modules/web-streams-polyfill/dist/ponyfill.js:511:23)
      at AcquireReadableStreamDefaultReader (/workspace/node_modules/web-streams-polyfill/dist/ponyfill.js:472:16)
      at AcquireReadableStreamAsyncIterator (/workspace/node_modules/web-streams-polyfill/dist/ponyfill.js:866:22)
      at ReadableStream.values (/workspace/node_modules/web-streams-polyfill/dist/ponyfill.js:4181:20)
      at from (node:internal/streams/from:31:45)
      at Readable.from (node:internal/streams/readable:1727:10)
      at Object.toNodeRequest (/workspace/node_modules/@remix-run/dev/dist/vite/node-adapter.js:71:36)
      at nodeHandler (/workspace/node_modules/@remix-run/dev/dist/vite/plugin.js:845:35)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async /workspace/node_modules/@remix-run/dev/dist/vite/plugin.js:847:15

When run in production with for example the deploy/arq/docker-compose.yml it does return status code 404


test: add method to create the sui coin metadata

Description
It adds a test only function to create Sui currency in tests.
Test Plan
How did you test the new or updated feature?
We added a a file to test the function.

If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.
Type of Change (Check all that apply)

[ ] protocol change
[ ] user-visible impact
[ ] breaking change for a client SDKs
[ ] breaking change for FNs (FN binary must upgrade)
[ ] breaking change for validators or node operators (must upgrade binaries)
[ ] breaking change for on-chain data layout
[ ] necessitate either a data wipe or data migration

Release notes

Hi @josemvcerqueira, have you seen the coin::mint_for_testing and balance::create_for_testing helpers? They allow you to create coins and balances for any coin type in the test environment:
https://github.com/MystenLabs/sui/blob/473e4e25615fb47cd788aaba849a372189df52bc/crates/sui-framework/packages/sui-framework/sources/coin.move#L409-L413
https://github.com/MystenLabs/sui/blob/473e4e25615fb47cd788aaba849a372189df52bc/crates/sui-framework/packages/sui-framework/sources/balance.move#L118-L122

Hi @josemvcerqueira, have you seen the coin::mint_for_testing and balance::create_for_testing helpers? They allow you to create coins and balances for any coin type in the test environment:
https://github.com/MystenLabs/sui/blob/473e4e25615fb47cd788aaba849a372189df52bc/crates/sui-framework/packages/sui-framework/sources/coin.move#L409-L413
https://github.com/MystenLabs/sui/blob/473e4e25615fb47cd788aaba849a372189df52bc/crates/sui-framework/packages/sui-framework/sources/balance.move#L118-L122

Hello @amnn, yes I have.
These updates are to allow developers to access the CoinMetadata in tests.
Thank you for the response.

@damirka / @tzakian / @tnowacki, could you double check for me that this seems okay? It looks fine to me, and is serving a niche that I can't think of another way of serving, but given that it's related to SUI initialisation, wanted to get some more eyes on it.

Thanks for submitting the PR @josemvcerqueira!
The method you're suggesting won't allow simple tests without TestScenario, and relies on storage. Do you think it would work better if sui::coin had a method like new_coin_metadata_for_testing, and sui::sui could implement it in test_only mode? Eg. sui::sui::new_metadata_for_testing()?

Hello Damir, thank you for your suggestion.
I believe this would be better yes. Should the new sui::sui::new_metadata_for_testing()be a friend-only function for sui::sui similar to sui::balance::destroy_supply?

I updated the PR based on @damirka's feedback. Please let me know if there are additional changes :)


fix the bug with number format

fix the bug with number format

@hackers267
Thanks for the PR.
we will merge it.


Linux instructions incorrect

http://asciidocfx.com/#trueinstall-on-linux states that you need to cd into a bin directory and then run ./AsciidocFX.sh, but if you download the tarball then there's no bin directory and the top level script is call AsciidocFX

Thank you for the report Richard.

this is still the case, despite it is now correct in README.adoc in the github repository asciidocfx/AsciidocFX


[BUG] Torque "pulsates" on some drivetrain/engine configurations

Short Description
For as long as I've used ACF, I've constantly encountered a bug where torque seems to pulsate on and off as I accelerate, depending on the vehicle's drivetrain configuration, and what type of vehicle it is. The best description I can give is that it feels like I'll get a burst of torque to the wheels, and then the gearbox acts as if it's clutched for a second, delivering zero torque, and then I get another burst.
How to Reproduce (Optional)
I haven't found a way to perfectly reproduce this yet but I notice it happens on tanks with complicated drivetrains, and on almost every RWD/FWD vehicle I make to some degree.
Video
https://streamable.com/4wgxnb (the brief wheel spazzes are a symptom of this, not the cause)
Additional Information (Optional)
This problem is occasionally fixed by placing a differential between the engine(s) and primary gearbox, with ratios of 1/1. I don't have a 100% success rate with doing this though.

Better video of it happening - in the top right, I'm printing Link.ReqTq from the gearbox connected to the engine (added print(Ent, Link.ReqTq) after line 604 in entities/acf_engine/init.lua)
https://streamable.com/nt6stg
In the middle of the video, I shift into a high gear and turn hard. The torque seems to become consistent when the engine is under very high load and stuck at near idle RPM. Throttle is 100% the entire time.


Help on mfm port

Describe your problem here.
Im trying to port Mid-Fight Masses mod into Psych Engine and i need help  making the first cutscene on Parish
yknow that one when sarv is calling ruv and theres just a bg with nokiaPhoneCall sprite
how do i do it?
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No response

Not possible in lua


ghcid is slow to :reload

Hi Neil, for some reason ghcid is slow to reload on one of my laptops. Here is what I see for about a second after touching a file, when running with -v:
%WAITING: /home/mitchell/haskell/generic-websockets-server/.ghcid generic-websockets-server.cabal cabal.project /home/mitchell/haskell/generic-websockets-server/Main.hs
%WAITING: /home/mitchell/haskell/generic-websockets-server
%NOTIFY: Added "/home/mitchell/haskell/generic-websockets-server/4913" 2018-05-10 23:54:33.586551059 UTC
%NOTIFY: Modified "/home/mitchell/haskell/generic-websockets-server/4913" 2018-05-10 23:54:33.586556259 UTC
%NOTIFY: Removed "/home/mitchell/haskell/generic-websockets-server/4913" 2018-05-10 23:54:33.586557934 UTC
%WAITING: Notify signaled
%WAITING: Extra wait due to file removal
%NOTIFY: Added "/home/mitchell/haskell/generic-websockets-server/Main.hs~" 2018-05-10 23:54:33.58715762 UTC
%NOTIFY: Removed "/home/mitchell/haskell/generic-websockets-server/Main.hs" 2018-05-10 23:54:33.587152404 UTC
%NOTIFY: Added "/home/mitchell/haskell/generic-websockets-server/Main.hs" 2018-05-10 23:54:33.588629137 UTC
%NOTIFY: Modified "/home/mitchell/haskell/generic-websockets-server/Main.hs" 2018-05-10 23:54:33.590324537 UTC
%NOTIFY: Removed "/home/mitchell/haskell/generic-websockets-server/Main.hs~" 2018-05-10 23:54:33.590705737 UTC

Do these two lines indicate that ghcid is waiting for an extra second intentionally?
%WAITING: Notify signaled
%WAITING: Extra wait due to file removal

Thanks!

Yep, the extra wait is deliberate to work around the pattern where a text editor writes to A by first deleting it then writing it afresh. Are you using the same editor/OS on the laptop? What is the file 4913 doing - does that have any relevance to you? It may be a temp file I should ignore when it notifies.

Cool, I'm happy to implement this if you'd like!

Reading the messages more carefully, your ghcid is being woken up by the 4913 file, but in all likelihood that's happening roughly the same time as an edit, so isn't going to be a problem (if it does turn out to be the issue, we can filter it out). However, the 1s extra wait is only triggered for files it really was watching. I've just pushed fafdc5858eac24dae7498139e872ccd9495fcedf giving a better log message in that situation, saying which file had disappeared. Can you try that?

Ah! Thanks for that. nvim is actually deleting the haskell file itself before writing it back:
%NOTIFY: Removed "/home/mitchell/haskell/generic-websockets-server/Main.hs" 2018-05-10 23:54:33.587152404 UTC
%NOTIFY: Modified "/home/mitchell/haskell/generic-websockets-server/Main.hs" 2018-05-10 23:54:33.590324537 UTC

Yep, that's what it indicates - are you perhaps using an older nvim on the troublesome machine? Or some configuration that causes it to do the delete first? If there's no way to avoid it with nvim, then I could change from sleep 1s, to sleep in 0.1s increments waiting for the file to come back, which should make the latency way better.

There should be a way to configure nvim to behave differently, I just haven't figured it out yet. It's a bit of a puzzler since my .vimrc file is fairly bare-bones on all of my machines, and I've never had to explicitly opt out of this behavior in the past.
I'll track it down eventually, but in the meantime - perhaps ghcid could, instead of sleeping upon noticing a relevant file was removed, also act on that deleted file coming back? In my case, this would fix the latency issue, as Main.hs is written immediately after being removed.

Ghcid can do that, and it does. But you'll really upset ghci which is super buggy if files are deleted and then return...

Instead of waiting 1s, I wait up to 1s, but check every 0.05s if the file has returned, and if so stop waiting then. Hopefully that means in the common case you are only sleeping 0.05s which should be mostly imperceptible.

Much appreciated


Add SQL style guide

Added an example for SQL queries embedded in code to this upcoming

Reviewed and looks good to me.


Check for null focusedWindow before showing notification

Unity's documentation states that EditorApplication.focusedWindow is allowed to be null. Before referencing that value to call ShowNotification, MRTK should confirm that the value is valid.

/azp run


productPricing.getListingOffersBatch Error: Duplicate requests submitted.  Cannot run batch request.

Hi,
I'm getting the error Duplicate requests submitted.  Cannot run batch request.  with the following body on productPricing.getListingOffersBatch operation.
 getlistingoffersbatch documentation
related issues but there aren't solutions:

https://github.com/amzn/selling-partner-api-docs/issues/2410
https://github.com/amzn/selling-partner-api-docs/issues/2502

Is there something I overlooked?
{
    body: {
      requests: [
        {
          uri: '/products/pricing/v0/items/B07XD6F2ZK/offers',
          method: 'GET',
          MarketplaceId: 'A1PA6795UKMFR9',
          ItemCondition: 'New',
          CustomerType: 'Consumer',
        },
        {
          uri: '/products/pricing/v0/items/B08YF4W8S9/offers',
          method: 'GET',
          MarketplaceId: 'A1PA6795UKMFR9',
          ItemCondition: 'New',
          CustomerType: 'Consumer',
        },
        {
          uri: '/products/pricing/v0/items/B09ZYCV3JN/offers',
          method: 'GET',
          MarketplaceId: 'A13V1IB3VIYZZH',
          ItemCondition: 'New',
          CustomerType: 'Consumer',
        },
        {
          uri: '/products/pricing/v0/items/B00C1RXC64/offers',
          method: 'GET',
          MarketplaceId: 'A1PA6795UKMFR9',
          ItemCondition: 'New',
          CustomerType: 'Consumer',
        },
      ],
    },

Operation: getListingOffersBatch
Description
Returns the lowest priced offers for a batch of listings by SKU.

Must use getItemOffersBatch for ASIN

Operation: getItemOffersBatch
Description
Returns the lowest priced offers for a batch of items based on ASIN.


See issue #15631 and PR #15632 Fix a bug in return simulation for an …

…explanation. In summary the PR fixed a bug in #return:from: which allowed a simulation to execute an illegal return to a dead or nil context - unlike the VM which would call #cannotReturn: . The previous incorrect semantics of #return:from: was unfortunately taken advantage of if in some code and tests (see the references). In case of these two tests I'm not sure how the authors if the tests would like to fix them so leaving them as expected failures.
PS: I only pushed expected failures method;  I don't know why all symbols are being replaced with strings - this is not coming from my image... no idea why this is happening.
This is my change:
SindarinDebuggerTest >> expectedFailures [

	^ #( testIsExecutionFinished testStepOverFinishedExecution )
]

{ #category : 'helpers' }

@adri09070 could you look at the conflicts?

Actually, there is something I don't understand. Since the other PR (Pharo #15632) has been merged, I've never seen these twot tests fail. So I don't see the point in putting them as expected failures?

Hi @adri09070 , @StevenCostiou ,
thanks for checking, if the tests pass, we can indeed close the PR; I may have erroneously thought my PR15632 made the tests fail. Sorry for the noise.


Is animated(FunctionalComponent) not supported?

Sorry if this has been brought up before! I looked all over the docs and GitHub issues but couldn't find any mention of this
Is animated(Component) where Component is functional component a known limitation?
Here's a minimal example reproducing the issue: https://codesandbox.io/s/1mp77p57l
It also gives this warning which I assume is relevant
Warning: Function components cannot be given refs. Attempts to access this ref will fail. Did you mean to use React.forwardRef()?

But even when attempting to use a simple functional component with React.forwardRef (also done in the example above) it seems to fail silently

That used to be a limitation in React before hooks but you still need to opt in essentially. See: https://reactjs.org/docs/hooks-reference.html#useimperativehandle
This seems to work:
const FunctionalComponentWithRef = React.forwardRef(({ color }, ref) => {
  const myRef = React.useRef();
  React.useImperativeHandle(ref, () => ({}));

  return (
    <div
      ref={myRef}
      style={{
        background: color,
        width: 250,
        height: 250
      }}
    />
  );
});

Thank you @mcernusca that does work! I am rather confused how does it work though since as far as I understand useImperativeHandle we are basically doing nothing? (Not modifying the ref at all since we return an empty object)
Also why do we need to create and pass our own Ref that as far as I can tell is not used by anything?

All great questions.. actually even this works:
const FunctionalComponentWithRef = React.forwardRef(({ color }, ref) => {
  React.useImperativeHandle(ref, () => ({}));
  return (
    <div
      style={{
        background: color,
        width: 250,
        height: 250
      }}
    />
  );
});

🤷♂

I think we should investigate this further, really weird behaviour

This is because of https://github.com/react-spring/react-spring/blob/master/src/animated/createAnimatedComponent.tsx#L91-L94
It needs the reference because it has to write values to it, though for a function component it could probably just detect that it's one and skip this step, it would call forceUpdate anyway to update it, since it doesn't have a dom representation. But that's the weird part, with forwardRef fn-comps could actually made so that they return their inner div as ref - in that case "native" animation could work.
Basically ... is it possible to detect a function component that does not forward refs?

It should be yes!
Here is the implementation from is-react:
https://github.com/treyhuffine/is-react/blob/master/index.js#L12

that's hella dirty for a check that's basically running 60fps, though. 😕

I solved this just by wrapping my "stateless functional components" which were coming from a library inside  an HOC... I don't know if it's a proper fix
function makeClassComponent(WrappedComponent) {
  return class extends React.Component {
    render() {
      return <WrappedComponent {...this.props} />;
    }
  };
}
// make my SVG animated
const AnimatedCircle = animated(makeClassComponent(Circle));

https://codesandbox.io/s/0y1ooj8n3l?fontsize=14

While writing my original answer mentioning is-react I did not realise that the React team published their own library react-is which has a much better implementation, though it doesn't have a way to tell functional components from class based ones, but it's easy to tell that something is a class based component component.prototype.isReactComponent so any other function ought to be a functional component?
Not sure if this solution is satisfactory or performant enough though, so pardon me!

@Mathspy That looks quite good, didn't know about it thanks for digging it up! I will take another look after fixing some issues that were piling up here.

@drcmda Good luck and thank you for the amazing work! ❤

@mcernusca's version is working but I think there are other ways to workaround the forwardRef warning.
Please have a look at the following Codesandbox.
As mentioned, it will also work with-out myRef but I've used that version in the example.
The useImperativeHandle Hook is a bit diffcult to understand for me - I'm pretty new to hooks. But as mentioned in the docs it should be avoided. As I understand, it will make the reference available for React-Spring but the warning is still present - maybe because of the empty object.
With-out useImperativeHandle the animation is not working.

I read that forwardRef will be made obsolete soon in React, that would be one worry less. :-)

I just ran into the same issue, and after more than a week of re-writing code probably 100 times, I ended up also using useImperativeHandle(), and that led me to this issue (that I wish I had found a week ago).

Sorry for letting you wait so long. I hope it's "fixed" now, refs in React are still weird. I do a simple check and if it's a plain function, it doesn't try to set a reference on it.