Notes on shuffling, sharding, and batchsize

(I'm writing this down here to have a written trace, but I'm looking forward to discuss this with you all in our upcoming meetings :) )
I spent some time porting the torchvision training recipes to use datapipes, and I noticed that the model I trained on ImageNet with DPs was much less accurate than the one with regular datasets. After a lot of digging I came to the following conclusion:

the datapipe must be shuffled before it is sharded
the DataLoader does not behave in the same way with a datapipe and with a regular indexable dataset, in particular when it comes to size of the last batches in an epoch. This has a dramatic effect on accuracy (probably because of batch-norm).

Details below. Note: for sharding, I used this custom torchvision sharder which takes DDP and dataloader workers into account, + the TakerIterDataPipe below it.

Shuffle before shard
First, some quick results (training a resnext50_32x4d for 5 epochs with 8 GPUs and 12 workers per GPU):
Shuffle before shard: Acc@1 = 47%  -- this is on par with the regular indexable dataset version (phew!!)
Shuffle after shard: Acc@1 = 2%
One way to explain this is that if we shuffle after we shard, then only sub-parts of the dataset get shuffled. Namely, each of the 8 * 12 = 96 dataloader workers receive ~1/96th of the dataset, and each of these parts get shuffled. But that means that the shuffling is far from uniform and for datasets in which the layout is all_samples_from_class1, all_samples_from_class2, ... all_samples_from_classN, it's possible that some class i is never in the same batch as class j.
So it looks like we need to shuffle before we shard. Now, if we shuffle before sharding, we still need to make sure that all of the 96 workers shuffle the dataset with the same RNG. Otherwise we risk sampling a given sample in more than one worker, or not at all. For that to happen, one can set a random seed in worker_init_fn, but that causes a second problem: the random transformations of each worker will also be the same, and this will lead to slightly less accurate results; on top of that, all epochs will start with the same seed, so the shuffling is the same across all epochs. I do not know how to solve this problem yet.
Note that TF shuffles the dataset before storing it. We might do something similar, but that would still not solve the issue for custom users datasets.

Size of the batches at the end of an epoch
Some quick results (same experiment as above):
with drop_last=True: Acc@1 = 47%
with drop_last=False: Acc@1 = 11%
Near the end of the epoch, the dataloader with DP will produce a lot of batches with size 1 if drop_last is False. See the last batches of an epoch on indices from [0, len(imagenet)) with a requested batch size of 32: https://pastebin.com/wjS7YC90. In contrast, this does not happen when using an indexable dataset: https://pastebin.com/Rje0U8Dx.
I'm not too sure of why this has such a dramatic impact, but it's possible that this has to do with batch-norm, as @fmassa pointed out offline. Using drop_last will make sure that the 1-sized batches are eliminated, producing a much better accuracy.
I guess the conclusion here is that it's worth unifying the behaviour of the DataLoader both DPs and regular indexable datasets regarding the batch size, because with indexable datasets and drop_last=False we still get ~47% acc.

Thank you for adding the detailed notes.

So it looks like we need to shuffle before we shard. Now, if we shuffle before sharding, we still need to make sure that all of the 96 workers shuffle the dataset with the same RNG. Otherwise we risk sampling a given sample in more than one worker, or not at all.

I totally understand this problem. This is one thing we want to solve during this half. For each shuffler, it should contain their own RNG rather than using a global RNG. For the same seed, I am actually not sure if we want to have the same seed across processes, I believe you want the same seed to make sure all data are covered by a single epoch. In the long run, if the seed for each shuffler is different with each other, the performance may be similar. (Need an experiment)

Near the end of the epoch, the dataloader with DP will produce a lot of batches with size 1 if drop_last is False. See the last batches of an epoch on indices from [0, len(imagenet)) with a requested batch size of 32: https://pastebin.com/wjS7YC90. In contrast, this does not happen when using an indexable dataset, where the dataloader will produce batches of size 18: https://pastebin.com/Rje0U8Dx.

Dataset produces 8 batches with size 18. IterDataPipe produces 96 batches with size 1 at the last iteration. This should be some discrepancy over sampling mechanism in DataLoader. Will take a look

So it looks like we need to shuffle before we shard. Now, if we shuffle before sharding, we still need to make sure that all of the 96 workers shuffle the dataset with the same RNG. Otherwise we risk sampling a given sample in more than one worker, or not at all.

I might be missing something obvious here. If you shuffle before sharding, can't you make worker_0 take the first 1/96 samples, worker_1 takes the second 1/96 and so on?
This should make sure each sample is used by at least one worker.
In that case, each worker can still have its own RNG for random transformations.

Dataset produces 8 batches with size 18. IterDataPipe produces 96 batches with size 1 at the last iteration. This should be some discrepancy over sampling mechanism in DataLoader. Will take a look

I went through the example again and found the reason about this discrepancy.
For Dataset, it relies on the sampler on main process to distribute the workload. For the last batch, the main process would send a batch of indices to a single worker process per distributed process. That's why each rank would send 1 batch with size 18 at the end.
For IterDataPipe, each process would own 1/96 dataset on each process evenly. Each worker process would take care of the last batch by themselves -> 1 sample. Then, each rank (distributed process) would return 12 batches with size 1.
In order to solve it, IMHO, we have two options:

Make sure batching only happens on a single process a.k.a. non-shardable DataPipe
At the end of the pipeline, create a non-shardable DataPipe to aggregate samples from workers to generate batches. (This would be similar to the internal data source structure -> making sure batching only happens at the end.)

cc: @VitalyFedyunin

Do you think adding a batch_size to DataLoaderV2 to guarantee batching at the end can be an option? Then, users can still do batching/unbatching in the pipeline.
cc: @VitalyFedyunin

I believe you want the same seed to make sure all data are covered by a single epoch. In the long run, if the seed for each shuffler is different with each other, the performance may be similar. (Need an experiment)

@ejguan I think ensuring that each image is processed only once per training epoch is a very common and reasonable assumption that we should maintain. Note that there techniques such as Repeated Augmentation that rely on agreeing on using a single subset of data across processes and then generate augmentations on top of them. Though there might be a way to go around it with your new API/approach, this technique shows that using different seeds across processes has a material effect on accuracies and can't be assumed to be necessarily similar.

I'm not too sure of why this has such a dramatic impact, but it's possible that this has to do with batch-norm, as @fmassa pointed out offline.

@NicolasHug This is a very reasonable guess especially since you proven that many batches get a sample size of 1. This is very likely to make the mini-batch statistics extremely noisy and break training. There is a way to confirm that this is the problem. You can replace BN with SyncBatchNorm to synchronize the statistics across the different batches. It's going to be slower (so it's not a viable solution to your problem) but it's going to give you evidence on whether the BN is to blame here. Not sure how useful this would be for you though as I agree with you that at any case we should consider aligning the behaviour with the old DataLoader.

@datumbox Thanks for your insights on the effect of random seed. I agree the shuffler should share the same seed across processes to make sure all images are processed by an epoch.
Here is my thinking about the random seed. Whatever random operations before sharding should use the same random seed to make sure the whole datasets are the same across process.
I wish I can gather your opionion on the random operations like random augmentation after sharding? Do you expect they use different random seed?  For example, we have two shards:

shard1: images [0, 2, 4, 6, 8]
shard2: images [1, 3, 5, 7, 9]
Do we want to have image_0 and image_1 having the same augmentation effect?
I think current implementation of DataLoader would use different random seeds based on the worker number. Do you still expect the similar behavior?

I think ensuring that each image is processed only once per training epoch is a very common and reasonable assumption that we should maintain.

@datumbox Hi! I'm an external user and was just following this thread as I am also very interested in sharding/shuffling behavior with DataPipes. I noticed your comment above and just wanted to clarify one thing:
The assumption of "each image is processed only once per training epoch" is not guaranteed true when DDP is used along with drop_last: False, even for indexed datasets. PyTorch requires equal device batch sizes at each step, and so at the end of the epoch, if some GPUs have a smaller final batch than others, they repeat some samples to get a full final batch.
You can see the codepath in DistributedSampler here: https://github.com/pytorch/pytorch/blob/9d05ce602e1610e43f2aaa904b34d6946f8e7a1a/torch/utils/data/distributed.py#L104-L110
Example with 10 samples, 4 GPUs, device_batch_size=2, global_batch_size=8, shuffle=False, drop_last=False:
Step 1: device batches are {0: [0, 4], 1: [1, 5], 2: [2, 6], 3: [3, 7]}
Step 2: device batches are {0: [8], 1: [9], 2: [0], 3: [1]}

Devices 2 and 3 have to repeat some samples on Step 2 in order for PyTorch DDP to work. So within a single epoch, a total of 12 samples have been processed, not 10. Note that if only 1 or 2 GPUs had been used, then exactly 10 samples would have been processed.
Anyways, just wanted to share, the difference in behavior is pretty small, especially when shuffe=True, the 2 repeated samples are drawn randomly each epoch. But basically I think it would be OK for DataPipes to also repeat some samples per-device in DDP mode as we never had that guarantee to begin with.

@abhi-mosaic Thanks for the input.
Though I agree with the technical details you provided, it's important to note how the two points being made here differ:

Currently it's true that Batch padding is very common and at the end of the epoch more samples are drawn if drop_last=False. In this case as you correctly noted above, it's going to be a very small number of images as the number of duplicate images can't exceed the total batch size. This is not likely to affect training much but still creates issues on the inference side (@NicolasHug has this documented nicely at https://github.com/pytorch/vision/issues/4559).
This discussion focuses primarily  on whether the seed on the shuffler should be the same across all processes. If this is not true, then the amount of duplicate images sampled within the epoch will be much more significant than on the above.

So the assumption of having each image is processed only once per training epoch, is in practice true at the moment, with exception the corner-case of batch padding listed above and that's what I'm advocating we should maintain. If we don't keep the seed the same, we risk being unable to reproduce past results using the new API, introducing large systematic biases on training and being unable to implement techniques similar to Repeated Augmentation properly.

I think it would be OK for DataPipes to also repeat some samples per-device in DDP mode

Agreed and that's why I think we should make sure that the new solution behaves similarly and predictably to the old implementation.

@ejguan I think your suggestion of keeping the seed constant across workers up until the sharding is reasonable. As @datumbox   re-itereated, we need different seeds after that. Let's discuss this more in depth in today's meeting, along with a robust user-friendly API, following up on https://github.com/pytorch/data/issues/352#issuecomment-1091758594

The user friendly API would be welcome :smile:
My understanding was that DataLoader could be freed from shuffling if using datapipes, but this is not the case as shown in the example below.
I'm trying to implement a quite simple datapipe (which involves only shuffling and mapping) and it doesn't really work.
In the following simplified example, I can't get the shuffling behavior (nor the expected worker behavior, but this was already reported in #352)
from torch.utils.data.datapipes.iter import IterableWrapper
from torch.utils.data import DataLoader

def dummy_dp(n=10):
    dp = IterableWrapper(range(n))
    dp = dp.shuffle()
    return dp

dp = dummy_dp(10)
dl = DataLoader(dp, batch_size=None, shuffle=False, num_workers=1, pin_memory=False, drop_last=False)
print([e for e in dl])

would give
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

I would really appreciate a simple working example of the above. This is, with shuffling in the datapipe only and num_workers>1. Or an explanation of what to do before DataLoaderV2

Answering my own question, setting shuffle=False in the DataLoader will stop the shuffling in the IterDataPipe. Setting shuffle=None will let things be as they should be. This is now the default.
What is non intuitive to me is the false separation between the datapipe and the dataloader. The dataloader still has control over the shuffling! I leave this only as an anecdote, you know way better than me the reasons behind this choice

Find here one example showing how things are not yet solved :cry:
I would love to know if you came up with a solution in your meeting
from typing import Iterator, Any
from torch.utils.data.datapipes.iter import IterableWrapper
from torch.utils.data import DataLoader
import torch
from torch.utils.data.datapipes.datapipe import IterDataPipe
import torch.distributed as dist

class SharderDataPipe(torch.utils.data.datapipes.iter.grouping.ShardingFilterIterDataPipe):
    def __init__(self, source_datapipe: IterDataPipe) -> None:
        super().__init__(source_datapipe)
        self.rank = 0
        self.world_size = 1
        if dist.is_available() and dist.is_initialized():
            self.rank = dist.get_rank()
            self.world_size = dist.get_world_size()
        self.apply_sharding(self.world_size, self.rank)

    def __iter__(self) -> Iterator[Any]:
        num_workers = self.world_size
        worker_id = self.rank
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is not None:
            worker_id = worker_id + worker_info.id * num_workers
            num_workers *= worker_info.num_workers
        self.apply_sharding(num_workers, worker_id)
        yield from super().__iter__()

def dummy_dp_0(n, num_workers):
    dp = IterableWrapper(range(n))
    dp = dp.shuffle()
    dp = SharderDataPipe(dp)
    return dp

def dummy_dp_1(n, num_workers):
    dp = IterableWrapper(range(n))
    dp = SharderDataPipe(dp)
    dp = dp.shuffle()
    return dp

def dummy_dp_2(n, num_workers):
    dp = IterableWrapper(range(n))
    dp = dp.shuffle(buffer_size=n)
    dp = SharderDataPipe(dp)
    dp = dp.shuffle()
    return dp

num_workers = 2
for dp in [dummy_dp_0(n=10, num_workers=num_workers),
           dummy_dp_1(n=10, num_workers=num_workers),
           dummy_dp_2(n=10, num_workers=num_workers)]:
    torch.manual_seed(0)
    dl = DataLoader(dp, batch_size=None, shuffle=None, num_workers=num_workers, pin_memory=False, drop_last=False)
    print([e for e in dl])

gives
[3, 7, 0, 1, 5, 6, 6, 0, 9, 4]  # shuffled but with repeated items
[2, 3, 4, 7, 0, 5, 6, 1, 8, 9]  # shuffled but with even number at even indices
[5, 7, 9, 0, 3, 4, 0, 6, 6, 1]  # shuffled but with same repeated items as in the first one

What seems intuitive to me is that each worker has access to a different local seed and to a global seed. Then in .shuffle the user could be able to define which seed to use. I know managing random numbers is not easy, but this could be useful for other purposes (e.g. if we want to apply the same random operation over the whole batch). It would be nice to know how you solved this

@franchesoni
Thank you for taking exploration of DataPipe and DataLoader.

dl = DataLoader(dp, batch_size=None, shuffle=False, num_workers=1, pin_memory=False, drop_last=False)

Answering my own question, setting shuffle=False in the DataLoader will stop the shuffling in the IterDataPipe. Setting shuffle=None will let things be as they should be. This is now the default.

Yeah. Either setting shuffle=True or shuffle=None would make turn on shuffle in DataPipe graph. I think the reanson that the default value if shuffle=None (enable shuffle by default) is that it should reflect the same behavior if users iterate directly over DataPipe rather than using DataLoader.

@franchesoni

Find here one example showing how things are not yet solved 😢 I would love to know if you came up with a solution in your meeting
from typing import Iterator, Any
from torch.utils.data.datapipes.iter import IterableWrapper
from torch.utils.data import DataLoader
import torch
from torch.utils.data.datapipes.datapipe import IterDataPipe
import torch.distributed as dist

class SharderDataPipe(torch.utils.data.datapipes.iter.grouping.ShardingFilterIterDataPipe):
    def __init__(self, source_datapipe: IterDataPipe) -> None:
        super().__init__(source_datapipe)
        self.rank = 0
        self.world_size = 1
        if dist.is_available() and dist.is_initialized():
            self.rank = dist.get_rank()
            self.world_size = dist.get_world_size()
        self.apply_sharding(self.world_size, self.rank)

    def __iter__(self) -> Iterator[Any]:
        num_workers = self.world_size
        worker_id = self.rank
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is not None:
            worker_id = worker_id + worker_info.id * num_workers
            num_workers *= worker_info.num_workers
        self.apply_sharding(num_workers, worker_id)
        yield from super().__iter__()

def dummy_dp_0(n, num_workers):
    dp = IterableWrapper(range(n))
    dp = dp.shuffle()
    dp = SharderDataPipe(dp)
    return dp

def dummy_dp_1(n, num_workers):
    dp = IterableWrapper(range(n))
    dp = SharderDataPipe(dp)
    dp = dp.shuffle()
    return dp

def dummy_dp_2(n, num_workers):
    dp = IterableWrapper(range(n))
    dp = dp.shuffle(buffer_size=n)
    dp = SharderDataPipe(dp)
    dp = dp.shuffle()
    return dp

num_workers = 2
for dp in [dummy_dp_0(n=10, num_workers=num_workers),
           dummy_dp_1(n=10, num_workers=num_workers),
           dummy_dp_2(n=10, num_workers=num_workers)]:
    torch.manual_seed(0)
    dl = DataLoader(dp, batch_size=None, shuffle=None, num_workers=num_workers, pin_memory=False, drop_last=False)
    print([e for e in dl])

gives
[3, 7, 0, 1, 5, 6, 6, 0, 9, 4]  # shuffled but with repeated items
[2, 3, 4, 7, 0, 5, 6, 1, 8, 9]  # shuffled but with even number at even indices
[5, 7, 9, 0, 3, 4, 0, 6, 6, 1]  # shuffled but with same repeated items as in the first one

What seems intuitive to me is that each worker has access to a different local seed and to a global seed. Then in .shuffle the user could be able to define which seed to use. I know managing random numbers is not easy, but this could be useful for other purposes (e.g. if we want to apply the same random operation over the whole batch). It would be nice to know how you solved this

As a work around for now, you have to provide worker_init_fn to DataLoader to make sure the global seed is shared across workers. And, shuffle is using python random rather than torch.random.
For

[2, 3, 4, 7, 0, 5, 6, 1, 8, 9]  # shuffled but with even number at even indices

This is the result if you do sharding before shuffle as you are shuffling each shard of dataset rather than the whole dataset.
And, the ultimate solution should be something like the following (This needs DataLoader2 landed into TorchData):
dp = IterableWrapper(range(n))
dp = dp.shuffle()
dp = SharderDataPipe(dp)
dl = DataLoader2(dp, ...)

for epoch in range(2):
    dl.set_seed(0)
    print(list(dl))

There are a few features need to be done:

For each shuffle DataPipe, an isolated RNG is attached to this instance. The random op (shuffle) is bounded to this specific rng.
DataLoader2 would generate seed and send seed to each DataPipe with RNG state.

Thank you for the explanation. If I understand well, in the future, a shuffle datapipe will carry a random number generator (RNG) that will be in sync when the datapipe is distributed across workers. This RNG will be initialized with a seed provided by DataLoader2. This would also solve the augmentation problem, as by default will depend on different seeds on different workers (this solution is unchanged).
Do we have today any solution that solves both shuffling and augmentation?
A modification of your example

dp = IterableWrapper(range(n))
dp = dp.shuffle()
dp = SharderDataPipe(dp)
dp = dp.map(lambda x: random_augment(x))  # franchesoni added this line
dl = DataLoader2(dp, ...)

for epoch in range(2):
    dl.set_seed(0)
    print(list(dl))

should work well. But where (or when) can I access all these new cool functionalities?

@franchesoni
My current plan is to add a DataPipe to wrap a RNG for argumentation. So the example will become:
dp = IterableWrapper(range(n))
dp = dp.shuffle()
dp = SharderDataPipe(dp)
dp = dp.map(lambda x: random_augment(x))  # franchesoni added this line
dp = dp.random_op(rng=random.Random())  # the API is not determined yet.
dl = DataLoader2(dp, ...)

for epoch in range(2):
    dl.set_seed(0)
    print(list(dl))

LMK if this design works for you. And, for random_op, DataLoader2 will provide different seed based on workers.
And, this functionality only applies to DataLoader2. For BC of DataLoader, I might provide another API for shuffle to set seed for the isolated RNG, and you can still use worker_init_fn to control the random state for augmentation.
The example should be:
dp = IterableWrapper(range(n))
dp = dp.shuffle().set_seed(0)
dp = SharderDataPipe(dp)
dp = dp.map(lambda x: random_augment(x))  # franchesoni added this line
dp = dp.random_op(rng=random.Random())  # the API is not determined yet.
dl = DataLoader2(dp, ...)

for epoch in range(2):
    dl.set_seed(0)
    print(list(dl))

But where (or when) can I access all these new cool functionalities?

Adding isolated RNG and API would be my high-pri next week. DataLoader2 should be landed to TorchData around this week.

dl.set_seed(0)

This might be ambiguous with all the other types of RNGs that are at play here. It's not clear from the API that this only affects the shuffling seed. Perhaps set_epoch() would work here, and be consistent with the DistributedSampler()?

This might be ambiguous with all the other types of RNGs that are at play here. It's not clear from the API that this only affects the shuffling seed.

The seed for DataLoader2 is not designed to only provide seed for shuffling ops. The seed is used by DataLoader2 to generate seeds for both shuffle and random_op. And, the difference would be the seeds varies per process for random_op but shuffle would share the same seed for all processes.

But, this requires users to add this random_op wrapper DataPipe to let DataLoader2 that there is random operation in the wrapped DataPipe.

Is there a way for the DataLoader2 to be aware of a specific shuffling seed?
The proposed solution above exposes various implementation details that IMHO are still fairly complex and easy for users to get wrong.
In the case of map-style datasets, all these details are properly handled by the sampler or by the DataLoader, and users don't really need to worry about the RNG at all (except maybe with DistributedSampler.set_epoch()). From a user perspective, it'd be fantastic if the datapipe API could be as simple as for map-style datasets.

I am not sure about it. Users still need to specify seeds for all global RNGs using worker_init_fn right?

I don't believe so. In the torchvision training references for example (which currently use map-style datasets), we don't need to worry about seeds at all. And yet:

we're sure that all workers properly shuffle the data, and
we're sure that all the transforms performs within the datasets __getitem__() methods all use different seeds.

Ideally the same simplicity would be available with datapipes.

And, even with Sampler, if users want to control shuffle state, users have to manually set seed to torch

Are you sure? I believe that the 2 points above are still verified even if we were to call torch.manual_seed() somewhere in the training script.

we're sure that all workers properly shuffle the data, and

This is not the mechanism for Map-style Dataset + DataLoader. Shuffling is carried out in main process using Sampler rather than in worker process.

Are you sure? I believe that the 2 points above are still verified even if we were to call torch.manual_seed() somewhere in the training script.

You mean users don't need to call torch.manual_seed() right?
See RandomSampler: https://github.com/pytorch/pytorch/blob/9bfa7e9791a13e786d39d938ccce0798d3f18ffd/torch/utils/data/sampler.py#L117-L124
If not specified, the seed for the generator object is created by tensor.random_(), which relies on torch.manual_seed(seed) to control the reproducibility in the main process.
If you don't do manual seed to torch, the order of data will vary for different runs of the training script.

we're sure that all the transforms performs within the datasets __getitem__() methods all use different seeds.

I am not 100% sure about how TorchVision augmentation works. But, if all random augmentation relies on the global Torch RNG, to control reproducibility, you have to specify worker_init_fn to make the random augmentations deterministic in worker processes.
If users don't want to preserve reproducibility, users don't need to specify any of torch.manual_seed or worker_init_fn. It's same for DataLoader2, users don't need to specify anything. But, in order to achieve reproducibility, we do need some hurdles in the code.

I agree with everything you wrote @ejguan , but I think we're not entirely talking about the same thing.
My main concern here is to provide an API for datapipes that is just as simple as what we currently have for map-style datasets.
Considering the non-reproducible use-case:

users don't set a seed
users don't need to worry about implementations details (except for set_epoch()).
Shuffling works as expected as the main process properly shuffles locally and is then able to request specific indices from the workers
The transforms (executed in __getitem__()) all use a different RNG, as needed.

Now considering the reproducible use-case, e.g. when users call torch.manual_seed():

users set a seed (as above)
users don't need to worry about implementations details (except for set_epoch()).
Shuffling works as expected as the main process properly shuffles locally and is then able to request specific indices from the workers
The transforms (executed in __getitem__()) all use a different RNG, as needed.

In both these use-cases with map-style datasets, things are kept extremely simple and hidden from the user. At no point do they need to worry about a specific seed for shuffling, or a specific seed for transforms. They can just call manual_seed() once if they want to for reproducibility, but that's it.
Perhaps I'm missing some important detail?

Great summary! And, thanks for bringing this discussion. It triggers me some thinking about BC @NicolasHug
For non-reproducible use case

DataLoader2 and DataPipe don't force users on manual seeding. And, from user perspective, it will be the same behavior. In terms of the implementation detail, DataLoader2 would create own seed by default and make sure seed for shuffle being same across worker processes because the shuffle using DataPipe happens in workers not in main process.
For transforms:

Executed in __getitem__: (1) as long as users use random, torch or numpy to carry out random ops, they can rely the default behavior in worker_loop to set seed. (2) However, if users want to change the default seeding strategy or use other random libraries, they have to provide set_seed explicitly in worker_init_fn.
Executed in __iter__: (1) We would provide the same global RNG per process for users. So, the behavior will be the same. (2) We can still provide something similar to worker_init_fn for users to specify their own seeding strategy. But, we might require users to register their RNGs for checkpointing.

For reproducible use case

The way to set seed will be different. Previously, users would do torch.manual_seed to control seed for both manipulate and random ops in the model. Now, users have to do dl.set_seed to control the seed within the data pipeline.
The random seed for shuffle will be generated by the DataLoader2 deterministically based on the seed provided by users. And all shuffle in worker processes would share the same seed. So, users don't need to do anything for shuffle. (The reason I talked about shuffle().set_seed() is for people who want the RNG for shuffle isolated from global RNG when using DataLoader not DataLoader2. And, it is orthogonal to our discussion IMO)
For transforms, as stated above, if users want to rely on global RNG, we would still provide the same mechanism by setting seeds for global RNGs by default. However, if users want to have isolated RNG per random op, they have to let us know by using

dp = dp.map(lambda x: random_augment(x))
dp = dp.random_op(rng=random.Random())  # this is the isolated RNG will override global RNG and reset global RNG per random op

You can see the only difference from users perspective is when they want to have an isolated RNG per random op (IterDataPipe). And, you might ask why I would come up with this design - IterDataPipe becomes modular components rather than monolithic object like Dataset, we want to have each component having their own state. It's more deterministic and easier for doing checkpoint. Considering we don't want to limit users to only use torch.random using the torch random generator, we need to have a way for us to register all different RNGs user want to use that need to be serialized.
LMK if this makes more sense.

@ejguan and I just had a chat offline where we discussed some of the points above. Here's a summary of our discussion thus far. The points below are either re-hashing, or updating / correcting the ones above. @ejguan please feel free to edit / correct if this isn't accurate. And thanks again for your time and all your work on this!

For both datapipes and map-style datasets, DataLoaderV2 will provide a similar logic to what currently exists in DataLoader regarding the RNG of the workers: each worker will have its own RNG, different from the other workers. This allows (among other things) that each worker has different random transforms.
However, in order for shuffling to be done correctly by all workers on a datapipe, DataLoaderV2 will have a special handling for the shuffling RNG. This will be transparent and users won't need to worry about the implementation details. Erjia will be sharing a proposal in the following week(s).
For reproducibility: calling torch.manual_seed() will allow reproducible results for all RNGs that comes from torch. But the first point above still applies: while the RNG will be reproducible, workers will still get a different RNG.

@ejguan, quick Q: currently, the Shuffler's RNG comes from Python's random builtin module. Do you think it would make sense to allow torch.manual_seed() to also control that RNG? From a user perspective, I feel like it should be expected that torch.manual_seed() affects all random components that come from pytorch, including the Shuffler. In addition, the current RandomSampler and DistributedSampler(shuffle=True) are all controllable through torch.manual_seed. Perhaps we could still keep the implementation based on Python's builtin, but yet allow torch.manual_seed() to control it as well?

quick Q: currently, the Shuffler's RNG comes from Python's random builtin module. Do you think it would make sense to allow torch.manual_seed() to also control that RNG?

Yeah. It will be one TODO in the proposal. I will make the RNG attached to shuffler can be seeded by torch

LMK if this design works for you

Sorry for the delay. You figured things out already in a nice way. Given that shuffle is most of the times used to shuffle the full input data it makes sense that it is handled specially, sounds great!
If I understood well, the last proposal
dp = IterableWrapper(range(n))
dp = dp.shuffle().set_seed(0)
dp = SharderDataPipe(dp)
dp = dp.map(lambda x: random_augment(x))  # franchesoni added this line
dp = dp.random_op(rng=random.Random())  # the API is not determined yet.
dl = DataLoader2(dp, ...)

for epoch in range(2):
    dl.set_seed(0)
    print(list(dl))

changed and it is now
n = 10
dp = IterableWrapper(range(n))
dp = dp.shuffle()  # one per worker using shared seed
dp = SharderDataPipe(dp)  # data subsampling depending on worker num (function defined way above)
dp = dp.map(lambda x: random_augment(x))  # one per worker using worker seed
dl = DataLoader2(dp, ...)

for epoch in range(2):  # will print the same correctly shuffled range twice
    torch.manual_seed(0)
    print(list(dl))

1- is this right?
2- could you implement an official .shard method so the users can import it instead of defining SharderDataPipe function?
3- calling shuffle twice will use different shared seeds, right?
4- although I can not think right now about any other function that could benefit from a shared seed, there is probably some usecase. In such a case it would be nice to still provide the user the option to use shared or worker seed for each operation if she wants to (e.g. I imagine a seed='shared'|'worker' param). Is it too much?
I came here because my code said
    assert (
        num_workers <= 1
    ), "this should be 1 until https://github.com/pytorch/data/issues/302 is solved (check if it is)"

happy to see it has advanced. Let me know when I can remove this assertion! :stuck_out_tongue_closed_eyes:


Contribution

Good evening,
I am interested in contributing to the project you are developing.
Do you need help?
Thank you very much
Sophia Sotiropoulou

@kuberti can I have service account credentials;


Company Local Landscape, intended use case ?

This might be a bit fuzzy, but I was playing around with this as a tool to create an overview/intro/index for what FOSS we make use of at my company.
We are a small company so limited resources which means that realistic alternatives could be something along the line of a wiki page or similar, which for sure would lack the same polished look!
So what I did was pull tooling, and then I cut down the ´https://github.com/cncf/landscape/blob/master/landscape.yml´ to only contain the projects that we make use of, nice !
Update logo, foundation/company information and I can generate a decent looking informative website... so far so good.
Then ofc I would like to add some other FOSS projects to complete the picture ... this very much doable, one concern here is however that I now have a fork from the official data to maintain.
Additionally, I would like to tone down some parts and extend replace some parts.
For example, I would like to be able to represent some sort of adoption 'status' internally i.e. we are looking at the project we are using the project we have stopped using the project ... I tried overloading this information in the project maturity (CNCF Sandbox, Incubation Adopted) but this is an ugly hack and kind of interfere with the intended use for these fields.
Or maybe I am more interested in other stats or some subset more relevant for our use case ...
Now I realize I might be kind of outside the scope for the tool, but since it is used to generate both CNCF and some other Landscape sites. I guess the question is ...

...  if this is something that should be possible to support ?
... would it be a wanted use case ?
... intended ?

Are there other tools or approached that better fit my usecase already ? I don't know of any (suggestions welcome !)

Hi @senare 👋
Like you mentioned, this level of customization is probably out of the scope for this tool. We'd like to keep it as simple as possible, so we focus mainly on features that align well with the landscapes we support.
Maybe you could experiment with a fork and see how it goes for your use case. We know of other users that have created their own to adapt the landscapes generator to suit their needs. If you only need to make some changes here and there, it may not be a problem to maintain the fork going forward 🙂


Performance issue in the definition of sparse_rgdcn_layer, gnns/rgdcn.py

Hello, I found a performance issue in the definition of sparse_rgdcn_layer, gnns/rgdcn.py, tf.initializers.truncated_normal  will created repeatedly during program execution, resulting in reduced efficiency. I think it should be created before the loop in sparse_rgdcn_layer.
Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

This is TF1, so this code is building a computation graph that will be re-used. The loop is creating many different weights, and I'm not sure how you are proposing to avoid this?

I am sorry for that I  propose a wrong issue. I think  tf.initializers.truncated_normal will create repeatedly in the graph by mistake. Thank you for replying me.

It will be called exactly once for each weight that requires initialization, so I think the code is correct.


pip install not working with newer cython

Upgrade to cython 0.22 and then install via pip.
First error says no *.pyx files found (line 59). Edited setup.py to change to *.py and then build continues and files with compile error in agglo.py

Maybe the wrong reason. The features/*.pyx files are not downloading to the tmp dir for the pip install.

Hi @elhuhdron. Thanks for reporting! It looks like the Cython .pyx files were not packaged in the pip source distribution for some reason. Sorry, first time distributing Cython code! I'm exploring the solution. In the meantime, if you want to get something working quickly, you can download the current master from here and build with python setup.py install.

No problem. Thanks for making the project available!
Yup, I was able to workaround by running pip install, letting it fail, then copying the *.pyx files over from a local git clone, then re-running pip install (which for me doesn't re-download, just reruns setup from the tmp folder).
Although the README is quite good compared to other projects I've seen, I noticed a couple missing dependencies:
> sudo pip install viridis
> sudo pip install libtiff
Then I modified gala/imio.py libtiff import for possible name change for pylibtiff:
try:
from pylibtiff import TIFF
except:
try:
from libtiff import TIFF
except:
print("pylibtiff not available: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pylibtiff")
Also, incase anyone else is still using Centos 6.5 and hasn't upgraded python, looks like python 2.6 will work fine except:
> sudo pip install unittest2
and the known deep copy issue (use patch to fix copy.py):
http://bugs.python.org/issue1515

@elhuhdron Thanks for commenting on the README, too! =) See also the readthedocs site: https://gala.readthedocs.org.
This issue should be fixed in 0.2.2, now available on PyPI. Can you confirm that it works? I'm closing this for now, but feel free to reopen if you encounter more issues.
To address your other comments:

additional dependencies: Oops! There's various places where the requirements are listed and they fell through the cracks in the readme. I've added them.
pylibtiff import: this is also fixed. I guess I haven't used it in a while.
2.6 support: I'm glad you've got it working with some mods! Gala was originally developed on 2.6, so this isn't surprising. However, I'm not interested in maintaining such backwards compatibility. Gala is definitely future-looking and intended to be used with the full scientific Python stack, and often with dependencies on the latest library releases (e.g. scikit-learn 0.15). I recommend using conda to manage environments with more recent Python version without messing with your system Python install.

Perfect. Tried on another machine that I hadn't installed on yet and it worked. The upgrade on the same machine also worked fine. Also the libtiff change works (or at least no warning printed). Thanks!

Awesome! For usage questions, feel free to post to the mailing list! =)


Do we need to check chrome.runtime.lastError when popup opens

If background page crashes, and user opens the popup, would it just show default redux state?

https://github.com/oasisprotocol/oasis-wallet-ext/blob/227fa0df99e3ab0ba01ab298b172ba14fdabff47/src/utils/popup.js#L13-L24

from @buberdds:

It looks like we don't need it in new extension for now. lastError  is used when extension use callback based extension API. Old extension is using checkForError only for windowType === 'dapp' which is not a case now (interaction with dapps via runtime.onMessage.addListener?)

So lastError wouldn't be set if background page crashed with Cannot read properties of undefined in normal code. And I didn't find another flag that would get set. Luckily we don't have asynchronous code in the background page that can fail, other than sagas that are handled by fatalError. And if synchronous code before wrapStore fails then popup doesn't initialize.
(And webext-redux and webextension-polyfill do check lastError after sendMessage
https://github.com/tshaddix/webext-redux/blob/95ff156b4afe9bfa697e55bfdb32ec116706aba3/src/store/Store.js#L161
https://github.com/tshaddix/webext-redux/blob/95ff156b4afe9bfa697e55bfdb32ec116706aba3/src/wrap-store/wrapStore.js#L169
https://github.com/mozilla/webextension-polyfill/blob/1649d84ad93c6ebb7225a3db44e96c33e6bbb4a7/src/browser-polyfill.js#L100 )


get_project_dir() is buggy :lady_beetle:

The purpose of get_project_dir()  is to return the root directory where the project is taken place, basically where the .cytosnake file is located. However, there is a problem with this implementation here is the source code:
def get_project_root() -> Path:
    """Returns complete path where cytosnake performs the analysis. The function
    will check if `.cytosnake` folder exists, if not an error will be raised.

    Return
    ------
    Path
        Returns absolute path of the project folder

    Raises
    ------
    FileNotFoundError
        Raised with the current directory is not a project folder
    """

    # get current working directory
    project_dir = Path().absolute()

    # check if the `.cytosnake` folder exist
    project_folder = project_dir / ".cytosnake"
    if not project_folder.exists():
        raise FileNotFoundError("Current directory is not a project folder")

    return project_dir

The problem with this function is that it assumes that you're only at the project directory. Therefore if this code is executed elsewhere, it will raise and error that the project directory is not found.

There is a work around this issue and it has been implemented #26.
The function find_project_dir() which recursively finds the root project directory.
This means that get_project_dir() must contain this function call or be deprecated.


Resize Filter?

I used the  | resize(x, y) filter within an old project some month ago.
Now I started a new project using laravel and twigbrige because I like the way twig works, but I tried to use the resize filter without success.
Is there any way to get the resize filter work using twigbridge?

Do you have a link to the filter? Not sure what it does?

Timber for WordPress had such a function

Sorry not familiar with that. I would assume it would also need something like Glide to actually resize the images, or generate thumbnails

Ok, realizing that this is an feature that does not come out of the box with twig.
Are there any plans adding this feature to TwigBridge or can we do this on our own? I think I have to create my own filter, right?


No module named 'graphite'

Would someone look at this error as I have seen it asked twice on StackOverflow with no response.  It's an installation issue:
https://stackoverflow.com/questions/57932901/graphite-django-admin-py-raise-improperlyconfiguredthe-secret-key-setting-must

Hello @werowe,
0.9.10 is very old version (May 2012, over 7 years ago) and it's not supporting Python 3.
if you want install it from sources please try latest release (1.1.5) instead.
But you can also install it from packages (although bit older version). We're avoiding specific distros in installation manual, but you can use more specific manual written by community. E.g. https://www.scaleway.com/en/docs/install-and-configure-graphite-on-ubuntu/ or https://www.vultr.com/docs/how-to-install-and-configure-graphite-on-ubuntu-16-04
You can also try Graphite in Docker - it's quick and convenient way of running latest version with minimal hassle.

Too complicated.  Will use Docker.  But if I was having to install it
across a cluster or with memcached I probably would have to work my way
through this error.  I got this error on the first try from those
instructions:
Job for carbon-cache.service failed because the control process exited with
error code. See "systemctl status carbon-cache.service" and "journalctl
-xe" for details.
invoke-rc.d: initscript carbon-cache, action "start" failed.
● carbon-cache.service - Graphite Carbon Cache
Loaded: loaded (/lib/systemd/system/carbon-cache.service; enabled;
vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-09-18 11:30:02 UTC;
5ms ago
Process: 9228 ExecStart=/usr/bin/carbon-cache
--config=/etc/carbon/carbon.conf --pidfile=/var/run/carbon-cache.pid
--logdir=/var/log/carbon/ start (code=exited, status=1/FAILURE)
Sep 18 11:30:02 paris2 carbon-cache[9228]:
self.subOptions.parseOptions(rest)
Sep 18 11:30:02 paris2 carbon-cache[9228]:   File
"/usr/lib/python2.7/dist-packages/twisted/python/usage.p...ions
Sep 18 11:30:02 paris2 carbon-cache[9228]:     self.postOptions()
Sep 18 11:30:02 paris2 carbon-cache[9228]:   File
"/usr/lib/python2.7/dist-packages/carbon/conf.py", line ...ions
Sep 18 11:30:02 paris2 carbon-cache[9228]:     if whisper.CAN_FALLOCATE:
Sep 18 11:30:02 paris2 carbon-cache[9228]: AttributeError: 'module' object
has no attribute 'CAN_FALLOCATE'
Sep 18 11:30:02 paris2 systemd[1]: carbon-cache.service: Control process
exited, code=exited status=1
Sep 18 11:30:02 paris2 systemd[1]: Failed to start Graphite Carbon Cache.
Sep 18 11:30:02 paris2 systemd[1]: carbon-cache.service: Unit entered
failed state.
Sep 18 11:30:02 paris2 systemd[1]: carbon-cache.service: Failed with
result 'exit-cod
On Wed, Sep 18, 2019 at 1:19 PM Denis Zhdanov<EMAIL_ADDRESS>wrote:

Hello @werowe https://github.com/werowe,
0.9.10 is very old version (May 2012, over 7 years ago) and it's not
supporting Python 3.
if you want install it from sources please try latest release (1.1.5)
instead.
But you can also install it from packages (although bit older version).
We're avoiding specific distros in installation manual, but you can use
more specific manual written by community
https://www.google.com/search?q=graphite-web+ubuntu+16.04. E.g.
https://www.scaleway.com/en/docs/install-and-configure-graphite-on-ubuntu/
or
https://www.vultr.com/docs/how-to-install-and-configure-graphite-on-ubuntu-16-04
You can also try Graphite in Docker
https://graphite.readthedocs.io/en/latest/install.html#docker - it's
quick and convenient way of running latest version with minimal hassle.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/graphite-project/graphite-web/issues/2486?email_source=notifications&email_token=AB2AXCJ27VCJTWZLQHCCVHTQKH6BNA5CNFSM4IXKV5K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD67SOBA#issuecomment-532621060,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB2AXCPDT2PNPVKTN6QTDQLQKH6BNANCNFSM4IXKV5KQ
.

Strange. CAN_FALLOCATE should be initialized that way or another.
Try to disable it using WHISPER_FALLOCATE_CREATE = False in carbon.conf


allow to manage zones and records (with opt out) with puppet

allow managing zones and records with puppet for a specific zone.
The service name change commit simplifies the autorequire dependencies.

🤯
I will try to free up some time to review this. Massive feature. Thanks so much for contributing.

youre welcome.
if you like, I can make the first commit (use service name as a parameter) a separate merge request.
I now fixed/commented all rubocop mentions (I use pdk for my modules, which has a different .rubocop.yaml)

@trefzer that's not necessary. The changelog is written by hand and I generally base it on what has been committed. I am going to try and go through it all this Monday so we can get this merged ASAP.

Great, I will try to make checks happy over the weekend !
Currently I cannot reproduce them and I have no idea what this error is:
Error: Style/FetchEnvVar: Use `ENV.fetch('GEM_SOURCE')` or `ENV.fetch('GEM_SOURCE', nil)` instead of `ENV['GEM_SOURCE']`. (https://rubystyle.guide/#hash-fetch-defaults)`

Do you have an idea ? (I can also not figure out at which position the error occurs)

Hm I think the error of the tests is in the Gemfile and not in my code ! The second pending merge request #121 has the same error !

Yes !!! ;)
I made a separate request for the rubocop fix (#123). So you can merge it separately (a rebase on #120 will fix the error there as well).

I think this is ready to roll!

Thanks, but now you have been to fast ! I found a problem: there is case that can occur so puppet updates the zone each time. I'm just testing a patch that sorts the records, so this does not occur anymore....

I think so, I did not find any issues anymore. And its running on production.


Offloading to GPU is not working for DBSCAN

Describe the bug
Following the example in the documentation about GPU offloading, I noticed that it did run, but that there was CPU load and that it didnt seem to be using the GPU (didnt hear any fans ramp up or anything). The example is:
from sklearnex import patch_sklearn, config_context
patch_sklearn()

from sklearn.cluster import DBSCAN

X = np.array([[1., 2.], [2., 2.], [2., 3.],
            [8., 7.], [8., 8.], [25., 80.]], dtype=np.float32)
with config_context(target_offload="gpu:0"):
   clustering = DBSCAN(eps=3, min_samples=2).fit(X)

I have also tried to more explicitly offload by using the general context
from sklearnex import patch_sklearn, set_config
patch_sklearn()

set_config(offload="gpu:0")

from sklearn.cluster import DBSCAN

X = np.array([[1., 2.], [2., 2.], [2., 3.],
            [8., 7.], [8., 8.], [25., 80.]], dtype=np.float32)
clustering = DBSCAN(eps=3, min_samples=2).fit(X)

But that didn't change much. I also considered the opportunity to offload the object:
import dpctl
from sklearnex import patch_sklearn, set_config
patch_sklearn()

# Offloading
q = dpctl.SyclQueue("gpu")
print(q.sycl_device.is_gpu)
print(q.sycl_device.is_cpu)
print(q.print_device_info())
set_config(offload="gpu:0")

from sklearn.cluster import DBSCAN

X = np.array([[1., 2.], [2., 2.], [2., 3.],
            [8., 7.], [8., 8.], [25., 80.]], dtype=np.float32)
clustering = DBSCAN(eps=3, min_samples=2).fit(X)

Which prints out nicely:
True
False
    Name            Intel(R) Arc(TM) A770 Graphics
    Driver version  1.3.26241
    Vendor          Intel(R) Corporation
    Filter string   level_zero:gpu:0

The best argument I could find that GPU offloading is not working as it should is because:

There is clear CPU load and increased thermals (so there is activity)
Changing the offloading from GPU to CPU doesn't change execution time for whatever reason. I used timeit to compare 10 runs each time, and they are the same regardless whether I offload to CPU or GPU.

To Reproduce
Already provided above
Expected behavior
Execution time should change when you run it on different hardware
Environment:
Ubuntu 23.04
CPU: 16-core AMD Ryzen 9 5950X (-MT MCP-) speed/min/max: 2258/2200/5083 MHz
Kernel: 6.2.0-24-generic x86_64 Up: 1h 27m Mem: 7957.1/128724.3 MiB (6.2%)
Storage: 931.51 GiB (37.3% used) Procs: 576 Shell: Bash inxi: 3.3.25

Maybe a relevant side-question could be: how can I validate that I am actually offloading to the GPU?

You can use verbose mode to see which device was used: https://intel.github.io/scikit-learn-intelex/verbose.html

@Alexsandruss, awesome! So I can confirm that it is running on CPU, regardless whether I set_config(target_offload = "gpu:0") or not. The logging returns:
SKLEARNEX INFO: sklearn.utils.validation._assert_all_finite: running accelerated version on CPU

@psmgeelen  you can try other algorithms meanwhile. DBSCAN have some specifics that put it apart

@psmgeelen could you please list your conda env as well? It would be very useful

@napetrov, thanks for responding. So I am trying to do some benchmarking with Intel GPU and followed the compatability list in the documentation here https://intel.github.io/scikit-learn-intelex/algorithms.html; So the algorithms that I have tried running are:

RandomForestClassification
SVC with rbf kernel
SVC with exponential (poly) kernel
SVC with linear kernel
PCA
KNeighborsRegressor
LinearRegression
RandomForestRegressor
DBSCAN
KMeans

@samir-nasibli , I listed my environment using conda list and it returned:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                hb9a14ef_9    intel
ca-certificates           2023.01.10           h06a4308_0    intel
certifi                   2022.12.7        py39h06a4308_0    intel
daal4py                   2023.1.1        py39_intel_48679    intel
dal                       2023.1.1            intel_48679    intel
dpcpp-cpp-rt              2023.1.0            intel_46305    intel
dpcpp_cpp_rt              2023.1.0            intel_46305    intel
dpctl                     0.14.2           py39ha23a21d_9    intel
fortran_rt                2023.1.0            intel_46305    intel
glob2                     0.7                        py_0    conda-forge
icc_rt                    2023.1.0            intel_46305    intel
impi_rt                   2021.9.0            intel_43482    intel
intel-cmplr-lib-rt        2023.1.0            intel_46305    intel
intel-cmplr-lic-rt        2023.1.0            intel_46305    intel
intel-fortran-rt          2023.1.0            intel_46305    intel
intel-opencl-rt           2023.1.0            intel_46305    intel
intel-openmp              2023.1.0            intel_46305    intel
intelpython               2023.1.0                      1    intel
joblib                    1.2.0              pyh3f38642_0    intel
libffi                    3.3                          14    intel
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
mkl                       2023.1.0            intel_46342    intel
mkl-service               2.4.0           py39h75d02e3_15    intel
mkl_fft                   1.3.5           py39h28f0b46_11    intel
mkl_random                1.2.2           py39h0b06908_51    intel
mkl_umath                 0.1.1           py39h450dca2_61    intel
ncurses                   6.4                  h6a678d5_0    intel
numpy                     1.23.5           py39h52df89b_7    intel
numpy-base                1.23.5           py39ha03f565_7    intel
openssl                   1.1.1t               h7f8727e_0    intel
pandas                    1.5.3            py39h6cd0baa_0    intel
pip                       23.0.1           py39h06a4308_0    intel
python                    3.9.16               h2722d68_1    intel
python-dateutil           2.8.2                    py39_2    intel
pytz                      2022.7           py39h06a4308_0    intel
readline                  8.2                  h5eee18b_0    intel
scikit-learn              1.2.1            py39h6a678d5_0    intel
scikit-learn-intelex      2023.1.1        py39_intel_48679    intel
scipy                     1.7.3            py39h4ca98da_8    intel
setuptools                65.6.3           py39h06a4308_0    intel
six                       1.16.0             pyhd3eb1b0_1    intel
sqlite                    3.41.1               h5eee18b_0    intel
tbb                       2021.9.0            intel_43484    intel
tbb4py                    2021.9.0        py39_intel_43484    intel
threadpoolctl             2.2.0              pyh0d69192_0    intel
tk                        8.6.12               h1ccaba5_0    intel
tqdm                      4.64.0           py39h06a4308_0    intel
wheel                     0.38.4           py39h06a4308_0    intel
xz                        5.2.8                h5eee18b_0    intel
zlib                      1.2.13               h5eee18b_0    intel

Hi @samir-nasibli, python -m dpctl -f  returned:
Platform  0 ::
    Name        Intel(R) OpenCL
    Version     OpenCL 3.0 LINUX
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                AMD Ryzen 9 5950X 16-Core Processor
        Version             2<IP_ADDRESS>.22_223734
        Filter string       opencl:cpu:0
Platform  1 ::
    Name        Intel(R) FPGA Emulation Platform for OpenCL(TM)
    Version     OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                Intel(R) FPGA Emulation Device
        Version             2<IP_ADDRESS>.22_223734
        Filter string       opencl:accelerator:0
Platform  2 ::
    Name        Intel(R) OpenCL Graphics
    Version     OpenCL 3.0
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                Intel(R) Arc(TM) A770 Graphics
        Version             23.17.26241.33
        Filter string       opencl:gpu:0
Platform  3 ::
    Name        Intel(R) FPGA Emulation Platform for OpenCL(TM)
    Version     OpenCL 1.2 Intel(R) FPGA SDK for OpenCL(TM), Version 20.3
    Vendor      Intel(R) Corporation
    Backend     opencl
    Num Devices 1
      # 0
        Name                Intel(R) FPGA Emulation Device
        Version             2<IP_ADDRESS>.22_223734
        Filter string       opencl:accelerator:1
Platform  4 ::
    Name        Intel(R) Level-Zero
    Version     1.3
    Vendor      Intel(R) Corporation
    Backend     ext_oneapi_level_zero
    Num Devices 1
      # 0
        Name                Intel(R) Arc(TM) A770 Graphics
        Version             1.3.26241
        Filter string       level_zero:gpu:0

I think I might be doing something wrong when running ls -al $OCL_ICD_VENDORS as it just lists the current directory..
I have updated the conda environmen running conda update -c intel -c conda-forge --all and the environment is now:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    intel
_openmp_mutex             4.5                       2_gnu    intel
bzip2                     1.0.8                hb9a14ef_9    intel
ca-certificates           2023.5.7             hbcca054_0    intel
certifi                   2022.12.7        py39h06a4308_0    intel
colorama                  0.4.6              pyhd8ed1ab_0    intel
daal4py                   2023.2.0        py39_intel_49572    intel
dal                       2023.2.0            intel_49572    intel
dpcpp-cpp-rt              2023.2.0            intel_49495    intel
dpcpp_cpp_rt              2023.2.0            intel_49495    intel
dpctl                     0.14.5          py39he78b74f_24    intel
fortran_rt                2023.1.0            intel_46305    intel
glob2                     0.7                        py_0    conda-forge
icc_rt                    2023.2.0            intel_49495    intel
impi_rt                   2021.9.0            intel_43482    intel
intel-cmplr-lib-rt        2023.2.0            intel_49495    intel
intel-cmplr-lic-rt        2023.2.0            intel_49495    intel
intel-fortran-rt          2023.2.0            intel_49495    intel
intel-opencl-rt           2023.2.0            intel_49495    intel
intel-openmp              2023.2.0            intel_49495    intel
intelpython               2023.2.0                      0    intel
joblib                    1.2.0              pyh3f38642_0    intel
level-zero                1.11.0               h00ab1b0_0    intel
libffi                    3.4.2                h7f98852_5    intel
libgcc-ng                 12.2.0              h65d4601_19    intel
libgomp                   12.2.0              h65d4601_19    intel
libnsl                    2.0.0                h7f98852_0    intel
libsqlite                 3.42.0               h2797004_0    intel
libstdcxx-ng              12.2.0              h46fd767_19    intel
libuuid                   2.38.1               h0b41bf4_0    intel
libzlib                   1.2.13               hd590300_5    intel
mkl                       2023.2.0            intel_49495    intel
mkl-service               2.4.0           py39h75d02e3_15    intel
mkl_fft                   1.3.6           py39h173b8ae_56    intel
mkl_random                1.2.2           py39h1595b48_76    intel
mkl_umath                 0.1.1           py39hd987cd3_86    intel
ncurses                   6.4                  hcb278e6_0    intel
numpy                     1.24.3           py39hed7eef7_0    intel
numpy-base                1.24.3           py39he88ecf9_0    intel
openssl                   3.1.1                hd590300_1    intel
pandas                    1.5.3            py39h6cd0baa_0    intel
pip                       23.1.2             pyhd8ed1ab_0    intel
python                    3.9.16              hef7c979_23    intel
python-dateutil           2.8.2                    py39_2    intel
pytz                      2022.7           py39h06a4308_0    intel
readline                  8.2                  h8228510_1    intel
scikit-learn              1.2.1            py39h6a678d5_0    intel
scikit-learn-intelex      2023.2.0        py39_intel_49572    intel
scipy                     1.7.3            py39h4ca98da_8    intel
setuptools                67.7.2             pyhd8ed1ab_0    intel
six                       1.16.0             pyhd3eb1b0_1    intel
sqlite                    3.41.1               h5eee18b_0    intel
tbb                       2021.9.0            intel_43484    intel
tbb4py                    2021.9.0        py39_intel_43484    intel
threadpoolctl             2.2.0              pyh0d69192_0    intel
tk                        8.6.12               h1ccaba5_0    intel
tqdm                      4.65.0             pyhd8ed1ab_1    intel
tzdata                    2023c                h71feb2d_0    intel
wheel                     0.40.0             pyhd8ed1ab_0    intel
xz                        5.2.8                h5eee18b_0    intel
zlib                      1.2.13               hd590300_5    intel

I think the updating broke the environment, as I now get this error:
AttributeError: module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

I resolved this running: conda install -c intel -c conda-forge numpy=1.22 and now have an environment like this:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    intel
_openmp_mutex             4.5                       2_gnu    intel
bzip2                     1.0.8                hb9a14ef_9    intel
ca-certificates           2023.5.7             hbcca054_0    intel
certifi                   2023.5.7           pyhd8ed1ab_0    intel
colorama                  0.4.6              pyhd8ed1ab_0    intel
daal4py                   2023.2.0        py39_intel_49572    intel
dal                       2023.2.0            intel_49572    intel
dpcpp-cpp-rt              2023.2.0            intel_49495    intel
dpcpp_cpp_rt              2023.2.0            intel_49495    intel
dpctl                     0.14.5          py39he78b74f_24    intel
fortran_rt                2023.1.0            intel_46305    intel
glob2                     0.7                        py_0    conda-forge
icc_rt                    2023.2.0            intel_49495    intel
impi_rt                   2021.9.0            intel_43482    intel
intel-cmplr-lib-rt        2023.2.0            intel_49495    intel
intel-cmplr-lic-rt        2023.2.0            intel_49495    intel
intel-fortran-rt          2023.2.0            intel_49495    intel
intel-opencl-rt           2023.2.0            intel_49495    intel
intel-openmp              2023.2.0            intel_49495    intel
intelpython               2023.2.0                      0    intel
joblib                    1.2.0              pyh3f38642_0    intel
level-zero                1.11.0               h00ab1b0_0    intel
libffi                    3.4.2                h7f98852_5    intel
libgcc-ng                 12.2.0              h65d4601_19    intel
libgomp                   12.2.0              h65d4601_19    intel
libnsl                    2.0.0                h7f98852_0    intel
libsqlite                 3.42.0               h2797004_0    intel
libstdcxx-ng              12.2.0              h46fd767_19    intel
libuuid                   2.38.1               h0b41bf4_0    intel
libzlib                   1.2.13               hd590300_5    intel
mkl                       2023.2.0            intel_49495    intel
mkl-service               2.4.0           py39h75d02e3_15    intel
mkl_fft                   1.3.1           py39hcab1719_22    intel
mkl_random                1.2.2           py39hbf47bc3_22    intel
mkl_umath                 0.1.1           py39hf66a691_32    intel
ncurses                   6.4                  hcb278e6_0    intel
numpy                     1.22.3           py39hf0956d0_5    intel
numpy-base                1.22.3           py39h45c9ace_5    intel
openssl                   3.1.1                hd590300_1    intel
pandas                    1.5.3            py39h6cd0baa_0    intel
pip                       23.1.2             pyhd8ed1ab_0    intel
python                    3.9.16              hef7c979_23    intel
python-dateutil           2.8.2                    py39_2    intel
pytz                      2022.7           py39h06a4308_0    intel
readline                  8.2                  h8228510_1    intel
scikit-learn              1.2.1            py39h6a678d5_0    intel
scikit-learn-intelex      2023.2.0        py39_intel_49572    intel
scipy                     1.7.3            py39h4ca98da_8    intel
setuptools                67.7.2             pyhd8ed1ab_0    intel
six                       1.16.0             pyhd3eb1b0_1    intel
sqlite                    3.41.1               h5eee18b_0    intel
tbb                       2021.9.0            intel_43484    intel
tbb4py                    2021.9.0        py39_intel_43484    intel
threadpoolctl             2.2.0              pyh0d69192_0    intel
tk                        8.6.12               h1ccaba5_0    intel
tqdm                      4.65.0             pyhd8ed1ab_1    intel
tzdata                    2023c                h71feb2d_0    intel
wheel                     0.40.0             pyhd8ed1ab_0    intel
xz                        5.2.8                h5eee18b_0    intel
zlib                      1.2.13               hd590300_5    intel

When I run my script I still get: INFO:sklearnex: sklearn.utils.validation._assert_all_finite: running accelerated version on CPU even though the I offloaded to the GPU.

Is there anything else I can do to support the process?

Hi @psmgeelen! Unfortunately I didn't reproduce your issue. I am getting GPU offloading. Let me investigate it more.  I ran this:
from sklearnex import patch_sklearn, config_context
import numpy as np
import logging
logger = logging.getLogger('sklearnex')
logger.setLevel(logging.INFO)
patch_sklearn()

from sklearn.cluster import DBSCAN

X = np.array([[1., 2.], [2., 2.], [2., 3.],
            [8., 7.], [8., 8.], [25., 80.]], dtype=np.float32)
with config_context(target_offload="gpu:0"):
   clustering = DBSCAN(eps=3, min_samples=2).fit(X)

In my new environment and got this print out:
Intel(R) Extension for Scikit-learn* enabled (https://github.com/intel/scikit-learn-intelex)
INFO:sklearnex: sklearn.cluster.DBSCAN.fit: running accelerated version on GPU

So it seems to be working. For some reason, this doesn't work on my larger benchmark test yet. Please give me 24 hours to debug myself before closing this issue. I'll get back to you soon!

Maybe small inbetween question: what does the log:
INFO:sklearnex: sklearn.utils.validation._assert_all_finite: running accelerated version on CPU exactly mean?

After reinstalling the environment again, the issue is not reproducible anymore. I guess the error was transient. I noticed that the GPU support for Intel is not accurately described in the documentation. I found that:

Randomforest Classification and Regression
Linear Regression
PCA
DBSCAN

are supported on an ARC 770, while the so-called supported algorithms for GPUs in the documentation:

SVC with rbf, exp, and linear kernels
KNN classifier
KMeans
are not supported on the ARC770. I assume that these are supported on the integrated graphics of Intel CPUs? Is there more information on whether this is on the roadmap?

Furthermore I had some issues with the methods that are associated to the models. For example the fit_predict method for DBSCAN threw an error:
RuntimeError: Cannot use target offload option inside daal4py.oneapi.sycl_context

While using the fit method works just fine.
Regardless, closing the issue. Thanks for the support!


We need NPCs

At the current state of this project, Mr. X is very lonely (not to mention claustrophobic) in the 16 x 16 world we have given him. He needs friends (or enemies)!

Please elaborate

There should be a "hero" character (I suggest an 'N') that has special powers.

Is this a reference to the flash game 'N'?

No it was actually a reference to some guy's name. The flash game was pretty cool though, I remember playing that years ago.

N is still active

Centralizer too OP.

@Skorezore lel. Also, no


Display total number of records

It would be nice to have the total number of records without actually showing them all in the window since the display of all the records could crash the editor.
Displaying 1000 records of 123456 from /path/to/file.avro
NB : Very usefull tool to check easily content files... Thx for that ! 👍

Thanks this is a great suggestion. I'll look into implementing it in the next week or so.

This change is now live on the marketplace so you should be prompted to update the plugin via your IDE.


Don't include non-installable addons when retrieving list

Give the option to include in the base function, but we won't want them by default, both when testing modules and when resetting the DB.

It seems you forgot to pre-commit.

Sorry, done now :+1:


influx cli write fails if _value is empty in csv

Steps to reproduce:
List the minimal actions needed to reproduce the behaviour.
We do the following between two instances, but it also works within the same instance to make things easier.

Write a row to the db with an empty value in _value column using python API
Read the row using influx read, and use the --raw argument
Pipe the output to influx write command

influx query --raw 'from(bucket: "your_bucket") |> range(start: -1h)' > influx write --bucket YOUR_BUCKET --format csv

Expected behaviour:
Row will be written successfully
Actual behaviour:
Error is thrown, and row is not written.
Error is:
...no field data found...

Environment info:
Using docker image for v2.7.1

We have also encountered this error.
Our exported data could not be re-imported due to this error.
Also an output of the faulty lines with “influx write ... --errors-file” does not work.
The file simply remains empty although there were errors.
A workaround to be able to import the data again is to set the default Annotation
https://docs.influxdata.com/influxdb/v2/reference/syntax/annotated-csv/#annotations
e.g:
#default,_result, , , , , , ,
Now the import runs through completely but any value that was previously an empty string or $null is now a space.
With flux it is possible to replace these spaces back to an empty string or $null.


rule: no-return-wrap (fixes #11)

Anyone want to give some comments on this before it goes in?

+<EMAIL_ADDRESS>


Update discipline-scalatest to 1.0.0-RC4

Updates org.typelevel:discipline-scalatest from 1.0.0-RC2 to 1.0.0-RC4.
Release Notes/Changelog
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.typelevel", artifactId = "discipline-scalatest" } ]

labels: library-update, semver-pre-release

Superseded by #808


What about a much larger collection?

Hi, as radio station we have a much larger collection of lossless encoded audio (FLAC). Would it be interesting to see the performance on our collection?

Hi, that sounds indeed very interesting. Not only to see the performance, but more importantly to train and evaluate. The problem with music is to be able to distribute the audio to researchers, because of copyright. Are you able to redistribute your collection ? Otherwise, how do you imagine to provide access ?

So Dutch law would exclude education and scientific research. But if I would offer the set, including computing power. I would like to keep the contribution anonymous at this time :)

5 Onder een voordracht, op- of uitvoering of voorstelling in het openbaar wordt niet begrepen die welke uitsluitend dient tot het onderwijs dat vanwege de overheid of vanwege een rechtspersoon zonder winstoogmerk wordt gegeven, voor zover de voordracht, op- of uitvoering of voorstelling deel uitmaakt van het schoolwerkplan of leerplan voor zover van toepassing, of tot een wetenschappelijk doel.

Would this be valid worldwide for your entire collection ? I'm not knowledgeable with copyright laws. The thing is I don't want to release it and then for copyright holders to come down on us and force us to remove everything. That would waste everybody's time. The songs hosted by FMA are under a permissive Creative Commons license which allows redistribution, so we are sure.
BTW, how many songs and what kind of songs are we talking about ?


Update django.contrib.auth.decorators.py

Having a default login_url of settings.LOGIN_URL is a bit more explicit and readable than giving it a default value of None and later in _wrapped_view resolving the login_url or settings.LOGIN_URL if the first was none.
login_required and permission_required don't really need to be changed, but it's better for consistency.

Thanks for the PR, but could you create a new ticket at https://code.djangoproject.com/newticket first? The patch also needs tests and documentation updates.

buildbot, test this please (i'm curious what the failures are)

@berkerpeksag  You're right. I jumped the gun, my bad!
Is this correct https://code.djangoproject.com/ticket/24188 ?

I think that's fine. Thanks!

Closing per ticket.

Alright.


Add optional scope parameter when namespacing

This is going to get a bit confusing to explain, but I'll do my best.
I've run into an issue where I want to add a namespaced reducer for a subspaced component, but the reducer will be a sibling of parent component's reducer, instead of a child.
Component structure:

App

Parent (namespace: 'parent')

Child (namespace: 'child')

Reducer structure:

Root

Parent (namspace: 'parent')
Child (namespace: 'child')

The reason why the reducer is a sibling and not being combined into the parent's reducer is because the reducers are being added dynamically using the `replaceReducer' which, so far, I have only been able to get reducers to be added to the top level of the store.
The problem is, when an action (e.g. 'SET_VALUE') is dispatched from the child, as it breaks out of each subspace, the namespace gets prepended so the final action type is 'parent/child/SET_VALUE' but the action is ignore by both reducers:

The parent reducer will successfully remove the 'parent' namespace but none of it's reducers are listening for the resulting action type ('child/SET_VALUE').
The child reducer is looking for an action that starts the 'child\' so it won't pass on an action that starts with 'parent/child'.

I propose the namespacing functionality be expanded to include an scope parameter to apply the namespace to.  If passed undefined, if will fallback to the default behaviour of inferring the scope from the nesting hierarchy.  I also propose that the same functionality be available anywhere a namespace can be provided (namspace, subspaced, SubspaceProvider)
It could be implemented as:

a seperate optional parameter

namespace(reducer, 'child', 'parent')
subspaced(mapState, 'child', 'parent')(Component)
<SubspaceProvider mapState={mapState} namespace='child' scope='parent'>

optionally accepting an object for the `namespace parameter

namespace(reducer, { id: 'child', scope: 'parent' })
subspaced(mapState, { id: 'child', scope: 'parent' })(Component)
<SubspaceProvider mapState={mapState} namespace={{ id: 'child', scope: 'parent' }}>

Another thing to consider what are acceptable values for scope.  I think it should accept:

a string

most basic way to add a scope
namespace(reducer, { id: 'child', scope: 'parent' }) would result in the reducer being namespaced to 'parent/child'

an array

can add multiple levels without knowing the 'prefix/...' convention
namespace(reducer, { id: 'child', scope: ['other', 'parent'] }) would result in the reducer being namespaced to 'other/parent/child'

To enable this, it may be necessary for the subspaced store to carry around it's current namespace (or scope?)  to be accessed by others when needing to identify the current scope.
If there are no objections to this, I will start working on this soon, or if someone else want's to take a crack, let me know and you can have first dibs.

I've been thinking about this more since writing it up and and I'm not sure if it's actually a good thing to add.
There's something nice and simple about the current method of inferring scope from nesting and introducing explicit scope adds a complexity that is probably not necessary in most cases.
I've decided to put this on hold until I have time to think about it more, or explore alternatives to the dynamic reducer solution that is actually causing the issue here.

I've thought about this and all I need is the current namespace of the subspaced store to be queryable on the store, e.g. add a fully qualified namespace value to the store in the redux context.
With that, I can wrap the problematic reducer in another namespaced reducer to handle the missing layers.
This should result in hierarchies as follows:
Component structure:

App

Parent (namespace: 'parent')

Child (namespace: 'child')

Reducer structure:

Root

Parent (namespace: 'parent')
Child (namespace: 'parent/child')


client: fix naming of rados objects

Closed by https://github.com/uccross/arrow/commit/b9de891206d7b40cf033164f27773a11ce9cd8a2


Transfer yjs-widgets

Target

yjs-widgets is the TypeScript counterpart of ypywidgets. It instantiates models and views of widgets in JupyterLab's frontend, and keeps them in sync with the backend using the Y protocol.

The code repository: https://github.com/davidbrochart/yjs-widgets
The npm package on npmjs.com (if available): https://www.npmjs.com/package/yjs-widgets
The python package on pypi.org (if available): https://pypi.org/project/yjs-widgets
The conda-forge recipe on https://github.com/conda-forge (if available): https://github.com/conda-forge/yjs-widgets-feedstock
The documentation website (if available): N/A

Reason

yjs-widgets goes hand in hand with ypywidgets.
Checklist

[x] You are owner of the repository you want to transfer
Which role do you want to have in the jupyter-widgets-contrib organization (check one of):

[ ] Help with all jupyter-widgets-contrib repositories
[x] Help with the transferred repository only
[ ] I don't have time anymore to keep helping the project

The transferred repository (check one of):

[x] Must keep the extension name
[ ] Cannot use the original name
[ ] I don't care

[x] The code is available under a Open Source license
[x] A license file is present in the code repository
[x] The repository contains a README file describing:

[ ] The provided feature(s)
[ ] A picture illustrating the feature(s)
[ ] The available options/settings (if applicable)
[x] How to install the extension
[ ] optional a Binder link to test the feature online

[ ] optional The frontend code (Typescript/Javascript) is tested
[ ] optional The backend code (Python) is tested
[x] I certify the extension is not using exclusively paid-plan of any commercial service
[x] I will help the transfer process on the code repository AND the built packages

You must enable two-factor authentication on GitHub

I don't think that yjs-widgets fits in this organization, since it's a new implementation of Jupyter widgets, and thus not compatible.


Readme

fix npm version link
remove redundant options section
Now I am reading the README again, I see the "Options" above "Sub generators". Should all "Options" just be the "Sub generators"?

fix npm version link

:+1:

remove redundant options section

Why is this redudant? Do you mean redudant with the subgenerator listing? That's not really the same as the options section lists all options of node:app while some subgenerators aren't options.

Hm.. why are they not options? They all should be able to included or not included
EG: I could then add
jshint (Boolean, default true) include or not the .jshintrc files
gulp (Boolean, default true) include or not the gulpfile

to the options.

I can see https://github.com/yeoman/generator-node/blob/master/generators/app/index.js#L11-L30
there are only three items that are optional. But why can't I exclude jshint while I can exclude travis?

I'd also keep the option section.

Ok, I instead fix the options.

Done, also added a missing node:license

Awesome, thanks

include or not a README.md file.

Maybe something like

Scaffold a README.md file.

would have been better.

I just followed the original tone...


Change stderr to stdout on non-error output

As discussed in https://github.com/allinurl/goaccess/issues/2283

Merged. Thanks very much!


Zipkin auto configuration fix

Motivation:
We add Zipkin dependency,we only configure com.alipay.sofa.tracer.zipkin.baseUrl. It's not need configure com.alipay.sofa.tracer.zipkin.enabled=true and consider compatible the default value is true
Modification:
com.alipay.sofa.tracer.zipkin.enabled is  true  when zipkin and zipkin-reporter is in classpath
Result:
Fixes #28

Pull Request Test Coverage Report for Build 63

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.03%) to 49.331%

Totals

Change from base Build 62:
0.03%

Covered Lines:
1658

Relevant Lines:
3361

💛  - Coveralls


Upgrade to Parrot ARSDK 3.12.4

[ ] Update https://github.com/AutonomyLab/parrot_arsdk to 3.12.4 (blocked by #113)
[ ] Update all auto-generated code  (blocked by #113)
[ ] Document any API changes from 3.11.0 -> 3.12.4
[ ] Make any necessary changes to the driver
[ ] Test

Finished update of autonomy lab parrot arsdk to 3.12.6 and submitted PR. I tried modifying and running the generate.py script and install.sh in bebop_driver/scripts/meta folder but some errors occur on the build of bebop_autonomy due to new names that do not fit the autogenerate script. Are there extra instructions that I may be missing (#113)?

@thomas-bamford Have you checked this part of the documentation? http://bebop-autonomy.readthedocs.io/en/latest/dev.html#upgrading-the-bebop-sdk (Step 6)?

@mani-monaj Thanks for pointing this out. Managed to fix these names (and some others) and everything works well. Submitted PR, but still have to test on vehicles.

@thomas-bamford You are welcome! Both AutonomyLab/parrot_arsdk#5 and #120 look great. Please check my comments on both those issues.

sorry,
In the last version I tried to read the states, such as the battery changed but I get this error
ERROR: Cannot load message class for [bebop_msgs/CommonCommonStateBatteryStateChanged]. Are your messages built?
I installed everything as says in instructions, even so I could not read the information published in any of the bebop / states /.. topics
Could you help me with this?

Hello @VINICIOSTALIN, could you please open an issue for this?
Quick guess at this: If you are updating to latest version (not clean install) you may need to clean the workspace before rebuilding, i.e.
catkin clean
catkin build
Or you may need to update the parrot-arsdk package to the latest version before building. If this does not work than this may be another issue but we should discuss it in the issues section.

Excuse me @thomas-bamford  I could not solve it, could you help me in issue #143 that I have raised?


Remove brigmedic hypospray

About the PR

Additionally removes the small chance to spawn omnizine in their locker.
The hypospray is supposed to be reserved as a special item for the CMO. I understand that brigmedic is a bit of a nonstandard medical role, but I don't think that justifies giving it such a powerful tool, especially given that it spawns in a locker with simple security access. I removed the omnizine because, similarly, it isn't meant  to be so plainly accessible (even if it's under a rare chance).
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl:

remove: The brigmedic no longer has a hypospray.

I didn't think it would be an issue since Puro made it suck so bad mechanically for anything other than a one time quick injection of 5u of inaprovaline or PAX.  The hypo has a 2.5 second cool down, which makes syringes more efficient for applying other medicines.
On the locker thing, brig medic definitely needs a specialized locker for their specialized gear (looking at hardsuit and, not hypospray anymore lol).


Issue 31/is empty property

resolves #31
Add is_empty to node to query if the file has no bytes.
Note this depends on #26 and #27 getting merged. A merge conflict is expected and this branch will likely fail the CI build until those PRs are merged.

I'm thinking of going with empty over is_empty to be consistent with pandas' similar method for DataFrames https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.empty.html and since we have other properties like encrypted instead of is_encrypted.


Download multiple videos on Windows not in a playlist

Checklist

[x] I'm asking a question
[x] I've looked through the README and FAQ for similar questions
[x] I've searched the bugtracker for similar questions including closed ones

Question

How can one download multiple videos on Windows that are not in a playlist?
I made searches at large (on Startpage) with different queries, as well as through this README with "multiple" without being able to download the multiple videos that are not connected. Those were my attempts:
C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" [URL1]; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" [URL2]; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" [URL3]
ERROR: '[URL1];' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:[URL1];" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" [URL1] ; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" [URL2] ; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" [URL3]
[youtube] [URL1]: Downloading webpage
[download] Destination: [FILE NAME]
[download] 100% of [SIZE] in [TIME]
[ffmpeg] Correcting container in "[FILE NAME]"
ERROR: ';' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:;" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "[URL1]" ; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "[URL2]" ; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "[URL3]"
[youtube] [URL1]: Downloading webpage
[download] [FILE NAME] has already been downloaded
[download] 100% of [SIZE]
[ffmpeg] Correcting container in "[FILE NAME]"
ERROR: ';' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:;" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" {"[URL1]","[URL2]","[URL3]"}
ERROR: '{[URL1],[URL2],[URL3]}' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:{[URL1],[URL2],[URL3]}" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "[URL1]"; !:-"[URL2]"; !:-"[URL3]"
ERROR: '[URL1];' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:[URL1];" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "[URL1]"; !:- "[URL2]"; !:- "[URL3]"
ERROR: '[URL1];' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:[URL1];" ) to search YouTube

Thank you kindly for your help

By masking the URLs, youve made this essentially impossible to troubleshoot. If you actually want help, please post commands that are actually valid.
You dont need to use the actual videos you want, but they need to be valid URLs.

I made a mistake, [URL#] should have been [YOUTUBE ID#], so the 1st command would be something like C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" abc123; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" def456; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" ghi789

By masking the IDs, youve made this essentially impossible to troubleshoot. If you actually want help, please post commands that are actually valid.
You dont need to use the actual videos you want, but they need to be valid IDs.

The only difference between this and what was in my terminal was I masked the user name in the directory:
C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" lLZxspPwAs0; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" Gml3EdLPNME
ERROR: 'lLZxspPwAs0;' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:lLZxspPwAs0;" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" lLZxspPwAs0 ; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" Gml3EdLPNME
[youtube] lLZxspPwAs0: Downloading webpage
[download] Destination: In Memoriam - Grindcore Goat-lLZxspPwAs0.webm
[download] 100% of 730.17KiB in 00:13
ERROR: ';' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:;" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "lLZxspPwAs0" ; "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "Gml3EdLPNME"
[youtube] lLZxspPwAs0: Downloading webpage
[download] In Memoriam - Grindcore Goat-lLZxspPwAs0.webm has already been downloaded
[download] 100% of 730.17KiB
ERROR: ';' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:;" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" {"lLZxspPwAs0","Gml3EdLPNME"}
ERROR: '{lLZxspPwAs0,Gml3EdLPNME}' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:{lLZxspPwAs0,Gml3EdLPNME}" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "lLZxspPwAs0"; ; !:-"Gml3EdLPNME"
ERROR: 'lLZxspPwAs0;' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:lLZxspPwAs0;" ) to search YouTube

C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" "lLZxspPwAs0"; ; !:- "Gml3EdLPNME"
ERROR: 'lLZxspPwAs0;' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:lLZxspPwAs0;" ) to search YouTube

It looks like you using Command Prompt (cmd.exe). If thats the case, you can fix this two ways:

stop using Command Prompt. Use something like PowerShell or NuShell instead.

If you insist on using Command Prompt, use & instead of ;

Thanks again
C:\Users\[USER NAME]\Downloads>"%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" lLZxspPwAs0 & "%ProgramFiles(x86)%\youtube-dl\youtube-dl-2021.06.06.exe" -f "bestaudio" Gml3EdLPNME
[youtube] lLZxspPwAs0: Downloading webpage
[download] In Memoriam - Grindcore Goat-lLZxspPwAs0.webm has already been downloaded
[download] 100% of 730.17KiB
[youtube] Gml3EdLPNME: Downloading webpage
[youtube] Gml3EdLPNME: Downloading MPD manifest
[download] Destination: Gore Bath 'Fake' Grindcore_Noise_Crustpunk (OFFICIAL MUSIC VIDEO)-Gml3EdLPNME.m4a
[download] 100% of 1.28MiB in 00:18
[ffmpeg] Correcting container in "Gore Bath 'Fake' Grindcore_Noise_Crustpunk (OFFICIAL MUSIC VIDEO)-Gml3EdLPNME.m4a"


An error is reported when using a plug-in in the UniApp framework

Are you using Nuxt?

[ ] Check this box if you encountered the issue using Nuxt and the included module.

Describe the bug

Reproduction
This is the project address：https://github.com/4uansi/test.git
System Info
MACOS

Used Package Manager
pnpm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] The provided reproduction is a minimal reproducible of the bug.

I cannot access the repo, is it public ? 🤔

用的是pnpm吧，加个.npmrc内容是shamefully-hoist=true就行了

用的是pnpm吧，加个.npmrc内容是shamefully-hoist=true就行了

是的，成功解决啦，谢谢

great closing this 🎉


fix bug of orphaned CS machine

Issue #, if available:
There are some CloudStack VMs that does not have assoicated CloudStackMachines in kubernetes.
Description of changes:
When CloudStackMachines resource is deleted, there is a possibility that its Spec.InstanceID has not been set due to race condition. This will cause CloudStackMachine resource being deleted without destroy CloudStack VM.
The fix is to return temp error and reconciler will retry and CloudStackMachines'  InstanceID will be set or can be retrieved by using CloudStack API call.
Testing performed:
Unit testing, manual testing.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Codecov Report
Base: 36.92% // Head: 34.02% // Decreases project coverage by -2.90% :warning:

Coverage data is based on head (6c51d26) compared to base (ea48080).
Patch coverage: 13.35% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #191      +/-   ##
==========================================
- Coverage   36.92%   34.02%   -2.91%
==========================================
  Files          42       43       +1
  Lines        3355     3862     +507
==========================================
+ Hits         1239     1314      +75
- Misses       1949     2369     +420
- Partials      167      179      +12

Impacted Files
Coverage Δ

api/v1beta2/cloudstackmachine_types.go
28.57% <ø> (ø)

api/v1beta2/zz_generated.deepcopy.go
12.29% <12.29%> (ø)

controllers/cloudstackmachine_controller.go
58.64% <53.84%> (+3.34%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

/run-e2e

A job created for running /run-e2e, will keep you posted when the result is ready

/ok-to-test

Setting up environment failed