Error while following docs to load the `ted_talks_iwslt` dataset

@skyprince999 as you authored the PR for this dataset, any comments?

This has been fixed in #2064  by @mariosasko (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)

I am currently trying to load the `ted_talks_iwslt` dataset into google colab.

The [docs](https://huggingface.co/datasets/ted_talks_iwslt) mention the following way of doing so.

```python
dataset = load_dataset("ted_talks_iwslt", language_pair=("it", "pl"), year="2014")
```

Executing it results in the error attached below.

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-7dcc67154ef9> in <module>()
----> 1 dataset = load_dataset("ted_talks_iwslt", language_pair=("it", "pl"), year="2014")

4 frames
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    730         hash=hash,
    731         features=features,
--> 732         **config_kwargs,
    733     )
    734

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in __init__(self, writer_batch_size, *args, **kwargs)
    927
    928     def __init__(self, *args, writer_batch_size=None, **kwargs):
--> 929         super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
    930         # Batch size used by the ArrowWriter
    931         # It defines the number of samples that are kept in memory before writing them

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in __init__(self, cache_dir, name, hash, features, **config_kwargs)
    241             name,
    242             custom_features=features,
--> 243             **config_kwargs,
    244         )
    245

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    337             if "version" not in config_kwargs and hasattr(self, "VERSION") and self.VERSION:
    338                 config_kwargs["version"] = self.VERSION
--> 339             builder_config = self.BUILDER_CONFIG_CLASS(**config_kwargs)
    340
    341         # otherwise use the config_kwargs to overwrite the attributes

/root/.cache/huggingface/modules/datasets_modules/datasets/ted_talks_iwslt/024d06b1376b361e59245c5878ab8acf9a7576d765f2d0077f61751158e60914/ted_talks_iwslt.py in __init__(self, language_pair, year, **kwargs)
    219             description=description,
    220             version=datasets.Version("1.1.0", ""),
--> 221             **kwargs,
    222         )
    223

TypeError: __init__() got multiple values for keyword argument 'version'
```

How to resolve this?

PS: Thanks a lot @huggingface team for creating this great library!


Is it possible to convert a `tfds` to HuggingFace `dataset`?

I was having some weird bugs with `C4`dataset version of HuggingFace, so I decided to try to download `C4`from `tfds`. I would like to know if it is possible to convert a tfds dataset to HuggingFace dataset format :)

I can also open a new issue reporting the bug I'm receiving with `datasets.load_dataset('c4','en')` in the future if you think that it would be useful.

Thanks!


update link to ZEST dataset

Updating the link as the original one is no longer working.


issue with opus100/en-fr dataset

@lhoestq  I also deleted the cache and redownload the file and still the same issue, I appreciate any help on this. thanks

Here please find the minimal code to reproduce the issue @lhoestq  note this only happens with MT5TokenizerFast

```
from datasets import load_dataset
from transformers import MT5TokenizerFast

def get_tokenized_dataset(dataset_name, dataset_config_name, tokenizer):
    datasets = load_dataset(dataset_name, dataset_config_name, script_version="master")
    column_names = datasets["train"].column_names
    text_column_name = "translation"
    def process_dataset(datasets):
        def process_function(examples):
            lang = "fr"
            return {"src_texts": [example[lang] for example in examples[text_column_name]]}
        datasets = datasets.map(
            process_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True,
        )
        return datasets
    datasets = process_dataset(datasets)
    text_column_name = "src_texts"
    column_names = ["src_texts"]
    def tokenize_function(examples):
            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
    tokenized_datasets = datasets.map(
            tokenize_function,
            batched=True,
            num_proc=None,
            remove_columns=column_names,
            load_from_cache_file=True
    )

if __name__ == "__main__":
     tokenizer_kwargs = {
        "cache_dir": None,
        "use_fast": True,
        "revision": "main",
        "use_auth_token": None
     }
     tokenizer = MT5TokenizerFast.from_pretrained("google/mt5-small", **tokenizer_kwargs)
     get_tokenized_dataset(dataset_name="opus100", dataset_config_name="en-fr", tokenizer=tokenizer)
~
```

as per https://github.com/huggingface/tokenizers/issues/626 this looks like to be the tokenizer bug, I therefore, reported it there https://github.com/huggingface/tokenizers/issues/626 and  I am closing this one.

Hi
I am running run_mlm.py code of huggingface repo with opus100/fr-en pair, I am getting this error, note that this error occurs for only this pairs and not the other pairs. Any idea why this is occurring? and how I can solve this?

Thanks a lot  @lhoestq for your help in advance.

`
thread '<unnamed>' panicked at 'index out of bounds: the len is 617 but the index is 617', /__w/tokenizers/tokenizers/tokenizers/src/tokenizer/normalizer.rs:382:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
 63%|██████████████████████████████████████████████████████████▊                                   | 626/1000 [00:27<00:16, 22.69ba/s]

Traceback (most recent call last):
  File "run_mlm.py", line 550, in <module>
    main()
  File "run_mlm.py", line 412, in main
    in zip(data_args.dataset_name, data_args.dataset_config_name)]
  File "run_mlm.py", line 411, in <listcomp>
    logger) for dataset_name, dataset_config_name\
  File "/user/dara/dev/codes/seq2seq/data/tokenize_datasets.py", line 96, in get_tokenized_dataset
    load_from_cache_file=not data_args.overwrite_cache,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/dataset_dict.py", line 448, in map
    for k, dataset in self.items()
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/dataset_dict.py", line 448, in <dictcomp>
    for k, dataset in self.items()
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1309, in map
    update_data=update_data,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 204, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1574, in _map_single
    batch, indices, check_same_num_examples=len(self.list_indexes()) > 0, offset=offset
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1490, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/user/dara/dev/codes/seq2seq/data/tokenize_datasets.py", line 89, in tokenize_function
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2347, in __call__
    **kwargs,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2532, in batch_encode_plus
    **kwargs,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 384, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
pyo3_runtime.PanicException: index out of bounds: the len is 617 but the index is 617

`


is there a way to override a dataset object saved with save_to_disk?

Hi
You can rename the arrow file and update the name in `state.json`

I tried this way, but when there is a mapping process to the dataset, it again uses a random cache name. atm, I am trying to use the following method by setting an exact cache file,

```
            dataset_with_embedding =csv_dataset.map(
                partial(self.embed, ctx_encoder=ctx_encoder, ctx_tokenizer=self.context_tokenizer),
                batched=True,
                batch_size=1,
                features=new_features,
                cache_file_name=cache_arrow_path,
                load_from_cache_file=False
            )
```
So here we set a cache_file_name , after this it uses the same  file name when saving again and again.

I'm not sure I understand your issue, can you elaborate ?

`cache_file_name` is indeed an argument you can set to specify the cache file that will be used for the processed dataset. By default the file is named with something like `cache-<fingerprint>.arrow` where the fingerprint is a hash.

Let's say I am updating a set of embedding in a dataset that is around 40GB inside a training loop every 500 steps (Ex: calculating the embeddings in updated ctx_encoder in RAG and saving it to the passage path).  So when we use **dataset_object.save_to_disk('passage_path_directory')** it will save the new dataset object every time with a random file name, especially when we do some transformations to dataset objects such as map or shards. This way, we keep collecting unwanted files that will eventually eat up all the disk space.

But if we can save the dataset object every time by a single name like **data_shard_1.arrow**, it will automatically remove the previous file and save the new one in the same directory.  I found the above-mentioned code snippet useful to complete this task.

Is this clear?

At the moment when I use save_to_disk, it uses the arbitrary name for the arrow file.  Is there a way to override such an object?


Could not find file for ZEST dataset

dataset bug

A bug in a dataset script provided in the library

The zest dataset url was changed (allenai/zest#3) and #2057 should resolve this.

This has been fixed in #2057 by @matt-peters (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)

I am closing this issue since its fixed!

I am trying to use zest dataset from Allen AI using below code in colab,
```
!pip install -q datasets
from datasets import load_dataset
dataset = load_dataset("zest")
```

I am getting the following error,
```
Using custom data configuration default

Downloading and preparing dataset zest/default (download: 5.53 MiB, generated: 19.96 MiB, post-processed: Unknown size, total: 25.48 MiB) to /root/.cache/huggingface/datasets/zest/default/0.0.0/1f7a230fbfc964d979bbca0f0130fbab3259fce547ee758ad8aa4f9c9bec6cca...
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-18dbbc1a4b8a> in <module>()
      1 from datasets import load_dataset
      2
----> 3 dataset = load_dataset("zest")

9 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    612             )
    613         elif response is not None and response.status_code == 404:
--> 614             raise FileNotFoundError("Couldn't find file at {}".format(url))
    615         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    616         raise ConnectionError("Couldn't reach {}".format(url))

FileNotFoundError: Couldn't find file at https://ai2-datasets.s3-us-west-2.amazonaws.com/zest/zest.zip
```


Add bAbI QA tasks

Hi @lhoestq,

Should I remove the 160 configurations? Is it too much?

EDIT:
Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

Thanks for the changes !

> Should I remove the 160 configurations? Is it too much?

Yea 160 configuration is a lot.
Maybe this dataset can work with parameters `type` and `task_no` ?
You can just remove the configuration in BUILDER_CONFIGS to only keep a few ones.
Also feel free to add an example in the dataset card of how to load the other configurations
```
load_dataset("babi_qa", type="hn", task_no="qa1")
```
for example, and with a list of the possible combinations.

> Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

It looks appropriate, thanks :)

Hi @lhoestq

I'm unable to test it locally using:
```python
load_dataset("datasets/babi_qa", type="hn", task_no="qa1")
```
It raises an error:
```python
TypeError: __init__() got an unexpected keyword argument 'type'
```
Will this be possible only after merging? Or am I missing something here?

Can you try adding this class attribute to `BabiQa` ?
```python
BUILDER_CONFIG_CLASS = BabiQaConfig
```
This should fix the TypeError issue you got

Hi @lhoestq

I have added the changes. Only the "qa1" task for each category is included. Also, I haven't removed the size categories and other description because I think it will still be useful. I have updated the line in README showing the example.

Thanks,
Gunjan

- **Name:** *The (20) QA bAbI tasks*
- **Description:** *The (20) QA bAbI tasks are a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. The aim is to classify these tasks into skill sets,so that researchers can identify (and then rectify) the failings of their systems.*
- **Paper:** [arXiv](https://arxiv.org/pdf/1502.05698.pdf)
- **Data:** [Facebook Research Page](https://research.fb.com/downloads/babi/)
- **Motivation:** This is a unique dataset with story-based Question Answering. It  is a part of the `bAbI` project by Facebook Research.

**Note**: I have currently added all the 160 configs. If this seems impractical, I can keep only a few. While each `dummy_data.zip` weighs a few KBs, overall it is around 1.3MB for all configurations. This is problematic. Let me know what is to be done.

Thanks :)

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Timit_asr dataset repeats examples

Hi,

this was fixed by #1995, so you can wait for the next release or install the package directly from the master branch with the following command:
```bash
pip install git+https://github.com/huggingface/datasets
```

Summary

When loading timit_asr dataset on datasets 1.4+, every row in the dataset is the same
Steps to reproduce

As an example, on this code there is the text from the training part:

Code snippet:
```
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")
timit['train']['text']
#['Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
# 'Would such an act of refusal be useful?',
```
The same behavior happens for other columns

Expected behavior:

Different info on the actual timit_asr dataset

Actual behavior:

When loading timit_asr dataset on datasets 1.4+, every row in the dataset is the same. I've checked datasets 1.3 and the rows are different
Debug info

    Streamlit version: (get it with $ streamlit version)
    Python version: Python 3.6.12
    Using Conda? PipEnv? PyEnv? Pex? Using pip
    OS version: Centos-release-7-9.2009.1.el7.centos.x86_64

Additional information

You can check the same behavior on https://huggingface.co/datasets/viewer/?dataset=timit_asr


Add MDD Dataset

Hi @lhoestq,

I have added changes from review.

- **Name:** *MDD Dataset*
- **Description:** The Movie Dialog dataset (MDD) is designed to measure how well models can perform at goal and non-goal orientated dialog centered around the topic of movies (question answering, recommendation and discussion), from various movie reviews sources such as MovieLens and OMDb.
- **Paper:** [arXiv](https://arxiv.org/pdf/1511.06931.pdf)
- **Data:**  https://research.fb.com/downloads/babi/
- **Motivation:** This is one of the popular dialog datasets, a part of Facebook Research's "bAbI project".

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

**Note**: I haven't included the following from the data files: `entities` (the file containing list of all entities in the first three subtasks), `dictionary`(the dictionary of words they use in their models), `movie_kb`(contains the knowledge base of information about the movies, actors and other entities that are mentioned in the dialogs). Please let me know if those are needed, and if yes, should I make separate configurations for them?


Build custom dataset to fine-tune Wav2Vec2

dataset request

Requesting to add a new dataset

@lhoestq - We could simply use the "general" json dataset for this no?

Sure you can use the json loader
```python
data_files = {"train": "path/to/your/train_data.json", "test": "path/to/your/test_data.json"}
train_dataset = load_dataset("json", data_files=data_files, split="train")
test_dataset = load_dataset("json", data_files=data_files, split="test")
```

You just need to make sure that the data contain the paths to the audio files.
If not, feel free to use `.map()` to add them.

Many thanks! that was what I was looking for.

Thank you for your recent tutorial on how to finetune Wav2Vec2 on a custom dataset. The example you gave here (https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) was on the CommonVoice dataset. However, what if I want to load my own dataset?  I have a manifest (transcript and their audio files) in a JSON file.


Fix text-classification tags

There are different tags for text classification right now: `text-classification` and `text_classification`:
![image](https://user-images.githubusercontent.com/29076344/111042457-856bdf00-8463-11eb-93c9-50a30106a1a1.png).

This PR fixes it.


github is not always available - probably need a back up

Yesterday morning github wasn't working:

```
:/tmp$ wget https://raw.githubusercontent.com/huggingface/datasets/1.4.1/metrics/sacrebleu/sacrebleu.py--2021-03-12 18:35:59-- https://raw.githubusercontent.com/huggingface/datasets/1.4.1/metrics/sacrebleu/sacrebleu.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2021-03-12 18:36:11 ERROR 500: Internal Server Error.
```

Suggestion: have a failover system and replicate the data on another system and reach there if gh isn't reachable? perhaps gh can be a master and the replicate a slave - so there is only one true source.


Multilingual dIalogAct benchMark (miam)

Hello. All aforementioned changes have been made. I've also re-run black on miam.py. :-)

I will run isort again. Hopefully it resolves the current check_code_quality test failure.

Once the review period is over, feel free to open a PR to add all the missing information ;)

Hi! I will follow up right now with one more pull request as I have new anonymous citation information to include.

My collaborators (@EmileChapuis, @PierreColombo) and I within the Affective Computing team at Telecom Paris would like to anonymously publish the miam dataset. It is assocated with a publication currently under review. We will update the dataset with full citations once the review period is over.


add_faisis_index  gets very slow when doing it interatively

I think faiss automatically sets the number of threads to use to build the index.
Can you check how many CPU cores are being used when you build the index in `use_own_knowleldge_dataset` as compared to this script ? Are there other programs running (maybe for rank>0) ?

Hi,
 I am running the add_faiss_index during the training process of the RAG from the master process (rank 0). But at the exact moment, I do not run any other process since I do it in every 5000 training steps.

 I think what you say is correct. It depends on the number of CPU cores. I did an experiment to compare the time taken to finish the add_faiss_index process on use_own_knowleldge_dataset.py vs the training loop thing.  The training loop thing takes 40 mins more. It might be natural right?

 at the moment it uses around 40 cores of a 96 core machine (I am fine-tuning the entire process).

Can you try to set the number of threads manually ?
If you set the same number of threads for both the `use_own_knowledge_dataset.py` and RAG training, it should take the same amount of time.
You can see how to set the number of thread in the faiss wiki: https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls

Ok, I will report the details too soon. I am the first one on the list and currently add_index being computed for the 3rd time in the loop. Actually seems like the time is taken to complete each interaction is the same, but around 1 hour more compared to running it without the training loop. A the moment this takes 5hrs and 30 mins.  If there is any way to faster the process, an end-to-end rag will be perfect. So I will also try out with different thread numbers too.

![image](https://user-images.githubusercontent.com/16892570/111453464-798c5f80-8778-11eb-86d0-19d212f58e38.png)

@lhoestq  on a different note, I read about using Faiss-GPU, but the documentation says we should use it when the dataset has the ability to fit into the GPU memory. Although this might work, in the long-term this is not that practical for me.

https://github.com/matsui528/faiss_tips

@lhoestq

Hi,  I executed the **use_own_dataset.py** script independently and ask a few of my friends to run their programs in the HPC machine at the same time.

 Once there are so many other processes are running the add_index function gets slows down naturally.  So basically the speed of the add_index depends entirely on the number of CPU processes.  Then I set the number of threads as you have mentioned and got actually the same time for RAG training and independat running.  So you are correct! :)

 Then I added this [issue in Faiss repostiary](https://github.com/facebookresearch/faiss/issues/1767). I got an answer saying our current **IndexHNSWFlat** can get slow for 30 million vectors and it would be better to use alternatives. What do you think?

It's a matter of tradeoffs.
HSNW is fast at query time but takes some time to build.
A flat index is flat to build but is "slow" at query time.
An IVF index is probably a good choice for you: fast building and fast queries (but still slower queries than HSNW).

Note that for an IVF index you would need to have an `nprobe` parameter (number of cells to visit for one query, there are `nlist` in total) that is not too small in order to have good retrieval accuracy, but not too big otherwise the queries will take too much time. From the faiss documentation:
> The nprobe parameter is always a way of adjusting the tradeoff between speed and accuracy of the result. Setting nprobe = nlist gives the same result as the brute-force search (but slower).

From my experience with indexes on DPR embeddings, setting nprobe around 1/4 of nlist gives really good retrieval accuracy and there's no need to have a value higher than that (or you would need to brute-force in order to see a difference).

@lhoestq

Thanks a lot for sharing all this prior knowledge.

Just asking what would be a good nlist of parameters for 30 million embeddings?

When IVF is used alone, nlist should be between `4*sqrt(n)` and `16*sqrt(n)`.
For more details take a look at [this section of the Faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#how-big-is-the-dataset)

Thanks a lot. I was lost with calling the index from class and using faiss_index_factory.

@lhoestq  Thanks a lot for the help you have given to solve this issue. As per my experiments, IVF index suits well for my case and it is a lot faster. The use of this can make the entire RAG end-to-end trainable lot faster.  So I will close this issue. Will do the final PR soon.

As the below code suggests, I want to run add_faisis_index in every nth interaction from the training loop. I have 7.2 million documents. Usually, it takes 2.5 hours (if I run an as a separate process similar to the script given in rag/use_own_knowleldge_dataset.py). Now, this takes usually 5hrs. Is this normal?   Any way to make this process faster?

@lhoestq

```
   def training_step(self, batch, batch_idx) -> Dict:

        if (not batch_idx==0) and (batch_idx%5==0):

            print("******************************************************")
            ctx_encoder=self.trainer.model.module.module.model.rag.ctx_encoder
            model_copy =type(ctx_encoder)(self.config_dpr) # get a new instance  #this will be load in the CPU
            model_copy.load_state_dict(ctx_encoder.state_dict()) # copy weights and stuff

            list_of_gpus = ['cuda:2','cuda:3']
            c_dir='/custom/cache/dir'

            kb_dataset = load_dataset("csv", data_files=[self.custom_config.csv_path], split="train", delimiter="\t", column_names=["title", "text"],cache_dir=c_dir)

            print(kb_dataset)

            n=len(list_of_gpus) #nunber of dedicated GPUs
            kb_list=[kb_dataset.shard(n, i, contiguous=True) for i in range(n)]

            #kb_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/haha-dir')

            print(self.trainer.global_rank)
            dataset_shards = self.re_encode_kb(model_copy.to(device=list_of_gpus[self.trainer.global_rank]),kb_list[self.trainer.global_rank])
            output = [None for _ in list_of_gpus]

            #self.trainer.accelerator_connector.accelerator.barrier("embedding_process")
            dist.all_gather_object(output, dataset_shards)

            #This creation and re-initlaization of the new index
            if (self.trainer.global_rank==0):  #saving will be done in the main process

                combined_dataset = concatenate_datasets(output)

                passages_path =self.config.passages_path

                logger.info("saving the dataset with  ")
                #combined_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/MY-Passage')
                combined_dataset.save_to_disk(passages_path)
                logger.info("Add faiss index to the dataset that consist of embeddings")

                embedding_dataset=combined_dataset
                index = faiss.IndexHNSWFlat(768, 128, faiss.METRIC_INNER_PRODUCT)
                embedding_dataset.add_faiss_index("embeddings", custom_index=index)

                embedding_dataset.get_index("embeddings").save(self.config.index_path)


Preserve column ordering in Dataset.rename_column

Not sure why CI isn't triggered.

@lhoestq Can you please help me with this?

I don't know how to trigger it manually, but an empty commit should do the job

Currently `Dataset.rename_column` doesn't necessarily preserve the order of the columns:
```python
>>> from datasets import Dataset
>>> d = Dataset.from_dict({'sentences': ["s1", "s2"], 'label': [0, 1]})
>>> d
Dataset({
    features: ['sentences', 'label'],
    num_rows: 2
})
>>> d.rename_column('sentences', 'text')
Dataset({
    features: ['label', 'text'],
    num_rows: 2
})
```
This PR fixes this.


Add CBT dataset

Hi @lhoestq,

I have added changes from the review.

This PR adds the [CBT Dataset](https://arxiv.org/abs/1511.02301).

Note that I have also added the `raw` dataset as a separate configuration. I couldn't find a suitable "task" for it in YAML tags.

The dummy files have one example each, as the examples are slightly big. For `raw` dataset, I just used top few lines, because they are entire books and would take up a lot of space.

Let me know in case of any issues.


Support pickle protocol for dataset splits defined as ReadInstruction

@lhoestq But we don't perform conversion to a `NamedSplit` if `_split` is not a string which means it **will** be a `ReadInstruction` after reloading.

Yes right ! I read it wrong.
Perfect then

Fixes #2022 (+ some style fixes)


Fix arrow memory checks issue in tests

The tests currently fail on `master` because the arrow memory verification doesn't return the expected memory evolution when loading an arrow table in memory.
From my experiments, the tests fail only when the full test suite is ran.
This made me think that maybe some arrow objects from other tests were not freeing their memory until they do and cause the memory verifications to fail in other tests.

Collecting the garbage collector before checking the arrow memory usage seems to fix this issue.
I added a context manager `assert_arrow_memory_increases` that we can use in tests and that deals with the gc.


Doc2dial update data_infos and data_loaders


ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Hi ! To help me understand the situation, can you print the values of  `load_from_disk(PATH_DATA_CLS_A)['train']._indices_data_files` and `load_from_disk(PATH_DATA_CLS_B)['train']._indices_data_files` ?
They should both have a path to an arrow file

Also note that from #2025 concatenating datasets will no longer have such restrictions.

Sure, thanks for the fast reply!

For dataset A: `[{'filename': 'drive/MyDrive/data_target_task/dataset_a/train/cache-4797266bf4db1eb7.arrow'}]`
For dataset B: `[]`

No clue why for B it returns nothing. `PATH_DATA_CLS_B` is exactly the same in `save_to_disk` and `load_from_disk`... Also I can verify that the folder physically exists under 'drive/MyDrive/data_target_task/dataset_b/'

In the next release you'll be able to concatenate any kinds of dataset (either from memory or from disk).

For now I'd suggest you to flatten the indices of the A and B datasets. This will remove the indices mapping and you will be able to concatenate them. You can flatten the indices with
```python
dataset = dataset.flatten_indices()
```

Indeed this works. Not the most elegant solution, but it does the trick. Thanks a lot!

Hi there,

I am trying to concat two datasets that I've previously saved to disk via `save_to_disk()` like so (note that both are saved as `DataDict`, `PATH_DATA_CLS_*` are `Path`-objects):
```python
concatenate_datasets([load_from_disk(PATH_DATA_CLS_A)['train'], load_from_disk(PATH_DATA_CLS_B)['train']])
```
Yielding the following error:
```python
ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```
Been trying to solve this for quite some time now. Both `DataDict` have been created by reading in a `csv` via `load_dataset` and subsequently processed using the various `datasets` methods (i.e. filter, map, remove col, rename col). Can't figure out tho...

`load_from_disk(PATH_DATA_CLS_A)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 785
})
```
`load_from_disk(PATH_DATA_CLS_B)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 3341
})
```


Doc2dial rc

Added fix to handle the last turn that is a user turn.


outdated dataset_infos.json might fail verifications

Hi ! Thanks for reporting.

To update the dataset_infos.json you can run:
```
datasets-cli test ./datasets/doc2dial --all_configs --save_infos --ignore_verifications
```

Fixed by #2041, thanks again @songfeng !

The [doc2dial/dataset_infos.json](https://github.com/huggingface/datasets/blob/master/datasets/doc2dial/dataset_infos.json) is outdated. It would fail data_loader when verifying download checksum etc..

Could you please update this file or point me how to update this file?

Thank you.


Fix: Wikipedia - save memory by replacing root.clear with elem.clear

The error you got is minor and appeared in the last version of pyarrow, we'll fix the CI to take this into account. You can ignore it

see: https://github.com/huggingface/datasets/issues/2031

What I did:
- replace root.clear with elem.clear
- remove lines to get root element
- $ make style
- $ make test
  - some tests required some pip packages, I installed them.

test results on origin/master and my branch are same. I think it's not related on my modification, isn't it?
```
==================================================================================== short test summary info ====================================================================================
FAILED tests/test_arrow_writer.py::TypedSequenceTest::test_catch_overflow - AssertionError: OverflowError not raised
============================================================= 1 failed, 2332 passed, 5138 skipped, 70 warnings in 91.75s (0:01:31) ==============================================================
make: *** [Makefile:19: test] Error 1

```

Is there anything else I should do?


Cannot load wikitext

when I execute these codes
```
>>> from datasets import load_dataset
>>> test_dataset = load_dataset("wikitext")
```

I got an error,any help?

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 487, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/wikitext/wikitext.py
```


wiki40b/wikipedia for almost all languages cannot be downloaded

Dear @lhoestq for wikipedia dataset I also get the same error, I greatly appreciate if you could have a look into this dataset as well. Below please find the command to reproduce the error:

```
dataset = load_dataset("wikipedia", "20200501.bg")
print(dataset)
```

Your library is my only chance to be able training  the models at scale and I am grateful for your help.

Hi @dorost1234,
Try installing this library first, `pip install 'apache-beam[gcp]' --use-feature=2020-resolver` followed by loading dataset like this using beam runner.

`dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')`

 I also read in error stack trace that:

> Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc.

Worked perfectly fine after this (Ignore these warnings)

![image](https://user-images.githubusercontent.com/19718818/110908410-c7e2ce00-8334-11eb-8d10-7354359e9ec3.png)

For wikipedia dataset, looks like the files it's looking for are no longer available. For `bg`, I checked [here](https://dumps.wikimedia.org/bgwiki/). For this I think `dataset_infos.json` for this dataset has to made again? You'll have to load this dataset also using beam runner.

Hello @dorost1234,

Indeed, Wikipedia datasets need a lot of preprocessing and this is done using Apache Beam. That is the reason why it is required that you install Apache Beam in order to preform this preprocessing.

For some specific default parameters (English Wikipedia), Hugging Face has already preprocessed the dataset for you (and it is stored in the cloud). That is the reason why you do not get the error for English: the preprocessing is already done by HF and you just get the preprocessed dataset; Apache Beam is not required in that case.

Hi
I really appreciate if huggingface can kindly provide preprocessed
datasets, processing these datasets require sufficiently large resources
and I do not have unfortunately access to, and perhaps many others too.
thanks

On Fri, Mar 12, 2021 at 9:04 AM Albert Villanova del Moral <
***@***.***> wrote:

> Hello @dorost1234 <https://github.com/dorost1234>,
>
> Indeed, Wikipedia datasets need a lot of preprocessing and this is done
> using Apache Beam. That is the reason why it is required that you install
> Apache Beam in order to preform this preprocessing.
>
> For some specific default parameters (English Wikipedia), Hugging Face has
> already preprocessed the dataset for you (and it is stored in the cloud).
> That is the reason why you do not get the error for English: the
> preprocessing is already done by HF and you just get the preprocessed
> dataset; Apache Beam is not required in that case.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-797310899>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXACFQZAGMK4VGXRETTDHDI3ANCNFSM4ZA5R2UA>
> .
>

Hi everyone
thanks for the helpful pointers, I did it as @bhavitvyamalik suggested, for me this freezes on this command for several hours,

`Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /users/dara/cache/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
`

Do you know how long this takes? Any specific requirements the machine should have? like very large memory or so? @lhoestq

thanks

HI @dorost1234,
The dataset size is 631.84 MiB so depending on your internet speed it'll take some time. You can monitor your internet speed meanwhile to see if it's downloading the dataset or not (use `nload` if you're using linux/mac to monitor the same). In my case it took around 3-4 mins. Since they haven't used `download_and_extract` here that's why there's no download progress bar.

Hi
thanks, my internet speed should be good, but this really freezes for me, this is how I try to get this dataset:

`from datasets import load_dataset
dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')`

the output I see if different also from what you see after writing  this command:

`Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /users/dara/cache/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...`

do you have any idea why it might get freezed? anything I am missing @lhoestq @bhavitvyamalik. Do I need maybe to set anything special for apache-beam?

thanks a lot

On Tue, Mar 16, 2021 at 9:03 AM Bhavitvya Malik ***@***.***>
wrote:

> HI @dorost1234 <https://github.com/dorost1234>,
> The dataset size is 631.84 MiB so depending on your internet speed it'll
> take some time. You can monitor your internet speed meanwhile to see if
> it's downloading the dataset or not (use nload if you're using linux/mac
> to monitor the same). In my case it took around 3-4 mins. Since they
> haven't used download_and_extract here that's why there's no download
> progress bar.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-800044303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMQIHNNLM2LGG6QKZ73TD4GDJANCNFSM4ZA5R2UA>
> .
>

I tried this on another machine (followed the same procedure I've mentioned above). This is what it shows (during the freeze period) for me:
```
>>> dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')
Downloading: 5.26kB [00:00, 1.23MB/s]
Downloading: 1.40kB [00:00, 327kB/s]
Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Connecting anonymously.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
```
After around 10 minutes, here's the loading of dataset:
```
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.42s/sources]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12sources/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14sources/s]
Dataset wiki40b downloaded and prepared to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f. Subsequent calls will reuse this data.
```

Hi
I honestly also now tried on another machine and nothing shows up after
hours of waiting. Are you sure you have not set any specific setting? maybe
google cloud which seems it is used here, needs some credential setting?
thanks for any suggestions on this

On Tue, Mar 16, 2021 at 10:02 AM Bhavitvya Malik ***@***.***>
wrote:

> I tried this on another machine (followed the same procedure I've
> mentioned above). This is what it shows (during the freeze period) for me:
>
> >>> dataset = load_dataset("wiki40b", "cs", beam_runner='DirectRunner')
> Downloading: 5.26kB [00:00, 1.23MB/s]
> Downloading: 1.40kB [00:00, 327kB/s]
> Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
> WARNING:apache_beam.internal.gcp.auth:Unable to find default credentials to use: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
> Connecting anonymously.
> WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
>
> After around 10 minutes, here's the loading of dataset:
>
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:16<00:00, 16.42s/sources]
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.12sources/s]
> 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.14sources/s]
> Dataset wiki40b downloaded and prepared to /home/bhavitvya/.cache/huggingface/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f. Subsequent calls will reuse this data.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2035#issuecomment-800081772>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMX6A2ZTRZUIIZVFRCDTD4NC3ANCNFSM4ZA5R2UA>
> .
>

Hi
I am trying to download the data as below:

```
from datasets import load_dataset
dataset = load_dataset("wiki40b", "cs")
print(dataset)
```

I am getting this error. @lhoestq I will be grateful if you could assist me with this error. For almost all languages except english I am getting this error.

I really need majority of languages in this dataset to be able to train my models for a deadline and your great scalable super well-written library is my only hope to train the models at scale while being low on resources.

thank you very much.

```
(fast) dara@vgne046:/user/dara/dev/codes/seq2seq$ python test_data.py
Downloading and preparing dataset wiki40b/cs (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to temp/dara/cache_home_2/datasets/wiki40b/cs/1.1.0/063778187363ffb294896eaa010fc254b42b73e31117c71573a953b0b0bf010f...
Traceback (most recent call last):
  File "test_data.py", line 3, in <module>
    dataset = load_dataset("wiki40b", "cs")
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/load.py", line 746, in load_dataset
    use_auth_token=use_auth_token,
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/datasets/builder.py", line 1105, in _download_and_prepare
    import apache_beam as beam
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/apache_beam-2.28.0-py3.7-linux-x86_64.egg/apache_beam/__init__.py", line 96, in <module>
    from apache_beam import io
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/apache_beam-2.28.0-py3.7-linux-x86_64.egg/apache_beam/io/__init__.py", line 23, in <module>
    from apache_beam.io.avroio import *
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/apache_beam-2.28.0-py3.7-linux-x86_64.egg/apache_beam/io/avroio.py", line 55, in <module>
    import avro
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/avro_python3-1.9.2.1-py3.7.egg/avro/__init__.py", line 34, in <module>
  File "/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/avro_python3-1.9.2.1-py3.7.egg/avro/__init__.py", line 30, in LoadResource
NotADirectoryError: [Errno 20] Not a directory: '/user/dara/libs/anaconda3/envs/fast/lib/python3.7/site-packages/avro_python3-1.9.2.1-py3.7.egg/avro/VERSION.txt'
```


Fix typo

Change `ENV_XDG_CACHE_HOME ` to `XDG_CACHE_HOME `


Raise an error for outdated sacrebleu versions

The `sacrebleu` metric seem to only work for sacrecleu>=1.4.12

For example using sacrebleu==1.2.10, an error is raised (from metric/sacrebleu/sacrebleu.py):
```python
    def _compute(
        self,
        predictions,
        references,
        smooth_method="exp",
        smooth_value=None,
        force=False,
        lowercase=False,
        tokenize=scb.DEFAULT_TOKENIZER,
        use_effective_order=False,
    ):
        references_per_prediction = len(references[0])
        if any(len(refs) != references_per_prediction for refs in references):
            raise ValueError("Sacrebleu requires the same number of references for each prediction")
        transformed_references = [[refs[i] for refs in references] for i in range(references_per_prediction)]
>       output = scb.corpus_bleu(
            sys_stream=predictions,
            ref_streams=transformed_references,
            smooth_method=smooth_method,
            smooth_value=smooth_value,
            force=force,
            lowercase=lowercase,
            tokenize=tokenize,
            use_effective_order=use_effective_order,
        )

E       TypeError: corpus_bleu() got an unexpected keyword argument 'smooth_method'
/mnt/cache/modules/datasets_modules/metrics/sacrebleu/b390045b3d1dd4abf6a95c4a2a11ee3bcc2b7620b076204d0ddc353fa649fd86/sacrebleu.py:114: TypeError
```

I improved the error message when users have an outdated version of sacrebleu.
The new error message tells the user to update sacrebleu.
cc @LysandreJik


Use Arrow filtering instead of writing a new arrow file for Dataset.filter

enhancement

New feature or request

Currently the filter method reads the dataset batch by batch to write a new, filtered, arrow file on disk. Therefore all the reading + writing can take some time.

Using a mask directly on the arrow table doesn't do any read or write operation therefore it's significantly quicker.

I think there are two cases:
- if the dataset doesn't have an indices mapping, then one can simply use the arrow filtering on the main arrow table `dataset._data.filter(...)`
- if the dataset an indices mapping, then the mask should be applied on the indices mapping table `dataset._indices.filter(...)`

The indices mapping is used to map between the idx at `dataset[idx]` in `__getitem__` and the idx in the actual arrow table.

The new filter method should therefore be faster, and allow users to pass either a filtering function (that returns a boolean given an example), or directly a mask.

Feel free to discuss this idea in this thread :)

One additional note: the refactor at #2025 would make all the pickle-related stuff work directly with the arrow filtering, so that we only need to change the Dataset.filter method without having to deal with pickle.

cc @theo-m @gchhablani

related issues: #1796 #1949


wikipedia.py generator that extracts XML doesn't release memory

Hi @miyamonz
Thanks for investigating this issue, good job !
It would be awesome to integrate your fix in the library, could you open a pull request ?

I tried downloading Japanese wikipedia, but it always failed because of out of memory maybe.

I found that the generator function that extracts XML data in wikipedia.py doesn't release memory in the loop.

https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L464-L502

`root.clear()` intend to clear memory, but it doesn't.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L490
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L494
I replaced them with `elem.clear()`, then it seems to work correctly.

here is the notebook to reproduce it.
https://gist.github.com/miyamonz/dc06117302b6e85fa51cbf46dde6bb51#file-xtract_content-ipynb


Implement Dataset from text

I am wondering why only one test of "keep_in_memory=True" fails, when there are many other tests that test the same and it happens only in pyarrow_1...

Implement `Dataset.from_text`.

Analogue to #1943, #1946.


Loading a faiss index KeyError

documentation

Improvements or additions to documentation

In your code `dataset2` doesn't contain the "embeddings" column, since it is created from the pandas DataFrame with columns "text" and "label".

Therefore when you call `dataset2[embeddings_name]`, you get a `KeyError`.

If you want the "embeddings" column back, you can create `dataset2` with
```python
dataset2 = load_from_disk(dataset_filename)
```
where `dataset_filename` is the place where you saved you dataset with the embeddings in the first place.

Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I copy-pasted it here.

> When you are done with your queries you can save your index on disk:
>
> ```python
> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
> ```
> Then reload it later:
>
> ```python
> ds = load_dataset('crime_and_punish', split='train[:100]')
> ds.load_faiss_index('embeddings', 'my_index.faiss')
> ```

Hi !

The code of the example is valid.
An index is a search engine, it's not considered a column of a dataset.
When you do `ds.load_faiss_index("embeddings", 'my_index.faiss')`, it attaches an index named "embeddings" to the dataset but it doesn't re-add the "embeddings" column. You can list the indexes of a dataset by using `ds.list_indexes()`.

If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
This looks misleading indeed, and we should add a note to make it more explicit that it doesn't store the column that was used to build the index.

Feel free to open a PR to suggest an improvement on the documentation if you want to contribute :)

> If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
Yes. I was trying to use the dataset in RAG and it complained that the dataset didn't have the right columns. No problems when loading the dataset with `load_from_disk` and then doing `load_faiss_index`

What I learned was
1. column and index are different
2. loading the index does not create a column
3. the column is not needed to be able to use the index
4. RAG needs both the embeddings column and the index

If I can come up with a way to articulate this in the right spot in the docs, I'll open a PR

I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2
Tesla K80

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.


Adding PersiNLU reading-comprehension

@lhoestq I think I have addressed all your comments.

Thanks! @lhoestq  Let me know if you want me to address anything to get this merged.


Update format columns in Dataset.rename_columns

Fixes #2026


KeyError on using map after renaming a column

Hi,

Actually, the error occurs due to these two lines:
```python
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')
```
`Dataset.rename_column` doesn't update the `_format_columns` attribute, previously defined by `Dataset.set_format`, with a new column name which is why this new column is missing in the output.

Hi @mariosasko,

Thanks for opening a PR on this :)
Why does the old name also disappear?

I just merged a @mariosasko 's PR that fixes this issue.
If it happens again, feel free to re-open :)

Hi,

I'm trying to use `cifar10` dataset. I want to rename the `img` feature to `image` in order to make it consistent with `mnist`, which I'm also planning to use. By doing this, I was trying to avoid modifying `prepare_train_features` function.

Here is what I try:

```python
transform = Compose([ToPILImage(),ToTensor(),Normalize([0.0,0.0,0.0],[1.0,1.0,1.0])])
def prepare_features(examples):
    images = []
    labels = []
    print(examples)
    for example_idx, example in enumerate(examples["image"]):
        if transform is not None:
            images.append(transform(examples["image"][example_idx].permute(2,0,1)))
        else:
            images.append(examples["image"][example_idx].permute(2,0,1))
        labels.append(examples["label"][example_idx])
    output = {"label":labels, "image":images}
    return output

raw_dataset = load_dataset('cifar10')
raw_dataset.set_format('torch',columns=['img','label'])
raw_dataset = raw_dataset.rename_column('img','image')

features = datasets.Features({
            "image": datasets.Array3D(shape=(3,32,32),dtype="float32"),
            "label": datasets.features.ClassLabel(names=[
                            "airplane",
                            "automobile",
                            "bird",
                            "cat",
                            "deer",
                            "dog",
                            "frog",
                            "horse",
                            "ship",
                            "truck",
                        ]),
        })
train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)
```
The error:
```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-54-bf29672c53ee> in <module>()
     14                         ]),
     15         })
---> 16 train_dataset = raw_dataset.map(prepare_features, features = features,batched=True, batch_size=10000)

2 frames
/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1287         test_inputs = self[:2] if batched else self[0]
   1288         test_indices = [0, 1] if batched else 0
-> 1289         update_data = does_function_return_dict(test_inputs, test_indices)
   1290         logger.info("Testing finished, running the mapping function on the dataset")
   1291

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1258             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1259             processed_inputs = (
-> 1260                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1261             )
   1262             does_return_dict = isinstance(processed_inputs, Mapping)

<ipython-input-52-b4dccbafb70d> in prepare_features(examples)
      3     labels = []
      4     print(examples)
----> 5     for example_idx, example in enumerate(examples["image"]):
      6         if transform is not None:
      7             images.append(transform(examples["image"][example_idx].permute(2,0,1)))

KeyError: 'image'
```

The print statement inside returns this:
```python
{'label': tensor([6, 9])}
```
Apparently, both `img` and `image` do not exist after renaming.

Note that this code works fine with `img` everywhere.

Notebook: https://colab.research.google.com/drive/1SzESAlz3BnVYrgQeJ838vbMp1OsukiA2?usp=sharing


[Refactor] Use in-memory/memory-mapped/concatenation tables in Dataset

There is one more thing I would love to see. Let's say we iteratively keep updating a data source that loaded from **load_dataset** or **load_from_disk**. Now we need to save it to the same location by overriding the previous file inorder to save the disk space. At the moment **save_to_disk** can not assign a name. So I do not see an easy way to override the previous files.  @lhoestq  is this possible?

p.s one last thing?

Is there a way to flush out any connection to a data source loaded from **load_from_disk** or **load_dataset** methods?  At the moment I suspect when we use any of those functions, it will always keep a pointer although we override it again with a new version of the dataset source. This is really useful in an iterative process.

> There is one more thing I would love to see. Let's say we iteratively keep updating a data source that loaded from **load_dataset** or **load_from_disk**. Now we need to save it to the same location by overriding the previous file inorder to save the disk space. At the moment **save_to_disk** can not assign a name. So I do not see an easy way to override the previous files. @lhoestq is this possible?

In the new save_to_disk, the filename of the arrow file is fixed: `dataset.arrow`.
This way is will be overwritten if you save your dataset again

> Is there a way to flush out any connection to a data source loaded from **load_from_disk** or **load_dataset** methods? At the moment I suspect when we use any of those functions, it will always keep a pointer although we override it again with a new version of the dataset source. This is really useful in an iterative process.

If you update an arrow file, then you must reload it with `load_from_disk` for example in order to have the updated data.
Does that answer the question ? How does this "pointer" behavior manifest exactly on your side ?

Apparently the usage of the compute layer of pyarrow requires pyarrow>=1.0.0 (otherwise there are some issues on windows with file permissions when doing dataset concatenation).

I'll bump the pyarrow requirement from, 0.17.1 to 1.0.0

> If you update an arrow file, then you must reload it with `load_from_disk` for example in order to have the updated data.
> Does that answer the question? How does this "pointer" behavior manifest exactly on your side?

Yes,  I checked this behavior.. if we update the .arrow file it kind of flushes out the previous one. So your solution is perfect <3.

Sorry for spamming, there's a a bug that only happens on the CI so I have to re-run it several times

Alright I finally added all the tests I wanted !
I also fixed all the bugs and now all the tests are passing :)

Let me know if you have comments.

I also noticed that two methods in pyarrow seem to bring some data in memory even for a memory mapped table: filter and cast:
- for filter I took a look at the C++ code on the arrow's side and found [this part](https://github.com/apache/arrow/blob/55c8d74d5556b25238fb2028e9fb97290ea24684/cpp/src/arrow/compute/kernels/vector_selection.cc#L93-L160) that "builds" the array during filter. It seems to indicate that it allocates new memory for the filtered array but not 100% sure.
- regarding cast I noticed that it happens when changing the precision of an array of integers. Not sure if there are other cases.

Maybe we'll need to investigate this a bit for your PR on improving `filter` @theo-m , since we don't want to fill the users memory.

> Maybe we'll need to investigate this a bit for your PR on improving `filter` @theo-m , since we don't want to fill the users memory.

I'm a bit unclear on this, I thought the point of the refactor was to use `Table.filter` to speed up our own `.filter` and stop using `.map` that offloaded too much stuff on disk.
At some point I recall we decided to use `keep_in_memory=True`  as the expectations were that it would be hard to fill the memory?

> I'm a bit unclear on this, I thought the point of the refactor was to use Table.filter to speed up our own .filter and stop using .map that offloaded too much stuff on disk.
> At some point I recall we decided to use keep_in_memory=True as the expectations were that it would be hard to fill the memory?

Yes it's ok to have the mask in memory, but not the full table. I was not aware that the table returned by filter could actually be in memory (it's not part of the pyarrow documentation afaik).
To be more specific I noticed that every time you call `filter`, the pyarrow total allocated memory increases.
I haven't checked on a big dataset though, but it would be nice to see how much memory it uses with respect to the size of the dataset.

I have addressed your comments @theo-m @albertvillanova ! Thanks for the suggestions

I totally agree with you. I would have loved to use inheritance instead.
However because `pa.Table` is a cython class without proper initialization methods (you can't call `__init__` for example): you can't instantiate a subclass of `pa.Table` in python.
To be more specific, you actually can try to instantiate a subclass of `pa.Table` with no data BUT this is not a valid table so you get an error.
And since `pa.Table` objects are immutable you can't even set the data in `__new__` or `__init__`.

EDIT: one could make a new cython class that inherits from `pa.Table` with proper initialization methods, so that we can inherit from this class instead in python. We can do that in the future if we plan to use cython in `datasets`.
(see: https://arrow.apache.org/docs/python/extending.html)

@lhoestq, but in which cases you would like to instantiate directly either `InMemoryTable` or `MemoryMappedTable`? You normally use one of their `from_xxx` class methods...

Yes I was thinking of these cases. The issue is that they return `pa.Table` objects even from a subclass of `pa.Table`

I guess that in this case, the best approach is as you did, using composition over inheritance...

https://github.com/apache/arrow/pull/5322

@lhoestq I think you forgot to add the new classes to the docs?

## Intro

Currently there is one assumption that we need to change: a dataset is either fully in memory (dataset._data_files is empty), or the dataset can be reloaded from disk with memory mapping (using the dataset._data_files).
This assumption is used for pickling for example:
- in-memory dataset can just be pickled/unpickled in-memory
- on-disk dataset can be unloaded to only keep the filepaths when pickling, and then reloaded from the disk when unpickling

## Issues

Because of this assumption, we can't easily implement methods like `Dataset.add_item` to append more rows to a dataset, or `dataset.add_column` to add a column, since we can't mix data from memory and data from the disk.
Moreover, `concatenate_datasets` doesn't work if the datasets to concatenate are not all from memory, or all form the disk.

## Solution provided in this PR

I changed this by allowing several types of Table to be used in the Dataset object.
More specifically I added three pyarrow Table wrappers: InMemoryTable, MemoryMappedTable and ConcatenationTable.
The in-memory and memory-mapped tables implement the pickling behavior described above.
The ConcatenationTable can be made from several tables (either in-memory or memory mapped) called "blocks". Pickling a ConcatenationTable simply pickles the underlying blocks.

## Implementation details

The three tables classes mentioned above all inherit from a `Table` class defined in `table.py`, which is a wrapper of a pyarrow table. The `Table` wrapper implements all the attributes and methods of the underlying pyarrow table.

Regarding the MemoryMappedTable:
Reloading a pyarrow table from the disk makes you lose all the changes you may have applied (slice, rename_columns, drop, cast etc.). Therefore the MemoryMappedTable implements a "replay" mechanism to re-apply the changes when reloading the pyarrow table from the disk.

## Checklist

- [x] add InMemoryTable
- [x] add MemoryMappedTable
- [x] add ConcatenationTable
- [x] Update the ArrowReader to use these new tables depending on the `in_memory` parameter
- [x] Update Dataset.from_xxx methods
- [x] Update load_from_disk and save_to_disk
- [x] Backward compatibility of load_from_disk
- [x] Add tests for the new tables
- [x] Update current tests
- [ ] Documentation

----------

I would be happy to discuss the design of this PR :)

Close #1877


Remove print statement from mnist.py

Thanks for noticing !
#2020 fixed this earlier today though ^^'

Closing this one


Add Romanian to XQuAD

Hi ! Thanks for updating XQUAD :)

The slow test is failing though since there's no dummy data nor metadata in dataset_infos.json for the romanian configuration.

Could you please generate the dummy data with
```
datasets-cli dummy_data ./datasets/xquad --auto_generate --json_field data
```
This will update all the dummy data files, and also add the new one for the romanian configuration.

You can also update the metadata with
```
datasets-cli test ./datasets/xquad --name xquad.ro --save_infos
```
This will update the dataset_infos.json file with the metadata of the romanian config :)

Thanks in advance !

Hello Quentin, and thanks for your help.

I found that running

```python
datasets-cli test ./datasets/xquad --name xquad.ro --save_infos
```

was not enough to pass the slow tests, because it was not adding the new `xquad.ro.json` checksum to the other configs infos and becuase of that an `UnexpectedDownloadedFile` error was being thrown, so instead I used:

```python
datasets-cli test ./datasets/xquad --save_infos --all_configs --ignore_verifications
```

`--ignore_verifications` was necessary to bypass the same `UnexpectedDownloadedFile` error.

Additionally, I deleted `dummy_data_copy.zip` and the `copy.sh` script because they both seem now unnecessary.

The slow tests for both the real and dummy data now pass successfully, so I hope that I didn't mess anything up :)

You're right, you needed the `--ignore_verifications` flag !
Thanks for updating them :)

Although I just noticed that the new dummy_data.zip files are quite big (170KB each) because they contain the json files of all the languages, while only one json file per language is necessary. Could you remove the unnecessary json files to reduce the size of the dummy_data.zip files if you don't mind ?

Done. I created a script (`remove_unnecessary_langs.sh`) to automate the process.

On Jan 18, XQuAD was updated with a new Romanian validation file ([xquad commit link](https://github.com/deepmind/xquad/commit/60cac411649156efb6aab9dd4c9cde787a2c0345))


ValueError when rename_column on splitted dataset

Hi,

This is a bug so thanks for reporting it. `Dataset.__setstate__`  is the problem, which is called when `Dataset.rename_column` tries to copy the dataset with `copy.deepcopy(self)`. This only happens if the `split` arg in `load_dataset` was defined as `ReadInstruction`.

To overcome this issue, use the named splits API (for now):
```python
train_ds, test_ds = load_dataset(
    path='csv',
    delimiter='\t',
    data_files=text_files,
    split=['train[:90%]', 'train[-10%:]'],
)

train_ds = train_ds.rename_column('sentence', 'text')
```

This has been fixed in #2043 , thanks @mariosasko
The fix is available on master and we'll do a new release soon :)

feel free to re-open if you still have issues

Hi there,
I am loading `.tsv` file via `load_dataset` and subsequently split the rows into training and test set via the `ReadInstruction` API like so:

```python
split = {
    'train': ReadInstruction('train', to=90, unit='%'),
    'test': ReadInstruction('train', from_=-10, unit='%')
}

dataset = load_dataset(
    path='csv',               # use 'text' loading script to load from local txt-files
    delimiter='\t',           # xxx
    data_files=text_files,    # list of paths to local text files
    split=split,              # xxx
)

dataset
```

Part of output:
```python
DatasetDict({
    train: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 900
    })
    test: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 100
    })
})
```
Afterwards I'd like to rename the 'sentence' column to 'text' in order to be compatible with my modelin pipeline. If I run the following code I experience a `ValueError` however:
```python
dataset['train'].rename_column('sentence', 'text')
```
```python
/usr/local/lib/python3.7/dist-packages/datasets/splits.py in __init__(self, name)
    353         for split_name in split_names_from_instruction:
    354             if not re.match(_split_re, split_name):
--> 355                 raise ValueError(f"Split name should match '{_split_re}'' but got '{split_name}'.")
    356
    357     def __str__(self):

ValueError: Split name should match '^\w+(\.\w+)*$'' but got 'ReadInstruction('.
```
In particular, these behavior does not arise if I use the deprecated `rename_column_` method. Any idea what causes the error? Would assume something in the way I defined the split.

Thanks in advance! :)


Interactively doing  save_to_disk and load_from_disk corrupts the datasets object?

Hi,

Can you give us a minimal reproducible example? This [part](https://huggingface.co/docs/datasets/master/processing.html#controling-the-cache-behavior) of the docs explains how to control caching.

 dataset_info.json file saved after using  save_to_disk gets corrupted as follows.

![image](https://user-images.githubusercontent.com/16892570/110568474-ed969880-81b7-11eb-832f-2e5129656016.png)

Is there a way to disable the cache that will save to /tmp/huggiface/datastes ?
I have a feeling there is a serious issue with cashing.


Remove unnecessary docstart check in conll-like datasets

Related to this PR: #1998

Additionally, this PR adds the docstart note to the conll2002 dataset card ([link](https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/ned.train) to the raw data with `DOCSTART` lines).


Replace print with logging in dataset scripts

@lhoestq Maybe a script or even a test in `test_dataset_common.py` that verifies that a dataset script meets some set of quality standards (print calls and todos from the dataset script template are not present, etc.) could be added?

Replaces `print(...)` in the dataset scripts with the library logger.


Md gender card update

Link to the card: https://github.com/mcmillanmajora/datasets/blob/md-gender-card/datasets/md_gender_bias/README.md

Ahah that's what I wanted to say @lhoestq, thanks for fixing. Not used to review the Datasets side ;-)

I updated the descriptions of the datasets as they appear in the HF repo and the descriptions of the source datasets according to what I could find from the paper and the references. I'm still a little unclear about some of the fields of the different configs, and there was little info on the word list and name list. I'll contact the authors to see if they have any additional information or suggested changes.


Add TF-based Features to handle different modes of data

Hi,

I am creating this draft PR to work on add features similar to [TF datasets](https://github.com/tensorflow/datasets/tree/master/tensorflow_datasets/core/features). I'll be starting with `Tensor` and `FeatureConnector` classes, and build upon them to add other features as well. This is a work in progress.


Not all languages have 2 digit codes.


Fix ipython function creation in tests

The test at `tests/test_caching.py::RecurseDumpTest::test_dump_ipython_function` was failing in python 3.8 because the ipython function was not properly created.

Fix #2010


more explicit method parameters

re: #2009

not super convinced this is better, and while I usually fight against kwargs here it seems to me that it better conveys the relationship to the `_split_generator` method.


Add Cryptonite dataset

cc @aviaefrat who's the original author of the dataset & paper, see https://github.com/aviaefrat/cryptonite


No upstream branch

What's the issue exactly ?

Given an `upstream` remote repository with url `https://github.com/huggingface/datasets.git`, you can totally rebase from `upstream/master`.

It's mentioned at the beginning how to add the `upstream` remote repository

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L10-L14

~~What difference is there with the default `origin` remote that is set when you clone the repo?~~ I've just understood that this applies to **forks** of the repo 🤡

Feels like the documentation on adding a new dataset is outdated?

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L49-L54

There is no upstream branch on remote.


Add RoSent Dataset

This PR adds a Romanian sentiment analysis dataset. This PR also closes pending PR #1529.

I had to add an `original_id` feature because the dataset files have repeated IDs. I can remove them if needed. I have also added `id` which is unique.

Let me know in case of any issues.


Local testing fails

bug

Something isn't working

I'm not able to reproduce on my side.
Can you provide the full stacktrace please ?
What version of `python` and `dill` do you have ? Which OS are you using ?

```
co_filename = '<ipython-input-2-e0383a102aae>', returned_obj = [0]

    def create_ipython_func(co_filename, returned_obj):
        def func():
            return returned_obj

        code = func.__code__
>       code = CodeType(*[getattr(code, k) if k != "co_filename" else co_filename for k in code_args])
E       TypeError: an integer is required (got type bytes)

tests/test_caching.py:152: TypeError
```

Python 3.8.8
dill==0.3.1.1

I managed to reproduce. This comes from the CodeType init signature that is different in python 3.8.8
I opened a PR to fix this test
Thanks !

I'm following the CI setup as described in

https://github.com/huggingface/datasets/blob/8eee4fa9e133fe873a7993ba746d32ca2b687551/.circleci/config.yml#L16-L19

in a new conda environment, at commit https://github.com/huggingface/datasets/commit/4de6dbf84e93dad97e1000120d6628c88954e5d4

and getting

```
FAILED tests/test_caching.py::RecurseDumpTest::test_dump_ipython_function - TypeError: an integer is required (got type bytes)
1 failed, 2321 passed, 5109 skipped, 10 warnings in 124.32s (0:02:04)
```

Seems like a discrepancy with CI, perhaps a lib version that's not controlled?
Tried with `pyarrow=={1.0.0,0.17.1,2.0.0}`


Ambiguous documentation

Hi @theo-m !

A few lines above this line, you'll find that the `_split_generators` method returns a list of `SplitGenerator`s objects:

```python
datasets.SplitGenerator(
    name=datasets.Split.VALIDATION,
    # These kwargs will be passed to _generate_examples
    gen_kwargs={
        "filepath": os.path.join(data_dir, "dev.jsonl"),
        "split": "dev",
    },
),
```

Notice the `gen_kwargs` argument passed to the constructor of `SplitGenerator`: this dict will be unpacked as keyword arguments to pass to the `_generat_examples` method (in this case the `filepath` and `split` arguments).

Let me know if that helps!

Oh ok I hadn't made the connection between those two, will offer a tweak to the comment and the template then - thanks!

https://github.com/huggingface/datasets/blob/2ac9a0d24a091989f869af55f9f6411b37ff5188/templates/new_dataset_script.py#L156-L158

Looking at the template, I find this documentation line to be confusing, the method parameters don't include the `gen_kwargs` so I'm unclear where they're coming from.

Happy to push a PR with a clearer statement when I understand the meaning.


Fix various typos/grammer in the docs

What do yo think of the documentation btw ?
What parts would you like to see improved ?

I like how concise and straightforward the docs are.

Few things that would further improve the docs IMO:
* the usage example of `Dataset.formatted_as` in https://huggingface.co/docs/datasets/master/processing.html
* the "Open in Colab" button would be nice where it makes sense (we can borrow this from the transformers project + link to HF Forum)

This PR:
* fixes various typos/grammer I came across while reading the docs
* adds the "Install with conda" installation instructions

Closes #1959


How to not load huggingface datasets into memory

So maybe a summary here:
If I could fit a large model with batch_size = X into memory, is there a way I could train this model for huge datasets with keeping setting the same? thanks

The `datastets` library doesn't load datasets into memory. Therefore you can load a dataset that is terabytes big without filling up your RAM.

The only thing that's loaded into memory during training is the batch used in the training step.
So as long as your model works with batch_size = X, then you can load an even bigger dataset and it will work as well with the same batch_size.

Note that you still have to take into account that some batches take more memory than others, depending on the texts lengths. If it works for a batch with batch_size = X and with texts of maximum length, then it will work for all batches.

In your case I guess that there are a few long sentences in the dataset. For those long sentences you get a memory error on your GPU because they're too long. By passing `max_train_samples` you may have taken a subset of the dataset that only contain short sentences. That's probably why in your case it worked only when you set `max_train_samples`.
I'd suggest you to reduce the batch size so that the batches with long sentences can be loaded on the GPU.

Let me know if that helps or if you have other questions

Hi
I am running this example from transformers library version 4.3.3:
(Here is the full documentation https://github.com/huggingface/transformers/issues/8771 but the running command should work out of the box)

 USE_TF=0  deepspeed  run_seq2seq.py --model_name_or_path google/mt5-base --dataset_name wmt16 --dataset_config_name ro-en --source_prefix "translate English to Romanian: " --task translation_en_to_ro   --output_dir /test/test_large  --do_train --do_eval --predict_with_generate  --max_train_samples 500   --max_val_samples 500  --max_source_length 128 --max_target_length 128 --sortish_sampler --per_device_train_batch_size 8   --val_max_target_length 128 --deepspeed ds_config.json --num_train_epochs 1 --eval_steps 25000 --warmup_steps 500 --overwrite_output_dir

(Here please find the script: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py)

If you do not pass max_train_samples in above command to load the full dataset, then I get memory issue on a gpu with 24 GigBytes of memory.

I need to train large-scale mt5 model on large-scale datasets of wikipedia (multiple of them concatenated or other datasets in multiple languages like OPUS), could you help me how I can avoid loading the full data into memory? to make the scripts not related to data size?

In above example, I was hoping the script could work without relying on dataset size, so I can still train the model without subsampling training set.

thank you so much @lhoestq for your great help in advance


Don't gitignore dvc.lock

The benchmarks runs are [failing](https://github.com/huggingface/datasets/runs/2055534629?check_suite_focus=true) because of
```
ERROR: 'dvc.lock' is git-ignored.
```

I removed the dvc.lock file from the gitignore to fix that