Table · dataset · 2022
Saibo-creator/bookcorpus_deduplicated
Listed in Hugging Face Datasets
Dataset Card for "bookcorpus_deduplicated" Dataset
Description
Summary
This is a deduplicated version of the original Book Corpus dataset. The Book Corpus (Zhu et al., 2015), which was used to train popular models such as BERT, has a substantial amount of exact-duplicate documents according to Bandy and Vincent (2021) Bandy and Vincent (2021) find that thousands of books in BookCorpus are duplicated, with only 7,185 unique books out of 11,038 total. Effect of deduplication Num of… See the full description on the dataset page: huggingface.co/datasets/Saibo-creator/bookcorpus_deduplicated.
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/Saibo-creator/bookcorpus_deduplicated ↗
landing page · from Hugging Face
Documentation and papers
- arXiv:2105.05241 arxiv.org/abs/2105.05241 ↗
publication · from Hugging Face
- arXiv:2107.06499 arxiv.org/abs/2107.06499 ↗
publication · from Hugging Face
- arXiv:2209.00099 arxiv.org/abs/2209.00099 ↗
publication · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/Saibo-creator/bookcorpus_deduplicated ↗
metadata API · from Hugging Face
Topics
- Stated by source
- text
- From keywords
- Computer Science & AI
Provenance · 1 source records, 8 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | Saibo-creator/bookcorpus_deduplicated | 15 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:text | source · Hugging Face | connector:huggingface@1.0.0 | |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |