Text · dataset · 2022
shjwudp/chinese-c4
Listed in Hugging Face Datasets
Introduction Chinese-C4 is a clean Chinese internet dataset based on Common Crawl.
Description
The dataset is 46.29GB and has undergone multiple cleaning strategies, including Chinese filtering, heuristic cleaning based on punctuation, line-based hashing for deduplication, and repetition removal. The dataset is open source and free for commercial use, and you are welcome to use the data and the cleaning strategies provided and contribute your cleaning strategies.
You can find the cleaning… See the full description on the dataset page: huggingface.co/datasets/shjwudp/chinese-c4.
Links
Where it is published
- Hugging Face dataset page huggingface.co/datasets/shjwudp/chinese-c4 ↗
landing page · from Hugging Face
Catalogue records · 1
- Hub API huggingface.co/api/datasets/shjwudp/chinese-c4 ↗
metadata API · from Hugging Face
Topics
- Stated by source
- text
- From keywords
- Computer Science & AI
Provenance · 1 source records, 9 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Hugging Face Datasets | shjwudp/chinese-c4 | 15 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · Hugging Face | connector:huggingface@1.0.0 | /gated |
| concepts[field].local:field:computer-science-ai | mapping · Hugging Face | connector:huggingface@1.0.0 | |
| concepts[modality].hf_modality:text | source · Hugging Face | connector:huggingface@1.0.0 | |
| created_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| description | source · Hugging Face | connector:huggingface@1.0.0 | /description |
| license | source · Hugging Face | connector:huggingface@1.0.0 | /tags[license:*] |
| publication_date | source · Hugging Face | connector:huggingface@1.0.0 | |
| title | source · Hugging Face | connector:huggingface@1.0.0 | /id |
| updated_date | source · Hugging Face | connector:huggingface@1.0.0 |