Tibetan News Text Classification and Continuation Dataset （2020–2026年）

Data preprocessing includes sentence boundary segmentation based on Tibetan double hammer symbols or paragraphs, filtering out short phrases, unified encoding and deduplication, effectively removing noise from the original data. It is recommended to use a fixed random seed (42) to randomly shuffle all samples and dynamically divide them according to a ratio of 90% training and 10% validation. To verify the data quality, two types of experiments were conducted: text classification and text generation.

In the classification task, baseline models such as Naive Bayes, logistic regression, random forest, SVM, and TextCNN achieved the highest accuracy of 81.68% on the validation set, with a macro average F1 score of 0.7968. The generation task uses a two-layer LSTM language model (syllable level) to verify that the perplexity has decreased from 750.66 to 32.82. Unlike simply collected raw news data, this dataset has undergone systematic cleaning, annotation, and quality verification, with clear category boundaries and learnable language patterns.

It can be used for downstream tasks such as Tibetan text classification, text continuation, and language model fine-tuning, providing a standardized and reusable important data foundation for low resource language natural language processing research.