Fraser/mnist-text-no-spaces

MNIST dataset adapted to a text-based representation. This allows testing interpolation quality for Transformer-VAEs. System is heavily inspired by Matthew Rayfield's work youtu.be/Z9K3cwSL6uM Works by quantising each MNIST pixel into one of 64 characters.

Every sample has an up & down version to encourage the model to learn rotation invarient features. Use `.array_to_text(` and `.text_to_array(` methods to test your generated data. Removed spaces to get better BPE compression on sequences. **Should only be used with a trained tokenizer.** Data format: - text: (30 x 28 tokens, 840 tokens total): Textual representation of MNIST digit, for example: ``` 00down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 01down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 02down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 03down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 04down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 05down!!!!!!!!!!!!!%%%@CL'Ja^@!!!! 06down!!!!!!!!(*8GK`````YL`]Q1!!!! 07down!!!!!!!-\\````````_855/*!!!!! 08down!!!!!!!%W`````RN^]!!!!!!!!!! 09down!!!!!!!!5H;``T#!+G!!!!!!!!!! 10down!!!!!!!!!$!G`7!!!!!!!!!!!!!! 11down!!!!!!!!!!!C`P!!!!!!!!!!!!!! 12down!!!!!!!!!!!#P`2!!!!!!!!!!!!! 13down!!!!!!!!!!!!)]YI<!!!!!!!!!!! 14down!!!!!!!!!!!!!5]``>'!!!!!!!!! 15down!!!!!!!!!!!!!!,O``F'!!!!!!!! 16down!!!!!!!!!!!!!!!%8``O!!!!!!!! 17down!!!!!!!!!!!!!!!!!_`_1!!!!!!! 18down!!!!!!!!!!!!!!,AN``T!!!!!!!! 19down!!!!!!!!!!!!*FZ```_N!!!!!!!! 20down!!!!!!!!!!'=X````S4!!!!!!!!! 21down!!!!!!!!&1V````R5!!!!!!!!!!! 22down!!!!!!%KW````Q5#!!!!!!!!!!!! 23down!!!!.LY````^B#!!!!!!!!!!!!!! 24down!!!!C```VBB%!!!!!!!!!!!!!!!! 25down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 26down!!!!!!!!!!!!!!!!!!!!!!!!!!!! 27down!!!!!!!!!!!!!!!!!!!!!!!!!!!! ``` - label: Just a number with the texts matching label.