AI / ML · PyTorch · open source
GPT-2 transformer Open source
A mini GPT-2 written from the ground up: custom tokenizer, token and positional embeddings, causal multi-head self-attention, residual feed-forward layers and autoregressive generation. Trained on Tiny Shakespeare.
I built a GPT-2 model from scratch using the Tiny Shakespeare dataset. This project is a decoder-only transformer that generates text in Shakespearean English. I implemented key components like tokenization, self-attention, and autoregressive text generation, ensuring the model could learn patterns effectively.
Instead of using pre-built libraries, I wrote everything from the ground up, focusing on understanding each part of the architecture. Training was done on character-level inputs, making it a great exercise in working with sequence models. The model successfully generates Shakespeare-like text, capturing the style and flow of the original dataset.
I also wrote a detailed 6.2k-word blog explaining my implementation, which you can find here. This project was both challenging and rewarding, reinforcing my understanding of transformers and deep learning.