Translator Transformer

This project aims to implement a Transformer-based machine translation system from scratch.

It is developed purely for educational purposes. My primary goal is to apply and deepen the theoretical knowledge I gained during my university studies by implementing the underlying components myself.

The initial translation task is English ↔ German. I'm using the bentrevett/multi30k dataset.

I am trying to use as few packages as possible. I'm using Hugging Face's datasets package to import the dataset, and I am going to use PyTorch for tensor mathematics. However, I am going to implement automatic differentiation myself and will not use PyTorch's implementation.

Roadmap

• Implement a Byte Pair Encoding (BPE) tokenizer
• Implement automatic differentiation
• Use the automatic differentiation system to train a simple neural network with backpropagation
• Implement masked self-attention
• Implement positional encoding
• Implement the Transformer encoder–decoder model
• Train and evaluate the model on an English–German translation dataset
• Optionally explore automatic hyperparameter tuning using Bayesian optimization

Modules

• Tokenizer: tokenizer.ipynb Contains the BPE tokenizer implementation. Given a text dataset (3,819,184 characters and 58 unique characters after normalization), it learns a token vocabulary of 256 tokens and provides:

  • enc: converts text into token IDs
  • dec: converts token IDs back into text

  For normalization, all sentences are lowercased, and rare characters such as à, ñ, and \t are replaced or removed.

Additional notebooks and source files will be added as the project progresses.

AI Usage Disclosure

I do not use generative AI to implement the project code.

I do, however, use AI tools (primarily ChatGPT) for assistance with Python syntax, conceptual questions, and review of code correctness. The implementation decisions and the code itself are written by me.