In the summer of 2023 - as a weekend project - I built a small language model from scratch that has 1.5 million parameters, trained^ for 20 minutes on 400 scientific paper abstracts* , capable of generating paper abstracts (well, sort of) based on a given prompt. One thing to mention is that this is just a very proof of concept that was created by reverse-engineering the Llama2 model, a week after its release. To build this, I reviewed the Llama2 paper (as well as the Python Object of the model's Network - after I uploaded the 7b model locally from HuggingFace) and generated the corresponding code based on the architecture described in the paper and the code object. All of this was done with the assistance of GPT-4. Below is the associated notebook, and here.
^trained on M2 Apple chip with 48GB. abstracts collected from papers related to the topic of 'LLMs Generative AI' between 2020 and 2023.*
Samples of the generation - Of course, it is not coherent and mostly rubbish. It was just a 20-minute training model with around 200 pages of text data.
The complete notebook is embedded (scrollable) below.
https://nbviewer.org/gist/iamaziz/171170dce60d9cd07fab221507fd1d52
In December 2023, I revisited the notebook and adapted it to generate Arabic news articles (trained using first 100 articles of this dataset) instead of scientific abstracts.
Notebook (scrollable)
https://nbviewer.org/gist/iamaziz/7b7a61120f3418b3511078f0fb2a0ec1
In 2016, during my PhD, I built my first language model from scratch, based on word2vec, capable of completing Arabic analogies by performing arithmetic operations on word embeddings e.g. king - man + woman = queen. And published a paper on this work, also released the dataset, code, and weights. The repo is here.
ℹ️ More details
