Building a programming dataset from scratch

Hey there! First Blog of the Year!

I’ve been working on something really nice lately, and I couldn’t wait to share it with you all! It’s called Espada↗, and it’s a dataset I’ve been building specifically to train AI models in programming domains. Now, datasets might not sound like the most exciting thing ever, but trust me, this one’s different. It’s packed with programming concepts, Q&A pairs, and more.

So, as many of you might know, I like to make stuff from scratch. My projects don't really have any any reason of why I made them, most of them were just me one day thinking- "How can I make my own {Thing}?"- My inspiration for making my own OS from scatch was the same, me wondering one day- "How did they make an entire OS?", and it was the same here. I was looking for a good AI model for Ollama When I found a Model that was trained using the OpenOrca↗ Dataset. The dataset seamed facinating to me, so I decided to try to make my own.

Now, let me tell you about how I built this dataset. First, I spent many days hunting down high-quality content. I pulled material from textbooks, programming forums, and even my own notes. I wanted the dataset to cover everything from the basics, like min-heaps, to some pretty advanced stuff like advanced cryptography. I even used some AI to get the data for the dataset. Here’s a little example from the dataset:

{
  "id": "qa_170",
  "task": "question_answering",
  "system_prompt": "You are a technical expert. Please answer the given question.",
  "question": "What is a min-heap and how is it used?",
  "response": "A min-heap is a complete binary tree where the value of each node is greater than or equal to the value of its parent, with the minimum element at the root. It is used to implement priority queues and in algorithms like Dijkstra's shortest path."
}

It’s all structured like this, so it’s clean and ready to be fed into AI models. I wanted the content to be high quality, so I didn’t just throw in random stuff. Each question-answer pair had to make sense and actually teach something useful.

Now, sourcing the data was just one part. The real challenge came with formatting and organizing it. I decided to go with JSONL because, honestly, it’s just easier to work with. I had to make sure things like token classification were correct to train my own AI.

There were also many challenges. One of the biggest ones was deciding what to include. I had to decide whether I should go deep into niche topics, or keep it broad? I ended up doing a bit of both because I wanted Espada to be versatile. I also had to go back and fix formatting issues. There were a lot of them.

I uploaded it to HuggingFace so anyone can use it. Here’s the link: Espada Dataset↗. Feel free to mess around with it, train some models, and let me know how it goes.

That’s what I’ve been up to. Making a dataset from scratch has taught me many things, and I’m really happy with how it turned out. I’m already thinking of ways to expand it – maybe add more examples, or even make it multilingual. Who knows where this will go?

Anyway, that’s all I am working on for now. Good luck on your programming projects!

Building a programming dataset from scratch

Comments