Experiment: How far can a 28M model go in business email generation?

I’ve been experimenting with training a small (~28M parameter) Transformer model on synthetic business email data.

It’s definitely not perfect and still struggles with instruction-following, but I was surprised that it can sometimes produce reasonably coherent email-like text.

The model is very small compared to typical LLMs, so this was more of an experiment to see how far structured generation can go under tight parameter constraints.

Some generations are messy or drift off-topic, but occasionally it produces outputs that almost look usable.

I’d be interested in any feedback, especially ideas on improving consistency or instruction following in small models.

Here’s one sample output:

Prompt: "Write a polite refusal email"

Output:

I understand this is a Friday evening, but I'm happy to provide more information.
I’ll do my best to discuss the details and explore possible alternatives.

We’ll keep you updated on our progress. Please let me know if this is something you’d be interested in.

Best,

[name]

This is from a ~28M parameter model, so it's still inconsistent but occasionally gets close.

If anyone’s interested:
GitHub: https://github.com/kamisori-daijin/textrm
HuggingFace: https://huggingface.co/Kamisori-daijin/textrm-28M-bizmail

(Implementation is loosely based on some TRM experiments and mlx-trm implementations.)