 | Hi everyone, I saw this post (28m model email experiment) and decided to train on this dataset but with a different architecture. Test prompt same as the above post: Prompt: Output: ``` Correcting Reason for Recent Experience Dear [Officer's Name], I hope this email finds you well. As you know, [Company Name] has recently received your experience with [Product Name], a hugely successful funding round, and we’re committed to providing you with the billing records and targeted a highly value of [Client Name]. As you know, our rebranding initiative is currently undergoing a significant rebranding phase, and we want to sincerely apologize for the timing. We value your time and appreciate your understanding. We understand your concerns swiftly and want to assure you that we will be moving forward. Please let us know your availability. Sincerely, [Name] Junior Developer [Company Name]<|eot|> ```
I've been experimenting with a simple idea. That is, completely removing FFN and replacing the Linear layers in Swiglu FFN with Attention layers. Thus converting Swiglu into something I call Silia (Silu in attention). It achieved similar loss and performance (compared to a standard Attention + Swiglu architecture) on same dataset & training config with much less parameters. This is the architecture diagram:
I trained on email-datasets-20k dataset which was used in the post I linked above. This is the model training config: The model has 0.8M total params out of which 0.3M are non-embedding params. The model has 2 blocks (4 attention layers & 2 activations in total), 4 attention heads. I used my custom tokenizer with 8k vocab size. It is just Regex + BPE tokenizer which Andrej Karpathy made in one of his videos, the only difference is I'm using After tokenization the dataset had 5.5M total tokens, after splitting by 80/20 rule, I had 4.4M train tokens, 1.1M val tokens. The dataset had ~20M chars in total. I trained on the dataset for ~10 epochs. The final train & val loss were 1.65 & 1.68 respectively. I've attached some screenshots of loss & demo generations. Here's the github repo link: https://github.com/SrijanSriv211/Strawberry You can download the model from here: https://github.com/SrijanSriv211/Strawberry/releases/tag/s0.2a Thank you :) [link] [comments] |
Trained a 0.8M model on business email generation.
Reddit r/LocalLLaMA / 3/21/2026
💬 OpinionTools & Practical UsageModels & Research
Key Points
- A 0.8M parameter model for business email generation was trained using a novel architecture called Silia that replaces FFN with attention blocks to reduce parameters while preserving performance.
- The training used the email-datasets-20k dataset with an 80/20 data split and prompts like "Write a polite refusal email" to demonstrate outputs.
- The architecture combines 2x Strawberry Blocks, scaled dot-product attention, rotary positional embeddings, QK norm, multi-head attention, SiLU non-linearity, and a weight-tied output projection, forming a Swiglu-into-attention variant named Silia.
- The work references a prior 28M-model email experiment and shows that the smaller model can achieve comparable loss and performance on the same data/config, signaling potential for practical business email automation.
