Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali

arXiv cs.CL / 4/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The study benchmarks linguistic adaptation for Romanized Nepali using three similar open-weight LLMs—Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B—under both zero-shot and fine-tuned (SFT) conditions.
Across multiple evaluation metrics (PPL, BERTScore, chrF++, ROUGE-1/2/L, and BLEU), all models fail to generate Romanized Nepali in zero-shot mode, each showing different, architecture-specific failure patterns.
After fine-tuning with QLoRA using rsLoRA (r=32) while training only ~1% of parameters for under 27 GPU-hours, all models substantially improve and converge toward BERTScore ≈ 0.75 and chrF++ > 23.
Dimension-wise results recommend Qwen3-8B overall, as it is the only model producing semantically relevant Romanized Nepali output in zero-shot and it leads structural alignment metrics after SFT.
The authors confirm an “adaptation headroom” effect: Llama-3.1-8B, though weakest in zero-shot, delivers the largest absolute fine-tuning gains in PPL (Δ=-49.77) and BERTScore (Δ=+0.3287), making it attractive for iterative low-resource development.

Abstract

Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal, yet it remains critically underresourced in the landscape of Large Language Models (LLMs). This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models: Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B. We evaluate these architectures under zero-shot and fine-tuned settings using a curated bilingual dataset of 10,000 transliterated instruction-following samples. Performance is quantified across five metrics spanning seven measurement dimensions: Perplexity (PPL), BERTScore, chrF++, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU, capturing fluency, phonetic consistency, and semantic integrity. Models were fine-tuned using Quantized Low-Rank Adaptation (QLoRA) with Rank-Stabilized LoRA (rsLoRA) at rank r=32 on dual NVIDIA Tesla T4 GPUs, training only approximately 1% of each model's parameters in under 27 total GPU-hours. At zero-shot, all three models fail to generate Romanized Nepali, each exhibiting a distinct architecture-specific failure mode. Following fine-tuning, all three resolve these failures and converge to BERTScore approximately 0.75 and chrF++ greater than 23. Overall dimension-wise assessment across ten criteria identifies Qwen3-8B as the overall recommended architecture, being the only model to produce semantically relevant zero-shot output and leading all structural alignment metrics post-SFT. The adaptation headroom hypothesis is confirmed: Llama-3.1-8B, despite its weakest zero-shot baseline, achieves the largest absolute fine-tuning gains in PPL (Delta = -49.77) and BERTScore (Delta = +0.3287), making it the preferred choice for iterative low-resource development pipelines. This work establishes the first rigorous baseline for Romanized Nepali adaptation in comparable-sized open-weight LLMs.