The nextAI Solution to the NeurIPS 2023 LLM Efficiency Challenge

arXiv cs.LG / 4/13/2026

💬 OpinionSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper describes how the authors participated in the NeurIPS 2023 LLM Efficiency Challenge by fine-tuning a LLaMA 2 70B model under strict compute and time constraints.
Their workflow used a custom, benchmark-aligned dataset assembled from diverse open-source sources and refined through multiple dataset iterations to improve generalization.
They fine-tuned with QLoRA while incorporating Flash Attention 2 and experimented with different LoRA configurations to balance efficiency and accuracy.
The resulting model met the challenge goal by running on a single NVIDIA A100 40GB GPU within a 24-hour limit while maintaining strong performance on QA benchmarks.
The authors conclude that large-scale LLMs can be efficiently adapted in resource-constrained settings, supporting practical deployment with reduced resource demands.

Abstract

The rapid evolution of Large Language Models (LLMs) has significantly impacted the field of natural language processing, but their growing complexity raises concerns about resource usage and transparency. Addressing these challenges, we participated in the NeurIPS LLM Efficiency Challenge, aiming to fine-tune a foundation model within stringent constraints. Our focus was the LLaMa2 70 billion model, optimized on a single A100 40GB GPU within a 24-hour limit. Our methodology hinged on a custom dataset, carefully assembled from diverse open-source resources and benchmark tests, aligned with the challenge's open-source ethos. Our approach leveraged Quantized-Low Rank Adaptation (QLoRA) Fine tuning, integrated with advanced attention mechanisms like Flash Attention 2. We experimented with various configurations of the LoRA technique, optimizing the balance between computational efficiency and model accuracy. Our fine-tuning strategy was underpinned by the creation and iterative testing of multiple dataset compositions, leading to the selection of a version that demonstrated robust performance across diverse tasks and benchmarks. The culmination of our efforts was an efficiently fine-tuned LLaMa2 70B model that operated within the constraints of a single GPU, showcasing not only a significant reduction in resource utilization but also high accuracy across a range of QA benchmarks. Our study serves as a testament to the feasibility of optimizing large-scale models in resource-constrained environments, emphasizing the potential of LLMs in real-world applications.