BERTology of Molecular Property Prediction

arXiv cs.LG / 3/17/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper systematically investigates how dataset size, model size, and standardization influence pre-training and fine-tuning performance of chemical language models for molecular property prediction.
It addresses inconsistent and contradictory results reported in CLMs across various MPP benchmarks by conducting hundreds of carefully controlled experiments.
The study highlights the lack of well-established scaling laws for encoder-only masked language models and provides numerical evidence to elucidate the underlying mechanisms affecting CLM performance in MPP.
By offering deeper insights and guidelines, the work aims to improve reproducibility and reliability of CLMs for molecular property prediction.

Abstract

Chemical language models (CLMs) have emerged as promising competitors to popular classical machine learning models for molecular property prediction (MPP) tasks. However, an increasing number of studies have reported inconsistent and contradictory results for the performance of CLMs across various MPP benchmark tasks. In this study, we conduct and analyze hundreds of meticulously controlled experiments to systematically investigate the effects of various factors, such as dataset size, model size, and standardization, on the pre-training and fine-tuning performance of CLMs for MPP. In the absence of well-established scaling laws for encoder-only masked language models, our aim is to provide comprehensive numerical evidence and a deeper understanding of the underlying mechanisms affecting the performance of CLMs for MPP tasks, some of which appear to be entirely overlooked in the literature.