TL;DR: The default QLoRA learning rate of 2e-4, commonly cited, is often too high for fine-tuning on datasets under 10,000 samples, leading to overfitting.
Summary: A developer found that the widely recommended QLoRA learning rate of 2e-4, originating from the 52k-sample Alpaca dataset, causes rapid overfitting when fine-tuning on smaller datasets (under 10k samples). Reducing the learning rate to 1e-4 and increasing epochs significantly improved evaluation metrics in their experiments, suggesting the default is a 'trap' for many common use cases.
Why it matters: AI builders using QLoRA for fine-tuning smaller custom datasets should reconsider the default 2e-4 learning rate. Experiment with lower rates like 1e-4 and adjust epochs to prevent overfitting and achieve better model performance.
Source: reddit