Optimizing Deeper Transformers on Small Datasets
Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang,
Chenyang Huang, Jackie Chi Kit Cheung, Simon JD Prince, and Yanshuai Cao
ACL 2021
It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train layers of transformers, comprising fine-tuned layers from pre-trained RoBERTa and relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.