LV-BERT: Exploiting Layer Variety for BERT

Modern pre-trained language models are mostly built upon backbones stacking self-attention and feed-forward layers in an interleaved order. This paper aims to improve pre-trained models by exploiting layer variety from two aspects: the layer type set and the layer order.