Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence
arXiv Preprint Archive December 1, 2024 Qianren Mao, Weifeng Jiang, Junnan Liu et al.
Semi-supervised learning (SSL) in lightweight models faces performance limits due to few training labels. PS-NET, a new framework for SSL text mining with small models, uses online distillation so a lightweight student model imitates a teacher model, plus an ensemble of student peers that teach each other, and constant adversarial perturbations for self-augmentation. With a 2-layer distilled BERT, PS-NET outperforms state-of-the-art lightweight SSL frameworks FLiText and DisCo in text classification when labeled data is extremely scarce.