Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence
Qianren Mao, Weifeng Jiang, Junnan Liu, Chenghua Lin, Qian Li, Xianqing Wen, Jianxin Li, Jinhu Lu
arXiv Preprint Archive December 1, 2024 via arXiv
Summary
AI-generated from the abstractSemi-supervised learning (SSL) in lightweight models faces performance limits due to few training labels. PS-NET, a new framework for SSL text mining with small models, uses online distillation so a lightweight student model imitates a teacher model, plus an ensemble of student peers that teach each other, and constant adversarial perturbations for self-augmentation. With a 2-layer distilled BERT, PS-NET outperforms state-of-the-art lightweight SSL frameworks FLiText and DisCo in text classification when labeled data is extremely scarce.
Study at a glance
| Characteristics | Theoretical or computational paper Peer reviewed |
|---|---|
| Keywords | Cs.cl Cs.lg |
| Key finding | Proposes that PS-NET, combining online distillation, peer ensemble, and adversarial perturbation, improves semi-supervised text classification performance over existing lightweight SSL frameworks under extremely scarce labeled data. |
Abstract
The semi-supervised learning (SSL) strategy in lightweight models requires reducing annotated samples and facilitating cost-effective inference. However, the constraint on model parameters, imposed by the scarcity of training labels, limits the SSL performance. In this paper, we introduce PS-NET, a novel framework tailored for semi-supervised text mining with lightweight models. PS-NET incorporates online distillation to train lightweight student models by imitating the Teacher model. It also integrates an ensemble of student peers that collaboratively instruct each other. Additionally, PS-NET implements a constant adversarial perturbation schema to further self-augmentation by progressive generalizing. Our PS-NET, equipped with a 2-layer distilled BERT, exhibits notable performance enhancements over SOTA lightweight SSL frameworks of FLiText and DisCo in SSL text classification with extremely rare labelled data.