1. Introduction In natural language processing (NLP), text classification is widely used for language understanding tasks such as sentiment analysis, intent recognition, and occupation classification [SO21, CTG*20, ECCB23]. For these tasks, NLP practitioners commonly adopt deep learning models like CNNs, LSTMs, and pre-trained large language models (LLMs) [LPL*22]. Although these models score high in performance metrics like accuracy, they are well-known to be difficult to interpret and trust.