Attending to the Room: Sentiment Classification with Masked Language Modeling
Sentiment Classification is a common NLP task with multiple applications. Its use for bias detection and content moderation makes illustrates the importance of developing robust tools to contribute to the safety of online narratives. Social media is a common recipient of these tools, and thus, the cental focus of the work presented. Our final project for a Natural Language Class at CU Boulder was the creation of sentiment classification with masked language modeling.
Method: Using the Sentiment 140 dataset a masked transformer was developed with the task of positive, negative, or neutral classification. Over 1,500,000 tweets compose this dataset and were used to train the masked transformer. The architecture of the transformer utilized causal attention with masked inputs over a variety of training parameters and epochs. A classifier head was then applied to complete the task.
Result: Our best performing model achieved an F1 score of 0.81 and a validation accuracy of 0.81, which shows us that even a compact transformer architecture with causal attention can effectively capture sentiment patterns in short social media text.
Conclusion: Our findings support the viability of lightweight transformer models for sentiment analysis tasks and also highlight their potential as efficient tools for scalable content moderation and bias detection in online environments.
Note: The below manuscript is NOT peer reviewed, it was our final write up and proof of concept for our sentiment classification architecture. The GitHub repository can be viewed in our shared repository csci-5832. I collaborated with Mukun Mahesan to complete this project.
Our report from this work can be found at this page: Attending to the Room: Sentiment Classification with Masked Language Modeling, or by clicking the teaser image below!
