Taxonomy and automated detection of counterspeech strategies using deep learning models
Jana Lasser | jana.lasser@tugraz.at | @janalasser
Which counterspeech strategies are most effective?
The research pipeline
The research pipeline
The research pipeline
The research pipeline
The research pipeline
What I am going to talk about today
What I am going to talk about today
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Taxonomy of strategies
Informed by Benesch (2016), Mathew et al. (2018), Ziegele et al. (2018), Buerger (2021).
Social-psychological aspects
Informed by Burnap et al. (2016), Cikara et al. (2011), Turner et al. (1992), Kirke (2010).
Social-psychological aspects
Informed by Burnap et al. (2016), Cikara et al. (2011), Turner et al. (1992), Kirke (2010).
Social-psychological aspects
Informed by Burnap et al. (2016), Cikara et al. (2011), Turner et al. (1992), Kirke (2010).
Social-psychological aspects
Informed by Burnap et al. (2016), Cikara et al. (2011), Turner et al. (1992), Kirke (2010).
Creating a labelled data set of counterspeech strategies
Labelling process
Labelling process
Labelling process
Labelling process
Labelling process
Strategy category prevalence
Twitter's language classification is not very good
Strategy category prevalence
Some categories are very rare
Strategy category prevalence
Some categories are very rare
Merged categories
"Active" learning
(1) train a classifier with available data (2) predict classes in unlabelled data
(3) bias sample for labelling towards minority classes
"Active" learning
(1) train a classifier with available data (2) predict classes in unlabelled data
(3) bias sample for labelling towards minority classes
Strategy breakdown by speech type
Training a machine learning model to detect counterspeech strategies
Large Language Models for complex classification
Large Language Models for complex classification
Large Language Models for complex classification
Large Language Models for complex classification
Current classifier performance: strategies
Models outperform random guessing but not accurate enough (yet).
Current classifier performance: all dimensions
Ingroup/outgroup and speech type classification already work pretty well.
Preliminary results: linear mixed effects model
Outlook: improving the classifiers
(1) More data: 40% of labelled data still missing. Labelled data contains an increasing amount of minority classes.
(2) Hyperparameter tuning: Current classifier training pipeline is not very sophisticated yet.
(3) Data augmentation: Creating more training examples for minority classes by translation & synonym replacement.
Slides
