Two-Stage Bootstrap Toxicity Classifier on Reddit Data

Paper context

Paper title: Reducing Toxicity in Language Models | Lil'Log Abstract: The figure illustrates a two-stage data bootstrapping pipeline for training a toxicity classifier, showing subreddit sorting and utterance sampling. Source body: Table of Contents Categorization of Toxic Content Data Collection Human Annotations Semi-supervised Dataset Toxicity Detection Adversarial Attacks Perspective API Prompt-based Detection Detoxification Blacklisting Prompt-based Detox Text Style Transfer Controllable Generation System-level Safety Solution Appendix: Datasets Citation References Large pretrained language models are trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process. Many challenges are associated with the effort to diminish various types of unsafe content: First, there are a variety of unsafe content types, such as toxicity, abusiveness, hate speech, biases, stereotypes, cyberbullying, identity attacks and more, which may or may not demand different treatment. Second, there is no clearly and widely agreed-upon categorization and definition of unsafe behavior in pretrained language models. Individual perceptions could vary a lot due to different social backgrounds. In this post, we delve into the issue of toxicity in language models. As I’m still struggling to find a concrete definition of toxic content, I list a couple in the literature below. [ Perspective API ] A rude, disrespectful, or unreasonable comment; likely to make people leave a discussion. [ Kurita et al. 2019 ] Content that can offend or harm its recipients, including hate speech, racism, and offensive language. [ Pavlopoulos et al. 2020 ] We use the term ’toxic’ as an umbrella term, but we note that the literature uses several terms for different kinds of toxic language or related phenomena: ‘offensive’, ‘abusive’, ‘hateful’, etc. Overall, toxicity is a broad term to describe several types of unsafe content. Methodologies in this post can be applied given some form of definition of toxicity; e.g. presented in the instruction for annotators. How to properly define the concept of toxicity and thus collect accurate annotation labels is out of the scope of this post. Categorization of Toxic Content # How to categorize toxic content is not a straightforward task. Which content should be considered toxic and what types of toxic content exist can be very subjective. Language that does not look offensive to one group might seem inappropriate to another. One popular categorization of offensive language is proposed by Zampieri et al. (2019) , a three-level hierarchical taxonomy considering both the type and the target of offense. The Offensive Language Identification Dataset ( OLID ) dataset is collected based on this taxonomy. The three-level hierarchical taxonomy for categorizing offensive language, proposed by Zampieri et al. (2019) . Level A: “Is it offensive?” [OFF] Offensive: Inappropriate language, insults, or threats. [NOT] Not offensive: No offense or profanity. Level B: “Is the offensive text targeted?” [TIN] Targeted Insult: Targeted insult or threat towards an individual, a group or other. [UNT] Untargeted: Non-targeted profanity and swearing. Level C: What is the target? [IND] The offense targets an individual, often defined as “cyberbullying”. [GRP] The offense targets a group of people based on ethnicity, gender, sexual orientation, religion, or other common characteristic, often defined as “hate speech”. [OTH] The target can belong to other categories, such as an organization, an event, an issue, etc. Data Collection # Preparing a dataset of samples labelled as “safe” vs “unsafe” is the foundation for training a toxic language classifier and further providing signals for model detoxification. Human Annotations # Vidgen & Derczynski (2020) summarized that training data annotations for toxicity detection on the high level can be collected by: Expert coding : An expert has enough knowledge or training to complete the annotation tasks with good quality, such as a researcher who studies prejudice, a student with moderate level of training, or a NLP practitioner. It is more expensive but produces high-quality data. Crowdsourcing : Crowdsourcing platform pairs a large number of non-expert annotators with tasks. It is easier to scale up but demands more attention on quality control. Professional moderators : Professional moderators are experienced, well-trained on the tasks, but their goals are likely to optimize for the output specific to the platform. Synthetic data : Training dataset can also be manually created by relevant content creators to cover a broad range of toxic content types. Crowdsourcing is the most common approach among them ( Davidson et al. 2017 , Zampieri et al. 2019 ) and there are several good practices to improve the data quality: Test data : A small set of annotations collected from a few experts can be used as test questions ( Zampieri et al. 2019 ) to filter out human annotators on the crowdsourcing platform who cannot achieve a certain threshold. Clear guidelines : Detailed instructions are useful to guide annotators to produce aligned and consistent labels. Without any guideline, annotators are encouraged to apply their personal perceptions, which could be problematic because (1) subjective interpretation of toxic content varies across individuals greatly and (2) it is tricky to mark certain types of noise like sarcasm and irony without any guideline. Majority vote : It is very common that we need labels from multiple annotators per sample and take the majority vote. Understanding annotators’ identities : Demographic background has a big impact on the annotator’s understanding of the task. We should aim to recruit diverse and qualified annotators. Semi-supervised Dataset # Khatri et al. (2018) pr