What is Red Teaming Generative AI?

Paraphrased from the HBR article How to Red Team a Gen AI Model -

A red team is a group that pretends to be an enemy, attempts a physical or digital intrusion against an organization at the direction of that organization, then reports back so that the organization can improve their defenses. Red teams work for the organization or are hired by the organization. 

It is a process of testing your cybersecurity effectiveness through the removal of defender bias by applying an adversarial lens to your organization.

The term was popularized during the Cold War when the U.S. Defense Department tasked “red teams” with acting as the Soviet adversary, while blue teams were tasked with acting as the United States or its allies. 

In October 2023, Biden's AI executive order mandated "red teaming" - a structured hacking effort to uncover flaws in certain high-risk generative models.

Red teaming generative AI is specifically designed to generate harmful content that has no clear analogue in traditional software systems — from generating demeaning stereotypes and graphic images to flat out lying. 

There is no one-size-fits-all approach to creating red teams for generative AI.

The key to effective red teaming lies in triaging each system for risk. Different risk levels can be used to guide the intensity of each red teaming effort: the size of the red team.

Degradation objectives are pre-defined goals or "targets" that specify the most critical failures, harms, and vulnerabilities red team security testers should attempt to purposely produce when probing an AI system. Tightly scoped degradation objectives guide red teams to methodically uncover an AI application's most impactful weaknesses.

When aligning degradation objectives for generative AI systems based on past incidents and risks, the AI Incident Database is a best practice resource.

Few common degradation objectives:

1. Helping users engage in illicit activities
2. Bias in the model - for example, performing differently for members of different groups (native English speakers vs. non-native speakers, for example).
3. Toxicity - As generative AI models are shaped by vast amounts of data scraped from the internet — a place not known for its decorum — toxic content plagues many generative AI systems.
4. Privacy harms

There are a wide variety of methods red teams can use to attacking the model including manual and automated attacks.

Having clear plans for addressing the vulnerabilities that red teaming efforts discover is another central but often-overlooked part of the red teaming process.