The AI Race Needs People Who Can Break What Others Build

Artificial intelligence is advancing quickly, but building more capable AI systems is only one part of the challenge. Companies must also make sure these systems behave as expected, protect sensitive information and do not create new security risks. This is increasing the importance of professionals who can test technology by looking for weaknesses before malicious users find them.

The issue gained attention after a White House meeting on September 29, 2026, where leaders from major technology companies, including OpenAI, Google, Meta, Anthropic, Nvidia and xAI, signed a voluntary AI safety agreement. The agreement called for stronger internal safety controls, continuous monitoring and independent external audits of advanced AI systems.

It also highlighted risks involving cybersecurity, biosecurity, chemical threats and unintended actions by AI models.

This development points to an important area of work in technology: AI red teaming. It involves testing AI systems from an attacker's perspective to identify weaknesses, understand how they could be misused and help developers improve their safeguards.

AI Red Teaming Is Becoming Important for Technology Safety

AI red teaming is a structured process in which professionals test an AI system to discover flaws, vulnerabilities and unexpected behaviour. Instead of using a system only as intended, testers deliberately try different approaches to find situations in which it may fail.

For example, a red team may check whether an AI chatbot can be tricked into revealing sensitive information, following harmful instructions or ignoring its safety restrictions. It may also test whether an AI agent can access files, tools or computer systems beyond the permissions it has been given.

The National Institute of Standards and Technology (NIST) describes AI red teaming as a structured testing effort that uses adversarial methods to identify flaws, vulnerabilities and potential risks associated with the misuse of AI systems.

Some common areas covered by AI red teaming include:

• Prompt injection: Testing whether carefully written instructions can make an AI system ignore its original instructions or follow unauthorised commands.

• Data privacy: Checking whether a model reveals confidential, personal or sensitive information.

• Unauthorised access: Testing whether an AI agent can access files, applications or systems outside its permitted limits.

• Harmful outputs: Identifying situations in which a model generates unsafe, misleading or inappropriate responses.

• Tool misuse: Checking whether an AI system uses connected tools or external services in ways that were not intended.

The purpose of red teaming is not to damage a product but to discover its weaknesses in a controlled and authorised environment. The findings help developers understand what needs to be fixed before a system is made available to more users.

Want to see how AI can spot security gaps before attackers exploit them? Read how AI is being used in cybersecurity.

AI Security Needs More Than Developers and Basic Software Testing

Developers focus on building AI products, improving their performance and adding new features. However, a system that works correctly during normal use may still have security weaknesses that appear only under unusual or deliberately harmful conditions.

This is why AI safety requires different types of testing and security checks. Software testing, penetration testing, ethical hacking and AI red teaming each examine a different part of a technology product.

Software testing checks whether an application works as expected, while security testing examines how it behaves when someone tries to exploit its weaknesses. AI red teaming adds another layer by examining the behaviour of AI models, including their responses to misleading instructions and attempts to misuse their capabilities.

For example, a software tester may check whether an AI-powered application produces the expected output for a particular input. An ethical hacker may examine whether the application has weak authentication or an exposed database. An AI red teamer may test whether the model can be manipulated into disclosing information or performing an unauthorised action.

These activities can work together to improve the security of an AI product.

Professionals working in this area may need to understand:

• Penetration testing: Finding and testing security weaknesses in applications, networks and systems.

• Vulnerability assessment: Identifying security gaps and understanding the risks they create.

• Software quality testing: Checking whether an application performs reliably across different situations.

• AI security evaluation: Testing model behaviour, safeguards and possible misuse scenarios.

• Security monitoring: Observing systems for unusual activity and identifying potential threats.

Discover how AI-powered tools are transforming the way cybersecurity teams detect, analyse, and respond to threats through AI-powered cybersecurity services.

AI Industry Will Need People Who Understand Both AI and Cybersecurity

AI red teaming requires a combination of technical knowledge and problem-solving skills. Professionals need to understand how software works, how AI models respond to instructions and how attackers may try to exploit weaknesses.

A background in computer science, programming, cybersecurity or software testing can help build the foundation for this work. Knowledge of AI and machine learning can further help professionals understand how models are trained, evaluated and deployed.

Some important skills for this field include:

1. Programming and technical fundamentals

Knowledge of programming languages such as Python can help professionals understand application logic, automate tests and examine how software behaves. Familiarity with data structures, APIs and basic system architecture is also useful.

Learners who want to strengthen their programming foundation can explore the Decode Python with DSA Course.

2. Cybersecurity and ethical hacking

Professionals should understand common security vulnerabilities, network fundamentals, web application security and authorised penetration testing. Ethical hacking helps them learn how to identify weaknesses without causing unauthorised damage.

The Certified Ethical Hacking Course can help learners build knowledge of security concepts and ethical hacking practices.

3. AI and machine learning fundamentals

Understanding how AI models process information, generate responses and interact with external tools is important when testing their behaviour. Knowledge of generative AI, model limitations and AI applications can help professionals identify risks that traditional software testing may not cover.

Learners can explore the Gen AI Engineering Course to develop their understanding of generative AI and its applications.

4. Software testing and quality assurance

Testing skills help professionals create test cases, identify errors, document results and check whether issues have been resolved. These abilities are useful when evaluating both traditional software and AI-powered applications.

The Software Testing with AI Bootcamp is relevant for learners looking to understand software testing practices and the role of AI in testing workflows.

Along with technical knowledge, red team professionals need curiosity, analytical thinking and clear communication. They must be able to explain what went wrong, how a weakness could affect users and what developers can do to address it.

These capabilities are also part of the broader set of AI skills employers want beyond ChatGPT.

Why AI Safety Is Becoming a Shared Responsibility

The White House agreement signed by major technology companies in September 2026 reflects growing attention to how advanced AI systems should be monitored and evaluated. It calls for companies to maintain internal controls, use independent external auditors and establish board-level oversight of safety reviews.

The agreement is voluntary and does not establish that every participating company has completed independent audits or resolved all identified risks. 

This also highlights why AI safety cannot depend only on the teams building these systems. Developers understand how a product is designed, but independent reviewers and security professionals can examine it from a different perspective.

As AI becomes part of software development, business operations, customer services and other digital systems, the need to evaluate its behaviour will continue to grow. Professionals who understand both AI and cybersecurity can contribute to this work by testing systems, identifying vulnerabilities and helping organisations reduce risks.

Key Takeaways

• AI red teaming helps identify vulnerabilities, unsafe behaviour and possible misuse of AI systems.

• The September 2026 White House AI safety agreement called for internal controls, monitoring and independent external audits.

• AI security requires a combination of software testing, ethical hacking, penetration testing and AI evaluation.

• Programming, cybersecurity, AI fundamentals and problem-solving are useful skills for professionals entering this field.

• AI safety is not only the responsibility of developers; security professionals and independent evaluators also play an important role.

The AI Race Also Needs People Who Can Test Its Limits

The development of advanced AI systems will continue to create new opportunities, but it will also introduce new technical and security challenges. Finding these weaknesses before they affect users is an important part of building reliable technology.

AI red teaming brings together programming, cybersecurity, ethical hacking and software testing. For learners interested in this area, developing these skills can provide a foundation for exploring roles in cybersecurity, AI evaluation and software quality assurance.

The future of AI will depend not only on what technology can do, but also on how effectively people can test, secure and control it.

Want to understand what could make you valuable in an AI-driven workplace? Explore AI skills and the future of work to identify where your learning journey could take you next.