Trust & Safety Evaluator with English from Canada I 6- months project
Trust & Safety Evaluator conduct adversarial testing and safety evaluation of generative AI features. Main tasks are crafting queries, evaluating the safety of generated content and providing critical feedback. This role requires creative thinking about potential misuse, deep cultural and linguistic knowledge, and the ability to identify subtle safety risks.
Write, review and evaluate diverse and challenging queries designed to test the system's limits and expose problematic outputs. Queries will target specific risk topics including explicit and/or offensive content
Design and execute sequences of queries simulating realistic, unfolding conversations.
Craft attack scenarios using techniques like crescendo attacks and context manipulation to test the system
Age-Appropriate Safety Evaluation: to guide adversarial query crafting and safety evaluation.
Assign risk ratings to AI Generated content based on safety guidelines.
Detect and articulate potential biases in system-generated content
Identify subtle unsafe elements, inconsistencies, or problematic implications
Evaluate specific cultural and language knowledge across different cultural contexts.
Leverage familiarity with relevant domains, genres, cultural references, and industry context.
Complete adversarial testing tasks within time constraints
Maintain high accuracy and attention to detail while working at pace.
Minimum 6 years of studying/living or working in South Africa
Language skills: English from South Africa as primary language is required
Deep cultural awareness of South African English language: context, social norms, and values
Understanding of cultural references and industry context
Knowledge of how different cultural groups interpret content
Familiarity with behavioral patterns of younger audiences
Ability to evaluate content across different cultural contexts
Query crafting and scenario design
Techniques like crescendo attacks and context manipulation
Risk rating assignment based on guidelines
Bias detection in AI outputs
Attention to detail + ability to work fast under time constraints
Ability to produce creative content
Understanding of slang and youth target language
PREFERRED BACKGROUND Creative Writing, Social Media Management, Community Management, Content Moderation, Digital Marketing, Communications, Journalism, Public Relations, Copywriting, or related fields.
Trust & Safety Background: Prior experience in trust and safety, or similar roles.
Understanding of how safety systems work at scale and common failure modes.
Fluency in additional languages beyond English is an advantage for this role.
Familiarity with Machine Learning Concepts
This role involves exposure to sensitive and potentially disturbing content, including examples of toxic and vulgar language, sexually explicit content, descriptions of violence and self-harm, child abuse, exploitation, and other harmful material. You will encounter this content as part of adversarial testing and safety evaluation.