


Das könntest du laut kununu User:innen als Software Analyst in Deutschland verdienen
Der Stellenanzeigentext wurde mithilfe von kununu intelligence (ki) analysiert und strukturiert.
Deine Aufgaben
- Flash and configure Android devices to precise build versions based on specific testing requirements.
- Install, configure, and maintain a diverse suite of consumer applications across multiple test devices.
- Ensure environment consistency to guarantee reproducible test results.
- Run automated scripts to simulate and execute complex user queries on physical or virtual Android devices.
- Manage and execute diverse test suites containing golden & underspecified queries.
- Generate detailed execution trajectories (logs, UI states, and action sequences) from test runs.
- Analyze trajectories to verify if all user actions and target apps were correctly identified, mapped, and executed by the system.
- Maintain, apply, and iteratively update objective rating criteria to evaluate execution success.
- Evaluate automated script execution results, UI action sequences, and AI agent outputs against strict Guidelines to ensure rigorous quality control and compliance.
- Systematically identify, flag, and classify execution failures or trajectory deviations according to specific criterias (e.g., incorrect app selection, wrong action order, UI navigation errors, or policy breaches).
- Document evaluation findings clearly to provide actionable data for engineering teams, contributing to the continuous refinement of evaluation and guideline alignment.
- Verify that the agent correctly parses complex user intent and executes target actions on local apps without violating safety policies or diverging from baseline guidelines.
Dein Profil
- Android Ecosystem Mastery: Strong experience with Android OS, including flashing ROMs/builds, working with adb (Android Debug Bridge), and managing Android environments.
- Code-Level Debugging: Ability to read, interpret, and debug Python/Bash scripts and log files to diagnose errors.
- Data & Trajectory Analysis: Experience analyzing structured logs, JSON outputs, or UI trees to evaluate system behavior.
- Experience with mobile automation tools
- Familiarity with evaluating AI agents, LLM-based tool-use, or complex intent-parsing systems is a massive plus.
- Strong analytical mindset with a rigorous approach to data validation and benchmarking (handling edge cases, negative testing, and ambiguous inputs).
- Excellent documentation skills for updating evaluation rubrics and writing clear bug reports.
- Experience working with mobile app profiling tools.
- Background in AI/LLM evaluation or benchmarking frameworks.
- LLM Tool-Use Knowledge: Conceptual understanding of how LLM agents interact with third-party tools, APIs, and mobile app interfaces.
- Ability to describe app glitches or unexpected behavior clearly.
- Prior experience with quality checking or rating content.
Über den Arbeitgeber
TELUS Digital is the customer experience transformation partner to the world's most admired brands. Our diverse team weaves data, technology, and human ingenuity to deliver differentiated customer journeys, drive operational effectiveness, and scale AI solutions with meaningful value and positive impact.
We craft real-world solutions in the moments that matter, from customer acquisition to lifelong loyalty. Enabled by our global reach - spanning 78,000 experts in 33 countries - and deep industry expertise, we help over 600 organizations make the customer experience feel effortless.
Our solutions span Data & AI, Digital Experience & IT, CX Management and Trust & Safety. At the core of our innovation is Fuel iX™, an enterprise-grade generative AI platform that helps clients safely access and optimize leading LLMs to scale their own AI from pilot to production.
Bewerbungsprozess
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.
At TELUS Digital, we are proud to be an equal opportunity employer and are committed to creating a diverse and inclusive workplace. All aspects of employment, including the decision to hire and promote, are based on applicant's qualifications, merits, competence and performance without regard to any characteristic related to diversity.
Description
KEY RESPONSIBILITIES
1. Device Provisioning & Configuration
- Flash and configure Android devices to precise build versions based on specific testing requirements.
- Install, configure, and maintain a diverse suite of consumer applications across multiple test devices.
- Ensure environment consistency to guarantee reproducible test results.
2. Test Execution & Device Orchestration
- Run automated scripts to simulate and execute complex user queries on physical or virtual Android devices.
- Manage and execute diverse test suites containing golden & underspecified queries.
3. Trajectory Generation & Analysis
- Generate detailed execution trajectories (logs, UI states, and action sequences) from test runs.
- Analyze trajectories to verify if all user actions and target apps were correctly identified, mapped, and executed by the system.
- Maintain, apply, and iteratively update objective rating criteria to evaluate execution success.
4. Output Evaluation
- Evaluate automated script execution results, UI action sequences, and AI agent outputs against strict Guidelines to ensure rigorous quality control and compliance.
- Systematically identify, flag, and classify execution failures or trajectory deviations according to specific criterias (e.g., incorrect app selection, wrong action order, UI navigation errors, or policy breaches).
- Document evaluation findings clearly to provide actionable data for engineering teams, contributing to the continuous refinement of evaluation and guideline alignment.
- Verify that the agent correctly parses complex user intent and executes target actions on local apps without violating safety policies or diverging from baseline guidelines.
REQUIRED QUALIFICATION & SKILLS
Technical Skills
- Android Ecosystem Mastery: Strong experience with Android OS, including flashing ROMs/builds, working with adb (Android Debug Bridge), and managing Android environments.
- Code-Level Debugging: Ability to read, interpret, and debug Python/Bash scripts and log files to diagnose errors.
- Data & Trajectory Analysis: Experience analyzing structured logs, JSON outputs, or UI trees to evaluate system behavior.
Experience & Competencies
- Experience with mobile automation tools
- Familiarity with evaluating AI agents, LLM-based tool-use, or complex intent-parsing systems is a massive plus.
- Strong analytical mindset with a rigorous approach to data validation and benchmarking (handling edge cases, negative testing, and ambiguous inputs).
- Excellent documentation skills for updating evaluation rubrics and writing clear bug reports.
Nice to Have
- Experience working with mobile app profiling tools.
- Background in AI/LLM evaluation or benchmarking frameworks.
- LLM Tool-Use Knowledge: Conceptual understanding of how LLM agents interact with third-party tools, APIs, and mobile app interfaces.
- Ability to describe app glitches or unexpected behavior clearly.
- Prior experience with quality checking or rating content.
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.
About Company:
TELUS Digital is the customer experience transformation partner to the world's most admired brands. Our diverse team weaves data, technology, and human ingenuity to deliver differentiated customer journeys, drive operational effectiveness, and scale AI solutions with meaningful value and positive impact.
We craft real-world solutions in the moments that matter, from customer acquisition to lifelong loyalty. Enabled by our global reach - spanning 78,000 experts in 33 countries - and deep industry expertise, we help over 600 organizations make the customer experience feel effortless.
Our solutions span Data & AI, Digital Experience & IT, CX Management and Trust & Safety. At the core of our innovation is Fuel iX™, an enterprise-grade generative AI platform that helps clients safely access and optimize leading LLMs to scale their own AI from pilot to production.
Equal Opportunity Employer Statement
At TELUS Digital, we are proud to be an equal opportunity employer and are committed to creating a diverse and inclusive workplace. All aspects of employment, including the decision to hire and promote, are based on applicant's qualifications, merits, competence and performance without regard to any characteristic related to diversity.

Ähnliche Jobs, die dich interessieren könnten


