zscaler logo

Senior Staff Machine Learning Engineer

zscaler • Remote - USA; Santa Clara, California, USA


No Relocation

Posted: September 30, 2026

Job Description

Role

We are looking for a Senior Staff Machine Learning Engineer to join our team. This is a remote (USA) role, reporting to the Manager AI Platform and Data Science in the AI Platform and Data Science department. This team’s mission is high-fidelity risk identification in customer data, with goals to catch all threats while minimizing noise. To achieve this, you will develop solutions applying data analysis and threat-research while leveraging AI, machine learning, and data engineering to build quality, data-driven components to automate security analysis.

What you’ll do (Role Expectations)

  • Translate risk identification methods into agent logic, understanding the benefits and limitations of agents and ensuring quality across risk analysis, explanations, and recommendations
  • Collaborate closely with threat-research to understand data, threats, and their approach to developing security heuristics
  • Identify and solve data requirements for analysis, developing and collaborating with data engineering teams for pipelines, enrichments, and aggregations
  • Follow a data-driven quality approach to threat detection, including backtesting, balancing precision vs recall, tuning, and quality control
  • Deploy and monitor your solutions in production within our CI/CD framework

Who You Are (Success Profile)

  • You thrive in ambiguity and build the path as you walk it, seeing unstructured challenges as the raw material to construct meaningful systems.
  • You act like an owner with a deep passion for the mission, operating with integrity and navigating seamlessly between high-level strategy and hands-on execution.
  • You are a continuous learner with a growth mindset who actively seeks feedback to develop yourself, elevate your partners, and execute with purpose.
  • You are a positive force who approaches complex technical challenges with contagious, constructive energy and a focus on solutions.
  • You are data-driven, using analytics and concrete evidence over assumptions to find the truth, measure what matters, and guide informed decisions.

What We’re Looking for (Minimum Qualifications)

  • Experience building LLM-powered agents in production: tool and function calling, prompt and context engineering, multi-step orchestration frameworks (LangGraph, LangChain, or equivalent), and evaluation of non-deterministic output against ground truth
  • 8+ years of professional Python development with demonstrated ability to design and maintain production-quality systems with validated inputs, data contracts, and comprehensive unit and integration tests, alongside hands-on expertise with SQL and Python data analytics libraries (e.g., pandas, Polars, NumPy)
  • Demonstrated ownership of production reliability: CI/CD, containerization, structured logging and metrics, observability and alerting, on-call participation, incident debugging in live distributed systems, and familiarity with cloud and infrastructure-as-code in an AWS environment
  • Sr. Staff level ownership and leadership of emergent requirements, architecture, and complex engineering projects

What Will Make You Stand Out (Preferred Qualifications)

  • Experience with production AI or ML systems where cost, latency, and accuracy are competing constraints, including handling model selection, routing, and regression testing
  • Background in cyber threat research, threat modeling, threat hunting, detection engineering, or previous experience in data-driven risk analysis, fraud detection, adversary profiling and targeting, actuarial risk, or close collaboration with risk analysis teams
  • Data engineering experience building and operating pipelines over large-volume event data using SQL, columnar or search-backed stores (OpenSearch/Elasticsearch, Athena/Presto), schema evolution, backfills, and data quality validation

#LI-KM9 #LI-Remote

Additional Content

Role

We are looking for a Senior Staff Machine Learning Engineer to join our team. This is a remote (USA) role, reporting to the Manager AI Platform and Data Science in the AI Platform and Data Science department. This team’s mission is high-fidelity risk identification in customer data, with goals to catch all threats while minimizing noise. To achieve this, you will develop solutions applying data analysis and threat-research while leveraging AI, machine learning, and data engineering to build quality, data-driven components to automate security analysis.

What you’ll do (Role Expectations)

  • Translate risk identification methods into agent logic, understanding the benefits and limitations of agents and ensuring quality across risk analysis, explanations, and recommendations
  • Collaborate closely with threat-research to understand data, threats, and their approach to developing security heuristics
  • Identify and solve data requirements for analysis, developing and collaborating with data engineering teams for pipelines, enrichments, and aggregations
  • Follow a data-driven quality approach to threat detection, including backtesting, balancing precision vs recall, tuning, and quality control
  • Deploy and monitor your solutions in production within our CI/CD framework

Who You Are (Success Profile)

  • You thrive in ambiguity and build the path as you walk it, seeing unstructured challenges as the raw material to construct meaningful systems.
  • You act like an owner with a deep passion for the mission, operating with integrity and navigating seamlessly between high-level strategy and hands-on execution.
  • You are a continuous learner with a growth mindset who actively seeks feedback to develop yourself, elevate your partners, and execute with purpose.
  • You are a positive force who approaches complex technical challenges with contagious, constructive energy and a focus on solutions.
  • You are data-driven, using analytics and concrete evidence over assumptions to find the truth, measure what matters, and guide informed decisions.

What We’re Looking for (Minimum Qualifications)

  • Experience building LLM-powered agents in production: tool and function calling, prompt and context engineering, multi-step orchestration frameworks (LangGraph, LangChain, or equivalent), and evaluation of non-deterministic output against ground truth
  • 8+ years of professional Python development with demonstrated ability to design and maintain production-quality systems with validated inputs, data contracts, and comprehensive unit and integration tests, alongside hands-on expertise with SQL and Python data analytics libraries (e.g., pandas, Polars, NumPy)
  • Demonstrated ownership of production reliability: CI/CD, containerization, structured logging and metrics, observability and alerting, on-call participation, incident debugging in live distributed systems, and familiarity with cloud and infrastructure-as-code in an AWS environment
  • Sr. Staff level ownership and leadership of emergent requirements, architecture, and complex engineering projects

What Will Make You Stand Out (Preferred Qualifications)

  • Experience with production AI or ML systems where cost, latency, and accuracy are competing constraints, including handling model selection, routing, and regression testing
  • Background in cyber threat research, threat modeling, threat hunting, detection engineering, or previous experience in data-driven risk analysis, fraud detection, adversary profiling and targeting, actuarial risk, or close collaboration with risk analysis teams
  • Data engineering experience building and operating pipelines over large-volume event data using SQL, columnar or search-backed stores (OpenSearch/Elasticsearch, Athena/Presto), schema evolution, backfills, and data quality validation

#LI-KM9 #LI-Remote