September 22, 2025 · 1 min read
Building an RL Evaluation Pipeline
Lessons from designing reproducible RL evaluation infrastructure at scale.
reinforcement-learning
mlops
evaluation
Reproducible RL evaluation is harder than it looks.
The Problem
When running hundreds of RL experiments, we encountered non-determinism, inconsistent logging, and difficulty comparing policies.
Architecture
class RLEvalHarness:
def evaluate(self, policy, config_id: str) -> EvalResult:
results = []
for seed in range(self.n_seeds):
env = self.env_factory(seed=seed)
metrics = self._run_episodes(policy, env)
results.append(metrics)
return EvalResult.aggregate(results, config_id)Best Practices
- Fix evaluation seeds separately from training seeds
- Log raw episode returns, not just aggregates
- Version environment configurations explicitly