This project develops an offline reinforcement learning framework for optimizing speed guidance of connected autonomous vehicles (CAVs) at signalized intersections under mixed traffic conditions. The framework learns safe and efficient policies from real-world trajectory data without requiring risky online interactions. The system employs Critic Regularized Regression (CRR) combined with real-world driving trajectories and Signal Phase and Timing (SPaT) information from the UCF-SST-CitySim dataset. This approach bridges the sim-to-real gap by learning directly from naturalistic human driving behavior while incorporating safety-critical features often neglected in prior work.