Oliver was born in Hong Kong, but grew up in New York City. He graduated from Stanford University with a Bachelor's in Public Policy. As part of his studies, he completed a senior honors thesis examining the impact of test-based student promotion policies on achievement in New York City. After graduation, Oliver returned to New York where he worked at the New York City Department of Education (NYCDOE) for nearly four years. At the NYCDOE, he worked on numerous analytic, data collection and verification projects related to Mayor Bloomberg's Children First Reform efforts. His work focused largely on improving systems for measuring teacher effectiveness and providing meaningful teacher evaluation. Oliver is a die-hard Yankees fan who played baseball, soccer, and basketball in high school as well as four years of rugby at Stanford.