Automatic Differential Testing of the PHP Interpreter

Abstract

The PHP interpreter, powering over 70% of websites on the internet, plays a crucial role in web development. Existing approaches to finding bugs in PHP primarily focus on detecting explicit security issues through crashes or sanitizer-based oracles, but fail to identify logic bugs that silently lead to incorrect results. We observe that the introduction of Just-In-Time (JIT) compilation mode in PHP presents an opportunity for differential testing, as it provides an alternative implementation of the same language specification. To leverage this opportunity, we propose DiffPHP, an automatic differential testing framework that efficiently detects logic bugs in the PHP interpreter by comparing JIT and non-JIT execution results. Our differential testing incorporates two key techniques: program state instrumentation for fine-grained execution state comparison, and dual verification to handle non-deterministic behaviors in PHP programs. Combined with a state-of-theart test case generation, these two techniques significantly reduce false alarms while maintaining high bug detection capability. Experimental results demonstrate that DiffPHP outperforms the official test suite used in PHP’s continuous integration, achieving higher code coverage and executing more Zend opcodes. Through ablation studies, we validate the effectiveness of both program state instrumentation and dual verification components. To date, DiffPHP has identified 35 previously unknown logic bugs in the PHP interpreter, with 25 already fixed and 5 confirmed by PHP developers. DiffPHP has been acknowledged by the PHP developers, and offers a practical tool for automatically discovering logic bugs in the PHP interpreter.