Siloso: Finding Logic Bugs in RDBMS via Dialect-Adaptable Reference Engine Construction

Abstract

Relational DBMSs (RDBMSs) are ubiquitous, so any bugs or inconsistencies within RDBMSs are highly consequential. Particularly, logic bugs, which can cause an incorrect result to be returned for a given query evaluation, are critical because they are likely to be unnoticed by users. Furthermore, the correctness of RDBMSs can also be undermined by specification inconsistencies, arising from under-documented or undocumented behavior, which can lead to ambiguous results for a given query execution. In our work, we investigated and conceptualized key correctness variabilities across a diverse set of RDBMSs. Based on this insight, we propose an approach for detecting logic bugs and inconsistencies in RDBMSs via the design and implementation of an extensible reference engine, which we term as Siloso, that can adapt the behavior of a query execution depending on the specified dialect. We evaluated Siloso extensively by using it in an automated RDBMS testing setting as a differential test oracle, finding 20 bugs across six RDBMSs.