What Leads to a Confirmatory or Disconfirmatory Behavior of Software Testers?

Creative Commons License

Salman I., Rodriguez P., Turhan B., Tosun Kühn A., Gureller A.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, vol.48, no.4, pp.1351-1368, 2022 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 48 Issue: 4
  • Publication Date: 2022
  • Doi Number: 10.1109/tse.2020.3019892
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index (SSCI), Scopus, Academic Search Premier, PASCAL, ABI/INFORM, Aerospace Database, Applied Science & Technology Source, Business Source Elite, Business Source Premier, Communication Abstracts, Compendex, Computer & Applied Sciences, EMBASE, INSPEC, MEDLINE, Metadex, zbMATH, Civil Engineering Abstracts
  • Page Numbers: pp.1351-1368
  • Keywords: Software, Software testing, Interviews, Electronic mail, Companies, Decision making, Software testing, cognitive biases, confirmation bias, grounded theory, interviews, BIAS, INTERVIEWS, SATURATION
  • Istanbul Technical University Affiliated: Yes


Background: The existing literature in software engineering reports adverse effects of confirmation bias on software testing. Confirmation bias among software testers leads to confirmatory behavior, which is designing or executing relatively more specification consistent test cases (confirmatory behavior) than specification inconsistent test cases (disconfirmatory behavior). Objective: We aim to explore the antecedents to confirmatory and disconfirmatory behavior of software testers. Furthermore, we aim to understand why and how those antecedents lead to (dis)confirmatory behavior. Method: We follow grounded theory method for the analyses of the data collected through semi-structured interviews with twelve software testers. Results: We identified twenty antecedents to (dis)confirmatory behavior, and classified them in nine categories. Experience and Time are the two major categories. Experience is a disconfirmatory category, which also determines which behavior (confirmatory or disconfirmatory) occurs first among software testers, as an effect of other antecedents. Time Pressure is a confirmatory antecedent of the Time category. It also contributes to the confirmatory effects of antecedents of other categories. Conclusion: The disconfirmatory antecedents, especially that belong to the testing process, e.g., test suite reviews by project team members, may help circumvent the deleterious effects of confirmation bias in software testing. If a team's resources permit, the designing and execution of a test suite could be divided among the test team members, as different perspectives of testers may help to detect more errors. The results of our study are based on a single context where dedicated testing teams focus on higher levels of testing. The study's scope does not account for the testing performed by developers. Future work includes exploring other contexts to extend our results.