Investigating Research Practice and Methods in Usable Security and Privacy

Abstract

Usable Security and Privacy (USP) is a fairly young discipline, with unique challenges in conducting user studies because its research is situated at the intersection of Human Computer Interaction (HCI) and IT security. This makes meta-scientific work in this area especially beneficial. There have been individual examples of meta-scientific work in USP for some time but only recently has meta-science explicitly started being included in calls for papers. This work contributes to metascience in USP in three areas: Qualitative research, quantitative research, and replicability. For the first two areas, it also considers the broader area of HCI research. <br/>

For qualitative research, this thesis explores the influence of different qualitative coders and data with differing degrees of complexity on the results of the qualitative analysis process and expectations in the review process, surrounding the number of coders used. The work describes a series of studies with two types of data and coders with different amounts of experience, as well as a survey with reviewers. Results were more similar for simple data than complex, and the results of more experienced researchers were more abstract and focused on the research questions than those of students with less experience. Reviewers had varied opinions on how many coders were appropriate and many expected clear justifications of methods rather than following set rules. <br/>

This work investigates two quantitative topics: The use of power analysis to determine necessary sample size prior to conducting studies, and the reporting, understanding and interpretation of effect sizes. Both topics are situated in the null hypothesis significance testing framework of statistical analysis. A literature analysis in a specialized field (Developer-Centered Usable Security) showed that power analysis is rarely conducted and often, the data necessary to conduct a power analysis based on literature is not reported there. Extending this analysis to HCI consolidates issues with reporting. The resulting data sets can be used to inform future power analyses. <br/>

Regarding effect sizes, the analysis of HCI literature suggests that the size of effects varies between fields. The thesis uses a subset of this sample to derive approaches to interpret effect sizes. Further, interview and survey studies with researchers in USP and HCI were used to identify influence factors on researchers' size and importance judgments of effect sizes. The results from these studies also suggest that there are misconceptions around the investigated effect sizes. <br/>

On the topic of replicability, this work discusses developments in recruitment and analysis but also in the area of study, using the third published iteration of a study on experts' and non-experts' security-related behaviors and advice as an example. <br/>

From the results from these three areas of studies, the thesis draws recommendations on how methods and reporting in USP can be improved.