In this paper, a role play scenario experiment of people’s ability to differentiate between phishing and genuine emails demonstrated limitations in the generalisability of phishing studies. This involves issues around the priming of participants and the diversity of emails used. Only half of our 117 participants were explicitly informed that the study was assessing the ability to identify phishing emails. Results indicate that the informed participants were significantly better at discriminating between phishing and genuine emails than the uninformed participants. This has implications for the interpretation of phishing studies. Specifically, studies where participants are directly asked to identify phishing emails may not represent the performance of real world users, because people are rarely reminded about the risks of phishing emails in real life. Our study also used emails from a larger number and greater diversity of industries than previous phishing studies. Results indicate that participants’ performance differs greatly in terms of category (e.g., type of sender) of emails. This demonstrates that caution should be used when interpreting the results of phishing studies that rely on only a small number of emails and/or emails of limited diversity. Hence, when designing and interpreting phishing studies, researchers should carefully consider the instructions provided to participants and the types of emails used.