The goal of this study is to develop a ranking of code snippets Self-contained Java methods from real-world open-source projects. based on the difficulty that software developers have in understanding them. This ranking will be used in future research on code comprehension.
To create this ranking, this study will use the Delphi method. Multiple software developers (including you) will first rank a set of code snippets independently. Then, we will provide anonymized feedback summarizing the rankings from other participants to guide a consensus-based final ranking. We estimate that one or two rounds of ranking will be sufficient, depending on the level of agreement among participants.
This is the first round of the study. You will review 8 code snippets and provide your ranking along with brief explanations. The estimated time to complete this round is approximately 60 minutes.
If a second round is needed, you will be invited to participate and it should take you less than 30 minutes. For details, see the What Happens After You Complete the First Round section below.
This round is divided into three phases:
| Phase | Description | Time |
|---|---|---|
| Phase 1 | Review each code snippet and provide a short summary, along with your assessment of how easy or difficulty it is to understand. | 35 mins |
| Phase 2 | Rank the snippets from easiest to hardest to understand, grouping similar snippets together. | 10 mins |
| Phase 3 | Explain your reasoning behind the ranking and the groupings. | 10 mins |
| Submission | Review and submit your answers. You may revise and resubmit them at any time. | 5 mins |
Code Comprehensibility: the difficulty a software developer experiences when trying to understand a code snippet. This is influenced by the clarity of its structure, naming, and logic.
Code Snippet: A self-contained Java method selected from a real-world open-source software project. All 8 snippets in this study are independent of one another and were carefully selected.
Please contact us if you have any questions or issues: ea442@njit.edu