A Pilot Study
A pilot study was conducted in the Summer 2021 semester at the School of
Information, University of Texas at Austin (Texas iSchool). This was
mainly a feasibility study to determine the technical logistics and
participant retention rates. It included PHASE1, PHASE2, and PHASE3 from the final study procedure (Figure 5.1). There was no recording of individual
differences questionnaires. Eight students from two courses
at the Texas iSchool – Academic Success in the Digital University
(ACS), and Information in Cyberspace (CYB) – participated in the
pilot study. The study ran from start of June 2021 to mid-August 2021.
There was no participant drop-off. Synchronous sessions (SES1 and SES3)
were conducted over the Zoom video conferencing platform. Log data was
captured using the YASBIL browser extension (Bhattacharya & Gwizdka, 2021).
All setup and technical logistics worked out properly, without any major
technological issues. Details of the search task descriptions are
presented below. Participants were compensated with USD 15 for SES1, 15
points of extra course credit for longitudinal tracking in SES2, and USD
15 for SES3.
A.1 Pilot Study Phase 1: Initial Phase
Participants performed a training task to familiarize themselves with the YASBIL browser extension. Then they performed two search tasks as described below. Each search task was followed by measurement of mental workload using NASA-TLX.
Prompt for Task 1: Financial Literacy (Repeated in PHASE3)
Money management and financial literacy are essential life skills, and what better time to learn about them than in college? Write a note to your future self, about essential money-related advice and skills that college students should know and practice.
What to do:
Find at least 3 unique, good quality online resources that are relevant to this topic
Look for resources that help establish connections and develop a narrative
What to deliver:
Write a summary of the lessons, advice, and/or tips you found across the different resources. This is a note to your future self, so the narrative can be in a format that is most useful and interesting to YOU
Paste the links of ALL the resources that you finally selected to develop your narrative, in the second text box, one link per line
Prompt Task 2: Social Media during COVID-19 (Topic was part of course
content in ACS and CYB)
“What was the role of Social Media during the COVID-19 pandemic? How did it affect people’s lives during quarantine and social distancing?” Suppose a family member (say your aunt) or a friend asked you these question over a phone call, and you want to talk to them on this topic for a couple of minutes.
What to do:
Find at least 3 unique, good quality online resources that are relevant to this topic
Look for resources that help establish connections and develop a narrative
What to deliver:
Write a short summary of the content that you found across the different resources. The length and writing style can be such that you can read it out to your family member/friend over a phone call, without them losing interest.
In the summary, briefly mention your thoughts about each resource - do you agree or disagree with the content in the resource? Anything else?
Paste the links of ALL the resources that you finally selected to develop your narrative, in the second text box, one link per line
A.2 Pilot Study Phase 2: Longitudinal Tracking
The longitudinal tracking phase Phase 2 involved student participants submitting log data for two final-project assignments for the ACS course, and four final project assignments for the CYB course. Participants received reminder emails to log and sync their data a few days before each assignment was due. Seven (out of 8) participants logged their data and synced it with our data server in a timely fashion, without major technical issues. One participant CYB course forgot to log their data for the first two assignments, despite the email reminder. However, upon following up with them, they remembered to log their data for the third and fourth sessions.
A.3 Pilot Study Phase 3: Final Phase
All eight participants from Phase 1 completed Phase 3 (no drop off). Participants performed two search tasks.
Prompt for Task 1: Financial Literacy (Repeated from SES1)
At the start of the semester, you wrote a note to your future self, about essential money-related advice and skills that college students should know and practice.
Here is what you wrote:
{dynamic content showing participants’ previous responses}
Here are the resources you took help from:
{dynamic content showing participants’ previous responses}
Now you have a chance to update or revise the note with more information. You can either choose to write afresh, or copy-paste the note from above into the first textbox below and add to it /edit it. Feel free to search the web if you need to, after turning YASBIL on. You can choose NOT to search, as well.
If you do choose to search, please paste the links of ALL the resources that you finally selected for updating your note, one link per line, in the second textbox. The links can be the same ones you visited earlier, or different.
Did you need to search the web for updating the note? Why?
Prompt for Task 2: HTML CSS (Topic was part of course content in ACS and CYB)
In your course, you studied about websites, HTML, and CSS. Therefore, for answering the questions below, you may choose NOT to search the web, if you feel you can answer the questions reasonably well. If you do need to search the web, feel free to do so, after turning on YASBIL.
As you understand these concepts, please explain (with examples if necessary)
what is the purpose of HTML?
what is the purpose of CSS?
how do HTML and CSS come together when someone visits a website?
List as many HTML tags as you can, one per line
List as many CSS properties as you can, one per line.
Did you need to search the web for this task? Why?