Skip to main content

LongSAL: A Longitudinal Search as Learning Study With University Students

B QSNR: Questionnaires

B.1 QSNR0: Recruitment Questionnaire

Thank you SO much for your willingness to participate in the LongSAL research study. The aim of this study is to identify how search engines can better support the needs of university students’ learning and education. To be eligible for this study, you must be enrolled in the I 303 Ethical Foundations for Informatics course for the Spring 2022 semester. Please fill out the information requested below. We will get back to you if you are selected to participate in the study. The principal investigator, Nilavra Bhattacharya, can be reached at <email-address> for any questions or concerns.

  1. Please select which section of the I-303 Ethical Foundations for Informatics course you are enrolled in.

    • TUE: FLEISCHMANN, VERMA
    • WED: FLEISCHMANN, GURSOY
    • THU: FLEISCHMANN, BAUTISTA
    • FRI: FLEISCHMANN, DAY

  2. Please select the degree level/name of the program you are in.

    • Bachelor’s
    • Master’s
    • Integrated Bachelor’s and Master’s
    • PhD
    • Other …..

  3. Please state which year of the program you are in.

    • Freshman
    • Sophomore
    • Junior
    • Senior
    • Graduate Year 1
    • Graduate Year 2
    • Other …..

  4. Please state your major(s) …..

  5. Do you have native-level familiarity with English language?

    • Yes
    • No
    • Other: …..

  6. Please state your age (in years) …..

  7. Please state your gender …..

  8. With which ethnicities do you identify? Please select all that apply:

    • African
    • African American / Black
    • Asian - East
    • Asian - South East
    • Asian - South
    • Asian - Middle East
    • Caucasian / White
    • Hispanic / Latinx
    • Native American
    • Pacific Islander
    • Mixed
    • Other …..

  9. Are you an international student? If “yes”, where are you originally from?

    • Yes …..
    • No

  10. We need your contact information to communicate with you over the semester (if you are selected). Your contact information will not be used in any other way, and will be kept private. Please enter an email address that you check regularly. We will use this email address to send communications and Amazon Gift Cards as payment. …..

  11. Your name as you would like us to address you (solely for communication). …..

B.2 QSNR1 - QSNR3: Entry, Mid-term and Exit Questionnaires

B.3 Motivation

Adapted from Intrinsic Motivation Inventory (IMI) (Ryan, 1982). Items will be randomly ordered.

Scoring directions: Score each response from 1 (not at all true) to 5 (very true). Then reverse score the items marked with (R). To do that, subtract the item response from 6, and use the resulting number as the item score. Then, calculate subscale scores by averaging across all the items on that subscale. The subscale scores are then used in the analyses of relevant research questions.

For each of the following statements, please indicate how true it is for you, using the following scale:
(1) not at all true — somewhat true — very true (5)

B.3.1 Interest/Enjoyment

  1. I will enjoy taking this course very much.
  2. This course will be fun to do.
  3. I think this will be a boring course. (R)
  4. This course will not hold my attention at all. (R)
  5. I would describe this course as very interesting.
  6. I think this course will be quite enjoyable.

B.3.2 Perceived Competence

  1. I think I will be pretty good at this course.
  2. I think I will be doing pretty well at this course, compared to other students.
  3. After working at this course for awhile, I will feel pretty competent.
  4. I think I will be satisfied with my performance in this course.
  5. I think I am pretty skilled at this course.
  6. This is a course that I think would not be able to do very well. (R)

B.3.3 Effort/Importance

  1. I plan to put a lot of effort into this course.
  2. I don’t think I will try very hard to do well at this course. (R)
  3. I will try very hard on this course.
  4. It is important to me to do well in this course.
  5. I do not plan to put much energy into this course. (R)

B.3.4 Value/Usefulness

  1. I believe the course and the final project activities could be of some value to me.
  2. I think that doing the final project activities is useful for me.
  3. I think the final project is important activity to do because it can equip me with skills that are necessary for making ethical decisions in my adult and professional life.
  4. I would be willing to do research on the final project topic again because it has some value to me.
  5. I think doing the final project activities will help me in my adult and professional life
  6. I believe doing the final project activities will be beneficial to me.
  7. I think this is an important course.

B.3.5 Pressure / Tension (not in QSNR1)

  1. I do not feel nervous while doing the final project activities. (R)
  2. I feel very tensed while doing the final project activities.
  3. I am very relaxed while doing the final project activities. (R)
  4. I feel anxious while working on the final project parts.
  5. I feel pressured while doing the final project activities.

B.3.6 Perceived Choice (not in QSNR1)

  1. I believe I have some choice about doing the final project activities.
  2. I feel like it is not my own choice to do the final project parts. (R)
  3. I don’t really have a choice about doing the final project tasks. (R)
  4. I feel like I have to do the final project tasks. (R)
  5. I do the final project activities because I have no choice. (R)
  6. I do the final project activities because I want to.
  7. I do the final project activities because I have to. (R)

B.4 Self-regulation

Self-Regulation Questionnaire (SRQ) by J. M. Brown et al. (1999).

Please answer the following questions by selecting the option that best describes how you are. There are no right or wrong answers. Work quickly and don’t think too long about your answers.

(1) Strongly Disagree – Disagree – Neutral – Agree – Strongly Agree (5)

  1. I usually keep track of my progress toward my goals.
  2. My behavior is not that different from other people’s. (R)
  3. Others tell me that I keep on with things too long. (R)
  4. I doubt I could change even if I wanted to. (R)
  5. I have trouble making up my mind about things. (R)
  6. I get easily distracted from my plans. (R)
  7. I reward myself for progress toward my goals.
  8. I don’t notice the effects of my actions until it’s too late. (R)
  9. My behavior is similar to that of my friends. Evaluating
  10. It’s hard for me to see anything helpful about changing my ways. (R)
  11. I am able to accomplish goals I set for myself.
  12. I put off making decisions. (R)
  13. I have so many plans that it’s hard for me to focus on any one of them. (R)
  14. I change the way I do things when I see a problem with how things are going.
  15. It’s hard for me to notice when I’ve “had enough” (alcohol, food, sweets, internet, social media) (R)
  16. I think a lot about what other people think of me.
  17. I am willing to consider other ways of doing things.
  18. If I wanted to change, I am confident that I could do it.
  19. When it comes to deciding about a change, I feel overwhelmed by the choices. (R)
  20. I have trouble following through with things once I’ve made up my mind to do something. (R)
  21. I don’t seem to learn from my mistakes. (R)
  22. I’m usually careful not to overdo it when working, eating, drinking, or being on social media.
  23. I tend to compare myself with other people.
  24. I enjoy a routine, and like things to stay the same. (R)
  25. I have sought out advice or information about changing.
  26. I can come up with lots of ways to change, but it’s hard for me to decide which one to use. (R)
  27. I can stick to a plan that’s working well.
  28. I usually only have to make a mistake one time in order to learn from it.
  29. I don’t learn well from punishment. (R)
  30. I have personal standards, and try to live up to them.
  31. I am set in my ways. (R)
  32. As soon as I see a problem or challenge, I start looking for possible solutions.
  33. I have a hard time setting goals for myself. (R)
  34. I have a lot of willpower.
  35. When I’m trying to change something, I pay a lot of attention to how I’m doing.
  36. I usually judge what I’m doing by the consequences of my actions.
  37. I don’t care if I’m different from most people. (R)
  38. As soon as I see things aren’t going right I want to do something about it.
  39. There is usually more than one way to accomplish something.
  40. I have trouble making plans to help me reach my goals. (R)
  41. I am able to resist temptation.
  42. I set goals for myself and keep track of my progress.
  43. Most of the time I don’t pay attention to what I’m doing. (R)
  44. I try to be like people around me.
  45. I tend to keep doing the same thing, even when it doesn’t work. (R)
  46. I can usually find several different possibilities when I want to change something.
  47. Once I have a goal, I can usually plan how to reach it.
  48. I have rules that I stick by no matter what.
  49. If I make a resolution to change something, I pay a lot of attention to how I’m doing.
  50. Often I don’t notice what I’m doing until someone calls it to my attention. (R)
  51. I think a lot about how I’m doing.
  52. Usually I see the need to change before others do.
  53. I’m good at finding different ways to get what I want.
  54. I usually think before I act.
  55. Little problems or distractions throw me off course. (R)
  56. I feel bad when I don’t meet my goals.
  57. I learn from my mistakes.
  58. I know how I want to be.
  59. It bothers me when things aren’t the way I want them.
  60. I call in others for help when I need it.
  61. Before making a decision, I consider what is likely to happen if I do one thing or another.
  62. I give up quickly. (R)
  63. I usually decide to change and hope for the best. (R)

Scoring Directions: Score each response from 1 (strongly disagree) to 5 (strongly agree), and calculate the following seven subscale scores by summing the items on that subscale. Items marked (R) are reverse-coded (i.e. 1 = strongly agree and 5 = strongly disagree). To do that, subtract the item response from 6, and use the resulting number as the item score.

  1. Receiving relevant information: 1, 8, 15, 22, 29, 36, 43, 50, 57
  2. Evaluating the information and comparing it to norms: 2, 9, 16, 23, 30, 37, 44, 51, 58
  3. Triggering change: 3, 10, 17, 24, 31, 38, 45, 52, 59
  4. Searching for options: 4, 11, 18, 25, 32, 39, 46, 53, 60
  5. Formulating a plan: 5, 12, 19, 26, 33, 40, 47, 54, 61
  6. Implementing the plan: 6, 13, 20, 27, 34, 41, 48, 55, 62
  7. Assessing the plan’s effectiveness: 7, 14, 21, 28, 35, 42, 49, 56, 63

Based on our clinical and college samples, we tentatively recommend the following ranges for interpreting SRQ total scores with the 63-item scale:

  • >= 239: High (intact) self-regulation capacity (top quartile)
  • 214 - 238: Intermediate (moderate) self-regulation capacity (middle quartiles)
  • <= 213: Low (impaired) self-regulation capacity (bottom quartile)

B.5 Metacognition

Metacognitive Awareness Inventory (MAI) proposed by Schraw & Dennison (1994) and revised by Terlecki & McMahon (2018).

Think of yourself as a learner. Read each statement carefully, and rate it as it generally applies to you when you are in the role of a learner (student, attending classes, university etc.) Please indicate how true each reason is for you using the following scale:

Score Response
1 I NEVER do this
2 I do this infrequently
3 I do this inconsistently
4 I do this frequently
5 I ALWAYS do this
  1. I ask myself periodically if I am meeting my goals.
  2. I consider several alternatives to a problem before I answer.
  3. I try to use strategies that have worked in the past.
  4. I pace myself while learning in order to have enough time.
  5. I understand my intellectual strengths and weaknesses.
  6. I think about what I really need to learn before I begin a task.
  7. I know how well I did once I finish a test.
  8. I set specific goals before I begin a task.
  9. I slow down when I encounter important information.
  10. I know what kind of information is most important to learn.
  11. I ask myself if I have considered all options when solving a problem.
  12. I am good at organizing information.
  13. I consciously focus my attention on important information.
  14. I have a specific purpose for each strategy I use.
  15. I learn best when I know something about the topic.
  16. I know what the teacher expects me to learn.
  17. I am good at remembering information.
  18. I use different learning strategies depending on the situation.
  19. I ask myself if there was an easier way to do things after I finish a task.
  20. I have control over how well I learn.
  21. I periodically review to help me understand important relationships.
  22. I ask myself questions about the material before I begin.
  23. I think of several ways to solve a problem and choose the best one.
  24. I summarize what I’ve learned after I finish.
  25. I ask others for help when I don’t understand something.
  26. I can motivate myself to learn when I need to.
  27. I am aware of what strategies I use when I study.
  28. I find myself analyzing the usefulness of strategies while I study.
  29. I use my intellectual strengths to compensate for my weaknesses.
  30. I focus on the meaning and significance of new information.
  31. I create my own examples to make information more meaningful.
  32. I am a good judge of how well I understand something.
  33. I find myself using helpful learning strategies automatically.
  34. I find myself pausing regularly to check my comprehension.
  35. I know when each strategy I use will be most effective.
  36. I ask myself how well I accomplish my goals once I’m finished.
  37. I draw pictures or diagrams to help me understand while learning.
  38. I ask myself if I have considered all options after I solve a problem.
  39. I try to translate new information into my own words.
  40. I change strategies when I fail to understand.
  41. I use the organizational structure of the text to help me learn.
  42. I read instructions carefully before I begin a task.
  43. I ask myself if what I’m reading is related to what I already know.
  44. I reevaluate my assumptions when I get confused.
  45. I organize my time to best accomplish my goals.
  46. I learn more when I am interested in the topic.
  47. I try to break studying down into smaller steps.
  48. I focus on overall meaning rather than specifics.
  49. I ask myself questions about how well I am doing while I am learning something new.
  50. I ask myself if I learned as much as I could have once I finish a task.
  51. I stop and go back over new information that is not clear.
  52. I stop and reread when I get confused.

Scoring Directions: Score each response from 1 (never) to 5 (always), and calculate the following subscale scores by summing the items on that subscale.

Knowledge about Cognition:

  1. Declarative Knowledge: 5, 10, 12, 16, 17, 20, 32, 46 (score out of \(8\times5 = 40\))
  2. Procedural Knowledge: 3, 14, 27, 33 (score out of \(4\times5 = 20\))
  3. Conditional Knowledge: 15, 18, 26, 29, 35 (score out of \(5\times5 = 25\))

Regulation of Cognition:

  1. Planning: 4, 6, 8, 22, 23, 42, 45 (score out of \(7\times5 = 35\))
  2. Information Management Strategies: 9, 13, 30, 31, 37, 39, 41, 43, 47, 48 (score out of \(10\times5 = 50\))
  3. Comprehension Monitoring: 1, 2, 11, 21, 28, 34, 49 (score out of \(7\times5 = 35\))
  4. Debugging Strategies: 25, 40, 44, 51, 52 (score out of \(5\times5 = 25\))
  5. Evaluation: 7, 19, 24, 36, 38, 50 (score out of \(6\times5 = 30\))

References

Brown, J. M., Miller, W. R., & Lawendowski, L. A. (1999). The self-regulation questionnaire. In V. L. & J. T. L. (Eds.), Innovations in clinical practice: A sourcebook (Vol. 17, pp. 281–292). Professional Resource Press/Professional Resource Exchange.
Ryan, R. M. (1982). Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43(3), 450.
Schraw, G., & Dennison, R. S. (1994). Assessing Metacognitive Awareness. Contemporary Educational Psychology, 19(4), 460–475. https://doi.org/10.1006/ceps.1994.1033
Terlecki, M., & McMahon, A. (2018). A Call for Metacognitive Intervention: Improvements Due to Curricular Programming in Leadership. Journal of Leadership Education, 17(4), 130–145. https://doi.org/10.12806/V17/I4/R8