8 Revisiting Research Questions
After discussing the findings from the LongSAL study in detail in the previous chapters, we now revisit the research questions introduced in Chapter 4. The chapter presents a discussion of the implications of the findings of the study, in light of the research questions proposed.
8.1 RQ1: Individual Differences and Longitudinal Search Behaviour
How do (changing) individual differences of students affect their longitudinal information search behaviour?
The study found that there were (often significant) differences in information search behaviour between student groups who rated high versus low on individual traits such as motivation, metacognition, self-regulation, and memory-span.
Metacognition and self-regulation are closely related concepts that involve thinking about and controlling one’s own cognitive processes and learning strategies (Ambrose et al., 2010; Garcı́a et al., 2023). They can help students plan, monitor and evaluate their information searching and learning goals, and increase motivation and engagement (Williamson, 2015). Metacognition and self-regulation can improve students’ ability to learn independently and overcome barriers to learning (Winne & Azevedo, 2022).
In the context of search as learning, metacognition can affect information searching behaviour in several ways. Metacognition help students develop effective search queries, compare and evaluate the information obtained, identify relevant websites and sources, and reformulate queries as needed. Students with higher levels of metacognition monitor and regulate their own search process, such as setting goals, planning strategies, checking progress and reflecting on outcomes. Metacognition also help students improve their search performance and learning outcomes by enhancing their self-awareness, confidence and motivation (Reisoğlu et al., 2020; Zhou & Lam, 2019).
A. Cole & O’Brien (2023) found that metacognitive strategies shift students’ thinking away from search as a routine task towards reflections on their learning, as well as how they might apply their learning to future tasks. One participant in their study commented that metacognitive nudges made the participant think more about what they were doing, instead of just aimlessly searching and reading a sentence or two. Metacognition helped to give purpose to their activities, in the form of asking questions like why they were searching for a certain piece of information, and whether they would be able to talk about what they found later in group discussion settings (A. Cole & O’Brien, 2023).
In our study, we found that students in the high group, with higher values of metacognition, self-regulation, and motivation, demonstrated more efficient search behaviours with time. The high group were able to better refine their search strategies and become more efficient as the semester progressed, resulting in fewer clicks per query when writing the final paper. They had higher counts of word substitutions and lower counts of repeat queries, which are both indicative of more focused and targeted search strategies. The randomness of their search behaviour decreased, indicating they became more efficient and strategic in their search processes. They engaged more with content pages than search results, and reported overall better levels of learning and search outcomes. In contrast, the low group, with lower levels of motivation, metacognition, self-regulation, and memory-span, showed less efficient search behaviours. They showed signs of struggling, when moving across different stages of the research paper, and more clicks per query when writing the final paper. They had more randomness and higher entropies of search tactics, more engagement with search results than content pages at later stages of the semester (which was done in earlier stages by the high group), and a general lower level of learning and search outcome.
As the high group progressed through the semester, their search behaviour gradually changed as described by established models of information seeking behaviour, such as Vakkari’s searching-as-learning model (Vakkari, 2016) and Kuhlthau’s Information Seeking Model (Kuhlthau, 1991). Vakkari outlines three stages in the search process: (1) assimilation, (2) restructuring, and (3) tuning. According to the model’s predictions, in the initial stages, students acquire knowledge through assimilation. Bogers & Kaya (2021) reports that searchers in this stage perform some basic `quick-and-dirty’ searching in the beginning to get to know the space of available resources. As students progressed in their search task, they increasingly demonstrate more deliberate and strategic querying behaviour. This implies students progressively enhanced their search efficiency across the session, which has also been reported by other studies (Bogers & Kaya, 2021; Kaya & Bogers, 2023) This transition from divergent to convergent behaviour is also in line with Kuhlthau’s information seeking model(Kuhlthau, 1991).
These findings also complement prior work on identifying struggling search behaviour. For instance, Hassan et al. (2014) investigates search behaviours of users to identify signs of struggling versus exploring. Their findings indicated that certain predictors, such as minimal similarity between consecutive queries, increased clicks per query, as well as differences in the nature of query reformulation patterns (i.e., less query term substitution and more addition/removal with exploring), were indicative of struggling search sessions.
It is important to note that the use of scholarly publications may not necessarily guarantee better quality information or higher self-perceived learning outcomes. The relevance and credibility of the sources used, as well as the ability to critically evaluate and synthesize the information, are important factors that can impact the effectiveness of the search and learning outcomes.
The quality of information is not solely determined by the type of source, but rather by the relevance, credibility, and reliability of the information found. While scholarly publications may provide more specialized and in-depth information, they may not always be the most relevant or up-to-date sources for a given research topic. In these situations, web search results may provide a broader range of sources, including news articles, blogs, and websites, that can offer alternative perspectives and insights on the research topic. However, the quality and reliability of the information found in these sources may vary, and it is important to critically evaluate and verify the information before using it in a research paper.
In addition, sensemaking – the ability to synthesize and integrate information from different sources, regardless of their type – is a key skill in conducting research and writing a research paper. This involves the ability to critically evaluate the relevance and credibility of the sources, as well as to identify and articulate the relationships between different pieces of information.
Overall, the choice of sources and the search strategies used should be based on the research question, the scope of the project, and the specific information needs of the researcher. Both scholarly publications and web search results can provide valuable information, and the key is to use them effectively and efficiently in order to achieve the desired learning outcomes.
8.2 RQ2: Repeated vs Non-repeated Search Tasks
What are the similarities and differences in information search behaviours for tasks where the learning goals are new (non-repeated search tasks), versus those where the learning goals are repeated (repeated search tasks)?
Non-repeated search tasks are those where the learning goals are new and require exploration and discovery of new information. Repeated search tasks are those where the learning goals are repeated and require reinforcement and retrieval of existing information. The findings from this study suggest that there are similarities and differences in information search behaviours for tasks where the learning goals are new versus those where the learning goals are repeated.
Regarding similarities, both the high and low groups showed a decrease in query reformulation types, and count of clicks per query across the semester, regardless of whether the tasks were repeated or non-repeated. However, there were some differences as well. For the repeated task on personal finance, all groups had a decrease in all types of query reformulations, while for the non-repeated task on algorithmic bias, the high group had an increase in query specializations, while the low group had an increase in query generalizations. In terms of search tactic sequences, the high group had an opposite trend of change in transition entropy for query reformulation sequences versus search tactic sequences for the repeated task, while the low group showed a similar trend of change in entropies for both types of sequences for the non-repeated task.
These findings complement and extend prior work that has linked topic familiarity expertise with search strategies and tactics, in influencing search behaviour. Task familiarity and intention impacts the types of query reformulations and search tactics used (Rha et al., 2016), as well as the entropy of search sequences (He et al., 2016). For instance, Qu et al. (2010) found that task type and familiarity influenced search behaviours, such as completion time and query count, but not habitual behaviours, such as the search entrance. Li (2008) reported that users’ topic familiarity and task experience affected their task performance, which could lead to higher searching efficiency and effectiveness. Finally, Karimi et al. (2011) reported that topic familiarity affected query formulation strategies, such as query length, query count, use of Boolean operators and use of quotation marks.
8.3 RQ3: Searching Behaviour and Learning Outcomes
How do (longitudinal) information search behaviour of students relate to their (self-perceived) learning outcomes?
The study found that the participant group with higher levels of metacognition, motivation, and self-regulation demonstrated significantly higher perceived learning outcomes and search outcomes compared to the participant group which were low on these individual traits. This also affected their longitudinal information search behaviour.
These findings are also in line with prior research, that has linked search behaviour with learning outcomes. For instance, Weber et al. (2019) examined a large sample of German students from all academic fields in a two-phase longitudinal study, and found that advanced levels of search behaviour and search tactics predicted better grades (Zlatkin-Troitschanskaia et al., 2021).
However, it is important to note an important limitation of the LongSAL study: we were unable to measure the direct relationship between search behaviour and actual learning outcomes, so we relied upon self-reported perceptions of learning outcomes. Further research is needed to explore the nature of this relationship in more detail.