Skip to main content

LongSAL: A Longitudinal Search as Learning Study With University Students

7 Results and Discussion: Repeated vs Non-repeated Search Tasks

Apart from the longitudinal phase of the study discussed in the previous chapter, participants also completed two search tasks at the beginning of the semester (PHASE1), and two search tasks at the end of the semester (PHASE3). One set of tasks, on the topic of “personal finance for college students”, was repeated at the end of the semester (repeated search tasks). The other set of tasks was not repeated (non-repeated search tasks), and the topics were “ubuntu ethics” in the beginning of the semester, and “algorithmic bias” at the end of the semester. These topics for the non-repeated search tasks were selected from the content of the I303 course that the participants were enrolled in.

The initial and final phases of the study aimed to compare the information search behaviours of students across repeated and non-repeated search tasks, at the beginning and end of the semester. This chapter presents the results of the analysis, which involved comparing the entropies of search tactics and query reformulation sequences, clicks per query, and interaction with different categories of webpages. Let us now discuss some of the findings for these repeated versus non-repeated tasks We use the same low-high groups from the LPA analysis discussed in Section 6.3.

7.1 Learning Outcomes

Learning and Search Outcomes – Longitudinal Phase.

Figure 7.1: Learning and Search Outcomes – Longitudinal Phase.

Figure 7.1 shows the differences in perceived learning outcomes and perceived search outcomes for the two sets of tasks for the low and high groups. The high group reported higher learning and search outcomes for both repeated and non-repeated tasks, at both the beginning and the end of the semester. For the low group, their perceived learning and search outcomes decreased for all the tasks at the end of the semester. The differences between the groups were statistically significant for the repeated task (on the topic of Personal Finance) – \((U = 26.5, p = .02, ES = 0.20)\) for perceived learning outcome, and \((U = 10.5, p = .001, ES = 0.08)\) for perceived search outcome.

There could be several factors contributing to the differences in perceived learning and search outcomes between the two groups. One possible explanation is that the high group had better information-seeking strategies and habits that allowed them to more effectively find and evaluate relevant information, leading to higher perceived learning outcomes. As discussed in the longitudinal phase findings, the high group engaged more with web search results and content pages, while also spending less time on search results and academic publications. This suggests that they were able to efficiently navigate through search results and find the most relevant information. Another possible explanation is that the high group had better prior knowledge and familiarity with the topics, which allowed them to more quickly and effectively identify relevant information. Additionally, it is possible that the high group had more motivation and interest in the topics, leading to more engagement and effort in the search process, and ultimately higher perceived learning outcomes.

For the low group, the decrease in perceived learning and search outcomes at the end of the semester could be due to a variety of factors, such as burnout or fatigue from the research paper writing process, or a lack of engagement or motivation. The decrease in their search tactics entropy and dwell time on academic publications and search results over the course of the semester suggests that they may have become more narrow in their search strategies and less exploratory, which could have limited their ability to find and evaluate relevant information.

7.2 Q: Query Formulations

7.2.2 Length and Count of Queries per Search Task

Query Lengths and Counts – Repeated vs Non-repeated Tasks.

Figure 7.2: Query Lengths and Counts – Repeated vs Non-repeated Tasks.

Figure 7.2 shows the differences in total query length (a, b), average query length (c, d), and count of queries (e, f) for the two sets of tasks for the low and high groups. The high group had a decrease in query count, total query length, and average query length, for all the tasks, from the beginning of the semester to the end of the semester. The low group had a decrease in query count as well, but their average query length increased for both the repeated and the non-repeated tasks. For the non-repeated tasks, the Algorithmic Bias task at the end of the semester was based on course content that students learnt during the semester. So ideally they would not have needed to search for new information if they felt confident of their own knowledge.

The fact that the Algorithmic Bias task was based on course content could have influenced the search behaviour of the participants. As we saw in Section 7.2.1, it is possible that the high group felt more confident in their understanding of the topic. This was also indicated by their higher perceived learning outcome, which led them to issue fewer queries of shorter length. On the other hand, the low group may have felt less confident with their knowledge (and presumably, search abilities), and still felt the need to explore and search for new information, leading them to issue longer queries on average at the end of the semester, in an effort to find the information they needed.

7.2.3 Query Reformulation Types (QRTs)

Number of Query Reformulation Types (QRTs) – Repeated vs Non-repeated Tasks.

Figure 7.3: Number of Query Reformulation Types (QRTs) – Repeated vs Non-repeated Tasks.

Figure 7.3 shows the differences in the five categories of query reformulations (QRTs) for the two sets of tasks, for the low and high groups. Both the groups showed a decrease in all query reformulation types from the start of the semester to the end of the semester, for both the task categories, except for certain specific cases. For the non-repeated task, the high group had an increase in query specializations (Figure 7.3 (f)), while the low group had an increase in query generalizations (Figure 7.3 (d)).

These findings suggest that both groups improved their search skills over the course of the semester, as they needed to perform fewer query reformulations to find relevant information. However, the high group demonstrated a more nuanced approach to query reformulation, as evidenced by their increase in query specializations for the algorithmic bias task. This may suggest that the high group was able to better hone in on specific information needs and tailor their queries accordingly. On the other hand, the low group’s increase in query generalizations for the same task may suggest a more broad and less focused approach to information seeking.

For the repeated task on personal finance, the decrease in all types of query reformulations for all groups indicates that participants had already acquired a baseline knowledge on the topic from the first round of searching earlier in the semester, and were able to more efficiently retrieve relevant information with fewer queries and reformulations.

7.2.4 Entropy of Query Reformulation Types

Entropy of Query Reformulation Types (QRTs) – Repeated vs Non-repeated Tasks.

Figure 7.4: Entropy of Query Reformulation Types (QRTs) – Repeated vs Non-repeated Tasks.

Figure 7.4 shows the differences in the entropies of query reformulation sequences for the low and high groups, across the two sets of tasks. For the repeated tasks on personal finance, both the groups showed an increase in transition entropy of query reformulations from the beginning of the semester to the end. Regarding the stationary entropy of query reformulations, it increased for the high group across the semester, and decreased for the low group. Coming to the non-repeated tasks of Ubuntu Ethics and Algorithmic bias, the high group had a decrease in transition entropy and increase in stationary entropy. The low group showed increase in both stationary and transition entropy.

The increase in transition entropy of query reformulations for both groups in the repeated task on personal finance suggests that participants were exploring different avenues and perspectives as they continued their search. This could be attributed to the fact that they were trying to find alternate information than what they had previously encountered in the beginning of the semester. The increase in stationary entropy of query reformulations for the high group across the semester indicates that their search queries became more diverse and exploratory as they gained a deeper understanding of the topic. On the other hand, the decrease in stationary entropy for the low group suggests that they became more focused and narrowed down their search queries as they progressed through the semester.

For the non-repeated tasks, the decrease in transition entropy and increase in stationary entropy for the high group on Ubuntu Ethics and Algorithmic bias indicate that they were able to refine their search queries and find more relevant information as they gained a deeper understanding of the topics. On the other hand, the increase in both stationary and transition entropy for the low group suggests that they struggled with finding relevant information and had to explore more avenues to refine their search queries.

7.3 Number of Clicks per Query

Number of Clicks Per Query – Repeated vs Non-repeated Tasks.

Figure 7.5: Number of Clicks Per Query – Repeated vs Non-repeated Tasks.

Figure 7.5 shows the differences in total, average, and variability of clicks per for the low and high groups, across the two sets of tasks. Across the board, all the groups had a decrease in the count of clicks per query for all the tasks, from the beginning to the end of the semester.

The decrease in clicks per query for both groups over time could be due to an increase in search experience and familiarity with the topics, as well as improved search strategies and tactics developed over the course of the semester. As participants became more familiar with the topics, they likely needed to click on fewer search results to find the information they were looking for. This is because they may have developed a better understanding of what search terms to use, what types of sources to look for, and how to evaluate the relevance and reliability of search results. This increased familiarity and experience likely contributed to the decrease in clicks per query observed in both groups over time. Additionally, as participants developed more effective search strategies and tactics over the course of the semester, they may have been able to find the information they needed more quickly, resulting in a decrease in the count of clicks per query.

7.4 L vs I: Interaction with Search Results vs Content Pages

Differences in interactions with search results vs. content pages - repeated and non-repeated tasks.

Figure 7.6: Differences in interactions with search results vs. content pages - repeated and non-repeated tasks.

Figure 7.6 shows the differences in interactions with search results and interaction with content pages for the low and high groups, across the two sets of tasks. Similar to the number of clicks per query, there was an overall decrease from semester beginning to semester end, for number of pages visited, total dwell time, and average dwell time, across all types of webpages – search results and content pages. This could be due to an increase in search experience and familiarity with the topics, as well as improved search strategies and tactics developed over the course of the semester. As students become more familiar with the topics and develop better strategies for searching, they may have been able to quickly identify and access the information they needed, leading to a decrease in dwell time, and the number of pages visited. These decreases may also be related to the fact that students may have become more efficient in their search process over time. They may have learned to quickly identify and assess the relevance of search results and content pages, leading to a more targeted approach in their search process.

7.5 Entropy of Search Tactic Sequences

Entropy of Search Tactics – Repeated vs Non-repeated Tasks.

Figure 7.7: Entropy of Search Tactics – Repeated vs Non-repeated Tasks.

Figure 7.7 shows the differences in the entropies of search tactics for the low and high groups, across the two sets of tasks. Across the semester, the high group had a decrease in both transition entropy and stationary entropy for the repeated task, while the low group had an increase in transition entropy, but decrease in stationary entropy. For the non-repeated task, both the groups had an increase in transition entropy, and decrease in stationary entropy.

The decrease in transition entropy and stationary entropy for the high group on the repeated tasks indicate that they were able to develop more focused and efficient search strategies over time, leading to more predictable patterns in their search tactic sequences. On the other hand, the increase in transition entropy for the low group on the repeated task suggests that they may have struggled to find effective search strategies, resulting in more varied and unpredictable sequences of search tactics. However, the decrease in stationary entropy for the low group suggests that they were still able to find and stick to effective search tactics over time. For the non-repeated tasks, the increase in transition entropy for both groups may indicate that they were exploring a wider range of search tactics and approaches to find relevant information. The decrease in stationary entropy suggests that they were still able to identify and use effective search tactics, but that they were more flexible in their approach overall.

Comparing between the entropies for query reformulation sequences (Figure 7.4) and entropies of search tactic sequences (Figure 7.7), it is interesting to note that the trends of change in these two classes of entropy values were opposite for the high group, but was similar for the low group. The high group had an increase in the transition entropy of query reformulations, but decrease in the transition entropy of search sequences, and so on. The high group showed an increase in transition entropy of query reformulations – meaning that they were exploring a wider range of query reformulation types over time – but they showed a decrease in transition entropy of search tactic sequences – indicating that they were relying on a smaller set of search tactics as they became more familiar with the topics. Conversely, the low group showed similar trends for both types of entropies. They had an increase in both transition entropy of query reformulations and transition entropy of search tactic sequences. This suggests that the low group was still exploring different types of search strategies and query reformulation types over time, perhaps due to a lack of confidence in their own knowledge or search skills.

7.6 Summary

We summarize the findings from this chapter, on repeated vs. non-repeated search tasks across the semester as follows.

Both the low and high groups demonstrated a general decrease in query behaviour from the start to the end of the semester (The high group showed an increase in query specializations for the non-repeated task, and the low group showing an increase in query generalizations for the same task). Search tactics became more random for the high group on the repeated task, and less random on the non-repeated task. For the low group, their tactics became more random regardless of the task type.

Regarding learning and search outcomes, the high group reported higher learning and search outcomes for both repeated and non-repeated tasks at both the beginning and end of the semester. For the low group, their perceived learning and search outcomes decreased for all tasks at the end of the semester. The qualitative responses from participants for needing to search again at the end of the semester indicated that the high group had a stronger sense of confidence and perceived understanding of the subject-matter, compared to the low group.

The findings from the initial and final phase of the study suggest that there are differences in information search behaviours for repeated and non-repeated search tasks. Overall, the high group demonstrated more effective and efficient search behaviour, with better search outcomes and lower entropy values, while the low group struggled with search behaviour and outcomes, showing increases in entropy values and decreased confidence in their knowledge. This indicates that repeated engagements with search task topics may have improved the information search behaviours for the group with higher values of motivation, metacognition, self-regulation, and memory span.