Publications

Hassan Khosravi

The updated list of my publications can be found on my Google Scholar page

Learnersourcing in the age of AI: Student, educator and machine partnerships for content creation

Journal Paper 2023
Hassan Khosravi and Paul Denny and Steven Moore and John Stamper (2023). In the journal of Computers and Education: Artificial Intelligence
Engaging students in creating novel content, also referred to as learnersourcing, is increasingly recognised as an effective approach to promoting higher-order learning, deeply engaging students with course material and developing large repositories of content suitable for personalised learning. Despite these benefits, some common concerns and criticisms are associated with learnersourcing (e.g., the quality of resources created by students, challenges in incentivising engagement and lack of availability of reliable learnersourcing systems), which have limited its adoption. This paper presents a framework that considers the existing learnersourcing literature, the latest insights from the learning sciences and advances in AI to offer promising future directions for developing learnersourcing systems. The framework is designed around important questions and human-AI partnerships relating to four key aspects: (1) creating novel content, (2) evaluating the quality of the created content, (3) utilising learnersourced contributions of students and (4) enabling instructors to support students in the learnersourcing process. We then present two comprehensive case studies that illustrate the application of the proposed framework in relation to two existing popular learnersourcing systems.
@article{khosravi2023learnersourcing,
      title={Learnersourcing in the age of AI: Student, educator and machine partnerships for content creation},
      author={Khosravi, Hassan and Denny, Paul and Moore, Steven and Stamper, John},
      journal={Computers and Education: Artificial Intelligence},
      year={2023},
      publisher={Elsevier}
    }
    
    


Are Deeper Reflectors Better Goal-Setters? AI-Empowered Analytics of Reflective Writing in Pharmaceutical Education

Journal Paper 2023
Yuheng Li and Mladen Rakovic and Wei Dai and Jionghao Lin and Hassan Khosravi and Kirsten Galbraith and Kayley Lyons and Dragan Gasevic and Guanliang Chen (2023). In the journal of Computers and Education: Artificial Intelligence
Reflection and goal-setting are interrelated processes in well-established educational theories to promote in-depth self-reflection and self-regulated learning. Prior studies have considered reflection to be an important antecedent for meaningful goal-setting. Yet, there lacks empirical evidence to shed light on how students' abilities to reflect inform their abilities to set goals. Hence, in the present study, we aimed to quantify the connection between students' retrospective reflection and their subsequent goal-setting, and derive more in-depth insights to benefit educators in their teaching to promote deeper reflection, more specific goal-setting and better self-regulation. To this end, we utilised two fine-grained coding schemes, adapted from well-established reflection and goal-setting theories, respectively, as well as pertinent prior studies, to annotate the reflective and goal-setting elements within 600 student responses in pharmacy curricula. We visualised such elements as a network graph to study students' joint behavioural patterns in reflecting and setting goals. Then, we statistically analysed the correlation between students' reflective levels and the goal specificities using a Mann Whitney U test. We found that (1) descriptive reflection and goals that included content and actions with additional details more commonly presented jointly; (2) students who reflected deeply tended to set more specific goals. These findings are further summarised and discussed to guide educators to adopt reflective and goal-setting practices when designing teaching activities. Moreover, driven by these findings, we emphasised the significance of aiding instructors to provide timely assessment to students' written reflections so as to further ameliorate students' reflective abilities. Therefore, we attempted to automate such assessments using five traditional machine learning algorithms and one deep learning approach based on Bidirectional Encoder Representation of Transformers (BERT), and discovered that BERT gave the best performance in terms of identifying reflective sentences and differentiating various reflective elements.
@article{li2023deeper,
      title={Are deeper reflectors better goal-setters? AI-empowered analytics of reflective writing in pharmaceutical education},
      author={Li, Yuheng and Rakovi{\'c}, Mladen and Dai, Wei and Lin, Jionghao and Khosravi, Hassan and Galbraith, Kirsten and Lyons, Kayley and Ga{\v{s}}evi{\'c}, Dragan and Chen, Guanliang},
      journal={Computers and Education: Artificial Intelligence},
      volume={5},
      pages={100157},
      year={2023},
      publisher={Elsevier}
    }
    


Aligning the Goals of Learning Analytics with its Research Scholarship: An Open Peer Commentary Approach

Journal Paper 2023
Rebecca Ferguson and Hassan Khosravi and colleagues (2023). In the Journal of Learning Analytics
Engaging students in creating novel content, also referred to as learnersourcing, is increasingly recognised as an effective approach to promoting higher-order learning, deeply engaging students with course material and developing large repositories of content suitable for personalised learning. Despite these benefits, some common concerns and criticisms are associated with learnersourcing (e.g., the quality of resources created by students, challenges in incentivising engagement and lack of availability of reliable learnersourcing systems), which have limited its adoption. This paper presents a framework that considers the existing learnersourcing literature, the latest insights from the learning sciences and advances in AI to offer promising future directions for developing learnersourcing systems. The framework is designed around important questions and human-AI partnerships relating to four key aspects: (1) creating novel content, (2) evaluating the quality of the created content, (3) utilising learnersourced contributions of students and (4) enabling instructors to support students in the learnersourcing process. We then present two comprehensive case studies that illustrate the application of the proposed framework in relation to two existing popular learnersourcing systems.
@article{Ferguson_Khosravi_Kovanović_Viberg_Aggarwal_Brinkhuis_Buckingham Shum_Chen_Drachsler_Guerrero_Hanses_Hayward_Hicks_Jivet_Kitto_Kizilcec_Lodge_Manly_Matz_Meaney_Ochoa_Schuetze_Spruit_van Haastrecht_van Leeuwen_van Rijn_Tsai_Weidlich_Williamson_Yan_2023,
      title={Aligning the Goals of Learning Analytics with its Research Scholarship: An Open Peer Commentary Approach}, 
      volume={10}, 
      DOI={10.18608/jla.2023.8197}, 
      number={2}, journal={Journal of Learning Analytics}, 
      author={Ferguson, Rebecca and Khosravi, Hassan and Kovanović, Vitomir and Viberg, Olga and Aggarwal, Ashish and Brinkhuis, Matthieu and Buckingham Shum, Simon and Chen, Lujie Karen and Drachsler, Hendrik and Guerrero, Valerie A. and Hanses, Michael and Hayward, Caitlin and Hicks, Ben and Jivet, Ioana and Kitto, Kirsty and Kizilcec, René and Lodge, Jason M. and Manly, Catherine A. and Matz, Rebecca L. and Meaney, Michael J. and Ochoa, Xavier and Schuetze, Brendan A. and Spruit, Marco and van Haastrecht, Max and van Leeuwen, Anouschka and van Rijn, Lars and Tsai, Yi-Shan and Weidlich, Joshua and Williamson, Kimberly and Yan, Veronica X.},
      year={2023},
      pages={14-50} 
    }
    


Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources

Journal Paper 2023
Paul Denny and Hassan Khosravi and Arto Hellas and Juho Leinonen and Sami Sarsa (2023) e-print on arXiv
As an increasing number of students move to online learning platforms that deliver personalized learning experiences, there is a great need for the production of high-quality educational content. Large language models (LLMs) appear to offer a promising solution to the rapid creation of learning materials at scale, reducing the burden on instructors. In this study, we investigated the potential for LLMs to produce learning resources in an introductory programming context, by comparing the quality of the resources generated by an LLM with those created by students as part of a learnersourcing activity. Using a blind evaluation, students rated the correctness and helpfulness of resources generated by AI and their peers, after both were initially provided with identical exemplars. Our results show that the quality of AI-generated resources, as perceived by students, is equivalent to the quality of resources generated by their peers. This suggests that AI-generated resources may serve as viable supplementary material in certain contexts. Resources generated by LLMs tend to closely mirror the given exemplars, whereas student-generated resources exhibit greater variety in terms of content length and specific syntax features used. The study highlights the need for further research exploring different types of learning resources and a broader range of subject areas, and understanding the long-term impact of AI-generated resources on learning outcomes.
@article{
    }
    


Data Management of AI-Powered Education Technologies: Challenges and Opportunities

Journal Paper 2023
Hassan Khosravi and Shazia Sadiq and Amer-Yahia Sihem (2023) in the Journal of Learning Letters
The use of AI-powered educational technologies (AI-EdTech) offers a range of advantages to students, instructors, and educational institutions. While much has been achieved, several challenges in managing the data underpinning AI-EdTech are limiting progress in the field. This paper outlines some of these challenges and argues that data management research has the potential to provide solutions that can enable responsible and effective learner-supporting, teacher-supporting, and institution-supporting AI-EdTech. Our hope is to establish a common ground for collaboration and to foster partnerships among educational experts, AI developers and data management researchers in order to respond effectively to the rapidly evolving global educational landscape and drive the development of AI-EdTech.
@article{
    }
    


Explainable Artificial Intelligence in Education

Journal Paper 2022
Hassan Khosravi and Simon Buckingham Shum and Guanliang Chen and Cristina Conati and Yi-Shan Tsai and Judy Kay and Simon Knight and Roberto Martinez-Maldonado and Shazia Sadiq and Dragan Gasevic (2022). In the journal of Computers and Education: Artificial Intelligence
There are emerging concerns about the Fairness, Accountability, Transparency, and Ethics (FATE) of educational interventions supported by the use of Artificial Intelligence (AI) algorithms. One of the emerging methods for increasing trust in AI systems is to use eXplainable AI (XAI), which promotes the use of methods that produce transparent explanations and reasons for decisions AI systems make. Considering the existing literature on XAI, this paper argues that XAI in education has commonalities with the broader use of AI but also has distinctive needs. Accordingly, we first present a framework, referred to as XAI-ED, that considers six key aspects in relation to explainability for studying, designing and developing educational AI tools. These key aspects focus on the stakeholders, benefits, approaches for presenting explanations, widely used classes of AI models, human-centred designs of the AI interfaces and potential pitfalls of providing explanations within education. We then present four comprehensive case studies that illustrate the application of XAI-ED in four different educational AI tools. The paper concludes by discussing opportunities, challenges and future research needs for the effective incor- poration of XAI in education.
@article{khosravi2022explainable,
      title={Explainable Artificial Intelligence in education},
      author={Khosravi, Hassan and Shum, Simon Buckingham and Chen, Guanliang and Conati, Cristina and Tsai, Yi-Shan and Kay, Judy and Knight, Simon and Martinez-Maldonado, Roberto and Sadiq, Shazia and Ga{\v{s}}evi{\'c}, Dragan},
      journal={Computers and Education: Artificial Intelligence},
      volume={3},
      pages={100074},
      year={2022},
      publisher={Elsevier}
    }
    


Beyond item analysis: Connecting student behaviour and performance using e-assessment logs

Journal Paper 2022
Hatim Lahza and Tammy Smith and Hassan Khosravi (2022). In the British journal of Educational Technologies (BJET)
Traditional item analyses such as classical test theory (CTT) use exam-taker responses to assessment items to approximate their difficulty and discrimination. The increased adoption by educational institutions of electronic assessment platforms (EAPs) provides new avenues for assessment analytics by capturing detailed logs of an exam-taker's journey through their exam. This paper explores how logs created by EAPs can be employed alongside exam-taker responses and CTT to gain deeper insights into exam items. In particular, we propose an approach for deriving features from exam logs for approximating item difficulty and discrimination based on exam-taker behaviour during an exam. Items for which difficulty and discrimination differ significantly between CTT analysis and our approach are flagged through outlier detection for independent academic review. We demonstrate our approach by analysing de-identified exam logs and responses to assessment items of 463 medical students enrolled in a first-year biomedical sciences course. The analysis shows that the number of times an exam-taker visits an item before selecting a final response is a strong indicator of an item's difficulty and discrimination. Scrutiny by the course instructor of the seven items identified as outliers suggests our log-based analysis can provide insights beyond what is captured by traditional item analyses.
@article{lahza2022beyond,
      title={Beyond item analysis: Connecting student behaviour and performance using e-assessment logs},
      author={Lahza, Hatim and Smith, Tammy G and Khosravi, Hassan},
      journal={British Journal of Educational Technology},
      year={2022},
      publisher={Wiley Online Library}
    }
    


Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer Review Models

Journal Paper 2022
Ali Darvishi and Hassan Khosravi and Afshin Rahimi and Shazia Sadiq and Dragan Gasevic (2022). In the IEEE Transactions on Eductational Technologies (TLT)
Engaging students in creating learning resources has demonstrated pedagogical benefits. However, to effectively utilise a repository of student-generated content (SGC), a selection process is needed to separate high- from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common and scalable approach is to use a peer review process where students are asked to assess the quality of resources authored by their peers. Given that judgements of students, as experts-in-training, cannot wholly be relied upon, a redundancy-based method is widely employed where the same assessment task is given to multiple students. However, this approach introduces a new challenge, referred to as the consensus problem: How can we assign a final quality to a resource given ratings by multiple students? To address this challenge, we investigate the predictive performance of 18 inference models across five well-established categories of consensus approaches for inferring the quality of SGC at scale. The analysis is based on the engagement of 2,141 undergraduate students across five courses in creating 12,803 resources and 77,297 peer reviews. Results indicate that the quality of reviews is quite diverse, and students tend to overrate. Consequently, simple statistics such as mean and median fail to identify poor quality resources. Findings further suggest that incorporating advanced probabilistic and text analysis methods to infer the reviewers' reliability and reviews' quality improves performance; however, there is still an evident need for instructor oversight and training of students to write compelling and reliable reviews.
@ARTICLE{9984957,
      author={Darvishi, Ali and Khosravi, Hassan and Rahimi, Afshin and Sadiq, Shazia and Gašević, Dragan},
      journal={IEEE Transactions on Learning Technologies}, 
      title={Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer Review Models}, 
      year={2022},
      volume={},
      number={},
      pages={1-15},
      doi={10.1109/TLT.2022.3229022}
    }
    


Analytics of learning tactics and strategies in an online learnersourcing environment

Journal Paper 2022
Hatim Lahza and Hassan Khosravi and Gianluca Demartini (2022). Journal of Computer Assisted Learning
Background The use of crowdsourcing in a pedagogically supported form to partner with learners in developing novel content is emerging as a viable approach for engaging students in higher-order learning at scale. However, how students behave in this form of crowdsourcing, referred to as learnersourcing, is still insufficiently explored.

Objectives To contribute to filling this gap, this study explores how students engage with learnersourcing tasks across a range of course and assessment designs.

Methods We conducted an exploratory study on trace data of 1279 students across three courses, originating from the use of a learnersourcing environment under different assessment designs. We employed a new methodology from the learning analytics (LA) field that aims to represent students' behaviour through two theoretically-derived latent constructs: learning tactics and the learning strategies built upon them.

Results The study's results demonstrate students use different tactics and strategies, highlight the association of learnersourcing contexts with the identified learning tactics and strategies, indicate a significant association between the strategies and performance and contribute to the employed method's generalisability by applying it to a new context.

Implications This study provides an example of how learning analytics methods can be employed towards the development of effective learnersourcing systems and, more broadly, technological educational solutions that support learner-centred and data-driven learning at scale. Findings should inform best practices for integrating learnersourcing activities into course design and shed light on the relevance of tactics and strategies to support teachers in making informed pedagogical decisions.

@article{lahza2022beyond,
      title={Beyond item analysis: Connecting student behaviour and performance using e-assessment logs},
      author={Lahza, Hatim and Smith, Tammy G and Khosravi, Hassan},
      journal={British Journal of Educational Technology},
      year={2022},
      publisher={Wiley Online Library}
    }
    


Incorporating AI and learning analytics to build trustworthy peer assessment systems

Journal Paper 2022
Ali Darvishi and Hassan Khosravi and Shazia Sadiq and Dragan Gasevic (2022). In the British Journal of Educational Technology
Peer assessment has been recognised as a sustainable and scalable assessment method that promotes higher-order learning and provides students with fast and detailed feedback on their work. Despite these benefits, some common concerns and criticisms are associated with the use of peer assessments (eg, scarcity of high-quality feedback from peer student-assessors and lack of accuracy in assigning a grade to the assessee) that raise questions about their trustworthiness. Consequently, many instructors and educational institutions have been anxious about incorporating peer assessment into their teaching. This paper aims to contribute to the growing literature on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn can increase their trustworthiness and adoption. In particular, we present and evaluate our AI-assisted and analytical approaches that aim to (1) offer guidelines and assistance to student-assessors during individual reviews to provide better feedback, (2) integrate probabilistic and text analysis inference models to improve the accuracy of the assigned grades, (3) develop feedback on reviews strategies that enable peer assessors to review the work of each other, and (4) employ a spot-checking mechanism to assist instructors in optimally overseeing the peer assessment process.
@article{darvishi2022incorporating,
        title={Incorporating AI and learning analytics to build trustworthy peer assessment systems},
        author={Darvishi, Ali and Khosravi, Hassan and Sadiq, Shazia and Ga{\v{s}}evi{\'c}, Dragan},
        journal={British Journal of Educational Technology},
        year={2022},
        publisher={Wiley Online Library}
      }
      


Assessment in the Age of Artificial Intelligence

Journal Paper 2022
Zachari Swiecki and Hassan Khosravi and Guanliang Chen and Roberto Martinez-Maldonado and Jason Lodge and Sandra Milligan and Neil Selwyn and Dragan Gasevic (2022). In the journal of Computers and Education: Artificial Intelligence
In this paper, we argue that a particular set of issues mars traditional assessment practices. They may be difficult for educators to design and implement; only provide discrete snapshots of performance rather than nuanced views of learning; be unadapted to the particular knowledge, skills, and backgrounds of participants; be tailored to the culture of schooling rather than the cultures schooling is designed to prepare students to enter; and assess skills that humans routinely use computers to perform. We review extant artificial intelligence approaches that at least partially address these issues and critically discuss whether these approaches present additional challenges for assessment practice.
@article{swiecki2022assessment,
        title={Assessment in the age of artificial intelligence},
        author={Swiecki, Zachari and Khosravi, Hassan and Chen, Guanliang and Martinez-Maldonado, Roberto and Lodge, Jason M and Milligan, Sandra and Selwyn, Neil and Ga{\v{s}}evi{\'c}, Dragan},
        journal={Computers and Education: Artificial Intelligence},
        volume={3},
        pages={100075},
        year={2022},
        publisher={Elsevier}
      }
      


Incorporating Training, Self-monitoring and AI-Assistance to Improve Peer Feedback Quality

Conference Paper 2022
Ali Darvishi and Hassan Khosravi and Solmaz Abdi and Shazia Sadiq and Dragan Gasevic (2022). In the proceedings of the 9th International Conference on Learning at Scale
Peer review has been recognised as a beneficial approach that promotes higher-order learning and provides students with fast and detailed feedback on their work. Still, there are some common concerns and criticisms associated with the use of peer review that limits its adoption. One of the main points of concern is that feedback provided by students may be ineffective and of low quality. Previous works supply three explanations for why students may fail to provide effective feedback: They lack (1) the ability to provide high-quality feedback, (2) the agency to monitor their work or (3) the incentive to invest the required time and effort as they think the quality of the reviews are not reviewed. To help mitigate these shortcomings, this paper presents a complementary peer review approach that integrates training, self-monitoring and AI quality-control assistance to improve peer feedback quality. In particular, informed by higher education research, we built a set of training materials and a self-monitoring checklist for students to consider while writing their reviews. Also, informed by work from natural language processing, we developed quality control functions that automatically assess feedback submitted and prompt students to improve, if necessary. A between-subjects field experiment with 374 participants was conducted to investigate the approach's efficacy. Findings suggest that offering training, self-monitoring, and quality control functionalities to students assigned to the complementary peer review approach resulted in longer feedback that was perceived as more helpful than those who utilised the regular peer review interface. However, this complementary approach does not seem to affect students judgement (leniency or harshness) or confidence in grading. Directions are suggested to further evaluate and refine peer review systems.
@ inproceedings{darvishi2022incorporating,
        title={Incorporating training, self-monitoring and AI-assistance to improve peer feedback quality},
        author={Darvishi, Ali and Khosravi, Hassan and Abdi, Solmaz and Sadiq, Shazia and Ga{\v{s}}evi{\'c}, Dragan},
        booktitle={Proceedings of the Ninth ACM Conference on Learning@ Scale},
        pages={35--47},
        year={2022}
      }
      


Incorporating Explainable Learning Analytics to Assist Educators with Identifying Students in Need of Attention

Conference Paper 2022
Shiva Shabaninejad and Hassan Khosravi and Solmaz Abdi and Marta Indulska Shazia Sadiq (2022). In the proceedings of the 9th International Conference on Learning at Scale
Increased enrolments in higher education, and the shift to online learning that has been escalated by the recent COVID pandemic, have made it challenging for instructors to assist their students with their learning needs. Contributing to the growing literature on instructor-facing systems, this paper reports on the development of a learning analytics (LA) technique called Student Inspection Facilitator (SIF) that provides an explainable interpretation of students learning behaviour to support instructors with the identification of students in need of attention. Unlike many previous predictive systems that automatically label students, our approach provides explainable recommendations to guide data exploration while still reserving judgement about interpreting student learning to instructors. The insights derived from applying SIF in an introductory Information Systems course with 407 enrolled students suggest that SIF can be utilised independent of the context and can provide a meaningful interpretation of students' learning behaviour towards facilitating proactive support of students.
 @inproceedings{shabaninejad2022incorporating,
          title={Incorporating Explainable Learning Analytics to Assist Educators with Identifying Students in Need of Attention},
          author={Shabaninejad, Shiva and Khosravi, Hassan and Abdi, Solmaz and Indulska, Marta and Sadiq, Shazia},
          booktitle={Proceedings of the Ninth ACM Conference on Learning@ Scale},
          pages={384--388},
          year={2022}
        }        
      


Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing

Conference Paper 2022
Hatim Lahza and Hassan Khosravi and Gianluca Demartini and Dragan Gasevic (2022). In the proceedings of the 12th International Conference on Learning Analytics and Knowledge
The benefits of incorporating scaffolds that promote strategies of self-regulated learning (SRL) to help student learning are widely studied and recognised in the literature. However, the best methods for incorporating them in educational technologies and empirical evidence about which scaffolds are most beneficial to students are still emerging. In this paper, we report our findings from conducting an in-the-field controlled experiment with 797 post-secondary students to evaluate the impact of incorporating scaffolds for promoting SRL strategies in the context of assisting students in creating novel content, also known as learnersourcing. The experiment had five conditions, including a control group that had access to none of the scaffolding strategies for creating content, three groups each having access to one of the scaffolding strategies (planning, externally-facilitated monitoring and self-assessing) and a group with access to all of the aforementioned scaffolds. The results revealed that the addition of the scaffolds for SRL strategies increased the complexity and effort required for creating content, were not positively assessed by learners and led to slight improvements in the quality of the generated content. We discuss the implications of our findings for incorporating SRL strategies in educational technologies.
@inproceedings{lahza2022effects,
      title={Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing},
      author={Lahza, Hatim and Khosravi, Hassan and Demartini, Gianluca and Gasevic, Dragan},
      booktitle={LAK22: 12th International Learning Analytics and Knowledge Conference},
      pages={542--548},
      year={2022}
    }                                    
    


Intelligent Learning Analytics Dashboards: Automated Drill-Down Recommendations to Support Teacher Data Exploration

Journal Paper 2021
Hassan Khosravi and Shiva Shabaninejad and Aneesha Bakharia and Shazia Sadiq and Marta Indulska and Dragan Gasevic (2021). In the Journal of Journal of Learning Analytics
Learning analytics dashboards commonly visualize data about students with the aim of helping students and educators understand and make informed decisions about the learning process. To assist with making sense of complex and multidimensional data, many learning analytics systems and dashboards have relied strongly on AI algorithms based on predictive analytics. While predictive models have been successful in many domains, there is an increasing realization of the inadequacies of using predictive models in decision-making tasks that affect individuals without human oversight. In this paper, we employ a suite of state-of-the-art algorithms, from the online analytics processing, data mining, and process mining domains, to present an alternative human-in-the-loop AI method to enable educators to identify, explore, and use appropriate interventions for subpopulations of students with the highest deviation in performance or learning process compared to the rest of the class. We demonstrate an application of our proposed approach in an existing learning analytics dashboard (LAD) and explore the recommended drill-downs in a course with 875 students. The demonstration provides an example of the recommendations from real course data and shows how recommendations can lead the user to interesting insights. Furthermore, we demonstrate how our approach can be employed to develop intelligent LADs.
 
        @article{khosravi2021intelligent,
          title={Intelligent Learning Analytics Dashboards: Automated Drill-Down Recommendations to Support Teacher Data Exploration},
          author={Khosravi, Hassan and Shabaninejad, Shiva and Bakharia, Aneesha and Sadiq, Shazia and Indulska, Marta and Ga{\v{s}}evi{\'c}, Dragan},
          journal={Journal of Learning Analytics},
          volume={8},
          number={3},
          pages={133--154},
          year={2021}
        }
    


Bridging the Gap between Theory and Empirical Research in Evaluative Judgment

Journal Paper 2021
Hassan Khosravi and George Gyamfi and Barbara Hanna and Jasohn Lodge and Solmaz Abdi (2021). In the Journal of Journal of Learning Analytics
The value of students developing the capacity to make accurate judgments about the quality of their work and that of others has been widely studied and recognized in higher education literature. To date, much of the research and commentary on evaluative judgment has been theoretical and speculative in nature, focusing on perceived benefits and proposing strategies seen to hold the potential to foster evaluative judgment. Their efficacy remains largely untested. The rise of educational tools and technologies that generate data on learning activities at an unprecedented scale, alongside insights from the learning sciences and learning analytics communities, provides new opportunities for fostering and supporting empirical research on evaluative judgment. Accordingly, this paper offers a conceptual framework and an instantiation of that framework in the form of an educational tool called RiPPLE for data-driven approaches to investigating the enhancement of evaluative judgment. Two case studies, demonstrating how RiPPLE can foster and support empirical research on evaluative judgment, are presented.
 @article{khosrav2021bridging,
                            title={Bridging the Gap between Theory and Empirical Research in Evaluative Judgment.},
                            author={Khosrav, Hassan and Gyamf, George and Hanna, Barbara E and Lodge, Jason and Abdi, Solmaz},
                            journal={Journal of Learning Analytics},
                            volume={8},
                            number={3},
                            pages={117--132},
                            year={2021},
                          }                                  
                        


Neurophysiological Measurements in Higher Education: A Systematic Literature Review

Journal Paper 2021
Ali Darvishi and Hassan Khosravi and Shazia Sadiq and Barbara Weber (2021). In the International Journal of Artificial Intelligence in Education
The use of neurophysiological measurements to advance the design, development, use, acceptance, influence and adaptivity of information systems is receiving increasing attention. Within the field of education, neurophysiological measurements have commonly been used to capture a learner’s psychological constructs such as cognitive load, attention and emotion, which play an important role in student learning. This paper systematically examines the literature on the use of neurophysiological measurements in higher education. In particular, using a well-established Systematic Literature Review (SLR) method, we identified 83 papers reporting empirical evidence about the outcome of employing neurophysiological measurements within educational technologies in higher education. The findings of the SLR are divided into three main themes discussing the employed measurements, experimental settings and constructs and outcomes. Our findings identify that (1) electroencephalography and facial expression recognition are the dominantly employed types of measurement, (2) the majority of the experiments used a pre-experimental design, (3) attention and emotion are the two foremost cognitive and non-cognitive constructs under investigation, while less emphasis is paid to meta-cognitive constructs and (4) the reported results mostly focus on monitoring learners’ states, which are not always the same as the intended purpose, such as developing an adaptive system. On a broader term, the review of the literature provides evidence of the effective use of neurophysiological measurements by educational technologies to enhance learning; however, a number of challenges and concerns related to the accuracy and validity of the captured construct, the intrusiveness of the employed instruments as well as ethical and privacy considerations have surfaced, that need to be addressed before such technologies can be employed and adopted at scale.
 @article{darvishi2021neurophysiological,
        title={Neurophysiological Measurements in Higher Education: A Systematic Literature Review},
        author={Darvishi, Ali and Khosravi, Hassan and Sadiq, Shazia and Weber, Barbara},
        journal={International Journal of Artificial Intelligence in Education},
        doi = {10.1007/s40593-021-00256-0},
        pages={1--41},
        year={2021},
        publisher={Springer}
      }
    


Repositioning students as co-creators of curriculum for online learning resources

Journal Paper 2021
Aaron McDonald and Heath McGowan and Mollie Dollinger and Ryan Naylor and Hassan Khosravi (2021). In the Australian Journal of Educational Technologies
Amid increasing calls for universities to transition to online learning, there is a need to explore how platforms and technology can provide positive student experiences and support learning. In this paper, we discuss the implementation of an online peer learning and recommender platform in a large, multi-campus, first-year health subject (n = 2095). The Recommendation in Personalised Peer Learning Environments (RiPPLE) platform supports student’s co-creation of learning resources and allows for students to provide feedback and rate their peers’ submissions. Our results indicated that both student engagement and academic performance were positively impacted for users by the introduction of the RiPPLE platform, but that academic preparedness, in the form of students’ ATAR scores, strongly influenced their engagement and the benefits received.
 @article{mcdonald2021repositioning,
        title={Repositioning students as co-creators of curriculum for online learning resources},
        author={McDonald, Aaron and McGowan, Heath and Dollinger, Mollie and Naylor, Ryan and Khosravi, Hassan},
        journal={Australasian Journal of Educational Technology},
        doi = {10.14742/ajet.6735},
        pages={102--118},
        year={2021}
      }  
    


Open Learner Models for Multi-Activity Educational Systems

Conference Paper 2021
Solmaz Abdi and Hassan Khosravi and Shazia Sadiq and Ali Darvishi (2021). In the proceedings of the 22nd International Conference on Artificial Intelligence in Education
Engaging students in the creation of learning resources has been demonstrated to have pedagogical benefits and lead to the creation of large repositories of learning resources which can be used to complement student learning in different ways. In recent years, there has been an increasing trend in the use of student-centred approaches within educational systems that en- gage students in various higher-order learning activities such as creating resources, creating solutions, rating the quality of resources, and giving feedback. In response to this trend, this paper proposes an interpretable and open learner model called MA-Elo that capture an abstract repre- sentation of a student's knowledge state based on their engagement with multiple types of learning activities. We apply MA-Elo to three data sets obtained from an educational system supporting multiple student activ- ities. Results indicate that the proposed approach can provide a higher predictive performance compared with baseline and some state-of-the-art learner models.
      @inproceedings{abdi2021open,
        title={Open Learner Models for Multi-activity Educational Systems},
        author={Abdi, Solmaz and Khosravi, Hassan and Sadiq, Shazia and Darvishi, Ali},
        booktitle={International Conference on Artificial Intelligence in Education},
        pages={11--17},
        doi = {10.1007/978-3-030-78270-2_2},
        year={2021},
        organization={Springer}
      }  
    


Employing Peer Review to Evaluate the Quality of Student Generated Content at Scale: A Trust Propagation Approach

Conference Paper 2021
Ali Darvishi and Hassan Khosravi and Shazia Sadiq (2021). In the proceedings of the 8th International Conference on Learning at Scale
Engaging students in the creation of learning resources has been demonstrated to have pedagogical benefits and lead to the creation of large repositories of learning resources which can be used to complement student learning in different ways. However, to effectively utilise a learnersourced repository of content, a selection process is needed to separate high-quality from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common and scalable approach to evaluating the quality of learnersourced content is to use a peer review process where students are asked to assess the quality of resources authored by their peers. However, this method poses the problem of “truth inference” since the judgements of students as expertsin- training cannot wholly be trusted. This paper presents a graph-based approach to propagate the reliability and trust using data from peer and instructor evaluations in order to simultaneously infer the quality of the learnersourced content and the reliability and trustworthiness of users in a live setting. We use empirical data from a learnersourcing system called RiPPLE to evaluate our approach. Results demonstrate that the proposed approach can propagate reliability and utilise the limited availability of instructors in spot-checking to improve the accuracy of the model compared to
@inproceedings{darvishi2021employing,
        title={Employing Peer Review to Evaluate the Quality of Student Generated Content at Scale: A Trust Propagation Approach},
        author={Darvishi, Ali and Khosravi, Hassan and Sadiq, Shazia},
        booktitle={Proceedings of the Eighth ACM Conference on Learning@ Scale},
        pages={139--150},
        year={2021}
      }                                      
      


The Effects of Rubrics on Evaluative Judgement: A Randomised Controlled Experiment

Journal Paper 2021
George Gyamfi and Barbara Hanna and Hassan Khosravi (2021). In the Journal of Journal of Assessment & Evaluation in Higher Education
Rubrics have been suggested as a means to foster students’ evaluative judgement, the capacity to appraise their own work and that of others; however, empirical evidence of rubrics’ effectiveness is still emerging. This paper contributes findings from a randomised controlled experiment on the effect of rubrics on evaluative judgement. Participants were randomly assigned to one of two groups: a control group which evaluated peer-authored learning resources without the use of a rubric and an experiment group which carried out the evaluation using a three-item rubric based on 1. alignment with course content, 2. accuracy and 3. clarity. Both groups were asked to rate their confidence and provide comments to justify their scoring. The results showed a small effect size in increasing average agreement on the quality of learning resources in the experiment group. Analysis of comments reveals that criteria in and beyond the rubric guided participants’ ratings of quality. The study provides evidence of the impact of rubrics on students’ evaluative judgement and an example of how data-driven approaches and learning analytics can inform actionable design choices for embedding pedagogically supported strategies derived from the literature into actively operating educational technologies.
 @article{dgyamfi2021rubrics,
                            author = {George Gyamfi and Barbara E. Hanna and Hassan Khosravi},
                            title = {The effects of rubrics on evaluative judgement: a randomised controlled experiment},
                            journal = {Assessment \& Evaluation in Higher Education},
                            volume = {0},
                            number = {0},
                            pages = {1-18},
                            year  = {2021},
                            publisher = {Routledge},
                            doi = {10.1080/02602938.2021.1887081}
                            }                                      
                        


Evaluating the Quality of Learning Resources: A Learnersourcing Approach

Journal Paper 2021
Solmaz Abdi and Hassan Khosravi and Shazia Sadiq and Gianluca Demartini (2021). In the Journal of IEEE Transactions on Learning Technologies
Learnersourcing is emerging as a viable approach for mobilizing the learner community and harnessing the intelligence of learners as creators of learning resources. Previous works have demonstrated that the quality of resources developed by students is quite diverse with some resources meeting rigorous judgmental criteria while other resources are ineffective, inappropriate, or incorrect. Consequently, to effectively utilize these large repositories of resources in student learning, there is a need for a selection and moderation process to separate high-quality resources from low-quality ones in such repositories. Instructors and domain experts are potentially the most reliable source for doing this task; however, their availability is often quite limited. This paper explores whether and how learnersourcing, as an alternative approach, can be used for evaluating the quality of learning resources. To do so, we first follow a data-driven approach to explore students’ ability in judging the quality of learning resources. Results from this study suggest that, overall, ratings provided by students strongly correlate with ratings from experts; however, students’ ability in evaluating learning resources can also vary significantly. We then present a consensus approach based on matrix factorization (MF) and indicate how it can be used for improving the accuracy of aggregating learnersourced decisions. We also demonstrate how utilizing information on student performance and incorporating ratings from domain experts on a limited number of learning resources can
 @@ARTICLE{abdi2021learnersourcing,
                    author={Abdi, Solmaz and Khosravi, Hassan and Sadiq, Shazia and Demartini, Gianluca},
                    journal={IEEE Transactions on Learning Technologies}, 
                    title={Evaluating the Quality of Learning Resources: A Learnersourcing Approach}, 
                    year={2021},
                    volume={14},
                    number={1},
                    pages={81-92},
                    doi={10.1109/TLT.2021.3058644}}                                       
                


Charting the Design and Analytics Agenda of Learnersourcing Systems

Conference Paper 2021
Hassan Khosravi and , Gianluca Demartini and Shazia Sadiq and Dragan Gasevic Shazia Sadiq (2021). In the proceedings of the 11th International Conference on Learning Analytics and Knowledge
Learnersourcing is emerging as a viable learner-centred and pedagogically justified approach for harnessing the creativity and evaluation power of learners as experts-in-training. Despite the increasing adoption of learnersourcing in higher education, understanding students' behaviour while engaged in learnersourcing and best practices for the design and development of learnersourcing systems are still largely under-researched. This paper offers data-driven reflections and lessons learned from the development and deployment of a learnersourcing adaptive educational system called RiPPLE, which to date, has been used in more than 50-course offerings with over 12,000 students. Our reflections are categorised into examples and best practices on (1) assessing the quality of students' contributions using accurate, explainable and fair approaches to data analysis, (2) incentivising students to develop high-quality contributions and (3) empowering instructors with actionable and explainable insights to guide student learning. We discuss the implications of these findings and how they may contribute to the growing literature on the development of effective learnersourcing systems and more broadly technological educational solutions that support learner-centred learning at scale.
@inproceedings{khosravi2021learnersourcing,
                                          author = {Khosravi, Hassan and Demartini, Gianluca and Sadiq, Shazia and Gasevic, Dragan},
                                          title = {Charting the Design and Analytics Agenda of Learnersourcing Systems},
                                          year = {2021},
                                          isbn = {9781450389358},
                                          publisher = {Association for Computing Machinery},
                                          address = {New York, NY, USA},
                                          doi = {10.1145/3448139.3448143},
                                          booktitle = {LAK21: 11th International Learning Analytics and Knowledge Conference},
                                          pages = {32–42},
                                          numpages = {11},
                                          }                                       
                                        


Modelling Learners in Adaptive Educational Systems: A Multivariate Glicko-based Approach

Conference Paper 2021
Solmaz Abdi and Hassan Khosravi and Shazia Sadiq Shazia Sadiq (2021). In the proceedings of the 11th International Conference on Learning Analytics and Knowledge
The Elo rating system has been recognised as an effective method for modelling students and items within adaptive educational systems. A common characteristic across Elo-based learner models is that they are not sensitive to the lag time between two consecutive interactions of a student within the system. Implicitly, this characteristic assumes that students do not learn or forget between two consecutive interactions. However, this assumption seems insufficient in the context of adaptive learning systems where students could have improved their mastery through practising outside of the system or that their mastery may be declined due to forgetting. In this paper, we extend the existing works on the use of rating systems for modelling learners in adaptive educational systems by proposing a new learner model called \mvg that builds on the Glicko rating system. \mvg is sensitive to the lag time between two consecutive interactions of a student within the system and models it as a parameter that captures the confidence of the system in the current inferred rating. We apply \mvg on three public data sets and three data sets obtained from an adaptive learning system and provide evidence that \mvg outperforms other conventional models in estimating students' knowledge mastery.
 @inproceedings{abdi2021glicko,
                                            author = {Abdi, Solmaz and Khosravi, Hassan and Sadiq, Shazia},
                                            title = {Modelling Learners in Adaptive Educational Systems: A Multivariate Glicko-Based Approach},
                                            year = {2021},
                                            isbn = {9781450389358},
                                            publisher = {Association for Computing Machinery},
                                            address = {New York, NY, USA},
                                            doi = {10.1145/3448139.3448189},
                                            booktitle = {LAK21: 11th International Learning Analytics and Knowledge Conference},
                                            pages = {497–503},
                                            numpages = {7}
                                            }
                                            
                                                                                     
                                        


Identifying Cohorts: Recommending Drill-Downs Based on Differences in Behaviour for Process Mining

Conference Paper 2020
Sander J. J. Leemans and Shiva Shabaninejad and Kanika Goel and Hassan Khosravi and Shazia Sadiq and Moe Thandar Wynn (2020). In the proceedings of the International Conference on Conceptual Modeling
Process mining aims to obtain insights from event logs to improve business processes. In complex environments with large variances in process behaviour, analysing and making sense of such complex processes becomes challenging. Insights in such processes can be obtained by identifying sub-groups of traces (cohorts) and studying their differences. In this paper, we introduce a new framework that elicits features from trace attributes, measures the stochastic distance between cohorts defined by sets of these features, and presents this landscape of sets of features and their influence on process behaviour to users. Our framework differs from existing work in that it can take many aspects of behaviour into account, including the ordering of activities in traces (control flow), the relative frequency of traces (stochastic perspective), and cost. The framework has been instantiated and implemented, has been evaluated for feasibility on multiple publicly available real-life event logs, and evaluated on real-life case studies in two Australian universities.
 @InProceedings{Leemans2020cohort,
                                            author="Leemans, Sander J. J.
                                            and Shabaninejad, Shiva
                                            and Goel, Kanika
                                            and Khosravi, Hassan
                                            and Sadiq, Shazia
                                            and Wynn, Moe Thandar",
                                            editor="Dobbie, Gillian
                                            and Frank, Ulrich
                                            and Kappel, Gerti
                                            and Liddle, Stephen W.
                                            and Mayr, Heinrich C.",
                                            title="Identifying Cohorts: Recommending Drill-Downs Based on Differences in Behaviour for Process Mining",
                                            booktitle="Conceptual Modeling",
                                            year="2020",
                                            publisher="Springer International Publishing",
                                            address="Cham",
                                            pages="92--102",
                                            isbn="978-3-030-62522-1"
                                            }                                    
                                        


Utilising Learnersourcing to Inform Design Loop Adaptivity

Conference Paper 2020 Best Paper Nomination
Ali Darvishi and Hassan Khosravi and Shazia Sadiq (2020). In the 14th European Conference on Technology Enhanced Learning
Design-loop adaptivity refers to data-driven decisions that inform the design of learning materials to improve learning for student populations within adaptive educational systems (AES). Commonly in AESs, decisions on the quality of learning material are based on students’ performance, i.e., whether engaging with the material led to learning gains. This paper investigates an alternative approach for design adaptivity, which utilises students’ subjective ratings and comments to infer the quality of the learning material. This approach is in line with the recent shift towards learner-centred learning and learnersourcing, that aim to transform the role of students from passive recipients of content to active participants that engage with various higher-order learning tasks including evaluating the quality of resources. In this paper, we present a suite of aggregation-based and reliability-based methods that can be used to infer the quality of learning material based on student ratings and comments. We investigate the feasibility and accuracy of the methods in a live learnersourcing educational platform called RiPPLE that provides the capacity to capture subjective ratings and comments from students. Empirical data from the use of RiPPLE in a first-year course on information systems are used to evaluate the presented methods. Results indicate that the use of a combination of reliability-based methods provides an acceptable level of accuracy in determining the quality of learning resources.
 @InProceedings{darvishi2020designloop,
                                            author="Darvishi, Ali
                                            and Khosravi, Hassan
                                            and Sadiq, Shazia",
                                            editor="Alario-Hoyos, Carlos
                                            and Rodr{\'i}guez-Triana, Mar{\'i}a Jes{\'u}s
                                            and Scheffel, Maren
                                            and Arnedillo-S{\'a}nchez, Inmaculada
                                            and Dennerlein, Sebastian Maximilian",
                                            title="Utilising Learnersourcing to Inform Design Loop Adaptivity",
                                            booktitle="Addressing Global Challenges and Quality Education",
                                            year="2020",
                                            publisher="Springer International Publishing",
                                            address="Cham",
                                            pages="332--346",
                                            isbn="978-3-030-57717-9"
                                            }                                            
                                        


Recommending Insightful Drill-Downs based on Learning Processes for Learning Analytics Dashboards

Conference Paper 2020
Shiva Shabaninejad and Hassan Khosravi and Sander Leemans and Shazia Sadiq and Marta Indulska (2020). In the 21st Conference Artificial Intelligence in Education
Learning Analytics Dashboards (LADs) make use of rich and complex data about students and their learning activities to assist educators in understanding and making informed decisions about student learning and the design and improvement of learning processes. With the increase in the volume, velocity, variety and veracity of data on students, manual navigation and sense-making of such multi-dimensional data have become challenging. This paper proposes an analytical approach to assist LAD users with navigating the large set of possible drill-down actions to identify insights about learning behaviours of the sub-cohorts. A distinctive feature of the proposed approach is that it takes a process mining lens to examine and compare students’ learning behaviours. The process-oriented approach considers the flow and frequency of the sequences of performed learning activities, which is increasingly recognised as essential for understanding and optimising learning. We present results from an application of our approach in an existing LAD using a course with 875 students, with high demographic and educational diversity. We demonstrate the insights the approach enables, exploring how the learning behaviour of an identified sub-cohort differs from the remaining students and how the derived insights can be used by instructors.
 @InProceedings{10.1007/978-3-030-52237-7_39,
                        author="Shabaninejad, Shiva
                        and Khosravi, Hassan
                        and Leemans, Sander J. J.
                        and Sadiq, Shazia
                        and Indulska, Marta",
                        editor="Bittencourt, Ig Ibert
                        and Cukurova, Mutlu
                        and Muldner, Kasia
                        and Luckin, Rose
                        and Mill{\'a}n, Eva",
                        title="Recommending Insightful Drill-Downs Based on Learning Processes for Learning Analytics Dashboards",
                        booktitle="Artificial Intelligence in Education",
                        year="2020",
                        publisher="Springer International Publishing",
                        address="Cham",
                        pages="486--499",
                        isbn="978-3-030-52237-7"
                        } 
                


Modelling Learners in Crowdsourcing Educational Systems

Conference Paper 2020
Solmaz Abdi and Hassan Khosravi and Shazia Sadiq (2020). In the 21st Conference Artificial Intelligence in Education
Traditionally, learner models estimate a student’s knowledge state solely based on their performance on attempting assessment items. This can be attributed to the fact that in many traditional educational systems, students are primarily involved in just answering assessment items. In recent years, the use of crowdsourcing to support learning at scale has received significant attention. In crowdsourcing educational systems, in addition to attempting assessment items, students are engaged with other various tasks such as creating resources, creating solutions, rating the quality of resources, and giving feedback. Past studies have demonstrated that engaging students in meaningful crowdsourcing tasks, also referred to as learningsourcing, has pedagogical benefits that can enhance student learning. In this paper, we present a learner model that leverages data from students’ learnersourcing contributions alongside attempting assessment items towards modelling of students’ knowledge state. Results from an empirical study suggest that indeed crowdsourced contributions from students can effectively be used in modelling learners.
 @InProceedings{abdi2020learnersourcing,
                                      author="Abdi, Solmaz
                                      and Khosravi, Hassan
                                      and Sadiq, Shazia",
                                      editor="Bittencourt, Ig Ibert
                                      and Cukurova, Mutlu
                                      and Muldner, Kasia
                                      and Luckin, Rose
                                      and Mill{\'a}n, Eva",
                                      title="Modelling Learners in Crowdsourcing Educational Systems",
                                      booktitle="Artificial Intelligence in Education",
                                      year="2020",
                                      publisher="Springer International Publishing",
                                      address="Cham",
                                      pages="3--9"                            
                                      }
                                


Complementing Educational Recommender Systems with Open Learner Models

Conference Paper 2020
Solmaz Abdi and Hassan Khosravi and Shazia Sadiq and Dragan Gasevic (2020). in the proceedings of the 10th International Conference on Learning Analytics and Knowledge
Educational recommender systems (ERSs) aim to adaptively recommend a broad range of personalised resources and activities to students that will most meet their learning needs. Commonly, ERSs operate as a “black box" and give students no insight into the rationale of their choice. Recent contributions from the learning analytics and educational data mining communities have emphasised the importance of transparent, understandable and open learner models (OLMs) that provide insight and enhance learners’ understanding of interactions with learning environments. In this paper, we aim to investigate the impact of complementing ERSs with transparent and understandable OLMs that provide justification for their recommendations. We conduct a randomised control trial experiment using an ERS with two interfaces (“Non-Complemented Interface" and “Complemented Interface") to determine the effect of our approach on student engagement and their perception of the effectiveness of the ERS. Overall, our results suggest that complementing an ERS with an OLM can have a positive effect on student engagement and their perception about the effectiveness of the system despite potentially making the system harder to navigate. In some cases, complementing an ERS with an OLM has the negative consequence of decreasing engagement, understandability and sense of fairness.
 
                              @inproceedings{abdi2020complementing,
                              author = {Abdi, Solmaz and Khosravi, Hassan and Sadiq, Shazia and Gasevic, Dragan},
                              title = {Complementing Educational Recommender Systems with Open Learner Models},
                              year = {2020},
                              isbn = {9781450377126},
                              publisher = {Association for Computing Machinery},
                              address = {New York, NY, USA},
                              url = {https://doi.org/10.1145/3375462.3375520},
                              doi = {10.1145/3375462.3375520},
                              booktitle = {Proceedings of the Tenth International Conference on Learning Analytics & Knowledge},
                              pages = {360–365},
                              numpages = {6},
                              keywords = {user models, open learner models, educational recommender systems},
                              location = {Frankfurt, Germany},
                              series = {LAK ’20}
                              }
                          


Fostering and Supporting Empirical Research on Evaluative Judgement via a Crowdsourced Adaptive Learning System

Conference Paper 2020 Best Short Research Paper Nomination
Hassan Khosravi and George Gyamfi and Barbara Hanna and Jason Lodge (2020). in the proceedings of the 10th International Conference on Learning Analytics and Knowledge
The value of students developing the capacity to make accurate judgements about the quality of their work and that of others has been widely recognised in higher education literature. However, despite this recognition, little attention has been paid to the development of tools and strategies with the potential both to foster evaluative judgement and to support empirical research into its growth. This paper provides a demonstration of how educational technologies may be used to fill this gap. In particular, we introduce the adaptive learning system RiPPLE and describe how it aims to (1) develop evaluative judgement in large-class settings through suggested strategies from the literature such as the use of rubrics, exemplars and peer review and (2) enable large empirical studies at low cost to determine the effect-size of such strategies. A case study demonstrating how RiPPLE has been used to achieve these goals in a specific context is presented.
 @inproceedings{khosravi2020fostering,
                              author = {Khosravi, Hassan and Gyamfi, George and Hanna, Barbara E. and Lodge, Jason},
                              title = {Fostering and Supporting Empirical Research on Evaluative Judgement via a Crowdsourced Adaptive Learning System},
                              year = {2020},
                              isbn = {9781450377126},
                              publisher = {Association for Computing Machinery},
                              address = {New York, NY, USA},
                              url = {https://doi.org/10.1145/3375462.3375532},
                              doi = {10.1145/3375462.3375532},
                              booktitle = {Proceedings of the Tenth International Conference on Learning Analytics & Knowledge},
                              pages = {83–88},
                              numpages = {6},
                              keywords = {crowd-sourcing, educational technologies, evaluative judgement, student-authored materials},
                              location = {Frankfurt, Germany},
                              series = {LAK ’20}
                              }                              
                          


Automated Insightful Drill-Down Recommendations for Learning Analytics Dashboards

Conference Paper 2020 Best Short Research Paper Award
Shiva Shabaninejad and Hassan Khosravi and Marta Indulska and Aneesha Bakharia and Pedro Isaias (2020). In the proceedings of the 10th International Conference on Learning Analytics and Knowledge
The big data revolution is an exciting opportunity for universities, which typically have rich and complex digital data on their learners. It has motivated many universities around the world to invest in the development and implementation of learning analytics dashboards (LADs). These dashboards commonly make use of interactive visualisation widgets to assist educators in understanding and making informed decisions about the learning process. A common operation in analytical dashboards is a ‘drill-down’, which in an educational setting allows users to explore the behaviour of sub-populations of learners by progressively adding filters. Nevertheless, drill-down challenges exist, which hamper the most effective use of the data, especially by users without a formal background in data analysis. Accordingly, in this paper, we address this problem by proposing an approach that recommends insightful drill-downs to LAD users. We present results from an application of our proposed approach using an existing LAD. A set of insightful drill-down criteria from a course with 875 students are explored and discussed.
 @inproceedings{shabaninejad2020aid,
                              author = {Shabaninejad, Shiva and Khosravi, Hassan and Indulska, Marta and Bakharia, Aneesha and Isaias, Pedro},
                              title = {Automated Insightful Drill-down Recommendations for Learning Analytics Dashboards},
                              year = {2020},
                              isbn = {9781450377126},
                              publisher = {Association for Computing Machinery},
                              address = {New York, NY, USA},
                              url = {https://doi.org/10.1145/3375462.3375539},
                              doi = {10.1145/3375462.3375539},
                              booktitle = {Proceedings of the Tenth International Conference on Learning Analytics & Knowledge},
                              pages = {41–46},
                              numpages = {6},
                              keywords = {learning analytics dashboards, drill-down analysis, decision trees, exploratory data analysis},
                              location = {Frankfurt, Germany},
                              series = {LAK ’20}
                              }            
                          


Development and Adoption of an Adaptive Learning System: Reflections and Lessons Learned

Conference Paper 2020
Hassan Khosravi and Shazia Sadiq and Dragan Gasevic (2020). In the proceedings of 51 Special Interest Group on Computer Science Education
Adaptive learning systems (ALSs) aim to provide an efficient, effective and customised learning experience for students by dynamically adapting learning content to suit their individual abilities or preferences. Despite consistent evidence of their effectiveness and success in improving student learning over the past three decades, the actual impact and adoption of ALSs in education remain restricted to mostly research projects. In this paper, we provide a brief overview of reflections and lessons learned from developing and piloting an ALS in a course on relational databases. While our focus has been on adaptive learning, many of the presented lessons are also applicable to development and adoption of educational tools and technologies in general. Our aim is to provide insight for other instructors that are interested in adopting ALSs or are involved in implementation of educational tools and technologies.
 @inproceedings{khosravi2020development,
                        author = {Khosravi, Hassan and Sadiq, Shazia and Gasevic, Dragan},
                        title = {Development and Adoption of an Adaptive Learning System: Reflections and Lessons Learned},
                        year = {2020},
                        isbn = {9781450367936},
                        publisher = {Association for Computing Machinery},
                        address = {New York, NY, USA},
                        url = {https://doi.org/10.1145/3328778.3366900},
                        doi = {10.1145/3328778.3366900},
                        booktitle = {Proceedings of the 51st ACM Technical Symposium on Computer Science Education},
                        pages = {58–64},
                        numpages = {7},
                        keywords = {adaptive learning systems, crowdsourcing, educational technologies},
                        location = {Portland, OR, USA},
                        series = {SIGCSE ’20}
                        }                         
                    


Multilevel Visualisation of Topic Dependency Models for Assessment Design and Delivery: A Hypergraph Based Approach

Journal Paper 2019
Hassan Khosravi and Kendra Cooper (2019). Journal of Visual Languages and Computing
The effective design and delivery of assessments in a wide variety of evolving educational environments remains a challenging problem. Proposals have included the use of learning dashboards, peer learning environments, and grading support systems; these embrace visualisations to summarise and communicate results. In an on-going project, the investigation of graph based visualisation models for assessment design and delivery has yielded promising results. Here, an alternative graph foundation, a two-weighted hypergraph, is considered to represent assessment results and their explicit mapping to one or more learning objective topics. The visualisation approach considers the hypergraph as a collection of levels; the content of these levels can be customized (i.e., filtered) and presented according to user preferences. A case study on generating hypergraph models using commonly available assessment data and a flexible visualisation approach using historical data from an introductory programming course is presented.


RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities

Journal Paper 2019
Hassan Khosravi and Kirsty Kitto and Joseph Jay Williams (2019). In the Journal of Learning Analytics
This paper presents a platform called RiPPLE (Recommendation in Personalised Peer-Learning Environments) that recommends personalized learning activities to students based on their knowledge state from a pool of crowdsourced learning activities that are generated by educators and the students themselves. RiPPLE integrates insights from crowdsourcing, learning sciences, and adaptive learning, aiming to narrow the gap between these large bodies of research while providing a practical platform-based implementation that instructors can easily use in their courses. This paper provides a design overview of RiPPLE, which can be employed as a standalone tool or embedded into any learning management system (LMS) or online platform that supports the Learning Tools Interoperability (LTI) standard. The platform has been evaluated based on a pilot in an introductory course with 453 students at The University of Queensland. Initial results suggest that the use of the\name platform led to measurable learning gains and that students perceived the platform as beneficially supporting their learning.
 @article{Khosravi_Kitto_Williams_2019, 
                    title={RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities},
                    author={Khosravi, Hassan and Kitto, Kirsty and Williams, Joseph Jay}, 
                    journal={Journal of Learning Analytics}, 
                    volume={6}, 
                    number={3},
                    url={https://learning-analytics.info/index.php/JLA/article/view/6373}, 
                    DOI={10.18608/jla.2019.63.12}, 
                    year={2019},
                    pages={91–105} 
                    } 
                


Profiling Language Learners in the Big Data Era

Conference Paper 2019
Mauro Ocana and Hassan Khosravi and Aneesha Bakharia (2019). In the proceedings of 36th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education
The educational data revolution has empowered universities and educational institutes with rich data on their students, including information on their academic data (e.g., program completion, course enrolment, grades), learning activities (e.g., learning materials reviewed, discussion forum interactions, learning videos watched, projects conducted), learning process (i.e., time, place, path or pace of learning activities), learning experience (e.g., reflections, views, preferences) and assessment results. In this paper, we apply clustering to profile students from one of the largest Massive Open Online Courses (MOOCs) in the field of Second Language Learning. We first analyse the profiles, revealing the diversity among students taking the same course. We then, referring to the results of our analysis, discuss how profiling as a tool can be utilised to identify atrisk students, improve course design and delivery, provide targeted teaching practices, compare and contrast different offerings to evaluate interventions, develop policy, and improve selfregulation in students. The findings have implications for the fields of personalised learning and differentiated instruction.


Multilevel Visualisation of Topic Dependency Models for Assessment Design and Delivery: A Hypergraph Based Approach

Conference Paper 2019
Kendra Cooper and Hassan Khosravi (2019). Proceedings of the International DMS Conference on Visualization and Visual Languages
Rapidly evolving classroom environments continue to drive research on methods to support the design and delivery of educational assessments. These methods, such as dashboards, peer learning environments, and grading support systems, embrace visualisations to effectively summarise and communicate results. In an on-going project, the investigation of graph based visualisation models has yielded promising results: a collection of topic dependency models based on a two-weighted, undirected graph has been proposed. In these original models, the vertices represent topics in a course; the edges represent assessment material (e.g., questions) that address the topics the edges are related to. As the graphs can only represent edges with up to two vertices, questions involving three or more topics must be redistributed in the models as combinations, in order to visualise them. Here, an alternative graph foundation is considered to explicitly represent the learning objective material (e.g., questions) and their mapping to one or more topics: a two-weighted hypergraph. The hypergraph overcomes the limitation of a traditional graph. However, their more general hyperedge relationships may be more difficult to understand in the broader community; approaches to address this complexity are needed. Here, we present preliminary results on a method to generate a hypergraph model using commonly available assessment data and a flexible visualisaton approach to help address the complexity of the graphs. The approach is a multilevel filtering approach that considers the hypergraph as a collection of levels. The content of these levels can be customized and presented according to a users preference.


A Multivariate Elo-based Learner Model for Adaptive Educational Systems

Conference Paper 2019
Solmaz Abdi, Hassan Khosravi , Shazia Sadiq and Dragan Gasevic (2019). In the proceeding of the 12th International Conference on Educational Data Mining
The Elo rating system has been recognised as an effective method for modelling students and items within adaptive educational systems. The existing Elo-based models have the limiting assumption that items are only tagged with a single concept and are mainly studied in the context of adaptive testing systems. In this paper, we introduce a multivariate Elo-based learner model that is suitable for the domains where learning items can be tagged with multiple concepts, and investigate its fit in the context of adaptive learning. To evaluate the model, we first compare the predictive performance of the proposed model against the standard Elo-based model using synthetic and public data sets. Our results from this study indicate that our proposed model has superior predictive performance compared to the standard Elo-based model, but the difference is rather small. We then investigate the fit of the proposed multivariate Elo-based model by integrating it into an adaptive learning system which incorporates the principles of open learner models (OLMs). The results from this study suggest that the availability of additional parameters derived from multivariate Elo-based models have two further advantages: guiding adaptive behaviour for the system and providing additional insight for students and instructors.
@inproceedings{abdi2019multivariate,
                    title={A Multivariate ELO-based Learner Model for Adaptive Educational Systems},
                    author={Abdi, Solmz and Khosravi, Hassan and Sadiq, Shazia and Gasevic, Dragan},
                    booktitle={Proceedings of the Educational Data Mining Conference},
                    pages={462-467},
                    year={2019}
                  } 
                  


Topic Dependency Models: Graph-Based Visual Analytics for Communicating Assessment Data

Journal Paper 2018
Hassan Khosravi Kendra Cooper (2018). Journal of Learning Analytics
Educational environments continue to rapidly evolve to address the needs of diverse, growing student populations, while embracing advances in pedagogy and technology. In this changing landscape ensuring the consistency among the assessments for different offerings of a course (within or across terms), providing meaningful feedback about students’ achievements, and tracking students’ progression over time are all challenging tasks, particularly at scale. Here, a collection of visual Topic Dependency Models (TDMs) is proposed to help address these challenges. It uses statistical models to determine and visualise students’ achievements on one or more topics and their dependencies at a course level reference TDM (e.g., CS 100) as well as assessment data at the classroom level (e.g., students in CS 100 Term 1 2016 Section 001) both at one point in time (static) and over time (dynamic). The collection of TDMs share a common two-weighted graph foundation. Exemplar algorithms are presented for the creation of the course reference and selected class (static and dynamic) TDMs; the algorithms are illustrated using a common symbolic example. Studies on the application of the TDM collection on data sets from two university courses are presented; these case studies utilise the open-source, proof of concept tool under development.
@inproceedingskhosravi2018tdms,
                        author = { Khosravi, Hassan and Kendra Cooper},
                        title = {Topic Dependency Models: Graph-Based Visual Analytics for Communicating Assessment Data},
                        journal={Journal of Learning Analytics},
                        volume={5},
                        number={3},
                        pages={136--153},
                        year={2018}
                        year = {2018},
                      } 
                    


Students as partners in action: Evaluating a university-wide initiative

Journal Paper 2018
Leanne Coombe, Jasmine Huang, Stuart Russell, Karen Sheppard, Hassan Khosravi (2018). International Journal for Students As Partners
This case study was designed as one of many pilot projects to inform the scaling-up of Students as Partners (SaP) as a whole-of-institution strategy to enhance the student learning experience. It sought to evaluate the other pilots in order to understand the phenomena of partnerships and how students and staff perceive the experience of working in partnership. It also sought to explore the extent of benefits and challenges experienced by staff and students throughout the process and identify potential implications for future implementation.
@article{coombe2018students,
                          title={Students as partners in action: Evaluating a university-wide initiative},
                          author={Coombe, Leanne and Huang, Jasmine and Russell, Stuart and Sheppard, Karen and Khosravi, Hassan},
                          journal={International Journal for Students As Partners},
                          volume={2},
                          number={2},
                          pages={85--95},
                          year={2018}
                        }
                    


Predicting Student Performance: The Case of Combining Knowledge Tracing and Collaborative Filtering

Conference Paper 2018
Solmaz Abdi, Hassan Khosravi and Shazia Sadiq (2018). Proceeding of the 11th International Conference on Educational Data Mining
In the past few years, many competing learning models have been proposed for improving the accuracy of predicting student performance (PSP). A well-studied subclass of algorithms focused on PSP uses temporal models to determine the knowledge state of users. Bayesian Knowledge Tracing (BKT), as one of the leading models in this subclass, uses Hidden Markov Models to capture the student knowledge states. An emerging new subclass of algorithms focused on PSP uses collaborative filtering, which is used primarily by recommender systems. Matrix Factorization (MF), a leading model in this subclass, can be presented as a rating prediction problem where students, tasks, and performance information are mapped to users, items and ratings, respectively. BKT and MF complement each other’s strengths and limitations quite effectively. In particular, BKT relies on four skill-specific parameters for learning the sequential behavior of learners on each concept, but it does not capture the similarities among users and items. In contrast, MF uses latent factors to exploit the similarities among users and items from learner-item performance, but disregards any temporal effect in modeling student learning. In this paper, we aim to investigate the effect of combining variations of BKT and MF using a proposed algorithm that exploits the power of MF in modeling the implicit similarities among learners and items while using the explicit parametrization of BKT towards improving PSP. Our results on four bench-mark educational datasets show that our approach outperforms the base classes as well as traditional techniques such as linear regression, logistic regression and Neural Networks for combining BKT and MF.
@inproceedings{abdi:2018:RPR:3170358.3170400,
                        author = {Abdi, Solmaz, and Khosravi, Hassan and Sadiq, Shazia},
                        title = {Predicting Student Performance: The Case of Combining Knowledge Tracing and Collaborative Filtering},
                        booktitle = {Proceeding of the 11th International Conference on Educational Data Mining},
                        series = {EDM2018},
                        pages="545--549",
                        year = {2018},
                      } 
                    


Reciprocal Content Recommendation for Peer Learning Study Sessions

Conference Paper 2018
Boyd Potts, Hassan Khosravi and Carl Reidsema (2018). Proceeding of the 19th International Conference on Artificial Intelligence in Education
Recognition of peer learning as a valuable supplement to formal education has lead to a rich literature formalising peer learning as an institutional resource. Facilitating peer learning support sessions alone however, without providing guidance or context, risks being ineffective in terms of any targeted, measurable effects on learning. Building on an existing open-source, student-facing platform called RiPPLE, which recommends peer study sessions based on the availability, competencies and compatibility of learners, this paper aims to supplement these study sessions by providing content from a repository of multiple-choice questions to facilitate topical discussion and aid productiveness. We exploit a knowledge tracing algorithm alongside a simple Gaussian scoring model to select questions that promote relevant learning and that reciprocally meet the expectations of both learners. Primary results using synthetic data indicate that the model works well at scale in terms of the number of sessions and number of items recommended, and capably recommends from a large repository the content that best approximates a proposed difficulty gradient.
                        @InProceedings{Potts2018,
                          author="Potts, Boyd A. and Khosravi, Hassan and Reidsema, Carl",
                          editor="Penstein Ros{\'e}, Carolyn
                          and Mart{\'i}nez-Maldonado, Roberto and Hoppe, H. Ulrich and Luckin, Rose and Mavrikis, Manolis and Porayska-Pomsta, Kaska and McLaren, Bruce and du Boulay, Benedict",
                          title="Reciprocal Content Recommendation for Peer Learning Study Sessions",
                          booktitle="Artificial Intelligence in Education",
                          year="2018",
                          publisher="Springer International Publishing",
                          address="Cham",
                          pages="462--475",
                          isbn="978-3-319-93843-1"
                          }
                  


Reciprocal Peer Recommendation for Learning Purposes

Conference Paper 2018
Boyd Potts, Hassan Khosravi , Carl Reidsema, Aneesha Bakharia, Mark Belonogof, Melanie Fleming (2018). Proceeding of the 8th International Learning Analytics and Knowledge (LAK) Conference
Larger student intakes by universities and the rise of education through Massive Open Online Courses has led to less direct contact time with teaching staff for each student. One potential way of addressing this contact deficit is to invite learners to engage in peer learning and peer support; however, without technological support they may be unable to discover suitable peer connections that can enhance their learning experience. Two different research subfields with ties to recommender systems provide partial solutions to this problem. Reciprocal recommender systems provide sophisticated filtering techniques that enable users to connect with one another. To date, however, the main focus of reciprocal recommender systems has been on providing recommendation in online dating sites. Recommender systems for technology enhanced learning have employed and tailored exemplary recommenders towards use in education, with a focus on recommending learning content rather than other users. In this paper, we first discuss the importance of supporting peer learning and the role recommending reciprocal peers can play in educational settings. We then introduce our open-source course-level recommendation platform called RiPPLE that has the capacity to provide reciprocal peer recommendation. The proposed reciprocal peer recommender algorithm is evaluated against key criteria such as scalability, reciprocality, coverage, and quality and shows improvement over a baseline recommender. Primary results indicate that the system can help learners connect with peers based on their knowledge gaps and reciprocal preferences, with designed flexibility to address key limitations of existing algorithms identified in the literature.
@inproceedings{Potts:2018:RPR:3170358.3170400,
                        author = {Potts, Boyd A. and Khosravi, Hassan and Reidsema, Carl and Bakharia, Aneesha and Belonogoff, Mark and Fleming, Melanie},
                        title = {Reciprocal Peer Recommendation for Learning Purposes},
                        booktitle = {Proceedings of the 8th International Conference on Learning Analytics and Knowledge},
                        series = {LAK '18},
                        year = {2018},
                        isbn = {978-1-4503-6400-3},
                        location = {Sydney, New South Wales, Australia},
                        pages = {226--235},
                        numpages = {10},
                        url = {http://doi.acm.org/10.1145/3170358.3170400},
                        doi = {10.1145/3170358.3170400},
                        acmid = {3170400},
                        publisher = {ACM},
                        address = {New York, NY, USA},
                        keywords = {peer learning, peer support, reciprocal recommender, recsysTEL},
                        } 
                        
                    


Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale

Conference Paper 2018 Best Short Research Paper Nomination
Kendra Cooper and Hassan Khosravi (2018). Proceeding of the 8th International Learning Analytics and Knowledge (LAK) Conference
Educational environments continue to rapidly evolve to address the needs of diverse, growing student populations, while embracing advances in pedagogy and technology. In this changing landscape the design and delivery of high-quality assessments become even more challenging. In particular, ensuring that there is consistency among the assessments for di erent o erings of a course, providing meaningful feedback about students’ achievements, and tracking students’ progression over time are all challenging tasks at scale. Here, a collection of visual Topic Dependency Models (TDMs) is proposed that addresses the three facets of t he challenges noted above. It visualises the required topics and their dependencies at a course level (e.g., CS 100); visualises assessment achievement data at a classroom level (e.g., CS 100 Term 1 2016 Section 001) both at one point in time ( static) and over time (dynamic). The collection of TDMs share a common, two-weighted graph foundation; the models are illustrated using examples drawn from a high-enrolment introductory course on C programming for engineering students. An algorithm is presented to create a TDM (static achievement for a cohort). A proof of concept tool is under development; the current version is described brie y in terms of its support for visualising existing (historical, test) and synthetic data generated on demand.
 @inproceedings{Cooper:2018:GVT:3170358.3170418,
                        author = {Cooper, Kendra and Khosravi, Hassan},
                        title = {Graph-based Visual Topic Dependency Models: Supporting Assessment Design and Delivery at Scale},
                        booktitle = {Proceedings of the 8th International Conference on Learning Analytics and Knowledge},
                        series = {LAK '18},
                        year = {2018},
                        isbn = {978-1-4503-6400-3},
                        location = {Sydney, New South Wales, Australia},
                        pages = {11--15},
                        numpages = {5},
                        url = {http://doi.acm.org/10.1145/3170358.3170418},
                        doi = {10.1145/3170358.3170418},
                        acmid = {3170418},
                        publisher = {ACM},
                        address = {New York, NY, USA},
                        keywords = {graph algorithms, student assessment, visual analytics},
                        } 
                    


Recommendation in Personalised Peer Learning Environments

Report 2017
Hassan Khosravi
Recommendation in Personalised Peer Learning Environments (RiPPLE) is an adaptive, crowdsourced, web-based, student-facing, open-source platform that employs exemplary techniques from the fields of machine learning, crowdsourcing, learning analytics and recommender systems to provide personalised content and learning support at scale. RiPPLE presents students with a repository of tagged multiple-choice questions and provides instant feedback in response to student answers. The repository of the questions is created in partnership with the students through the use of crowdsourcing. RiPPLE uses students responses to the questions to approximate their knowledge states. Based on their knowledge state and learning needs, each student is recommended a set of personalised questions. For students that are interested in providing learning support, seeking learning support or finding study partners, RiPPLE recommends peer learning sessions based on their availability, knowledge state and preferences. This paper describes the RiPPLE interface and an implementation of that interface that has been built at the University of Queensland. The RiPPLE platform and a reference implementation
 @article{khosravi2017recommendation,
                    title={Recommendation in Personalised Peer-Learning Environments},
                    author={Khosravi, Hassan},
                    journal={arXiv preprint arXiv:1712.03077},
                    year={2017}
                  }
                


Towards the Addition of Recommendation to Visualising Learning Dashboards

Conference Paper 2017
Solmaz Abdi, Hassan Khosravi (2017). Presented at the 5th Australian Learning Analytics Summer Institute
There have been significant contributions from the learning analytics community on creating visualisations within student-facing learning dashboards to provide insight and enhance learners’ understanding of interactions with learning environments. While learning dashboards have been well-received in the research com- munity, most of the developed dashboards, to date, have had limited ability in providing actionable insight for improving learning. In a separate body of work, inspired by the success of recommender systems, researchers have utilised the digital traces left by learners towards providing recommendation of resources that will assist students in overcoming their shortcomings. Despite the success of recommender systems in many other domains, they have not been well-adopted in the context of higher education. This may be because recommender systems often do not provide rationale for their recommendations. As a potential solution for addressing the twin challenges of visualisation without recommendation and recommendation without justification, we have designed, implemented, and validated an open-source student-facing learning platform called RiPPLE that couples visualisation and recommendation. We have evaluated the approach using synthetic data sets. Our results indicate that RiPPLE can provide accurate personalised and justified recommendation for learners.


Competency, Compatibility and Preferences in Reciprocal Peer Recommendation

Conference Paper 2017
Boyd Potts, Hassan Khosravi Carl Reidsema (2017). Presented at the 5th Australian Learning Analytics Summer Institute
Inviting learners to engage in peer learning and peer support has established bene ts for both students and providers of education. Reciprocal recommender systems provide sophisticated altering techniques that enable users to connect with one another. Recommender systems for technology enhanced learning have employed and tailored recommenders towards use in education, with a focus on recommending learning content rather than other users. In this paper, we discuss the role recommending reciprocal peers can play in educational settings and introduce our open-source course- level recommendation platform called RiPPLE and its capacity to provide reciprocal peer recommendation. The proposed algorithm is evaluated against key criteria such as scalability, reciprocality and coverage, showing improvement over a non-reciprocal recommender. Primary results indicate that the system can help learners connect with peers based on their knowledge gaps and reciprocal preferences, with designed exibility to address key limitations of existing algorithms identified in the literature.


Analysing the Learning Pathways of Students in a Large Flipped Engineering Course

Conference Paper 2017
Carl Reidsema, Hassan Khosravi , Melanie Fleming, Nick Achilles, Esther Fink (2017). Proceedings of the 34th International Conference on Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education
Recent advancements in educational technologies (learning management systems, online discussion forums, peer-learning tools) coupled with new methods of course delivery (e.g. blended, flipped, MOOCs) provide significant opportunities for universities to deliver challenging, high quality, yet engaging curriculum for students. In this paper, we examine the variations and similarities of student’s approaches to learning (learning pathways) by examining how well they performed in a large (N ~ 1000 student) first year engineering flipped classroom. The analysis focused on student’s performance in their assessment (formative and summative) as well as their online interaction with a range of online tools purposely built to support students through peer learning and acquisition of resources and expertise. Analysis using k-means clustering reveals that students do in fact adopt a variety of successful pathways through the course. The unique aspects of this work lie in the use of analytics algorithms that whilst perhaps routinely utilised in data mining, are not as well utilised in better understanding patterns (successful or otherwise) of student interactions within a technology enhanced active learning environment that integrates theory with engineering practice.
 @inproceedings{Reidsema2017LearningPathways,
                        title={Analysing the Learning Pathways of Students in a Large Flipped Engineering Course},
                        author={Reidsema, Carl and Khosravi, Hassan and Fleming, Melanie and Achilles, Nick and Fink,  Esther},
                        booktitle={Proceedings of the 34th International Conference of Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education},
                        pages={372--382},
                        year={2017},
                      }


RiPLE: Recommendation in Peer-Learning Environments Based on Knowledge Gaps and Interests

Journal Paper 2017
Hassan Khosravi , Kendra Cooper, Kirsty Kitto (2017). Journal of Educational Data Mining
Various forms of Peer-Learning Environments are increasingly being used in post-secondary education, often to help build repositories of student generated learning objects. However, large classes can result in an extensive repository, which can make it more challenging for students to search for suitable objects that both reflect their interests and address their knowledge gaps. Recommender Systems for Technology Enhanced Learning (RecSysTEL) offer a potential solution to this problem by providing sophisticated filtering techniques to help students to find the resources that they need in a timely manner. Here, a new RecSysTEL for Recommendation in Peer-Learning Environments (RiPLE) is presented. The approach uses a collaborative filtering algorithm based upon matrix factorization to create personalized recommendations for individual students that address their interests and their current knowledge gaps. The approach is validated using both synthetic and real data sets. The results are promising, indicating RiPLE is able to provide sensible personalized recommendations for both regular and cold-start users under reasonable assumptions about parameters and user behavior.
 @article{khosravi2017,
                    title={RiPLE: Recommendation in Peer-Learning Environments Based on Knowledge Gaps and Interests},
                    author={Khosravi, Hassan and Cooper, Kendra and Kitto, Kirsty},
                    journal={JEDM-Journal of Educational Data Mining},
                    volume={9},
                    number={1},
                    pages={42-67},
                    year={2017}
                  }
                  


Using Learning Analytics to Investigate Patterns of Performance and Engagement in Large Classes

Conference Paper 2017
Hassan Khosravi , Kendra Cooper (2017). Proceedings of the Special Interest Group on Computer Science Education conference.
Educators continue to face significant challenges in providing high quality, post-secondary instruction in large classes including: motivating and engaging diverse populations (e.g., academic ability and backgrounds, generational expectations); and providing helpful feedback and guidance. Researchers investigate solutions to these kinds of challenges from alternative perspectives, including learning analytics (LA). Here, LA techniques are applied to explore the data collected for a large, flipped introductory programming class to (1) identify groups of students with similar patterns of performance and engagement; and (2) provide them with more meaningful appraisals that are tailored to help them effectively master the learning objectives. Two studies are reported, which apply clustering to analyze the class population, followed by an analysis of a subpopulation with extreme behaviours.
 @inproceedings{khosravi2017largeclasses,
                          title={Using Learning Analytics to Investigate Patterns of Performance and Engagement in Large Classes},
                          author={Khosravi, Hassan and Cooper, Kendra ML},
                          booktitle={Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education},
                          pages={309--314},
                          year={2017},
                          organization={ACM}
                        }


An Overview of Learning Analytics in Higher Education

Report 2017
Amy Wong, Marcel Lavrencic, Hassan Khosravi (2017) Institute for Teaching and Learning Innovation, The University of Queensland
This paper provides an overview of the recent national and international initiatives, adoptions, and challenges around learning analytics. It aims to assist the Learning Analytics Steering Group at the University of Queensland in setting the context and priorities in the adoption of learning analytics, as we embrace the opportunities and challenges big data creates in higher education.


Enhanced security authentication methods, systems and media

Patent 2016
Andrew Csinger, Ildar Muslukhov, Hassan Khosravi, Peter Luong (2016)
A transaction authorization apparatus includes a processor in communication with a communications interface. The processor is configured to receive a request for a transaction requested by a user with whom a plurality of user devices are associated, to obtain respective transaction measurements from at least some available devices from among the plurality of user devices, and to confirm approval of the request for the transaction in response to confirmation that the transaction measurements satisfy a multi­device authorization policy associated with the transaction.
 @misc{csinger2016enhanced,
                                        title={Enhanced security authentication methods, systems and media},
                                        author={Csinger, A. and MUSLUKHOV, I. and KHOSRAVI, H. and LUONG, P.T.},
                                        url={http://www.google.com.pg/patents/WO2016115633A1?cl=en},
                                        year={2016},
                                        month=jul # "~28",
                                        publisher={Google Patents},
                                      }

                                    


Retiring the Red Pen: Marking Exams Digitally

Poster 2016
Hassan Khosravi, Donald Acton (2016), UBC Annual Science Education Open House
Over the last several years members of the Computer Science department have been marking exams, scanning them, recording grades using optical character recognition, and then returning online digital copies to students. This has greatly decreased the amount of time spent handing back exams and recording marks. With the advent of cloud-based software like Gradescope and Crowdmark two of us decided to switch the workflow and load scanned exams into grading software and then marked them online. During the 2015W2 term we collectively marked over 1500 exams (including group exams), across 3 courses (APSC 160, CPSC 313, and CPSC 317) utilizing over 30 TAs.


Method and System for Decoupling User Authentication and Data Encryption on Mobile Devices

Patent 2015
Hassan Khosravi, Ildar Muslukhov, Peter Luong (2015)
Over a billion people today use smartphones - a portable and highly mobile personal computer. Various data is stored on such devices for the benefit of being accessible "on-the-go". This creates the necessity to protect sensitive data that are stored on smartphones, i.e., "data-at-rest". Data encryption with a randomly generated encryption key can be applied. However, since the device has to be able to work in off-line scenarios, the encryption key has to be stored on the device, along with data. In order to overcome this limitation, all major mobile platforms encrypt the encryption key with so called key encryption key. The key encryption key is derived from a secret that is used to authenticate smartphone users (e.g., a PIN-code, or password). Unfortunately, PIN-codes are weak and are prone to brute-force attacks. In this work we present the KeyVault system, that decouples user authentication and data encryption. The results of the evaluation suggest that the KeyVault system is a practical solution that requires low maintainability from users.
@misc{khosravi2015coupling,
                                        title={Method and system for decoupling user authentication and data encryption on mobile devices},
                                        author={Khosravi, Hassan and Muslukhov, Ildar and Luong, Peter},
                                        year={2015},
                                        month=sep # "~15",
                                        publisher={Google Patents},
                                        note={US Patent 9,137,659}
                                      }

                                    


Modelling Relational Statistics With Bayes Nets Using Pseudo-Likelihood

Journal Paper 2014
Oliver Schulte, Hassan Khosravi , Arthur Kirkpatrick, Tong Man, Tianxiang Gao, Yi Xiong, Yuke Zhu, Jianxing Ling (2014). Journal of Machine Learning.
Learning statistical patterns based on relational frequencies is a machine learning task that supports applications like strategic planning and query optimization. Bayes nets are a widely used generative model class for representing statistical patterns. Bayes net parameters are often learned using classic maximum likelihood estimation. Applying this to relational data is difficult, however, due to the dependencies introduced by cyclic relationships. To address cyclic relations, a Bayes Net pseudolikelihood measure was recently introduced for relational data. This measure can be used to construct maximum pseudolikelihood estimation for estimating Bayes Net parameters. A naive implementation is intractable due to the complexity imposed by negated relational links. We render this computation tractable by using the fast Mobius transform. Evaluation on four benchmark datasets shows that MPLE is consistent, and provides accurate
 @article{Schulte2014statisticalstats,
                                        author = {Schulte, Oliver and Khosravi, Hassan and Kirkpatrick, Arthur E. and Gao, Tianxiang and Zhu, Yuke},
                                        title = {Modelling Relational Statistics with Bayes Nets},
                                        journal = {Mach. Learn.},
                                        issue_date = {January   2014},
                                        volume = {94},
                                        number = {1},
                                        month = jan,
                                        year = {2014},
                                        issn = {0885-6125},
                                        pages = {105--125},
                                        numpages = {21},
                                        url = {http://dx.doi.org/10.1007/s10994-013-5362-7},
                                        doi = {10.1007/s10994-013-5362-7},
                                        acmid = {2558593},
                                        publisher = {Kluwer Academic Publishers},
                                        address = {Hingham, MA, USA},
                                        keywords = {Bayes Nets, M\"{o}bius transform, Pseudo-likelihood, Statistical-relational learning, Structured data},
                                        } 
                                    


Transaction-Based Link Strength Prediction Using Matrix Factorization

Conference Paper 2013
Hassan Khosravi , Ali Bozorgkhan, Oliver Schulte (2013). Computational Intelligence and Data Mining (CIDM), IEEE.
The revolution of online social networks and methods of analyzing them have attracted interest in many research fields. Predicting whether a friendship holds in a social network between two individuals or not, link prediction, has been a heavily researched topic in the last decade. In this paper we research a related problem, link strength prediction, which aims to assign ratings or strengths to friendship links. A basic approach would be matrix factorization applied to friendship ratings. However, the existence of extensive transactions among users may be used for better predictions. We propose a new type of multiple-matrix factorization model for incorporating a transaction matrix. We derive gradient descent update equations for learning latent factors that predict values in the target rating matrix. To evaluate the model, we introduce data from Cloob which is a popular Iranian social network as well as synthetic data.
 @inproceedings{khosravi2013transaction,
                                          title={Transaction-based link strength prediction in a social network},
                                          author={Khosravi, Hassan and Bozorgkhan, Ali and Schulte, Oliver},
                                          booktitle={Computational Intelligence and Data Mining (CIDM), 2013 IEEE Symposium on},
                                          pages={191--198},
                                          year={2013},
                                          organization={IEEE}
                                        }
                                    


Fast Learning of Markov Logic Networks Via Moralization

Conference Paper 2012
Hassan Khosravi , Oliver Schulte, Tong Man (2012). proceeding of Inductive Logic programming .
Scaling MLN learning to medium-sized datasets with many descriptive attributes is a challenge. For structure learning, one of the most efficient and effective methods is the moralization approach, which uses Bayes net algorithms to learn a directed graphical model, then converts to an undirected one. Previously the MLN parameters were then learned using current MLN methods. In this paper we extend the moralization approach to parameter learning in addition to structure learning, where MLN weights are directly inferred from Bayes net CP-table entries. Empirical evaluation indicates that this approach to parameter learning is orders of magnitude faster while performing as well or better on prediction metrics.
 @inproceedings{khosravi2012fastlearning,
                                        title={Fast parameter learning for markov logic networks using bayes nets},
                                        author={Khosravi, Hassan},
                                        booktitle={International Conference on Inductive Logic Programming},
                                        pages={102--115},
                                        year={2012},
                                        organization={Springer}
                                      }
                                    


Learning Graphical Models for Relational Data via Lattice Search

Journal Paper 2012
Oliver Schulte, Hassan Khosravi (2012). Journal of Machine Learning .
This paper is an extension on "Structure Learning for Markov Logic Networks with Many Descriptive Attributes". In this paper we introduce a lattice search framework to present an efficient new algorithm for learning a Parametrized Bayes Net that performs a level-wise search through the table joins for relational dependencies. The previous version did not use the lattice search framework. Our new version adds constraints on the model search, makes all constraints explicit, and provides rationales and discussion of each. We have also added more comparison with other Markov Logic Network learning methods (e.g., BUSL, LSM), with well known inductive logic programming methods and a lesion study that assesses the effects of using only part of the components of our main algorithm.
 @article{schulte2012learning,
                                        title={Learning graphical models for relational data via lattice search},
                                        author={Schulte, Oliver and Khosravi, Hassan},
                                        journal={Machine Learning},
                                        volume={88},
                                        number={3},
                                        pages={331--368},
                                        year={2012},
                                        publisher={Springer}
}
                                    


Learning Compact Markov Logic Networks With Decision Trees

Journal Paper 2012
Hassan Khosravi, Oliver Schulte, Jianfeng Hu, and Tianxiang Gao (2012) Journal of Machine Learning .
Statistical-relational learning combines logical syntax with probabilistic methods. Markov Logic Networks (MLNs) are a prominent model class that generalizes both first-order logic and undirected graphical models (Markov networks). The qualitative component of an MLN is a set of clauses and the quantitative component is a set of clause weights. Generative MLNs model the joint distribution of relationships and attributes. A state-of-the-art structure learning method is the moralization approach: learn a set of directed Horn clauses, then convert them to conjunctions to obtain MLN clauses. The directed clauses are learned using Bayes net methods. The moralization approach takes advantage of the high-quality inference algorithms for MLNs and their ability to handle cyclic dependencies. A weakness of the moralization approach is that it leads to an unnecessarily large number of clauses. In this paper we show that using decision trees to represent conditional probabilities in the Bayes net is an effective remedy that leads to much more compact MLN structures. In experiments on benchmark datasets, the decision trees reduce the number of clauses in the moralized MLN by a factor of 5-25, depending on the dataset. The accuracy of predictions is competitive with the unpruned model and in many cases superior.
 @article{khosravi2012learningcompact,
                                        title={Learning compact Markov logic networks with decision trees},
                                        author={Khosravi, Hassan and Schulte, Oliver and Hu, Jianfeng and Gao, Tianxiang},
                                        journal={Machine Learning},
                                        volume={89},
                                        number={3},
                                        pages={257--277},
                                        year={2012},
                                        publisher={Springer}
                                        }
                                    


Learning Directed Relational Models With Recursive Dependencies

Journal Paper 2012
Oliver Schulte, Hassan Khosravi, Tong Man(2012) Journal of Machine Learning .
Recently, there has been an increasing interest in generative models that represent probabilistic patterns over both links and attributes. A key characteristic of relational data is that the value of a predicate often depends on values of the same predicate for related entities. For directed graphical models, such recursive dependencies lead to cycles, which violates the acyclicity constraint of Bayes nets. In this paper we present a new approach to learning directed relational models which utilizes two key concepts: a pseudo likelihood measure that is well defined for recursive dependencies, and the notion of stratification from logic programming. An issue for modelling recursive dependencies with Bayes nets are redundant edges that increase the complexity of learning. We propose a new normal form format that removes the redundancy, and prove that assuming stratification, the normal form constraints involve no loss of modelling power. Empirical evaluation compares our approach to learning recursive dependencies with undirected models (Markov Logic Networks). The Bayes net approach is orders of magnitude faster, and learns more recursive dependencies, which lead to more accurate predictions. A preliminary version of this work was published in the proceeding of Inductive Logic programming 2011
 @article{schulte2012learningrecursive,
                                          title={Learning directed relational models with recursive dependencies},
                                          author={Schulte, Oliver and Khosravi, Hassan and Man, Tong},
                                          journal={Machine learning},
                                          volume={89},
                                          number={3},
                                          pages={299--316},
                                          year={2012},
                                          publisher={Springer}
                                        }
                                    


Directed Models For Statistical Relational Learning

Thesis 2012
Hassan Khosravi
Statistical Relational Learning is a new branch of machine learning that aims to model a joint distribution over relational data. Relational data consists of different types of objects where each object is characterized with a different set of attributes. The structure of relational data presents an opportunity for objects to carry additional information via their links and enables the model to show correlations among objects and their relationships. This dissertation focuses on learning graphical models for such data. Learning graphical models for relational data is much more challenging than learning graphical models for propositional data. One of the challenges of learning graphical models for relational data is that relational data, unlike propositional data, is non independent and identically distributed and cannot be viewed in a single table. Relational data can be modeled using a graph, where objects are the nodes and relationships between the objects are the edges. In this graph, there may be multiple edges between two nodes because objects may have different types of relationships with each other. The existence of multiple paths of different length among objects makes the learning procedure much harder than learning from a single table. We use a lattice search approach with lifted learning to deal with the multiple path problem. We focus on learning the structure of Markov Logic Networks, which are a first order extension of Markov Random Fields. Markov Logic Networks are a prominent undirected statical relational model that have achieved impressive performance on a variety of statistical relational learning tasks. Our approach combines the scalability and efficiency of learning in directed relational models, and the inference power and theoretical foundations of undirected relational models. We utilize an extension of Bayesian networks based on first order logic for learning class-level or first-order dependencies, which model the general database statistics over attributes of linked objects and their links. We then convert this model to a Markov Logic Network using the standard moralization procedure. Experimental results indicate that our methods are two orders of magnitude faster than, and predictive metrics are superior or competitive with, state-of-the-art Markov Logic Network learners.
 @phdthesis{khosravi2012phdthesis,
                                        title={Directed Models For Statistical Relational Learning},
                                        author={Khosravi, Hassan},
                                        year={2012},
                                        school={Simon Fraser University}
                                      }
                                    


Structure Learning for Markov Logic Networks with Many Descriptive Attributes

Conference Paper 2010
Hassan Khosravi Oliver Schulte, Tong Man, Xiaoyuan Xu, and Bahareh Bina (2010). Proceedings of the Twenty-Fourth Conference on Artificial Intelligence
Many machine learning applications that involve relational databases incorporate first-order logic and probability. Many of the current state-of-the-art algorithms for learning MLNs have focused on relatively small datasets with few descriptive attributes, where predicates are mostly binary and the main task is usually prediction of links between entities. This paper addresses what is in a sense a complementary problem: learning the structure of a graphical model that models the distribution of discrete descriptive attributes given the links between entities in a relational database. Descriptive attributes are usually non-binary and can be very informative, but they increase the search space of possible candidate clauses. We present an efficient new algorithm for learning a directed relational model (parametrized Bayes net). From the Bayes net we obtain an MLN structure via a standard moralization procedure for converting directed models to undirected models. Learning MLN structure by moralization is 200-1000 times faster and scores substantially higher in predictive accuracy than benchmark MLN algorithms on three relational databases.
 @inproceedings{Khosravi2010structure,
                                        author = {Khosravi, Hassan and Schulte, Oliver and Man, Tong and Xu, Xiaoyuan and Bina, Bahareh},
                                        title = {Structure Learning for Markov Logic Networks with Many Descriptive Attributes},
                                        booktitle = {Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence},
                                        series = {AAAI'10},
                                        year = {2010},
                                        location = {Atlanta, Georgia},
                                        pages = {487--493},
                                        numpages = {7},
                                        url = {http://dl.acm.org/citation.cfm?id=2898607.2898686},
                                        acmid = {2898686},
                                        publisher = {AAAI Press},
                                        } 
                                    


A Survey on Statistical Relational Learning

Conference Paper 2010
Hassan Khosravi, and Bahareh Bina (2010) In Proceedings of Canadian Conference of AI
The vast majority of work in Machine Learning has focused on propositional data which is assumed to be identically and independently distributed, however, many real world datasets are relational and most real world applications are characterized by the presence of uncertainty and complex relational structure where the data distribution is neither identical nor independent. An emerging research area, Statistical Relational Learning (SRL), attempts to represent, model, and learn in relational domain. Currently, SRL is still at a primitive stage in Canada, which motivates us to conduct this survey as an attempt to raise more attention to this ?eld. Our survey presents a brief introduction to SRL and a comparison with conventional learning approaches. In this survey we review four SRL models as an attempt to raise more attention to this field. Our survey presents a brief introduction to SRL and a comparison with conventional learning approaches. In this survey we review four SRL models (PRMs, MLNs, RDNs, and BLPs) and compare them theoretically with respect to their representation, structure learning, parameter learning, and inference methods. We conclude with a discussion on limitations of current methods
 @inproceedings{khosravi2010survey,
                                        title={A Survey on Statistical Relational Learning},
                                        author={Khosravi, Hassan and Bina, Bahareh},
                                        booktitle={Advances in Artificial Intelligence: 23rd Canadian Conference on Artificial Intelligence, Canadian AI 2010, Ottawa, Canada, May 31-June 2, 2010, Proceedings},
                                        volume={6085},
                                        pages={256},
                                        year={2010},
                                        organization={Springer}
                                      }
                                    


The Imap Hybrid Method for Learning Gaussian Bayes Nets

Conference Paper 2010 Best paper award
Oliver Schulte, Gustavo Frigo, Russell Greiner, Hassan Khosravi (2010) In Proceedings of Canadian Conference of AI
This paper presents the I-map hybrid algorithm for selecting, given a data sample, a linear Gaussian model whose structure is a directed graph. The algorithm performs a local search for a model that meets the following criteria: (1) The Markov blankets in the model should be consistent with dependency information from statistical tests. (2) Minimize the number of edges subject to the ?rst constraint. (3) Maximize a given score function subject to the ?rst two constraints. Our local search is based on Graph Equivalence Search (GES); we also apply the recently developed SIN statistical testing strategy to help avoid local minima. Simulation studies with GES search and the BIC score provide evidence that for nets with 10 or more variables, the hybrid method selects simpler graphs whose structure is closer to the target graph.
 @inproceedings{schulte2010imap,
                                        title={The IMAP hybrid method for learning Gaussian Bayes nets},
                                        author={Schulte, Oliver and Frigo, Gustavo and Greiner, Russell and Khosravi, Hassan},
                                        booktitle={Canadian Conference on Artificial Intelligence},
                                        pages={123--134},
                                        year={2010},
                                        organization={Springer}
                                      }
                                    


A social network model of investment behaviour in the stock market

Journal Paper 2010
L Bakker, W Hare, Hassan Khosravi, B Ramadanovic (2010) Physica A: Statistical Mechanics and its Applications
This paper presents the I-map hybrid algorithm for selecting, given a data sample, a linear Gaussian model whose structure is a directed graph. The algorithm performs a local search for a model that meets the following criteria: (1) The Markov blankets in the model should be consistent with dependency information from statistical tests. (2) Minimize the number of edges subject to the ?rst constraint. (3) Maximize a given score function subject to the ?rst two constraints. Our local search is based on Graph Equivalence Search (GES); we also apply the recently developed SIN statistical testing strategy to help avoid local minima. Simulation studies with GES search and the BIC score provide evidence that for nets with 10 or more variables, the hybrid method selects simpler graphs whose structure is closer to the target graph.
 @article{bakker2010social,
                                        title={A social network model of investment behaviour in the stock market},
                                        author={Bakker, Laurens and Hare, Warren and Khosravi, Hassan and Ramadanovic, Bojan},
                                        journal={Physica A: Statistical Mechanics and its Applications},
                                        volume={389},
                                        number={6},
                                        pages={1223--1229},
                                        year={2010},
                                        publisher={Elsevier}
                                      }
                                    


Class-Level Bayes Nets for Relational Data

Report 2009
Oliver Schulte, Hassan Khosravi, Flavia Moser, Martin Ester
This paper presents the I-map hybrid algorithm for selecting, given a data sample, a linear Gaussian model whose structure is a directed graph. The algorithm performs a local search for a model that meets the following criteria: (1) The Markov blankets in the model should be consistent with dependency information from statistical tests. (2) Minimize the number of edges subject to the ?rst constraint. (3) Maximize a given score function subject to the ?rst two constraints. Our local search is based on Graph Equivalence Search (GES); we also apply the recently developed SIN statistical testing strategy to help avoid local minima. Simulation studies with GES search and the BIC score provide evidence that for nets with 10 or more variables, the hybrid method selects simpler graphs whose structure is closer to the target graph.
 @article{schulte2009class,
                                          author    = {Oliver Schulte and
                                                      Hassan Khosravi and
                                                      Flavia Moser and
                                                      Martin Ester},
                                          title     = {Join Bayes Nets: {A} new type of Bayes net for relational data},
                                          journal   = {CoRR},
                                          volume    = {abs/0811.4458},
                                          year      = {2008},
                                          url       = {http://arxiv.org/abs/0811.4458},
                                          timestamp = {Wed, 07 Jun 2017 14:40:54 +0200},
                                          biburl    = {http://dblp.uni-trier.de/rec/bib/journals/corr/abs-0811-4458},
                                          bibsource = {dblp computer science bibliography, http://dblp.org}
                                        }
                                    


LNBC: A Link-Based Naive Bayes Classifier

Conference Paper 2009
Bahareh Bina, Oliver Schulte, Hassan Khosravi (2009) IEEE International Conference on Data Mining Workshops
Many databases store data in relational format, with different types of entities and information about links between the entities. Link-based classification is the problem of predicting the class label of a target entity given information about features of the entity and about features of the related entities. A natural approach to link-based classification is to upgrade standard classification methods from the propositional, single-table testing. In this paper we propose a new classification rule for upgrading naive Bayes classifiers (NBC). Previous work on relational NBC has achieved the best results with link independency assumption which says that the probability of each link to an object is independent from the other links to the object. We formalize our method by breaking it into two parts: (1) the independent influence assumption: that the influence of one path from the target object to a related entity is independent of another. We consider object-path independency and (2) the independent feature assumption of NBC: that features of the target entity and a related entity are probabilistically independent given a target class label. We derive a new relational NBC rule that places more weight on the target entity features than formulations of the link independency assumption. The new NBC rule yields higher accuracies on three benchmark datasets-Mutagenesis, MovieLens, and Cora-with average improvements ranging from 2% to 10%
 @inproceedings{bina2009lnbc,
                                        title={LNBC: A Link-Based Naive Bayes Classifier},
                                        author={Bina, Bahareh and Schulte, Oliver and Khosravi, Hassan},
                                        booktitle={Data Mining Workshops, 2009. ICDMW'09. IEEE International Conference on},
                                        pages={489--494},
                                        year={2009},
                                        organization={IEEE}
                                      }
                                                                          


A New Hybrid Method for Bayesian Network Learning With Dependency Constraints

Conference Paper 2009
Oliver Schulte, Gustavo Frigo, Russell Greiner, Wei Luo, Hassan Khosravi (2009). Proceedings IEEE CIDM Symposium
A Bayes net has qualitative and quantitative aspects: The qualitative aspect is its graphical structure that corresponds to correlations among the variables in the Bayes net. The quantitative aspects are the net parameters. This paper develops a hybrid criterion for learning Bayes net structures that is based on both aspects. We combine model selection criteria measuring data fit with correlation information from statistical tests. We rely on the statistical test only to accept conditional dependencies, not conditional independencies. We show how to adapt local search algorithms to accommodate the observed dependencies. Simulation studies with GES search and the BDeu/BIC scores provide evidence that the additional dependency information leads to Bayes nets that better fit the target model in distribution and structure.
 @inproceedings{schulte2009hybrid,
                                          title={A new hybrid method for Bayesian network learning with dependency constraints},
                                          author={Schulte, Oliver and Frigo, Gustavo and Greiner, Russell and Luo, Wei and Khosravi, Hassan},
                                          booktitle={Computational Intelligence and Data Mining, 2009. CIDM'09. IEEE Symposium on},
                                          pages={53--60},
                                          year={2009},
                                          organization={IEEE}
                                        }
                                  


Exploratory Analysis of Co-Change Graphs for Code Refactoring

Conference Paper 2009
Hassan Khosravi and Recep Colak (2008). In Proceedings of Canadian conference of AI
A Bayes net has qualitative and quantitative aspects: The qualitative aspect is its graphical structure that corresponds to correlations among the variables in the Bayes net. The quantitative aspects are the net parameters. This paper develops a hybrid criterion for learning Bayes net structures that is based on both aspects. We combine model selection criteria measuring data fit with correlation information from statistical tests. We rely on the statistical test only to accept conditional dependencies, not conditional independencies. We show how to adapt local search algorithms to accommodate the observed dependencies. Simulation studies with GES search and the BDeu/BIC scores provide evidence that the additional dependency information leads to Bayes nets that better fit the target model in distribution and structure.
 @inproceedings{khosravi2009exploratory,
                                        title={Exploratory Analysis of Co-Change Graphs for Code Refactoring},
                                        author={Khosravi, Hassan and Colak, Recep},
                                        booktitle={Advances in Artificial Intelligence: 22nd Canadian Conference on Artificial Intelligence, Canadian AI 2009, Kelowna, Canada, May 25-27, 2009 Proceedings},
                                        volume={5549},
                                        pages={219},
                                        year={2009},
                                        organization={Springer}
                                      }
                                  


TACtic- A Multi Behavioral Agent for Trading Agent Competition

Conference Paper 2008
Hassan Khosravi M. E.shiri, Hamid. Khosravi, E. Iranmanesh A. Davoodi (2008). In Advances in Computer Science and Engineering, Communications in Computer and Information Science
Software agents are increasingly being used to represent humans in online auctions. Such agents have the advantages of being able to systematically monitor a wide variety of auctions and then make rapid decisions about what bids to place in what auctions. They can do this continuously and repetitively without losing concentration. To provide a means of evaluating and comparing (benchmarking) research methods in this area the trading agent competition (TAC) was established. This paper describes the design, of TACtic. Our agent uses multi behavioral techniques at the heart of its decision making to make bidding decisions in the face of uncertainty, to make predictions about the likely outcomes of auctions, and to alter the agent's bidding strategy in response to the prevailing market conditions.
 @incollection{khosravi2008tactic,
                                          title={TACtic-A Multi Behavioral Agent for Trading Agent Competition},
                                          author={Khosravi, Hassan and Shiri, Mohammad E and Khosravi, Hamid and Iranmanesh, Ehsan and Davoodi, Alireza},
                                          booktitle={Advances in Computer Science and Engineering},
                                          pages={811--815},
                                          year={2008},
                                          publisher={Springer}
                                        }
                                  


An Automated Negotiation Technique for Self-interest Agents

Conference Paper 2006
Hassan Khosravi, Mohammad Shiri, Hamid Khosravi, Ehsan Iranmanesh (2006) 12th International CSI Computer Conference
Due to the rapid growth of electronic environments, much research is currently being performed on autonomous trading mechanisms. Also in the last few years there has been an increasing interest from the agent community in the use of techniques from decision theory and game theory. Our paper connects theses two fields. In real world, when two agents negotiate, it seems rational that they emphasize on their highest priorities first. We give a strategyin which an agent can gain more profit by sacrificing or delaying some high priority tasks. The focus of this paper is on negotiation of self interest agents.