A Framework to Identify Non-Achievers in eLearning Business Informatics Lab Courses

: Learning Management Systems store valuable data in their repositories. Analyzing such data could contribute to identifying non-achievers in eLearning courses. This study presents an integrated framework (encompassing stages from a risk management process) to predict non-achievers in eLearning Business Informatics Lab Courses through a discriminant function analysis of student engagement data. In detail, data regarding with student interaction with the learning activities were elicited from the Moodle LMS log files. The paper also presents a specific eLearning Business Informatics Lab Course, designed upon Business Informatics competencies, tailored to a Business Informatics Curriculum for undergraduate Accounting Students. A discriminant function analysis was used to develop a competent prediction model. Linear discriminant functions were generated for achievers and non-achievers respectively. Students were classified into non-achievers or achievers according to the maximum score of the discriminant functions. The high classification percentage of our model indicates that our framework could be used to identify non-achievers in any eLearning Business Informatics Lab course sharing the same structure. A linear discriminant analysis (LDA) was also employed to indicate the training potential of our model. The evaluation metrics of our trained model indicate that our model could potentially be used to develop an alert system.


Introduction
Business Informatics is a discipline, designed upon IT skills and Business Competencies (Zabukovšek et al., 2020;Zakopoulos, 2021).Business Informatics courses aim at equipping all entities involved in an organization to solve enterprise problems.Therefore, Business Informatics competencies play a cardinal role in a business sustainability Zabukovšek et al., 2020;Tumbas et al., 2019;Zakopoulos, 2021;Zakopoulos et al., 2022).
In general, competences and skills in Business Informatics encompass (Helfert, 2007;Seres et al., 2017): • Understanding and evaluating the business process.
• Analyzing and applying scientific knowledge in a business context.
• Creating added value through innovation and software development.
• Managing operational processes and utilizing software technology.

•
Understanding strategy and implementing technology in e-business.
In this spirit, most Business Informatics courses are designed upon the principles of Informatics, Statistics, Finance and Business Administration (Korczak et al., 2011;Kuz'mina et al., 2020;Martz et al., 2011;Paul et al., 2017;Pavlicevic et al., 2015;Tumbas et al., 2019;Zabukovšek et al., 2020).Such courses are designed to provide students with the requisite theoretical framework in the previously mentioned fields, helping them to apply the respective knowledge to solve real enterprise problems (Kuz'mina et al., 2020;Korczak et al., 2011;Martz et al., 2011;Pavlicevic et al., 2015;Tumbas et al., 2019;Zakopoulos, 2022).Getting perspective on things, case studies drawn from real enterprise experiences, and project simulations are transformed into learning activities (Bower et al., 2010).Therefore, Business Informatics courses include theoretical and practical activities.These activities are designed to promote critical thinking, problem-solving, and communication skills (Bower et al., 2010).eLearning Business Informatics courses implement such activities on Learning Management Systems (Zakopoulos, 2022).
Various methods are employed in the learning process, such as collaborative assignments, blended learning, and eportfolios (Garrison & Kanuka, 2004;Laurillard, 2002).Business Informatics lab courses focus on practical skills without brushing theoretical knowledge aside.Typical Business Informatics Lab courses familiarize students with Business Informatics tools (Morozov et al., 2019;Paul et al., 2017;Pavlicevic et al., 2015;Tumbas et al., 2019).Hence, specific learning activities are created to familiarize students with spreadsheets, programming languages, visualization tools, simulation tools, and Management Information Systems (Morozov et al., 2019;Tumbas et al., 2019).In parallel, students are given insight into the disciplines related to these tools.Therefore, statistical, administrative, financial, accounting, and informatics fundamentals stand out in the structure of the Business Informatics Lab Courses.
Generally, in the case of Universities, Business Informatics Lab Courses comply with the standards dictated by the Curricula.In this sense, the tools presented in these courses also vary according to each school student's needs.To ensure that the entire course will be tailored to the industry's needs, some Universities involve Industries and organizations in the design process.Such courses revolve around project management, business intelligence, and digital transformation (Buřita & Rosman, 2014;Helfert, 2011;Helfert & Duncan, 2005;Klašnja-Milićević et al., 2019;Suša, 2014) .Moreover, since Business Informatics Courses aspire to foster innovation, artificial intelligence, and machine learning are gradually incorporated into the new Business Informatics curricula (Klašnja-Milićević et al., 2019).
Since, Business Informatics courses taught in universities are designed to meet educational goals, the risk of student failure should be controlled.Moreover, Business Informatics courses involve informatics competencies that can help any professional.In this sense, to assess digital competencies is of outmost importance.Therefore, nonachievers should be early predicted to avoid a possible dropout in a Business Informatics course (Casanova et al., 2021;Ortiz-Lozano et al., 2018;Singell & Waddell, 2010).
Non-achievement denotes academic failure regarding educational objectives.In some studies, lack of motivation and engagement are reported to affect student achievement in universities.Therefore, focusing on student effort (behavioral engagement) can contribute to identifying educational academic factors related to non-achievers (Kuh et al., 201;Kuh., 2005;Jia & Maloney, 2015;Mäkinen et al., 2004;Sarra et al., 2019;Seidman, 2012).
Non-achievement critically impacts the entire educational foundation.First, non-achievement can curb student academic and professional prospects.Second, the lack of academic success can cause temporary or even permanent psychological disorders in students.Moreover, students who try hard to meet educational goals suffer from intensive stress that could lead to a possible dropout (Kehm et al., 2019;Pascarella & Terenzini, 2005;Srairi, 2022).
Furthermore, the lack of professional prospects can bring about poverty, doing huge economic damage to students and their families, who will strive to support them.In parallel, non-achievers cannot offer their professional services to the State, increasing the unemployment rate, and forcing the government to back them up financially (Latif et al., 2015).
In a nutshell, academic failure has social, psychological, and economic implications (Aina et al., 2022;Stăiculescu & Ramona, 2018).Hence, identifying non-achievers, and taking remedial action is of utmost importance.To answer this purpose, many studies refer to competent warning systems that predict non-achievers considering student engagement.Statistical and machinelearning methods can be used to develop efficient warning systems.In e-learning, measurable data can be elicited from the Learning Management Systems.However, this is not the case in conventional teaching.Although many warning systems are reported in the literature, the development process has not been based on a concrete framework.
This study presents a Business Informatics Lab course designed to cover the needs of an Accounting School at a Greek University (University of West Attica).Additionally, the study demonstrates a framework to identify nonachievers in the specific Business Informatics Lab course through a proper analysis of student behavioral engagement data.This framework could be used to achieve the same goal in each Business Informatics Lab course, sharing the same structure.It is essential to underline that our study contributes to the educational field in the following ways: 1.It accentuates the need for predicting non-achievers.
2. It proposes a robust framework (encompassing the stages of risk analysis and risk control) that can be used to predict non-achievers.
3. It proves that student engagement is a strong predictor of student final achievement.
4. It shows (through a machine learning process) an efficient way to generate competent predictive models with high training potential.
Nevertheless, a few educational forecast models are reported in the case of e-learning courses (Finnegan et al., 2005;Macfadyen & Dawson, 2010;Ya-Han Hu et al., 2014).In detail, one study proved that student behavioral engagement is a strong predictor of students' attrition in elearning courses (Morris et al., 2005).In parallel, another study indicates that non-achievers could be monitored and predicted by analyzing LMS engagement data whereas another study proposes the analysis of e-Portfolio data to yield the same result (Ya-Han Hu et al., 2014).
In the case of eLearning courses, this interaction is determined by the completion of specific learning activities.Students who fail to complete the requisite learning activities are non-achievers and represent students who fail the course.Since these activities are implemented through Learning Management Systems, analyzing the data stored in their log files could lead to measurable factors.
There are some endeavors to predict non-achievers in Business Informatics Lab courses, taking advantage of the capabilities of predictive analytics.Predictive analytics uses historical data and machine learning algorithms to predict future outcomes, including learner performance.In the context of e-learning lab courses, predictive analytics can identify students at risk by analyzing various parameters such as previous academic performance, level of engagement and interaction with course material (Richardson et al. 2012;DeBerard et al., 2004;Jia & Maloney, 2015;Liz-Domínguez et al., 2019).In a case study by Smith et al. (2018), predictive analytics was applied to an e-learning business IT lab course to identify low achieving students.By analyzing data from previous cohorts, including grades, assignment submissions, and online activities, a predictive model was developed to predict student performance.The model identified students at risk of failure early in the course, allowing educators to intervene with individualized support strategies.In parallel, a study by Rakic et al. (2018) used behavior analysis to identify non-learners in an experimental e-learning business informatics course.By analyzing patterns of learner interaction on the e-learning platform, such as clickstream data and time spent on tasks, the study identified learners who showed signs of poor learning motivation and difficulty.
The studies mentioned above have proved that risk drivers are course oriented.Nevertheless, similar predictors of student failure are reported for courses with the same learning design.In most of studies, the results are based on various independent analyses without presenting an integrated framework (Alam & Mohanthy, 2022;Aleksandrova, 2019;Büyüköztürk & Çokluk-Bökeoğlu, 2008;Fauszt et al., 2023;Kabathova et al., 2021;Lourens & Bleazard, 2016;Macfadyen & Dawson, 2010).In this study, a robust framework is developed to support our analysis.In detail, our method encompasses stages from a generic risk management process (Georgakopoulos et al., 2018;Vose, 2008).In this sense, our method can be used in many Business Informatics Lab courses.However, the dataset could vary among courses.
Our Business Informatics Lab course is demonstrated in section 2.1.Sections 3 and 4 focus attention on the proposed framework and the results.It is essential to underline that although our main research interest is to predict non-achievers, our research investigates the effect of student engagement in predicting student final achievement, since the studies mentioned above have revealed the student engagement predicting potential.Therefore, our research assumption could be shaped as follows: Student interaction with the learning activities on Moodle is a strong predictor of non-achievers in our Business Informatics lab course.

Our business informatics lab
Our Business Informatics lab is delivered at the Accounting department, at the University of West Attica, tailored to a Business Informatics curriculum for undergraduate students.The respective lab course combines theory with practice.The theory consists of the following activities implemented through Moodle: 1. Slides (PDF) on fundamental Business Informatics concepts (Enterprise scenarios, Business Informatics Tools, Business Statistics, Risk Management, Business Administration, Accounting and Finance, and Management Informatics Systems).
3. Appropriate videos (links on YouTube) on specific Business Informatics Tools.
The practical part consists of the following activities, also implemented through Moodle: 1.A set of exercises (enterprise scenarios) on Excel and Statistical Packages (SPSS), devoted to student practice.
2. A small-scale enterprise problem that is assigned to students.It constitutes a collaborative assignment that can be done using an appropriate Business Informatics Tool and uploaded to Moodle before the final online test.
The final online test includes theoretical questions (Multiple Choice or True/ False) that should be answered on Moodle.The test score is 60 % of the final grade, whereas the practical exercise (assignment) makes up 40 % of the final grade.
The self-assessment quizzes are completed if the maximum score is greater than 5. Three (tries) are allowed.Therefore, these quizzes do not contribute to the final grade, but they familiarize students with theoretical and practical aspects.Non-achievers are students who get a final grade below 5.
Comparing our lab course to similar Business Informatics courses taught in Universities, we can deduce that our lab course complies with the standards of a typical Business Informatics course design since it fosters problem-solving, and collaboration.In parallel, our course familiarizes students with specific Accounting Business Informatics tools (such as Excel).Moreover, electronic assessment methods are used to evaluate students.Moodle LMS (an electronic platform) is used to help students take in the rudimentary knowledge, through well-designed and orchestrated activities.
However, our course is not connected to the Industry needs like similar courses delivered in universities (Buřita & Rosman, 2014;Helfert, 2011;Helfert & Duncan, 2005;Klašnja-Milićević et al., 2019;Suša, 2014).Although, the course presents the capabilities of Business Informatics tools, it does not present Professional Accounting tools.
In parallel, the assessment process revolves around examining specific learning goals related to business informatics competencies, but these competencies are not linked to occupational needs of students.Moreover, the competencies are assessed in light of theoretical knowledge and practical skills, without students being categorized according to a specific level of competencies' development.This is because the entire course is designed in the philosophy of the "encyclopedia", in the context of which many curricula provide fundamental knowledge, and not focus on market needs.

Methodology
Our framework encompasses stages from a general Risk Management Process, executed in the following steps depicted in Figure 1 (Georgakopoulos et al., 2018;Vose, 2008).Two distinct phases stand out in a typical risk management framework, risk analysis and risk control.The risk analysis process revolves around identifying factors that contribute to the risk occurrence within the risk area.Efficient risk models are built using qualitative and quantitative techniques to identify these factors (Vose, 2008).Since, student engagement affects student performance (as indicated in many studies), in our case, the risk area, in which risks factors will be identified is student engagement.In parallel, risk models are validated to generate prediction models based on risk factors.The validation outcome leads to an efficient warning system.The enactment of the warning system achieves risk control (Vose, 2008).
However, in our case, the objective is not to identify the risk factors, but to predict non-achievers.In this spirit, the stage of building the risk model can be omitted.Therefore, qualitative, or quantitative methods should be employed to generate an efficient prediction model.In most cases, the selection of the method depends on the data type (Vose, 2008).In our case, a quantitative data set is elicited from the log files of Moodle, and thus quantitative methods can be used.In this sense, non-achievers can be identified after the enactment of the prediction model.

Figure 1 The proposed framework
As shown in Figure 1, the first step includes a data collection process.Data regarding all activities implemented through Moodle are collected using the appropriate Learning Analytics Tool.The second step is devoted to generating the prediction Model by applying a competent statistical or machine-learning method to the collected data set.The quantitative collected data set is appropriately formatted to comply with the method used.The prediction model will classify students into achievers or not achievers.
In our case, the risk control process is also omitted since our research objective is to predict non-achievers, and not to develop a remedial strategy.Nevertheless, the prediction model could be applied to many courses, and after being validated, it can lead to an early alert system, through which risk control will be fully achieved.

Applying our framework
Our framework was applied to a specific Business Informatics Lab course, delivered at the University of West Attica in the "Accounting and Finance" Department.All activities were implemented through Moodle.The data collected are: 1. Total number of slides (PDF) viewed 2. Total number of self-assessment exercises (quizzes) completed.
3. Total number of practical exercises completed.4. Total number of YouTube videos watched. 5. Final score on the assignment.6. Final score on the final online test.7. Total logins to Moodle.300 Accounting students participated in the final online test, constituting our sample.The course was mandatory, and therefore, all registered students took the final exams.In this sense, no students were excluded from our research, denoting that no outliers were detected in our sample.Since, our main interest was to predict student final achievement regardless of demographic features, no specific sampling method was used, and all participants in the final online test were included in our research.The grades were collected after the final online test, whereas the rest of the data were collected one week before the final online test.Variables reflecting the data set collected were created.In parallel, the binary variable "achr" was modeled to reflect student performance (Georgakopoulos et al., 2018;Macfadyen & Dawson, 2010).The value "1" indicated nonachievers, whereas "0" indicated students who passed the course.The values were calculated according to the student's final grade (non-achievers got a final grade below 5).
A linear discriminant function analysis was employed.A competent prediction model was developed, to classify students into achievers and non-achievers (Georgakopoulos et al., 2018;Izenman, 2013;Manly et al., 2002;Xanthopoulos et al, 2013;Ye et al., 2004).In detail, two discriminant functions were generated (a function for nonachievers (state 1), and a function for achievers (state 0)).After the final online test, the classification functions' scores were calculated in a student-oriented approach.Therefore, students were classified into achievers if the discriminant function for achievers had the maximum score.Students were classified into non-achievers in any other case (Izenman, 2013;Manly et al., 2002;Xanthopoulos et al, 2013).The following section (section 4) focuses on the prediction model and the entire process.
The prediction model was generated by means of competent statistical packages (SPSS/JASP).The variables analyzed are listed in the following SPSS instance (Image 1).
In detail, the variable "grade" reflected the final grade (constituting by the assignment grade and the final online test grade).The variable "achr" was modeled to determine achievers (state 0, grade>5) and non-achievers (state 1, grade<5).Therefore, the variable "achr" was the dependent variable, and all other variables were the independent variables (coefficients) in the discriminant function analysis scheme.
Image 1 SPSS Variables view

Results
The coefficients of the linear discriminant functions for achievers and non-achievers are shown in Table 1.Table 1 shows the classification functions' coefficients for achievers and non-achievers respectively (states 0, and 1).
As shown in Table 2, the prediction model achieves a 94.7 % correct classification percentage, indicating that only a small number of cases is not correctly classified.The values of the discriminant functions are replaced by the respective means (z-scores), and the density of the columns reflects the number of students classified to achievers or non-achievers, providing a graphical illustration of the classification table (Izenman, 2013;Manly et al., 2002;Xanthopoulos et al, 2013;Ye et al., 2004).As shown in Figure 2, our model works better in case of achievers.Nevertheless, the high classification percentage shown in Table 2 is a proof that our model works sufficiently in all cases.
Although our model is course-dependent, a linear discriminant analysis (LDA) proved that it can be trained with promising results as shown in Table 3: Table 3 Training Performance Characteristics: Therefore, our model can be trained to achieve 87.8 % accuracy if applied to many courses.Figure 4 (ROC Curves Plot) depicts the training potential of our model indicating the true and false positive rate (Izenman, 2013;Manly et al., 2002;Xanthopoulos et al, 2013;Ye et al., 2004).

Figure 4 ROC Curves Plot: True and False Positive Rate
The ROC-Curves plot indicates the sensitivity (True Positive Rate), and the specificity (False Positive Rate) of our trained model, indicating the number of real cases that are correctly classified, and the number of real cases that are incorrectly classified (Izenman, 2013;Manly et al., 2002;Xanthopoulos et al, 2013;Ye et al., 2004).
Table 4 explains the ROC Curves plot:

Table 4 Evaluation Metrics
As shown in Table 4, the true Positive Rate is 0.878, and the False Positive Rate is 0.5.Looking at the other evaluating metrics (shown in Table 4), it appears that our trained model works better in case of achievers.However, it seems that our trained prediction model works sufficiently in all cases.

Discussion
Our prediction model achieves a high classification percentage (Table 2).Figures 3 and 4 indicate that our model works sufficiently in all cases.Our model's accuracy is course-dependent.That finding is supported by the literature where the prediction models reported are also course-dependent (Macfadyen & Dawson, 2010;Georgakopoulos et al., 2018).However, our model could be trained with promising results, as shown in Table 3.The evaluation metrics in Table 4 indicate that our trained model works sufficiently in all cases, and therefore it can be applied to many courses.
However, the possibility that our model will work better in courses with similar structure cannot be ruled out.Our model has not been tested in various courses to ensure its accuracy.In this sense, our model should be validated being applied to many different courses before it constitutes the basis on which a prediction model can be developed.Therefore, the intrinsic characteristic of prediction models, that are course-dependent is the main limitation of our study, denoting that the results cannot be easily generalized.
In parallel, if our model is applied to different courses, the possibility of emerging risk factors (coefficients of the model) will not be ruled out.In this case, the classification potential of our model could vary among courses.This is also a finding supported by literature (Anagnostopoulos et al., 2020;Macfadyen & Dawson, 2010;Zakopoulos, 2022).
Moreover, the data used in the linear discriminant function analysis are calculated after the final online test.Therefore, our model predicts non-achievers after the final exams, indicating another limitation of our study.Hence, it constitutes a prediction model after the first course run, a finding also mentioned in some studies (Georgakopoulos et al., 2018;Georgakopoulos & Tsakirtzis, 2021).However, the training potential of our model proves that our model could constitute the basis of an early warning system on the condition that the discriminant functions will be calculated before the final exams to facilitate remedial action.
At this point, it should be noted that achievers and non-achievers are identified according to their final grade.Therefore, the competencies of achievers are reflected in their grade.However, Business Informatics Labs are designed upon the principles of Business Informatics competencies.A more efficient way to assess the students' competencies might be needed.Therefore, another essential limitation of our study is that the assessment of the course success is not implemented considering students' competencies, since the student failure is only related to a negative final achievement (reflected by low grades).However, a sufficient student performance could indicate that a student has developed the requisite competencies.In parallel, a risk model (built before the prediction model) could reveal risk drivers, focusing on competencies that have not been developed in the case of non-achievers.
From an educational perspective, the students' interaction with self-assessment quizzes, practical exercises, and interactive learning material was among the predictors of student performance in our study, a finding that is also supported by many studies in literature, especially in the case of e-learning courses.Therefore, student interaction with the learning activities on Moodle proved to be a strong predictor of students' final achievement, verifying our research assumption.In parallel, predictors of student achievement stemmed from both theoretical and practical part of the course, a finding that is also reported in a specific study.Moreover, the e-learning part (reflecting students' interaction with learning activities on Moodle) proved to play a cardinal role in predicting student critical achievement, a finding that is aligned with some studies in literature (Rakic et al., 2018;Zakopoulos et al., 2021;Zakopoulos, 2022).
In our case, the training potential of our model is high, indicating that our model could constitute the pillar of an early warning system, accentuating our model's dynamics, a finding that is also reported in some studies based on predictive analytics (Izenman, 2013;Jia & Maloney, 2015;Manly et al., 2002;Smith et al., 2018;Xanthopoulos et al, 2013;Ye et al., 2004).

Conclusions and Future Expectations
The paper demonstrates a concrete framework to identify non-achievers that encompasses stages from a generic risk analysis process.Therefore, our study does not address the issue of students at risk in a fragmented way, but the identification of non-achievers is well-grounded.In parallel, the paper presents a specific Business Informatics Course, designed upon the Business Informatics competencies.The results proved that student engagement (reflected by the students' interaction with the theoretical material and practical exercises) proved to be a decisive factor in our prediction model, denoting that a proper analysis of student engagement data could lead to efficient prediction models.Our model could be applied to any Business Informatics Course to identify non-achievers after the first course-run.
Our team is currently working on validating our model in terms of different courses to develop an early warning system for non-achievers.In addition, our team is working on applying our framework in courses delivered in the context of face-to-face teaching to indicate emerging risk factors and a possible change in the accuracy maximum percentage.

Figure 2 .
Figure 2., and Figure 3., graphically illustrates the classification results for achievers and non-achievers respectively.
Figure 2 Classification Outcome for achievers