GENERATIVE AI IN ENGINEERING AND COMPUTING EDUCATION: A SCOPING REVIEW OF EMPIRICAL STUDIES AND EDUCATIONAL PRACTICES

A. Background / Context Since the release of ChatGPT, a pre-trained text generation language model by OpenAI on November 30, 2022, much debate has emerged surrounding the implications and impacts of these kinds of generative AI (GenAI) tools could bring into education. Quickly, other big tech companies such as Google or Microsoft have joined in creating their own GenAI products such as Gemini, GitHub Copilot, Visual Studio, Open AI Codex, or Azure AI. As pointed out by Chan et al., the term “generative” refers to the capability of AI to produce novel outputs rather than simply reproducing, categorizing, processing, and analyzing inputs. These outputs can come from several sources, including not only text but audio, video, and images, or in the case of engineering and computer science, programming code or design assistance. Similar definitions emphasize the capability of GenAI to create new and unique content, starting from pre-trained data. At the technical level, GenAI works based on Large Language Models (LLMs) such as Generative Pretrained Transformer (GPT-4) or (GPT-5), Gemini, Claude, Or LLaMa 2. These models are sophisticated AI systems that can process and generate human-like language responses, working through prompt engineering. The idea behind prompt engineering is to enable computers to interpret and understand human language and it has a close relationship with Natural Language Processing (NLP), which is an Artificial Intelligence field that examines how computer systems can interpret and control the natural language such as text or speech. B. Problem Statement Beyond the innovation obsession to use GenAI in educational institutions and their promoters or detractors, we have noticed a gap in the current literature in describing how engineering and computing Educators are integrating GenAI in their courses, or what educational outcomes students have developed. In addition, we observed a plethora of student-centered proposals during the review, but not teachercentered ones that show their perceptions or initiatives. Therefore, there is a need for an integrated, evidence-based review that can assist educators and researchers in designing comprehensive strategies that encompass both technical skills and AI literacy. C. Objective / Aim Our objectives with this research are the following: (1) Identify the overall features of the studies in terms of bibliometric trends, such as cited articles, articles per year, leading countries, journals, authors, and cluster topics. (2) Inform researchers and/or Educators about the educational features of the studies identified, including features such as type of intervention, participants, research approach, or main learning outcomes. (3) Expose educational strategies elicited by the researchers utilizing GenAI. And (4) Expose issues, challenges and improvement opportunities for GenAI in engineering and computing education based on the evidence. D. Significance / Contribution These features of GenAI have drawn the attention of Educators and foremost Higher Education Institutions (HEIs), apparently revolutionizing the way to teach and learn and providing more educational tools to the students to complement their formation. Among the advantages of GenAI for education are highlighted instant feedback, personalized learning, multilingual support, explaining complex concepts, and enhanced student engagement. This study introduces a unified scoping review that maps out how educators are leveraging these capabilities while documenting the associated risks and learning outcomes across various educational tiers. E. Paper Organization The remainder of this paper presents related work, the proposed system and methodology, implementation design, evaluation, limitations, future scope, and conclusion.

A. Overview of Existing Research We detected several related literature reviews and experiential studies concerning the scope of engineering, computing, and education, indicating mainly GenAI’s affordances, challenges, and opportunities. Crompton and Burke identified the affordances of ChatGPT for education based on 44 studies. The affordances for educators include task automation and professional development, allowing the creation of effective instructional materials. For students, the affordances comprise explaining complex concepts, providing personalized feedback, and facilitating engagement. Andrade-Girón et al. examined the current scholarly landscape concerning the use of ChatGPT within higher education. The review emphasizes the positive impact of AI tools such as ChatGPT on student learning outcomes and time management. However, a significant research gap in rigorous experimental studies is identified, underscoring the need for more robust evidence through controlled settings. B. Summary of Key Research Works Chen and Chen evaluated the impact of AI on Education. The authors describe a set of categories in their review for AI in education as follows. Adoption and implementation, indicating that AI has been adopted through embedded systems, chatbots, and web-based, and online intelligent education systems. Customization and Personalization, which pose the idea that AI meets the individual needs and capabilities of students. Barrier Elimination assures that AI enables global access to education through online and web platforms. Robert and Mohamed conducted a literature review based on 22 papers with SWOT analysis from 2022 onwards in computer science education. Concerning Strengths (S), Generative AI models, such as ChatGPT and Codex, demonstrate a solid ability to generate code effectively. Studies have shown promising results in solving fundamental programming problems. As for Weaknesses (W), GenAI models may exhibit issues related to reliability, leading to the generation of incorrect or inconsistent code without warning. As Opportunities (O) are highlighted helping beginner programmers, improvement of educator’s productivity and workload, personalized learning, accessibility, and error feedback. Finally, as Threats (T), the authors indicate plagiarism and over-reliance on generated information through GenAI rather than on learned knowledge or other information sources. C. Identified Research Gap Bahroun et al. synthesized key insights of GenAI in educational settings based on selecting 207 research papers through a bibliometric analysis. The study points out several meaningful implications for Educators, researchers, and policymakers. For Educators, GenAI could bring pedagogical innovation, creating interdisciplinary collaboration, but at the same time, Educators need to dive into the ethical implications of GenAI, e.g., in data privacy, biases, and academic integrity. Regarding this latter part with regulatory frameworks or pre-emptive policies, Ifenthaler et al. presents a Delphi study based on a survey administered to 33 international professionals, indicating as outstanding results three trends: (1) privacy and ethical use of AI, (2) the importance of trustworthy algorithms, and (3) equity and fairness. The study highlights the importance of precautionary pre-emptive policies preceding practice for AI in education. Decision-makers may introduce such policies in response to societal concerns, rapid technological advancements, and the need to align educational practices with ethical and legal standards. D. Contribution of the Proposed Work AI literacy is the ability to comprehend, assess, interact with, and make informed decisions using AI technologies. AI literacy involves understanding the basic principles of AI, recognizing its applications, and being aware of the ethical, social, and privacy implications of the system using it. This review highlights how the concept of AI literacy is helpful if we analyze the limitations and pitfalls of GenAI reported by educational researchers, which include plagiarism and cheating, privacy issues, potential bias, diversity issues, or spread of false information. Rather than follow-up disruptive technologies such as GenAI, it is better to develop skills in the students and teachers to evaluate and understand the affordances and limitations of thereof.

A. System Architecture (Scoping Review Method) We performed a scoping review based on PRISMA-SCR guidelines. Since the field of the review (GenAI) is relatively new, we performed a scoping review instead of a systematic literature review (SLR). In addition, the nature of scoping reviews is exploratory aiming at rapidly mapping the key concepts in a research area. Scoping reviews summarize and disseminate research findings, identifying research gaps in the existing literature, or examining the range and nature of research activity. We tailored for the study the stages for scoping reviews proposed by Arksey and O’Malley, which are the following: Identifying the research question; identifying relevant studies; study selection; charting the data; and collating, summarizing, and reporting the results. B. System Design (Research Questions) To meet the objectives of this study, we proposed a set of research questions (RQs). RQ1: What are the bibliometric characteristics of the studies in terms of citations, journals, authors, and research topics? RQ2: What are the educational characteristics of the selected studies and their outcomes in the cognitive, affective, or behavioral learning domains? RQ3: What Gen IA integration strategies and activities have Educators and/or researchers constructed? RQ4: What are the challenges, issues, and opportunities for improvement of GenAI in engineering and computing education? For the study, we adopted an approach starting from the general features of the studies (bibliometric trends and research topics) and navigating towards their pedagogical particularities that help understand how Educators utilize GenAI in engineering and computing education from K-12 to tertiary levels. It is worth mentioning that this scoping review focuses on empirical research with evidence of application and evaluation whether or not the final educational results were positive or negative. C. Main System Modules (Identification of Studies) To identify relevant studies, we searched them in three databases: SCOPUS, Web of Science (WoS), and Education Resources Information Center (ERIC). Specifically, we included the terms “engineering education” and “computing education” with the tags in each database to search directly in the title, abstract, or keywords of every record obtained with the boolean connectors AND and OR. Regarding generative AI, we prefer not to include abbreviations such as GenAI or GAI since studies typically include the complete term, even in their indexing terms. Also, we do not add a specific timeframe because GenAI is relatively a new phenomenon. However, the studies retrieved are between 2022 and 2024. We obtained 146 records with this procedure. D. Data Integration (Screening and Appraisal Criteria) We downloaded the records of each database (n = 146) in the format of research information system (RIS). Then, these records were added to the software Rayyan.ai, which is a software specialized in the screening process for systematic and scoping reviews. With this software, firstly, we checked duplicates (n = 14), and secondly, we screened the (n = 132) remaining records. Nonetheless, (n = 5) articles could not be retrieved because the missing DOI or publisher was unavailable. Finally, (n = 127) records were retrieved. In the latter studies, we searched for integration of GenAI in the educational settings in engineering and computing through an approach such as quantitative, qualitative, or mixed in any educational level (K-12 to tertiary). In addition, the studies must exhibit the development of any of the learning outcomes in the levels of cognitive (knowledge and skills), affective (attitudes, motivation), or behavioral (developed behaviors, e.g., socialization, communication, and organization skills), which are called domains of learning.

A. Technology and Component Specification To respond to RQ1, VOSViewer v.1.6.14 and Crossref REST API were employed. VOSViewer is a software for bibliometric analysis based on network data that is focused on items and clusters with two overall functions: creating maps and visualizing them. VOSViewer was focused on the keywords of every study approved. Crossref REST API is a tool that allows the extraction of bibliometric information such as citations, authors, publishers, type of study (journal article, conference proceedings, book chapter), or articles per year, among others. The API was used with Python language to extract the bibliometric features of the studies. To respond to RQ2, we extracted information from each approved article such as study aim, research approach, participants, educational level, learning outcomes, and conclusions. To respond to RQ3, we identified some strategies and curricular activities described by the researchers in the learning settings. The information for RQ2 and RQ3 was coded using MAXQDA 24. We adopted the following procedure. Two researchers coded 640 fragments for the 24 studies, reading the articles and extracting the information. B. System Implementation (Bibliometric Characteristics) The trend shows an increase in proposals from 2023 to 2024 consistent with other prior systematic reviews. None proposals were identified for 2022. In 2023, the most cited article 'Unleashing ChatGPT’s Power: A Case Study on Optimizing Information Retrieval in Flipped Classrooms via Prompt Engineering' describes a GenAI methodology with ChatGPT for computer science students, specifically, for the topic of TCP socket programming. Using the CRISPE framework, students could create more comprehensive and complete prompts to get in-depth information about the topics addressed. Another trend for this year is the concentration of proposals addressed from the teachers’ perspective. These studies show the teachers’ perceptions regarding GenAI, their possibilities, and the concerns related to their deployment in education. In 2024, the most cited article 'Prompt Problems: A New Programming Exercise for the Generative AI Era' deploys a GenAI methodology for CS1 and CS2. The methodology only offered visual representations of the problems, and students should use GenAI (ChatGPT) as a guide to achieve a solution. Apart from the methodology, the authors created an authoritative prompt tool called Promptly. It is interesting to observe that by 2024, several studies have proposed their own prompting tools with particular educational features for the courses. In addition, proposals with GenAI for children as cocreators of digital media with music, storytelling, and arts were identified in this year. C. Dataset and API Configuration (Countries & Sources) Of 24 studies analyzed, 11 proposals came from the U.S (45.83%), two proposals from the U.K. (8.33%), two proposals from Guatemala (8.33%), two proposals from Germany (8.33%), and countries such as New Zealand, Spain, Chile, Sweden, China, Finland, and Serbia each one with a proposal (4.16%) respectively. In the case of Guatemala, both proposals entail the teachers’ perceptions regarding GenAI in computing and engineering education. Then, mainly, GenAI educational research has been led by the U.S. As for publication type in the studies, (n = 19, 82.6%) are conference proceedings while (n = 5, 21.74%) are journal articles. Most of the proposals were published in Association for Computing Machinery (ACM) conferences (n = 11, 48%) followed by IEEE (n = 5, 22%). This manifests the trend that the central part of the studies is related to computing education and computer science (n = 17); meanwhile, several proposals (n = 7) are for engineering education. D. Data Flow (Research Topics and Keywords) There are some noticeable aspects to remark regarding the cluster map for the keywords in the study. Firstly, AI literacy and AI pedagogy are related to arts, computing, and music for elementary school students. In the same cluster, AI literacy appears as a 21st-century skill to be developed by students. Also, for high school students, proposals describe a set of activities to strengthen AI knowledge and literacy. Secondly, the green cluster (Computing Education) is related to code generation, prompt engineering, conflict resolution, chatbots, and flipped classrooms. For instance, many studies for computing education in the ScR used GenAI tools for code generation, making that students paid attention to learning the logical structure of their programs and contrasting the new knowledge acquired by using GenAI with the prior one. Conversely, for the same feature of GenAI, researchers have reported problems, specifically, hallucinations and missing details and critical technical information in the case of ChatGPT. Seemingly, ChatGPT can work well with 'generic information' for CS1 and CS2 courses, but for more specific topics such as TCP sockets or encryption, retrieved information can fail, even providing wrong guidance.

A. Test Cases (Educational Characteristics & Outcomes) We classified the synthesis into student-centered proposals and teacher-centered ones. We also divided the synthesis into several educational levels from K-12 to university. Readers can notice the prevalence of student-centered computing education proposals focused on cognitive learning outcomes at the university level. Nonetheless, for elementary and middle schools’ students, there is an emphasis on affective learning outcomes. For K-12 levels, strategies explored how elementary-aged Latinx students historically excluded in CS establish connections between community, heritage, and technology using a text-to-image Generative AI (DALL.E). The qualitative focus groups showed that elementary students can make deeper connections when discussing imagined futures of their community and technology, creating prompts that embed technology into cultural practices. B. Performance Evaluation (Students' Perceptions) Reported positive tool-related perceptions from students are associated with the creative capability of GenAI to generate ideas and answers; personal tutoring with the features of convenient feedback and knowledgeable assistant; condensed and structured ways for programming purposes; support for students with limited experience in project management; debugging assistant and easy checking process for programming; support for decision-making process; massive, detailed, and immediate information availability; trustworthy information comparable to experts; and wiring catalyst in terms of improving professional writing style and concise and error-free language. In some proposals, students have reported the GenAI features of the help approach comparable to human interaction and not ego, not judgment referring that GenAI tools do not judge the type and level of questions that users do. Positive learning process perceptions from students include the discovery of new concepts and functions through understanding the logical structure of programs, the possibility to contrast previous knowledge with new information provided by GenAI, improving learning curve, developing analytical, problemsolving, computational, and innovative thinking, writing catalysis by improving writing skills, and enjoyment and engagement in activities. Concerning negative tool-related perceptions, students reported hallucinations in terms of difficulty in explaining ambiguous concepts, misinterpretation, slow processing, ignoring errors in programming, misleading suggestions, or inconsistent generative responses. Besides, repetitive and uncreative information; limited interpretation and analysis; unreliable and not trustworthy information for specific CS and engineering topics. In addition, students exposed the problem of weighting the answers given by the IA because the responses provided are generic and have not been reviewed. C. Discussion (Teachers' Perceptions) Teachers have indicated different tool-oriented possibilities and benefits such as support for the creation of personalized programming tasks, multiple approaches to validate students’ comprehension, production of large and feasible software systems, an increase of productivity, and idea generation for learning activities, content generation, productivity with time optimization, and proof reading. From a pedagogical standpoint, teachers remark on the possibility that GenAI brings to rethinks and redesign the educational structure of courses through GenAI adaptation to learning goals, defining curriculum activities that incorporate GenAI, and change in traditional assessment. Nevertheless, teachers indicate several concerns and threats. For instance, teachers manifested the misbehavior practices that students could have such as cheating and plagiarism, violation of academic integrity rules, missing out the focus on learning, over-reliance which could hinder the development of skills, e.g., in fundamental programming concepts, lack of preparedness to cope with GenAI properly, and privacy, bias, and ethics issues. D. Comparative Analysis (GenAI Tools and Strategies) The typical role of GenAI tools was assistant for the learning process. Some of these strategies used their own personalized GenAI chatbots, which were tailored to the particular learning goals in the courses. GenAI tools reported by the researchers in their methodologies include ChatGPT 3.5 and 4.0, DALL-E, Promptly (AI Web-based tool for visual representation of CS1 and CS2 problems designed by researchers), CS50 Duck (AI assistant tailored for introductory programming), GitHub Copilot, GAIPowered Slack Bot, Runway ML, AIVA, and Talking Head Anime from a Single Image 3. Instructional activities for K-12 levels are characterized by participatory sessions, some of them directed by questions such as the following: How do you use AI in your daily life? What are some interesting things you could do with AI? What are some potential use cases of AI? Also, some activities reported included creating a simple ML model using the 'Teachable Machine' platform development by Google. Concerning instructional activities for the tertiary level, these are featured by teaching topics in programming and engineering in which GenAI tools are used typically as assistants. Instructional activities include project-based learning, StandUp reports, Kata code approach, and feedback of students interactions.

A. Limitations We observed several issues and challenges regarding the pedagogical component of GenAI. There is an overestimation of students and teachers’ perceptions regarding Gen AI tools but an underestimation of how Gen AI tools influence the teaching-learning process. We observe that many studies are focused on perceptions, but few report how GenAI tools benefit students’ learning process, and how these tools are aligned with the skills and outcomes expected in the curricula of engineering and computing. There is no balance between the number of student-centered and teacher-centered proposals. It is missing the standpoint and criteria of teachers as co-creators of Gen AI methodologies. In fact, we did not identify educational programs focused on teachers’ formation to effectively cope with Gen AI. AI literacy competencies are ambiguous. For instance, some AI literacy competencies proposed can work with the tertiary level (university) and not with K-12 ones, e.g., when we explain LLMs from a technical perspective. Then, it is unclear what learning outcomes and skills are expected to be developed by students with Gen AI at different educational levels. B. Future Scope We need to reflect on learning by doing during the whole process of using Gen AI. Many proposals only identified the students perceptions once the process finished and not before. The observation concerning students’ interaction should be throughout the educational process to measure the value of GenAI in creating meaningful learning. In this, positive and negative outcomes derived from GenAI usage should be analyzed. A better conceptualization of AI literacy and its competencies should be provided. Yet, this term and what it means for engineering and computing education is in an early development stage. A proper conceptualization for engineering and computing should be created, analyzing the sociocultural context and the implications of GenAI in the curricula. Teachers’ voices must be considered in the whole process of educational co-creation with GenAI. Here, the teachers’ formation must be crucial for leveraging and properly integrating GenAI tools in the teaching-learning process. We distinguish between the formation in Gen AI usage and the pedagogical formation to deploy it effectively in the learning settings.

This scoping review analyzed and described GenAI methodologies and strategies for engineering and computing education. We identified twenty-four proposals, most of them for computing education at the tertiary level with few for K-12 levels. At the same time, we detected that the central part of the proposals is describing students’ and teachers’ perceptions and affordances of GenAI tools dominated by ChatGPT. Only a few studies describe methodologies with evidence-based outcomes from quantitative, qualitative, or mixed approaches. Then, it is tough to analyze how GenAI intervenes in the learning process from cognitive, affective, or behavioral domains, including sociocultural and identity aspects of students. Another aspect that we consider crucial to expose is that teachers’ voices are missing out. Most of the proposals are student-centered and have diverse outcomes. Still, the standpoints of the teachers and formation programs for them in GenAI at the technical and pedagogical levels are missing. Several studies asserted this issue as significant to the successful leverage of GenAI in learning settings. It seems that in the discourse, GenAI tools by themselves were self-described and self-explanatory for students when evidence shows just the opposite: Students can struggle with AI-based systems and prompt engineering, even when considered digital natives. Teacher support is essential to unveil the implications of GenAI in engineering and computing education. Similarly, extant literature remarks on two concepts: AI literacy and prompt engineering. On one hand, seemingly, AI literacy and prompt engineering do not have a close relationship or correlation. This indicates that students can learn how to make prompts without even knowing about AI literacy. The previous aspects point out that is necessary to take a critical pedagogical perspective of GenAI systems, and we need to think beyond the GenAI hype. GenAI is only a tool that should be oriented towards our pedagogical purposes and not vice versa. We encourage educators and researchers to take the outcomes and findings of this ScR to co-create methodologies, frameworks, and critical positions about what GenAI means and entails for engineering and computing education.