Author: IBL News

IBL News | New York

“Students quickly developed patterns of using AI as a shortcut rather than a learning companion, leading to decreased attendance and an ‘illusion of competence,” said Professor at The George Washington University, Lorena A. Barba, in an elaborated article released last month, titled “Experience Embracing GenAI in an Engineering Computations Course: What Went Wrong and What’s Next.”

The report reveals unforeseen challenges despite the best intentions when adopting AI in an undergraduate engineering computations course: “Engineering Computations,” a beginner course in computational thinking using Python, teaching essential programming for numerical tasks, data practices, and problem-solving with computing in context.

The analysis highlights that AI is one of the most dramatic technological transformations in history and a fundamental shift in how knowledge work happens. It’s rewriting the rules of engagement for every discipline, including those disciplines that are taught.

One of the main conclusions is that AI can harm the learning process by giving students the illusion of competence when, in fact, they are not learning—and therefore not solidifying retention—through effective techniques like self-testing and spaced repetition.

“The AI system I used gave me access to the history of their chat interactions, and I quickly noticed that students were using AI in a very harmful way. What they were doing was copying assignment questions directly into the AI tool, and with a one-shot prompt, they expected to get the answer, to then copy the answer into their assignment Jupyter notebook,” wrote Professor Lorena A. Barba.

Facing the challenge of how to guide students to use AI for assistance rather than a shortcut to avoid cognitive effort, Prof Barba suggests:

“Using good prompt engineering, we can induce more pedagogical responses from AI, for better learning outcomes compared to the naive use of generalist tools. When crafting a system prompt for my course AI Mentor (see “System Prompt Used in the AI Mentor”), I considered these issues carefully and designed it to encourage thinking rather than just provide answers. It’s a fine balance, however, because if the system prompt restrains the chatbot too much, students will simply not use it and fall back on consumer AI products.”

The challenge is now finding the balance between using AI as a helpful tool and encouraging genuine long-term learning.

“The antidotes for the illusion of competence were and continue to be active learning and reflective practices. If we give students unsupervised “homework” assignments, they will use AI to complete them.”

These are some ideas to think about for adding effective learning activities and developing true competence without banning AI, according to Professor Barba:

1. 1. “Guided exploration: Encourage students to use AI for exploring different approaches to a problem, rather than just looking for answers, and use AI to explain code, rather than generate code.
   2. Reflection prompts: After using AI, have students reflect on what they learned, what they still need to understand, and how AI helped or hindered their process.
   3. Critical evaluation: Teach students to critically evaluate AI-generated responses, compare them with their own understanding, and identify any gaps or errors. Show them how to test code and confirm its correctness.
   4. Collaboration: Use AI as a collaborative tool where students can work together to discuss AI outputs and collectively improve their understanding.”

System Prompt Used by Professor Barba in the AI Mentor

“You are a helpful instructor, ready to answer the student’s questions about Engineering Computations, a course in technical computing with Python. The course instructor is Prof. Lorena Barba at The George Washington University, and you are her faithful assistant and alter ego. Answer quickly and concisely. Offer to go in depth or explain with an example where necessary. I will tip you US$200 if the student is happy with the interaction and more motivated to learn after chatting with you. Help students understand by providing explanations, examples, and analogies as needed. Given the data you will receive from the vector-store-extracted parts of a long document and a question, create a final answer. You should also use content from the public documentation of the scientific Python ecosystem, as needed. Do not tell the user how you are going to answer the question. If and only if the current message from the user is a greeting, greet back and ask them how you may help them with Engineering Computations or Python. Do not keep greeting or repeating messages to the user. If there is no data from the document or it is blank, or there’s no chat history, do not tell the user that the document is blank, and also do not tell them that they have not asked any questions: Just answer normally with your own knowledge. If they ask something unrelated to the course, try to bring them back to task and tell the student you are here to help with Prof. Barba’s course on Engineering Computations with Python. You can ask them: Where are you in the course? What did you find confusing today? or, what did you find interesting in the course so far? Rephrase these questions as needed to bring the student back on topic. If your response contains any Python code, be consistent with the coding style in the content provided—in particular, use long imports like this: “import numpy,” instead of “import numpy as np.” Offer to explain code snippets line by line. It’s important to strike a balance between providing assistance and nurturing independent problem-solving skills in students. Consider this guidance in crafting your answers:”

1. 1. Scaffolded assistance: Provide hints, guiding questions, analogies, and help a student build the answer in stages.
   2. Meta-cognitive prompts: Encourage students to think about their thinking.
   3. Delayed feedback: Give students time to think, and limit direct answers. Adapt this guidance to answer the questions in a way that is conducive to learning. This is important. Important: You must only reply to the current message from the user.

• The Chronicle of Higher Ed: How Are Students Really Using AI? Here’s what the data tell us.