Author: IBL News

“I’ve seen this topic discussed a few times now in relation to Instructure’s recent press release about partnering with OpenAI on a new integration. I attended the InstructureCon conference last week, where among other things Instructure gave a tech demo of this integration to a crowd of about 2,500 people. I don’t think they’ve released video of this demo publicly yet, but it’s not like they made us sign an NDA or anything, so I figured I’d write up my notes. I’m recreating this based on hastily-written notes, so they may not be perfectly accurate recreations of what we were shown.

During the demonstrations they made it clear that these were very much still in development, were not finished products, and were likely to change before being released. It was also a carefully controlled, partially pre-programmed tech demo. They did disclose which parts were happening live and which parts were pre-recorded or simulated.

In the tech demo they showed off three major examples.

1. Course Admin Assistant. This demo had a chat interface similar to every LLM, but its function was specifically limited to canvas functions. The example they showed was typing in a prompt like, “Emily Smith has an accommodation for a two-day extension on all assignments, please adjust her access accordingly,” and the AI was able to understand the request, access the “Assign To” function of every assignment in the class, and give the Emily student extended access.

In the demo it never took any action without explicitly asking the instructor to approve the action. So it gave a summary of what it proposed to do, something like “I see twenty-five published assignments in this class that have end dates. Would you like me to give Emily separate “Assign to” Until Dates with two extra days of access in each of these assignments?” It’s not clear what other functions the AI would have access to in a canvas course, but I liked the workflow, and I liked that it kept the instructor in the loop at every stage of the process.

The old “AI Sandwich,” principle. Every interaction with an AI tool should with a human and end with a human. I also liked that it was not engaging with student intellectual property at any point in this process, it was targeted solely at course administration settings.

My analysis: I think this feature could be genuinely cool and useful, and a great use case for AI agents in Canvas. Streamline the administrative busywork so that the instructor can spend more time on instruction and feedback. Interesting. Promising. Want to see more.

AI Assignment Assistant. Another function was a little more iffy, and again a tightly controlled demo that didn’t provide many details. The demo tech guy created a new blank Assignment in Canvas, and opened an AI assistant interface within that assignment. He prompted it with something like, “here is a PDF document of my lesson. turn it into an assignment that focuses on the Analysis level of Bloom’s Taxonomy,” and then he uploaded his document.

We were not shown what the contents of the document looked like, so this is very vague, but it generated what looked like a competent-enough analysis paper assignment. One thing that I did like about this is that whenever the AI assistant generates any student-facing content, it surrounds it with a purple box that denotes AI-generated content, and that purple box doesn’t go away unless and until the instructor actually interacts with that content and modifies or approves it. So AI Sandwich again, you can’t just give it a prompt and walk away.

The demo also showed the user asking for a grading rubric for the assignment, which the AI also populated directly into the Rubric tool, and again every level, criteria, etc. was highlighted in purple until the user interacted with that item.

My analysis: This MIGHT useful in some circumstances, with the right guardrails. Plenty of instructors are already doing things like this anyway, in LLMs that have little to no privacy or intellectual property protections, so this could be better, or at least less harmful. But there’s a very big, very scary devil in the details here, and we don’t have any details yet. My unanswered questions about this part surrounds data and IP. What was the AI trained on in order to be able to analyze and take action on a lesson document? What did it do with that document as it created an assignment? Did that document then become part of its training data, or not? All unknown at this point.

AI Conversation Assignment. They showed the user creating an “AI Conversation” assignment, in which the instructor set up a prompt, something like “You are to take on the role of the famous 20th century economist John Keynes, and have a conversation with the student about Supply and Demand.” Presumably you could give it a LOT of specific guidance on how the AI is to guide and respond to the conversation, but they didn’t show much detail.

Then they showed a sequence of a student interacting with the AI Keynes inside of an LLM chat interface within a Canvas assignment. It showed the student trying to just game the AI and ask for the answer to the fundamental question, and the AI told it that the goal was learning, not getting the answer, or something like that. Of course, there’s nothing here that would stop a student from just copying and pasting the Canvas AI conversation into a different AI tool, and pasting the response back into Canvas. Then it’s just AI talking to AI, and nothing worthwhile is being accomplished.

Then the part that I disliked the most was that it showed the instructor SpeedGrader view of this Conversation assignment, which showed a weird speedometer interface showing “how engaged” the student was in the conversation. It did allow the instructor to view the entire conversation transcript, but that was hidden underneath another button. Grossest of all, it gave the instructor the option of asking for the AI’s suggested grade and written feedback for the assignment. Again, AI output was purple and wanted instructor refinement, but… gross.

My analysis: This example, I think, was pure fluff and hype. The worst impulses of AI boosterism. It wasn’t doing anything that you can’t already do in copilot or ChatGPT with a sufficient starting prompt. It paid lip service to academic integrity but didn’t show any actual integrity guardrails. The amount of AI agency being used was gross. The faith it put in the AI’s ability to actually generate accurate information without oversight is negligent. I think there’s a good chance that this particular function is either going to never see the light of day, or is going to be VERY different after it goes through some refinement and feedback processes.”