LectureLoom

An AI tool that turns lecture videos into study notes

LectureLoom takes a lecture or lesson video, from a YouTube or Vimeo link or an uploaded file, and returns one structured study aid:

• a short plain-language summary
• key takeaways
• key terms with definitions
• an outline of timestamped sections
• referenced resources
• generated study questions

It's aimed at students, note-takers and course creators, and its output is shaped so learning platforms can ingest it.

We built it at the AMD Developer Hackathon: ACT II as a sibling of RecipeReel. The architecture is the same, and only the domain model and prompts changed. The code is public on GitHub.

Key features

• Slide-aware notes — terms, formulas and dates come from the text on the slides, while explanations and section order come from the narration.
• Clip-ready sections — each section has a start and an end timestamp, so it can be opened at the right second or cut out as a clip.
• No invented definitions — a term the lecturer mentions but never defines keeps a null definition, and every term is flagged as spoken, on screen or inferred.
• Study questions — generated questions for self-testing, based on the lecture content.
• LearningResource JSON-LD — sections export as schema.org Clip parts with start and end offsets, a standard format that search engines and learning management systems can read.

Architecture and tech stack

The backend is FastAPI with an async job worker, live progress over Server-Sent Events and SQLite storage. A job runs these steps:

1. Download the video with yt-dlp.
2. Extract the audio with FFmpeg.
3. Sample timestamped keyframes.
4. Transcribe the audio with Whisper-large-v3.
5. Read the slides with Qwen2.5-VL.
6. Have Gemma produce schema-constrained JSON at temperature 0.
7. Post-process the notes: deduplicate terms, order and renumber sections by timestamp and infer the lecture's duration.

Both perception stages, transcription and slide reading, are built for AMD Instinct MI300X GPUs through ROCm.

As in RecipeReel, every stage is a pluggable provider with a mock implementation. The synthesizer speaks the OpenAI API, so it can point at Gemma on Google AI Studio, a model on Fireworks AI or Gemma served by vLLM on AMD hardware without code changes. Mock mode runs the whole pipeline offline from a bundled photosynthesis lecture, which is also how the 11-test suite runs. Slim and ROCm Dockerfiles are included.

What we delivered

In a single day we built the lecture-video-to-study-notes version on top of the shared pipeline. It includes the lesson data model (key terms, sections, resources and study questions), rewritten prompts, the LearningResource export, lecture fixtures, tests and Docker packaging. Because it shares almost everything with RecipeReel, most of our demo and deployment effort went into the cooking product.

Related work

LectureLoom is part of our AI integration and automation work. For a full education product, see the BiliMind AI course generator. Other builds from the same hackathon are Legible, Testify and Keystone.