tl;dr: Our 2-year experiment at MIT hiring students to scribe lectures demonstrates the potential of lecture scribing as a cost-effective method to capture lecture content at scale for open courseware.

Intro

Our goal at SOUL is to make college-level course sequences, such as undergraduate majors or minors, realistically learnable from free online resources. Achieving this involves capturing, among other things, the content of lectures.¹ While video recordings of classroom lectures are one way to accomplish this, they can be logistically challenging to do at scale, as I’ve previously discussed. So, what alternative methods could we use to capture lecture content effectively?

Lecture scribing

At MIT – and, I’m sure, at many other universities – it’s not uncommon to see a student in class taking extremely detailed notes in LaTeX, essentially creating a live transcript of the class. These students are fast typers and very comfortable with LaTeX. This is a particularly common occurence in computer science, physics, and math classes, where LaTeX is used often. In some cases, these notes end up doing a better job at explaining the material than the lecture, offering cleaner diagrams and clearer explanations, especially when the notes are refined further after class. Over time, the notes of some students who do an exceptional job become well-known, such as Andrew Lin’s math notes and Evan Chen’s math notes (these are the ones I’m most familiar with from the time I was at MIT, and I’m sure there are many more like these). High-quality, openly-shared scribed notes are a great resource for students in the class, and they’re also a great resource for the instructors, who may use them to improve future iterations of the class or may adopt them as the class’s official course notes. For example, my friend Sanath Devalapurkar’s scribed notes from Fall 2016’s 18.905: Algebraic Toplogy I and notes from Spring 2020’s 18.906: Algebraic Topology II came to form the basis of the professor’s official course notes on algebraic topology.

While students usually scribe on their own initiative, there are cases where scribing is encouraged or formalized by instructors. In new courses that don’t have official typed course notes yet, instructors may ask a TA or student if they’d be interested in scribing the lectures to help create a first draft of official course notes as a voluntary task outside of class work. Alternatively, instructors may structure lecture scribing into the course and assign a subset of students to scribe each lecture, either as a required graded activity or as an opportunity to earn extra credit.

For example, here’s an excerpt from the syllabus of Spring 2022’s 6.435: Bayesian Modeling and Inference (a class I took), in which scribing one lecture was a required activity:

As another example, here’s an excerpt from the syllabus of Fall 2022’s 6.246: Reinforcement Learning (a class I audited), in which scribing was an extra credit activity:

In some cases, notes created as part of class work are posted online for anyone to learn from, such as the scribed lecture notes from Fall 2004’s 6.895: Esssential Coding Theory. But in most cases where scribing is a required activity, the notes are not posted online (they weren’t in the case of 6.435 and 6.246), usually because the instructors are concerned students might copy significantly from the notes from previous years.²

Scribed lectures as a possibly acceptable substitute for live lecture

When the scribed notes are of high quality (imo: when they include everything the professor covered, and possibly more), they’re like a mini textbook, and they can be a really good substitute for the live classroom lecture. With this in mind, if we can’t video record the lectures, either because it’s not logistically possible or because the professor doesn’t want the recordings online, we can try to have them scribed. Notably, there are a lot of instances where a professor doesn’t want to share their video recordings online but is comfortable with sharing their scribed lectures online.³

Before I discuss how we might implement lecture scribing at scale, I’ll briefly acknowledge some of the drawbacks of lecture notes as a substitute for live lectures:

Regardless, really good notes are better than nothing and are easily better than mediocre lectures. Furthermore, everyone’s learning styles are different, and some people learn better from reading a text than from listening to a lecture.

Crowdsourcing lecture scribing

When I was thinking about this a few years ago, I had three main thoughts:

1. Many students at MIT scribe lectures while they take classes. More generally, students generate many different kinds of high-quality learning materials during the semester, but most of these materials are never shared widely and would do a lot of good if they were. For example, it’s not uncommon for staff solutions for homework problems to be not-that-great. But among all the students in a class, there are always several excellent student solutions. Aggregating these student solutions could be a great way to show multiple ways of solving a problem. Instead of posting a subpar staff solution online (if we had permisson to), we could post the good student ones. Overall, we can do a better job of asking for student-generated course materials and making them available to the public. Someone with subject-matter expertise might have to do some quality control before we post them.
2. We can get more lectures scribed by nudging professors to include lecture scribing in the course as a required or extra credit activity.
3. We can hire and pay students to scribe lectures.

@1: We’ve been doing this for the past few semesters. At the end of each semester, we send a few emails that end up reaching most undergrads, asking if they have any good notes they could share. This gets us good course notes for a handful of courses per year. For example, one student recently sent us good notes for 18.405, notes for SP23’s 18.218, and notes for SP24’s 18.218.

@2: Every professor I’ve asked has not wanted to do this.⁴ I’m not exactly sure why – I think professors have a clear idea of what tasks they want their students to focus on to maximize the learning objectives given the time constraints of the semester, and they don’t want to introduce distractions. I also think many professors can be resistant to change. But many professors are open to experimentation and we should be asking more professors if they’d be open to this. It’s also worth noting that making lecture scribing part of the course in some way might be the only viable strategy in situations where funds are insufficient to pay student scribes. Requiring students to scribe for free purely to generate notes for public posting seems unethical, but if scribing is truly believed to be a meaningful learning activity aligned with the course objectives, and if the scribing work is distributed across the class so that each student is not overburneded (e.g. each student does one lecture), it may be justifiable.

@3: I’ll talk about this in detail next.

Implementing crowdsourced lecture scribing

I thought paying students to scribe lectures was a reasonable idea to test out for several reasons:

• Many students are already scribing to some decent level of quality, and might be willing to do it to a high level of quality if they are paid.
• There are significant advantages to hiring scribes from within a class versus outside. As current students in the class, they experience firsthand which concepts are challenging to grasp and this informs the degree of detail they include in the notes (not skipping steps, elaborating where necessary), and they naturally incorporate student perspectives. Also, they can leverage class resources like the office hours with the teaching staff and study groups to clarify confusing points. While we could theoretically hire external scribes to work from lecture recordings, they might miss valuable insights that come from being an active participant in all aspects of the class, which includes doing the homework and taking the exams.⁵
• Many students seek part-time work during their studies. There are many jobs students can get on campus (Tech Callers – calling alums to ask for donations, UROPs – paid research opportunities, TAing a class, being a tour guide for the admissions department, etc.), and based on what I hear, there’s non-zero unmet demand for on-campus jobs. Lecture scribing seems like a great job – you basically get paid to learn, and the produced lecture notes can help you get a TAship, which are often highly sought after. But maybe students prefer a side gig that requires less mental work when they’re already busy with hard classes.
• If we pay enough, students should be interested. Everyone has a price?
• Students don’t demand super high wages (as a reference point, highly-qualified TAs in the EECS department make $25/hour), so this should be pretty cost-effective. If a 1.5-hour lecture takes 4 hours to scribe, producing course notes for a course with 24 1.5-hour lectures should cost $2,400. That’s significantly lower than the cost of hiring MIT Video Productions to record the lectures; they charge something like $10,000-$20,000 per course (and remember, the professor might only be open to scribed notes, not video recordings). $2,400 is great if the notes are high-quality and can be used for several years until the course, if ever, gets significantly updated.

In 2021, while I was a student at MIT, I proposed this idea to MIT OpenCourseWare (OCW) and began part-time work on their team to experiment with hiring students to scribe lectures.⁶ OCW funded the project for the first year. That year, we experimented with (1) paying students $15/hour, which was a few dollars more than the standard rate for on-campus research assistant jobs (so it was somewhat competitive), and (2) in some cases, hiring multiple students per class to work together on the notes (to reduce the amount of work each student had to do and to do quality control via peer review). In the second year, I ran the project under SOUL. That year, we experimented with (1) paying students $25/hour and (2) hiring only one student per class in all cases.

The process was as follows:

1. I reached out to professors asking if they’d be open to this.
2. For those that were, in their classes, I sent a job announcement to the class during the first week of the semester, requesting interested folks to send in a sample of high-quality typed notes they’d taken before or a sample of high-quality typed homework solutions if they didn’t have notes. Everyone who applied provided either LaTeX-ed notes or LaTeX-ed solution samples.
3. I reviewed the submissions, had brief interviews with promising candidates, and hired the candidate(s) who had the best notetaking skills and sufficient bandwidth to do the work.⁷
4. During onboarding, I provided examples of good notes and my instructions for scribing mainly emphasized including all the content from lecture (e.g. student questions and the professor’s response should be included!). I also provided a LaTeX template.
5. Most scribes cannot live-scribe a lecture. If a class was in a classroom equipped with auto-capture recording technology, we asked the professor if we could video record the lectures and make them available to the scribes as a reference, if not to the entire class. About half the time, the professors agreed to do this; in the other half, the professors were recording the lectures to make them available to the entire class anyways. There were no cases where a professor said no to recording the lectures solely for the scribes.
6. I had a virtual meeting with each hired student regularly (e.g. once a week or once every two weeks) to briefly check in about their progress and keep them accountable. I had them keep an excel sheet up to date with the status of their notes for each lecture. Our first meeting was in person so we could put faces to names and see each other as individuals, connecting on a more human level.
7. I reviewed the notes they produced before each meeting and provided feedback for revisions. Feedback concentrated on organization, writing quality, and diagrams, with little feedback provided on the core technical content (because I wasn’t familiar with all of it).
8. In the first year, we paid scribes every two weeks. In the second year, we paid them after the notes were complete.
9. When we were done with the notes, we asked the professor if they wanted to proofread the notes (about half did) and if they had any revisions they wanted us to make. Once the professor gave us final approval, we posted them online.

Results

In all cases, we were able to provide video recordings of the lectures to the scribes via the free auto-capture recording technology in many classrooms, though a handful of the recordings for each course got messed up due to technical issues.

Here are links to lecture notes we created and published online. I was very happy with the quality of these.

• 18.700
• 18.701
• 18.702

The scribes did a great job including questions asking during lecture and their answers (page 31 of the 18.702 notes):

They also did a great job recreating diagrams from lecture (pages 173-175 of the 18.701 notes):

Here’s the status of each scribing project:

I think all the scribes did a great job, but in about half the cases, the work did not meet the quality threshold I was looking for, which was a pretty high bar. Perhaps we could’ve still posted those notes online (better than nothing?), but I was planning to improve the notes myself or by hiring someone else and post them once they were ready.⁸ 3 of the 4 notes we didn’t post were about 80% of the way to publishable. Unfortunately, we got busy and had some funding issues, so that work got put on hold.

It seems noteworthy that the notes that were successful were from math classes. I think this is because

• Math students scribe more often than other students. All the math students we hired had scribed classes before and were going to scribe their classes regardless of whether or not we hired them. This was not true for students we hired in non-math classes. This fact influences the quality of the notes in two ways:
  • The more you scribe, the better you get at it.
  • Scribing and being a strong student is correlated, especially in math. If you scribe, you’re probably a strong student, and if you’re a strong student, you’re probably a better-than-average scribe. (Andrew Lin and Evan Chen, whose notes I linked earlier as examples of excellent scribed lecture notes, were both math students.) If I had to judge, I think the math students we hired were stronger in their subject than the non-math students we hired.
• Scribing is easier in math classes for many reasons. Hypothesis: (1) math lectures tend to have less variation in media to incorporate (just numbers, symbols, and line-based diagrams), whereas non-math classes have more (e.g. a biology class might have anatomical illustrations, life cycle diagrams, phylogenetic trees, bar/line graphs, heat maps, molecular structures, reaction pathways, flowchats, etc.); (2) math is less subjective, so you can get by with more of copying the professor’s words verbatim; (3) math lectures tend to follow a more linear and consistent structure compared to non-math lectures; (4) math is easier to live-scribe because it has more lecture content embedded in what’s written on the chalkboard (easier to type) relative to what’s spoken (harder to type).

Here’s some data about how recruiting went:

• I didn’t track the exact percentages, but about 1 in 10 classes I asked were open to us scribing their lectures to put online, which was about double the percentage of classes I found were open to sharing video recordings online (~ 1 in 20). Note that I didn’t pick random classes – I mostly stuck to STEM classes in standard lecture format.
• In many classes where we advertised the job, no one was interested. I was very disappointed by this because I think the hardest part is getting the professors to agree to this (10% chance of success), and writing the notes is the easy part (~50% chance of success based on our results).
• In the classes where students were interested in the job, the number of interested students was only 1-5 out of 50-100 students. Usually just 1-2. I think this isn’t surprising – students are busy and have other priorities.
• Those interested in the job were more likely to be younger than older. We got more applicants who were freshman and sophomores than juniors, and no applicants who were seniors. I think this isn’t surprising – the older you get, the more you realize how valuable your time is and the more you prioritize what you spend your time on. This is kind of unfortunate because I think the older students might be better scribes.

Here’s some data about how scribing went:

• Scribes relied heavily on the video recordings to rewatch anything they couldn’t live-scribe or found confusing. I think it would be very difficult for a student in the class to write good lecture notes without a recording of the class. For classrooms without auto-capture recording technology, perhaps it would be possible to get by with (1) having the student take phone pictures of the chalkboard (to capture chalkboard content), (2) using an audio recorder (e.g. Sony ICD-PX470) to capture any spoken content, and (3) recording the instructor’s laptop screen on Zoom to capture any content shared to the projector screen.
• Scribing took way longer than expected. It took us about a year to complete the notes for each class, rather than a single semester (4 months). It was easy to fall behind schedule during the semester when the students got busy. While I had emphasized to students that it’s important that we stay on schedule, I had also told them that they should prioritize their studies during crunch times like exam weeks. I didn’t make payment contingent on the notes being completed on time.
• I held an in-person group hangout with all the scribes at the end of the year to discuss the experience. All the scribes said that scribing helped them learn the material better.
• There’s not enough data to tell whether the payment schedule (every two weeks vs after the notes were complete) had an effect on anything. Our scribes all said they didn’t really care when they were paid as long as they got paid some time that year, although I think this was by chance – if we did this again, I’d expect a few people who preferred to be paid during the job.
• Hiring multiple students to scribe a single class went ok. They noted that they each had to do less work when they divided the work between them, that they indeed consistently caught mistakes and made general improvements through peer review, and that giving the notes a single voice took some extra effort. There was also more coordination overhead for me.

Reflection on hiring

Looking forward, we want to figure out how to do this better. I think our results pose the following questions:

• Can we train new hires without significant time investment?
• How do we identify stellar candidates during the hiring process?
• How do we do quality control?

The distribution of outcomes from our students was about the same as the distribution of outcomes from students hired as UROPs (undergrad research assistants). I, and many others, think of UROPs as falling into one of three categories:⁹

• They’re stars. They do insanely great work with minimal supervision. They’re basically like a fellow graduate student collaborator.
• They’re good. They’re hardworking students with great potential. They do good work and continue to grow and do better and better work as they get more experience and are provided good mentorship.
• They’re not worth the time. They do mediocre work no matter how much you supervise them (e.g. they could be overcommited, they could be having a rough semester, etc.).

I think the students we hired were mostly in the second category, but we had students from all three. I spent a lot of my own time supervising the notetaking, and this won’t scale. If we want to scale, we have to hire more star students and spend less time supervising the good students. Note that supervision was about 30% feedback on notes but 70% checking in on them and making sure they were on schedule, and bugging them when they fell behind.

I initially thought that having one student take the notes to 80% done would be ok – I or someone else could do the remaining 20% work – but I’ve realized this is a bad idea. It’s not like the student did the first 80% lectures really well, and didn’t do anything for the last 20% of the lectures – it’s more like they did all the lectures kinda poorly, and we have to go through and rewrite everything to be a bit better, and that actually takes a similar amount of time (less time, but similar) as doing the entire thing ourselves and not hiring them in the first place. In the future, we should have a trial period where we give the student a few lectures to scribe and see how they do, and then make a more informed decision about whether or not to extend the contract. The sample notes we ask applicants to provide aren’t always enough to forecast how they’ll do.

Next steps

We had paused this project for a little bit, but we’re going to resume it soon. Here’s what we have in mind for next time:

Workflow improvements:

• Reduce how much supervising and feedback we have to give new hires by providing much clearer documentation on what good notes look like. Rather than just having an example of good notes, provide an example of scribed notes before and after we gave our feedback, highlighting the changes we suggested.
• Provide clearer and stricter expectations about the role. It’s a real job, and things will be due at specific times. We’ll be less lenient with deadlines. Consider having delays reduce payment.
• Structure the contract with a trial period to see how they do. We’ll be more strict about hiring great candidates.

Things to explore and experiment:

• Consider hiring students outside the class or even outside of MIT when we aren’t able to get any applicants from the class.
• Consider doing this in classes at other universities.
• Research the use of LLMs for giving feedback to scribes and for lecture scribing itself.

Overall, I think our experiment went well. We learned a lot and have a very clear idea of what to do in the next iteration. I hope this writeup is helpful for anyone interested in implementing paid lecture scribing at their university, and more generally, figuring out how to cost-effectively capture lecture content for open courseware!