"I always prefer video to a pile of reading materials, but I am not sure which one helps me learn better." –Lin Dong

‘Reinforcement’ has become a buzz word in the machine learning and artificial intelligence communities. It has wide application from winning a video game to automating a car.

If you are not familiar with reinforcement learning, here is what it is. First of all, it is a sub-area of machine learning. In supervised learning, the task is to learn to predict or classify something from a training dataset. For example, you want to decide if an image is showing an apple. You will receive some pictures of apples and get an idea of what an apple looks like. In reinforcement learning, the task is not simply to predict or classify but to learn what to do to maximize our reward in a complex dynamic system. In this setting, we don’t have the nicely labelled training set to teach us. So what should we do? Well, we can learn through trial-and-error interactions with the system. This is like making an apple pie. Try different kinds of apples and various amounts of sugar. You may puke several times, but eventually you will learn to make a perfect-tasting apple pie.

You may wonder how this is related to education. Think of students taking a course to learn some skills. The complex system is the interaction between the instructor and the students, as well as how student learn the knowledge. The reward of this system is how much the students actually learn.

Nowadays, the common practice in education is a one-fit-all method. That is, each student in a course is treated identically for all the teaching activities–the same content, same way of teaching, and same tests. However, some students may learn better from a video illustration, whereas some students may learn more from a well-organized handout. Or, some students may perform better on a project but others are good at exams. Therefore, a better strategy would be to develop a personalized educational scheme that takes into account the inherent differences between students, and the scheme should be able to change dynamically according to feedback from the student.

The study process can be formally modeled as a Markov decision process. Each student entering the course has his/her own initial status, which may include the student’s own characteristics and previous proficiency level. The process starts by an assessment (A), say a quiz. The assessment is so important here as instructors normally cannot read minds. They need to give a quiz to estimate how much he/she really understands the content. The result of this quiz is observed (X) and serves as an estimate to the student’s true proficiency level. Then the instructor gives an intervention (I) by choosing one of the teaching resources for this student. This intervention leads the student to a new proficiency level. The data triple (A, X, I) accumulates until the student reaches the end of the course. What we care about most is the final assessment result, which is a measurement of a student’s final proficiency level after the course.

This process is slightly different from ordinary Markov decision processes in the sense that there are two completely different decisions to make: how to assess the student’s understanding and how to select the teaching resources for the student. Therefore, our goal is to maximize the final outcome by finding the optimal policy of both assessment and instructor’s intervention.

The next step is how to solve for the optimal policy using the accumulated data of students. We will use approximate dynamic programming, a tool from reinforcement learning, to learn the optimal teaching plan. Check out my next post for details!