This book is designed to be used as the primary text for a one- or two-semester
course on reinforcement learning. For a one-semester course, the first ten chapters should
be covered in order and form a good core, to which can be added material from the
other chapters, from other books such as Bertsekas and Tsitsiklis (1996), Wiering and
van Otterlo (2012), and Szepesv´ari (2010), or from the literature, according to taste.
Depending of the students’ background, some additional material on online supervised
learning may be helpful. The ideas of options and option models are a natural addition
(Sutton, Precup and Singh, 1999). A two-semester course can cover all the chapters as
well as supplementary material. The book can also be used as part of broader courses
on machine learning, artificial intelligence, or neural networks. In this case, it may be
desirable to cover only a subset of the material. We recommend covering Chapter 1 for a
brief overview, Chapter 2 through Section 2.4, Chapter 3, and then selecting sections
from the remaining chapters according to time and interests. Chapter 6 is the most
important for the subject and for the rest of the book. A course focusing on machine
learning or neural networks should cover Chapters 9 and 10, and a course focusing on
artificial intelligence or planning should cover Chapter 8. Throughout the book, sections
and chapters that are more dicult and not essential to the rest of the book are marked
with a ?. These can be omitted on first reading without creating problems later on. Some
exercises are also marked with a ? to indicate that they are more advanced and not
essential to understanding the basic material of the chapter