Reinforcement Learning: An Introduction, Second Edition (Draft)

This textbook provides a clear and simple account of the key ideas and algorithms of reinforcement learning that is accessible to readers in all the related disciplines. Familiarity with elementary concepts of probability is required.

**Tag(s):**
Machine Learning

**Publication date**: 30 Sep 2016

**ISBN-10**:
n/a

**ISBN-13**:
n/a

**Paperback**:
455 pages

**Views**: 2,655

Reinforcement Learning: An Introduction, Second Edition (Draft)

This textbook provides a clear and simple account of the key ideas and algorithms of reinforcement learning that is accessible to readers in all the related disciplines. Familiarity with elementary concepts of probability is required.

Admin's note:

This is a draft of the second edition, a work in progress. When this book is completed, there is a possibility that this draft will no longer be publicly and freely accessible.

From the Preface to the First Edition:

From the Preface to the Second Edition:

This is a draft of the second edition, a work in progress. When this book is completed, there is a possibility that this draft will no longer be publicly and freely accessible.

From the Preface to the First Edition:

Our goal in writing this book was to provide a clear and simple account of the key ideas and algorithms of reinforcement learning. We wanted our treatment to be accessible to readers in all of the related disciplines, but we could not cover all of these perspectives in detail. For the most part, our treatment takes the point of view of artificial intelligence and engineering. Coverage of connections to other fields we leave to others or to another time. We also chose not to produce a rigorous formal treatment of reinforcement learning. We did not reach for the highest possible level of mathematical abstraction and did not rely on a theorem–proof format. We tried to choose a level of mathematical detail that points the mathematically inclined in the right directions without distracting from the simplicity and potential generality of the underlying ideas.

The book is largely self-contained. The only mathematical background assumed is familiarity with elementary concepts of probability, such as expectations of random variables. Chapter 9 is substantially easier to digest if the reader has some knowledge of artificial neural networks or some other kind of supervised learning method, but it can be read without prior background. We strongly recommend working the exercises provided throughout the book. Solution manuals are available to instructors. This and other related and timely material is available via the Internet.

From the Preface to the Second Edition:

The nearly twenty years since the publication of the first edition of this book have seen tremendous progress in artificial intelligence, propelled in large part by advances in machine learning, including advances in reinforcement learning. Although the impressive computational power that became available is responsible for some of these advances, new developments in theory and algorithms have been driving forces as well. In the face of this progress, we decided that a second edition of our 1998 book was long overdue, and we finally began the project in 2013. Our goal for the second edition was the same as our goal for the first: to provide a clear and simple account of the key ideas and algorithms of reinforcement learning that is accessible to readers in all the related disciplines. The edition remains an introduction, and we retain a focus on core, on-line learning algorithms. This edition includes some new topics that rose to importance over the intervening years, and we expanded coverage of topics that we now understand better. But we made no attempt to provide comprehensive coverage of the field, which has exploded in many different directions with outstanding contributions by many active researchers. We apologize for having to leave out all but a handful of these contributions.

Tweet

About The Author(s)

Andrew Barto is Professor Emeritus in the College of Information and Computer Sciences at University of Massachusetts Amherst. He is a Co-Director at Autonomous Learning Laboratory. His research interests are theory and application of methods for learning and planning in stochastic sequential decision problems; algebraic approaches to abstraction; psychology, neuroscience, and computational theory of motivation, reward, and addiction; computational models of learning and adaptation in animal motor control systems.

Richard S. Sutton is Professor and iCORE chair Department of Computing Science at University of Alberta. Dr. Sutton is considered one of the founding fathers of modern computational reinforcement learning, having several significant contributions to the field, including temporal difference learning, policy gradient methods, the Dyna architecture.

Book Categories

Computer Science
15
Introduction to Computer Science
32
Introduction to Computer Programming
52
Algorithms and Data Structures
24
Artificial Intelligence
24
Computer Vision
28
Machine Learning
6
Neural Networks
22
Game Development and Multimedia
25
Data Communication and Networks
5
Coding Theory
14
Computer Security
8
Information Security
34
Cryptography
3
Information Theory
17
Computer Organization and Architecture
22
Operating Systems
1
Image Processing
10
Parallel Computing
4
Concurrent Programming
20
Relational Database
3
Document-oriented Database
13
Data Mining
16
Big Data
17
Data Science
23
Digital Libraries
22
Compiler Design and Construction
26
Functional Programming
11
Logic Programming
26
Object Oriented Programming
21
Formal Methods
69
Software Engineering
3
Agile Software Development
7
Information Systems
5
Geographic Information System (GIS)

Mathematics
66
Mathematics
13
Algebra
1
Abstract Algebra
27
Linear Algebra
3
Number Theory
8
Numerical Methods
2
Precalculus
10
Calculus
2
Differential Equations
5
Category Theory
10
Proofs
19
Discrete Mathematics
24
Theory of Computation
14
Graph Theory
2
Real Analysis
1
Complex Analysis
14
Probability
43
Statistics
7
Game Theory
5
Queueing Theory
13
Operations Research
16
Computer Aided Mathematics

Supporting Fields
19
Web Design and Development
1
Mobile App Design and Development
28
System Administration
2
Cloud Computing
9
Electric Circuits
6
Embedded System
26
Signal Processing
4
Network Science
3
Project Management

Operating System
Programming/Scripting
6
Ada
13
Assembly
34
C / C++
8
Common Lisp
2
Forth
35
Java
12
JavaScript
1
Lua
15
Microsoft .NET
1
Rexx
12
Perl
6
PHP
66
Python
12
R
1
Rebol
13
Ruby
2
Scheme
3
Tcl/Tk

Miscellaneous