Human-in-the-Loop Machine Learning is a guide to optimizing the human and machine parts of your machine learning systems, to ensure that your data and models are correct, relevant, and cost-effective. 20-year machine learning veteran Robert Munro lays out strategies to get machines and humans working together efficiently, including building reliable user interfaces for data annotation, Active Learning strategies to sample for human feedback, and Transfer Learning. By the time you’re done, you’ll be able to design machine learning systems that automatically select the right data for humans to review and ensure that those annotations are accurate and useful.

what's inside

Active Learning to sample the right data for humans to annotate

Annotation strategies to provide the optimal interface for human feedback

Techniques to select the right people to annotate data and ensure quality control

Supervised machine learning design and query strategies to support Human-in-the-Loop systems

Advanced Adaptive Learning approaches that use machine learning to optimize each step in the Human-in-the-Loop process

Real-world use cases from well-known data scientists

Robert Munro has built Annotation, Active Learning, and machine learning systems with machine learning-focused startups and with larger companies including Amazon, Google, IBM, and most major phone manufacturers. If you speak to your phone, if your car parks itself, if your music is tailored to your taste, or if your news articles are recommended for you, then there is a good chance that Robert contributed to this experience.

Robert holds a PhD from Stanford focused on Human-in-the-Loop machine learning for healthcare and disaster response, and is a disaster response professional in addition to being a machine learning professional. A worked example throughout this text is classifying disaster-related messages from real disasters that Robert has helped respond to in the past.