Course Overview

Robotic Perception combines subfields of computer vision and signal processing that focus on the development of sensing algorithms for robots. That is, these algorithms usually have to run as part of a perception-action loop. This can involve time, energy, and computation constraints and the robot might be operating in potentially unstructured environments and challenging conditions. To address these challenges robotic platforms use a variety of sensors and algorithms for processing their data.

This class focuses on some of the perception algorithms that are relevant for robotic applications and on sensors commonly used in robotics. The aim is to provide an understanding of the challenges encountered when deploying perception systems on a robot and to introduce some of the tools and algorithms typically used to address these challenges. The class will combine a variety of sensors presenting a mix of “classical” and learning-based approaches. Among these sensors, the main focus will be on vision as one of the richest sensing modalities. Additionally, we will consider processing data from inertial measurement units (IMUs) and depth information e.g. from lidars.

Course Details

Learning Outcomes

This course aims to expose students to different types of sensor data in robotics and some of the problems commonly encountered when using such sensors. The goal of this course is to enable the derivation and implementation of several basic algorithms for each of the considered sensor types as well as when these algorithms can be utilized and what their shortcomings are. Moreover, the students will also be enabled to implement deep learning pipelines involving data from each of the considered sensing modalities.

Prerequisites

To answer the questions posed above, we will rely on concepts from linear algebra, optimization, and probabilistic reasoning, which are some of the pillars of modern AI systems. Tutorials will be provided by the teaching assistant to help students develop a solid grasp of this important background material. Algorithm implementation will be done mainly in Python or C++/C if you prefer. If your software engineering background does not include these languages, please talk to the instructor as soon as possible.

Formerly, we recommend the following courses as prerequisites:

Schedule

Week Date Lecture Tutorial
1 Jan 11 Introduction, Course Overview, Logistics NA
2 Jan 18 Math & Probability Recap NumPy & PyTorch Recap

Course Components

The course will consist of the following main components:

Evaluation

The course evaluation is based on the performance in the quizzes, homework assignments, and the final project using the following breakdown:

The quizzes will be (virtually) in-class throughout the semester and cover material from any of the previous lectures. Each student will have 3 grace days for late assignment submissions. Late submissions that exceed those grace days will lose 33% of their value for every late day beyond the allotted grace days. Late submissions that exceed three days of delay after the grace days have been used will unfortunately not be accepted. The official policy of the Registrar’s Office at UTM regarding missed exams can be found at here.

Communication Logistics

References

The course will not follow a particular textbook and we will provide information for further reading on a per-lecture basis.

Acknowledgments

Our work stands on the shoulders of giants and this course makes use of existing teaching material that was generously provided by Profs. Geiger, Kelly, Waslander, and Wetzstein.