Course Overview

Robotic Perception combines subfields of computer vision and signal processing that focus on the development of sensing algorithms for robots. That is, these algorithms usually have to run as part of a perception-action loop. This can involve time, energy, and computation constraints and the robot might be operating in potentially unstructured environments and challenging conditions. To address these challenges robotic platforms use a variety of sensors and algorithms for processing their data.

This class focuses on some of the perception algorithms that are relevant for robotic applications and on sensors commonly used in robotics. The aim is to provide an understanding of the challenges encountered when deploying perception systems on a robot and to introduce some of the tools and algorithms typically used to address these challenges. The class will combine a variety of sensors presenting a mix of “classical” and learning-based approaches. Among these sensors, the main focus will be on vision as one of the richest sensing modalities. Additionally, we will consider processing data from inertial measurement units (IMUs) and depth information e.g. from lidars.

Course Details

See Communication Logistics section below for how to contact the instructor and TA.

Learning Outcomes

This course aims to expose students to different types of sensor data in robotics and some of the problems commonly encountered when using such sensors. The goal of this course is to enable the derivation and implementation of several basic algorithms for each of the considered sensor types as well as when these algorithms can be utilized and what their shortcomings are. Moreover, the students will also be enabled to implement deep learning pipelines involving data from each of the considered sensing modalities.

Prerequisites

To answer the questions posed above, we will rely on concepts from linear algebra, optimization, and probabilistic reasoning, which are some of the pillars of modern AI systems. Tutorials will be provided by the teaching assistant to help students develop a solid grasp of this important background material. Algorithm implementation will be done mainly in Python or C++/C if you prefer. If your software engineering background does not include these languages, please talk to the instructor as soon as possible.

Formerly, we recommend the following courses as prerequisites:

Schedule

Below is the tentative schedule of this course (updated Feb 7, 2023).

Week Date Lecture Tutorial Note
1 Jan 13 Introduction, Course Overview, Logistics -  
2 Jan 20 Math & Probability Recap NumPy & PyTorch Recap  
3 Jan 27 Image Formation Python Debugger & Colab Intro A1 Release
4 Feb 3 Object Detection Quiz 1 & Attention Mechanism  
5 Feb 10 Object Tracking Quiz 2 & Discuss Transformations (e.g. Homography Estimation in A1)  
6 Feb 17 ICP Quiz 3 & QA for A1 and project A1 Due & A2 Release
7 Feb 24 Reading Week -  
8 Mar 3 IMU Quiz 4 & Discuss A1 solutions & Hint on project topics A2 Due
9 Mar 10 Coordinate-based Networks Quiz 5 & Discuss A2 solutions & QA for project A3 Release
10 Mar 17 Shape-from-X Quiz 6 & Tips on working on ML/CV projects Project Proposal Due
11 Mar 24 Self-Supervised Learning Quiz 7 & Discuss A3 solutions A3 Due
12 Mar 31 Frontiers in Robotic Perception Quiz 8 & QA for project  
13 Apr 10 Project Presentation (Mon next week instead of Fri!) - Project Report & Code Due

Course Components

The course will consist of the following main components:

Lectures & Tutorials will be recorded and posted on Quercus. Students are not required to attend classes in-person.

Evaluation

The course evaluation is based on the performance in the quizzes, homework assignments, and the final project using the following breakdown:

The quizzes will be (virtually via Quercus) in-class throughout the semester and cover materials from any of the previous lectures. They will be easy questions regarding basic concepts in the course.

TA will hold tutorials right after the lectures to discuss the quiz and provide guidance for the assignments.

Each student will have 3 grace days for late assignment submissions. Late submissions that exceed those grace days will lose 33% of their value for every late day beyond the allotted grace days. Late submissions that exceed three days of delay after the grace days have been used will unfortunately not be accepted. The official policy of the Registrar’s Office at UTM regarding missed exams can be found at here.

Communication Logistics

References

The course will not follow a particular textbook and we will provide information for further reading on a per-lecture basis.

Acknowledgments

Our work stands on the shoulders of giants and this course makes use of existing teaching material that was generously provided by Profs. Geiger, Kelly, Waslander, and Wetzstein.