CS 6501: 3D Reconstruction and Understanding

Fall 2017

Instructor: Connelly Barnes

Course Summary

Lectures:

Office hours:

Connelly	Tues 1 to 2 PM, Thurs 1 - 2 PM, Rice 510, or by appointment via email
Xinge (Vinson)	Wed 10 - 11 AM, Rice 404, Fri 10 - 11 AM, Rice 514

Course Description

This graduate seminar will focus on topics within 3D computer vision and graphics related to reconstruction, recognition, and visualization of 3D data. There has been a trend towards 3D sensors, such as stereo and depth cameras. This information can be used to reconstruct 3D geometry, find locations and poses of humans, and reason about object relationships. We will discuss also applications such as virtual and augmented reality and robotics.

Textbook

The following textbooks are not required, but allow you to understand the course topics in greater depth. The first two hardcopy books have been placed on reserve at the Brown Science and Engineering library. The third and fourth are available online and in hardcopy.

Computer Vision: A Modern Approach by Forsyth and Ponce
Multiple View Geometry by Richard Hartley and Andrew Zisserman
Computer Vision: Algorithms and Applications by Szeliski
Deep Learning by Goodfellow, Bengio and Courville. See also [2].

Prerequisites

Requires experience writing code independently (for projects) in a language such as Python and JavaScript (although you can use a language of your choice, the projects will be set up assuming these languages). Linear algebra is required. Any graphics, machine learning, or computer vision courses will be helpful, but not required.

Grading

There will be student paper presentations (20%), programming assignments (40%), a final course project (30%), and participation (10%). For participation, points will be awarded for discussion of in-class topics.

Student Paper Presentations

The presentation format should be:

About 25 minutes (20 for presentation, 5 minutes discussion)
With slides
Attempt to show the video for the paper (if it has one)
Include a few open-ended questions for class discussion

Programming Policy

individual

group

individual

group

Late Policy

Programming assignments are due at 11:59 PM. Late projects are marked down 25% per day.

Lectures

Date	Topic	Reading (legend)	Student Presentation	Assignments
1. Aug 22	Basics: Introduction / overview
2. Aug 24	Basics: camera models and calibration	FP: I 1.1-1.2 or S Ch. 2, 6.2-6.3.1
3. Aug 29 / Aug 31	Basics: linear filters, feature detection and matching	FP II.4, II.5 or S 3.2, 4.1, and Lowe 2004 (esp Sect 3-6)
4. Sept 5	Basics: stereo / epipolar geometry	Epipolar geometry notes by George Bebis or S 11.1, also S 11.3, 11.4, blog
5. Sept 7	Basics: structure from motion (presented from Photo Tourism slides 1-30)	S 7.1-7.2, 7.4 or FP 8.1
6. Sept 12	Basics: Simultaneous Localization and Mapping (SLAM), Intro neural nets	Lie Groups tutorial, 1-2.2, 3.1, 4.1, 5.1, 6.1, LSD-SLAM, GBC 6-6.2
7. Sept 14	Basics: deep learning of visual representations (CNNs)	GBC 8-8.3
8. Sept 19	Basics: deep learning of visual representations (CNNs)	GBC 9-9.4, MVCNN
9. Sept 21	3D: dense correspondence and depth estimation	[1], [2], [3], [4]
10. Sept 26	3D: shape datasets	ModelNet, ShapeNet, ScanNet
11. Sept 28	3D: shape analysis	O-CNN, [2], OctNet, PointNet
12. Oct 5	3D: shape segmentation, shape synthesis	[1], SyncSpecCNN, [3], [4]		Assignment 1 due (MVCNN)
Reading day (no class): Tues Oct 3
13. Oct 10	3D: scene reconstruction I	[1], BundleFusion, Tanks and Temples
14. Oct 12	3D: scene reconstruction II	3DMatch, Virtual Rephotography, [3]
15. Oct 17	3D: dynamic shape reconstruction	DynamicFusion, VolumeDeform, KillingFusion (video)
16. Oct 19	3D: object detection and tracking	[1], [2] (video), deep sliding shapes
17. Oct 24	Graphics: view interpolation	[1], [2], [3]
18. Oct 26	Graphics: reconstruction of geometry, materials, and lighting	MonoFVV, SA-Net, Wang et al. 2016
19. Oct 31	Final project proposal presentations
20. Nov 2	Humans: pose estimation	VNect, [2], Gesture3D
21. Nov 7	Humans: clothing, and animal shape estimation	ClothCap, 3D Menagerie		Assignment 2 due (VR)
22. Nov 9	VR/AR: Augmented reality, face shape estimation	[1], [2], [3]
23. Nov 14	VR/AR: Face shape estimation, segmentation, transfer	[1], [2], [3]
24. Nov 16	VR/AR: Hand pose estimation	[1], [2], [3]
25. Nov 21	Robotics: Autonomous driving	[1] (video), [2], [3], [4]
Thanksgiving (no class): Thurs Nov 23
27. Nov 28	Robotics: UAVs, robotic reconstruction	[1], Horus, [3]
28. Nov 30	Final project presentations I
29. Dec 5	Final project presentations II

* Legend: FP = Forsyth and Ponce, S = Szeliski, GBC = Goodfellow, Bengio, Courville.

Final Project Proposal

Due Tues Oct 31, 11:59 PM. Present proposal in class (up to 6 minutes long). Submit a brief writeup (a third of a page to one page) to Collab consisting of:

Students who are part of the project (group size 3 to 4)
Project topic and goal
The target deliverables. These should include at least several hundred lines of code per student that compiles and runs (written by the students, any third-party code should be clearly indicated in the submission), and a final project writeup explaining the results obtained (the final writeup can be 2 to 4 pages).

Final Project Example Topics

You could select one of the below topics, or come up with another project idea on your own.

In our Project 1, many students found the MVCNN test accuracy was improved over that reported result in the paper due to simply fine-tuning from a more modern ImageNet-pretrained architecture (ResNet-50). The paper Qi et al. 2016 also reported that MVCNN test accuracy could be improved by using a multi-size sphere rendering as a regularization (see their Figure 9 and Section 5). You could investigate whether combining these two improvements: (1) pretrained ResNet-50 or other architectures and (2) multi-size sphere rendering results in even better performance.
Virtual reality painting or sculpting interfaces. See the "vive / paint" and "vive / sculpt" examples from three.js. How to improve these? You could look at the Tilt Brush game for examples of more sophisticated painting or sculpting tools.
Virtual reality stereo reconstruction for non-rectified cameras. We have a StereoLabs ZED camera in the Rice 224 lab, but it is not rectified. You could explore rectifying this imagery and then trying to obtain a high-quality reconstruction using block-based matching algorithms that search for similar patches in each horizontal scanline (similar to Project 2). Using OpenCV, for a stereo camera, one could detect keypoints, estimate fundamental matrix (using cv2.findFundamentalMat()), perform stereo rectification (using cv2.stereoRectifyUncalibrated()), and then warp both images to be rectified. The black background regions would have to be rejected to eliminate spurious matches, which may require a custom matching algorithm to be implemented. Like in Project 2, the resulting geometry could be imported as a point cloud into virtual reality. One could also repeat this process with a video (although the rectification fundamental matrix and homography should not be re-estimated for each frame) to result in a video point cloud (although JavaScript memory limitations may require either low point cloud resolution or only a few number of frames such as 30).
Virtual reality games based on Firefox and WebVR, similar to Project 2. For many games, you may want to implement a camera controller that moves forward when the trigger is pressed. You may also find it interesting to experiment with how to best use the touch controllers for game dynamics.
Shape synthesis using GANs. The paper Achlioptas et al. 2017 shows some interesting results for point cloud synthesis using GANs. You could implement something similar.
Character tracking using VNect. The VNect model can be used for tracking of human pose even in outdoor environments. You might try to do some outdoor sports related captures, such as of a person playing tennis, and then you could write some code to play these back as animated skeletons in three.js.
3D reconstruction using depth camera. There are 2 Kinect sensors in the Rice 224 lab that teams can use. There are some commercial 3D scanning tools such as ReconstructMe and open source ones such as LiveScan3D. One fun project would be to reconstruct a few objects, load them in three.js, and then assign reasonable physics to them (e.g. mass, elasticity, ability to detach or break into different component objects, ...) using a JavaScript physics engine such as Cannon.
Motion controllers for video games. One could imagine training deep networks that learns to map the input image to reasonable actions for a given video game: e.g. steering and speed for a driving game, facing direction, motion and firing for a shooting game, and so forth. These could be trained either from human input or from an existing AI. The game would have to be scriptable for ease of logging such information, and ideally using the resulting controller in the game.
Reimplement any of the above papers.
3D procedural art projects. You can see an example of Blortaria, an app that has procedurally created 3D geometry and textures. One approach would be to express the geometry as an implicit surface in 3D (i.e. write down a function f(x, y, z) and extract the level set f(x, y, z) = 0 using Marching Cubes). Then one could use fragment shaders to render an interesting animated texture on the surface, as a function of (x, y, z) position, surface normal, and time. You can see ShaderToy.com for some examples of how to write fragment shaders.