CS 6501: Large-scale Data-driven Graphics and Vision

Course Summary

Lectures: TuTh 5 PM - 6:15 PM, Room: Thornton Hall E304

Office hours: Tu 4-5 PM, Thurs 4-5 PM, Rice 224 Graphics Lab (alternate location: Rice 510 my office), or by appointment

Course Description

In this seminar, we will study Big Data graphics and computer vision algorithms. These include large-scale 3D reconstruction, machine learning techniques for photo editing, Internet-scale photo collections, deep learning, and computer vision techniques for training and inference on images with depth. A key feature of these algorithms is that they process many images, videos, 3D points, etc. For example, one might use a dataset containing hundreds to millions of images.

Prerequisites

Requires experience writing code independently (for projects). Recommended math course is linear algebra and probability. Any graphics, machine learning, or computer vision courses will be helpful, but not required.

Grading

There will be quizzes in class (15%), student paper presentations (20%), programming assignments (35%), and a final course project (30%).

For in-class discussion you should sign up to present a paper one of the class days below (or join someone else's presentation). The presentation is informal so no slides are needed. I encourage also presenting demos of startups, websites, or other examples of technologies that are related to the topic under discussion. Have a few questions to discuss for the class. Credit is also given for contributing to discussions.

Late Policy

Programming assignments are due at 11:59 PM. Late projects are marked down 10% per day.

Lectures

Date	Topic	Reading	Student Presentation	Student	Assignments
1. Aug 25	Introduction
2. Aug 27	Basics of image/video/optics
3. Sept 1	Classical computer vision: feature engineering		SIFT paper
4. Sept 3	Classical computer vision: modeling cameras/objects		Lucas-Kanade 20 Years On
5. Sept 8	Languages/frameworks: Halide, HIPI, Python, Caffe	Halide paper	HIPI
6. Sept 10	Deep learning for image classification and object detection	Krizhevsky et al. 2012	Girshick et al. 2014
7. Sept 15	Deep learning for video classification	Karpathy et al. 2014
8. Sept 17	Deep learning for feature extraction and face recognition	Razavian et al. 2014	Taigman et al. 2014		Project 1 due
9. Sept 22	Deep learning for segmentation, saliency	Pinheiro et al. 2013	Long et al. 2015, Vig et al. 2014
10. Sept 24	Deep learning: visualizing networks	Google blog post, Simonyan et al. 2014	Nguyen et al. 2015
11. Sept 29	Photo collections: scene completion and transient attributes	Scene completion	Transient attributes
12. Oct 1	Photo collections: style and enhancement	Bychkovsky et al. 2011, Gatys et al. 2015	Shih et al. 2014
No class Oct 6 (reading day)
13. Oct 8	Photo collections: patches	PatchTable	AverageExplorer, What Makes Paris...
14. Oct 13	Photo collections: intrinsic images	Intrinsic Images ITW
15. Oct 15	Photo collections: correspondence graphs	FlowWeb	Yancheshmeh et al. 2015
16. Oct 20	Images with depth: datasets	Sun3D	NYU Depth		Project 2 due
17. Oct 22	Images with depth: deep learning	SUN RGBD	Su et al., Flynn et al. (video)
18. Oct 27	Going 3D: structure from motion	Bundler	PMVS (paper)
19. Oct 29	Going 3D: city reconstruction from photo collections	Building Rome
20. Nov 3	Going 3D: scalable SLAM/SFM	Resch et al. 2015	LSD-SLAM
21. Nov 5	Going 3D: scene-space video, time-lapse	Klose et al. 2015	Time lapse mining, 3D time-lapse
22. Nov 10	Cars I: Road and trajectory estimation	Lee et al. 2013	He et al. 2004
23. Nov 12	Final Project Proposals
Nov 13					Project 3 due
24. Nov 17	Cars II: Object detection, DARPA challenges [1], [2], [3]	Trivedi et al. 2007	Dahlkamp et al. 2006
25. Nov 19	Humanoid, animal, insect robots [1], [2], [3], [4], , [6], [7]
26. Nov 24	Connectomics 1, [1], [2], [3], [4]
No class Nov 26 (Thanksgiving break)
27. Dec 1	Connectomics 2, EyeWire	Lichtman et al.	Alivisatos et al., Wang et al.
28. Dec 3	Medical, biological applications [1], [2], [3]
29. Dec 8	Final Project Presentations
30. Dec 18	Final Project Submission due on Collab at 11:59 PM (there is no final exam, so no need to show up this week)				Final project due

Programming Policy

Programming Assignments

The assignments are:


1. Mini-Project 1: Google Deep Dream Vacation, due Thurs, Sept 17 (11:59 PM)
2. Project 2: Train Your Own Artificial Neural Network, due Tues, Oct 20 (11:59 PM)
3. Project 3: Scene Completion, due Fri, Nov 13 (11:59 PM)

I recommend Python or MATLAB when possible because they have more image-processing facilities (arrays, libraries) than C++. However, use whatever language you are comfortable with.

Here is a Python Tutorial showing image processing.

• Many tiny images. Explore similar ideas as the 80 million images project, which collected 80 million tiny images. You might collect fewer images, but could explore similar applications, such as: (1) Finding similar photos, (2) Colorizing grayscale input photographs by copying colors from similar photos, (3) Orienting a photo so that it is upright. See their presentation for explanations of these applications. To get many small images you can use Bing search API or find thumbnails from ImageNet.
• Facial recognition. For example, you could collect a dataset of thousands of face photos both of yourself and your friends. You could label each photograph with the identity of the person that it is from. You could then use existing face detection technologies to extract the crop region where the face is in the image. You could investigate training different neural network classifiers that take the cropped face image as input and predict the label (identity) of the face in the training dataset. Finally, for a new test dataset containing people that were not previously seen by the system, you could investigate a Facebook-style tag transfer algorithm. For example, you could assume only 10% of the people in the test dataset have valid tags, representing the person's identity, and guess the tag for all the other photos by transferring the tag from the nearest image that does have a tag. (Here "nearest" could be defined as a Euclidean distance in a feature space, e.g. the output of the previously trained neural network). You could measure the accuracy of the tag transfer process for different trained neural networks.
• Reconstructing University buildings using structure-from-motion tools such as Bundler or SLAM tools such as LSD-SLAM. It might be especially nice to make a WebGL viewer that allows a viewer to "walk around" in the reconstructed point clouds (if using WebGL, the library three.js is strongly recommended). The visualization might be improved by (1) using simple game-like physics constraints to make a mode where one "walks" on the ground or floor, (2) growing textured quadrilaterals around each point in the point cloud, to better represent the scene geometry and texture, where each quad takes its texture from the appropriate region of an image seen from a nearby camera, or (3) similarly to the Photo Tourism project, one could instead visualize textured quadrilaterals in 3D that correspond to the camera and what each camera saw. (Note: one might have to limit the number of quadrilaterals or the resolution of the texture images used in each, based on GPU memory limits).
• Exploring average images. An interface similar to AverageExplorer would be fun. It might be particularly interesting to focus on some particular image domains, such as frontal face photos, frontal full-body photos, or side photos of different kinds of dogs.
• Neural-network based artistic style. One could generalize the paper Gatys et al. to video. One way to do this would be to first processing every frame independently. However, this will result in temporal flicker, because there is no temporal coherence. A simple solution to improve temporal coherence would be to take an average of the colors along the optical flow direction, using both forward and backwards optical flow. (One could use the existing optical flow algorithms in libraries such as OpenCV).
• Quadcopter identification. Quadcopters have been causing increasing problems for aircraft recently, and it can also be hard to identify who owns which quadcopter. One solution to increase their safety might be to have license plates. This could be done with RFID, but that would require a custom reader. Another alternative would be to use computer vision. In this project, you could first train a classifier to recognize quadcopter vs non-quadcopter images (for example, you could train a linear SVM on top of the features output by the pre-trained AlexNet deep convolutional neural network). Then you could run this classifier over many sliding windows extracted from a video containing a quadcopter, to identify where the quadcopter is most likely to be. Finally, one could experiment with placing on the quadcopter either a barcode or a blinking light (e.g. a video played on a cellphone that has a long blink, short blink, long blink, etc, to contain a hidden code). Then you could use computer vision to read the barcode or the blinking light, and obtain the "license plate."
• Visual classifiers for online shopping, online dating. It might be interesting to train say neural network classifiers (or regressors) to predict which photo of a product is most attractive, or other properties such as category of object. This could also be applied to photos on online dating websites. (Or you can come up with other problems where a classifier might be helpful!) Since just training a classifier will not involve that much programming, I also suggest to explore whether averages of large numbers of photographs can improve the "attractiveness" score output by the classifier. In particular, this could be formulated as a search process, where one starts with one photograph, and greedily tries to add every other photograph to the set of photos being averaged, and repeat as long as the attractiveness keeps increasing.
• Time-lapse mining from Internet photographs. Implement a project similar to time-lapse mining from Internet photographs.
• Scene-spacing video processing. Implement one or more applications of the scene-space video processing project.