CS 6501 - Computational Photography

Course Summary

Lectures: TuTh 12:30-1:45 Rice 340

Office hours: Tu 3-4 PM, Thurs 2-3 PM, Rice 224 Graphics Lab (alternate location: Rice 510 my office), or by appointment

Course Description

Computational photography uses computers to enhance the capabilities of photography. It is at the intersection of computer graphics, computer vision, and cameras. Computational photography is used in scientific imaging, and products, e.g. Adobe Photoshop, Google Glass, NVidia mobile cameras.

We will emphasize the software side and cover topics such as:
• Camera image formation
• Multiscale decompositions
• Object removal (e.g. PatchMatch / content-aware fill)
• Retargeting (e.g. seam carving)
• Texture synthesis
• Gradient-domain editing
• High dynamic range
• Internet datasets
• Visual perception
• Non-photorealistic rendering
• Panorama stitching
• And more!

Prerequisites

Requires programming experience for projects, and basic knowledge from linear algebra (e.g. Math 3351) and calculus (Math 1310). Any knowledge from computer graphics or vision will be helpful.

Grading

There will be a couple programming assignments (50%), in class discussions (15%) and a final project (35%).

For in-class discussion you should sign up to present a paper one of the class days below (or join someone else's presentation). The presentation is informal so no slides are needed. We can bring the paper up on the projector. Have a few questions to discuss for the class. Credit is also given for contributing to paper discussions.

Late Policy

Assignments are due at 11:59 PM. Late assignments are marked down 10% per day.

Lectures

Date	Topic	Student Presentations	Student	Assignments
1. Aug 26	Introduction
2. Aug 28	Cameras and optics
3. Sept 2	Sampling and reconstruction	Smith 1995, Szeliski 2.3-2.3.1	_
4. Sept 4	Color perception, light spectra	Szeliski 2.3.2	_
5. Sept 9	Blending and compositing	Porter and Duff 1984, Perez et al 2003
6. Sept 11	Texture synthesis (basic)	Efros and Leung 1999, Efros and Freeman 2001	_
7. Sept 16	Texture synthesis (PatchMatch)	Barnes et al 2009	_
8. Sept 18	Image retargeting	Seam carving, Improved seam carving	_
9. Sept 23	Wavelets / frequency decompositions	Chen et al 2007	_
10. Sept 25	Image warping / morphing	Beier and Neely 1992	_
11. Sept 30	MRFs, BP, Graph cut	Kwatra et al 2003		Assignment 1 due (Tues, Sept 30)
12. Oct 2	Guest talk -- David Luebke, NVidia		_
13. Oct 7	Matting	Levin et al 2006	_
14. Oct 9	High dynamic range, discussion of final project requirements	Debevec and Malik 1997, Kalantari et al 2013	_
Oct 11-14	No class -- reading days
14. Oct 16	Project proposal presentations			Final project proposal due (+present)
16. Oct 21	Visual data and the Internet	Hays and Efros 2007	_	Assignment 2 due (Tues, Oct 21)
17. Oct 23	Non-photorealistic rendering	Praun et al 2001	_
18. Oct 28	RANSAC and homographies	RANSAC, Lowe 2004	_
19. Oct 30	Panorama stitching	Agrawala et al 2006	_
20. Nov 4	Light fields	Levoy and Hanrahan 1996, Gortler et al 1996	_
21. Nov 6	Motion magnification, [1], [2], [3]	Wadhwa et al 2013	_
22. Nov 11	Face detection	Viola-Jones 2001, Being John Malkovich	_
23. Nov 13	Project progress presentations			Present progress, discuss problems
24. Nov 18	Gigapixel imaging	Kopf et al 2007	_
25. Nov 20	Deblurring	Shan et al
26. Nov 25	Guest talk -- Tom Cogill on photography
Wed 26-30	No class -- Thanksgiving break
27. Dec 2	Coded Exposures and Apertures	Raskar et al, Levin et al	_
28. Dec 4	Convolutional Neural Networks, [1], [2], [3], [4], [5]		_
30. Dec 10	Final project presentation (Rice 340, 2-5 PM)			Final project writeup due (+present)

Assignments

The assignments are:


1. Image quilting / PatchMatch for image correspondence
2. Gradient domain blending

I recommend Python or MATLAB when possible because they have more image-processing facilities (arrays, libraries) than C++. However, use whatever language you are comfortable with.

Here is a Python Tutorial showing image processing.

• Photographic style transfer
• High dynamic range app for tablet
• Focus stack -- capture many focal depths on a tablet and allow user to simulate adjusting focus after the fact. See Jason Lawrence's startup app FocusTwist which does this. A simple method to determine for a given pixel which image in the stack is most in focus is to calculate for a window around the pixel a contrast measure such as standard deviation, and select the photo with the highest contrast. See also the paper Focal Stack Compositing.
• Panorama stitching
• Video textures
• Make up feature -- detect face region, detect skin color, bilateral filter against skin mattes to remove blemishes. Or reshape faces or bodies using warping as in [1]. Could use a face detector source code from OpenCV, and features inspired from commercial software.
• Computational rephotography
• Software for helping photographers create cinemagraphs. See papers such as [2], [3].
• Image object removal, using scene completion (big data approach), or Criminisi et al (small data approach).
• Image retargeting, for example seam carving using CUDA (a GPU programming language).

Proposal:

• Who will work on the project (groups of 1-3)
• What the project aims to accomplish, and any related papers that will help you read up on the technical approach
• Deliverables that you strongly expect to accomplish (e.g. a few features of the project). As well as any optional deliverables.
• What parts each person plans to code

Final Project Writeup