CS 6501 - Computational Photography

Course Summary

Lectures: TuTh 12:30-1:45 Rice 340

Office hours: TuTh 2-3 PM, Rice 510, or by appointment

Course Description

Computational photography uses computers to enhance the capabilities of photography. It is at the intersection of computer graphics, computer vision, and cameras. Computational photography is used in scientific imaging, and products, e.g. Adobe Photoshop, Google Glass, NVidia mobile cameras.

We will emphasize the software side and cover topics such as:
• Camera image formation
• Multiscale decompositions
• Object removal (e.g. PatchMatch / content-aware fill)
• Retargeting (e.g. seam carving)
• Texture synthesis
• Gradient-domain editing
• High dynamic range
• Internet datasets
• Visual perception
• Non-photorealistic rendering
• Panorama stitching
• And more!

Prerequisites

Requires programming experience for projects, and basic knowledge from linear algebra (e.g. Math 3351) and calculus (Math 1310). Any knowledge from computer graphics or vision will be helpful.

Grading

There will be a couple programming assignments (35%), in class discussions (15%) and a final project (50%).

For in-class discussion you should sign up to present a paper one of the class days below (or join someone else's presentation). The presentation is informal so no slides are needed. We can bring the paper up on the projector. Have a few questions to discuss for the class. Credit is also given for contributing to paper discussions.

Late Policy

Assignments are due at 11:59 PM. Late assignments are marked down 10% per day.

Lectures

Date	Topic	Student Presentations	Student	Assignments
1. Aug 27	Introduction
2. Aug 29	Cameras and optics
3. Sept 3	Sampling and reconstruction	Smith 1995, Szeliski 2.3-2.3.1	Rafael
4. Sept 5	Color perception, light spectra	Szeliski 2.3.2	Richard
5. Sept 10	Blending and compositing	Porter and Duff 1984, Perez et al 2003
6. Sept 12	Texture synthesis (basic)	Efros and Leung 1999, Efros and Freeman 2001	Jin
7. Sept 17	Texture synthesis (PatchMatch)	Barnes et al 2009	Hui Ying
8. Sept 19	Image retargeting	Seam carving, Improved seam carving	Chi
9. Sept 24	Wavelets / frequency decompositions	Chen et al 2007	Wang, Jacob	Assignment 1 due (Tues, Sept 24)
10. Sept 26	Image warping / morphing	Beier and Neely 1992	Liu, Abbas
11. Oct 1	Guest talk -- David Luebke, NVidia
12. Oct 3	MRFs, BP, Graph cut	Kwatra et al 2003	Puneet
				Assignment 2 due (Mon, Oct 7)
13. Oct 8	Matting	Levin et al 2006	Hou
14. Oct 10	Project proposal presentations			Final project proposal due (+present)
Oct 15	No class -- reading day
Oct 17	Finish project proposal presentations
15. Oct 22	High dynamic range, tone mapping	Debevec and Malik 1997, Kalantari et al 2013	Rafael
16. Oct 24	Visual data and the Internet	Hays and Efros 2007	Ritambhara
17. Oct 29	Non-photorealistic rendering	Praun et al 2001	Samyukta
18. Oct 31	RANSAC and homographies	RANSAC, Lowe 2004	Akshay
19. Nov 5	Panorama stitching	Agrawala et al 2006	Jeremy
20. Nov 7	Light fields	Levoy and Hanrahan 1996, Gortler et al 1996	Michael
21. Nov 12	Guest -- Rick Stillings on photography
22. Nov 14	Project progress presentations			Present progress, discuss problems
23. Nov 19	Motion magnification, [1], [2], [3]	Wadhwa et al 2013	Paul
24. Nov 21	Face detection	Viola-Jones 2001, Being John Malkovich	Chen
25. Nov 26	Gigapixel imaging	Kopf et al 2007	Yong
Wed 27-29	No class -- Thanksgiving break
26. Dec 3	Deblurring	Shan et al
27. Dec 5	Coded Exposures and Apertures	Raskar et al, Levin et al	Shanshan
28. Dec 10	Final project presentation (Rice 120, 2-5 PM)			Final project writeup due (+present)

Assignments

Assignments are above. I propose only 2 assignments because I'd like more time spent on the final project and discussing research. The assignments are:


1. Image quilting / PatchMatch for image correspondence
2. Gradient domain blending

I recommend Python or MATLAB when possible because they have more image-processing facilities (arrays, libraries) than C++. However, use whatever language you are comfortable with.

Here is a Python Tutorial showing image processing.

• Photographic style transfer
• High dynamic range app for tablet
• Focus stack -- capture many focal depths on a tablet and allow user to simulate adjusting focus after the fact. See Jason Lawrence's startup app FocusTwist which does this. A simple method to determine for a given pixel which image in the stack is most in focus is to calculate for a window around the pixel a contrast measure such as standard deviation, and select the photo with the highest contrast. See also the paper Focal Stack Compositing.
• Panorama stitching
• Video textures
• Make up feature -- detect face region, detect skin color, bilateral filter against skin mattes to remove blemishes. Or reshape faces or bodies using warping as in [1]. Could use a face detector source code from OpenCV, and features inspired from commercial software.
• Computational rephotography
• Software for helping photographers create cinemagraphs. See papers such as [2], [3].
• Image object removal, using scene completion (big data approach), or Criminisi et al (small data approach).
• Image retargeting, for example seam carving using CUDA (a GPU programming language).

Proposal:

• Who will work on the project (groups of 1-3)
• What the project aims to accomplish, and any related papers that will help you read up on the technical approach
• Deliverables that you strongly expect to accomplish (e.g. a few features of the project). As well as any optional deliverables.
• What parts each person plans to code

Final Project Writeup