Project 1: 3D Object Classification via MVCNN

• The official Python tutorial, for those who are not that familiar with Python
• The Python array facility (numpy) documentation: user guide, array objects
• Keras documentation
• StackOverflow post which describes how to check if Keras is using the GPU. If your GPU instance is configured correctly then this test should pass!
• On the Amazon instance, you may need to run this command to upgrade your Keras library: sudo pip install --upgrade keras

Policies

Assignment Overview

Part I: Single view classifier (50%)

1. For the dataset, we will be using the same ModelNet-40 dataset as used in the MVCNN paper. Since the MVCNN paper requires models to be rendered to images, to make your life easier, we make available a ModelNet-40 dataset with all models pre-rendered to images. Please download this to your project 1 directory on your cloud compute and/or local compute.
   
   
2. Again, to make your life easier, we have provided a Python ModelNet-40 loader for the above dataset. Please download this to your project directory and rename the extension to .py. The main function in this module is modelnet40_generator. This function returns a generator, which produces training/testing images and their corresponding classes for ModelNet-40. Please see the docstring for modelnet40_generator for extensive documentation on how it works, and the bottom of the module for an example of how to use it.
   
   If you are not familiar with the concept of a generator, this is important computer science knowledge, so please see the Wikipedia article on generators. You can also see this wiki article on how to implement generators in Python.
   
   
3. For this project, we will fine-tune an existing CNN. We suggest to use ResNet-50 (described in the residual networks paper), which is already included with Keras in keras.applications, with weights that have been pre-trained on ImageNet classification. We suggest to start from this example fine-tuning code, although you may also want to look through this tutorial on fine-tuning if you are not very familiar with the concept of fine tuning.
   
   
4. (25 points) Fine-tune your CNN (e.g. ResNet-50) on the ModelNet-40 dataset. You can do this by following along the fine-tuning example, and (1) instantiating a ResNet50 instance without the top layers, (2) adding a flat layer followed by a dense (fully connected) layer with 40 outputs that uses softmax, for the classification, (3) setting the first p fraction of layers to be not trainable (I used p=0.7, but you can experiment and see what gets the best results), and then (4) calling model.fit_generator() with arguments that are the ModelNet-40 training and testing set generators. Make sure this initial fine-tuning network runs (but you do not need to run it until convergence). Note: you will want to use the categorical_crossentropy loss.
   
   
5. (10 points) The above network is not very efficient because the ModelNet-40 generator produces only a single image at a time, and therefore the model effectively uses only a batch size of 1. Implement a "batching" generator that takes as input an existing ModelNet-40 generator and produces mini-batches of size n. This could be done for example by collecting n elements from the ModelNet-40 generator, and concatenating them along the batch size dimension (dimension 0 in our case). The produced data should be in the same format as the original ModelNet-40 generator, just with a larger batch size. Use the "batching" generators for training and testing to improve the efficiency.
   
   
6. (5 points) In your writeup text file, report (A) the test accuracy of this initial fine tuned network after convergence (you can stop training once the test accuracy levels off).
   
   
7. (10 points) In the MVCNN paper, dataset augmentation is also used to improve generalization (the paper calls this "jittering"). The paper reports two kinds of augmentation were used: (A) rotations by -45 degrees, 0 degree, or 45 degrees (equal probability), and (B) horizontal flips (50% probability). Given that the models are already oriented to be upright, which of these two augmentations (or both) do you think would be beneficial? Report (B) this in your writeup. Implement one of the dataset augmentation methods (each element of the mini-batch should be augmented independently). Report (C) in your writeup the test accuracy after convergence when augmentation is used.
   
   
8. Save your final code for the 1 view classifier as fit_1view.py.
   
   For reference, I obtained a test accuracy of about 89% with and without augmentation.

Part II: Multi view classifier (50%)

1. For the second part, we will implement the full network. Please see the figure above for a reminder of how this works. There are two CNN components: CNN1, which has shared parameters along all views, and CNN2, which is a single network used to output the class scores.
   
   
2. We suggest to create the CNN1 by truncating ResNet-50, so that you can easily modify your existing code from part I to solve part II. Make a copy of your part I code, and call it fit_nview.py, which implements the multi-view classifier.
   
   
3. (20 points). To construct this more complex network, we suggest to use the Keras functional API. In particular, you can create a truncated ResNet model that can become CNN1 by code such as the following:

       resnet = keras.applications.resnet50.ResNet50(include_top=False)
       resnet = keras.models.Model(resnet.input, resnet.layers[STOP_LAYER].output)
   

   Here STOP_LAYER is the layer at which to truncate CNN1. In your writeup, report (D) at which layer you decided to truncate CNN1. You may want to see the ResNet paper, Table 1 for the list of layers, and in your code you can also print resnet.layers for the list of layers. Next, you can construct a list of Input instances to represent the 12 input images for the multiple views, and pass each of these through your shared-parameter CNN1 (resnet(x) in the functional API will apply to tensor x). For details of how to do this, see the sections about multiple inputs in the Keras functional API.
   
   
4. (20 points) Implement the CNN2 part of the network. The same as the MVCNN paper, use the Keras element-wise maximum function to take the maximum over all views, and reduce your 12 tensors to a single tensor. Run this through one or more CNN layers (you can experiment with the best architecture and kernel size). We suggest to use batch normalization after each CNN to prevent vanishing and exploding gradients. Then, the same as in part I, run the CNN result through layers that flatten and apply softmax to give 40 classes. As a first step, train and test on the ModelNet-40 generators directly (no augmentation nor batching yet), and make sure this works. The generators accept an argument single=False to indicate multi-view data is being generated.
   
   
5. (10 points) Implement dataset augmentation and batching, the same as in part I. Report (E) in your writeup the final test accuracy achieved by your network, the architecture choices you made for CNN2, and how this accuracy compares with state of the art methods as reported in the O-CNN paper. For reference, my network achieved 92.5% test accuracy after 20 epochs.

Extra credit: Best performance

1. We will give +7 points of extra credit to the student who achieves the highest test accuracy after 20 epochs, +5 points to the student who achieves second-highest test accuracy, and +3 points to the student who achieves third-highest test accuracy. We will compute the accuracies ourselves based upon a single run of your multi-view CNN code (code that does not run will be disqualified from the contest). This reward is designed to encourage you to experiment with different architectures and hyperparameter settings to obtain the best performance.

Submission