Page 1

The Art of Neural Nets Marco Tavora

Preamble The challenge of recognizing artists given their paintings has been, for a long time, far beyond the capability of algorithms. Recent advances in deep learning, specifically the development of convolutional neural networks, have made that task possible.

Challenge Given a painting can machines identify the painter? Can you identify these painter or maybe their period/style?

Challenge Given a painting can machines identify the painter?

Both are from Picasso (1923, 1935)

Woman in Hat [Olga] by Pablo Picasso, 1923

Portrait of Olga Picasso, by Pablo Picasso, 1935

Challenge Complex but useful problem: ● Painters change styles along their career ● It is not just object or face recognition (painters can paint many different things) ● Methods employed by specialists can be invasive and cause damage do the painting ● High performance classifiers can also help identify forgeries

Deep learning to the rescue!

Deep learning ● DL methods of image processing can help. ● Most previous work on artist identification start by choosing the painters’ characteristics as features ● Since NN do not need “feature hand-crafting” they are ideal for this task

Particularly proficient at image classification

Methodology: Convolutional Neural Networks (CNNs)

They make assumption that inputs are images and that vastly reduces number of parameters

Layers are feature identifiers: Convolutional layers filter input and they identify edges, curves, ‌

Methodology: Convolutional Neural Networks (CNNs) Pooling layers shrink input dimension by an integer factor

Procedure â—? Multi-class classification problem â—? Painters must have a minimum number of paintings in the training set to ensure sufficient sample size â—? Running time increases when the number of classes increases, so I ended up choosing 3 painters (contrast with 1 painter = binary classification)

Procedure ● We have a small sample (for image recognition standards) ● Tactics to attack this problem of lack of data: ○ Image augmentation ○ Transfer learning

Image Augmentation ● Combats high expense of getting new data by creating new data from existing data ● Keras very easily manipulates images to get many altered versions of the same image ● Modifications include rotations, shifts, shears, flips, whitening and others

What is Transfer Learning? ● When knowledge gained during training in one type of problem is used to train in other type of problem (e.g. knowledge gained learning to recognize cars applies when trying to recognize trucks) ● TL and CNNs: nowadays, training CNNs from scratch is quite rare because of sample size limitations ● Pre-train a CNN on a large dataset (e.g. ImageNet with 1.2M images, 1K categories) and apply it to our task

Feature Extractor â—? We remove only the last full-connected layer of the pre-trained CNN and treat the rest as a fixed feature extractor â—? I replace the final fully-connected layer with a new one to obtain a score for each painter in my dataset instead of a score for ImageNet classes.

Feature Extractor To verify if features from Google’s InceptionV3 were useful as a starting point, I used it directly on my training data

Inception V3 in Keras

Feature Extractor

Implementation Metrics: I looked at the accuracy and (binary cross-entropy) loss for both training and validation set over the training epochs I only used rectified linear activation functions (reLU) and the Adam optimizers

Implementation My fully-connected top model had convolutional layers with dimensions 512, 256, 128,..., 16 and 8 with 50% drop-outs in between. Adding layers was very helpful with both accuracy and losses. Then I inserted a small CNN on top of the pre-trained network and trained it on those features get the classification I needed.

Metrics I Accuracy for each epoch approaches 76% and baseline is 33.3%. Also train and validation accuracies track very close indicating little overfitting

Metrics II Loss for each epoch - training and validation are also very close together (binary cross-entropy)

Conclusion I ● I used CNNs to identify the artist of a given painting. The dataset contains (after filtering) around 2000 paintings by 3 painters. Code structure allows this number to be any value (I started with 27 painters but it was too slow) ● I used transfer learning, fine-tuning the top layers of a deep pre-trained networks (Inception V3) and obtained: ○ reasonably high accuracy 76% ○ No overfitting

Conclusion II ● Confusion matrix analysis (partially done but not included in the slides) ● More thorough analysis using the KerasClassifier( ) wrapper for a classification neural network created in Keras and used in the scikit-learn library (keras.wrappers.scikit_learn) ● More painters, more paintings, larger networks, etc

Profile for marcotav65

Painter Identification using Convolutional Neural Nets  

The goal of this project is to use Transfer Learning to build a Convolutional Neural Network (or ConvNet) to identify the artist of a given...

Painter Identification using Convolutional Neural Nets  

The goal of this project is to use Transfer Learning to build a Convolutional Neural Network (or ConvNet) to identify the artist of a given...