November 16, 2017

Deep Learning Note

Deep Learning Note

Lesson 1 - MNIST

Code

import numpy as np
from keras.datasets import mnist
from keras.layers import Dense, Activation, Dropout
from keras.models import Sequential
from keras.optimizers import Adam
from keras.utils import np_utils
from keras.callbacks import EarlyStopping


(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(x_train.shape[0], 28*28).astype(np.float32)/255
x_test = x_test.reshape(x_test.shape[0], 28*28).astype(np.float32)/255
y_train = np_utils.to_categorical(y_train, 10)
y_test = np_utils.to_categorical(y_test, 10)

model = Sequential()

model.add(Dense(input_dim=28*28, units=500))
model.add(Activation('relu'))
model.add(Dropout(0.8))
model.add(Dense(units=500))
model.add(Activation('relu'))
model.add(Dropout(0.8))
model.add(Dense(units=10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['accuracy'])
early_stopping = EarlyStopping(monitor='acc', patience=2)
model.fit(x_train, y_train, batch_size=100, epochs=20, callbacks=[early_stopping])

score = model.evaluate(x_test, y_test)
print('Total loss on Testing Set: ', score[0])
print('Accuracy of Testing Set: ', score[1])

Optimizations

  1. Loss Function(mse -> crossentropy)
  2. Activation Function(sigmoid -> ReLU)
  3. Adavanced Adagrad + Momentum
  4. Early Stop
  5. Dropout

Result

Step No Opt 1 2 3 4 5
Total loss 0.0483 0.1994 0.0651 0.1039 0.0877 0.1031
Accuracy 0.6903 0.9404 0.9804 0.9797 0.9823 0.9704

Optimization Result