Introduction to Deep Learning
Module 8 · Introduction to Deep Learning

Introduction to Deep Learning

Understand the fundamentals of neural networks and deep learning — from perceptrons and activation functions through training with backpropagation and building your first model with TensorFlow/Keras.

What Is Deep Learning?

Deep learning is a subset of machine learning that uses artificial neural networks with many layers to learn representations from data. It has revolutionized image recognition, natural language processing, speech, and generative AI.

TermMeaning
Machine LearningAlgorithms that learn from data
Deep LearningML using deep (multi-layer) neural networks
Neural NetworkLayers of connected neurons that transform inputs to outputs
LayerA group of neurons; each layer extracts higher-level features
WeightLearnable parameters; adjusted during training
BiasAdditional learnable parameter per neuron

The Neuron

A single neuron computes:

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

Where x are inputs, w are weights, b is bias, and the activation function introduces non-linearity.

Common Activation Functions

FunctionFormulaUse Case
ReLUmax(0, x)Hidden layers (default choice)
Sigmoid1 / (1 + e⁻ˣ)Binary classification output
SoftmaxNormalizes to probability distributionMulti-class output
Tanh(eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)Occasionally for hidden layers

Building a Neural Network with Keras

import tensorflow as tf
from tensorflow import keras
import numpy as np

# Load data (MNIST: handwritten digits)
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Preprocess
x_train = x_train / 255.0    # normalize to [0, 1]
x_test  = x_test  / 255.0

# Build model
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),   # 784 inputs
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dropout(0.2),                    # regularization
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(10, activation="softmax")  # 10 digit classes
])

# Compile
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

model.summary()

Training

history = model.fit(
    x_train, y_train,
    epochs=10,
    batch_size=32,
    validation_split=0.1,
    verbose=1
)

Evaluation and Prediction

test_loss, test_acc = model.evaluate(x_test, y_test)
print(f"Test accuracy: {test_acc:.4f}")

predictions = model.predict(x_test[:5])
predicted_classes = np.argmax(predictions, axis=1)

The Training Loop

ConceptWhat Happens
Forward passInput flows through layers to produce output
Loss functionMeasures error between prediction and truth
BackpropagationComputes gradient of loss w.r.t. each weight
OptimizerUpdates weights to reduce loss (Adam, SGD, RMSProp)
EpochOne full pass through the training dataset
Batch sizeNumber of samples processed before a weight update

Visualizing Training

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 4))

plt.subplot(1, 2, 1)
plt.plot(history.history["accuracy"], label="Train")
plt.plot(history.history["val_accuracy"], label="Val")
plt.title("Accuracy")
plt.legend()

plt.subplot(1, 2, 2)
plt.plot(history.history["loss"], label="Train")
plt.plot(history.history["val_loss"], label="Val")
plt.title("Loss")
plt.legend()

plt.tight_layout()
plt.show()
⚠️ Common Pitfall: Overfitting Without Regularization

A model that achieves 99% training accuracy but 70% validation accuracy has overfit — it memorized training data rather than learning generalizable patterns. Counter with: (1) Dropout layers (randomly zero out neurons during training); (2) L2 regularization (kernel_regularizer=keras.regularizers.l2(0.01)); (3) Early stopping (stop training when validation loss stops improving); (4) more training data or data augmentation.


🧠 Knowledge Check

Your model's training accuracy is 95% but validation accuracy plateaus at 72% after epoch 5, then starts to decline. What is happening, and what are three specific changes you would make to the model or training process?

Reveal Answer & Explanation

The model is overfitting — it is learning the training set too specifically. Three specific changes: (1) Add Dropout between dense layers (Dropout(0.3–0.5)) to force the network to learn redundant representations that generalize better. (2) Early stopping — add keras.callbacks.EarlyStopping(monitor="val_loss", patience=3, restore_best_weights=True) to the fit() call so training stops when validation loss starts rising, restoring the best weights seen. (3) Reduce model capacity — fewer neurons or layers reduce the model’s ability to memorize; if the problem doesn’t require deep capacity, smaller is often better. Also evaluate whether training data is representative and whether data augmentation could help.