Beyond the Loss https://ashudva.github.io/blog/ A blog built with Quarto quarto-1.8.24 Sun, 21 Sep 2025 21:36:10 GMT Solution Approach - Analytics Vidhya November Jobathoon https://ashudva.github.io/blog/posts/2021-11-22-Approach_AV_Jobathon_Nov2021/

Table of Contents

  • Problem Statement
  • Data Wrangling
  • Results from the EDA
  • Feature Engineering
  • Modeling

Problem Statement

Develop a Machine Learning model to aid HR Department in predicting the attrition of employees.

Given

Train Dataset - Given with the following features: 1. Demographics of the employee 2. Tenure information 3. Historical data regarding the performance

Target - There is no apparent target variable given in the dataset. The target is given in the column LastWorkingDate of the dataset.

Filled the missing values in LastWorkingDate column with the ReportingDate column.

Test Dataset - Given only the Employee ID’s for which we need to predict if they will leave the company or not in the next two quarters of year 2018.

Evaluation Metric - F1 Score

Data Wrangling

To make the data more suitable for Machine Learning models, EDA, and Feature Engineering, a few Data Wrangling steps were taken: 1. Creating the Target Variable from the LastWorkingDate column 2. Fill missing values in the LastWorkingDate column with the ReportingDate column 3. Convert all the Date columns to datetime format

Results from the EDA

Following are the most prominent observations made from the EDA: 1. KDE Plots showed that the Age and Salary were normally distributed. 2. Total Business Value had a lot of zero values. 3. Salary and Age have a similar distribution for each Gender, City Category. 4. Employees with less Quarterly Rating tend to leave the company. 5. Older Employees have a much less probability of leaving the company, So JoiningYear, Tenure (in days, months, and years) would be great features to use in the model. 6. Employees who did not leave the company have a much higher number of Positive Total Business Value.

Feature Engineering

Features that were created based on EDA: 1. JoiningYear - Year in which the employee joined the company. 2. WorkingDays - Number of working days till the reporting date. 3. WorkingMonths - Number of working months till the reporting date. 4. WorkingYears - Number of working years till the reporting date. 5. Promotions - Number of promotions till the reporting date. 6. Quarterly_Rating_RA - Running Average of the QuarterlyRating. 7. Quarterly_Rating_CumSum - Cumulative sum of the QuarterlyRating. 8. Total_Business_Value_CumSum - Cumulative sum of the Total Business Value. 9. PBVCount - Number of positive Total Business Value till the reporting date. 10. NBVCount - Number of negative Total Business Value till the reporting date. 11. SalaryGrowth - Growth in Salary till the reporting date. 12. SalaryGrowthRatio - Growth in Salary till the reporting date as a ratio. 13. SalaryGrowth_WorkingDays - Ratio of SalaryGrowth to WorkingDays 14. SalaryGrowth_WorkingMonths - Ratio of SalaryGrowth to WorkingMonths 15. SalaryGrowth_WorkingYears - Ratio of SalaryGrowth to WorkingYears 16. Designation_Count - Total number of employees with the same designation till the reporting date. 17. ReportingDate_Count - Total number of employees reported on a reporting date. 18. City_ReprotingDate_Count - Total number of employees reported in a particular city on a reporting date. 19. City_Count - Total number of employees in a particular city.

Steps to assess the features: 1. Calculate the Mutual Information between the features and the target variable. 2. Create a Bar Plot to show the Mutual Information. 3. Get 5-Fold Cross Validation Score for a CatBoost Base model (without Hyper Parameter Tuning). 4. Plot the Feature Importance of the CatBoost model.

Discarded Features based on assessment: 1. SalaryGrowth_WorkingDays 2. SalaryGrowth_WorkingMonths 3. Designation_Count 4. SalaryGrowthRatio 5. SalaryGrowth 6. SalaryGrowth_WorkingYears 7. NBVCount 8. City_Count

Modeling

List of models used during the model-building: 1. XGBoost 2. CatBoost 3. LightGBM 4. Random Forest 5. MLP (with EaryStopping and ReduceLROnPlateau)

Results from the model building: 1. Used 5-Fold StratifiedKFold cross validation to assess the model at every step. 2. MLP did not perform well due to less data available for training, and quickly overfit. 3. XGBoost, LightGBM, and Radom Forest performed worse than the CatBoost model. 4. Hyper Parameter Tuning was done for all the models using Optuna. 5. All models were trained using Early Stopping Rounds. 6. Models other than CatBoost did not improve the score even after Hyper Parameter Tuning. 7. CatBoost model was the best performing model.

Final Model

  1. Ensemble of 14 CatBoost Models with different Hyper Parameters and Random SEEDs.
  2. Average of 5-Fold out-of-fold scored predictions from each model to create Meta Features.
  3. Trained a Logistic Regression model on the meta features.
  4. Final Inference on the Test Dataset - Using the Logistic Regression model.
]]> competition solution https://ashudva.github.io/blog/posts/2021-11-22-Approach_AV_Jobathon_Nov2021/ Sun, 21 Sep 2025 21:36:10 GMT Authorship Identification: Part-1 (The baseline) https://ashudva.github.io/blog/posts/2021-04-27-ConvnetBiLSTM-C50/

Abstract

Authorship identification is the task of identifying the author of a given text from a set of suspects. The main concern of this task is to define an appropriate characterization of texts that captures the writing style of authors.
As a published author usually has a unique writing style in his/her work. The writing style is mostly context independent and is discernible by a human reader.
In previous studies various stylometric models have been suggested for the aforementioned task e.g. BiLSTM, SVM, Logistic Regression, several other Deep Learning Models. But most of them fail or show poor results for either short passages or long passages and none of them were able to perform well in both cases.
Previously the best performance at authroship identification is achieved by LSTM and GRU model.

Baseline Model

For setting up a baseline for the task, I used a combination of stack of 1D-CNN with BiLSTM which gives a validation accuracy: ~62% and test accuracy: ~54% while using a fairly simple BiDirectional LSTM and CNN Architecture. And unsurprisingly the results of baseline model are pretty close to the past best performing model without any type of Tuning.

Model uses pretrained GloVe word Embeddings for text representation. GloVe uncased word embeddings were trained using Wikipedia 2014 + Gigaword 5 and it consists of 6B tokens, 400K vocab. Embeddings are available as 50d, 100d, 200d, & 300d vectors. [Source]

In most of the cases training word embeddings for the downstream task is a good idea and gives better results, albeit because of computational requirements I have used pretrained GloVE embeddings.

Utilitiy functions

These are the functions that I’ll be using to do redundant tasks in this part like: 1. Plotting train history 2. Saving figures 3. Saving and Loading pickle objects

take a look if interested!

#collapse
import matplotlib.pyplot as plt
import numpy as np
import pickle
from pathlib import Path
import keras

def save_object(obj: object, file_path: Path) -> None:
    """
    Save a python object to the disk and creates the file if does not exists already.
    Args:
        file_path - Path object for pkl file location
        obj       - object to be saved

    Returns:
        None
    """
    if not file_path.exists():
        file_path.touch()
        print(f"pickle file {file_path.name} created successfully!")
    else:
        print(f"pickle file {file_path.name} already exists!")

    with file_path.open(mode='wb') as file:
        pickle.dump(obj, file, protocol=pickle.HIGHEST_PROTOCOL)
        print(f"object {type(obj)} saved to file {file_path.name}!")


def load_object(file_path: Path) -> object:
    """
    Loads the pickle object file from the disk.
    Args:
        file_path - Path object for pkl file location

    Returns:
        object
    """
    if file_path.exists():
        with file_path.open(mode='rb') as file:
            print(f"loaded object from file {file_path.name}")
            return pickle.load(file)
    else:
        raise FileNotFoundError


def vectorize_sequence(sequences: np.ndarray, dimension: int = 10000):
    """
    Convert sequences into one-hot encoded matrix of dimension [len(sequence), dimension]

    Args: 
        sequences - ndarray of shape [samples, words]
        dimension = number of total words in vocab
    Return:
        vectorized sequence of shape [samples, one-hot-vecotor]
    """
    # Create all-zero matrix
    results = np.zeros((len(sequences), dimension))

    for (i, sequence) in enumerate(sequences):
        results[i, sequence] = 1.

    return results


def plot_history(
    history:  keras.callbacks.History,
    metric:  str = 'acc',
    save_path: Path = None,
    model_name: str = None
) -> None:
    """
    Plots the history of training of a model during epochs

    Args: history:
        model history - training history of a model
        metric - 

    Plots:
    1. Training and Validation Loss
    2. Training and Validation Accuracy
    """
    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
    ax1.plot(history.epoch, history.history.get(
        'loss'), "o", label='train loss')
    ax1.plot(history.epoch, history.history.get(
        'val_loss'), '-', label='val loss')
    ax2.plot(history.epoch, history.history.get(
        metric), 'o', label='train acc')
    ax2.plot(history.epoch, history.history.get(
        f"val_{metric}"), '-', label='val acc')
    ax1.set_xlabel("epoch")
    ax1.set_ylabel("loss")
    ax2.set_xlabel("epoch")
    ax2.set_ylabel("accuracy")
    ax1.set_title("Loss")
    ax2.set_title("Accuracy")
    f.suptitle(f"Training History: {model_name}")
    ax1.legend()
    ax2.legend()
    if save_path is not None:
        f.savefig(save_path)

Structure of notebook

  1. Data Preprocessing
    1. Load dataset
    2. Text Vectorization
    3. Configure Dataset for faster training

  2. Modelling
    1. Parse Glove Embeddings
    2. Define ConvnetBiLSTM model
    3. Load Embedding matrix

  3. Training and Evaluation

Data Preprocessing

Dataset: UCI C50 Dataset(small subset of origin RCV1 dataset) [Source] C50 dataset is widely used for authorship identification.
Dataset Specifications: >Catagories/Authors: 50
>Datapoints per class: 50
>Total Datapoints: 5000 (4500 train, 500 test)

Loading and Preprocessing the dataset

  1. 80-20 train and validation split and 500 holdout datapoints.
  2. I’ll use text_dataset_from_directory utility of keras library to load dataset which is faster than manually reading the text.
  3. In the preprocessing step, numbers and special characters except {.} {,} and {'} are removed from the dataset.
# collapse
# Import Python Regular Expression library
import re

src_dir = Path('data/C50_raw/')
src_test_dir = src_dir / 'test'
src_train_dir = src_dir / 'train'

dst_dir = Path('data/C50/')
dst_test_dir = dst_dir / 'test'
dst_train_dir = dst_dir / 'train'


test_sub_dirs = src_test_dir.iterdir()
train_sub_dirs = src_train_dir.iterdir()

for i, author in enumerate(test_sub_dirs):
    author_name = author.name
    dst_author = dst_test_dir / author_name
    dst_author.mkdir()
    for file in author.iterdir():
        file_name = file.name
        dst = dst_author / file_name
        raw_text = file.read_text(encoding='utf-8')
        out_text = re.sub("[^A-Za-z.',]+", " ", raw_text)
        dst.write_text(out_text, encoding='utf-8')

for i, author in enumerate(train_sub_dirs):
    author_name = author.name
    dst_author = dst_train_dir / author_name
    dst_author.mkdir()
    for file in author.iterdir():
        file_name = file.name
        dst = dst_author / file_name
        raw_text = file.read_text(encoding='utf-8')
        out_text = re.sub("[^A-Za-z.',]+", " ", raw_text)
        dst.write_text(out_text, encoding='utf-8')
# collapse
nfiles_test = len(list(dst_test_dir.glob("*/*.txt")))
nfiles_train = len(list(dst_train_dir.glob("*/*.txt")))
print(f"Number of files in processed test dataset: {nfiles_test}")
print(f"Number of files in processed train dataset: {nfiles_train}")
Number of files in processed test dataset: 500
Number of files in processed train dataset: 4500

Load Dataset

#collapse
import keras
import numpy as np
import tensorflow as tf
from keras import models, layers
from keras.preprocessing import text_dataset_from_directory
from keras.layers.experimental.preprocessing import TextVectorization
import keras.callbacks as cb

ds_dir = Path('data/C50/')
train_dir = ds_dir / 'train'
test_dir = ds_dir / 'test'
seed = 123
batch_size = 32

train_ds = text_dataset_from_directory(
    train_dir,
    label_mode='categorical',
    seed=seed,
    shuffle=True,
    batch_size=batch_size,
    validation_split=0.2,
    subset='training'
)

val_ds = text_dataset_from_directory(
    train_dir,
    label_mode='categorical',
    seed=seed,
    shuffle=True,
    batch_size=batch_size,
    validation_split=0.2,
    subset='validation')

test_ds = text_dataset_from_directory(
    test_dir,
    label_mode='categorical',
    seed=seed,
    shuffle=True,
    batch_size=batch_size)
Found 4500 files belonging to 50 classes.
Using 3600 files for training.
Found 4500 files belonging to 50 classes.
Using 900 files for validation.
Found 500 files belonging to 50 classes.

Inspect dataset

class_names = test_ds.class_names
class_names = np.asarray(class_names)
print(f"nclasses: {len(class_names)}")
print(f'first 4 classes/users: {class_names[:4]}')
for texts, labels in train_ds.take(1):
    print("Shape of texts", texts.shape)
    print(f'Class of 2nd data point: {class_names[labels.numpy()[1].astype(bool)]}')
nclasses: 50
first 4 classes/users: ['AaronPressman' 'AlanCrosby' 'AlexanderSmith' 'BenjaminKangLim']
Shape of texts (32,)
Class of 2nd data point: ['GrahamEarnshaw']
#collapse
MAX_LEN_TRAIN = 0
MAX_LEN_TEST = 0
for file in test_dir.glob('*/*.txt'):
    with file.open() as f:
        seq_len = 0
        for line in f.readlines():
            seq_len += len(line.split())
#         print(seq_len)
        if MAX_LEN_TEST < seq_len:
            MAX_LEN_TEST = seq_len
for file in train_dir.glob('*/*.txt'):
    with file.open() as f:
        seq_len = 0
        for line in f.readlines():
            seq_len += len(line.split())
#         print(seq_len)
        if MAX_LEN_TRAIN < seq_len:
            MAX_LEN_TRAIN = seq_len
print(f"length of largest article in train dataset: {MAX_LEN_TRAIN}")
print(f"length of largest article in test dataset: {MAX_LEN_TEST}")
length of largest article in train dataset: 1498
length of largest article in test dataset: 1474
#collapse
for batch, label in iter(val_ds):
    index = np.argmax(label.numpy(), axis=1).astype(np.int)
    print(f'Users of first batch: {class_names[index]}')
    break
Users of first batch: ['BradDorfman' 'JaneMacartney' 'RobinSidel' 'JanLopatka' 'GrahamEarnshaw'
 'SamuelPerry' 'KouroshKarimkhany' "LynneO'Donnell" 'JaneMacartney'
 'FumikoFujisaki' 'MarkBendeich' 'LynnleyBrowning' 'JanLopatka'
 'EdnaFernandes' 'SimonCowell' 'KirstinRidley' 'MatthewBunce'
 'MichaelConnor' 'KeithWeir' 'HeatherScoffield' 'MarcelMichelson'
 'PatriciaCommins' 'MureDickie' 'TanEeLyn' 'MichaelConnor' 'MureDickie'
 'MartinWolk' 'TanEeLyn' 'ScottHillis' 'KirstinRidley' 'ToddNissen'
 'MichaelConnor']

Text Vectorization

Text vectorization includes the following tasks using TextVectorization layer: 1. Standardization 2. Tokenization 3. Vectorization

Initial run of the vectorization layers

  1. Make a text-only dataset (without labels), then call adapt
  2. Do not call adapt on test dataset to prevent data-leak
  3. train and save vocab to disk

Note: Use it only for the first time or if vocab is not saved

#collapse
from utils import save_object

### Define vectorization layers
VOCAB_SIZE = 34000
MAX_LEN = 1450

vectorize_layer = TextVectorization(
    max_tokens=VOCAB_SIZE,
    output_mode='int',
    output_sequence_length=MAX_LEN
)

# Train the layers to learn a vocab
train_text = train_ds.map(lambda text, lables: text)
vectorize_layer.adapt(train_text)


# Save the vocabulary to disk
# Run this cell for the first time only
vocab = vectorize_layer.get_vocabulary()
vocab_path = Path('vocab/vocab_C50.pkl')
save_object(vocab, vocab_path)
vocab_len = len(vocab)
print(f"vocab size of vectorizer: {vocab_len}")
pickle file vocab_C50.pkl already exists!
object <class 'list'> saved to file vocab_C50.pkl!
vocab size of vectorizer: 34000

Vectorization layers from saved vocab

Only run after first saving the vocabulary to the disk!

# collapse
# # Load vocab
# from utils import load_object
# vocab_path = Path('vocab/vocab_C50.pkl')
# vocab = load_object(vocab_path)

# VOCAB_SIZE = 34000
# MAX_LEN  = 1500

# vectorize_layer = TextVectorization(
#     max_tokens=VOCAB_SIZE, 
#     output_mode='int',
#     output_sequence_length=MAX_LEN,
#     vocabulary=vocab
# )

Configure dataset

This is the final step of the data-processing pipeline where the text is converted into vectors, and then to train the model faster, dataset is prefetched and cached before each epoch. Prefetch is especially efficient when training on a GPU as the CPU fetch and cache the dataset while GPU is training in parallel and after finishing current epoch GPU doesn’t have to wait for CPU to load the data for next epoch.

def vectorize(text, label):
    text = tf.expand_dims(text, -1)
    return vectorize_layer(text), label

AUTOTUNE = tf.data.AUTOTUNE
def prepare(ds):
    return ds.cache().prefetch(buffer_size=AUTOTUNE)

train_ds = train_ds.map(vectorize)
val_ds = val_ds.map(vectorize)
test_ds = test_ds.map(vectorize)

# Configure the datasets for fast training 
train_ds = prepare(train_ds)
val_ds = prepare(val_ds)
test_ds = prepare(test_ds)

Modelling

Now comes the most interesting part!
Below cell first creates an emb_index dictionary which maps words to a 100-Dimensional embedding vector and then emb_matrix is created which maps each word in C50 vocab to corresponding embedding.

##### Pretrained Glove Embeddings #####
## Parse the weights
from utils import load_object
emb_dim = 100
glove_file = Path(f'vocab/glove/glove.6B.{emb_dim}d.txt')
emb_index = {}
with glove_file.open(encoding='utf-8') as f:
    for line in f.readlines():
        values = line.split()
        word = values[0]
        coef = values[1:]
        emb_index[word] = coef

##### Getting embedding weights #####
vocab = load_object(Path('vocab/vocab_C50.pkl'))
emb_matrix = np.zeros((VOCAB_SIZE, emb_dim))
for index, word in enumerate(vocab):
    # get coef of word
    emb_vector = emb_index.get(word)
    if emb_vector is not None:
        emb_matrix[index] = emb_vector
print(f"Embedding Dimensionality: {emb_matrix.shape}")
loaded object from file vocab_C50.pkl
Embedding Dimensionality: (34000, 100)
import keras.backend as K
from keras.utils import plot_model
from keras import regularizers
K.clear_session()

lstm_model = models.Sequential([
    layers.Embedding(VOCAB_SIZE, emb_dim, input_shape=(MAX_LEN,)),
    layers.SpatialDropout1D(0.3),
    layers.Conv1D(256, 11, activation='relu'),
    layers.MaxPooling1D(7),
    layers.Dropout(0.4),
    layers.BatchNormalization(),
    layers.Bidirectional(layers.LSTM(128, return_sequences=True)),
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Conv1D(128, 3, activation='relu'),
    layers.MaxPooling1D(3),
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Conv1D(64, 3, activation='relu'),
    layers.GlobalMaxPooling1D(),
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Dense(128, activation='relu', kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4)),
    layers.BatchNormalization(),
    layers.Dense(50, activation='softmax')
])

Detailed Architecture of the model

Load the emb_matrix as wieghts of the embedding layer of model and then set them as non-trainable.

lstm_model.layers[0].set_weights([emb_matrix])
lstm_model.layers[0].trainable = False
plot_model(lstm_model, show_layer_names=False, show_shapes=True, to_file="models/base.png")

Training the model

# from keras.optimizers import RMSprop

K.clear_session()
# optim = RMSprop(lr=1e-2)

################# Configure Callbacks #################
# Early Stopping
es = cb.EarlyStopping(
    monitor='val_loss',
    min_delta=5e-4,
    patience=5,
    verbose=1,
    mode='auto',
    restore_best_weights=True
)

# ReduceLROnPlateau
reduce_lr = cb.ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.4,
    patience=3,
    verbose=1,
    mode='auto',
    min_delta=5e-3,
    min_lr=1e-6
)

# Tensorboard
tb = cb.TensorBoard(
    log_dir="./logs",
    write_graph=True,
)


################# Model Training #################
lstm_model.compile(
    loss='CategoricalCrossentropy',
    optimizer='adam',
    metrics=['acc']
)

lstm_history = lstm_model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=100,
    callbacks=[es, reduce_lr, tb]
)

lstm_model.save('models/base.h5')
Epoch 1/100
113/113 [==============================] - 35s 220ms/step - loss: 4.5336 - acc: 0.0188 - val_loss: 3.9713 - val_acc: 0.0200
Epoch 2/100
113/113 [==============================] - 13s 114ms/step - loss: 4.2080 - acc: 0.0312 - val_loss: 3.8599 - val_acc: 0.0278
Epoch 3/100
113/113 [==============================] - 13s 116ms/step - loss: 3.8997 - acc: 0.0403 - val_loss: 3.5842 - val_acc: 0.0544
Epoch 4/100
113/113 [==============================] - 13s 118ms/step - loss: 3.6826 - acc: 0.0454 - val_loss: 3.4762 - val_acc: 0.0467
Epoch 5/100
113/113 [==============================] - 13s 118ms/step - loss: 3.4679 - acc: 0.0585 - val_loss: 3.2102 - val_acc: 0.0900
Epoch 6/100
113/113 [==============================] - 13s 119ms/step - loss: 3.2866 - acc: 0.0862 - val_loss: 3.1259 - val_acc: 0.0844
Epoch 7/100
113/113 [==============================] - 13s 119ms/step - loss: 3.1113 - acc: 0.0972 - val_loss: 2.9419 - val_acc: 0.1533
Epoch 8/100
113/113 [==============================] - 17s 153ms/step - loss: 2.9159 - acc: 0.1459 - val_loss: 2.9488 - val_acc: 0.1456
Epoch 9/100
113/113 [==============================] - 17s 153ms/step - loss: 2.7023 - acc: 0.1737 - val_loss: 2.8818 - val_acc: 0.1544
Epoch 10/100
113/113 [==============================] - 17s 151ms/step - loss: 2.4595 - acc: 0.2387 - val_loss: 2.2731 - val_acc: 0.2889
Epoch 11/100
113/113 [==============================] - 17s 153ms/step - loss: 2.3077 - acc: 0.2764 - val_loss: 2.0811 - val_acc: 0.3033
Epoch 12/100
113/113 [==============================] - 17s 154ms/step - loss: 2.1691 - acc: 0.3154 - val_loss: 2.2417 - val_acc: 0.2622
Epoch 13/100
113/113 [==============================] - 18s 156ms/step - loss: 2.0237 - acc: 0.3493 - val_loss: 2.1130 - val_acc: 0.3178
Epoch 14/100
113/113 [==============================] - 17s 152ms/step - loss: 1.9448 - acc: 0.3544 - val_loss: 1.8156 - val_acc: 0.4133
Epoch 15/100
113/113 [==============================] - 17s 154ms/step - loss: 1.8491 - acc: 0.3828 - val_loss: 1.8381 - val_acc: 0.3789
Epoch 16/100
113/113 [==============================] - 17s 150ms/step - loss: 1.7508 - acc: 0.3922 - val_loss: 1.8668 - val_acc: 0.3789
Epoch 17/100
113/113 [==============================] - 18s 156ms/step - loss: 1.6681 - acc: 0.4339 - val_loss: 1.7776 - val_acc: 0.4022
Epoch 18/100
113/113 [==============================] - 17s 153ms/step - loss: 1.6276 - acc: 0.4328 - val_loss: 1.6306 - val_acc: 0.4411
Epoch 19/100
113/113 [==============================] - 17s 154ms/step - loss: 1.5808 - acc: 0.4466 - val_loss: 1.6069 - val_acc: 0.4522
Epoch 20/100
113/113 [==============================] - 17s 155ms/step - loss: 1.4749 - acc: 0.4696 - val_loss: 1.7228 - val_acc: 0.4089
Epoch 21/100
113/113 [==============================] - 17s 152ms/step - loss: 1.4379 - acc: 0.4849 - val_loss: 1.5438 - val_acc: 0.4567
Epoch 22/100
113/113 [==============================] - 17s 150ms/step - loss: 1.4126 - acc: 0.4781 - val_loss: 1.5012 - val_acc: 0.4667
Epoch 23/100
113/113 [==============================] - 17s 153ms/step - loss: 1.3508 - acc: 0.5022 - val_loss: 1.5094 - val_acc: 0.4467
Epoch 24/100
113/113 [==============================] - 17s 152ms/step - loss: 1.3220 - acc: 0.5124 - val_loss: 1.3901 - val_acc: 0.5067
Epoch 25/100
113/113 [==============================] - 17s 151ms/step - loss: 1.2896 - acc: 0.5279 - val_loss: 1.3944 - val_acc: 0.5056
Epoch 26/100
113/113 [==============================] - 17s 153ms/step - loss: 1.2639 - acc: 0.5314 - val_loss: 1.3389 - val_acc: 0.5244
Epoch 27/100
113/113 [==============================] - 17s 155ms/step - loss: 1.2449 - acc: 0.5533 - val_loss: 1.4641 - val_acc: 0.5044
Epoch 28/100
113/113 [==============================] - 17s 154ms/step - loss: 1.2253 - acc: 0.5574 - val_loss: 1.2490 - val_acc: 0.5756
Epoch 29/100
113/113 [==============================] - 17s 153ms/step - loss: 1.1860 - acc: 0.5534 - val_loss: 1.2377 - val_acc: 0.5744
Epoch 30/100
113/113 [==============================] - 17s 153ms/step - loss: 1.1334 - acc: 0.5763 - val_loss: 1.3690 - val_acc: 0.5100
Epoch 31/100
113/113 [==============================] - 17s 154ms/step - loss: 1.1058 - acc: 0.5856 - val_loss: 1.3175 - val_acc: 0.5678
Epoch 32/100
113/113 [==============================] - 17s 152ms/step - loss: 1.0955 - acc: 0.5932 - val_loss: 1.2472 - val_acc: 0.5667

Epoch 00032: ReduceLROnPlateau reducing learning rate to 0.0004000000189989805.
Epoch 33/100
113/113 [==============================] - 17s 151ms/step - loss: 1.0554 - acc: 0.6074 - val_loss: 1.2565 - val_acc: 0.5433
Epoch 34/100
113/113 [==============================] - 17s 153ms/step - loss: 1.0444 - acc: 0.6218 - val_loss: 1.2164 - val_acc: 0.5711
Epoch 35/100
113/113 [==============================] - 17s 154ms/step - loss: 0.9743 - acc: 0.6205 - val_loss: 1.1523 - val_acc: 0.5956
Epoch 36/100
113/113 [==============================] - 17s 150ms/step - loss: 0.9554 - acc: 0.6343 - val_loss: 1.2389 - val_acc: 0.5656
Epoch 37/100
113/113 [==============================] - 18s 157ms/step - loss: 0.9462 - acc: 0.6412 - val_loss: 1.1904 - val_acc: 0.5767
Epoch 38/100
113/113 [==============================] - 18s 163ms/step - loss: 0.9341 - acc: 0.6527 - val_loss: 1.1958 - val_acc: 0.5922

Epoch 00038: ReduceLROnPlateau reducing learning rate to 0.00016000000759959222.
Epoch 39/100
113/113 [==============================] - 18s 156ms/step - loss: 0.8854 - acc: 0.6792 - val_loss: 1.1309 - val_acc: 0.6078
Epoch 40/100
113/113 [==============================] - 17s 153ms/step - loss: 0.8495 - acc: 0.6747 - val_loss: 1.1413 - val_acc: 0.6022
Epoch 41/100
113/113 [==============================] - 18s 157ms/step - loss: 0.8854 - acc: 0.6635 - val_loss: 1.1481 - val_acc: 0.6044
Epoch 42/100
113/113 [==============================] - 17s 154ms/step - loss: 0.8528 - acc: 0.6802 - val_loss: 1.1488 - val_acc: 0.5967

Epoch 00042: ReduceLROnPlateau reducing learning rate to 6.40000042039901e-05.
Epoch 43/100
113/113 [==============================] - 18s 155ms/step - loss: 0.8528 - acc: 0.6895 - val_loss: 1.1268 - val_acc: 0.6056
Epoch 44/100
113/113 [==============================] - 17s 154ms/step - loss: 0.8339 - acc: 0.6923 - val_loss: 1.1091 - val_acc: 0.6167
Epoch 45/100
113/113 [==============================] - 18s 156ms/step - loss: 0.8147 - acc: 0.6870 - val_loss: 1.1009 - val_acc: 0.6122
Epoch 46/100
113/113 [==============================] - 17s 153ms/step - loss: 0.8440 - acc: 0.6834 - val_loss: 1.1055 - val_acc: 0.6167
Epoch 47/100
113/113 [==============================] - 18s 157ms/step - loss: 0.8263 - acc: 0.6752 - val_loss: 1.1083 - val_acc: 0.6122
Epoch 48/100
113/113 [==============================] - 18s 158ms/step - loss: 0.7950 - acc: 0.7115 - val_loss: 1.1292 - val_acc: 0.6078

Epoch 00048: ReduceLROnPlateau reducing learning rate to 2.560000284574926e-05.
Epoch 49/100
113/113 [==============================] - 18s 157ms/step - loss: 0.8392 - acc: 0.6807 - val_loss: 1.1152 - val_acc: 0.6133
Epoch 50/100
113/113 [==============================] - 17s 154ms/step - loss: 0.8248 - acc: 0.7110 - val_loss: 1.1076 - val_acc: 0.6167
Restoring model weights from the end of the best epoch.
Epoch 00050: early stopping
# Model evaluation
print('Model evaluation on test dataset')
lstm_model.evaluate(test_ds)

# Plot training history
plot_history(
    lstm_history,
    model_name="ConvnetBiLSTM",
    save_path=Path('plots/base.jpg')
)
Model evaluation on test dataset
16/16 [==============================] - 1s 58ms/step - loss: 1.4009 - acc: 0.5360

Summary

The simple BiLSTM + Conv1D model performs fairly well considering that it was not tuned for performance and provides a good baseline to work with. A clear thing to note here is that, While performance of the model on validation dataset is quite good but it performs poorly on the test. Model doesn’t generalize well on the holdout dataset which is our primary goal, in the text part I’ll try to improve the accuracy to make it better than the baseline.

]]> project NLP LLM https://ashudva.github.io/blog/posts/2021-04-27-ConvnetBiLSTM-C50/ Sun, 21 Sep 2025 21:36:10 GMT Enable Fira Code and Ligatures in code cells https://ashudva.github.io/blog/posts/2020-12-30-Fira-Code/ This tutorial is beginner friendly so if you already know about fira code you might want to skip this part and move to Using Fira Code and Ligatures.

For those who are not familiar with fira code, it is one of the most popular fonts for coding, and the reason for that is - not only it’s an artistic font, it also provides Ligatures which are symbols for common programming multi-character combinations.

Let’s take a look at some examples

Character Combination: ->

Ligature:

->

Character Combination: !=

Ligature:

!=

Character Combination: ==

Ligature:

==

There’s no reason to discuss all the details about the font, head to wikipedia for more information or go to the FiraCode GitHub link if you want to use fira code anywhere else and for other examples of ligatures and use cases.

Using Fira Code and Ligatures

@import url('https://fonts.googleapis.com/css2?family=Fira+Code&display=swap');

.input_area pre, .input_area div, code {
  font-family: 'Fira Code' !important;
  font-variant-ligatures: initial !important;
}

Add these lines in /_sass/minima/custom-styles.scss at root of your repository and you are good to go.

{% include info.html text=“Fira Code font is only used in code cells of both notebook and markdown” %}

{% include alert.html text=“Clear browser cache and wait for github actions to finish executing if the effects do not appear immediately” %}

Use this link to get my other font-styles and if you decide to create a new font-style.scss file as in my case, do not forget to add @import "minima/font-style"; in custom-styles.scss

]]> tutorial productivity https://ashudva.github.io/blog/posts/2020-12-30-Fira-Code/ Sun, 21 Sep 2025 21:36:10 GMT Authorship Identification - Part-2 (DistilBERT Transformer) https://ashudva.github.io/blog/posts/2021-05-26-DistilBERT/

Abstract

This is a follow-up post on the authorship identification project.
I regard the past few years as the inception of the era of Transformers which started with the popular Research Paper “Attention is all you need” by “somebody” in 2020. Several transformer architectures have shown up since then. Some of the famous ones are - GPT, GPT2, and the latest GPT3 which has outperformed many previous state-of-the-art models at several tasks in NLP, BERT (by Google) is also one of the most popular transformers out there.
Transformers are very large models with multi-billions of parameters. Pretrained transformers have shown tremendous capability when used with a downstream task head in Transfer Learning similar to the CNNs in Computer Vision.
In this part, I’ll use fine-tuned DistilBERT transformer which is a smaller version of the original BERT for the downstream classification task.
I’ll use the transformers library from Huggingface which consists of numerous state-of-the-art transformers and supports several downstream tasks out of the box. In short, I consider Huggingface a great starting point for a person engrossed in NLP and it offers tons of great functionalities.
I’ll provide links to resources for you to learn more about these technologies. 

{% raw %}
<div class="input_area">
import keras
import tensorflow as tf
import numpy as np
from pathlib import Path
from utils import plot_history
from keras.preprocessing import text_dataset_from_directory

ds_dir = Path('data/C50/')
train_dir = ds_dir / 'train'
test_dir = ds_dir / 'test'
seed = 1000
batch_size = 16


train_ds = text_dataset_from_directory(train_dir,
                                     label_mode='int',
                                     seed=seed,
                                     shuffle=True,
                                     validation_split=0.2,
                                     subset='training')

val_ds = text_dataset_from_directory(train_dir,
                                      label_mode='int',
                                      seed=seed,
                                      shuffle=True,
                                      validation_split=0.2,
                                     subset='validation')

test_ds = text_dataset_from_directory(test_dir,
                                       label_mode='int',
                                       seed=seed,
                                       shuffle=True,
                                       batch_size=batch_size)

class_names = train_ds.class_names

</div>
{% endraw %}

{% raw %}
<div class="input_area">
from utils import prepare_batched
from transformers import DistilBertTokenizerFast

AUTOTUNE = tf.data.AUTOTUNE

tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')

batch_size = 2

train_ds = prepare_batched(train_ds, tokenizer, batch_size=batch_size)
val_ds = prepare_batched(val_ds, tokenizer, batch_size=batch_size)
test_ds = prepare_batched(test_ds, tokenizer, batch_size=batch_size)

</div>
{% endraw %}

{% raw %}
<div class="input_area">
from transformers import TFAutoModelForSequenceClassification
keras.backend.clear_session()

model = TFAutoModelForSequenceClassification.from_pretrained(
    'distilbert-base-uncased', num_labels=50)

model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=5e-5),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=tf.metrics.SparseCategoricalAccuracy()
)

history = model.fit(train_ds, validation_data=val_ds, epochs=20)

plot_history(history, 'sparse_categorical_accuracy')
model.save("DistilBERT_finetuned.h5")

</div>
{% endraw %}

{% raw %}
<div class="input_area">
print("Evaluate the model on test dataset")
model.evaluate(test_ds)

</div>
{% endraw %}
]]> project NLP LLM https://ashudva.github.io/blog/posts/2021-05-26-DistilBERT/ Sun, 21 Sep 2025 21:36:10 GMT Credit Card Lead Prediction - EDA https://ashudva.github.io/blog/posts/2021-11-30-EDA/

Problem Statement

Credit Card Lead Prediction

Happy Customer Bank is a mid-sized private bank that deals in all kinds of banking products, like Savings accounts, Current accounts, investment products, credit products, among other offerings. The bank also cross-sells products to its existing customers and to do so they use different kinds of communication like tele-calling, e-mails, recommendations on net banking, mobile banking, etc. In this case, the Happy Customer Bank wants to cross sell its credit cards to its existing customers. The bank has identified a set of customers that are eligible for taking these credit cards.

Now, the bank is looking for your help to understand various patterns among the data that might be useful in identifying customers that could show higher intent towards a recommended credit card, given: 1. Customer details (gender, age, region etc.) 2. Details of his/her relationship with the bank (Channel_Code,Vintage, ’Avg_Asset_Value etc.)

Imports and Display Options

# collapse
# Imports
import warnings
import numpy as np
import pandas as pd
import seaborn as sns
from pathlib import Path
import matplotlib.pyplot as plt

# Set output display options
warnings.filterwarnings('ignore')
%matplotlib inline

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.options.display.float_format = '{:.3f}'.format

# Set color palette for plots
sns.set_palette('Set2')

# Plots Background color
bg_color = '#f6f5f5'

Data Eyeballing

# Load data and drop ID column
data_dir = Path('./data')
df = pd.read_csv(data_dir / 'train.csv')
df.drop('ID', axis=1, inplace=True)

# Convert Is_Lead column into categorical variable
df['Is_Lead'] = df['Is_Lead'].astype('category')
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 245725 entries, 0 to 245724
Data columns (total 10 columns):
 #   Column               Non-Null Count   Dtype   
---  ------               --------------   -----   
 0   Gender               245725 non-null  object  
 1   Age                  245725 non-null  int64   
 2   Region_Code          245725 non-null  object  
 3   Occupation           245725 non-null  object  
 4   Channel_Code         245725 non-null  object  
 5   Vintage              245725 non-null  int64   
 6   Credit_Product       216400 non-null  object  
 7   Avg_Account_Balance  245725 non-null  int64   
 8   Is_Active            245725 non-null  object  
 9   Is_Lead              245725 non-null  category
dtypes: category(1), int64(3), object(6)
memory usage: 17.1+ MB

Dataset can be considered small, as there are only 9 Independant variables and ~250K Entries. This information can be used to select the model and cross-validation strategy.

df.head()
Gender Age Region_Code Occupation Channel_Code Vintage Credit_Product Avg_Account_Balance Is_Active Is_Lead
0 Female 73 RG268 Other X3 43 No 1045696 No 0
1 Female 30 RG277 Salaried X1 32 No 581988 No 0
2 Female 56 RG268 Self_Employed X3 26 No 1484315 Yes 0
3 Male 34 RG270 Salaried X1 19 No 470454 No 0
4 Female 30 RG282 Salaried X1 33 No 886787 No 0 
# collapse
# Categorical cols
cat_cols = df.select_dtypes(include=['category', 'object']).columns

# Numerical cols
num_cols = df.select_dtypes(include=['int64']).columns
print(f'''
    {len(cat_cols)}-Categorical Columns: {cat_cols.tolist()},
    {len(num_cols)}-Numerical Columns: {num_cols.tolist()}
    ''')

    7-Categorical Columns: ['Gender', 'Region_Code', 'Occupation', 'Channel_Code', 'Credit_Product', 'Is_Active', 'Is_Lead'],
    3-Numerical Columns: ['Age', 'Vintage', 'Avg_Account_Balance']
    
# Check for missing values
df.isnull().sum()
Gender                     0
Age                        0
Region_Code                0
Occupation                 0
Channel_Code               0
Vintage                    0
Credit_Product         29325
Avg_Account_Balance        0
Is_Active                  0
Is_Lead                    0
dtype: int64
print(f'{df.isnull().sum().sum() / df.shape[0] * 100: .3f}%\
    values in \'Credit_Product\' are missing')
 11.934%    values in 'Credit_Product' are missing
print(df['Vintage'].nunique(), f'Unique values in \'Vintage\'',
      df['Age'].nunique(), f'Unique values in \'Age\'')
66 Unique values in 'Vintage' 63 Unique values in 'Age'

Descriptive Statistics

# Descriptive statistics for numerical columns
df.describe()
Age Vintage Avg_Account_Balance
count 245725.000 245725.000 245725.000
mean 43.856 46.959 1128403.101
std 14.829 32.353 852936.356
min 23.000 7.000 20790.000
25% 30.000 20.000 604310.000
50% 43.000 32.000 894601.000
75% 54.000 73.000 1366666.000
max 85.000 135.000 10352009.000 
# Descriptive statistics for categorical columns
df.describe(include=['O', 'category'])
Gender Region_Code Occupation Channel_Code Credit_Product Is_Active Is_Lead
count 245725 245725 245725 245725 216400 245725 245725
unique 2 35 4 4 2 2 2
top Male RG268 Self_Employed X1 No No 0
freq 134197 35934 100886 103718 144357 150290 187437

Univariate Analysis

# Plot KDE Plots for all numerical columns
fig, axes = plt.subplots(nrows=3, ncols=3, figsize=(15, 9))
for i, col in enumerate(num_cols):
    sns.kdeplot(df[col], shade=True, label=col, ax=axes[0, i], color='Tomato')
    sns.distplot(df[col], hist=True, kde=False, label=col, bins=20, ax=axes[1, i])
    sns.boxplot(x=col, data=df, ax=axes[2, i], color='#D1E4CD')

    # set title and remove x-axis labels
    axes[0, i].set_xlabel("")
    axes[1, i].set_xlabel("")
    axes[2, i].set_xlabel("")
    axes[0, i].set_ylabel("")
    axes[0, i].set_title(col)

    # Remove Spines
    for spine in ['right', 'top']:
        axes[0, i].spines[spine].set_visible(False)
        axes[1, i].spines[spine].set_visible(False)
        axes[2, i].spines[spine].set_visible(False)
plt.tight_layout()

We can use log-transform to make the distribution of ‘Avg_Account_Balance’ more normal, as it approximately follows a log-normal distribution.

# Using log transformation to normalize the 'Avg_Account_Balance'
log_aab = np.log(df['Avg_Account_Balance'])
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(12, 3), dpi=100)
fig.suptitle('Distribution of log(Avg_Account_Balance)')
sns.distplot(log_aab, label='log(Avg_Account_Balance)',
            kde=True, hist=True, bins=20, color='#19A789', ax=axes[0], kde_kws={'color': 'Tomato'})
sns.boxplot(x=log_aab, ax=axes[1], color='#69A789')
axes[0].set_xlabel("")
axes[1].set_xlabel("")

# Remove Spines
for spine in ['right', 'top']:
    axes[0].spines[spine].set_visible(False)
    axes[1].spines[spine].set_visible(False)
plt.show()

# Plot Violen plot for all numerical columns
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(12, 2), dpi=300)
for i, col in enumerate(num_cols):
    sns.violinplot(x=col, data=df, ax=axes[i])
    axes[i].set_xlabel("")
    axes[i].set_ylabel("")
    axes[i].set_title(" ".join(col.split('_')), weight='bold')

    # Remove Spines
    for spine in ['left', 'right', 'top']:
        axes[i].spines[spine].set_visible(False)
        axes[i].axes.get_yaxis().set_visible(False)

# Plot correlation matrix for all numerical columns
corr = df[num_cols].corr()
fig, ax = plt.subplots(figsize=(6, 6))
sns.heatmap(corr, annot=True, fmt='.2f', ax=ax)
plt.show()

There is a positive correlation between Vintage and Age which can be further explored using boxplot and scatterplot. No other numerical variables have a strong correlation.

# collapse
# plot() for all categorical columns
cat_feats = list(cat_cols)
cat_feats.remove('Region_Code')

plt.rcParams['figure.dpi'] = 600
fig = plt.figure(figsize=(15, 9), facecolor='#f6f5f5')
gs = fig.add_gridspec(2, 3)
gs.update(wspace=0.25, hspace=.25)

for i, col in enumerate(cat_feats):
    ax = fig.add_subplot(gs[i // 3, i % 3])

    # Get the sorted value_counts and plot the bar plot
    col_data = df[col].value_counts(normalize=True)\
        .rename('Percentage').mul(100)\
        .reset_index().sort_values(ascending=False, by='Percentage')

    # Plot customizations
    for spine in ['right', 'top', 'left']:
        ax.spines[spine].set_visible(False)
    ax.set_facecolor(bg_color)
    ax.axes.yaxis.set_visible(False)

    ax_sns = sns.barplot(x=col_data['index'], y=col_data['Percentage'], ax=ax, saturation=1)
    # Customize plot
    if i % 3 == 0:
        ax_sns.set_ylabel("Count(%)", weight='bold')
    else:
        ax_sns.set_ylabel("")
    ax_sns.set_xlabel(" ".join(col.split('_')), weight='bold')

    # Add patches for data percentages
    for p in ax.patches:
        value = f'{p.get_height(): .0f}%'
        x = p.get_x() + p.get_width() / 2 - 0.1
        y = p.get_y() + p.get_height() + 2
        ax.text(x, y, value, ha='left', va='center', fontsize=7, 
                bbox=dict(facecolor='none', edgecolor='black', boxstyle='round', linewidth=0.5))

Dataset is highly imbalanced, as there are only 24% entries which are leads and 76% entries which are not. Due to the small size of the dataset, Upsampling Strategy to hadle imbalanced data could work well.

There is also a large imbalance in the dataset for Occupation and Channel_Code categories as there are very few entries for Enterpreneur and X4 respectively.

plt.figure(figsize=(16, 5), dpi = 600)
col_data = df['Region_Code'].value_counts(normalize=True)\
    .rename('Percentage').mul(100).reset_index().sort_values(ascending=False, by='Percentage')
sns.barplot(x=col_data['index'], y=col_data['Percentage'], saturation=1)
plt.xticks(rotation=90)
plt.xlabel('Region Code', weight='bold')
plt.ylabel('Count(%)', weight='bold')
plt.title('Ranked Frequency Plot - Region Code', weight='bold', fontsize=15)

for spine in ['top', 'right']:
    plt.gca().spines[spine].set_visible(False)
plt.show()

Bivariate Analysis

# Plot count of Leads by Region Code
fig = plt.figure(facecolor=bg_color, dpi = 600)
g = sns.FacetGrid(df, col='Region_Code', col_wrap=6)
g.map(sns.countplot, 'Is_Lead', saturation=1)
plt.show()
<Figure size 3600x2400 with 0 Axes>

# Plot KDE Plots for all numerical columns with hue=cat_col
fig, axes = plt.subplots(nrows=len(cat_feats), ncols=3, figsize=(12, len(cat_cols)*2), dpi=600)
for i, cat_col in enumerate(cat_feats):
    for j, num_col in enumerate(num_cols):
        ax_sns = sns.kdeplot(x = num_col, data=df, hue=cat_col, label=col, ax=axes[i, j])
        
        # Customize the plot
        ax_sns.tick_params(axis='y', labelsize=0)
        axes[i, j].set_xlabel(" ".join(num_col.split('_')), weight='bold')
        if j != 0:
            axes[i, j].set_ylabel("")
            axes[i, j].legend_.remove()
        else:
            axes[i, j].set_ylabel("Density", weight='bold')

        for spine in ['top', 'right']:
            axes[i, j].spines[spine].set_visible(False)
plt.tight_layout()

Almost all the categories in each categorical variable follows the same distribution.

# Plot Box-Plot for all numerical columns with hue=cat_col
fig, axes = plt.subplots(nrows=len(cat_feats), ncols=3, figsize=(12, 3*len(cat_feats)), dpi=600)
for i, cat_col in enumerate(cat_feats):
    for j, num_col in enumerate(num_cols):
        ax_sns = sns.boxplot(y=num_col, x=cat_col, data=df, ax=axes[i,j])

        # Customize the plot
        axes[i, j].set_ylabel(num_col, weight='bold')
        axes[i, j].set_xlabel(cat_col, weight='bold')

        for spine in ['top', 'right']:
            axes[i, j].spines[spine].set_visible(False)
plt.tight_layout()

There are huge number of outliers for each categorical variable in Avg_Account_Balance.

# Plot Scatter-Plot between Age and Vintage
grid = sns.FacetGrid(df, row='Occupation', col='Is_Active', hue='Is_Lead', aspect=1.5, size=4, palette='Set2')
grid.map(plt.scatter, 'Age', 'Avg_Account_Balance', alpha=0.5)
grid.add_legend()
plt.show()

# Plot Scatter-Plot between Age and Vintage
grid = sns.FacetGrid(df, row='Occupation', col='Credit_Product', hue='Is_Lead', aspect=1.5, size=4, palette='Set2')
grid.map(plt.scatter, 'Age', 'Avg_Account_Balance', alpha=0.5)
grid.add_legend()
plt.show()

# Plot Scatter-Plot between Age and Vintage
grid = sns.FacetGrid(df, row='Occupation', col='Gender', hue='Is_Lead', aspect=1.5, size=4, palette='Set2')
grid.map(plt.scatter, 'Age', 'Avg_Account_Balance', alpha=0.5)
grid.add_legend()
plt.show()

# Plot Scatter-Plot between Age and Vintage
grid = sns.FacetGrid(df, row='Occupation', col='Channel_Code', hue='Is_Lead', aspect=1.5, size=4, palette='Set2')
grid.map(plt.scatter, 'Age', 'Avg_Account_Balance', alpha=0.5)
grid.add_legend()
plt.show()

From above all Scatter-Plots, we can see that almost every Self_Employed individual whose Age is above 40 is a lead. We can use this information to create a new feature.

Relationship between Missing Values and Target

# collapse
# Fraction of customers with missing values who are leads
df['Credit_Product'][df['Credit_Product'].isnull()] = 'Null'
na_df = df['Credit_Product'].value_counts()\
    .rename('Fraction').reset_index().sort_values(ascending=False, by='Fraction')

fig = plt.figure(dpi = 600, figsize=(13,3))
ax_sns = sns.barplot(y=na_df['index'], x=na_df['Fraction'], orient='h',
saturation=1, palette='flare')
# Remove Spines
for spine in ['top', 'right', 'bottom']:
    plt.gca().spines[spine].set_visible(False)

# Disable ticks
plt.tick_params(axis='x', which='both', bottom=False, top=False, labelbottom=False)

# Customize the plot
plt.xlabel('Fraction of Customers', weight='bold', fontsize=13)
plt.ylabel('Credit Product', weight='bold', fontsize=13)

# Add patches
for p in ax_sns.patches:
    value = f'{p.get_width(): .0f} | {p.get_width() / df.shape[0] * 100: 0.2f}%'
    x = p.get_x() + p.get_width() + 5e3
    y = p.get_y() + p.get_height() / 2
    ax_sns.text(x, y, value, ha='left', va='center', fontsize=9,
            bbox=dict(facecolor='none', edgecolor='black', boxstyle='round', linewidth=0.5))
plt.show()

na_df = df.groupby('Credit_Product')['Is_Lead'].value_counts(normalize=True)
# convert na_df to dataframe
na_df = na_df.reset_index()
na_df.columns = ['Credit_Product', 'Is_Lead', 'Fraction']
# swap rows at index 2 and 3
na_df.loc[2], na_df.loc[3] = na_df.loc[3], na_df.loc[2]
na_df
Credit_Product Is_Lead Fraction
0 No 0 0.926
1 No 1 0.074
2 Null 0 0.148
3 Null 1 0.852
4 Yes 0 0.685
5 Yes 1 0.315 
# collapse
fig = plt.figure(dpi=600, figsize=(13, 3))
ax_sns = sns.countplot(y='Credit_Product', data=df, hue='Is_Lead', palette='flare', saturation=1, orient='h')
ax_sns.set_xlabel('Customers', weight='bold', fontsize=13)
ax_sns.set_ylabel('Credit Product', weight='bold', fontsize=13)

# Remove ticks
plt.gca().tick_params(axis='x', which='both', bottom=False, labelbottom=False)

# Remove Spines
for spine in ['top', 'right', 'bottom']:
    ax_sns.spines[spine].set_visible(False)

# Add patches
na_f = na_df['Fraction'].values
idx = 0
for p in ax_sns.patches:
    if idx <= na_df.shape[0] // 2:
        value = f"{na_f[idx] * 100: 0.2f}%"
        idx += 2
    elif idx == na_df.shape[0] // 2 + 1:
        value = f"{na_f[idx] * 100: 0.2f}%"
        idx = 1
    else:
        value = f"{na_f[idx] * 100: 0.2f}%"
        idx += 2
    x = p.get_x() + p.get_width() + 3e3
    y = p.get_y() + p.get_height() / 2
    ax_sns.text(x, y, value, ha='left', va='center', fontsize=8,
                bbox=dict(facecolor='none', edgecolor='black', boxstyle='round', linewidth=0.5))
plt.show()

Calculate Feature Importance using Mutual Information

# collapse
from sklearn.feature_selection import mutual_info_classif
from sklearn.preprocessing import OrdinalEncoder, RobustScaler

cat_cols = cat_cols.tolist()
cat_cols.remove('Is_Lead')

# Scale the numerical columns
scaler = RobustScaler()
df[num_cols] = scaler.fit_transform(df[num_cols])

# Encode categorical columns
encoder = OrdinalEncoder()
df[cat_cols] = encoder.fit_transform(df[cat_cols])

# Get the target variable
target = df.pop('Is_Lead')

# Get the indices of the categorical features
cols = df.columns.tolist()
cat_idx = [i for i in range(len(cols)) if cols[i] in cat_cols]

# Calculate Mutual Information
mi = mutual_info_classif(df, target, discrete_features=cat_idx)
mi_df = pd.DataFrame(mi, index=df.columns, columns=['MI'])
mi_df = mi_df.sort_values(by='MI', ascending=False)
mi_df.head()
MI
Credit_Product 0.161
Age 0.051
Channel_Code 0.048
Vintage 0.047
Occupation 0.011 
# Plot MI from mi_df
fig = plt.figure(dpi=600, figsize=(6, 2))
sns.barplot(x='MI', y=mi_df.index, data=mi_df, palette='crest', saturation=1)

# Remove ticks
# plt.gca().tick_params(axis='x', which='both', bottom=False, labelbottom=False)

# Remove Spines
for spine in ['top', 'right']:
    plt.gca().spines[spine].set_visible(False)

# Remove "_" from yticklabels
plt.gca().set_yticklabels(mi_df.index.str.replace('_', ' '), fontsize=5, weight='bold')
plt.gca().set_xticklabels(plt.gca().get_xticks(), fontsize=5)

plt.xlabel('Mutual Information', fontsize=7)
plt.show()

]]> EDA Visualization https://ashudva.github.io/blog/posts/2021-11-30-EDA/ Sun, 21 Sep 2025 21:36:10 GMT Principal Component Analysis https://ashudva.github.io/blog/posts/2021-06-10-PCA/

Structure

I have used R for some of the tasks and Python for the implementation of the PCA from scratch. If you are not familiar with R, you can still understand all of the material.

You can see the code by clicking on show code befor each code cell.

  1. Introduction to PCA
  2. How PCA reduces dimensionlity
  3. Scree Plot
  4. Dimensionality reduction
  5. PCA Code in python
  6. Application - 1
  7. Application - 2
  8. Application - 3
  9. Summary
  10. References

Introduction to PCA

PCA is a dimensionality reduction technique used in Data Analysis and Machine Learning.

Why we need PCA?
In Data Analysis, we can easily visualize a dataset with upto three features but for four or more features we can’t envisage the data thus we use PCA to make it 3-D, 2-D or even 1-D with minimum loss of inherent distribution of the data.

In Machine Learning, a dataset with large number of features is computationally expensive and thus PCA can be used to reduce the number of features that do not account for much variance in the dataset. > PCA requires working knowledge of eigenvalues and eigenvectors.

Code in the below cell generates a synthetic dataset with 3 Features and 50 Samples generated from multivariate random distribution and without any covariance between the features.

# collapse
options(warn=-1)
library(repr)
library(MASS)
library(ggplot2)
library(ggthemes)
library(ggrepel)
library('Cairo')
CairoWin()

nsamples <- 50
nfeatures <- 3
data <- matrix(nrow = nsamples, ncol = nfeatures)
colnames(data) <- paste("Feature", 1:nfeatures, sep = "")
rownames(data) <- paste("Sample", 1:nsamples, sep = "")

# Create
data[1:25,] = mvrnorm(n = 25, mu = c(3,3,0), Sigma = diag(3))
data[26:50,] = mvrnorm(n = 25, mu = c(-3,-3,0), Sigma = diag(3))
head(data)
Feature1 Feature2 Feature3
Sample1 2.895416 1.458052 -0.2471825
Sample2 2.802611 3.021491 0.1058062
Sample3 1.406073 2.437702 -0.7030344
Sample4 1.315465 4.477110 1.0596977
Sample5 2.754785 1.844019 0.2963604
Sample6 4.415007 4.525606 0.5257132

Scatter Plot

Feature3 is represented by the size of the point i.e larger the point => larger the Feature3

As it can be seen in the plot, there are two clusters in the data. So we would like to reduce dimensionality in such a way that it preserves these clusters.

# collapse
p <- ggplot() +
geom_point(data=data.frame(data), 
           aes(x=Feature1, y=Feature2, size=Feature3, color="red", alpha=0.24)
) +
guides(color=F, alpha=F)  +
ggtitle("Scatter plot")
options(repr.plot.width = 5, repr.plot.height = 4)
p

How PCA reduces dimensionality

Our goal is to remove the 3rd dimension(Feature3) from the generated data with minimal loss in the structure of the data.

  1. A dataset with N-Samples and M-Features can be considered as a [N X M] matrix and we know that every matrix has a set of pairs of Eigenvalues and their corresponding Eigenvectors associated with it.
  2. Eigenvecotrs are usual vectors and thus we know they refer to some direction.
  3. Intuitively the Eigenvalues tells the amount of variance in data, in the direction of its corresponding Eigenvector.
  4. Therefore the the Eigenvector with the highest Eigenvalue is the principal component of the data i.e. direction of maximum variance in the data.
  5. The maximum variance Eigenvector will be the one for which the Sum of Squared Distances (SSD) from points to the vector is minimum.
  6. To find the Principle Component, we try to find the best fitting line for the data, similar to linear regression.
  7. We might start with randomly drawing a line and then iteratively try to reduce the SSD to find best fitting line (In practice we use the Closed-Form Solution not iterative).
    (contin.)

In the figure below, the blue lines represent the distance b/w line and points, and our goal in PCA is to minimize their squared sum.

# collapse
slope <- 0

# Get the endpoints of points
perp.segment.coord <- function(x0, y0, a=0,b=1){
 #finds endpoint for a perpendicular segment from the point (x0,y0) to the line
 # defined by lm.mod as y=a+b*x
  x1 <- (x0+b*y0-a*b)/(1+b^2)
  y1 <- a + b*x1
  list(x0=x0, y0=y0, x1=x1, y1=y1)
}

ss<-perp.segment.coord(data[,1], data[,2], 0, slope)


p <- ggplot() +
geom_point(data=data.frame(data), 
           aes(x=Feature1, y=Feature2, size=Feature3, color="red", alpha=0.24)
) +
geom_abline(slope = slope, intercept = 0) +
guides(color=F, alpha=F)  +
ggtitle("Random line") +
geom_segment(data=as.data.frame(ss), aes(x = x0, y = y0, xend = x1, yend = y1), colour = "steelblue")
options(repr.plot.width = 5, repr.plot.height = 4)
p

Clearly we can do better than this, Line-1 is approximately the best fitting line. Line-1(the best fit) is called the PC1 (Principle Component 1) which accounts for the most variance.

# collapse
slope <- 1
ss<-perp.segment.coord(data[,1], data[,2], 0, slope)

p <- ggplot() +
geom_point(data=data.frame(data), 
           aes(x=Feature1, y=Feature2, size=Feature3, color="red", alpha=0.24)
) +
geom_abline(slope = slope, intercept = 0) +
annotate("text", x= 2, y = 0 , label="Line-1 / PC1") +
guides(color=F, alpha=F)  +
ggtitle("Approximate best fit") +
geom_segment(data=as.data.frame(ss), aes(x = x0, y = y0, xend = x1, yend = y1), colour = "steelblue")
options(repr.plot.width = 5, repr.plot.height = 4)
p

(contin.)
8. Next step is to rotate the axes around origin to reorient the data and project data onto the Principle Component to get PC1.

# collapse
theta <- 45
# x_new = xcosθ + ysinθ
# y_new = ycosθ – xsinθ
x = data[,1]
y = data[,2]
x_new = x*cos(theta) + y*sin(theta)
y_new = y*cos(theta) - x*sin(theta)
new_data = data
new_data[,1] = x_new
new_data[1:25,2] = y_new[1:25] + 1
new_data[26:50, 2] = y_new[26:50] - 1



slope <- 0
ss<-perp.segment.coord(new_data[,1], new_data[,2], 0, slope)

require(gridExtra)
p1 <- ggplot() +
geom_point(data=data.frame(new_data), 
           aes(x=Feature1, y=Feature2, size=Feature3, color="red", alpha=0.24)
) +
geom_abline(slope = slope, intercept = 0) +
annotate("text", x= 0, y = -.5 , label="PC1", angle=90) +
guides(color=F, alpha=F, size=F)  +
ggtitle("Rotatation") +
geom_segment(data=as.data.frame(ss), aes(x = x0, y = y0, xend = x1, yend = y1), colour = "steelblue")+
xlab("PC1")

p2 <- ggplot() +
geom_point(data=data.frame(new_data), 
           aes(x=Feature1, y=rep(0, 50), size=Feature3, color="red", alpha=0.24)
) +
geom_abline(slope = slope, intercept = 0) +
guides(color=F, alpha=F, size=F)  +
ggtitle("Projection onto PC1") +
xlab("PC1")+
theme(axis.title.y=element_blank(),
    axis.text.y=element_blank(),
    axis.ticks.y=element_blank())
grid.arrange(p1, p2, ncol=2)
options(repr.plot.width = 8, repr.plot.height = 3)
Your code contains a unicode char which cannot be displayed in your
current locale and R will silently convert it to an escaped form when the
R kernel executes this code. This can lead to subtle errors if you use
such chars to do comparisons. For more information, please see
https://github.com/IRkernel/repr/wiki/Problems-with-unicode-on-windows

  1. Once we have found the PC1, PC2 will be a line perpendicular to PC1 and PC3 will be perpendicular to PC2 and so on, because for a symmetric matrix Eigenvectors are always orthogonal to each other and Eignevalues are always real.
  2. We get other PC’s by simply projecting the data onto them.
  3. Maximum number of PC’s = minimum(features, samples)
  4. The projected data will act as new Feature and we can discard the Features/PC’s which accounts for less variance.

Why PC’s are always orthogonal (optional)

In practice, if the data has a shape [n x m] i.e. n-samples and m-features, PC’s are calculated using either Scatter Matrix or Covariance Matrix for the data. Both of these are symmetric matrices with shape [m x m] (as covariance is calculate between features and the diagonal represents the variance) and therefore their Eigenvectors must be orthogonal, this property is proved using Spectral Theorem.

In a real world situation, we measure values of features on the original scales/axes i.e. Real Number line but this may not capture maximum essence of the data and therfore through PCA we essentially change our axes such that it now explains the data better than the original axes.

Computing PCA and Scree Plot

I’ll use prcomp() utility of R to calculate Principal Components of the data and then discard the component with less variance the reduce dimensionality.

# collapse
pca <- prcomp(data)
# prcomp returns x, stdev, rotation
# where x is the data transposed on PC's

Scree Plot: it shows the variance captured by each principal components.

# collapse
pca.var = pca$sdev
# Percentage of variation each component accounts for
pca.var.per =  round(pca.var / sum(pca.var) * 100, 1)

ggplot(mapping = aes(x = paste("PC", 1:3, sep = ""), y = pca.var.per)) +
geom_bar(stat = "identity", fill='steelblue') +
xlab("Principal Components") +
ylab("Percentage Variance") +
geom_text(aes(label=pca.var.per), vjust=1.6, color="white", size=3.5)
options(repr.plot.width = 5, repr.plot.height = 4)

Dimensionality Reduction

Finally we plot the data with reduced dimensions and it is evident from the figure that even if we discard the PC3, PC1 and PC2 still captures most of the distribution of the data.

Data is still divided into two clusters.

pca.data = data.frame(Sample=rownames(pca$x),
                      X = pca$x[,1],
                      Y = pca$x[,2])
ggplot(data=pca.data, aes(x = X, y = Y, label = Sample)) +
geom_point(color='tomato') +
geom_text_repel() + 
xlab(paste("PC1 - ", pca.var.per[1], "%", sep = "")) +
ylab(paste("PC2 - ", pca.var.per[2], "%", sep = "")) +
ggtitle("Principal component plot")

PCA form scratch using Python

Overview of functions used:

  1. center_data() - to scale the data such that each feature in the dataset has mean 0
  2. principal_component() - returns the PC’s (eigenvectors) of the data
  3. project_onto_PC() - perform dimensionality reduction by projecting the data onto PC’s
  4. plot_PC - used() to visualize the resulting image after dimensionality reduction
  5. reconstruct_PC() - reconstruct an image from PCA

Feel free to read the docstrings of functions to get an overview of their functionality.

import numpy as np
import matplotlib.pyplot as plt


def project_onto_PC(X, pcs, n_components):
    """
    Given principal component vectors pcs = principal_components(X)
    this function returns a new data array in which each sample in X
    has been projected onto the first n_components principcal components.

    Args:
        X - n x d Numpy array
        pcs - d x d Numpy array with each column as an eigenvector sorted in 
        decreasing order of their corresponding eigenvalues.
        n_components - (scalar) top principal components
    Returns:
        projected_data - n x n_components Numpy array with n samples and n_components features
    """
    # Step1: Center the data such that for each feature of a sample, mean = 0.
    # This step is often called scaling
    X_bar = center_data(X)

    # Step2: Projection onto the n_components principal components.
    n_pcs = pcs[:, :n_components]
    projected_data = X_bar @ n_pcs

    return projected_data



def center_data(X):
    """
    Returns a centered version of the data, where each feature now has mean = 0
    Example:         X  = [[2, 2, 1],
                           [0, 1, 2],
                           [1, 0, 3]]
                   
                   mean =  [1, 1, 2]
             centered_X = [[2 - 1, 2 - 1, 1 - 2],
                           [0 - 1, 1 - 1, 2 - 2],
                           [1 - 1, 0 - 1, 3 - 2]]

    Args:
        X - n x d NumPy array of n data points, each with d features

    Returns:
        n x d NumPy array X' where for each i = 1, ..., n and j = 1, ..., d:
        X'[i][j] = X[i][j] - means[j]
    """
    feature_means = X.mean(axis=0)
    return(X - feature_means)


def principal_components(X):
    """
    Returns the principal component vectors of the data, sorted in decreasing order
    of eigenvalue magnitude. This function first caluclates the covariance matrix
    and then finds its eigenvectors.
    
    Why center the data?
    for a matrix with d-features, covariance matrix of features is given by:
                    Cov(X) = (X - mu_X).T @ (X - mu_X)
    by centering the data it becomes,
                    Cov(X) = X.T @ X
                    
    Note: If we do not center the data then the resulting matrix is called Scatter Matrix

    Args:
        X - n x d NumPy array of n data points, each with d features

    Returns:
        d x d NumPy array whose ****columns are the principal component directions**** sorted
        in descending order by the amount of variation each direction (these are
        equivalent to the d eigenvectors of the covariance matrix sorted in descending
        order of eigenvalues, so the first column corresponds to the eigenvector with
        the largest eigenvalue
    """
    centered_data = center_data(X)  # first center data
    scatter_matrix = centered_data.T @ centered_data
    eigen_values, eigen_vectors = np.linalg.eig(scatter_matrix)
    # Re-order eigenvectors by eigenvalue magnitude:
    idx = eigen_values.argsort()[::-1]
    eigen_values = eigen_values[idx]
    eigen_vectors = eigen_vectors[:, idx]
    return eigen_vectors


def plot_PC(X, pcs, labels):
    """
    Given the principal component vectors as the columns of matrix pcs,
    this function projects each sample in X onto the first two principal components
    and produces a scatterplot where points are marked with the digit depicted in
    the corresponding image.
    labels = a numpy array containing the digits corresponding to each image in X.
    """
    pc_data = project_onto_PC(X, pcs, n_components=2)
    text_labels = [str(z) for z in labels.tolist()]
    fig, ax = plt.subplots()
    ax.scatter(pc_data[:, 0], pc_data[:, 1], alpha=0, marker=".")
    for i, txt in enumerate(text_labels):
        ax.annotate(txt, (pc_data[i, 0], pc_data[i, 1]))
    ax.set_xlabel('PC 1')
    ax.set_ylabel('PC 2')
    plt.show()

def reconstruct_PC(x_pca, pcs, n_components, X):
    """
    Given the principal component vectors as the columns of matrix pcs,
    this function reconstructs a single image from its principal component
    representation, x_pca.
    X = the original data to which PCA was applied to get pcs.
    """
    # X - (X - mu_X) = mu_X = feature means
    # Alternatively, 
    # feature_means = X.mean(axis=0)
    feature_means = X - center_data(X)
    feature_means = feature_means[0, :]
    x_reconstructed = np.dot(x_pca, pcs[:, range(n_components)].T) + feature_means
    return x_reconstructed

Application-1: Data Analysis

For the demonstration of capability of PCA, I’ll use MNIST Dataset with 60000 images of size - 28x28. Each image consists of 28*28 = 784 features, and using PCA I’ll reduce the number of features to only 2 so that we can visualize the dataset.

Even when we reduce the the data to such low dimensionality, the data will still contain some level of structure. That’s the power of PCA.

Later on we’ll see how do we know, how many Components are enough to properly describe the dataset, and how many are redundant.

from keras.datasets import mnist
(data, labels), (_, _) = mnist.load_data()
# Scale the dataset
data = data.astype('float') / 255
print(f"Loaded mnist dataset with shape: {data.shape}")

n_components=2
flatten_data = data.reshape((60000, 28*28))
pcs = principal_components(flatten_data)
pca_data = project_onto_PC(flatten_data, pcs, n_components)
print(f"Original data shape: {data[0].shape}")
print(f"Projected data shape: {pca_data[0].shape}")

plt.scatter(pca_data[:1000, 0], pca_data[:1000, 1],
            c=labels[:1000], edgecolor='none', alpha=0.8,
            cmap=plt.cm.get_cmap('Spectral', 10))
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.colorbar();
Loaded mnist dataset with shape: (60000, 28, 28)
Original data shape: (28, 28)
Projected data shape: (2,)

Essentially, we have found the optimal stretch and rotation in 784-dimensional space that allows us to see the layout of the digits in two dimensions, and have done this in an unsupervised manner—that is, without reference to the labels.

Decide the Number of Components

For this task I’ll utilize the sklearn.decomposition.PCA to get explained_variance_ratio_

from sklearn.decomposition import PCA
pca = PCA().fit(flatten_data)
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlabel('Number Of Components')
plt.ylabel('Cumulative Explained Variance');

As you can see, with using just 100 out of 784 components we can capture more than 90% of the variance in the data.

Application-2: Machine Learning

With a simple visualization I’ll explain how reducing the dimensionality with PCA is useful.

# collapse
n_components=18
pcs = principal_components(flatten_data)
pca_data = project_onto_PC(flatten_data, pcs, n_components)

reduced_image = reconstruct_PC(pca_data[10], pcs, n_components, flatten_data)
reduced_image = reduced_image.reshape((28, 28))

fig, ax = plt.subplots(1, 2, figsize=(10, 5))
ax[0].imshow(data[10], cmap='Greys')
ax[1].imshow(reduced_image, cmap='Greys')
ax[0].set_title("Original Image")
ax[1].set_title("Reduced Image")
fig.suptitle("Application of PCA on Digit Image");

The original image could be reconstructed if we use all of the PC’s but here I used only 18 PC’s and as you can see the reconstructed image still resembles the digit - three.

Reducing the dimensionality to sheer 18-Dimensions will still have fair amount of structure to be useful in a machine learning algorithm. This will save a lot of computational cost especially in case where dimensionality of data is very high. PCA often improves accuracy by eliminating the irrelevant features from the data.

Application-3: Denoising

def plot_digits(data):
    fig, axes = plt.subplots(4, 10, figsize=(10, 4),
                             subplot_kw={'xticks':[], 'yticks':[]},
                             gridspec_kw=dict(hspace=0.1, wspace=0.1))
    for i, ax in enumerate(axes.flat):
        ax.imshow(data[i].reshape(28, 28),
                  cmap='Greys')
print("Origina Digits without any noise")
plot_digits(flatten_data)
Origina Digits without any noise

#Introduce some gaussian noise into the images.
np.random.seed(42)
noisy = np.random.normal(flatten_data, 0.1)
plot_digits(noisy)
print("Noisy Images")
Noisy Images

pca = PCA(0.50).fit(noisy)
print(f"Components accountable for 50% variance: {pca.n_components_}")
Components accountable for 50% variance: 15

Here 50% of the variance amounts to 15 principal components. Now we compute these components, and then use the inverse of the transform to reconstruct the filtered digits.

components = pca.transform(noisy)
filtered = pca.inverse_transform(components)
plot_digits(filtered)

Summary

PCA has several application apart from those mentioned above. It’s even used in facial recognition. PCA plays useful role especially when we want to reduce dimensionality with low computational cost. PCA is very light algorithm and takes only few seconds to process very large datasets. Because of the versatility and interpretability of PCA, it has been shown to be effective in a wide variety of contexts and disciplines. Given any high-dimensional dataset, try to start with PCA in order to visualize the relationship between points (as we did with the digits) and to understand the intrinsic dimensionality (by plotting the explained variance ratio). Certainly PCA is not useful for every high-dimensional dataset, but it offers a straightforward and efficient path to gaining insight into high-dimensional data.

Scree-Plot and Cumulative Explained Variance prove to be very useful in every scenario.

References

  1. PDF Resources from MIT OCW
  2. Video Lecture by Philippe Rigollet MIT
  3. Easy to understand video by StatQuest
  4. Intuitive explanation of Eigenvalues and Eigenvectors by 3Blue1Brown
  5. sklearn.decomposition.PCA
  6. prcomp()
]]> tutorial https://ashudva.github.io/blog/posts/2021-06-10-PCA/ Sun, 21 Sep 2025 21:36:10 GMT Perceptron, Average Perceptron and Pegasos https://ashudva.github.io/blog/posts/2020-02-13-Linear-Classifiers/

Abstract

The goal of this project is to write three Supervised Learning Algorithms - Perceptron, Average Perceptron, and Pegasos - in pure python code. I’ll use numpy for handling and creating numerical arrays. The parameter estimation is done through Stochastic Gradient Descent algorithm.

import numpy as np

Test Cases

I’ve created some test cases for each algorithm that I’ll be writing, to test the correctness of the algorithm. e.g.  check_get_order() check_hinge_loss_single() check_hinge_loss_full()

Algorithms

Hinge Loss

HINGE_LOSS (, , ):
  loss 0
  for i 1….n:
     $ y^{(i)} (. x^{(i)} + _0 ) $
    
  return 

#collapse
def hinge_loss_single(feature_vector, label, theta, theta_0):
    y = theta @ feature_vector + theta_0
    return np.maximum(0, 1 - y * label)
#collapse
def hinge_loss_full(feature_matrix, labels, theta, theta_0):
    loss = 0
    for i in range(len(labels)):
        loss += hinge_loss_single(feature_matrix[i], labels[i], theta, theta_0)
    return loss / len(labels)

1. Perceptron

For perceptron algorithm, I’ll use 0-1 loss for simplicity.
Algo takes feature matrix, T - no. of times to run the algo on the given data, labels as input and returns estimated

Proof for perceptron update:
Loss function(for a particular sample ):
Aggrement: $ z = y^i*(x^i.+ _0)$, Learning rate:

$ _ =

$


therefore the gradient descent update will be:

PERCEPTRON (, T):
 
 
  for t 1….T:
   Randomly shuffle indices
   for i in indices:
    if $ y^{(i)} (. x^{(i)} + _0 ) $:
    
     

#collapse
def perceptron_single_step_update(
        feature_vector,
        label,
        current_theta,
        current_theta_0):
    if label * (np.dot(current_theta, feature_vector) + current_theta_0) <= 1e-7:
        current_theta += label * feature_vector
        current_theta_0 += label
    return current_theta, current_theta_0
#collapse
def perceptron(feature_matrix, labels, T):
    import random
    random.seed(1)
    nsamples, nfeatures = feature_matrix.shape
    theta = np.zeros(nfeatures)
    theta_0 = 0.0
    for t in range(T):
        indices = list(range(nsamples))
        random.shuffle(indices)
        for i in indices:
            theta, theta_0 = perceptron_single_step_update(feature_matrix[i], labels[i], theta, theta_0)
    return theta, theta_0

2. Average Perceptron

The Average Perceptron algorithm gives better results than the simple Perceptron algorithm in most of the cases.
Average Perceptron is analogous to the simple Perceptron except that we also need to keep track of the sum of ’s got after each run of the algorithm and then finally return the Average of those.

#collapse
def average_perceptron(feature_matrix, labels, T):
    import random
    random.seed(1)
    nsamples, nfeatures = feature_matrix.shape
    theta = np.zeros(nfeatures)
    theta_sum =  np.zeros(nfeatures)
    theta_0 = 0.0
    theta_0_sum = 0.0
    for t in range(T):
        indices = list(range(nsamples))
        random.shuffle(indices)
        for i in indices:
            theta, theta_0 = perceptron_single_step_update(feature_matrix[i], labels[i], theta, theta_0)
            theta_sum += theta
            theta_0_sum += theta_0
    return theta_sum / (nsamples * T), theta_0_sum / (nsamples * T)

3. Pegasos

#collapse
def pegasos_single_step_update(feature_vector, label, L, eta, current_theta,
                               current_theta_0):
    c = 1 - eta * L
    d = eta * label
    if label * (current_theta @ feature_vector + current_theta_0) <= 1:
        return c * current_theta + (d * feature_vector), current_theta_0 + d
    return c * current_theta, current_theta_0
#collapse
def pegasos(feature_matrix, labels, T, L):
    import random
    random.seed(1)
    nsamples, nfeatures = feature_matrix.shape
    theta = np.zeros(nfeatures)
    theta_0 = 0
    count = 0
    for t in range(T):
        indices = np.array(range(nsamples))
        random.shuffle(indices)
        for i in indices:
            count += 1
            eta = 1 / np.sqrt(count)
            theta, theta_0 = pegasos_single_step_update(
                feature_matrix[i], labels[i], L, eta, theta, theta_0)
    return theta, theta_0

Testing

Apply algos on synthetic data

Generate Synthetic Data

#collapse
def synth_data_2d(n):
    import numpy as np
    import scipy.stats as ss
    np.random.seed(1)
    positive_data = ss.norm.rvs(loc=(4,5), scale=(1,1), size=(n,2))
    negative_data = ss.norm.rvs(loc=(-2,-2), scale=(1,1), size=(n,2))
    data = np.concatenate((positive_data, negative_data), axis=0)
    class1 = np.repeat(1, positive_data.shape[0])
    class2 = np.repeat(-1, negative_data.shape[0])
    labels = np.concatenate([class1, class2])
    return data, labels
X, y = synth_data_2d(1000)
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib notebook
mpl.style.use("ggplot")

colors = ['darkcyan' if label == 1 else 'salmon' for label in y]

plt.scatter(X[:, 0], X[:, 1], c=colors, s=10, alpha=0.75)
plt.xlabel("$X_1$")
plt.ylabel("$X_2$")
plt.title("Scatter Plot - Synthetic data")
plt.show()

Decesion Boundaries

Get parameters and for each algorithm and plot the corresponding decesion boundary.

#collapse
ap_theta, ap_theta_0 = average_perceptron(X, y, 10)
p_theta, p_theta_0 = perceptron(X, y, 10)
peg_theta, peg_theta_0 = pegasos(X, y, 10, 0.01)
plt.scatter(X[:, 0], X[:, 1], s=10, c=colors)
xmin, xmax = plt.axis()[:2]

# plot the decision boundary
xs = np.linspace(xmin, xmax)
ap_ys = -(ap_theta[0]*xs + ap_theta_0) / (ap_theta[1] + 1e-16)
p_ys = -(p_theta[0]*xs + p_theta_0) / (p_theta[1] + 1e-16)
peg_ys = -(peg_theta[0]*xs + peg_theta_0) / (peg_theta[1] + 1e-16)

plt.plot(xs, ap_ys, '--', label="Avg Perceptron")
plt.plot(xs, p_ys, '--', label="Perceptron")
plt.plot(xs, peg_ys, '-', label="Pegasos")
plt.legend(loc="upper left")
plt.xlabel("$X_1$")
plt.ylabel("$X_2$")
plt.title("Decesion Boundary Comparison")
plt.show()

Because the Pegasus Algorithm is a Linear Support Vector Machine Algorithm, so let’s plot its Decision and Margin Boundaries.

%matplotlib notebook
import matplotlib as mpl
mpl.style.use("ggplot")
colors = ['darkcyan' if label == 1 else 'salmon' for label in y]
plt.scatter(X[:, 0], X[:, 1], s=30, c=colors)
xmin, xmax = plt.axis()[:2]



# plot the decision boundary
xs = np.linspace(xmin, xmax)
ys = -(peg_theta[0]*xs + peg_theta_0) / (peg_theta[1] + 1e-16)
upper_margin = -(peg_theta[0]*xs + peg_theta_0 + np.linalg.norm(peg_theta)) / (peg_theta[1] + 1e-16)
lower_margin = -(peg_theta[0]*xs + peg_theta_0 - np.linalg.norm(peg_theta)) / (peg_theta[1] + 1e-16)
plt.plot(xs, ys, '-', label="Decision Boundary")
plt.plot(xs, upper_margin,'--', color = "grey", label="Margin Boundary")
plt.plot(xs, lower_margin, '--', color = "grey")
plt.legend(loc="upper left")
plt.xlabel("$X_1$")
plt.ylabel("$X_2$")
plt.title("Pegasos Decesion and Margin Boundaries")
plt.show()
]]> machine-learning tutorial https://ashudva.github.io/blog/posts/2020-02-13-Linear-Classifiers/ Sun, 21 Sep 2025 21:36:10 GMT Physical activity classification using smartphone-data https://ashudva.github.io/blog/posts/2020-01-22-HAR/

Data Loading and Exploration

load essential libraries

import pandas as pd
import numpy as np
import matplotlib as mpl
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
X = pd.read_csv("data/har/time_series.csv") 
y = pd.read_csv("data/har/labels.csv").label

activities = {1:'standing', 2:'walking', 3:'stairs-down', 4:'stairs-up'}
labels = []
for i in range(len(y)):
    label = np.repeat(y[i], 9)
    labels.extend([*label, y[i]])
    
X['label'] = labels[:-6]
y = X.label
X.head()
Unnamed: 0 timestamp UTC time accuracy x y z label
0 20586 1565109930787 2019-08-06T16:45:30.787 unknown -0.006485 -0.934860 -0.069046 1
1 20587 1565109930887 2019-08-06T16:45:30.887 unknown -0.066467 -1.015442 0.089554 1
2 20588 1565109930987 2019-08-06T16:45:30.987 unknown -0.043488 -1.021255 0.178467 1
3 20589 1565109931087 2019-08-06T16:45:31.087 unknown -0.053802 -0.987701 0.068985 1
4 20590 1565109931188 2019-08-06T16:45:31.188 unknown -0.054031 -1.003616 0.126450 1 
X.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3744 entries, 0 to 3743
Data columns (total 8 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   Unnamed: 0  3744 non-null   int64  
 1   timestamp   3744 non-null   int64  
 2   UTC time    3744 non-null   object 
 3   accuracy    3744 non-null   object 
 4   x           3744 non-null   float64
 5   y           3744 non-null   float64
 6   z           3744 non-null   float64
 7   label       3744 non-null   int64  
dtypes: float64(3), int64(3), object(2)
memory usage: 234.1+ KB
y
0       1
1       1
2       1
3       1
4       1
       ..
3739    4
3740    4
3741    4
3742    4
3743    4
Name: label, Length: 3744, dtype: int64
standing = X.label == 1
walking = X.label == 2
stairs_down = X.label == 3
stairs_up = X.label == 4

x = np.linspace(0, len(labels)-6, len(labels)-6)

mpl.style.use("fivethirtyeight")
%matplotlib notebook

fig, ax = plt.subplots(2, 2, figsize=(12, 8))
ax[0, 0].plot(x[standing], X.x[standing], x[standing],
              X.y[standing], x[standing], X.z[standing], '-', alpha=0.4)
ax[0, 0].set_title(activities[1])

ax[0, 1].plot(x[walking], X.x[walking], x[walking],
              X.y[walking], x[walking], X.z[walking], '-', alpha=0.4)
ax[0, 1].set_title(activities[2])

ax[1, 0].plot(x[stairs_down],
              X.x[stairs_down], x[stairs_down],
              X.y[stairs_down], x[stairs_down],
              X.z[stairs_down], '-', alpha=0.4)
ax[1, 0].set_title(activities[3])

ax[1, 1].plot(X.timestamp[stairs_up], X.x[stairs_up], X.timestamp[stairs_up],
              X.y[stairs_up], X.timestamp[stairs_up], X.z[stairs_up], '-', alpha=0.4)
ax[1, 1].set_title(activities[4])

fig.suptitle("Tri-Axial Linear Acceleration", fontsize=25)
plt.gcf().autofmt_xdate()
fig.text(0.5, 0.05, 'time', ha='center', fontsize=16)
fig.text(0.01, 0.5, 'acceleration', va='center', rotation='vertical', fontsize=16)
fig.show()
mpl.style.use("fivethirtyeight")
plt.plot(X.timestamp, X.x, X.timestamp, X.y, X.timestamp, X.z, '-', alpha=0.4)
plt.title("Tri-Axial Linear Acceleration")
plt.xlabel("time")
plt.ylabel("acceleration")
plt.gcf().autofmt_xdate()
plt.show()
walking = X.label == 1
standing = X.label == 2
stairs_down = X.label == 3
stairs_up = X.label == 4
%matplotlib notebook
fig,axs = plt.subplots(4,1, figsize = (16,12), sharex=True)
sns.kdeplot(X.x[walking], shade=True, ax=axs[0])
sns.kdeplot(X.y[walking], shade=True, ax=axs[0])
sns.kdeplot(X.z[walking], shade=True, ax=axs[0])

sns.kdeplot(X.x[standing], shade=True, ax=axs[1])
sns.kdeplot(X.y[standing], shade=True, ax=axs[1])
sns.kdeplot(X.z[standing], shade=True, ax=axs[1])

sns.kdeplot(X.x[stairs_down], shade=True, ax=axs[2])
sns.kdeplot(X.y[stairs_down], shade=True, ax=axs[2])
sns.kdeplot(X.z[stairs_down], shade=True, ax=axs[2])

sns.kdeplot(X.x[stairs_up], shade=True, ax=axs[3])
sns.kdeplot(X.y[stairs_up], shade=True, ax=axs[3])
sns.kdeplot(X.z[stairs_up], shade=True, ax=axs[3])

axs[0].set_title(activities[1])
axs[1].set_title(activities[2])
axs[2].set_title(activities[3])
axs[3].set_title(activities[4])

axs[0].set_xlim((-3,2))
fig.suptitle("Tri-Axial Acceralometer Data", fontsize=20)
fig.show()

Modelling

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import r2_score
from sklearn.model_selection import cross_val_score

train_covariates = X[['x', 'y', 'z']]
target = X.label
clf = RandomForestClassifier(max_depth=10, random_state=0)

def correlation(estimator, X, y):
    estimator.fit(X,y)
    y_pred = estimator.predict(X)
    return r2_score(y, y_pred)

def accuracy(estimator, X, y):
    estimator.fit(X,y)
    y_pred = estimator.predict(X)
    return accuracy_score(y, y_pred)

test_score = accuracy(clf, train_covariates, target)

val_scores = cross_val_score(clf,
                         train_covariates,
                         target,
                         cv=10,
                         scoring=accuracy)
scores
array([0.97066667, 0.968     , 0.96      , 0.976     , 0.93582888,
       0.9973262 , 0.94919786, 0.98930481, 0.98128342, 0.99197861])
test_score
0.7641559829059829
scores.mean()
0.9719586452762924
]]> project machine-learning https://ashudva.github.io/blog/posts/2020-01-22-HAR/ Sun, 21 Sep 2025 21:36:10 GMT Birds Migration Patterns https://ashudva.github.io/blog/posts/2020-12-27-Bird-Migration/

Same old quotidian work of importing libraries and the data

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib notebook
plt.style.use('ggplot')

birddata = pd.read_csv("data/bird/bird_tracking.csv", index_col=0)
birddata.head()
altitude date_time device_info_serial direction latitude longitude speed_2d bird_name
0 71 2013-08-15 00:18:08+00 851 -150.469753 49.419860 2.120733 0.150000 Eric
1 68 2013-08-15 00:48:07+00 851 -136.151141 49.419880 2.120746 2.438360 Eric
2 68 2013-08-15 01:17:58+00 851 160.797477 49.420310 2.120885 0.596657 Eric
3 73 2013-08-15 01:47:51+00 851 32.769360 49.420359 2.120859 0.310161 Eric
4 69 2013-08-15 02:17:42+00 851 45.191230 49.420331 2.120887 0.193132 Eric 
birddata.info
<bound method DataFrame.info of        altitude               date_time  device_info_serial   direction  \
0            71  2013-08-15 00:18:08+00                 851 -150.469753   
1            68  2013-08-15 00:48:07+00                 851 -136.151141   
2            68  2013-08-15 01:17:58+00                 851  160.797477   
3            73  2013-08-15 01:47:51+00                 851   32.769360   
4            69  2013-08-15 02:17:42+00                 851   45.191230   
...         ...                     ...                 ...         ...   
61915        11  2014-04-30 22:00:08+00                 833   45.448157   
61916         6  2014-04-30 22:29:57+00                 833 -112.073055   
61917         5  2014-04-30 22:59:52+00                 833   69.989037   
61918        16  2014-04-30 23:29:43+00                 833   88.376373   
61919         9  2014-04-30 23:59:34+00                 833  149.949008   

        latitude  longitude  speed_2d bird_name  
0      49.419860   2.120733  0.150000      Eric  
1      49.419880   2.120746  2.438360      Eric  
2      49.420310   2.120885  0.596657      Eric  
3      49.420359   2.120859  0.310161      Eric  
4      49.420331   2.120887  0.193132      Eric  
...          ...        ...       ...       ...  
61915  51.352572   3.177151  0.208087     Sanne  
61916  51.352585   3.177144  1.522662     Sanne  
61917  51.352622   3.177257  3.120545     Sanne  
61918  51.354641   3.181509  0.592115     Sanne  
61919  51.354474   3.181057  0.485489     Sanne  

[61920 rows x 8 columns]>
birddata.tail()
altitude date_time device_info_serial direction latitude longitude speed_2d bird_name
61915 11 2014-04-30 22:00:08+00 833 45.448157 51.352572 3.177151 0.208087 Sanne
61916 6 2014-04-30 22:29:57+00 833 -112.073055 51.352585 3.177144 1.522662 Sanne
61917 5 2014-04-30 22:59:52+00 833 69.989037 51.352622 3.177257 3.120545 Sanne
61918 16 2014-04-30 23:29:43+00 833 88.376373 51.354641 3.181509 0.592115 Sanne
61919 9 2014-04-30 23:59:34+00 833 149.949008 51.354474 3.181057 0.485489 Sanne

The data consists of almost 62,000 data points and 9 features or columns

birddata.bird_name.value_counts()
Nico     21121
Sanne    21004
Eric     19795
Name: bird_name, dtype: int64

There are 3 types of birds in our dataset, named Nico, Sanne, Eric

Linear estimation - because the earth is not flat - of flight trajectory of bird migration of a particular bird “Eric”. The trajectory will be substantially distorted because we have not done any Cartographic Projection of the flight trajectory.

This plot is just to get a rought look at the flight trajectory of a bird

ind = birddata.bird_name == "Eric"
x, y = birddata.longitude[ind], birddata.latitude[ind]
plt.figure(figsize=(5,5), dpi=100)
plt.plot(x, y, "o", ms=2)
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.title("Eric flight trajectory")
plt.show()

Let’s plot the flight trajectory for all of three birds

birds = birddata.bird_name.unique()
plt.figure(figsize=(5,5), dpi=100)
for bird in birds:
    ind = birddata.bird_name == bird
    x, y = birddata.longitude[ind], birddata.latitude[ind]
    plt.plot(x, y, "o", ms=2, label=bird)
plt.xlabel("Longitude")
plt.ylabel("Latitude")
plt.title("Birds flight trajectory")
plt.legend(loc="lower right")
plt.show()

To further proceed, we would like to chech if our data consists of missing values and handle them accordingly We’ll be using sklearn for the preprocessing of the data and handling the missing values

birddata.isnull().sum()
altitude                0
date_time               0
device_info_serial      0
direction             443
latitude                0
longitude               0
speed_2d              443
bird_name               0
dtype: int64

Two columns direction and speed_2d consists of same no. of missing values but for direction column mean is not an appropriate approximation. Therefor we’ll first impute speed_2d with mean and then we’ll use n_neighbours strategy for imputation of direction

from sklearn.impute import SimpleImputer, KNNImputer
# default args are what we want i.e. missing_values=nan, strategy='mean'
imputer = SimpleImputer()
birddata["speed_2d"] = imputer.fit_transform(birddata[['speed_2d']])
birddata.isnull().sum()
altitude                0
date_time               0
device_info_serial      0
direction             443
latitude                0
longitude               0
speed_2d                0
bird_name               0
dtype: int64

Let’s impute the direction column with default args

imputer = KNNImputer()
imputer.fit(birddata.loc[:, 'direction':'speed_2d'])
birddata.loc[:, 'direction':'speed_2d'] = imputer.transform(birddata.loc[:, 'direction':'speed_2d'])
birddata.tail()
altitude date_time device_info_serial direction latitude longitude speed_2d bird_name
61915 11 2014-04-30 22:00:08+00 833 45.448157 51.352572 3.177151 0.208087 Sanne
61916 6 2014-04-30 22:29:57+00 833 -112.073055 51.352585 3.177144 1.522662 Sanne
61917 5 2014-04-30 22:59:52+00 833 69.989037 51.352622 3.177257 3.120545 Sanne
61918 16 2014-04-30 23:29:43+00 833 88.376373 51.354641 3.181509 0.592115 Sanne
61919 9 2014-04-30 23:59:34+00 833 149.949008 51.354474 3.181057 0.485489 Sanne

Ommit the last row as it’s unnecessarily introduced into the dataset.

birddata = birddata.iloc[:-1, :]
birddata.tail()
altitude date_time device_info_serial direction latitude longitude speed_2d bird_name
61914 -10 2014-04-30 21:29:45+00 833 -10.057916 51.352661 3.177122 5.531148 Sanne
61915 11 2014-04-30 22:00:08+00 833 45.448157 51.352572 3.177151 0.208087 Sanne
61916 6 2014-04-30 22:29:57+00 833 -112.073055 51.352585 3.177144 1.522662 Sanne
61917 5 2014-04-30 22:59:52+00 833 69.989037 51.352622 3.177257 3.120545 Sanne
61918 16 2014-04-30 23:29:43+00 833 88.376373 51.354641 3.181509 0.592115 Sanne 
birddata.isnull().sum()
altitude              0
date_time             0
device_info_serial    0
direction             0
latitude              0
longitude             0
speed_2d              0
bird_name             0
dtype: int64

Let’s try plotting a histogram of speed_2d for a particular bird Eric

# ind is already defined above for "Eric"
speed = birddata.speed_2d[ind]
plt.figure(figsize=(5,5), dpi=100)
plt.hist(speed, bins=np.linspace(0,30,20), density=True)
plt.title("Eric 2D speed Histogram")
plt.xlabel("Speed (m/s)")
plt.ylabel("Frequency")
plt.show()

Notice that in our dataset we have a column that consists of datetime, so lets check what is the datatype of this column

type(birddata.date_time[0])
str
birddata.date_time[0]
'2013-08-15 00:18:08+00'

datetime in our dataset is in str format and to be able to perform computation - computing time interval between two data points - on datetime we would like it convert to a datetime object

import datetime as dt
# remove '+00 from the strings as the time is already in UTC'
timestamps = birddata.date_time
timestamps = [stamp[:-3] for stamp in timestamps]
timestamps[:3]
['2013-08-15 00:18:08', '2013-08-15 00:48:07', '2013-08-15 01:17:58']
# convert str to a datetime object to be able to perform arithmetic operation on it
timestamps = list(map(lambda str_stamp: dt.datetime.strptime(
    str_stamp, "%Y-%m-%d %H:%M:%S"), timestamps))
birddata["timestamp"] = pd.Series(timestamps, index=birddata.index)
SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  birddata["timestamp"] = pd.Series(timestamps, index=birddata.index)
birddata.tail()
altitude date_time device_info_serial direction latitude longitude speed_2d bird_name timestamp
61914 -10 2014-04-30 21:29:45+00 833 -10.057916 51.352661 3.177122 5.531148 Sanne 2014-04-30 21:29:45
61915 11 2014-04-30 22:00:08+00 833 45.448157 51.352572 3.177151 0.208087 Sanne 2014-04-30 22:00:08
61916 6 2014-04-30 22:29:57+00 833 -112.073055 51.352585 3.177144 1.522662 Sanne 2014-04-30 22:29:57
61917 5 2014-04-30 22:59:52+00 833 69.989037 51.352622 3.177257 3.120545 Sanne 2014-04-30 22:59:52
61918 16 2014-04-30 23:29:43+00 833 88.376373 51.354641 3.181509 0.592115 Sanne 2014-04-30 23:29:43 
birddata.timestamp[0]
Timestamp('2013-08-15 00:18:08')
birddata.timestamp[4] - birddata.timestamp[3]
Timedelta('0 days 00:29:51')

Now that we have our timestamp in place, we’d like to see how often or when the data was collected in the process. Also for this we’ll limit ourselves to Eric

times = birddata.timestamp[birddata.bird_name == "Eric"]
elapsed_time = [time - times[0] for time in times]
plt.figure(figsize=(5,5), dpi=100)
plt.plot(np.array(elapsed_time) / dt.timedelta(days=1))
plt.xlabel("Observations")
plt.ylabel("Elapsed Time")
plt.title("Elapsed time for Eric")
plt.show()

Our next goal is to find when does “Eric” migrate. To achieve that we’ll plot the daily mean speed of Eric. The data is recorded unevenly i.e. on some days data was collected more times and some days it was collected less no. of times. We’ll start by getting indices of speed_2d that were collected on the same day and then take mean of those speeds, followed by plotting them to see if there’s any pattern.

data = birddata[birddata.bird_name == "Eric"]
elapsed_days = np.array(elapsed_time) / dt.timedelta(days=1)
daily_mean_speed = []
next_day = 1
inds = []
for i,t in enumerate(elapsed_days):
    if t < next_day:
        inds.append(i)
    else:
        daily_mean_speed.append(np.mean(data.speed_2d[inds]))
        next_day += 1
        inds = []
plt.figure(figsize=(7,5), dpi=100)
plt.plot(daily_mean_speed)
plt.xlabel("Days")
plt.ylabel("Speed (m/s)")
plt.title("Eric Daily Mean Speed")
plt.show()

Migration Pattern

from the 2D-Speed of Eric it can be argued that during days 90 - 100 and 230 - 240, speed of Eric was significantly higher than other days. So it can be said that Eric migrated during those days. To corroborate our beliefs about the migration we would like to look at the place at which Eric ended up during those days.

Cartographic Projection using Cartopy

Earlier we tried plotting migration pattern of birds but it was not quite what we were looking for because it was not a cartographic projection. So now we’ll use Cartopy for cartographic projection of flight patterns of the birds.

import cartopy.crs as ccrs
import cartopy.feature as cfeature

proj = ccrs.Mercator()

plt.figure(figsize=(8,8), dpi=100)
ax = plt.axes(projection=proj)
ax.set_extent((-25.0, 20.0, 52.0, 10.0))
ax.add_feature(cfeature.LAND)
ax.add_feature(cfeature.OCEAN)
ax.add_feature(cfeature.COASTLINE)
ax.add_feature(cfeature.BORDERS, linestyle=':', alpha = 0.95)

for bird in birds:
    ix = birddata["bird_name"] == bird
    x, y = birddata.longitude[ix], birddata.latitude[ix]
    ax.plot(x,y, '.', transform=ccrs.Geodetic(), label=bird)
    
plt.legend(loc="upper left")
plt.savefig("map.pdf")
plt.show()

Analysis for each bird

We’ll now group the data by bird_name to get the average 2D speed of the birds

data = birddata.groupby('bird_name')
names = pd.Series(birds, name="Bird Name")
mean_speeds = data.speed_2d.mean()
data.speed_2d.describe().set_index(names)
count mean std min 25% 50% 75% max
Bird Name
Eric 19795.0 2.301654 3.558977 0.0 0.344819 1.012719 2.511553 63.488066
Nico 21121.0 2.906855 3.726812 0.0 0.490408 1.560000 3.701148 48.381510
Sanne 21003.0 2.451794 3.366275 0.0 0.411096 1.174564 2.823376 57.201748 
mean_altitudes = data.altitude.mean()
data.altitude.describe().set_index(names)
count mean std min 25% 50% 75% max
Bird Name
Eric 19795.0 60.249406 115.333013 -830.0 7.0 26.0 106.0 4808.0
Nico 21121.0 67.900478 153.498842 -965.0 2.0 16.0 112.0 6965.0
Sanne 21003.0 29.160882 133.453211 -1010.0 1.0 8.0 22.0 6145.0

Mean Altitude at each day

We’ll now group our data by each date to get the mean altitude of each day

# Convert date_time to pd.datetime objects
birddata.date_time = pd.to_datetime(birddata.date_time)
birddata["date"] = birddata.date_time.dt.date
grouped_bydates = birddata.groupby("date")
mean_altitudes_perday = grouped_bydates.altitude.mean()
mean_altitudes_perday
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py:5168: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[name] = value
<ipython-input-30-975ec523ff9b>:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  birddata["date"] = birddata.date_time.dt.date
date
2013-08-15    134.092000
2013-08-16    134.839506
2013-08-17    147.439024
2013-08-18    129.608163
2013-08-19    180.174797
                 ...    
2014-04-26     15.118012
2014-04-27     23.897297
2014-04-28     37.716867
2014-04-29     19.244792
2014-04-30     13.982857
Name: altitude, Length: 259, dtype: float64
grouped_birdday = birddata.groupby(["bird_name", "date"])
mean_altitudes_perday = grouped_birdday.altitude.mean()

mean_altitudes_perday.head()
bird_name  date      
Eric       2013-08-15     74.988095
           2013-08-16    127.773810
           2013-08-17    125.890244
           2013-08-18    121.353659
           2013-08-19    134.928571
Name: altitude, dtype: float64
eric_daily_speed  = grouped_birdday.speed_2d.mean()["Eric"]
sanne_daily_speed = grouped_birdday.speed_2d.mean()["Sanne"]
nico_daily_speed  = grouped_birdday.speed_2d.mean()["Nico"]

plt.figure(figsize=(8,5), dpi=100)
eric_daily_speed.plot(label="Eric")
sanne_daily_speed.plot(label="Sanne")
nico_daily_speed.plot(label="Nico")
plt.xlabel("Date")
plt.ylabel("Mean 2D Speed (m/s)")
plt.title("Mean 2D Speeds")
plt.legend(loc="upper left")
plt.show()
]]> case-study Visualization EDA https://ashudva.github.io/blog/posts/2020-12-27-Bird-Migration/ Sun, 21 Sep 2025 21:36:10 GMT Health and world economical growth https://ashudva.github.io/blog/posts/2020-10-10-Gapminder/

Abstract

The objective of this case study is to understand the change in World Health and Economics using Data Visualization, EDA, and Summarization. In this study two main questions, Is it a fair characterization of today’s world to say that it is divided into a Westorn Rich Nations (Europian Countries, USA et cetera), and Developing Countries (Asia, Africa et cetera)? Has the Income Inequality worsened during the last 40 years? The study involves data from Gapminder Foundation about trends in world health and economics. Study emphasizes the use of data visualization to better understand the trends and insights. This study is purely based on the Gapminder TED talks New Insights on poverty

Load and explore data

library(dslabs)
library(tidyverse)
library(ggthemes)
library(ggrepel)
data(gapminder)
gapminder %>% head()
str(gapminder)
country year infant_mortality life_expectancy fertility population gdp continent region
Albania 1960 115.40 62.87 6.19 1636054 NA Europe Southern Europe
Algeria 1960 148.20 47.50 7.65 11124892 13828152297 Africa Northern Africa
Angola 1960 208.00 35.98 7.32 5270844 NA Africa Middle Africa
Antigua and Barbuda 1960 NA 62.97 4.43 54681 NA Americas Caribbean
Argentina 1960 59.87 65.39 3.11 20619075 108322326649 Americas South America
Armenia 1960 NA 66.86 4.55 1867396 NA Asia Western Asia 
'data.frame':   10545 obs. of  9 variables:
 $ country         : Factor w/ 185 levels "Albania","Algeria",..: 1 2 3 4 5 6 7 8 9 10 ...
 $ year            : int  1960 1960 1960 1960 1960 1960 1960 1960 1960 1960 ...
 $ infant_mortality: num  115.4 148.2 208 NA 59.9 ...
 $ life_expectancy : num  62.9 47.5 36 63 65.4 ...
 $ fertility       : num  6.19 7.65 7.32 4.43 3.11 4.55 4.82 3.45 2.7 5.57 ...
 $ population      : num  1636054 11124892 5270844 54681 20619075 ...
 $ gdp             : num  NA 1.38e+10 NA NA 1.08e+11 ...
 $ continent       : Factor w/ 5 levels "Africa","Americas",..: 4 1 1 2 2 3 2 5 4 3 ...
 $ region          : Factor w/ 22 levels "Australia and New Zealand",..: 19 11 10 2 15 21 2 1 22 21 ...

Data consists of 10545 Observations and 9 Variables, it consists of varibles like country, region, health outcomes (life_expectancy, fertility), economic aspects (gdp), Add gdp_cp var i.e. gdp per capita which represents the wealth of a country

# Add gdp_cp var i.e. gdp per capita which represents wealth of a country
gapminder <- gapminder %>% mutate(gdp_pc = gdp / population, dollars_per_day = gdp_pc/365)
head(gapminder)
country year infant_mortality life_expectancy fertility population gdp continent region gdp_pc dollars_per_day
Albania 1960 115.40 62.87 6.19 1636054 NA Europe Southern Europe NA NA
Algeria 1960 148.20 47.50 7.65 11124892 13828152297 Africa Northern Africa 1242.992 3.405458
Angola 1960 208.00 35.98 7.32 5270844 NA Africa Middle Africa NA NA
Antigua and Barbuda 1960 NA 62.97 4.43 54681 NA Americas Caribbean NA NA
Argentina 1960 59.87 65.39 3.11 20619075 108322326649 Americas South America 5253.501 14.393153
Armenia 1960 NA 66.86 4.55 1867396 NA Asia Western Asia NA NA

Analysis

Infant Mortality

Getting started with testing our knowledge regarding differences in infant mortality across differnt countries, for each of the pairs of countries given below, Which country do you think had the highest child mortality rate in 2015? and Which pairs do you think are the most similar?

Country1 Country2
Sri Lanka Turkey
Poland South Korea
Malaysia Russia
Pakistan Vietnam
Thialand South Africa

It is commonly percieved that the non-europian countries like Sri Lanka, South Korea have higher mortality rates than their Europian counterparts. Also the developing countries like Pakistan are considered to have high mortality rates. Lets take a look at the data to see whether it is just a superstition or a fact.

countries <- c("Sri Lanka","Turkey","Poland","South Korea","Malaysia","Russia","Pakistan","Vietnam","Thailand","South Africa")
mortality <- data.frame()
for (i in seq(1,10,2)) {
    mortality1 <- gapminder %>%
        filter(year == 2015 & country %in% countries[c(i,i+1)]) %>%
        select(country, infant_mortality)
    mortality1 <- cbind(mortality1[1,], mortality1[2,])
    mortality1 <- mortality1[-2,]
    mortality <- rbind.data.frame(mortality,mortality1)
}
mortality
country infant_mortality country infant_mortality
Sri Lanka 8.4 Turkey 11.6
South Korea 2.9 Poland 4.5
Malaysia 6.0 Russia 8.2
Pakistan 65.8 Vietnam 17.3
South Africa 33.6 Thailand 10.5

We see that the European countries on this list have higher child mortality rates: Poland has a higher rate than South Korea, and Russia has a higher rate than Malaysia. We also see that Pakistan has a much higher rate than Vietnam, and South Africa has a much higher rate than Thailand. The reason for this stems from the preconceived notion that the world is divided into two groups: the western world (Western Europe and North America), characterized by long life spans and small families, versus the developing world (Africa, Asia, and Latin America) characterized by short life spans and large families.

Life Expectancy, Fertility

scatterplot of life expectancy versus fertility rates (average number of children per woman) 50 years ago

#basic scatterplot of life expectancy versus fertility in year 1962
ds_theme_set()
gapminder %>%
    filter(year == 1962) %>%
    ggplot(aes(fertility, life_expectancy)) +
    geom_point(size = 1) +
    ylab("Life Expectancy") +
    xlab("Fertility") +
    ggtitle("Life Expectancy vs Fertility (1962)") +
    theme_gdocs()

Most points fall into two distinct categories: Life expectancy around 70 years and 3 or fewer children per family, and Life expectancy lower than 65 years and more than 5 children per family.

To confirm that indeed these countries are from the regions we expect, we can use color to represent continent.

# Add color based on continent
gapminder %>%
    filter(year == 1962) %>%
    ggplot(aes(fertility, life_expectancy, color = continent)) +
    geom_point() +
    labs(title = "Life Expectancy vs Fertility",
         subtitle = "1962",
         caption = "Data from Gapminder foundation study",
         x = "Fertility",
         y = "Life Expectancy",
         color = "Continent") +
    theme_gdocs()

In 1962, “the West versus developing world” view was grounded in some reality. Is this still the case 50 years later?

Changes over time

Facet life expectancy vs fertility by continent and year to see how it changed from 1962 to 2012 for different continents using side-by-side plots

gapminder %>%
    filter(year %in% c(1962, 2012)) %>%
    ggplot(aes(fertility, life_expectancy, color = continent)) +
    geom_point() +
    facet_grid(continent ~ year) +
    labs(title = "Life Expectancy vs Fertility",
         subtitle = "Coparison between 1962 and 2012",
         caption = "Data from Gapminder foundation study",
         x = "Fertility",
         y = "Life Expectancy",
         color = "Continent") +
    theme_gdocs()

Except for countries in Africa continent almost all of the countries had significant increase in life expectancy and reduced fertility and Europian countries has the most significant increase of all thus **It’s quite clear from the plot, notion that Europian and American countries have a higher life-expectancy is somewhat correct

Facet by year only

gapminder %>% filter(year %in% c("1962", "2012")) %>%
    ggplot(aes(fertility, life_expectancy, col = continent)) +
    geom_point(size = 2) +
    facet_grid(.~ year) +
    ylab("Life Expectancy") +
    xlab("Fertility")

Faceting by year 1962 and 2012 shows, though all of the countries had a increase in life-expectancy, European and American countries had the highest life-expectancy. This plot clearly shows that the majority of countries have moved from the developing world cluster to the western world one. In 2012, the western versus developing world view no longer makes sense. This is particularly clear when comparing Europe to Asia, the latter of which includes several countries that have made great improvements.

Facet by year, plots wrapped onto multiple rows to see changes over the years in life-expectancy to explore how this transformation happened through the years, we can make the plot for several years. This plot clearly shows how most Asian countries have improved at a much faster rate than European ones.

gapminder %>%
    filter(year %in% c(1962, 1980, 1990, 2000, 2012), continent %in% c("Asia", "Europe")) %>%
    ggplot(aes(fertility, life_expectancy, color = continent)) +
    geom_point() +
    facet_wrap(~year) +
    labs(title = "Life Expectancy vs Fertility",
         subtitle = "How the old superstition changed over time",
         caption = "Data from Gapminder foundation study",
         x = "Fertility",
         y = "Life Expectancy",
         color = "Continent") +
    theme_gdocs()

gapminder %>%
    filter(year %in% c(1962, 1980, 1990, 2000, 2012), continent %in% c("Asia", "Europe")) %>%
    ggplot(aes(fertility, life_expectancy, color = continent)) +
    geom_point() +
    facet_wrap(~year, scales = "free") +
    labs(title = "Life Expectancy vs Fertility",
         subtitle = "Trend will be unobservable if the scale is free",
         caption = "Take a closer look at the scales",
         x = "Fertility",
         y = "Life Expectancy",
         color = "Continent") +
    theme_gdocs()

gapminder %>%
    filter(year %in% c(1962, 2012)) %>%
    ggplot(aes(fertility, life_expectancy, color = continent)) +
    geom_point() +
    facet_grid(continent ~ year) +
    labs(title = "Life Expectancy vs Fertility",
         subtitle = "Coparison between 1962 and 2012",
         caption = "Data from Gapminder foundation study",
         x = "Fertility",
         y = "Life Expectancy",
         color = "Continent") +
    theme_gdocs()

Time Series Analysis

The visualizations above effectively illustrate that data no longer supports the western versus developing world view. Once we see these plots, new questions emerge. For example, which countries are improving more and which ones less? Was the improvement constant during the last 50 years or was it more accelerated during certain periods? For a closer look that may help answer these questions, we are going to use time series plots.

# scatterplot of US fertility by year
gapminder %>%
    filter(country == "United States", !is.na(fertility)) %>%
    ggplot(aes(year, fertility)) +
    geom_point() +
    theme_gdocs()

We see that the trend is not linear at all. Instead there is sharp drop during the 1960s and 1970s to below 2. Then the trend comes back to 2 and stabilizes during the 1990s. When the points are regularly and densely spaced, as they are here, we create curves by joining the points with lines, to convey that these data are from a single series, here a country.

Line plot - US fertility

# line plot of US fertility by year
gapminder %>%
    filter(country == "United States", !is.na(fertility)) %>%
    ggplot(aes(year, fertility)) +
    geom_line() +
    theme_gdocs()

Line plot - Korea and Germany

This is particularly helpful when we look at two countries. If we subset the data to include two countries, one from Europe and one from Asia.

countries <- c("South Korea", "Germany")
gapminder %>% filter(country %in% countries & !is.na(fertility)) %>%
    ggplot(aes(year, fertility, group = country)) +
    geom_line() +
    theme_gdocs()

The plot clearly shows how South Korea’s fertility rate dropped drastically during the 1960s and 1970s, and by 1990 had a similar rate to that of Germany.

labels <- data.frame(country = countries, x = c(1986, 1975), y = c(2.5,2.0))
gapminder %>% filter(country %in% countries & !is.na(fertility)) %>%
    ggplot(aes(year, fertility, col = country)) +
    geom_text(data = labels, aes(x, y, label = country), size = 4) +
    theme(legend.position = "none") +
    geom_line()

# life expectancy time series - lines colored by country and labeled, no legend
labels <- data.frame(country = countries, x = c(1975, 1965), y = c(60, 72))
gapminder %>% filter(country %in% countries) %>%
    ggplot(aes(year, life_expectancy, col = country)) +
    geom_line() +
    geom_text(data = labels, aes(x, y, label = country), size = 5) +
    theme(legend.position = "none") +
    ylab("Life Expectancy") +
    xlab("Fertility")

countries <- c("Germany", "United States")
gapminder %>%
    filter(country %in% countries & !is.na(gdp_pc)) %>%
    ggplot(aes(gdp_pc, life_expectancy, col = country)) +
    geom_point() +
    xlab("Per Capita GDP") +
    ylab("Life Expectancy")

past_year <- 1970
gapminder %>%
    filter(year == past_year & !is.na(gdp)) %>%
    ggplot(aes(dollars_per_day)) +
    geom_histogram(binwidth = 1, col = "black") +
    xlab("Income (Dollars/Day)") +
    ylab("Count")

past_year <- 1970
gapminder %>%
    filter(year == past_year & !is.na(gdp)) %>%
    ggplot(aes(dollars_per_day)) +
    geom_histogram(binwidth = 1, color = "black") +
    scale_x_continuous(trans = "log2") +
    xlab("Income (Dollars/Day)") +
    ylab("Count")

# Add dollars_per_day or Income per day Variable/Column to the data
gapminder <- gapminder %>% mutate(dollars_per_day = gdp/population/365)

# Number of regions 
length(levels(gapminder$region))

# Plot Boxplot
gapminder %>%
    filter(year == past_year & !is.na(gdp)) %>%
    ggplot(aes(region, dollars_per_day)) +
    geom_boxplot() +
    scale_y_continuous(trans = "log2") +
    theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
    ylab("Income (Dollars/Day)") +
    xlab("Region")
22 

# Reorder and Color Regions for better comparison
gapminder %>%
    filter(year == past_year & !is.na(gdp)) %>%
    # Reorder region by median income
    mutate(region = reorder(region, dollars_per_day, FUN = median)) %>%
    ggplot(aes(region, dollars_per_day, fill = continent)) +
    geom_boxplot() +
    scale_y_continuous(trans = "log2") +
    theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
    geom_point(show.legend = FALSE) +
    ylab("Income (Dollars/Day)") +
    xlab("Region")

# add dollars per day variable and define past year
gapminder <- gapminder %>%
    mutate(dollars_per_day = gdp/population/365)
past_year <- 1970

# define Western countries
west <- c("Western Europe", "Northern Europe", "Southern Europe", "Northern America", "Australia and New Zealand")

# facet by West vs devloping
gapminder %>%
    filter(year == past_year & !is.na(gdp)) %>%
    mutate(group = ifelse(region %in% west, "West", "Developing")) %>%
    ggplot(aes(dollars_per_day)) +
    geom_histogram(binwidth = 1, color = "black") +
    scale_x_continuous(trans = "log2") +
    facet_grid(. ~ group)

# facet by West/developing and year
present_year <- 2010
gapminder %>%
    filter(year %in% c(past_year, present_year) & !is.na(gdp)) %>%
    mutate(group = ifelse(region %in% west, "West", "Developing")) %>%
    ggplot(aes(dollars_per_day)) +
    geom_histogram(binwidth = 1, color = "black") +
    scale_x_continuous(trans = "log2") +
    facet_grid(year ~ group)

# define countries that have data available in both years
country_list_1 <- gapminder %>%
    filter(year == past_year & !is.na(dollars_per_day)) %>% .$country
    country_list_2 <- gapminder %>%
    filter(year == present_year & !is.na(dollars_per_day)) %>% .$country
    country_list <- intersect(country_list_1, country_list_2)

# make histogram including only countries with data available in both years
gapminder %>%
    filter(year %in% c(past_year, present_year) & country %in% country_list) %>%    # keep only selected countries
    mutate(group = ifelse(region %in% west, "West", "Developing")) %>%
    ggplot(aes(dollars_per_day)) +
    geom_histogram(binwidth = 1, color = "black") +
    scale_x_continuous(trans = "log2") +
    facet_grid(year ~ group)

p <- gapminder %>%
    filter(year %in% c(past_year, present_year) & country %in% country_list) %>%
    mutate(region = reorder(region, dollars_per_day, FUN = median)) %>%
    ggplot() +
    theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
    xlab("") + scale_y_continuous(trans = "log2")
    
 p + geom_boxplot(aes(region, dollars_per_day, fill = continent)) +
     facet_grid(year ~ .)
 
 # arrange matching boxplots next to each other, colored by year
 p + geom_boxplot(aes(region, dollars_per_day, fill = factor(year)))

west <- c("Western Europe", "Northern Europe", "Southern Europe", "Northern America", "Australia and New Zealand")

dat <- gapminder %>%
    filter(year %in% c(2010, 2015) & region %in% west & !is.na(life_expectancy) & population > 10^7)

dat %>%
    mutate(location = ifelse(year == 2010, 1, 2),
           location = ifelse(year == 2015 & country %in% c("United Kingdom", "Portugal"),
                             location + 0.22, location),
           hjust = ifelse(year == 2010, 1, 0)) %>%
    mutate(year = as.factor(year)) %>%
    ggplot(aes(year, life_expectancy, group = country)) +
    geom_line(aes(color = country), show.legend = FALSE) +
    geom_text(aes(x = location, label = country, hjust = hjust), show.legend = FALSE) +
    xlab("") +
    ylab("Life Expectancy") +
    ggtitle("Change in Life Expectancy") +
    theme_gdocs()

dat %>%
    mutate(year = paste0("life_expectancy_", year)) %>%
    select(country, year, life_expectancy) %>% spread(year, life_expectancy) %>%
    mutate(average = (life_expectancy_2015 + life_expectancy_2010)/2,
                difference = life_expectancy_2015 - life_expectancy_2010) %>%
    ggplot(aes(average, difference, label = country)) +
    geom_point(color = "red") +
    geom_text_repel() +
    geom_abline(lty = 2) +
    xlab("Average of 2010 and 2015") +
    ylab("Difference between 2015 and 2010")

# see the code below the previous video for variable definitions

# smooth density plots - area under each curve adds to 1
gapminder %>%
    filter(year == past_year & country %in% country_list) %>%
    mutate(group = ifelse(region %in% west, "West", "Developing")) %>% group_by(group) %>%
    summarize(n = n()) %>% knitr::kable()

# smooth density plots - variable counts on y-axis
p <- gapminder %>%
    filter(year == past_year & country %in% country_list) %>%
    mutate(group = ifelse(region %in% west, "West", "Developing")) %>%
    ggplot(aes(dollars_per_day, y = ..count.., fill = group)) +
    scale_x_continuous(trans = "log2")
p + geom_density(alpha = 0.2, bw = 0.75) + facet_grid(year ~ .)


|group      |  n|
|:----------|--:|
|Developing | 87|
|West       | 21|

# add group as a factor, grouping regions
gapminder <- gapminder %>%
    mutate(group = case_when(
            .$region %in% west ~ "West",
            .$region %in% c("Eastern Asia", "South-Eastern Asia") ~ "East Asia",
            .$region %in% c("Caribbean", "Central America", "South America") ~ "Latin America",
            .$continent == "Africa" & .$region != "Northern Africa" ~ "Sub-Saharan Africa",
            TRUE ~ "Others"))

# reorder factor levels
gapminder <- gapminder %>%
    mutate(group = factor(group, levels = c("Others", "Latin America", "East Asia", "Sub-Saharan Africa", "West")))
# note you must redefine p with the new gapminder object first
p <- gapminder %>%
  filter(year %in% c(past_year, present_year) & country %in% country_list) %>%
    ggplot(aes(dollars_per_day, fill = group)) +
    scale_x_continuous(trans = "log2")

# stacked density plot
p + geom_density(alpha = 0.2, bw = 0.75, position = "stack") +
    facet_grid(year ~ .)

# weighted stacked density plot
gapminder %>%
    filter(year %in% c(past_year, present_year) & country %in% country_list) %>%
    group_by(year) %>%
    mutate(weight = population/sum(population*2)) %>%
    ungroup() %>%
    ggplot(aes(dollars_per_day, fill = group, weight = weight)) +
    scale_x_continuous(trans = "log2") +
    geom_density(alpha = 0.2, bw = 0.75, position = "stack") + facet_grid(year ~ .)
Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"Warning message in density.default(x, weights = w, bw = bw, adjust = adjust, kernel = kernel, :
"sum(weights) != 1  -- will not get true density"

The plot clearly shows how an improvement in life expectancy followed the drops in fertility rates. In 1960, Germans lived 15 years longer than South Koreans, although by 2010 the gap is completely closed. It exemplifies the improvement that many non-western countries have achieved in the last 40 years.

]]> case-study Visualization EDA https://ashudva.github.io/blog/posts/2020-10-10-Gapminder/ Sun, 21 Sep 2025 21:36:10 GMT