Improving Dog Emotion Classification in Shelter Environments
A study by Gauri Shetty
With Help from Ayush Pandit
July 31, 2024
Abstract:
In 2023, approximately 6.5 million cats and dogs entered rescue shelters across America.
Due to the overwhelming number, many animals were neglected and not properly cared for,
leading to around 690,000 being euthanized (Shelter Animals Count, 2024). Unfortunately, many
dogs suffer in silence due to the stress and anxiety of changing environments. Animal welfare is
declining as humans often fail to understand how behavior changes correspond to the dog’s
emotions. We can improve animal welfare by classifying shelter animals' emotions based on their
behavior. This paper attempts to use an image classifier model to detect 4 emotions in dogs:
happy, relaxed, sad, and angry. This pre-trained model could be deployed with cameras or
sensors with machine learning to detect specific animal features and connect them to the emotion
the animal is feeling. An alert would then be sent to a shelter worker when an animal is
experiencing negative emotions (sad and angry). This model can identify the four typical dog
emotions with an accuracy of 67.65%. This approach can guarantee owners and shelter staff a
better understanding of their pets' well-being.
Introduction:
When animals are placed in unfamiliar and crowded surroundings like shelters, they often
become scared and anxious due to the new environment (Taking the Fear Out of Sheltering,
n.d.). Unfortunately, this results in a decline in their overall welfare, as they are more susceptible
to sicknesses. It is challenging for shelter staff to check up on every animal they have, as shelters
are often both overcrowded and understaffed. It is crucial that we can quickly recognize these
negative emotions in animals and provide appropriate care. However, interpreting animal
emotions, especially in shelter settings where they may exhibit different behaviors than usual,
can be complicated. My research aims to address this challenge by using the ResNet-50 machine
learning model to classify animal emotions (happy, sad, relaxed, angry) based on the behavior
captured in an image of the animal. By developing an accurate dog emotion classifier system,
shelter staff can quickly identify animals who are in distress and provide them with the necessary
care to make them happy and healthy again.
Background:
Previous approaches to detect animal emotions consist of wearable devices to track heart
rate and movement, along with systematic approaches to identify behaviors. One approach
conducted by a group of researchers in 2019 used a collar equipped with machine-learning
algorithms and sensors to detect and analyze dog behavior. This approach had very high
accuracy in detecting both activity and emotion in dogs with 96.58% accuracy in activity
detection and 92.87% in emotion detection (Aich et al., 2019). A limitation of this study is that it
focused on tail wagging as its main source of indicating the emotion exhibited by a dog.
However, this approach doesn’t capture the full range of emotions exhibited by dogs, as tail
wagging is not the most accurate way to provide insight into a dog’s emotional state (Subbiah,
2023).
Another study conducted in 2019 used FBAs (functional behavior assessments) to
understand animal behavior. Researchers observed the behavior of dogs and used a systematic
process to identify the specific factors that identify the function of their behavior (DipCbst,
2023). This approach is simple and inexpensive, but it is limited in its ability to understand
animal emotions, as it only focuses on subjective interpretations.
These approaches offered insights into how analyzing dog emotions specific to each
breed using facial features is the most effective and accurate way of classifying their emotions.
Dataset:
The dataset we found was a “Dog Emotions Prediction” image set found on Kaggle. The
original dataset consisted of angry, happy, sad, and relaxed images, all sourced straight from the
Google Search Engine. The exact numbers are shown in the table below.
Before Data Augmentation Image Count by Emotion
Angry Happy Sad Relaxed
2256 4784 4532 4349
The first thing that needed to be done was data cleaning. A few of the images from the
original dataset were not dogs, and therefore should not be included in the dataset as they have
the potential to limit accuracy. We finetuned an existing ResNet-50 ML model to detect when
there were non-dogs in the image based on a confidence threshold and then remove them from
the dataset. After the dataset consisted of only easily identifiable images of dogs, we moved on
to oversampling and image augmentations.
The dataset was originally unbalanced, so oversampling had to be done to represent each
emotion equally. Each emotion image set consisted of ~4k images after oversampling was
completed. After this, 5% of the images from each emotion were transferred to the testing
dataset, and another 5% was transferred to the validation dataset.
To ensure the highest accuracy from the model, all images were augmented (blurred,
rotated, zoomed in or out, mirrored) at least once. The final number of images in the training
dataset was ~8k per emotion. The table below shows the final number of images in the training,
testing, and validation datasets.
After Data Augmentation Image Count by Emotion
Angry Happy Sad Relaxed
Training (90%) 8483 8874 8832 8730
Testing (5%) 113 113 113 113
Validation (5%) 92 92 92 92
The images in this dataset (like the augmented one shown to the left)
contain visual information that can help the model classify a dog’s
emotion based on visual cues such as facial expressions and body
language. The model needed to be trained to notice the subtle
differences in the dog's expressions to accurately portray what
emotion it is feeling.
Methodology/Models:
As stated before, after finishing data preprocessing, the data was split into three sections
(training (90%), testing (5%) and validation (5%)). The training data was used to experiment
with different pre-trained models of image classifiers and transfer learning methods. The
validation data was used to tune the hyperparameters of the model to improve accuracy. Finally,
the testing data was used when I was completely confident with the model and its accuracy.
A Convolutional Neural Network (CNN) was used for emotion detection. Specifically,
the ResNet-50 was pre-trained and tweaked to fit this specific task. We defined the CNN model
architecture by replacing the final fully connected layer with a new linear layer with 4 outputs,
corresponding to the 4 emotion classes (happy, sad, relaxed, and angry). We used the
cross-entropy loss function, which measures the difference between the predicted and actual
probability distributions of the emotion classes, and the Adam optimizer with a learning rate of
0.0001 to train the model. This approach allowed us to effectively classify emotions in images of
dogs, achieving accurate results for this dataset. The model was trained for 10 epochs and the
best-performing model was tested on the testing data.
Results:
The results indicated that the model was reasonably accurate, with a training accuracy of
72.21%, a testing accuracy of 67.65%, and a validation accuracy of 65.92%.
Overall Accuracy
Training Accuracy Testing Accuracy Validation Accuracy
0.7221 → 72.21% 0.6765 → 67.65% 0.6592 → 65.92%
However, these results indicate some overfitting because the training accuracy is higher
than the testing and validation accuracies. Factors that may have influenced the results include
the quality of the dataset, which was sourced from the internet, and the difficulties in detecting
subtle differences in the expressions of these dogs. The Res-Net 50 model might not fully be able
to capture the different nuances in dog emotion.
The limitations of this study are the lack of diversity in the dataset images as they were
all sourced from the internet. The very moderate accuracy may not allow the model to prove
applicable in settings like shelters because higher accuracy is needed. Future work should be
focused on expanding the dataset with more varied images to improve the model's accuracy and
prove more effective in the real world.
Despite these shortcomings, the model proves to provide a valuable step in enhancing
animal welfare in shelter environments.
Conclusion:
In this study, we addressed the challenge of detecting and classifying dog emotions based
on their behavior, to improve animal welfare in shelters. Using a Convolutional Neural Network
(CNN) with the ResNet-50 architecture, we developed a model that can accurately classify four
emotions (angry, happy, relaxed, sad) based on images of dogs' faces.
This approach utilizes pre-trained models to efficiently classify a dog’s emotions, making
it possible to implement cameras in shelters in the real world. When negative emotions in dogs
are detected, shelter workers are quickly notified and can provide the necessary care to improve
their well-being.
The dataset used for training was sourced from Kaggle and underwent data cleaning,
oversampling, and augmentation to ensure a balanced representation of emotions and improve
model accuracy. The trained model achieved an accuracy of 67.65% on the test dataset,
demonstrating its effectiveness in classifying dog emotions.
Overall, this research contributes to the field of animal welfare by providing a practical
solution to detect and address negative emotions in shelter dogs. Future work could involve
training the model to detect more emotions, exploring different pre-trained models, or collecting
more diverse datasets to improve emotion detection accuracy. The most important next step
would be integrating the model into shelter environments and evaluating its impact on animal
welfare in real time.
Acknowledgements:
Thank you to Ayush Pandit for guiding me through this process, giving me important resources to
aid me in my research, and reviewing my final submission.
References:
Aich, S., Chakraborty, S., Sim, J., Jang, D., & Kim, H. (2019). The Design of an Automated
System for the Analysis of the Activity and Emotional Patterns of Dogs with Wearable
Sensors Using Machine Learning. Applied Sciences, 9(22), 4938.
https://doi.org/10.3390/app9224938
Cuglovici, D. A. (2023). What can associations between emotional states and management-based
measures teach us about shelter dogs’ welfare? Journal of Veterinary Behavior, 69–70,
32–38. https://doi.org/10.1016/j.jveb.2023.11.001
DipCbst, A. F. C. (2023, July 9). Using functional analysis for dog behavior problems.
PetHelpful.
https://pethelpful.com/dogs/Using-Functional-Analysis-For-Dog-Behavior-Problems
Dog emotions prediction. (2022, October 3). Kaggle.
https://www.kaggle.com/datasets/devzohaib/dog-emotions-prediction
Shelter Animals Count. (2024, February 15). 2023 Statistics - Shelter animals count.
https://www.shelteranimalscount.org/stats
Subbiah, K. (2023, October 24). Dog Tail Wags and Movements: A guide to Canine
communication. Supertails.
https://supertails.com/blogs/expert-led-blogs/dog-tail-wags-and-movements-a-guide-to-c
anine-communication
Taking the fear out of sheltering. (n.d.). HumanePro.
https://humanepro.org/magazine/articles/taking-fear-out-sheltering