Face Photograph Memorability Score (A project for Machine Learning Course - Department of CS at Temple University)

Project Overview

In the digital era, faces significantly influence marketing, social interactions, and security. Understanding which facial features contribute to memorability can enhance brand recall, social media engagement, and visual content effectiveness. This project introduces a novel hybrid approach combining visual and textual data to predict face memorability more accurately than previous models. Traditional image memorability models often fall short when applied specifically to facial images, leaving a critical gap in our understanding of face memorability. This project addresses this gap by developing a model that captures complex facial attributes through both visual and textual descriptions.

Methodology

Dataset: Over 2,000 facial images annotated with memorability scores (http://www.wilmabainbridge.com/facememorability2.html).

Preprocessing: Face detection and cropping via MTCNN, resizing to 224x224, text tokenization.

Hybrid Model Architecture:

- Visual Features: SE-ResNet model pretrained on VGGFace dataset.

- Textual Features: BERT embeddings of detailed facial descriptions generated by LLMs (llama3.2).

- Cross-Attention Mechanism: A multi-head attention layer integrates textual queries with visual keys/values to identify features that enhance memorability prediction.

Results

The proposed model outperformed existing state-of-the-art models in predicting face memorability, achieving a Spearman’s rank correlation of 0.678. Visualization of attention weights revealed specific features associated with higher or lower memorability:

- High Memorability: Unique attributes such as scars, goatees, and tan skin tones.

- Low Memorability: Common features like full beards or baldness