Fashion Matching Recommendation — Portfolio Project by Muhammad Daffa Ashdaqfillah

A multimodal fashion recommendation system using a fine-tuned CLIP model to provide visual compatibility matching and smart product tagging.

Fashion Matching Recommendation — Portfolio Project by Muhammad Daffa Ashdaqfillah

Published on June 15, 2026

Project Details

Image
Image
Image

Description

This project is a multimodal fashion recommendation system designed to improve product discovery in fashion e-commerce. It uses visual compatibility matching to recommend fashion items that style well together and automatically generates descriptive tags from product images and text metadata. It includes a custom fine-tuned CLIP backend and an iOS application for the user interface.

Problem

Traditional e-commerce platforms rely heavily on keyword-based metadata (e.g., category, color) for search and recommendations. However, fashion purchasing decisions often depend on visual and contextual attributes like style compatibility, outfit matching, color harmony, and occasion. This makes it difficult for users to discover complementary products that visually work well together using conventional filters alone.

Role

As a Machine Learning Engineer, I designed the system architecture, prepared the dataset, and fine-tuned a LAION CLIP base model to better capture fashion-specific relationships. I also integrated the backend services to serve model inferences to the frontend iOS application.

Solution

The solution leverages a LAION CLIP base model fine-tuned on the FashionProductImagesDataset using contrastive loss. The backend processes product images and text to generate structured tags (e.g., category, color, style, occasion) and provides visually compatible recommendations (e.g., matching a white oversized t-shirt and black sneakers with blue straight jeans). The iOS app allows users to select an item from a catalog or snap a photo and input a style prompt (like "outfit for gym workout") to receive real-time recommendations.

Key Features

  • Smart Product Tagging: Automatically generates structured tags such as category, style, fit, and occasion from visual and textual inputs.
  • Visual Compatibility Matching: Recommends complementary fashion items that form cohesive outfits, moving beyond basic similarity search.
  • Multimodal Inputs: Supports searching for outfits using both catalog item selection and user-uploaded images combined with text prompts.
  • iOS Client: A native SwiftUI application for interacting with the backend recommendation engine.

Implementation

  • Model Fine-Tuning: Fine-tuned the laion/CLIP-ViT-B-32-laion2B-s34B-b79K model using contrastive learning to better map fashion semantics compared to the base model.
  • Zero-Shot Tagging: Utilized the CLIP model's zero-shot capabilities for categorizing and labeling fashion items without explicit prior training on every label.
  • Data Preparation: Processed a dataset of 44,441 product images and associated metadata, handling text cleaning and caption building.
  • Backend Service: Implemented an API providing endpoints for health checks, random product retrieval, catalog recommendations, and image-based recommendations.