Data Automation & Cleaning Intermediate

Fuzzy Match & Deduplicate Messy Vendor/Customer Names in Accounting CSVs (Python)

Detect duplicate vendor entries like 'Reliance Ind Ltd' and 'Reliance Industries Limited' in ERP master CSVs using token similarity scoring in Python.

#pandas

Problem Statement

ERP systems accumulate duplicate vendor accounts when clerks type 'ABC Pvt Ltd' vs 'ABC Private Limited', splitting ledger balances and causing duplicate payments.

Formula / Core Logic

Strips legal suffixes (PVT, LTD, LIMITED, INC, LLP), normalizes whitespace, and computes SequenceMatcher similarity ratio >= 0.88.

Installation

pip install pandas
fuzzy-vendor-name-deduplication-master-data-python.py
import pandas as pd
import re
from difflib import SequenceMatcher

def normalize_entity_name(name: str) -> str:
    s = re.sub(r'[^A-Z0-9 ]', ' ', str(name).upper())
    s = re.sub(r'\b(PVT|PRIVATE|LTD|LIMITED|INC|LLP|CO|CORP)\b', '', s)
    return ' '.join(s.split())

def find_duplicate_vendors(vendor_list, threshold=0.88):
    cleaned = [(v, normalize_entity_name(v)) for v in vendor_list]
    matches = []
    for i in range(len(cleaned)):
        for j in range(i + 1, len(cleaned)):
            sim = SequenceMatcher(None, cleaned[i][1], cleaned[j][1]).ratio()
            if sim >= threshold:
                matches.append({'Vendor_A': cleaned[i][0], 'Vendor_B': cleaned[j][0], 'Similarity': round(sim, 3)})
    return pd.DataFrame(matches)

Get the Complete 100+ Python Quant & Automation Bundle

Instant lifetime access to all production-ready scripts, valuation models, and trading bots ($19).

Buy Full Bundle on Gumroad ($19) →

Frequently Asked Questions

Q: Why strip 'PRIVATE LIMITED' before running SequenceMatcher?

Two completely different companies sharing 'PRIVATE LIMITED' would otherwise score an artificially high character similarity.