Data Automation & Cleaning
Intermediate
Fuzzy Match & Deduplicate Messy Vendor/Customer Names in Accounting CSVs (Python)
Detect duplicate vendor entries like 'Reliance Ind Ltd' and 'Reliance Industries Limited' in ERP master CSVs using token similarity scoring in Python.
#pandas
Problem Statement
ERP systems accumulate duplicate vendor accounts when clerks type 'ABC Pvt Ltd' vs 'ABC Private Limited', splitting ledger balances and causing duplicate payments.
Formula / Core Logic
Strips legal suffixes (PVT, LTD, LIMITED, INC, LLP), normalizes whitespace, and computes SequenceMatcher similarity ratio >= 0.88.
Installation
pip install pandas
fuzzy-vendor-name-deduplication-master-data-python.py
import pandas as pd
import re
from difflib import SequenceMatcher
def normalize_entity_name(name: str) -> str:
s = re.sub(r'[^A-Z0-9 ]', ' ', str(name).upper())
s = re.sub(r'\b(PVT|PRIVATE|LTD|LIMITED|INC|LLP|CO|CORP)\b', '', s)
return ' '.join(s.split())
def find_duplicate_vendors(vendor_list, threshold=0.88):
cleaned = [(v, normalize_entity_name(v)) for v in vendor_list]
matches = []
for i in range(len(cleaned)):
for j in range(i + 1, len(cleaned)):
sim = SequenceMatcher(None, cleaned[i][1], cleaned[j][1]).ratio()
if sim >= threshold:
matches.append({'Vendor_A': cleaned[i][0], 'Vendor_B': cleaned[j][0], 'Similarity': round(sim, 3)})
return pd.DataFrame(matches)
Get the Complete 100+ Python Quant & Automation Bundle
Instant lifetime access to all production-ready scripts, valuation models, and trading bots ($19).
Buy Full Bundle on Gumroad ($19) →Frequently Asked Questions
Q: Why strip 'PRIVATE LIMITED' before running SequenceMatcher?
Two completely different companies sharing 'PRIVATE LIMITED' would otherwise score an artificially high character similarity.