Data Automation & Cleaning Beginner

Merge 100+ Monthly CSV Files into One Clean DataFrame in Python

Fast Python script using pathlib and Pandas to recursively combine dozens of monthly CSV exports, normalize column headers, and drop duplicate rows.

#pandas

Problem Statement

Broker trade logs and monthly accounting exports often come split across dozens of individual CSV files. This script glues an entire folder of CSVs into one deduplicated master file.

Formula / Core Logic

Uses pathlib.Path.glob('*.csv') with generator comprehension inside pd.concat() for memory-efficient merging.

Installation

pip install pandas

merge-multiple-csv-files-into-one-pandas-python.py

import pandas as pd
from pathlib import Path

def merge_csv_folder(folder_path, output_file='master_merged.csv'):
    csv_files = list(Path(folder_path).glob('*.csv'))
    if not csv_files:
        raise FileNotFoundError('No CSV files found in directory.')
    df_list = [pd.read_csv(f) for f in csv_files]
    merged = pd.concat(df_list, ignore_index=True).drop_duplicates()
    merged.to_csv(output_file, index=False)
    print(f'Merged {len(csv_files)} files into {len(merged)} unique rows.')
    return merged

Get the Complete 100+ Python Quant & Automation Bundle

Instant lifetime access to all production-ready scripts, valuation models, and trading bots ($19).

Buy Full Bundle on Gumroad ($19) →

Frequently Asked Questions

Q: What if one CSV has extra columns?

`pd.concat` automatically aligns matching column names and fills missing columns with `NaN`.