Data Automation & Cleaning Beginner

Compress & Query 10GB+ Tick Data CSVs using Parquet & PyArrow in Python

Shrink massive historical tick-data CSV files by 80% and speed up Pandas read times 15x by converting CSVs to Snappy-compressed Parquet in Python.

#pandas#pyarrow

Problem Statement

Loading multi-year 1-minute or tick-by-tick options CSVs takes minutes and crashes RAM. Converting to columnar Apache Parquet reduces file size by ~80% and loads in milliseconds.

Formula / Core Logic

Reads CSV in chunks or direct DataFrame, casts float64 to float32 where appropriate, and writes Snappy-compressed `.parquet`.

Installation

pip install pandas pyarrow

convert-large-csv-to-parquet-duckdb-python.py

import pandas as pd
import os

def csv_to_parquet(csv_path, parquet_path='compressed_data.parquet'):
    df = pd.read_csv(csv_path)
    df.to_parquet(parquet_path, engine='pyarrow', compression='snappy', index=False)
    orig_mb = os.path.getsize(csv_path) / (1024 * 1024)
    new_mb = os.path.getsize(parquet_path) / (1024 * 1024)
    print(f'Compressed {orig_mb:.2f} MB -> {new_mb:.2f} MB ({100 - (new_mb/orig_mb)*100:.1f}% saved)')
    return parquet_path

Get the Complete 100+ Python Quant & Automation Bundle

Instant lifetime access to all production-ready scripts, valuation models, and trading bots ($19).

Buy Full Bundle on Gumroad ($19) →

Frequently Asked Questions

Q: Can I load only specific columns from a Parquet file?

Yes! Use `pd.read_parquet('file.parquet', columns=['Date', 'Close'])` to read only needed columns without loading the full file into memory.