πΎ Input/Output Engineering: Breaking the Data Ingestion Bottleneck
In modern machine learning platforms, model training loops and feature transformation pipelines often spend more time waiting for I/O disk reads and memory deserialization than executing gradient calculations. While legacy text files like CSV remain human-readable, they lack schema enforcement, force costly ASCII-to-binary float parsing, and waste massive disk space. Modern data engineering relies on binary columnar storage formats. Leveraging Apache Parquet (optimized for compressed cold storage and analytical cloud queries), Feather / Arrow IPC (designed for blazing zero-copy inter-process RAM transfers), and SQL stream chunking via SQLAlchemy eliminates memory bottlenecks, reduces network egress transfer costs by up to 85%, and allows pipelines to ingest millions of records per second on commodity hardware.
ROW-ORIENTED TEXT (CSV / JSON):
[ Row 0: ID=1, Age=28, City="Pune", Balance=52000.50 ]
[ Row 1: ID=2, Age=45, City="Mumbai", Balance=120000.00 ]
[ Row 2: ID=3, Age=33, City="Pune", Balance=85000.75 ]
Disadvantages:
* Inefficient queries: Must parse every preceding character to read 'Balance'
* Expensive ASCII-to-float conversions; zero data type preservation
* Incompressible across mixed categorical/numeric row strings
COLUMNAR BINARY STORAGE (Apache Parquet / Feather / Arrow):
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β METADATA FOOTER: Schema, Column Offsets, Row Group Stats (Min/Max/Nulls) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β COLUMN CHUNK 1 (ID): [ 1, 2, 3 ] (Bit-Packed Int32) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β COLUMN CHUNK 2 (Age): [ 28, 45, 33 ] (Run-Length Encoded) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β COLUMN CHUNK 3 (City): [ Dictionary: 0="Pune", 1="Mumbai" ] -> [ 0, 1, 0]β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β COLUMN CHUNK 4 (Balance): [ 52000.50, 120000.00, 85000.75 ] (IEEE Float64) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Advantages:
* Predicate Pushdown: Skips entire row groups if target not in Min/Max range
* Projection Pruning: Reads strictly columns requested; ignores the rest
* Zero-Copy Ingestion: Feather buffers map directly into RAM without parsing
import os
import time
import tempfile
import pandas as pd
import numpy as np
# Generate 1,000,000 financial transactional records (~80MB in memory)
N = 1000000
np.random.seed(42)
df_benchmark = pd.DataFrame({
"transaction_id": [f"TX-{i:08d}" for i in range(N)],
"user_id": np.random.randint(1000, 50000, size=N, dtype=np.int32),
"amount_inr": np.random.exponential(scale=2500.0, size=N).astype(np.float32),
"category": np.random.choice(["Retail", "Food", "Travel", "Utility", "Healthcare"], size=N),
"is_fraud": np.random.choice([False, True], size=N, p=[0.98, 0.02])
})
temp_dir = tempfile.gettempdir()
csv_path = os.path.join(temp_dir, "transactions.csv")
parquet_path = os.path.join(temp_dir, "transactions.parquet")
feather_path = os.path.join(temp_dir, "transactions.feather")
def benchmark_format(write_fn, read_fn, file_path: str, label: str):
# Benchmark Write
t0 = time.perf_counter()
write_fn(file_path)
t_write = time.perf_counter() - t0
file_size_mb = os.path.getsize(file_path) / (1024 * 1024)
# Benchmark Read
t0 = time.perf_counter()
df_read = read_fn(file_path)
t_read = time.perf_counter() - t0
return {"Format": label, "Size (MB)": file_size_mb, "Write (s)": t_write, "Read (s)": t_read}
results = []
results.append(benchmark_format(
lambda p: df_benchmark.to_csv(p, index=False),
lambda p: pd.read_csv(p),
csv_path, "Standard CSV"
))
results.append(benchmark_format(
lambda p: df_benchmark.to_parquet(p, engine="pyarrow", compression="snappy"),
lambda p: pd.read_parquet(p, engine="pyarrow"),
parquet_path, "Apache Parquet"
))
results.append(benchmark_format(
lambda p: df_benchmark.to_feather(p),
lambda p: pd.read_feather(p),
feather_path, "Feather (Arrow)"
))
# Clean up benchmark files
for p in [csv_path, parquet_path, feather_path]:
if os.path.exists(p): os.remove(p)
df_summary = pd.DataFrame(results)
print("=" * 65)
print(f"STORAGE FORMAT INGESTION BENCHMARK ({N:,} Rows)")
print("=" * 65)
print(df_summary.to_string(index=False))
import os
import time
import tempfile
import pandas as pd
import numpy as np
# Create 500,000 rows with 20 columns
N_ROWS = 500000
data = {f"metric_{i:02d}": np.random.randn(N_ROWS) for i in range(20)}
df_wide = pd.DataFrame(data)
temp_dir = tempfile.gettempdir()
csv_file = os.path.join(temp_dir, "wide_data.csv")
parquet_file = os.path.join(temp_dir, "wide_data.parquet")
df_wide.to_csv(csv_file, index=False)
df_wide.to_parquet(parquet_file, engine="pyarrow")
TARGET_COLS = ["metric_00", "metric_19"] # Only first and last column needed
# 1. Read Subset from CSV via usecols (Still parses full file stream internally)
t0 = time.perf_counter()
df_csv_subset = pd.read_csv(csv_file, usecols=TARGET_COLS)
t_csv = time.perf_counter() - t0
# 2. Read Subset from Parquet via columns (Loads strictly target chunk offsets)
t0 = time.perf_counter()
df_parquet_subset = pd.read_parquet(parquet_file, columns=TARGET_COLS, engine="pyarrow")
t_parquet = time.perf_counter() - t0
for p in [csv_file, parquet_file]:
if os.path.exists(p): os.remove(p)
print("=" * 65)
print("PROJECTION PRUNING: READING 2 OF 20 COLUMNS (500,000 Rows)")
print("=" * 65)
print(f"CSV Ingestion Duration : {t_csv * 1000:6.2f} ms")
print(f"Parquet Ingestion Duration : {t_parquet * 1000:6.2f} ms")
print(f"Projection Read Speedup : {t_csv / t_parquet:6.1f}x Faster!")
print(f"Extracted DataFrame Shape : {df_parquet_subset.shape}")
filters parameter, pyarrow evaluates conditions against these metadata statistics, skipping non-matching blocks entirely.
import os
import time
import tempfile
import pandas as pd
import numpy as np
# Partitioned sales records across calendar years
N_RECORDS = 600000
years = np.repeat([2024, 2025, 2026], N_RECORDS // 3)
df_sales = pd.DataFrame({
"sale_id": np.arange(N_RECORDS),
"sale_year": years,
"revenue": np.random.uniform(100.0, 5000.0, size=N_RECORDS)
})
temp_dir = tempfile.gettempdir()
partitioned_parquet = os.path.join(temp_dir, "sales_partitioned.parquet")
# Save with smaller row groups to allow granular skipping
df_sales.to_parquet(partitioned_parquet, engine="pyarrow", row_group_size=50000)
# 1. Full Ingestion + In-Memory Filtering (Anti-Pattern: Loads all 600,000 rows into RAM)
t0 = time.perf_counter()
df_full = pd.read_parquet(partitioned_parquet, engine="pyarrow")
df_filtered_mem = df_full[df_full["sale_year"] == 2026]
t_mem = time.perf_counter() - t0
# 2. Predicate Pushdown Filtering (PyArrow reads ONLY row groups containing 2026)
t0 = time.perf_counter()
df_pushdown = pd.read_parquet(
partitioned_parquet,
filters=[("sale_year", "==", 2026)],
engine="pyarrow"
)
t_pushdown = time.perf_counter() - t0
if os.path.exists(partitioned_parquet): os.remove(partitioned_parquet)
print("=" * 65)
print("PREDICATE PUSHDOWN VS. IN-MEMORY FILTERING")
print("=" * 65)
print(f"Full Ingestion + Filter Time : {t_mem * 1000:6.2f} ms (Loaded 600,000 rows)")
print(f"Predicate Pushdown Read Time : {t_pushdown * 1000:6.2f} ms (Loaded only target groups)")
print(f"Filter Throughput Speedup : {t_mem / t_pushdown:6.1f}x Faster!")
print(f"Filtered Result Parity Check : {len(df_filtered_mem) == len(df_pushdown)} ({len(df_pushdown):,} rows)")
pd.read_csv() triggers an Out-Of-Memory (OOM) crash. Process large files out-of-core using chunksize iterators to compute summary aggregations in $O(1)$ memory.
import os
import tempfile
import pandas as pd
import numpy as np
temp_dir = tempfile.gettempdir()
large_csv_path = os.path.join(temp_dir, "massive_telemetry_feed.csv")
# Generate a 200,000 row sample file
N_SIMULATED = 200000
df_temp = pd.DataFrame({
"sensor_id": np.random.choice(["SENSOR_A", "SENSOR_B", "SENSOR_C"], size=N_SIMULATED),
"voltage": np.random.normal(loc=230.0, scale=4.5, size=N_SIMULATED),
"status": np.random.choice(["OK", "ALERT"], size=N_SIMULATED, p=[0.95, 0.05])
})
df_temp.to_csv(large_csv_path, index=False)
del df_temp
# Process file in 50,000 row chunks to maintain constant low memory usage
CHUNK_SIZE = 50000
total_alerts = 0
running_voltage_sum = 0.0
total_records = 0
print("=" * 65)
print(f"OUT-OF-CORE CHUNKED STREAMING PIPELINE (Chunk Size: {CHUNK_SIZE:,})")
print("=" * 65)
# pd.read_csv with chunksize returns a TextFileReader generator
chunk_iterator = pd.read_csv(large_csv_path, chunksize=CHUNK_SIZE)
for chunk_num, chunk in enumerate(chunk_iterator, start=1):
total_records += len(chunk)
total_alerts += (chunk["status"] == "ALERT").sum()
running_voltage_sum += chunk["voltage"].sum()
print(f" Processed Chunk {chunk_num:2d} -> Loaded {len(chunk):,} rows | Current Alerts: {total_alerts:,}")
mean_voltage = running_voltage_sum / total_records
if os.path.exists(large_csv_path): os.remove(large_csv_path)
print("-" * 65)
print(f"Total Stream Records Processed : {total_records:,}")
print(f"Total Anomaly Alerts Found : {total_alerts:,} ({(total_alerts/total_records)*100:.2f}%)")
print(f"Global Streaming Mean Voltage : {mean_voltage:.2f} V")
pd.read_sql_table() buffers entire result sets in memory. Using SQLAlchemy connection streaming and the chunksize parameter streams records directly from relational engines.
import sqlite3
import pandas as pd
import numpy as np
from sqlalchemy import create_engine, text
# 1. Setup in-memory SQLite database populated with transactional data
db_engine = create_engine("sqlite:///:memory:")
# Populate mock database table
N_ACCOUNTS = 100000
df_accounts = pd.DataFrame({
"account_number": [f"ACC-{i:06d}" for i in range(N_ACCOUNTS)],
"branch_code": np.random.choice(["PUN-01", "MUM-04", "BLR-02"], size=N_ACCOUNTS),
"balance_inr": np.random.uniform(5000.0, 500000.0, size=N_ACCOUNTS).round(2),
"kyc_verified": np.random.choice([1, 0], size=N_ACCOUNTS, p=[0.92, 0.08])
})
df_accounts.to_sql("customer_accounts", con=db_engine, index=False, if_exists="replace")
# 2. Execute Chunked Stream Query with Parameterized Query
QUERY = text("""
SELECT branch_code, balance_inr, kyc_verified
FROM customer_accounts
WHERE balance_inr > :min_balance
""")
# Stream in 25,000 row chunks
SQL_CHUNK_SIZE = 25000
total_qualified_balance = 0.0
total_verified_kyc = 0
print("=" * 65)
print("SQLALCHEMY BATCH STREAMING QUERY EXECUTION")
print("=" * 65)
with db_engine.connect() as conn:
sql_chunks = pd.read_sql_query(
QUERY,
con=conn,
params={"min_balance": 100000.0},
chunksize=SQL_CHUNK_SIZE
)
for batch_idx, batch_df in enumerate(sql_chunks, start=1):
total_qualified_balance += batch_df["balance_inr"].sum()
total_verified_kyc += (batch_df["kyc_verified"] == 1).sum()
print(f" Streamed Batch {batch_idx:2d} -> {len(batch_df):,} Rows | KYC Verified Count: {total_verified_kyc:,}")
print("-" * 65)
print(f"Total High-Balance Accounts Analyzed : βΉ {total_qualified_balance:,.2f}")
print(f"Total KYC-Verified Customers : {total_verified_kyc:,}")
import time
import pyarrow as pa
import pandas as pd
import numpy as np
# Create PyArrow Columnar Arrays
N_ITEMS = 1000000
arrow_id = pa.array([f"REC-{i:07d}" for i in range(N_ITEMS)])
arrow_metric = pa.array(np.random.randn(N_ITEMS).astype(np.float64))
arrow_status = pa.array(np.random.choice([True, False], size=N_ITEMS))
# Construct a native PyArrow Table
arrow_table = pa.Table.from_arrays(
[arrow_id, arrow_metric, arrow_status],
names=["record_id", "metric_val", "is_active"]
)
# 1. Zero-Copy Interop: Convert Arrow Table to Pandas with PyArrow-backed dtypes
t0 = time.perf_counter()
df_arrow_backed = arrow_table.to_pandas(types_mapper=pd.ArrowDtype)
t_convert = time.perf_counter() - t0
print("=" * 65)
print("PYARROW TABLE TO PANDAS ZERO-COPY CONVERSION")
print("=" * 65)
print(f"PyArrow Table Record Count : {arrow_table.num_rows:,} rows across {arrow_table.num_columns} columns")
print(f"PyArrow In-Memory Byte Size : {arrow_table.nbytes / (1024 * 1024):.2f} MB")
print(f"Conversion Execution Time : {t_convert * 1000:6.2f} ms (Near-instantaneous pointer transfer)")
print("-" * 65)
print("Pandas Arrow-Backed Column Dtypes:")
print(df_arrow_backed.dtypes)
print("-" * 65)
print(f"Sample First Row Inspection :\n{df_arrow_backed.iloc[0].to_dict()}")
.jsonl) stores one independent JSON object per line, enabling streaming reads using lines=True.
import os
import tempfile
import pandas as pd
temp_dir = tempfile.gettempdir()
jsonl_file_path = os.path.join(temp_dir, "web_access_logs.jsonl")
# Generate line-delimited JSON stream
mock_logs = [
'{"timestamp": "2026-10-30T10:00:01", "ip": "192.168.1.1", "endpoint": "/api/v1/auth", "code": 200}',
'{"timestamp": "2026-10-30T10:00:03", "ip": "10.0.0.15", "endpoint": "/api/v1/checkout", "code": 500}',
'{"timestamp": "2026-10-30T10:00:05", "ip": "172.16.0.4", "endpoint": "/api/v1/catalog", "code": 200}',
'{"timestamp": "2026-10-30T10:00:08", "ip": "192.168.1.1", "endpoint": "/api/v1/payment", "code": 403}'
]
with open(jsonl_file_path, "w") as f:
for line in mock_logs:
f.write(line + "\n")
# Stream JSONL in chunks of 2 lines
log_chunks = pd.read_json(jsonl_file_path, lines=True, chunksize=2)
print("=" * 65)
print("LINE-DELIMITED JSON (JSONL) STREAM INGESTION")
print("=" * 65)
total_errors = 0
for batch_idx, batch_df in enumerate(log_chunks, start=1):
errors = batch_df[batch_df["code"] >= 400]
total_errors += len(errors)
print(f" Chunk {batch_idx}: Read {len(batch_df)} lines -> Error Codes Found: {list(errors['code'].values)}")
if os.path.exists(jsonl_file_path): os.remove(jsonl_file_path)
print("-" * 65)
print(f"Total Ingested Log Failures (HTTP >= 400): {total_errors}")
scan_parquet() builds an execution graph that defers reading and combines filter, projection, and aggregation steps before touching disk.
import os
import time
import tempfile
import polars as pl
import numpy as np
temp_dir = tempfile.gettempdir()
polars_parquet_path = os.path.join(temp_dir, "large_sensor_dataset.parquet")
# Create 500,000 sensor observations
N_POINTS = 500000
df_raw = pl.DataFrame({
"device_id": np.random.choice(["DEV_NORTH", "DEV_SOUTH", "DEV_EAST"], size=N_POINTS),
"temperature": np.random.normal(25.0, 3.0, size=N_POINTS),
"humidity": np.random.uniform(40.0, 90.0, size=N_POINTS)
})
df_raw.write_parquet(polars_parquet_path)
del df_raw
# Execute a Lazy Scan: Builds a query plan, delays loading until .collect()
t0 = time.perf_counter()
lazy_query = (
pl.scan_parquet(polars_parquet_path)
.filter(pl.col("device_id") == "DEV_NORTH")
.select(["device_id", "temperature"])
.group_by("device_id")
.agg([
pl.col("temperature").mean().alias("avg_temp"),
pl.col("temperature").max().alias("max_temp")
])
)
# Execute optimized query plan
result_df = lazy_query.collect()
t_lazy = time.perf_counter() - t0
if os.path.exists(polars_parquet_path): os.remove(polars_parquet_path)
print("=" * 65)
print("POLARS LAZY SCAN QUERY EXECUTION")
print("=" * 65)
print(f"Lazy Evaluation & Query Time : {t_lazy * 1000:6.2f} ms")
print("-" * 65)
print("Aggregated Query Result:\n", result_df)
DATA INGESTION FORMATS ARCHITECTURAL MATRIX:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β Ingestion Format β Typical Compression Ratio β Read/Write Throughputβ Recommended Production Use Case β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Comma-Separated Values (CSV) β 1x (Uncompressed Baseline) β Slow (ASCII Parsing) β Small ad-hoc exports (< 50 MB) β
β Apache Parquet (Snappy/ZSTD) β 4x to 8x vs. CSV β Fast Columnar Read β Cloud Data Lakes (S3/GCS), ML DB β
β Feather / Arrow IPC β 2.5x to 4x vs. CSV β Ultra-Fast Zero-Copy β Local caching, IPC across models β
β SQLite / Relational SQL β Variable (B-Tree Indexing) β Moderate Streaming β Transactional data extraction β
β Line-Delimited JSON (JSONL) β 1x to 2x vs. CSV β Streamable Parsing β Unstructured event logs, NLP API β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ€
β INGESTION BEST PRACTICES: β
β 1. Never store training datasets in CSV for production ML; use Parquet to preserve schemas and speed up reads. β
β 2. Always use `columns=[...]` when reading Parquet files to trigger projection pruning and save bandwidth. β
β 3. Apply `filters=[...]` in `pd.read_parquet` to push down filtering predicates to file footer metadata. β
β 4. When datasets exceed physical RAM, use `chunksize` iterators or lazy query engines (Polars / DuckDB). β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ