✓ Test Progress
01 · API Access Logs ✓
02 · Kubernetes / Container ✓
03 · Application Logs ✓
04 · Auth / Security ✓
05 · Infrastructure / System ✓
06 · Streaming Pipeline ✓
07 · Operations / CI/CD ✓
08 · Cloud Provider ✓
09 · Network / Firewall ✓
10 · E-Commerce / Transactions ✓
The Storage Cost Problem — Solved
Cold Storage That Queries
Like Hot Storage.
Today, engineering teams keep logs on expensive hot/warm infrastructure for weeks — not because they query them daily, but because they might need to. Cold storage was never queryable. PFC-JSONL changes that: ship logs to cheap S3 on day one, query any 1-hour window in under 6 seconds. The warm tier becomes unnecessary.
| Storage Tier | Typical Tools | Cost / TB / month | Queryable? | Time-Range Query | Ideal Retention |
|---|---|---|---|---|---|
| Hot | Elasticsearch, Datadog, Loki, Splunk | $20 – $50 | ✓ Yes real-time | < 1 second | Last 0 – 3 days |
| Warm | Cheaper ES tier, S3 + custom index | $8 – $15 | ~ Partial | seconds – minutes | Days 3 – 30 (often kept too long) |
| Cold (traditional) | S3 Glacier, gzip/zstd archives | $2 – $4 | ✗ No full decompress needed | minutes – hours | Days 30+ (only for compliance) |
| PFC Cold pfc-jsonl + DuckDB |
S3 Standard / S3 IA | $2 – $4 cold price | ✓ Full SQL via DuckDB | 3 – 6 seconds per 1h window / 1 GB file | Day 1 onwards |
Traditional: 30-day hot retention
$9,000
per month · 10 TB/month · Elasticsearch pricing
With PFC: 3 days hot + rest PFC Cold
$1,030
$900 hot (3 days) + $130 PFC S3-IA (27 days)
Monthly savings — same query capability
~$8,000
≈ 89% cost reduction · no tooling changes · full SQL via DuckDB
What this means for your team:
Compress logs to PFC on day one. Store directly on S3.
When an incident surfaces three weeks later, run the same DuckDB query
your SREs already know — read_pfc_jsonl() — and get results in seconds.
No re-hydration. No waiting. No $40/TB Elasticsearch bill for data you check twice a year.
Compatible with Fluent Bit, Vector, Kafka, OpenTelemetry, Telegraf, Grafana,
and all five major time-series databases.
Fits into existing pipelines without infrastructure changes.
1 API Access Logs
Sources: nginx · Caddy · AWS ALB · Cloudflare · SaaS backends
|
File: 01_api_access_logs_1gb.jsonl · 1.00 GB · ~4.5M lines
PFC-JSONL Ratio
12.76%
of original size
vs gzip-9
28% smaller
gzip-9: 17.82%
vs zstd-3
33% smaller
zstd-3: 19.02%
Compress Speed
40.1 MB/s
3.8× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 12.76% | 130 MB | 40.1 MB/s | 25.5s | 45.7 MB/s | ✅ Time-range BIDX |
| gzip -9 | 17.82% | 183 MB | 10.5 MB/s | 99.3s | 151.1 MB/s | ❌ Full scan only |
| zstd -3 | 19.02% | 195 MB | 113.2 MB/s | 10.1s | 468.6 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2.2 GB RAM / ~380 MB per worker) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 32-block file): 5.3 s — 183,726 rows
— BIDX reads 1 of 32 blocks only. gzip full scan: ~99s. zstd full scan: ~10s. PFC is the only format with block-level random access.
API Access Logs: The highest-volume log type — and PFC handles it.
At 12.76% ratio, a 30-day archive of 1 GB/day costs just 3.8 GB on S3 instead of 5.4 GB with zstd-3.
When a customer reports a 429 or a spike in 500s from last Tuesday, DuckDB returns the relevant
hour in 5.3 seconds — without downloading or decompressing the full archive.
For SaaS teams running nginx, Caddy, or AWS ALB: this is the log type you produce the most of,
store the longest, and query the least — until something breaks.
2 Kubernetes / Container Logs
Sources: k8s pods · Docker · containerd · kubectl logs
|
File: 02_kubernetes_container_logs_1gb.jsonl · 1.00 GB · ~5.5M lines
PFC-JSONL Ratio
7.97%
of original size
vs gzip-9
33% smaller
gzip-9: 11.85%
vs zstd-3
43% smaller
zstd-3: 13.93%
Compress Speed
53.8 MB/s
3.6× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 7.97% | 81 MB | 53.8 MB/s | 19.0s | 49.9 MB/s | ✅ Time-range BIDX |
| gzip -9 | 11.85% | 122 MB | 16.2 MB/s | 71.8s | 201.1 MB/s | ❌ Full scan only |
| zstd -3 | 13.93% | 143 MB | 162.9 MB/s | 8.5s | 513.0 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 4.3 s — 175,837 rows
— BIDX skips almost all blocks. gzip/zstd require full decompression before any query.
Kubernetes Logs: 43% smaller than zstd-3 — the format Loki and Promtail use by default.
Repetitive pod states (Running, Pending, Failed), service names, and namespace strings make
Kubernetes logs exceptionally well-suited for PFC. At 7.97%, a 100-node cluster generating 50 GB/day
stores 4 GB compressed instead of 7 GB with zstd-3.
Pod crash investigations — "what was this container doing at 03:14?" — return in 4.3 seconds.
Move K8s logs to S3 IA on day one and cut your observability storage bill by up to 43%.
3 Application Logs
Sources: Spring Boot · Node.js (winston/bunyan) · Python (structlog) · Java (logback)
|
File: 03_application_logs_1gb.jsonl · 1.00 GB · ~5M lines
PFC-JSONL Ratio
10.56%
of original size
vs gzip-9
28% smaller
gzip-9: 14.68%
vs zstd-3
35% smaller
zstd-3: 16.41%
Compress Speed
44.7 MB/s
3.2× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 10.56% | 108 MB | 44.7 MB/s | 22.9s | 49.7 MB/s | ✅ Time-range BIDX |
| gzip -9 | 14.68% | 151 MB | 14.1 MB/s | 75.0s | 141.1 MB/s | ❌ Full scan only |
| zstd -3 | 16.41% | 169 MB | 104.2 MB/s | 10.1s | 552.6 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 3.9 s — 159,403 rows
— BIDX filters to 1-2 of 32 blocks. No full decompression needed.
Application Logs: Structured logs from your services, queryable across weeks — not hours.
Spring Boot, Node.js structlog, Python logging, Go zap — all produce consistent field sets
that PFC compresses to 10.56%, 35% smaller than zstd-3. A trace_id correlation across multiple
services from three days ago returns in under 4 seconds, without re-hydrating anything to Elasticsearch.
Teams running on 7–14 day hot retention keep logs warm purely because queries would be too slow otherwise.
With PFC Cold, that window shrinks to hours.
4 Auth / Security Logs
Sources: OAuth providers · Keycloak · WAF · fail2ban · SSO gateways · API key auth
|
File: 04_auth_security_logs_1gb.jsonl · 1.00 GB · ~4.8M lines
PFC-JSONL Ratio
9.13%
of original size
vs gzip-9
31% smaller
gzip-9: 13.34%
vs zstd-3
39% smaller
zstd-3: 15.12%
Compress Speed
51.2 MB/s
3.3× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 9.13% | 93 MB | 51.2 MB/s | 20.0s | 54.7 MB/s | ✅ Time-range BIDX |
| gzip -9 | 13.34% | 137 MB | 14.2 MB/s | 78.4s | 181.6 MB/s | ❌ Full scan only |
| zstd -3 | 15.12% | 154 MB | 161.7 MB/s | 10.1s | 558.9 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 3.3 s — 77,162 rows
— BIDX filters to 1 of 32 blocks. No full decompression needed.
Auth & Security Logs: Compliance keeps them for 90–365 days. Now they stay queryable the whole time.
Authentication events, failed logins, privilege escalations — at 9.13% ratio, a year of security
logs fits in a fraction of what gzip needs. The real win: when a security incident is discovered
six weeks after it happened, DuckDB finds the offending user_id or source IP in 3.3 seconds
from cold storage, without paying for hot-tier retention.
Audit trail queries that used to require a multi-hour Elasticsearch restore now run on S3 IA directly.
5 Infrastructure / System Logs
Sources: systemd · syslog · cloud-init · kernel · node-exporter · auditd · cron
|
File: 05_infrastructure_system_logs_1gb.jsonl · 1.00 GB · ~5M lines
PFC-JSONL Ratio
5.31%
best result in suite 🏆
vs gzip-9
46% smaller
gzip-9: 9.82%
vs zstd-3
57% smaller
zstd-3: 12.52%
Compress Speed
61.3 MB/s
3.3× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio 🏆 | 5.31% | 54 MB | 61.3 MB/s | 16.7s | 62.4 MB/s | ✅ Time-range BIDX |
| gzip -9 | 9.82% | 101 MB | 18.4 MB/s | 55.6s | 210.6 MB/s | ❌ Full scan only |
| zstd -3 | 12.52% | 129 MB | 183.2 MB/s | 6.1s | 525.5 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 2.9 s — 113,385 rows
— BIDX filters to 1 of 32 blocks. No full decompression needed.
Infrastructure Logs: 5.31% — the most compressible log type in the benchmark. 57% smaller than zstd-3.
Syslog, host metrics, daemon events, and process logs share highly repetitive fields:
hostname, facility, severity, service name, numeric metrics. PFC's log-aware preprocessing
identifies these recurring patterns and encodes them efficiently before BWT — the result is
nearly 20× compression on raw data. For ops teams collecting system logs from hundreds of nodes:
a full year of infrastructure telemetry compressed with PFC costs less than two months with zstd-3.
Query time: 2.9 seconds per hour window.
6 Streaming Pipeline Logs
Sources: Kafka · Fluent Bit · Vector · Logstash · Flink · Spark Streaming
|
File: 06_streaming_pipeline_logs_1gb.jsonl · 1.00 GB · ~5M lines
PFC-JSONL Ratio
7.49%
of original size
vs gzip-9
34% smaller
gzip-9: 11.33%
vs zstd-3
44% smaller
zstd-3: 13.47%
Compress Speed
53.8 MB/s
3.1× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 7.49% | 76 MB | 53.8 MB/s | 19.0s | 59.1 MB/s | ✅ Time-range BIDX |
| gzip -9 | 11.33% | 116 MB | 17.3 MB/s | 64.8s | 156.3 MB/s | ❌ Full scan only |
| zstd -3 | 13.47% | 138 MB | 120.1 MB/s | 7.0s | 506.9 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 3.3 s — 144,267 rows
— BIDX filters to 1-2 of 32 blocks. No full decompression needed.
Streaming Pipeline Logs: Kafka, Fluent Bit, Vector — high-throughput sources that compound fast.
At 7.49%, 34% smaller than gzip-9 and 44% smaller than zstd-3. Streaming pipelines generate
large, structured logs — consumer offsets, topic names, partition IDs, lag values — with fields
that repeat across millions of events. These archives grow quickly and are rarely queried
after the first 24 hours, making them ideal for immediate cold placement on S3.
Consumer lag investigations, dead-letter queue analysis, and partition rebalance events
are all queryable by timestamp window in 3.3 seconds.
7 Operations / CI-CD Logs
Sources: GitHub Actions · GitLab CI · Jenkins · ArgoCD · Terraform · Helm · PagerDuty
|
File: 07_operations_cicd_logs_1gb.jsonl · 1.00 GB · ~4.5M lines
PFC-JSONL Ratio
6.54%
of original size
vs gzip-9
35% smaller
gzip-9: 10.05%
vs zstd-3
47% smaller
zstd-3: 12.45%
Compress Speed
52.7 MB/s
2.9× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 6.54% | 66 MB | 52.7 MB/s | 19.4s | 59.5 MB/s | ✅ Time-range BIDX |
| gzip -9 | 10.05% | 103 MB | 17.9 MB/s | 59.9s | 201.7 MB/s | ❌ Full scan only |
| zstd -3 | 12.45% | 128 MB | 185.0 MB/s | 8.3s | 548.9 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 2.8 s — 92,010 rows
— BIDX filters to 1 of 32 blocks. No full decompression needed.
Operations & CI/CD Logs: 2.8 second queries — the fastest of all 10 log types.
Build logs, deployment events, feature flag changes, rollback triggers: at 6.54%, 47% smaller
than zstd-3. Ops logs are dominated by state transitions — created, updated, deleted, success,
failure, enabled, disabled — short strings PFC encodes with single-byte tokens.
When a release from last sprint introduced a regression, query the exact deployment window
by timestamp in 2.8 seconds. No log aggregation service required for historical ops data.
Archive build artifacts and deploy events together in the same queryable PFC file.
8 Cloud Provider Logs
Sources: AWS CloudTrail · GCP Audit Logs · Azure Monitor · IAM events
|
File: 08_cloud_provider_logs_1gb.jsonl · 1.00 GB · ~4.5M lines
PFC-JSONL Ratio
10.43%
of original size
vs gzip-9
30% smaller
gzip-9: 14.90%
vs zstd-3
37% smaller
zstd-3: 16.61%
Compress Speed
51.7 MB/s
4.2× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 10.43% | 106 MB | 51.7 MB/s | 19.8s | 52.2 MB/s | ✅ Time-range BIDX |
| gzip -9 | 14.90% | 153 MB | 12.2 MB/s | 87.9s | 164.9 MB/s | ❌ Full scan only |
| zstd -3 | 16.61% | 171 MB | 147.4 MB/s | 7.2s | 406.2 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 3.8 s — 85,440 rows
— BIDX filters to 1-2 of 32 blocks. No full decompression needed.
Cloud Provider Logs: CloudTrail, GCP Audit, Azure Monitor — compliance archives that finally answer questions.
At 10.43%, 30% smaller than gzip and 37% smaller than zstd-3. Cloud audit logs are required
for compliance in virtually every regulated industry — and they must be retained for 1–7 years.
With traditional tools, those archives are write-only: compressed on S3, impossible to query
without a full download. PFC changes that: a CloudTrail query for all API calls from a specific
IAM role in a specific hour returns in 3.8 seconds, directly from S3 IA.
Cut cloud audit storage costs by 37% and keep every byte queryable for the full retention period.
9 Network / Firewall Logs
Sources: HAProxy · iptables · UFW · AWS VPC Flow Logs · Cloudflare WAF · Cisco ASA
|
File: 09_network_firewall_logs_1gb.jsonl · 1.00 GB · ~5.5M lines
PFC-JSONL Ratio
6.14%
2nd best in suite 🥈
vs gzip-9
47% smaller
gzip-9: 11.54%
vs zstd-3
52% smaller
zstd-3: 12.86%
Compress Speed
55.0 MB/s
3.3× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio 🥈 | 6.14% | 62 MB | 55.0 MB/s | 18.6s | 53.3 MB/s | ✅ Time-range BIDX |
| gzip -9 | 11.54% | 119 MB | 15.5 MB/s | 69.1s | 186.5 MB/s | ❌ Full scan only |
| zstd -3 | 12.86% | 132 MB | 190.7 MB/s | 5.6s | 494.3 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 4.5 s — 239,740 rows
— BIDX filters to 2 of 32 blocks. No full decompression needed.
Network & Firewall Logs: 52% smaller than zstd-3 — second-best ratio across all log types.
IP addresses, port numbers, protocol identifiers (TCP, UDP), firewall actions (ACCEPT, DROP, REJECT)
and connection states recur millions of times per hour. PFC's pattern library covers these
network-specific fields natively — the encoder recognises them and compresses to 6.14%.
For security teams: trace a suspicious IP, find lateral movement patterns, or audit firewall
policy violations across 30 days of data in 4.5 seconds per query.
Network log archives are often the largest and least queried — exactly the use case PFC is built for.
10 E-Commerce / Transaction Logs
Sources: Payment gateways · Order systems · Stripe/PayPal · Fraud detection · Checkout
|
File: 10_ecommerce_transaction_logs_1gb.jsonl · 1.00 GB · ~4.5M lines
PFC-JSONL Ratio
10.18%
of original size
vs gzip-9
25% smaller
gzip-9: 13.70%
vs zstd-3
35% smaller
zstd-3: 15.78%
Compress Speed
50.1 MB/s
2.9× faster than gzip-9
Compression Ratio — lower is better
Compress Speed — higher is better
| Tool | Ratio | Output Size | Compress | Wall-Clock | Decompress | Query Support |
|---|---|---|---|---|---|---|
| PFC-JSONL v5.6.5 best ratio | 10.18% | 104 MB | 50.1 MB/s | 20.4s | 54.7 MB/s | ✅ Time-range BIDX |
| gzip -9 | 13.70% | 141 MB | 15.0 MB/s | 71.3s | 172.1 MB/s | ❌ Full scan only |
| zstd -3 | 15.78% | 162 MB | 128.9 MB/s | 8.3s | 456.5 MB/s | ❌ Full scan only |
⚡ PFC runs 6 parallel workers (~2 GB RAM total) — gzip and zstd run single-threaded. Wall-clock times reflect real user wait time.
DuckDB Extension
Time-range query (1h window / 30-day dataset): 3.4 s — 73,719 rows
— transaction logs queryable by time range in milliseconds. Critical for fraud investigation.
E-Commerce & Transaction Logs: PCI compliance keeps them for 7 years. PFC keeps them queryable.
Payment events, order state transitions, cart actions — at 10.18%, 35% smaller than zstd-3.
Transaction logs have the longest mandatory retention of any log type, and they are queried
almost exclusively for forensics: a dispute from four months ago, a fraud pattern across
specific user IDs, a failed payment window during a promotions event.
With PFC on S3 Glacier Instant Retrieval, seven years of transaction logs cost a fraction
of traditional warm storage — and a timestamp query still returns in 3.4 seconds.
No restore window. No re-hydration. Just SQL.
i
Methodology
Test Environment
Dedicated Linux VPS · 6-Core CPU · 12 GB RAM · Ubuntu 22.04 LTS · No other workloads running during tests.
Test Data
1 GB synthetic JSONL per log type. Seeded (seed=42) for reproducibility. Representative of real production log schemas and value distributions.
PFC-JSONL Config
32 MiB blocks · 6 parallel workers · Sparse rANS O2 entropy coding · purpose-built JSONL preprocessor.
Competitors
gzip -9 (max compression, default for many log pipelines) · zstd -3 (streaming standard, used by Fluent Bit / Vector / Kafka defaults).
Timing
Wall-clock time measured per run. Each tool run once after OS page-cache warm-up. RAM via peak RSS.
Query Test
DuckDB + pfc-jsonl extension. 1-hour time window queried from a 30-day compressed dataset. Roundtrip verified byte-identical on all runs.