Technical Overview
How SubstackAPI provides reliable, high-performance access to Substack publication data.
SubstackAPI on GitHub
SubstackAPI is open source and available on GitHub.
Challenge
Substack does not offer a public API. There are currently two ways to access data from a Substack publication:
- Use the Substack publication RSS feed
- Use the Substack unofficial/internal API
Both of these options come with their own set of pros and cons.
RSS Feed:
- ✓ Publicly available and stable
- ✓ Official support from Substack
- ✗ Limited data access
- ✗ Slower query performance
Unofficial API:
- ✓ Rich data access
- ✓ Fast query performance
- ✗ No official support
- ✗ Could break without notice
- ✗ Restrictive rate limits
Neither option alone provides a complete solution for content creators to access their data reliably and efficiently.
Solution
To address these limitations, SubstackAPI implements a hybrid approach:
-
Primary Data Source: Unofficial API
- Used to fetch publication data
- Rate limiting handled through minimizing requests
-
Fallback Mechanism: RSS Feed
- Provides backup data access
- Ensures service continuity if unofficial API changes
-
Caching Layer: Redis
- Minimizes calls to the unofficial API
- Provides fast, low-latency data access
- Handles large amounts of requests efficiently
This architecture ensures:
- Reliable and unlimited access to publication data
- Minimal dependency on the unofficial API
- Fast response times
- High availability through fallback mechanisms
The main tradeoff with this solution is a potential delay of up to a few hours for the latest publication updates due to caching.
If you are interested in seeing the source code, you can view it on GitHub.