Technical Overview

​

Challenge

Substack does not offer a public API. There are currently two ways to access data from a Substack publication:

1. Use the Substack publication RSS feed
2. Use the Substack unofficial/internal API

Both of these options come with their own set of pros and cons.

RSS Feed:

• ✓ Publicly available and stable
• ✓ Official support from Substack
• ✗ Limited data access
• ✗ Slower query performance

Unofficial API:

• ✓ Rich data access
• ✓ Fast query performance
• ✗ No official support
• ✗ Could break without notice
• ✗ Restrictive rate limits

Neither option alone provides a complete solution for content creators to access their data reliably and efficiently.

​

Solution

To address these limitations, SubstackAPI implements a hybrid approach:

1. Primary Data Source: Unofficial API

   • Used to fetch publication data
   • Rate limiting handled through minimizing requests

2. Fallback Mechanism: RSS Feed

   • Provides backup data access
   • Ensures service continuity if unofficial API changes

3. Caching Layer: Redis

   • Minimizes calls to the unofficial API
   • Provides fast, low-latency data access
   • Handles large amounts of requests efficiently

This architecture ensures:

• Reliable and unlimited access to publication data
• Minimal dependency on the unofficial API
• Fast response times
• High availability through fallback mechanisms

The main tradeoff with this solution is a potential delay of up to a few hours for the latest publication updates due to caching.

If you are interested in seeing the source code, you can view it on GitHub.