SubstackAPI on GitHub

SubstackAPI is open source and available on GitHub.

Challenge

Substack does not offer a public API. There are currently two ways to access data from a Substack publication:

  1. Use the Substack publication RSS feed
  2. Use the Substack unofficial/internal API

Both of these options come with their own set of pros and cons.

RSS Feed:

  • ✓ Publicly available and stable
  • ✓ Official support from Substack
  • ✗ Limited data access
  • ✗ Slower query performance

Unofficial API:

  • ✓ Rich data access
  • ✓ Fast query performance
  • ✗ No official support
  • ✗ Could break without notice
  • ✗ Restrictive rate limits

Neither option alone provides a complete solution for content creators to access their data reliably and efficiently.

Solution

To address these limitations, SubstackAPI implements a hybrid approach:

  1. Primary Data Source: Unofficial API

    • Used to fetch publication data
    • Rate limiting handled through minimizing requests
  2. Fallback Mechanism: RSS Feed

    • Provides backup data access
    • Ensures service continuity if unofficial API changes
  3. Caching Layer: Redis

    • Minimizes calls to the unofficial API
    • Provides fast, low-latency data access
    • Handles large amounts of requests efficiently

This architecture ensures:

  • Reliable and unlimited access to publication data
  • Minimal dependency on the unofficial API
  • Fast response times
  • High availability through fallback mechanisms

The main tradeoff with this solution is a potential delay of up to a few hours for the latest publication updates due to caching.

If you are interested in seeing the source code, you can view it on GitHub.