Reddit Meme Scraper

A terminal-based Python tool that pulls posts from a target subreddit using Reddit's official Data API, stores their metadata in a local SQLite database, and downloads the associated media.

Note

This project uses Reddit's Data API via asyncpraw, not raw HTML scraping. It requires an approved Reddit developer app and complies with Reddit's Developer Terms and Data API Terms, including rate limits and deleted-content handling.

Features

• Fetches posts (hot / new / top) from any subreddit via Reddit's official API
• Stores post metadata in a local SQLite database, with automatic deduplication on post_id
• Downloads media files from stored post links into a local folder
• Async throughout (asyncpraw, aiosqlite, aiohttp) for non-blocking I/O

Tech Stack

• Python 3.14
• asyncpraw — async Reddit API client
• aiosqlite — async SQLite access
• aiohttp — async HTTP requests for media downloads
• python-dotenv — environment variable management

Project Structure

Web_Scrapper_Memes/
├── scrapper.py       # Fetches posts from a subreddit and stores metadata in SQLite
├── downloader.py      # Downloads media files for posts already stored in the database
├── memes.db           # SQLite database (generated on first run, git-ignored)
├── downloads/          # Downloaded media files (git-ignored)
├── .env                # API credentials (git-ignored, see setup below)
└── .env.example        # Template showing required environment variables

Setup

1. Install dependencies

pip install asyncpraw aiosqlite aiohttp python-dotenv

2. Register a Reddit developer app

Reddit's app registration currently requires manual approval through their request form rather than instant self-serve creation. Choose the script app type when registering.

3. Configure environment variables

Create a .env file in the project root:

REDDIT_CLIENT_ID=your_client_id
REDDIT_CLIENT_SECRET=your_client_secret
REDDIT_USER_AGENT=script:meme-scraper:v1.0 (by /u/your_username)

Usage

Scrape posts from a subreddit:

python scrapper.py <subreddit> --sort hot --limit 25

Argument	Required	Default	Description
subreddit	Yes	—	Subreddit name, no r/ prefix
--sort	No	hot	One of hot, new, top
--limit	No	25	Number of posts to fetch

Example commands:

# Top 10 posts of all time from r/memes
python scrapper.py memes --sort top --limit 10

# Most recent 25 posts from r/dankmemes (defaults used for --limit)
python scrapper.py dankmemes --sort new

# Current hot posts from r/ProgrammerHumor, small batch
python scrapper.py ProgrammerHumor --sort hot --limit 15

# No flags at all — falls back to defaults (--sort hot --limit 25)
python scrapper.py memes

Download media for stored posts:

python downloader.py

No arguments needed — it reads directly from memes.db and downloads anything not already present in downloads/.

Database Schema

Column	Type	Description
post_id	TEXT (primary key)	Reddit's unique post ID
subreddit	TEXT	Source subreddit
title	TEXT	Post title
author	TEXT	Username, or [deleted]
url	TEXT	Direct media link
permalink	TEXT	Full Reddit comments URL
score	INTEGER	Post score at time of fetch
created_utc	REAL	Post creation timestamp
nsfw	INTEGER	1 if marked NSFW, else 0
fetched_at	REAL	Timestamp this row was scraped

Roadmap

Unified scrape-and-download pipeline

Currently scrapper.py and downloader.py run as two separate steps. The plan is to combine them into a single pipeline: fetch a post, evaluate it, and download its media immediately if it qualifies — rather than scraping everything first and downloading in a second pass.

Post decision-making logic

Before downloading, each post will be scored and filtered rather than downloaded unconditionally. This decision layer will sit between the fetch step and the download step in the unified pipeline.

Weighted scoring metrics

Three metrics are planned to drive the decision logic, listed in current priority order:

1. Interaction Score (highest priority) — a measure of post engagement, likely derived from a combination of upvotes, upvote ratio, and comment count.
2. String Frequency — frequency analysis of recurring words or phrases in post titles (and possibly comments), used to identify trending formats or topics.
3. Sentiment Score — sentiment analysis of a post's title or top comments, used to gauge audience reception.

Each metric will eventually be weighted and combined into a single composite score; exact weighting values and formula are still to be determined.

Other known gaps

• Gallery and video posts — post.url only reliably captures single-image/GIF posts; Reddit galleries and native video need separate handling.
• Deleted-content compliance — a periodic job to purge stored posts/comments that have since been deleted on Reddit, per Reddit's Data API Terms.
• Multi-subreddit support — currently one subreddit per run; extending to a list of subreddits is a planned enhancement.