RL-based Stock Trading Support System

An experimental reinforcement learning project for studying stock-trading decision policies with historical market data from Yahoo Finance.

The project compares classic tabular reinforcement learning agents and deep reinforcement learning agents in a simplified buy/sell/hold environment. It is designed as a learning and experimentation system, not as production trading software or financial advice.

Example DQN + epsilon-greedy evaluation result against a buy-and-hold baseline.

---

Table of Contents

• Overview
• Features
• Architecture
• How It Works
• Example Results
• Project Structure
• Installation
• Usage
• Testing
• Research Limitations
• Future Work
• License

---

Overview

The goal of this project is to explore how reinforcement learning agents can learn trading behaviors from historical price and volume signals.

The environment is intentionally simple:

• State: price trend categories, volume state, and portfolio state.
• Actions: buy, sell, or hold.
• Reward: price movement and portfolio-aware trading outcome.
• Baseline: buy-and-hold strategy.
• Output: portfolio-value plots, text reports, learned Q-tables, and model weights.

This makes the repo useful for practicing RL concepts such as state design, reward shaping, exploration strategies, policy evaluation, and backtesting caveats.

---

Features

• Fetches historical OHLCV data with yfinance.
• Converts raw stock data into discrete RL states.
• Supports classic tabular RL agents:
  • Q-learning
  • SARSA
• Supports deep RL agents:
  • Deep Q-Network, DQN
  • Deep SARSA
• Supports two action-selection strategies:
  • epsilon-greedy
  • softmax exploration
• Compares agent portfolio value against a buy-and-hold baseline.
• Saves generated outputs under local runtime directories:
  • images/
  • documents/
  • model_weights/
• Includes stored example artifacts under best_performance/.
• Includes pytest coverage for preprocessing, Q-table updates, action selection, and evaluation behavior.
• Includes GitHub Actions CI for branch pushes and pull requests.

---

Architecture

Yahoo Finance data
        |
        v
Data preprocessing
        |
        |-- trend states from moving averages
        |-- volume state from quantiles
        |-- portfolio state: empty / holding
        v
RL environment
        |
        |-- actions: buy / sell / hold
        |-- reward: price movement + portfolio outcome
        v
Agents
        |
        |-- Q-learning / SARSA
        |-- DQN / Deep SARSA
        v
Evaluation
        |
        |-- compare with buy-and-hold
        |-- save plots, reports, Q-tables, weights

---

How It Works

1. Preprocess market data

The preprocessing pipeline keeps the core price columns and creates categorical state features:

Open, Close, Trend_0, Trend_1, volume_state

Trend features are derived from moving averages, while volume_state is derived from volume quantiles.

2. Train agents

Classic agents use a tabular Q-table over combinations of:

trend states × volume states × portfolio states

Deep agents one-hot encode the state representation and learn Q-values with TensorFlow/Keras models.

3. Evaluate against a baseline

The evaluation compares the learned policy with a buy-and-hold baseline using the same historical period.

Generated outputs are written locally so runtime artifacts do not have to be committed:

images/
documents/
model_weights/

---

Example Results

The repository includes stored example outputs in best_performance/ for demonstration.

Example historical run on a tracked dataset:

Agent	Final agent value	Buy-and-hold value	Notes
Q-learning + epsilon-greedy	92,987.29	46,781.40	Stored example artifact
DQN + epsilon-greedy	95,501.89	46,827.80	Stored example artifact
SARSA + softmax	48,521.56	46,827.80	Stored example artifact

These results are useful for comparing experimental behavior, but they should not be interpreted as evidence of a profitable live-trading strategy. See Research Limitations.

Example screenshots:

---

Project Structure

.
├── .github/workflows/ci.yml       # GitHub Actions CI
├── best_performance/              # Stored example outputs and weights
├── packages/
│   ├── agent.py                   # Tabular Q-learning and SARSA agent
│   ├── deep_learning_agent.py     # DQN and Deep SARSA agents
│   └── preprocess.py              # Data preprocessing and state construction
├── readme_images/                 # README screenshots
├── tests/                         # Unit tests
├── main.py                        # CLI entry point
├── pyproject.toml                 # Project metadata and dependencies
├── uv.lock                        # Locked uv dependency graph
├── requirements.txt               # pip-compatible dependency export
└── save_best.sh                   # Helper script for saving selected outputs

Runtime directories are intentionally git-ignored:

images/
documents/
model_weights/

---

Installation

Prerequisites

• Python >=3.12,<3.13
• uv recommended

Clone

git clone https://github.com/StevenHuang41/RL-based_Stock_Trading_Support_System.git
cd RL-based_Stock_Trading_Support_System

Install with uv

uv sync --locked

Alternative: install with pip

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

---

Usage

Train and evaluate the agents for a Yahoo Finance ticker:

uv run python main.py AAPL

Taiwan market tickers can also be used, for example:

uv run python main.py 0050.TW

After training, generated artifacts are written to:

images/
documents/
model_weights/

Save selected best results

If a run produces a result you want to preserve, use:

bash save_best.sh

---

Testing

Run the unit tests locally:

uv sync --dev --locked
uv run pytest

Current tests cover:

• preprocessing without making network calls;
• deep-agent one-hot preprocessing;
• Q-learning update behavior;
• SARSA next-action validation;
• deterministic evaluation-time action selection;
• portfolio evaluation behavior for a simple hold-only policy.

CI runs the same pytest suite on branch pushes and pull requests to main.

---

Research Limitations

This project is an RL/backtesting learning project, not a trading recommendation system.

Important limitations:

• No transaction costs or brokerage fees are modeled.
• No slippage or market-impact assumptions are modeled.
• Historical performance may overfit the selected period.
• A high backtest return does not imply future profitability.
• The environment uses simplified state and action spaces.
• Real markets include liquidity, volatility regimes, survivorship bias, and execution constraints.
• Yahoo Finance data may contain missing data, adjustment issues, or delayed split information.

A production-grade trading research system would need stricter train/test splits, walk-forward validation, risk-adjusted metrics, transaction-cost modeling, and robust experiment tracking.

---

Future Work

• Add transaction-cost and slippage modeling.
• Add result tables generated from evaluation artifacts.
• Add hyperparameter-search scripts with fixed seeds.
• Add Docker support for reproducible local execution.

---

License

This project is licensed under the MIT License.