Research note · full experiment log
We tested 6 RL allocation algorithms — then chose not to use them
Published 2026-10-24 · out-of-sample window 2010–2026 (202 months) · verdict: keep static 50/50. Most "AI strategy" sites never publish failed experiments; we think negative results are exactly what belongs in due-diligence material.
Setup
- <b>Decision:</b> monthly momentum-vs-defensive sleeve weight (0–100%) — equivalent to timing.
- <b>State:</b> 8 market features — SPY trend (px/MA200), 21d annualized vol, 63d return, Top-10 momentum breadth and median crowding, TLT/GLD trend, momentum–TLT 63d correlation.
- <b>Reward:</b> log(next-month net return) with intra-month drift, monthly resets, 10bp equity / 2bp ETF costs plus reallocation fees.
- <b>Entrants:</b> DQN / PPO / A2C / SAC / TD3 / DDPG; baselines static50, static60, voltarget, MCPI (prior round).
Headline results · 202 OOS months (seed=7)
| Strategy | CAGR | MaxDD | Sharpe | Vol |
|---|---|---|---|---|
| DQN | 13.97% | −19.11% | 0.99 | 14.20% |
| PPO | 12.46% | −16.67% | 0.89 | 14.16% |
| A2C | 15.71% | −25.95% | 0.80 | 20.65% |
| SAC | 12.13% | −15.57% | 0.89 | 13.80% |
| TD3 | 14.93% | −19.62% | 0.81 | 19.33% |
| DDPG | 11.16% | −32.80% | 0.67 | 18.05% |
| static50 · current | 12.84% | −16.63% | 0.90 | 14.43% |
| static60 | 14.34% | −20.41% | 0.86 | 17.08% |
| voltarget | 11.18% | −16.07% | 0.87 | 13.05% |
| MCPI | 12.78% | −20.36% | 0.87 | 14.89% |
Single-seed metrics lie
DQN's 0.99 Sharpe on seed=7 looks like a win — until you change the seed:
| Algo | Sharpe ×3 seeds | Verdict |
|---|---|---|
| DQN | 0.99 / 0.98 / 0.82 | high variance — seed 123 loses to static50 |
| PPO | 0.89 / 0.94 / 0.90 | stable but no edge (≈static50) |
| A2C | 0.80 / 0.91 / 0.82 | unstable |
The learned policies tell the same story: PPO collapses back to 0.49 (≈50/50) on its own, A2C pins to its 0.75 cap (≈75/25), DQN alone times the market but its rules are seed-fragile. At monthly frequency there is no transferable conditional information in "how much momentum to hold."
Verdict & artifacts
- Production spec unchanged — static 50/50 stays.
- Six RL algorithms + MCPI + 3 seeds: none beat the static split out-of-sample robustly.
- The only variant worth keeping is voltarget (lowest vol, 13.05%, at ~1.7pt CAGR cost) — a possible conservative tier, not a replacement.
- Reproduction artifacts: balanced_drl.py (--seed/--algos/--years/--steps), balanced_drl_oos.csv (monthly OOS returns), balanced_drl_actions.csv (monthly action log).
Research note, not investment advice; backtests are simulated and not indicative of future results. · ← Research index · Balanced I methodology →