Maximizing Over a Long Horizon

I'm starting this blog to document things as I work through them rather than after I've tidied them up. Three threads, roughly:

AI — reinforcement learning, POMDPs, LLMs. Mostly notes on what I'm building and reading.
Business — what I've learned founding companies, and what I got wrong.
Life — the occasional thing I care about that isn't either of the above.

It seems right to open with the equation I keep coming back to:

\pi^* = \arg\max_{\pi} \; \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \right]

Find the policy that maximizes expected reward, summed not over the next step but over every step, out to infinity. Nearly everything else in RL — value functions, Bellman backups, policy gradients — falls out of taking this seriously.

It's worth unpacking, because the notation compresses a lot. Read right to left. $R(s_t, a_t)$ is the reward you get for taking action $a_t$ in state $s_t$ , and the thing to notice is that it isn't a number you can look up. You choose the action, but the world chooses what happens next: $s_{t+1}$ is drawn from a distribution $P(\cdot \mid s_t, a_t)$ . So the state you're in two steps from now is random, which makes the reward you collect there random too. Each $R(s_t, a_t)$ is a random variable, and the whole sequence of them across time is a stochastic process.

That's why the expectation $\mathbb{E}$ has to be there. For any given way of behaving, there isn't one future — there's a distribution over futures, some lucky and some not. You can't optimize the trajectory you hope for, so you optimize the average across all of them, weighted by how likely each one is.

And $\pi$ is a policy, which is the word that trips people up. A policy isn't a sequence of actions, like a plan or a recipe. It's a rule: what you would do in each situation you might find yourself in, including the ones you don't expect. You're not searching for a route. You're searching for a way of deciding. That's what makes the answer robust to a world that refuses to go how you planned.

What I like about it is $\gamma$ , the discount factor — a number between 0 and 1 that gets raised to the power $t$ , so a reward $t$ steps away counts for $\gamma^t$ of its face value. It sets how far ahead you're willing to look. Push $\gamma$ toward zero and the optimal policy becomes greedy: take whatever pays now. Push it toward one and the same agent in the same world will accept a negative reward right now, because the state it lands in is worth more later. (It's also quietly doing bookkeeping — with $\gamma < 1$ and bounded rewards, that infinite sum stays finite.)

That greedy case is worth being precise about, because it's easy to misread. Setting $\gamma = 0$ does not zero out the rewards. The $t = 0$ term is multiplied by $\gamma^0 = 1$ , so the immediate reward always counts at full value no matter what $\gamma$ is — only the future gets discounted away. The whole objective collapses to:

\pi^* = \arg\max_{\pi} \; \mathbb{E}_{\pi} \left[ R(s_0, a_0) \right]

The agent still collects a reward at every step of its life. It simply refuses to let any of them influence what it does right now. A rough handle for the other direction: the effective horizon is about $1/(1 - \gamma)$ , so $\gamma = 0.9$ is planning roughly ten steps ahead and $\gamma = 0.99$ about a hundred.

That willingness to take a loss now isn't a quirk of the math. It's the whole point. An agent that can't take a bad step on purpose can't cross a valley to reach a higher peak — it gets stuck at the first local maximum it finds and stays there, optimally, forever.

Most of the decisions I've gotten right came from raising $\gamma$ . Most of the ones I got wrong came from quietly letting it drop.

More soon.