`
Roll mouse over text in dark red to reveal explanatory boxes
Click on text in blue to jump to relevant section of this overview.
Click on 
to link to a passage from a related article.
Click on references in brown to bring up complete citation
Picoeconomics (micro-micro-economics) is the study of intertemporal bargaining: how conflicting motives interact with each other when they will be dominant at different times. It is not a psychology of economics, but rather an economic approach to psychology and its related behavioral sciences, including philosophy. This website is a hyperlinked, word-searchable overview of how people’s innate time perspective creates these conflicts, and how it limits the way they can be overcome. I comment on much of the relevant literature, but for copyright reasons only my own articles are excerpted here.
Until recently the behavioral sciences never contradicted the common assumption that choices have a certain Newtonian momentum—that they’ll stay in place unless influenced by some new factor. But now a widely repeated experimental finding has contradicted that assumption. Our perspective of the future is warped in such a way that 1) we overvalue imminent events and 2) we have incentive to predict and forestall this overvaluation before it develops. Hence intertemporal bargaining. As it plays out, this simple dynamic can be found at the root of many familiar experiences, including impulsiveness, addiction, strength of will, actual freedom of will, compulsiveness and other side effects of will, the near-irresistibility of pain, and how some goals become important in the absence of external incentives. Analysis of intertemporal bargaining also offers solutions to several theoretical problems of motivational science that are not obvious in daily experience:
The gist is that analyzing disparate motives by the timing of a single dimension, rewardingness, lets us see them as parts of an integrated whole. Other explanatory factors that have been held to be necessary—especially classical conditioning, negative motives as the inverse of positives, and overriding cognitive controls—reduce to consequences of this timing.
Each topic is summarized in a numbered paragraph. These paragraphs generally contain a link indicated . In some cases there are additional links marked 
or 
. Clicking on these links will bring up the relevant section from a published article. You can read the highlighted section, you can click a button that displays a copyable APA-style citation for that article, and you can click a button to take you to the top, to read the entire article or to open a PDF version of that article which can be printed or downloaded. Clicking the button marked "Return to previous location" will step you back to the Nutshell paragraph you had been reading.
Within Nutshell itself, there are frequent links to related paragraphs, indicated by blue text. If you follow any of these links, you can click "Back to previous location" or use your browser's Back button to take you back to where you were before clicking the link.
You can click on references printed in brown, such as (Skinner, 1950) to bring up a complete citation for the source. The citation can be highlighted and copied, to be pasted in another document. (Problems with copy-paste function? Click here.) Clicking outside the citation box will cause it to disappear.
I have avoided coining words, but when I limit an ordinary word in a technical way, the first use of the word is italicized in dark red, and hovering the mouse over the word will display a definition.
Introduction: picoeconomics is a model of intertemporal bargaining. A person is a population of learned processes—interests-- that compete or combine in strict accordance with how well they maximize prospective reward, including reward from emotion and imagination. Because of the warp in how rewards lose value with delay, interests have to include tactics to deal with the shifting outcomes that they themselves partially determine. Such internal marketplace models have been proposed since Bentham’s “two sovereign masters, pain and pleasure,” but formerly they depicted people as simply estimating their best options and keeping their original choices unless new inputs occur. Critics have complained that these agents are mere throughputs, totally manipulated by the reward contingencies they face. (Russell, 1978) But even granting reward to be the inescapable, neurological cause of choices, the warp in how its effect declines with expected delay invalidates this criticism. This warp creates a limited warfare relationship among interests, and the resulting time-based bargaining is chaotic in the technical sense: Its outcome is unpredictable with certainty even from a knowledge of all the prior incentives.
The need for stable choice-making is the basic issue in humans’ evolutionary advance from instinct-based foraging to planning. With intertemporal bargaining the person is no longer a throughput but rather a literally self-conscious agent, who makes decisions by finding equilibria among not only external incentives but also her shifting self-predictions. This internal process has some properties in common with how social groups such as legislatures and financial markets achieve stability. Such analogies have usually been thought of as more poetic than functional, but limited warfare gives all these processes a common operation with disparate outcomes. Our strictly reward-based pursuits can combine to make us, as Hamlet said, “noble in reason and infinite in faculty.” Or they can trap us in hopeless addiction.
How much do we know about intertemporal bargaining so far? The phenomenon on which the theory is based is now well established, and some of its implications have been tested by controlled experiment; but many have not, and some may not be experimentally testable—for instance, when they involve a mental process that predicts its own future activity. So the methods that this model relies on have included not only
but also
New mechanisms are proposed here only when 1) they are coherent and parsimonious; 2) they accord with what observations are now available, including common experience; and 3) conventional models fail to account for important observations or are contradicted by them.
Our lives are built of choices among alternatives. A few things that we do are not chosen, such as reflexive knee jerks. But we face a large set of alternatives that could all replace each other, even if we don’t usually think of them that way. This is the set of substitutable behaviors. People can sometimes even learn not to mind pain, not to feel sexual attraction, not to believe what we see. Everything that occupies our mind could be forced out by something else. We don’t stand above this competition and choose; we are this competition.
1.1 A definition: Reward selects the processes it follows. Reward tells you “you’re getting warmer,” but this isn’t just information. Reward actually selects the pattern of activity it follows, making you more likely to repeat that pattern. Such selection can be called choice, but it does not imply the conscious attention that would get it called deliberate. Chosen merely implies motivated.
A point about usage: The selection process obviously takes place inside your head, but in ordinary English external events that lead to reward get called rewards, a potentially confusing usage; nevertheless I’ll follow it here rather than coin words. I’ll call the external events rewards or a reward, and the internal process just reward. Reward may or may not depend on rewards.
1.2 Behaviors are selected for reward, not directly for fitness. Some organisms always do the same thing in a given situation: Moths always fly toward the light. As longer-lived organisms evolved, so did a way that individual choices could be selected instead of whole pre-programmed organisms. Reward evolved as a proxy for Darwinian adaptiveness—giving rise to the motives that were most apt to produce adaptive choices. Genes came to program kinds of reward, rather than specific choices in a particular situation. Thus, although people have some inborn action tendencies—for instance to be afraid of heights— these operate through motives, and can be overcome. It is reward that selects behaviors. Importantly, humans often discover sources of reward in thought and imagination that are independent of external rewards, but these are still constrained by innate properties. When we seem to pick what activities will reward us, we are actually making gambles that will pay off or fail according to those properties.
Because of the great changes that have come from civilization, often the rewards that attract us do not still confer an evolutionary advantage. If an inborn reward starts to be maladaptive in the Darwinian sense, for instance motivating us to seek opiates, rich foods-- or birth control-- this will create selective pressure against that reward. But in the meanwhile, we are not motivated to maximize adaptiveness, only reward. Over the centuries that it takes evolutionary pressure to change the inborn reward mechanism, rewards will still select the behaviors that they used to. Nor do logical, moral, or other cognitive processes per se have any necessary call on our choice-making. We are creatures whose only job is to forage for reward.
1.3 The marketplace of vicarious trial and error (VTE). Reward is what makes mental processes attractive. Alternative mental processes compete with each other on the basis of expected reward. They vie for adoption by how much they lure you, one after the other, a process called vicarious trial and error (VTE). Of course an option often wins because it is moral, neat, or accurate, but such standards have to have won out in turn by giving you satisfaction—that is, by being rewarded. An option may also win because it is intensely attractive only briefly or intermittently, an unwelcome urge that lures you to pay attention to obsessions, panic, or pain.
1.4 Reward is a unitary phenomenon. There may or may not be separate neural centers for different kinds of reward. The question doesn't matter much when you have only one set of limbs-- or, more to the point, a finite channel of attention that directs those limbs. If a given behavior can be influenced by more than one center, those centers must compete for the exercise of this influence, and whatever process governs this competition constitutes a single effective reward center. To the extent that one behavior can be replaced by another it has to compete with the other for expression, and this competition operates as a clearinghouse for all substitutable behaviors-- all behaviors among which you can choose. This is the constraint that unifies a person's behavior at any given moment.
1.5 Reward implies just selective power, not pleasure. The only thing that will overcome an expected reward is a greater competing reward. Sometimes reward selects patterns that are the opposite of pleasures, when you’re drawn into a painful process such as the urge to panic or to brood over your failures. Reward can be found to underlie processes that aren’t usually regarded as behaviors at all—for instance when you “have” an emotion. Subjectively an option may offer variously the “click” of a cognitive solution, the satisfaction of a desire, or the dysphoric engagement with an urge. The most irresistible lure may feel like ecstasy, or it may feel like an unwanted but unopposable urge. Some of these cases don’t look comparable; but since each one can draw attention from each other one, they are substitutable, and so we must conclude that they compete on a common basis.
1.6 The gold standard for analyzing choice has been rational choice theory (RCT). Until recently behavioral science assumed that people were naturally consistent over time. This was explicit in economics, but also implied in psychology and the philosophy of mind. According to RCT a person may discount (devalue) events that won’t happen until later, but she does so in the way keeps their relative values the same. This would be the same formula for discounting that a bank uses, subtracting a percentage of value for each unit of delay (exponentially). If the prospect of a new car was worth $20,000 to you when you could have it today, but $18,000 if you couldn’t have it for a year, that would be said to imply that you discount the prospect at 10% a year, so a two year prospect would be worth $16,600 to you, a three year prospect $14,940, and so on.
1.7 RCT fails to account for temporary preferences.People in experiments often show intertemporal inconsistency, for instance preferring $50 in 3 months to $100 in 2 years and 3 months, but not preferring $50 in 4 years to $100 in 6 years—the same choice at a different distance. Such findings imply that a subject’s discount rate changes over time. The same pattern holds in real life for choices ranging over time scales from seconds-- not pausing to put on a condom—to years-- not saving money for retirement. We put off going to sleep even though we’ll spend the next day tired, buy at the higher price that doesn’t have to be paid until later, give in to pressure knowing it will invite worse pressure in the future, and generally fail to make the stitch in time that will save nine. RCT doesn’t account for intertemporal inconsistency, and certainly not why it occurs repeatedly-- in addictions, or even in the ordinary bad habits that you keep following while knowing they’re bad. In order to preserve RCT, conventional theories explain temporary preferences by a lower, unreasoning mental process that overcomes rational valuations.
1.8 Theories using dual choice mechanisms only roughly correct RCT. Most approaches to irrationality have been dualistic: From the time of Plato’s chariot horses of the soul--well-behaved vs. unruly-- we have served God or Satan, been governed by ego or id, or, nowadays, use cool or hot thinking and choose with our neocortices or midbrains. Certainly our choices can often be divided into patient and hasty, planned and impulsive, wise and foolish. But hot thinking-- aroused appetite or emotion-- plays a role in only some kinds of temporary preference. Dualism makes explanations easy—Dr. Jekyll behaves inconsistently because Mr. Hyde sometimes takes over—but on the other hand it forces our ideas of motivation into discrete categories where the evidence rather points to an open field of competition. (Ainslie, 2018)
2.1 Hyperbolic delay discounting causes impulses. In contrast to RCT’s percentage-per-unit-delay, controlled experiments suggest an elementary selecting power of reward* that is inversely proportional to its expected delay. That is, a graph with delay on the x axis and selecting power on the y axis describes a hyperbola. A hyperbolic discount curve is different from an exponential one, not just in being steeper at short delays, but also in being shallower at long delays. The steep part makes nearby rewards disproportionately effective, creating impulses. The shallower part describes motivation for preventing later impulsiveness. Four kinds of finding support a hyperbolic shape:
Brain imaging has further confirmed the evidence of expressed preferences, that the motivational weight of future events diminishes proportionately to their expected delays (Kable & Glimcher, 2007; Wittman et.al., 2010) It gives the near future a temporary advantage over the far future, the same proportionality effect that makes a shorter, nearer building begin to hide a taller, more distant building as we approach them.
Two foundational experiments:
Change of preference over time in pigeons. When choosing between an SS food reward and an LL reward four seconds later, birds shift from choosing SS to LL as the delay before the SS goes from 0 to 12 seconds. This finding in nonhumans demonstrates that the hyperbolic shape of the discount curve is elementary, a property of motivation that dates well back in evolution. (Ainslie & Herrnstein, 1981)
Change of preference over time in people. A majority of both substance abuse patients and random adults reported preference for an SS of $50 over an LL of $100 changed their preference as the delay before the SS was increased. You can easily reproduce this finding by asking your friends. (Ainslie & Haendel, 1983)
Historical background:
Ancient theories of impulsiveness A basic part of religious theories was to explain impulsiveness. Perhaps the first was the doctrine of original sin.
Economists’ observations of delay discounting. Economists first thought that any future discounting was irrational, then realized that inconsistent discounting was what caused problems.
Psychoanalysts’ observations of delay discounting. Freud’s “pleasure principle” implied temporary preference for SS rewards, but his followers did not develop the idea.
Behavioral psychologists’ observations of delay discounting led to picoeconomics.. B. F. Skinner’s behavioristic psychology developed precise ways to measure motivation, but was philosophically opposed to hypothesizing about internal—mental—processes. Subsequent cognitive psychologies repudiated that philosophy, but also got rid of Skinner’s idea that choice is strictly determined by motivation. Modern economics also measures motivation precisely, but it, too, avoids looking inside the person. Picoeconomics (micro-micro-economics) infers mental processes from an application of game theory to elementary laboratory findings. The predicted motivational interactions offer a coherent beginning for a comprehensive science of choice.
2.2 Hyperbolic discount curves did not hurt nonhumans during natural selection of species. The temporary preference effect creates obvious planning problems for humans. However, hyperbolic discounting is part of the basic way that animals perceive stimuli; and in nonhuman species long range adaptiveness doesn’t depend on foresight but on obedience to instincts that reward right away. When human foresight evolved, we had inherited the basic hyperbolic tooling.
Alternative theories without hyperbolic discounting. There are other theories besides hyperbolic discounting of why preference changes as expected rewards get closer. They do not fit the data as well.
Framing effects sometimes look like hyperbolic discounting. The decrease in apparent discount rates as delay increases have been attributed to various cognitive framing processes, based particularly on:
But the effects are small, and do not entail reversal of preferences as delay changes.
2.3 Some theorists prefer a variant of hyperbolic discounting: Hyperboloid (beta/delta) discounting. Some kinds of temporary preference for SS rewards occur when your emotions or appetites are aroused. Theorists have proposed accommodating these cases with an extra, beta spike on top of a rational, exponential (delta) basic discount curve. However, hyperboloid curves are not sufficient to explain temporary preference in general After I present more background, I’ll discuss how the supposedly conditioned arousal of appetite and emotion is itself based on reward, and how positively fed back motives account for sudden arousals.
2.4 The hyperbolic shape also raises the value of very long term prospects. In findings that are converse to impulsiveness, the present value of projects that will take decades is usually reported as higher than any competitive exponential discount rate could produce. Such projects have included saving lives (Cropper et.al., 1992), preserving individual health (Chapman, 2002), public works projects (Harvey, 1994), and, especially noteworthy, measures to prevent climate change (Gollier & Weitzman, 2010).
3.1 Hyperbolic discounting creates competing internal interests. The mental processes that are learned because they get a particular reward can be described as that reward’s interest—just like an economic interest in a marketplace, such as the coal interest or the environmental interest. Interests whose rewards are dominant at different times, such as a temptation to get drunk and a wish to avoid a hangover, wage war on each other—to forestall the other before it gets the upper hand. It is usually limited war, however. The interest in getting drunk will fail if it includes the prospect of becoming an alcoholic, and the interest in staying sober will fail if it rules out too much current comfort.
3.2 Controlling impulses requires commitment. An interest based on delayed rewards will survive only to the extent that it forestalls impulses-- interests that would become dominant as their rewards get closer. The most obvious methods act by precommitment, either external or internal, but they suffer from limitations. And internal (mental) commitments that involve effort—“holding your breath”—are unstable, as we shall see.
3.3 External precommitments may be physical or social. External precommitments include contracts, medications to change your appetite, investments that can’t be cashed in easily, and the main commitment method before modern times, social pressure. These commitments may be long-lasting but are also limited—For instance, social commitments don’t protect you from impulses you can conceal, people who exploit you, or people who share your impulse. [AddixRes’08]* Social approval was the greatest constraint on impulsive choice until around 1600, when a focus on individual conscience arose.
A foundational experiment
Precommitment occurs even in pigeons. Some but not all pigeons will learn to peck a key, the only effect of which is to prevent an SS reward from becoming available later. Since the birds presumably have no insight into their impulsiveness, this choice must be based only on the different relative strengths of the rewards at the earlier and later times. Conversely, rats will accept a smaller, sooner foot shock to prevent a larger one later, but only if the SS shock is also delayed from the moment of choice (Deluty et.al., 1983)
3.4 Two kinds of internal precommitments act only in advance.
3.4.1 Attention control blocks out alternatives. You can keep your mind away from tempting subjects, or look away from the early sign of an impulse before it becomes strong. This action ranges from simple vigilance (attention filtering) to sophisticated mental blocking (as in hypnotic suggestion). It is necessary for moment-to-moment consistency—You don’t re-evaluate your incentives (VTE) continuously, but rather sample them, more or less frequently as the volatility of your situation calls for. The “willpower” elicited in lab experiments usually involves putting off the re-valuation of an urge from moment to moment, much as the chairman of a meeting avoids holding a vote. This process can only work in the short term. It may now be seen by brain imaging.
3.4.2 You can cultivate emotions in advance, or nip them in the bud. The activity of some emotions leads to more of the same activity. So you can cultivate an emotion contrary to the impulse you fear (reaction formation, or inhibition of appetite), or stop emotions from arising before they gain momentum (isolation of affect). When the relevant emotion is imminent, avoiding it may be the same process as attention control, in that both avoid rehearsing (VTE of) alternative options in the short term.
3.5 Hyperbolic valuation enables long term interests to combine. Hyperbolic curves are higher than exponential curves not only just before a reward; they’re also higher than exponential curves at longer delays. So if curves from a whole series of LL rewards are added together, it increases your relative incentive to reject each SS reward when it gets close. This means that when you choose between a whole series (or bundles) of SS/LL pairs at once you’ll tend to choose the LL rewards more than when you choose between each single SS/LL pair. Crucially, the increase in patience when series of SS/LL choices are bundled together would not happen if your discount curves were exponential. You discount very long term expectations—your later health, your retirement, climate change—not from experience but through imagination, but the shape of your discount curve still tends to be hyperbolic.
A foundational experiment
Bundling choices increases patience, even in rats. When rats chose between SS and LL amounts of sugar water, they opted for LL more when they chose a bundle of the next three deliveries at once than when they made one choice at a time. This shows that the anti-impulsive effect of adding up series of discount curves doesn’t depend on suggestion or some other aspect of human culture. It also shows that summing the effects of delayed rewards does not require a cognitive skill such as adding numbers. (Ainslie & Monterosso, 2003)
4.1 Strength of will is distinct from the initiation or ownership of actions. The term “will” is most often used also to mean 1) resolution or strength of intent, but sometimes also to mean 2) the bridge from thought to the initiation of an action, as in willing your hand to move or 3) your sense of having authorized the action, like moving your hand deliberately. Intertemporal bargaining mostly involves only the first meaning—the process that maintains an intention against alternatives.
4.2 Strength of will comes from intertemporal bargaining. If you realize that your current choice is a test case for how you can expect yourself to make similar choices in the future, this perception creates a bundle of your expected outcomes. You then face a variant of a repeated prisoner’s dilemma game among your expected successive selves, with your bundle of expected future outcomes at stake in each choice. If you defect to the SS option this time, your best guess will usually be that you will go on doing so, a prospective loss you might experience as guilt. A personal rule about when to avoid SS options represents your proposal of a truce in the limited warfare between your present and future selves. These selves enforce such a truce by recursive self-prediction. 
4.3 Recursive self-prediction is a common experience. Many partially voluntary activities turn on the knife-edge between two opposite positive feedback cycles of self-prediction: intertemporal cooperation leads to more of this cooperation, and defections (lapses) lead to more defections. The old Darwin-James-Lange theory is partially correct, in that our perception of whether emotions such as fear and nausea are increasing or decreasing is positively fed back to the development of those emotions. When such activities as falling asleep or urinating (for prostate patients) are tenuous, they are also subject to recursive self-prediction. However, self-prediction is sometimes used merely for mnemonic purposes, such as leaving your hat where you expect you’ll remember it.
4.4 Alternative models of will. Several alternative theories of how willpower works have been published, such as 1. diverting attention, 2. flexing a muscle-like organ, or 3. simply realizing the value of choosing according to principle. Each lack a key property: 1. Willpower works with full consciousness of the alternatives; 2. a muscle-like organ that didn’t itself consist of motivation would need a separate source of motivation to direct it; and 3. respect for principle that did not put the principle at stake would not provide the incentive to stick to it in the face of temptation. However, some recent experiments have suggested that intertemporal bargaining works synergistically with “holding your breath,” that is, avoiding VTE, which would look like #1and/or #2 but necessarily be unstable.
4.5 Brain imaging: Will involves synergy of choice bundling and moment-to-moment control of attention/emotion. The restraint of an SS choice from moment to moment does involve keeping attention diverted from the rewards involved. When human subjects are trying to resist a temptation, brain activity that suggests attention filtering and inhibition of appetite are observed. This activity often gets called “willpower expenditure,” but it is different from true willpower, the intertemporal bargaining process that is necessary for stable resolutions. “Willpower expenditure” is effortful, and is greater the less the superiority of the bundled LL rewards to the bundle of their SS alternatives. That is, the bigger the bundle of LL incentives, the less “willpower expenditure” is needed. Because experiments with concrete temptations in the laboratory are necessarily short term, they involve a large component of “willpower expenditure,” a capacity for which has been described as analogous to muscle strength. However, your decision about whether this expenditure is worthwhile depends on how you bundle your expected rewards.
4.6 Willpower need not be deliberate. You inevitably engage in self-prediction whenever you notice that you keep encountering similar choices. The process won’t usually take the form of an actual resolution. For instance, you may go to the gym only if you think you’ll continue to go to the gym; and this expectation survives only if you see yourself actually go.
4.7 Willpower is new in evolution. Overvaluation of imminent outcomes was not an important disability in nonhumans From an evolutionary viewpoint intertemporal bargaining must be a recent kludge. We use our newly evolved awareness of our own past choices (Smith et.al., 2003) and our imagination of our future states of mind—“episodic prospection” (Szpunar, Spring, and Schacter, 2014) to jerry-rig a way to keep our hyperbolic discount curves from changing our minds.
4.8 There are several kinds of evidence about intertemporal bargaining. Experimental data about intertemporal bargaining is hard to get, because the self-observation is internal. Suggestive evidence has come:
A foundational experiment
A single new defection affects a repeated prisoner’s dilemma game more than a single new cooperation. Anonymous subjects in a two person game repeatedly chose between 100 points (exchangeable for money) for themselves versus 70 points each for themselves and their partners. After choices stabilized in one direction, false feedback that the partner had suddenly switched changed each subject’s subsequent responding, much more so if the new (false) responses were defections than if they were cooperations. This models how trust in self-enforcing contracts is damaged more readily than it is repaired. (Monterosso et al., 2002)
4.9 Thought experiments show what we already know about recursive self-prediction. Self-observation (“introspection”) has been in disrepute ever since early psychologists carried it to extremes, but many modern experiments rely on self-reported preferences. One school of philosophy has potentially sharpened our self-observation, but we have to be careful to distinguish findings that we actually observe from how we interpret them. Key thought experiments are:
Personal rules include “mental accounts” Economists responded early to the issue of intertemporal bargaining, but after initial dalliance with the idea of an underlying hyperbolic discount curve they were soon at pains to avoid any such thing.
4.10 Bright lines keep you from hedging on personal rules. Intertemporal contracts (personal rules) are self-enforcing, but are apt to be unstable unless they include a line between cooperation and defection that stands out from other possible lines. Such bright lines, like the one between some smoking and none, deter you from proposing new terms for a personal rule when temptation is high. Bright lines limit the most effective weapon used by short term interests, which is to propose alternative rules that permit the SS choice just in the current case—that is, rationalization.
4.11 Your inborn discount rate is not open to learning. If your inborn discount function could be modified, that fact would motivate you to modify it as much as possible, just to make all future rewards seem larger. Then there would be no such thing as impulses.
4.12 You achieve something close to exponential discounting through both intertemporal and interpersonal bargaining. The unstable preferences caused by hyperbolic discounting not only get in the way of your own plans, but also put you in danger of being exploited by more consistent people—“money-pumped.” In response, intertemporal bargaining makes summed discount curves shallower and less concave than curves from single rewards—closer to exponential. You can also define which examples are more important and which less, “mental accounts” as proposed by Thaler & Shefrin (1981) , which mean you don’t always have to be ascetic as long as you protect an “investment” account. Finally, where you’re in competition with other people you have an additional incentive not to follow your spontaneous, hyperbolically discounted preference.
People’s effective discount rates have been modified by their bargaining experience. Various human experiments and questionnaires have elicited discount rates that differ by hundredfolds between subjects. By comparison, discount rates in nonhumans vary only by single digits. The wide human variation probably reflects different self-control practices, although still with an inborn, genetic component (see Anokhin et.al., 2011)
4.13 Even private, one-shot choices can be test cases (self-signals). Social psychologists have had trouble explaining why subjects are so altruistic or cooperative in anonymous games that will happen only once, such as the single-play Prisoner’s Dilemma. But any choice can be a test of your character, and an experimenter can’t reassure you otherwise.
PREVIEW
Recursive self-prediction has many implications. Unlike theories of self-control that rely on narrowing your range of choice or of attention, the recursive self-prediction model accounts for a number of common observations about self-control that are otherwise puzzling. These include:
Self-control is just one topic in the broad field of choice-making. More or less related to self-control are other motivational patterns that arise from hyperbolic delay discounting, many of which necessarily include recursive self-prediction:
Finally, this Nutshell will interpret current theories of addiction and discuss the problems of planning for welfare in light of hyperbolic delay discounting.
5.1 Willpower has serious side effects. Seeing your current choices as test cases to create bundles recruits more incentive for LL choices, which necessarily means increasing the cost of the SS choices that are lapses. The risk of this increased cost
1) makes it harder to focus on the here-and-now;
2) may lead to specific areas where, after a failure, your will gives up;
3) motivates you to avoid noticing lapses (repression and denial); and
4) motivates you not to chooses richer options when they are ambiguous in terms of whether they follow a rule (compulsiveness).
5.2 Side effect #1: Bundles of choice overshadow their individual members. Personal rules may lead you to develop a lawyerly wariness of your current desires.
5.3 Compulsions are bundles of choices that have become rigid. A choice that belongs to, or even just symbolizes, a larger category of similar choices takes on additional significance. If you haven’t defined this category deliberately, the resulting feeling of being compelled will be unaccountable. It may even seem to represent the intervention of a supernatural being.
5.4 Side effect #2: Rules magnify lapses. Successful obedience to personal rules makes further success in the same circumstances easier; but failures may result in specific areas where you don’t dare to risk the stake of your personal rule again—lapse districts.
5.5 Side effect #3: Rules motivate misperception. The expectations that a personal rule puts at risk create incentives to avoid noticing a lapse, lest losing this stake damage intertemporal cooperation. There are two obvious means to do so before you’re even consciously aware of the risk: to divert your attention when you first begin to suspect that a choice is a lapse, before that diversion itself would demand your notice (repression); or to acknowledge the facts but re-interpret them to not constitute a lapse (denial). If you use these tactics deliberately they would be called suppression and rationalization, respectively. Repeated use of any of these dodges will inevitably reduce the expected stake of your rule—because they make the rule less effective—and may impair your perception of reality.
5.6 Side effect #4: Rules don’t necessarily serve your longest range interest. Personal rules recruit the most motivation when their criteria are countable and easy to test. This gives explicit plans an advantage over subtle ones, which might be richer were it not for the risk that they might open the way for lapses. For instance, a strict, explicit diet might win out over a rule to stop eating when you’re no longer hungry, because the latter rule is ambiguous. Since the 1800s many philosophers and psychologists have been suspicious of making personal rules a paramount strategy.
Combined motives for repressing lapses and partitioning off lapse districts may cause dissociation. Dissociation is prolonged mental (attentional) blocking, leading to your unawareness at some times of what you experience at other times. Examples range from “blackouts” to multiple personality disorder. Conventional theory says that dissociation (A) serves to get away from disturbing thoughts, including conscience; but dissociation could also be (B) a way of protecting personal rules or even (C) a way of serving long term interests by evading overly strict personal rules.
Dissociation may sometimes serve your long term interest. To the extent that you can’t revise your personal rules to be compromises between a present desire and your long range intentions, dissociation offers a crude expedient (situation C).
5.7 Summary: Side effects limit the rationality of using willpower. Self-control is not the same thing as rationality. Rationality is best defined as what serves your longest range interest, but because of the danger of rigidity there is no general strategy that you can systematically pursue to serve it. However, the high tails of hyperbolic delay discount curves do predict that personal rules and other forms of foresight can lead your long term interests to prevail.
6.1 Recursive self-prediction creates freedom of will. When the value of options as test cases affects their choosability, even your imminent choices are unpredictable from a knowledge of your prior incentives. Even by you. This unpredictability, and your consequent participation in recursive self-prediction, fulfill the usual philosophical requirements for having free will.
6.2 Newcomb’s Problem: How diagnostic thinking becomes causal. Newcomb posed his famous two box problem to demonstrate the psychological effect of a belief in predestination. My take: Avoiding temptation shows you are a one-boxer (or good person), and this remains true even if you’re aware of avoiding temptation to pass the test. Two boxers (or bad people) are those who can’t pass it even with this awareness.
6.3 Determinism does not negate moral responsibility. A deterministic chain of prior causes is sometimes argued to excuse people from social blame. By contrast, self-blame from breaking an intertemporal bargain* happens automatically when it reduces your expectations for future self-control. Perhaps social blame is vicarious self-blame—“if I were in her place I would blame myself”—and thus also compatible with strict determinism.
Conventional theory says that pain and negative emotions must have a selective principle different from reward, such as conditioned association, to explain why you have strong urges to pay attention to painful experiences. On the contrary, hyperbolic discounting explains how all mental processes may be governed by temporal patterns of a single selective principle, reward, which is variously experienced as a welcome attraction, an unwelcome urge, or a mixed temptation. There is a wide range of experiences that we don’t want but which we’re drawn into, ranging from seemingly senseless compulsions to obeying the urge to panic. Hyperbolic delay discounting suggests that such experiences are related by how long their rewards are typically dominant. At one end of the continuum is overcontrol—You find yourself trapped in personal rules that cost you harmless reward, by requiring you give impulses too wide a berth. The mixture of attraction and avoidance is clearest in impulses, temporary preferences with finite durations-- for instance for consuming something addictively. At a distance you prefer to avoid the activity, but you strongly want to seek it when it’s close. The briefest conscious attractions are itch-like activities that you keep engaging in even while not enjoying them. Finally there are mental activities that you’re never aware of even seeking, but that capture you by attracting your attention—the negative emotions and pain. All of these attractions trade in the market of reward, in its broad definition – whatever makes the mental process it follows more likely to recur – but their different time courses create very different experiences.
7.1 Impulses are activities that you value negatively, but consciously prefer for a while. The most obvious consequence of hyperbolic delay discounting is temporary preference for addictive activities. Once you discover the costs and try to avoid them, the duration of your preference for the activity becomes limited to periods varying from minutes (for forbidden cigarettes) to weeks (for some kinds of binge). Arousal of appetites or emotions may augment your impulsive motivation, but even this arousal depends on reward. The topic of addiction will get a whole section later.
7.2 There are options you never consciously prefer but which are hard to turn down when very close (itches). There are many kinds of unpleasant urge that go away if you never respond them, but which people continue to obey. Examples include tics, mannerisms, obsessive thoughts, those itches that lack a physical cause, and consumption of an addictive substance near its satiation point. Similarly, laboratory subjects may persistently self-induce brain stimulation that they dislike.
7.3 Itches are brief recurring impulses. Itch-like activities produce a combination of SS reward and longer-term nonreward, making them “wanted but not liked” – psychologist Kent Berridges description of an option such as brain stimulation that a subject seeks to repeat only if it will be immediate. (Berridge, 2009) These activities follow the same cycle of seeking and avoiding that addictions do, but the wanting is renewed much faster.
7.4 Pains may consist of very rapidly recurring impulses. Even pains are not the direct opposite of pleasures. The observation that pains attract attention but not physical approach suggests a cycle of brief reward alternating with longer nonreward—the same cycle as an addiction or an itch but alternating so rapidly that they are experienced simultaneously, as in the flicker fusion that motion pictures depend on. There could be other models that describe how reward and nonreward fail to integrate at very short delays—why they don’t just average out-- but such models would still have to account for how an aversive experience competes for your attention.
Not even pain can get a foothold without luring you. Experiences such as physical pain, panic, and distressing intrusive memories are neither liked nor wanted, but still require your participation. People can even be taught to withhold the emotional (“protopathic”) component of pain or their panic response to phobic objects-- Thus even processes that are supposed to be entirely negative must gain entry into the marketplace by giving you urges—incentives-- rather than by some automatic trigger like a reflex.
Historical background: the age-old dichotomy between pain and pleasure is inadequate. Many writers have divided motives simply into seeking pleasure and avoiding pain, but since Plato some have described a category of flawed pleasures, and, since Freud’s death wish, an attractive component of pain itself. Even today, however, there is no agreement about how the attractive component of pain is integrated with aversion.
7.5 A unifying rationale: Unwelcome behaviors differ according to how long they are preferred. Preferences that are unwelcome in the longest view have a continuum of durations:
7.6 Brain regions may specialize in evaluating different delays, before being integrated. (As of 2014) Within specific brain pleasure centers such as the ventral striatum and the medial prefrontal cortex, evaluation of rewards at different delays seems to be localized. However, overall activities in the main reward centers track expected rewards at all delays, and their activities decline with delay hyperbolically at the same rates as each other. Furthermore, pain centers largely overlap pleasure centers, suggesting some commonality of the selective process—reward as the universal basis of selection.
This section is not essential to the rest of picoeconomics. It points out the flaws in a longstanding psychological dichotomy, and argues that hyperbolic discounting makes this dichotomy unnecessary to begin with. You can skip to Humans generate most of their reward endogenously without losing any information you need.
Conventional theory holds that there are two separate kinds of selective factors for behaviors, rewards and the triggers of reflexes (“unconditioned stimuli”). You seek rewards, but in conventional theory you are said to have involuntary responses imposed on you by hardwired reflexes activated by stimuli—or, through association (conditioning), by predictors of these stimuli. The stimuli that are supposed to produce hardwired responses are called “unconditioned,” the predictors are called “conditioned.” The kinds of responses said to be selected by conditioning include the arousable appetites and emotions. Significantly, all the processes that can serve as conditioning stimuli have been pointed out to also have motivational value (Miller, 1969). That is, they can serve as rewards or punishments for behaviors. However, since the motivations are often negative, such as pain, panic, and anger, it was always thought that people (and nonhumans) would never choose them-- so they would have to have peremptory power to elicit responses reflexively. The existence of a hyperbolic discounting rationale makes a separate “conditioning” factor unnecessary: Negative experiences can lure you into participating in them by a mixture of reward and nonreward,
8.1 Conditioning does not explain involuntary behaviors. We have a wide range of involuntary, ostensibly conditioned, behaviors. However, “conditioned responses” are not just transferred reflexes, but behaviors with new specific components, which must have been shaped by a selection mechanism—some unknown third mechanism if this is not reward.
8.2 Supposedly conditioned responses such as appetites and emotions are based on reward. Emotions and other involuntary mental processes are conventionally supposed to be forced on us by conditioning, that is, by the pairing of new stimuli with innate hardwired releasers; but this explanation relies on a crude analogy to laboratory conditioning procedures. Instead there is evidence that emotion is a reward-seeking mental process. This can be partially depicted by a mechanical model.
8.3 Even vegetative bodily responses are at least partially reward-seeking processes. All options that involve the obedience to urges function within the internal marketplace. For some behaviors the urge is noticeable only when absent or opposed, as in the urge to breathe; for some it only partially affects a hardwired process, as when vegetative functions such as heart rate and blood pressure are altered by biofeedback or hatha yoga, but will continue without motivation.
8.4 Tastes are preferences for appetites, based on how those are rewarded. The inborn wiring that makes some kinds of reward dependent on arousal was probably an early evolutionary strategy to counteract the hyperbolic overvaluation of immediate rewards. Where activities need an aroused appetite to be rewarding, or where they become more rewarding with arousal, the urge for them can be avoided by avoiding arousal. Arousal is a reward-seeking process itself; avoiding it is a precommitting device. Complete avoidance of an appetite may result in a loss of taste for it, as sometimes happens with human alexithymia and sexual frigidity.
8.5 Systematic avoidance may prevent appetites from arising. Although arousal of appetite is usually regarded as involuntary, it responds to evidence about whether it will be satisfied. Arousal does not take much attention, but never getting its object will make even that attention not worthwhile—unless the appetite is rewarding in its own right, as in the cases of anger and sex. If you are entirely confident of not giving in to an appetite it will stop arising; whereas if you see signs of wavering, the appetite will be encouraged. Total confidence usually requires a long period of consistent abstinence, but can be created by an iron-clad personal rule (Dar et.al., 2005) or radical reinterpretation of your options (Miller & C’de Baca, 2001). By contrast, counter-conditioning does not keep addictive appetites from arising (Conklin & Tiffany, 2002).
8.6 Suddenly aroused appetites are positive feedback phenomena. In cases where a reminder arouses an appetite for consumption without predicting any increase in the availability of its object, even conventional theory could not say that the arousal has been literally conditioned to the reminder. This case poses a puzzle unless we recognize appetite as a reward-seeking process. Reward-seeking offers a mechanism for the explosive appetites that seem to arise from nowhere, or from mere reminders of their objects, a phenomenon that is not explained either by hyperbolic curves alone or by classical conditioning. The mechanism is another example of recursive self-prediction. Briefly, when consumption is limited by your own choice, noticing appetite offers your short term interests an occasion to challenge your impulse controls.
8.7 Negative appetites habituate less than positive ones. The readiness to be lured by unwanted emotions could be called negative appetite. We define negative emotions by our incentive to avoid or escape them. Nature may give them less tendency to habituate than positive emotions have, since they might otherwise stop warning people of harms; also, they do not habituate to the extent that we avoid them. Appetites for negative emotion are the basis of phobias, performance anxiety, hypochondria and other suggestible fears in ordinary life. Other emotions may be only partially negative: Anger to overcome more dysphoric emotions may be addictive, as less commonly may grief; and there are ways of provoking negative emotions that undo the habituation of positive ones.
9.1 Self-Control is only half of rationality. A society on the edge of starvation needs to make sure that people control impulses-- avoid dangers and save enough seed corn for next year’s crops. By contrast, a rich society has at least as much incentive to refresh people’s appetites, but this obviously can’t involve simply undoing the work of impulse control.
9.2 The most obvious non-physical reward is emotional. Most significant experiences include emotions. Conventional theory depicts emotions as packages of mental process, like reflexes, triggered by specific kinds of situation—anger by frustration, fear by threat, grief by loss, and so on. Certainly the components of a major emotion are innately linked, and there are particular situations that naturally facilitate one emotion or another. However, I have argued that emotion is best understood as a rewarding behavior that is itself selected on the basis of this reward. It is thus the most prominent kind of endogenous reward.
9.3 Reward power does not always depend on chains of secondary reward. The conventional theory of reward is that it must come from unlocking innate sources with specific turnkeys, “primary rewards,” and that “secondary rewards” get their power get their power by predicting this unlocking. That theory sees the success that people value—wealth, status, power, relationships—as a soft currency that must be backed by the hard currency of primary reward. This is an unwarranted generalization from laboratory findings. Reward in imagination can function as a fiat currency, as long as it is protected from the inflation that comes of too-ready availability.
9.4 Most human reward is endogenous, but competes on the same footing as external rewards. The internal marketplace works by trying out options in imagination. Most human reward comes from mental (endogenously rewarded) scenarios.*
9.5 Habituation is what limits endogenous reward. Endogenous reward is limited by exhaustion of appetite, and especially by a tendency to satisfy the appetite before it grows to significant size, habituation. Even where the potential for an appetite is undiminished, for instance in the longing that persists after a daydream, there is a refractory phase that briefly blocks the appetite—demonstrated in cases of brain damage where the absence of a refractory phase causes patients to perseverate the same thought endlessly.
9.6 Hyperbolic discounting creates the mechanism of endogenous reward. Mental processes that both reward and are rewarded might seem to be paradoxical, a circular and potentially explosive phenomenon. However, entertaining endogenous rewards at will prematurely reduces the appetite for them, so the most direct routes to them fade, as in daydreams. Arbitrary occasions for them lose out to routes that are limited by external occasions, which reward proportionately as gratification is delayed or frustrated. Hyperbolic discounting makes early gratification of appetite an SS alternative to waiting for the appetite to grow.
The demon at the calliope: a primitive model of endogenous reward. As a discipline to check the internal consistency of my theory of endogenous reward, I originally proposed a mechanical model in which a demon presses keys on a steam organ (calliope) to maximize the loudness of its sound. Loudness stands for reward, so it might now be understood as a hypothesis about activity in the ventromedial prefrontal cortex, together with whatever other centers feed into the parietal centers that govern choice (Glimcher, 2009). The model did not deal with hedonic importance, which would be information to the demon about how to best use occasions for key-pressing.
9.7 “Negative” emotions have hedonic uses. Although the staying power of the emotions we want to avoid is largely due to their lesser tendency to habituate {Negative appetites habituate less than positive ones}, we also learn to provoke them in order to undo the habituation of positive emotions.
Our culture’s lack of attention to appetite has been puzzling. Neither our culture nor behavioral scientists have tended to notice the need for appetite. This is at least partly because a reward is more conspicuous than its appetite.
Many goals ostensibly involve external rewards, but functionally depend on how the benchmarks to these goals affect appetite for them. Stripped of its imagery, a typical activity is like pure gambling—not needing money—which rewards outcomes according to how rare and surprising they are, and how much importance the person has invested in them. I propose here that the process of betting on singular and surprising occasions over time can generate very high levels of hedonic importance, the value of a fiat currency of endogenous reward. In subsequent sections I describe how we often find occasions for endogenous reward in benchmarks for instrumental reward and in empathic experience, and I propose how these factors come together in the experience of belief.
10.1 We cultivate endogenous reward by building up the importance of occasions for it. The importance of an activity is the degree to which components of the activity occasion endogenous reward, which in turn depends on how much importance you have bet on them. When the degree is high you often experience this importance as an emotion—for instance curiosity, suspense, hope, and doubt. At lower levels the importance is just interestingness.
10.2 The most effective occasions are singular and surprising. The hedonic importance of an activity will build in proportion as its occasions are singular and surprising—singular meaning that they both are infrequent and stand out from other potential occasions. For instance, the outcome of a sports event is more singular if it is a championship game, and more surprising if odds had been strongly against it. Endogenous reward is limited by appetite, so it becomes attached to external events in proportion to their singularity and their surprisingness.
10.3 Texture is the aptness of an environment to provide effective occasions. How well an environment provides good subjects for bets can be called its texture. A rich texture contains an optimal variety of potential occasions for endogenous reward.
10.4 The hedonic importance of bets depends on betting history.A history of repeated betting makes an occasion that has been bet on increasingly singular, and the increasing reward from this singularity motivates repeated betting, a recursive process. This positive feedback system differs from that of a personal rule in that it is not necessarily tied to long term outcomes, and may in fact build addictions, phobias, paranoid suspicions, and other psychiatric symptoms.
10.5 Hedonic importance is internally fed back, but not necessarily in regard to personal rules. The recursiveness that affects hedonic importance overlaps the recursive self-observations that are the basis of personal rule-making; but occasions for endogenous reward may grow and shrink in importance as mere tastes, without giving rise to the issue of self-control. A change in the hedonic importance of a bet doesn’t necessarily depend on how this change can be expected to affect future importance. Examples include loves and fads, and also the many processes that endure despite having a negative value in recursive self-prediction: addictions, obsessions, intrusive memories, phobias…
11.1 Instrumental steps often have hedonic value. Events in the environment can be valuable as both stepping stones to other events (instrumentally, as secondary rewards) and as occasions for endogenous reward (hedonically) at the same time. It is often hard in practice to discriminate these values.
11.2 Endogenously rewarded activities may parasitize instrumental ones, creating an incentive for indirection. Benchmarks of progress toward external goals may provide occasions of great singularity for endogenous reward. This effect creates incentive to falsely interpret the most hedonically useful benchmarks as being the most instrumentally useful, and to avoid detection of this shift.
Endogenous reward may be a factor in the obstinacy of addictive lifestyles. Some authors argue that addictive habits gain force by becoming “automatic.” But habits that become automatic are merely mindless, and do not resist correction by experience like addictions do. The challenge of getting away with forbidden activities can build endogenous reward, especially visible in shoplifting.
11.3 Another problem for RCT: Optimal satisfaction often requires indirection. Both instrumental and hedonically motivated activities may serve long-range interests—the former in getting goods that satisfy appetites, the latter in maintaining and refreshing those appetites. Deliberate optimization of this combination may be impossible in practice because of the need for indirection.
11.4 Summary of endogenous reward: It is governed differently from exogenous reward. The prospect of endogenous reward is discounted in the same way as external reward, but the determinants of endogenous reward are different. I propose a new framework for understanding endogenous reward.
11.5 Reward by imagination is an evolutionary work in progress. Superior intelligence confers adaptive advantages in computation and planning. But the rewarding power that lets imagination compete with concrete rewards can make people unrealistic. This may be a limiting factor in the evolution of brain capacity.
12.1 Vicarious reward is endogenous, which fully explains altruism. Care for other people’s welfare has often been attributed to moral self-discipline, but personal rules neither explain nor are necessary for the phenomenon of altruism. Vicarious experience gives rise to value in its own right, by providing occasions for endogenous reward.
12.2 Empathy is based on vicarious reward, not a reflex. Despite some evidence that our brains are innately prepared to relate other people’s situations to our own, empathic emotional responses are not automatic. Rather, potential emotional responses compete on the basis of their prospective rewardingness.
12.3 Vicarious reward is the richest source of occasions for endogenous reward. However helpful other people may be for instrumental purposes, they have a primary value in providing occasions for endogenous reward. Human relationships are apt to resist habituation because of their complexity and surprisingness. Most importantly, continuing interaction with particular other people builds the hedonic importance of the relationship with them, through both recursive interaction and recursive self-prediction.
12.4 Mental models of other people—geists—become interests in their own right. Vicarious experience creates the imaginary characters which are our perceptions of other people. The emotions that these characters have are physically “ours”—information from the others supplies only occasions—so the characters may form interests that seek expression even when the people aren’t present. Call these semi-autonomous models geists. They are the motivational basis of phenomena ranging from ordinary companionship, to transference and projection, to the felt presence of ghosts. [What Good Are People p9]
12.5 Our sense of fairness in competition comes from intertemporal bargains. In cases where hurting other people pays off in the short run but has a greater long run cost, personal rules can form to motivate fairness. Their force does not come from a specially evolved sensibility, but rather like any self-enforcing contract.
12.6 Personal rules may hinge on approval by mental models of others, which may include supernatural beings. Direct social control is a form of external commitment, but the mental models we make to predict living people may be valued when the people are absent or dead, or even just imagined. Testing choices by whether such others would approve, or be distressed or angry, can effectively create a personal rule, but one that may “forgive” lapses in a way that a logical principle cannot, potentially preventing the bankruptcy of personal rules.
12.7 Negative empathy is also rewarding, perhaps as a form of self-control.Others’ pain may serve as an occasion for endogenous reward. You may cultivate such negative empathy as a warning for your own behavior, but it is not apt to be rewarding in the long run. [What Good Are People p11]
13.1 Top-down explanations are intuitively appealing but limited. Our self-image depends on our higher mental functions: understanding, planning, willing, and taking responsibility as autonomous agents, as opposed to just being buffeted by lower processes like desires and fears. But this image is short on specifics. How do these subtle processes get there, and what moves us to adopt them? Hyperbolic discounting predicts that internal market forces build higher processes from the interaction of lower ones; and it defines “higher” and “lower” by how farsighted they are. Roughly, higher processes are those that have learned to broker access to lower processes.
