Endogenous Reward, 2013
Ainslie, G., (2013) Grasping the impalpable: The role of endogenous reward in choices, including process addictions. Inquiry 56, 446-469. doi: 10.1080/0020174X.2013.806129
http://www.picoeconomics.org/HTarticles/Endog/Endog2.html

 

`

 

Grasping the impalpable: The role of endogenous reward in choices, including process addictions

 

George Ainslie
Veterans Affairs Medical Center, Coatesville PA, USA
University of Cape Town, South Africa
George.Ainslie@va.gov

 

Presented at Agency and Addiction
Center for the Study of Mind in Nature
Oslo, Norway
November 10 – 11, 2011

 To be published in Inquiry, 2013.

This material is the result of work supported with resources and the use of facilities at the Department of Veterans Affairs Medical Center, Coatesville, PA, USA. The opinions expressed are not those of the Department of Veterans Affairs of the US Government.

Abstract


The list of proposed addictions has recently grown to include television, video games, shopping, day trading, kleptomania, and use of the Internet.  These activities share with a more established entry, gambling, the property that they require no delivery of a biological stimulus that might be thought to unlock a hardwired brain process.  I propose a framework for analyzing that class of incentives that do not depend on the prediction of physically privileged environmental events:  People have a great capacity to coin endogenous reward; we learn to cultivate it, and, where it is entrapping, to minimize it, by managing internally generated appetites for it.  The basic method of cultivating endogenous reward is to learn cues that predict when to best harvest the reward that has been made possible by the growth of these appetites.  This hedonic management occurs in the same motivational marketplace as the instrumental planning that seeks environmental goods in the conventional manner, and presumably obeys the same laws of temporal difference learning; but these laws are no longer what is limiting.  Furthermore, instrumental contingencies often provide the most productive structure for hedonic management as well, for reasons that I discuss; but the needs of hedonic management create incentives both to pursue instrumental goals in a suboptimal manner and to avoid noticing how the hedonic incentive affects this pursuit.  The result is the apparent irrationality that is often observed in process addictions.

 

I. ”Process” reward is endogenous

Addiction is now a preoccupation of behavioral science, undoubtedly because addictions to substances are the greatest preventable cause of death—in youth (Robins & Regier, 1992), or, if we adopt the recent insight that addiction to food should be included (Gearhardt et.al., 2011), in adults generally.  Addiction was long thought to be a physical process, in which specific substances were necessary to unlock reward centers in the brain, leading to an overriding motivation to ingest them, as well as to tolerance on repeated ingestion and to withdrawal symptoms when ingestion ceases.  This model may have delayed the recognition that substances are not necessary for addictive behaviors—or even for tolerance and withdrawal (Blaszczynski et.al., 2008).  However, in the 1980s the potential of activities unassociated with substances to become motivational traps began to be recognized (Marks, 1990).  The list of frequently proposed addictions has recently grown to include such substance-unrelated activities as television, video games, shopping, day trading, kleptomania, and use of the Internet, which share with a more established entry, gambling, the property that they require no delivery of any stimulus that might be thought to unlock a hardwired brain process.  Many writers are still uncomfortable with classifying them as addictions, undoubtedly because there is no bright line that separates these conflictual preoccupations from mere bad habits, or, imaginably, even “positive addictions” such as playing a musical instrument (Becker & Murphy, 1988).  But even with the consumption of substances, no such line exists; certainly a person’s apparent inability to stop even opiate addiction is only relative (Heyman, 2009).  As inconvenient as this conclusion is for legal and moral debate, strongly motivated activities that the person herself wants to avoid seem to occupy one end of a continuum, not a distinct category.  There is no reason that any intensely motivating activity cannot become addictive.  The puzzle is how non-substance goals become intensely motivating, when behavioral science has no clear idea of how they become motivating at all.

Much of what people seek in general is independent of concrete goods, or is connected to such goods only arbitrarily.  This is commonplace, but only the controversy over how seeking biologically neutral events can become addictive has forced behavioral science to confront the phenomenon.  “Process” goals are usually external events, successes of one kind or another, but the person must have endowed these events with meaning, and the constraints on this endowment remain unknown.  Students of process goals have mostly been social psychologists (e.g. Gollwitzer & Bargh, 1996), who have avoided reductionism—the search for mechanisms that can be built of simple elements.  Reductionist approaches to choice such as temporal difference learning have achieved great precision and have increasingly identified neural correlates (e.g. Dayan et.al., 2006), but for non-concrete goals the field has not moved beyond the old behaviorists’ chains of prediction, whether secondary reward or computational valuation, built toward innately rewarding external events (e.g. Skinner, 1953, pp. 76-81).  Even allowing for generalization, the connection of common endeavors with such events often strains credibility.  .

The conceptual device that has been lacking is a reductionist framework for analyzing those incentives that do not depend on the prediction of physically privileged environmental events.  I start in the same place as behavioral psychology and economics:  All choice is determined in an internal marketplace that weighs competing options according to their expected generation of a selective process—call it reward. Reward from a given source may sometimes comprise disparate components, but it always has a valence that can be weighed against reward from other sources on a common dimension (Ainslie, 1992, pp. 28-32; Montague & Berns, 2002; Shizgal & Conover, 1996).  Parametric experiments have suggested that a hyperbolic shape characterizes all organisms’ inborn delay discount curves, resulting in a tendency to form temporary preferences for imminent but inferior rewarding events (Ainslie, 1975, 2005; Green et.al., 1994; Kirby, 1997).  This tendency is accentuated by sudden surges of appetite that depend on recursive self-prediction when a person is not entirely confident of controlling herself (Ainslie, 2010).   I have also made an argument for going beyond conventional reward theory to include among rewards the conditioned stimuli that appear to govern involuntary behavior and attention, ascribing their automatic and sometimes aversive qualities to an effect of hyperbolic valuation (Ainslie, 2001, pp. 48-70; 2005, 2010).  That argument is relevant here only because I will include whatever draws us to attend to aversive experiences in this article’s discussion of reward. 

What I am proposing here is an approach that recognizes the great human capacity to coin endogenous reward.  Our huge cortical apparatus does not just evaluate rewarding stimuli, but directly activates whatever brain centers create reward-- indeed, that the large, overlapping cortical areas that are active in autobiographical memory, fantasy, and prospection (Buckner et.al., 2008; Spreng et. al., 2009) must generate reward in order to compete for our attention.  When this reward is conspicuous we often name it an emotion (see Rick & Loewenstein, 2008), but we can also be aware of motivation that does not have much emotional quality, which we describe in terms such as importance or interest.  Our experience of this reward is often, indeed usually, attached to an external event; but this does not mean that the event either has intrinsic rewarding properties or predicts the occurrence of another event that does.  What I propose is an alternative rationale for this attachment.  It falls far short of the parametric models that have been developed on the basis of predicting extrinsic rewards, but I hope it opens the way to explore rewards that are constrained differently.

 

II. Endogenous reward is limited by appetite(s) for it

Behavioral scientists began to think about reward as a discrete mechanism around the time of William James, but the first to look at how we might generate our own reward was the psychologist Raymond Dodge.  Dodge had just begun to study the phenomenon of “autogenic reinforcement” (1917, p. 103) when the behaviorists banished introspection as a source of observation, making him not only first but just about last such psychologist, until educational psychologists recently took up a similar task (Renninger & Hidi, 2011; Silvia, 2006).  He thought it was “quite as significant for psychology as the action of adrenin to which Cannon has introduced us” (ibid.).  The property that offered him a handle on this process was its limiting factor, “fatigue,” which he thought might be characteristic of all nervous tissue.  “The routine is regularly less alluring than the unusual.  Mankind in general prefers new scenes, new plays, new walks, new jokes, new styles, new investigations.” (ibid., p. 104).  He ascribed this observation to a “relative refractory phase” in the relevant mental process.  “Repetitions of all sorts seem to be avoided whenever practicable.  The repetition of questions, courses, lectures, phrases, and even words is possible enough, but except for special reinforcing circumstances, it is postponed until the effect of the initial case is somewhat worn off” (ibid., p. 103).  Certainly such a refractory phase occurs in mental activity, as shown by the perseveration syndrome that occurs in its absence—some brain-damaged patients repeat the same phrase hundreds of times in a row, as if they had freshly come upon it each time (Sandson & Albert, 1984).  But this disinhibition of perseveration shows also that the refractory phase does not come literally from neuronal fatigue. 

Dodge missed the important distinction between the rewarding power that is renewed by rest—as in recovery from a refractory phase—and the rewarding power of a learned sequence that is never entirely renewed, which is actually characteristic of most of his examples: “The popular song, the clever phrase, the good joke, soon finds us refractory to the point of desperation” (ibid., p. 104).  Rest never restores to such rewards the impact that they had when new.  The loss of rewarding power because of learning is not fatigue but habituation. Dodge did observe a telling counterexample to the decline of rewarding power due to repetition, that “one of the marks of good art is the constancy of its appeals.”  In contrast to the transitory examples he had just given, “the great classics in music and literature may be heard over and over with increasing satisfaction” (ibid, p. 104).   Here he raised a question that still must be answered:  When we have learned how to harvest endogenous reward from, say, a good story, why do we not rehearse this story repeatedly, getting better and better at experiencing its rewards each time?  Conversely, why are we able to do just that with some stories, pieces of music, and paintings?

I have suggested a mechanism elsewhere (Ainslie, 2001, pp. 164-174; 2003), the gist of which is that our reward from imagination is limited by some kind of appetite, exhaustion of which is Dodge’s “fatigue.”  Hyperbolic delay discount curves make us impatient to think ahead in whatever scenario we are rehearsing, giving us the urge to harvest a small amount of appetite early rather than waiting for a larger harvest later.  Thus, for instance, experimental subjects who receive unexplained gifts enjoy them more than when their curiosity is immediately satisfied (Wilson et.al., 2005).  We have the same problem of appetite management with a physical reward in unlimited supply-- If we can eat whenever we want we have an incentive to set up limits to snacking or grazing, so as to let appetite build-- but the consumption of exogenous rewards is easily controlled, while thinking ahead is not.  Learning to anticipate is habituation, a valuable process when learning a task that will produce reward somewhere else, but a limiting factor for endogenous reward.  Art arguably functions by creating a complexity or subtlety that prevents complete anticipation.

Although the key to the effectiveness of endogenous reward is the appetite for it, this appetite is so impalpable that no common name seems to specify it well.  Sometimes there are names for particular appetites within this category: curiosity, suspense, impatience, a wish to resolve ambiguity.  In general this appetite could be called a sense of importance; but this may differ radically from a person’s objective judgment of importance.  It might be called interest or even interestingness-- to distinguish it from the meaning of her objective or “best” interest.  Interest has begun to be studied again by educational psychologists, but not usually with reference to the concept of reward, which is taken to imply external manipulation (e.g. Ryan & Deci, 2000, p. 16; Silvia, 2001, p. 278)—an unnecessary theoretical limitation for which behaviorism has only itself to blame.  The concept of interest in educational, vocational, and aesthetic psychology is clearly what we are discussing.  A student’s interest in a new content area is first “triggered” as a “situational interest” (Renninger, 1992)—presumably by relevance to an existing interest—and then develops in stages into an “individual interest” in its own right.  Renninger and Hidi even say that interest functions as a reward, and note its correspondence with activity in brain reward circuits (2011).  Still, when the authors in these fields speak of intrinsic motivation they just mean whatever governs free choice among interests in various topics; they do not ask where this motivation comes from.

Both Dodge’s and the educators’ examples are complex, involving sensory stimulation, social considerations, and larger purposes, which might admit of many explanations.  To get closer to the mechanisms involved, we need to simplify.  Many illustrations of the minimal requirements of endogenous reward are provided by the psychology of boredom, which has recently been augmented by social psychology experiments on strength of will.  Here a subject must stay vigilant on a simple, repetitive task such as detecting letter patterns, adding numbers, or reporting stimuli while disregarding a prominent quality (as in the Stroop task, reading color names printed in a different color), thereby depleting her willingness to do similar tasks and eventually her ability to do them correctly (Hagger et.al., 2010).  Subjects get an external reward—money or course credit—for the whole task, but need to find ways from moment to moment to keep their attention from straying (Sansone & Harackiewicz, 1996).  In analogous real life tasks even an overwhelming reward contingency, such as detecting or failing to detect enemy airplanes on a radar screen, has often failed to maintain vigilance (Mackworth, 1948).  Presumably what ongoing vigilence requires is finding, or generating, endogenous reward with minimal input.

Vigilance tasks still involve contingencies that are externally imposed.   It will be better to take as our exemplar a solo activity with no exogenous consequences—for instance, the game of solitaire.  I create my appetite for solitaire out of a small, unnamed purse of motivation that is at my arbitrary disposal.  I make the game important to the extent that I bet this motivation on an unknown outcome.  As I play, the interestingness or importance of the game increases; in effect my bet increases.  The words are generic, but suggest something about the nature of the process:  I could say I have taken an interest in the outcome, or that I have made the outcome important, at least to the extent that it commands my attention.  The significance of these words is in their ambiguity about my degree of control.  Once I have taken an interest in the game I find it interesting.  Once I have given it importance, that importance lingers to be perceived when I re-evaluate it.  In effect, I can pull up a rewarding activity by its own bootstraps—but I am constrained to do this gradually.  If an eccentric billionaire offers me a million dollars to make the next turn of the cards extremely important to me, and will pay me for physiological evidence of this importance regardless of what card comes up, I will be unable generate the necessary suspense.  Yet in stages people cultivate the importance of intrinsically worthless outcomes all the time.

Once created, the interest or importance has an inertial quality, as is evident in a game that is more complex than solitaire, or that can be a component of some larger game.  If I do Sudoku puzzles my appetite may grow with repetition, so when I begin a new game I find it more important than I found an earlier game.  I can add or take away importance deliberately, but my appetite is also intrinsically limited by some particular properties.  If I play several games in a row my interest will wane; it will become less important to me. This is Dodge’s “refractory period.”  Conversely, if I conclude that the game is a waste of time and forbid myself to play any more that day, the importance I formerly built is refreshed, so the opportunity to play becomes a recurring temptation.  And as I become a better player, puzzles at a difficulty level that was once exciting become boring, so I must find tougher ones; this is habituation.  The process of investing importance in activities lacks an inclusive name.  It seems to be what Freud meant by a person’s directing libido to “sit on” an activity, which Ernest Jones translated as cathecting it (Freud, 1916-1917, pp. 335-357); but this has complex theoretical connotations.  By analogy to our solitaire example I will refer to it here as betting on the activities.

 

III. Endogenous reward is paced by occasions

Solo games are trivial examples, but illustrative because they cannot be interpreted as cues that predict some other reward.  They reveal something about the basic machine language of endogenous reward, particularly that we can make an elementary game out of whatever cue-response pattern we choose to designate, but also that we cannot arbitrarily designate whether the game will build, sustain, or lose our interest. To pay even a modicum of attention requires a modicum of interest, i.e. appetite for the relevant endogenous reward.

Given that there is no social or monetary contingency involved in solitaire, the difference must be in how it generates occasions for endogenous reward:  Turns of the cards are good or bad, and moves are clever or not, partially because of how they augur for the outcome but also because of their mathematical pattern:  Is the run of cards rare, is the choice of response difficult?  Each turn, and response, creates an occasion for more or less reward.   The setup of an endogenous payoff is an ongoing, recursive process: The player starts out taking a little interest in the game and becomes more interested as it gives her occasions for reward—not proportionately to the raw frequency of these occasions, but to the product of frequency and intensity of the resulting reward, the latter in turn decreasing with frequency—thus a minimax effect.  Were this game to become a major source of reward for her, the word importance would become appropriately substitutable for interest:  She gives the game importance, and to the extent that she has done so, finds it important.  This interestingness, or importance, becomes emotional capital, a resource for generating endogenous reward. 

Even in the simple example of solitaire, the constraints on harvesting reward confront us with a mystery.    If a player can win by making one illegal play she will have an urge to make it.  What keeps her—usually-- from falsely claiming an occasion for too much reward in the moment, because of her hyperbolic over-valuation of the near future?   The integrity of her basic betting pattern must be protected by the threat of imminent nonreward if she cheats.  In practice she soon learns that cheating deflates the importance of the game.  The dynamics of building and maintaining interestingness seem to be the same in that respect as I have discussed elsewhere for cooperation in intertemporal prisoner’s dilemmas (Ainslie, 2010), except that here the stake is created by her history of cooperation itself-- the stake is the interestingness that she has created by taking an interest.  Such a model of endogenous reward that is paced by adequately rare occasions is a radical departure from the conventional model, in which even seemingly endogenous reward is based strictly on how each stimulus pattern predicts an exogenous reward-- that is, on its value as a secondary reward.  A theorist who depends entirely on exogenous reward would have to see the win at solitaire as predicting some larger success in life, and see this success as valued because it predicts some ultimate turnkey reward.  Granted that steps to later goals often become goals in their own right; if this happens just through association or conditioning, these steps should lose their value to the extent that they stop predicting the later goal.  When a secondary goal acquires value independently of its predictive value, an explanation is needed. 

If, instead, we see endogenous reward as an active process—as a behavior, of sorts—then its effective performance depends on appetite.  The well-worked-out laws of temporal difference learning still apply, but when extrinsic rewards are not limited, maximization of their ultimate selective factor, the internal reward process, depends on optimizing appetite.  Temporal difference learning of the cues that predict reward in this circumstance are unexplored, although a recent Bayesian model of how a person’s expectation of future motives affects her choice includes appetites such as dread and regret (Pezzulo & Rigoli, 2011).  We have to speculate that during the preschool years when a child’s experience is largely fantasy she learns that to harvest reward totally ad lib spoils the harvest almost immediately.  That is, to do the self-reward behavior without an occasion fails in general because it undermines appetite, and by the same token some occasions are better than others at indicating where to exploit appetite.  “Magnification” of interestingness with practice does not take place by association, as its students have assumed (e.g. Silvia, 2001), but by finding new and more productive occasions for reward.  As for the other side of the coin-- entrapment into aversive experiences by brief reward-- negative appetites survive because they have much less tendency to fatigue than positive ones.  The disliked appetites that draw us into phobias, for instance, or “the pain of paying” (Prelec & Loewenstein, 1998), intrusive memories, hypochondria, paranoid fear, or obsessional doubt—do not “demagnify” with repetition, like boredom, as Silvia has it (2001, p. 284), but will stay punishing unless they are either avoided like temptations or worked into strategies for augmenting other appetites—as they sometimes are in horror movies or stormy romantic relationships.

We might ask how much motivation can be built up without any external incentive at all.  Real life examples exist.  The radical isolation of the Christian anchorite saints was probably one. Some schizotypal and schizophrenic patients stay in their rooms and withdraw into fantasy worlds.  However, the clearest illustration is the fictional example that novelist Robert Coover created: J. Henry Waugh, a lonely accountant who interests himself exclusively in a lifelong dice-driven game of fantasy baseball (Coover, 1968).  Henry increasingly ties his emotional reward to the career of one of his players.  When a rare throw of the dice requires him to kill the player with a pitched ball, he experiences an extreme version of the solitaire player’s urge to cheat, a crisis that overwhelms his life.  If Henry existed someone would surely say he was addicted to fantasy baseball.  He would also be called schizotypal, and might be a good model of a video game addict. His value as an illustration is that he created colossal importance out of throws of dice that had no external payoffs. 

As an example of cultivating importance in total isolation Henry’s case is extreme, but if we enrich our illustrations just as far as including spectator sports, examples abound.  Here again the first requirement is that people take an interest.  The moves of a hockey player weaving up and down the rink have to seem more important than the moves of a tropical fish swimming about a tank.  And again, having given importance leads to finding importance.  Our investment of importance in the sport builds; increasingly it lets us get reward from a win, but commits us to endure disappointment at a loss.  A recent essay gives a good description of the incentives for “[investing] a huge amount of emotional capital” in baseball:


It allows you to feel real emotional investment in something that has no actual real-world consequences.  In any other contest (presidential campaigns, for example), the outcome can be exhilarating or dispiriting to its followers and, by the way, when we wake up the next day, the course of history has been changed.  As for fictional stories, you can certainly get swept up in them, but their outcomes don’t hinge on the unpredictability of real life.  Sports stories, on the other hand, are never guaranteed to end happily.  In fact, as we’ve seen, some end in a highly unsatisfying way.  As a fan, you will feel actual joy or actual pain—this is precisely what non-sports-fans usually ridicule about being a sports fan—in relation to events that really don’t affect your life at all.  (Sternbergh, 2011, 18-20)


Events that give no information about exogenous rewards can become great hedonic forces just by serving as occasions for endogenous reward.

 

IV. The best occasions are singular and surprising

Given interest, events within a game pay off in proportion to their singularity and their surprisingness.  Singularity is the combination of their rarity and the extent to which they stand out from other kinds of event that are less rare.  In baseball a hit by a pitcher at bat is more singular than a hit by a fielder, and a number of scoreless innings in a row is more singular than a haphazard sequence.  Announcers try to increase the appearance of singularity by finding ways that the current play is statistically unusual—the first hit by a left-handed batter against this pitcher this season—but the net effect is to deflate the singularity of similar statistical finds by increasing their frequency.  The game itself may be more or less singular: a real game more than one generated by a video simulation program, a game with your home team more than with another, a game in post-season playoffs more than a regular one, today’s game or one shown within the day more than a game from the past, even if you do not know the outcome of the past game.  In general, the emotional impact of the outcome of a gamble is inversely proportional to its probability (Mellers et.al., 1999). 

Surprisingness is important because anticipating outcomes undermines your appetite for them (suspense).  If you know the outcome of the past game its occasions will pay off less, and less still if you have previously seen the whole game.  The same principles can be seen operating in other spectator activities.  Encountering a singular Stephen Sondheim rhyme such as “the hands of the clock turn but don’t play a nocturne” occasions more reward than hearing moon rhymed with June, but most when you first encounter it, and of course proportionately as you have taken interest in musical theater.  News items occasion reward according to surprisingness but also according to singularity, as measured by how recently and how close to home the events occurred.  If you watch the news only once a day it will have more impact than if you frequent all-news channels; the hedonic deterioration from waiting for news to consuming “infotainment” at will is essentially the same as from belief to fantasy.

When psychologist D. E. Berlyne reviewed a behavioral implication of interestingness—attention-- from a behavioral viewpoint, he found that it was “selected for” (?rewarded) by four stimulus properties: novelty, uncertainty, conflict, and complexity (1960, pp. 18-44).  All four can be characterized as blocking anticipation.  Ability to participate in an activity also makes it more interesting, but by inviting you into specific bets on outcomes and thus increasing their singularity; it is more fun to play solitaire than to watch it being played.  Recent experiments have pointed out that the pleasantness of stimuli does not necessarily go along with their interestingness (Turner & Silvia, 2006), but this finding merely confirms that unpleasant experience can reward attention

 

V. Texture is the aptness of an environment to provide good occasions

Endogenous reward becomes arbitrary without occasions, and then interestingness cannot build.  In prisons solitary confinement is a major punishment, more aversive than the din of a cellblock (Suedfeld et.al., 1982).  Experimental subjects deprived of sensory input develop psychotic symptoms in a matter of hours (Bexton et.al., 1954).  Even a child whose fantasy life has only begun to habituate needs a ball thrown or a stick to manipulate or mud to sculpt.  Conversely, even one-year-olds can be observed to use minimal cues such as circles moving on a screen to occasion emotions (Johnson et. al., 2007).  With experience we gain skill at investing importance—betting-- but our results depend on the availability of good occasions in the environment, a property that might be called its texture.  Rich texture permits complex occasioning, for instance in a challenging puzzle or plot or human relationship.  Durable texture is not worn away by familiarity, for instance in art.  As endeavors and environments compete for our investment, this durability over time may be the determining factor for which endeavors survive.    In Dodge’s examples, the popular song, the clever phrase and the good joke are rewarding in the short run, but with repetition anticipation erodes their usefulness as occasions.  Tic-tac-toe (naughts and crosses) engages a child’s interest, but is replaced by draughts/checkers and then chess as the contingencies become predictable.  As she becomes more able to predict complex patterns of play, these become the elements of wider puzzles.  The most satisfying puzzles are those whose solution reveals new puzzles.  Silvia describes the same pattern of unfolding “cognitive conflict” in the “selective persistence of interest” (2001, pp., 280-281).
 
The texture of occasions for endogenous reward provides a framework on which interestingness can grow, much as a trellis attracts vines by increasing the surface area that can be exposed to sunlight. But in addition to the obvious differences, trellises merely have to increase exposure to sunlight with no appetite-like limitation, whereas good texture has to improve a person’s combination of appetite and reward.  Development of such texture for endogenous reward has become a growth industry.  For instance, video games have grown from being the distractions of a few minutes to foci of engagement that last for hours or days.  This growth has historical parallels in the growth of short literary forms such as ballads and stories to novels, but automated games add responsiveness to players’ moves, and research on the behavioral effects of various textures permits systematic increases in their motivational impact (Jackson, 2012; King et.al., 2011).  Manipulation of texture alone has brought to video games an endogenously rewarding potential similar to that of gambling for money (Griffiths, 2005).  Here we are reminded that, although many textures invite a growth of interest that increases long term reward, some may yield intense reward up front that is followed by emotional impoverishment, just as in substance addictions.  The case of gambling should have shown society that the absence of a substance does not preclude the development of addictive preferences.  Computers have created many new patterns of occasioning reward, including not only video games but also simulated reality, media on demand,  internet cruising, and social networking.  Entrapping effects have been described by Sherry Turkle and others since the 1980s (Turkle, 2011), but the most direct evidence for an impulse control problem is the popularity of precommitting programs such as Self-Control and Freedom, which lock out a person’s own access during specified times.  I will not catalog the activities where reward that is obviously endogenous may be addictive, but I will now point out that its role is not apparent at first in the field where it is spread widest.

 

VI. Background for indirection: Prediction of exogenous reward also creates singularity

So far we have not considered the activities that engage most people most of the time, the ones with instrumental value—those that obtain rewarding consequences without necessarily being rewarding in their own right.  We have been talking about hedonic value—the value of an activity in its own right—and within this category only endogenous rewards, those that are not governed by the expectation of stimuli that are hardwired to be rewarding (food, sex, pain…).  We have seen in commonplace examples that endogenous reward does not have the properties that have been demonstrated in exogenously based rewards.  However, this does not mean that the two forms of reward usually occur in isolation from each other.  Even well-off people do not spend most of their time in activities with no instrumental purpose. 

Identification of endogenous reward in the presence of external incentive is difficult.  There are many activities in which the determination of reward by exogenous events is arguable or simply unknowable.  To what extent is money an endogenous reward?  Psychologists Stephen Lea and Paul Webley showed several ways that money has a “drug” effect over and above its “tool” value as a means of buying goods, especially a “money illusion” in which people adjust money’s subjective value in accordance with its nominal value or even physical properties (2006).  The irrational impatience people sometimes report for money—annualized discount rates of thousands of percent (Ainslie & Haendel, 1983; Kirby, 1997; Rosati et.al., 2007)—may sometimes reflect simply an impatience to spend it, for instance in the case of customers for an English loan company that advertizes a 1734% annual interest rate (www.QuickQuid.co.uk, June, 2012); but recent experiments trading off small amounts of money against delays of just seconds have found even steeper discount curves—hyperbolic in one case (Wittmann et.al., 2010), ambiguous in another (Gregorios-Pippas et.al., 2009)—despite the fact that the money was not to be delivered until the end of the experiments.  The valuations were consistent with fMRI brain recordings of subjects’ striatal reward centers, but by definition reflected drug-like rather than tool-like utility, which Wittmann et.al. analogized to points in a video game (2010).  With longer delays experimental dissociation of money’s hedonic value from its instrumental value will obviously be harder.

The example of money raises the general question of how much we should count as endogenous those token rewards that are ostensibly exogenous.  Outcomes in activities such as solitaire are obvious because they lack any credible connection with exogenous rewards.  However, what about the large variety of activities that are performed in the name of exogenous rewards, but that are often so distant that their discounted value would be negligible?  Using conventional financial discount rates, planners of welfare wonder how investments over decades are ever motivated (Karp, 2005).  The relatively high tails of hyperbolic discount curves improve this picture only partially (ibid.), especially if we consider the discount rate for actual experiences, which may be close to those of nonhuman animals.  Experiments with middle class subjects that use money as a reward are offering surplus value, not something that affects the subjects’ current comfort.  The discount rate for obviously exogenous rewards would be how much bad traffic, smoking deprivation, boredom, or noxious noise people would accept at a delay to avoid it in the immediate future.  At least in the case of noise, undergraduates choose immediate relief for a fraction of the duration they could get only tens of seconds later (Navarick, 1982; Solnick et.al., 1980), numbers similar to the cost in effort that pigeons will accept to postpone current effort (Mazur, 1996).  In one study of immediate versus delayed food, humans’ patience was on the same order of magnitude as chimpanzees’ (Rosati et.al., 2007).   Compared with the value of current comfort, the present value of a good that is twenty years away is vanishingly small.  When we seek a distant, expected reward whose discounted value is virtually nil, some process other than simply weighing it must be making it desirable.

Furthermore, distant prospects are often so unlike what we have actually experienced that they must be subject to extensive interpretation, or even wholly constructed: retirement, a dream vacation, getting a dread disease, getting a heavenly or infernal reward.  Discounting aside, explanation in terms of secondary or conditioned reward--by analogy to behavioral experiments--is not adequate for novel scenarios constructed from imagination.  In some cases an endogenous component can be shown to exceed the value that the expected rewards would have if imminent, as seen in the process of savoring desirable scenarios (Loewenstein, 1987) or dreading painful ones (Berns et.al., 2006).  It may arise even when the person does not believe the scenarios will actually take place, as in savoring the vacation that could never fit into her schedule or dreading the phobic object that she knows is not dangerous.  These believed-and-not-believed scenarios differ from fantasy to the extent that they are singular—licensed, as it were, by whatever has made them stand out from the common ruck of the person’s imaginings.  Such licensure may even lead her to seek signs of progress toward a goal that detain her from getting the goal itself, as when she prefers good grades in a subject to actually mastering it, or acquiring valuables for sale over actually selling them (see Sansone, 2009). The connection between current choices and the ostensibly exogenous rewards in the above examples is too tenuous to be accounted for by conditioned or secondary reward.  The obvious alternative is reward that is not induced, released, or limited by external events—reward that is endogenous.  Distant expectations are apt to be shaped by the endogenous reward they permit.  That is, when we can construct singular scenarios that map the future, these can serve as present occasions for reward, somewhat like the process that Rick and Loewenstein call “immediatizing” (2008).  The observation that hyperbolic value relationships are preserved within all ranges of delay must depend on the way we discern singularity, but this has not been specifically modeled (see Ainslie, 2006).  My point is that, although these choices are usually made in the name of maximizing some exogenous reward, this cannot be what they actually depend on.

VII. Endogenously rewarded activities may parasitize instrumental ones, leading to indirection

Even if we assume that activities with an expressed instrumental purpose are rewarded by the expectation of achieving that purpose—exogenously-- we should expect a component of endogenous reward as well.  Recall that endogenously rewarded interest tends to grow around occasions that are singular and surprising.  Occasions chosen purely for their hedonic value tend to deteriorate because their arbitrariness tempts the person to redefine them or replace them when they are paying off poorly.  This consideration suggests that occasions that are tied to actual productiveness in the real world will be more robust than those that are contingent on their near-term emotional effects.  Granted that a dedicated fantasist such as Coover’s J. Henry Waugh can commit himself to strict imaginary rules, and video games such as Second Life have discovered how to invite extensive commitment; in most people instrumental demands are apt to induce even stricter commitment.  Therefore steps toward exogenous rewards will often be the best occasions for endogenous reward.  The pursuit of exogenous rewards then offers two separate incentives:  One is the discounted value of the expected exogenous reward; the other is the endogenous reward occasioned by steps toward the exogenous reward. 

Here is the crux of why reward may function differently in people than in both nonhuman animals and in expected utility theory.  People seek external events somewhat as conventionally believed, but not entirely so.  As we seek events we learn that the pursuit itself creates occasions for additional reward:  As long as we follow a strict convention for belief, we can pursue distant goals and experience substantive current reward.  We can occasion reward with prospects that we have merely deduced as expectable.  As just described, we often experience more reward than the expected event would induce if present.  And we may find that the occasions we have authenticated as milestones can compete against the goals that provided the rationale for them to begin with.  However, these departures from instrumental efficiency must not be so great as to undermine our belief that they are realistic:  To cut them loose from the path to exogenous rewards would be to undermine their singularity, just like cheating at solitaire.

Furthermore, belief itself can depend on other factors than predictive realism.  I have argued elsewhere that the experience of belief—as opposed to make-believe—is the awareness of constraints on what we can choose (Ainslie, 2009, pp. 151-155).  Recursive self-prediction reaches equilibrium in personal rules, which, to the extent that we are committed by them, may be felt as facts.  The following example shows how they reward by singularity, quite apart from the ostensible rewards involved:

The belief that you have found a bargain can be instantly rewarding…  However, maintenance of this belief requires behavior that is consistent with it.  If you have stocked up on a food at a good price or bought a concert series at a discount, you may face an incentive to eat the food when you are tired of it or attend a concert you do not expect to enjoy in order to avoid recognizing a loss.  And yet you may be fully conscious of the unpleasant prospect.  The belief in the bargain is really a personal rule for playing a game, the wins in which occasion [endogenous] reward that is related only tangentially to the reward of tasting the food or listening to the concert.  The relationship is that the prospect of this consumption authenticates the bargain-hunting as an instrumental activity rather than a mere game, even though, once so authenticated, the bargain-hunting is a self-sufficient source of reward and has requirements that sometimes contradict those of optimally consuming the ostensible reward (ibid., p. 154).

Differences in the principles governing exogenous and endogenous reward create two kinds of motive—to maximize instrumental effectiveness and to refresh appetite.  Even though both may serve long range goals, they tend to be incompatible.  Instrumental effectiveness is best served by willpower, which lets us get to well-defined goals as efficiently as possible (see Ainslie, 2001, pp. 161-174; 2005).  Appetite is best refreshed by being confronted with delay and surprise.  The singularity of milestones in realistic instrumental plans make them excellent occasions for endogenous reward—but just to the extent that we have appetite for that reward.  If our plans have worked so well that they no longer excite us, conventional wisdom tells us to take this as the best possible accomplishment; however, from a hedonic standpoint it is suboptimal.  If realistic obstacles arise, creating suspense, the accomplishment will be less but the hedonic outcome may be better.  The hedonic problem, of course, is that we have been relying on the instrumental importance of our plan to make its occasions for reward singular, that is, more than just the results of a game or fantasy.   If we catch ourselves introducing obstacles to make the task more exciting, we belie that importance, a disincentive more significant than the obstacles’ immediate cost.  To maximize the plan’s effectiveness in pacing our reward, we have to believe both in its instrumental value and in the necessity of its obstacles.  Thus the experiences that have had the most sustained importance in a person’s life—the ones that she might say have “given it meaning”—are those that have entailed the most massive obstacles, experiences that she might never deliberately seek to have again (well illustrated in George Loewenstein’s rebuttal to a study basing well-being on the absence of unpleasantness—2009, especially pp. 92-100).  A major source of post-traumatic stress disorder in war veterans, besides traumatic memories, is difficulty in taking civilian pursuits seriously again (see Broyles, 1984).

Probably the largest component of endogenous reward is not what we indulge in deliberately for consciously hedonic reasons.  The largest component is what we cultivate in conjunction with instrumental tasks that depend on exogenous reward.  And we must expect these tasks to actually produce exogenous reward, unless they have some equally good claim to singularity-- a hallowed cultural imperative, for instance, or a longstanding habit. Once they can no longer be seen as instrumental (or otherwise dictated by a singular rationale), they lose most of their value as occasions for endogenous reward.  This consideration creates an incentive not to notice the endogenous contribution to the value of the activity, an incentive that pushes it into the Freudian dynamic unconscious; that is, it invites repression and denial of the endogenous motive.    

To restate this hypothesis:  Refreshing your emotional appetite without having to contradict what you have resolved often requires you to believe in some seemingly rational, or arguably necessary, activity that is incompatible with the direct routes to a reward.  That is, you need to find indirect routes to success: dummy activities that are not actually optimal for their ostensible purpose, but stay desirable insofar as they maintain appetite by creating good gambles.  In general you will need to believe in some larger quest that requires you to put your satisfaction at risk.  Instrumental and hedonic effectiveness can both be bases for choosing the same activity, but the hedonic component motivates you to under-perceive its own value and over-perceive that of the instrumental component.  

The incentive for indirection created by endogenous reward makes it hard to discriminate the two rationales in practice.  The best empirical indicator is probably the stubborn irrationality of the choices people make in ostensibly instrumental tasks—stubborn in the sense that people persist in displaying it even after extensive argument, reflection, and experience, that is, at “reflective equilibrium” (Slovic & Tversky, 1974).   Examples of obvious endogenous reward patterns embedded in instrumental activities are often trivial.  For instance, some people habitually risk being late.  Ostensibly they are optimizing use of their time: maximizing the value of [efficiency from tighter scheduling minus risk of objective loss].  However, they may also be getting an endogenous payoff from a solitary gambling game, the interestingness of which is proportional to the risk, and inversely proportional to the idle time spent after arriving early.  The determinant of actual behavior will be the sum of the exogenous and endogenous prospects. 

 I know of no controlled research on subjects’ paying more for outcomes than the outcomes are objectively worth—some analog of hunting for pennies off on gasoline, or of defending a belief in bargain purchases as in the above passage—but a recent experiment does show the value of having a rationale per se for an endogenously rewarded activity:  Students were promised one of two kinds of chocolate for turning in some paperwork, either at a nearby site or at a place that was a 15 minute walk away.  If (and only if) one kind of chocolate could be had just at the distant site, a majority of subjects chose to walk to it rather than stay idle—whichever that kind was, and even though the subjects had been found to value both kinds equally (Hsee et.al., 2010).  In the language of the present discussion, the experimenters’ suggestion that walking to a dummy goal had an instrumental purpose let subjects give it importance, a way for them to occasion endogenous reward for the task.  Significantly, the subjects who walked reported a happier experience than those who chose idleness, indicating that the mere pretext conferred value.  Elsewhere I have presented an argument that gambling for money is a paradigmatic example of an ostensibly instrumental activity that is actually dominated by the endogenous contingencies of reward, with the chance of winning money serving as a MacGuffin to give the activity a rationale (Ainslie, in press).

 

VIII. Summary: Endogenous reward is governed differently from exogenous reward

In this paper I have suggested that humans can generate a great deal of reward—the selective factor for choice—independently of any information predicting causes that lie outside of our control.  That is, we are physically able to coin reward through imagination, and this reward stands on its own, without needing endowment by an external releaser.  Except for its endogenous origin, this is the same reward conceived by classical behaviorism.  An ability to cultivate it jibes with common experience, but in conventional theory this ability should be impossible because of the circularity of reward’s selecting itself.  I have hypothesized that this kind of vicious circle is usually prevented by our hyperbolic impatience to gratify our appetite, which undermines ad lib self-reward and creates an incentive to set up obstacles to gratification.  I have outlined how constraints must operate on any kind of reward that depends only on the appetite for it, and how both conflict and symbiosis with instrumental reward is apt to result, sometimes creating process addictions. 

The important factors in endogenous reward have been partially recognized in behavioral psychology and classical economics, but beyond reward itself the concepts increasingly diverge from those proposed here (table 1).

 

Table 1: key concepts in endogenous reward and their closest equivalents in two other approaches
     
Picoeconomics Behavioral Psychology Classical Economics
Reward Process reward Utility
Endogenous reward Process reward Unaccounted tastes
Interestingness Drive, appetite Demand
Occasion (as a noun) Secondary reward Predictive information
Singularity [no concept] Scarcity (but negative)
Texture [no concept] [no concept]
Indirection [no concept] X-inefficiency (March, 1978)


              Notes       

1. The word “reward” may denote either the internal selective factor that selects the mental process it follows for repetition, or an external event that activates this factor.  Unfortunately, there are no common words that distinguish these cases.  I use “reward” to denote the internal factor, and “a reward” or “rewards” to denote the external events..

2. Briefly, pain, negative emotion and other vivid but aversive experiences comprise two motivational components, reward for attention and a consequent lowering of general reward level.  The simplest model of this combination is a rapid cycle of brief, high reward and longer unreward, hyperbolic discounting of which can enable the brief rewards to capture choice despite the aversiveness of the pattern as a whole.  This explanation obviates classical conditioning as a separate selective principle.  Other models are possible, for instance the competition of Pavlovian and instrumental “values” in temporal difference theory (Dayan et. al., 2006), but all must entail what amounts to a positive reward for attention.

3. The discrimination of appetites from each other is also an ambiguous process.  Dodge provided a start:  “Whenever in mental processes fatigue of one is regularly accompanied by fatigue of another there must be some dynamic factor common to both.  Conversely, whenever the fatigue of one mental process does not show a fatigue of another, the two must depend on different dynamic conditions” (1917, p. 92).  This is to say that psychologist Richard Herrnstein’s test of separate satiability to delimit concrete appetites (1969) could be applied to modalities of endogenous reward.  The eight or ten major categories of arousable emotion (e.g. Izard, 1972), which, because they fatigue, must also have (or be) appetites, are obvious examples; but otherwise practical benchmarks that might define separately fatiguing categories seem to be almost entirely lacking.

4. I thank Ole Rogeberg for finding this reportage.

5. A social analog was the Great Plains Indians’ practice of “counting coup,” arising from their valuation of striking the first blow against enemies more highly than actually overcoming them (McGinnis, 2010).

6. After years of experience any of these activities becomes less vulnerable to a recognition that it is not instrumentally effective, because the longstanding taking of this particular risk or collecting that particular object have made the activities singular in their own right.

 

References

Ainslie, G.  (2001)  Breakdown of Will.  Cam­bridge U.

Ainslie, G.  (2003)  Uncertainty as wealth.  Behavioural Processes 64, 369-385.

Ainslie, G. (2005) Précis of Breakdown of Will. Behavioral and Brain Sciences 28(5), 635-673.

Ainslie, G. (2006)  Motivation Must Be Momentary.  In J. Elster, O. Gjelsvik, A. Hylland and K. Moene  (Eds), Understanding Choice, Explaining Behaviour: Essays in Honour of Ole-Jorgen Skog.  Unipub Forlag, pp. 11-28.

Ainslie, G.  (2009)  Recursive self-prediction in self-control and its failure.  In T. Gruene-Yanoff and S. O. Hansson (Eds), Preference Change: Approaches from Philosophy, Economics, and Psychology. Springer, pp. 139-158.

Ainslie, G.  (2010)  The core process in addictions and other impulses: Hyperbolic discounting versus conditioning and cognitive framing.  In D. Ross, H. Kincaid, D. Spurrett, and P. Collins, (Eds)., What Is Addiction? MIT, pp. 211-245.

Ainslie, G.  (2012)  Pure hyperbolic discount curves predict “eyes open” self-control.  Theory and Decision 73, 3-34. 10.1007/s11238-011-9272-5

Ainslie, G.  (in press)  Money as MacGuffin: A factor in gambling and other process addictions.  In N. Levy (Ed), The Mechanisms of Self-Control: Lessons from Addiction. Oxford U.

Ainslie, G. and Haendel, V. (1983) The motives of the will.  In E. Gottheil, K. Druley, T. Skodola, H. Waxman (Eds), Etiology Aspects of Alcohol and Drug Abuse,  Charles C. Thomas, pp. 119-140.

Becker, G. and Murphy, K. (1988) A theory of rational addiction. Journal of Political Economy 96, 675-700.

Berlyne, D.E. (1960)  Conflict, Arousal, and Curiosity.  McGraw-Hill.

Berns, G. S., Chappelow, J., Cekic, M., Zink, C. F., Pagnoni, G., and Martin-Skurski, M. E.  (2006)  Neurobiological substrates of dread.  Science 312, 754-758.

Bexton, W. A., Heron, W., and Scott, T. H.  (1954)  Effects of decreased variation in the sensory environment.  Canadian Journal of Psychology 8, 70-76.

Blaszczynski, A., Walker, M., Sharpe, L., and Nower, L. (2008)  Withdrawal and tolerance phenomenon in problem gambling.  International Gambling Studies 8, 179-192.

Broyles, W. (1984) Why men love war.  Esquire, November.

Buckner, R. L., Andrews-Hanna, J. R., and Schacter, D. L.  (2008)  The brain’s default network: anatomy, function, and relevance to disease.  Annals of the New York Academy of Sciences 1124, 1-38, doi: 10.1196/annals.14410.011.

Coover, R. (1968) Universal Baseball Association, Inc.,  J. Henry Waugh, Prop. Random.

Dayan, P., Niv, Y., Seymour, B., & Daw, N. (2006). The misbehavior of value and the discipline of the will. Neural Networks, 19(8), 1153-1160.

Dodge, R, (1917)  The laws of relative fatigue.  Psychological Review 24, 89-113.

Gearhardt, A. N., Yokum, S., Orr, P. T., Stice, E., Corbin, W. R., and Brownell, K. D. (2011)  Neural correlates of food addiction.  Archives of General Psychiatry 68, 808-816.

Gollwitzer, Peter M. and Bargh, John A.  (1996)  The Psychology of Action: Linking Cognition and Motivation to Behavior.  Guilford.

Gregorios-Pippas, L., Tobler, P. N., and Schultz, W. (2009)  Short-term temporal discounting of reward value in human ventral striatum.  Journal of Neurophysiology 101, 1507-1523.

Griffiths, M.  (2005)  Relationship between gambling and video game playing: A response to Johansson and Gotestam.  Psychological Reports 96, 644-646.

Hagger, M. S., Wood, C., Stiff, C., and Chatzisaarantis, N. L. D.  (2010)  Ego depletion and the strength model of self-control: A meta-analysis.  Psychological Bulletin. 136, 495-525.

Heyman, G. M.  (2009)  Addiction: A Disorder of Choice.  Harvard U.

Herrnstein, R. J. (1969) Method and theory in the study of avoidance. Psychological Review 76, 49-69.

Hidi, S. and Renninger, K. A. (2006)  The four phase model of interest development.  Journal of Educational Psychology 41, 111-127.

Hsee, Christopher K., Yang, Adelle X., and Wang, Liangyan  (2010)  Idleness aversion and the need for justifiable busyness.  Psychological Science 21, 926-930.

Izard, C.E. (1972) Patterns of Emotion: A New Analysis of Anxiety and Depression, New York:  Academic Press.

Jackson, B.  (2012)  The Zynga abyss.  www.theatlantic.com/technology /print/2012/01/the-zynga-abyss/251920.

Johnson, S. C., Shimizu, A., and Ok, S.-J. (2007)  Actors and actions: The role of agent behavior in infants’ attribution of goals.  Cognitive Development 22, 310-322.

Karp, L.  (2005)  Global warming and hyperbolic discounting.  Journal of Public Economics 89, 261-282.

King, D. L., Delfabbro, P. H., and Griffiths, M. D.  (2011)  The role of structural characteristics in problematic video game play: An empirical study.  International Journal of Mental Health and Addiction 9, 320-333.

Kirby, K. N.  (1997)  Bidding on the future: Evidence against normative discounting of delayed rewards.  Journal of Experimental Psychology: General 126, 54-70.

Lea, S. E.G. and Webley, P.  (2006)  Money as tool, money as drug: The biological psychology of a strong incentive.  Behavioral and Brain Sciences 29, 161-209.

Loewenstein, G. (1987) Anticipation and the valuation of de­layed consumption. The Economic Journal 97, 666-685. 

Loewentstein, G. (2009)  That which makes life worthwhile.  In Alan B. Kreuger, (Ed), Measuring the Subjective Well-Being of Nations: National Accounts of Time Use and Well-Being. University of Chicago, pp. 87-106.

Mackworth, N. H.  (1948)  The breakdown of vigilance during prolonged visual search.  Quarterly Journal of Experimental Psychology 1, 6-21.

Marks, I. (1990)  Behavioural (non-chemical) addictions.  British Journal of Addictions 85, 1389-1394).

Mazur, J.E.  (1996)  Procrastination by pigeons: Preference for larger, more delayed work requirements.  Journal of the Experimental Analysis of Behavior 65, 159-171.

McGinnis, A. R.  (2010)  Counting Coup and Cutting Horses: Intertribal Warfare on the Northern Plains, 1738-1889.  Bison Books.

Mellers, B., Schwartz, A. and Ritov, I.  (1999)  Emotion-based choice.  Journal of Experimental Psychology: general 128, 332-345.

Montague, P. Read and Berns, Gregory S.  (2002)  Neural economics and the biological substrates of valuation.  Neuron 36, 265-284.

Navarick, D.J. (1982) Negative reinforcement and choice in humans. Learning and Motivation 13, 361-377.

Pezzulo, G. and Rigoli, F.  (2011)  The value of foresight: how prospection affects decision-making.  Frontiers in Neuroscience 5, 79.  Doi: 10.3389/fnins.2011.00079.

Prelec, D. and Loewenstein, G. F.  (1998)  The red and the black: Mental accounting of savings and debt.  Marketing Science 17, 4-28.

Renninger, K. A. (1992)  Individual interest and development: Implications for theory and practice.  In K. A. Renninger, S. Hidi, and A. Krapp (Eds), The Role of Interest in Learning and Development.  Erlbaum.

Renninger, K. A. and Hidi, S. (2011)  Revisiting the conceptualization, measurement, and generation of interest.  Educational Psychologist 46, 168-184.

Rick, S. and Loewenstein, G.  (2008)  Intangibility in intertemporal choice.   Philosophical Transactions of the Royal Society B 363, 3813-3824.

Robins, L..N. and Regier, D..A. (1991) Psychiatric Disorders in A­merica: The Epidemiologic Catchment Area Study.  Free Press.

Rosati, A. G., Stevens, J. R., Hare, B., and Hauser, M. D.  (2007)  The evolutionary origins of human patience: Temporal preferences in chimpanzees, bonobos, and human adults.  Current Biology 17, 1663-1668

Ryan, R. M., and Deci, E. L.  (2000)  When rewards compete with nature: The undermining of intrinsic motivation and self-regulation.  In C. Sansone and J. M. Harackiewicz, Eds., Intrinsic and Extrinsic Motivation: The Search for Optimal Motivation and Performance. Academic, 13-54.

Sandson, J., and Albert, M. L.  (1984)  Varieties of perseveration.  Neuropsychologia 22, 715-732.

Sansone, C. (2009)  What’s interest got to do with it?  Potential trade-offs in the self-regulation of motivation.  In  J. P. Forgas, R. Baumeister and D. Tice (Eds), Psychology of Self-Regulation: Cognitive, Affective, and Motivational Processes.  Psychology Press, pp. 35-51.

Sansone, C., and Harackiewicz, J. M.  (1996)  “I don’t feel like it”: The function of interest in self-regulation.  In L. L. Martin and A. Tesser (Eds), Striving and Feeling. 203-228.  Erlbaum.

Shizgal, P., and Conover, K. (1996)  On the neural compu­tation of utility.  Current Directions in Psychological Science 5, 37-43.

Skinner, B.F. (1953) Science and Human Behavior, New York: Free Press.

Solnick, J., Kannenberg, C., Eckerman, D. and Waller, M.  (1980) An experimental analysis of impulsivity and impulse control in humans. Learning and Motivation 2, 61-77. review, 217-225.

Silvia, P. J.  (2001)  Interest and interests: The psychology of constructive capriciousness.  Review of General Psychology 5, 270-290.

Silvia, P. J.  (2006)  Exploring the Psychology of Interest. Oxford.

Slovic, P., and Tversky, A. (1974)  Who accepts Savage’s axioms?  Behavioral Science 19, 368-373. 

Spreng, R. N., Mar, R. A., and Kim, A. S. N.  (2009)  A common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: A quantitative meta-analysis.  Journal of Cognitive Neuroscience 21 (3), 489-510.

Sternbergh, A. (2011)   The thrill of defeat for sports fans.   New York Times Magazine October 23, 2011, pp. 18-20)

Suedfeld, P., Ramirez, C., Deaton, J., and Baker-Brown, G.  (1982)  Reactions and attributes of prisoners in solitary confinement.  Criminal Justice and Behavior 9, 303-340.

Turkle, S. (2011)  Alone Together. Basic

Turner, S. A., and Silvia, P. J.  (2006)  Must interesting things be pleasant?  A test of competing appraisal structures.  Emotion 6, 670-674.

Wilson, T. D., Centerbar, D. B., Kermer, D. A., and Gilbert, D. T.  (2005)  The pleasures of uncertainty: Prolonging positive moods in ways people do not anticipate.  Journal of Personality and Social Psychology 88, 5-21.

Wittman, M., Lovero, K. L., Lane, S. D., and Paulus, M. P.  (2010)  Now or later?  Striatum and insula activation to immediate versus delayed rewards.  Journal of Neuroscience, Psychology and Economics 1, 15-26.  Doi: 10.1037/a0017252.

1. The word “reward” may denote either the internal selective factor that selects the mental process it follows for repetition, or an external event that activates this factor.  Unfortunately, there are no common words that distinguish these cases.  I use “reward” to denote the internal factor, and “a reward” or “rewards” to denote the external events.
2. Briefly, pain, negative emotion and other vivid but aversive experiences comprise two motivational components, reward for attention and a consequent lowering of general reward level.  The simplest model of this combination is a rapid cycle of brief, high reward and longer unreward, hyperbolic discounting of which can enable the brief rewards to capture choice despite the aversiveness of the pattern as a whole.  This explanation obviates classical conditioning as a separate selective principle.  Other models are possible, for instance the competition of Pavlovian and instrumental “values” in temporal difference theory (Dayan et. al., 2006), but all must entail what amounts to a positive reward for attention.
3. The discrimination of appetites from each other is also an ambiguous process.  Dodge provided a start:  “Whenever in mental processes fatigue of one is regularly accompanied by fatigue of another there must be some dynamic factor common to both.  Conversely, whenever the fatigue of one mental process does not show a fatigue of another, the two must depend on different dynamic conditions” (1917, p. 92).  This is to say that psychologist Richard Herrnstein’s test of separate satiability to delimit concrete appetites (1969) could be applied to modalities of endogenous reward.  The eight or ten major categories of arousable emotion (e.g. Izard, 1972), which, because they fatigue, must also have (or be) appetites, are obvious examples; but otherwise practical benchmarks that might define separately fatiguing categories seem to be almost entirely lacking.
4. I thank Ole Rogeberg for finding this reportage.
5. A social analog was the Great Plains Indians’ practice of “counting coup,” arising from their valuation of striking the first blow against enemies more highly than actually overcoming them (McGinnis, 2010).
6. After years of experience any of these activities becomes less vulnerable to a recognition that it is not instrumentally effective, because the longstanding taking of this particular risk or collecting that particular object have made the activities singular in their own right..