Positive Reinforcement Frontiers with Zoo Animals

For this post, I take you to a recent entry on one of the blogs in my Links page:  Mary Hunter's at stalecheerios.com.   Mary covers positive reinforcement training, and she helped advise an artist on the development of a poster featuring "twelve great examples of wild animals who have been trained to do extraordinary tasks without any use of force or punishment."  The first one certainly grabbed my attention:  training a whale to pee in a cup.  Way cool!  Some feats are covered in my own chapter on animal applications, such as birds that spy and animals that willingly weigh themselves on scales, or allow blood samples to be taken.  Here's the link.  It's a great poster!  (Note:  I have no connection with the artist.)  Don't miss the fish that plays soccer.

Risky Choices

Risky, that is, in the sense of passing up a sure thing in favor of a much larger reward--maybe.  In The Science of Consequences, I note that people and animals alike often prefer variable schedules of reinforcement over fixed ones even when they don't pay off as well.  What about when it’s the amount of the reward that's unpredictable?  Do we still take a chance on the riskier variable choice?

(Note:  I regularly post descriptions of new or classic research that are a bit more technical than my regular posts.  This is a "research post" that I hope is of general interest.)  

In a new study, Carla Lagorio and Tim Hackenberg report that past research results have been inconsistent.  They also note that this type of research has been viewed as a way to approach the study of gambling.  Obviously, not every gamble pays off:  Success is variable and (often) unpredictable in amount.  Problem gambling is a large and expensive problem worldwide, and laboratory analogs that help us understand some of its contributing factors are valuable. 

The researchers looked at pigeons working for "token" symbols:  lights on a panel.  Each light could be exchanged for a short period of access to food, but only during signaled "exchange" periods a short while after each choice.  This approach helped make this research setup more analogous to real-life human gambling:  People also frequently get tokens like chips that can be exchanged only later for money (which itself is an exchangeable token, of course!).

In this particular study, seven pigeons pecked to initiate a trial, then pecked again to make their choice.  The fixed choice payoff stayed constant at either 2, 4, 6, or 8 tokens.  The variable payoff could be anything from 0 to 12 tokens, offered on one of a number of different distributions ("rectangular" or "exponential" for those of you who are mathematically inclined).  A bird might earn 9 tokens after a "variable" choice, then only 2 after another "variable" choice.

This was a thorough "parametric study" in which different combinations of fixed vs variable schedules were run.  Once an individual's choice pattern was stable for one combo, that bird would be switched to a different one, and so on.  The outcome?--a strong preference for variable rewards rather than fixed ones, similar to the results for variable vs fixed schedules of reinforcement.  And again, that was frequently the case even when the birds lost by their risky choices:  that is, even when switching to the fixed choice would have provided substantially more reward over time. In a way, they're like problem gamblers in this respect.

Just as interesting:  When the token signals were removed and the birds simply worked for direct access to food, these skewed results were less likely; the birds made more rational choices instead.  What's going on?  Stay tuned.

One final finding I have to mention:  Some of my own past research examined the influence of one particular reinforced schedule choice on the next choice, in a process called "sequential analysis."  If you’ve just TV-surfed to a baseball game and happened to catch a home run or a spectacular double play, are you more likely to stay with the game than if you tuned into a batter engaging in boring warm-up swings?  These moment-to-moment influences on our choices make intuitive sense in our daily lives.  That was also the case here:  Pigeons were significantly more likely to go with the "variable" option if they had just enjoyed a handsome payoff for choosing "variable."  If they’d received no tokens for the "variable" choice, they were very likely to switch to fixed for their next choice.  How human of them . . .

An Odd Consequence Value Effect (Or, Why Do I Mop the Kitchen Floor When I Get Good News?)

There's no doubt about it:  I can go for weeks without mopping the kitchen floor.  (Sweeping is so much easier.)  It weighs on my mind.  A little.

What nudges its consequence value up enough to tip it over the threshold of it-can-wait to let's-get-it-over-with?  It's often getting good news.  Suddenly, chores become easy as their aversiveness mysteriously declines. 

My kitchen floor

  Social psychologists verified years ago what our own
  experience tells us:  "Be happy to be good."  When is
  a charity more likely to successfully appeal to you (in
  other words, when does giving hard-earned cash become
  sufficiently rewarding)?--after you've had a horrible day
  at work, or a great one?  How does this effect fit in with the
  principles of consequences?  Well, we already know
  that the relationship between emotions and consequences
  is complex.  This is another way in which they interact.


I'm sure our past experiences--our history of consequences--and the rules we've learned are involved. As someone particularly aware of the motivators in my life (teaching this science and then writing this book), I can see the value of consequences vary with my mood.  As always, the situation is complex:  For household chores, I've also noticed that the other extreme can work too.  That is, when very little in my life is reinforcing, again, it can be easier than usual to do chores.  After all, they provide some reward value, if only by escaping that annoying feeling that I ought to do them, and, of course, escaping the unpleasantness of seeing dust balls in the corner. I couldn't find any research about this in a quick check, but I'm sure there's some out there.  (Let me know.)

Most effective of all, when I get a visitor, wow, see me reach for that mop.  Serious consequences (both positive and negative).

Am I just weird, or does this phenomenon apply to you too?

Book Launch at the Fair!

The Miami Book Fair International, that is--one of the largest in the United States!  I'm pleased to announce that I will be launching my book there over the weekend of November 17-18 (the weekend before Thanksgiving).  When I get around to it, I'll be opening an Events webpage that will include the specifics (which I don't know yet myself).

I'm especially pleased because I spent five happy years in Miami as a research faculty member at Florida International University, working on nature-nurture relations and the mathematical modeling of behavior.  A behavioral enrichment project from those years is described in my book.

Extra Signals

Compose a new message in gmail and it's hard to click on the Send button by mistake:  It's the only button in red.  (Wonder why the programmers didn't set up this "extra" color-coded signal for the "Reply" function as well . . .)

In a similar manner, I appreciate the "extra" raised dots on the "f" and "j" on my keyboard, helping me touch-type truly by touch, no peeking needed.  They signal when my fingers are on the right keys, indicating that certain key strokes will be reinforced by enabling me to type what I need to (such as this blog post).  I've often wished there were more such extra signals on the standard keyboard.  In fact, to help me move the arrow keys efficiently, I put sticky paper on one--and that lets me navigate to the adjacent ones easily just by touch.  My focus can stay on the monitor, rather than switching back and forth to the keyboard. I get things done faster.

Such redundancy in signaling has lots of practical applications.  A couple years ago, I happened to get into conversation with a motorcyclist who commented on his efforts not to disturb the neighbors when he rode in late at night.  I'd wondered why more work hadn't been done to muffle the roar of the engine.  Turns out that motorcyclists--naturally enough--want to make a big commotion when they're on the roads risking their lives among the heavy bruisers like trucks and SUVs.  Even subcompacts, for that matter.  Like ambulances, they want to be both seen and heard so that other drivers can give them a safe berth. (Of course, the attention all the noise brings them can sometimes be rewarding in itself.)

That makes sense, but wouldn't it be great if motorcyclists could turn off the roar when they don't need it, like ambulance drivers do?

If the Sky was Pink All Day

People who live near mountains get so used to them, they hardly bother to look.  These are the same mountains others might drive hundreds of miles to see.  What's happening to the reinforcer value here?

California sunset ©Ken R. Schneider

  Imagine if the sky was pink all day, and only blue at sunrise
  and sunset. People would gather to watch the wonderful
  blue emerge, and sigh as it slowly faded away--much as
  they do now for glorious pink sunrises and sunsets.

  Frequently, things that are easy to get, like a blue sky, are
  less powerful rewards than things that are harder to get, like
  a pretty sunset.  The schedule of reinforcement is different,
  for sure: You can't just walk outside and marvel at a sunset
  any time of day.   

Another factor is satiation:  For many rewards, too much of a good thing is not a better thing.  Two or three delicious candies might be a longed-for treat, but a whole box is a surfeit with no extra immediate value.  Indeed, being forced to eat a whole box might be a punisher.

I don't know that anyone's ever had a surfeit of sunsets, but I suppose a photographer trying to capture the perfect one, evening after evening after evening, could find blue skies a lot more appealing!

Jays in My Backyard

Western Scrub-Jay (©Ken R. Schneider)

Earlier this summer, I had the pleasure of watching two fledgling Western Scrub-Jays learn from consequences.  They were perched on my brush pile underneath my redwood when I spotted them, and clearly had left the nest not long before.  They were waiting impatiently for their parents, who stopped by regularly with food that got eagerly grabbed.  Meanwhile, they explored their surroundings.  One found a small yellow leaf, trapped it under one foot and pecked at it, as if checking whether it was good to eat (no).  It was discarded.  The other youngster poked around a bit too, also without success.  Still, it was a start on the path to independence.

One week later, I watched what was probably one of these youngsters in my yard again, this time foraging more actively, picking up dead grass stalks systematically.  Once again, its efforts did not appear to be crowned with success, but it was trying.

The adults are omnivorous, kind of like feathered bears:  They’ll go after anything that’s edible.  And they readily learn their way around minor obstacles.  Someone in my neighborhood feeds peanuts in the shell, and the birds come to particular flat fencepost stumps in my backyard to hold them down and peck them open with their all-purpose beaks.  I'm sure the youngsters will be up to that--someday.

Update:   I happened to catch a cool scrub jay post on animal expert Sophia Yin's blog (link here).  As she reports, two scientists published a 1999 Nature article on how jays learn from consequences when they store food like nuts in hidden "caches."  The nuts last a long time, but cached foods like waxworms don't.  It turns out that "the birds had to learn that food such as waxworms degrade after long intervals. A separate set of scrub jays whose rotten waxworms were secretly replaced with fresh ones after long intervals, never learned that waxworms go bad."  And they behaved accordingly, very differently from the jays that had learned otherwise.  The flexibility provided by learning--even if it's researcher-assisted and "unnatural"--helps the birds survive.

The Joy of Escape

Shelter, London Blitz (public domain)

I’m not talking about a trip to Hawaii here.  Escaping a big negative is a powerful reward, one that can be accompanied by powerful emotions.  My community’s "One Book" selection this year is Rebecca Solnitt’s A Paradise Built in Hell: The Extraordinary Communities that Arise in Disaster (which I thought was excellent, by the way; 2009, Viking).  Solnitt quotes a young woman describing her feelings of relief after making it to a shelter and surviving a bomb during the World War II London Blitz:  "It seems a terrible thing to say, when many people must have been killed and injured last night; but never in my whole life have I ever experienced such pure and flawless happiness" (p. 103).  She was not alone.

For most of us, the joys of escape come on a much smaller scale in less tragic circumstances.  When I read this passage, I flashed back to a sensation I’d experienced years ago.  I was a commuter cyclist for decades, but the statistics eventually caught up with me and I got hit by a car, fracturing my back as well as suffering many minor injuries.  The accident occurred during a stressful time in my life and, weirdly, even during the immediate aftermath, I had a feeling of liberation from those other problems.  I’d escaped from them--albeit not in a way I would have chosen--and could simply appreciate being taken care of and working on recovering. My bicycling had been punished pretty strongly by this powerful negative consequence, but when I recovered, I did eventually get back to it. 

Fun with Hummingbirds

Male Anna's (©Ken R. Schneider)

One of the delights of living in California is hosting hummingbirds year-round--including one of the most spectacular North American species, the Anna’s Hummingbird.  My brother took this picture of an adult male.

Last year I rescued a young, probably female Anna’s that hit a back window (despite the warning stickers).  She didn’t seem to  be injured, just stunned.  It was cold, so to help her recover--and keep her away from predators--I held her in my hand in the sun, while she stared at me and one wing quivered.  How could I reassure her?  I decided to take the opportunity to play a game that’s described in my book.  It entails rewarding an animal’s eyeblinks by giving unusually long eyeblinks in immediate response--a sort of communication, if you will.  My hummer didn’t blink much at first, but I took advantage of every blink.  Then she caught on, and within 3 minutes, every time I opened my eyes after my own blinks, she immediately blinked!  Great fun, and best of all, after 5 min, she lifted off, apparently back to normal.  What a magical experience. 

Juvenile Anna's (public domain)

Having a hummingbird feeder lets me enjoy behavior-watching as well as beauty.  I get to laugh at hummingbirds learning to find the nectar.  (Sometimes they try in what seems like every possible place before they succeed.)  When I take the feeder in for cleaning and refilling, I see hummers fly to where they’ve learned it should be, casting about fruitlessly before buzzing off.  Talk about well-learned habits!  And I breathlessly witness frequent dominance battles over access to this prime resource.  In most U.S. hummingbirds, females are larger than males, and my own feeder tends to be dominated by one of them.  I still get to enjoy the beautiful magenta iridescence on the throat "gorget," though, because adult female Anna’s have them.  Maybe the hummer I rescued has developed one by now, attracted a mate, and raised little ones of her own.