April 20, 2014

Conveying Forecast Error

Whenever you forecast, sometime you'll want to see where you're off a little.  And sometime, you'll want to see where you're off a lot.  You'll need a functional way to express these errors, that let you analyze them in a variety of ways.

I can pretty much step you through what most first-time forecasters do, and then explain where you'll sooner-or-later end up: Using a logarithmic scale.

Bad Method 1:  Using a Linear Scale
Crappier than it seems
At first glance, you'll be tempted to just do it like the picture to the right.  ...So, in this chart, the 130%-140% bucket would contain a forecast of $6.65 for an actual of $5 (6.65/5 = 133%), and the 60%-70% bucket would contain a forecast of $3.25 for an actual of $5 (3.25/5 = 65%).  

This works, right?   Sort of, but it has two big drawbacks:


Static Level of Precision
You probably don't want all of your error buckets to be the same size;  this gets ridiculous pretty quickly.

I mean, reporting upon a 100-110% bucket makes sense -- but what about the 170-180% bucket (are these sufficiently different from 180-190% to warrant their own bucket?).   And it makes zero sense to report a 780%-790% bucket or (even more ridiculous) a 8,920%-8,930% bucket.  And that's what you're gonna get with this approach.


Ideally, your buckets will grow along with your error:  You need fine precision when your forecasts are close, and coarser precision for your big misses -- which you don't get.

Asymmetrically Bounded

On a percentage chart like the one above, the smallest error bucket is fixed:  0-10%.  Your forecast can't get any lower than 0% of actual.  However, your forecast could be a bazillion times higher than actual.  

As a result, when you plot out your error, you'll get a very short "low-tail", and a very long "high-tail".  Among other problems, this makes it impossible to visually scan for forecast bias, because your too-high and too-low forecasts aren't reported in a consistent fashion.  

 
Bad Method 2:  Using a Custom Scale
From there, most people move on to creating customized buckets.  So you'll write statements like this:

If err = 0 then "Match"
else if err > 0 and err <= 1% then "1%"
else if err > 1% and err <= 5% then "1-5%"
else if err > 5% and err < 10% then "5-10%"
else if err > 10% and err < 25% then "10-25%"

...But you'll probably discover that your buckets are always somewhat arbitrary and not quite  pleasing.  And, of course, any time you want more or less-granular buckets, you gotta mess with your equation.

Proper Method:  Use a Log Scale
Reporting upon forecasting error on a logarithmic scale solves all of these problems.
 
But before I jump into using  log-based error groups, let me remind you about logarithms.  (For the rare few of you who are already savvy with logs, feel free to jump ahead.)


Logs 101
Easy, once you get the hang of it.
Logs are the opposite of exponentials, just like subtraction is the opposite of addition, or division is the opposite of multiplication.  So, ya know how since 5 * 2 = 10, then by definition 10 / 2 = 5?  Well, if 53 = 125  then log5(125) = 3.  ...Btw, that five in subscript is called the log's "base."

Logs are much easier to understand if you just see them in action.  The chart to the right shows the log values for certain bases. As you can see, for log base 5, each value represents five times the previous value.   For log base 2, each value represents two times the previous value.

To convert your forecasting error to a log scale, just take the log of the forecast/actuals.  Any base is fine.  With base 2, if your forecast were double your actual, log(200%,2)= 1.   If your forecast were half of your actual, log(50%,2)= -1.  If your forecast matched your actual, log(100%,2)=0.  

What does this buy us?  A lot!

Dynamic Level of Precision
First and foremost, each log value represents a bigger group, which can go from very constrained to enormous.  

Let's say that your forecast is often off by 30%, but you're occasionally off by upwards of 100,000%.  (This especially happens when your actual value unexpectedly drops near zero, where the forecast/actual ratio can be sky-high even for modest forecasts.)

On a linear scale, you can't meaningfully show 30% and 100,000% in the same chart, without the chart being enormous.  Yet on a log base 2 chart, the 30% gets a value of .37, and the 100,000% gets a value of 13.2. 

Amazingly, a log scale allows for fine-tuned precision for accurate forecasts (i.e, you can see the difference between a 5% and a 10% miss), even when reported right next to a 100,000% miss!

Symmetrically Bounded
As I described earlier, on a linear scale, too little precision for forecast that are very low (they're all lumped into the 0-10% bucket), and too much precision for forecasts that are too high (each one gets its own useless bucket, like 8050-8060%).

Logarithms don't have that problem: They represent high and low values on an equal terms.  If you were ever 100x too high, using a log base 2, you'd get a log value of 6.6.   If you were 100x too low, you'd get a log value of -6.6. 

Working Example:
Here, I used Excel to generate 1000 random numbers (each from 1 to 100), and then "forecast" each with another set of semi-random numbers -- based partly upon the original number, and partly upon another random number.  

For kicks, I gave my forecast a subtle positive bias -- it's higher than the actual number a bit too often.

Now, let's plot the error both linearly and logarithmically, and see what we find.

Linear Scale

Ugh, what a disaster. 

First, for one thousand data points, we have 629 buckets.  

Second, ten "low" buckets (0-10%, 10-20%, etc) averaged 28 entries each, while six hundred and nineteen "high" buckets averaged just one entry each.  

Third, because the chart is naturally lopsided, we can't visually see the overt bias in the forecast.

Logarithmic Scale

This is more like it!

First, our chart has only 100 points, instead of six hundred and twenty nine.  Yet our precision was precise where it mattered (i.e., forecasts close to 100% of actuals), and rough when it didn't matter.

Second, you can see that both our high and low values slope away, roughly in symmetry.  Instead of our low forecast crammed into ten groups, and our high values spread out over 619 groups, they're much closer to equal.

Third, you can see instantly that there is a positive forecast bias.  (Focus on the center bar, which represents forecasts near 100% of actuals.  Now compare the bars immediately to its right and left, and then bars two to the right and left.  See how the right bars are always higher?  That's a tell-tale sign of positive forecast bias.)

Conclusion
I probably could have written this entire article in a single sentence:  When reporting forecast error, use a logarithmic scale.

The trouble is, other people had told me that in the past, and I never quite understood why I was doing it, or even knew if I was doing it right.  Hopefully, by describing the drawbacks of the more conventional linear approach, you'll have a bit better understanding of why you should embrace logarithmic error reporting sooner, rather than later!

April 4, 2014

Makin' It


I'm kind of ashamed of my finances.   I should be saving more.  I buy too many toys and spend too much on needless shit.   I fear my friends are saving WAY more than I am.  Plus, I gotta think about retirement and college accounts and all that.

It's not that I'm not saving anything, but I'm probably not saving as much as I could.   That's why I'm ashamed.

But it's all good.  I'm pretty sure 80% of my friends feel this way, too.  We're the demographic that frets over this sort of thing.  We're Makin' It.

Makin' It.
As far as I can tell, people fall into one of several demographics: Children, Students, Paycheck-to-Paycheck, Makin' It, and Have It.

My Co-Workers: the Makin' It Demographic
I'm in the Makin' It class, which is roughly comprised of people who can afford to purchase a luxury car, or some other large-ish indulgence -- but will wince a bit doing it, because there are far better things for them to be spending money upon.

We have career plans and debate the merits of Masters degrees: either lusting after them, as a faux blue ribbon on our resumes, or resenting them as a faux blue ribbon on other people's resumes.  We save.  We dabble in the stock market.  We have 401Ks and retirement plans and SEPs and Roth IRAs.   Sometimes we spring for Brooks Brothers no-iron shirts.   We fret about buying a house, or adding an addition to our house, or upgrading to a bigger house, even though we can't afford it quite yet.

...And we occasionally assess our total assets and think, "Shit, really?  Is that it?  Well...I guess if I count the house...that's something."

I someday hope to transfer to the Have It class, although I'm not quite sure how the hell that happens.  But for now, I'm happily in the Makin' It category.

Theme Song
Now, most of you know that each class has its own theme song.  For example, the working-class  Midwestern demographic fawns over Bruce Springsteen's Born to Run.   The former-varsity-football-playing overweight insurance salesmen demographic favors Bruce Springsteen's Glory Days.   (Note: Bruce Springsteen's songs are disproportionately reflected in demographic theme songs; sociologists cannot agree upon why.)

My demographic's theme is obvious:  The eponymous Makin' It, by David Naughton.  This song totally "gets" my cohort.  Plus, it has an awesome disco vibe, of which we unanimously approve.

This song was so awesome that it spurred its own television series, which ran for seventeen years.  Fun fact:  Makin' It is the only show to be nominated for an Emmy in the Primetime and Documentary categories.

Seriously, if you know any corporate executives, ask them and they will gaily confirm that many meetings start off with Makin' It playing on a Tivoli desk speaker.  We hum it to ourselves in Starbucks, and sometimes they play it over the loudspeaker.   We're climbin' the ladda.

Earlier today,  I passed by my colleague Jasmie, emerging from her big business presentation and I high-fived her:   An executive had nodded and smiled during her conclusion slide-- a sure sign of office success.  Jasmine is Makin' it

Academic Underpinnings
Even before joining the corporate workforce, some of you learned this song in corporate-minded undergraduate business and engineering schools.  Others, no doubt, ran across it in MBA programs, where verse memorization is often required.

This is more common in MBA programs than many people realize.  They certainly covered it when I got my MBA at Duke. I happen to know that the Harvard and MIT MBA programs require memorization as well.  A dear friend of mind (who does Data Warehousing at Akamai) is getting his MBA at Chicago (Booth), and expressed surprise at how much this song was reflected in its general curriculum.

The Lyrics of Makin' It
Okay, for those of you who don't know what I'm talking about, I'm just going to put the words to the song right here.  Obviously, people who already know the lyrics can skip this section.  However, it's worth a read for the rest of you.  This is the voice that speaks for my demographic.
Makin' it ooh...Makin' it
I´m solid gold, I´ve got the goods
They stand when I walk through the neighborhoods  

I´m makin' it, I´ve got the chance, I´m takin´ it, no more
.. no more fakin´ it, this time in life I´m makin´ it ooh makin´ it
Hello Uptown, Goodbye Poverty
The Top of the ladder is waiting for me
I´m makin´ it, I´ve got the chance, I´m takin´ it no more
No more fakin´ it, this time in life, I´m makin´ it, ooh
Makin´ it, Makin´ it
Listen everyone here, this coming year´s gonna be my year
I´m as fast as they come, number two to no one, I´ve got looks
I´ve got brains and I´m breakin´ these chains, make some room now
Dig what you see, success is mine, ´cause I´ve got the key
I´m makin´ it, I´m makin´ it, I´ve got the chance, I´m takin´ it no more

No more fakin´ it, this time in life, I´m makin´ it , ooh makin´ it, non stop, 
Makin´ it to the top... Makin´ it ...this time in life I´m makin´ it
Makin' it...I´m makin' it ..Makin' it....I'm makin' it
Listen Everyone here...This coming year's gonna be my year
I´m makin' it...I´ve got the chance I'm takin it in ...no more...no more
Makin' it, this time in life I'm makin' it..ooh...I'm makin' it...non stop...
I'm here...I'm here...Makin' it...to the top...ahh .. ahh..

Makin' it....non stop....right here...right now
Even when judged by contemporary lyric-writing standards, this song is a masterpiece.  Observe how "makin' it" rhymes with both "...chance, I'm takin' it" and with "no more fakin' it."  Sheer brilliance.

Word From the Boardroom
I know this might seem a bit odd, if this is the first time you're hearing about this.  I mean, people from other cultures sometimes think the Easter Bunny and Santa are a bit bizarre.  This is kind of like that.

So don't take my word for it; let's see what other people are saying.  My co-workers, the other people Makin' It.   Let's hear their tales, about how this year is going to be their year.  Are they as fast as they come?  Are they second to no one?   Let's find out:

For the last time:  I don't.  Know what.  You're talking about.  Please get out of my office.
Kate C., Senior Director
Awesome!  You go Kate -- the sky's the limit!

Yes, I remember that song.  I detest that song.
Kathryn P., Sr. Manager
Yeah Kathryn, you said it!  That corner office is as good as yours.

 Seriously, Brett, this is bordering on a behavioral disorder.
Clint H., Principal Analyst
Haaa!  Word to that, Clint.  You're on your way to the top!

Anyway, it's understandable that we all think Makin' It is our own personal theme song.  But, the truth is, everybody's really Makin' It, so it's all cool.  If someone happens to burst into song a few more times than someone else at work, or maybe if somebody plays the song a little louder in their cubicle, because they landed that big presentation, s'cool.  It's all cool.

April 1, 2014

Building Modular Forecasts

The Pain of Smarter Forecasting
Pundits love to expound about how we should use our data "smarter" -- and often they're talking about making smarter forecasts.   Seriously, is there a person alive who would publicly disagree to the importance of making forecasts smarter?   It's quite fashionable to nod our heads vigorously.

...But I wonder how many people have pondered what exactly that means.   What does it mean to make our forecasts smarter?  What does a "smarter" forecast look like?

A lot of people assume it means fancier-shmancier math.  Ya know, eigenvalues and matrix math and whatnot.  That's rarely the case.   ...Indeed (and this topic begs for a separate post), I've witnessed lots of Directors and VP's place undue faith that a statistician can solve behavioral or process problems -- which invariably ends in tears.  Math only gets you so far.

Using the data smarter usually means two things:  1. Performing more steps.  2. Using more disparate data sets.  Basically, a "smarter" forecast does more comparing, inferring, measuring, assessing, extrapolating, interpolating, checking.  A smarter forecast is a more complicated forecast.

Net-net, a "smarter" forecast isn't free; its cost is reduced system explainability.

These days, the goal of staying simple and explainable isn't even ignored, it's practically scorned.  Companies rush towards unfathomably-complicated  forecasts, as if they're the path to salvation.  On some cosmic level, executives believe that as soon as it's so friggin' complicated that nobody understands how the damn thing works, the riches will surely start rolling in.

And here's the kicker:  Forecasts naturally gravitate towards incomprehensibility anyway, which leads me to my first assertion:

Left Unchecked, Forecasts Become Integrated, Inscrutable and Deteriorated.

There isn't one reason for this.  There are probably ten; here are a few that spring to mind:
  1. People are more prone to add logic to, rather than remove logic from, a forecast.
  2. Forecasting algorithms can work in unexpected ways, especially on volatile input data, which means that a forecast is never fully understood, even right after it is written.
  3. The forecast's logic is often split over several systems and/or sets of code. As a result, changes in one area can inadvertently affect the results of the other.
  4. The forecast's data inputs naturally change over time, which causes input data to become stale, inconsistent or even corrupted. 
  5. It's hard to tell when there even is a bug, because it's predicting the freakin' future and past performance is never a predictor of future results.  As a result, problems can fester.
  6. The people who have mastered the heady math might not be good at explaining what they're doing.  ...Worse yet, they might not fully understand how the dang forecast is even used.  As a result, problems might not be detected.
Worst case scenario (which I have seen many, many times):  The forecast integrates a myriad of different data sources, each of which slowly deviates from the forecast's working assumptions about it.  Fewer and fewer people claim to grasp how the forecast works, and nobody wants to take responsibility for (or rely upon) a forecast that they don't understand.  Finally, its performance begins to deteriorate, and nobody is sure how to fix it. 

The Proposal:  Stay Simple and Modular
At the heart of the problem is that many companies build monolithic forecasts:  Too much functionality fed into a big black box, with multiple data feeds coming out the other side.  See that picture above?  That's how it ends up. 

So what do you do about this?

Well, you can ensure your forecasts do not become too complicated by breaking them into simple and well-understood parts, that can be mixed-and-matched.  In other words, resist the natural temptation to build a big monolithic forecast, and focus upon building a series of smaller, simpler, more well-understood components that can be used independently.

I'm not 100% sure how easy/possible/feasible this will be in your circumstances.  But I've gotten it to work on many forecasts.  Let me describe the general approach I took here, using Akamai as an example.

Case Study: Revenue Forecasting

About this client...
Before I describe forecasting revenue I conducted at a client (back in my consulting days), allow me to provide some detail about the company's business model.

This company provides content delivery service, and its customers' invoices are similar to cell phone bills -- only instead of delivering minutes of phone service, they delivers internet traffic.   Just like with cell phone plans, customers can "Commit" to a certain level of usage; generally, the higher a customer's Commit, the lower the per-unit price.  If they push more than their Commit, they pay a per-GB Overage rate -- just like the per-minute Overage when you exceed your cell phone minutes.

Customers with steady traffic typically prefer high Commits (to get the lowest per-GB rate), customer with volatile traffic typically have low Commits (so that they don't pay for traffic that they don't use).  But any customer's traffic can gyrate unpredictably in any given month.

However, one way in which this company is unlike a typical cell phone plan is that the contracts can get rather complicated.  (This is a common phenomenon in B2B companies, where premier customers receive tailored contracts.)   Point being:  Even if you know a given customer's traffic, calculating their invoice amount can be very tricky.

Revenue Forecasting Data
So, when attempting to forecast quarterly revenue, what factors must be considered?  As it turns out, A LOT OF FACTORS.  However, they can generally be broken down into four categories, each with three sub-categories:
  • Traffic Factors:  the levels of content that customers push
  • Contract Factors: the customer contracts Akamai has, and the terms of each contract
  • Invoice Factors: the amounts that appear on the actual invoices
  • Revenue Factors: the amount of invoicing that Akamai recognizes (versus deferrals, rev reserve, credit memos, etc)
 Within each of these categories:
  • Current Data:  The most recent month's data.
  • Historical Patterns:  Patterns in how this data has changed over time.
  • Known Current Adjustments: One-off events scheduled to happen within the forecasted period.
Old vs. New Forecasting Methodologies
Until 2009, the forecast took all of these quite-different inputs, and incorporated them all into a single mega-algorithm, like so:
 Since then, Akamai has transitioned to a new modular forecast.   It works like this:
1. Use only traffic information to make a traffic forecast.
2. Next, use only contract information to forecast what our contracts will look like.
3. Then input our forecasted traffic and forecasted contracts into our invoicing system, to calculate forecasted invoicing.
4. Finally, use historical revenue data to forecast how much of this invoicing we will recognize as revenue.
Benefits of a Modular Approach
You might be wondering:  "Okay, so you went from one super-complicated forecast, to a series of four less-complicated ones.  Did you really come out ahead?"

The answer: Way ahead.  Here are some of the benefits:
  1. Forecast accuracy improved significantly.  As you might imagine, Revenue is really just a bunch of adjustments to Invoicing -- and in this case, we're calculating forecasted Invoicing, based upon forecasted Traffic and Contracts:  We literally use our production Invoicing system to say "Okay, if the contracts look like this, and the traffic looks like that, then how much money will we invoice?"  For a variety of confidentiality reasons, I can't cite specific accuracy levels, but believe me: nobody is complaining about accuracy.
  2. Forecast intuition is very high.  Each part of the forecast (e.g., our traffic forecast, our contract forecast, etc) had sufficiently few inputs that people could wrap their heads around how it worked, and what events would affect the forecast.   As a result, the forecast hasn't turned into a black box, like many other forecasts I've observed. 
  3. Forecasting modules can be swapped in and out.  Sooner or later, we'll need to revamp one of the forecasting modules diagrammed above; but because the forecasts are modularized, it won't introduce risk to the other forecasting modules.
  4. Our intermediary forecasts now sing for their supper.  Originally, we modularized our forecast to yield a simpler and more-accurate model -- but we attained another benefit:  We discovered that groups within the company could use these intermediary forecasts!   Engineering, Finance and Sales use our Traffic, Contracts and Invoicing forecasts -- and even spot opportunities to improve them, which yields a more accurate revenue forecast, to boot!
Conclusion
In most organizations, forecasts naturally gravitate towards becoming monolithic beasts that nobody fully understands, and which then deteriorate over time.  However, by moving towards more modular forecasts, a company can fight this trend, and maintain understandable, accurate forecasts for much longer.