A short history of standardized test preparation

By Jose Ferreira, Ceo Of Knewton
05/29/2015
Share

The test prep industry began in Stanley Kaplan’s Brooklyn basement just after World War II. The son of Jewish immigrants, Stanley believed standardized tests were some kind of elitist establishment plot to keep minorities out of the Ivy Leagues. His philosophy was that anybody could improve at standardized tests. The testing industry and larger education establishment loudly proclaimed that to be wrong, foolish, and practically immoral. But Stanley, standing alone for a long time, was proven right. In 1984, The Stanley H. Kaplan Company was bought by the Washington Post/Newsweek Company. The New York Times laughed. Today Kaplan, if it were a standalone company, would be bigger than the rest of The Post and The Times combined.

I joined Kaplan in 1991. I had just graduated from college and moved to San Francisco without a job. It was the middle of a pretty bad recession, but the good news was that I had majored in Philosophy. Turns out that made me unemployable. I also began taking standardized tests for fun—in the morning with a cup of coffee, the way other people do crosswords. I found a hidden structure to standardized tests that makes them a lot easier.

A friend said, “You should teach for Stanley Kaplan.” So I did.

At the time, Kaplan’s product consisted of 36 hours of live instruction in 9 four-hour classes. The curriculum was heavy on content and light on strategy. Or anything fun. By hour four, the class resembled not so much a roomful of go-getters as it did a clinic for narcolepsy. The course also featured a lot of practice material you could only access at the local Kaplan center. Audiotaped explanations were provided via bulky old tape players from kindergarten. Students from other parts of the country found the heavy Brooklyn accents incomprehensible. It was like having Luca Brasi from The Godfather explain algebraic equations to you. But Kaplan management was very proud of the tapes. Without irony they referred to them as “technology.”

Students weren’t allowed to keep their lesson books. At the end of each class, they were required to hand that day’s booklet back in. This drove students insane. But Kaplan management believed that students only took the course because of “the secrets” in the lesson booklets — yes, arcana like “arithmetic” and “triangles.” By forbidding students from keeping content they’d already paid for, Kaplan was saying in effect, “We’re more worried about our students stealing our intellectual property than about improving their scores. Which, of course, giving them the materials would tend to do.

While Kaplan was sharing this revolutionary customer service philosophy with the world, a small company was being started by Princeton grads John Katzman and Adam Robinson. The Princeton Review (“TPR”) electrified the test prep world with its “Joe Bloggs” system, based on how a perfectly average shmoe would answer any given question. Easy questions had answers that looked right to Joe Bloggs; difficult questions had answers that looked wrong. Joe Bloggs was fun to teach and to learn. But it was fixable; the test-makers could change their tests to kill Joe Bloggs. And, since John Katzman loved nothing so much as mocking the tests publically, that’s precisely what they did. RIP Joe Bloggs.

The smartest thing Katzman did was create test prep guides for bookstores. Kaplan, with their General Jack D. Ripper-esque obsession with secrecy, had of course never published a bookstore product. So TPR had the shelves all to themselves; Katzman used bookstores as a marketing channel to spread the TPR brand. Instead of bleeding cash to build the TPR brand, Katzman found a way to get paid to do it. TPR rose quickly, and Kaplan, with their audio tapes and lesson book lockdown, quickly lost market leadership in SAT prep.

Fortunately for Kaplan, The Post Co’s Don Graham began to make changes, replacing the ancien régime with a team of Harvard Business School grads. Kaplan’s new CEO, Greg Rorke, had previously turned around Danskin. Now he got Kaplan focused on technology for the first time. He brought in a lot of charm and energy. He also brought in Jonathan Grayer, who would eventually replace him, from Newsweek’s ad division. Stanley was still around, but only in an avuncular, non-executive kind of way. Eventually he lost the corner office.

The new Kaplan, minus Stanley, faced a challenge other than new competition: computerized testing. The Educational Testing Service (ETS), long time publisher of the GRE, SAT, and GMAT, was moving the GRE and GMAT to computer format. Since computer tests are given five days a week rather than five times a year, this would inevitably disrupt Kaplan’s entire sales infrastructure and field operations. It would presumably also require us to develop technology more sophisticated than audiotapes.

I was made GRE Product Director. My marching orders were to “figure out this computer thing,” and quickly make product for a brand new GRE question type called “Pattern ID”—a bewildering cocktail of math and logic that befuddled everyone who saw it. Most people couldn’t even understand the directions. But it could be batch-created by computers in vast quantities. Since computerized testing requires lots of questions in order to be secure, Pattern ID was a godsend for ETS.

Almost accidentally, I created a Pattern ID strategy that was so powerful that students could get all the questions right without doing any work. I was conflicted. If I kept quiet and let ETS administer Pattern ID on scored exams, ETS would eventually be compelled to cancel every score, affecting hundreds of thousands of students. I had delusions of making it into Time magazine. But doing so would force all those students to retake the GRE. Instead, I gave ETS a heads-up. They wound up throwing away hundreds of thousands of test booklets and tried to forget about Pattern ID. One of them offered me a job.

ETS admitted to a major periodical, and by that I mean the Louisiana State University Daily Reveille, that I “broke the code and published it, so we are removing the questions from the test.” This marked the only time in the history of standardized testing that a major test-maker admitted they removed questions because they’d been decrypted by a coaching strategy. Kaplan put that quote on every direct mail postcard they sent out for something like ten years. Even nursing students in the Philippines got postcards about it.

ETS’s problems with Pattern ID did not slow their drive to administer tests on computer. They launched the computerized GRE in 1994, with plans for the GMAT to follow soon. Pattern ID had taught me that the more tests are based on programmatic rules, the easier it is to break open their underlying structure. I sent researchers to take the new test and immediately discovered two new Pattern ID-sized fiascos.

First, I reverse-engineered the scoring algorithm. I found multiple weaknesses and began coaching students to skip the last questions in each section and spend the extra time on the first questions. This produced score gains of up to 100 points per section. ETS later admitted as much to U.S. News and started recommending it themselves to level the playing field. That was another first: the test-maker giving advice to students on how to game their own system! Eventually they gave up and rebuilt the algorithm to make this approach less effective.

Second, I reverse-compiled the question pool. The way it works is: computerized tests require the same pool of questions to be used for weeks at a time. So question pools have to be enormous to prevent cheating. But my researchers were all seeing a dozen or more of the same questions—almost mathematically impossible if the pool is large enough. Since ETS had testified under oath before the New York State Senate that the pool was indeed large enough, this had the potential to be perhaps the most fascinating statistical anomaly of all time.

The question pool we handed over to our accounting firm, Price Waterhouse. Then we contacted ETS and said something along the lines of, “Umm… wow, this is a little awkward, but we have your test.” At first, ETS issued a press release thanking us. Two weeks later they sued us on the basis of copyright infringement and named me personally as a defendant. Don Graham, in a visit thereafter to Kaplan headquarters, placed his hand on my shoulder and said, “You know, you’re the youngest person ever to get the Washington Post/Newsweek Company sued!” Lucky for me, he seemed more bemused than anything else.

Deans and docents around the country had been calling ETS asking if the GRE could be trusted. Not questions you want to hear asked about your eight-figure investment in computerized testing. So they sued Kaplan and told everyone else not to worry, that only a fiendish mastermind could compromise their test’s security. In reality, though, pretty much any buffoon at your college fraternity could have compromised that test.

The parties agreed to a token settlement. Later on it emerged that ETS had for some time been referring to me as “the antichrist.” Needless to say, we didn’t put that on any postcards.

Jonathan Grayer, now CEO, tabbed me with leading a Kaplan-wide initiative called “Project Footprint,” the stated purpose of which was to build everything from scratch the way it ought to be built. We streamlined the course to 8 three-hour classes. We replaced the audiotapes with CDs. And, of course, we let students keep their materials.

I left Kaplan shortly thereafter to go to Harvard Business School. My timing was impeccable. Kaplan went on a tear that continues unabated today, growing from around $70 million in annual revenue to around $2.5 billion.

Kaplan’s new leaders weren’t education people. Some of them referred to our products as “widgets,” admitting they would be just as happy selling Charmin or dreaming up new marketing campaigns for the Frito Bandito. It often seemed to me that they considered test prep just about the most stupid little industry ever devised by man. And maybe it is, though they have done a marvelous job of making it a bigger industry. They’ve accomplished this by being obsessed with numbers, calling area managers every day to get the latest. Unfortunately, the numbers they’re obsessed with are daily revenues, not student score increases.

Today’s Kaplan course is not very different from the one I built in Project Footprint: still 20-odd hours, with the same Preview-Classroom-Review architecture. As far as I can tell, Kaplan’s biggest innovation in the last decade has been to realize that many people aren’t ultimately price sensitive when it comes to determining their entire future. When I first started at Kaplan, the price of a GMAT course was $590. Today it’s $1,490.

What’s now clear is that the traditional business model of local “Strip Mall Test Prep” is hopelessly archaic. Nearly all the money you spend on old-school test prep is wasted on local teachers, centers, center staff, and marketing. And all the paper, printing, shipping, and commuting requirements are about as eco-friendly as a styrofoam Hummer. These things used to be necessary for product delivery, but never added to product quality. Now they’re just the waste byproducts of a bloated, outdated infrastructure, propped up by your hard-earned dollars. The future of test prep is online, and moving there quickly. Instead of having a mediocre teacher, you’ll get the very best teachers and test experts in the country, any time you want, for as much time as you need, for a lot less money. I started Knewton.com to do exactly that.

People always ask me whether I think standardized tests actually measure anything. I used to give nerdy statistical answers about scoring validity and concordance tables. Eventually I realized that what these people want to hear is: “Standardized tests combine the diagnostic efficiency of a banana republic health care system with the personal touch of the airport immigration line.” It helps to throw in a reference to the Nazis for good measure.

The reality is that standardized tests are very effective predictors of academic success in college or graduate school. That’s a fact of statistics, whether we like it or not. It’s also just an average, and you may be an exception one way or the other. That’s why standardized tests are just part of the application process, and are never (or, should never be) the most important part.

Another pop psychology protest is that standardized tests are ipso facto biased against minorities since African Americans and Hispanics on average underperform the mean. This argument is about as logical as divorcing your spouse upon reading that most billionaires are single. Other minority groups perform at or above the mean, so the issue isn’t ethnic identity. And it isn’t the tests; every question is now statistically vetted to eliminate any cultural bias. The failure is in society itself.

Yet a growing number of colleges say standardized tests don’t measure anything and are making the SAT voluntary. Some of these schools use intellectually dishonest arguments like: “high school grades are a better predictor of college performance.” That’s true, but high school grades combined with SAT scores are a better predictor still. So why not use them? After all, getting more information can’t possibly hurt—can it?

My suspicion is that schools making the SAT voluntary are doing so for affirmative action purposes. In 2003, the Supreme Court significantly narrowed the allowable use of affirmative action in university admissions. Shortly thereafter, admissions rates across the country declined for Hispanics and African Americans. Since these groups historically underperform the mean, SAT scores tend to be a mildly deflationary force on their overall admissions rates. The ivory tower solution? Drop the test so that affirmative action admits are easier to justify. And, wherever possible, shoot the messenger. It’s come full circle: The folks who marginalized good ol’ Stanley Kaplan, who once demanded standardized tests in order to keep minorities out, now attack the tests so they can let more minority candidates in.

Discussion

s