Two weeks ago, after reading the New York Times Article which judged the best high school writers harshly, Chad Orzel came up with an idea that was so good it just had to be tried:
Somebody ought to get a bunch of bloggers together, and give them the writing SAT under timed conditions, and see what they come up with.
I took Chad up on the challenge, and together we created the Blogger SAT Challenge, giving writers from across the blogosphere the chance to show that they can do better than high school students. How did they do?
Well, 500 people looked at the essay question, but just 109 were able to complete a scoreable response under the time limit. That alone should demonstrate that the SAT challenge was no easy task. But what about the responses? Can seasoned bloggers do better than high schoolers?
We sent the essays, with the identities of the writers concealed, to 6 volunteer graders with experience teaching college writing or working for an SAT prep service. Each essay was graded by two different people. For the real SAT, the scores (on a scale of 0 to 6) would simply be added together, but we also want to know what blog readers think of the essays. So we created a single composite score based on the two grades by averaging the scores and rounding up: 4 and 5 would average to 5, while 3 and 5 would result in a final score of 4.
Now you get to rate the essays on the same scale: Chad and I -- with a little help from some friends who actually know how to code -- have designed a site which presents each of the essays and lets you score them yourself. You can then see the score the essay was given by our professional volunteers. The site is live now, and you can visit it here. We only ask that you rate at least 10 of the essays, using the "Random Entry" feature, so that eventually all of the essays get rated.
If you wrote an essay, you can seach for it and claim it by linking to it from your blog, or by writing a comment below your entry. You'll even find a badge with your score which you can post anywhere you've got password access. Send your friends over to rate your entry and those of your competitors. It's completely unscientific, but it's also the way of the Web, and we want to know how Web readers respond to the essays. Your entry will remain anonymous until you claim it, so if you're not pleased with the result, you can slink quietly away and no one will be the wiser.
There will be much more analysis of the scoring in the days to come, but I did want to give you a taste of the results. The average score was 2.9. When put into SAT form (adding the grades, rather than averaging them), the average was just 5.32, which compares rather unfavorably to the average high schooler's score: 7.2. The median grade was 3, and here's how the scores broke down:
As you can see, nearly everyone scored a 2, 3, or 4. There were just 21 5s or 1s, and only one score of 6. Even the 6 isn't SAT-perfect, because it was composed of one 5 and one 6. In the real SAT, less than 1 percent of students scored a perfect 12; in our test, no one did. The initial goal of our test was to compare the best high schoolers' writing -- tests scoring a perfect 12, with the best bloggers can do. In a future post, I'll take a closer look at the essays in our test that scored the highest.
For now, I invite you to read them yourself; then let us know what you think, either in the comments below each essay, or here in CogDaily's comments section.
- Log in to post comments
That is not surprising...everyday that I read Digg.com it proves to me how many idiots there are out in the world. Digg.com is the perfect example of "If you don't know the subject, then just shut the hell up"...too many people think they know something but really don't know jack! Actually the biggest problem is "Not Re-Reading and Correcting Your Text before hitting SEND"...same goes for people sending emails. It is very tough to write something good in a short amount time.
I would make two points about by this study. First, most blogs are crap but the web does a great job of creating a repuatation for each author. The effect of this reputation system is that the blogs we're most likely to read are quite a bit better than the best high-school student's essay.
The second point is it's unlikely the bloggers who took this test take this even half as seriously as the high school students for which this test is a life defining, life changing event.
So SAT tests and blogs are two quite different animals. You'd be much better off testing high school students' blogs versus the blog population as a whole. How good is the average blog? How many of our most read blogs are written by high schoolers? How does age or writing-industry experience relate to popularity?
Awww, I wish I'd found this earlier!
I would have shown those upity High School-ers who's boss.
Cognitive Daily,
Wow! This challenge/exercise/open rating system was brilliantly conceived and well executed. I'm astonished at the variance in my ratings vs. the professionals--sometimes the pros and I were smack on, at others, we were miles apart. I suspect some of this is due to only two "pro" graders per paper. I also suspect some of it is due to the training of me (I looked at the pro grades before moving on to the next essay--you should ban this in future similar exercises.)
Also, I found this exercise personally very timely. My son, a junior in high school, is treading into the world of placement exams and standardized tests. He loathes the essay, but perhaps, by witnessing how poorly avocationally inclined writers scored, he can better bear the essay writing coming his way.
"I also suspect some of it is due to the training of me "
This sentance is onstructed awkwardly. Please use: "I also suspect some of it is due to my training"
"This sentance is onstructed awkwardly."
Constructed has a 'c' at the beginning and "This sentence is awkward" would have been more succinct.
Now you might understand how hard it is to write a quality essay in 25 minutes. Granted, the bloggers received only 21 minutes, and therefore did not have the advantage of time to outline and plan out their essays as we "uppity High School-ers" do, but I'm not sure they would have used the time to outline anyways.
I think I only saw the techniques you are supposed to use in an SAT essay in a couple of the essays (and yes, I went through quite a few of them).
1) The clearly defined thesis statement illustrating your argument - not your position, your argument - was missing.
2) Specific examples and explication (i.e., how does this support your argument?)
3) At least one example from literature, one example from history, and maybe (if time permits) one from personal experience.
The 5 paragraph essay format is advisable, but not required, so I'll give that a break.
Anyways, you bloggers have an advantage - you don't have to stay in the exam room for another 3 hours asnd 20 minutes to do more math, science, and writing...
"This sentance is onstructed awkwardly."
Does spelling count? In addition to 'constructed', 'sentence' shouldn't have an 'a'.
Out of five words, three were spelled incorrectly:
"This sentance is onstructed awkwardly."
Oh, the irony, simply mind boggling.
"Out of five words, three were spelled incorrectly"
Ummm... actually only two were. That's why this sort of discussion in comments boards usually infuriates me. It's actually not so bad in this case, since at least we're talking about writing, but still...
Keep in mind, also, the students writing this had no computer access while writing, whereas the bloggers did. Believe me, it makes a difference.
"Keep in mind, also, the students writing this had no computer access while writing, whereas the bloggers did. Believe me, it makes a difference."
Absolutely. That's why the schoolkids did better. They weren't distracted by the porn.
D Medberry wrote: "I also suspect some of it is due to the training of me ..."
grammar guy suggested: "Please use: "I also suspect some of it is due to my training""
Note that Mr. Medberry's original sentence did not mean "I suspect that my previous education caused some of" the variances and correlations he observed between his scoring and that of the experts. What he meant was "I suspect that seeing the expert's scores on early essays trained me to imitate their scoring methods as I rated later essays." Mr. Medberry's original sentence disobeys no grammar rules and has the virtues of being succinct and conveying the writer's individual tone. The suggested correction, while it employs a more commonplace construction, changes the meaning.
Just because I didn't have a computer doesn't mean I wasn't distracted by porn.