The scores for Romance Writer’s of America’s (RWA) unpublished manuscript contest, The Golden Heart, have been sent and there’s already some discussion on loops about what the scores mean and whether it’s working.
First, some background. The Golden Heart’s mission from RWA’s website is to:
…promote excellence in the romance genre by recognizing outstanding romance manuscripts
This is a coveted contest final in the Romance world. After finalists are announced, agents regularly give automatic requests for fulls to them. At the national conference, the finalists are announced and winner awarded during the final gala night in an Oscars-like setting.
This is the first year a new scoring system has been used. In the past, the judge gave one overall score based on how they viewed it. There’s been criticism in the past that this wasn’t based on any specific criteria. This year, judges were asked to break down their score into the following categories:
Romance (1-20)
Writing (1-10)
Characters (1-10)
The Story/Plot (1-10)
For a total score of 50.
On the loops I belong to, writers are asking what it means when they get a wide range. Some have said it’s as varied as 11 (as a total score!) to 48, for the same manuscript, and others are reporting the same wide range. I’ve also seen some writers say they received a 1 or a 5 for their Writing category. In the instance of one of the ones who got a 5, I actually Beta read that manuscript and their writing (which should be based on the craft, i.e. grammar, command of language, etc) was not 5, IMO.
It’s a true adage, that in contest feedback, large swings in opinion can mean that you have a strong voice and so you’re alienating some folks who just hate your voice.
Golden Heart this year has taken care of some of the unfairness in these large swings by dropping the lowest AND the highest score, but I’m wondering if there’s more to it than this. I judged entries this year and here’s my thoughts:
There was no grading scale given to help orientate the judge on what a 1 as opposed to a 5 or a 10 means (other than that 1 was on the low end, and 10 was a perfect score). Lacking this, I made one up for myself by taking the grading scales used in local chapter contests. So when I judged the entries, this was the criteria I used by taking the 1-5 scale used in local contests and extrapolating it out to:
9-10 Ready to Publish, no changes needed.
7-8 Almost there.
5-6 Several minor problems.
3-4 This area could be strengthened with some significant rework.
1-2 Major problems in this area.
And for the Romance category, I used this:
17-20 Ready to Publish, no changes needed.
13-16 Almost there.
9-12 Several minor problems.
5-8 This area could be strengthened with some significant rework.
1-4 Major problems in this area.
And so I gave my scores accordingly. But do you see the problem here? I did this on my own. Who knows whether this is what the coordinators had in mind? Who knows what other judges used to assign their numbers?
In many local contests that use a scale, they give what the judge should look for in each category. Perhaps a way to improve this would be to give some kind of scale guideline for each category in order to take out this part of the subjective equation. Because yes, every judges opinion is subjective, but how to use the numbering system shouldn’t be subjective.
Also, some folks had high scores in all categories except romance, with their romance number being 7s and 8s, consistent with what they were getting in the other categories. So, it makes one wonder if the judge didn’t realize the scale went up to 20?
I also had an interesting phenomenon happen. I had one entry I judged that I thought was so great, I gave it a perfect score (the only one I gave). The writing was great–sharp writing, sizzling sexual tension (I was literally squirming) and the synopsis was well done in that the plot was crystal clear, plausible and the character’s goals and motivations were all clear and made sense (it was the only one that did). I was surprised it didn’t final and then I saw that it was published, so it must’ve been disqualified. Anyway, I bought it, so I could read it, and the story completely did NOT hold up. The prose was still technically flawless, but man, for the Black moment/final climax, it totally hinged on the character doing something completely out of character as it was written (but which in the synopsis made it sound like it was totally their character) and also the characters never really got fleshed out past cardboard cutouts to serve the plot. Just goes to show how only reading first 50 and a synopsis truly do not help pick the best. Jami Gold’s post yesterday touches on this in her post “Why Is Storytelling Ability So Important?” based on judging a recent contest.
So here’s some thoughts I have on how the contest could be improved for the future:
- Give a scale on what each number means for each category
- If Romance will still count double, perhaps double the score after the scores are turned in. I’m sorry, but there’s some people who just don’t pay attention. How many folks missed out on finaling because they were unlucky enough to get more than one judge who didn’t look close enough to see the scores went to 20 in this one category only?
- Unfortunately, I think asking to read a full will be too much work, so fixing the instance I found where the problems exploded only once the full was read probably can’t be addressed. RWA had a hard enough time getting enough judges this past year
What do you think? I see this as a place to discuss the new scoring system and whether it worked or didn’t. Is it an improvement on the old system? Do you have some suggestions on how it could be improved for next year?
I both judged and entered this year(different categories). I found the judging difficult for just the reason you state. I like your idea of a scale or at least some point of reference. I also nearly got caught out by the romance being out of 20 but corrected it on time. I took my time with the judging and read each entry 3 times to be sure I was happy with the score I gave. It’s difficult to find the happy medium. I am wondering if judges shouldn’t be asked to give a short account for their scoring? I know that means more work but a score, at the end of the day, is not that helpful to an entrant.
Thanks for you comment Sarah! I wonder how many others didn’t catch the 20 scale like you did? You raise a good point. I think it might come down to what the GH is for, and I’m not sure whether detailed feedback is it. I think chapter contests are great for that. But that said, it would be nice to maybe have at least one line or two from the judge maybe, saying why it didn’t hit the mark?
In the past, I have judged the Golden Heart and experienced the same issues you did on scoring. RWA needs to assign some kind of weight to the scoring rather than making it so subjective or rather than 1 – 10 move it to 1 – 5. I didn’t judge this year, but it sounds like the 1 – 20 category would have thrown people off. I don’t think this is the first time this issue is arisen. BTW, nice blog. :0)
Yeah, I think the 1-20 for just ONE category was potentially an issue. I know of one person who had more than one judge score that category between 1 and 10, so that might have knocked her out of finaling, and she deserved to final. I do like the 1-10, but the 1-5 might make it easier to explain the differences in weight. Though perhaps the could group them like I did and that would give the judge leeway on putting it on a lower or higher end of that 2-point range for that particular description/weight. Thanks for commenting!
I’m all for any system that makes the judge’s numbers mean something. Since for the vast majority of us, the scores are all we get for our $50 (I choose to think of the GH along the lines of getting a lottery ticket) it would sure be nice if the numbers were a little less opaque. Nice for the judge AND the entrant.
or if the entrant at least was privy to the weight descriptions for each number? That way they could judge how close they are. GH is like a lottery, but I think the $50 is for a chance to final and get that agent/editor, not necessarily for the feedback, though that might just be me. I tend to see the chapter contests as the avenue for feedback and a way to polish up the MS to have a chance in the GH. Thanks for commenting!
I judged for the first time this year and boy, I wish I’d had your scale to go by. I think I may have subconsciously used something similar. I kept in mind published books that I thought were well done and sort of held up the entries to those.
I had a strange score also. One judge gave me very high marks (8) for everything except the romance portion where she gave me a 7. I wondered if she thought it was 1-10 also. Overall my scores were evenly split. One high, one low, three in the middle.
I’m going to keep your scale for next year. Thanks!
A scale would be good, though I wonder if putting it up to published books is fair, since they had the benefit of a professional editor? I tended to look at it as to whether it was ready to be submitted to an agent/query
I did not judge or enter (I’d already sold, so there was no point). The problem, as I saw it, was that one was not allowed to judge in the catagory in which they write. Quite frankly, there is a reason I don’t read some genres, I just don’t get them, so I felt I could not be a fair judge.
I thought you only couldn’t judge in a category if you were entered in that category?
You can’t. I’m talking about those not entered.
That’s what I’m saying too– you should’ve been able to judge since you were not entered in the contest this year…
Yes. Yes, yes, and yes.
I agree, Angela. Well said. Limiting subjectivity by giving meaning to the numbers would be wonderful. Great post.
Yep, there’s already enough subjectivity to make it a crap shoot, so seeking out ways to pare down unnecessary subjectivity I think is worth working on for next year.
Very interesting post. As with some other readers, I both judged and entered this year. As judge, I likewise tried to create some definition for how I was awarding scores, but also realize this was what everyone was doing. A bit more definition would have been helpful. I’ve also wondered if potentially some of the scores could have been affected by the heavy load of entries each judge was surprised by this year (I had 9 and have heard this wasn’t uncommon). As with the scoring, how individuals handled their “judging duties” could also have differed – all at once, leading to exhaustion and impatience might have affected scores. Anyway, very interesting post, and I’m sure RWA will be considering some of these things, as the GH remains such a prestigious contest for romance.
Yep, so many factors go into it, that I think giving some kind of weighted guideline for the scores would help the judges more. I only got 4 entries, though I might have marked that I could only do 1-2, so that might be why I got saddled with less than others?
I recently judged my first contest, not the Golden Heart. They not only provided a scale similar to the one Angela posted, but a sheet explaining each type and how to interpret. I felt pretty good about my judging and evals because I had something to fall back on. I was also able to write comments on the judging sheet to explain specifically what made me score other than full value to a section. I hope with good intentions on my part, that the authors are able to get some help from this feedback. It’s still subjective, but at least the author has something to go on to improve their work.
Yep, I loved doing chapter contests last year for exactly that kind of feedback. I think it might be too much to do that kind of thing for GH, since I think the goals for entering it are different (IMO), but at least some kind of weighted description would be good to help guide judges. Thanks for commenting!
This was the first year I didn’t enter but wanted to judge. I could not figure out the website where I was supposed to register to be a judge (was it myRWA? another site I was rerouted to? Don’t remember). I contacted RWA to ask for help, but didn’t hear back. I know they had trouble getting judges this year, and I wonder how many people had trouble just offering to help. (I do have one friend who told me later she had the same problem this year.)