Reminder that nominations for the 2009 Reader Survey Wine of the Year and 2010 Wine to Watch are due by Monday 12/7 at 9pm. Read about the survey here.
Last month I wrote two posts comparing how the major publications review Washington wine. In the first, I compared the breadth of coverage of Washington wines and ratings of wines for Wine Advocate, Wine Spectator, and Wine Enthusiast. In the second, I looked at the publication rate of Washington wine reviews for these three publications. Today I will focus on their sampling, review, and publication practices.
My goal in looking at this was to determine, as a reader of these publications, how each determines what to review and publish and how they do it. I also wanted to provide background information and context for some of the numbers written about in my previous posts. To this end, I discuss here for these three publications: how wines are solicited and sampled; how wines are rated; and how reviews are published. For those interested in the nitty gritty details, this one is for you. For those who are not, the summary starts in the third to last paragraph.
Writer Jay Miller, PhD has covered Washington for Wine Advocate since 2007. Miller brings an extensive knowledge of the state’s wineries, having sampled Washington wines since 1990. For Wine Advocate, Miller solicits samples from specific wineries of interest. Wines are solicited based on previous experience and current information. In addition to specific, solicited wines, Miller works with two local organizations – the Washington Wine Commission and Walla Walla Wine Alliance – to obtain samples. For these samples, the Commission and
Miller spends approximately two weeks each year in Washington reviewing wine. During this time, he samples wines both at wineries and the Commission’s offices. Every wine that is submitted is sampled and rated.
To rate wines, Wine Advocate uses the 100 point scale institutionalized by Robert Parker. Wine Advocate describes its ratings as follows:
96-100: An extraordinary wine of profound and complex character displaying all the attributes expected of a classic wine of its variety. Wines of this caliber are worth a special effort to find, purchase, and consume.
90 – 95: An outstanding wine of exceptional complexity and character. In short, these are terrific wines.
80 – 89: A barely above average to very good wine displaying various degrees of finesse and flavor as well as character with no noticeable flaws.
70 – 79: An average wine with little distinction except that it is a soundly made. In essence, a straight forward, innocuous wine.
60 – 69: A below average wine containing noticeable deficiencies, such as excessive acidity and/or tannin, an absence of flavor, or possibly dirty aromas or flavors.
50 – 59: A wine deemed to be unacceptable.
To determine which of the sampled wines to publish, Wine Advocate uses a rating cut-off. Wines that receive a rating of 85 points or above receive a written review that is published in print and on-line. Wines that receive a rating of 84 points or below are not published. No additional criteria are used. In deciding on the 85 point cut-off Miller says “We (The Wine Advocate contributors) feel that a wine scoring 85 or better can be recommended…We have debated listing wines tasted but not recommended but this is what we have settled on.” Miller estimates that in 2009 60-67% of wines sampled received ratings of 85 points or above. In 2009, Wine Advocate’s published ratings for Washington wines have ranged from 85 to 99 points.
For Wine Spectator, Editor-at-Large Harvey Steiman, who has been covering Washington since the 1970s, reviews wines that were either submitted by wineries or were purchased by Wine Spectator. Submitted samples include wines Wine Spectator has requested from wineries as well as unsolicited samples.
Tastings for official review take place only at Wine Spectator offices or other venues Wine Spectator controls. While wines may be sampled under non-blind conditions, such as at a winery, tastings for official reviews – those that are published in print or on-line – are always conducted under blind conditions (read more about the tasting format here). Executive Editor Thomas Matthews states “We consider that adhering to a consistent, independent, blind tasting policy is the essential foundation for delivering fair, unbiased and objective wine reviews.”
All wines that are solicited for tasting are officially reviewed. For Washington, in the case of unsolicited samples, Steiman may “screen” wines in unofficial blind tastings to determine whether to include them in official blind tastings. Wines are screened out that 1) are of insufficient quality 2) are made in insufficient quantities or 3) are overpriced based on the rating in the unofficial blind tasting. In 2009, Steiman estimates that one hundred Washington wines were screened out based on these criteria.
In rating wines, Wine Spectator uses the 100 point system. The publication describes its rating system as follows:
95-100 Classic: a great wine
90-94 Outstanding: a wine of superior character and style
85-89 Very good: a wine with special qualities
80-84 Good: a solid, well-made wine
75-79 Mediocre: a drinkable wine that may have minor flaws
50-74 Not recommended
All wines that are officially reviewed are subsequently published in print or on-line. Wine Spectator does not make any rating cut-off in determining what reviews to publish. In 2009, Wine Spectator’s ratings for Washington wines have ranged from 78 points to 98 points.
Contributing Editor Paul Gregutt has been reviewing Washington wines for Wine Enthusiast since 1998 and has been writing on Washington wine since the 1980s. Gregutt lives in Washington and splits his time between homes in Seattle and eastern Washington. For samples, wineries submit wines directly to Gregutt (see Wine Enthusiast ‘Wine and Spirit Submission Guidelines’ here and a ‘Tasting and Review FAQ’ here). Gregutt also works with the Washington Wine Commission and the Walla Walla Wine Alliance to source samples for review. All wineries are encouraged to submit samples.
Gregutt samples wines at the Commission offices, at wineries, and in private settings. All wines submitted for review are sampled and rated. In terms of rating, Wine Enthusiast also uses a 100 point system which it describes as follows:
Classic 98-100: The pinnacle of quality
Superb 94-97: A great achievement
Excellent 90-93: Highly recommended
Very Good 87-89: Often good value; well recommended
Good 83-86: Suitable for everyday consumption; often good value
Acceptable 80-82: Can be employed in casual, less critical circumstances
All wines submitted for review are published either in print or on-line with Gregutt listing the following exceptions: “wines that have already been reviewed (I don’t go back); wines from really old, back-dated vintages; wines with mysterious bacterial issues; wines that are extremely limited, wine club only type releases; wines from brand new wineries that I have nothing good to say about (I like to give them a break).” In 2009, Wine Enthusiast’s published ratings for Washington wines have ranged from 80 to 100 points.
Reviewing this information, the sampling, rating, and publication practices of each publication are similar in some ways and unique in others. In terms of similarities, speaking generally, each publication employs a ‘beat’ writer dedicated to covering Washington with a long history with the area’s wines. Each publication aims to cover as broad a selection of Washington wines as possible by soliciting samples in some manner. The rating system each uses, while the definitions have some variation, is based on a 100 point system. Finally, each uses some method to screen out certain wines they do not believe will be of interest to readers.
In terms of differences, again speaking generally, each takes a different approach in terms of how they obtain samples for review. Wine Spectator puts a premium on blind sample tasting under controlled conditions (I won’t discuss here the arguments for or against as it has been covered elsewhere by others). The publications differ in terms of how they screen out certain wines they do not believe will be of interest to readers. Wine Advocate is the most distinct in its approach where all wines that receive a rating lower than 85 points are not published.
In closing, I would like to thank the individuals at each publication and organization for their assistance with this information. I will conclude the series in the next post with some final thoughts. In the meantime, as always, please feel free to send along any comments.
Sean, I very much appreciate your thoroughness and objectivity – look forward to your final thoughts. A couple of notes to add – Dr. Jay Miller, who did in fact visit WA in the early 1990s, did not revisit the state (or sample its wines methodically, at least to my knowledge) until Pierre Rovani resigned about four years ago. Harvey Steiman has indeed consistently covered the state's wines for about as long as I have. I believe that I am the only reviewer among the three who also writes about WA wines without using the 100 point system. In my upcoming book I abandon numbers altogether; I have written extensively on the limitations of the system. Nonetheless, it dominates the national press and I am comfortable using it for the publications that prefer it.
WS may claim that they only taste blind, but if you prod enough of the top ranked producers you will find that they never submit their wines for blind tasting. If you want to review the wine then either buy it, or come and taste with the owner/winemaker.
Sean, This report is an accurate description of Wine Spectator's tasting policies. Thank you.
Re: "anonymous" above — I am not sure what point you are trying to make, but I can assure you that if you read a wine review in Wine Spectator, it is the result of a blind tasting in our offices (unless explicitly noted in the tasting note). Our editors frequently visit wineries and talk and taste with winemakers and owners, but we never review wines officially except in blind tastings in venues we control. We believe that is the only way to guarantee a level playing field and unbiased judgment.
Thomas Matthews
Executive editor
Wine Spectator
Paul, thanks for the comment. Looking forward to the next edition of the book.
Thomas, thanks for comment and follow-up.
Good article Sean! Helpful to those of us without years of experience to see the judging procedures all laid out together along with the description of the various ratings for each of the big 3.
Thanks for the comment Don. Glad to hear you enjoyed the post.
Well-researched, compelling piece, Sean. Perhaps your best yet.
Something that seems interesting (if I am reading this correctly) is that the three publications have different definitions for the grading scale. It seems that if the same person reviewed an average wine that was drinkable but nothing special, would score a 76 for WA, 80 for WS and 85 for for WE. (Just making up the numbers to illustrate the point)
Anon, I agree that I found the differences in definitions along the 100 point scale quite interesting. Even something as subtle as the difference between Wine Parker's 96-100 "extraordinary" and Wine Spectator's 95-100 "Classic". Thanks for the comment.