Inside the Meter
The difference between Stalled and Promise Broken
We've been publishing lots of updates to our Obameter database of campaign promises and have been getting some questions about our rating system -- particularly how we distinguish between a Stalled and a Promise Broken. So we thought it would be helpful to explain our ratings.
First, we should emphasize that because the Obameter is our creation, we've had to develop the rating system ourselves. They don't teach this stuff in journalism school -- at least not yet! We've tried to make the system thorough and even-handed. We want to show the relative progress of a campaign promise and account for many outcomes.
It's also important to note that Obameter rulings are not final. We have moved many ratings from In the Works to Stalled, or from Promise Kept to Promise Broken. (By contrast, our Truth-O-Meter rulings are final and don't change unless we find we've made an error.)
We gave the Obameter six ratings. The first three (Not Yet Rated, In the Works and Stalled) provide a broad picture of whether Obama is making progress; the final three indicate whether he kept the promise (Promise Kept, Compromise, Promise Broken).
Many promises include a time frame, which gives us an end point to judge whether the promise is kept. But many others don't have a time frame, so we have to use our best judgment about when Obama has had a reasonable time to fulfill the promise.
In general, we rate items as In the Works or Stalled when we feel that the promise is in progress and there is still the possibility of action. Once we decide that action has been concluded (or if there is no sign of action), we rate whether it's Kept, Broken or Compromised.
This can be a difficult call. One person's Stalled is someone else's Broken. We use our best judgment. But we recognize that reasonable people can disagree. That's why we are fully transparent about our sources and we publish our e-mail address -- email@example.com -- so that you can let us know when you think we've made a bad call.
Here are our definitions for each rating:
Not Yet Rated – Every promise begins at this level until we've rated it.
In the Works – This is our broad category to indicate the promise has been proposed or is being considered. We move the status to In the Works when Obama makes a formal proposal, as he did with the economic stimulus plan, and it could remain at that status until it is ultimately approved or rejected by Congress. For some promises, it's possible that the status could initially go to In the Works, but then move back to Stalled if we decide the proposal has hit a lull, and then go back to the In the Works.
Stalled – When there's no action for a promise. We expect a fair number of Obama's promises will stall at some point because of limitations on money or opposition from congressional Republicans. Some may stay stalled and ultimately be rated Promise Broken, but others will probably move back to In the Works.
Promise Broken – There are several ways a promise could earn this rating. Congress might reject the proposal outright through votes in the House or Senate. It's also possible that the proposal could be determined to be dead before an actual vote. It might be referred to a committee but never get a vote, or congressional leaders might announce that the proposal won't be considered.
Compromise – Promises will earn this rating when something substantially less than Obama's original statement occurs but when there is still a significant accomplishment that is consistent with the goal of his original promise.
Promise Kept – They'll earn this rating when the original promise is mostly or completely fulfilled.
An important point about Obameter ratings: A Promise Broken rating does not necessarily mean that Obama failed to be an advocate for his promises. He could put tremendous effort for any given promise but it could still die because of opposition in Congress. That might be a perfect example of the legislative checks and balances on the executive branch, or the impact of public opinion. A promise that was popular during the campaign could be less popular now because of changes in the economy. But for consistency, we are still rating every promise he made during the campaign. Individually and collectively, our ratings show the progress of Obama's presidency during changing times.