Nov. 19, 2008
BETA
Here are some of the most frequent categories of comments about the NewsTrust review tool, in response to the question: "How could this tool be improved?"
Rank |
Comment Category | # of comments | % of comments |
| 1 | Story is not credible | 19 | 8% |
| 2 | Good tool | 14 | 6% |
| 3 | Compare with other stories | 13 | 6% |
| 4 | More questions | 12 | 5% |
| 5 | Fix viewpoint questions | 11 | 5% |
| 6 | More comment boxes | 11 | 5% |
| 7 | Name author and source | 10 | 4% |
| 8 | Show known facts | 10 | 4% |
| 9 | Questions are limiting | 10 | 4% |
| 10 | Check specific quotes | 9 | 4% |
| 11 | Not sure what it's for | 9 | 4% |
| 12 | Not for me | 8 | 3% |
| 13 | Not a tool | 8 | 3% |
| 14 | Fewer questions | 7 | 3% |
| 15 | Provide "n/a" options | 6 | 3% |
| 16 | Define quality | 6 | 3% |
| 17 | Guidelines and examples | 5 | 2% |
| 18 | Use buttons, not drop-down | 5 | 2% |
| 19 | Helps me think | 5 | 2% |
| 20 | Too generic | 5 | 2% |
| 21 | Great idea | 4 | 2% |
| 22 | Let me rate a topic I know | 4 | 2% |
| 23 | Different questions for opinions | 4 | 2% |
| 24 | Feedback from others | 3 | 1% |
| 25 | Larger story frame | 3 | 1% |
| 26 | Shorter story | 3 | 1% |
| 27 | Ask what's missing | 3 | 1% |
| 28 | Ask for objective answers | 3 | 1% |
| 29 | Useful educational tool | 3 | 1% |
| 30 | More interesting topics | 3 | 1% |
| 31 | Single page | 3 | 1% |
| 32 | Rate writing quality | 3 | 1% |
| 33 | Three-tiered tool | 2 | 1% |
| 34 | Highlights | 2 | 1% |
| 35 | Email editors | 2 | 1% |
| 36 | Ask to verify evidence | 2 | 1% |
| 37 | Rate brevity | 2 | 1% |
| 38 | Let us 'edit' the story | 1 | 0% |
| Total Frequent Comments | 233 | 100% |
After rating their assigned story, survey respondents were asked this question:
"How could this tool be improved?"
About 544 survey respondents made tool comments, out of 1,011 respondents total. Of those, 233 made substantial comments echoed by one or more members, in the categories listed above.
Here are samples of these comments about the NewsTrust tool, presented below in 4 categories:
Excellent tool, just as it is.
I'm impressed. I think it's fine as it is. If I use it more I'll be able to come up with ways to improve it.
I think the tool has been well thought out and accomplishes the goal of the review. It allows for a critique of the reliability to both the print media and the writer, but does not inject personal values.
It's quite good, actually. The story itself seemed a bit of a self-parody, though. Its lack of subtlety made the review tool perhaps deceptively easy to use, than might have been the case with a more ambivalent story.
It was handy to be able to reference the story on the half-screen while filling out the questionnaire.
I appreciate it as an educational tool. It makes me think about the information I'm receiving in a clearer, more critical way. You could tweak it to make it more understandable to your average Joe, because right now your language is too abstract.
The tool made me think about the story, rather than to just accept it as fact.
It made me concentrate on what I was reading rather than just perusing it
It caused me to realize that, too often, I scan an article without giving it the kind of analysis that I could and should.
I liked the fact that by having to concentrate on that article, I learned more than I normally would just scanning through an article. I thought the length was also appropriate, so it wasn't TOO much for my brain to absorb.
I see that it makes me think a little more by requiring me to question and focus more on what I am reading. I am not sure, though, how this tool would be used. Would people really be willing to take this extra step and how does it really help you figure out if there is bias? Is this something that would create a rating (such as feedback on book reviews)? I find that I only look at those to confirm my own thoughts - not to question them.
Thanks. I appreciate thinking a bit about what I'm consuming. It's a good reminder that we should always use our little 'review tool' when consuming any news source.
Provide a less outlandish news piece.
It didn't do much for me at all. The article was so silly.
Use articles that are more challenging. It is too easy to see the bias in this article.
I did not like the conservative vs. liberal continuum. The story just does not fit that continuum. It's not a liberal/conservative issue. Although it seems likely that conservatives might be more sympathetic with a first strike by Israel.
Question on political orientation of article was not good. The story clearly favored an Israeli Likud perspective, but the only choices were on a spectrum from 'liberal' to 'conservative', which is not especially relevant to the story. There should at least have been an 'other' option with space for a comment.
The only thing that I found a little frustrating was the question about whether the article represents a political viewpoint. I believe that it does. But when asked what viewpoint, I was presented with a list of choices, none of which described my opinion.
Multiple-choice responses usually leave something to be desired...
The multiple-choice questions speed up the process, but the available choices don't always cover enough ground. Sometimes 'none of the above' feels like the most appropriate response. Something more refined would be desirable.
The number of questions is fine, but I found them frustrating to try to answer. I kept wanting to add clarifications and qualifications. In nearly every case, the most truthful answer would be 'yes and no,' and I would then want to explain the 'yes' and the 'no,' rather than average them to some muddy response in the middle that does little to convey either the article's strengths or its weaknesses.
Not sure what the purpose of the review tool is.
Depends on the purpose of the tool. Is the tool to guide the taker to critically evaluate an article? Is the tool to guide the investigators on what the taker is thinking? Part of the problem is the same as with the article- what is the agenda of those responsible for its origination?
I'm still a bit fuzzy on the purpose of the 'tool' itself. One would think that thoughtful readers would be able to analyze and evaluate articles such as this on their own, but perhaps a 'tool' would be helpful to some of them. Blog and/or newspaper reviewers would surely, it seems to me, be able to do such analysis without such a tool. But I could be wrong, of course. I was once.
I personally have no need for such a review tool.
I'm not sure how much the protocol actually helped me review the article; I suspect my own reading methods and proclivities did that.
I don't feel the need to use a 'tool' like this at all. My assessment of the completeness, accuracy, or bias of the article did not change noticeably as a result of going through the exercise -- certainly not enough to warrant spending the additional time doing it.
I think it would be interesting to do a side-by-side comparison of articles from different points of view.
I have nothing to measure it against. It seems to be just right. Perhaps compare the differences between two stories on the same subject.
You need to present 2 stories on the same issue from different perspectives and ask us to evaluate the merits of each, where we may be able to point to the good and bad rating in each of the stories.
More places to comment when a multiple-choice answer just doesn't fit with what you wanted to answer.
More room for comments initially and room for comments during review process. I felt the multiple-choice format was too general to get my opinion.
It would be helpful to have more areas for comment, particularly when the answers from which one must choose don't quite describe the way one feels.
An optional comment space could come earlier.
I'd like to see the author and source publication at the time of the review.
I need to know the author, publisher, and other information about where it came from. I have no way to check it out.
Give the news source and the journal in which it was published, including the name of the journalist and his/her affiliation with a news syndicate, if applicable. Also, include space for additional commentary on your questions, (much like this space).
I would like a scorecard to rate the author. I was under the impression that the intent for this program was to create 'Nielsen rating system' for pundits. If enough people gave scores to pundits on accuracy than it can be used as tool to measure their 'BS' level.
Provide more information, sources, facts as well as a counter position.
Provide as many known facts about the story as possible so one could compare with the story.
Links to resources and articles. Background on author (previous postings and political leanings), etc.
Allow comments to be tied to specific parts of the story.
Ask for evaluation on quoted elements in the story. Your questions are too general.
Possibly rating or selecting parts of the article that are good/bad/convincing, etc.
The users choose general topic areas, so they can evaluate articles of which they have some knowledge. Ask more about factual accuracy and appropriate contextualization of article.
Categorize the stories as foreign or domestic subject, science, politics, environmental, economic, etc. Then have the people who are interested in that topic by downloading the story be the evaluators.
Ask what should have been addressed (unanswered questions)
You might include a question about what specific information might improve the article.
Ask, 'what kind of information is missing from this article, if any?' and 'whose position is being under-represented, if any?'
Questions should be created that call for more objective answers, which I think may be possible. This tool is more subjective than it needs to be.
It could do more with some actual counting of named sources, questions that get at diversity of point of view, and some method of rating how contextualized or de-contextualized the story is.
I rather liked the earlier tool that asked the user, for example, to compare the number, content, and quality of 'pro' vs. 'con' quotes and citations.
Could it include some keys to look for actual documented facts including the usual who, what when where and why?
Providing information about ways to test the validity of stories rather than asking me to evaluate a story with questions like yours would be more useful to me than this review tool.
You could give the reviewer the ability to edit. For instance, my version of this 'story' would be: There's a rumor going around that the Bush administration wants Lieberman on their team. We haven't confirmed it yet. Anyone interested in going after the real story?
The entire article review form should be on a single page and needs refinement. I think the smoothness and flow need work. It's a very good start though.
Make the selections radio-buttons instead of drop -down lists, making it faster to select a response; and arrange the radio buttons all in one line, and in the same horizontal place for each question, making it faster to select a response.
One page. Radio buttons or checkboxes. Tab/keyboard accessible. FAST... I want to see, point, click and be done. Click and scroll... pain in the ass. Also, want to be able to click SOME attributes and skip others. Like, suppose I think this is an important article with NEW information... but I don't want to rate the partisanship or credibility of the article.
Fewer pages, perhaps radio buttons instead of drop down menus (see Yahoo's star rating for an easy to use design). Should be quicker to use, I have a hard time finding time to read what I want to read (despite knowing where to find it). Adding a minute to each article will not make me use the tool.
Give a choice for 3 levels of questions, brief, moderate, detailed, based on our time, interest, and passion for the story.
Second or third-tier tools would be quite useful. This tool takes a few minutes for a beginner to use. I propose an approach where a tool like this is the middle tier. There would be a 'quick review' tool that could be used casually and more often by a large number of people. The existing review tool would be in the middle tier, used to get a little more depth from a smaller set of readers, particularly if the first-tier reviews indicate controversy, challenge, or some other 'red flag' interest. A third tool might be available for those willing to take tens of minutes or longer -- with this, individual claims might be refuted, or vague citations might be pinned down.
The real key for me is whether the tool was super easy to access -- the jump from reading a page to commenting on it should be just a bookmark away (see bugmenot.com for how they do bookmarklet code). I'd be much more likely to rate stories then. It'd also be nice to see how others rate stories, after I've rated it myself of course.
After submitting 'your view' to be able to see the views of others....
After I complete it I would like to read feedback from other reviewers.
Some way of assuring that the editors and reporters will actually get the feedback. ...
Are the journalists who wrote the piece going to see the review? What about their editors or publishers where applicable? Can we see other reviews of the same article? Different articles?