Proceedings talk:MW2

This is the discussion page for Martin Walker's BOF at Wikimania 2006, Validation on Wikipedia: How do I Know this Article is Accurate?. Please join the discussion below!

For discussion arising from this session, please see w:Wikipedia talk:Pushing to validation and the related "article".

Absolutely positively brilliant. 68.81.169.102 02:10, 26 July 2006 (UTC)[reply]

The trouble then becomes who is the expert

The trouble is that expertise can be extremely narrow. For example, I'm pretty good at reviewing an article on type II supernova collapse, but I know nothing about the interstellar medium, and an undergraduate that has done about a week of research can develop more expertise than I do in type I supernova. I can write a lot of useful stuff about the Evidential School of Confucianism in the Qing dynasty in Zhejiang, but I know nothing about the Cheng-Chu school or the Han dynasty. Which raises another question, how do you know that I know about the Evidential school? Which raises another question, what do you do when the experts are wrong? One problem I see in science is that having the "cult of the expert" gives them attributes to them much more power and authority than they actually should be given. Experts have known to be wrong. The scientific and scholarly consensus has also known to be very, very wrong.

The other problem is that in a lot of cases you don't need an expert to review. Experts are extremely expensive, and in a lot of cases there is just overkill to bring in the best experts. For example, in editing a basic article on astrophysics, I think it would work a lot better to have non-experts write the article, and then bring in "experts" only when there is major unresolvable issue.

(I think that there is a science versus engineering culture clash here. In engineering, the principles used by the community are considered inviolate, whereas in science the name of the game is to encourage people to question the principles of the community.)

Something that to keep in mind is that anything that makes participation more difficult for "non-experts" keeps out "experts" and anything that creates a division between "non-experts" and "experts" makes it more difficult for a non-expert to be an expert.

Roadrunner 16:05, 29 July 2006 (UTC)[reply]

Sorry I missed seeing this till now! You raise a good point, the classic quis custodiet ipsos custodes problem! In a volunteer community, it is particularly hard! I think we should delegate the job to the WikiProjects, and have people choose to review articles they feel comfortable reviewing. Within the projects, active editors are known and respected for certain areas of knowledge, and these people would carry the weight of the project with them. A disruptive know-it-all will be known to the group, and their opinions in any review would no doubt be given less weight. We can talk more on this tomorrow! Walkerma 03:43, 4 August 2006 (UTC)[reply]

How do I know the article is accurate?

The answer is that you don't, but that goes for anything that you read.

Roadrunner 16:09, 29 July 2006 (UTC)[reply]

Stable versions

Prompted by your note on en:Wikipedia talk:Stable versions, I'd like to point out that there is a wider applicability of stable versions than just for validated versions.

Validation is good, and needed, but of necessity takes significant time and comes late in the evolution of an article. Clearly a mechanism like stable versions is needed to assist in handling of validation. But, in the view of at least some, stable versions (or similar) is also needed for a wider use: improved quality of articles without expectation of complete (or even significant) fact-checking. After the toddlerhood of an article, it can start being lightly reviewed so that Wikipedia readers can encounter better articles generally, that is, articles which are free from obvious graffiti and absurdities.

Stable versions can provide this by having a "lightly-stablized" version presented to the average reader. This mode of presentation is quite different than simply having a "Validated version" tab at the top of today's page. Although the latter is approriate for a formally controlled (and rarely updated) version, the lightly-stablized versions need to be presented by default in order to generally raise the level of the articles presented by Wikipedia.

So, I hope in your discussion you make it clear that the stable version mechanism, although needed for formal validation, has much wider uses. -R. S. Shaw 22:13, 3 August 2006 (UTC)[reply]

Yes, this is very true. Validation needs stable versions to work, IMHO, that's why it's raised here, but stable versions have much wider application. It may be that an initiative on validation might help the development of stable versions in general. If there are people there tomorrow who can say, "Yes, we can add a new tab" then we may need to discuss how that would be implemented. Would it be on one tab labelled "Stable", with a "validated version" link showing when available, or would it be done differently?

One point - I had the impression that the consensus at WP:STABLE is for the stable version to be the default - as you suggest above. My suspicion, though, is that the wider Wikipedia community would not take kindly to that - I think many would feel strongly that the editable version should always be the default version. For validation, I'm OK with that; for use of stable versions as you mention, I can see why you might want it as default. I think we'll have to see what the consensus is - if and when it is implemented. Right now we need the infrastructure! Walkerma 03:53, 4 August 2006 (UTC)[reply]

Validation of Credentials

Validation of articles necessarily leads to questions of who will be the experts. How do you validate their credentials? This is a more important, yet simpler, problem that could more easily be addressed by a central organization. Users could submit proof of their Harvard PhD or other credentials to Wikipedia head office, and would then be allowed to have the official Harvard PhD userbox (or whatever) on their user page. That allows everyone in the community to make their own judgement about the value of each user's opinion, instead of appointing elite panels of experts. Dr. Martin Walker has suggested that we can simply use reputation within the Wikipedia community as credentials, but that denies the much larger body of works, accreditation, etc. that already exists outside Wikipedia. Relying solely on reputation within the Wikipedia community sets up an elitism of experienced wikipedia users and an unfair barrier to entry onto review panels.-Ytrottier

After the discussion today, I came away with the opinion that probably some blend of the two would be best. Your comments this afternoon helped to persuade me, thanks for raising them! We should perhaps have a group of respected Wikipedians, some of whom may have appropriate credentials anyway (by all means verified by Wikipedia), but also invite one or two outside experts. That way we allow a voice for the community, but also have input from people that our readers would regard as expert. Do you think that could work? Along these lines, you may be interested in this proposal from Sj, our wonderful Wikimania organiser! Walkerma 04:18, 5 August 2006 (UTC)[reply]

Thanks, I like Sj's proposal and I will comment on its discussion page. I don't think it matters so much whether the reviewers are internal or external to Wikipedia, as long as readers can somehow trace their credentials back to the reader's trust network. Your trust network includes several members of the Wikipedia organic chemistry community, so that is an easy trust trace for you. But being from a completely different field, I'd like to be able to trace an article to an author to a school to an accreditation authority which is supported by my school which is supported by my teachers. There is plenty of room to argue that not all schools, and not even all accreditation authorities are equivalent, and that's why Sj's idea of "choosing a set of blessings" appeals to me. -Ytrottier

I discussed this with a lot of people yesterday, and the most workable solution would be a two step process:

Members of the relevant WikiProject perform a fact-check on every fact in the article, check every reference. You don't need to have a PhD to check that the fact matches the literature - you just need to be trustworthy and knowledgable in the field - but any credentials reviewers can share would be noted (I think we have about 10 active PhDs in chemistry). These fact-checks would be available to users (just a click or two away from the main page). Thus any reader who doubts a particular fact could find out that User:XXXX had verified this fact in this book on this date.
An outside person with credentials and reputation (a "big name") in the field would check over the article looking for balance, completeness, etc. They could trust that the individual facts were correct (and they could see exectly where any given fact was found and verified). Their valuable time would not be wasted on trivial fact checking, they could utilise their skill very efficiently. They would sign off, and afterwards any reader could then see that BIG NAME at FAMOUS UNIVERSITY has declared this article to be accurate.

We have to make sure that the big name doesn't just rewrite the entire article to fit their own views - perhaps we could encourage them to do any edits before the review began.

I think this is the most workable solution - it would show the world exactly where every fact comes from, and the expert would confirm its relevance - and the user and even the skeptical librarian would be able to TRUST the article. We'd have much more credibility. Walkerma 13:37, 6 August 2006 (UTC)[reply]

Stability, readability, accuracy: a proposal of least apparent change

I strongly support the idea of a move towards article stabilization and authentication, but I'm sympathetic to the idea we need to move conservatively. Let me start with some observations:

There are three basic metrics of article quality:

content (completeness/appropriateness/etc.)
accuracy (fact-checking, balance, and citation)
readability (clarity, style, grammer, and mechanics)

Few editors care equally about all three.

When refining an article in one of these regards, a frustrating eventuality is that another well-meaning editor will come along and negate your work to further one of these ends. This is obviously the case for readability and accuracy: careful fact-checking and careful grammer correction work often gets mauled when someone decides to change the content. To a lesser extent, content can get unfairly negated too: if I add x but don't cite it, someone else is just as likely to come along and just delete it rather than improve the content by finding a citation (which can be understandable, but still frustrating); whole sub-topics can go undeveloped this way because the only person who thought to bring it up got shut down. I would say, though, that the work of readability and accuracy are far more delicate than content development, and they are therefore more fragile to the wiki process. You really can't do those kinds of work well when the content sands shift underneath you.

Ideally, these three aims could be treated in seperate processes to minimize the frustration. In fact, the ideal development goes something like:

first, stabilize content
then stabilize accuracy
then stabilize readability
then do a quick pass on accuracy again
rinse, repeat

The seeming problem with adopting this model is that it would introduce down time for the various factions. For instance, those working on the content will have to wait for the accuracy and readability folks to put their stamp on an article before continuing work. But is this really necessary?

I think it's safe to assume that the majority of Wikipedeans much prefer to work on content, for most people find the minutia of writing and verification to be onerous burdens. If there is to be a stability feature in Wikipedia, it must respect that primary motivation.

Here's what I propose:

I'm going to sidestep the whole issuse of authority selection and just say that the readability- and accuracy- stamps-of-approval will be in the hands of priviledged committees, perhaps divided into subject areas or perhaps not.

Whatever the selection process, the authority will be excercised with a simple extra checkbox on an edit page-- 'this is a stable version'-- which commits the edited page and gives it the special mark of a stable version. So the stable pages are just versions of a page within the normal main namespace.
If the current version is not a stable version, readers are presented with a link near the article name to 'view the last stable version' (assuming there is one for that article). Viewing the last stable version is just like reading a page in the history, and if you look in the history, stable versions are marked as such.
It's up to these priviliged editors to indicate at the top of the page the kind and degree of stableness that is being asserted.
It will be up to the priviliged committees to coordinate their efforts (esp. so that readability people and accuracy people don't negate each others work).
It may be desirable with some pages to present the 'last stable version' to visitors by default rather than the current version. This could replace much of the need for page locking.
These commitees will often have the need to work on branches of the page in a seperate, protected namespace. This should be done by copying the last decent version of the regular article and working from there. A problem I forsee is that, if committees take too long to do their work on a page, they will frustrate regular editors who contribute edits between the time the committee starts their work and when they then finish because their edits will be discarded when the stable version is checked in. There are three ways to mitigate this:
1. Strict time limits should be imposed on the committee's work.
2. When starting the process, committees should post a notice on the regular page.
3. After commiting the stable version, committees should be responsible for going over the edits made while they did their work and incorporating the reasonable edits that can be reasonably reconciled with the stable version; the reconciled version will be commited as a nonstable version (so this would not be official committee endorsement of the material, just a courtesy); the flaws in these edits will be worked out the next time the committee works on the page.

I think a key virtue of this system is that it leaves open the question of what stability means, for it is something that is indicated on each particular stable version. The kind of assurance being given can be specified as appropriate.

A problem that arises, though, that users looking for one kind of assurance may not necessarily want the last stable version but rather an earlier stable version. It could be a history option to see only stable versions, but if the assurance semantics are in the page rather than in some tagging system in the software, the reader would have to look at the actual pages to find the one with the best kind of assurance he's looking for. This won't be such a problem, however, if the committees are sure to incorporate the virtues of the last stable version; as long as the stable versions get progressively better on all counts, readers will always want the last one.

Another solution is that, if a committee creates a new stable version that is in some way inferior to a previous stable version, they could note this at the top with a link to that other stable version.

Taking the committee system one step further, there could be a mechanism for committees to authenticate their pages. One idea is to have certain special category-like links for each committee with the restriction that only a member of a committee can add its special link to a page. This would make it easy to browse the category of stable pages given a seal of approval by some particular group. This mechanism would protect the 'this is a stable version' checkbox privilige be not allowing anyone to claim to speak for any particular committee they don't really represent.

Isn't this a form of elitism?

Yes. I don't see how you could have a stability/quality/accuracy system in Wikipedia without some kind of privilege. However, the system I propose, I think, creates only a shallow kind of elite, one based on reputation that is not so very different from what already exists with well-known members. Wikipedia, like other free/open projects, does, in fact, have a very definite hierarchical structure, and this is fine as long as they remain open hierarchies. --Apantomimehorse 14:26, 8 August 2006 (UTC)[reply]