Certifier Nullification and the Advogato Trust Metric

Posted 12 Jan 2003 at 07:44 UTC by jbucata Share This

If there is, in fact, a problem with people being certified who shouldn't be, we need to go back to the trust metric definition to find out how it happened, before going about trying to fix it.

This article is my response to the recent article and subsequent discussion about the trust metric--you'll note the influence here of several of the good replies posted there.

Journeyer status has become far too common. People's certifications tend toward Journeyer status, seemingly independent of whether they deserve it.

When I first signed (back) up, and Denny certed me as an Apprentice, I figured that that was the most appropriate slot for me to be in. By my own personal metric (largely influenced by Software Craftsmanship by Pete McBreen), I'd be a strong journeyer, in terms of my technical abilities to get software work accomplished, but just about all of that skill is applied for closed-source work for my day job, so according to Advogato's metric, my occasional helping out with Debian isn't enough for a higher cert. So I'm not an apprentice, but I am an Apprentice. Fair enough.

Then I started receiving a few Journeyer certs, and my bright green changed to cyan. I'm grateful that a few people think I'm Journeyer material, and I agree that on a broader level that's where I belong, but from a Free Software perspective I don't merit that title at all. Maybe that means that the definition needs to change, or else extra ranking options need to be added in parallel to what we have right now--to account for general technical skill along with prominence in the Free Software community. But more on all that later...

Having people who are certified higher than they should be is actually just the general case of having people being certified who should never have been--in that case they shouldn't be certified above Observer!--which was the focus of the trust metric at its inception. This means we can examine the trust metric and see what wisdom we can glean from looking at it in the light of the more general case.

In terms of the trust metric security proof, our problem is that there are a lot of "confused" nodes who are certifying "bad" nodes. Anybody who is giving out Journeyer or Master certs to one or more users who shouldn't be receiving any is "confused"--which is a generalization of the original specific case that the trust metric definition paper discusses, of should-be Observers being certified at Apprentice or above by "confused" users.

In the trust metric definition paper, Raph proves that the number of bad nodes certified is bounded by the number of confused nodes (Theorem 1). The point is that it's not ultimately a function of the number of bad nodes--so Bad Guys(TM) can create as many accounts as they want, but unless they can trick other, certified users into certifying them it won't make a difference.

However, it points to another problem one level removed from that one: The number of confused nodes isn't itself bounded! So if a large number of people are confused and start certifying people they shouldn't (and/or, same thing, certifying them higher than they should be), then we're going to have a (potentially) large number of certified bad nodes. So the plain answer is, we need to reduce the number of confused nodes in the graph, in order to reduce the number of certified bad nodes.

But wait a minute... let's go back to basics and review the key definitions here. Raph's distinction between "good", "confused", and "bad" wasn't a part of the formal mathematical definition of the trust metric--it was introduced for the security proof--so we need to scrutinize carefully what he, and we, mean by those terms. Saith he:

The bad nodes are under the attackers [sic] control. The confused nodes themselves represent valid accounts, but may contain certificates to the bad nodes. The good nodes are both valid accounts and have certificates only for other good nodes and confused nodes.

Raph's accidental omission of the apostrophe here is interesting: We presume it should be "attacker's", meaning one malicious individual is creating a bunch of accounts to try to wreak havoc. However, it could easily be "attackers'", if there is a large group of people, creating one account per person, performing the attack. It makes no difference to the resilience of the underlying mathematical model.

The key insight comes from that observation. What is an attacker, anyway? "Well, an attacker is, um, one who attacks." Spiffy. What do you mean by "attack"? The word isn't explicitly defined in the paper. We can always fall back to its dictionary definition, but... what does that have to do with the model? Nothing. The paper didn't define it because it didn't need to. The security proof holds independent of what you're securing against.

Instead of saying that the bad nodes are "attackers", let's generalize it to say that the bad nodes have Property X. Confused nodes don't have Property X (or else they'd be bad), but are certified themselves and have certified (or overcertified) bad nodes; good (certified) nodes never do so. Let's say that the certified users that are confused about Property X will themselves have Property C(X). By the generalized trust metric security proof, reducing the number of users who have Property C(X) reduces the number of (over)certified users who have Property X.

Let's say that Property X ceases to be "is actively attempting to ruin the Advogato experience maliciously or for personal gain" and is replaced with "does any sort of closed-source software development". That makes me a bad node, and all those people that certified me are confused. Use instead "works on *BSD", and a raft of other people become bad, and I become good once again (unless I happen to have certified a *BSD developer, in which case I'm now confused). Or we could use properties about where people live, their hat size, or, more ominously, their skin color or such like. If Property X is defined to be "is a kernel developer", then alan is a bad node, and the root node is confused for certifying him as a Master!

What does all this mean? The trust metric is value-neutral per se. The values that Advogato chooses to hold to come from the users--the certifiers--not the trust metric. The officially endorsed values of Advogato aren't encoded in the trust metric, and so can only be enforced by the individual users. (You could say that there are some value choices implied by the selection of certain users as trusted seed users, but that selection is an external process, and not inherent in the trust metric itself. The trust metric mandates the existence of seed users, but it doesn't dictate who they are.)

This has two implications:

  1. All complaints along the lines of, "The trust metric is broken because we're seeing $CLASS_OF_PEOPLE being certified", in fact say nothing whatsoever about the trust metric itself. The trust metric ain't broke; if anything, the trusters are.

  2. The things that Advogato values and rewards is a function of the users of the system, up to the limits of what their individual certification levels allow and disallow. It has nothing to do with the published guidelines of who should be certified how, unless the certifiers decide that it should.

That second point is worth a bit more examination. In the United States and elsewhere there's a legal principle called the right of jury nullification, which allows jurors in a trial to acquit a defendant of violating a law that they believe to be morally wrong. In effect it allows them to judge the law, and render it ineffective (nullify it) by refusing to convict even if the defendant did what the law said not to do.

This is effectively what's going on on Advogato. Even though the law has been laid down concerning whom we should certify and how, it's ultimately up to "We, the People" to make judgments in individual cases whether somebody ought to be certified or not. If we decide to ignore the law, then people will be certified as Journeyers and Masters when by the rules they belong at most as Apprentices--and there's nothing that the rules (ultimately just an HTML file on a server somewhere) can do about it.

So what do we do? I see four options. We can handle the problem of too many confused users:

  1. ...by proclaiming it "not a problem": Continue with the existing system as is. Stop complaining and get on with our projects and our lives.

  2. ...by reducing confusion: Initiate a massive campaign to educate the userbase as to what the certification guidelines mean, how to apply them in particular situations, encouraging and thanking them when correct decisions are made, etc., etc.

  3. ...by removing their certification: Modify the certification guidelines to make explicit what's already implicit in the trust metric: When you say that somebody is a Journeyer, you're not just making a judgment about their contributions to Free Software, you're pronouncing them fit and capable of competently judging others the same way. In other words, Property X (badness) becomes "not (works on Free Software AND doesn't have Property C(X) (confusion))". In other words, what's "confused" today becomes "bad" tomorrow--which makes sense, if we esteem accurate handling of certifications, and not just involvement in a Free Software project. If somebody you certified starts certifying people in a way you don't think is right, yank their cert.

    The present glut of Journeyers means we've probably got at least a few users with bad judgment, including some somewhere close to the root node, who need to have their own Journeyer certs downgraded or removed altogether.

  4. ...by changing the trust metric to make them a non-issue: Along the lines of what was mentioned in passing above (and what has been discussed several times over Advogato's history), modify the trust metric to split between the functional certification of "you work on an important Free Software project" and the trust certification of allowing their certs to have an impact on others.

I advocate option 4. It doesn't do away with the problem of certifier nullification (if it's in fact a problem--if essentially nobody agrees with Raph's guidelines, then maybe they should be trashed!), but it makes it easier to comply with the letter of the law without other consequences rippling throughout the trust network: "Yes, I believe you're a Journeyer according to the Free Software measure, but no, I don't believe you have enough good sense to make accurate judgments of other people!"

Option 3 has some problematic Gödelian implications, since if confused nodes are now relabelled "bad", then the formerly good nodes that had certified them must now be relabelled "confused". Another iteration is required so that we can relabel those nodes as "bad", and propagate the confusion further up the chain of formerly good users towards the root of certification. Ultimately the seed users themselves must be labelled "bad" (sorry, Raph!).

Besides which, any sort of attempt to implement option 3 would still require that sort of ranking, on both criteria, on the part of the certifier, but keeping today's certification system means he/she must necessarily choose the lower of the two ranks as the ultimate rank to give. In the face of such a task, a certifier is much more likely to say, "No, forget that, they deserve more than Apprentice," and overcertify. Option 4 doesn't require any more thought on the certifier's part than Option 3; it merely allows for the expression in the system of the full results of that thought process, so no such compromises need to be made in the process of force-fitting them to one metric. Option 4 also offers the possibility of requiring less effort than option 3, if it's implemented in such a way that either of the two rankings can be defaulted to "no opinion". So any concerns that option 4 would require "too much effort" are largely unfounded.

In today's reality, there are probably several factors (such as overall technical competence, entertainment value of diary and article postings, etc.) that influence each individual decision. Breaking out several rankings from today's solitary Observer->Master scale would allow more fine-grained evaluation of each other's abilities. (The comparatively new diary rating system is a step in this direction.) How to integrate multiple scales together into a meaningful system is an open question for discussion.

(Kudos to Raph for creating such an interesting trust metric in the first place. In the paper Raph effectively said he hoped it would extrapolate to other real-world issues and not just be confined to one Web site somewhere. It looks like it does, though perhaps not necessarily in quite the way he was expecting.)

Well said., posted 12 Jan 2003 at 20:28 UTC by kbob » (Master)

That was well-written. As I was reading it, I thought of several points I wanted to reply to, but you addressed all but two of them.

The first, most important point is, for the purposes of Advogato as a social/technical discussion forum, what value does an absolutely accurate trust metric give? My experience is that the more egalitarian a group is, i.e., the less rank is emphasized, the more effectly people in the group cooperate. The trust metric serves its primary purpose, which is to keep out spambots and headhunters, perfectly. (Anybody remember headhunters? (-: )

The secondary point is that we could regard the system's shortcomings as a UI issue. Raph's description of master reads,

A Master is the principal author or hard-working co-author of an "important" free software project, i.e. one that many people depend on, or one that stands out in quality. A Master has command of the tools and is an excellent programmer. Generally, a Master works equivalent to full time (or more) on free software. Ideally, a Master writes clearly about the work and its broader context, and serves as a mentor to others in the free software community.
As you pointed out, that confuses a whole bunch of separate values: programming ability, commitment to free software, ability to pick good projects (-: , hours spent, writing ability and focus, and mentoring. There is also the implied authentication part of a cert: Does this account really belong to the person it claims to be?

But when you're actually certifying a person, you're not looking at any of this stuff. Instead, you just see, "Certify this person as...". You have to remember what a master certification means from way back when, and come up with your own method of composing all of the person's attributes into a single rank.

Instead, how about a questionnaire type cert process?

  • How certain are you that this person is who they say they are?
  • What is this person's level of commitment to free software?
  • How good an engineer is this person?
  • How much you trust this person's judgement to certify others?
(And provide a "no opinion" option for each question.) That way, the person bestowing certification knows exactly what his certification means.

You could fine tune the question list quite a bit. You could give each person a composite rank based on a weighting of the scores or you could just display the vector of scores. You could generalize it so that influential non-hacker free software advocates such as Jon "maddog" Hall or Roblimo could participate.

Good article, thanks., posted 12 Jan 2003 at 22:00 UTC by Denny » (Journeyer)

From the article:

When I first signed (back) up, and Denny certed me as an Apprentice

w00t! Fame at last ;)

I do make a habit of wandering through non-certified users from time to time and looking them with Google, to see if I can find grounds to give them basic posting abilities. It's hard to get noticed with no voice :)

The trust metric is value-neutral per se. The values that Advogato chooses to hold to come from the users--the certifiers--not the trust metric.

That is an excellent summary of the whole 'issue'.

The article in general was well thought-out and well-written, thanks.

From kbob:

You could generalize it so that influential non-hacker free software advocates such as Jon "maddog" Hall or Roblimo could participate.

I'd like to reply to that with a quote from the page you linked to: A Journeyer is generally a competent programmer, but significant contributions of documentation, artwork, or other non-code goodies counts too.

I've always taken that as applying to the other certification levels too - i.e. I certify not for contributions to the free software codebase, but for contributions to the free software community (which can be code, docs, running LUGs or websites, etc).

Like jbucata said, the values the site recognises come from the users, not the technology - what I count as valuable is broader than just how many kloc someone released this year.

Re: Well Said., posted 12 Jan 2003 at 22:08 UTC by jbucata » (Apprentice)

Thank you. I definitely agree on your first point: We keep out "undesirables" quite effectively, by keeping them at Observer status where they can't spoil our fun. Why even bother, then? I've noticed complaints from various corners (and I'm one of those corners) that once you get past Observer the levels are handed out without much regard for what they're supposed to mean. Picking through the replies of the previous article, you'll see several such comments. It's not a new issue, either: lilo posted an article asking to be taken from Master back down to Journeyer. lilo has since reenlisted as rlevin to get around the "stale certs and cert inflation". There are probably other discussions along these lines that I haven't dug up yet.

A related fact is that from Apprentice on up, the privileges on the system seem to all be the same. As far as I've been able to tell, there are no practical differences now that I'm a Journeyer versus when I was an Apprentice. People seem to be judicious in pulling someone up from the ranks of Observer, since that has an effect on the system, but above that it doesn't seem to matter, so little thought goes into granting Master status. (That obviously doesn't apply to everybody--those who are particular about the meaning of the certs given to them don't feel that way when certing others!) Being a Journeyer now instead of an Apprentice has to actually mean something for people to stop giving out higher certs willy-nilly.

So for that case, if higher levels ever ultimately start conferring more benefits, we'll want to be more particular in how we bestow them--and we might as well start doing it now.

So that's the two main reasons: We, as computer geeks, tend to want things, especially software and computer systems, to Do The Right Thing(TM), and the Right Thing(TM) here is for certs to match the posted guidelines; and one day higher levels might actually mean something.

I agree with your comment on egalitarianism. I'm glad I didn't have to wait to be a Journeyer to start posting articles. So no, I don't have any ideas for what kinds of benefits to confer upon Masters or Journeyers that Apprentices wouldn't get. Maybe that means we should condense the entire set of rankings down to Observer and Participant... but we've had these finer distinctions for years now, and we've been using them, so that seems to mean we want to have them around. So since simplifying isn't appealing, if we don't want to stay where we are today, it looks like we need to head in the other direction: Towards more ranking scales, perhaps with finer-grained levels.

Interpretation, posted 13 Jan 2003 at 06:55 UTC by Pseudonym » (Journeyer)

Just a quick note on what kbob said: I interpret Advogato ranking as level of commitment or contribution to the community and nothing more. I also rank this way. In particular, an Observer or Apprentice may well be a more competent practitioner of their art than a Master. Otherwise it's just too hard to come up with a rank for the world's greatest technical writer who hasn't contributed a lot.

That's how I interpret raph's definitions, anyway. Admittedly the names of the ranks don't really fit, but something has to give.

nice analysis, posted 13 Jan 2003 at 07:37 UTC by nixnut » (Journeyer)

However, option 4 will not solve the "problem", but only create two (basically the same) new "problems" in other places. People will still have to make a value judgement about others. And in making such a judgement most people will strongly influenced by social factors.
In others words do I know/like you is more important than do I judge your skill set in field N to be level M.
Increasing N may yield more satisfying certification results, but imho not very much more.

btw, I'm definitely not a journeyer, barely an apprentice. I was certified as master for a while :-)

Social factors, posted 13 Jan 2003 at 13:11 UTC by Omnifarious » (Journeyer)

I've noticed the social factors myself. I think the appropriate certification level for me is Journeyer, but I've noticed in 2 or 3 cases, people I've certified as Apprentice have certified me back as Apprentice. This hasn't happened with people I've certified as Journeyer or Master, they've all certified me back as Journeyer.

I'm not intending to complain about being certified as Apprentice, I'm just pointing something out. :-)

Signal to Noise Ratio, posted 13 Jan 2003 at 14:43 UTC by sarum » (Journeyer)

I don't know I have not seen what the big deal is, the S2N ratio on Advogato is good. It just seams to me that some people either want to be cleeky or don't like some indiviuals PoVs and are using this as part of some vendetta against them. To quote B5 "You forgot the first rule of the fanatic: When you become obsessed with the enemy, you *become* the enemy."

Perhaps an &quoobserver&quo or &quocommentator&quo status?, posted 14 Jan 2003 at 16:12 UTC by roblimo » (Journeyer)

"You could give each person a composite rank based on a weighting of the scores or you could just display the vector of scores. You could generalize it so that influential non-hacker free software advocates such as Jon "maddog" Hall or Roblimo could participate."

Actually, this already seems to be happening in practice. I can (but rarely do) post on Advogato, and I assume maddog can too if he's so inclined.

The reason I rarely post here (aside from having enough sites to post on already) is that I don't usually feel I have much to contribute. I am not a developer, so why should I chime in with my two cents on Advogato? I feel it would be rude more often than not, almost like horning in on a private conversation, albeit one taking place in public. :)

Perhaps there should be an "observer" or "commentator" status to cover people like me. Or perhaps not. How many non-developers are there who are known well enough to most Advogato readers that they/we are going to be listened to anyway? 5? 10? 20? Enough to matter? Probably not.

I personally believe the Advogato trust metric works reasonably well as it is, and that any attempts to change it might end up doing as much harm as good.

- Robin

Trust and context, posted 14 Jan 2003 at 17:54 UTC by MikeCamel » (Journeyer)

Hmm - it's been a while since I've been posting much, but the recent couple of threads on trust models overlap very closely with a field of research that I'm dabbling further and further in: trust in P2P networks. You can ask whether Advogato is a P2P network - in some ways it is, and in some ways it isn't - but lots of the issues that are coming up are ones I've been thinking lots about. You can find some of my thoughts on my website - www.p2ptrust.org, in particular in the "trust" section. You'll notice from the bibliography that I come at the issue from a strongly sociological viewpoint.

I think that what's coming out more and more clearly here is that trust only makes sense in a context. Part of the problem is that we use the word trust rather loosely in English (and I don't know of any other good alternatives). We say "I trust X", and usually the context of a situation is clear - you might have asked "who do you trust to service your scuba gear?", or "who do you trust to write a favourable reference about me?" However, we really trust someone to do something. Our trust is contextualised. In my opinion, this isn't always clearo on Advogato (or in most of our machine-mediated communications).

On Advogato, I see several different contexts, and I'm as guilty as anyone of miscertification, I suspect. Among the contexts that get confused are:

  • I trust A to be the person I identify as an Free Software/Open Source (FS/OS) guru - note that I'm not trusting them to do anything much, just to be a hero
  • I trust B to work to further the aims of the FS/OS movement
  • I trust C to have interesting diary entries
  • I trust D not to spam this site
  • I trust E to commit interesting and well thought-out views to the Advogato site.
I don't think we can truly expect people certify people to only one of those (or another) unless the interactions they have with those they are certifying are exclusively (or overwhelmingly) in that context - i.e. that the interactions in the Advogato context are reduced to a simpler context.

This seems hard - it may be impossible, and it's almost certainly undesirable. So, what should we do (if anything needs doing)? I'd advocate increasing the number of levels, trying to explain context more, and possibly increasing the contexts applied. And, possibly, removal of the showing of people's level in contexts where it has little bearing on what's going on (if there are any such contexts).

I hope this makes sense - it's a little garbled, and I'm still trying to get my head round it, too, but if the PhD proposal gets accepted, I'll be doing a whole lot more thinking!

Random Replies, posted 14 Jan 2003 at 18:06 UTC by jbucata » (Apprentice)

I'm not complaining that the social factors exist; I'm saying that they would be included as attributes to rate on. I don't want to replace 1 technical rating with n technical ratings; those social factors are important, too (else we wouldn't have the diary rating system in parallel with the certifications).

Part of that is "I trust your certifications": I'm more inclined to believe that you're a Master if raph claims you are, rather than any other arbitrary member of Advogato. There are other people, too (particularly those who justify their ratings with a note in their diaries: "for his excellent work on the Acme Activator Plus project"), at varying levels of me trusting their trust. So you might just be an apprentice but I still feel your judgment of other people is keen, so I'd like some of my flow of points to go to the people you deem worthy instead of having to judge all of them myself--and especially when today you can only meaningfully cert others as Apprentice. (Is it always the case that it only means something when another Master calls you a Master? Especially when your Apprentice-ship is based on your participation, not on your abilities?)

It definitely feels rude to find that somebody has certified you highly, and when you do an honest assessment of where they are, you certify them lower than that. So it doesn't surprise me that a lot of people don't do that. Having multiple ratings tends to make such evaluations fairer--witness the fact that most corporate performance reviews, or at least those that try to be fair at all, force the reviewer to rate each of several skills on a numeric scale, precisely to avoid such tendencies to automatically inflate (or deflate!) people's final score.

I'm also thinking that we should include in the list a WACGYR ("What A Cool Guy/Gal You aRe") rating to account for those "warm fuzzies", which basically allows me to say, "I know you can't code your way our of a wet paper bag, and none of the other factors apply here, either, but I still think you're cool."

Since people tend to be a bit conservative in giving Apprentice certs to brand-new fresh-in-from-the-street accounts, because Apprentice certs actually mean something, if we were to somehow make Journeyer and Master mean something (in terms of abilities on the site), that would get rid of a lot of inflation. It would also mean transforming the existing metric into what it ultimately reduces to: "How involved are they w/Free Software?" becomes "Do they merit the kind of privileges I'm thinking of bestowing upon them?" We've got that already to a degree with giving Apprentice certs (as Denny mentioned). So if Masters have (warning: random meaningless idea ahead) some extra ability to post that others don't, we stop thinking in terms of "Do they do enough with Free Software?" and start thinking "Do I want to read the posts from this guy?" And that's ultimately what our high SNR is about.

If higher certs don't confer higher privileges, then what's the point? Serious, non-rhetorical question here: Why don't we just go to an "Observer"/"Participant" scheme, and condense Apprentice on up into "Participant"? They must mean enough of something to enough somebodies that nobody's suggested this before--why?

roblimo: The guidelines already explicitly list various "non-code contributions". Though from appearances there's an awful lot of people with enough non-code contributions to merit Journeyer status... You definitely belong, tho ;). BTW, we do have "Observer": That's what you're called when you first sign up and you haven't even been certified an Apprentice yet, where all you can do is post diary entries. I assume you meant "observer" more along the lines of "one who makes wry observations"...

And no, I don't know the details of how to integrate several metrics together into one (if indeed we still want one linear "final score" at that point). Nor how that affects posting ability. Maybe we should just have a "I want to hear your posts" ranking, independent of achievement or recognition within the Free/Open community.

On the WACGYR factor, posted 15 Jan 2003 at 07:38 UTC by nixnut » (Journeyer)

I'm not complaining that the social factors exist; I'm saying that they would be included as attributes to rate on. I don't want to replace 1 technical rating with n technical ratings;

The point I was trying to make is that I doubt that even if there are several technical and non-technical (i.e. social) attributes a large number of people will still mainly use the WACGYR factor to rate the attributes, even the technical ones. The hard problem imho is getting people to turn of their socio-emotional biases for technical attributes.

Now I'm a natural cynic, so I don't really believe there's a solution for this, but I try to keep an open mind so if any of you can think of a solution...

On the WACGYR factor, posted 15 Jan 2003 at 07:38 UTC by nixnut » (Journeyer)

I'm not complaining that the social factors exist; I'm saying that they would be included as attributes to rate on. I don't want to replace 1 technical rating with n technical ratings;

The point I was trying to make is that I doubt that even if there are several technical and non-technical (i.e. social) attributes a large number of people will still mainly use the WACGYR factor to rate the attributes, even the technical ones. The hard problem imho is getting people to turn of their socio-emotional biases for technical attributes.

Now I'm a natural cynic, so I don't really believe there's a solution for this, but I try to keep an open mind so if any of you can think of a solution...

Re: On the WACGYR factor, posted 16 Jan 2003 at 16:41 UTC by Malx » (Journeyer)

The answer is the same as in FIDO - your boss will check you for correctness (or it's boss will degrade him).

That means if you certify anyone - you must controll all activity of people you "trust to certify others". If not - you must uncertify them (if you whould not do this - the person certify you will uncert you (and up to root)).

Yes, and...., posted 17 Jan 2003 at 07:15 UTC by nixnut » (Journeyer)

does this actually happen? I don't think so.

Option 3, posted 17 Jan 2003 at 16:46 UTC by jbucata » (Apprentice)

Malx's solution is basically my option 3--which doesn't work even if everybody is diligent to keep up with everybody. Better is option 4, which means you don't have to trust other's certs at all if you don't want to.

enhancements, posted 17 Jan 2003 at 21:56 UTC by lkcl » (Master)

ha, cool. a clearly well-written article. raised interesting points, i particularly liked the one about the metric system not being broken, but: the trusters are.

(counter-answer later to that one :)

we decide to ignore the law, then people will be certified as Journeyers and Masters when by the rules they belong at most as Apprentices--and there's nothing that the rules (ultimately just an HTML file on a server somewhere) can do about it.

i have been talking to people about this one recently: a number of tme question the usage - at all! hated them! - of trust metrics; one of them pointed out that it's pointless to Certify someone in perl coding if it has nothing to do with the mission statement or the certification levels of the site.

how many people _read_ the mission statement or the purpose of certifications before actually clicking?

should it be made bluntly obvious to people via the user-interface - a two-stage clicking "you are about to Certify this person as xyz, read this statement describing what you are saying they do, if you know that they do not fulfil it do NOT proceed"?

regarding the permissions thing:

If higher certs don't confer higher privileges, then what's the point? Serious, non-rhetorical question here: Why don't we just go to an "Observer"/"Participant" scheme, and condense Apprentice on up into "Participant"? They must mean enough of something to enough somebodies that nobody's suggested this before--why?

yes, permissions have been raised before (in articles dating back to early 2000). however, as i've mentioned a couple of times, the site is basically laissez-faire by raph and the site pretty much Does Its Job, so the issue of making permissions Do More on _this_ site is moot.

however, on a more sophisticated trust-metric-controlled such as linux.edu, where mailman mailing list memberships are controlled by trust metrics, it becomes critically important to make appropriate usage of Certification Levels.

finally, regarding the "broken trust or broken metric" issue: i do not know if you are aware of this but one of the weaknesses of the trust metric calculation is that the "maximum flow" system in each "level" is unrestricted, so to speak.

i do not want to go into details here unless someone asks or indicates that it is worthwhile, but here is an example:

raph certifies joe as master, joe certifies fred as master, fred certifies mary as master, mary certifies jane as master, ..... ..... .... .... .... ... ..... .... NameX as master.

if any Certification chain is as long as the "capacity degree" chain length, where the capacities i believe are presently defined to 800,200,50,25,4,2,1 or something similar, then it is entirely possible that the last person in the chain WILL be Certified as Master.

seven degrees away from the root!

raph and i discussed this in some depth two years ago, and we concluded that the solution was to add in a requirement for _more than one_ Certificate for a level to be accepted.


raph certifies joe as master federico certifies jo as master

therefore, joe is master.

joe and only joe certifies fred as master, fred is NOT a master.

now, whilst conceptually it seems quite simple to add in multi-cert requirements, algorithmically it's a Pig. it's entirely possible that the only way to implement the algorithm is:

- define MIN_CERTS to 2, 3, 4 whatever.

- to start with the seed nodes, count the number of incoming Certs between the seeds themselves, REMOVE any seed nodes that have less than MIN_CERTS

- to add in the 2nd-degree nodes (i.e. all those people that the seed nodes Certify), and perform a Max Flow calculation. REMOVE any of the 2nd-degree nodes that do not have Flow coming in via at least MIN_CERTS paths.

- to add in the 3rd-degree nodes (i.e all those people that the 2nd-degree nodes Certify), repeat, repeat until no more nodes added.

i have a good feeling that this will _work_, but have you _any_ idea how long it takes to do a Trust Metric calculation for 10,000 nodes??? the pauses on this site for over a minute every 15 minutes are due to a tmetric/.lock global lock being created.

so if the present trust metric algorithm is order N**3, the above algorithm is Order N**4. EEK! :)

the only redeeming factor about the algorithm is that N will be, at each time, a smaller number.

i did recommend to raph another method: he said that it had weaknesses that i could not immediately understand that he didn't have enough time to explain in enough detail, ah well, c'est la vie.

believe me when i say i am still looking, and it's been two years.

i would be delighted to hear from anyone who has any possible solutions or ideas.

commas, posted 17 Jan 2003 at 22:04 UTC by lkcl » (Master)

i missed out some commas!

i did recommend to raph another method: he said that it had weaknesses, that i could not immediately understand, ...


Re: enhancements, posted 18 Jan 2003 at 04:54 UTC by jbucata » (Apprentice)

Well, I'm not exactly proposing that we go out of our way to find/invent higher-order privileges to dole out. My serious, non-rhetorical question was: Since we're not going to find things to reward higher-level users with, then why not just get rid of all the higher levels to begin with? Why not just have Observer and Participant (= Apprentice ∪ Journeyer ∪ Master)?

Your suggestion doesn't change the fact that people will tend to overcertify, when faced with having to pick exactly one ranking for another person that they want to certify. Option 4, the one I'm proposing here, would split it up into something like: "General technical ability", "Free/Open software participation", "Ability to certify others", "WACGYR", etc. If a lot of people like Joe, then Joe can get lots of high WACGYR certs, but that doesn't necessarily make him a Master (or even much of a coder).

Or else ditch the upper-level certs altogether and stick with "Participant", since there's no practical benefit in having finer distinctions than that. Why not? Why do we want to keep "Journeyer" and "Master" around as mere titles? Is it just because it makes for more interesting research problems (not a bad reason)? Is it an ego thing? Do we all need to go back and reread our Gerald Weinberg? :)

On to the interesting digression...

I don't quite get how it's supposed to work from your example. I don't get the part about removing seed nodes, for one.

Also, it's quite unclear how it scales up to the general case. Raph and Alan certify you a Master. You and Raph both certify me as Master. I only have one Master cert in distance 1 (just from Raph), so I'm "removed"--does that mean I can't get Master certification at all, since I didn't get it from a second seed node?

I'm not sure why the whole metric has to be recalculated each and every time an update is to take place--I'd think it'd be possible to start with the network flow as of the last run, and just look at the changes to people's certs for this run, and recompute it incrementally. Nor does it seem necessary to lock the whole site down while it's being recomputed--a slightly stale copy of reality would be fine (indeed, we've already got that now, for the up to 14 minutes and 59 seconds immediately after you've certified somebody).

Also, without really trying, I managed to dig up this statement:

Although Ford-Fulkerson was the foundation for almost every other flow algorithm, numerous studies have shown that Ford-Fulkerson is the slowest in execution.

But all this (a threshold of N certs to be certified) is like extracting information from a noisy, low-resolution source of data (such as from a high-tech scientific instrument) by taking moving averages, discarding outliers, etc. My option 4 is akin to getting better scientific equipment to provide more, higher-resolution data--although that leaves you as the scientist with the new problem of figuring out what exactly to do with all this data you're now collecting. At this point I've run out of brilliant ideas--I don't know how to take the multiple certification axes I'm proposing and collapse them down to one "can-I-post-or-not" ranking.

social implications of Certs, posted 18 Jan 2003 at 12:58 UTC by lkcl » (Master)

hiya, you are dead on about there being no point, on advogato.org as it stands, to any Certs other than "participant", which is why i mentioned linux.edu as an example where the trust metric levels actually _mean_ something:

  • Anyone may create a group, but only a Group that is successfully Certified as Master will get its own DNS name and Mailing List.
  • only an Ambassador can Certify someone as a member of a Group, thereby inviting them to join the Mailing List for the group.
  • Only Novices and above may post Messages.
  • Anyone may write an Article or News, but only Articles and News that is Certified at above Journeyer will appear on the front page.

the list goes on: it's a hell of a lot more sophisticated than what is equivalent to "Participant".

so that leaves only one remaining purpose for the advogato.org Certifications, or two at a push:

  • social and informational purposes to uninformed individuals: it is possible to draw, albeit not a very _good_ one, an approximate judgement value of a person's role in fulfilling the site's charter.

    it would help enormously to be able to refer to people by their deeds not their standing, but then we get in to the smacks-of-elitism-and-i-got-a-bigger-head-than-thou thing, which i don't want to go into right now.

  • for other sites to make HTTP queries to advogato.org (and maybe XMLRPC yukk at some point) of the Certification levels and for THOSE sites to make more use of the user Certs than is at present made on advogato.org itself.


i did indeed miss out the point of the top-level seeds self-certifying each other. if they do not do so, then the "quorum" breaks down. how can you trust the top-level seeds if they do not indicate to you that they trust each other??

wouldn't it be great if it was Law that politicians had to be trust-led Certified to make decisions?

and then they fell out, had to revoke the Certifications of their top-level peers and this was Legally taken as a requirement for a General Election to be called?


does that give you some idea of why it is important that the top-level seeds self-certify?

also i believe you may have misunderstood about the repeated Certification calculation: it is only one (very bad) algorithm that demonstrates the point.

perhaps i should explain it again: the purpose is to ensure that MORE THAN ONE certificate is required for an individual to be successfully Certified. i believe that this stops the "7 degrees away" people getting Master or Journeyer Certs that they do not even themselves feel that they deserve.

in order to ensure that more than one certificate is required, you can only trust those people who have already been Certified by people who alalready have more than one certificate.

consequently, you have a recursive algorithm that requires a recalculation of the Trust Metric, first of all including only the top-level seeds, then including the 1st degree people and re-Certifying, then the 2nd degree people and re-Certifying, then the 3rd etc.

now, your point about Ford-Fulkerson being inappropriate is well taken: some of the other algorithms especially the Edmunds-Karp one which is breadth-first (by degrees) _anyway_ it might be possible to adopt that algorithm to achieve the desired goal much more readily than Ford-Fulkerson can (especially as the advogato.org implementation uses depth-first searches for augmenting paths)

regarding updates and changes

it would be _great_ to be able to do incremental changes: it's fairly easy to establish the differences; it's also fairly easy to record the state information between runs, to record in the XML files the capacities of the edges.


i realise that things are not perfect. on advogato.org itself, It Does The Job: i believe that the chances of anything radical happening to advogato.org itself are fairly slim. as people have often stated here: we don't get much spam and that's GREAT and it's good enough.

however, there _are_ other sites out there that use trust metrics, and there are other uses other than even for web sites, so any additional improvements in this field _will_ be of use.

also if you find c-code intimidating, i did port the trust metric code to python, it can be found at http://sf.net/projects/pymmetry

option 4, posted 18 Jan 2003 at 13:05 UTC by lkcl » (Master)

okay, i re-read things: yes, your option 4 idea would be good, and on the code that is running linux.edu it's trivial to do: just create some xml files with the definition of the levels and the names, and start using them in xml tags (see http://sf.net/projects/virgule

however, and please do not consider this to be discouragement, but just a realistic assessement of the way things presently are: on mod_virgule, the code running advogato.org, a) it's not trivial, b) there's no point because the site is deliberately left _well_ alone by its maintainers.

being more encouraging: if you believe that trust metric based projects _other_ than advogato might benefit (e.g. the sourceforge ranking system darn i almost swore there is "based" on advogato trust metrics) then go for it.

i'd be delighted to have something more robust for linux.edu for example, which requires a quorum of Certifiers: it is more vital on linux.edu that people understand and correctly certify than it is on advogato.

Summary, posted 21 Jan 2003 at 04:54 UTC by jbucata » (Apprentice)

What I was hoping for when I posted this was one or a combination of:

  • a sound refutation of my proposal (option 4): "No, that won't work because you didn't think of this, this, and that.";
  • a groundswell of support for the idea (ideas of exactly what fine-grained rankings to employ would be even better); or
  • a good reason why we like today's redundant (practically speaking) set of non-Observer levels, and therefore don't want to flatten them down to just one Participant level (I suppose that's Option 5), even if we're at the same time unwilling to go fine-grained (option 4).

Unexpectedly, I managed to get a smattering of voices in favor of each of the four options, and the Option 6 that I neglected to notice originally: Hopeless Existentialism ("It is a problem, contrary to option 1, but there's no way we can ever fix it, so why even bother trying?")

And another interesting, and vastly more practical, idea about a double-certification approach which would help ameliorate the symptoms, even if it doesn't fully address the underlying causes.

A lot of what I'm hearing is simply inertia/status quo. I was hoping to get past the fact that it's not coded that way now, and find out what people would really like to see. However, I'm starting to think that a lot of people (including a significant chunk of the silent majority) prefer Meta-Option 1: Show Me The Code. If we coded it up and tried it, much like the oft-requested-but-tried-and-dropped Dimwit cert, then people could comment based on their practical experience with the new system, instead of speculating en masse on how it would be if we ever tried it. That shouldn't surprise me (this is a Free Software site after all), but it did--so if I surmised right, this conclusion is probably more enlightening than any of the ones I was expecting to get.

Thoughts on Other Options, posted 22 Jan 2003 at 20:31 UTC by robocoder » (Journeyer)

1. If Observer is a non-participant, there are only 3 remaining levels to represent participation. Central tendency would explain the assertion that there are many Journeyers. (A quick scan of the 50 most recent diary entries shows 4 by Observers, 2 by Apprentice, 27 by Journeyers, and 17 by Masters.) If there's a desire for further differentiation, why not add more participant levels? Some candidates include: Novice (arguably the same as Apprentice), Rookie, Expert, Grand Master, and Legend.

2. Taking option 3 to the extreme. If there are concerns about the accuracy of the metric (given its subjectiveness), why not yank all certs in one fell swoop, and let re-certification begin anew? It shouldn't take long to reestablish.

3. The certification guidelines suggests recency of contribution is a consideration (i.e., "...work done in the last year"). From what I've read, there are concerns about the currency of the metric. Perhaps the metric should consider aging the certifications and/or certifiers? (Member since, last visit, most recent post, number of posts, frequency of posts, etc.)

4. I can't remember where I read it, but someone mentioned that it took at least two certs by individuals with at least those certs to receive that cert. That number seems low. Does the minimum increase with the size of the population? (up to some arbitrary upper bound of course, e.g., 500)

5. How about other forms of popularity contest or peer recognition program (beyond peer certification)? Like being able to award a "star" to peers (perhaps limited to those a level below you).

Yes, when I have time, I'll go back and review my certs. And try as I might, I can't cert myself back down to Observer.

Another thought, posted 22 Jan 2003 at 20:46 UTC by robocoder » (Journeyer)

Instead of a single measure (i.e., certification level), why not poll a number of measures and combine these into a (weighted?) score as your input? Example:

Rank from 1 (Low/Poor) to 10 (High/Good), this individual's competency and contributions:

  • Soft skills (communications, interpersonal)
  • Design, Coding
  • Documentation
  • Project Management
  • Entertainment value
  • Signal to noise
  • Open source advocacy
  • Popularity of open source projects

And one more..., posted 22 Jan 2003 at 21:46 UTC by robocoder » (Journeyer)

If you can measure activity or recency of certs, you could introduce "Honorary" as a qualifier, e.g. "Honorary Journeyer".

robocoder's Thoughts, posted 23 Jan 2003 at 02:57 UTC by jbucata » (Apprentice)

Needing at least two certs to be certified: No, that was a proposal made by lkcl in this thread (and probably made elsewhere too previously). I know it's not reality, both because that's not what the trust metric definition says, and because I became an Apprentice after receiving only one Apprentice cert from Denny.

Using a number of measures: That's my option 4, the proposal I was setting forth in the article. You might want to take a closer look at what I originally wrote :). Nice to see that you agree with me ;).

Further differentiation: Well, it seems that just about everybody is out of ideas for useful things to do to distinguish between the upper levels in terms of what they can do around the site. Hence my sidebar question asking why we don't just go to "Observer" and "Participant" and get rid of any finer distinctions.

The problem isn't that the existing levels are too coarse in and of themselves; it's that they're not being strictly adhered to, and a lot of people would prefer to not be certed a Journeyer or Master if per Raph's definition they really only deserve Apprentice. The proposed solution, my option 4, calls for multiple axes--not one finer-grained scale but several (probably) coarse-grained scales--to make it easier to "say what you mean" so that cert inflation doesn't happen. Coarseness isn't the problem with the current system: Having multiple new levels like "Journeyer, Second Class" doesn't make it any easier to rate somebody who's highly skilled technically but doesn't contribute much to Free Software.

Scales, posted 23 Jan 2003 at 04:37 UTC by robocoder » (Journeyer)

I was just trying to offer some examples -- while I talked to myself ;) -- that made it clearer (at least to me) that many alternatives are non-trivial (e.g., require more input from certifiers and/or involve more computation on the backend).

Personally, I'm in favor of option 4. A scoring system could quantify differing value systems (composed of multiple axes) ... allowing the actual level to be determined by the system. Let's say the computed score is between 0 and 100. The site admin would determine the number of levels, thresholds, and weights. That in turn would become input to the nodes.

And personally, I think the current scale is sufficiently middle ground to not require changing, i.e., not too many, not too few.

Re: Scales, posted 27 Jan 2003 at 06:05 UTC by jbucata » (Apprentice)

Requiring more input from certifiers: Is that a bad thing? I'm not convinced that it is, if we make people think harder before certifying somebody. "More computation" is a hardware problem ;). Or else it's SMOP (= Simple Matter of Programming).

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page