Advogato: Community-Moderated Mailing Lists

Community-Moderated Mailing Lists

Posted 5 Jan 2002 at 09:38 UTC by zak

Open Source mailing lists can easily spiral into a high noise-to-signal state. The following is a rough idea on how to make a system that would allow low-effort group moderation of a mailing list.

Basically, the system involves integrating a trust metric system into a mailing list. The idea is based (strongly) on advogato, moderated mailing lists and (roughly) the concept of process priority.

The intent of the system is to improve the quality of posts to the list, as well as reward users and threads that follow implicit and explicit list etiquette.

The system could operate in the following fashion:

All email addresses - possibly with the exception of a small group of seed accounts - start with an empty karma. Additionally, each individual message/thread on the list will have a rating associated with it. Karma levels and message ratings would be included in the message headers - allowing great flexibility in filtering received mail.

Karma could consist of one or more markers - it is an abstract rating that would need to roughly represent the following information:

Activity : How frequently the address posts
Quality : How other addresses rate the address's posts
Trust : How other adresses rate the address's user

The karma associated with an address alters characteristics of the address's posts.

Addresses with high activity, quality and trust can post without impediments. Their ratings of other address and posts can have significant consequence - even on an individual basis.

Additionally, they receive messages sent to the list before other users - providing them with a chance to enter discussions early on, modify the ratings of messages, etc. The changes that they make to karma and ratings will be applied to users and messages as they are made, thus affecting how messages that they receive will be delivered to other users.

Users with lower karma levels have delays imposed on the delivery and receipt of messages. Like users with high karma, they to can modify the ratings of messages, users, etc. Individually, their ratings will only have moderate effect, but as a group or over time, they can have very significant effect.

The lower the karma, the more impediments imposed on the address.

To reduce cycling of addresses, accounts with empty karma are fairly limited and have little effect on karma and message rating.

Regardless of address karma, the ratings that its owner makes on others karma and messages are logged - if nothing else, they can provide advisory information for the address being rated.

The tools to modify karma/message ratings would be embedded in the footer of the email message. Users would be encouraged to use these tools, rather than replying with a "me to" or "I beg to differ" or "don't waste our time" message. Modifications to account karma would likely require a confirmation message for users without high karma - this would help reduce accidental/poorly considered ratings.

I believe that a system like this would allow a list to be community moderated with minimal overhead. It would reward careful messages and politeness with responsiblity and a publicly viewable indication of respect.

Comments are welcome!

A snag but not too bad :), posted 5 Jan 2002 at 13:25 UTC by darkewolf » (Journeyer)

Well, at least one potential snag.. Maybe more. Depends what I think of as I type.
The most notiable snag, is that low to zero karma'd people would have a hard time contributing new ideas to a thread. Those with higher karma would get in first, and say it. This would lead to lower ratings for their articles (using the slashdot.org rating system, they'd get a redundant rating).

However, this in itself would not be a too large problem using the technique described above. Coupled with frequency of posting, and the chance of getting something unique in, it would be reasonably possible for someone new to increase karmic value.

Even the delay technique, although it would be initially painful, would give those new to the forum time to observe and consider the general list feel and mode of communication.

Of course, well known individuals could be seeded high when they join the list.

At least from my point of view (being on many theological mailing lists and history lists), this technique would be beneficial beyond the technical field.

implementation advice, posted 6 Jan 2002 at 11:29 UTC by lkcl » (Master)

pymmetry is the trust metrics ported to python. you may find this easier to use, adding it to a python-based list manager or email system than to a c-code-based one.
there are examples in pymmetry - FileCerts class, and FilePeople, which store user Certification info in file storage. it's very very easy to implement your own storage / database mechanism whatever. if you get stuck i may just help say implementing a sql db example or something.
i hate perl. whoopsie. can't advise there.
i wouldn't advise using xvl for this: the back-end is in xml hard-coded: there's not all that much documentation or usage base for any such program based on it to become well established.

implementation advice, posted 6 Jan 2002 at 11:30 UTC by lkcl » (Master)

pymmetry is the trust metrics ported to python. you may find this easier to use, adding it to a python-based list manager or email system than to a c-code-based one.
there are examples in pymmetry - FileCerts class, and FilePeople, which store user Certification info in file storage. it's very very easy to implement your own storage / database mechanism whatever. if you get stuck i may just help say implementing a sql db example or something.
i hate perl. whoopsie. can't advise there.
i wouldn't advise using xvl for this: the back-end is in xml hard-coded [i.e. i mean you can only use xml as the back-end, not that that isn't very powerful and flexible, it would just be a pain to add SQL to xvl, for example]: there's not all that much documentation or usage base for any such program based on it to become well established.

something from nothing & getting what you pay for, posted 6 Jan 2002 at 17:43 UTC by wnewman » (Master)

zak writes
All email addresses - possibly with the exception of a small group of seed accounts - start with an empty karma.

It seems to me that you need some sort of seed of validity/relevance/whatever in order to get started. The quote above seems be the only such seed mentioned in the proposal. If you want to follow the proposal closely and make things work, I don't think you can just skip it. Trying to bootstrap validity from nothing -- empty karma in all accounts -- seems broken. Then fundamentally what would validity be based on? Volume of input?

Aside: You might be able to make things work with a separate karma from the point of view of each user. Then each user could rate articles for himself, and see articles based ultimately on his own ratings. Under such a system, anyone trying to screw up the system by bogusly giving karma to trolls would only pollute his own input, instead of polluting everyone's input. But that's not what I'm going to write about today.

There are other ways to get started on validity not mentioned in the proposal. I can think of two, and I'd be interested to hear about others.

First, you can build on some existing somewhat-valid set. Fundamentally, this is pretty much the "seed accounts" idea in the proposal, except it's orthogonal to the "small group" idea zak associated with it. E.g. in a discussion group on education you might give one validity vote to every graduate of every accredited college everywhere, or every alumnus of your high school. That'll potentially skew the discussion away from the experience of people with different experiences, so it's certainly not perfect. However, at least it protects you from many kinds of manipulation, since most trolls are only going to be able to get control of a few votes.

Second, you could do something which seems to me rather different from web of trust: base results on monetary input, so you get what you pay for. Then, if trolls or other adversaries want to cause confusion, they'll succeed, but they'll have to pay enough much to do so -- in money transferred to the other users -- that the other users will still be content. "Yes, I skimmed 14 confusing/annoying/worthless articles, but at least I got $8 for my pains."

I haven't figured out how to use this second approach for most discussion boards, and I'm not sure it's worth trying to figure it out without actually implementing it. Instead, since people who actually implement this kind of thing are reading this, instead I'll discuss an easier example, and suggest that it could be generalized.

My easier example is e-mail spam. This seems to be a sufficiently parallel problem that I've seen people writing about using web-of-trust techniques to suppress spam. However, such techniques seem to be somewhat tricky to get right. The proposals I heard about seemed complicated. Furthermore, I wasn't completely convinced either that they stopped all possible spammer strategies effectively or that they let in all sufficiently important messages.

The you-get-what-you-pay-for solution to spam seems much simpler. (Caveat: You have to build it on top of a payments system, like e-cash or at least a legal system which allows binding contracts through e-mail. Such a payment system is complicated, but that's complexity that someone else has already worked out for you.) You set up your mail handler so that it looks for mail containing an offer that says "I'm confident that this message is worthy of your attention. If not, please cash the $5.00 certificate attached." If it sees this offer, it drops the mail into your inbox. (Also, probably it uses some other rules, like if you've gotten worthy mail from an address before, it drops future mail from that address into your inbox whether or not it has a worthiness certificate.) Otherwise, it automatically replies to the sender "I'm sorry, Mr. Newman doesn't accept unsolicited mail without worthiness certifications of at least $5.00.") Voila! As far as I can see, this is a simple and robust system that gets a lot of the behavior you'd want.

I think something analogous to the get-what-you-pay-for behavior here would be a nice property of discussion boards. For boards where people are willing to pay (or offer to pay) for attention, it's easy to see how to apply the scheme above. For boards where instead people are paying for knowledge, it's not immediately obvious how to apply it, but my intuition is that some suitable generalization of this scheme is possible. Comments?

Another aside: I've was told long ago that Robert Heinlein proposed something parallel to the e-mail scheme above, but for telephone answering systems. I don't know any reference for this, and I'd be interested to know whether it's true. (It mightn't suffice to keep some schmuck from being granted a 17+-year US monopoly on it, but it couldn't hurt.:-)

re: A snag but not too bad :), posted 7 Jan 2002 at 04:30 UTC by zak » (Observer)

Thank you for the feedback!
> The most notiable snag, is that low to zero karma'd people would
> have a hard time contributing new ideas to a thread. Those with
[cut]

Addresses with higher karma will have the chance to comment first. I would guess that the dynamics of the system would allow subscribers to receive the high-karma comments and original message at the same time. This would not stop similarly karma'd addresses from posting duplicate ideas, but little can prevent this. :)

re: implementation advice, posted 7 Jan 2002 at 04:33 UTC by zak » (Observer)

Thank you for the advice. I too would rather not use C! :) My concern is the speed of the system. The overhead could be quite significant for large mailing lists - C might be needed to get reasonable performance.

Re: something from nothing..., posted 7 Jan 2002 at 05:01 UTC by zak » (Observer)

It seems to me that you need some sort of seed of validity/relevance/whatever in order to get started. [...] Then fundamentally what would validity be based on? Volume of input?

If there were no seed accounts, there would certainly be an interesting dynamic to the list. I would guess that users would start 'trading' ratings - making things much more political and easy to poison. You are likely right that seed accounts are likely to be a neccessity - however, trying it without seeds would certainly be interesting. :)

Aside: You might be able to make things work with a separate karma from the point of view of each user. [...]

A very good suggestion. I would bet that you could derive decent group ratings from these personal ratings as well.

First, you can build on some existing somewhat-valid set. [...]

That could work... and gives me a funny idea. What if granting karma meant that you were allowing that persons karma to affect your own. :) Trolls would not get much support... :)

Second, you could do something which seems to me rather different from web of trust: base results on monetary input...

It might be tough to implement with current payment systems, however, it would certainly cut down on spam. Hell, users would even sign up for spam. ;)

For use on community mailing lists, what about basing the system on some other finite quality? Message posters would bet a quanity of something on their message. A net negative rating would mean that they would lose the amount that they bet. :) Given that they only have a certain amount to be with, they might consider posting more cautiously. However, if they don't bet enough to get past people's filters....