Older blog entries for abram (starting at number 17)

test post

Last night I added preliminary support for filters to Hep. You can copy
and/or redirect incoming messages to other places besides your Inbox,
based on criteria you specify.

I was hoping to finish the configuration UI for filters and protocols,
and get a new alpha release of Hep out today. But I ran into some
issues. First, I learned that the default Jabber server configuration
doesn't like it when a connection sends too many messages in a short
period of time (it gives you a bad 'karma' rating and stops accepting
messages for a few seconds), which was a problem since I had set up a
filter to copy all incoming messages to a jabber address. Then I found
that Jabber (with the default server configuration) also doesn't like it
when you have more that 5 connections open at a time from a single IP
address, which breaks my one-connection-per-destination setup. So I'm
going to have to rethink the way the Jabber bot will work.

Also I'm occasionally getting some kind of race condition where Hep uses
100% CPU, which definitely wasn't happening before.

I've been adding a lot of new code to Hep over the past few days, and I
think soon it will be time to step back and think about the design a bit
before I get sloppy in my rush to add cool new features. I'll release a
new alpha version with all my latest changes early next week. Then I
might pause for a week or two, let people do some testing, collect
feedback and bug reports and suggestions, get outside and enjoy the fall
weather instead of spending so much time coding.

I was home sick from work on Tuesday, so I took the opportunity to hack on Hep a bit.  And since I wasn't feeling well I didn't feel bad about ignoring the important-but-not-fun items on my TODO list, and playing with some interesting things instead.

So Jabber support is now working in Hep.  You give Hep a jabber account and password, and it logs in as username@server/Hep (with priority 2), and once for each of your destinations (username@server/DestinationName, priority 1).  Messages send to username@server/Hep (or just username@server since it Hep has the highest priority) go to your Hep Inbox.  Messages sent to the other resources get posted to the appropriate destinations (you configure a list of jabber accounts to accept messages from, to prevent strangers from sending messages to your weblog).

I also added the concept of "protocol handlers" to Hep, so that it can figure out how to send a message to a mailto: or jabber: address.  A protocol works much like a destination, except that every time you send a message you also specify an address.  That way you don't have to set up destinations for every e-mail or jabber address you might want to send a message to - you just set up your e-mail configuration once, and then specify the address when you send the message.  (I haven't figured out all the details of how this will work, but in the short term it should let Hep handle filters like "Copy all messages from Slashdot to this e-mail address").

Hep's architecture is making me happy.  Adding Jabber support was only 100 lines of code (thanks to the nice jabber.py library).  The rest of the work is all generic support for protocol handlers, which can be added in the future by dropping modules in the messaging/protocols directory.

Also I made a diagram of how Hep works, which may or may not help other people understand it.

I'll try to get another snapshot release ready this week, so that other people can start playing with this stuff.  And I really should write up some developer documentation...

I've put a new version of Hep up for testing (see the bottom of the page).  This is an alpha release, meant for developers and curious people.  It has some nice new features:

  • Uses Mark Pilgrim's rssfinder module to find the RSS feed for a web page, searching syndic8's database if the page itself doesn't contain a link to the feed.  This means that Hep should be able to find the RSS feed for almost any site that has one.
  • Also uses Mark's rssparser module to parse RSS feeds, which works around some buggy feeds and eliminates the problems some people were having with Python's XML libraries.
  • Lets you set up message destinations, and has an SMTP server that accepts mail sent to those destinations.   Now you can send e-mail to your weblog (through the Blogger or MetaWeblog APIs), or your advogato diary, or both at once.
  • Lets you view your Inbox on a web page, so you can read and delete messages without using your mail client if you want.
A complete changelog is here.

But before you download, let me remind you that there are reasons that this is an alpha release:
  • The SMTP server has NO SECURITY, and only works for the first Hep user you create.  Don't run it on a shared or public system unless you want the whole world to be able to post to your weblog.
  • The SMTP server hasn't been thoroughly tested and might occasionally lose an outgoing mail message.  A suggestion:  put yourself on the bcc list for any important messages that you send.  That way you can verify that your message got through.
  • The first time you run the new version you'll probably get a lot of messages you've already seen (the way it generates message IDs is a little different, so all RSS messages look new).
These things will all be fixed soon enough, so if you're just using Hep as an RSS aggregator and you're happy with the way it's working I recommend you wait to upgrade, at least until this release gets into the beta stage.

If you're an existing user and you want to upgrade, here's what you need to know to get started fast.   Edit the hep.ini file that comes with the distribution (don't replace it with your old one - there's some new stuff in there related to the SMTP server, and it won't work without it).  Set the SMTP upstream_server=your.smtp.server.   Set your mail client to use localhost, port 5337 for outgoing mail.   Send mail to [destination_name]@hep.   And when you set up your destinations don't put spaces or special characters in your destination name.

If you're interested in seeing an almost complete list of things that need to be done before 0.3 is ready, check out the TODO file in the distribution.

Sitting on a bench outside the coffee shop this afternoon, I think I finally figured out how Jabber instant messaging can be connected to Hep.  I'm putting my thoughts up here for comments.   This is a first draft.  Let me know what you think.

  1. You create a Jabber account.  Say joesmith@jabber.org
  2. You set up some message destinations in Hep:  your weblog, your advogato diary.
  3. Hep will log into Jabber using your account, once for itself, and once for each destination you've got set up.  For each login it uses a different Jabber resource: joesmith@jabber.org/Hep, joesmith@jabber.org/Weblog, joesmith@jabber.org/Advogato.
  4. You give Hep a list of Jabber accounts it should allow posts to come from.
  5. Now anyone can send messages to joesmith@jabber.org/Hep.  These messages go to your Hep Inbox.  The /Weblog and /Advogato connections only accept messages from the accounts you've specified.  These messages get routed to the appropriate destinations.
22 Aug 2002 (updated 22 Aug 2002 at 21:42 UTC) »

I've been playing with my new wireless card (as you may have guessed
from my last post). I may write up something about what I've learned -
as a wireless newbie I was disappointed by the lack of good real-world
documentation on what to expect from wireless - things like what kind of
signal strength you need to get a connection, how to tell if the network
you're on is using encryption, etc. Maybe I just haven't looked in the
right places, but I've spent the time on Google.

I'm still working on Hep 0.3, but I think I'm going to have to push the
release back to next week, and even then it will probably be an
"0.3-alpha" release. Hep 0.3 has a built in SMTP proxy server that
watches messages going to your ISP's SMTP server. If it sees a message
addressed to <address>@hep, it delivers it to a destination you've
configured. I'm sending this message to weblog@hep (my weblog), and
cc'ing advogato@hep (my advogato diary). It seems to be working so far,
but the thought of people sending all their e-mail through Hep makes me
nervous: suddenly Hep will be handling information that's actually
important. So I want to do some good testing, and let the brave and
curious use it for a while before I do a "real" release.

posting from my e-mail client...

I added a new feature to Hep. Here's how it works. You give Hep the URL of a web page you'd like to subscribe to. Hep grabs the page from the URL, and passes it (along with the original URL and HTTP headers) to each of its message source drivers. Each driver gets a chance to analyze the page information and try to get messages from that page. If it can, it returns a working configuration. If not, Hep goes on to the next driver.

Currently I've added support for community sites using Slash, Virgule, or PHP-Nuke, weblogs that use either a <link> tag or an xml.gif image to point to their RSS feed, and individual diaries on Virgule-powered sites. And I wrote a simplified "Subscribe" bookmarklet that just passes Hep the URL of the web page you're currently looking at. This makes the process of adding a message source almost completely painless:

  1. While browsing, you come accross a page you'd like to subscribe to.
  2. You hit the Subscribe link on your browser's Links toolbar.
  3. You're transferred to Hep, which verifies that you want to add the page as a message source.
  4. You press OK, and are returned to the original page.
  5. From that point forward you get a message in your Inbox every time the page is updated.

All this and more in Hep 0.2.3, coming soon!

26 Jul 2002 (updated 26 Jul 2002 at 19:29 UTC) »

There's a new version of Hep (now officially called Hep Messaging Server) available for download on my website. The GUI from version 0.1 is gone, replaced by web and POP3 servers that let you configure Hep through your browser, and get messages through your e-mail client.

Sending messages to weblogs still isn't supported, but it will be in the next couple of weeks. So Hep 0.2 doesn't really do anything more than 0.1 did - it works as a news aggregator, and that's about it. But internally it has a lot of new features - built in POP3 and web servers, support for multiple users, and a cleaned up API, which gives me a good platform to build on.

Now that it doesn't have its own client interface, Hep should run on any operating system that supports Python and the Python standard libraries. I've been developing it on Linux, but this morning I tested it on Windows and it ran, without errors, the first try. Hooray for Python!

I've released a new hep tarball that fixes a bugs in the 0.1 version, where messages with long Subject lines and/or IDs would get corrupted. The new also version stores it's messages in maildir directories, which is faster and much less prone to corruption. Also, it uses a slightly different format for the ID numbers it generates for RSS newsfeeds.

This means that, if you're currently using Hep 0.1, you'll lose whatever messages are currently in your Inbox, and the first time you run this version it will re-download messages that you've already read and deleted. All your accounts will be intact, though.

Thanks to everyone who's given me feedback on Hep over the past two days!

8 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!