Older blog entries for mindcrime (starting at number 38)

10 Jun 2010 (updated 10 Jun 2010 at 06:07 UTC) »

Introducting Project Shelley (with lots of screenshots!)

So, if you follow the OpenQabal Twitter stream or my personal twitter stream, you've heard some chatter about something called "Project Shelley." You may assume it had something to do with my recent post about restarting OpenQabal development, and you would be correct.

So, what is this "Project Shelley?" Well, for starters, Project Shelley is just a temporary code-name. The project will get a better name later (maybe we'll have a contest or something) but it works for now. Someday I'll explain how the name came about... for now I'll just mention that it's a nod to our friend Mary Shelley the author of Frankenstein.

Now with that out of the way, let's get down to the nitty-gritty. Project Shelley is the first bit to come out of rethinking the direction of OpenQabal. In the past, I was more focused on the broader social-networking aspect, thinking about decentralized, federated social-networks. But my personal interest was always more in enterprise applications of this kind of technology, and NOT in trying to build a "facebook killer" ala Diaspora or whoever.

So, while working for a semi well-known self-publishing company in the Raleigh area back in 2008, I started playing with the Open Reddit code, looking at how that type of technology could be a complement to some other things "behind the firewall" vis-a-vis knowledge management. Some people showed interest, but no real champion ever stepped up to push its use, then the economic collapse happened and side-projects became less emphasized, and then I left the company in 2009. But that experience planted a seed, and so the first new OpenQabal sub-project is - essentially - a very Reddit (or Digg if you prefer) like application that uses voting, tagging, sharing, filtering, etc. of articles and documents.

In it's present form it looks a lot like "just a Reddit clone" but the intention is to move beyond that, and I'll talk more about the more advanced features later (and to be fair, Shelley already has things that Reddit doesn't, but it also lacks things that Reddit does have) but tackling something that starts of as a "reddit like" gave me a chance to get started with a well known problem domain, and a chance to get something tangible out the door to start poking and prodding and playing with.

The current version has a lot of functionality, but will need a fair amount of "cleanup" work to be anywhere near production ready. The intent here really was to blast through as much as possible in a short period of time just to make this project feel real again.

So... with no further ado, here are screenshots and details about what this stuff does.


The front page, which is the default view of submitted links for the "default" channel. ("Channels" in Shelley lingo are like "sub reddits" in Reddit lingo)

The Login page, awaiting login:

The front page after login. Notice the new tab in the upper right hand corner. That takes a user to their user profile "stuff."

Sharing a link. It's kinda hard to make out, but the first entry in that field is a plain email address (it's mine, don't spam me, ok?) and the second is the same address but prefixed with "xmpp:" Yes, we support XMPP messaging.

As you can see here, I did receive an XMPP message about the shared link.

And now we're looking at tagging support. It's primitive right now, but this page shows a user a list of all the tags they've used:

And clicking on said tag displays a list of the posts you've tagged with that tag. What hasn't been done yet is any work on dealing with tags on a "global" basis (eg, can I see links that somebody else tagged with a given tag?)

Notice here that the Search dialog has been filled in with the string "information retrieval".

And here we get our search results:

Click the "comments" link for a given entry, and you come to this page, which lets a user view and add comments to an entry, and also shows other similar links. Right now the "recommended links" stuff is built just using the MoreLikeThis class from Lucene Contrib, but this is one of the areas that's going to get some interesting work in the future. In particular, I want to supplement this by using knowledge of tags and social-graph connections, to (hopefully) get better results than from just a strict content similarity score.

About to enter a comment:

With a new comment added:

The "User Profile" page. It's pretty spartan right now:

The "Saved entries" page. Things go here if you click the "save" button under an entry.

Adding a tag to an entry. The input field is hidden until you click the "tag" link under an entry.

Same thing, after putting some tag text in.

So there you go...a quick look at what Project Shelly is, currently. Coming later, more on what it will become in the future, and some info on the even more mysterious "Project Poe."

Syndicated 2010-06-10 04:46:32 (Updated 2010-06-10 05:17:08) from openqabal

13 May 2010 (updated 10 Jun 2010 at 04:15 UTC) »

Latest OpenQabal News

I'm sure most everyone is assuming that OpenQabal is dead. That would be a reasonable assumption given the lack of activity on my part for the past while. And it may even be true, for a certain value of "dead" and a certain value of "OpenQabal." Certainly since the last time I was heavily engaged in this project, the world around us has changed, and context and assumptions have changed. To everyone who wanted a decentralized, federated, open-source social-networking platform 2 years ago, I apologize... I let myself get distracted by other concerns and let this languish.

This is doubly sad given the recent burst of interest in that very space (see: Diaspora), after the Facebook's Gone Rogue bit; but things are what they are and there's no use crying over spilt milk.

So, the future then... As I said almost a year ago, the goal - in my mind - had already changed to being more about APIs for building "social applications." I believe that is still the case, and that is still the direction that I'll be pursuing with code I write in the near future. There is another subtle (or not so subtle) shift in focus as well... More than before, my intent is to focus on APIs, interfaces and tools for building socially aware applications in an enterprise context. As much as I hate buzzwords and labels, I'd say the goal is a Enterprise 2.0 suite.

As my current interests have moved heavily in the area of social ranking / voting, collaborative filtering, tagging, etc., I've started coding up a front-end to work with those techniques. As that evolves, some of the same APIs that were part of the OpenQabal vision 2 years ago will begin to evolve again. Some new things may appear, some old things may get dropped, as I explore some ideas about how all of this will work. At any rate, there *will* be new work coming out of the OpenQabal project, it just might not be what anybody thought it would be before.

Some things I know I'll be doing:

  1. The newest code I'm writing is in Groovy / Grails. Any user facing / UI stuff I work on in the near-term will probably be in Grails. Backend APIs will *probably* be plain Java, but don't rule out seeing Scala or Clojure show up somewhere. And depending on what kind of stuff we wind up needing, some C++, Python or Erlang or "other" could wind up in the mix as well.
  2. The new code will be going up on GitHub soon. Sometime after that, the existing code probably will as well (this is assuming that the existing code base is worth keeping around). I'm convinced that DVCSs are the way forward, but if a convenient way can be found to do it, we can mirror the code into java.net svn as well.
  3. I've been working with Hudson at my day job, and realize I like it better than Cruise-Control. I'll see about getting a VM setup somewhere, to host Hudson, Bugzilla, etc. Having that stuff running on an old box in my home office, sitting on a cable modem, is not ideal.
  4. Soon I'll see about getting an instance of Agilefant, Ice Scrum or XPlanner+ setup for managing the backlog
  5. At some point, I will take another stab at making an actual "road map" and some documentation that illustrates what it is that I'm on about here. I know I haven't done a very good job of articulating what the goal of this project was in the past (that's because *I* didn't really know either) and when I have articulated something, I've diverged from that more than once. Such are the perils of an exploratory approach. It will all become clearer eventually. No, really...

Of course all of this just reflects what *I* intend to do. I am not the project, per-se. If members of the community want to do different things, well.. standard OSS project terms apply: submit ideas to the list, and/or write code, and send it in. If it seems reasonable for inclusion, it could very well get included in the main code-base. And if I - in the "benevolent dictator" role - reject your pet idea, patch or suggestion, then you can fork the code. Once we go DVCS, that becomes even easier! Win for everybody.

So... OpenQabal is dead, long live OpenQabal!

Syndicated 2010-05-13 04:58:48 (Updated 2010-06-10 04:03:45) from openqabal

Is OpenQabal dead?

A few emails have shown up in my inbox over the past month or two, which I've been sadly negligent in replying too. If you sent one of them, I will reply to you directly in short order, but I did want to post a quick update here.

The question lately has been: "What's going on with OpenQabal? Is the project (dead|inactive|stalled|etc) or alive?"

The short answer is, the project IS still alive. For some value of "alive" anyway. This project has not received much attention from me lately, for a variety of reasons. Late last year I was involved in a political campaign, and that bled over into the holiday season, and I basically said "I'll work on it after the beginning of the year." Then January came and went, and my passion for hacking code in my every spare hour got pushed aside for some other interests for a while. But at no point did I decide to drop this project, and I fully expect some work to resume shortly.

Of course my life just keeps getting more complicated, as I'm currently in the middle of starting a new IT consulting and custom development shop: Fogbeam Labs. The bad news about that is, it takes time and cycles that I could be spending on OpenQabal. The good part is, if things work out well and I prove I can support myself consulting, I should (in theory) have more freedom to dedicate some time to OQ, unlike when I was doing the "40 hours a week for The Man" job.

One point of interest though: The focus of OpenQabal might change. After a very enlightening discussion a few weeks ago - with a friend who's opinions I respect greatly - I'm thinking that OpenQabal should probably drop most of the directly user facing "stuff" (for now at least) and focus on become a set of enabling APIs for building social applications. Of course those APIs were always part of the plan, but what I was trying to build was the APIs and a default suite of applications to use them, all at the same time. And since this is still mostly a one-person project, it's proving somewhat overwhelming.

So, the net-net is: OpenQabal is still alive, but the focus may be changing, and the roadmap along with it. And I can't make any specific promises at the moment about what will be done when. Unless you want to hire Fogbeam Labs to do some OpenQabal work, that is. :-)

Also, to the couple of folks who have asked "I want to use OpenQabal in a production site, can I?" The answer is a qualified "maybe." The blogging and forums stuff works fairly well, since it's all based on existing code from established projects. And SSO between the Roller App and the JavaBB App using CAS works fairly well. The last thing I was looking into before things got stalled out was how to make CAS into an OpenID Relying Party so one could sign in using OpenID and still get SSO between apps. That may get dropped for now, once work resumes however.

Syndicated 2009-06-02 09:15:13 from openqabal

Some Notes on Identity Management in OpenQabal

Some quick notes on what's going on with the Identity related stuff in OQ. Not necessarily in order:

1. One of the key concepts in OQ is the notion of federated identity... that is, an identity associated with one install of OQ should be "usable" from another install, where "usable" might mean "user at Site A can list user at Site B as a friend" or it might mean "user from Site B can login at Site A" or it might mean both, or it might mean neither.

2. That said, part of the idea of OQ, at any given install, is to be a sort of "social operating system" that enables application integration based on "social" technologies (the social graph, social ranking/voting, etc).

3. The use cases around (2) above are where some actual work has been done, and stuff works. We currently integrate apps running in the same domain by using JA-SIG CAS, with the authentication backend being our own simplistic IdM system (refered to as the IdentityEngine).

4. At one time it seemed to make sense to have the IdentityEngine project there to act as a mediator / bridge to possible other existing enterprise IdM systems. In retrospect, I'm no longer sure that adds much value.

5. It's entirely possible that a better approach would be to rip out the existing OQ IdM 'stuff' and just plop in Sun's OpenSSO / OpenFederation / OpenDS / OpenPTK stack. I'm doing some research on that point at this very moment, while also continuing to work on the existing OQ stuff based on CAS and the IdentityEngine.

6. Part of the point in rolling out this initial, somewhat naive, built-in IdM was to give us a test platform to experiment with issues around (1) above. To that point, I've been working on the possibility of making CAS an OpenID Relying Party, so that a user can login to OQ through OpenID and get the full CAS SSO experience.

7. Much research still needs to be done to sort out the best approach(es) to handling (1) above. OpenID may well be part of that solution, along with OAuth, but don't hold me to that. SAML, WS-Federation and some of the Liberty Alliance stuff may also be useful. Still need to do more research there.

8. What works now is this: SSO using CAS, successfully integrates Roller, JavaBB and our "User Console" app. Roller has a mechanism for fully externalizing user management, so it's not technically required to provision a new user into the Roller user table, although that code is still in place at the moment. JavaBB, on the other hand, does not (yet) have that ability, so when we create a new user, we have to populate the JavaBB user table.

9. One of the big things I want to determine is "what are the implicatiosn for user provisioning, when using an external authentication source like OpenID?" See comment (6) above. Getting that done will enable some experimenting around how this will work in principle.

10. User provisioning with SPML is also of interest.

11. The IdentityEngine exposes remote interfaces using EJB3 SLSBs, this is used by our implementation of the pluggable UserManager in Roller (there are use cases which require exposing APIs like "get list of users" and "get user by name" etc. In addition, we have to be able to authenticate "out of band" to enable authentication for the MetaweblogAPI interface (and presumably APP as well).

12. Another point that has not been addressed at all is two-factor (or multi-factor in general) authentication.

If anything else comes to mind, I'll make a follow up post. Please send other questions to the dev@ mailing-list...

Syndicated 2008-11-18 04:37:53 (Updated 2008-11-18 04:46:42) from openqabal

Finally back to work on OpenQabal

So after a rather long hiatus - due mainly to a political campaign - we're finally getting back to work on OpenQabal here. Of course our old road-map is now shot to heck, so it's going to take a while to sit down and put together a new plan and get some momentum going again. But I can say that a lot of work has been done the past week. Some of the needed work done includes: swapping all of the EJB2 SLSBs that were in place for remoting with EJB3 SLSB's instead, plugging in some of the JMS event messaging code that had been planned (and prototyped like 2 years ago!), renaming some modules, some serious refactoring in the IdentityEngine project, and - last but not least - getting the CruiseControl build-loop all sorted out so that everything builds continuously again.

So what's next in the short-term? Some or all of the following:

  • update CAS to the latest release
  • major improvements to the IdentityManager module
  • OpenID login support
  • support for provisioning users who are logging in using OpenID (or other 3rd party authentication services)

Getting those things knocked out will represent major progress. After that, it'll be time to start on the GraphEngine, PolicyEngine and ProfileEngine stuff for managing the social-graph, profiles and privacy / access control issues. After that, hopefully we can finally get started on the federated social-graph using FOAF... and after that, start on the SIOC stuff for distributed conversations. Oh and something needs to be done with the user-dashboard module to support plugging in applications. That may turn out to be using Portlets, or possibly the Google Gadgets / OpenSocial stuff.

After that... well, let's not look quite *that* far ahead just yet.

Syndicated 2008-11-04 04:22:17 (Updated 2008-11-04 04:26:52) from openqabal

So, what's going on with OpenQabal anyway?

Anyone following the OpenQabal project may be wondering: "What the heck is going on? No recent commits to SVN, no blog posts... what's up, is this dead or what?" Well, the answer is "No, the project is hardly dead." It has been on something of a hiatus however, as the main developer is currently running a campaign for NC Lieutenant Governor and has been a touch occupied the past month or two.

Rest assured though, that things will get back to normal. I'm hoping to get some work done this weekend, and once the election is over, the pace should pick up again. Unfortunately the old road-map is totally hosed now, at least as far as schedule goes, so that will need to be re-worked.

In the meantime, feel free to grab the code, experiment, research, submit proposals, patches, etc. And definitely email me with any questions or concerns.

Syndicated 2008-08-13 02:58:47 from openqabal

OpenQabal 0.0.3 available

OpenQabal 0.0.3 has been released and is now available from the SVN repo. As with previous releases, there are not yet pre-built binary releases available... you will have to check the code out from SVN and build it. Thankfully this process is now *much* easier as a result of massive work on the build system and the addition of scripts to automate most of the tedious stuff. You can now essentially build and install OpenQabal with 4 commands.

Changes in this release include:

  • New, more modular build system using Ivy for dependency management and incorporating useful tools such as FindBugs, JDepend, PMD, TestNG, Cobertura, etc.
  • New "User Dashboard" component that provides the main point of entry and ties the various components together visually
  • Concordantly with the introduction of the new User Dashboard, all of the old Sitemesh stuff has been ripped out
  • New configure and install scripts to automate most of the tedious parts of building and installing OpenQabal
  • Addition of many unit tests (we still don't have 100% test coverage, but progress has been made)
  • New "IdentityEngine" component introduced (more on this later)
  • Posting blogs using MetaWeblogAPI now works with the OpenQabal IdentityEngine component
  • Various minor bug fixes and tweaks

This release is a huge step in the direction of having a really usable system, but there is still a LOT of work to be done. Some things that had originally been planned for this release had to be deferred to 0.0.4, with OpenID support being the most notable such item. And of course there are still plenty of ideas on the roadmap that we haven't even gotten started on yet. But we're now in a lot better position to be able to start on some of the more interesting stuff, such as the "distributed conversation" support, tools for building a managing the "social graph," etc.

Look for a new post soon with more discussion of what the roadmap will look in the near future, as well as some discussion of some of the important changes that did make it into 0.0.3. And with any luck, the long-awaited "demo server" will be coming soon, as well as some screencasts and other ways of exploring OpenQabal.

Syndicated 2008-05-10 22:24:31 (Updated 2008-05-10 22:25:45) from openqabal

2 more tasks until OpenQabal 0.0.3 can be released

Finally getting closer and closer to getting this OpenQabal release out. The last two tasks to do are: A. verify the new SQL which includes a "seed" admin user, and B. make the install/config/uninstall scripts and any utility scripts working on Windows. Most of that stuff is in Ruby so it should be pretty portable, but there are some Bash scripts which will need equivalents written as DOS batch files. Ugh. Still, this stuff should be done soon, maybe tonight, which means the release should happen in the next day or two.

In other news, I've started working on getting setup with Amazon EC2, in order to host a "demo" instance of OpenQabal.

And after that, I think I need to go on holiday for a while. :-)

Syndicated 2008-05-04 21:46:48 (Updated 2008-05-04 21:49:26) from openqabal

29 Apr 2008 (updated 4 May 2008 at 22:10 UTC) »

OpenQabal Update

Ok, the new OpenQabal release that was mentioned a few weeks ago is *finally* eminent. No really, it's coming soon. The big hold-up turned out to be a bit of Roller integration that had been overlooked. Getting single-sign-on and integration with the centralized identity stuff working using Acegi and CAS takes care of web access; and that stuff has been in place for a while now. But at some point we realized that API access for posting new blog entries also needed to be wired into the new authentication stuff. And that's a whole new can of worms.

Basically, deciding to implement that in this release meant moving forward with implementing an initial version of what we're calling the IdentityEngine, which will eventually be the centralized source for all things identity, vis-a-vis OpenQabal. Even for OpenID logins or other logins using an external authentication provider, OpenQabal has to keep track of that that that such-and-such identity exists, and map it to resources, etc. So getting this piece in place was kind of a big deal, even if this first cut at it is fairly naive.

Anyway, with that in place and API access now working, the last things left to do before a release are basically all "fit and polish" work. Clean up some loose ends here and there, more testing, tweaking the SQL scripts and the install scripts. Oh, and the install stuff needs to be tested on Windows and any relevant tweaks made. At any rate, if things go well, look for the new release maybe this weekend, or early next week.

Syndicated 2008-04-29 04:09:50 (Updated 2008-05-04 21:50:28) from openqabal

29 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!