Older blog entries for mindcrime (starting at number 40)

11 Jun 2010 (updated 11 Jun 2010 at 03:08 UTC) »

OpenQabal, Project Shelley, Project Poe, Project Collins, WTF?

If you were paying attention to the previous post and its predecessor you should kinda have an idea of what Project Shelley is. At least, you may be thinking to yourself, "Ok, it's a Reddit clone, written in Groovy/Grails, and incomplete, what's the big deal?" And that's a perfectly fine attitude to have at the moment. The point of this post is to start clarifying where this is going, how Project Shelley relates to Project Poe (namesake) and Project Collins (namesake) and how it all fits into the OpenQabal vision.

Let's start by looking at what OpenQabal is meant to be. In the past, I had described OQ as:

...an open-source social-networking and collaboration platform / suite driven by the principles of federation, composition, and openness; with a special emphasis on enabling "distributed conversations" and the "federated social graph."

So, let's start by throwing that paragraph out, while keeping some of the essence of it. This is still - in part - about social-networking and collaboration, but "distributed conversations" (especially at "Internet scope") aren't really part of what I'm interested in tackling right now. Not because it's not interesting, or because it's a solved problem, but because I don't see as much value for it in the setting(s) that I'm focusing on. But, again, if members of the community see a place to do that kind of work within the OQ umbrella, then that would be great by me. Now, the "federated social graph" bit... I wouldn't say that's being thrown out, but rather placed on the "back burner."

So, that leaves us with, what, "an open-source social-networking and collaboration platform / suite driven by the principles of federation, composition, and openness?" Well, yes. But to make the scope of intended application(s) clearer, and the way things will decompose, I'd re-word that now as something like:

"OpenQabal is an open-source suite of APIs and applications - driven by the principles of federation, composition, open protocols, and open standards - for building and enabling intelligent enterprise applications for collaboration, social-networking, knowledge management and discovery, organizational learning, Information Retrieval and decision support."

No, that definition isn't perfect, and it'll morph over time. But I think that gets closer to the heart of things. "Intelligent enterprise applications" is really what interests me right now. And what I mean by that is, using technologies like collaborative filtering, tagging, social-graph mining, explicit semantics, data mining, machine learning, etc. to build (or integrate) enterprise applications in a way that makes them "smarter" or better able to help humans find the information they need, when they need it (even if they don't know they need it yet!) I've said before that I don't really like the label Enterprise 2.0, but for lack of a better term, you could say that that's what this is. Except we should probably call it Enterprise 3.0 just to one-up the competition, eh?

With that said, what OpenQabal becomes is sort of an "umbrella" or over-arching structure for sub-projects that handle different parts of this vision. Think of how Glassfish has become an "umbrella" project for a series of related projects: OpenMQ, OpenESB, SailFin, Portal Server, etc.

With OQ as the overall structure, sub-projects will deal with provide various APIs and/or applications / subsystems that are part of this overall "intelligent application" vision. A number of the pieces we talked about in the old OpenQabal model will live on now, pretty much as they always would have. I still see the need for an API / system for managing tags, something for doing recommendations, something for managing a social graph and allowing queries against it, etc. Of course, as before, in a lot of areas existing open-source code exists to do these things. In that case, we may (within the letter and spirit of the respective licenses) just "borrow" existing code, possibly wrapping or modifying it to fit the model of what we're doing here. Other bits will have to be written entirely from scratch, and that's OK.

So, what about these sub-projects? Well, here's what I think I know about them so far:

  • Project Shelley - An interface that builds on the APIs for tagging, ranking and recommending items, to provide a platform for sharing and discovering useful links, documents, people, etc. What we have right now is pretty simplistic, but there's a LOT of room for growth in this. Expect another post soon just dealing specifically with "what's coming" for Project Shelley.
  • Project Poe - a sort of "mini Facebook" like social-networking interface. Builds on the APIs for social-graph management, activity-stream, activity profiling, tagging, etc. Provides the front-end for managing connections and for letting users provide information about themselves, their interests, etc. But unlike Facebook, no silly Pirates vs. Ninjas or Farmville stuff.
  • Project Collins - This one we haven't talked about before. It may not come into existence as a standalone project, but the general vision is a search engine that leverages all of the various bits of information from Project Shelley, Project Poe, and "TBD" to provide better / deeper / more insightful search results than you can get from simple document content analysis. Intranet search in the Enterprise usually sucks, largely because page-rank type algorithms don't work well due to the lack of links between documents. But by supplementing the content analysis with scoring based on tags, social graph connections, activity-stream information, etc., it should be possible to do a much better job. This is not, by the way, a totally novel idea. It's something people are referring to as Social Search.
  • GraphEngine - as before, an API for storing and managing the "social graph stuff." Doing this on a large scale is still a problem, and I'm intrigued by the idea of using an incremental evaluation system approach for this, but haven't done much on this yet.
  • ProfileEngine - name says it all, really.
  • RecommenderEngine
  • ActivityEngine
  • TagEngine
  • Etc.

Of course since this is all intended for an enterprise setting, a big focus will be on integrating with other systems (see the point about "open protocols" and "open standards" above). Notice that the description for Project Shelley mentions discovering documents and people, despite the screen shots only showing stuff about links so far. That's where integration comes into play... part of the vision is to integrate with, for example, a document management system like Alfresco, a forum system like JForum, a groupware / calendar system, and/or a HRM system like OrangeHRM (or Project Poe). That, of course, also plays into the vision for Project Collins, which is all about searching across all the different domains, and using the knowledge aggregated across all of them, to enhance our search capability.

And we haven't even had time to talk about the event correlation stuff, personalization (filters, attenuators and amplifiers) or social-network-analysis, or prediction markets. Whew.

Anyway, that's the quick and dirty on how some of this fits together. As far as status: Project Shelley already exists and has a lot of features implement, as seen from the screen-shots. There is already Project Poe code as well, but it's much less feature complete. Some of the backend API code already exists, but it needs to be refactored, munged, moved around, etc.

Also, one last note... Nothing about this new direction or "roadmap" precludes the possibility of doing a "distribution" as we talked about before. If there's demand/need, the idea of bundling one or more, or all, of the OQ "pieces" with another "piece" like Roller or JavaBB or Alfresco, etc., could certainly be doable. It might even make sense to do some pre-configured bundles that make everything work together seamlessly. But my focus at the moment is really on the new bits. And more documentation (in the form of blog posts, at least) if you hadn't noticed. Don't forget to follow OpenQabal on Twitter.

Syndicated 2010-06-11 01:34:58 (Updated 2010-06-11 02:58:16) from openqabal

10 Jun 2010 (updated 11 Jun 2010 at 02:07 UTC) »

More on what Project Shelley does

In case last nights post didn't sate your appetite for Project Shelley information, here's another morsel. These screen shots illustrate the various ranking views that are available and demonstrate the RSS feed support.

This is the "hot" view. "Hotness" is a basically a metric that combines age and activity... if something is both new and has votes (or comments), it's hotter than if it's older and has fewer votes/comments. In other words, hotness decays with age (on a logarithmic basis) from the moment an entry is posted, and hotness rises in response to votes or comments.


Here we see an older entry has been voted up, and with the limited activity on the test system, that 1 vote was enough to move it into the top part of the Hot view.

This is the "Top" view, which simply shows the entries with the highest score, over all time. Since this is a test system, there aren't a lot of votes, and nothing has a score higher than 1, but you can see how all the 1's are at the top, above all the 0's. An entry starts with a score of 0, and up votes add 1 to the score, while downvotes subtract 1.:

Viewing one of the RSS feeds in RSSOwl:

More looking at RSS feeds.

Syndicated 2010-06-10 19:49:39 (Updated 2010-06-11 01:59:55) from openqabal

10 Jun 2010 (updated 10 Jun 2010 at 06:07 UTC) »

Introducting Project Shelley (with lots of screenshots!)

So, if you follow the OpenQabal Twitter stream or my personal twitter stream, you've heard some chatter about something called "Project Shelley." You may assume it had something to do with my recent post about restarting OpenQabal development, and you would be correct.

So, what is this "Project Shelley?" Well, for starters, Project Shelley is just a temporary code-name. The project will get a better name later (maybe we'll have a contest or something) but it works for now. Someday I'll explain how the name came about... for now I'll just mention that it's a nod to our friend Mary Shelley the author of Frankenstein.

Now with that out of the way, let's get down to the nitty-gritty. Project Shelley is the first bit to come out of rethinking the direction of OpenQabal. In the past, I was more focused on the broader social-networking aspect, thinking about decentralized, federated social-networks. But my personal interest was always more in enterprise applications of this kind of technology, and NOT in trying to build a "facebook killer" ala Diaspora or whoever.

So, while working for a semi well-known self-publishing company in the Raleigh area back in 2008, I started playing with the Open Reddit code, looking at how that type of technology could be a complement to some other things "behind the firewall" vis-a-vis knowledge management. Some people showed interest, but no real champion ever stepped up to push its use, then the economic collapse happened and side-projects became less emphasized, and then I left the company in 2009. But that experience planted a seed, and so the first new OpenQabal sub-project is - essentially - a very Reddit (or Digg if you prefer) like application that uses voting, tagging, sharing, filtering, etc. of articles and documents.

In it's present form it looks a lot like "just a Reddit clone" but the intention is to move beyond that, and I'll talk more about the more advanced features later (and to be fair, Shelley already has things that Reddit doesn't, but it also lacks things that Reddit does have) but tackling something that starts of as a "reddit like" gave me a chance to get started with a well known problem domain, and a chance to get something tangible out the door to start poking and prodding and playing with.

The current version has a lot of functionality, but will need a fair amount of "cleanup" work to be anywhere near production ready. The intent here really was to blast through as much as possible in a short period of time just to make this project feel real again.

So... with no further ado, here are screenshots and details about what this stuff does.


The front page, which is the default view of submitted links for the "default" channel. ("Channels" in Shelley lingo are like "sub reddits" in Reddit lingo)

The Login page, awaiting login:

The front page after login. Notice the new tab in the upper right hand corner. That takes a user to their user profile "stuff."

Sharing a link. It's kinda hard to make out, but the first entry in that field is a plain email address (it's mine, don't spam me, ok?) and the second is the same address but prefixed with "xmpp:" Yes, we support XMPP messaging.

As you can see here, I did receive an XMPP message about the shared link.

And now we're looking at tagging support. It's primitive right now, but this page shows a user a list of all the tags they've used:

And clicking on said tag displays a list of the posts you've tagged with that tag. What hasn't been done yet is any work on dealing with tags on a "global" basis (eg, can I see links that somebody else tagged with a given tag?)

Notice here that the Search dialog has been filled in with the string "information retrieval".

And here we get our search results:

Click the "comments" link for a given entry, and you come to this page, which lets a user view and add comments to an entry, and also shows other similar links. Right now the "recommended links" stuff is built just using the MoreLikeThis class from Lucene Contrib, but this is one of the areas that's going to get some interesting work in the future. In particular, I want to supplement this by using knowledge of tags and social-graph connections, to (hopefully) get better results than from just a strict content similarity score.

About to enter a comment:

With a new comment added:

The "User Profile" page. It's pretty spartan right now:

The "Saved entries" page. Things go here if you click the "save" button under an entry.

Adding a tag to an entry. The input field is hidden until you click the "tag" link under an entry.

Same thing, after putting some tag text in.

So there you go...a quick look at what Project Shelly is, currently. Coming later, more on what it will become in the future, and some info on the even more mysterious "Project Poe."

Syndicated 2010-06-10 04:46:32 (Updated 2010-06-10 05:17:08) from openqabal

13 May 2010 (updated 10 Jun 2010 at 04:15 UTC) »

Latest OpenQabal News

I'm sure most everyone is assuming that OpenQabal is dead. That would be a reasonable assumption given the lack of activity on my part for the past while. And it may even be true, for a certain value of "dead" and a certain value of "OpenQabal." Certainly since the last time I was heavily engaged in this project, the world around us has changed, and context and assumptions have changed. To everyone who wanted a decentralized, federated, open-source social-networking platform 2 years ago, I apologize... I let myself get distracted by other concerns and let this languish.

This is doubly sad given the recent burst of interest in that very space (see: Diaspora), after the Facebook's Gone Rogue bit; but things are what they are and there's no use crying over spilt milk.

So, the future then... As I said almost a year ago, the goal - in my mind - had already changed to being more about APIs for building "social applications." I believe that is still the case, and that is still the direction that I'll be pursuing with code I write in the near future. There is another subtle (or not so subtle) shift in focus as well... More than before, my intent is to focus on APIs, interfaces and tools for building socially aware applications in an enterprise context. As much as I hate buzzwords and labels, I'd say the goal is a Enterprise 2.0 suite.

As my current interests have moved heavily in the area of social ranking / voting, collaborative filtering, tagging, etc., I've started coding up a front-end to work with those techniques. As that evolves, some of the same APIs that were part of the OpenQabal vision 2 years ago will begin to evolve again. Some new things may appear, some old things may get dropped, as I explore some ideas about how all of this will work. At any rate, there *will* be new work coming out of the OpenQabal project, it just might not be what anybody thought it would be before.

Some things I know I'll be doing:

  1. The newest code I'm writing is in Groovy / Grails. Any user facing / UI stuff I work on in the near-term will probably be in Grails. Backend APIs will *probably* be plain Java, but don't rule out seeing Scala or Clojure show up somewhere. And depending on what kind of stuff we wind up needing, some C++, Python or Erlang or "other" could wind up in the mix as well.
  2. The new code will be going up on GitHub soon. Sometime after that, the existing code probably will as well (this is assuming that the existing code base is worth keeping around). I'm convinced that DVCSs are the way forward, but if a convenient way can be found to do it, we can mirror the code into java.net svn as well.
  3. I've been working with Hudson at my day job, and realize I like it better than Cruise-Control. I'll see about getting a VM setup somewhere, to host Hudson, Bugzilla, etc. Having that stuff running on an old box in my home office, sitting on a cable modem, is not ideal.
  4. Soon I'll see about getting an instance of Agilefant, Ice Scrum or XPlanner+ setup for managing the backlog
  5. At some point, I will take another stab at making an actual "road map" and some documentation that illustrates what it is that I'm on about here. I know I haven't done a very good job of articulating what the goal of this project was in the past (that's because *I* didn't really know either) and when I have articulated something, I've diverged from that more than once. Such are the perils of an exploratory approach. It will all become clearer eventually. No, really...

Of course all of this just reflects what *I* intend to do. I am not the project, per-se. If members of the community want to do different things, well.. standard OSS project terms apply: submit ideas to the list, and/or write code, and send it in. If it seems reasonable for inclusion, it could very well get included in the main code-base. And if I - in the "benevolent dictator" role - reject your pet idea, patch or suggestion, then you can fork the code. Once we go DVCS, that becomes even easier! Win for everybody.

So... OpenQabal is dead, long live OpenQabal!

Syndicated 2010-05-13 04:58:48 (Updated 2010-06-10 04:03:45) from openqabal

Is OpenQabal dead?

A few emails have shown up in my inbox over the past month or two, which I've been sadly negligent in replying too. If you sent one of them, I will reply to you directly in short order, but I did want to post a quick update here.

The question lately has been: "What's going on with OpenQabal? Is the project (dead|inactive|stalled|etc) or alive?"

The short answer is, the project IS still alive. For some value of "alive" anyway. This project has not received much attention from me lately, for a variety of reasons. Late last year I was involved in a political campaign, and that bled over into the holiday season, and I basically said "I'll work on it after the beginning of the year." Then January came and went, and my passion for hacking code in my every spare hour got pushed aside for some other interests for a while. But at no point did I decide to drop this project, and I fully expect some work to resume shortly.

Of course my life just keeps getting more complicated, as I'm currently in the middle of starting a new IT consulting and custom development shop: Fogbeam Labs. The bad news about that is, it takes time and cycles that I could be spending on OpenQabal. The good part is, if things work out well and I prove I can support myself consulting, I should (in theory) have more freedom to dedicate some time to OQ, unlike when I was doing the "40 hours a week for The Man" job.

One point of interest though: The focus of OpenQabal might change. After a very enlightening discussion a few weeks ago - with a friend who's opinions I respect greatly - I'm thinking that OpenQabal should probably drop most of the directly user facing "stuff" (for now at least) and focus on become a set of enabling APIs for building social applications. Of course those APIs were always part of the plan, but what I was trying to build was the APIs and a default suite of applications to use them, all at the same time. And since this is still mostly a one-person project, it's proving somewhat overwhelming.

So, the net-net is: OpenQabal is still alive, but the focus may be changing, and the roadmap along with it. And I can't make any specific promises at the moment about what will be done when. Unless you want to hire Fogbeam Labs to do some OpenQabal work, that is. :-)

Also, to the couple of folks who have asked "I want to use OpenQabal in a production site, can I?" The answer is a qualified "maybe." The blogging and forums stuff works fairly well, since it's all based on existing code from established projects. And SSO between the Roller App and the JavaBB App using CAS works fairly well. The last thing I was looking into before things got stalled out was how to make CAS into an OpenID Relying Party so one could sign in using OpenID and still get SSO between apps. That may get dropped for now, once work resumes however.

Syndicated 2009-06-02 09:15:13 from openqabal

Some Notes on Identity Management in OpenQabal

Some quick notes on what's going on with the Identity related stuff in OQ. Not necessarily in order:

1. One of the key concepts in OQ is the notion of federated identity... that is, an identity associated with one install of OQ should be "usable" from another install, where "usable" might mean "user at Site A can list user at Site B as a friend" or it might mean "user from Site B can login at Site A" or it might mean both, or it might mean neither.

2. That said, part of the idea of OQ, at any given install, is to be a sort of "social operating system" that enables application integration based on "social" technologies (the social graph, social ranking/voting, etc).

3. The use cases around (2) above are where some actual work has been done, and stuff works. We currently integrate apps running in the same domain by using JA-SIG CAS, with the authentication backend being our own simplistic IdM system (refered to as the IdentityEngine).

4. At one time it seemed to make sense to have the IdentityEngine project there to act as a mediator / bridge to possible other existing enterprise IdM systems. In retrospect, I'm no longer sure that adds much value.

5. It's entirely possible that a better approach would be to rip out the existing OQ IdM 'stuff' and just plop in Sun's OpenSSO / OpenFederation / OpenDS / OpenPTK stack. I'm doing some research on that point at this very moment, while also continuing to work on the existing OQ stuff based on CAS and the IdentityEngine.

6. Part of the point in rolling out this initial, somewhat naive, built-in IdM was to give us a test platform to experiment with issues around (1) above. To that point, I've been working on the possibility of making CAS an OpenID Relying Party, so that a user can login to OQ through OpenID and get the full CAS SSO experience.

7. Much research still needs to be done to sort out the best approach(es) to handling (1) above. OpenID may well be part of that solution, along with OAuth, but don't hold me to that. SAML, WS-Federation and some of the Liberty Alliance stuff may also be useful. Still need to do more research there.

8. What works now is this: SSO using CAS, successfully integrates Roller, JavaBB and our "User Console" app. Roller has a mechanism for fully externalizing user management, so it's not technically required to provision a new user into the Roller user table, although that code is still in place at the moment. JavaBB, on the other hand, does not (yet) have that ability, so when we create a new user, we have to populate the JavaBB user table.

9. One of the big things I want to determine is "what are the implicatiosn for user provisioning, when using an external authentication source like OpenID?" See comment (6) above. Getting that done will enable some experimenting around how this will work in principle.

10. User provisioning with SPML is also of interest.

11. The IdentityEngine exposes remote interfaces using EJB3 SLSBs, this is used by our implementation of the pluggable UserManager in Roller (there are use cases which require exposing APIs like "get list of users" and "get user by name" etc. In addition, we have to be able to authenticate "out of band" to enable authentication for the MetaweblogAPI interface (and presumably APP as well).

12. Another point that has not been addressed at all is two-factor (or multi-factor in general) authentication.

If anything else comes to mind, I'll make a follow up post. Please send other questions to the dev@ mailing-list...

Syndicated 2008-11-18 04:37:53 (Updated 2008-11-18 04:46:42) from openqabal

Finally back to work on OpenQabal

So after a rather long hiatus - due mainly to a political campaign - we're finally getting back to work on OpenQabal here. Of course our old road-map is now shot to heck, so it's going to take a while to sit down and put together a new plan and get some momentum going again. But I can say that a lot of work has been done the past week. Some of the needed work done includes: swapping all of the EJB2 SLSBs that were in place for remoting with EJB3 SLSB's instead, plugging in some of the JMS event messaging code that had been planned (and prototyped like 2 years ago!), renaming some modules, some serious refactoring in the IdentityEngine project, and - last but not least - getting the CruiseControl build-loop all sorted out so that everything builds continuously again.

So what's next in the short-term? Some or all of the following:

  • update CAS to the latest release
  • major improvements to the IdentityManager module
  • OpenID login support
  • support for provisioning users who are logging in using OpenID (or other 3rd party authentication services)

Getting those things knocked out will represent major progress. After that, it'll be time to start on the GraphEngine, PolicyEngine and ProfileEngine stuff for managing the social-graph, profiles and privacy / access control issues. After that, hopefully we can finally get started on the federated social-graph using FOAF... and after that, start on the SIOC stuff for distributed conversations. Oh and something needs to be done with the user-dashboard module to support plugging in applications. That may turn out to be using Portlets, or possibly the Google Gadgets / OpenSocial stuff.

After that... well, let's not look quite *that* far ahead just yet.

Syndicated 2008-11-04 04:22:17 (Updated 2008-11-04 04:26:52) from openqabal

So, what's going on with OpenQabal anyway?

Anyone following the OpenQabal project may be wondering: "What the heck is going on? No recent commits to SVN, no blog posts... what's up, is this dead or what?" Well, the answer is "No, the project is hardly dead." It has been on something of a hiatus however, as the main developer is currently running a campaign for NC Lieutenant Governor and has been a touch occupied the past month or two.

Rest assured though, that things will get back to normal. I'm hoping to get some work done this weekend, and once the election is over, the pace should pick up again. Unfortunately the old road-map is totally hosed now, at least as far as schedule goes, so that will need to be re-worked.

In the meantime, feel free to grab the code, experiment, research, submit proposals, patches, etc. And definitely email me with any questions or concerns.

Syndicated 2008-08-13 02:58:47 from openqabal

OpenQabal 0.0.3 available

OpenQabal 0.0.3 has been released and is now available from the SVN repo. As with previous releases, there are not yet pre-built binary releases available... you will have to check the code out from SVN and build it. Thankfully this process is now *much* easier as a result of massive work on the build system and the addition of scripts to automate most of the tedious stuff. You can now essentially build and install OpenQabal with 4 commands.

Changes in this release include:

  • New, more modular build system using Ivy for dependency management and incorporating useful tools such as FindBugs, JDepend, PMD, TestNG, Cobertura, etc.
  • New "User Dashboard" component that provides the main point of entry and ties the various components together visually
  • Concordantly with the introduction of the new User Dashboard, all of the old Sitemesh stuff has been ripped out
  • New configure and install scripts to automate most of the tedious parts of building and installing OpenQabal
  • Addition of many unit tests (we still don't have 100% test coverage, but progress has been made)
  • New "IdentityEngine" component introduced (more on this later)
  • Posting blogs using MetaWeblogAPI now works with the OpenQabal IdentityEngine component
  • Various minor bug fixes and tweaks

This release is a huge step in the direction of having a really usable system, but there is still a LOT of work to be done. Some things that had originally been planned for this release had to be deferred to 0.0.4, with OpenID support being the most notable such item. And of course there are still plenty of ideas on the roadmap that we haven't even gotten started on yet. But we're now in a lot better position to be able to start on some of the more interesting stuff, such as the "distributed conversation" support, tools for building a managing the "social graph," etc.

Look for a new post soon with more discussion of what the roadmap will look in the near future, as well as some discussion of some of the important changes that did make it into 0.0.3. And with any luck, the long-awaited "demo server" will be coming soon, as well as some screencasts and other ways of exploring OpenQabal.

Syndicated 2008-05-10 22:24:31 (Updated 2008-05-10 22:25:45) from openqabal

31 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!