Erlang, distribution & bspawner

Sriram Krishnan was enquiring about distributed programming tools and all things distributed. No doubt, the first thing that came into my mind was "erlang", and taking into consideration that he was inviting feedback & his recent work on cacheman and on server /app performace i thought it would be a good time to talk about bspawner - my own pet project among other things erly.

Load Balancing & Introducing bspawner

Erlang nodes can communicate over the network if they find each other. You can send a message to any process (differ from OS process) through its Pid – even if the process is on another node.

Although you don't have to know where processes are, it is still up to the erlang programmer to decide which node initially spawns a task on which node as there is no inherent way to make use of multiple nodes , even though they recognize each other & can message-pass between them.

bspawner is a project i've open sourced - that attempts to load-balance the task of spawning across multiple nodes . The steps involved can be isolated into a couple of distinct problems.

1. deciding which node needs to spawn a task
2. communicating across these nodes
   
3. maintaining a record of nodes, added /removed nodes ,etc

This project in its essence, deals with the first part and the implementation of the message passing begins with the "messenger.erl" sample program and modifiedto handle the intended message-passing, load-balancing and node-information

Cheers to the feedback, suggestions , comments, and further changes inspired by the growing involvement of the erlang community & encouragement from #erlang in particular.

However, since the project is still in its early stages - and although it was a fantastic learning experience - I learned that Erlang already had inbuilt load-balancing module that I quote:

pool can be used to run a set of Erlang nodes as a pool of computational processors. It is organized as a master and a set of slave nodes and includes the following features:

• The slave nodes send regular reports to the master about their current load.
• Queries can be sent to the master to determine which node will have the least load.

How cool can it get! Since it's even build in a master-slave basis - when one worker goes down, restart strategies can be configured courtesy of Erlang's OTP supervisor and behaviours. this basically allows you to horizontally scale and have distribute processing among box's. I'm even wondering to test this setup at hover.in on weekends by maybe doing some bizarrely wild clustered processing job like ...

• finding the largest prime number in Pune maybe ? :D
• any thing from distributing cron jobs to non-blocking or blocking I/O to ... free your mind!
  

Testing & Test suites

I'm not much into the whole test suite setup as yet, but there a good test suite that is widely adopted would be

• EUnit - a Lightweight Unit Testing Framework for Erlang
• An excellent error report evaluation, testing and debugging paper can also be found here (although a little old '92 , still gives valuable info on general practices)
  

Distributed Monitoring & Debugging

Regarding utilities for debugging & profiling ,etc apart from user-contributed packages, and there are loads of them on process-one, trapexit , jungerl, google code ,etc - I thought I would list few of the interesting utilties . ( see more on the left side of the documentation at erlang.org under Tool Applications )

• appmon — a graphical utility to observe and manipulate supervisiontrees.
• debugger — an Erlang source code debugger.
• erl_interface — a set of libraries for communicating with distributed Erlang nodes.
• et — the event tracer and tools to record and give a graphical presentation of event data.
• eva — the “event and alarm” handling application.
• observer — tools for tracing and observing the behaviour of a distributed system.
• os_mon — a tool to monitor resource usage in the external operating system.
• pman — a graphic tool to inspect the state of the system, at local or remote Erlang nodes.
• runtime_tools — miscellaneous small routines needed in the runtime system.
• toolbar — a graphical toolbar from which applications can be started.
• webtool — a system for managing web-based tools (such as inets)
• tools — a package of stand-alone applications for analysing and monitoring Erlang programs. This includes tools for profiling, coverage analysis, cross reference analysis etc.

These go hand in hand with other utilities like the distributed database also written in erlang called Mnesia, or other open-source erlang implementations of everything from bloom filters and decisssion trees , to bayeux protocol-comet servers and cron jobs.

Preserving State in Datastrutctures, Process's or Servers

Behaviours are templates or formalizations of common design patterns. The three inbuilt behaviours are gen_server (client-server paradigms), gen_event (event-driven paradigms) and gen_fsm ( finite state machine paradigms). In addition you can create your own behaviours and have modules implement them. (Emacs in erlang mode, even gives nice skeletons for all behaviours, common design patterns)

You can also have stateless or stateful process's which can run indefintely, or be blocking in nature or exit after a timeout. Which really opens up a whole lot of possibilities, but again - since the key is in message passing - regardless of the process's being on your local node or half way across the globe.

To think about ....

• load- balancing two or multiple yaws servers each of which are capable of handling 80,000 parallel connections each.
  
  
• Having erlang communicate via a port to say a python or perl for doing abstracting cross-language functionaliry ( Facebook Chat seems to be the largest xmpp-erlang based web application , and communicates with c++ for logging)
  

So there you have, I'm still a beginner in Erlang - but it's pretty evident to see the kind of flexibility to mix and match with the OS and other languages, the inherent encouragement towards distribute and concurrent programming across multiple nodes, the facets of a functional programming language , as well as an active and growing developer community.


Keep Clicking,
Bhasker V Kode

Syndicated 2008-05-24 08:06:00 (Updated 2008-05-24 10:17:12) from Bosky

Updates on Identities on the web

Some thoughts on identities and their related data , online accounts and related news over the past few weeks.

1. The Data Portability group amidst trying to get the big-wigs to come together and discuss how to share user data ,gets warned twice about infringement related cases in its own logo. But otherwise they've been pretty busy, with several announcements with the right intention, but left to too many influential market dynamics and vested invterests ( as illustrated in the points below)
   
   
2. OpenID, seems to keep itself out of trouble, and with a bunch of active adopters and hackers working together - there seems to be hope after all . Still don't know how long before the authentication itself can be made a asynchronous call, rather than the multi-step process right now. A ycombinator startup called clickPass seems to have some traction in its favour.
   
   
3. Google 's Orkut makes the news with the indian who was jailed for commenting on a particular politician ( im not touching this with a 10 foot pole! :D ) . Rumours from the local Pune papers, also suggest that girifthar'ed the wrong guy! ( that's right , newspapers in Pune report more rumours and Bollywood gossip that actual news)
   
   
4. And as if offering for your product for FREE was'nt good enough, early adopters are ploughing away in a bid to yank their data out as well . Thoughts on who owns your data within the facebook network, have brought in highly-engrossing discussions as well. Its your id, your profile, your pics, on their servers , powered by their cash and their vc's credibility. If the beacon showed what 3rd party developers could do with your data, the latest trend in 'yanking the data out' could give as unpredictable and "no-one-strategy-suits-anyone" results. ( It might also be interesting to check out the school talk / DHH talk on how to make money online - charging your customers works! )
   
   
5. What do you do when a startup's product becomes so popular that people get inspired enought to want to take their data elsewhere or distribute it as well! Thats been the case with Twitter and several comments on how to de-centralise twitter.
   
   
6. FaceBook and Google not hitting it off with the launch of Google FriendConnect, with Scoble 's insights into how Microsoft wants to keep the web closed, and how FriendFeed is trying to filter signal from the noise. Most users are used to the noise btw( when was the last time you went to cnn or bbc VS a twitter or a valleywag) . Facebook also makes news in the #erlang channels btw, for probably becoming possibly the largest erlang powered chat app. Will be interesting to see what technique they're using in handling unicode - something that hover.in is working to integrate as well .
   
   
7. Zoho announcing that you can now login to Zoho with your google or yahoo accounts. Very inspiring to see the pace at which Zoho takes ideas and implements them. Keep 'em coming!
   
   
   
8. And amidst all this , a friend pointed out to me that someone was posting comments under the alias kode, without their full names , giving several blogs and people the impression that it was infact me who was commenting, etc. What happened to the days when stalkers believed in linkback :D . Either ways, I hope its nothing as serious and on the contrary, it does bring up several unique advantages that a sezwho, a disqus , friendfeed or an openid-enabled commenting system can provide.
   
   But it helps to remember that denial is'nt just a river in Africa .
   
   8 )
   

Keep Clicking,
Bhasker V Kode


PS : Btw, if you're a fresher looking to join a startup in Pune , send in a mail to kode at hover dot in

Syndicated 2008-05-20 12:32:00 (Updated 2008-05-20 15:07:21) from Bosky

Developing developer platforms Part-1

It's no secret that I've always been an ardent admirer of companies that invest in impressive internal tools, developer resources, and internal things that make your developers go mmmh! Perhaps thats the secret to how they manage to release public api's and platforms for developers quickly in their product life cycle than companies with more mature products,but spend more time and effort trying to open up API's even for internal use in a plug-n-play, easy to lookup&hack fashion. Facebook & YUI will always be remembered for the part they've played in this very paradigm.



So, I thought it was time to kickstart my own soul searching, API intimidating, developer friendly environment right here at hover tech.. . What follows is a of the platform for the developers ( thats emp#1 srivatsan, and upcoming recruit thyaga- both theoretical CS & functional programming fanatics ) who one day will i hope influence the lives of several thousands of users, just as my own programming rockstar idols have done in the past.

Firstly, I jotted down a number of things that would seem fairly familiar in an environment where you track api changes, features, test them in real-time ,and discuss/update them. Things that came to mind in the first version of db_api were

• api Id
• developer
  
• module name
• path to source
• path to RESTfull call (so that development is independent of server side language/es)
  
• params, and type for the GET/POST
• return type
  
• description

An important element that allows to use this system outside the walls of your startup, is ofcourse the accessibility of the api information to either internal use or public api developers ,etc , which was added into db_api .

• (.. added to db_api)
  
• accessibility
  

I also thought a created_on, updated_on would help. To add more clarity regarding the params sent into the GET/POST call, i then normalized the prev fields into another table called db_api_params that would be of the form

• id
• api Id
• param name
  
• type
• description (tried to be synonymous to the javaDoc/jsDoc syntax)

Now today i might be in a position to look into each of my developer 's api declared into the system, but felt a more transparent yet "scope for delegation" strategy was best implemented by introducing a reviewing sytem. Hence the db_api_reviewinfo

• id
• api Id
• reviewed by
• last reviewed on
• comments
• isFlagged

Notice the luxury of comments, and a warning signal to quickly be able to search flagged or high priority api calls . Come to think of it, it could even be a rating based priority ,instead of a boolean. Also there may be several comments to the same api, review and moreover i wanted to have a "comment stream" , possibly displayed on one of the 21" monitors i'm planning to have at a centrally accessible area in the office. ;) So synchronous threading model for comments seemed warranted enough to spawn a new table db_api_comments, and re-edit the last two fields into:

• (...edited db_api_reviewinfo)
  
• comment Thread id
• flaggedRating

• (db_api_comments)
• api id
• title
• by
• content
• date
• comment type ( so u can comment on a developer | api call | review | module )
  

Since i was allowing multiple reviewers for an api call, considering that we do want to hire experts to help in multiple api and platform initiatives, i decided to have a similar developer db, for information on possibly mutiple developers working on the same api, as well to see a nice report on which developers are working on what , and hence db_api_devinfo and db_api_developers

• (db_api_devinfo)
  
• id
• api id
• developer id
• created on
• flagged by developer

• (db_developers)
• id
• name
• email
• hashed password
  
• permission level ( for admin/developer/ manager / testing positions /etc)

Since a greater part of the developers out there appreciate the usage of modules to wrapper function calls (as against global PHP calls or on the contrary the depreciation of global calls in actionscript 3.0 !) , we are going to make a simple table for module information called db_api_modules so that we could list api calls per module, or even drill down to public/internal api calls within a module)

• (db_api_modules)
• id
  
• name
• description
• created by
• flagged priority
• ( todoID perhaps implemented later ?...)

So there you have it, the meat of the monster in place, and once you hack up a login system in the server side language of your choice ( erlang & mnesia with traces of python and loads of javascript, in the case of hover.in ) - you should have the following very impressive reports at your disposal:

1. List of Developers , filter by permission or name
   Maybe you could integrate your bugzilla/ wiki/forum or mashup ur own flavoured list of developers and their information.
   
   
2. List of Reviewers , filter by name or flagged priority or module
   Nice for delegating and promoting a peer-review atmosphere at work where people are ranked not by experience , but by expertise on domain. Could even integrate HR data once u cross the 20-30 mark and gauge who to allocate where, etc
   
   
3. List of Modules, filtered by perhaps date or your own custom fields like "show-stopper " status
   
   and of course the big momma...
   
   
4. List of API Calls , filtered by
   by developer
   OR by flag priority
   OR by module (hack a drop down like facebook maybe )
   OR by accessibility ( useful to use same system to show for your platform developers as well as for internal developers) OR
   OR ask an intern to hack up a api search with autocomplete input like YUI.

It might not be perfect, but it's a start and a very satisfying night's work 8 ).

If you'd like to get the wicked details of we're going to implement or if you would like to implement it yourself along with the rest of us- Arun ,our CEO and an active blogger, Srivatsan and Thyaga (yup that's about it ) , let me know. Send a mail over to kode at hover dot in . We are just settling into the startup ecosystem in Pune, India and would like to hear from you as well as keep everyone in the loop of how it turns out in subsequent parts. And maybe will do a talk about the progress at the next barcamp pune as well .

Keep Clicking,

Bhasker V Kode
Founder & CTO - Hover Technologies ( prev. nick = Orange cube )
http://hover.in

Technorati Tags: programming, api, facebook, yui, bosky, hover

Syndicated 2008-04-01 21:23:00 (Updated 2008-04-01 23:46:32) from Bosky

it doesnt matter what language you write in, if you can write the future

I often hear about an upcoming platform going to revolutionize the next decade. Now I have no qualms with this , and infact biased towards big-thinkers . But i just think that on the contrary this is an inherent potential that any platform / technology can claim. How big was it designed to be when it was visualized ? Were they thnking big enough when the users were thought of ? I think these are interesting questions. Because for all practical reasons ,it is you who writes the future. Literally in your code , in your packaging, and its scaling before it goes out there for better or for worse. Who would you blame for X's success or failure? seems reasonable enough to claim X to please stand up. But looking deeper I think it's only fair to say that you may have limited yourself to build something just big enough for a million people to notice that it's doing well. Might sound cruel? but thats the way the way consumer web products space works.

Vision of a startup
What would you think of a 3 month old startup for sale on ebay ? TechCrunch would put you in "the deadpool". The founders could call it an exit. But the obvious reason in the difference in reactions is that the founders may have wanted just to hack a product in a couple of months, when they started off and get bought over . On the other hand, i completely understand why the founders over at Yahoo! were still not satisfied with 65$bn dollars. I can see why Zoho had'nt diluted much when they had numerous bids,much to the misery of the biggest players and investors in the world. Because that's not the future they want to write, and it simply is'nt time yet, they are'nt there yet, ... and miles to go before they sleep.

Writing the future
If you think you can "write the future", please let me know . I think i can too and maybe we could start a forum for us ;) . Maybe we could go back and think of all the things we are'nt doing. Maybe we could think of all the things that show you that there are so many things that we still don't know. But at least we know what we don't know , as cliched or odd as it sounds. We know that we suck at some things. And these are the kind of things that are needed to get you doing the things you want to do .

I call it a "reverse engineering process". It's a fantastic feeling when you can identify what you want in life. "What you want" ,not the how or the if's. But the long-term dream. Although that question itself more often has more to do with what you don't want . That's what incidently provoked my entrepreneurial ventures . I wanted to be responsible for my own failure /success by taking things in my own hand. And fortunately or unfortunately - recognition can only come in two scenarios. I can't remember what the second one is , but the first one is - as Paul graham said once -is a startup . You don't need to be a chemist to realise that the implications of your actions is best seen in a isolated environment. So that definitely convinced me that joining a 100,000 services company would be conflicting to my interest (no offense they are amazing wealth & welfare creators ) . Agreed that youngsters and executives from today's indian middle class can afford to take the risk. But they don't. I have nothing against people who don't want to write their future. But ignorance is something that something i love to fix. And I have no problem admitting to it- because i think it's a human right violation to not find out how ignorant you are. And i love finding out how ignorant i am. Just love it! Because then it's upto you to decide what ignorance you want to rectify , before moving onto the next ignorance to rectify,...and so on! And the earlier you start , the better. They should have dropped you in a pool,the day you were born . Hows that for a claim to evolve for survival. ( jk! )

Will every moment in your life bring you in a position where you are an expert. Nope, so like i said, the sooner you start the better, and believe me - everyone in the industry is watched. And people only just want you to keep doing what you have been doing, but only with fancier names. Now depending on your appetite for risk , you could either take the road that puts you in the situation at par of your natural growth-path - in a rat race just without recognition. OR purposely take yourself through the road less traveled ,where the solution is not known, but what is eminent is that you will not know what to do, and you'll have to find out and get through it & bultly put - be a creator of something out of nothing. And my definition of getting through it is simple. I dont care if that dream is getting into being a Google or getting into the payroll of the indian post office's . can it be done ? if it can be - then great . Take solace in the fact and accept that there will situations of embarrassing ignorance, and a cost to innovation which you will learn the hard way. But if nobody has done it - then that's a couple of patents right there ! You might not be able to build a real estate search engine for the moon yet, but you could improvise on the fact that you've come up with it . Depending on how many years the concept is realistically away "crossing the chasm" - you could probably patent, or discuss with your peers. Get recognition,possibly mocked at, but keep doing it until you are recognized by your peers as someone willing to try everything atleast twice, and keep innovating within the community. And if you can't even get to your peers who are looking out the slightest glimpse of innovation & market erach - then how the hell do you plan on reaching your consumers half way across the world!?.

Which bring me to the what you want to do again . Do you want to build years of expertise in a technology before launching your own company in a totally different technology ? ( Remember everyone in the industry is being watched - whether you notice it or not and the people who invest in you or your company - predominantly want you to continue doing what you have been doing). Tricky, tricky,tricky....

Does the language/medium/timer matter ?
On the other hand a hacker or an artist would know that it really does'nt matter what technology/medium you're working on. I'd love to could pick up ice-sculpting as a career if i thought that's the future i wanted to see myself in . What would linus torvalds, be doing if he was a kid today or if windows was 'nt closed source, would he have still hacked linux ? or build a great product that revolutionizes the next 50 years ? would the hexa hackers of yesteryears be working on hacking pixels on 384's or would they be building a holographic 3d second life ? would paul graham have manifested his ideas of "move to the web" in the form of a haptic interface to yahoo stores / amazon / ebay if he was hacking today?

I think that great innovators can innovate independent of their times, and the languages they use ,and were'nt in any more a gifted environment in their days , as the one we are in today, or anytime in future. This btw, i reiterate is my view on all those flame wars and rants on programming languages . I think it could be best explained in this great little xkcd strip that shows how different editors and talk about how old school they are, from the more prominent ones ,to and end up saying how "real hackers" write in assembly language or else straight away magnetic tape or conjure voltage from the sky and binary. hilarious, and it's the first thought that comes to my mind when i see a language /flame war or rant.


Reverse Engineering the future
The fact of the matter is that we dont look to reverse engineer our future, and that my friend is the only way to write the future. How much are you going to keep studying before you think you 're ready to change the world ? I may not know the answer to that, but i'm happy to question the same. To question what the optimal qualifications, or the geography is to do what you want to do in life. In all the articles ive read on startup founders, and innovators - i'm surprised that i've never read of anyone mentioning "reverse engineering" your future. If you wanted to see yourself as the official advisor to the queen of england on quantum computing , or lead the indian ultimate frisbee team, or the ambassador to bermuda in secondlife or trade shoes with peter norvig.

An example of a "reverse engineering your Future" log
-------------------------------------------
eg : Final Destination : ?
ideally , before I ...Final destination, ... i'd like to .... "leanr the intricacies of an IPO ", "learn to do a 'ballmer'"
ideally , before ...
ideally , before I ...0 , i'd like to "build an awesome product" and Y
ideally , before I ...Y , i'd like to "build a rockstar team" ...
ideally , before ...
ideally , before ...
Ideally , before i get/do N, i'd like to do get/do( N -1 )

Start off with a N = 5 goal , or a N = 2 goal . If i were to be scobleized next week , what would i have to liked to do this week, how many twitter message would i have to ping him with , etc ? If you wanted to get an grueling feature X done , what unknown thing would i have to read in the last 10 minutes before sleeping for the next month.

And since the final destination keeps escalating as you progress through the N things, based on your ability to execute what you want to do . The truth is that trying to start off with the Final Destination from day1 may not be quantifiable, but what is quantifiable is documenting if doing something is likely to get you closer to the eventual goal. Be it something trivial or absolutely drop-dead common sense. Thats the reason why It really struck me as to why people havent thought of how their personal identity on the internet would be jeopardized in step X of your reverse engineering lifecycle because of something that sounds silly today. That's also another reason why it's pretty darn easy for one risk taker- to find another hign-risk -high-gain motivated person. Perhaps thats how companies flock in the ecosystem themselves. Ofcourse, it all makes sense. How many people woudl want to let go of opportunities that they dont take today ,when they think about it 10 years down the line - you'd give anything to go back and do it right ? The thought of all the things i could do in that each day, in one week , or even one year keeps me up several nights. Honestly, i'm particularly happy with the way things have gone over the last 6 months with hover tech as well. It only has if anything only strengthened my conviction that you really can write the future if you want to.

The question for you/me really is and was always supposed to be , what kind of future do want to write...

Keep Clicking,
Bhasker V Kode

Blogged with Flock

Syndicated 2008-03-13 15:09:00 (Updated 2008-04-02 22:37:07) from Bosky

pipesCamp, unconference for yahoo! pipes enthusiasts [del.icio.us]

pipesCamp, is an unconference for yahoo! pipes enthusiasts, and will be held in Chennai, India for the first time on december 23, 2007 .

Syndicated 2007-12-19 03:08:07 from Bosky