Older blog entries for eMBee (starting at number 12)

watch for the fish shell

The following command is used to watch the progress of files being updated in a directory.

watch 'du -h *| tail -20 | cut -c -$(($COLUMNS-5))'

cut is used so we don't get linewraps in the output. And we need to subtract 5 from the width because there is a tabstop in there which cut counts as 1 instead of the width of the tabstop (which is 8 minus the width of the size column)

This command works fine as long as files are updated in alphabetical order, but when this is not the case we need to sort files by time.

We need to do something like this instead:

watch 'du -h $(ls -tr)| tail -20 | cut -c -$(($COLUMNS-5))'

Unfortunately this only works if the filenames don't contain spaces.

fish handles spaces in filenames just fine:

du -h (ls -tr)| tail -20 | cut -c -(math $COLUMNS - 5)

So instead of trying to find a solution for dealing with spaces in bash, lets just use a better shell, shall we?

However, watch insists on executing the command with sh -c, so we need to devise our own watch loop for fish instead. That's not really hard:

while true
  clear
  du -h (ls -tr) | tail -20 | cut -c -(math $COLUMNS - 5)
  sleep 2
end

Unfortunately using clear causes an annoying flicker, especially if the du command takes a bit longer.

ANSI escape-sequences help:

clear
while true
  echo \e\[H
  du -h (ls -tr) | tail -20 | cut -c -(math $COLUMNS - 5)
  sleep 2
end

This causes the cursor to be moved into the top-left corner without clearing the screen. Now we are almost there. Two problems still:
The tab character used to align the columns does not overwrite anything in its space. Likewise at the end of the line, if the newly written line is shorter then the remaining part of the old line is not cleared.

We can fix this with some sed trickery to clear the path:

clear
while true
    echo \e\[H
    du -h (ls -tr) | tail -20 | sed -e 's/^/'\e'\[K/' | cut -c -(math $COLUMNS - 2)
    sleep 2
end

The ANSI escape sequence ESC[K clears the line just before it is rewritten. This has almost the same effect as clearing the screen, but without the flicker because we only start clearing after du has done its work.
And because the escape sequence adds 3 characters to the output we need to adjust the width accordingly.

At the end we can also add a command to clear the rest of the screen.

Leaves one last issue: $COLUMNS doesn't get updated if the terminal is resized. Granted, it's a nit-pick really, because how often does one resize the terminal. But to make this command generally usable, let's fix this too:

clear
while true
    echo \e\[H
    du -h (ls -tr) | tail -20 | sed -e 's/^/'\e'\[K/' | cut -c -(math (tput cols) - 2)
    echo \e\[0J
    sleep 2
end

This is now pretty usable, so we'll leave it at that. There is still some room for improvement though. For example currently we are estimating that the size column is at least 2 characters wide so the tabstop adds at most 5 characters worth of space which cut does not count. Should it be less then we would get a linewrap again and if it is more then we get empty space at the end of the line. cut likely also has problems with multibyte unicode characters. This can probably be solved by switching the terminal in a no-wrap mode while the command is running, or finding a replacement for cut that handles these issues.

Syndicated 2013-09-19 05:44:43 (Updated 2013-09-19 07:11:05) from DevLog

basic http file server

I needed to quickly serve some files, and i didn't want to install a full webserver, knowing that this was only temporary. So instead here is a simple http file-server in Pike. I call it a file-server because it serves static files, and nothing else.

Based on this example for a simple web-server, it just takes a few additions to turn this into a file-server:

#!/usr/local/bin/pike

constant default_port = 8080;
constant my_version = "0.0";

Protocols.HTTP.Server.Port port;

string basedir = "/srv/www";

int main(int argc, array(string) argv)
{
  int my_port = default_port;
  if(argc>1) my_port=(int)argv[1];

  write("SocServe starting on port %d\n", my_port);

  port = Protocols.HTTP.Server.Port(handle_request, my_port);
  return -1;
}

void handle_request(Protocols.HTTP.Server.Request request)
{
  write(sprintf("got request: %O\n", request));

  mapping response = ([]);

  response->server="SocServe " + my_version;

  string target = Stdio.append_path(basedir, request->not_query);
  mixed tstat = file_stat(target);
  write("target: %O\n", tstat);

  if (tstat 
&
& tstat->isdir)
  {
    response->type = "text/html";
    response->error = 200;
    response->data = dirlist(target);
  } 
  else if (tstat 
&
& tstat->isreg)
  {
    response->type = MIME.ext_to_media_type((target/".")[-1]) || "octet/stream";
    response->error = 200;
    response->file = Stdio.File(target);
  }
  else
  {
    response->type = "text/html";
    response->error = 200;
    response->request = sprintf("%O", mkmapping(indices(request), values(request)));
    response->data = "<h1>SocServe " + my_version + "</h1>n<pre>"
    + response->request + "</pre>n";
  }

  request->response_and_finish(response);
}

string dirlist(string path)
{ 
  array dir = get_dir(path);
  return sprintf("%{<a href=\"%s\">%s</a><br>\n%}", ({ dir[*], dir[*] })); 
}

These are the changes: first combine the request path with our basedir which will reduce any embedded /../ to not go beyond that base. Then we check if it's a directory or a file. For a directory we make a simple listing, and for a file we find the mime-type and then just open the file and pass it to the request. This will cause the file to be served with non-blocking I/O, allowing you to handle multiple requests in parallel without blocking the server while a file is being downloaded.

That's all there is to it. Start it up, and it's ready to serve files.

Syndicated 2013-07-07 17:51:04 from DevLog

vim tips: capitalize the first word of every sentence

To solve this problem i found this useful solution, but discovered that it didn't cover all cases i had.

s/\v(\U)([^\.]*\.)/\u\1\L\2/g

To start with, \U does not mean every not-uppercase letter, but every character from the whole set that is not uppercase. So it includes spaces and everything else. This causes the expression to match " hello world!" if the sentence doesn't start at the beginning of the line, which is not quite what we want. To get every non-uppercase (that is lowercase) letter use \l. But even that does not really work, because it means that now it matches "Hello world." as "ello world.", and we get a transformation as "HEllo world!". Again, not what we are looking for. Unfortunately, until someone can suggest a method to skip already capitalized sentences we have to stick to \w.

Next, the expression only excludes periods, but not question-marks, exclamations or other sentence ending characters. We can extend this by simply including the respective characters: [^\.?!:;]. We also do not need to enforce the terminating character, we can simply drop that. What we really want to match is the beginning of the sentence, we don't care about the end.

Also, unless the text is in all uppercase, lower-casing the second group could be counter productive as it would affect upper-cased acronyms etc. that are already there.

Lastly, we want to capture sentences spanning multiple lines, lest every line gets matched as a separate sentence. This is achieved using \_

Putting it all together we get s/\v(\w)(\_[^\.?!:;]*)/\u\1\2/g

Syndicated 2013-03-06 06:09:00 (Updated 2013-03-06 06:12:07) from DevLog

(moved) vim tips: capitalize the first word of every sentence

this article moved

Syndicated 2013-03-05 09:52:53 (Updated 2013-03-06 07:12:02) from DevLog

javascript frameworks

Since i already found last week that creating views with jquery is not easy i decided to explore alternatives first. Already from early check-ins i received recommendations for angular.js, knockout.js and backbone.js.

Taking this frame work comparison as a guide i excluded backbone.js, and because a friend from the singapore hackerspace had also recommended angular.js i started with that.

Implementation was straight forward. What i really liked was that i didn't have to redo my datastructure that i had made up beforehand, but angular.js allowed me to access and traverse the data in the form that it was available. After some trials i even managed to build that 3 levels deep nested loop. (Iterate over columns, then projects and then task-types)

Then i briefly tried knockout.js but dropped it quickly because it would have caused me to put ko.observable() all over the place. angular.js handles this way better in my opinion. It simply observes the whole dataset and checks for updates.

Other contenders from the above comparison table were ember.js and batman.js. I had a brief look at ember.js and found that similar to knockout it would force me to write a new object-based datastructure, so i dropped the idea. batman.js is written in coffeescript, which is interesting but i want to stick to pure javascript for now, so i'll explore that later.

Syndicated 2012-08-22 15:06:23 from DevLog

learning phonegap cordova

This weeks goal was to learn how to work with PhoneGap/Cordova. Because of that i figured i could just link a Phonegap tutorial for this weeks goal. The report video is worth watching though. It covers the PhoneGap installation with eclipse, which went smoothly following the instructions on the website. Another Highlight this week was that we received 6 "awesome" votes, which is enough to get us over the minimum of 10 needed to access mentors.

Only two details are worth noting about the installation: The instructions call for installation of the android SDK, but it turned out that when installing the eclipse android plugin, the SDK was downloaded and installed right along with it. So you can skip the SDK step. The other point to note was that if you want to use the emulator you need to use the SDK manager to download a system image and then create a virtual device using the device manager. After that you are ready to roll.

Since the install was smooth and painless i moved on to the next step and quickly implemented the mockup in static html/css. i also tried to build the html from a function using jquery, but had no luck with that.

Why PhoneGap?

I have looked briefly at whats out there and somehow PhoneGap came to my attention. I watched some introductions and found that made a good case being all Free Software and supporting many platforms.

One great feature is that you can mix native and html views

Now they see it as allowing a transition from a native app to an html app, but i see also the reverse as ultimately we wanted to build native apps. This will allow us to do so without having to start from scratch. Instead we can focus on adding native code where it matters, where for example html is to limited to achieve a certain effect that we need or to slow. (It may turn out that we never need that, but it is good to have that option)

Also their stance of believing in the vision that all future mobile apps should be based on html and css and that they really want to make themselves irrelevant by expecting that eventually the browsers will pick up that functionality (so that all you need to deploy is html, css and js) is quite interesting. I am not necessarily agreeing with their idea of the future, but i certainly like their approach to it.

The phonegap guys even claim that they don't have competition.

Syndicated 2012-08-22 15:05:40 from DevLog

presenting nReduce at the beijing open party

Today's Open Party had a low turnout due to strong rain. Nonetheless it still had 11 talks in 4 tracks. Topics were Firefox OS, Geek Park, backbone.js conference report among others.

My presentation about nReduce gathered about half a dozend people. Given 3 competing talks and not more than 20-30 visitors overall this is actually quite a high turnout. They were also quite engaged and asked many questions. Maybe we'll get some more beijing teams when nReduce opens a new batch. One person was also interested in the email-task-manager.

nReduce is an online incubator. That is, the key benefits of joining an incubator: meeting other teams and sharing your experience, as well as talking to mentors, are brought online. Every team makes weekly goals and reports with short (less than 1 minute) videos. Teams group with other teams to work with and then discuss each others weekly progress. Our experience with nReduce was great from the start. Every week we received valuable feedback. A few times our check-in had the most discussion activity from all the teams we grouped with. Our progress is slow because we work on this only part-time for now, but we managed to come up with suitable goals that invited discussion.

We have now reached the halfway point and also reached a significant milestone for the email-task-manager. Until last week we were not clear whether we should start with a webinterface or focus on mobile clients first. The opinions were divided with me siding for the web. But one short comment by Hugh Mason clarified what we should do:

the folk who started with mobile first were forced to focus on the ONE really important feature of their product because of the small screen and the demand for instant gratification that mobile users show. That was really valuable. I'd always generally going for mobile first now if you plan on it being mobile ever

After reading that the goal was as clear as day. We absolutely have to start with mobile. There is no doubt about it now. We need to focus on the features that we can fit onto a mobile screen first, before expanding onto larger devices where we can use the space to add more features. It is also interesting to note that while we found related and competing products, none of those seem to focus on mobile.

Syndicated 2012-07-21 09:46:33 from DevLog

open graphics hardware is not dead

Phoronix declares the death of the open-source graphics card, but i disagree. Now, i am not really following the projects mentioned in the article (though i pay attention when i come across the topic), but i believe that open-source graphics is far from dead and it will come back. I am certain. We just need some patience.

In the long term, since the first computers, more and more sophisticated devices are being created by hobbyists. It is all a matter of the cost of components and the availability of affordable tools. For example, just recently oshpark.com was launched. A site that will print custom circuit boards at an affordable price. This site may not be the first of its kind, but it certainly is not the last. And just like the 3d printers, circuitboard printing is becoming accessible to more people, and it will be followed eventually by the ability to create small batches of affordable ICs, opening up room for more development and creation of open hardware.

Incidently, just today i attended a meeting here in beijing of openbrd. A group of young enthusiastic developers intent on making a completely open version of a miniature form factor arm based computer. (Like the raspberry pi or the beagleboard, but all specs, drivers, etc completely open and designed to allow anyone to build the device for themselves). So they are not designing the chips themselves, but another group like them, in a few years, may. They are meeting weekly in a restaurant in the northwest of beijing. About half a dozen people sat around a table, watching an unscripted presentation and discussing the topic. Today's topic wasn't about open hardware but introduced wget and curl and then by a second speaker, emacs. This group formed less than a year ago, but it is not the first group about open hardware in beijing. Before it, the beijing linux user group was leading an effort to create an open hardware quadcopter. But like the open-source graphics projects, that project eventually ceased (because some key members left beijing). Worldwide, there are many more groups like it. And one day, one of these groups will produce that open source graphics card.

As i said, i am not closely following the open-source graphics card efforts, and it may be a long time before these efforts gains some traction again, as well as the focus may shift to other components in a digital device. But the failure of existing projects to produce a result is a setback, it's not the end. just wait!

This topic is also discussed on LXer. In particular, khamul makes some compelling counter arguments regarding the prospects of accessible IC production.

Update:

A short while after this posting the developer behind Project VGA has written a lengthy email explaining why the open-source graphics card is no more. It appears to be that part of the problem was lack of openness from a company supporting the project. And then, just a few days later a university group announces that they are developing an open source graphics accelerator as part of their master thesis.

Syndicated 2012-06-25 07:40:32 from DevLog

can remote teamwork be productive?

Because i usually find potential clients through the internet i am occasionally confronted with the question on dealing with remote work. In this post i will try to relay my experience and hopefully answer some questions.

I will write about four types of collaboration:
Effective local collaboration and effective remote collaboration as well as complete lack of local collaboration and ineffective remote collaboration.
I have experienced all four.

The key to each one is communication. if communication does not happen, it will fail. Let me detail all the types above and explain what i learned from them. Starting with the complete lack of local collaboration.

The situation here was that i was hired to work on a specific project by myself. There was no team, just a manager who was unfortunately very busy. I was seated in a different room and left alone to do my work. The result was a complete desaster. I didn't have enough guidance, there was no oversight, no progress checking, nothing. This was probably as much my fault as the managers. As i said he was busy, and i should have been more upfront to ask.

I believe that if i had done this project from home, it would have been more effective for that situation. When you are in an office it is easy to assume that you are disciplined and working, and making progress. When you work at home your manager will be more concerned to find out if you are really working, and at least check in once in a while. Also working from home makes it more obvious that actual working time needs to be documented as opposed to mentally ticking a clock when you enter and leave the office.

Later, when i started to work from home my first client asked for hourly time sheets. At first i thought that it would be a drag, but then i realized that it was quite liberating because i could take a break whenever i wanted without feeling guilty and simply subtracted that time from the time sheet. It helped with the discipline :-)

The next type is ineffective remote collaboration. The situation here was that i joined a team that was working on a Free Software project at a university. I was the only outsider in that team. I was welcomed with open arms. The personal relationship with the other developers was great. When i visited them we had parties together, and when i joined them in the office we were working together.

But that office became the problem in the long run. As everyone else was there, most communication happened accross the desk face to face, so not only was i not involved in the communication or decisions, i often didn't even find out about plans, goals, opportunities etc...

Not enough effort was made to keep me in the loop and as a result i didn't really see much opportunity to contribute. Opportunity sure was there, and if i had just done what i wanted i could have ignored the communication problem and just send in code. But i like to work with a team, and not despite it.

Now let us look at the types where collaboration worked.

There are actually two types of local collaboration. One is the usual with regular meetings, everyone does their thing, if you have a question you go and ask, you help each other, everything as you'd expect.

The second is more interesting. What was unusual here was that we were 50 people in the company working on a large project with a client on the other side of the planet. All communication was done through a project internal irc server that we shared with the client. Once a day we had a project wide irc meeting where everyone, yes, all 50 odd people, would report on what they were working on. Within 5 minutes there was a flodd of issue tracker numbers and short comments, so everyone got to see what everyone else was doing. The meeting was in two parts: first a plain status report, and then a problem report. A bot was there to track that everyone had made their status report before problem reports were allowed.

The rest of the day, work was done in small teams. Each team had their own daily 5 minute stand up meeting, which was done face-to-face and extra meetings if they needed it.

The office was a large open space, and because of that talking was usually not convenient and if you had a question you'd ping your colleague on irc even if she was sitting right behind you.

Using irc in this manner had another sideeffect. It was simply less disruptive. If you were diving deep into a coding problem you could simply ignore the irc window and you wouldn't be disturbed. On the other hand if i didn't get a response even though i can see my collegue if i turn around i would know that he was busy and so i'd either work on something else else or find another person for my question.

(Here are some articles on why interruptions are bad: http://37signals.com/svn/archives2/the_science_of_interruptions.php http://www.paulgraham.com/makersschedule.html http://www.futurepundit.com/archives/004149.html)

This setup had another sideeffect. It didn't actually matter if you were in the office. If the situation called for it, you could work from home without affecting productivity. Also occasionally some of the senior developers were sent to the clients location, but they'd continue to work with their team from remote.

Because all communication was written, it could be logged, and so you could for example review the irc meeting even if you missed it.

Which brings us to the last type: fully working from remote. Above all else, consider the vast amount of Free Software developed by volunteers all across the world. You can pretty much guess that most of them work from remote. And successfully so. Myself, i am involved in several international Free Software projects and i have worked with clients in new zealand, singapore, germany and austria all from remote.

the most recent project for example was managed using trac with work shared among half a dozend people in almost as many locations in two timezones (6h difference).

we used trac, email, irc, jabber and voip to communicate.

working from remote forces you to write down more, which in turn gives more opportunity to think before expressing something, or to correct it.

There is an interesting paper on this topic called: "Can Absence Make a Team Grow Stronger?" (A. Majchrzak, A. Malhotra, J. Stamps and J. Lipnack), Harvard Business Review, May 2004, pp. 137-144 ( (there are more papers on this topic at http://public.kenan-flagler.unc.edu/faculty/malhotra/))

So it all comes down to the willingness of team-members to use suitable communication tools and make sure that remote team members are kept in the loop. positive sideeffects include better documentation or more doccumentation of informal decisions, a quieter office and more flexibility for the team members without negatively affecting the teams productivity.

This article in itself demonstrates one of the advantages of remote work.

It was written as a direct response to one potential client. now, imagine someone asking me this question in a personal meeting. I'd give them an answer, but it would be much shorter and not contain as much details or even reference links.

So not only did i answer the clients question, but i produced a general document that will answer this question for everyone else.

I wrote documentation.

Syndicated 2012-05-14 18:00:27 from DevLog

better email

LXer today posted an article that compares email to MPI.

What i find interesting about this article is not the realization that MPI is like email, but the reverse, that human collaboration using messages is like MPI. The suggestion to use SMTP as a transport for things much more complex than emails is something worth looking at. It relates to my belief that the instant messaging difference does not come from the new protocols like XMPP for jabber, but from the new client interface that facilitates short messages in a conversational style. SMTP is equally (if not more) capable of transporting these messages. Sure, SMTP is not designed for "instant" messaging. but todays use of SMTP where my local SMTP server talks directly to your SMTP server, makes email is just as instant.

We don't need another protocol, at best we need improved SMTP servers that handle multicast messaging. But the sender/recipient envelope is as simple and timeless as it gets. Any additional data can be added in headers, and here is where there is plenty of room for change.

We can make email messages more sophisticated, with additional headers indicating for example whether a message requires an action, or a response or whether it is merely informational. A reply message could then contain headers to show if the message is a response, or a followup question, or just an acknowledgement of the original message.

All we need for this is to add these features into email clients to allow more sophisticated messages to be sent and received.

Syndicated 2012-03-28 10:29:38 from DevLog