Older blog entries for eMBee (starting at number 15)

31 Dec 2014 (updated 31 Dec 2014 at 18:10 UTC) »

learning smalltalk with Google Code In

For years i have been meaning to learn smalltalk. my first exploration started about 10 years ago while teaching two children to make a game with squeak. Then i worked through a tutorial about making a simple game. Unfortunately it didn't capture my interest. So the my attempts to learn smalltalk were stalled as i searched for a project that i could do with it.

Why do i want to learn smalltalk? Because it is the first object-oriented language. Many of the OO concepts were invented in smalltalk. There is also the concept of working in an image that not only contains my code but also a full IDE which is used to update my code at runtime. Updating code at runtime is a concept that has been with me for more than 20 years now, ever since i started programming MUDs in LPC and writing modules for the Spinner/Roxen webserver in Pike. Pike allows recompiling classes at runtime. Any new instances will be made from the new class, while old instances remain as is. If the compilation fails, the class is not replaced and the old class continues to work. This way it is possible to make changes on a live server without restarting and disrupting ongoing requests. A decade later i discovered sTeam, the platform that also drives this very website. It takes this process even further: sTeam persists code and objects in a database. While in Roxen objects live as long as it takes to process a request, in sTeam objects are permanent, much like in a smalltalk image. sTeam then adds the capability to update live objects with new class implementations. The image concept of smalltalk is therefore already very familiar, and the major difference is smalltalk's GUI.

Recently a friend asked me what it would take to build a text search application for the Baha'i writings in chinese. There is one for english and other western languages, but not for chinese, and it does not run on mobile devices. It is also not Free Software, so i can't use it as a base to improve. But i didn't really want to take on a new project either so i just filed the idea for the time being.

One of my customers is managing access to several internal resources through htaccess and htpasswd. Because they have many interns who need to have access to some of these, and because they are now spread over multiple servers, it is becoming more and more cumbersome to manage them manually via these files. It also does not help that a salt module which we could use to help depends on apache helpers, which we can not install because apache conflicts with nginx which we are using. So i started exploring alternatives. One such alternative is a different way for nginx to verify access. It can make a request to an external service which then grants or rejects access depending on the resource and credentials. This could be implemented as a webservice with a webinterface to manage the users. I looked for some existing applications that would get me part of the way but i found nothing suitable.

Enter Google Code-In: FOSSASIA invited the BLUG to join them as mentors.

At first i put up tasks for the community-calendar project, but then i realized that this was an opportunity to explore new ideas. Figuring that teaching is the best way to learn i put up those project ideas as tasks for the students. I could ask students to learn and explore, and finally work on those projects. I would pick the technology and guide the students through a sequence of tasks to acquire the skills needed to implement the actual applications. This was my chance to get back into smalltalk. Since code-In targets middle and highschool students, it is quite unlikely that any of them already know smalltalk, or have even heared about it. so in a way this will introduce a few students to smalltalk. I picked pharo because i feel it is going in the right direction trying to improve itself and also adding things like commandline support.

The desktop application was straight-forward: find out how to embed text-documents in the image and make them searchable.

The web application took more exploration. I wanted to do it with a RESTful api and a javascript frontend. Again, the frontend was easy to define: create a user management interface. For the backend, the question was which webframework to use? AIDA/web has builtin user management and REST style url support by default. Seaside includes a REST module, but both are strong on generating html which i am not interested in. Then there is iliad, which appears more lightweight. Eventually i figured i could just let the students explore each, and i created a task for each tutorial that i could find:

(some of these i repeated because the student who did the them first time didn't pick up the follow-up tasks.)

Finally i discovered that Zinc, the HTTP server used by most frameworks is powerful enough to build a RESTful API without all the templating extras that the above frameworks provide. I also discovered teapot, a microframework that might be useful.

Once the students are familiar with the smalltalk environment, they can move on to the next steps:

Of course there are also tasks for the front-end

Related is also this task about a file editor, which i believe should make it easier to edit static assets like html and css pages from within the image:

Syndicated 2014-12-31 06:24:17 (Updated 2014-12-31 18:10:38) from DevLog

31 Dec 2014 »

leaning smalltalk through Google Code In

For years i have been meaning to learn smalltalk. my first exploration started about 10 years ago while teaching two children to make a game with squeak. Then i worked through a tutorial about making a simple game. Unfortunately it didn't capture my interest. So the my attempts to learn smalltalk were stalled as i searched for a project that i could do with it.

Why do i want to learn smalltalk? Because it is the first object-oriented language. Many of the OO concepts were invented in smalltalk. There is also the concept of working in an image that not only contains my code but also a full IDE which is used to update my code at runtime. Updating code at runtime is a concept that has been with me for more than 20 years now, ever since i started programming MUDs in LPC and writing modules for the Spinner/Roxen webserver.

Recently a friend asked me what it would take to build a text search application for the Baha'i writings in chinese. There is one for english and other western languages, but not for chinese, and it does not run on mobile devices. It is also not Free Software, so i can't use it as a base to improve. But i didn't really want to take on a new project either so i just filed the idea for the time being.

One of my customers is managing access to several internal resources through htaccess and htpasswd. Because they have many interns who need to have access to some of these, and because they are now spread over multiple servers, it is becoming more and more cumbersome to manage them manually via these files. It also does not help that a salt module which we could use to help depends on apache helpers, which we can not install because apache conflicts with nginx which we are using. So i started exploring alternatives. One such alternative is a different way for nginx to verify access. It can make a request to an external service which then grants or rejects access depending on the resource and credentials. This could be implemented as a webservice with a webinterface to manage the users. I looked for some existing applications that would get me part of the way but i found nothing suitable.

Enter Google Code-In: FOSSASIA invited the BLUG to join them as mentors.

At first i put up tasks for the community-calendar project, but then i realized that this was an opportunity to explore new ideas. Figuring that teaching is the best way to learn i put up those project ideas as tasks for the students. I could ask students to learn and explore, and finally work on those projects. I would pick the technology and guide the students through a sequence of tasks to acquire the skills needed to implement the actual applications. This was my chance to get back into smalltalk. Since code-In targets middle and highschool students, it is quite unlikely that any of them already know smalltalk, or have even heared about it. so in a way this will introduce a few students to smalltalk. I picked pharo because i feel it is going in the right direction trying to improve itself and also adding things like commandline support.

The desktop application was straight-forward: find out how to embed text-documents in the image and make them searchable.

The web application took more exploration. I wanted to do it with a RESTful api and a javascript frontend. Again, the frontend was easy to define: create a user management interface. For the backend, the question was which webframework to use? AIDA/web has builtin user management and REST style url support by default. Seaside includes a REST module, but both are strong on generating html which i am not interested in. Then there is iliad, which appears more lightweight. Eventually i figured i could just let the students explore each, and i created a task for each tutorial that i could find:

(some of these i repeated because the student who did the them first time didn't pick up the follow-up tasks.)

Finally i discovered that Zinc, the HTTP server used by most frameworks is powerful enough to build a RESTful API without all the templating extras that the above frameworks provide. I also discovered teapot, a microframework that might be useful.

Once the students are familiar with the smalltalk environment, they can move on to the next steps:

Of course there are also tasks for the front-end

Related is also this task about a file editor, which i believe should make it easier to edit static assets like html and css pages from within the image:

Syndicated 2014-12-31 06:11:01 from DevLog

31 Dec 2014 »

Google Code In with FOSSASIA

FOSSASIA is a mentor organization at Google Code-In, and the Beijing GNU/Linux User Group has been invited to join them as mentors.

Two of us joined and created tasks for our projects.

At first i created tasks for our community-calendar project, but then i took the opportunity to get students to work on new projects that i had been hoping to do. For a long time i wanted to learn smalltalk, but i lacked good project ideas. This changed recently, as a friend asked me about a text search application, and one of my customers needed a better solution than htpasswd to manage users for nginx. I decided that both could be done in smalltalk.

So i created tasks for three new projects: A text search application to run on the desktop, and one on mobile, and a user-management web-application. For the desktop and the web-application i stipulated pharo smalltalk as the implementation platform. For good measure i also threw in my idealist for sup, a reimplementation of the frontend for this weblog in angular.js, a t-shirt design for the BLUG, packaging pike, and exploration of the meetup.com api. I also proposed a new structure of the files for the fossasia api, and helped mentor a few tasks relating to getting chinese communities added to the api.

  • community-calendar (7 tasks)
  • desktop text search application (2 tasks)
  • mobile text search application (2 tasks)
  • user-management web-application (12 tasks)
  • sup ideas (more than 50 ideas, create tasks as needed)
  • fossasia (1 task)
  • sTeam weblog ui (1 task)
  • blug t-shirt (1 task)
  • amber (2 tasks)
  • file-editor (1 task)
  • packaging pike (1 task)
  • meetup.com api (3 tasks)
new tasks will be added as needed, when i get another idea for improvements on one of the projects, or if i feel a task needs to be redone.

Syndicated 2014-12-31 03:53:36 from DevLog

19 Sep 2013 (updated 19 Sep 2013 at 07:11 UTC) »

watch for the fish shell

The following command is used to watch the progress of files being updated in a directory.

watch 'du -h *| tail -20 | cut -c -$(($COLUMNS-5))'

cut is used so we don't get linewraps in the output. And we need to subtract 5 from the width because there is a tabstop in there which cut counts as 1 instead of the width of the tabstop (which is 8 minus the width of the size column)

This command works fine as long as files are updated in alphabetical order, but when this is not the case we need to sort files by time.

We need to do something like this instead:

watch 'du -h $(ls -tr)| tail -20 | cut -c -$(($COLUMNS-5))'

Unfortunately this only works if the filenames don't contain spaces.

fish handles spaces in filenames just fine:

du -h (ls -tr)| tail -20 | cut -c -(math $COLUMNS - 5)

So instead of trying to find a solution for dealing with spaces in bash, lets just use a better shell, shall we?

However, watch insists on executing the command with sh -c, so we need to devise our own watch loop for fish instead. That's not really hard: 

while true
  clear
  du -h (ls -tr) | tail -20 | cut -c -(math $COLUMNS - 5)
  sleep 2
end

Unfortunately using clear causes an annoying flicker, especially if the du command takes a bit longer.

ANSI escape-sequences help: 


clear
while true
  echo \e\[H
  du -h (ls -tr) | tail -20 | cut -c -(math $COLUMNS - 5)
  sleep 2
end

This causes the cursor to be moved into the top-left corner without clearing the screen. Now we are almost there. Two problems still:
The tab character used to align the columns does not overwrite anything in its space. Likewise at the end of the line, if the newly written line is shorter then the remaining part of the old line is not cleared.

We can fix this with some sed trickery to clear the path: 


clear
while true
    echo \e\[H
    du -h (ls -tr) | tail -20 | sed -e 's/^/'\e'\[K/' | cut -c -(math $COLUMNS - 2)
    sleep 2
end

The ANSI escape sequence ESC[K clears the line just before it is rewritten. This has almost the same effect as clearing the screen, but without the flicker because we only start clearing after du has done its work.
And because the escape sequence adds 3 characters to the output we need to adjust the width accordingly.

At the end we can also add a command to clear the rest of the screen.

Leaves one last issue: $COLUMNS doesn't get updated if the terminal is resized. Granted, it's a nit-pick really, because how often does one resize the terminal. But to make this command generally usable, let's fix this too: 


clear
while true
    echo \e\[H
    du -h (ls -tr) | tail -20 | sed -e 's/^/'\e'\[K/' | cut -c -(math (tput cols) - 2)
    echo \e\[0J
    sleep 2
end

This is now pretty usable, so we'll leave it at that. There is still some room for improvement though. For example currently we are estimating that the size column is at least 2 characters wide so the tabstop adds at most 5 characters worth of space which cut does not count. Should it be less then we would get a linewrap again and if it is more then we get empty space at the end of the line. cut likely also has problems with multibyte unicode characters. This can probably be solved by switching the terminal in a no-wrap mode while the command is running, or finding a replacement for cut that handles these issues.

Syndicated 2013-09-19 05:44:43 (Updated 2013-09-19 07:11:05) from DevLog

7 Jul 2013 »

basic http file server

I needed to quickly serve some files, and i didn't want to install a full webserver, knowing that this was only temporary. So instead here is a simple http file-server in Pike. I call it a file-server because it serves static files, and nothing else.

Based on this example for a simple web-server, it just takes a few additions to turn this into a file-server: 

#!/usr/local/bin/pike

constant default_port = 8080;
constant my_version = "0.0";

Protocols.HTTP.Server.Port port;

string basedir = "/srv/www";

int main(int argc, array(string) argv)
{
  int my_port = default_port;
  if(argc>1) my_port=(int)argv[1];

  write("SocServe starting on port %d\n", my_port);

  port = Protocols.HTTP.Server.Port(handle_request, my_port);
  return -1;
}

void handle_request(Protocols.HTTP.Server.Request request)
{
  write(sprintf("got request: %O\n", request));

  mapping response = ([]);

  response->server="SocServe " + my_version;

  string target = Stdio.append_path(basedir, request->not_query);
  mixed tstat = file_stat(target);
  write("target: %O\n", tstat);

  if (tstat 
&
& tstat->isdir)
  {
    response->type = "text/html";
    response->error = 200;
    response->data = dirlist(target);
  } 
  else if (tstat 
&
& tstat->isreg)
  {
    response->type = MIME.ext_to_media_type((target/".")[-1]) || "octet/stream";
    response->error = 200;
    response->file = Stdio.File(target);
  }
  else
  {
    response->type = "text/html";
    response->error = 200;
    response->request = sprintf("%O", mkmapping(indices(request), values(request)));
    response->data = "<h1>SocServe " + my_version + "</h1>n<pre>"
    + response->request + "</pre>n";
  }

  request->response_and_finish(response);
}

string dirlist(string path)
{ 
  array dir = get_dir(path);
  return sprintf("%{<a href=\"%s\">%s</a><br>\n%}", ({ dir[*], dir[*] })); 
}

These are the changes: first combine the request path with our basedir which will reduce any embedded /../ to not go beyond that base. Then we check if it's a directory or a file. For a directory we make a simple listing, and for a file we find the mime-type and then just open the file and pass it to the request. This will cause the file to be served with non-blocking I/O, allowing you to handle multiple requests in parallel without blocking the server while a file is being downloaded.

That's all there is to it. Start it up, and it's ready to serve files.

Syndicated 2013-07-07 17:51:04 from DevLog

6 Mar 2013 (updated 6 Mar 2013 at 06:12 UTC) »

vim tips: capitalize the first word of every sentence

To solve this problem i found this useful solution, but discovered that it didn't cover all cases i had.

s/\v(\U)([^\.]*\.)/\u\1\L\2/g

To start with, \U does not mean every not-uppercase letter, but every character from the whole set that is not uppercase. So it includes spaces and everything else. This causes the expression to match " hello world!" if the sentence doesn't start at the beginning of the line, which is not quite what we want. To get every non-uppercase (that is lowercase) letter use \l. But even that does not really work, because it means that now it matches "Hello world." as "ello world.", and we get a transformation as "HEllo world!". Again, not what we are looking for. Unfortunately, until someone can suggest a method to skip already capitalized sentences we have to stick to \w.

Next, the expression only excludes periods, but not question-marks, exclamations or other sentence ending characters. We can extend this by simply including the respective characters: [^\.?!:;]. We also do not need to enforce the terminating character, we can simply drop that. What we really want to match is the beginning of the sentence, we don't care about the end.

Also, unless the text is in all uppercase, lower-casing the second group could be counter productive as it would affect upper-cased acronyms etc. that are already there.

Lastly, we want to capture sentences spanning multiple lines, lest every line gets matched as a separate sentence. This is achieved using \_

Putting it all together we get s/\v(\w)(\_[^\.?!:;]*)/\u\1\2/g

Syndicated 2013-03-06 06:09:00 (Updated 2013-03-06 06:12:07) from DevLog

5 Mar 2013 (updated 6 Mar 2013 at 07:12 UTC) »

(moved) vim tips: capitalize the first word of every sentence

this article moved

Syndicated 2013-03-05 09:52:53 (Updated 2013-03-06 07:12:02) from DevLog

13 Dec 2012 »

javascript frameworks

Since i already found last week that creating views with jquery is not easy i decided to explore alternatives first. Already from early check-ins i received recommendations for angular.js, knockout.js and backbone.js.

Taking this frame work comparison as a guide i excluded backbone.js, and because a friend from the singapore hackerspace had also recommended angular.js i started with that.

Implementation was straight forward. What i really liked was that i didn't have to redo my datastructure that i had made up beforehand, but angular.js allowed me to access and traverse the data in the form that it was available. After some trials i even managed to build that 3 levels deep nested loop. (Iterate over columns, then projects and then task-types)

Then i briefly tried knockout.js but dropped it quickly because it would have caused me to put ko.observable() all over the place. angular.js handles this way better in my opinion. It simply observes the whole dataset and checks for updates.

Other contenders from the above comparison table were ember.js and batman.js. I had a brief look at ember.js and found that similar to knockout it would force me to write a new object-based datastructure, so i dropped the idea. batman.js is written in coffeescript, which is interesting but i want to stick to pure javascript for now, so i'll explore that later.

Syndicated 2012-08-22 15:06:23 from DevLog

13 Dec 2012 »

learning phonegap cordova

This weeks goal was to learn how to work with PhoneGap/Cordova. Because of that i figured i could just link a Phonegap tutorial for this weeks goal. The report video is worth watching though. It covers the PhoneGap installation with eclipse, which went smoothly following the instructions on the website. Another Highlight this week was that we received 6 "awesome" votes, which is enough to get us over the minimum of 10 needed to access mentors.

Only two details are worth noting about the installation: The instructions call for installation of the android SDK, but it turned out that when installing the eclipse android plugin, the SDK was downloaded and installed right along with it. So you can skip the SDK step. The other point to note was that if you want to use the emulator you need to use the SDK manager to download a system image and then create a virtual device using the device manager. After that you are ready to roll.

Since the install was smooth and painless i moved on to the next step and quickly implemented the mockup in static html/css. i also tried to build the html from a function using jquery, but had no luck with that.

Why PhoneGap?

I have looked briefly at whats out there and somehow PhoneGap came to my attention. I watched some introductions and found that made a good case being all Free Software and supporting many platforms.

One great feature is that you can mix native and html views

Now they see it as allowing a transition from a native app to an html app, but i see also the reverse as ultimately we wanted to build native apps. This will allow us to do so without having to start from scratch. Instead we can focus on adding native code where it matters, where for example html is to limited to achieve a certain effect that we need or to slow. (It may turn out that we never need that, but it is good to have that option)

Also their stance of believing in the vision that all future mobile apps should be based on html and css and that they really want to make themselves irrelevant by expecting that eventually the browsers will pick up that functionality (so that all you need to deploy is html, css and js) is quite interesting. I am not necessarily agreeing with their idea of the future, but i certainly like their approach to it.

The phonegap guys even claim that they don't have competition.

Syndicated 2012-08-22 15:05:40 from DevLog

13 Dec 2012 »

presenting nReduce at the beijing open party

Today's Open Party had a low turnout due to strong rain. Nonetheless it still had 11 talks in 4 tracks. Topics were Firefox OS, Geek Park, backbone.js conference report among others.

My presentation about nReduce gathered about half a dozend people. Given 3 competing talks and not more than 20-30 visitors overall this is actually quite a high turnout. They were also quite engaged and asked many questions. Maybe we'll get some more beijing teams when nReduce opens a new batch. One person was also interested in the email-task-manager.

nReduce is an online incubator. That is, the key benefits of joining an incubator: meeting other teams and sharing your experience, as well as talking to mentors, are brought online. Every team makes weekly goals and reports with short (less than 1 minute) videos. Teams group with other teams to work with and then discuss each others weekly progress. Our experience with nReduce was great from the start. Every week we received valuable feedback. A few times our check-in had the most discussion activity from all the teams we grouped with. Our progress is slow because we work on this only part-time for now, but we managed to come up with suitable goals that invited discussion.

We have now reached the halfway point and also reached a significant milestone for the email-task-manager. Until last week we were not clear whether we should start with a webinterface or focus on mobile clients first. The opinions were divided with me siding for the web. But one short comment by Hugh Mason clarified what we should do:

the folk who started with mobile first were forced to focus on the ONE really important feature of their product because of the small screen and the demand for instant gratification that mobile users show. That was really valuable. I'd always generally going for mobile first now if you plan on it being mobile ever
After reading that the goal was as clear as day. We absolutely have to start with mobile. There is no doubt about it now. We need to focus on the features that we can fit onto a mobile screen first, before expanding onto larger devices where we can use the space to add more features. It is also interesting to note that while we found related and competing products, none of those seem to focus on mobile.

Syndicated 2012-07-21 09:46:33 from DevLog

6 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!