# Older blog entries for shlomif (starting at number 689)

Sherlock Holmes about the Awk Programming Language

I enjoyed reading some of Sir Arthur Conan Doyle’s writings about the fictional detective Sherlock Holmes when I was younger, which were entertaining (although possibly distanced from the way actual crime investigation actually works), and interesting. I vividly recall one excerpt from the very first Sherlock Holmes story A Study in Scarlet:

His ignorance was as remarkable as his knowledge. Of contemporary literature, philosophy and politics he appeared to know next to nothing. Upon my quoting Thomas Carlyle, he enquired in the naivest way who he might be and what he had done. My surprise reached a climax, however, when I found incidentally that he was ignorant of the Copernican Theory and of the composition of the Solar System. That any civilized human being in this nineteenth century should not be aware that the earth travelled round the sun appeared to be to me such an extraordinary fact that I could hardly realize it.

"You appear to be astonished," he said, smiling at my expression of surprise. "Now that I do know it I shall do my best to forget it."

"To forget it!"

"You see," he explained, "I consider that a man's brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things, so that he has a difficulty in laying his hands upon it. Now the skilful workman is very careful indeed as to what he takes into his brain attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones."

"But the Solar System!" I protested.

"What the deuce is it to me?" he interrupted impatiently; "you say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work."

(Chapter 2 of A Study in Scarlet by Sir Arthur Conan Doyle, under the public domain in most countries.)

Conan Doyle was naturally exaggerating here in portraying the ideal of Sherlock Holmes (as few, if any, human beings can forget that the Earth revolves around the Sun), but the principle still stands: we need to make a conscious decision of how to manage our memory, because there is a limit to how many different aspects can put inside our resident memory, or otherwise we'll forget more important stuff.

So what does it has to do with the Awk programming language? Many decades after A Study in Scarlet, Eric S. Raymond had this to say in his book The Art of Unix Programming:

A case study of awk is included to point out that it is not a model for emulation; in fact, since 1990 it has largely fallen out of use. It has been superseded by new-school scripting languages—notably Perl, which was explicitly designed to be an awk killer. The reasons are worthy of examination, because they constitute a bit of a cautionary tale for minilanguage designers.

The awk language was originally designed to be a small, expressive special-purpose language for report generation. Unfortunately, it turns out to have been designed at a bad spot on the complexity-vs.-power curve. The action language is noncompact, but the pattern-driven framework it sits inside keeps it from being generally applicable — that's the worst of both worlds. And the new-school scripting languages can do anything awk can; their equivalent programs are usually just as readable, if not more so.

For a few years after the release of Perl in 1987, awk remained competitive simply because it had a smaller, faster implementation. But as the cost of compute cycles and memory dropped, the economic reasons for favoring a special-purpose language that was relatively thrifty with both lost their force. Programmers increasingly chose to do awklike things with Perl or (later) Python, rather than keep two different scripting languages in their heads.[90] By the year 2000 awk had become little more than a memory for most old-school Unix hackers, and not a particularly nostalgic one.

Falling costs have changed the tradeoffs in minilanguage design. Restricting your design's capabilities to buy compactness may still be a good idea, but doing so to economize on machine resources is a bad one. Machine resources get cheaper over time, but space in programmers' heads only gets more expensive. Modern minilanguages can either be general but noncompact, or specialized but very compact; specialized but noncompact simply won't compete.

(Emphasis mine.)

(Case Study: awk in minilanguages in The Art of Unix Programming by Eric Steven Raymond, text available under the Creative Commons Attribution-NonDerivatives licence, and hopefully quoted here (with attribution) under fair use auspices.)

Back in 1996, after I first learned Perl and started working on Unix, I asked one of my co-workers if I should learn Awk and he said “Forget it! Perl can do everything Awk does and more, and is a much better language”. (That was some time before other of the so-called “scripting languages” that gained popularity after Perl, were notable and/or mature enough to be considered by most sane people.) While I was not entirely convinced, and also ended up using GNU awk (gawk) to write a small text processing script for Microsoft Windows at one point (because I preferred not to investigate how to make the perl executable more self-contained). For a while, I felt guilty about not being fluent in Awk, until I read what Raymond said, when I realised why he, my co-worker, and Conan Doyle’s words of Sherlock Holmes, have been right all along.

Some people take more radical approaches to managing their memory. A friend of mine mostly converted from Perl 4 to Python, which due to syntactic limitations is not very suitable for one-off scripts on the command line, as his scripting language. He told me that whenever he has to perform a text processing or a similar task from the command-line, he edits a new file in his text editor, which also gives him some boilerplate to write his script, edits it, saves the file, and finally calls it from the command line. If I did something like that whenever I wrote something on the command line, I would quickly become extremely unhappy, but I suppose it is a useful approach if one is most comfortable with Python for such tasks.

Awk is not completely useless, and may sometimes need to be used for extra portability when old, antiquated or kept-minimal-on-purpose Unix systems, are involved, and is of important historical significance. However, in my case, I don't see a point in knowing it. If I need to learn it, I learn it enough to write what I need, and, like Sherlock Holmes, try to quickly forget it because I know I won't readily need this knowledge.

Naturally, this extends to other fields aside from computing. One of my pupils for private lessons testified that he had photographic memory, and for the history matriculation examination, he memorised the entire books, and during the exam wrote an paraphrased answer based on his memory, and as a result, got a very high grade, and eventually forgot most of it. Similarly, my sister, who now studies medicine, told me that she and her fellow students often memorised a lot of material in preparation for the examinations, only to forget it and then learn it again for a different examination, that also covered the same material. This makes me question the effectiveness of the methodology behind medical education, but still reinforces the original point.

## The Other Side of the Coin

On the other hand, your knowledge and understanding of it should not be too specialised either, because one can infer many parallels from different fields of knowledge, and reach conclusions, because all knowledge is contiguous. By learning a little of everything and anything, you can often handle situations and have clearer thinking and greater creativity.

In my screenplay Star Trek: “We, the Living Dead”, I describe an optimal situation of this in the “Planet of the Hebrews” where scholars each take different units of study and learn any that they want, and eventually are judged based on the number of units they learned, and the amount of useful contributions they have done. And you still shouldn't rule out that someone less experienced, younger, or less qualified, than you will be able to do as well, or even better than you (see what Paul Graham wrote about “amateurs” in “What business can learn from open source?”).

This document is Copyright by Shlomi Fish, 2006, and is available under the terms of the Creative Commons Attribution License 3.0 Unported (or at your option any later version of that licence).

## Meta

Despite enjoying captioned images for a long time, I am late to the game of creating them. You may know them as lolcats, and they are also sometimes called “memes”, although the term “meme” is used for any unit of thought and more than just. However, I recently created three of them using Wikimedia Commons, or Google Image Search, as well as GIMP and Inkscape, and realised it is incredibly easy to do. I now truly understand why their low barrier to entry - almost everyone can take a photo of a cat or whatever and caption it - makes them so subversive, and why the Cheezburger network is being blocked by both Iran and China.

I have done some work on Star Trek: “We, the Living Dead” (which is now close to being in a mostly usable state) and “Selina Mandrake - The Slayer”, which combines a Buffy the Vampire Slayer parody and tribute (with a conscious and constant referencing of the original show) with many more elements. An Indian software developer, with whom I talked on the Internet, and who did not watch Buffy, said it was still very funny, so there may be hope for me yet.

Cheers, all.

Syndicated 2013-02-14 17:47:19 from shlomif

Joke: “NSA Facts”

Yesterday, I had the idea of creating some “facts” about the NSA - the United States National Security Agency, similar to the Chuck Norris Facts meme. And I got some pretty good factoids so far, which I will share here:

• The NSA doesn't publish. They perish.
• The NSA employs the largest number of mathematicians with Ph.D. And the most stupid and incompetent ones.
• The NSA has a patent for an efficient process for collecting a lot of information and doing nothing with it.
• The Bajoran scholars have positively identified Benjamin Sisko as The Emissary. They also positively identified the NSA headquarters as The Dungeon.
• One of Buffy the Vampire Slayer’s episodes took place in the NSA headquarters, but had to be destroyed, because all of the test audience had uncontrollable panic attacks.
• The NSA knows what you did last summer. But no one, in the NSA or outside it, knows why they should.
• The more the NSA think, the less they want to be able to think. So they think less and less.

Hope you enjoyed it and naturally - “ha ha, only serious!”. Now the ball is in your court - can you think of more? Put them in the comments below.

Happy NSA bashing!

Syndicated 2013-02-06 06:28:51 from shlomif

Invitation for the Israeli Perl Workshop 2013 - 25 February 2013

## Israeli Perl Workshop 2013 - 25 February 2013

(The Hebrew text will be followed by an English one).

סדנת הפרל הישראלית לשנת 2013 תתקיים ב-25 בפברואר, 2013. זהו כנס יומי שיתקיים במכללת שנקר ברמת-גן. לעיונכם נתונה רשימת המצגות שתינתנה בסדנה, בה יהיו שני מסלולים.

הפעם יידרש תשלום עבור הכנס, כדי לכסות את ההוצאות, ויש צורך להירשם, אז הקדימו להירשם היום. בנוסף, אם אתם, או חברה מסחרית אחרת שאתם מכירים מעוניינים לתת חסות לסדנה, אנא פנו למארגניה.

הפעם סימן הכנס הוא "פרל מעשית" (Perl in Practice) ואנו מקווים שרוב ההרצאות בכנס תהיינה על יישומים מעשיים שנכתבו או יכתבו בשפת פרל.

פרל הינה שפת תכנות דינמית וחזקה שנמצאת תחת פיתוח פעיל, ושיש לה מאגר הרחבות פעיל ופורה בשם CPAN (Comprehensive Perl Archive Network). דף הבית של שוחרי הפרל הישראליים מכיל מידע נוסף וקישורים נוספים לגבי הפעילות של חובבי השפה בישראל.

הפעם יתארחו בסדנה מספר אורחים מחו"ל וביניהם לארי וול, הידוע כיוצר שפת פרל (ומספר מיזמים חשובים מוקדמים וביניהם הגרסה הראשונה של תוכנת ההטלאה patch) שגם יציג בכנס; אליו תתלווה רעייתו גלוריה. בהמשך נעדכן את שמותיהם של האורחים הנוספים.

### English Version

The Israeli Perl Workshop of 2013 will take place at 25 February, 2013. It will be a daily conference which will take place in Shenkar College in Ramat Gan. One can view the list of talks that will be given in the conference, where there are going to be two tracks.

‎This time the conference will cost some money in order to cover the costs and, so there is a need to register. Therefore, please register as soon as possible. In addition, if you, or a different company that you know, are willing to sponsor the workshop, please contact the organisers.

Perl is a dynamic and powerful programming language which is under active development, and which has an active and comprehensive extension repository called CPAN (Comprehensive Perl Archive Network). The home page of the Israeli Perl Mongers contains other information and more links about the activity of Perl's Israeli enthusiasts.

This year's conference will sport some guests from abroad, including Larry Wall, who is known as the creator of the Perl programming language (and some earlier software projects such as the original patch program for UNIX), and who will give a talk at the conference; he will be accompanied by his wife, Gloria. We will publish the names of the other guest speakers as we learn about them.

Everyone are welcome to attend.

Syndicated 2013-01-24 18:47:03 from shlomif

Announcing the Vim Beginners’ Site

I am glad to announce that I, along with some help from some other people, have set up the Vim Beginners’s Site - http://vim.begin-site.org/ (or Vim-Begin for short). It aims to be a centrally managed, yet fully open content/open source site, for concentrating the Internet’s best material for learning about the Vim text editor and expanding one’s knowledge. The site was inspired by the Perl Beginners’ site (or “Perl-Begin” for short), and I set up the domains begin-site.org (and begin-site.com as a future redirect) to concentrate other similar high-quality sites introducing people to various technologies and topics. So if you want python.begin-site.org, emacs.begin-site.org, linux.begin-site.org, dotnet.begin-site.org, cooking.begin-site.org etc. then contact me and I’ll see what I can do.

The site is incomplete, and there's still a lot to do, but we have a Bitbucket mercurial repository, an issue tracker there, a TODO list, and we accept pull requests. The text for the site’s pages is under the Creative Commons Attribution 3.0 licence (unless noted otherwise) and whatever original source code is found there is under the MIT/X11 licence, and both were chosen to allow for maximal reuse. Nevertheless, we may mirror, restore, or link to, resources under different licences.

Cheers, happy new year, and happy Vimming!

Syndicated 2012-12-30 12:48:39 from shlomif

What’s New on Shlomi Fish’s Homepage

Here are the recent updates for Shlomi Fish’s Homepage. Most of the work this time was done on the look, feel, and infrastructure of the site, such as the navigation menus, but there is still some new (and hopefully interesting) content. So without further ado, here is what is new:

1. The main navigation menu to the right now comprises of most of the pages that were navigable and previously were present only in the section navigation menus. Since its HTML markup was quite excessive, I decided to load most of the content using an AJAX (= “Asyncrhonous JavaScript And XML” or “XMLHTTPRequest”) fetch of a JSON document, while keeping a smaller subset still usable as plain HTML for browsers with JavaScript disabled and for search engines and other web user agents.

I made sure that the expansion state of the navigation menu is preserved between the pages. Moreover, the much maligned section navigation menus are now hidden by default, but can be enabled using the button and should remember their state between pages.

2. There is a new HTML Tutorial in Hebrew under work. Currently, there is only one section, and some aspects of it are lacking.

3. There is a new list of text processing tools and a list of networking clients in the software resources section.

5. Yaakov: I LOVE YOU ALL WITH A GREAT HUGE LOVE
6. rindolf: Yaakov: HOW MUCH DOES YOUR GREAT HUGE LOVE COST?
7. Yaakov: It's on offer half price today.
8. rindolf: Yaakov: I WILL OFFER YOU 200,000 VIRTUAL DOLLARS FOR YOUR GREAT HUGE LOVE.
9. Yaakov: The regular price is free of charge.
10. rindolf: Yaakov: oh nice.
11. * rindolf buys Yaakov's GREAT HUGE LOVE.
12. rindolf: I LOVE YOU ALL WITH YAAKOV'S GREAT HUGE LOVE.
13. The third version of my essay “Open source, Free software, and Other Beasts” is now live. It was converted from DocBook 4 to DocBook 5 and greatly enhanced.

14. Also new is the countdown program which is similar to the UNIX sleep command, only displays the amount of time remaining.

15. There’s a new and open source solver for the so-called “Binary Puzzles”. It is still incomplete.

16. There is a new geek song parody (under work) titled “Can I SCO Now?” sang to the music of Jennifer Love Hewitt’s “Can I Go Now?”:

Can I SCO now?
Sue who you wanna sue,
it doesn't matter anyhoo,
it's time to litigate.
Can I SCO Now?
Say what you wanna say,
we don't care anyway,
we're going to inflate.

Can I, can I, SCO now….

17. I now mirror The Fountainhead’s parody “The Fountainhead Starring Skull Force”, whose original link went offline, but which I was able to find a mirror of. Enjoy.

18. Support for the MathJax JavaScript library was added to the pages in the MathVentures section enabling most modern JavaScript-enabled browsers to pretty view the mathematical formulae there.

19. There are some new additions to the original Aphorisms and Quotes page:

The English Wikipedia: now you don’t see it - now you do.

20. I restored the display of the images in my “There are Too Many Ways to Do it” lightning talk.

Syndicated 2012-12-28 07:50:29 from shlomif

Tech Tip: Overriding the Audio Track of a Video using ffmpeg

In this tip I will cover how to use ffmpeg to override the audio track of a video from a different audio track (such as the one in a WAV, an OGG or an MP3 file). To do that use the following recipe (based on this out-of-date blog post and some help from ubitux on #ffmpeg, with a lot of trial and error):

ffmpeg -i in_audio.wav -i in_video.ogv -map 0:0 -map 1:1 -shortest \
-c:a libvorbis -q:a 7 -c:v copy output.ogv


Some of these files are not needed if you're not dealing with Ogg Video files. -shortest is useful for making sure that the length of the output is trimmed to that of the shortest input.

I ended up using this command for the latest screencast I prepared. Cheers, and happy holidays.

Syndicated 2012-12-10 16:59:01 from shlomif

Freecell Solver 3.16.0 was Released + New Screencast

Freecell Solver version 3.16.0, has been released. Freecell Solver 3.16.0 is available in the form of a source archive, from the download page. Freecell Solver is an open source framework (library and some command line applications), for automatically solving several variants of card Solitaire / Patience games, including Freecell.

This new release took about a month to prepare, and also contains less profound changes than the previous release, but still quite a lot to look for. It features the -l micro-finance-improved and -l qualified-seed presets, which improve upon the average solution length (at the expense of speed). The --flares-choice flag was added and one can specify --flares-choice fcpro to choose based on the lenght of the Freecell Pro moves. Another additional flag is --flares-iters-factor which multiplies the time the flares run at by a factor for a speed/solution-length trade-off.

Finally, the inidividual flares are now recycled and their memory is reused when they are no longer needed which yields a RAM optimisation in certain cases.

In addition to all that, a new screencast of PySolFC solving Freecell using Freecell Solver is now available on YouTube. Make sure you expand the details for all the credits of the software and other material that was used for its creation.

Enjoy and happy holidays.

Syndicated 2012-12-10 13:19:00 from shlomif

Why I Don’t Trust Non-Open Source Software

Today I’m going to explain in a dedicated entry, why I do not trust programs that are not Free and open-source software (FOSS) and instead are more restricted. I have previously told that story as part of a different essay, but it was easy to miss.

In any case, the BitKeeper version control system is now quite obscure, due to the advent of quality distributed open-source systems such as Git and Mercurial, but some years ago it was used by many developers of the Linux kernel, and the demise of its gratis version was in fact the impetus for the creation of Git, and later Mercurial. Some time, before Linus Torvalds adopted it, I ran into a limitation of CVS, an open-source version control system, that was then popular, and was looking for an alternative, and after reading an out-of-date article about BitKeeper (which said its source was available under a mostly open source licence), decided to use it and its bkbits.net service.

At first, I was quite happy using it for some of my projects, but then I posted a question to the mailing list, asking where I can find the source, which was implied to be available on the BitKeeper site. Larry McVoy (BitKeeper's main creator and maintainer) answered by saying that they decided for removing the source code, because some users modified it to remove the restrictions, and allowed them to abuse the licensing of the gratis version and prevent them from paying for the commercial version. He also noted that availability of source, meant that, in practice, your software was Public Domain, and that they provide the sources for people they can trust, in private.

At that point, I figured out that I don't have an immediate need for the source, and that perhaps in the future, I can win BitMover (= BitKeeper’s parent company) trust and gain access to it. So I continued using BitKeeper.

That changed, however, when Mr. McVoy announced a licence change to BitKeeper (while requiring all users of the gratis version to upgrade) that I found unacceptable, and caused me to seek a different alternative. This caused an unpleasant exchange between me and the BitKeeper developers, and made me lose some of the repositories I hosted on bkbits.net.

From that moment on, I realised that I cannot really trust non-open-source software, because even if I am allowed to continue to use its previous version after a licence change, then it may accumulate bugs or stop being runnable on my systems, or stop being supported, and I cannot risk it. To quote Richard Stallman: “Every non-free [= non-FOSS] program has a lord, a master — and if you use the program, he is your master”.

The end of the BitKeeper story, was that after evaluating a few open-source alternatives, I settled on using the open-source Subversion, and later on also started using Mercurial and Git. Furthermore, from then on, I often refused to look at and evaluate proprietary programs. Lately, many open-source developers have been infatuated with Sublime Text, but I am not willing to even try it, because it is not open-source, so I will never have to depend on it.

Despite all that, I still license my original software under non-Copyleft licences, because the GPLv2 and the GPLv3 are incompatible, both with one another, and with many other open-source licences, and because I know of at least three different interpretations to the GPL ( GNU’s one in the GPL FAQ, the Linux kernel's one, and the draconian Nmap interpretation), and because I want my code to be of the maximal available use without the need to consult a lawyer, and because I don't want to be worried about how it will possibly be abused, when I don't care if it will. I'm still using GPLed software, in the hope that I won't get sued.

To sum up, I do not wish to rely on non-FOSS, because it may mean these software applications later becomes unavailable to me, in a similar manner to what was the case with BitKeeper. I hope you can relate to that, and if not, you may likely run into a similar situation in the future, as well also the case for the Linux kernel project, with the demise of the gratis BitKeeper altogether.

Syndicated 2012-11-19 10:34:13 from shlomif

Freecell Solver 3.14.1 was Released

Freecell Solver version 3.14.0, shortly followed by Freecell Solver 3.14.1, which fixed a build problem on Microsoft Windows, have been released. Freecell Solver 3.14.1 is available in the form of a source archive, and a Win32 self-installing executable, from the download page. Freecell Solver is an open source framework (library and some command line applications), for automatically solving several variants of card Solitaire / Patience games, including Freecell.

This release features several new features: we added a 6th Best-First-Search weight of the inverse of the number of cards not above parents, which has proven useful. We now also allow test groups inside the --tests-order and --depth-tests-order with the random-dfs scan to be ordered using the function =asw(…) based on those BeFS weights.

As a result, the preset -l amateur-star or -l as for short has been added, that uses that and is our fastest preset yet. Another new preset is -l micro-finance (or -l mf for short), which yields especially short solutions.

There are also some not-as-major features: input boards can now contain a leading colon (":") at the beginning of the line of columns, so they can be copy-and-pasted directly from the output of fc-solve with the -p flag. The dbm_fc_solver and depth_dbm_fc_solver now store the positions more compactly (which aided in researching two-freecell deals), and the core libfreecell-solver code was made more 64-bit enabled and many of the limits were converted to 64-bit friendly ones.

A final note: we have dropped support for building Freecell Solver with Microsoft Visual C++ and other non-GCC compatible compilers, which do not support the newer C standards, and other useful features of GCC. Building Freecell Solver with GCC, clang and other compatible compilers will be continued to be supported on Microsoft Windows as well as on Linux and other UNIX-like systems.

Enjoy!

Syndicated 2012-11-04 16:22:00 from shlomif

Why Java Sucks for Cross-Platform GUI

When the Java programming language first became publicised, it was heralded as the dawn of a new age of writing cross-platform programs. “Write once, run everywhere” they said. Now, 16 years after 1996, which was roughly when the Java hype started, I’d like to check if it delivered on this promise, by seeing if one can easily write a cross-platform GUI (= Graphical User-Interace) application using it. What I’m looking for is a Java GUI library that: 1. Compiles. 2. Is cross-platform. 3. Does not suck. Teaser: I was not able to find one.

The standard Java SWING toolkitis out-of-the-question due to its weird look and feel and crappy behaviour everywhere. Joel on Software has this to say about SWING:

If you still think that something as small as how long you hold down the Alt key when you activate a menu command doesn't matter, well, your software is going to make people unhappy. These tiny inconsistencies are what makes Swing applications so unbearably annoying to use, and in my opinion it's why there are virtually no commercially successful Java GUI applications.

(From the book reviews, there's more criticism of it in “Lord Palmerston on Programming”.)

I too have been frustrated by the unusable Java SWING GUIs, and in one of the latest projects I am contributing to, which has already been written in Java, I ran into a particularly buggy SWING UI.

The Eclipse people implemented something called SWT, but according to the wikipedia, it has several limitations, and require writing a lot of platform-dependent code.

Next are the Qt bindings to Java - Qt Jambi. These require customising the build.properties (whereas the error and documentation says one should edit the buildpath.properties file), and after that fails compiling with this error:

[make] ../../generator/out/cpp/com_trolltech_qt_core/
qtjambishell_QFutureIterator.cpp:21:15: error: no matching
function for call to ‘QFutureIterator::QFutureIterator()’


Next I looked into java-gnome, but apparently it does not work on Windows, so it's not cross-platform.

And the Java bindings for wxWidgets offer wxJava, which is Windows-only and unmaintained, and wx4j and jwx!, which are unmaintained and no longer build.

So I think that leaves us with nothing. How sad. Compare that to the comprehensive coverage of Mono’s GUI toolkits, many of which are cross-platform, and to the situation in other languages such as Perl or Python, and you'll see that Java sucks the most in this respect.

Syndicated 2012-10-20 10:49:19 from shlomif

680 older entries...