Older blog entries for Nafai77 (starting at number 14)

How to redirect stderr and stdout to a file plus display at the same time

The biggest search term that has brought people to my blog this month has been "redirecting stderr". I realize that I probably should expound more on Redirecting stderr and stdout to a file plus displaying them so that it makes more sense.

The original example script:

  #!/bin/bash

OUTPUT_LOG=output.log
OUTPUT_PIPE=output.pipe

if [ ! -e $OUTPUT_PIPE ]; then
    mkfifo $OUTPUT_PIPE
fi

if [ -e $OUTPUT_LOG ]; then
    rm $OUTPUT_LOG
fi

exec 3>&1 4>&2
tee $OUTPUT_LOG < $OUTPUT_PIPE >&3 &
tpid=$!
exec > $OUTPUT_PIPE 2>&1

echo "This is on standard out"
echo "This is on standard err" >&2

exec 1>&3 3>&- 2>&4 4>&-
wait $tpid

rm $OUTPUT_PIPE

The Explanation

First,

  if [ ! -e $OUTPUT_PIPE ]; then
    mkfifo $OUTPUT_PIPE
fi

I needed to make a named pipe -- otherwise known as a FIFO (First In First Out) -- to provide means for the shell and tee to share the output. You're probably familiar with pipes, such as ls -l | grep myfile. Named pipes are merely a way for you to do the same thing via a "file" in the filesystem. That same command can actually be done this way:

  mkfifo tmpfifo && (ls -l > tmpfifo &) && grep myfile < tmpfifo

Yeah, it's a little messy for that simple command. But it can be used in cases where a simple pipe cannot be used.

Next,

  exec 3>&1 4>&2

This saves the file descriptors for stdout (file descriptor 1) and stderr (file descriptor 2) to file descriptors 3 and 4.

Then,

  tee $OUTPUT_LOG < $OUTPUT_PIPE >&3 &

tee will, to quote from the manpage tee(1): "read from standard input and write to standard output and files". So, let's break up this line into parts to describe what's going on:

  • tee $OUTPUT_LOG: tee will take from stdin and write to the file named in the variable $OUTPUT_LOG.
  • < $OUTPUT_PIPE: tee's stdin will be redirected from the named pipe named in the variable $OUTPUT_PIPE
  • >&3: redirect tee's stdout to file descriptor 3, which is the stdout of the shell script
  • &: start tee as a background process of the shell

Next,

  tpid=$!

This line saves the process id of the tee process to the variable tpid.

Then,

  exec > $OUTPUT_PIPE 2>&1

In parts:

  • exec > $OUTPUT_PIPE: redirects the stdout of the script to the named pipe named in the variable $OUTPUT_PIPE.
  • 2>&1: redirects stderr to stdout so both stderr and stdout are redirected to the named pipe.

Then comes whatever you need your script to do. I included a few echo statements to show that both stdout and stderr are shown both in the terminal and in the log file.

After the meat of the script:

  exec 1>&3 3>&- 2>&4 4>&-

  • 1>&3 and 2>&4 restore the original file descriptors for stdout and stderr from file descriptors 3 and 4.
  • 3>&~ 4>&~: now that they are not needed, close file descriptors 3 and 4

Then,

  wait $tpid

This "pauses" the script until the tee process (whose process id is saved in the variable $tpid) exits. It will exit because its stdin (the output from the named pipe) has been closed. The named pipe closed the stdin because its input (file descriptors 1) was closed when the file descriptor stored in descriptor 3 was restored back to the original stdin.

Finally,

  rm $OUTPUT_PIPE

remove the named pipe, because it is not needed any more.

Addtional Information

For additional information, Chapter 19. I/O Redirection and Appendix E. A Detailed Introduction to I/O and I/O Redirection of the Advanced Bash-Scripting Guide are very useful in understanding these concepts. Also, a good introduction to named pipes can be found in this Linux Journal article.

I hope this helped make things a little bit clearer.

Syndicated 2006-08-20 04:20:00 from Travis B. Hartwell / Software Craftsman

Handy script to search for and install Debian packages

I often find myself searching for a package with apt-cache search and then installing interesting matches with apt-get install. Whenever I find myself doing something over and over again, I want to automate it, to make it easier, faster, repeatable. To this end, I have come up with search_and_install.sh.

Some may say that I should just use synaptic or a similar tool. I find doing this is much quicker and easier to use for this use case.

To Use

To use, just run search_and_install.sh or search_and_install.sh <search terms>. If you leave off the search terms, a dialog box will prompt you for the search terms. If there are matches, a dialog box will come up listing the matching packages with their name and the short description returned by apt-cache search. Each will have a checkbox beside it. Check the packages you want to install and then hit OK. You will then be prompted for your password (or your root password, if you are on a non-Ubuntu system), and synaptic will run, downloading and installing the selected packages.

Caveat

Currently, packages you already have installed are shown in the list. In future versions, I'd like to filter those out.

Dependencies

bash, zenity, apt, synaptic, gksu

A standard Ubuntu install should have these, and these are easily installable on any other Debian-derived distro.

Download search_and_install.sh.

This and other scripts will be linked from my Random Code page. Hopefully something there will be useful.

Syndicated 2006-07-31 00:16:00 from Travis B. Hartwell / Software Craftsman

Quick pop-up of Tomboy notes

I've started using Tomboy to keep track of my tasks lists, to keep random notes, whatever. I've found it to be very useful.

I am always seeking to be more productive and find ways to reduce the amount of distraction in my life. One example: if I am in the middle of working on a project and remember I have to pick something up on my way home from work, I want to quickly jot that down so I don't forget. But I don't want to interrupt the concentration that I have on the task at hand. Tomboy is an ideal solution for this. I am trying to implement a GTD-like system, so I have a Tomboy note titled "GTD Inbox". The idea is to put all of those things that need to be processed and remembered in my "GTD Inbox" to be dealt with later.

Normally, to get to do this, I would have to click on the Tomboy icon in the notification area and select the note title. Alternately, I would hit Alt-F12 to pop up the same menu and move down with the arrow keys and hit Enter. If I am trying to minimize the interruption of the original task, that's too much. I'd rather rely on "muscle memory" -- something that is as natural as pressing Alt-Tab to switch between windows.

With the help of Alex Gravely (thanks Alex!), I was able to come up with a popup_note.sh that uses dbus-send to open up a specified note. With that, I could set a hotkey in Metacity for each of the important notes. For example, at work Windows-i opens "GTD Inbox", Windows-n opens "Next Actions" (my list of next actions for all of my projects, GTD-style), and Windows-t opens "Today" (my list of goals and tasks for the day).

So, back to my "remember to pick up the milk" example. All I'd do is hit Windows-i, type "Pick up Milk", and hit Escape. The Note goes away and I'm back at doing whatever I was doing previously. Wonderful.

To use the script, just pass the title of the note on the commandline, like so:

   $ popup_note.sh GTD Inbox

If you want to set up global hotkeys in Metacity, do this:

   $ gconftool-2 -t string -s /apps/metacity/global_keybindings/run_command_1 "<Mod4>i"
 $ gconftool-2 -t string -s /apps/metacity/keybinding_commands/command_1 "popup_note.sh GTD Inbox"

Download popup_note.sh.

This and other scripts will be linked from my Random Code page. Hopefully something there will be useful.

Syndicated 2006-07-25 04:29:00 from Travis B. Hartwell / Software Craftsman

New Blog

I'm moving my blog from LiveJournal to my own domain. I'm using PyBlosxom, and so far I'm pretty pleased.

I plan on blogging more now that I have greater flexibility and control.

Watch this space.

Syndicated 2006-07-21 15:08:00 from Travis B. Hartwell / Software Craftsman

Redirecting stderr and stdout to a file plus displaying them

It has taken me ages to figure this out. Googling has been fruitless. Finally, after some tinkering, I was able to figure out how to properly redirect stdout and stderr to a file plus display them from within a script. Here's the idiom:

  #!/bin/bash

OUTPUT_LOG=output.log
OUTPUT_PIPE=output.pipe

if [ ! -e $OUTPUT_PIPE ]; then
    mkfifo $OUTPUT_PIPE
fi

if [ -e $OUTPUT_LOG ]; then
    rm $OUTPUT_LOG
fi

exec 3>&1 4>&2
tee $OUTPUT_LOG < $OUTPUT_PIPE >&3 &
tpid=$!
exec > $OUTPUT_PIPE 2>&1

echo "This is on standard out"
echo "This is on standard err" >&2

exec 1>&3 3>&- 2>&4 4>&-
wait $tpid

rm $OUTPUT_PIPE

It works but it might not be entirely correct, particularly the exec &> /dev/null. Improvements or suggestions are welcome.

Update: Figured it out. I had to store stderr and then restore stdout and stderr and close the fd that I stored them in.

Update update: Looked back and realized I had a typo on my first exec line.

Syndicated 2006-07-14 08:23:00 from Travis B. Hartwell / Software Craftsman

Random Ideas

Frustrations

Okay, maybe this is ironic that I'm posting this on my Live Journal account, but I really hate how limited LJ is. I would love to be able to track site statistics and reads and so forth. I just got a Google Analytics account for my main domain. There is currently (and probably never will be) a way for me to put in the requisite javascript code into my template at LJ to have Google Analytics track things. Yet another reason to move off of it. After the wedding, I think I'll finally have time to get my blog set up on my own hosting account. I'm going to be using Pyblosxom as it seems to be most flexible and easiest to integrate with Twisted Web. I got Twisted Web 2 working with Pyblosxom via WSGI with a minimal amount of code and some help from #twisted.web. I'll post how I did that when I get that blog up.

Random Ideas

I figure this would be a good place to throw out my random ideas of things I want to hack on, things to figure out. Feel free to tell me my ideas are dumb. Feel even freer to tell me that they've already been implemented or that you want to write them. :)

In reference to my prior entry regarding my IRC and IM setup, here are some things I've been thinking about:

  • I want to write a programmable chat notifier, so I don't have to be constantly logged onto my session locally. I want an application that will connect to my proxy for me and, when certain conditions (regular expressions or something) are met, notifiy me via libnotify and the notification area of my Gnome panel. That way I can't let IM/IRC distract me.
  • Write a wrapper around XChat-Gnome to handle my setup more cleanly:
    • If my ssh tunnel is not started, start it, prompting me for the password
    • Turn off the notification plug-ins (sound, notification, and on screen display) to start. As bip is playing back my log to the session, I don't want to be notified.
    • Find out, via d-bus or a python plug-in or something, that the playback is done and then enable those plug-ins
    • If appropriate, on exit, kill the tunnel

I've started using Evolution as my primary mail client, so I have a few things I want to do with it:

  • Update my (not-yet-released) new mail notifier:

    • Make it more robust and clean code-wise.
    • Give more information, if possible. (Number of new messages, perhaps message information)
    • Leave an icon in the notification area until I click the pop-up notification or the system tray icon.
    • When clicking either, go to the new messages in Evolution
  • I get a lot of spam -- probably 1250 or so in the last week. I need to curb that.

    • We use spam bayes on my mail server. I need some way to retrain it, when it misses or mis-classifies something.
    • I also use spamassassin locally, which works for most of what I have.
    • First filter on spam bayes, and then spam assassin classifications.
    • I want to be able to set up something -- like search folders -- that will capture messages that are in the junk folder, but missed by one or the other.
    • I want to be able to automatically re-train spambayes and spamassassin appropriately on these.

So, for Evolution, I need to figure out the following:

  • In what order are the filters applied? Junk filters and then user-defined filters? Can I specfy the order of this? Can I specify the order of my user-defined filters?
  • Can I set up a filter that is only applied when I select it? For example, I need to write some shell scripts to handle re-training Spambayes on my remote server. I would like to be able to click on a message and then tell Evolution to pipe that message to a given program. Can I do this?

I think that's all of my random ideas for now. I probably won't get to any of this until after the wedding. Of well, priorities, you know.

Syndicated 2006-05-17 19:44:00 from Travis B. Hartwell / Software Craftsman

Ten Days and Counting

On Saturday, May 27th, my sweetheart Torie and I are getting married!

Ten days!

Torie and Travis engagement picture

Syndicated 2006-05-17 19:41:00 from Travis B. Hartwell / Software Craftsman

Iswitch-window handler update

I keep meaning to announce that my iswitch-window deskbar handler as described here, has now been included in the core Gnome Deskbar Applet distribution, as seen in the 2.15.1 announcement. Any further changes or improvements will go directly into deskbar cvs.

Enjoy! Thanks Raph and team for a great project and including my contribution!

Syndicated 2006-05-01 10:43:00 from Travis B. Hartwell / Software Craftsman

ncm: My lack of strength in C++ shows here. Why is using delete considered sloppy thinking? I would assume that the use of new is acceptable. For example, when you are allocating memory for a dynamically sized array (yes, I do know about the STL <vector> I thought it was prope, and in fact necessary, to pair new and delete calls as in C with malloc() and free. What am I missing? Standard C++ doesn't have garbage collection, so I'm a bit lost.

An update: in just 6 weeks or so, I will be done with school forever. My last final is on December 12th, and I will be finishing my bachelor's degree in Computer Science with a minor in Mathematics at Utah State University. Finally!

On the free software front, I've been doing a little hacking. I've worked more on my Gnome Alerts Panel Applet, now with clients for an e-mail watcher, web page monitor, cvs monitor, and a regular standalone messager. I need to finish the UI and then put it up for others to use.

I just recently started using Spam Bayes as Spam Assassin wasn't catching all of my spam. Spam Bayes is doing a great job of catching nearly all spam that SA does not, and only a few false positives and unsures. I need to tweak how I train it, but overall I am pleased. I was annoyed with having to start a Python process for every single message that comes through, so I wrote a spam bayes daemon that listens on named pipes and then a small C program that acts as the filter that procmail calls that interacts with the daemon via the pipes. I was pretty proud of myself. After doing that, I added the named pipe capability to the e-mail watcher program for my alerts applet. Anyway, I want to release this stuff to the Spam Bayes community, I just haven't taken the time to clean it up and put it out there.

Other than that, my free software stuff is slow. I'm waiting for all of the Gnome 2.4 packages to come through on Debian PPC Unstable, and am excited to try that out. As soon as the Linux 2.6.0 final kernel comes out, I will be trying it out as well. Hopefully it will make my boxes feel faster, as I have slow machines right now.

5 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!