11 Dec 2007 elanthis   » (Journeyer)

Creating Custom C++ Output Streams

In my younger, dumber days, I’d often write whole new classes in C++ when I had a need for output streaming, such as in a class handling a TELNET connection. Such classes required a ton of

I'm here today to tell you how not to do that, but to instead write a stream class that will work with all of the standard C++ output stream features. You might even learn a thing or two about input streams, but output streams are going to be my focus.

Note that I’m only going to touch on the basics here. C++ streams have a lot of features, most of which you won’t need to customize in the the majority of circumstances, so I’m ignoring those topics.

Introduction

Streaming output in C++ is accomplished by using the

One of the primary reasons to use a stream instead of directly writing bytes to a file is that streams allow for formatting and buffering. Formatting allows you to do something like the follow:

cout 

That will write out 123 in hexadecimal, or 7b. Without streaming, you’d have to create a byte buffer, format 123 yourself into that buffer, and then call system facilities like wite() to get your output on the screen. Kind of a pain.

C programmers will be familiar with the printf family of functions. These functions server a very similar role to the C++ streaming facilities. The above line of code, in C, could be written:

printf("%xn", 123);

The C++ streams offer several very distinct advantages over the printf family of functions, however. The first, and most widely known, is type safety. If the printf call had used the %s formatter instead of %x, then the program would likely have just crashed. The second advantage is that C++ streams have built-in support for user-controllable buffering. Buffering allows output to be stored up and sent to the OS facilities in larger chunks, which can both improve performance as well as allow for some special tricks which we’ll explore later. To provide user-controlled buffering in C, new functions which use printf functions internally must be created, and gracefully dealing with buffer overflows (that is, neither crashing nor losing output) can be a serious pain. A final advantage is that C++ streams maintain state, allowing you to more easily output a large number of identically formatted values without respecifying the format for each and every one.

The printf family of functions do have some advantages. They are, in most implementations, significantly faster than their C++ counterparts. Additionally, sometimes that “advantage” of C++ streams of maintaining state can actually be a problem, particularly if you set some state and never unset it. For example, the C++ example up above sets the number output format to hexadecimal, but never reverts it to decimal. All other output on cout will be formatted to hexadecimal until reset.

Since this is an article about C++ streams and not C string formatting, we’re going to assume that you actually want to use C++ streams and not printf, so let’s get on to the meat of creating a custom output stream.

Ostream and Streambuf

The std::ostream class does all the meat of formatting your streams. It stores the output state (like whether numbers should be displayed as decimal or hexadecimal) and processes your values to convert them into the correctly formatted output. This class is really the true core of all output formatting.

There’s absolutely no need to derive a class from std::ostream, either. The ostream class handles the formatting, but it doesn’t itself actually do anything with the output. Sure, there are derived classes like std::ofstream and std::ostringstream in the standard library, but these classes don’t actually change the behavior of std::ostream in any way. They are merely convenience wrappers that make use of a derived std::streambuf class.

All the actual work of outputing formatted data is performed by std::streambuf. Every ostream has a streambuf object associated with it. When you stream data to an ostream object, it formats the data and passes the results on to its streambuf. The streambuf then does the actual interesting work of writing the result out to your screen, into a file, or into an internal buffer. When you want to change the behavior of an output stream, what you actually need to do is make a new streambuf child class.

The std::ofstream class, for instance, creates a new std::filebuf object and associates it with the opened file. The ofstream class also offers a few other convenience methods on top of the base std::ostream, but all of these methods actually interact with the filebuf object.

To associate a streambuf object with an ostream, you can call the ostream::rdbuf() method. If called with no arguments, it returns the current streambuf. If called with a pointer to a streambuf, it sets that as the current streambuf. You can also pass a pointer to a streambuf to the constructor for an ostream. For example, let’s mimic ofstream using just ostream and filebuf.

filebuf file;
file.open("myfile.txt", ios::out);
ostream os(&file);
os 

That could behaves identically to code that uses ofstream. The only difference there is that the methods like open and close must be called on the filebuf object instead of the ostream object.

There is one catch to be wary of. The ostream class will not manage the memory for its streambuf object. That’s fine for the example above, but if you had created the filebuf object using the new operator, you would have to remember to delete the pointer yourself when you’re done.

Buffered and Unbuffered Output

There are two kinds of output you can perform using the std::streambuf class: buffered and unbuffered. Buffered output is when all data is stored temporarily in a buffer. The data is only sent to the actual output destination when the buffer fills up, or when the output stream is flushed. Unbuffered output sends all data to the output destination immediately.

When you create a new streambuf instance, it is by default unbuffered. If you wish to make it buffered, you must create a buffer for it, and then tell the streambuf about your buffer using the setp method, which is protected. So, to create a buffered streambuf object using a 100 character buffer:

class mybuf : public streambuf {
public:
  mybuf () {
    setp(buffer, buffer + sizeof(buffer));
  }

private:
  char_type buffer[100];
};

And voila! Your streambuf descendent is now buffered using your 100 character array. That’s all there is to it.

Note that buffer memory is not managed by the streambuf class. If you allocate a buffer with new, you are responsible for deleting it.

Custom Unbuffered Streams

You want to write a log stream facility that sends output both to cerr (standard error output) as well as a file, mylog.txt. It’s easy enough to do either - just stream your data to either cerr or a ofstream - but you’d rather not write each stream command twice. Writing a very simply streambuf class that performs both for you is, thankfully, quite easy.

A virtual method called xsputn is called on a streambuf whenever there is data to write. You need only override that one function to create a custom unbuffered streambuf. The function takes a pointer to an array of characters and the length of the array, and is expected to return the number of characters it was able to write. Since we’re just passing this on to a couple other streams we just return the length of the buffer given us.

// your log file, lazily declared as a global
ofstream logfile;

// logbuf forwards all output to cerr and logfile
class logbuf : public streambuf {
private:
  // write a string s of length n to standard
  // error and a log file
  int xsputn (char_type* s, streamsize n) {
    cerr.write(s, n);
    logfile.write(s, n);
    return n;
  }
};

int main () {
  // open our log file
  logfile.open("mylog.txt", ios::app);

  // create our log stream
  ostream log(new logbuf());

  // be friendly
  log 

That’s the gist of what you need, and nothing more. Pretty simple, eh? We could improve things a little further. For example, our logbuf object is leaked - we never delete it. That isn’t really vital for this example, since the memory is reclaimed at the end of the function anyhow, but we should handle it properly anyway. More importantly for our little example, however, we don’t control buffering properly. Our logbuf is unbuffered, but both cerr and logfile are buffered. We would expect the

class logbuf : public streambuf {
private:
  // flush both cerr and logfile; return 0 to
  // indicate there was no error, but we're
  // too lazy to check for errors ourselves
  int sync () {
    cerr.flush();
    logfile.flush();
    return 0;
  }

  // write a string s of length n to standard
  // error and a log file
  int xsputn (char_type* s, streamsize n) {
    cerr.write(s, n);
    logfile.write(s, n);
    return n;
  }
};

There, flushing is now supported!

I’m going take this moment to explain the char_type and streamsize types used above. streambuf::char_type is a typedef for the actual character type in use, which for streambuf would be char. The wstreambuf type is identical to streambuf, except it works with wchar_t (wide character support, for unicode), and char_type is different for that class. The streamsize type is similar in purpose to size_t - it’s just a typedef for the particular type of integer your STL implementation chose, and using streamsize makes your code portable.

Custom Buffered Streams

Unbuffered streams are great, but they’re not always what you’re looking for. Say that you are writing a network stream. You don’t want to call send() over and over for performance reasons; you’d rather buffer up your output and send it all at once. Once we set the buffer with setp, the streambuf class will do all the work of putting characters into the buffer and protecting against overruns. We don’t need to implement our own xsputn at all, since the default implementation does exactly what we want. We can simply override the sync method to take our buffer contents, write them to the socket, and then clear the buffer.

class sockbuf : public streambuf {
public:
  // initialize our sockbuf with a socket
  // descriptor, and setup a new buffer
  sockbuf (int _sockfd) : sockfd(_sockfd) {
    char_type* buf = new char_type[1024];
    setp(buf, buf + 1024);
  }

  // free our buffer
  ~sockbuf () {
    delete[] pbase();
  }

private:
  // dump our buffer to the socket and clear
  // the buffer
  int sync () {
    // for brevity's sake, not doing proper error
    // handling; we return a non-zero value(error)
    // if we failed to send the full buffer contents
    int ret = send(sockfd, pbase(), pptr() - pbase(), 0);
    if (ret != pptr() - pbase())
      return 1;
    // reset the buffer
    setp(pbase(), epptr());
    return 0;
  }

  // our socket descriptor
  int sockfd;
};

We’ve got a few new functions there. The pbase() method returns a pointer to the beginning of the buffer. The pptr() method returns the current position of the stream in the buffer. So, the number of characters in the buffer is equal to pptr() minus pbase().

Unfortunately, this little class has a problem. Our buffer is only 1024 characters long, and the data is only written out when the flush method is called on the ostream using this streambuf. When that buffer fills up, any further data we try to stream is just lost. It would be ideal if we could instead grow the buffer or try to flush the data we already have in the buffer. Let’s try growing the buffer.

When our buffer fills up, the overflow() method is called. This method takes the character that didn’t fit into the buffer as a parameter, and returns either EOF to indicate failure or any other value to indicate success. We’re just going to grow our buffer by 1024 elements and then call the standard sputc() function to add the character into the buffer.

class sockbuf : public streambuf {
public:
  // initialize our sockbuf with a socket
  // descriptor, and setup a new buffer
  sockbuf (int _sockfd) :
      sockfd(_sockfd), buf(0), buflen(1024) {
    buf = new char_type[buflen];
    setp(buf, buf + buflen);
  }

  // free our buffer
  ~sockbuf () {
    delete[] buf;
  }

private:
  // dump our buffer to the socket and
  // clear the buffer
  int sync () {
    // for brevity's sake, not doing proper error
    // handling; we return a non-zero value(error)
    // if we failed to send the full buffer contents
    int ret = send(sockfd, pbase(), pptr() - pbase(), 0);
    if (ret != pptr() - pbase())
      return 1;
    // reset the buffer
    setp(pbase(), epptr());
    return 0;
  }

  // we ran out of space, so grow
  // the buffer
  int overflow (int c) {
    // allocate a new buffer and copy our
    // current data into it, then swap it with
    // the old buffer
    char_type newbuf[buflen + 1024];
    memcpy(newbuf, buf, buflen);
    delete[] buf;
    buf = newbuf;

    // now we need to stuff c into the buffer
    sputc(c);
    return 0;
  }

  // our socket descriptor
  int sockfd;

  // our buffer
  char_type* buf;
  unsigned long buflen;
};

And there we have it. Our sockbuf class can now buffer up data until flushed without losing any data.

Complex Example

Alright, let’s go ahead and totally abuse the system now. We want to use our log class from before, but we want all of our log lines to include a date and time at the start as well as a log priority, but we don’t want to have to stream the time to log over and over. We’d like to be able to write code like this:

log 

There are a few important things going on here. First, we are setting the priority by streaming out a special priority value (e.g., DEBUG, ERROR). Note that on the third line we didn’t stream a priority, but the ERROR priority from the prior line didn’t carry over. It’s a piece of state that we’ll reset when a flush (or endl) occurs on the stream.

We could do this as an unbuffered stream. We would just write an xsputn method that wrote the time and then the log message. However, think what would happen in this example:

log 

We’d actually end up with the time and priority printed four times in the single line: once before “The,” once before the user’s name, once before ” logged,” and then a final time just before the newline. We wil need to buffer our output and then only write the time and the log once for each flush.

We’re also going to assume that flush won’t be called on our log directly, but instead we’ll always use endl. That way we know that our stream will contain a newline and we don’t need to worry about one ourself.

We’re also going to actually override ostream this time. We actually need to do that to get the priority feature to work, plus it’s kind of a pain to have to create an ostream object and then call rdbuf() on it all the time, and a custom ostream allows us to hide that in the constructor.

First, the priorities. This will just be a simple enum.

enum LogPriority {
  INFO, // regular unimportant log messages
  DEBUG, // debugging fluff
  ERROR, // it's dead, jim
};

Because it’s an enum, we can create a special

Our logbuf derived from streambuf should be fairly old news by this time. In fact, it's identical to our sockbuf above, except when sync() is called we spit out the time and the log message to cerr and a logfile ofstream, whch this time around we'll make a member. We also keep the current priority level, which defaults to INFO.

class logbuf : public streambuf {
public:
  // create a buffer and initialize our logfile
  logbuf (const char* logpath) :
      priority(INFO), buf(0), buflen(1024) {
    // create our buffer
    buf = new char_type[buflen];
    setp(buf, buf + buflen);

    // open the log file
    logfile.open(logpath, ios::app);
  }

  // free our buffer
  ~logbuf () {
    delete[] buf;
  }

  // set the priority to be used on the
  // next call to sync()
  void set_priority (LogPriority p) {
    priority = p;
  }

private:
  // spit out the time, priority, and the
  // log buffer to cerr and logfile
  int sync () {
    // nifty time formatting functions
    // from the C standard library
    time_t t = time();
    tm* tmp = localtime(&t);
    char timebuf[128];
    strftime(timebuf, sizeof(timebuf),
      "%Y-%m-%d %H:%M:%S", tmp);

    // now we stream the time, then the
    // priority, then the message
    cerr 

Well, there’s that. Now we need our customized ostream-derived class so that we can stream the LogPriority values and get the desired behavior. We’ll keep a logbuf object as a member variable, which we’ll setup as the streambuf for ostream in the constructor.

class logstream : public ostream {
public:
  // we initialize the ostream to use our logbuf
  logstream (const char* logpath) :
    ostream(&buf), logbuf(path) {}

  // set priority
  void set_priority (LogPriority pr) {
    buf.set_priority(pr);
  }

private:
  // our logbuf object
  logbuf buf;
};

// set the priority for a logstream/logbuf
// this must be a global function and not a
// member to work around C++'s type
// resolution of overloaded functions
logstream& operator

And there you have it! You can now easily create log and use logstreams. You can even have multiple such streams, giving each its own log file.

int main () {
  logstream log("logfile.txt");

  log 

Closing Thoughts

C++ streams are fairly simple to implement. We took a few liberties in the examples above with sockets and memory handling, but the core concepts of writing a custom output streambuf are there.

The complete source to the logstream example is available here.

Syndicated 2007-12-10 23:34:28 from Sean Middleditch

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!