On the flip side, I can reduce the need to deallocate memory simply by introducing memory leaks … yes, you read that right! Basically when you have many millions of objects allocated it can take quite a while to deallocate them all. There’s something like a 10-second pause after processing the 42G data file while my application de-allocates all of the objects, so an easy performance win is simply Not Do That. Every second counts, particularly when you have Java implementations to beat …

-T

And even with a 64-bit binary, you really don’t want to mmap an entire 42GB data file into memory, trust me.

Well I don’t trust you. I currently and routinely mmap in 10+TB in one shot on a 64bit machine. So what’s the problem ? Elaborate plesae?

Well I don’t trust you.

Well, as it turns out, Brian was right not to trust me. I was too hasty in condemning the all-at-once mmap.

Here’s a test app:

boost::iostreams::mapped_file_source source(argv[1]);
unsigned lines = std::count(source.begin(), source.end(), '\n');
std::cout << argv[1] << ": " << lines << " lines" << std::endl;

As you can see it just counts the number of \n characters in the requested file. I ran this on the Wide Finder 2 full data set, and here’s what happened:

~/wf2/ 512> time ./readmmap /wf1/data/logs/O.all
/wf1/data/logs/O.all: 218201129 lines

real    16m0.565s
user    7m42.454s
sys     7m53.235s

Two important things to note here. First, obviously mmap-ing the whole file works as advertised. But more importantly, it seems that my Wide Finder 2 implementation is already running at I/O speed.

But other Wide Finder 2 implementations are going faster, which raises the obvious question as to how. mmap is traditionally the fasted form of I/O, given that it doesn’t have to copy the data from kernel space into user space. But obviously that rule doesn’t hold any longer, at least for Solaris.

More investigation needed, I feel.

1. Have 1 thread keeping stats.
2. Have dynamically starting threads on block boundaries – i.e. read 4k at a time (or whatever the HDD’s block size is) and start the thread with that instead of explicitly searching for a newline. These will send messages to two threads. One for the statistics, and another for a “residual” (a message not ending a line).
3. Have another thread which waits for residuals and matches them up. Once it gets a bunch of em, it can dynamically start one of the threads in (2).

The only downside is the mallocs. IMO you need to copy data or else you’re hosed from a multi-threaded perspective. Just thinking out loud here…

what compiler options are you using? In particular, are you making use of prefetch? Specifying a page size? On something as simple as the exemple that counts the newlines it could make a difference.