I write therefore I think.

Answers on a postcard.

Take one webserver running Apache/1.3.27, with PHP/4.3.0, on 4.5-RELEASE. Aside from a small problem with it maxing out 4Mbps there is nothing wrong. Now, migrate to Apache/2.0.49, with PHP/4.3.7, on 5.2.1-RELEASE-p9. As the DNS is changed what the load on the new server creep sky high and, eventually, stop responding.

Try again with Apache/1.3.31 on 5.2.1-RELEASE-p9. Then with Apache/2.0.49 on 4.10-RELEASE. Each time what the load average go through the roof: up to 420 at one point: totally locked up. Run top and systat -vmstat 1 and see the box struggle - each of the numerous apache process running with at least 3% CPU.

Turn off softupdates (battery backed RAID controller so we should be fine). Install an Intel fxp NIC and enable interrupt mitigation. At last an improvement... the box grinds to a halt slightly slower.

In desperation copy the binaries off the old 4.5-RELEASE server. Perform some jiggery-pokery to get the dependencies sorted. And fire it up. Then watch top and systat and see as everything runs smoothly and the new server happily sucks up 8Mbps. Marvel at the reduced number of apache processes which only use 0.00% CPU. Wipe the sweat off your brow and mutter "thank f**k for that".

So. What the hell is going on?

The only thing I can possibly think of is Apache's DSO support. Looking at the configure command from the old:

./configure \
"--with-layout=Apache" \
"--prefix=/usr/local/apache-1.3.27" \
"--enable-module=so" \
"--enable-module=all" \
"--enable-shared=expires" \
"--enable-shared=headers" \
"--enable-shared=rewrite" \
"--enable-shared=speling" \
"--enable-shared=unique_id" \
"$@"

And comparing it with the new:

CC="cc" \
CFLAGS="-O -pipe -mcpu=pentiumpro \
    -DDOCUMENT_LOCATION=\\\\\"/usr/local/www/data\\\\\" \
    -DDEFAULT_PATH=\\\\\"/bin:/usr/bin:/usr/local/bin\\\\\" \
    -DHARD_SERVER_LIMIT=512" \
LDFLAGS="-L/usr/local/lib" \
LD_SHLIB="cc" \
INCLUDES="-I/usr/local/include" \
./configure \
"--prefix=/usr/local" \
"--server-uid=www" \
"--server-gid=www" \
"--with-perl=/usr/local/bin/perl" \
"--with-layout=FreeBSD" \
"--datadir=/usr/local/www" \
"--htdocsdir=/usr/local/www/data" \
"--cgidir=/usr/local/www/cgi-bin" \
"--without-confadjust" \
"--enable-module=most" \
"--enable-module=auth_db" \
"--enable-module=mmap_static" \
"--disable-module=auth_dbm" \
"--enable-shared=max" \
"--enable-rule=EXPAT" \
"$@"