I apologize ahead of time that I don't quite have the proper jargon to describe my problem, and that I have likely not given enough information.
I've been running my MPI code under gcc 4.4 and OpenMPI/MPICH2 for months now with no issue on a variety of platforms. Recently I upgrade a set of servers and my desktop to Ubuntu 11.04 (running gcc 4.5 now) and ran an 8 task job on a node with 8 processors. Typically I see nearly 100% user CPU utilization, and now I see only 60% user CPU and over 30% system cpu. This leads to a remarkable slowdown of my code when run in this fashion.
Investigating further, I simply ran a serial job, and noted that the process reported 150+% cpu time was being used. So, my program was multithreading itself over many processors. I verified this explicitly using 'ps -eLF' and looking at the per-processor loads.
This is an incredibly bad and inefficient thing for my MPI code, and I have no idea where it comes from. Nothing has changed other than the move to Ubuntu 11.04 and gcc 4.5. I have verified this against varying OpenMPI versions.
I also moved binaries around between two binary-compatible machines. If I compile on another machine (ubuntu 10.10/gcc 4.4) and run there, everything is fine. Moving the binary to the Ubuntu 11.04 machine, the same binary begins threading itself.
It is worth noting that I have explicitly disabled all optimizations (-O0), thinking my default (-O3) could include something I didn't understand in 4.5. I get identical behavior regardless of the optimization level.
Please let me know what further information I can provide to determine the source of this problem.
* ADDITIONAL INFO *
Results of ldd in response to request. Simply, it's OpenMPI, libconfig, and scalapack, along with standard gcc stuff:
linux-vdso.so.1 => (0x00007ffffd95d000)libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2bd206a000)libconfig.so.8 => /usr/lib/libconfig.so.8 (0x00007f2bd1e60000)libscalapack-openmpi.so.1 => /usr/lib/libscalapack-openmpi.so.1 (0x00007f2bd151c000)libmpi.so.0 => /usr/lib/libmpi.so.0 (0x00007f2bd126b000)libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2bd0ed7000)libblacsCinit-openmpi.so.1 => /usr/lib/libblacsCinit-openmpi.so.1 (0x00007f2bd0cd4000)libblacs-openmpi.so.1 => /usr/lib/libblacs-openmpi.so.1 (0x00007f2bd0aa4000)libblas.so.3gf => /usr/lib/libblas.so.3gf (0x00007f2bd022f000)liblapack.so.3gf => /usr/lib/liblapack.so.3gf (0x00007f2bcf639000)libmpi_f77.so.0 => /usr/lib/libmpi_f77.so.0 (0x00007f2bcf406000)libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f2bcf122000)libopen-rte.so.0 => /usr/lib/libopen-rte.so.0 (0x00007f2bceed3000)libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2bcecb5000)/lib64/ld-linux-x86-64.so.2 (0x00007f2bd22fc000)libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2bcea9f000)libopen-pal.so.0 => /usr/lib/libopen-pal.so.0 (0x00007f2bce847000)libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2bce643000)libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f2bce43f000)All the best.