Comity of errors

Continuing on from last time, I had made a quick overview of conical intersections and density functional theory (DFT). I also promised more about Configuration-Interaction methods (Wikipedia) and the new method developed in our research group, Constrained DFT-Configuration Interaction. But first, a bit about electronic correlation.
The standard way to write an electronic wavefunction is as a Slater determinant — I have N (identical) electrons that I need to put in N one-particle orbitals, and maintain the antisymmetry required since electrons are fermions. The natural way to do this turns out to be to write an NxN matrix, with increasing electron index along the columns, and increasing orbital index along the rows. The determinant of this matrix is the simplest N-electron wavefunction from that given set of one-electron orbitals that obeys the required symmetry. However, this is a fixed description of a snapshot of an electronic configuration, that ties down the electrons in particular places. Unfortunately, this is not always sufficient to describe a system, as electrons (with their wavelike nature) can be quite tricksy. The most notable cases when this occurs are when there are multiple degenerate (or nearly-degenerate) configurations for the electrons of the system that have the same (or nearly so) energy. In this case, the actual wavefunction is a linear combination of the Slater determinants corresponding to those configurations. The classic example is when a bond is being broken — dihydrogen at large internuclear separations should have one electron at each nucleus, but in a Slater “determinant” (no actual determinant being needed since there is only one electron of each spin), we have to pick one nucleus to have the spin-up electron and the other to have the spin-down one. Due to the quantum uncertainty principle, we can’t know in advance, and thus spin-up and spin-down must be in a linear combination until the wavefunction is collapsed. This need for multiple determinants is the result of “static correlation”, where there are multiple electronic configurations to be accounted for. There is also “dynamical correlation”, which reflects the fermionic tendency of the electrons to not be in the same place, i.e. to avoid each other. This is sometimes described as the time-dependent motion of the electrons to avoid each other, though this is not exactly right. If this all seems a bit confusing, don’t worry too much — there isn’t a precise way to distinguish “static” and “dynamic” correlation, and in fact, the correlation energy is usually defined as the difference between the Hartree-Fock energy (that is, the best single-determinantal energy) and the exact energy (in a given basis set). That’s not really a very useful description, but we make do. In any case, the takeaway point is that multiple determinants or electronic configurations are needed to accurately treat static correlation.
DFT has been a very popular method because of its ability to describe dynamical correlation (usually) with the same expense (or less!) than Hartree-Fock, which does not treat any dynamical correlation. The main idea of the CDFT-CI method is to add static correlation to a DFT method. It does so using Configuration Interaction, a method that starts off with a (usually large) set of electronic configurations/determinants and produces improved representations of the ground (and excited) electronic states as linear combinations of those states. (I will go on to call these states “basis states”, not to be confused with the one-particle basis functions that are used to create the one-particle orbitals that make up each determinant.) The details of the method are actually conceptually fairly simple: create a representation of the system’s Hamiltonian in this basis of states, and then diagonalize that Hamiltonian to produce its eigenstates. To do this, in addition to our basis states, all we need is the off-diagonal terms of the Hamiltonian, which we can think of as electronic couplings between the different states/determinants. For somewhat technical reasons, these couplings are easily obtained for determinants that correspond to Hartree-Fock wavefunctions, but there is less theoretical backing for using Kohn-Sham wavefunctions (which are what DFT provides) in computing these couplings. One of the main features of CDFT-CI is a somewhat clever way to compute these couplings.
Another feature of CDFT-CI is that the basis used for the configuration-interaction (which can easily reach tens of thousands of states for standard CI methods) is quite small, involving only three or four states for the systems we treated in our paper. To generate these states, instead of just taking the standard DFT ground state (which would not provide a very interesting basis set when taken multiple times), we apply a constraint, forcing charge or spin to localize on parts of the molecule. These calculations are still ground-state DFT calculations (just of a slightly perturbed system), and as such retain the accuracy and dynamic correlation benefits of DFT. However, if we choose these perturbations correctly, we can get a lot of the character of the lowest few excited states; combining a few such constrained calculations and taking a linear combination allows the ground state to be removed and each excited state in turn to be amplified, as the several eigenvectors of the CI Hamiltonian.
And that’s CDFT-CI: take a few of these tweaked DFT calculations, compute couplings between them, and diagonalize the CI Hamiltonian; out come energies that correspond to the ground and lowest few excited states. There’s dynamical correlation from DFT, and static correlation from the configuration-interaction. All in all, it’s enough to make a pretty picture of a conical intersection, which you can see in the paper I mentioned at the start of last week’s post.

At the moment, the article that my advisor and I wrote is on the front page of The Journal of Chemical Physics. Entitled “Conical intersections using constrained density functional theory–configuration interaction”, we discuss how we have used a computational method previously developed in the research group to study conical intersections (CIs). I mentioned these in a previous post, but didn’t do much other than link to Wikipedia to describe them. Essentially, one can think of the problem of electronic structure (the branch of computational chemistry/physics in which I work) as being a question of describing a many-dimensional hypersurface which maps the electronic energy as a function of the position of the atoms that make up the molecule. (There are complications, of course, but we’ll ignore them for now.) There are actually infinitely many possible values of the electronic energy for any given nuclear position, corresponding to electronic excited states; each such excited state (as well as the non-excited ground state) traces out its own manifold in this large-dimensional space. Usually, these surfaces are well-separated, and the distance between them corresponds to the amount of energy in a visible or soft ultraviolet photon. This means that when we shine a light (usually a laser when we think about it, but any light will do) with about that much energy per photon onto our molecule, we can start to excite electrons from the ground state to an excited state. The energies do need to be fairly close, as the coupling between states has an inverse dependency on the [energy gap between states - photo energy] quantity. Anyway, once we have prepared an excited state, there are a few things that can happen. The simplest, is that it just spontaneously relaxes back to the ground state, emitting another photon. If that doesn’t happen, other relaxation pathways are available — the nuclei can move along the particular potential energy surface (that is, the hypersurface corresponding to this excited state) to a lower energy configuration, and emit a photon there. (This is fluorescence, and is responsible for the glows you see under a blacklight — the ultraviolet photons from the lamp are not visible to human eyes, but the lower-energy photons coming from fluorphores after they relax are visible.) There are some other loss pathways back to the ground state that are not easily described at this level of theory, but the final way for an excitation to relax is for it to find a place where two of these energetic hypersurfaces actually cross each other. In these configurations, the molecule can readily transition from the “excited” state to the ground state. These configurations are the conical intersections; many workers, notably David Yarkony, have found that these are actually quite ubiquitous in this many-dimensional configuration space.

Unfortunately, most of the ways to compute the electronic structure (that is, these hypersurfaces) that can actually find these conical intersections between surfaces are quite computationally expensive, scaling as N^6 or N^7 with the number of electrons (or worse). These are usually “wavefunction methods” (Wikipedia has something of a list), methods that explicitly try to describe the (3N-dimensional) configuration of all the electrons at once. An alternative class of methods, the density functional theory (DFT) class (Wikipedia) reduces the scope of the problem down to just a 3-dimensional function (which happens to be something that can be experimentally observed, by methods like X-ray diffraction), the electronic density. All electrons are identical, so we can think of this electronic density as being an average of where all the wave-like electrons happen to be. The advantage in computational cost comes at the expense of some approximations — though DFT is “in principle” exact (as wavefunction methods can be), this exactness relies on a “exchange-correlation functional” (a mathematical object that takes as input a density (as a function of position) and produces as output an energy). This functional is not known, and is unlikely to ever be known in a general fashion, so we must use approximations to it. DFT methods have only been developed comparatively recently (compared to wavefunction methods), and do not really have fully-reliable ways to compute the electronic excited states. Time-dependent DFT (which is kind of neat in its own right, and not exactly what it sounds like, but I shouldn’t digress) is the leading competitor, but it has previously been shown to be quite lousy at describing conical intersections (this is what the Levine reference from our paper is about). Our method, “Constrained DFT-configuration interaction” (CDFT-CI) builds on top of ground-state DFT to form a technique that is capable of describing CIs, which is the main result of our paper.
More about configuration-interaction methods in general, and CDFT-CI in particular, will appear in a later post.

The past couple posts followed my adventures in compiling (and running) a parallel program on an updated system. I have since found a resolution to my problems, though it was not what I expected.
Backing up a little bit, I was experiencing what might nominally have been a bug at the fourth-order of interaction — I had my source tree, a (vastly) updated operating system, updated Intel compilers, the fftw libraries, and the MPI compiler/libraries to deal with. Complicated systems like this can easily lead to ridiculously subtle and hard-to-diagnose issues (of course, they can also have extremely simple bugs, too), so I was not really expecting to just be able to type my issues into google and find an answer. However, I did have a couple of useful diagnostic tools at my disposal — I still had old libraries and compilers from the machine we’re replacing. After struggling with my issues in an ab initio fashion for a while, I made use of these tools, copying over the fftw libraries and LAM toolchain to the new machine. I was then able to isolate my bug to the LAM system — I could compile my program using the new Intel compilers and a new fftw library on the new machine, and it ran successfully. (Lest you think that this was an easy, obvious thing to do, I will point out that it did not go entirely smoothly. The mpic++ compiler wrapper has logic to check that there are LAM libraries where it expects them to be, but apparently it does not always force those libraries to be the ones linked into the final binary. As such, the initial link stage failed, as it tried to pull in the LAM libraries from /usr/lib, which are from version 7.1.2 and known to be broken for static compilation. However, I could call mpic++ -showme to print out what it was going to do for the link step, and replace -llam and friends with absolute paths to the libraries I wanted. It was this resulting binary that ran successfully.) If I repeated that procedure exactly, changing only which LAM compiler and libraries I was using, the resulting executable suffered a runtime error. By using a modular debugging technique to track down in which component(s) my bug existed, I greatly reduced the complexity of the problem, and knew which external resources to reach out to for help.
I sent mail to the LAM user list describing my issues, expecting to have a few rounds of back-and-forth of testing small programs and applying patches, but instead received a reply to the effect of “LAM is dead — the developers have moved on”. The respondent suggested OpenMPI as an alternative, and in less than a day’s work I was able to compile and install OpenMPI back-end-ing to the Intel compilers, recompile the MPI fftw libraries against that OpenMPI, and compile my software. It runs just fine.
So, now I can move on to doing more interesting things with my time. But I’ll never know what the actual bug was … and I won’t really get my time back.

As I mentioned last time, I’ve been working to upgrade the software and development environment on our research group’s new login node.
One of the main troubles that when going from a serial-execution code to the MPI parallelized compiler, linkage errors popped up about the symbols fftw_version, fftw_malloc, and fftw_free. In addition to being defined in the fftw library, these symbols are also defined in an Intel MKL library. For some reason the change of compiler caused complaints about the duplicate symbols; I don’t see how the addition of the MPI additions to the fftw libraries could have affected this, as they do not involve the relevant symbols (as verified by objdump(1)). After playing around with the configure options for both fftw and lam (the MPI library), I eventually broke down and compiled a version of the fftw libraries that did not define those three symbols. This actually turned out to be quite more exciting that it ought to have been … it turns out that fftw likes to compile and run some test programs as part of its build. When you go and remove symbols from its libraries, these programs don’t like to compile, which causes the build to fail. It doesn’t help that the build system is quite autogoo-ified, making it hard to manually follow dependencies and see which make targets (if any) would just build the libraries and not test them. I ended up running a full build of the clean package (using debuild to get the debian packaging additions), keeping a log of the build to a file. Then, I went in and removed the definitions of these three symbols, replacing those statements with ‘extern’ declarations of them. After removing the compiled binary files of interest (and the final libraries), I failed to find ‘make’ invocations that would build the libraries I was really interested, so I just ran ‘make malloc.o’ and such by hand, and then copy/pasted the ar invocation to create the static libraries I wanted from the log of the normal build. I could then manually copy these libraries into a deployed tree (I just copied an existing one for the other libraries I wasn’t modifying), and build the software that has been giving me such trouble. The link step went uneventfully, and then I went to go and actually test this code.
Initial signs were promising, as the test jobs for the code I’ve already written worked okay, but jobs that explicitly tested the fft routines produced all NaNs as the result. (I am assuming that this stems from the MKL fftw_malloc’s failure to sufficiently align the memory it returns for use of the pipelined assembly instructions that fftw uses internally, but have not checked thoroughly.) For bonus points, if I pointed a serial compilation of my software at those libraries, the build failed, claiming that these symbols were not implemented!
This actually led me to note that the FFTW_LIBS were listed after the INTEL_LIBS on the link line, and thus the fftw library not picking up fftw_malloc from the Intel libraries made sense (since we recall that earlier libraries on the link line are not searched for symbols when resolving a particular object). Moving FFTW_LIBS before INTEL_LIBS allowed that static build to finish (though it still produced broken code). More interestingly, leaving FFTW_LIBS before INTEL_LIBS allowed the parallel compilation to link while using the normal (well, except for the underscores) fftw libraries. The most plausible explanation I have is that mpic++ and icpc differ in their treatment of -Wl,--begin-group and -Wl,--end-group, as INTEL_LIBS requires repeated searching to resolve inter-library dependencies, and the original complaint was for multiple implementations within the grouped objects. But that’s kind of a stretch — I’d really like to know what’s actually going on.
Now that the linking is mostly settled out, I have a new problem to deal with — the lam runtime environment for the parallel calculations is bailing on me after only one part of multi-part jobs. The displayed error is “bufferd (getroute): invalid node”. Source-diving the line printing the error is not too hard, but trying to back out where the error originates is not so easy. (The code itself is moderately interesting, doing things such as having functions that return function pointers, which results in a type signature of void(*(bufferd()))().) Playing around with my PATH and the hardcoded calls to mpirun results in several ominous WARNINGS about version mismatch between the runtime and the libraries that the binary were compiled with. I do have some reasons for having multiple version of the lam code installed on this machine, but I think I’ll save those for another post (hopefully one where I have solved all my problems and gotten back to the more interesting business of actually doing chemistry!).