Comity of errors

Having finally made some advances on the OpenAFS front, I had achieved a state that was able to copy, read, and write a large data set without error, hang, or crash. However, I was unable to run executables from AFS, which presented a serious obstacle to passing the lazy man’s filesystem stress test: ‘make buildworld’. This target recompiles from scratch an entire build toolchain, and uses that (updated) toolchain to rebuild the entire operating system from scratch. As such, it can put a fair bit of load on a filesystem (and a CPU, for that matter).
Asking on the freebsd-fs@FreeBSD.org mailing list, a simple suggestion was made that would account for the displayed symptoms. (This involved two different mechanisms for tracking what is effectively a file’s size, and only one of them being updated.) After applying that fix, and a workaround for some locking issues, I now have an OpenAFS installation that can survive the buildworld informal stress test.
To be fair, it’s not perfect — attempting a parallel make with simultaneous compilation processes still causes a deadlock, but it’s a big milestone, and cause for some celebration.

Continuing on from last time, I had made a quick overview of conical intersections and density functional theory (DFT). I also promised more about Configuration-Interaction methods (Wikipedia) and the new method developed in our research group, Constrained DFT-Configuration Interaction. But first, a bit about electronic correlation.
The standard way to write an electronic wavefunction is as a Slater determinant — I have N (identical) electrons that I need to put in N one-particle orbitals, and maintain the antisymmetry required since electrons are fermions. The natural way to do this turns out to be to write an NxN matrix, with increasing electron index along the columns, and increasing orbital index along the rows. The determinant of this matrix is the simplest N-electron wavefunction from that given set of one-electron orbitals that obeys the required symmetry. However, this is a fixed description of a snapshot of an electronic configuration, that ties down the electrons in particular places. Unfortunately, this is not always sufficient to describe a system, as electrons (with their wavelike nature) can be quite tricksy. The most notable cases when this occurs are when there are multiple degenerate (or nearly-degenerate) configurations for the electrons of the system that have the same (or nearly so) energy. In this case, the actual wavefunction is a linear combination of the Slater determinants corresponding to those configurations. The classic example is when a bond is being broken — dihydrogen at large internuclear separations should have one electron at each nucleus, but in a Slater “determinant” (no actual determinant being needed since there is only one electron of each spin), we have to pick one nucleus to have the spin-up electron and the other to have the spin-down one. Due to the quantum uncertainty principle, we can’t know in advance, and thus spin-up and spin-down must be in a linear combination until the wavefunction is collapsed. This need for multiple determinants is the result of “static correlation”, where there are multiple electronic configurations to be accounted for. There is also “dynamical correlation”, which reflects the fermionic tendency of the electrons to not be in the same place, i.e. to avoid each other. This is sometimes described as the time-dependent motion of the electrons to avoid each other, though this is not exactly right. If this all seems a bit confusing, don’t worry too much — there isn’t a precise way to distinguish “static” and “dynamic” correlation, and in fact, the correlation energy is usually defined as the difference between the Hartree-Fock energy (that is, the best single-determinantal energy) and the exact energy (in a given basis set). That’s not really a very useful description, but we make do. In any case, the takeaway point is that multiple determinants or electronic configurations are needed to accurately treat static correlation.
DFT has been a very popular method because of its ability to describe dynamical correlation (usually) with the same expense (or less!) than Hartree-Fock, which does not treat any dynamical correlation. The main idea of the CDFT-CI method is to add static correlation to a DFT method. It does so using Configuration Interaction, a method that starts off with a (usually large) set of electronic configurations/determinants and produces improved representations of the ground (and excited) electronic states as linear combinations of those states. (I will go on to call these states “basis states”, not to be confused with the one-particle basis functions that are used to create the one-particle orbitals that make up each determinant.) The details of the method are actually conceptually fairly simple: create a representation of the system’s Hamiltonian in this basis of states, and then diagonalize that Hamiltonian to produce its eigenstates. To do this, in addition to our basis states, all we need is the off-diagonal terms of the Hamiltonian, which we can think of as electronic couplings between the different states/determinants. For somewhat technical reasons, these couplings are easily obtained for determinants that correspond to Hartree-Fock wavefunctions, but there is less theoretical backing for using Kohn-Sham wavefunctions (which are what DFT provides) in computing these couplings. One of the main features of CDFT-CI is a somewhat clever way to compute these couplings.
Another feature of CDFT-CI is that the basis used for the configuration-interaction (which can easily reach tens of thousands of states for standard CI methods) is quite small, involving only three or four states for the systems we treated in our paper. To generate these states, instead of just taking the standard DFT ground state (which would not provide a very interesting basis set when taken multiple times), we apply a constraint, forcing charge or spin to localize on parts of the molecule. These calculations are still ground-state DFT calculations (just of a slightly perturbed system), and as such retain the accuracy and dynamic correlation benefits of DFT. However, if we choose these perturbations correctly, we can get a lot of the character of the lowest few excited states; combining a few such constrained calculations and taking a linear combination allows the ground state to be removed and each excited state in turn to be amplified, as the several eigenvectors of the CI Hamiltonian.
And that’s CDFT-CI: take a few of these tweaked DFT calculations, compute couplings between them, and diagonalize the CI Hamiltonian; out come energies that correspond to the ground and lowest few excited states. There’s dynamical correlation from DFT, and static correlation from the configuration-interaction. All in all, it’s enough to make a pretty picture of a conical intersection, which you can see in the paper I mentioned at the start of last week’s post.

At the moment, the article that my advisor and I wrote is on the front page of The Journal of Chemical Physics. Entitled “Conical intersections using constrained density functional theory–configuration interaction”, we discuss how we have used a computational method previously developed in the research group to study conical intersections (CIs). I mentioned these in a previous post, but didn’t do much other than link to Wikipedia to describe them. Essentially, one can think of the problem of electronic structure (the branch of computational chemistry/physics in which I work) as being a question of describing a many-dimensional hypersurface which maps the electronic energy as a function of the position of the atoms that make up the molecule. (There are complications, of course, but we’ll ignore them for now.) There are actually infinitely many possible values of the electronic energy for any given nuclear position, corresponding to electronic excited states; each such excited state (as well as the non-excited ground state) traces out its own manifold in this large-dimensional space. Usually, these surfaces are well-separated, and the distance between them corresponds to the amount of energy in a visible or soft ultraviolet photon. This means that when we shine a light (usually a laser when we think about it, but any light will do) with about that much energy per photon onto our molecule, we can start to excite electrons from the ground state to an excited state. The energies do need to be fairly close, as the coupling between states has an inverse dependency on the [energy gap between states - photo energy] quantity. Anyway, once we have prepared an excited state, there are a few things that can happen. The simplest, is that it just spontaneously relaxes back to the ground state, emitting another photon. If that doesn’t happen, other relaxation pathways are available — the nuclei can move along the particular potential energy surface (that is, the hypersurface corresponding to this excited state) to a lower energy configuration, and emit a photon there. (This is fluorescence, and is responsible for the glows you see under a blacklight — the ultraviolet photons from the lamp are not visible to human eyes, but the lower-energy photons coming from fluorphores after they relax are visible.) There are some other loss pathways back to the ground state that are not easily described at this level of theory, but the final way for an excitation to relax is for it to find a place where two of these energetic hypersurfaces actually cross each other. In these configurations, the molecule can readily transition from the “excited” state to the ground state. These configurations are the conical intersections; many workers, notably David Yarkony, have found that these are actually quite ubiquitous in this many-dimensional configuration space.

Unfortunately, most of the ways to compute the electronic structure (that is, these hypersurfaces) that can actually find these conical intersections between surfaces are quite computationally expensive, scaling as N^6 or N^7 with the number of electrons (or worse). These are usually “wavefunction methods” (Wikipedia has something of a list), methods that explicitly try to describe the (3N-dimensional) configuration of all the electrons at once. An alternative class of methods, the density functional theory (DFT) class (Wikipedia) reduces the scope of the problem down to just a 3-dimensional function (which happens to be something that can be experimentally observed, by methods like X-ray diffraction), the electronic density. All electrons are identical, so we can think of this electronic density as being an average of where all the wave-like electrons happen to be. The advantage in computational cost comes at the expense of some approximations — though DFT is “in principle” exact (as wavefunction methods can be), this exactness relies on a “exchange-correlation functional” (a mathematical object that takes as input a density (as a function of position) and produces as output an energy). This functional is not known, and is unlikely to ever be known in a general fashion, so we must use approximations to it. DFT methods have only been developed comparatively recently (compared to wavefunction methods), and do not really have fully-reliable ways to compute the electronic excited states. Time-dependent DFT (which is kind of neat in its own right, and not exactly what it sounds like, but I shouldn’t digress) is the leading competitor, but it has previously been shown to be quite lousy at describing conical intersections (this is what the Levine reference from our paper is about). Our method, “Constrained DFT-configuration interaction” (CDFT-CI) builds on top of ground-state DFT to form a technique that is capable of describing CIs, which is the main result of our paper.
More about configuration-interaction methods in general, and CDFT-CI in particular, will appear in a later post.

The past couple posts followed my adventures in compiling (and running) a parallel program on an updated system. I have since found a resolution to my problems, though it was not what I expected.
Backing up a little bit, I was experiencing what might nominally have been a bug at the fourth-order of interaction — I had my source tree, a (vastly) updated operating system, updated Intel compilers, the fftw libraries, and the MPI compiler/libraries to deal with. Complicated systems like this can easily lead to ridiculously subtle and hard-to-diagnose issues (of course, they can also have extremely simple bugs, too), so I was not really expecting to just be able to type my issues into google and find an answer. However, I did have a couple of useful diagnostic tools at my disposal — I still had old libraries and compilers from the machine we’re replacing. After struggling with my issues in an ab initio fashion for a while, I made use of these tools, copying over the fftw libraries and LAM toolchain to the new machine. I was then able to isolate my bug to the LAM system — I could compile my program using the new Intel compilers and a new fftw library on the new machine, and it ran successfully. (Lest you think that this was an easy, obvious thing to do, I will point out that it did not go entirely smoothly. The mpic++ compiler wrapper has logic to check that there are LAM libraries where it expects them to be, but apparently it does not always force those libraries to be the ones linked into the final binary. As such, the initial link stage failed, as it tried to pull in the LAM libraries from /usr/lib, which are from version 7.1.2 and known to be broken for static compilation. However, I could call mpic++ -showme to print out what it was going to do for the link step, and replace -llam and friends with absolute paths to the libraries I wanted. It was this resulting binary that ran successfully.) If I repeated that procedure exactly, changing only which LAM compiler and libraries I was using, the resulting executable suffered a runtime error. By using a modular debugging technique to track down in which component(s) my bug existed, I greatly reduced the complexity of the problem, and knew which external resources to reach out to for help.
I sent mail to the LAM user list describing my issues, expecting to have a few rounds of back-and-forth of testing small programs and applying patches, but instead received a reply to the effect of “LAM is dead — the developers have moved on”. The respondent suggested OpenMPI as an alternative, and in less than a day’s work I was able to compile and install OpenMPI back-end-ing to the Intel compilers, recompile the MPI fftw libraries against that OpenMPI, and compile my software. It runs just fine.
So, now I can move on to doing more interesting things with my time. But I’ll never know what the actual bug was … and I won’t really get my time back.