/usr/local/etc/rc.d/afsd onestart
/usr/local/etc/rc.d/afsd forcestop
/usr/local/etc/rc.d/afsd start

with multiple variations on the “start” and “stop” commands. In order to be started, the appropriate rc.conf variable must be set to enable that service; onestart is a way to start it manually regardless. In order for this to work, each rc script has to define several shell functions that hook into the (massive) rc.subr (that’s “subroutine”) infrastructure. Here’s what I ended up with in files/afsd.in:

#!/bin/sh
#
# we require afsserver for the (rare, untested) case when a client
# and server are running on the same machine -- the client must not
# start until the server is running.
#
# PROVIDE: afsd
# REQUIRE: afsserver named

These keywords are used to order all the rc scripts on system startup (and shutdown) — dependencies are declared explicitly.

. /etc/rc.subr

name="afsd"
rcvar="afsd_enable"
start_cmd="afsd_start"
start_precmd="afsd_prestart"
stop_cmd="afsd_stop"
command="%%PREFIX%%/sbin/${name}"
kmod="libafs"
vicedir="%%PREFIX%%/etc/openafs"

The reason for the .in suffix on this file is because it has variable substitution applied to it. Here, %%PREFIX%% gets expanded to the current prefix that the port is being built with; this is usually /usr/local but can be other things.

load_rc_config "$name"
eval "${rcvar}=\${${rcvar}:-'NO'}"

This is us checking if we’re listed in rc.conf; default to disabled if not mentioned.

afsd_prestart()

This is one of the functions that hooks into rc.subr — AFS requires several configuration files and a kernel module to be in place before it can start, so we check that they’re all there and give a useful error if not. This is quite helpful for users who are not familiar with how to start afsd manually.

{
        # not going very far without a kernel module
        if ! kldstat -qm afs; then
                echo "Loading AFS kernel module..."
                if ! kldload $kmod; then
                        echo "Failed to enable kernel support. Aborting."
                        return 1;
                fi
        fi
        # now we have a kernel module; check for conffiles
        for file in cacheinfo ThisCell CellServDB; do
                if [ ! -f ${vicedir}/${file} ]; then
                        echo "${vicedir}/${file} does not exist.  Not starting AFS client."
                        return 1
                fi
        done
        # need a mountpoint and a cache dir (well, if we have a disk cache)
        for dir in $(awk -F: '{print $1, $2}' ${vicedir}/cacheinfo); do
                if [ ! -d ${dir} ]; then
                        echo "${dir} does not exist. Not starting AFS client."
                        return 2
                fi
        done
}

afsd_start()
{
        # you probably don't want to change these
        afsd_default_args="-memcache -dynroot -fakestat-all -afsdb"
        # either set explicit extra args or just a size; default medium
        afsd_args=${afsd_args:-'MEDIUM'}
        case ${afsd_args} in
        LARGE)
                afsd_args="-stat 2800 -dcache 2400 -daemons 5 -volumes 128"
                ;;
        MEDIUM)
                afsd_args="-stat 2000 -dcache 800 -daemons 3 -volumes 70"
                ;;
        SMALL)
                afsd_args="-stat 300 -dcache 100 -daemons 2 -volumes 50"
                ;;
        esac
        ${command} ${afsd_default_args} ${afs_args}
}

The actual start function. We check to see if we’ve been given extra arguments, using sane defaults if not. There are also a few things that you basically always want (for example, non-memcache is currently broken), which are listed in a local variable.

afsd_stop()
{
        afsdir=$(awk -F: '{print $1}' ${vicedir}/cacheinfo)
        umount ${afsdir}
        _return=$?
        [ "${_return}" -ne 0 ] && [ -n "${rc_force}" ] && umount -f ${afsdir}
        kldunload ${kmod}
}

Stopping does not actually involve touching afsd at all — those processes will happily ignore whatever you throw at them. We must check that AFS is mounted (as someone might be erroneously running onestop), and then just run the umount command to stop things. We also check for whether force is being used, passing that on to umount if needed.

run_rc_command "$1"

This last line is very important! It looks very mundane, but it is how we actually interface with the rc system; rc.subr defines this function, which does all the necessary variable-name munging and calls the appropriate function(s) that we have defined.

At install time for the port, the substituted variables are replaced, and the script is installed into ${PREFIX}/etc/rc.d and added to the package list to be removed at deinstall time. All in all, we wrap a standard interface around the complicated afsd semantics.

    if (afs_globalVp && (afs_globalVp->f.states & CStatd)) {
        tvp = afs_globalVp;
        error = 0;
    } else {
tryagain:
        if (afs_globalVp) {
            afs_PutVCache(afs_globalVp);
            /* vrele() needed here or not? */
            afs_globalVp = NULL;
        }

The afs_PutVCache function sleeps, dropping the GLOCK in the process. So it is possible for some other thread to have come into this block of code at the same time, and also try to call afs_PutVCache on afs_glovalVp. When this happens, the core vputx routine used to implement afs_PutVCache sees that the reference count on the global vnode entry is negative, which violates an invariant of the VFS layer. (This means a kernel panic under my debugging options.)
The fix, then, is to make sure that all changes to afs_globalVp involve conditions and actions checked while the GLOCK is held and no sleeps have been made. If the thread sleeps, conditions must be re-checked.
For this block of code, the change is easy — set afs_globalVp to NULL before calling afs_PutVCache (storing the value in an intermediate variable), so that other threads will see that the removal has been queued, even if it has not actually taken place, yet. However, this is not the only bug of this form in the afs_root function — a reminder that great care must be taken with locking strategies, and that sleeps can come in surprising places.

afs_vop_reclaim: afs_FlushVCache failed code 16

Code 16 is EBUSY, which is actually returned from several places in that function. Placing print statements before all of them, and triggering the bug again, reveals that the reference count of the vnode (as determined by AFS’s VREFCOUNT() macro) was too large, implying that someone else was probably trying to use that vnode. For extra fun, later on the buildworld step would fail, usually claiming that it could not find a particular header file. Using the fs getfid command (from a different computer), it was clear that the file existed, and the fid used to identify that file is the same as the one that we failed to flush properly. Clearly, this bug was leaving a corrupt vnode floating around, and this corruption was later crashing the build.
Now, what does VREFCOUNT actually do? The relevant block of code is

 665 #if defined(AFS_XBSD_ENV) || defined(AFS_DARWIN_ENV)
 666 #define vrefCount   v->v_usecount
[...]
 673 #elif defined(AFS_XBSD_ENV) || defined(AFS_DARWIN_ENV)
 674 #define VREFCOUNT(v)          ((v)->vrefCount)
 675 #define VREFCOUNT_GT(v, y)    (AFSTOV(v)->v_usecount > (y))

(which is rather ugly); both VREFCOUNT and VREFCOUNT_GT use the v_usecount field of the vnode associated with the given vcache. Now, FreeBSD has a locking strategy for vnode elements (and for the vnodes themselves, but that’s getting ahead of ourselves), which is laid out in the sys/vnode.h system header.

/*
 * Reading or writing any of these items requires holding the appropriate lock.
 *
 * Lock reference:
 *      c - namecache mutex
 *      f - freelist mutex
 *      G - Giant
 *      i - interlock
 *      m - mntvnodes mutex
 *      p - pollinfo lock
 *      s - spechash mutex
 *      S - syncer mutex
 *      u - Only a reference to the vnode is needed to read.
 *      v - vnode lock
[...]
        /*
         * Locking
         */
        struct  lock v_lock;                    /* u (if fs don't have one) */
        struct  mtx v_interlock;                /* lock for "i" things */
        struct  lock *v_vnlock;                 /* u pointer to vnode lock */
        int     v_holdcnt;                      /* i prevents recycling. */
        int     v_usecount;                     /* i ref count of users */
        u_long  v_iflag;                        /* i vnode flags (see below) */
        u_long  v_vflag;                        /* v vnode flags */
        int     v_writecount;                   /* v ref count of writers */

As we can see, the locking strategy requires that accesses to v_usecount hold the vnode interlock; OpenAFS was failing to do so. Conveniently, there is a wrapper function vrefcnt() that takes the interlock, reads the use count into a local variable, and then returns the local variable. Changing the VREFCOUNT macros to use this function did eliminate the console warnings about afs_FlushVCache returning EBUSY … but it did not fix the buildworld. Still, we get compilation errors stemming from files that are mysteriously “missing”. The fix involves more locking, but the story is a bit more involved than I have space left, here; it’s on tap for next time.

I got around to doing some research, and none of the other systems I tested made a failure to represent the uid a fatal error: Linux simply truncated it to 335547, Solaris capped it at 600001, and OS X took the remainder modulo some power of 2 (between 8 and 25), leaving it as 217. FreeBSD’s libarchive infrastructure makes this a fairly easy patch to write, and I’m currently testing a patch that makes this condition non-fatal for submission to upstream.