Things have been kind of locked up, here, for the past few weeks. But I have finally gotten around to getting some interesting work done, on the OpenAFS front. During the long run of a “buildworld” in AFS, I would eventually get a warning on the console that the afs_vop_reclaim() function (i.e. the actual function that gets used when the VOP_RECLAIM() operation is performed on a vnode of type “afs”) had hit an error condition where the routine that should have removed all AFS content from that vnode failed:
afs_vop_reclaim: afs_FlushVCache failed code 16
Code 16 is EBUSY, which is actually returned from several places in that function. Placing print statements before all of them, and triggering the bug again, reveals that the reference count of the vnode (as determined by AFS’s VREFCOUNT() macro) was too large, implying that someone else was probably trying to use that vnode. For extra fun, later on the buildworld step would fail, usually claiming that it could not find a particular header file. Using the fs getfid command (from a different computer), it was clear that the file existed, and the fid used to identify that file is the same as the one that we failed to flush properly. Clearly, this bug was leaving a corrupt vnode floating around, and this corruption was later crashing the build.
Now, what does VREFCOUNT actually do? The relevant block of code is
665 #if defined(AFS_XBSD_ENV) || defined(AFS_DARWIN_ENV) 666 #define vrefCount v->v_usecount [...] 673 #elif defined(AFS_XBSD_ENV) || defined(AFS_DARWIN_ENV) 674 #define VREFCOUNT(v) ((v)->vrefCount) 675 #define VREFCOUNT_GT(v, y) (AFSTOV(v)->v_usecount > (y))
(which is rather ugly); both VREFCOUNT and VREFCOUNT_GT use the v_usecount field of the vnode associated with the given vcache. Now, FreeBSD has a locking strategy for vnode elements (and for the vnodes themselves, but that’s getting ahead of ourselves), which is laid out in the sys/vnode.h system header.
/*
* Reading or writing any of these items requires holding the appropriate lock.
*
* Lock reference:
* c - namecache mutex
* f - freelist mutex
* G - Giant
* i - interlock
* m - mntvnodes mutex
* p - pollinfo lock
* s - spechash mutex
* S - syncer mutex
* u - Only a reference to the vnode is needed to read.
* v - vnode lock
[...]
/*
* Locking
*/
struct lock v_lock; /* u (if fs don't have one) */
struct mtx v_interlock; /* lock for "i" things */
struct lock *v_vnlock; /* u pointer to vnode lock */
int v_holdcnt; /* i prevents recycling. */
int v_usecount; /* i ref count of users */
u_long v_iflag; /* i vnode flags (see below) */
u_long v_vflag; /* v vnode flags */
int v_writecount; /* v ref count of writers */
As we can see, the locking strategy requires that accesses to v_usecount hold the vnode interlock; OpenAFS was failing to do so. Conveniently, there is a wrapper function vrefcnt() that takes the interlock, reads the use count into a local variable, and then returns the local variable. Changing the VREFCOUNT macros to use this function did eliminate the console warnings about afs_FlushVCache returning EBUSY … but it did not fix the buildworld. Still, we get compilation errors stemming from files that are mysteriously “missing”. The fix involves more locking, but the story is a bit more involved than I have space left, here; it’s on tap for next time.