Subversion hanging?
Today, I received a report of subversion hanging any time commands were run against a specific repository. Upon stack tracing the process, it was clear that subversion was waiting on a lock to be released within the underlying Berkley DB. Below is a copy of the end of the stack trace (just before the process hung):
open(“/home/user/repos/db/strings”, O_RDWR|O_LARGEFILE) = 10
fcntl64(10, F_SETFD, FD_CLOEXEC) = 0
read(10, “\32\0\0\0\323\337\3\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0”…, 512) = 512
close(10) = 0
stat64(“/home/user/repos/db/strings”, {st_mode=S_IFREG|0644, st_size=12304384, …}) = 0
open(“/home/user/repos/db/strings”, O_RDWR|O_LARGEFILE) = 10
fcntl64(10, F_SETFD, FD_CLOEXEC) = 0
fstat64(10, {st_mode=S_IFREG|0644, st_size=12304384, …}) = 0
brk(0x9c11000) = 0x9c11000
futex(0xb7c2e704, FUTEX_WAIT, 1, NULL
Even running svnadmin commands such as verify and rmlocks caused the same result (hung process at the same point). I used the db_recover tool that is included in the db tools package to resolve the issue. Below is a copy of the steps required:
-bash-3.00$ cp -raf repos repos.orig
-bash-3.00$ db_recover -h repos/db/
-bash-3.00$ svnadmin verify ./repos/
* Verified revision 0.
* Verified revision 1.
* Verified revision 2.
* Verified revision 3.
* Verified revision 4.
* Verified revision 5.
* Verified revision 6.
* Verified revision 7.
* Verified revision 8.
* Verified revision 9.
* Verified revision 10.
* Verified revision 11.
* Verified revision 12.
* Verified revision 13.
* Verified revision 14.
-bash-3.00$
Have I recommended lately that everyone consider the newer fsfs backend of their subversion repository? We have found it to be very stable and reliable in our use of it. Subversion even provides simple instructions for converting from bdb to fsfs
Comments
-
I still had it hang once with fsfs but its much better than the before.