(Update: Fix is also available at http://permalink.gmane.org/gmane.comp.file-systems.btrfs/15906)
Symptom
When you're following instructions to debug btrfs with gdb and UML (User Mode Linux), you might come across this error when making btrfs on a file:
error checking <file name> mount status
And here're the steps before:
1. Download latest btrfs-prog(git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs.git)
2. Build the tools
3. Run mkfs.btrfs <image file>
Root cause
If you're sure that you just created the file with dd and not touching it, then it is quite likely you hit an issue in the mkfs.btrfs code which works well on later linux kernel.
mkfs.btrfs will check if the target device/file is mounted anywhere or it is formatted in btrfs before it do any write. One last thing is to check that the file is not backing a loop device. In utils.c, it resolves the loop device by parsing /sys/block/<loop name>/loop/backing_file, which doens't exist until 2.6.37 (I'm running on 2.6.29). Failure in opening the file results in failure of the whole operation.
Fix
Unmount any loop device
I'm not going to file any bug request or provide any fix myself, due to the fact that this issue won't happen in the real environment (should be running a later linux kernel). So just put it down with the workaround for your reference.