Another thing I might try is running FTM from a different user account in Windows to see what happens. If it runs successfully, it might reveal something about where the problem lies.
You might also consider calling into phone support.
Edit: Another idea below
Another idea is to start FTM as you do normally. When you get the message about compacting, don't click anything in it. While it's still up on the screen (and presumably you can see FTM.exe running in your task manager), try to start another instance of FTM. The first instance might lock the tree, so the second instance may open successfully if the problem lies with that tree.
You might also consider calling into phone support.
Edit: Another idea below
Another idea is to start FTM as you do normally. When you get the message about compacting, don't click anything in it. While it's still up on the screen (and presumably you can see FTM.exe running in your task manager), try to start another instance of FTM. The first instance might lock the tree, so the second instance may open successfully if the problem lies with that tree.