You have to right click and then click it again to open all and it only works on folders. Don't right click the folder to begin with and you'll never have to worry about accidentally hitting it.
Also, it seems to me you are using the speed dial as a replacement for bookmarks. Thats not really what its intended for. I think you would be much better off using bookmarks, which already allow nested folders. You could easily ad a link to your speed dial that goes to the bookmarks manager (or replace the speed dial with it via an extension).