I recommend you use the speed dial for what it is, a speed dial. Its not a bookmarks storage area.
Your words, and in the devs' ears…
Its for the links you need to open right after opening a web browser but don't want opened every single time by using the 'continue where I left off' option.
They serve different purposes. Entirely different. Pages opened where I left them last time are the ones I work with and want to keep working. GitHub issue trackers, WoWHead wishlists, corporate Wiki.
The speed Dial is for sites i visit often. The key word is "visit" as opposed to "work with". Different forums or links to normative references… I open the site, check for new messages or for relevant norms, and close it as it do not have anything left for me to keep it open any longer.