OK. So I got it to go away. I went to the new tab/speed dial. Then customize start page, and selected the drop down for the side bar. Then toggled "History" on off a couple of times and it went away.

Very odd.