I still think clicking the heart a second time or right clicking on it should give you the option to remove it too

There is an "undo" button when you click on the heart icon/menu.