It seems to work fine in a private tab both with and without my extensions activated. So it doesn't seem to be an extension problem. The tabs dump a lot of the memory and consistently stay below 100 MB.
The problem is much more pronounced in "Source" quality. It literally adds like 15 MB per second. In "Medium" quality it manages to handle the memory quite well but it still adds up and doesn't dump enough for the tab to stay small.
Very strange. I've run anti-virus/malware software and the computer is clean.
Any other idea what could cause this? Cookies?