Hi Igor!
I'm also having the same problem, especially on google docs and what not.
As another mac user thats been using Opera for a bit now, the only thing that I can recommend you to do is:
- not be in full screen mode
If you know what im saying, go ahead and just ignore this.
Basically what I would do is enlarge opera to fit the screen to its maximum BUT not be in full screen mode.
That way, when you press ESC button especially on websites like google docs/sheets, it won't become a small window.
I also find that this is quite annoying, so another thing I would also do is have 2 'desktops' open if you know what I mean, so you can switch to the other desktop to open other apps like finder and the like.
I know that it's not the best solution, but I hope that it helps a bit.