What you're noticing is a known problem with the checkbox being broken. To work around it, you have to close down Opera and edit operaprefs.ini in the "preferences" folder. See "Help -> About -> Opera Mail" for the location. Under "[User Prefs]", you have to set (add it if it's not there):

Show Notification For New Messages=0

There's also:

Show Notification For No New Messages=0

if you want to turn off the "No new messages" notification when checking for new mail.

Save the settings to the file and start Opera. Then, never mess with the checkbox again or it'll mess things up and you'll have to edit the file again.

Note though that Opera Mail was discontinued years ago. So, this issue and others will never be fixed.