I have a suspicion that this happens because the cursor 'focus' remains in the previous buffer after
switching to a new buffer, in which case the focused element should be updated on buffer switch. I
haven't confirmed this, but it is worth looking into. (I notice that in Firefox when you Ctrl+Pg[Up|Down]
to a new tab, then hit space to advance the page, the page in the buffer-switched-away-from scrolls
rather than the page in the current tab.)