Unable to reproduce problem.

  "... keybindings are declared twice, once for forward and backward of
   one character and the other for panning maybe this is the source of the
   problem?"

No, google_maps_keymap is at the top of the keymap stack, and therefore
any bindings in it shadow bindings in lower keymaps.