You're mixing up your modules. You want rabidvwm. Then, you can just hotkey the !bangs to change desktops like you want.