The part that takes longest is usually module loading. Unless it's in the cache, Windows has to load the DLL from the harddrive, rebase it, resolve the dependencies (ie. load all other DLLs that the module depends on), run initialization routines (CRT, global constructors, DllMain, ...), etc. Modules like label (if you use lots of [vars]) are especially bad.

If you want to speed up your "sub-theme switching" you should keep all those modules running and !(net)reloadmodule the few that are not used in all sub-themes. Or, if you really switch between them all the time, keep all modules from all sub-themes loaded.