First of all you need to determine which popup modules the themes in question are using (it's either popup2.dll or xpopup.dll). Then you need to add one of these lines to the theme's config file(s):

(if popup2)

(if xpopup)

Keep in mind that if you turn icons on you might also have to adjust text margins and icon size/position, otherwise your popup might end up looking really funky. Take a peek in the popup2/xpopup docs for further info.