If you do the worms will not fare to well, every frig is different some would kill them others would just about kill them. If you want to slow growth keep them in a cooler place in your house and feed them every other day.
That's the key, feed them less and they don't grow as quick, although they pupate at roughly the same time or slightly longer only smaller....from my experience