@martyfrost What do you call "worms"? Earthworms?
Most people call mealworms and superworms "worms" but they are in fact the larval stage of a beetle and are not worms at all. Silkworms and hornworms are also not "worms" either but caterpillars of moths.
I personally would not feed earthworms. They can carry several parasites, one being lungworms. I don't know if the lungworms carried by North American earthworms will infect chameleons or are species specific to pigs but having dealt with lungworms in imports, I'm not taking a chance because there is no way to clear lungworms from the lungs without killing the chameleon.
If you are talking about feeder bugs such as silkworms, hornworms, mealworms and superworms, they are fine with the exception of meal worms which have too high a chitin level to be fed a lot. Just make sure you feed your feeders well to make them as nutritious as possible.