Should not be handled, period.
You will have to move your animal for cage cleaning or to go outside in the sun or to the vet on occasion.
Other than that it should be left alone. It is not a kitten, and yes (regardless of the color) it stresses them.
A lot of people just do what they want regardless of the animal in question.
It makes sense that a singular, territorial animal, that doesn't even like others of it's own species around is not going to appreciate being handled ... excessively or otherwise.
Every single chameleon (if given the choice) no matter what anyone says, would choose to be left alone.