Just keep trying. Try giving him the worms first and then the crickets. Some chameleons are just picky about what they eat. Did you try letting the worms crawl around on the branches?
If he's eating crickets then that's a good thing. I've been told feeding them too many worms can cause them to have digestion problems as the shell to flesh ratio ain't as great compared to food like crickets. I don't feed mine worms, just crickets which I occasionally dust with vits and calcium. But vary his diet with locusts or lobster roaches.
soft bodied worms such as silks and horns are fine as long as you mix up his diet with something like crickets or roaches that has some chitin in it. you don't want to feed just soft bodied prey.
My panthers prefer the faster moving food like crickets. On a rare occasion they will eat a silkie. Once they have had one they appear to be more receptive to having another one in the future. So, I guess they remember the good stuff.
As long as he is eating his crickets (Staple food) with his appropriate dustings, he should be ok! My panther refuses to eat anything but crickets... I have tried superworms, hornworms, butterworms, waxworms... Everything. But he is picky and bland!
Your ok ,I had the other way didn't eat crix only worms ,dont leave worms in the cage in the morning before I feed them crix put a worm by him n c if they eat it.