the fruit is edible- we have mulberry trees and we eat the fruit all the time when it's in season. I feed the leaves and berries to many of our lizards that will eat such things...
Is it possible? maybe. As evolution go's, I'd have to say that the silkworms avoid the trees fruit, and stick to the leaves that their mouth parts seem taylor made for. You don't see too many worms associated with berry eating. Simply put, berry fruits attract birds and animals. So the silkworms in many cases would make a fine menu addition. Worms are much safer munching away on the underside of a leaf, and out of sight. I believe feeding fresh deep green mulberry leaves, work pretty good at passing on trace elements and nutrients