I would at least scrub the lichen off. Not just because of the lichen itself, but because of the other things that could be around/under it and elsewhere on the bark.
Many people bake branches to kill off bugs etc.
Others scrub and then rinse with boiling water.