Worms

In the Autumn l planted a patch of mustard 5x3m. Recently I turned it over and dug it in a full shovel deep but did not see any worms, and have not over the years. I’ve put in compost and horse manure and I have plenty of weeds. Should I be worried?