When St Joseph discovered that Mary, his wife, was with child, he did not in any way suspect her of adultery or sin, but he recognized that there was a mystery present beyond his comprehension. However, this at first troubled him so that he thought he would lose Mary - but the vision of the angel calmed all his fears and filled him with so much joy.
We ought not to suppose (as some do, in a rather impious way) that St Joseph was angry with Mary or that he accused her of sin or that he wanted to hurt her in any way. None of this is present in Scripture! What we see from the Bible is that St Joseph and Mary had made vows of virginity even before their marriage, and that they were both most committed to each other -- between Joseph and Mary there is nothing but love and fidelity!
Listen online [here]!