[Distress at missing the congregation due to fear of people’s blame]

Answer (with Allah’s enabling): It is a trait of the believer—indeed, of every person—to ward off accusations from himself. He dislikes becoming known among people—or some of them, or even a single one of them—for a false charge, or even for a true fault. This is human nature: anyone who belongs to this human reality loves perfection and honor and dislikes whatever mars that, and turns away from it; he is unable to rid himself of this nature. The Messenger of Allah—may Allah bless him and his family and grant them peace—was keen to avert what might lead to accusation against him.
It is narrated that two Anṣārīs passed by while he (may Allah bless himand his family and grant them peace) was standing with one of his wives, conversing; he (may Allah bless him and his family and grant them peace) called them and said, “She is Ṣafiyyah.” They said, “Glory be to Allah, O Messenger of Allah! How could we accuse you?” He (may Allah bless him and his family and grant them peace) replied, “Satan flows through the son of Ādam like blood in the veins.” (This is the meaning of the report, not its exact wording.)
From this it becomes clear that distress at the possibility of people’s suspicion is not ostentation, nor a sign of it. A believer must avoid the places of suspicion; indeed, there are reports from the Prophet—may Allah bless him and his family and grant them peace—forbidding and warning against standing in situations that invite suspicion.
Source : Min Thimār al-ʿIlm wa al-Ḥikmah vol.1