Shame is such a big topic that we'll cover it in a whole section after we cover intimacy. But lets touch on it here, primarily in the context of sexual intimacy though it certainly impacts all areas.
Shame associated with sex may have come from pornography or with other premature or inappropriate sexual exposure. In some way your innocence and sexual development were not protected.
I see this all the time with the women I encounter in the sexual abuse recovery group I lead. The premature sexual contact inflicted on these women damage their ability to rest comfortably in who they are, soul and body. One significant part of healing for one of the groups was the realization that there was a vast difference in sex each one chose and sex they didn't choose. Sexual abuse was not part of sexual sin. There was great freedom which allowed each woman to more fully enjoy sex in the present with her spouse.
For men we see similar dynamics. I can't tell you the number of times I've heard men who struggle today with porn say they received a subscription to Playboy or Penthouse at the age of 13 from their parent(s) to teach him about sex. What is this, a pornmitsvah? It creates a mindset that sex is a sport or a performance. It damages the mind, creating addictive channels that yearn to be filled with the pungent waters of porn. It is a form of sexual abuse and 95% of men who struggle with sexual addiction have been sexually abused.
Shame can make us hypersexual (oversexualized) or hyposexual (sexually avoidant). Either can lead to low self-worth. Feelings of low self-worth may make it difficult for you to take responsibility for sex and may keep you from sexually validating one another. If there is a gnawing sense internally of valuelessness why would words of affirmation from that internal darkness matter to anyone is the lie.
Given this, it makes it understandable why it is so difficult for some men and women to express their sexuality in marriage with spiritual, emotional and a playful intimacy. It can feel safer to express it outside of marriage where the relationship doesn't matter near as much and can be walked away from at any time.
This isn't an excuse for infidelity, but it does make it more understandable in certain situations.
The goal is to be able to touch and be touched in non-demanding ways. Giving and receiving physical love in a way that allows you to be enjoyed and your spouse to feel enjoyed is healing, and fun and worshipful.