In his book A Grace Disguised: How the soul grows through loss Gerald Sittser talks about the concept of grief and how it enlarges our souls. Sittser's mom, wife and daughter were all killed by a drunken driver so I trust what he says about going through that process. In fact, if you are going to read one book on grief make it this one.
Sittser says this, "The challenge [of catastrophic loss] is met when we learn to take the loss into ourselves and to be enlarged by it, so that our capacity to live life well and to know God intimately increases."
The losses surrounding your affair are painful. You have a choice in how you will deal with them. However you choose to deal with them those losses are part of your story now and there is appropriate grief that goes along with that. Couples that heal and grow through an affair learn to express that grief together. In so doing both of their souls are enlarged by accepting the pain, loss and change brought by the affair.
Facing the pain gives us the, "...capacity to live life well and to know God intimately." It's a truth I'm not happy about, but nonetheless God seems to have rigged the world this way.