Simon Cowell's FORMER good buddy claims Mr. 'X Factor' was sticking it to his wife before they parted ways ... and THAT'S why he's accusing Cowell .
We know real estate mogul Andrew Silverman isn't buying what Simon is selling -- that he didn't lay a finger or any other part on Lauren Silverman until after the split.
Andrew, we're told, just wants out of the marriage and away from Lauren and Simon, and he's willing to settle custody and financial issues quickly. For now, he's asking for full custody of their 7-year-old son, but it appears from what we know that's a negotiable point.
A source close to Lauren tells TMZ ... Lauren and her husband were over before she and Simon entered into any kind of relationship.
Adultery aside ... we checked with a bunch of English folk who tell us the "Bro Code" extends to the U.K., and Simon squarely violated it.
