He misses his ex.
Does he mention his ex a lot? Does he share a lot of her stories with you? If this is the case, then he is too caught up in his past and he isn’t really interested in you. He got with you assuming that he was over his last affair and he would now have a good time but it failed and now, he’s stuck with you so he treats you badly because he doesn’t have other options.