While the kids are in his care, she may call constantly or use the kids to spy on what's going on between the two of you.
Unless he's willing to set some boundaries, your relationship is going to have a third wheel.
If he doesn't take his duties as a parent seriously -- he is likely to shirk other responsibilities.
When you spend time with him and his children, you might find that you and he have different expectations for good behavior.
He may try to blame the situation on their mother or even say the kids just don't want to come over.