What to do? Should I end it? very confused.?

I am am having a lot of trouble figuring out what to do. I am with a guy I love more than anything. now this may take a while as its a bit complicated. First up I have 2 young children from a previous relationship, I work nearly full time and am the bread winner for the family, i pay the rent, power, ph, food as well as try to save to clothe my kids etc. My partner of nearly 18 months, lives with me, his work has been a bit slow recently and has not been bringing in any money, but i was happy to support him as i love him and this is what couples do. Anyway when he does get money he will pay off some of his debt he owes but also thinks its ok to go to the pub etc etc and not really contribute to the household. he will give me around $500 maybe every 3 months which is not much when i am paying $2000 per month at least in bills and food. He will take a day off work (he is his own boss) so he can play golf or go bush for a day, this really annoys me. but the main issue is, he has 2 children to 2 different mothers, they live in another town, one is still madly in love with him, she stayed with him one night and they had sex and she got pregnant, she was trying to get pregnant as she knew that he didn't really want anything more to do with her so this was her way of holding on, he doesn't really have much to do with her or the kid coz of the mind games she plays. The other ex who has his 1st child was really nice, we become friends and all of a sudden about 8 months ago she stopped calling me, we saw each other on the child's birthday and that was ok, but are no where near as close as what we were. My relationship with this guy is more serious than what he had with the other 2 and the one i was friends with has realized this and doesn't like it much. Anyway she trys to get my partner to stay at her place when he goes to pick up the child, as we have the child every 2nd weekend. I told him i don't feel comfortable with him staying at his exs place regardless of if she is there or not, she says she will go and stay with friends for the weekend. Im in a different town and their is no way of knowing this. she got all shitty last weekend coz he didn't stay there and brought his child back here which is what we have been doing for the past year, but now she has gone and put their child in swimming lessons on a sat, which means that on our weekend to have the child its a waste of time, him going 3 hours to pick him up bring him back only to drive all that way the next day to drop him off, so it means he will most likely stay at her place as he cant afford to stay in a motel, I trust him but i don't trust her, she has already made it clear she never loved anyone like she did him. she gets her digs in whenever possible. So she text him the other day and said, that she had put their son in classes on a sat so he needs to either cut his time short with their child (which she knows he wont do0 or sort something so he stays there. I cant handle the thought of him staying at her place, i know it will cause arguments between me and him, and he says im being selfish and unreasonable. I would never stand in the way of him seeing his kids. I have never asked anything of him except to not stay at his exs place and to be honest with me about everything. What do I do? Im not going to make him choose between me and his child and therefore thinking seriously about letting him go so it saves all the arguing that's about to come. My family have all said he just takes me for granted and its a free ride but i really love this guy. Please someone give me your advice, I want genuine replies only please. thanks in advance