Conflict between my children and my partner?

I have had an on and off relationship with a man I fell hopelessly in love with and still love but he lives nearly 300 miles away so the distance doesn't help. After 23 years I left my ex-husband for this man and he left his wife. Both of us were unhappy for a number of years and whilst his ex-wife who I get on with well remains friends with him my ex still feels animosity towards me despite living with his new partner a year after we separated. I thought our partnership would improve once he met someone but in fact it got worse as his new partner didn't want anything to do with his boys so he had very little contact with them despite my trying to be Friends for the children's sake. They were 11 and 13 when we separated so there should have been some willingness to put aside our differences in order to fully support our children. Whether this has affected the children who are now 17 and 19 and made them blame me for the break-up although youngest has said he understands why I left his Dad. My eldest is at university, youngest at college. They want their Mum to be happy but my eldest in particular doesn't like my partner and feels he doesn't make me happy. My unhappiness is there because I feel like piggy in the middle. They don't want to share even a small part of their lives with their Mum when he is around even as far as having dinner together or watching the TV/DVD....They are so out of character.It's not all about the children as I don't ever feel confident that my partner is making much effort with the boys and wants my sole attention. I've tried to explain to the boys and have success with everything else in my life as we are very close and they always say I'm not just Mum but a good friend. They feel he is boring and too old for me. He is ten years older and whilst he doesn't look it he will act older than me and is probably at a different stage of his life. In my heart I want to be with him but my mind tells me that whilst he says he accepts the children he openly says he wants me and that makes me wonder if we can ever be a family unit. His daughters are 9 years older than mine and so have left home. I still feel that until the youngest is at University in another years time that I have responsibilities and can't throw away 17 years of love and support to up sticks, find a new job and live with my partner. He is willing to move and live with me but whilst there is this tension especially when oldest at home I find it difficult to find happiness. I have finished with him so many times and now feel I should let him go and allow him to find someone that ideally hasn't got other commitments such as children/young adults. I know they're not children but they still need support, emotionally and financially. They're Dad has only started to make a small effort with his boys since the eldest went off to University but weeks can go by without any contact and its mainly the boys who contact him. I can't see an end to all of this and its making me ill. Although I'm very sad and tearful when we finish the turmoil I feel in having to choose between my boys and partner diminishes big time but I feel so lonely and sad and keep looking back at the good times i.e. when I spent weekends with him at his place and holidays. Equally on two occasions when he was probably unsure of our relationship he went out with his next door neighbour and another woman. both of these relationships went on for three months or so despite him contacting me when he had only just started seeing them and telling me he loved me. Those were painful times and have also affected the way I feel and think.. It wasn't nice when we got back together but he hadn't quite timed it right with the other women. I felt like it was me he was having the affair with. Whilst I forgive him I can't forget especially when flirting with them in front of me before he went out with them, the eye contact yet that wasn't with me any longer. Any advice would be so appreciated.