Should I dump my boyfriend, he never listens to how I feel?

My boyfriend and I have been dating for 8 months now. After 3 months of dating I found out his ex girl friend who is 37 years old is still chasing after him, he is 28 and Im 25. He promised to stop speaking to her, she used to call me and tell me that she spent the night at his place and he would deny and call her and she would say she was trying to get me to dump him. OK fine I accepted that she is a bit crazy so I have decided to ignore her, I think she still calls him but he is not telling me. That led to all our problems, Im paranoid Im always thinking he is with another girl, I get jelous even when he gets calls from other girl, that is the problem on my part and Im working on it as Im trying to trust him. The current problem in our relationship is that he is constantly using my car, I live alone and he lives alone, so we either sleep at my place or his place at leas 4 times a week. Everyday when we come back from work he has to go somewhere the excuse is that we are broke and he is going to try and look for money for petrol and food for the next day. So what happens every evening is that we get home from work and he has to go out and he comes back late or early hours of the morning, its as if he is avoiding spending time with me. These days the sex has also decreased to once a week. I told him about this and he has a way of turning things around and making me seem bad he will say things like I dont appreciate the fact that he is out there trying to get money for us, when it comes to sex he is stressed so I cant expect to get more sex from him. He never wants to spend time with me, he is constantly taking my car, he does not listen to what I say, I will tell him how I feel and the next day there will be no change in the situation. What has also angered me is that I have been at his place since Sunday and on Thursday it was his birhtday, I stayed at his place because I thought he ws trying to save on Petrol as it costs us more petrol when I go sleep at my place and drive to his place in the morning just to go drop of the car for him to use during the day. What hurt me is that he asked me to leave on Thursday (On his Birthday) as he was going to be spending the night with his friends. Ok fair enough I understand, but the manner he did it, it was as if he was tryin to get rid of me. I love this man, I dont want to loose him. I have decided to ignore him today, is this wise, he has sent me messages asking me to call him and I called him earlier an told him Im at work and very busy so I dont have time for the phone. I feel hurt, I want him to call me and apologise but at the same time I feel like I need to just call him. It does not seem like he appreciates me although he says he loves me. Is it wise for me to not speak to him for a few days or just rather wait for him to call. Im scared he will go get another girl who will prbably take all of this. Am I being unreasonable?