Please tell me this will not suffice to win custody of my daughter?

My ex and I had a very messy break up just recently. Our daughter is three, nearly four years old. He is still bitter and is threatening to get custody of our daughter. Right now, we have normal joint custody - he has her in the day time because I work (7am - 5pm) and I have her after I finish work. He's threatening to take me to court and take my child away from me on the grounds of neglect because I work. I DO NOT neglect my daughter. I don't know if he's bluffing, I'm getting somewhat scared. He's a lot richer than me and could get an amazon lawyer. I know he wants us to get back together but it's never going to happen.