You may should take once your time to listen to this guy: William Greenspan (Legendary Floor Trader):
https://www.youtube.com/watch?v=uLSkiAlQJ0w It is an old video and still it is a very new video. In the first ten minutes or so he is pointing to the problem you mentioned you face.
Coming to your question from my side: Some are great traders with five, ten or twenty lots but are miserable traders with more lots, as they face fear which they not will over come in some cases when trading bigger amounts of lots. If so, then do accept it and do only trade in your "Comfort Zone". Do forget about the rest and stay with the amounts of lots you feel comfortable to handle. Not all are born to trade big amounts of lots. No shame in this instead a good behavior to protect your self against huge losses when it comes to this specific mental point in trading.
Most important: The end result of your trades must be in green as you trade in your "Zone".
Adding this: If what you traded in the past (Trading plan with your trading strategy) did work well and you did slowly increase your size of lots you trade today, then in general you should not feel bad when you trade bigger amounts of lots, as the past has proved the success by what you trade. Do not go to your limits you could trade with, as when you make losses under such circumstances it will hurt.