Mahi, It is good that your short order was rejected. Option shorting is risky game and one should not venture into it untill one understands and ready to accept the risk involved with it.
For shorting 5100 Call, you might need margin of about 30 to 35k.. (chk with your broker directly cause it varies from broker to broker).
More importantly, you need to understand the risk involved in this shorting.
Plz spend time and understand what will be your profit or loss when nifty goes to 5300 / 5000 or 5150 etc.
and what will happen to your margin requirement from broker. Else at some stage, due to lack of margin funding, broker can close your position and you take the whole loss.
Happy and safe trading