Worried regarding PPF agent misusing my account

I had opened a PPF (public providend fund) account through a post office an year ago, through an agent.

I had deposited money in the account through cheque, via my agent. Also, i had once handed over my passbook to the agent so that he can get passbook entry for deposits made.

However, recently this thought has started coming to my mind.
Can the agent, since he knows my account number, issue a request for duplicate passbook (on the basis of giving false information regarding passbook loss), and hence after obtaining the passbook, withdraw money from my account (after lock-in period) ? I don't think post office ask for any verification like original PAN card/driving license to people looking for issuing of lost pasbook, they just want a form to be filled. And at withdrawal time, they just need the passbook !

I have never used PPF, but I certainly know that nobody would give away your money just on the basis of the passbook. Also, all amounts greater than Rs. 20000 are given by a crossed(a/c payee only) cheque. So even if he tries to withdraw that amount, he will only get a cheque that is in your name.

Dceite for first 6 years you cant withdraw.After that partial withdraw allowed

Also if you really worried why dont you take back your passbook and say you yourself will update it.