Amp meter will tell you how much power you are pulling. A 5Hp motor should be around 39.1 amps at 120 volts plus the VFD circuity adds a little more because transference isnt 100% efficient . Kinda weird that its only after 20 minutes, that makes me assume that something is heating up the circuit; increasing resistance and further heating the wire increasing resistance again, etc, that causes the overload. Also you didnt say what amperage for your sub panel, or what other loads are on it, assuming your VFD is set up correctly I would look into more mouse connection problems with all those splices, If it is wired correctly the mice should not be able to get near the wiring