In order to avoid crowds at some of the popular locations such as The Baths and White Bay, it is smart to check on the cruise ship schedules and avoid the days where ships are in port.  Check out the schedule here:

 

BVI Ports Cruise Schedule