Regular and chaotic classical dynamics in the U(5)-SU(3) quantum phase transition of the IBM