Nothing much to do in Villefranche although it was a pretty little town with nice views. Nice is only 15 minutes away and that's where you should go if you have enough time.
I wouldn't attempt this port without an excursion or private tour. Luckily we had a relative who was visiting his mother in Tunis at the time, and he picked us up and showed us around. It's a fascinating part of the world!