I love Bermuda. It doesn't seem to have the poverty plaguing so many other island destinations and at no time did we feel unsafe. Forget the cruise ship tours and buy a 3-day transportation pass. The busses will take you to Horseshoe Bay beach, the Ferry goes to Hamilton for shopping and St. George if you want go to the Crystal Caves, Tobacco Bay Beach, or shop there too.
One night we decided to go our own separate ways and have a "date-night" dinner. We took the Ferry into Hamilton and had dinner at Red (a new steakhouse which was wonderful). Unfortunately we missed the last ferry back to the ship and took the bus instead. Although longer, the bus ride back definitely better than a $50 cab ride. The bus station in Hamilton was a five minute walk from the restaurant, and the Bermudians are extremely nice and helpful when it comes to giving directions (which bus to take home, etc.) Of course, being cautious is prudent no matter where you go. But you shouldn't let fear or apprehension keep you from experiencing the island on your own.
Kings Wharf is pretty sparse when it comes to restaurants and shopping, but there are some nice choices there. This is a nice area to take a couple of hours to walk and explore on the last day of the cruise. There is a beach nearby, but there's a $10 cover charge to get in and the reviews I've read are spotty at best.