Tri Ky on Nguyen Duc Thuan St. offers tasty, fresh seafood at reasonable prices, and has an English menu available, although offerings are relatively sparse. If not well-versed in Vietnamese, the "pick and point" method is usually best for communicating to waitresses what seafood you want.
For lunchtime, you can also find some local restaurants that serve good noodle soups for 15000 dong. One is located on Nguyen Hue street, after the crossing with Vo Thi Sau street, when you drive towards East on the left side.
Most of the larger hotels have restaurants attached to them. The Seatravel Con Dao resort has an open-air lobby that doubles as a restaurant, and offers outside seating with excellent views of the ocean. Food prices are somewhat pricey (for Vietnam), but that is the case with most of the hotel restaurants on the island. English menu available, however, some of the better foods, such as their beef and veggie pho, are not listed and usually best ordered by simply asking the waitstaff directly. Western-style breakfast omelette is excellent as well.