If all of the targets are more or less the same distance away, the ships should target enemy ships that they have the best damage modifiers against (the exception to this siege frigates, especially if they are bombing one of your planets). I'm not sure how they choose if multiple targets have the same damage modifiers though.