I somehow think nanites aren't the answer to this. For one thing, if the "enemy" were nanites the lone surviving ship would continue being deconstructed after escaping as it already had the nanites on board, and given that the Vasari already used nanites offensively the crew would have known, at least to some extent, what had attacked them. Also, the only ways nanites can make people insane are either

1) Direct manipulation of the brain (though the back lines in the "final battle" would have known about these, as nanites aren't invisible and the front line would be affected first)

or 2) The sheer horror of seeing the entire Dark Fleet wiped out within moments, though anything that destroys the Dark Fleet would accomplish this.

And overall the Vasari seem to use nanites as mere tools, lower than slaves, like the TEC use bullets or other pieces of metal, so I doubt they would construct any with enough power to wipe out the Vasari Empire. (Even if you combined all known features of Vasari nanites as exhibited by the researches/abilities using them, the only way I see them threatening the Dark Fleet is as "grey goo", i.e. converting the entire fleet to raw resources, then again look at my first point)

And no nanites had any ability of phase travel, so unless someone deliberately created a horde of phase-traveling, self-replicating killer machines and unleashed them, they shouldn't be able to travel and destroy things. (Any experimental phase-travelling nanites would surely be built without deconstruction abilities, and a remote selfdestruct)

Another possibility would be an internal rebellion created by a Vasari group (which could of course use nanites ), attempting to overthrow the old empire to gain power for themselves

Aside from that, Phase Space simply is a more interesting option. Haywire robots of any size are somewhat boring next to multi-dimensional creatures or a huge wormhole. Considering the Dark Fleet crew were used to casualties and cruel deaths, including those caused by (their own) military nanites, a phase space anomaly also seems more likely to make them go mad (in Star Wars they said merely looking out of the window during hyperspace jumps would make you mad over time) And the Vasari, although experienced in using phase space for travel, and some weapons, never displayed any deeper knowledge into the subject. Seeing as they couldn't work out PJIs I doubt they truly "know" what phase space is and "how it works". They might even have a wrong/incomplete theory of it, as Humans had concerning light when they invented the first lightbulbs.

And then there's the "Experimental Transport Lab" from the opening cutscene. Now what do the Vasari use for transportation? Correct -phase space! (Of course it could be phase-traveling nanites too, but really that idea sounds stupid when you could just send a carrier ship) Though I must say whatever that lab was researching the Vasari were pretty stupid for placing it at the core of their empire.

Another argument I have in favor of phase space and against nanites is that phase space preceded the Vasari, while they created the nanites. Phase space "is" without the Vasari, and doesn't require their understanding to work, while nanites must be designed and built by sentient beings. While they can malfunction, they must first be created as a working system with the potential to malfunction. (To use electricity in any way, let alone as a weapon, humans needed batteries, generators and consumers, yet cavemen and probably dinosaurs were killed by lightning)

Of course there are numerous possibilities to this, though I suspect the "enemy" will be a sentient "race" of sorts for gameplay reasons.

actually photoloss,a nanite outbreak could spread through space to other systems without ftl travel.

a vasari rebelion wouldnt have been so fast and devastating.

When I wrote a short story about this game (actually, technically three sort stories) I addressed this question.  My concept of it was large (larger than the starbases, large enough to, when burrowing underground, show up from space when viewing the surface of a planet) dreadnaught-type warships with massively powerful and retractable deck guns (one-shot kills on Kol Battleships) and cloud-shield-coverings of an acidic base (maybe nanites? I never offered an explanation) capable of dissolving bullets, ships, and even planets with ease.  I implied them to be self-aware, but not necessarily AI.  They could be organic, and in fact characters note how they seem organic, and yet not, but they could be purely robotic.  They're terrifying in their mystique, in the fact that you just don't know what they're going to do, as well as their raw, unstoppable power.  They are very much so the hunter-type creatures, built solely for consumption.

But, I don't really think it matters what the things chasing the Vasari are, per se.  It matters that they are, not what.  They represent the "sin" of the Vasari Empire, whether by their own direct creation via a lab, or merely their release from somewhere else by the Empire's own action.  In any case, they are the personification of that sin, as they relentlessly and purposelessly chase the Vasari for no other reason than to consume and destroy.  The TEC's sin is represented as the Advent, who return after their unjust exile to impose their own brand of justice on the Traders.  And the Advent's sin is represented by their own self-consumed need to impose their view of order upon the chaos that is the universe, to the point where self-expressive thinking is lost to them.

As for reactions, I think being driven mad by fear would come less from any direct action and more from a lack of effective reactions.  If there was an enemy that either had no explanations that really fit rational thought or just had no conceivable and believable backstory, and yet they were before you, destroying everything you've ever known, would you stay sane?  Especially if they were something big and real, like, say, my "dark dreadnaughts," which have form and mass.   Freed and all-consuming nanites, meanwhile, would be scary maybe only because they aren't visible, because they can't be fought, only run from, unless you wish to stay and watch your own worlds and ships dissolve out from under you.

In my short stories, I predicted the Advent and TEC reactions to the chasers differently.  The Vasari I had just run, regardless of a need for resources or not.  It would be better to try running and finding resources later than stay and try and be greedy while your enemy slaughters you where you stand.  The TEC I showed as running away, similar to the Vasari, but attempting to stage a back vanguard to buy time, however futile a venture such an action would be.  The Advent I imagined as collapsing in on themselves, unable to accept that their so hard-sought-after revenge is unavailable as their prey runs and their prey's true hunters fall upon them.  I figured that the leadership of the Advent would be torn between running alongside their former enemies, or staying and trying to fight, and would fall to civil war.  A grave situation, indeed, if you consider their harmony with each other and group mind, etc. 

Yeah...I'm done with my random rant...lol...

My thoughts exactly neoshadow. lol

Commissar Yarrick.

Duh.