Ran into the same problem trying to give starbases more than 8 slots. I don't know for sure, but I suspect 8 is a hard limit due to UI reasons. Hopefully someone will prove me wrong. 
I even tried adding enough items so that adding all the hangar upgrades would go past 8, to see if it was just a UI thing, but no, it still stopped me at 8.
At least for starbases (and titans) we can look at squishing some of the upgrade tracks from 3 items to 1, to free up more slots.
For example, for the TEC starbase:
1. Edit "trader_starbase_hangar_0.unit_item" to add all 14 hangars. You can adjust the cost for balance, too.
2. Edit "trader_loyalist.player" and "trader_rebel.player" to remove "trader_starbase_hangar_1.unit_item" and "trader_starbase_hangar_2.unit_item".
You just gained +2 slots!
Just gotta be aware of attached research for some things, and adjust or remove them accordingly. So, doing this with the Hull/Armor/Shield upgrades on the TEC starbase means you have to either adjust how the Unbreachable Hull research works, or remove it entirely.
I'm trying to make it so you build the first Hull/Armor/Shield upgrade then upgrade the rest via research, to keep the progression but free up 2 more slots, but so far I can't get the starbase stats to improve via research.
