The Vex were in our solar system before we were, though. Well, they can time travel, so that isn't saying much.

There's a line from your Ghost on one of the story missions that's something like "It's like the Vex have dormant systems/networks/somethings buried under every planet in the solar system, just waiting to be revived!" and the ruins on Venus are said to predate humanity by billions of years.

Finally, they don't necessarily have an interstellar empire; the only ones who we know have that are the Cabal, and possible the Hive, and that the Fallen say they used to.

Obviously several of these points are moot because your thing was in response to above conjecture and this is partly conjecture of mine in another direction, but this is an interesting conversation.

Quick, we need an Awoken Titan, an Exo Hunter, and a human female Warlock. (We could also have another human male Warlock, but he'll spend FOREVER before pressing the button to revive.)