Has any game tried this?

We have had bots since the late 90s. Unreal Tournament even had bots that could auto adjust to your skill.

So when someone leaves a game, put a bot in their place that plays at their skill level. When someone joins swap them for the bot.

Downsides?

Whatever downside there is would be found and exploited within the first 5 minutes, spread through reddit, create mass chaos and prove that D2 is the worst game ever and Bungies r stoopids.