At this point, however, we're getting closer to our physical limits regarding what we can perceive visually, and since improved visuals have been the primary driver for improving consoles, perhaps hardware is going to matter less? That idea might sound really stupid in a few years. I don't know.

Better visuals are not the only things you can do with better hardware. The more power you have under the hood, the better you can make your games. Period. The quest for improvements will only end when it becomes physically impossible on a quantum level, or until designing games on said improved hardware is not financially viable.

I never really thought about this before and I find the idea fascinating. Labyrinthine hardware equals room to grow without changing the hardware.

There's definitely some room to grow regardless. Even if the hardware is easy to utilize in a basic sense of "we can use up the processor's time and fill the RAM with data", techniques develop as a generation progresses. Methods of crafting assets improve, understanding of the behavior of light and how to model it in ways that are convincing to the human viewer is expanded, new approaches to rendering 3D scenes which might reduce certain redundancies in some circumstances come into being, etc. Even when pure efficiency doesn't improve, the toolbox gets bigger, and things get more refined.

Human characters in some late-gen PS360 games have skin that looks warmer and less like waxy concrete than characters in some CGI from the early part of the gen. That's not because the games had more resources to spend on skin rendering than the CGI, it's because over eight years we improved our understanding of how light interacts with the various layers of stuff that make up skin, how to represent that "accurately" even with cheap approximations, and how to correctly author textures to represent these things.

Makes you wonder, though, how first-gen games managed to look as good as they did (like Halo)

The Xbox had a good development environment right out of the gate, and while the tech was very modern, it wasn't janky. Except for the disc drive, which is extremely janky.

Sixth-gen Halo sort of demonstrates the compromization and refinement that I'm talking about.

Halo 2's graphics are much more sophisticated than Halo 1's; dynamic objects received normal maps and analytic specular, rudimentary specular color appears to be added, then-modern post-processing techniques like bloom and depth-of-field were implemented, water has refraction, streaming is now a dynamic process, assets for dynamic objects look more realistic and detailed, and effects for explosions and the like are smoother. There are also changes like the physics system vaguely resembling real-world physics. The game even runs better, by and large.

These didn't come without cost, though. Halo 1's dynamic lighting is far more vibrant and much more of it includes specular on the environment; Halo 1's effects are larger and more deeply-layered; Halo 1 uses several times as many particles. While Halo 1 can't necessarily get away with as much high-quality unique textures as Halo 2, it still managed to have very sharp textures through lots of reuse, and it didn't suffer from texture popping.

In overall appearance, Halo 2 tends to be a more modern-looking game than Halo 1. But, this has the interesting caveat that a lot of people prefer how Halo 1 looks.

¯\_(ツ)_/¯

(And that entire situation has a lot of parallels with Halo's 3 and Reach.)