The game was designed from the get-go to support up to 6, but testing revealed that having 5+ players is bad for a player's first couple games, because their brain is still having to actively think about the contents of each and every land to parse what's there. With 5 or 6 boards, it just becomes visually overwhelming. (4 leans towards that, but seems to be on the OK side of acceptable. I feel like 3 may be optimal for learning games.)
Once players can read the board more easily, playing 5-6 is fine. (Exactly how many games it takes to get to that point depends on the player.) The coordination time/complexity doesn't scale up as badly as you might think, because the island gets so big you have more geographical separation - you're usually not coordinating with any more players than you would in 4p, with the occasional exception of "Target a Spirit" Powers. (The separation is kind of neat, actually; suddenly a new color of Presence crops up in this problem area that you and someone else have been struggling over, and you're like "Oh, hey! Visitor from far away! Welcome to the neighborhood; can you give us a hand?")
But for early plays, having too many players can make for a terrible, overly-drawn-out experience.
One could in theory print the game with "1-6 players" on the box followed by huge warnings in the rules to the effect of "never play with more than 4 until everyone feels well-acquainted with the game", but I'd expect a certain amount of justified annoyance at the perceived bait-and-switch - particularly because there are (deliberately) only 4 spirits appropriate for a teaching game.
But that's all moot, because of the other issue: components. Going to 6p would require half again as many Cities, Towns, Explorers, Blight, Dahan, and Presence pieces, not to mention two additional island boards and more Fear/Energy tokens. I have to assume that would balloon the cost of the game.
It also creates problems with the Power decks being too small relative to the playgroup - if everyone takes "Gain a Power" on a given turn, you can search through 24 cards in either Power deck, which allows for a sort of highly targeted digging that isn't intended to be a part of the game. ("So next turn, someone will take Reaching Grasp...") You could also in theory run out of Minor Powers by late-game, though practically speaking that's not likely.
But some groups would probably embrace the challenge.
Oh, certainly. There are some groups which prefer 4-6p. I'd love to support them in that, but I think it'll have to wait for a 5-6p expansion sometime in the future; in the meantime, there's always the "combine two sets" option for the groups which feel super-strongly about it.