Something that may surprise you, as it did me, was that added difficulty does not actually add more complexity by and large.
Adversary: Just playing with a baseline adversary adds an additional check during invader stage II. The reminder is easy to see on the invader cards with the fort/flag. While this adds some extra difficulty, it's not really added complexity.
As you move down the level list of adversaries, a large number of changes are set-up related and don't pile up things to remember or check in-game. Some things like extra health/damage on buildings (england/sweeden), extra blight (sweeden), exploration sources (england) might need to be remembered, but it is not a lot of extra information per adversary, and is more in the fiddly catagory than complexity.
Branch and Claw: The event deck can be considered added complexity since a significant portion of those events direct players to make sometimes difficult choices.
The 4 extra tokens are nice because no single game will blast you into reams of needing all 4 and dealing with them all. For instance, outside of a very few power cards or playing as Keeper, you might never see or deal with a Growth token in a game. Beast and Disease tokens start on the board since some event deck cards directly interact with them, but unless you are playing as Fang, the Beast tokens do nothing without Event direction. Strife tokens will need to come from the Fear deck or from a limited number of powers. You could argue that these tokens add complexity, and they do to a certain extent, but within a practical scope of any given game they really are easily managable and are sparcely seen (outside of Keeper or Fang who are centered around their respective tokens).
TL:DR - Don't be scared of the expansion. It may look overwhelming, but it does not add the "complexity" you might think. Where it does add complexity, it adds it in small bits in different phases of the game making the transition very easy.
In my personal experience, I was pretty scared to add the expansion in too quickly as I was floored by just the base game alone. But once I added it in, I was shocked at how such a large expansion just slid in seemlessly into the base game. It looked so overwhelming from a distance, but in practice the game does a great job of incorporating small bits of these new things at any given time, allowing you to parse the new information easily. The expansion really makes Spirit Island a "full game", and I suspect that once you play a game or two with it you will see what I mean.