Think of it like this.
You have a turbo that is only able to produce a certain VOLUME of air per minute. Specifically, CFM (cubic feet per minute). That said, since the output, or the "air" produced by the turbos are essentially static (not really, but for simplicity's sake let's say they are) and we change the volume of the "box" you are trying to fill (ie the charge volume, which is everything after the turbo compressor up to the intake valves, or even the cylinders themselves, arguably) then it will take longer to fill said "box" when you are making it bigger.
A much more simplified way to look at it is to grab a garden hose. Turn it on full blast, that can be your turbos max output of compressed air. Grab a gallon milk jug, put the hose in the top and let her rip. The time it takes to fill the jug and spill over the top is our pretend way of visualizing the time it takes to pressurize the charge piping and intercooler (ie spool to full boost) and subsequently spill into the engine. Now, if you switch to a 2 gallon or 5 gallon jug (ie get a larger intercooler) then obviously it will take a LOT longer for the hose to fill up the jug and spill over. Why? Because you only changed the charge volume (ie the jug) but not the output of the turbo (ie the water hose).
That said, ANY TIME you increase charge volume, your turbos will take longer to "spool" (ie build to full boost).
Now keep in mind that what I said is not taking into account flow of said pressurized gas. That's way more complicated and way above what I understand.
Oh also, one more thing. My analogy isn't perfect. It would seem like getting a bigger hose and more water pressure would be equivalent to just getting a bigger turbo. THIS ISNT THE CASE. Keep in mind that turbos use exhaust gasses to make them spool, so simply running a bigger turbo is NOT going to decrease your spool times. Instead, you'd need more displacement (ie exhaust gasses) to spool said bigger turbo (and probably have a larger intake as well) for it to be equivalent to having a larger hose with more water pressure in my analogy. Even so, again, this analogy has many flaws, so you can't directly translate everything to how our cars work, but this is just one, very simple, way of visualizing just one part of what is going on during boost.
Hope that makes sense.
Edit: clarification. grammar. etc.
