From what @Mr. D said..
...I assumed that the "with you" refers to the unassigned characters only, as opposed to those stationed in any base.
Otherwise, many aspects of the game would sound pointless, especially the enormous scale of the available classes and items. One would assume a necessity to widely experiment with different character configuration, to test them every now and then, to have like an offensive team and defensive team at various moments, to dwell on training policy (should I have one maxed out team, or several less trained teams - since we cannot train all at the same time).
If the total limit of characters is indeed 3, noone would experiment, since the cost of lost exp would be enormous. At most, people will search for a preferred set a bit longer at the beginning, but once you have a month or more invested exp-wise, you'd feel quite locked with the crew you have and not interested with collecting them anymore.
And late in the game, once you find that few-per-world "Unique" character, you would have a really tough nut to crack, to waste any of those two-year-fed characters to risk picking that single gold one and start from scratch? Would it ever get better than a two year older character of an "ordinary" type?
@Mr. D So, is it a bug or is it a feature?