They CAN'T assign everything to Gandalf. Each minion gains the assignment ability to be assigned to Gandalf, but after he uses that ability for one minion, Gandalf will already be assigned, and he can't assign the other ones. Only after every companion is assigned to a minion you can assign leftover minions to a given companion.