I still can't find where it where it specifies that the cards have to be from your draw deck in the rules.
Actually, I believe you are right. Here's the quote:
The Rule of 4
Some card effects allow you to draw cards in the
fellowship phase. You cannot draw (or take into
hand) more than 4 cards during your fellowship
phase.
This applies to cards taken into hand by any
means. This does not apply to cards drawn "at
the start of each of your turns."
That does indeed color the discussion a bit... Hmmm...
EDIT: Thanks for pointing that out! I'll have to rethink my vote.