I've missed you, Larry!
This is a very fair question. Put simply, I think the value of a card is determined by the general rules of supply and demand:
For demand: the more people who want the card, the more they will have to pay for it to stop someone else getting it. To my mind there are two main types of people who buy cards: players and collectors. A player will typically want to complete their decks, so will want the stronger cards (and four of some cards they need). A collector may be less interested in having duplicates, but may instead value things like foils over playability.
And then there's supply: if a card is very rare, there will be fewer of them in circulation, so again more competition over the card, driving up the price. Decipher did not print all cards in equal amounts. In particular, there were very few cards printed in the Hunters Block (and Siege of Gondor to a lesser extent). As a result it is less common to see these cards being sold, so when they are, they fetch a higher price.
And finally, on to your question. In terms of demand, the Hunters Ugluk will slightly outdo the Towers Ugluk. Both are great in their respective decks, but more people will play expanded than Towers block (set 4 Ugluk is not seen outside of that block, and even then, trackers aren't as popular as hunters are in expanded). However, the real difference in price is due to demand. As a rare card from Hunters,
Ugly Fellow is by a long way less available than
Servant of Saruman. This compounds with the fact that if you're buying him for your deck, you'll probably want four, taking even more of them out of the market.
$180 is a huge price. That must be foil as well, but if that's what some people will pay for it, then I guess that's its value. Seems like a lot for a piece of cardboard to me, though.