As the card is worded, this isn't a bug.
A site's text is only active when the fellowship is at that site (otherwise, for example,
The Prancing Pony could be used every turn). Since your opponent wasn't at that site, it could freely be replaced.
Mount Doom is different because it had the text "for the rest of the game", so once the text is activated (by a fellowship moving to it), it is in place forever. Even that site can be replaced before a fellowship moves there, however.
Perhaps this is one for the PC to consider rewording, as it's just as abusive as always as things stand.