Historically almost everyone built their castles in someone else's kingdom. The barons may have complained a lot, but in practice they were much more likely to build another palace in the capital than in some remote swampy place where the king's writ ran less.
>>zeteo+(OP)
Well given feudalism even chances are the count or baron's castle wound up somebody else's kingdom or empire by force if they had it beforehand.