This is not necessarily true. In Burdick v. United States it does say "an imputation of guilt and acceptance of a confession of it" but there is debate about whether it is binding of not.
Apparently, there is something in Lorance v. Commandant, U.S. Disciplinary Barracks that indicates that accepting a pardon does not imply guilt, but I am not very knowledge on that.