One way to cut out the middle man would be to convince journal editors to run sibling journals alongside the existing journals, so for each "Transactions on XYZ" there would be an "Open Transactions on XYZ" (as close in title as is legal). Importantly, each sibling journal would be run by the exact same academics (who are doing the real work on tax money anyway), and according to the same process as the original journal, just without involvement from a traditional publisher. PDFs would be hosted on a site like arXiv. The goal would be that submitting to the open "sibling" would be the obvious rational choice (same people, same decision-making, no fees, open access), which in time even the funding agencies and tenure committees would have to acknowledge.
So rather than a reviewer lending their reputation to a journal, and that journal then conferring a stamp of trustworthiness onto an academic work, reviewers lend their reputation directly to the works they review. The works themselves can still be shared far and wide, e.g. via ArXiv.
Of course, inertia is still a massive force to work against.
The biggest problem I see with the proposed system is that it's unfortunately often easy to recognize who wrote a paper (which is bad for ordinary peer review already) and in a personal endorsement system this would lead to collusion among researchers with low integrity. You wink through my papers, I wink through your papers. Also, imagine you don't endorse the paper of the senior researcher in charge of your postdoc funding...