Seems simple enough, while searching I came across this snippet you can paste in the console, and gives you a sorted list of most upvoted/downvoted comments: >>36107028
Sadly requires hitting news.ycombinator.com rather than the API, but only way to get the actual points as you mention.