zlacker

In my view, in most cases ACs now perform one of two roles (1) summarize what reviewers have said, and/or (2) if there is a high variance in scores, they request the reviewers to sort it out among themselves, and THEN summarize :). I am exaggerating here - sometimes they do engage, but these cases are far in between. Among my personal experiences, this has only happened twice: (a) during an ARR review where I was an author, we had requested the committee to intervene because one set of reviews went wildly off-track (along the lines of "more experiments", like you mention), (b) as a reviewer, I had pointed out some glaring benchmarking omissions, and the AC took time to understand the concern and decided to bring it up for discussion.

  > I crave for someone to actually _criticize_ my work

Yes! But if you are not doing mainstream DL, good constructive criticism is almost impossible to obtain. I feel like many reviewers expect half the paper to be a tutorial if it is not a trendy topic. Which I find is unfair for multiple reasons: (1) for a trendy topic much more complex topics are unexplained, because it is assumed the reviewer has heard of it, (2) yes, the previous point makes sense, but this is what cited materials or the Appendix is for, (3) most conferences have a page limit for the main paper, so you cannot go about rambling and arbitrarily explaining ideas, and (4) this is supposed to be a rigorous review process, it is not supposed to be easy. It shouldn't come as a shock that some papers take (sometimes a lot of) work to review!

>>abhgh+(OP)

  > I feel like many reviewers expect half the paper to be a tutorial if it is not a trendy topic. 

I tried to push a paper using a GAN to a workshop. I was asked to spend more time explaining what a GAN is "for those unfamiliar with the topic." I was baffled. Sure, ML moves fast, but that's catastrophic forgetting right there... (and in a fucking workshop?!)

I honestly believe that if someone is not intimately familiar with GANs then they should not be reviewing for a generative vision workshop.

  > I feel like many reviewers expect half the paper to be a tutorial if it is not a trendy topic. 

The difficulty of this gets harder with different topics. Generative works should show samples. But how much? How representative? These are crucial to evaluating the work but they devour your text limits.

It is always easy to ask for more. But with page limits there are clearly limits. I think it is too subjective. I wish we went more the direction of math papers which are often VERY short. Use as much space as you need. No more, no less. I think the formats are just too limiting (not to mention that paper isn't great for a lot of topics like video, point cloud, audio, pose estimation, and many others. But momentum is a powerful force.