Skip to yearly menu bar Skip to main content


Guidance for Reviewers

When reviewing for ICLR 2019, please keep the following in mind:

 

Publication standards. It’s important to maintain the high quality of ICLR papers. At the same time, we must be cognizant of the fact that different reviewers, authors, and even different areas will have different standards about what constitutes a high-quality paper. Reviewers are encouraged to use their best judgement, but also ask themselves the following questions:

  • Does the paper present substantively new ideas or explore an underexplored or highly novel question? Papers that take risks and study a less explored area are likely to have less polished results, papers that study a highly explored topic are likely to have more polished results. This phenomenon often results in reviewers excessively penalizing papers that explore underexplored topics, and it is worth accounting for this.

  • At the same time, papers that present state-of-the-art results on well-studied problems should be given consideration: addressing problems that are clearly of interest to a large segment of the community is important, when the results substantively advance the state of the art.

  • Ask yourself: will a substantial fraction of the ICLR attendees be interested in reading this paper?

  • Another question to ask yourself: would I send this paper to one of my colleagues to read? This is often a good indication that the paper is worth accepting, though not the only indication.

At the same time, it is critical to maintain a high standard in terms of scientific rigor: if you believe that a paper has flaws in terms of its evaluation or validation, proofs, or other parts of the discussion, it is critical to point this out to the authors. The authors will have an opportunity to address these concerns, and the iterative process of improving papers after reviewer feedback is important for ensuring the highest quality of ICLR papers.

 

Topic fit. The scope of ICLR has increased over the years, and covers a large segment of machine learning research and applications. If you are uncertain about whether a paper is a good topic fit for ICLR, feel free to contact your area chair.

 

Anonymity. ICLR is double-blind, which means that authors are not aware of reviewer identities and reviewers are not aware of author identities. If you believe a paper contains an anonymity violation, contact your AC immediately. Anonymity violations are not considered as part of reviewing criteria, they are requirements for submission. Unless your AC decides that the paper does indeed violate anonymity, proceed to review it as normal. Do not reveal your or the authors’ identities in the discussion.

Length. The recommended paper length is 8 pages, with unlimited additional pages for citations.  There is a strict upper limit of 10 pages.   Authors may use as many pages of appendices (after the bibliography) as they wish. Reviewers are not required to read all appendices. However, details that are necessary to understand whether a paper is reproducible or to validate proofs may be found in these appendices. This is acceptable, and it is up to the reviewers to verify they are present and correct if they have concerns about reproducibility and correctness. Papers may exceed 8 pages in the main text, but reviewers should encourage authors to avoid overly lengthy papers. In some cases, longer papers may be acceptable. For example, a paper with two pages of figures (e.g., image or video generation) might have a total of 10 pages in the main body. But the overall time to read a paper should be comparable to that of a typical 8-page conference paper. Reviewers may apply a higher reviewing standard to papers that substantially exceed this length. If reviewers believe that a paper is too lengthy, first it should be pointed out to the authors to see if they can fix it during the revisions phase.

 

Similar/concurrent submissions. ICLR often sees multiple concurrent submissions on similar topics from different authors. In such cases, special care should be taken to ensure that concurrent papers are reviewed to the same standard. If you are reviewing a paper and believe there is another ICLR submission that is concurrent and similar, please contact your AC to let them know.

 

Arxiv and prior work. ICLR considers unpublished arxiv papers to be prior work. While we encourage reviewers to apply the reasonable standards of the relevant community in considering what does and does constitute prior work, the following minimum standards will be enforced: no paper will be considered prior work if it appeared on arxiv, or another online venue, less than 30 days prior to the ICLR deadline.