Area Chair Guide
Area Chairs play a critical role in curating the technical programme for ICLR. Use this as a resource for any questions related to your role as an Area Chair. Please contact program chairs via email (iclr2022pc@gmail.com) with any questions or comments. Thank you for your contribution!
Here are the key dates, focused on AC duties, that we expect to work towards for ICLR 2022 (Please allow possibility of small last-minute adjustments):
- Sep 28: Abstract submission
- Oct 5: Paper submission
- Sep 29 - Oct 4: AC bidding
- Oct 8 - Oct 12: AC receiving and confirming paper assignments
- Oct 13 - Oct 15: AC vetting reviewer-paper assignments
- Oct 14 - Nov 1: Review period
- Nov 3 - Nov 8: Substitute review period
- Nov 9 - Nov 29: Paper discussion (authors, reviewers, ACs). Note: Authors can update their papers until Nov 22
- Nov 29 - Dec 6: AC Meta-review period
- Jan 24: Author notification
Abstract Submission
This year, we added an abstract submission deadline. Having the abstracts a few days earlier will allow ACs to bid on the abstracts, recommend appropriate reviewers, and check the reviewer fit early in the reviewing period.
Code of Ethics
This year, ICLR continues to practice of the Code of Ethics which needs to be acknowledged and adhered to by all participants including authors and reviewers. If any submission, review, or discussion comments raise ethical concerns, please flag the problematic content and contact the Program Chairs.
Multi-Stage Discussion
Similar to last year, the review process is designed to maximize discussions while clearly distinguishing the different stages of discussion. After the initial review period, during which each assigned reviewer is required to submit a formal review, there will be two stages of discussion.
-
In the first stage (Public Discussion), anyone can post a comment on a submission. Authors may post any clarification anonymously, and the assigned reviewers and AC may post further comments. Public commentators can also participate and leave comments, but cannot do so anonymously, of which the decision was made to avoid any potential adverse behavior.
-
In the second stage (Author/Reviewer/AC Discussion), the authors, assigned reviewers and AC are allowed to post their comments, while posts from the public will be blinded (they will eventually appear after the decision notifications are sent.). Reviewers and ACs may have private discussion threads throughout this period (which will remain private even after the final decision).
Your Roles and Action Items
Reviewing Process Manager
Your first role is to help the Program Chairs manage the reviewing process for the thousands of submissions we expect to receive. When you are assigned a batch of papers to handle based on your bids on the abstracts, please recommend a set of appropriate reviewers (you may skip this, and the reviewers will be assigned based on their bids only), and when some of those reviewers are delinquent or not responsive, please help us assign alternate reviewers in a timely manner so that every submission gets a chance to be judged fairly and expertly.
-
Bid on abstracts
-
Vet and recommend reviewer assignments
-
Identify delinquent reviewers and assign alternate reviewers
Decision Maker
From the moment all papers are submitted, perhaps the most important role as AC is making decisions for the ICLR program. For every submission that goes through the reviewing process, please recommend whether it should be accepted or rejected. The recommendation should be accompanied by a metareview summarizing the reviews and the three stages of discussion, optionally adding your own view of the merits and limitations of the paper. The Senior Area Chairs and Program Chairs will actively engage in this decision making and help with the metareviews, so please do not hesitate to contact your SAC or the PCs if you find the need to discuss any submission assigned to you.
-
Flag papers for desk reject
-
Watch out for reviewers’ flags for Code of Ethics violations. For all papers flagged, collect information, make an informed recommendation, and provide any evidence obtained (e.g., from the paper, discussion with author and/or reviewers). Please note the AC recommendation will be non-binding, and any of the flagged papers may be passed to the Ethics board for further review
-
Engage in discussion with the reviewers to ultimately help you reach an informed decision (see next section)
-
Write metareviews with recommendations for accept/reject
Among the important responsibilities of ACs is to ensure reviewers’ judgments are in line with the new ICLR 2022 reviewer guidelines at https://iclr.cc/Conferences/2022/ReviewerGuide. Especially pay attention to the FAQs (https://iclr.cc/Conferences/2021/ReviewerGuide#faq) for questions such as the definition of contemporaneous work for the purpose of citations and empirical comparisons.
Also, it is important to make sure that the language used in the reviews (including ACs’ own metareviews) are constructive and polite. Here are useful advice to review:
-
Daniel Dennet, Criticising with Kindness.
-
Comprehensive advice: Mistakes Reviewers Make
-
Views from multiple reviewers: Last minute reviewing advice
-
Perspective from instructions to Area Chairs: Dear ACs.
Discussion Moderator
Each submission is considered a forum on its own, and you as an AC has full responsibility in encouraging and moderating active discussions. When a submission does not receive enough attention that it deserves or requires, you should actively engage with the assigned reviewers as well as the authors and ask for clarification or argument. You should also “moderate” discussion by discouraging the participation in any discussion that is irrelevant to scientific claims and merits of a submission.
-
Encourage reviewers to respond to author rebuttals
-
Moderate the discussions so that they are not toxic and focus on the scientific merits, limitations, and clarifications
-
Identify any violations of Code of Ethics during the discussion phase
Discussion Participant
We have invited you to serve as an AC because of your expertise and reputation. In other words, your assessment of a submission is a critical factor behind the entire decision-making process, and we ask you to actively participate in discussions not only as a moderator but also as a scientific expert. You are encouraged to ask authors (as well as any other commentator of the submission including assigned reviewers) for clarification. In other words, please be an active participant in discussion.
-
Participate in discussions with your own view of the paper
-
Ask authors for clarifications when needed to understand and judge the contributions fairly
Meta-Review Guideline
As an AC, we trust you to make an informed recommendation based on sufficient knowledge and justified analysis of the paper and to clearly and thoroughly convey this recommendation and reasoning behind it to the authors. To this end, you have full freedom in writing your meta-reviews, although we list below a few items that have been found useful by authors when they were presented with meta-reviews. Aim to write a meta-review of at least 60 words.
1. A concise description of the submission’s main content (scientific claims and findings) based on your own reading and reviewers’ characterization, including the paper’s strengths and weaknesses. Ideally this description should contain both what is discussed in the submission and what is missing from the submission.
2. A concise summary of discussion. Unlike other conferences in which there is only a single round of back-and-forth between reviewers and authors, ICLR distinguishes itself by providing three weeks of discussion. These weeks of discussion not only serve the purpose of decision making but also to contribute scientifically to improve the submission. We thus encourage the AC to summarize the discussion in the meta-review. In particular, it is advised that the AC lists the points that were raised by the reviewers, how each of these points was addressed by the authors and whether you as the AC found each point worth consideration in decision making.
3. Your recommendation and justification. The meta-review should end with a clear indication of your recommendation. Your recommendation must be justified based on the content and discussion of the submission (i.e., the points you described above.)