As machine learning conferences grow, we as a community need to put more thought and planning into maintaining a fair review process. During the recent round of reviews, several issues pertaining to academic integrity came to our attention. This prompted us to conduct an investigation, including, e.g., partnering with sister conferences for checking for dual submissions.