The released ChatGPT as a powerful language model is capable of assisting with a wide range of tasks, including answering questions, summarizing, paraphrasing, proofreading, classifying, and integrating texts. In this study, we tested ChatGPT capability to assist researchers in evaluating the academic articles’ contribution. We suggest a dialogue schema in which ChatGPT is asked to answer research questions from the target article and then to compare its own answers with the answers from the article. Finally, ChatGPT is asked to integrate both solutions coherently. We experimented with Proceedings of ISM-2022 Conference on Industry 4.0 and Smart Manufacturing, utilizing explicit research questions. The chat context enabled assessing studied articles’ contributions to Industry 4.0, uncovering advancements beyond the state-of-the-art. However, ChatGPT demonstrates limitations in content understanding and contribution evaluation. We conclude that while it collaborates with humans on academic tasks, human guidance remains essential, while ChatGPT's assistance efficiently complements traditional academic processes. See presentation slides: https://ai.it.jyu.fi/ISM-2023-ChatGPT.pptx