{"id":6155,"date":"2023-05-05T10:02:42","date_gmt":"2023-05-05T09:02:42","guid":{"rendered":"https:\/\/wordpress.aber.ac.uk\/e-learning\/?p=6155"},"modified":"2023-05-11T11:00:04","modified_gmt":"2023-05-11T10:00:04","slug":"considerations-for-generative-ai-detection","status":"publish","type":"post","link":"https:\/\/wordpress.aber.ac.uk\/e-learning\/2023\/05\/05\/considerations-for-generative-ai-detection\/","title":{"rendered":"Considerations for Generative AI Detection\u00a0"},"content":{"rendered":"\n<p>This blog post is written by Generative AI Working Group members.&nbsp;<\/p>\n\n\n\n<p>The landscape for learning and teaching in the age of Generative AI has been developing rapidly. As staff will be aware, the <a href=\"https:\/\/www.aber.ac.uk\/en\/academic-registry\/handbook\/regulations\/uap\/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>UAP regulation<\/strong><\/a> has been updated to address the use of AI in student assessment. The UAP Form and penalty table have been updated to include \u2018Presenting work generated by AI as if it were your own\u2019 (approved by Academic Board March 2023).&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.aber.ac.uk\/en\/academic-registry\/handbook\/regulations\/uap\/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Regulation on Unacceptable Academic Practice<\/strong><\/a>&nbsp;&nbsp;<\/li>\n\n\n\n<li><a href=\"https:\/\/www.aber.ac.uk\/en\/academic-registry\/handbook\/taught-schemes\/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Template Form<\/strong><\/a>: See section 3.13&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>A Generative AI working group, chaired by Mary Jacob, was created in January 2023 to coordinate university efforts. Please see <a href=\"https:\/\/www.aber.ac.uk\/en\/lteu\/teaching\/teaching-cpd-workshops\/#generative-ai\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Generative AI<\/strong><\/a> for current guidance and resources. We are designing training materials for staff and students that will be available well before next academic year.&nbsp;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Advice for marking&nbsp;<\/h2>\n\n\n\n<p>On 3\/4\/2023, Turnitin enabled its AI detection tool. At present, staff can see the \u2018AI Score\u2019 but students cannot. This may change if Turnitin updates the tool later in the year. Please see <a href=\"https:\/\/wordpress.aber.ac.uk\/e-learning\/2023\/03\/27\/launch-of-turnitin-ai-writing-and-chatgpt-detection-capability\/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Launch of Turnitin AI writing and ChatGPT Detection Capability<\/strong><\/a><strong> <\/strong>on the LTEU<strong> <\/strong>blog and <a href=\"https:\/\/wordpress.aber.ac.uk\/e-learning\/files\/2023\/03\/AI-writing-detection-How-to-use-guide.pdf\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Turnitin\u2019s AI Writing Detection<\/strong><\/a> from Turnitin (note that sometimes the same passage can be identified as both AI generated and matching an external source).&nbsp;&nbsp;<\/p>\n\n\n\n<p>There is a clear consensus among experts in the sector that <strong>no AI detection tool can provide conclusive evidence<\/strong>.&nbsp;&nbsp;<\/p>\n\n\n\n<p>This comes from the QAA, the National Centre for AI in Tertiary Education (sponsored by Jisc), and others. You can find links to this evidence on the <a href=\"https:\/\/www.aber.ac.uk\/en\/lteu\/teaching\/teaching-cpd-workshops\/#generative-ai\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Generative AI page<\/strong><\/a>, including the QAA recording where Michael Webb from the National Centre explains why this is the case.&nbsp;<\/p>\n\n\n\n<p>If you face a potential UAP case, your professional judgement is key to making the right call. Here is the best advice we can give departments:&nbsp;<\/p>\n\n\n\n<ol class=\"wp-block-list\" start=\"1\">\n<li><strong>Use the Turnitin AI detection tool in conjunction with other indicators \u2013 <\/strong>The Turnitin detection tool can identify red flags for further investigation but cannot provide evidence in itself.&nbsp;&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"2\">\n<li><strong>Check sources \u2013 <\/strong>Gen AI often, but not always, produces fake citations. These can seem plausible at first sight \u2013 real authors and real journals, but the article doesn\u2019t exist. Check the sources cited to see if they are 1) real and 2) chosen appropriately for the assignment. Is the source on topic? Is it the type of source a student would have read when writing the assignment (e.g. not a children\u2019s book used as a source for a business case study)? This isn\u2019t conclusive proof of AI use, but it is solid evidence that the student didn\u2019t do things correctly.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"3\">\n<li><strong>Check facts \u2013<\/strong> Gen AI often produces plausible falsehoods. The text may sound reasonable but include some made-up \u2018facts\u2019. Gen AI is not intelligent, but merely a sophisticated predictive text machine, so if you spot something that seems a bit off, check to see if it is a plausible falsehood.&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"4\">\n<li><strong>Check level of detail \u2013<\/strong> AI tends towards overly-generic output, e.g. using abstract terms with no concrete definitions or examples. Is the essay or report written in generalities or does it include concrete examples in enough detail to support the conclusion that a student wrote it? Again, lack of detail isn\u2019t conclusive evidence that the student cheated but it can be a red flag in combination with other factors.&nbsp;&nbsp;<\/li>\n<\/ol>\n\n\n\n<ol class=\"wp-block-list\" start=\"5\">\n<li><strong>Hold an interview to determine authenticity \u2013 <\/strong>If you see strong indications of unacceptable academic practice, an interview or panel where the student is asked questions about their assignment may be a way to get conclusive evidence. We know this isn\u2019t feasible at large scale, however. This is a sticky problem not only for our university but across the sector.&nbsp;&nbsp;<\/li>\n<\/ol>\n\n\n\n<p>To find out more about Generative AI, see the <a href=\"https:\/\/wordpress.aber.ac.uk\/e-learning\/category\/weekly-resource-roundup\/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Weekly Resource Roundup<\/strong><\/a> for events and materials, e.g. this article specifically about a study on Turnitin\u2019s AI detection: Fowler, G. A. (3\/4\/2023),\u202f<a href=\"https:\/\/eur02.safelinks.protection.outlook.com\/?url=https%3A%2F%2Fwww.washingtonpost.com%2Ftechnology%2F2023%2F04%2F01%2Fchatgpt-cheating-detection-turnitin%2F&amp;data=05%7C01%7C%7C33a3ede90e764cf1493f08db424aac8f%7Cd47b090e3f5a4ca084d09f89d269f175%7C0%7C0%7C638176660709194889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=vlr%2BPysShZprUKUznauOmXvL4e2w6r2LGLciVOJXapg%3D&amp;reserved=0\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>We tested a new ChatGPT-detector for teachers. It flagged an innocent student<\/strong><\/a>,\u202f<em>Washington Post. <\/em>Fowler explains how they tested it, what they found, and why it generated the false results.&nbsp;&nbsp;<\/p>\n\n\n\n<p>In short, if staff don\u2019t see anything suspicious other than the Turnitin AI score, we would recommend against bringing a UAP case forward. There\u2019s too much potential for harm if the student really didn\u2019t cheat.&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This blog post is written by Generative AI Working Group members.&nbsp; The landscape for learning and teaching in the age of Generative AI has been developing rapidly. As staff will be aware, the UAP regulation has been updated to address the use of AI in student assessment. The UAP Form and penalty table have been [&hellip;]<\/p>\n","protected":false},"author":55903,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-6155","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9XLJV-1Bh","_links":{"self":[{"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/posts\/6155","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/users\/55903"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/comments?post=6155"}],"version-history":[{"count":2,"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/posts\/6155\/revisions"}],"predecessor-version":[{"id":6163,"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/posts\/6155\/revisions\/6163"}],"wp:attachment":[{"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/media?parent=6155"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/categories?post=6155"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.aber.ac.uk\/e-learning\/wp-json\/wp\/v2\/tags?post=6155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}