User-agent: Gigabot Disallow: / User-agent: Scrubby Disallow: / User-agent: Robozilla Disallow: / User-agent: Nutch Disallow: / User-agent: ia_archiver Disallow: /*.pdf$ User-agent: * Disallow: /elements/ Disallow: /_designs/ Crawl-delay: 120 User-agent: * Disallow: /howtoapply/courses/undergraduate/current-year/ User-agent: * Disallow: /about-us/uobdocuments/ User-agent: * Disallow: /__data/ Disallow: /media/ User-agent: * Disallow: /student-experience2 User-agent: * Disallow: /search/test-area User-agent: * Disallow: /resources User-agent: * Disallow: /makeithappen