User-agent: Scrapy Allow: / User-agent: * Allow: / Allow: /*.js* Allow: /*.css* Allow: /*.png* Allow: /*.jpg* Allow: /*.gif* Disallow: /QA/* Disallow: /QA-UAT/* Disallow: /IT-Information/* Disallow: /Video/* User-agent: Jooblebot Disallow: /