# SEO - Common robots.txt **URL:** \ **Category:** SEO and Website Performance\ **Tags:** seo\_meta\ **Created:** [April 26, 2022, 12:08pm UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677 "2022-04-26T12:08:29Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![AJ\_Joe](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@AJ\_Joe](https://community.sitejet.io/u/AJ_Joe)\ **Post date:** [April 26, 2022, 12:08pm UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/1 "2022-04-26T12:08:29Z")
What would be the common robots.txt for all SiteJet websites? Usually when we develop a website we may disallow cache folder for example and other internal dev folders. ---
**Author:** ![Lucian\_Dinu](https://dub1.discourse-cdn.com/flex017/user_avatar/community.sitejet.io/lucian_dinu/32/141_2.png) [@Lucian\_Dinu](https://community.sitejet.io/u/Lucian_Dinu)\ **Post date:** [April 27, 2022, 5:44am UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/2 "2022-04-27T05:44:40Z")
Hi! If you need, you can create a `robots.txt` file using the instructions from here: [Add a robots.txt file - Sitejet Help](https://help.sitejet.io/article/740-add-robots-txt) From what I know, there is no cache other internal dev. folders in Sitejet. Hope this helps! ---
**Author:** ![AJ\_Joe](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@AJ\_Joe](https://community.sitejet.io/u/AJ_Joe)\ **Post date:** [May 17, 2022, 2:55pm UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/3 "2022-05-17T14:55:12Z")
Hi @Lucian_Dinu, thank you for your reply. I have already followed the instructions before asking the question. My question is about the content of the `robots.txt` and not on how to create one (as in your shared link). Usually, when we develop a website, we must exclude all the internal folder that we do not want to share on the internet. In case of SiteJet, it is a black box for us, so I wonder why you don’t share a common `robots.txt` that we may use in our website as a base one. ---
**Author:** ![Lucian\_Dinu](https://dub1.discourse-cdn.com/flex017/user_avatar/community.sitejet.io/lucian_dinu/32/141_2.png) [@Lucian\_Dinu](https://community.sitejet.io/u/Lucian_Dinu)\ **Post date:** [May 17, 2022, 7:11pm UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/4 "2022-05-17T19:11:21Z")
Hi! This is what is currently available in Sitejet. I have a similar issue and I also want to exclude pages/collection items this is why I’ve created this feature request: [Add the possibility to exclude pages/collection item pages from sitemap.xml](https://community.sitejet.io/t/add-the-possibility-to-exclude-pages-collection-item-pages-from-sitemap-xml/416) If you want you can add your input (+1) there. Hope this helps! ---
**Author:** ![AJ\_Joe](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@AJ\_Joe](https://community.sitejet.io/u/AJ_Joe)\ **Post date:** [May 18, 2022, 11:50am UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/5 "2022-05-18T11:50:36Z")
Hi, this should be answered by the dev team of SiteJet. There should be a common base robots.txt file to share with us or to generate on the website and give us the possibility to modify it. Maybe @Andre or @Franzi could help us with this. As for your feature request. I totally agree with you. There should be a possibility to have draft pages along with this feature [Save Without Publishing](https://community.sitejet.io/t/save-without-publishing/1754). ---
**Author:** ![malte](https://dub1.discourse-cdn.com/flex017/user_avatar/community.sitejet.io/malte/32/28_2.png) [@malte](https://community.sitejet.io/u/malte)\ **Post date:** [May 18, 2022, 2:15pm UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/6 "2022-05-18T14:15:26Z")
Hi @AJ_Joe there is already a default robots.txt automatically generated for every website. You can download it (simply append `/robots.txt` to your website domain) and customize it to your needs and upload it again using the instructions mentioned above. Usually a search engine would index every (linked) page, so a customized robots.txt is not needed in most cases. If you want to disallow pages from being indexed, you can better simply untick the “Index” checkbox in the “Pages” dialog. ---
**Author:** ![AJ\_Joe](https://avatars.discourse-cdn.com/v4/letter/a/d26b3c/32.png) [@AJ\_Joe](https://community.sitejet.io/u/AJ_Joe)\ **Post date:** [May 18, 2022, 2:22pm UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/7 "2022-05-18T14:22:37Z")
> [@malte](#): > > simply append `/sitemap.xml` to your website domain Hi @malte, thank you for your reply. You mean simply add `/robots.txt` to the website domain? ---
**Author:** ![Myladuke\_d](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@Myladuke\_d](https://community.sitejet.io/u/Myladuke_d)\ **Post date:** [July 28, 2026, 10:47am UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/8 "2026-07-28T10:47:56Z")
I usually just allow all crawlers in robots.txt since Sitejet has no hidden folders. You can add custom rules in your dashboard. ---
**Author:** ![John\_Frey](https://avatars.discourse-cdn.com/v4/letter/j/97f17d/32.png) [@John\_Frey](https://community.sitejet.io/u/John_Frey)\ **Post date:** [July 31, 2026, 7:00am UTC](https://community.sitejet.io/t/seo-common-robots-txt/1677/9 "2026-07-31T07:00:38Z")
adding robots.txt is a good start. for broader seo improvements, i’ve been using link building as it helps to boost real organic traffic