Domainmonster.com Glossary
Please select the glossary subject area you require
If you require further assistance please contact our customer support team.
Domains Glossary
Robots.txt
A robots.txt file is a text file present in the root directory of a website which controls which pages search engines' spiders are permitted to index. This can be useful if you have secure pages, members-only pages, or products for sale such as software or ebooks held on your website. Only spiders which comply with the Robots Exclusion Standard will follow the instructions contained in this file.
Robots.txt files can be used in conjunction with the Robots meta tag.
Web Page Meta Tags - Meta Tags Robots Robots.txt Title Description Keywords
Domainmonster.com - Domain Name Registration Service
Domainmonster.com - Domain Name Registration. Domain Names Cheap Domain Renewal. Domain Transfer