When user-agent and allow or disallow are together, it really is regarded as a solitary entry in a file in which the action is only going to be employed to the user agent that is specified. An asterisk (*) if you’d like to direct this to multiple user-agents, list.
You shall then must make sure that the robots.txt file is conserved. Make certain you perform some after to ensure that internet crawlers should be able to find and recognize your file.
- Save the robots.txt file as being a text file.
- Put the file inside the directory that is highest-level of site ( or perhaps in the basis of this domain).
- The file needs to be called robots.txt.
- Example: a robots.txt file that is conserved in the cause of test.com with all the Address of http://www.sample.com/robots.txt is discoverable by internet crawlers, if the robots.txt file is situated at a URL like http://www.sample.com/not_root/robots.txt it’ll maybe maybe not manage to be found by internet crawlers. ادامه مطلب