Robots.txt Kya Hai?
robots.txt file ek spider ki marki ki tarh hai jo search engine ko batata hai ki mere website ya blog pe itne saare content hai aur content kis liye likhe gaye hai.
Robots.txt file example hai :-
Example 1
User-agent: Googlebot
Disallow: /nogooglebot/
Example 2
User-agent: *
Allow: /
Sitemap: http://www.example.com/sitemap.xml
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
User-agent: Mediapartners-Google
Yeh code hum Google Adsense Robots ke liye use karte hai jisse aapke blog ko ek better ads Google Adsense provide karta hai.
User-agent: *
Yeha jo aap ek start * Mark dekh rahe hai ye aapke blog pe jo labels links hai unhe Google search result pe index hone se rokta hai
Disallow: /search
Iska use hum isliye karte hai jab google apne search result pe hamare blog se koi content/post dikhayeha to uss result pe search Label ko na dikhaye
agar aap ise removed kar de to Google aisa Result dikhayega
Koi Post Hai jo aap Google search pe nahi dikhana chahte hai to iss robots.txt file code ko use kar sakte hai :-
Disallow: /yyyy/mm/post-url.html
Jaha yyyy year ko darshata hai aur mm month for example
Disallow: /2013/03/post-url.html
robot.txt file blogger pe Kaise Add kare:-
Blogger blog pe robots.txt file Kaise check kare?
robots.txt file Generate Kaise Kare?
