configuration - Add robots.txt to root

admin管理员组

文章数量:1130349

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but .txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite/robots.txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

• configuration
• robots.txt

Share Improve this question asked Feb 25, 2013 at 15:34 Mike DeckMike Deck 11711 silver badge88 bronze badges 2

• I just realized that the "Discourage search engines from indexing this site" setting adds a <meta content="noindex,nofollow" name="robots"> to the head section of each page which should accomplish my goal. I'd still like to know how to add a robots.txt too though. – Mike Deck Commented Feb 25, 2013 at 15:36
• Can't you just manually upload a robots.txt over FTP? WordPress doesn't use it anyway. – user26607 Commented Feb 25, 2013 at 15:44

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 3

First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu.

Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the reason the wordpress setting is ignored.

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite/robots.txt. The function, responsible for creation of this file, is do_robots.

If you wish to add your own directives, just write your hook for robots_txt filter, like this:

add_filter( 'robots_txt', 'wpse8170_my_robots_txt', 10, 2 );
function wpse8170_my_robots_txt( $output, $public ) {
    if ( '0' != $public ) {
        $output .= '
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
';
    }
    return $output;
}

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but .txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with Disallow: / for all user agents.

I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite/robots.txt still returns a 404.

Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?

• configuration
• robots.txt

Share Improve this question asked Feb 25, 2013 at 15:34 Mike DeckMike Deck 11711 silver badge88 bronze badges 2

• I just realized that the "Discourage search engines from indexing this site" setting adds a <meta content="noindex,nofollow" name="robots"> to the head section of each page which should accomplish my goal. I'd still like to know how to add a robots.txt too though. – Mike Deck Commented Feb 25, 2013 at 15:36
• Can't you just manually upload a robots.txt over FTP? WordPress doesn't use it anyway. – user26607 Commented Feb 25, 2013 at 15:44

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 3

First of all, in order for Wordpress to generate a robots.txt for you you must be using a non-default permalink structure. Make sure you've selected an option in the Settings > Permalinks menu.

Also, if a robots.txt file exists at your root directory it will override the setting in Wordpress. It looks like you already have a robots.txt file and that is the reason the wordpress setting is ignored.

You don't need to add robots.txt file to the root of your site. robots.txt file is generated in real time, when you visit http://mysite/robots.txt. The function, responsible for creation of this file, is do_robots.

If you wish to add your own directives, just write your hook for robots_txt filter, like this:

add_filter( 'robots_txt', 'wpse8170_my_robots_txt', 10, 2 );
function wpse8170_my_robots_txt( $output, $public ) {
    if ( '0' != $public ) {
        $output .= '
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
';
    }
    return $output;
}

本文标签： configurationAdd robotstxt to root