Support Center Visit Demo License manager Buy @ envato

Robots.txt file generated by unknown source, can't locate, can't delete, help!

web2seoweb2seo
in SEO
Hello support.

There is a robots.txt file that exists on the live site which disables access to entire website root.

I cannot locate this file using file explorer in CPanel and even using FTP. Its as if it does not exist.

I think there is something that's triggering it from BeTheme. 

I have also disabled Yoast but still no luck. I have not installed any additional SEO extensions either!

So I have no idea where this robots.txt file is generated from, where its located, how this is happening. Its a ghost file!

I even overwrote it with another blank robots.txt file but something is still blocking robots.

Please reply with detailed troubleshooting steps which will help me triangulate the issue and solve this. Otherwise I'm not sure if I should buy BeTheme again for my other clients because this is a very scary and embarrassing problem...

Thanks

Comments

  • web2seoweb2seo
    edited May 2015
    when I type in my URL then add /robots.txt it is showing up btw like below

    User-agent: *
    Disallow: /


    And its also note worthy to point out that this must be a 1 KB file but it does take some time to deliver from server (much longer than usual). Will really appreciate your help as soon as possible.
  • AlbertAlbert
    Hi,

    this is wordpress option and theme does not have to theme. Under Settings > Reading you have an options named "Discourage search engines from indexing this site".
    Learn more: Video Tutorials | How To | FAQ Vote on what comes next
  • MadMax94MadMax94
    I dont know if its a problem of the theme or the problem of yoast.

    But you cant edit the robots.txt file in yoast.
    Yoast > Tools > file editor
    If i want to creat a robots.txt file and click on creat new i get to a white site with many text and i cant edit it.

    Than i install the WP robots txt Plugin.
    And now it works.

    Is this a theme problem ?
  • AlbertAlbert
    No, this is not a theme problem for sure. We do not have any impact on files like robots.txt.
    Learn more: Video Tutorials | How To | FAQ Vote on what comes next
  • camstechnologiescamstechnologies
    It is quite RIDICULOUS to have a default setting which prevents search engines from indexing the site. I have made my site live 2 months back and just discovering this now, all the while wondering why google is not indexing my site

  • AlbertAlbert
    @camstechnologies We understand your thought but you supposed to tell it to WP developers because themes does not have any impact on it. Also, please notice, that while you install WP files themselves, you have a checkbox that inform you if you want to index your site or not. You must checked NOT and that's why your site has not been indexing.
    Learn more: Video Tutorials | How To | FAQ Vote on what comes next
  • JuttaGJuttaG
    I have also a problem with robots.txt.
    Google shows the following for the website I created: A description for this result is not available because of this site's robots.txt
     Discourage search engines from indexing this site is unchecked.
    So, what can I do next?
  • BryanBryan
    @JuttaG How would Discourage search engines from indexing this site help you with this?
    You can check it in settings>reading
    Learn more: Video Tutorials | How To | FAQ Vote on what comes next
  • JuttaGJuttaG
    I have never seen that Google shows "A description for this result is not available because of this site's robots.txt". I don't see a robots.txt on my site in FTP. Have read that BeTheme creates a virtual robots.txt. How can I delete it, so that I have site description?
  • BryanBryan
    You can change the meta description in theme options>SEO>General
    Learn more: Video Tutorials | How To | FAQ Vote on what comes next
  • JuttaGJuttaG
    I have solved the problem by installing Yoast and used the plugin for the metadata etc.
Sign In or Register to comment.