Skip to content

Where is robots.txt

The robots.txt file is a public text file that lives in the root directory of your website.

1. How to find (view) it

You can view any website’s robots.txt file by adding /robots.txt to the end of the domain name in your browser address bar.

  • Format: https://yourwebsite.com/robots.txt
  • Example: If your site is example.com, type https://example.com/robots.txt into Chrome or Safari.

If you see a plain text page with code like User-agent: *, it exists. If you get a 404 error, you do not have one yet.

2. How to edit it (by platform)

While the file “lives” at that URL, you usually cannot edit it there directly. You must edit it through your website builder or hosting file manager.

If you use WordPress

WordPress creates a “virtual” robots.txt automatically, so you might not see a physical file in your folders. To edit it, use a plugin (safest method):

  • Yoast SEO: Go to Yoast SEO > Tools > File Editor. You will see a button to “Create robots.txt file” if one doesn’t exist, or an editor if it does.
  • RankMath: Go to RankMath > General Settings > Edit robots.txt.
  • All In One SEO: Go to All In One SEO > Tools > Robots.txt Editor.

If you use Shopify

Shopify generates this file automatically, but you can override it using the code editor.

  1. Go to Online Store > Themes.
  2. Click the … (Three dots) button next to your current theme > Edit code.
  3. Under the “Templates” folder, click Add a new template.
  4. Select robots from the dropdown menu (it will create robots.txt.liquid).
  5. You can now add your custom allow/disallow rules here.
See also  How to disable "XML-RPC" - a popular backdoor that hackers often use to bypass login pages?

If you use Wix

Wix used to hide this, but now allows editing via the dashboard.

  1. Go to your Wix Dashboard.
  2. Navigate to Marketing & SEO > SEO > SEO Settings.
  3. Scroll down to Robots.txt and click Edit.
  4. It will show you the default rules; you can add your AI-specific rules at the bottom.

If you use Squarespace

Squarespace is more restrictive. You cannot manually edit the text file directly to add custom code for specific bots.

  • You generally only have toggle switches in Settings > Crawlers to “Block Artificial Intelligence crawlers.”
  • Warning: Toggling this off (allowing them) is good, but you cannot granularly invite specific bots (like GPTBot) while blocking others if Squarespace doesn’t provide that specific toggle.

If you have a Custom Website (HTML/PHP)

You need to access your server via FTP (like FileZilla) or your Hosting File Manager (cPanel).

  1. Open the public_html folder (or www folder).
  2. Look for a file named robots.txt.
  3. If it’s not there, create a new plain text file on your computer, name it robots.txt, paste your rules in, and upload it to that folder.

Robot.txt for subdomain

A subdomain (like store.yoursite.com or blog.yoursite.com) is treated by AI and Google as a completely separate website.

It does not obey the robots.txt of your main domain. It needs its own.

1. Where to find it

It is located at the root of the subdomain itself.

  • Main Domain: https://example.com/robots.txt
  • Subdomain: https://sub.example.com/robots.txt

If you visit https://sub.yourdomain.com/robots.txt and get a 404 error, that subdomain has no instructions for AI, which usually means “come in and take everything” (unless specific meta tags block it).

See also  Adding subtitles in multiple languages to videos on a WordPress website

2. How to edit it (It depends on your setup)

Since subdomains are often different software than the main site, you usually have to edit them in a different place.

Scenario A: It’s a different platform (Most Common)

Often, a main site is on WordPress, but the subdomain is something else (e.g., shop.ksml4.com on Shopify, or docs.ksml4.com on GitBook).

  • If it’s Shopify: You must edit the robots.txt.liquid in the Shopify theme code (Online Store > Themes > Edit Code).
  • If it’s a specialized app (HubSpot, GitBook, Teachable): These platforms usually have a specific “SEO” or “Tracking” settings page where you can paste robots.txt content. They rarely give you FTP access.

Scenario B: It’s a WordPress Multisite

If you run a network where site1.ksml4.com and site2.ksml4.com are all on one WordPress install:

  • You cannot just upload a file to FTP. WordPress virtualizes these files.
  • You must use a plugin like robots.txt Editor or Yoast SEO specifically while logged into that subdomain’s dashboard, not the main network dashboard.

Scenario C: It’s a simple folder on your server

If you created a subdomain in cPanel that just points to a folder (e.g., public_html/subdomain_folder):

  1. Open your File Manager.
  2. Go inside the specific folder for that subdomain.
  3. Create a file named robots.txt inside that folder.

Critical Note for AI

If you want AI to discover your subdomain, you should cross-reference it.

In the robots.txt of your Main Domain, add a reference to the subdomain’s Sitemap:

Plaintext

# Inside the MAIN domain's robots.txt
Sitemap: https://ksml4.com/sitemap_index.xml
Sitemap: https://shop.ksml4.com/sitemap.xml  <-- Add this!

This acts as a “bridge,” telling the bots crawling your main site that the subdomain exists and is relevant.

Leave a Reply

error: Content is protected !!