Teams reason like this: since a page is published, a search engine will find it and show it. But between publishing and showing lies a pipeline of several stages, and ranking - choosing the position in results - begins only at the very end, once the page is discovered, fetched, and understood. Stuck at an early stage, and no keywords and no backlinks help: there is nothing to rank.
The path to results is made of discovering the URL, fetching the content, rendering JavaScript when needed, extracting the content, choosing the canonical version, and finally indexing. Any stage can filter the page out, and then it never reaches ranking at all. Keep the whole pipeline in view: most search problems are not ranking but a break at an early step.
| Stage | What happens | Common break |
|---|---|---|
| Discovery | the URL is found via links and sitemap | nothing links to the page |
| Crawling | the robot downloads the HTML | the path is disallowed in robots.txt |
| Rendering | JavaScript runs | the content never appears |
| Canonicalization | the main version is chosen from duplicates | a different canonical is picked |
| Indexing | the page enters the index | noindex is set |
Discovery rests on links and sitemaps: the engine learns of a URL from a link on a known page or from a sitemap. A page with no inbound link barely exists for the crawler. Next is crawling - the robot makes an HTTP request and downloads the HTML. But first Googlebot reads robots.txt: if the path is disallowed, no request is sent and the robot never sees the content.
A modern site often serves an empty shell and lets JavaScript paint the content in the browser. Googlebot can handle this - it has a rendering service on the engine of current Chromium - but rendering is deferred and costs resources, so it does not happen at once. Take a product card in a multilingual shop: if the price, description, and title arrive only after a client-side request to an API, the engine sees them only once the render is done. Until the critical content and links survive in the rendered HTML, they do not exist for search - it is safer to deliver them from the server.
After extracting the content, the engine decides which version is canonical. One product is often reachable at several addresses - with sort parameters, with a campaign tag, in different letter case - and all are duplicates of the same content. The system picks one canonical and indexes it, while rel=canonical in the markup is a hint, not an order: with contradictory signals the engine may pick a different URL. Only after clearing this step does the page enter the index and become a candidate to be shown.
Two mechanisms are constantly confused, and the cost of the mistake is high. robots.txt governs crawling - whether to let the robot reach a URL - but you cannot remove a page from results with it. A disallowed address still enters the index if it is linked from elsewhere: the engine shows a bare URL with no description. To guarantee a page is not indexed you need a meta robots with noindex or an X-Robots-Tag header - and here is the paradox: the page must stay open to crawling. Block it in robots.txt and the robot never reaches the noindex and never learns of the prohibition.
# robots.txt - governs crawling, not indexing
User-agent: *
Disallow: /cart/
<!-- For a page not to be indexed, robots.txt must NOT block it -->
<meta name="robots" content="noindex">
# The same meaning via an HTTP header:
X-Robots-Tag: noindexA classic failure: a utility section is hidden by blocking it in robots.txt and adding noindex at once - and the result is the opposite. The robot, hitting the disallow, does not download the page, does not see the noindex, and through external links still enters the bare URL into results. The section they meant to hide ends up in plain view. The discipline comes down to the order of the stages: first make sure the page is discoverable and its content survives in the rendered HTML; set the canonical explicitly and consistently; apply robots.txt and noindex strictly for their own purpose, never muting with the second what the first has already blocked.