Technical SEO is rarely about the words on the page - it is about addresses and about the server's responses. A product page in a camera store serves three at once: the shopper in a browser, the search crawler, and the code that walks the site link by link. The goal is single - every valuable entity should have one stable, indexable URL, and its duplicates should not compete for a place in the results.

The naive view is convincing: publish the pages, add links, and the engine will sort it out. It does sort things out fairly well, but the page for a single camera is almost always reachable at a dozen addresses - with a sort parameter, with a campaign tag, over http and https, with and without a trailing slash, inside a catalog filter and in a language variant. To a person this is one page. To a machine - different URLs with identical content.

This is where the model breaks. The crawler spends its limited crawl budget on clones instead of new products. Signals - inbound links, behavior, authority - get smeared across the copies, and none gathers enough weight. A random variant lands in the index: an address with a tracking tag, or a filtered catalog instead of the clean product URL. The user sees an ugly address in the results, and the store sees its rankings slip.

The remedy is simple in principle and demanding in discipline: the URL is the identity of the entity, the server response is the contract with the crawler. Several tools divide the duties, and confusing them is dangerous. robots.txt controls crawling - which paths the robot may request. The sitemap.xml file is a discovery hint: a list of canonical addresses worth finding and recrawling. The canonical tag says which of several similar pages to treat as the main one. The meta robots tag with the value noindex controls indexing - it allows crawling but forbids the display. HTTP status codes report what happened to an address.

The key to all of it is telling crawling and indexing apart - this is where intuition fails. It seems logical to close a utility or duplicate page in robots.txt so it stays out of search. But the block only prevents the page from being requested, not from appearing in the index: with external links the engine may show the address with no description, on the URL alone. Worse, because fetching the page is forbidden, the robot never sees the noindex on it - and there is nothing left to remove it from the index. The rule: to guarantee a page is not indexed, allow the crawl and forbid the display through noindex.