Web robots.txt disallows indexing

Question

Hi, we are using Odoo 16 and we have problem with website generated in Odoo. Robots.txt have following content:

User-agent: *
Disallow: /
Even when I change the content of robots.txt in webpage Settings to:
Disallow: /web/login
Allow: *
I can see in the website source code:  meta name="robots" content="noindex"
How to solve it?
Many thanks in advance.

Martin Riley · Accepted Answer

I had the same issue. This is how I fixed it.

Go to * Website * > * Site * > * HTML/CSS Editor *  * *  * *  *
*  Click * Edit HTML Anyway *  * * * *
From the first dropdown, select * XML (HTML) *  *
*  From the second dropdown, select * Main Layout * . (The template ID of the main layout is * website.layout * )  * *  * *
Go to line 44.
Find where it says  * (website.domain and not website._is_indexable_url(request.httprequest.url_root)) *  *
*  Replace  * request.httprequest.url_root *  * *   w

Lumír Jasiok · Answer

Just for reference, I found the solution. Problem was, that Odoo is internally using http protocol and we had in the Website Settings in the field Domain URL with HTTPS, which was working with the combination of Nginx proxy, but it didn't fit the condition stated here:
https://github.com/odoo/odoo/blob/df3811e745cec7375ae4eba4ac2a82128fa6dab0/addons/website/views/website_templates.xml#L2401 [1]

So, when I changed the Domain URL from https://our-domnain.com to the http://our-domain.com, it worke

Câu hỏi này đã bị gắn cờ

Bạn có hứng thú với cuộc thảo luận không? Đừng chỉ đọc, hãy tham gia nhé!