feat: header, disable setting X-Robots-Tag header

[HADB]

🔗 Linked issue

resolves #133

❓ Type of change

• 📖 Documentation (updates to the documentation or readme)
• 🐞 Bug fix (a non-breaking change that fixes an issue)
• 👌 Enhancement (improving an existing functionality)
• ✨ New feature (a non-breaking change that adds functionality)
• 🧹 Chore (updates to the build process or auxiliary tools and libraries)
• ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

Add an option to disable setting X-Robots-Tag header.

[harlan-zw]

Thanks, I think there may be some runtime logic that also applies these headers. Do you think we should be covering that with this flag or should we have two separate flags?

Btw I think we we could disable this feature by default for cloudflare but I'll set up that up outside this PR.

[HADB]

Hi, I pushed a new commit which covers runtime header logic.

[harlan-zw]

Great work, thank you for the help. I'll aim to get this merged and released tomorrow.

If you have any time in the meantime to add a test it would be really appreicated.

[HADB]

I added a test for server response header. I have no idea how to test the generated _headers file. But it works as expected when I run generate script manually.