Boosting Blog Post Visibility: Building an Automation System with the IndexNow API

I'm sure many of you have experienced the frustration of publishing a new blog post only to find it's not immediately visible in search engine results. I recently learned that search engines like Bing and Yandex offer a way to quickly notify them of new posts via the IndexNow API. So, I decided to integrate this feature into my blog.

Attempts and Pitfalls

Initially, I created helper functions in services/indexnow_service.py to call the IndexNow API when a post was published. I structured the code to use asyncio.create_task to send a ping asynchronously whenever the post status changed to 'published' in the BlogRepository.update_status method.

# services/indexnow_service.py (partial)
import asyncio
import httpx

async def ping_urls(urls: list[str], api_key: str):
    async with httpx.AsyncClient() as client:
        for url in urls:
            try:
                response = await client.post(
                    "https://api.indexnow.org/submit-url",
                    json={"url": url, "key": api_key}
                )
                response.raise_for_status()
                print(f"Successfully pinged {url}")
            except httpx.HTTPStatusError as e:
                print(f"Error pinging {url}: {e}")
            except Exception as e:
                print(f"An unexpected error occurred for {url}: {e}")

async def ping_blog_post(post_url: str, api_key: str):
    await ping_urls([post_url], api_key)

# BlogRepository.update_status (partial)
async def update_status(self, post_id: int, new_status: str):
    # ... existing logic ...
    if new_status == 'published' and INDEXNOW_KEY:
        post = await self.get_post_by_id(post_id) # In reality, you'd get the URL from the post object
        asyncio.create_task(ping_blog_post(post.url, INDEXNOW_KEY))
    # ...

I also created an admin API endpoint to manually trigger pings. I set up the public/<KEY>.txt file and even configured middleware. But to my surprise, the pings just wouldn't go through, no matter what I tried. After about three hours of debugging, I discovered that the ownership verification file required by the IndexNow API had a different path than I expected. Sometimes, it needed to be accessed not as /public/<KEY>.txt, but simply as /KEY.txt.

The Cause

Ultimately, the problem lay in how the IndexNow API verifies ownership via the verification file. My setup placed the file inside the public/ directory, but IndexNow prefers it directly in the root directory, or it has stricter requirements for specific path configurations. Additionally, the INDEXNOW_KEY environment variable might not have been set correctly, disabling the feature.

The Solution

To resolve this, I made a few adjustments:

Corrected Ownership File Path: I removed the public/ directory and changed the configuration to place the KEY.txt file directly in the root directory. I configured the web framework's middleware to serve this file directly.
Enhanced Environment Variable Check: I added logic to explicitly check if the INDEXNOW_KEY environment variable was set and if it contained a valid value.
Improved Asynchronous Ping Logic: In BlogRepository.update_status, I continued to use asyncio.create_task to ensure the ping request wouldn't block the main request flow.

# services/indexnow_service.py (after modification)
import asyncio
import httpx
import os

INDEXNOW_KEY = os.environ.get("INDEXNOW_KEY")

async def ping_urls(urls: list[str]):
    if not INDEXNOW_KEY:
        print("INDEXNOW_KEY is not set. Skipping ping.")
        return

    async with httpx.AsyncClient() as client:
        for url in urls:
            try:
                response = await client.post(
                    "https://api.indexnow.org/submit-url",
                    json={"url": url, "key": INDEXNOW_KEY}
                )
                response.raise_for_status()
                print(f"Successfully pinged {url}")
            except httpx.HTTPStatusError as e:
                print(f"Error pinging {url}: {e}")
            except Exception as e:
                print(f"An unexpected error occurred for {url}: {e}")

async def ping_blog_post(post_url: str):
    await ping_urls([post_url])

# main.py or app.py (example middleware setup)
# from fastapi import FastAPI
# from fastapi.staticfiles import StaticFiles
#
# app = FastAPI()
#
# # Configure to serve KEY.txt file directly from the root directory
# app.mount("/", StaticFiles(directory=".", html=True), name="static")
#
# # BlogRepository.update_status (after modification)
# async def update_status(self, post_id: int, new_status: str):
#     # ... existing logic ...
#     if new_status == 'published' and INDEXNOW_KEY:
#         post = await self.get_post_by_id(post_id)
#         asyncio.create_task(ping_blog_post(post.url))
#     # ...

# Example admin API endpoint
# @router.post("/blog/indexnow-ping-all")
# async def indexnow_ping_all():
#     all_posts = await blog_repository.get_all_published_posts()
#     for post in all_posts:
#         asyncio.create_task(ping_blog_post(post.url))
#     return {"message": "Initiated ping for all published posts."}

Results

The time it takes for posts to appear in search engine results after publication has noticeably decreased.
The ability to enable or disable the feature at any time via the INDEXNOW_KEY environment variable allows for secure management.
Thanks to the admin API, initial setup scenarios and batch pinging of any missed posts have become much easier.
asyncio.create_task ensures that pings are handled in the background, having no impact on the user experience.

Summary — Avoiding the Same Pitfalls

[ ] When using the IndexNow API, always double-check the exact path configuration for the ownership verification file (KEY.txt). You need to verify your web framework's static file serving settings.
[ ] The INDEXNOW_KEY environment variable is mandatory; manage it securely for enabling/disabling the feature.
[ ] Process IndexNow pings for post publications asynchronously (asyncio.create_task) to avoid degrading user experience.
[ ] Building an admin API to add a batch ping function for all posts is extremely useful during initial setup and for re-processing.