Boosting Blog Post Visibility: Building an Automation System with the IndexNow API

I'm sure many of you have experienced the frustration of publishing a new blog post only to find it's not immediately visible in search engine results. I recently learned that search engines like Bing and Yandex offer a way to quickly notify them of new posts via the IndexNow API. So, I decided to integrate this feature into my blog.

Attempts and Pitfalls

Initially, I created helper functions in services/indexnow_service.py to call the IndexNow API when a post was published. I structured the code to use asyncio.create_task to send a ping asynchronously whenever the post status changed to 'published' in the BlogRepository.update_status method.

# services/indexnow_service.py (partial)
import asyncio
import httpx
async def ping_urls(urls: list[str], api_key: str):
async with httpx.AsyncClient() as client:
for url in urls:
try:
response = await client.post(
"https://api.indexnow.org/submit-url",
json={"url": url, "key": api_key}
)
response.raise_for_status()
print(f"Successfully pinged {url}")
except httpx.HTTPStatusError as e:
print(f"Error pinging {url}: {e}")
except Exception as e:
print(f"An unexpected error occurred for {url}: {e}")
async def ping_blog_post(post_url: str, api_key: str):
await ping_urls([post_url], api_key)
BlogRepository.update_status (partial)
async def update_status(self, post_id: int, new_status: str):
# ... existing logic ...
if new_status == 'published' and INDEXNOW_KEY:
post = await self.get_post_by_id(post_id) # In reality, you'd get the URL from the post object
asyncio.create_task(ping_blog_post(post.url, INDEXNOW_KEY))
# ...

I also created an admin API endpoint to manually trigger pings. I set up the public/<KEY>.txt file and even configured middleware. But to my surprise, the pings just wouldn't go through, no matter what I tried. After about three hours of debugging, I discovered that the ownership verification file required by the IndexNow API had a different path than I expected. Sometimes, it needed to be accessed not as /public/<KEY>.txt, but simply as /KEY.txt.

The Cause

Ultimately, the problem lay in how the IndexNow API verifies ownership via the verification file. My setup placed the file inside the public/ directory, but IndexNow prefers it directly in the root directory, or it has stricter requirements for specific path configurations. Additionally, the INDEXNOW_KEY environment variable might not have been set correctly, disabling the feature.

The Solution

To resolve this, I made a few adjustments:

1. Corrected Ownership File Path: I removed the public/ directory and changed the configuration to place the KEY.txt file directly in the root directory. I configured the web framework's middleware to serve this file directly.
2. Enhanced Environment Variable Check: I added logic to explicitly check if the INDEXNOW_KEY environment variable was set and if it contained a valid value.
3. Improved Asynchronous Ping Logic: In BlogRepository.update_status, I continued to use asyncio.create_task to ensure the ping request wouldn't block the main request flow.

# services/indexnow_service.py (after modification)
import asyncio
import httpx
import os
INDEXNOW_KEY = os.environ.get("INDEXNOW_KEY")
async def ping_urls(urls: list[str]):
if not INDEXNOW_KEY:
print("INDEXNOW_KEY is not set. Skipping ping.")
return
async with httpx.AsyncClient() as client:
    for url in urls:
        try:
            response = await client.post(
                "https://api.indexnow.org/submit-url",
                json={"url": url, "key": INDEXNOW_KEY}
            )
            response.raise_for_status()
            print(f"Successfully pinged {url}")
        except httpx.HTTPStatusError as e:
            print(f"Error pinging {url}: {e}")
        except Exception as e:
            print(f"An unexpected error occurred for {url}: {e}")

async def ping_blog_post(post_url: str):
await ping_urls([post_url])
main.py or app.py (example middleware setup)
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

app = FastAPI()

# Configure to serve KEY.txt file directly from the root directory
app.mount("/", StaticFiles(directory=".", html=True), name="static")

# BlogRepository.update_status (after modification)
async def update_status(self, post_id: int, new_status: str):
# ... existing logic ...
if new_status == 'published' and INDEXNOW_KEY:
post = await self.get_post_by_id(post_id)
asyncio.create_task(ping_blog_post(post.url))
# ...
Example admin API endpoint
@router.post("/blog/indexnow-ping-all")
async def indexnow_ping_all():
all_posts = await blog_repository.get_all_published_posts()
for post in all_posts:
asyncio.create_task(ping_blog_post(post.url))
return {"message": "Initiated ping for all published posts."}

Results

• The time it takes for posts to appear in search engine results after publication has noticeably decreased.
• The ability to enable or disable the feature at any time via the INDEXNOW_KEY environment variable allows for secure management.
• Thanks to the admin API, initial setup scenarios and batch pinging of any missed posts have become much easier.
• asyncio.create_task ensures that pings are handled in the background, having no impact on the user experience.

Summary — Avoiding the Same Pitfalls

• [ ] When using the IndexNow API, always double-check the exact path configuration for the ownership verification file (KEY.txt). You need to verify your web framework's static file serving settings.
• [ ] The INDEXNOW_KEY environment variable is mandatory; manage it securely for enabling/disabling the feature.
• [ ] Process IndexNow pings for post publications asynchronously (asyncio.create_task) to avoid degrading user experience.
• [ ] Building an admin API to add a batch ping function for all posts is extremely useful during initial setup and for re-processing.