我的 JavaScript 密集型网站对 AI 爬虫不可见——动态渲染能拯救我们吗？

Marcus, we went through this exact journey six months ago. Dynamic rendering was a game-changer for our AI visibility.

Our implementation:

Results after implementing Rendertron:

• AI crawler success rate: 0% → 98%
• ChatGPT citations: 0 → 47 in 90 days
• Perplexity mentions: 0 → 23 in 90 days

Key insight:

The critical part is user agent detection. You need to route these specific bots to pre-rendered pages:

• GPTBot
• ChatGPT-User
• ClaudeBot
• PerplexityBot
• Bytespider

Don’t forget to keep your cached pages fresh. Stale content is worse than no content.

The JavaScript rendering gap is massive and most sites don’t realize it.

Research data:

From Vercel’s crawler study:

• GPTBot: 569 million requests/month (0% JavaScript rendering)
• ClaudeBot: 370 million requests/month (0% JavaScript rendering)
• Googlebot: Renders JavaScript (but with delays)

What AI crawlers actually fetch:

The problem:

They fetch JavaScript files as text but can’t execute them. So if your content relies on JS execution, it’s invisible.

Critical check:

View your page source (not inspect element). If you see mostly empty divs and script tags, AI crawlers see the same thing.

We migrated from Create React App to Next.js specifically for this reason.

The migration path:

1. Week 1-2: Set up Next.js app router
2. Week 3-4: Migrate components (mostly copy-paste)
3. Week 5-6: Implement getServerSideProps/getStaticProps
4. Week 7-8: Testing and deployment

Before/After:

Before (CRA):

<div id="root"></div>
<script src="/static/js/main.chunk.js"></script>

After (Next.js):

<article>
  <h1>Full content here</h1>
  <p>All text visible to crawlers...</p>
</article>

Results:

• First AI citation appeared 3 weeks after launch
• Now averaging 15-20 AI citations per month
• Page load time improved too (bonus!)

The investment was worth it. Dynamic rendering is a band-aid. SSR/SSG is the proper fix.

One thing people miss: you need to actually verify AI crawlers are seeing your content.

How to test:

1. User agent testing:

   curl -A "GPTBot" https://yoursite.com/page
   

2. Check for actual content:

   • Look for your key content in the response
   • Not just a loading spinner or placeholder

3. Monitor in production:

   • Log AI crawler requests
   • Track render success/failure
   • Alert on anomalies

Common failures we’ve seen:

Use Am I Cited to track if it’s working. You can monitor whether you start appearing in AI answers after implementing dynamic rendering. That’s the ultimate validation.

Performance considerations that matter:

Rendering latency:

AI crawlers have timeouts. If your pre-rendered page takes too long:

• GPTBot: Appears to timeout around 30 seconds
• ClaudeBot: Similar behavior
• PerplexityBot: Slightly more patient

Optimization priorities:

1. Cache everything possible - First hit renders, subsequent hits serve cached
2. Prioritize above-fold content - Make sure critical content renders first
3. Lazy load images - But include alt text in initial HTML
4. Minimize third-party scripts - They slow rendering

Our metrics after optimization:

• Average render time: 2.3 seconds
• Cache hit rate: 87%
• Crawler success rate: 99.2%

Don’t forget structured data. Your pre-rendered pages should include schema markup. AI crawlers extract this for understanding content.

For anyone on a tight budget, here’s the quick-win approach:

Minimal viable dynamic rendering:

1. Use Cloudflare Workers - $5/month plan
2. Puppeteer in a Worker - Pre-render on demand
3. Cache in Cloudflare - Serve cached versions

Total cost: ~$10-15/month

Code structure:

• Worker intercepts AI crawler requests
• Puppeteer renders the page
• Cache stores result for 24 hours
• Subsequent requests serve from cache

Our results:

• Setup time: 1 weekend
• Cost: $12/month
• AI visibility: From zero to appearing in ChatGPT within 6 weeks

It’s not as robust as Prerender.io or Next.js, but it works for startups.

Client case study perspective:

Client situation:

• Large ecommerce site (50k products)
• Angular SPA with client-side rendering
• Zero AI visibility
• Competitors dominating AI recommendations

Implementation:

• Prerender.io (chose managed for their scale)
• Enterprise plan for high-volume caching
• Custom integration with their CMS

Timeline:

• Week 1-2: Integration
• Week 3-4: Cache warming (50k pages)
• Month 2: First AI citations detected
• Month 3: 340% increase in AI visibility

Cost-benefit:

• Prerender.io cost: $499/month (enterprise)
• Additional AI-driven traffic value: ~$15k/month
• ROI: Clear win

Key learning:

For large sites, the cache warming phase is critical. You can’t wait for AI crawlers to discover all your pages. Pre-render proactively.

Controversial take: maybe stop building JavaScript-heavy sites?

The broader picture:

• AI crawlers can’t render JS
• Some users have JS disabled
• Slow networks struggle with JS bundles
• Accessibility tools often struggle with SPAs

Progressive enhancement:

Consider building sites that work without JavaScript, then enhance with JS:

1. Server renders full HTML
2. JavaScript adds interactivity
3. Works for everyone - humans and bots

Modern tools that help:

• Astro (partial hydration)
• SvelteKit (SSR by default)
• Next.js (hybrid rendering)
• Nuxt (same approach)

Dynamic rendering is a workaround for a problem we created. The real solution is building accessible-by-default.

This thread gave me a clear path forward. Here’s our plan:

Short-term (next 2 weeks):

• Implement Rendertron for immediate AI visibility
• User agent detection for GPTBot, ClaudeBot, PerplexityBot
• 24-hour cache TTL with event-based invalidation

Medium-term (next quarter):

• Evaluate Next.js migration for key pages
• A/B test SSR vs dynamic rendering performance
• Build monitoring dashboard for AI crawler access

Long-term (6 months):

• Full migration to hybrid rendering framework
• Server-side rendering for all indexable content
• Client-side enhancement for interactivity

Key metrics I’ll track:

• AI crawler success rate (target: >95%)
• Time to first AI citation
• Citation volume over time
• Cache efficiency

The investment breakdown:

• Rendertron hosting: ~$50/month
• Engineering time: 2 weeks
• Expected ROI: AI visibility within 60 days

Thanks everyone. The data on crawler behavior and implementation details were exactly what I needed.

For anyone else with JS-heavy sites: this is no longer optional. AI crawlers are a significant traffic source and they can’t see your JavaScript content.