PHP में Search Engine Bots Detect करना

User-Agent string का विश्लेषण करके search engine crawlers (bots) की पहचान करने के तरीके।

विधि 1: Simple String Matching

<?php
function isBot($userAgent = null) {
    $userAgent = $userAgent ?? ($_SERVER['HTTP_USER_AGENT'] ?? '');
    $userAgent = strtolower($userAgent);
    
    $botKeywords = [
        'googlebot', 'bingbot', 'slurp', 'duckduckbot',
        'baiduspider', 'yandexbot', 'facebot', 'twitterbot',
        'linkedinbot', 'whatsapp', 'telegrambot', 'applebot'
    ];
    
    foreach ($botKeywords as $bot) {
        if (strpos($userAgent, $bot) !== false) {
            return true;
        }
    }
    return false;
}

// उपयोग
if (isBot()) {
    // Bot-specific logic (no analytics, simplified HTML, etc.)
    echo "Bot detected";
}

विधि 2: Regex Pattern (अधिक व्यापक)

<?php
function isBotRegex($userAgent = null) {
    $userAgent = $userAgent ?? ($_SERVER['HTTP_USER_AGENT'] ?? '');
    
    // Common bot patterns
    $pattern = '/bot|crawl|spider|slurp|fetch|archiver|monitor|check|validator|preview|api|uptime|ping|health|monitor/i';
    
    return (bool)preg_match($pattern, $userAgent);
}

विधि 3: Known Bot List के साथ (अधिक सटीक)

<?php
class BotDetector {
    private static $knownBots = [
        // Search engines
        'Googlebot' => 'Google',
        'Bingbot' => 'Bing',
        'Slurp' => 'Yahoo',
        'DuckDuckBot' => 'DuckDuckGo',
        'Baiduspider' => 'Baidu',
        'YandexBot' => 'Yandex',
        'Sogou' => 'Sogou',
        'Exabot' => 'Exalead',
        
        // Social media
        'facebookexternalhit' => 'Facebook',
        'Twitterbot' => 'Twitter',
        'LinkedInBot' => 'LinkedIn',
        'WhatsApp' => 'WhatsApp',
        'TelegramBot' => 'Telegram',
        
        // Tools & Services
        'AhrefsBot' => 'Ahrefs',
        'SemrushBot' => 'Semrush',
        'MJ12bot' => 'Majestic',
        'DotBot' => 'DotBot',
        'UptimeRobot' => 'UptimeRobot',
        'Pingdom' => 'Pingdom',
    ];
    
    public static function detect($userAgent = null): ?array {
        $userAgent = $userAgent ?? ($_SERVER['HTTP_USER_AGENT'] ?? '');
        
        foreach (self::$knownBots as $pattern => $name) {
            if (stripos($userAgent, $pattern) !== false) {
                return ['name' => $name, 'pattern' => $pattern, 'is_bot' => true];
            }
        }
        
        // Generic bot detection
        if (preg_match('/bot|crawl|spider|slurp/i', $userAgent)) {
            return ['name' => 'Unknown Bot', 'is_bot' => true];
        }
        
        return null; // Not a bot
    }
    
    public static function isBot($userAgent = null): bool {
        return self::detect($userAgent) !== null;
    }
}

// उपयोग
$bot = BotDetector::detect();
if ($bot) {
    error_log("Bot visit: {$bot['name']} from {$_SERVER['REMOTE_ADDR']}");
    // Don't track in analytics, serve simplified version, etc.
}

विधि 4: Verified Bot Detection (Reverse DNS) - उच्च सटीकता के लिए

<?php
function isVerifiedGooglebot($ip = null) {
    $ip = $ip ?? ($_SERVER['REMOTE_ADDR'] ?? '');
    
    // 1. User-Agent जांचें
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (stripos($ua, 'Googlebot') === false) {
        return false;
    }
    
    // 2. Reverse DNS lookup
    $hostname = gethostbyaddr($ip);
    
    // 3. Verify it's googlebot.com or google.com
    if (!preg_match('/\.(googlebot\.com|google\.com)$/i', $hostname)) {
        return false;
    }
    
    // 4. Forward DNS verification (optional but recommended)
    $forwardIp = gethostbyname($hostname);
    return $forwardIp === $ip;
}

// उपयोग
if (isVerifiedGooglebot()) {
    // Definitely Googlebot
}

उपयोग के मामले

• Analytics filtering: Bot traffic को exclude करें
• SEO optimization: Bots को pre-rendered/SSR content दें
• Rate limiting: Bots के लिए अलग limits
• Content protection: Email addresses, phone numbers को bots से छिपाएं
• Performance: Bots के लिए heavy scripts skip करें

चेतावनी

User-Agent spoof किया जा सकता है। महत्वपूर्ण निर्णयों के लिए (जैसे content cloaking), हमेशा Reverse DNS verification का उपयोग करें। Google, Bing, और अन्य प्रमुख search engines verification guidelines प्रदान करते हैं।