Exporting Google and Bing Results with PowerShell

24 May

Whilst doing a bit of SEO work, I needed to remove some URL’s from both Bing and Google using their webmaster tools.

Rather than doing it manually and copying the links from each page, I ended up writing some PowerShell, which may or may not slightly violate the TOC’s, so this is purely for educational use, ok? (The sleep is to prevent the Search Engines blocking you, which will happen if you make too many requests in a short space of time)

#variables
$mydomain = "www.mydomain.com"
$sleepinterval = 4
$loopmax = 30

#scrape google
clear-host
for ($i=0; $i -le $loopmax; $i++){
    start-sleep $sleepinterval
    $url = "https://www.google.com/search?q=site:$mydomain&start=" + $i*10
    ((Invoke-WebRequest –Uri $url).Links | Where-Object {$_.href -like “*://$mydomain*”} ).href | ForEach-Object {
    $temp = $_
    $temp = $temp.split(":")
    $temp2 = $temp[4]
        try{
            $temp2.replace("//","").split("%")[0]
        }catch{
        }
    }
}

#scrape bing
clear-host
for ($i=0; $i -le $loopmax; $i++){
    start-sleep $sleepinterval
    $page = ($i*10)+1
    $url = "https://www.bing.com/search?q=site%3a$mydomain&sp=1&first=$page" 
    #write-host $url
    ((Invoke-WebRequest –Uri $url).Links | Where-Object {$_.href -like “*://$mydomain*”} ).href | ForEach-Object {
    write-host $_
    }
}