Integrating Crawlerly into Your Site

Crawlerly is designed for developers who need to filter traffic. There could be many reasons such as securing a PayWall or filtering crawler traffic from log files. Crawlerly makes the task very easy, you make a call to our URL passing an IP address to check and we will return a boolean value telling you if it's a crawler or not.

All you really need to know is the URL: https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=[IPAddressToLookUp]. Simply replace the lables with your given token which you get when registering and the IP address you wish to look up.

The hardest part is making a web call to get the code from our page. This can be done from any language though. Here are a few examples. If you need help, please contact our support team.

The results from our URL are either True or False.

    True = it is associated with a Crawler
    False = no it is not associated with a Crawler



VB.net
Imports System.Net
Imports System.Xml
Imports System.IO

Private Function getHTMLFeed(ByVal URL As String) As String
    Try
        ServicePointManager.Expect100Continue = True
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls Or SecurityProtocolType.Tls11 Or SecurityProtocolType.Tls12 Or SecurityProtocolType.Ssl3
        
        Dim request As HttpWebRequest = CType(WebRequest.Create(URL), HttpWebRequest)
        Dim response As WebResponse = request.GetResponse()
        
        Dim rssStream As Stream = response.GetResponseStream()
        Dim rssDoc As New XmlDocument()
        
        Dim SNWC As New System.Net.WebClient()
        Dim reader As StreamReader = New StreamReader(rssStream)
        Dim responseFromServer As String = reader.ReadToEnd()
        
        Dim Contents As String = responseFromServer
        Contents = Contents.Trim()
        
        Return Contents.ToString()
        Catch ex As Exception
        Return "False"
        End Try
        End Function
        
        Dim ValidIPAddress as Boolean = False
        Dim InputIPAddress as string = "192.168.5.10"
        ValidIPAddress = getHTMLFeed("https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=" & InputIPAddress)




C#
Using System.Net;
Using System.Xml;
Using System.IO;


private string getHTMLFeed(string URL)
    {
        try
        {
        ServicePointManager.Expect100Continue = true;
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;

         HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
        WebResponse response = request.GetResponse();

         Stream rssStream = response.GetResponseStream();
        var rssDoc = new XmlDocument();

        var SNWC = new System.Net.WebClient();
        var reader = new StreamReader(rssStream);
        string responseFromServer = reader.ReadToEnd();

        string Contents = responseFromServer;
        Contents = Contents.Trim();

         return Contents.ToString();
        }
        catch (Exception ex)
        {
         return "False";
        }
    }
    bool ValidIPAddress = false;
    string InputIPAddress = "192.168.5.10";
    ValidIPAddress = getHTMLFeed("https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=" + InputIPAddress);




jQuery
$.ajax({ url: 'https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=[IPAddressToLookUp]', success: function(data) { alert(data); } });



JavaScript
function makeHttpObject() {
    try {return new XMLHttpRequest();}
    catch (error) {}
    try {return new ActiveXObject("Msxml2.XMLHTTP");}
    catch (error) {}
    try {return new ActiveXObject("Microsoft.XMLHTTP");}
    catch (error) {}

    throw new Error("Could not create HTTP request object.");
}

    var request = makeHttpObject();
    request.open("GET", "https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=[IPAddressToLookUp]", true);
    request.send(null);
    request.onreadystatechange = function() {
        if (request.readyState == 4)
        alert(request.responseText);
};




PHP using fopen wrappers
$html = file_get_contents('https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=[IPAddressToLookUp]');



PHP using cURL
$c = curl_init('https://www.crawlerly.com/isCrawler.aspx?Token=[YOURTOKEN]&IP=[IPAddressToLookUp]');
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
//curl_setopt(... other options you want...)

$html = curl_exec($c);

if (curl_error($c))
    die(curl_error($c));

// Get the status code
$status = curl_getinfo($c, CURLINFO_HTTP_CODE);

curl_close($c);