how to scrub websites?

[Homerboy]

We often hire 3rd party contractors to build us simple applications that allow us to do some data-mining en masse off of various public websites.

In most scenarios, we feed the program (or the 3rd party directly) a csv of the data fields we want entered into the forms on the website(s), and it basically automates the task of entering the information into the forms and capturing the results (either in a screen shot, or a return csv of the data etc etc)

I was curious as to how they go about doing this. Are they just parsing the HTML of the document itself and inserting where applicable or ???

** please note that these sites are set up for these sort of uses and have no problems with such practices.

 

[Ken g6]

It sounds like they're writing apps that use a web service. Basically, if you POST to a site with the right text, you'll get the right text back, whether as HTML or some other file.

If they're using an actual web browser and screen shots, that's some kind of automation system like Selenium.

 

[Hmongkeysauce]

I wrote a vb script for excel a few weeks back that did something similar. The script takes data from excel and pastes the data into a field on an online form and submits the form. That's far as I had to go, but I'd imaging grabbing the data would be a matter of waiting for the page to POST and grabbing the values from the correct elements. If the form uses GET, then it's probably even easier. A snippet of the script that I wrote is below:

    With CreateObject("InternetExplorer.Application")
        .Visible = True
        .Navigate "http://www.yourform.com"
        Do Until .ReadyState = 4
        Loop
        With .document
            .getElementById("sourceData").Value = sText
            .getElementById("submitbutton").Click
        End With
    End With

sText is the data from Excel

 

[DaveSimmons]

For sites without an API or web service, "screen scraping" is when you make HTML POST or GET requests to a web page, receive the HTML response back (with results like doing "view source" in your browser) then use string functions to get the data out of the response.

This approach is brittle since if the page from the server changes, your code to find the data may break.

The code for each page is also usually going to be different, both to send the request and to process the result.

 

[beginner99]

For sites without an API or web service, "screen scraping" is when you make HTML POST or GET requests to a web page, receive the HTML response back (with results like doing "view source" in your browser) then use string functions to get the data out of the response.

This approach is brittle since if the page from the server changes, your code to find the data may break.

The code for each page is also usually going to be different, both to send the request and to process the result.

or maybe the site / web service offers the option to get the data in a more usable format like JSON.

 

[DaveSimmons]

or maybe the site / web service offers the option to get the data in a more usable format like JSON.

Good point. I did say "For sites without an API or web service ....", but it's true that sometimes you can use scraping to make your own (brittle, undocumented) web service by figuring out what JSON calls a page sent from the server makes to fill in parts of itself.

You still need to impersonate a browser instead of using a stable and supported SOAP, REST etc. service with documented behavior, but you may be able to get a response that's easier to parse than a page full of HTML.

 

[Leros]

It sounds like they're writing apps that use a web service. Basically, if you POST to a site with the right text, you'll get the right text back, whether as HTML or some other file.

If they're using an actual web browser and screen shots, that's some kind of automation system like Selenium.

Be aware that a lot of websites won't accept a form submission without a valid CSRF token, which makes POSTing to the site a bit more difficult since you need to actually load the page with the form to get the CSRF token. Still completely doable though.

 

[beginner99]

Good point. I did say "For sites without an API or web service ....", but it's true that sometimes you can use scraping to make your own (brittle, undocumented) web service by figuring out what JSON calls a page sent from the server makes to fill in parts of itself.

You still need to impersonate a browser instead of using a stable and supported SOAP, REST etc. service with documented behavior, but you may be able to get a response that's easier to parse than a page full of HTML.

cURL or wget can be used for that. I know what you mean by undocumented. Recently had to do exactly that. Data displayed on page was loaded by AJAX as JSON and was pretty straight forward to make a cURL GET request that fetched all the data.

This is sure one thing anyone should also consider when creating web pages. Are users allowed to access data that way? IMHO this can easily be forgotten...

 

[Hmongkeysauce]

On a semi-related note, anybody know how to screen scrap desktop apps?

 

[JustAnAverageGuy]

Print Screen?

 

[DaveSimmons]

Scraping Windows apps is done by the bad guys for harvesting passwords and account security pics so it's not something I'd offer tips on. What's your intended use, or just curious?

 

[Hmongkeysauce]

I'm curious because a lot of ECM apps does screen scrape on desktop apps for integration purposes.