Easy HTTP GET from scratch?

[etherealfocus]

Alright, continuing my barrage of newbish questions...

I'm trying to pull a heap of data from a distributor's web service. I can access through the browser, but with 30k items and 55 attributes each, the resulting XML file makes both Chrome and IE choke.

Here's the sample GET code:

GET /path/Inventory.asmx/Resource?CustomerNumber=string&UserName=string&Password=string&LastUpdate=string&LastItem=string&Source=string HTTP/1.1
Host: sub.domain.com
HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Length: length

<?xml version="1.0" encoding="utf-8"?>
<string xmlns="http://sub.domain.com/path/Inventory.asmx">string</string>

--------

I understand that I can run this somehow through PowerShell as per http://stackoverflow.com/questions/13935218/how-to-use-http-get-in-powershell but had some trouble adapting it to fit my needs, and I'd imagine there's an easier way to do it.

All I really need is for it to dump that huge XML file on my desktop instead of into my browser. I had a couple years of comp sci back in the day but have no web programming skills beyond HTML/CSS.

Batch script or Python maybe? I'm a complete newbie at both (my comp sci was in C and Java) but need to learn Python anyway...

Thanks guys!

 

[Ken g6]

There's plenty of programs that do that. Curl and WGet to name two. Unless you're not allowed to install external software.

 

[MrScott81]

wget is really easy to use and has a ton of options.

 

[etherealfocus]

I'm playing with it now. There's a cool lifehacker post on it at http://lifehacker.com/161202/geek-to-live--mastering-wget that might be of interest to other curious readers.

So can I pass all the above sample code through it in a single statement? And how do I tell it where to dump the resulting XML? Sorry, I'm sure this is all covered somewhere in the manual at http://www.gnu.org/software/wget/manual/wget.html but I've checked the HTTP Options section and found nothing.

The web service supports SOAP 1.1 and 1.2, HTTP GET, and HTTP POST. Obviously I don't need POST for this, but maybe SOAP is more appropriate for my purposes...? I'm in a little over my head here. lol

 

[etherealfocus]

So far no luck.

U:\>wget /path/resource.asmx/reportname&customernumber=string&username=string...&source= HTTP/1.1 subdomain.domain.com

returns a bunch of errors. The string through customernumber is an unsupported scheme, and the attributes I'm passing are unrecognized as internal or external commands.

And yes, I replaced the 'string' for each attribute with valid data. I can access the web service just fine; the output is just too big for a web browser.

I put wget in the Windows folder so it runs fine; I can access the help command and such.

 

[Syran]

So far no luck.

U:\>wget /path/resource.asmx/reportname&customernumber=string&username=string...&source= HTTP/1.1 subdomain.domain.com

returns a bunch of errors. The string through customernumber is an unsupported scheme, and the attributes I'm passing are unrecognized as internal or external commands.

And yes, I replaced the 'string' for each attribute with valid data. I can access the web service just fine; the output is just too big for a web browser.

I put wget in the Windows folder so it runs fine; I can access the help command and such.

Just looking at your syntax, unless you are running this on the machine it's installed on, try wget with a properly formatted web address:
wget http://subdomain.domain.com/path/resource.asmx/reportname...etc

You may also want to encapsulate it in quotes if there are any spaces in the link somewhere.

 

[Crusty]

Just looking at your syntax, unless you are running this on the machine it's installed on, try wget with a properly formatted web address:
wget http://subdomain.domain.com/path/resource.asmx/reportname...etc

You may also want to encapsulate it in quotes if there are any spaces in the link somewhere.

spaces, semicolons, question marks, asterisks... all sorts of reasons to quote the whole string

 

[etherealfocus]

Alright here's the string I tried:

wget "sub.domain.com/smart.resource.asmx/name?att1=string&att2=string&att3=string"

It resolves and connects, then says:

HTTP request sent, awaiting response... 500 Internal Server Error
2013-05-08 ERROR 500: Internal Server Error.

The wget syntax looks a bit different than the distributor's example GET request. Could that be the problem?

Also, how would I specify a destination for the recieved information assuming the request was successful?

 

[DaveSimmons]

http://linux.die.net/man/1/wget

wget -O filename request-URL

500 Internal Server Error - this means the ASPX server either didn't like the request, or more likely had a crash or exception.

You might still be getting the request URL wrong, such as not url-encoding / escaping special characters like space, = , ? .

 

[Crusty]

As far as I know you have to specify either http or https with wget.

 

[etherealfocus]

@Crusty I get the same error using

wget "sub.domain.com/smart.resource.asmx/name?att1=string&att2=string&att3=string" HTTP/1.1

and

wget "http://sub.domain.com/smart.resource.asmx/name?att1=string&att2=string&att3=string" HTTP/1.1

@DaveSimmons thanks for the link! Digging through it now. Question: as I understand it, the ampersands between attributes are part of the http code and shouldn't be swapped, but values including ampersands and such should. Is that correct, or should I swap everything? ie:

wget "url.com/resource.asmx/resource%3Fatt1=string%26att2=string%26att3=string..."

 

[Syran]

wget "http://sub.domain.com/smart.resource.asmx/name?att1=string&att2=string&att3=string" HTTP/1.1

Dumb question, is the HTTP/1.1 supposed to be part of the link somehow? If so, it should be within the quotes, if it isn't, try dropping it from the wget. You may also, if possible, try some sort of smaller query, just to see if you get any data at all, if that's possible.

@DaveSimmons thanks for the link! Digging through it now. Question: as I understand it, the ampersands between attributes are part of the http code and shouldn't be swapped, but values including ampersands and such should. Is that correct, or should I swap everything? ie:

wget "url.com/resource.asmx/resource%3Fatt1=string%26att2=string%26att3=string..."

Basically, when you put anything after a ? in a link, it's put into the GET command structure. Each comparison on it's own can be in any order, so long as the values line up correctly.

ie:

http://www.test.com/index.php?base=1&subject=test&name=whomever
http://www.test.com/index.php?subject=test&base=1&name=whomever
http://www.test.com/index.php?name=whomever&base=1&subject=test

Those would all be processed the same way by the server.

 

[DaveSimmons]

@DaveSimmons thanks for the link! Digging through it now. Question: as I understand it, the ampersands between attributes are part of the http code and shouldn't be swapped, but values including ampersands and such should. Is that correct, or should I swap everything? ie:

page ? variable1 = value1 & variable2 = value2 ...

It's only the value1, value2, ... where you need to replace (url-encode) any " " / "&"

Dumb question, is the HTTP/1.1 supposed to be part of the link somehow?

No, and I didn't see it listed in the parameters for wget, but I didn't read them all.

 

[Net]

any chance you cost post the link?

try this:

curl -H "Accept: text/xml" -H "Content-Type: text/xml" -X GET "http://www.w3schools.com/dom/books.xml"

 

[etherealfocus]

Sadly I can't post the link - our supplier list is confidential. I did get it figured out though - pasted my wget statement into a batch file and ran it from CMD. Here's the working string:

wget "url.com/path/resource.asmx/query?att1=value1&att2=value2" HTTP/1.1

Downloaded the entire 110MB xml file.

The HTTP/1.1 thing is specified in their example. Not sure why it's formatted that way, seems weird to me too. The supplier in question is not a shining paragon of technical wizardry, to put it mildly, so they might just be using some obsolete or third party system that does things oddly.

Anyway, it works. Thanks guys!