.net httpwebrequest question-----(Descartes Maybe?)

[calpha]

've got a page scraper running----however, i run the pages to scrape from a database, and the scraping works for the first two lookups (one to login and get the auth cookie, and the second to scrape the particular page).

However, from the third try on, my httpwebrequest times out. I assume there's something not being closed, but I am really not sure.

here's my code

VB:
Function GetRequestStream(ByVal sURL As String) As String
Dim objRequest As HttpWebRequest = WebRequest.Create(sURL)
Dim objResponse As HttpWebResponse
Dim myWriter As StreamWriter
Dim sr As StreamReader
Dim objCookie As CookieContainer
Dim strRead As String
With objRequest
.Method = "PROPFIND"
.CookieContainer = Me.ckCont
Try
myWriter = New StreamWriter(.GetRequestStream())
Catch ex As Exception
RaiseEvent ErrorMessage(ex)
Return Nothing
End Try
End With

Try
objResponse = objRequest.GetResponse()
sr = New StreamReader(objResponse.GetResponseStream())
strRead = sr.ReadToEnd()
sr.Close()
objResponse.Close()
Return strRead
Catch ex As Exception
RaiseEvent ErrorMessage(ex)
Return Nothing
End Try
End Function

obviously I'm just returning a string----but this function is called on every iteration through my loop.

 

[Descartes]

I'm a bit puzzled about what you're doing with the PROPFIND. I don't see where you actually write anything to the response stream, and you're not closing myWriter.

There's nothing wrong with the code aside from the lack of functionality from what I can tell. Let me know what you're doing with the response stream and I'll try to help further. It's been a while since I've seen WebDav get some use though.

I did run your code through some tests of my own and it didn't fail on 20 consecutive requests.

 

[calpha]

Hmmmm.
Well, part one of the answer:
In order to set the cookiecontainer, you cannot use the method = "post". Runtime Error.
I'm a part of a Fantasy BB League (2 of them) and I'm trying to get a scraper going because I'm sick and tired of going to the site every day and looking at everyone's scores to see how I'm doing relative to the league. In order to get there, I have to login first (which works) and then do "gets" using the cookie from the login. Using the method "PropFind" was the only way I could get the httpwebrequest to let me set the cookie cont.

By The Way---
I did get a fix in (thanks to experts exchange).
Explicitly getting the stream object from the httpwebrequest object, and then explicitly closing both streams and streamwriter---
I admit, I do not know why that fixed it----and am still trying to find that part out. Having done socket programming before via the API I know that if you leave sockets open even outside of their scope you can get into trouble...... but this fixed the error I was getting. It's curious that yours didn't fail though----

My test was coming from a webform btw.


This is the code from the function that calls the function I was having a problem in:

Public Function GetWeeksLinks() As Boolean
Dim ds As DataSet
Dim dr As DataRow
Dim sText As String
Dim sReg As New Regex("<[^>]{1,}page=matchup[^/]{1,}")
Dim m As Match
Dim sSQL As String
Dim sLink As String
Dim team1 As Integer
Dim team2 As Integer
Dim myMatches As Match
Dim i As Integer
Dim j As Integer
Dim icount As Integer = 0

GetURLDetails(enmMode.MODE_DB)
'first make the request using the post method to get the auth cookie
sText = GetRequestStream(cSURL.strYahooURL, cSUSER.strFormData)
'get the links from all the weeks with an ending date < curr date and w/ no scores already in db
ds = cDB.ReturnDS("spPastWeeks")
'now for each link returned
for each dr in ds.Tables(0).rows
'dr = ds.Tables(0).Rows(myInt)
Debug.Write("My Link = " & dr("WeekLink"))
'Get the Stream from this link using the auth cookie & propfind
sText = GetRequestStream(dr("WeekLink"))

For Each m In sReg.Matches(sText)
Dim sreg2 As New Regex("/{1,}[^""]{1,}")
For Each myMatches In sreg2.Matches(m.ToString)
get the matchup links for this week
sLink = cSURL.strFantasyBase & myMatches.ToString
Next
Dim sr() As String
sreg2 = New Regex("mid1=\d{1,}")
For Each myMatches In sreg2.Matches(m.ToString)
'get the team ids
sr = Split(myMatches.ToString, "=")
team1 = CInt(sr(1))
Next
sreg2 = New Regex("mid2=\d{1,}")
For Each myMatches In sreg2.Matches(m.ToString)
'get the team ids
sr = Split(myMatches.ToString, "=")
team2 = CInt(sr(1))
Next
'return the found buggers to the db
sSQL = "insert into WeeksLinks (WeeksID, WeekMatchupLink, Team1ID, Team2ID) values " & _
" ( " & dr("ID") & ",'" & sLink & "'," & team1 & "," & team2 & ")"
cDB.ReturnCount(sSQL)

Next
Next
End Function


'This is my changed function

Function GetRequestStream(ByVal sURL As String) As String
Dim objRequest As HttpWebRequest = WebRequest.Create(sURL)
Dim objResponse As HttpWebResponse
Dim myWriter As StreamWriter
Dim sr As StreamReader
Dim objCookie As CookieContainer
Dim strRead As String
With objRequest
.Method = "PROPFIND"
.CookieContainer = Me.ckCont
Dim RequestStream As Stream
Try
'explicilty get the stream from the httpwebrequest
RequestStream = .GetRequestStream()
'now write the stream
myWriter = New StreamWriter(requeststream)
Catch ex As Exception
RaiseEvent ErrorMessage(ex)
Return Nothing
Finally
close both the stream and the streamwriter
myWriter.Close()
RequestStream.Close()

End Try
End With

Try
objResponse = objRequest.GetResponse()
sr = New StreamReader(objResponse.GetResponseStream())
strRead = sr.ReadToEnd()
sr.Close()
objResponse.Close()
Return strRead


Catch ex As Exception
RaiseEvent ErrorMessage(ex)
Return Nothing
End Try

End Function