2 hours ago
Hi,
That's some pretty advanced use right there.
A few remarks.
To me it sounds very much like you are hitting some anti-scraping protection.
There's a few points that make me believe so.
- Hitting the exact same issue when using a completely different mechanism (MSXML2.ServerXMLHTTP.6.0)
- Have this happen on the 19th download every time
- Receiving a HTTP 200 status code (e.g. all OK) instead of a HTTP error.
What I would do in your situation is:
- use (Telerik) Fiddler for inspecting the data stream both requests and responses. It's a great tool for diagnosing this kind of issue and best of all, it's free.
- try and see if the same issue happens when you download via a browser manually. If it doesn't then try to increase time between the downloads to say 30 seconds.
- Log out of the page after for example 15 downloads then login again and continue to download the next 15 documents and so on.
Some answers to your questions..
- I think you are using AntView correctly
- The XML parsing error is expected if the page contains non XML data. You could probably test the content to see if it starts with "<" character after the white space.
- If you use javascript fetch then I don't think that the browser component itself can investigate any HTTP headers as the data is at the javascript layer (I might be wrong on this). Normally you can retrieve header data on document navigation events when the page is navigated. eg. OnNavigationStarting has the request object. But that's in the browser document.
There's also NavigateWithWebResourceRequest which can give you access to header data and both request and response objects via the OnWebResourceRequested event.
But if the issue is bot/scrape detection it won't give you anything extra over your current methods.
First try and see if a Fiddler snapshot can get you some more answers.
That's some pretty advanced use right there.
A few remarks.
To me it sounds very much like you are hitting some anti-scraping protection.
There's a few points that make me believe so.
- Hitting the exact same issue when using a completely different mechanism (MSXML2.ServerXMLHTTP.6.0)
- Have this happen on the 19th download every time
- Receiving a HTTP 200 status code (e.g. all OK) instead of a HTTP error.
What I would do in your situation is:
- use (Telerik) Fiddler for inspecting the data stream both requests and responses. It's a great tool for diagnosing this kind of issue and best of all, it's free.
- try and see if the same issue happens when you download via a browser manually. If it doesn't then try to increase time between the downloads to say 30 seconds.
- Log out of the page after for example 15 downloads then login again and continue to download the next 15 documents and so on.
Some answers to your questions..
- I think you are using AntView correctly
- The XML parsing error is expected if the page contains non XML data. You could probably test the content to see if it starts with "<" character after the white space.
- If you use javascript fetch then I don't think that the browser component itself can investigate any HTTP headers as the data is at the javascript layer (I might be wrong on this). Normally you can retrieve header data on document navigation events when the page is navigated. eg. OnNavigationStarting has the request object. But that's in the browser document.
There's also NavigateWithWebResourceRequest which can give you access to header data and both request and response objects via the OnWebResourceRequested event.
But if the issue is bot/scrape detection it won't give you anything extra over your current methods.
First try and see if a Fiddler snapshot can get you some more answers.