Hi VWR Community!

following is my current problem:
i scrape a website with many requests. after app. 90min. i'm blocked by the website.
i assume, that a firewall is blocking me. in less than one hour the blocking is abandoned and i can continue.

what i already tried:
1. i used random page delay, with a delay time from 6000 to 10000 ms. -> no success
2. i used a proxy -> it works (but i don't want to use a proxy in my infrastructure)
3. i switched between ie, webbrowser and webcrawler -> no success
4. doing partial load (delay between 2-3seconds) -> 30min vwr + 5min pause + 30min vwr + 5min pause and so on --> success

what i like to have:
1. i want vwr to stop after a specific time / or a specific worklad, wait a specific amount of time and continue working after that. my idea is to have a global counter or a global variable, which i can increase using a script.
on my first page is a search form, after every search i like to increase my counter. if this counter is 500, i can use the .net SLEEP function to pause the script. unfortunately i don't know, how i can define a global variable and increase its value within my vwr template.

2. if vwr can not reach the website (because it it blocked by firewall), i like vwr to wait a specific amount of time and try again. for the time being vwr continues with lots of errors.

3. maybe someone has an alternative?

i appreciate any help :-)

greetings,
Marc

Hi,

thanks for your advice.
i've got a further question:

when i got locked from webite, my vwr log show this:

11-08-25 15:30 Navigation error (Click). Timeout waiting for browser to navigate. Action:
11-08-25 15:30 Error submitting form: Suche
11-08-25 15:30 Processing start URL http://www.xxxyyy.de
11-08-25 15:35 Navigation error (Direct URL). Timeout waiting for page load. Url: http://www.xxxyyy.de
11-08-25 15:35 Error processing start URL url (Timeout loading URL)
11-08-25 15:35 Processing start URL http://www.xxxyyy.de
11-08-25 15:35 Form element was not found: Artikel
11-08-25 15:35 Unable to connect to the target website. The website may be unreachable or you are not connected to the Internet.
11-08-25 15:35 Unable to recover from error by navigating to: http://www.xxxyyy.de/

Now this error message continues for each input data row.
how can i stop processing, when this error occurs. and how can i give a specified return code back to the script, which calls vwr.

thanks,
marc

hi,

thanks for help.
i want to grep exactly this error (timeout while loading), and give a user specified return code back to calling script (a php background process), so that further error handling can be processed by this script.

is this possible?