Hi

Before fully completing project and exporting
data to the specified format does VWR store
intermediate results somewhere?

So in case of failure ( VWR crash, OS crash or whatever )
I could get data already extracted but not exported?


I had an OS crash when VWR was working that is why
I am asking. And for some reason there is not dat files
in output folder of the project.

How can I be sure that results are writing to the
database during the extraction?

I installed MySQL server, chose VWR to write to
Mysql database.
When I press stop during extraction I see extracted data in
database, but when I just ended VWR via process manager
to simulate crash or whatever where is no data in the tables.


And I start playing with MySQL output after project after 15
hours of extraction gave me
Exception of type 'System.OutOfMemoryException' was thrown.
and all data was lost.

Hello!

I did as you write. However, I got "Exception of type 'System.OutOfMemoryException' was thrown." as an error message - while processing the very last URL. I write all data to a MySQL-DB, but it seems to miss so many data. These are my settings:

[Data cache]
Max memory usage (MB): 8000 {12GB are installed}
Data row cache: 0
Export data & clear memory: False
Restart web browser: True

[Data cache timer]
Cache timer interval (minutes): 0
Export data & clear memory at time interval: True
Restart web browser at time interval: True

I hope so much that you can help me!

Thank you!

You can try
"Data row cache" 1
"Export data & clear memory" true

Data Row Cache defines number of rows stored in memory before flushing to disk. The default value 0 means to cache all rows in memory.

Export Data & Clear Memory defines how Visual Web Ripper handle Row Cache full. So you will need to let Visual Web Ripper export data to destination data source.