Using an Input Data Source to Provide Start URLs

After you have added an input data source, you can configure the data extraction project to use it to provide multiple start URLs. Follow these steps:

  1. Open the Project Options window.
  2. Select the Start URLs options tab.
  3. Set the Feed URLs from Input Data Source option.
  4. Select the column in the input data source that contains the start URLs.

 

Using Link Transformation on the Start URL

Link transformation can be used to generate start URLs from values in the input data source.

Example

The following example shows a link transformation script that uses two columns in the input data source to generate start URLs.

  1. using  System;   
  2. using  VisualWebRipper.Internal.SimpleHtmlParser;   
  3. using  VisualWebRipper;   
  4. public   class  Script   
  5. {      
  6.      public   static  string TransformLink(WrLinkTransformationArguments args)   
  7.     {   
  8.          try   
  9.         {   
  10.              return   "http://www.coldwellbanker.com/agent?action=list&freeTextAddress="     
  11.                 + args.InputDataRow[ "State" ] +  "&CountryID="  + args.InputDataRow[ "CountryID" ];   
  12.         }   
  13.          catch (Exception exp)   
  14.         {   
  15.             args.WriteDebug(exp.Message);   
  16.              return   "Custom script error" ;   
  17.         }   
  18.     }   
  19. }