Visual Web Ripper Logo Visual Web Ripper Logo
Welcome Guest Search | Active Topics | Log In | Register

Tag as favorite
Outputting into remote MySQL
ScrapinChick
#1 Posted : Saturday, February 05, 2011 11:18:56 PM
Groups: Registered
Joined: 2/5/2011
Posts: 4
Hello, I want to schedule VWR to scrape and output the data into a remote MYSQL database every hour but I'm not sure how.

I have the scrape ready and is scraping the correct data but in the "destination data source"

settings box, there's a selection for MySQL in the drop down.

But the only options it gives you are

server name, port, username, password, and database.

Where do I tell it which table to put the data into?
RipperCode
#2 Posted : Sunday, February 06, 2011 12:45:09 AM
Groups: Registered
Joined: 11/27/2010
Posts: 28
Location: Chennai
Hi ScrapinChick ,

in order to execute the data for the specific table you need to create a custom post processing script for this either using VB.NET / C#.net ,

Please let me us if you have any difficulties ,thank you.

Regards,

Ashok

Ashok@Rippercode.com

Regards,

Ashokkumar Chandramohan

"Authorized reseller and service provider for Visual Web Ripper"

URL : http://www.rippercode.com/
GTalk : ashok@rippercode.com
Skype : ashok_kumarc
Call: 0091 994 114 4844
FB: http://www.facebook.com/...ppercode/177447045625349
ScrapinChick
#3 Posted : Sunday, February 06, 2011 1:30:07 AM
Groups: Registered
Joined: 2/5/2011
Posts: 4
I do not have any experience in VB.NET or C#.net other than writing macros in microsoft word and excel.

I'm not sure where this script would even go? I have not even tried to connect VWR to my mysql db yet.
RipperCode
#4 Posted : Sunday, February 06, 2011 1:42:16 AM
Groups: Registered
Joined: 11/27/2010
Posts: 28
Location: Chennai
well i can help you on this , Please check your email , thank you.
Regards,

Ashokkumar Chandramohan

"Authorized reseller and service provider for Visual Web Ripper"

URL : http://www.rippercode.com/
GTalk : ashok@rippercode.com
Skype : ashok_kumarc
Call: 0091 994 114 4844
FB: http://www.facebook.com/...ppercode/177447045625349
Sequentum Support
#5 Posted : Sunday, February 06, 2011 6:17:32 PM

Groups: Administrators
Joined: 4/10/2010
Posts: 1,239
Location: Sydney, Australia
Please see this topic in the manual for more information about post-processing scripts.

http://manual.visualwebr...fault.aspx?manual_id=945
ScrapinChick
#6 Posted : Sunday, February 06, 2011 10:05:56 PM
Groups: Registered
Joined: 2/5/2011
Posts: 4
I've looked over the manual and it is quite confusing.

My agent is suppose to collect the url of an image, which it does.


I'm testing it with XLS as the output and in the spreadsheet I get 4 columns.


Google_key google image image_file


The only one I care about is the image column.

All I want to do is have VWR upload this to a mysql database automatically on a scheduled run. Could it be that difficult?
ScrapinChick
#7 Posted : Sunday, February 06, 2011 10:12:40 PM
Groups: Registered
Joined: 2/5/2011
Posts: 4
In the photo it shows this

with the associated code.

but then it tells you it has to do this,

The Postprocess Method

A post-processing script must have one method as shown below.
Code:

public   static   bool  Postprocess(WrPostprocessArguments args)         
{         
     try           
    {                       
          return   true ;       
    }         
     catch  (Exception exp)         
    {         
        args.WriteDebug(exp.Message);         
          return   false ;         
    }         
}     



Which is completely different from the photo.



But then it says this


The .NET test application should load the data output for your project and then call the postprocess method as shown below. When your test application works, you can copy the Postprocess method directly into the Visual Web Ripper script editor.
Code:

using  System;   
using  System.Collections.Generic;   
using  System.Linq;   
using  System.Text;   
using  VisualWebRipper;   
 
namespace  Postprocessing   
{   
     class  Program   
    {   
         static   void  Main(string[] args)   
        {   
            WrProject project = WrProject.LoadByName( "projectName" );   
            WrDataProject data = project.GetCollectedData();               
            WrPostprocessArguments processArgs =  new  WrPostprocessArguments(data.StartUrls, project);   
            Postprocess(processArgs);   
        }   
 
         public   static   bool  Postprocess(WrPostprocessArguments args)     
        {     
             try     
            {     
                 //First we set the SQL we'll use to insert data into the database table.     
                 //The Database connection has already been set by defining a shared script       
                 //database. Visual Web Ripper will automatically open and close the       
                 //database connection.     
                args.Database.SetSql( "insert into properties (type,fors,title,description,area) values (@type,@fors,@title,@description,@area)" );     
                args.Database.PrepareSql();     
                             
                 //The first table contains the start URL. We only have one start URL in     
                 //this project. ChildTableRows returns all rows in the first child table     
                foreach(WrDataRow finditRow in args.DataTable[0].ChildTableRows)     
                {     
                     //The next child table is the page navigation data table     
                    foreach(WrDataRow pageRow in finditRow.ChildTableRows)     
                    {     
                         //The last child table is where the property data is located     
                        foreach(WrDataRow propertyRow in pageRow.ChildTableRows)     
                        {     
                            args.Database.SetParameterTextValue( "@type" ,       
                                finditRow[ "type" ]);     
                            args.Database.SetParameterTextValue( "@fors" ,       
                                finditRow[ "fors" ]);     
                            args.Database.SetParameterTextValue( "@title" ,       
                                propertyRow[ "title" ]);     
                            args.Database.SetParameterTextValue( "@description" ,       
                                propertyRow[ "description" ]);     
                            args.Database.SetParameterTextValue( "@area" ,     
                                propertyRow[ "area" ]);     
                            args.Database.ExecuteNonQuery();     
                        }     
                    }     
                }                 
                 return   true ;     
            }     
             catch (Exception exp)     
            {     
                args.WriteDebug(exp.Message);     
                 return   false ;     
            }     
        }       
    }   




I'm pretty confused...
Sequentum Support
#8 Posted : Tuesday, February 22, 2011 11:14:36 PM

Groups: Administrators
Joined: 4/10/2010
Posts: 1,239
Location: Sydney, Australia
You'll need .NET programming skills in order to understand the topic in the manual and create a post-processor. However, it doesn't sound like you necessarily need a post-processor.

In order to get only the field you are interested in, set the option "Denormalize data" in the "Destination Data Source" screen.

You cannot insert data into an existing table without using a post-processor, but if you just want the software to name the table differently, the easiest way is to rename the project, so it has the same name as the table you want created.
Users browsing this topic
Guest
Tag as favorite
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

Powered by YAF 1.9.4 RC1 | YAF © 2003-2009, Yet Another Forum.NET
This page was generated in 0.123 seconds.