$more input
3308191

The following program reads in the number from the file named 'input'
and builds a url form this number. Then it builds a url from this
number. I have lynx then dump the data into a file called 'out' and
then just grep the entire thing for the Product Number, Product ID,
SKU, UPC, and weight.

m-net% more parse.pl
#!/usr/bin/perl -w

my (@shit, $read, $build, @product, @id, @sku, @upc, @weight);
my $temp;

open(IN, '<', 'input') || die "cant open: $!";
$read = <IN>;
chomp($read);
$build = "http://www.doba.com/members/catalog/".$read.".html";
$temp = `lynx -accept_all_cookies -dump $build`;
open(OUTFILE, '>out');
print OUTFILE $temp;
close OUTFILE;

open(OUT, '<', 'out') || die "cant open: $!";
@shit = <OUT>;

@product = grep(/Product ID/, @shit);
@id = grep(/Item ID/, @shit);
@sku = grep(/SKU/, @shit);
@upc = grep(/UPC/, @shit); #this part doesn't grep UPC correctly. I
get some extra data after UPC.
@weight = grep(/Weight/, @shit);

print @product;
print @id;
print @sku;
print @upc;
print @weight;

  c> open(IN, '<', 'input') || die "cant open: $!";
  c> $read = <IN>;
  c> chomp($read);
  c> $build = "http://www.doba.com/members/catalog/".$read.".html";
  c> $temp = `lynx -accept_all_cookies -dump $build`;

why are you calling out to a program when perl can load web pages just
fine with LWP? did you even look for web stuff on cpan?

  c> open(OUTFILE, '>out');
  c> print OUTFILE $temp;
  c> close OUTFILE;

  c> open(OUT, '<', 'out') || die "cant open: $!";
  c> @shit = <OUT>;

why are you writing out the output of lynx JUST TO READ IT BACK IN
AGAIN? this is the most absurd part of this program.

you have the text in $temp. you know how to use backticks but why do you
do the file write and reading back in? if you assigned the backticks to
an array you would get the same thing as in @shit without the wasted
effort.

also calling it @shit is not a good thing.

  c> @product = grep(/Product ID/, @shit);
  c> @id = grep(/Item ID/, @shit);
  c> @sku = grep(/SKU/, @shit);
  c> @upc = grep(/UPC/, @shit); #this part doesn't grep UPC correctly. I
  c> get some extra data after UPC.

that is a problem with the format of the html page. html isn't line
oriented and you are grepping over lines. the proper way to deal with
html is with a parser. or in special very well defined cases with
regexes to actually grab what you want from the text. whole html lines
are almost never what you want.

uri

my (@shit, $read, $build, @product, @id, @sku, @upc, @weight);
my @temp;

open(IN, '<', 'input') || die "cant open: $!";
$read = <IN>;
chomp($read);
$build = "http://www.doba.com/members/catalog/".$read.".html";
@temp = `lynx -accept_all_cookies -dump $build`;

@product = grep(/Product ID/, @temp);
@id = grep(/Item ID/, @temp);
@sku = grep(/SKU/, @temp);
@upc = grep(/UPC/, @temp);
@weight = grep(/Weight/, @temp);

print @product;
print @id;
print @sku;
print @upc;
print @weight;

  c> I know I shouldn't critize free help, but you seem to have some anger
  c> management issues.

nope. i have bad code anger issues. i deal with this in code reviews all
the time. i just don't get how people come up with wacky and slow ways
to do things. i have seen worse code that read in files, parsed them,
wrote them out (untouched) and read them in again.

  >>
  c> open(IN, '<', 'input') || die "cant open: $!";
  c> $read = <IN>;
  c> chomp($read);
  c> $build = "http://www.doba.com/members/catalog/".$read.".html";
  c> $temp = `lynx -accept_all_cookies -dump $build`;
  >>
  >> why are you calling out to a program when perl can load web pages just
  >> fine with LWP? did you even look for web stuff on cpan?
  >>
  c> Would using LWP speed up the code? By the way, this code is meant to
  c> run on a server with restricted access. Ie, I can't install stuff from
  c> cpan on that server.

if you have access to load scripts you can load pure perl modules
too. this is an FAQ.

  c> open(OUTFILE, '>out');
  c> print OUTFILE $temp;
  c> close OUTFILE;
  >>
  c> open(OUT, '<', 'out') || die "cant open: $!";
  c> @shit = <OUT>;
  >>
  >> why are you writing out the output of lynx JUST TO READ IT BACK IN
  >> AGAIN? this is the most absurd part of this program.
  >>
  >> you have the text in $temp. you know how to use backticks but why do you
  >> do the file write and reading back in? if you assigned the backticks to
  >> an array you would get the same thing as in @shit without the wasted
  >> effort.
  >>
  >> also calling it @shit is not a good thing.
  >>
  c> Huh? Are you saying I don't need the 'out' file?

yes. why do you think you need that file? you call backticks and get the
html page in $temp. why do you think you need a file to process that
data? you already have it inside perl.

uri

yes.

  c> Maybe something like this?
  c> % more parse.pl
  c> #!/usr/bin/perl -w

  c> my (@shit, $read, $build, @product, @id, @sku, @upc, @weight);
  c> my @temp;

  c> open(IN, '<', 'input') || die "cant open: $!";
  c> $read = <IN>;
  c> chomp($read);
  c> $build = "http://www.doba.com/members/catalog/".$read.".html";
  c> @temp = `lynx -accept_all_cookies -dump $build`;

  c> @product = grep(/Product ID/, @temp);
  c> @id = grep(/Item ID/, @temp);
  c> @sku = grep(/SKU/, @temp);
  c> @upc = grep(/UPC/, @temp);
  c> @weight = grep(/Weight/, @temp);

  c> print @product;
  c> print @id;
  c> print @sku;
  c> print @upc;
  c> print @weight;

  c> However, I don't know how to use LWP. Again, would the code run faster
  c> if I used LWP?

better but forking off lynx is still slow. LWP should be much faster. if
you want speed (and with the data size you have, you want it), use LWP.

depending on how fast you need it (cpu usage will spike with the greps
you have) you can also change all that to parse out what you want with
regexes. (again, that assumes a known fixed html page layout which you
seem to have).

uri

As a simple answer, take a look at LWP:UserAgent
(http://search.cpan.org/~gaas/libwww-perl-5.812/lib/LWP/UserAgent.pm),
as a good start in the right direction.

In fact, I wrote a whole script that took only 0.8 seconds to download
and parse a single page (of course, with more id's in a file, the only
real limit on the speed is the network latency and transfer speed) but I
have decided not to post it as I do not know what his intentions are.

As for you, pick a posting id and stick with it.

PLONKETY PLONK!

Sinan

--
A. Sinan Unur <1...@llenroc.ude.invalid>
(remove .invalid and reverse each component for email address)

#!/usr/bin/perl -w

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Cookies;

my ($read, $build, @product, @id, @sku, @upc, @weight);
my @temp;

open(IN, '<', 'input') || die "cant open: $!";
$read = <IN>;
chomp($read);
$build = 'http://www.doba.com/members/catalog/'.$read.'.html';
#@temp = `lynx -accept_all_cookies -dump $build`;

my $ua = LWP::UserAgent->new;
$ua->agent("OMEGA SPARC DESTROYER/69");

my $request = HTTP::Request->new('GET');
$request->url($build);

my $cookie_jar = HTTP::Cookies->new;
$cookie_jar->add_cookie_header($request);

my $response = $ua->request($request);

my $code = $response->code;
print $code;

@temp = $request->content;

@product = grep(/Product ID/, @temp);
@id = grep(/Item ID/, @temp);
@sku = grep(/SKU/, @temp);
@upc = grep(/UPC/, @temp);
@weight = grep(/Weight/, @temp);

print @product;
print @id;
print @sku;
print @upc;
print @weight;

So, here is a fish for you:

C:\Temp> cat p.pl
#!/usr/bin/perl

use strict;
use warnings;

use HTML::TokeParser;
use LWP::Simple;

my ($input_file) = @ARGV;
die "No input file specified\n" unless defined $input_file;

open my $INPUT, '<', $input_file
    or die "Cannot open '$input_file': $!";

ID:
while ( my $id = <$INPUT> ) {
    chomp $id;

    my $url = make_url( $id );
    my $html = get $url;

    unless ( defined $html ) {
        warn "Error downloading from '$url'\n";
        next ID;
    }

    my $parser = HTML::TokeParser->new( \$html );

    TABLE:
    while ( my $token = $parser->get_tag('table') ) {
        if ( lc $token->[1]{id} eq 'product_details' ) {
            my $td = $parser->get_tag('td');
            last TABLE unless $td;
            my $cell = $parser->get_text('/td');
            my %data;
            while ( $cell =~ /\s*([^:]+?):\s+(\d+)\s+/g ) {
                $data{$1} = $2;
            }
            use Data::Dumper;
            print Dumper \%data;
        }
    }

C:\Temp> timethis p list

$VAR1 = {
          'Product ID' => '3308191',
          'UPC' => '896207999816',
          'Item ID' => '3653992',
          'SKU' => '8930'
        };

TimeThis :  Command Line :  p list
TimeThis :    Start Time :  Thu May 15 18:19:28 2008
TimeThis :      End Time :  Thu May 15 18:19:29 2008
TimeThis :  Elapsed Time :  00:00:01.062

Comparing this to the overhead of an empty script:

C:\Temp> cat t.pl
#!/usr/bin/perl

use strict;
use warnings;

C:\Temp> timethis t

TimeThis :  Command Line :  t
TimeThis :    Start Time :  Thu May 15 18:20:3