Import Upstream version 1.8.5
[hcoop/debian/openafs.git] / src / tests / OpenAFS / wrapper.pm
1 # CMUCS AFStools
2 # Copyright (c) 1996, 2001 Carnegie Mellon University
3 # All rights reserved.
4 #
5 # See CMU_copyright.ph for use and distribution information
6
7 package OpenAFS::wrapper;
8
9 =head1 NAME
10
11 OpenAFS::wrapper - AFS command wrapper
12
13 =head1 SYNOPSIS
14
15 use OpenAFS::wrapper;
16 %result = &wrapper($cmd, \@args, \@pspec, \%options);
17
18 =head1 DESCRIPTION
19
20 This module provides a generic wrapper for calling an external program and
21 parsing its output. It is primarily intended for use by AFStools for calling
22 AFS commands, but is general enough to be used for running just about any
23 utility program. The wrapper is implemented by a single function,
24 B<OpenAFS::wrapper::wrapper>, which takes several arguments:
25
26 =over 4
27
28 =item $cmd
29
30 The command to run. This can be a full path, or it can be a simple command
31 name, in which case B<wrapper()> will find the binary on its internal path.
32
33 =item \@args
34
35 A reference to the list of arguments to be passed to the command. Each
36 element of the list is passed as a single argument, as in B<exec()>.
37
38 =item \@pspec
39
40 A reference to the list describing how to parse the command's output.
41 See below for details.
42
43 =item \%options
44
45 A reference to a table of command execution and parsing options.
46
47 =back
48
49 On success, B<wrapper()> returns an associative array of data gathered
50 from the command's output. The exact contents of this array are
51 caller-defined, and depend on the parsing instructions given. On failure,
52 an exception will be thrown (using B<die>), describing the reason for the
53 failure.
54
55 The I<%options> table may be used to pass any or all of the following
56 options into B<wrapper()>, describing how the command should be executed
57 and its output parsed:
58
59 =over 4
60
61 =item pass_stderr
62
63 If specified and nonzero, the command's stderr will be passed directly
64 to the calling program's, instead of being parsed. This is useful when
65 we want to process the command's output, but let the user see any
66 diagnostic output or error messages.
67
68 =item pass_stdout
69
70 If specified and nonzero, the command's stdout will be passed directly
71 to the calling program's, instead of being parsed. This is useful when
72 the command being run produces diagnostic or error messages on stderr
73 that we want to parse, but provides bulk data on stdout that we don't
74 want to touch (e.g. B<vos dump> when the output file is stdout).
75
76 =item path
77
78 If specified, the path to be used for the program to execute, instead of
79 deriving it from the command name. This is useful when we want the
80 command's argv[0] (which is always I<$cmd>) to be different from the
81 path to the program.
82
83 =item errors_last
84
85 If specified and nonzero, the built-in instructions for catching errors
86 from the command will be added to the end of the instructions in @pspec
87 instead of to the beginning.
88
89 =back
90
91 =head1 PARSING COMMAND OUTPUT
92
93 The I<@pspec> list describes how to parse command output. Each element
94 of the list acts like an "instruction" describing how to parse the command's
95 output. As each line of output is received from the program, the parsing
96 instructions are run over that line in order. This process continues for
97 every line of output until the program terminates, or the process is
98 aborted early by flow-control operators.
99
100 Each parsing instruction is a reference to a list, which consists of a
101 regular expression and a list of "actions". As a line of output is
102 processed, it is compared to each instruction's regexp in turn. Whenever
103 a match is found, the actions associated with that instruction are taken,
104 in order. Each instruction's regexp may contain one or more parenthesized
105 subexpressions; generally, each "action" uses up one subexpression, but there
106 are some exceptions. Due to the current design of B<wrapper()>, each regexp
107 must have at least one subexpression, even if it is not used.
108
109 The acceptable actions are listed below, each followed by a number in brackets
110 indicating how many subexpressions are "used" by this action. It is an error
111 if there are not enough subexpressions left to satisfy an action. In the
112 following descriptions, I<$action> is the action itself (typically a string or
113 reference), I<$value> is the value of the subexpression that will be used, and
114 I<%result> is the result table that will be returned by B<wrapper> when the
115 command completes.
116
117 =over 4
118
119 =item string [1]
120
121 Sets $result{$action} to $value. Note that several specific strings have
122 special meaning, and more may be added in the future. To ensure compatibility
123 with future versions of B<wrapper>, use only valid Perl identifiers as
124 "string" actions.
125
126 =item scalar ref [1]
127
128 Sets $$action to $value.
129
130 =item list ref [*]
131
132 Pushes the remaining subexpression values onto @$action. This action uses
133 all remaining subexpression values.
134
135 =item hash ref [2]
136
137 Sets $$action{$value0} to $value1.
138
139 =item code ref [*]
140
141 Calls the referenced function, with all remaining subexpression values as
142 its arguments. Any values returned by the function will be used to refill
143 the (now empty) subexpression value list, and thus may be used as arguments
144 by subsequent actions. If only a few values are required, use a function
145 like this:
146
147 sub usetwo { # uses two values and preserves the rest
148 my($val1, $val2, @rest) = @_;
149
150 print STDOUT "Got $val1, $val2\n";
151 @rest;
152 }
153
154 =item '.' [0]
155
156 End processing for this line of output, ignoring any remaining instructions.
157 Remaining actions in this instruction will be processed.
158
159 =item '+n' [0]
160
161 Skip the next I<n> instructions. This, along with the '.' action, can be
162 used to build simple flow-control constructs based on the contents of
163 lines of output.
164
165 =item '-x' [0..1]
166
167 Signal an error after this instruction. Remaining actions in this instruction
168 will be processed, but no further instructions will be processed for this
169 line, and no further lines of output will be processed. If I<x> is given,
170 it will be used as a regexp to match against the B<previous> line of output,
171 and the first parenthesized subexpression resulting from that match will be
172 used as the error string. Otherwise, one subexpression from the current
173 line will be used up as the error string.
174
175 =item '?' [1]
176
177 Prints $value to STDOUT.
178
179 =back
180
181 =cut
182
183 use OpenAFS::CMU_copyright;
184 use OpenAFS::util qw(:DEFAULT :afs_internal);
185 use Exporter;
186 use Symbol;
187
188 $VERSION = '';
189 $VERSION = '1.00';
190 @ISA = qw(Exporter);
191 @EXPORT = qw(&wrapper);
192 @EXPORT_OK = qw(&wrapper &fast_wrapper);
193
194 sub wrapper {
195 my($cmd, $args, $instrs, $options) = @_;
196 my($prevline, $pid, $exception);
197 my(@instrs, $instr, $action, @values, $path);
198 local(%result);
199 my(@werrinstrs) = ([ '^(wrapper\:.*)', '-' ]);
200 my(@cerrinstrs) = ([ '^(' . $cmd . '\:.*)', '-' ],
201 [ '^(' . $path . '\:.*)', '-' ]);
202
203 if ($options->{errors_last}) {
204 @instrs = (@werrinstrs, @$instrs, @cerrinstrs);
205 } else {
206 @instrs = (@werrinstrs, @cerrinstrs, @$instrs);
207 }
208
209 if ($options->{path}) {
210 $path = $options->{path};
211 } elsif ($cmd =~ /^\//) {
212 $path = $cmd;
213 } else {
214 $path = $AFScmd{$cmd};
215 }
216
217 if ($AFS_Trace{wrapper}) {
218 print STDERR "Instructions:\n";
219 foreach $instr (@$instrs) {
220 print STDERR " /", $instr->[0], "/\n";
221 if ($AFS_Trace{wrapper} > 2) {
222 my(@actions) = @$instr;
223 shift(@actions);
224 print " => ",
225 join(', ', map { ref($_) ? "<" . ref($_) . " reference>"
226 : $_ } @actions),
227 "\n";
228 }
229 }
230 }
231
232 ## Start the child
233 if ($options->{pass_stdout}) {
234 open(REALSTDOUT, ">&STDOUT");
235 }
236 $pid = open(AFSCMD, "-|");
237 if (!defined($pid)) {
238 die "wrapper: Fork failed for $cmd: $!\n";
239 }
240
241 ## Run the appropriate program
242 if (!$pid) {
243
244 if ($AFS_Trace{wrapper} > 1) {
245 print STDERR "Command: $path ", join(' ', @$args), "\n";
246 }
247
248 open(STDERR, ">&STDOUT") if (!$options{pass_stderr});
249 if ($options{pass_stdout}) {
250 open(STDOUT, ">&REALSTDOUT");
251 close(REALSTDOUT);
252 }
253
254 { exec($path $cmd, @$args); }
255 # Need to be careful here - we might be doing "vos dump" to STDOUT
256 if ($options{pass_stdout}) {
257 print STDERR "wrapper: Exec failed for $cmd: $!\n";
258 } else {
259 print STDOUT "wrapper: Exec failed for $cmd: $!\n";
260 }
261 exit(127);
262 }
263 if ($options{pass_stdout}) {
264 close(REALSTDOUT);
265 }
266
267 ## Now, parse the output
268 line:
269 while (<AFSCMD>) {
270 my($skip) = 0;
271
272 print STDERR $_ if ($AFS_Trace{wrapper} > 3);
273 chop;
274
275 instr:
276 foreach $instr (@instrs) {
277 my($dot, $action, @actions);
278
279 if ($skip) {
280 $skip--;
281 next instr;
282 }
283 $dot = 0;
284 if ($instr->[0]) {
285 @values = ($_ =~ $instr->[0]);
286 next instr if (!@values);
287 } else {
288 @values = ();
289 }
290
291 act:
292 @actions = @$instr;
293 shift(@actions);
294 foreach $action (@actions) {
295 if (ref($action) eq 'SCALAR') {
296 if (@values) {
297 $$action = shift(@values);
298 } else {
299 last act;
300 }
301 } elsif (ref($action) eq 'ARRAY') {
302 push(@$action, @values);
303 @values = ();
304 } elsif (ref($action) eq 'HASH') {
305 if (@values > 1) {
306 $$action{$values[0]} = $values[1];
307 shift(@values); shift(@values);
308 } elsif (@values) {
309 $$action{shift @values} = '';
310 last act;
311 } else {
312 last act;
313 }
314 } elsif (ref($action) eq 'CODE') {
315 @values = &$action(@values);
316 } elsif (ref($action)) {
317 $exception = "Unknown reference to " . ref($action)
318 . "in parse instructions";
319 last line;
320 } else { ## Must be a string!
321 if ($action eq '.') {
322 $dot = 1;
323 } elsif ($action =~ /\+(\d+)/) {
324 $skip = $1;
325 } elsif ($action =~ /-(.*)/) {
326 my($pat) = $1;
327
328 if ($pat && $prevline) {
329 ($exception) = ($prevline =~ $pat);
330 } elsif (@values) {
331 $exception = shift(@values);
332 } else {
333 $exception = $_;
334 }
335 } elsif ($action eq '?') {
336 print STDOUT (@values ? shift(@values) : $_), "\n";
337 } elsif (@values) {
338 $result{$action} = shift(@values);
339 } else {
340 last act;
341 }
342 }
343 }
344
345 last line if ($exception);
346 last instr if ($dot);
347 }
348 $prevline = $_;
349 }
350 close(AFSCMD);
351 $exception .= "\n" if ($exception && $exception !~ /\n$/);
352 die $exception if ($exception);
353 %result;
354 }
355
356
357 ## Generate code for a fast wrapper (see example below)
358 sub _fastwrap_gen {
359 my($instrs, $refs) = @_;
360 my($SRC, $N, $N1, $X, $instr, $pattern, @actions, $action);
361
362 $N = $X = 0;
363 $N1 = 1;
364
365 $SRC = <<'#####';
366 sub {
367 my($FD, $refs) = @_;
368 my($prevline, @values, $skip, $exception);
369
370 line: while (<$FD>) {
371 #####
372
373 $SRC .= " print STDERR \$_;\n" if ($AFS_Trace{'wrapper'} > 3);
374 $SRC .= " chop;\n";
375
376 foreach $instr (@$instrs) {
377 ($pattern, @actions) = (@$instr);
378 $SRC .= ($pattern ? <<"#####" : <<"#####");
379
380 instr_$N:
381 die \$exception if \$exception;
382 if (\$skip) { \$skip-- } else {
383 \@values = (\$_ =~ /$pattern/);
384 if (\@values) {
385 #####
386
387 instr_$N:
388 die \$exception if \$exception;
389 if (\$skip) { \$skip-- } else {
390 \@values = ();
391 if (1) {
392 #####
393
394 foreach $action (@actions) {
395 if (ref($action) eq 'SCALAR') {
396 $refs[++$X] = $action;
397 $SRC .= <<"#####";
398
399 if (\@values) { \${\$refs[$X]} = shift (\@values) }
400 else { goto instr_$N1 }
401 #####
402
403 } elsif (ref($action) eq 'ARRAY') {
404 $refs[++$X] = $action;
405 $SRC .= <<"#####";
406
407 push(\@{\$refs[$X]}, \@values);
408 \@values = ();
409 #####
410
411 } elsif (ref($action) eq 'HASH') {
412 $refs[++$X] = $action;
413 $SRC .= <<"#####";
414
415 if (\@values > 1) {
416 \$refs[$X]{\$values[0]} = shift(\$values[1]);
417 shift(\@values); shift(\@values);
418 } elsif (\@values) {
419 \$refs[$X]{shift(\@values)} = '';
420 goto instr_$N1;
421 } else {
422 goto instr_$N1;
423 }
424 #####
425
426 } elsif (ref($action) eq 'CODE') {
427 $refs[++$X] = $action;
428 $SRC .= "\n \@values = \$refs[$X]->(\@values);\n";
429
430 } elsif (ref($action)) {
431 die "Unknown reference to " . ref($action) . "in parse instructions\n";
432
433 } elsif ($action eq '.') {
434 $SRC .= "\n next line;\n";
435
436 } elsif ($action eq '?') {
437 $SRC .= <<"#####";
438
439 if (\@values) { print STDOUT shift(\@values), "\\n" }
440 else { print STDOUT \$_, "\\n" }
441 #####
442
443 } elsif ($action =~ /\+(\d+)/) {
444 $SRC .= "\n \$skip = $1;\n";
445
446 } elsif ($action =~ /-(.*)/) {
447 $SRC .= $1 ? <<"#####" : <<"#####";
448
449 if (\$prevline) { (\$exception) = (\$prevline =~ /$1/) }
450 elsif (\@values) { \$exception = shift(\@values) }
451 else { \$exception = \$_ }
452 #####
453
454 if (\@values) { \$exception = shift(\@values) }
455 else { \$exception = \$_ }
456 #####
457
458 } else {
459 $SRC .= <<"#####";
460
461 if (\@values) { \$result{"\Q$action\E"} = shift(\@values) }
462 else { goto instr_$N1 }
463 #####
464 }
465 }
466
467 $N++; $N1++;
468 $SRC .= <<'#####';
469 }
470 }
471 #####
472 }
473
474 $SRC .= <<'#####';
475 } continue {
476 die $exception if $exception;
477 $prevline = $_;
478 }
479 }
480 #####
481
482 $SRC;
483 }
484
485 ####################### Example code #######################
486 # sub {
487 # my($FD, $refs) = @_;
488 # my($prevline, @values, $skip, $exception);
489 #
490 # line: while (<$FD>) {
491 # print STDERR $_; ## if ($AFS_Trace{'wrapper'} > 3);
492 # chop;
493 #
494 # ## Following block repeated for each instruction
495 # instr_N:
496 # die $exception if $exception;
497 # if ($skip) { $skip-- } else {
498 # @values = ($_ =~ /## pattern ##/); ## () if no pattern
499 # if (@values) { ## 1 if no pattern
500 # ## For each action, include one of the following blocks:
501 #
502 # ## SCALAR ref
503 # if (@values) { ${$refs[X]} = shift (@values) }
504 # else { goto instr_N+1 }
505 #
506 # ## ARRAY ref
507 # push(@{$refs[X]}, @values);
508 # @values = ();
509 #
510 # ## HASH ref
511 # if (@values > 1) {
512 # $refs[X]{shift(@values)} = shift(@values);
513 # } elsif (@values) {
514 # $refs[X]{shift(@values)} = '';
515 # goto instr_N+1;
516 # } else {
517 # goto instr_N+1;
518 # }
519 #
520 # ## CODE ref
521 # @values = $refs[X]->(@values);
522 #
523 # ## string '.'
524 # next line;
525 #
526 # ## string '?'
527 # if (@values) { print STDOUT shift(@values), "\n" }
528 # else { print STDOUT $_, "\n" }
529 #
530 # ## string '+DDD'
531 # $skip = DDD;
532 #
533 # ## string '-XXX'
534 # if ($prevline) { ($exception) = ($prefline =~ /XXX/) }
535 # elsif (@values) { $exception = shift(@values) }
536 # else { $exception = $_ }
537 #
538 # ## string '-'
539 # if (@values) { $exception = shift(@values) }
540 # else { $exception = $_ }
541 #
542 # ## anything else
543 # if (@values) { $result{XXX} = shift(@values) }
544 # else { goto instr_N+1 }
545 # }
546 # }
547 #
548 # } continue {
549 # die $exception if $exception;
550 # $prevline = $_;
551 # }
552 # }
553 ############################################################
554
555
556 ## The following does exactly the same thing as wrapper(),
557 ## but should be considerably faster. Instead of interpreting
558 ## parsing instructions, it translates them into perl code,
559 ## which is then compiled into the interpreter. The chief
560 ## benefit to this approach is that we no longer compile
561 ## one RE per instruction per line of input.
562
563 sub fast_wrapper {
564 my($cmd, $args, $instrs, $options) = @_;
565 my(@instrs, $SRC, $CODE, $path, $pid, $refs, $FD, $exception);
566 local(%result);
567 my(@werrinstrs) = ([ '^(wrapper\:.*)', '-' ]);
568 my(@cerrinstrs) = ([ '^(' . $cmd . '\:.*)', '-' ],
569 [ '^(' . $path . '\:.*)', '-' ]);
570
571 $FD = gensym;
572 $refs = [];
573 if ($options->{errors_last}) {
574 @instrs = (@werrinstrs, @$instrs, @cerrinstrs);
575 } else {
576 @instrs = (@werrinstrs, @cerrinstrs, @$instrs);
577 }
578 $SRC = _fastwrap_gen(\@instrs, $refs);
579 $CODE = eval $SRC;
580
581 if ($options->{path}) {
582 $path = $options->{path};
583 } elsif ($cmd =~ /^\//) {
584 $path = $cmd;
585 } else {
586 $path = $AFScmd{$cmd};
587 }
588
589 if ($AFS_Trace{'wrapper'}) {
590 print STDERR "Instructions:\n";
591 foreach $instr (@$instrs) {
592 print STDERR " /", $instr->[0], "/\n";
593 if ($AFS_Trace{'wrapper'} > 2) {
594 my(@actions) = @$instr;
595 shift(@actions);
596 print " => ",
597 join(', ', map { ref($_) ? "<" . ref($_) . " reference>"
598 : $_ } @actions),
599 "\n";
600 }
601 }
602 }
603
604 if ($AFS_Trace{'wrapper'} > 2) { print STDERR "Input parse code:\n$SRC\n" }
605
606 ## Start the child
607 if ($options->{pass_stdout}) {
608 open(REALSTDOUT, ">&STDOUT");
609 }
610 $pid = open($FD, "-|");
611 if (!defined($pid)) {
612 die "wrapper: Fork failed for $cmd: $!\n";
613 }
614
615 ## Run the appropriate program
616 if (!$pid) {
617 if ($AFS_Trace{'wrapper'} > 1) {
618 print STDERR "Command: $path ", join(' ', @$args), "\n";
619 }
620
621 open(STDERR, ">&STDOUT") if (!$options{pass_stderr});
622 if ($options{pass_stdout}) {
623 open(STDOUT, ">&REALSTDOUT");
624 close(REALSTDOUT);
625 }
626
627 { exec($path $cmd, @$args) }
628 # Need to be careful here - we might be doing "vos dump" to STDOUT
629 if ($options{pass_stdout}) {
630 print STDERR "wrapper: Exec failed for $cmd: $!\n";
631 } else {
632 print STDOUT "wrapper: Exec failed for $cmd: $!\n";
633 }
634 exit(127);
635 }
636 if ($options{pass_stdout}) {
637 close(REALSTDOUT);
638 }
639
640 ## Now, parse the output
641 eval { $CODE->($FD, $refs) };
642 $exception = $@;
643
644 close($FD);
645
646 $exception .= "\n" if ($exception && $exception !~ /\n$/);
647 die $exception if ($exception);
648 %result;
649 }
650
651
652 1;
653
654 =head1 EXAMPLES
655
656 The following set of instructions is used by B<wrapper> to detect errors
657 issued by the command, or by the child process spawned to invoke the command.
658 I<$cmd> is the name of the command to run, and I<$path> is the path to the
659 binary actually invoked.
660
661 [ '^(wrapper\:.*)', '-' ]
662 [ '^(' . $cmd . '\:.*)', '-' ]
663 [ '^(' . $path . '\:.*)', '-' ]
664
665 The following instruction is added by the B<OpenAFS::vos> module to catch errors
666 generated by B<vos> commands, which often take the form of a generic error
667 message (Error in vos XXX command), with a description of the specific problem
668 on the preceeding line:
669
670 [ 'Error in vos (.*) command', '-(.*)' ]
671
672 If the AFStools parameter I<vostrace> is nonzero, the following instruction
673 is added to force all lines of output to be copied to STDOUT. Note that this
674 is different from specifying the I<pass_stdout> option, which would pass the
675 command's STDOUT directly to ours without parsing it.
676
677 [ '', '?' ]
678
679 B<OpenAFS::vos::AFS_vos_listvldb> uses the following instructions to parse the
680 output of "vos listvldb". This is a fairly complex example, which illustrates
681 many of the features of B<wrapper>.
682
683 1 ['^(VLDB|Total) entries', '.']
684 2 ['^(\S+)', sub {
685 my(%vinfo) = %OpenAFS::wrapper::result;
686 if ($vinfo{name}) {
687 $vinfo{rosites} = [@rosites] if (@rosites);
688 $vlist{$vinfo{name}} = \%vinfo;
689 @rosites = ();
690 %OpenAFS::wrapper::result = ();
691 }
692 }],
693 3 ['^(\S+)', 'name' ],
694 4 ['RWrite\:\s*(\d+)', 'rwid' ],
695 5 ['ROnly\:\s*(\d+)', 'roid' ],
696 6 ['Backup\:\s*(\d+)', 'bkid' ],
697 7 ['Volume is currently (LOCKED)', 'locked' ],
698 8 ['server (\S+) partition /vicep(\S+) RW Site', 'rwserv', 'rwpart'],
699 9 ['server (\S+) partition /vicep(\S+) RO Site', sub {
700 push(@rosites, [$_[0], $_[1]]);
701 }],
702
703 Instruction 1 matchees the header and trailer lines printed out by B<vos>, and
704 terminates processing of those lines before instructions 2 and 3 have a chance
705 to match it. This is a simple example of a conditional - the next two
706 instructions are used only if this one doesn't match. If we wanted to consider
707 additional instructions even on lines that do match this one, we could place
708 them above this one, or use '+2' instead of '.', which would skip only the next
709 two instructions and allow remaining ones to be processed.
710
711 Instruction 2 matches the first line printed for each volume, stores away any
712 information that has been collected about the previous volume, and prepares for
713 the new one. Besides being a good example of use of a code reference as an
714 action, this instruction also takes advantage of the fact that B<wrapper>'s
715 %result array is a dynamically-scoped variable, and so can be modified by code
716 referenced in parsing instructions.
717
718 The remaining instructions are fairly simple. Instructions 3 through 8 use
719 simple strings to add information about the volume to %result. Instruction 9
720 is a bit more complicated; it uses a function to add a server/partition pair
721 to the current volume's list of RO sites.
722
723 =head1 COPYRIGHT
724
725 The CMUCS AFStools, including this module are
726 Copyright (c) 1996, 2001 Carnegie Mellon University. All rights reserved.
727 For use and redistribution information, see CMUCS/CMU_copyright.pm
728
729 =cut