merge Michaels wonderful "fix" for the currently unwanted ABI Break
[ntk/apt.git] / doc / apt-ftparchive.1.xml
1 <?xml version="1.0" encoding="utf-8" standalone="no"?>
2 <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
3 "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
4
5 <!ENTITY % aptent SYSTEM "apt.ent">
6 %aptent;
7
8 ]>
9
10 <refentry>
11
12 <refentryinfo>
13 &apt-author.jgunthorpe;
14 &apt-author.team;
15 &apt-email;
16 &apt-product;
17 <!-- The last update date -->
18 <date>17 August 2009</date>
19 </refentryinfo>
20
21 <refmeta>
22 <refentrytitle>apt-ftparchive</refentrytitle>
23 <manvolnum>1</manvolnum>
24 <refmiscinfo class="manual">APT</refmiscinfo>
25 </refmeta>
26
27 <!-- Man page title -->
28 <refnamediv>
29 <refname>apt-ftparchive</refname>
30 <refpurpose>Utility to generate index files</refpurpose>
31 </refnamediv>
32
33 <!-- Arguments -->
34 <refsynopsisdiv>
35 <cmdsynopsis>
36 <command>apt-ftparchive</command>
37 <arg><option>-hvdsq</option></arg>
38 <arg><option>--md5</option></arg>
39 <arg><option>--delink</option></arg>
40 <arg><option>--readonly</option></arg>
41 <arg><option>--contents</option></arg>
42 <arg><option>-o <replaceable>config</replaceable>=<replaceable>string</replaceable></option></arg>
43 <arg><option>-c=<replaceable>file</replaceable></option></arg>
44 <group choice="req">
45 <arg>packages<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
46 <arg>sources<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
47 <arg>contents <arg choice="plain"><replaceable>path</replaceable></arg></arg>
48 <arg>release <arg choice="plain"><replaceable>path</replaceable></arg></arg>
49 <arg>generate <arg choice="plain"><replaceable>config-file</replaceable></arg> <arg choice="plain" rep="repeat"><replaceable>section</replaceable></arg></arg>
50 <arg>clean <arg choice="plain"><replaceable>config-file</replaceable></arg></arg>
51 </group>
52 </cmdsynopsis>
53 </refsynopsisdiv>
54
55 <refsect1><title>Description</title>
56 <para><command>apt-ftparchive</command> is the command line tool that generates the index
57 files that APT uses to access a distribution source. The index files should
58 be generated on the origin site based on the content of that site.</para>
59
60 <para><command>apt-ftparchive</command> is a superset of the &dpkg-scanpackages; program,
61 incorporating its entire functionality via the <literal>packages</literal> command.
62 It also contains a contents file generator, <literal>contents</literal>, and an
63 elaborate means to 'script' the generation process for a complete
64 archive.</para>
65
66 <para>Internally <command>apt-ftparchive</command> can make use of binary databases to
67 cache the contents of a .deb file and it does not rely on any external
68 programs aside from &gzip;. When doing a full generate it automatically
69 performs file-change checks and builds the desired compressed output files.</para>
70
71 <para>Unless the <option>-h</option>, or <option>--help</option> option is given one of the
72 commands below must be present.</para>
73
74 <variablelist>
75 <varlistentry><term>packages</term>
76 <listitem><para>
77 The packages command generates a package file from a directory tree. It
78 takes the given directory and recursively searches it for .deb files,
79 emitting a package record to stdout for each. This command is
80 approximately equivalent to &dpkg-scanpackages;.</para>
81
82 <para>The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
83 </varlistentry>
84
85 <varlistentry><term>sources</term>
86 <listitem><para>
87 The <literal>sources</literal> command generates a source index file from a directory tree.
88 It takes the given directory and recursively searches it for .dsc files,
89 emitting a source record to stdout for each. This command is approximately
90 equivalent to &dpkg-scansources;.</para>
91 <para>
92 If an override file is specified then a source override file will be
93 looked for with an extension of .src. The --source-override option can be
94 used to change the source override file that will be used.</para></listitem>
95 </varlistentry>
96
97 <varlistentry><term>contents</term>
98 <listitem><para>
99 The <literal>contents</literal> command generates a contents file from a directory tree. It
100 takes the given directory and recursively searches it for .deb files,
101 and reads the file list from each file. It then sorts and writes to stdout
102 the list of files matched to packages. Directories are not written to
103 the output. If multiple packages own the same file then each package is
104 separated by a comma in the output.</para>
105 <para>
106 The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
107 </varlistentry>
108
109 <varlistentry><term>release</term>
110 <listitem><para>
111 The <literal>release</literal> command generates a Release file from a
112 directory tree. It recursively searches the given directory for
113 Packages, Packages.gz, Packages.bz2, Sources, Sources.gz,
114 Sources.bz2, Release and md5sum.txt files. It then writes to
115 stdout a Release file containing an MD5 digest and SHA1 digest
116 for each file.</para>
117 <para>
118 Values for the additional metadata fields in the Release file are
119 taken from the corresponding variables under
120 <literal>APT::FTPArchive::Release</literal>,
121 e.g. <literal>APT::FTPArchive::Release::Origin</literal>. The supported fields
122 are: <literal>Origin</literal>, <literal>Label</literal>, <literal>Suite</literal>,
123 <literal>Version</literal>, <literal>Codename</literal>, <literal>Date</literal>,
124 <literal>Architectures</literal>, <literal>Components</literal>, <literal>Description</literal>.</para></listitem>
125
126 </varlistentry>
127
128 <varlistentry><term>generate</term>
129 <listitem><para>
130 The <literal>generate</literal> command is designed to be runnable from a cron script and
131 builds indexes according to the given config file. The config language
132 provides a flexible means of specifying which index files are built from
133 which directories, as well as providing a simple means of maintaining the
134 required settings.</para></listitem>
135 </varlistentry>
136
137 <varlistentry><term>clean</term>
138 <listitem><para>
139 The <literal>clean</literal> command tidies the databases used by the given
140 configuration file by removing any records that are no longer necessary.</para></listitem>
141 </varlistentry>
142 </variablelist>
143 </refsect1>
144
145 <refsect1><title>The Generate Configuration</title>
146 <para>
147 The <literal>generate</literal> command uses a configuration file to describe the
148 archives that are going to be generated. It follows the typical ISC
149 configuration format as seen in ISC tools like bind 8 and dhcpd.
150 &apt-conf; contains a description of the syntax. Note that the generate
151 configuration is parsed in sectional manner, but &apt-conf; is parsed in a
152 tree manner. This only effects how the scope tag is handled.</para>
153
154 <para>
155 The generate configuration has 4 separate sections, each described below.</para>
156
157 <refsect2><title>Dir Section</title>
158 <para>
159 The <literal>Dir</literal> section defines the standard directories needed to
160 locate the files required during the generation process. These
161 directories are prepended certain relative paths defined in later
162 sections to produce a complete an absolute path.</para>
163 <variablelist>
164 <varlistentry><term>ArchiveDir</term>
165 <listitem><para>
166 Specifies the root of the FTP archive, in a standard
167 Debian configuration this is the directory that contains the
168 <filename>ls-LR</filename> and dist nodes.</para></listitem>
169 </varlistentry>
170
171 <varlistentry><term>OverrideDir</term>
172 <listitem><para>
173 Specifies the location of the override files.</para></listitem>
174 </varlistentry>
175
176 <varlistentry><term>CacheDir</term>
177 <listitem><para>
178 Specifies the location of the cache files</para></listitem>
179 </varlistentry>
180
181 <varlistentry><term>FileListDir</term>
182 <listitem><para>
183 Specifies the location of the file list files,
184 if the <literal>FileList</literal> setting is used below.</para></listitem>
185 </varlistentry>
186 </variablelist>
187 </refsect2>
188
189 <refsect2><title>Default Section</title>
190 <para>
191 The <literal>Default</literal> section specifies default values, and settings
192 that control the operation of the generator. Other sections may override
193 these defaults with a per-section setting.</para>
194 <variablelist>
195 <varlistentry><term>Packages::Compress</term>
196 <listitem><para>
197 Sets the default compression schemes to use
198 for the Package index files. It is a string that contains a space
199 separated list of at least one of: '.' (no compression), 'gzip' and
200 'bzip2'. The default for all compression schemes is '. gzip'.</para></listitem>
201 </varlistentry>
202
203 <varlistentry><term>Packages::Extensions</term>
204 <listitem><para>
205 Sets the default list of file extensions that are package files.
206 This defaults to '.deb'.</para></listitem>
207 </varlistentry>
208
209 <varlistentry><term>Sources::Compress</term>
210 <listitem><para>
211 This is similar to <literal>Packages::Compress</literal>
212 except that it controls the compression for the Sources files.</para></listitem>
213 </varlistentry>
214
215 <varlistentry><term>Sources::Extensions</term>
216 <listitem><para>
217 Sets the default list of file extensions that are source files.
218 This defaults to '.dsc'.</para></listitem>
219 </varlistentry>
220
221 <varlistentry><term>Contents::Compress</term>
222 <listitem><para>
223 This is similar to <literal>Packages::Compress</literal>
224 except that it controls the compression for the Contents files.</para></listitem>
225 </varlistentry>
226
227 <varlistentry><term>DeLinkLimit</term>
228 <listitem><para>
229 Specifies the number of kilobytes to delink (and
230 replace with hard links) per run. This is used in conjunction with the
231 per-section <literal>External-Links</literal> setting.</para></listitem>
232 </varlistentry>
233
234 <varlistentry><term>FileMode</term>
235 <listitem><para>
236 Specifies the mode of all created index files. It
237 defaults to 0644. All index files are set to this mode with no regard
238 to the umask.</para></listitem>
239 </varlistentry>
240 </variablelist>
241 </refsect2>
242
243 <refsect2><title>TreeDefault Section</title>
244 <para>
245 Sets defaults specific to <literal>Tree</literal> sections. All of these
246 variables are substitution variables and have the strings $(DIST),
247 $(SECTION) and $(ARCH) replaced with their respective values.</para>
248
249 <variablelist>
250 <varlistentry><term>MaxContentsChange</term>
251 <listitem><para>
252 Sets the number of kilobytes of contents
253 files that are generated each day. The contents files are round-robined
254 so that over several days they will all be rebuilt.</para></listitem>
255 </varlistentry>
256
257 <varlistentry><term>ContentsAge</term>
258 <listitem><para>
259 Controls the number of days a contents file is allowed
260 to be checked without changing. If this limit is passed the mtime of the
261 contents file is updated. This case can occur if the package file is
262 changed in such a way that does not result in a new contents file
263 [override edit for instance]. A hold off is allowed in hopes that new
264 .debs will be installed, requiring a new file anyhow. The default is 10,
265 the units are in days.</para></listitem>
266 </varlistentry>
267
268 <varlistentry><term>Directory</term>
269 <listitem><para>
270 Sets the top of the .deb directory tree. Defaults to
271 <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/</filename></para></listitem>
272 </varlistentry>
273
274 <varlistentry><term>SrcDirectory</term>
275 <listitem><para>
276 Sets the top of the source package directory tree. Defaults to
277 <filename>$(DIST)/$(SECTION)/source/</filename></para></listitem>
278 </varlistentry>
279
280 <varlistentry><term>Packages</term>
281 <listitem><para>
282 Sets the output Packages file. Defaults to
283 <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/Packages</filename></para></listitem>
284 </varlistentry>
285
286 <varlistentry><term>Sources</term>
287 <listitem><para>
288 Sets the output Sources file. Defaults to
289 <filename>$(DIST)/$(SECTION)/source/Sources</filename></para></listitem>
290 </varlistentry>
291
292 <varlistentry><term>InternalPrefix</term>
293 <listitem><para>
294 Sets the path prefix that causes a symlink to be
295 considered an internal link instead of an external link. Defaults to
296 <filename>$(DIST)/$(SECTION)/</filename></para></listitem>
297 </varlistentry>
298
299 <varlistentry><term>Contents</term>
300 <listitem><para>
301 Sets the output Contents file. Defaults to
302 <filename>$(DIST)/Contents-$(ARCH)</filename>. If this setting causes multiple
303 Packages files to map onto a single Contents file (such as the default)
304 then <command>apt-ftparchive</command> will integrate those package files
305 together automatically.</para></listitem>
306 </varlistentry>
307
308 <varlistentry><term>Contents::Header</term>
309 <listitem><para>
310 Sets header file to prepend to the contents output.</para></listitem>
311 </varlistentry>
312
313 <varlistentry><term>BinCacheDB</term>
314 <listitem><para>
315 Sets the binary cache database to use for this
316 section. Multiple sections can share the same database.</para></listitem>
317 </varlistentry>
318
319 <varlistentry><term>FileList</term>
320 <listitem><para>
321 Specifies that instead of walking the directory tree,
322 <command>apt-ftparchive</command> should read the list of files from the given
323 file. Relative files names are prefixed with the archive directory.</para></listitem>
324 </varlistentry>
325
326 <varlistentry><term>SourceFileList</term>
327 <listitem><para>
328 Specifies that instead of walking the directory tree,
329 <command>apt-ftparchive</command> should read the list of files from the given
330 file. Relative files names are prefixed with the archive directory.
331 This is used when processing source indexes.</para></listitem>
332 </varlistentry>
333 </variablelist>
334 </refsect2>
335
336 <refsect2><title>Tree Section</title>
337 <para>
338 The <literal>Tree</literal> section defines a standard Debian file tree which
339 consists of a base directory, then multiple sections in that base
340 directory and finally multiple Architectures in each section. The exact
341 pathing used is defined by the <literal>Directory</literal> substitution variable.</para>
342 <para>
343 The <literal>Tree</literal> section takes a scope tag which sets the
344 <literal>$(DIST)</literal> variable and defines the root of the tree
345 (the path is prefixed by <literal>ArchiveDir</literal>).
346 Typically this is a setting such as <filename>dists/woody</filename>.</para>
347 <para>
348 All of the settings defined in the <literal>TreeDefault</literal> section can be
349 use in a <literal>Tree</literal> section as well as three new variables.</para>
350 <para>
351 When processing a <literal>Tree</literal> section <command>apt-ftparchive</command>
352 performs an operation similar to:
353 <informalexample><programlisting>
354 for i in Sections do
355 for j in Architectures do
356 Generate for DIST=scope SECTION=i ARCH=j
357 </programlisting></informalexample></para>
358
359 <variablelist>
360 <varlistentry><term>Sections</term>
361 <listitem><para>
362 This is a space separated list of sections which appear
363 under the distribution, typically this is something like
364 <literal>main contrib non-free</literal></para></listitem>
365 </varlistentry>
366
367 <varlistentry><term>Architectures</term>
368 <listitem><para>
369 This is a space separated list of all the
370 architectures that appear under search section. The special architecture
371 'source' is used to indicate that this tree has a source archive.</para></listitem>
372 </varlistentry>
373
374 <varlistentry><term>BinOverride</term>
375 <listitem><para>
376 Sets the binary override file. The override file
377 contains section, priority and maintainer address information.</para></listitem>
378 </varlistentry>
379
380 <varlistentry><term>SrcOverride</term>
381 <listitem><para>
382 Sets the source override file. The override file
383 contains section information.</para></listitem>
384 </varlistentry>
385
386 <varlistentry><term>ExtraOverride</term>
387 <listitem><para>
388 Sets the binary extra override file.</para></listitem>
389 </varlistentry>
390
391 <varlistentry><term>SrcExtraOverride</term>
392 <listitem><para>
393 Sets the source extra override file.</para></listitem>
394 </varlistentry>
395 </variablelist>
396 </refsect2>
397
398 <refsect2><title>BinDirectory Section</title>
399 <para>
400 The <literal>bindirectory</literal> section defines a binary directory tree
401 with no special structure. The scope tag specifies the location of
402 the binary directory and the settings are similar to the <literal>Tree</literal>
403 section with no substitution variables or
404 <literal>Section</literal><literal>Architecture</literal> settings.</para>
405 <variablelist>
406 <varlistentry><term>Packages</term>
407 <listitem><para>
408 Sets the Packages file output.</para></listitem>
409 </varlistentry>
410
411 <varlistentry><term>Sources</term>
412 <listitem><para>
413 Sets the Sources file output. At least one of
414 <literal>Packages</literal> or <literal>Sources</literal> is required.</para></listitem>
415 </varlistentry>
416
417 <varlistentry><term>Contents</term>
418 <listitem><para>
419 Sets the Contents file output. (optional)</para></listitem>
420 </varlistentry>
421
422 <varlistentry><term>BinOverride</term>
423 <listitem><para>
424 Sets the binary override file.</para></listitem>
425 </varlistentry>
426
427 <varlistentry><term>SrcOverride</term>
428 <listitem><para>
429 Sets the source override file.</para></listitem>
430 </varlistentry>
431
432 <varlistentry><term>ExtraOverride</term>
433 <listitem><para>
434 Sets the binary extra override file.</para></listitem>
435 </varlistentry>
436
437 <varlistentry><term>SrcExtraOverride</term>
438 <listitem><para>
439 Sets the source extra override file.</para></listitem>
440 </varlistentry>
441
442 <varlistentry><term>BinCacheDB</term>
443 <listitem><para>
444 Sets the cache DB.</para></listitem>
445 </varlistentry>
446
447 <varlistentry><term>PathPrefix</term>
448 <listitem><para>
449 Appends a path to all the output paths.</para></listitem>
450 </varlistentry>
451
452 <varlistentry><term>FileList, SourceFileList</term>
453 <listitem><para>
454 Specifies the file list file.</para></listitem>
455 </varlistentry>
456 </variablelist>
457 </refsect2>
458 </refsect1>
459
460
461 <refsect1><title>The Binary Override File</title>
462 <para>The binary override file is fully compatible with &dpkg-scanpackages;. It
463 contains 4 fields separated by spaces. The first field is the package name,
464 the second is the priority to force that package to, the third is the
465 the section to force that package to and the final field is the maintainer
466 permutation field.</para>
467 <para>The general form of the maintainer field is:
468 <literallayout>old [// oldn]* => new</literallayout>
469 or simply,
470 <literallayout>new</literallayout>
471 The first form allows a double-slash separated list of old email addresses
472 to be specified. If any of those are found then new is substituted for the
473 maintainer field. The second form unconditionally substitutes the
474 maintainer field.</para>
475 </refsect1>
476
477
478 <refsect1><title>The Source Override File</title>
479 <para>
480 The source override file is fully compatible with &dpkg-scansources;. It
481 contains 2 fields separated by spaces. The first fields is the source
482 package name, the second is the section to assign it.</para>
483 </refsect1>
484
485 <refsect1><title>The Extra Override File</title>
486 <para>
487 The extra override file allows any arbitrary tag to be added or replaced
488 in the output. It has 3 columns, the first is the package, the second is
489 the tag and the remainder of the line is the new value.</para>
490 </refsect1>
491
492 <refsect1><title>options</title>
493 &apt-cmdblurb;
494
495 <variablelist>
496 <varlistentry><term><option>--md5</option></term>
497 <listitem><para>
498 Generate MD5 sums. This defaults to on, when turned off the generated
499 index files will not have MD5Sum fields where possible.
500 Configuration Item: <literal>APT::FTPArchive::MD5</literal></para></listitem>
501 </varlistentry>
502
503 <varlistentry><term><option>-d</option></term><term><option>--db</option></term>
504 <listitem><para>
505 Use a binary caching DB. This has no effect on the generate command.
506 Configuration Item: <literal>APT::FTPArchive::DB</literal>.</para></listitem>
507 </varlistentry>
508
509 <varlistentry><term><option>-q</option></term><term><option>--quiet</option></term>
510 <listitem><para>
511 Quiet; produces output suitable for logging, omitting progress indicators.
512 More q's will produce more quiet up to a maximum of 2. You can also use
513 <option>-q=#</option> to set the quiet level, overriding the configuration file.
514 Configuration Item: <literal>quiet</literal>.</para></listitem>
515 </varlistentry>
516
517 <varlistentry><term><option>--delink</option></term>
518 <listitem><para>
519 Perform Delinking. If the <literal>External-Links</literal> setting is used then
520 this option actually enables delinking of the files. It defaults to on and
521 can be turned off with <option>--no-delink</option>.
522 Configuration Item: <literal>APT::FTPArchive::DeLinkAct</literal>.</para></listitem>
523 </varlistentry>
524
525 <varlistentry><term><option>--contents</option></term>
526 <listitem><para>
527 Perform contents generation. When this option is set and package indexes
528 are being generated with a cache DB then the file listing will also be
529 extracted and stored in the DB for later use. When using the generate
530 command this option also allows the creation of any Contents files. The
531 default is on.
532 Configuration Item: <literal>APT::FTPArchive::Contents</literal>.</para></listitem>
533 </varlistentry>
534
535 <varlistentry><term><option>-s</option></term><term><option>--source-override</option></term>
536 <listitem><para>
537 Select the source override file to use with the <literal>sources</literal> command.
538 Configuration Item: <literal>APT::FTPArchive::SourceOverride</literal>.</para></listitem>
539 </varlistentry>
540
541 <varlistentry><term><option>--readonly</option></term>
542 <listitem><para>
543 Make the caching databases read only.
544 Configuration Item: <literal>APT::FTPArchive::ReadOnlyDB</literal>.</para></listitem>
545 </varlistentry>
546
547 <varlistentry><term><option>APT::FTPArchive::AlwaysStat</option></term>
548 <listitem><para>
549 &apt-ftparchive; caches as much as possible of metadata in it is cachedb. If packages
550 are recompiled and/or republished with the same version again, this will lead to problems
551 as the now outdated cached metadata like size and checksums will be used. With this option
552 enabled this will no longer happen as it will be checked if the file was changed.
553 Note that this option is set to "<literal>false</literal>" by default as it is not recommend
554 to upload multiply versions/builds of a package with the same versionnumber, so in theory
555 nobody will have these problems and therefore all these extra checks are useless.
556 </para></listitem>
557 </varlistentry>
558
559 <varlistentry><term><option>APT::FTPArchive::LongDescription</option></term>
560 <listitem><para>
561 This configuration option defaults to "<literal>true</literal>" and should only be set to
562 <literal>"false"</literal> if the Archive generated with &apt-ftparchive; also provides
563 <filename>Translation</filename> files. Note that it is currently not possible to create these
564 files with <command>apt-ftparchive</command>.
565 </para></listitem>
566 </varlistentry>
567
568 &apt-commonoptions;
569
570 </variablelist>
571 </refsect1>
572
573 <refsect1><title>Examples</title>
574
575 <para>To create a compressed Packages file for a directory containing
576 binary packages (.deb):
577
578 <programlisting>
579 <command>apt-ftparchive</command> packages <replaceable>directory</replaceable> | <command>gzip</command> > <filename>Packages.gz</filename>
580 </programlisting></para>
581
582 </refsect1>
583
584 <refsect1><title>See Also</title>
585 <para>&apt-conf;</para>
586 </refsect1>
587
588 <refsect1><title>Diagnostics</title>
589 <para><command>apt-ftparchive</command> returns zero on normal operation, decimal 100 on error.</para>
590 </refsect1>
591
592 &manbugs;
593
594 </refentry>