* doc/*:
[ntk/apt.git] / doc / apt-ftparchive.1.xml
1 <?xml version="1.0" encoding="utf-8" standalone="no"?>
2 <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
3 "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
4
5 <!ENTITY % aptent SYSTEM "apt.ent">
6 %aptent;
7
8 <!ENTITY % aptverbatiment SYSTEM "apt-verbatim.ent">
9 %aptverbatiment;
10
11 ]>
12
13 <refentry>
14
15 <refentryinfo>
16 &apt-author.jgunthorpe;
17 &apt-author.team;
18 &apt-email;
19 &apt-product;
20 <!-- The last update date -->
21 <date>17 August 2009</date>
22 </refentryinfo>
23
24 <refmeta>
25 <refentrytitle>apt-ftparchive</refentrytitle>
26 <manvolnum>1</manvolnum>
27 <refmiscinfo class="manual">APT</refmiscinfo>
28 </refmeta>
29
30 <!-- Man page title -->
31 <refnamediv>
32 <refname>apt-ftparchive</refname>
33 <refpurpose>Utility to generate index files</refpurpose>
34 </refnamediv>
35
36 &synopsis-command-apt-ftparchive;
37
38 <refsect1><title>Description</title>
39 <para><command>apt-ftparchive</command> is the command line tool that generates the index
40 files that APT uses to access a distribution source. The index files should
41 be generated on the origin site based on the content of that site.</para>
42
43 <para><command>apt-ftparchive</command> is a superset of the &dpkg-scanpackages; program,
44 incorporating its entire functionality via the <literal>packages</literal> command.
45 It also contains a contents file generator, <literal>contents</literal>, and an
46 elaborate means to 'script' the generation process for a complete
47 archive.</para>
48
49 <para>Internally <command>apt-ftparchive</command> can make use of binary databases to
50 cache the contents of a .deb file and it does not rely on any external
51 programs aside from &gzip;. When doing a full generate it automatically
52 performs file-change checks and builds the desired compressed output files.</para>
53
54 <para>Unless the <option>-h</option>, or <option>--help</option> option is given one of the
55 commands below must be present.</para>
56
57 <variablelist>
58 <varlistentry><term>packages</term>
59 <listitem><para>
60 The packages command generates a package file from a directory tree. It
61 takes the given directory and recursively searches it for .deb files,
62 emitting a package record to stdout for each. This command is
63 approximately equivalent to &dpkg-scanpackages;.</para>
64
65 <para>The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
66 </varlistentry>
67
68 <varlistentry><term>sources</term>
69 <listitem><para>
70 The <literal>sources</literal> command generates a source index file from a directory tree.
71 It takes the given directory and recursively searches it for .dsc files,
72 emitting a source record to stdout for each. This command is approximately
73 equivalent to &dpkg-scansources;.</para>
74 <para>
75 If an override file is specified then a source override file will be
76 looked for with an extension of .src. The --source-override option can be
77 used to change the source override file that will be used.</para></listitem>
78 </varlistentry>
79
80 <varlistentry><term>contents</term>
81 <listitem><para>
82 The <literal>contents</literal> command generates a contents file from a directory tree. It
83 takes the given directory and recursively searches it for .deb files,
84 and reads the file list from each file. It then sorts and writes to stdout
85 the list of files matched to packages. Directories are not written to
86 the output. If multiple packages own the same file then each package is
87 separated by a comma in the output.</para>
88 <para>
89 The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
90 </varlistentry>
91
92 <varlistentry><term>release</term>
93 <listitem><para>
94 The <literal>release</literal> command generates a Release file from a
95 directory tree. It recursively searches the given directory for uncompressed
96 <filename>Packages</filename> and <filename>Sources</filename> files and the ones
97 compressed with <command>gzip</command>, <command>bzip2</command> or <command>lzma</command>
98 as well as <filename>Release</filename> and <filename>md5sum.txt</filename> files by default
99 (<literal>APT::FTPArchive::Release::Default-Patterns</literal>). Additional filename patterns
100 can be added by listing them in <literal>APT::FTPArchive::Release::Patterns</literal>.
101 It then writes to stdout a Release file containing a MD5, SHA1 and SHA256 digest
102 for each file.</para>
103 <para>
104 Values for the additional metadata fields in the Release file are
105 taken from the corresponding variables under
106 <literal>APT::FTPArchive::Release</literal>,
107 e.g. <literal>APT::FTPArchive::Release::Origin</literal>. The supported fields
108 are: <literal>Origin</literal>, <literal>Label</literal>, <literal>Suite</literal>,
109 <literal>Version</literal>, <literal>Codename</literal>, <literal>Date</literal>,
110 <literal>Valid-Until</literal>, <literal>Architectures</literal>,
111 <literal>Components</literal>, <literal>Description</literal>.</para></listitem>
112
113 </varlistentry>
114
115 <varlistentry><term>generate</term>
116 <listitem><para>
117 The <literal>generate</literal> command is designed to be runnable from a cron script and
118 builds indexes according to the given config file. The config language
119 provides a flexible means of specifying which index files are built from
120 which directories, as well as providing a simple means of maintaining the
121 required settings.</para></listitem>
122 </varlistentry>
123
124 <varlistentry><term>clean</term>
125 <listitem><para>
126 The <literal>clean</literal> command tidies the databases used by the given
127 configuration file by removing any records that are no longer necessary.</para></listitem>
128 </varlistentry>
129 </variablelist>
130 </refsect1>
131
132 <refsect1><title>The Generate Configuration</title>
133 <para>
134 The <literal>generate</literal> command uses a configuration file to describe the
135 archives that are going to be generated. It follows the typical ISC
136 configuration format as seen in ISC tools like bind 8 and dhcpd.
137 &apt-conf; contains a description of the syntax. Note that the generate
138 configuration is parsed in sectional manner, but &apt-conf; is parsed in a
139 tree manner. This only effects how the scope tag is handled.</para>
140
141 <para>
142 The generate configuration has 4 separate sections, each described below.</para>
143
144 <refsect2><title>Dir Section</title>
145 <para>
146 The <literal>Dir</literal> section defines the standard directories needed to
147 locate the files required during the generation process. These
148 directories are prepended certain relative paths defined in later
149 sections to produce a complete an absolute path.</para>
150 <variablelist>
151 <varlistentry><term>ArchiveDir</term>
152 <listitem><para>
153 Specifies the root of the FTP archive, in a standard
154 Debian configuration this is the directory that contains the
155 <filename>ls-LR</filename> and dist nodes.</para></listitem>
156 </varlistentry>
157
158 <varlistentry><term>OverrideDir</term>
159 <listitem><para>
160 Specifies the location of the override files.</para></listitem>
161 </varlistentry>
162
163 <varlistentry><term>CacheDir</term>
164 <listitem><para>
165 Specifies the location of the cache files</para></listitem>
166 </varlistentry>
167
168 <varlistentry><term>FileListDir</term>
169 <listitem><para>
170 Specifies the location of the file list files,
171 if the <literal>FileList</literal> setting is used below.</para></listitem>
172 </varlistentry>
173 </variablelist>
174 </refsect2>
175
176 <refsect2><title>Default Section</title>
177 <para>
178 The <literal>Default</literal> section specifies default values, and settings
179 that control the operation of the generator. Other sections may override
180 these defaults with a per-section setting.</para>
181 <variablelist>
182 <varlistentry><term>Packages::Compress</term>
183 <listitem><para>
184 Sets the default compression schemes to use
185 for the Package index files. It is a string that contains a space
186 separated list of at least one of: '.' (no compression), 'gzip' and
187 'bzip2'. The default for all compression schemes is '. gzip'.</para></listitem>
188 </varlistentry>
189
190 <varlistentry><term>Packages::Extensions</term>
191 <listitem><para>
192 Sets the default list of file extensions that are package files.
193 This defaults to '.deb'.</para></listitem>
194 </varlistentry>
195
196 <varlistentry><term>Sources::Compress</term>
197 <listitem><para>
198 This is similar to <literal>Packages::Compress</literal>
199 except that it controls the compression for the Sources files.</para></listitem>
200 </varlistentry>
201
202 <varlistentry><term>Sources::Extensions</term>
203 <listitem><para>
204 Sets the default list of file extensions that are source files.
205 This defaults to '.dsc'.</para></listitem>
206 </varlistentry>
207
208 <varlistentry><term>Contents::Compress</term>
209 <listitem><para>
210 This is similar to <literal>Packages::Compress</literal>
211 except that it controls the compression for the Contents files.</para></listitem>
212 </varlistentry>
213
214 <varlistentry><term>Translation::Compress</term>
215 <listitem><para>
216 This is similar to <literal>Packages::Compress</literal>
217 except that it controls the compression for the Translation-en master file.</para></listitem>
218 </varlistentry>
219
220 <varlistentry><term>DeLinkLimit</term>
221 <listitem><para>
222 Specifies the number of kilobytes to delink (and
223 replace with hard links) per run. This is used in conjunction with the
224 per-section <literal>External-Links</literal> setting.</para></listitem>
225 </varlistentry>
226
227 <varlistentry><term>FileMode</term>
228 <listitem><para>
229 Specifies the mode of all created index files. It
230 defaults to 0644. All index files are set to this mode with no regard
231 to the umask.</para></listitem>
232 </varlistentry>
233
234 <varlistentry><term>LongDescription</term>
235 <listitem><para>
236 Sets if long descriptions should be included in the Packages file or split
237 out into a master Translation-en file.</para></listitem>
238 </varlistentry>
239 </variablelist>
240 </refsect2>
241
242 <refsect2><title>TreeDefault Section</title>
243 <para>
244 Sets defaults specific to <literal>Tree</literal> sections. All of these
245 variables are substitution variables and have the strings $(DIST),
246 $(SECTION) and $(ARCH) replaced with their respective values.</para>
247
248 <variablelist>
249 <varlistentry><term>MaxContentsChange</term>
250 <listitem><para>
251 Sets the number of kilobytes of contents
252 files that are generated each day. The contents files are round-robined
253 so that over several days they will all be rebuilt.</para></listitem>
254 </varlistentry>
255
256 <varlistentry><term>ContentsAge</term>
257 <listitem><para>
258 Controls the number of days a contents file is allowed
259 to be checked without changing. If this limit is passed the mtime of the
260 contents file is updated. This case can occur if the package file is
261 changed in such a way that does not result in a new contents file
262 [override edit for instance]. A hold off is allowed in hopes that new
263 .debs will be installed, requiring a new file anyhow. The default is 10,
264 the units are in days.</para></listitem>
265 </varlistentry>
266
267 <varlistentry><term>Directory</term>
268 <listitem><para>
269 Sets the top of the .deb directory tree. Defaults to
270 <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/</filename></para></listitem>
271 </varlistentry>
272
273 <varlistentry><term>SrcDirectory</term>
274 <listitem><para>
275 Sets the top of the source package directory tree. Defaults to
276 <filename>$(DIST)/$(SECTION)/source/</filename></para></listitem>
277 </varlistentry>
278
279 <varlistentry><term>Packages</term>
280 <listitem><para>
281 Sets the output Packages file. Defaults to
282 <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/Packages</filename></para></listitem>
283 </varlistentry>
284
285 <varlistentry><term>Sources</term>
286 <listitem><para>
287 Sets the output Sources file. Defaults to
288 <filename>$(DIST)/$(SECTION)/source/Sources</filename></para></listitem>
289 </varlistentry>
290
291 <varlistentry><term>Translation</term>
292 <listitem><para>
293 Set the output Translation-en master file with the long descriptions if they
294 should be not included in the Packages file. Defaults to
295 <filename>$(DIST)/$(SECTION)/i18n/Translation-en</filename></para></listitem>
296 </varlistentry>
297
298 <varlistentry><term>InternalPrefix</term>
299 <listitem><para>
300 Sets the path prefix that causes a symlink to be
301 considered an internal link instead of an external link. Defaults to
302 <filename>$(DIST)/$(SECTION)/</filename></para></listitem>
303 </varlistentry>
304
305 <varlistentry><term>Contents</term>
306 <listitem><para>
307 Sets the output Contents file. Defaults to
308 <filename>$(DIST)/Contents-$(ARCH)</filename>. If this setting causes multiple
309 Packages files to map onto a single Contents file (such as the default)
310 then <command>apt-ftparchive</command> will integrate those package files
311 together automatically.</para></listitem>
312 </varlistentry>
313
314 <varlistentry><term>Contents::Header</term>
315 <listitem><para>
316 Sets header file to prepend to the contents output.</para></listitem>
317 </varlistentry>
318
319 <varlistentry><term>BinCacheDB</term>
320 <listitem><para>
321 Sets the binary cache database to use for this
322 section. Multiple sections can share the same database.</para></listitem>
323 </varlistentry>
324
325 <varlistentry><term>FileList</term>
326 <listitem><para>
327 Specifies that instead of walking the directory tree,
328 <command>apt-ftparchive</command> should read the list of files from the given
329 file. Relative files names are prefixed with the archive directory.</para></listitem>
330 </varlistentry>
331
332 <varlistentry><term>SourceFileList</term>
333 <listitem><para>
334 Specifies that instead of walking the directory tree,
335 <command>apt-ftparchive</command> should read the list of files from the given
336 file. Relative files names are prefixed with the archive directory.
337 This is used when processing source indexes.</para></listitem>
338 </varlistentry>
339 </variablelist>
340 </refsect2>
341
342 <refsect2><title>Tree Section</title>
343 <para>
344 The <literal>Tree</literal> section defines a standard Debian file tree which
345 consists of a base directory, then multiple sections in that base
346 directory and finally multiple Architectures in each section. The exact
347 pathing used is defined by the <literal>Directory</literal> substitution variable.</para>
348 <para>
349 The <literal>Tree</literal> section takes a scope tag which sets the
350 <literal>$(DIST)</literal> variable and defines the root of the tree
351 (the path is prefixed by <literal>ArchiveDir</literal>).
352 Typically this is a setting such as <filename>dists/&stable-codename;</filename>.</para>
353 <para>
354 All of the settings defined in the <literal>TreeDefault</literal> section can be
355 use in a <literal>Tree</literal> section as well as three new variables.</para>
356 <para>
357 When processing a <literal>Tree</literal> section <command>apt-ftparchive</command>
358 performs an operation similar to:
359 <programlisting>
360 for i in Sections do
361 for j in Architectures do
362 Generate for DIST=scope SECTION=i ARCH=j
363 </programlisting></para>
364
365 <variablelist>
366 <varlistentry><term>Sections</term>
367 <listitem><para>
368 This is a space separated list of sections which appear
369 under the distribution, typically this is something like
370 <literal>main contrib non-free</literal></para></listitem>
371 </varlistentry>
372
373 <varlistentry><term>Architectures</term>
374 <listitem><para>
375 This is a space separated list of all the
376 architectures that appear under search section. The special architecture
377 'source' is used to indicate that this tree has a source archive.</para></listitem>
378 </varlistentry>
379
380 <varlistentry><term>LongDescription</term>
381 <listitem><para>
382 Sets if long descriptions should be included in the Packages file or split
383 out into a master Translation-en file.</para></listitem>
384 </varlistentry>
385
386 <varlistentry><term>BinOverride</term>
387 <listitem><para>
388 Sets the binary override file. The override file
389 contains section, priority and maintainer address information.</para></listitem>
390 </varlistentry>
391
392 <varlistentry><term>SrcOverride</term>
393 <listitem><para>
394 Sets the source override file. The override file
395 contains section information.</para></listitem>
396 </varlistentry>
397
398 <varlistentry><term>ExtraOverride</term>
399 <listitem><para>
400 Sets the binary extra override file.</para></listitem>
401 </varlistentry>
402
403 <varlistentry><term>SrcExtraOverride</term>
404 <listitem><para>
405 Sets the source extra override file.</para></listitem>
406 </varlistentry>
407 </variablelist>
408 </refsect2>
409
410 <refsect2><title>BinDirectory Section</title>
411 <para>
412 The <literal>bindirectory</literal> section defines a binary directory tree
413 with no special structure. The scope tag specifies the location of
414 the binary directory and the settings are similar to the <literal>Tree</literal>
415 section with no substitution variables or
416 <literal>Section</literal><literal>Architecture</literal> settings.</para>
417 <variablelist>
418 <varlistentry><term>Packages</term>
419 <listitem><para>
420 Sets the Packages file output.</para></listitem>
421 </varlistentry>
422
423 <varlistentry><term>Sources</term>
424 <listitem><para>
425 Sets the Sources file output. At least one of
426 <literal>Packages</literal> or <literal>Sources</literal> is required.</para></listitem>
427 </varlistentry>
428
429 <varlistentry><term>Contents</term>
430 <listitem><para>
431 Sets the Contents file output. (optional)</para></listitem>
432 </varlistentry>
433
434 <varlistentry><term>BinOverride</term>
435 <listitem><para>
436 Sets the binary override file.</para></listitem>
437 </varlistentry>
438
439 <varlistentry><term>SrcOverride</term>
440 <listitem><para>
441 Sets the source override file.</para></listitem>
442 </varlistentry>
443
444 <varlistentry><term>ExtraOverride</term>
445 <listitem><para>
446 Sets the binary extra override file.</para></listitem>
447 </varlistentry>
448
449 <varlistentry><term>SrcExtraOverride</term>
450 <listitem><para>
451 Sets the source extra override file.</para></listitem>
452 </varlistentry>
453
454 <varlistentry><term>BinCacheDB</term>
455 <listitem><para>
456 Sets the cache DB.</para></listitem>
457 </varlistentry>
458
459 <varlistentry><term>PathPrefix</term>
460 <listitem><para>
461 Appends a path to all the output paths.</para></listitem>
462 </varlistentry>
463
464 <varlistentry><term>FileList, SourceFileList</term>
465 <listitem><para>
466 Specifies the file list file.</para></listitem>
467 </varlistentry>
468 </variablelist>
469 </refsect2>
470 </refsect1>
471
472
473 <refsect1><title>The Binary Override File</title>
474 <para>The binary override file is fully compatible with &dpkg-scanpackages;. It
475 contains 4 fields separated by spaces. The first field is the package name,
476 the second is the priority to force that package to, the third is the
477 the section to force that package to and the final field is the maintainer
478 permutation field.</para>
479 <para>The general form of the maintainer field is:
480 <literallayout>old [// oldn]* => new</literallayout>
481 or simply,
482 <literallayout>new</literallayout>
483 The first form allows a double-slash separated list of old email addresses
484 to be specified. If any of those are found then new is substituted for the
485 maintainer field. The second form unconditionally substitutes the
486 maintainer field.</para>
487 </refsect1>
488
489
490 <refsect1><title>The Source Override File</title>
491 <para>
492 The source override file is fully compatible with &dpkg-scansources;. It
493 contains 2 fields separated by spaces. The first fields is the source
494 package name, the second is the section to assign it.</para>
495 </refsect1>
496
497 <refsect1><title>The Extra Override File</title>
498 <para>
499 The extra override file allows any arbitrary tag to be added or replaced
500 in the output. It has 3 columns, the first is the package, the second is
501 the tag and the remainder of the line is the new value.</para>
502 </refsect1>
503
504 <refsect1><title>options</title>
505 &apt-cmdblurb;
506
507 <variablelist>
508 <varlistentry><term><option>--md5</option>, <option>--sha1</option>, <option>--sha256</option></term>
509 <listitem><para>
510 Generate the given checksum. These options default to on, when turned off the generated
511 index files will not have the checksum fields where possible.
512 Configuration Items: <literal>APT::FTPArchive::<replaceable>Checksum</replaceable></literal> and
513 <literal>APT::FTPArchive::<replaceable>Index</replaceable>::<replaceable>Checksum</replaceable></literal> where
514 <literal><replaceable>Index</replaceable></literal> can be <literal>Packages</literal>, <literal>Sources</literal> or
515 <literal>Release</literal> and <literal><replaceable>Checksum</replaceable></literal> can be <literal>MD5</literal>,
516 <literal>SHA1</literal> or <literal>SHA256</literal>.</para></listitem>
517 </varlistentry>
518
519 <varlistentry><term><option>-d</option></term><term><option>--db</option></term>
520 <listitem><para>
521 Use a binary caching DB. This has no effect on the generate command.
522 Configuration Item: <literal>APT::FTPArchive::DB</literal>.</para></listitem>
523 </varlistentry>
524
525 <varlistentry><term><option>-q</option></term><term><option>--quiet</option></term>
526 <listitem><para>
527 Quiet; produces output suitable for logging, omitting progress indicators.
528 More q's will produce more quiet up to a maximum of 2. You can also use
529 <option>-q=#</option> to set the quiet level, overriding the configuration file.
530 Configuration Item: <literal>quiet</literal>.</para></listitem>
531 </varlistentry>
532
533 <varlistentry><term><option>--delink</option></term>
534 <listitem><para>
535 Perform Delinking. If the <literal>External-Links</literal> setting is used then
536 this option actually enables delinking of the files. It defaults to on and
537 can be turned off with <option>--no-delink</option>.
538 Configuration Item: <literal>APT::FTPArchive::DeLinkAct</literal>.</para></listitem>
539 </varlistentry>
540
541 <varlistentry><term><option>--contents</option></term>
542 <listitem><para>
543 Perform contents generation. When this option is set and package indexes
544 are being generated with a cache DB then the file listing will also be
545 extracted and stored in the DB for later use. When using the generate
546 command this option also allows the creation of any Contents files. The
547 default is on.
548 Configuration Item: <literal>APT::FTPArchive::Contents</literal>.</para></listitem>
549 </varlistentry>
550
551 <varlistentry><term><option>-s</option></term><term><option>--source-override</option></term>
552 <listitem><para>
553 Select the source override file to use with the <literal>sources</literal> command.
554 Configuration Item: <literal>APT::FTPArchive::SourceOverride</literal>.</para></listitem>
555 </varlistentry>
556
557 <varlistentry><term><option>--readonly</option></term>
558 <listitem><para>
559 Make the caching databases read only.
560 Configuration Item: <literal>APT::FTPArchive::ReadOnlyDB</literal>.</para></listitem>
561 </varlistentry>
562
563 <varlistentry><term><option>-a</option></term><term><option>--arch</option></term>
564 <listitem><para>Accept in the <literal>packages</literal> and <literal>contents</literal>
565 commands only package files matching <literal>*_arch.deb</literal> or
566 <literal>*_all.deb</literal> instead of all package files in the given path.
567 Configuration Item: <literal>APT::FTPArchive::Architecture</literal>.</para></listitem>
568 </varlistentry>
569
570 <varlistentry><term><option>APT::FTPArchive::AlwaysStat</option></term>
571 <listitem><para>
572 &apt-ftparchive; caches as much as possible of metadata in a cachedb. If packages
573 are recompiled and/or republished with the same version again, this will lead to problems
574 as the now outdated cached metadata like size and checksums will be used. With this option
575 enabled this will no longer happen as it will be checked if the file was changed.
576 Note that this option is set to "<literal>false</literal>" by default as it is not recommend
577 to upload multiply versions/builds of a package with the same versionnumber, so in theory
578 nobody will have these problems and therefore all these extra checks are useless.
579 </para></listitem>
580 </varlistentry>
581
582 <varlistentry><term><option>APT::FTPArchive::LongDescription</option></term>
583 <listitem><para>
584 This configuration option defaults to "<literal>true</literal>" and should only be set to
585 <literal>"false"</literal> if the Archive generated with &apt-ftparchive; also provides
586 <filename>Translation</filename> files. Note that the <filename>Translation-en</filename>
587 master file can only be created in the generate command.
588 </para></listitem>
589 </varlistentry>
590
591 &apt-commonoptions;
592
593 </variablelist>
594 </refsect1>
595
596 <refsect1><title>Examples</title>
597
598 <para>To create a compressed Packages file for a directory containing
599 binary packages (.deb):
600
601 <programlisting>
602 <command>apt-ftparchive</command> packages <replaceable>directory</replaceable> | <command>gzip</command> > <filename>Packages.gz</filename>
603 </programlisting></para>
604
605 </refsect1>
606
607 <refsect1><title>See Also</title>
608 <para>&apt-conf;</para>
609 </refsect1>
610
611 <refsect1><title>Diagnostics</title>
612 <para><command>apt-ftparchive</command> returns zero on normal operation, decimal 100 on error.</para>
613 </refsect1>
614
615 &manbugs;
616
617 </refentry>