* apt-pkg/deb/debrecords.cc:
[ntk/apt.git] / doc / apt-ftparchive.1.xml
1 <?xml version="1.0" encoding="utf-8" standalone="no"?>
2 <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
3 "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" [
4
5 <!ENTITY % aptent SYSTEM "apt.ent">
6 %aptent;
7
8 ]>
9
10 <refentry>
11
12 <refentryinfo>
13 &apt-author.jgunthorpe;
14 &apt-author.team;
15 &apt-email;
16 &apt-product;
17 <!-- The last update date -->
18 <date>17 August 2009</date>
19 </refentryinfo>
20
21 <refmeta>
22 <refentrytitle>apt-ftparchive</refentrytitle>
23 <manvolnum>1</manvolnum>
24 <refmiscinfo class="manual">APT</refmiscinfo>
25 </refmeta>
26
27 <!-- Man page title -->
28 <refnamediv>
29 <refname>apt-ftparchive</refname>
30 <refpurpose>Utility to generate index files</refpurpose>
31 </refnamediv>
32
33 <!-- Arguments -->
34 <refsynopsisdiv>
35 <cmdsynopsis>
36 <command>apt-ftparchive</command>
37 <arg><option>-hvdsq</option></arg>
38 <arg><option>--md5</option></arg>
39 <arg><option>--delink</option></arg>
40 <arg><option>--readonly</option></arg>
41 <arg><option>--contents</option></arg>
42 <arg><option>--arch <replaceable>architecture</replaceable></option></arg>
43 <arg><option>-o <replaceable>config</replaceable>=<replaceable>string</replaceable></option></arg>
44 <arg><option>-c=<replaceable>file</replaceable></option></arg>
45 <group choice="req">
46 <arg>packages<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
47 <arg>sources<arg choice="plain" rep="repeat"><replaceable>path</replaceable></arg><arg><replaceable>override</replaceable><arg><replaceable>pathprefix</replaceable></arg></arg></arg>
48 <arg>contents <arg choice="plain"><replaceable>path</replaceable></arg></arg>
49 <arg>release <arg choice="plain"><replaceable>path</replaceable></arg></arg>
50 <arg>generate <arg choice="plain"><replaceable>config-file</replaceable></arg> <arg choice="plain" rep="repeat"><replaceable>section</replaceable></arg></arg>
51 <arg>clean <arg choice="plain"><replaceable>config-file</replaceable></arg></arg>
52 </group>
53 </cmdsynopsis>
54 </refsynopsisdiv>
55
56 <refsect1><title>Description</title>
57 <para><command>apt-ftparchive</command> is the command line tool that generates the index
58 files that APT uses to access a distribution source. The index files should
59 be generated on the origin site based on the content of that site.</para>
60
61 <para><command>apt-ftparchive</command> is a superset of the &dpkg-scanpackages; program,
62 incorporating its entire functionality via the <literal>packages</literal> command.
63 It also contains a contents file generator, <literal>contents</literal>, and an
64 elaborate means to 'script' the generation process for a complete
65 archive.</para>
66
67 <para>Internally <command>apt-ftparchive</command> can make use of binary databases to
68 cache the contents of a .deb file and it does not rely on any external
69 programs aside from &gzip;. When doing a full generate it automatically
70 performs file-change checks and builds the desired compressed output files.</para>
71
72 <para>Unless the <option>-h</option>, or <option>--help</option> option is given one of the
73 commands below must be present.</para>
74
75 <variablelist>
76 <varlistentry><term>packages</term>
77 <listitem><para>
78 The packages command generates a package file from a directory tree. It
79 takes the given directory and recursively searches it for .deb files,
80 emitting a package record to stdout for each. This command is
81 approximately equivalent to &dpkg-scanpackages;.</para>
82
83 <para>The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
84 </varlistentry>
85
86 <varlistentry><term>sources</term>
87 <listitem><para>
88 The <literal>sources</literal> command generates a source index file from a directory tree.
89 It takes the given directory and recursively searches it for .dsc files,
90 emitting a source record to stdout for each. This command is approximately
91 equivalent to &dpkg-scansources;.</para>
92 <para>
93 If an override file is specified then a source override file will be
94 looked for with an extension of .src. The --source-override option can be
95 used to change the source override file that will be used.</para></listitem>
96 </varlistentry>
97
98 <varlistentry><term>contents</term>
99 <listitem><para>
100 The <literal>contents</literal> command generates a contents file from a directory tree. It
101 takes the given directory and recursively searches it for .deb files,
102 and reads the file list from each file. It then sorts and writes to stdout
103 the list of files matched to packages. Directories are not written to
104 the output. If multiple packages own the same file then each package is
105 separated by a comma in the output.</para>
106 <para>
107 The option <option>--db</option> can be used to specify a binary caching DB.</para></listitem>
108 </varlistentry>
109
110 <varlistentry><term>release</term>
111 <listitem><para>
112 The <literal>release</literal> command generates a Release file from a
113 directory tree. It recursively searches the given directory for
114 Packages, Packages.gz, Packages.bz2, Sources, Sources.gz,
115 Sources.bz2, Release and md5sum.txt files. It then writes to
116 stdout a Release file containing an MD5 digest and SHA1 digest
117 for each file.</para>
118 <para>
119 Values for the additional metadata fields in the Release file are
120 taken from the corresponding variables under
121 <literal>APT::FTPArchive::Release</literal>,
122 e.g. <literal>APT::FTPArchive::Release::Origin</literal>. The supported fields
123 are: <literal>Origin</literal>, <literal>Label</literal>, <literal>Suite</literal>,
124 <literal>Version</literal>, <literal>Codename</literal>, <literal>Date</literal>,
125 <literal>Architectures</literal>, <literal>Components</literal>, <literal>Description</literal>.</para></listitem>
126
127 </varlistentry>
128
129 <varlistentry><term>generate</term>
130 <listitem><para>
131 The <literal>generate</literal> command is designed to be runnable from a cron script and
132 builds indexes according to the given config file. The config language
133 provides a flexible means of specifying which index files are built from
134 which directories, as well as providing a simple means of maintaining the
135 required settings.</para></listitem>
136 </varlistentry>
137
138 <varlistentry><term>clean</term>
139 <listitem><para>
140 The <literal>clean</literal> command tidies the databases used by the given
141 configuration file by removing any records that are no longer necessary.</para></listitem>
142 </varlistentry>
143 </variablelist>
144 </refsect1>
145
146 <refsect1><title>The Generate Configuration</title>
147 <para>
148 The <literal>generate</literal> command uses a configuration file to describe the
149 archives that are going to be generated. It follows the typical ISC
150 configuration format as seen in ISC tools like bind 8 and dhcpd.
151 &apt-conf; contains a description of the syntax. Note that the generate
152 configuration is parsed in sectional manner, but &apt-conf; is parsed in a
153 tree manner. This only effects how the scope tag is handled.</para>
154
155 <para>
156 The generate configuration has 4 separate sections, each described below.</para>
157
158 <refsect2><title>Dir Section</title>
159 <para>
160 The <literal>Dir</literal> section defines the standard directories needed to
161 locate the files required during the generation process. These
162 directories are prepended certain relative paths defined in later
163 sections to produce a complete an absolute path.</para>
164 <variablelist>
165 <varlistentry><term>ArchiveDir</term>
166 <listitem><para>
167 Specifies the root of the FTP archive, in a standard
168 Debian configuration this is the directory that contains the
169 <filename>ls-LR</filename> and dist nodes.</para></listitem>
170 </varlistentry>
171
172 <varlistentry><term>OverrideDir</term>
173 <listitem><para>
174 Specifies the location of the override files.</para></listitem>
175 </varlistentry>
176
177 <varlistentry><term>CacheDir</term>
178 <listitem><para>
179 Specifies the location of the cache files</para></listitem>
180 </varlistentry>
181
182 <varlistentry><term>FileListDir</term>
183 <listitem><para>
184 Specifies the location of the file list files,
185 if the <literal>FileList</literal> setting is used below.</para></listitem>
186 </varlistentry>
187 </variablelist>
188 </refsect2>
189
190 <refsect2><title>Default Section</title>
191 <para>
192 The <literal>Default</literal> section specifies default values, and settings
193 that control the operation of the generator. Other sections may override
194 these defaults with a per-section setting.</para>
195 <variablelist>
196 <varlistentry><term>Packages::Compress</term>
197 <listitem><para>
198 Sets the default compression schemes to use
199 for the Package index files. It is a string that contains a space
200 separated list of at least one of: '.' (no compression), 'gzip' and
201 'bzip2'. The default for all compression schemes is '. gzip'.</para></listitem>
202 </varlistentry>
203
204 <varlistentry><term>Packages::Extensions</term>
205 <listitem><para>
206 Sets the default list of file extensions that are package files.
207 This defaults to '.deb'.</para></listitem>
208 </varlistentry>
209
210 <varlistentry><term>Sources::Compress</term>
211 <listitem><para>
212 This is similar to <literal>Packages::Compress</literal>
213 except that it controls the compression for the Sources files.</para></listitem>
214 </varlistentry>
215
216 <varlistentry><term>Sources::Extensions</term>
217 <listitem><para>
218 Sets the default list of file extensions that are source files.
219 This defaults to '.dsc'.</para></listitem>
220 </varlistentry>
221
222 <varlistentry><term>Contents::Compress</term>
223 <listitem><para>
224 This is similar to <literal>Packages::Compress</literal>
225 except that it controls the compression for the Contents files.</para></listitem>
226 </varlistentry>
227
228 <varlistentry><term>DeLinkLimit</term>
229 <listitem><para>
230 Specifies the number of kilobytes to delink (and
231 replace with hard links) per run. This is used in conjunction with the
232 per-section <literal>External-Links</literal> setting.</para></listitem>
233 </varlistentry>
234
235 <varlistentry><term>FileMode</term>
236 <listitem><para>
237 Specifies the mode of all created index files. It
238 defaults to 0644. All index files are set to this mode with no regard
239 to the umask.</para></listitem>
240 </varlistentry>
241 </variablelist>
242 </refsect2>
243
244 <refsect2><title>TreeDefault Section</title>
245 <para>
246 Sets defaults specific to <literal>Tree</literal> sections. All of these
247 variables are substitution variables and have the strings $(DIST),
248 $(SECTION) and $(ARCH) replaced with their respective values.</para>
249
250 <variablelist>
251 <varlistentry><term>MaxContentsChange</term>
252 <listitem><para>
253 Sets the number of kilobytes of contents
254 files that are generated each day. The contents files are round-robined
255 so that over several days they will all be rebuilt.</para></listitem>
256 </varlistentry>
257
258 <varlistentry><term>ContentsAge</term>
259 <listitem><para>
260 Controls the number of days a contents file is allowed
261 to be checked without changing. If this limit is passed the mtime of the
262 contents file is updated. This case can occur if the package file is
263 changed in such a way that does not result in a new contents file
264 [override edit for instance]. A hold off is allowed in hopes that new
265 .debs will be installed, requiring a new file anyhow. The default is 10,
266 the units are in days.</para></listitem>
267 </varlistentry>
268
269 <varlistentry><term>Directory</term>
270 <listitem><para>
271 Sets the top of the .deb directory tree. Defaults to
272 <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/</filename></para></listitem>
273 </varlistentry>
274
275 <varlistentry><term>SrcDirectory</term>
276 <listitem><para>
277 Sets the top of the source package directory tree. Defaults to
278 <filename>$(DIST)/$(SECTION)/source/</filename></para></listitem>
279 </varlistentry>
280
281 <varlistentry><term>Packages</term>
282 <listitem><para>
283 Sets the output Packages file. Defaults to
284 <filename>$(DIST)/$(SECTION)/binary-$(ARCH)/Packages</filename></para></listitem>
285 </varlistentry>
286
287 <varlistentry><term>Sources</term>
288 <listitem><para>
289 Sets the output Sources file. Defaults to
290 <filename>$(DIST)/$(SECTION)/source/Sources</filename></para></listitem>
291 </varlistentry>
292
293 <varlistentry><term>InternalPrefix</term>
294 <listitem><para>
295 Sets the path prefix that causes a symlink to be
296 considered an internal link instead of an external link. Defaults to
297 <filename>$(DIST)/$(SECTION)/</filename></para></listitem>
298 </varlistentry>
299
300 <varlistentry><term>Contents</term>
301 <listitem><para>
302 Sets the output Contents file. Defaults to
303 <filename>$(DIST)/Contents-$(ARCH)</filename>. If this setting causes multiple
304 Packages files to map onto a single Contents file (such as the default)
305 then <command>apt-ftparchive</command> will integrate those package files
306 together automatically.</para></listitem>
307 </varlistentry>
308
309 <varlistentry><term>Contents::Header</term>
310 <listitem><para>
311 Sets header file to prepend to the contents output.</para></listitem>
312 </varlistentry>
313
314 <varlistentry><term>BinCacheDB</term>
315 <listitem><para>
316 Sets the binary cache database to use for this
317 section. Multiple sections can share the same database.</para></listitem>
318 </varlistentry>
319
320 <varlistentry><term>FileList</term>
321 <listitem><para>
322 Specifies that instead of walking the directory tree,
323 <command>apt-ftparchive</command> should read the list of files from the given
324 file. Relative files names are prefixed with the archive directory.</para></listitem>
325 </varlistentry>
326
327 <varlistentry><term>SourceFileList</term>
328 <listitem><para>
329 Specifies that instead of walking the directory tree,
330 <command>apt-ftparchive</command> should read the list of files from the given
331 file. Relative files names are prefixed with the archive directory.
332 This is used when processing source indexes.</para></listitem>
333 </varlistentry>
334 </variablelist>
335 </refsect2>
336
337 <refsect2><title>Tree Section</title>
338 <para>
339 The <literal>Tree</literal> section defines a standard Debian file tree which
340 consists of a base directory, then multiple sections in that base
341 directory and finally multiple Architectures in each section. The exact
342 pathing used is defined by the <literal>Directory</literal> substitution variable.</para>
343 <para>
344 The <literal>Tree</literal> section takes a scope tag which sets the
345 <literal>$(DIST)</literal> variable and defines the root of the tree
346 (the path is prefixed by <literal>ArchiveDir</literal>).
347 Typically this is a setting such as <filename>dists/woody</filename>.</para>
348 <para>
349 All of the settings defined in the <literal>TreeDefault</literal> section can be
350 use in a <literal>Tree</literal> section as well as three new variables.</para>
351 <para>
352 When processing a <literal>Tree</literal> section <command>apt-ftparchive</command>
353 performs an operation similar to:
354 <programlisting>
355 for i in Sections do
356 for j in Architectures do
357 Generate for DIST=scope SECTION=i ARCH=j
358 </programlisting></para>
359
360 <variablelist>
361 <varlistentry><term>Sections</term>
362 <listitem><para>
363 This is a space separated list of sections which appear
364 under the distribution, typically this is something like
365 <literal>main contrib non-free</literal></para></listitem>
366 </varlistentry>
367
368 <varlistentry><term>Architectures</term>
369 <listitem><para>
370 This is a space separated list of all the
371 architectures that appear under search section. The special architecture
372 'source' is used to indicate that this tree has a source archive.</para></listitem>
373 </varlistentry>
374
375 <varlistentry><term>BinOverride</term>
376 <listitem><para>
377 Sets the binary override file. The override file
378 contains section, priority and maintainer address information.</para></listitem>
379 </varlistentry>
380
381 <varlistentry><term>SrcOverride</term>
382 <listitem><para>
383 Sets the source override file. The override file
384 contains section information.</para></listitem>
385 </varlistentry>
386
387 <varlistentry><term>ExtraOverride</term>
388 <listitem><para>
389 Sets the binary extra override file.</para></listitem>
390 </varlistentry>
391
392 <varlistentry><term>SrcExtraOverride</term>
393 <listitem><para>
394 Sets the source extra override file.</para></listitem>
395 </varlistentry>
396 </variablelist>
397 </refsect2>
398
399 <refsect2><title>BinDirectory Section</title>
400 <para>
401 The <literal>bindirectory</literal> section defines a binary directory tree
402 with no special structure. The scope tag specifies the location of
403 the binary directory and the settings are similar to the <literal>Tree</literal>
404 section with no substitution variables or
405 <literal>Section</literal><literal>Architecture</literal> settings.</para>
406 <variablelist>
407 <varlistentry><term>Packages</term>
408 <listitem><para>
409 Sets the Packages file output.</para></listitem>
410 </varlistentry>
411
412 <varlistentry><term>Sources</term>
413 <listitem><para>
414 Sets the Sources file output. At least one of
415 <literal>Packages</literal> or <literal>Sources</literal> is required.</para></listitem>
416 </varlistentry>
417
418 <varlistentry><term>Contents</term>
419 <listitem><para>
420 Sets the Contents file output. (optional)</para></listitem>
421 </varlistentry>
422
423 <varlistentry><term>BinOverride</term>
424 <listitem><para>
425 Sets the binary override file.</para></listitem>
426 </varlistentry>
427
428 <varlistentry><term>SrcOverride</term>
429 <listitem><para>
430 Sets the source override file.</para></listitem>
431 </varlistentry>
432
433 <varlistentry><term>ExtraOverride</term>
434 <listitem><para>
435 Sets the binary extra override file.</para></listitem>
436 </varlistentry>
437
438 <varlistentry><term>SrcExtraOverride</term>
439 <listitem><para>
440 Sets the source extra override file.</para></listitem>
441 </varlistentry>
442
443 <varlistentry><term>BinCacheDB</term>
444 <listitem><para>
445 Sets the cache DB.</para></listitem>
446 </varlistentry>
447
448 <varlistentry><term>PathPrefix</term>
449 <listitem><para>
450 Appends a path to all the output paths.</para></listitem>
451 </varlistentry>
452
453 <varlistentry><term>FileList, SourceFileList</term>
454 <listitem><para>
455 Specifies the file list file.</para></listitem>
456 </varlistentry>
457 </variablelist>
458 </refsect2>
459 </refsect1>
460
461
462 <refsect1><title>The Binary Override File</title>
463 <para>The binary override file is fully compatible with &dpkg-scanpackages;. It
464 contains 4 fields separated by spaces. The first field is the package name,
465 the second is the priority to force that package to, the third is the
466 the section to force that package to and the final field is the maintainer
467 permutation field.</para>
468 <para>The general form of the maintainer field is:
469 <literallayout>old [// oldn]* => new</literallayout>
470 or simply,
471 <literallayout>new</literallayout>
472 The first form allows a double-slash separated list of old email addresses
473 to be specified. If any of those are found then new is substituted for the
474 maintainer field. The second form unconditionally substitutes the
475 maintainer field.</para>
476 </refsect1>
477
478
479 <refsect1><title>The Source Override File</title>
480 <para>
481 The source override file is fully compatible with &dpkg-scansources;. It
482 contains 2 fields separated by spaces. The first fields is the source
483 package name, the second is the section to assign it.</para>
484 </refsect1>
485
486 <refsect1><title>The Extra Override File</title>
487 <para>
488 The extra override file allows any arbitrary tag to be added or replaced
489 in the output. It has 3 columns, the first is the package, the second is
490 the tag and the remainder of the line is the new value.</para>
491 </refsect1>
492
493 <refsect1><title>options</title>
494 &apt-cmdblurb;
495
496 <variablelist>
497 <varlistentry><term><option>--md5</option></term>
498 <listitem><para>
499 Generate MD5 sums. This defaults to on, when turned off the generated
500 index files will not have MD5Sum fields where possible.
501 Configuration Item: <literal>APT::FTPArchive::MD5</literal></para></listitem>
502 </varlistentry>
503
504 <varlistentry><term><option>-d</option></term><term><option>--db</option></term>
505 <listitem><para>
506 Use a binary caching DB. This has no effect on the generate command.
507 Configuration Item: <literal>APT::FTPArchive::DB</literal>.</para></listitem>
508 </varlistentry>
509
510 <varlistentry><term><option>-q</option></term><term><option>--quiet</option></term>
511 <listitem><para>
512 Quiet; produces output suitable for logging, omitting progress indicators.
513 More q's will produce more quiet up to a maximum of 2. You can also use
514 <option>-q=#</option> to set the quiet level, overriding the configuration file.
515 Configuration Item: <literal>quiet</literal>.</para></listitem>
516 </varlistentry>
517
518 <varlistentry><term><option>--delink</option></term>
519 <listitem><para>
520 Perform Delinking. If the <literal>External-Links</literal> setting is used then
521 this option actually enables delinking of the files. It defaults to on and
522 can be turned off with <option>--no-delink</option>.
523 Configuration Item: <literal>APT::FTPArchive::DeLinkAct</literal>.</para></listitem>
524 </varlistentry>
525
526 <varlistentry><term><option>--contents</option></term>
527 <listitem><para>
528 Perform contents generation. When this option is set and package indexes
529 are being generated with a cache DB then the file listing will also be
530 extracted and stored in the DB for later use. When using the generate
531 command this option also allows the creation of any Contents files. The
532 default is on.
533 Configuration Item: <literal>APT::FTPArchive::Contents</literal>.</para></listitem>
534 </varlistentry>
535
536 <varlistentry><term><option>-s</option></term><term><option>--source-override</option></term>
537 <listitem><para>
538 Select the source override file to use with the <literal>sources</literal> command.
539 Configuration Item: <literal>APT::FTPArchive::SourceOverride</literal>.</para></listitem>
540 </varlistentry>
541
542 <varlistentry><term><option>--readonly</option></term>
543 <listitem><para>
544 Make the caching databases read only.
545 Configuration Item: <literal>APT::FTPArchive::ReadOnlyDB</literal>.</para></listitem>
546 </varlistentry>
547
548 <varlistentry><term><option>-a</option></term><term><option>--arch</option></term>
549 <listitem><para>Accept in the <literal>packages</literal> and <literal>contents</literal>
550 commands only package files matching <literal>*_arch.deb</literal> or
551 <literal>*_all.deb</literal> instead of all package files in the given path.
552 Configuration Item: <literal>APT::FTPArchive::Architecture</literal>.</para></listitem>
553 </varlistentry>
554
555 <varlistentry><term><option>APT::FTPArchive::AlwaysStat</option></term>
556 <listitem><para>
557 &apt-ftparchive; caches as much as possible of metadata in a cachedb. If packages
558 are recompiled and/or republished with the same version again, this will lead to problems
559 as the now outdated cached metadata like size and checksums will be used. With this option
560 enabled this will no longer happen as it will be checked if the file was changed.
561 Note that this option is set to "<literal>false</literal>" by default as it is not recommend
562 to upload multiply versions/builds of a package with the same versionnumber, so in theory
563 nobody will have these problems and therefore all these extra checks are useless.
564 </para></listitem>
565 </varlistentry>
566
567 <varlistentry><term><option>APT::FTPArchive::LongDescription</option></term>
568 <listitem><para>
569 This configuration option defaults to "<literal>true</literal>" and should only be set to
570 <literal>"false"</literal> if the Archive generated with &apt-ftparchive; also provides
571 <filename>Translation</filename> files. Note that it is currently not possible to create these
572 files with <command>apt-ftparchive</command>.
573 </para></listitem>
574 </varlistentry>
575
576 &apt-commonoptions;
577
578 </variablelist>
579 </refsect1>
580
581 <refsect1><title>Examples</title>
582
583 <para>To create a compressed Packages file for a directory containing
584 binary packages (.deb):
585
586 <programlisting>
587 <command>apt-ftparchive</command> packages <replaceable>directory</replaceable> | <command>gzip</command> > <filename>Packages.gz</filename>
588 </programlisting></para>
589
590 </refsect1>
591
592 <refsect1><title>See Also</title>
593 <para>&apt-conf;</para>
594 </refsect1>
595
596 <refsect1><title>Diagnostics</title>
597 <para><command>apt-ftparchive</command> returns zero on normal operation, decimal 100 on error.</para>
598 </refsect1>
599
600 &manbugs;
601
602 </refentry>