September 2011 TriLUG meeting: Making Debian Packages

Motivation: Why bother with Debian packages

  • 8 word version: Gives you a simple way to manage dependencies
  • 4 word version: Integrates software with system
  • 2 word version: Easier Ops

Step 1: Setting up

$ sudo apt-get install build-essential devscripts dh-make`

Configure dh_make: add your name and email to your .bashrc:

echo 'DEBEMAIL="example@trilug.org"' >> ~/.bashrc
echo 'DEBFULLNAME="Example Name"' >> ~/.bashrc
echo 'export DEBEMAIL DEBFULLNAME' >> ~/.bashrc
source ~/.bashrc

Step 2: Preparing your codebase

The structure of your source tree should look like this:

trilug-simple/
doc/
src/
Makefile

Note: All the source code is located under the src/ directory, and not in the top level directory. doc/ may contain anything from README's to sample config files. The Makefile should include the build and install targets so that dpkg can use it to create the Debian package.

Note: Makefile is not the only way. We will talk about this later.

Step 3: Debianizing your codebase

"Debianizing" your codebase consists of adding a debian/ directory to your project, which will contain the configuration files necessary to create a Debian package. The dh-make package provides a script that will do most of this work for us:

mkdir -p ~/buildroot/ && cd ~/buildroot/
git clone git://github.com/ipartola/trilug-debianization.git ./
cd trilug-simple-1.0

We are now ready to add the debian/ directory. We use the dh_make command to do this. We also give it two hints: first we are telling it that we are going to create a single package. We are also going to tell it that we want to create a "native" package; that is we are not packaging a piece of software that was written by someone else.

$ dh_make --single --native 
Maintainer name : Example name
Email-Address   : example@trilug.org
Date            : Tue, 25 Jan 2011 11:16:16 -0500
Package Name    : trilug-simple
Version         : 1.0
License         : gpl
Using dpatch    : no
Type of Package : Single
Hit <enter> to confirm:

$ cd debian/

We are now ready to begin customizing the configuration files in the debian/ directory. The first step is to remove the templates we are not using.

$ rm emacsen* manpage* README* watch.ex trilug-simple.doc-base.EX menu.ex

On Debian 5 remove init.d and use init.d.lsb.ex instead:

$ rm init.d.ex && mv init.d.lsb.ex init.d

Now let's walk through the rest of the files and what they should contain:

changelog

The changelog for the project. This file is important: the version of the package will be derived from the top line of this file. Here is an example:

trilug-simple (1.1) unstable; urgency=low

  * Major feature addded
  - Minor feature or bug fix

-- Example Name <example@trilug.org>  Mon, 21 Aug 2011 21:07:16 -0400

trilug-simple (1.0) unstable; urgency=low

  * Initial Release.

-- Example Name <example@trilug.org>  Tue, 25 Jan 2011 21:06:35 -0400

The version does not have to be in the x.x format. In fact, it is a good idea to use the revision number from your version control system as the package version instead. You should probably also use the commit log from your VCS to automatically populate this file, instead of modifying it directly (that way you would be sure to fill it out).

compat

This file contains the compatibility level with debhelper (which we use indirectly). This should simply be set to 7.

control

The control file contains important information about your package such as its name, dependencies (build-time and run-time), and architecture. Here is an example:

Source: trilug-simple
Section: web
Priority: extra
Maintainer: Example Name <example@trilug.org>
Build-Depends: debhelper (>= 7), libcurl4-gnutls-dev
Standards-Version: 3.8.0

Package: trilug-simple
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}, libcurl3
Description: sample trilug package that does much of nothing
 The simple package is a good way to learn how to create simple debian 
 packages. It is meant as an example, but is by no means exhaustive.

For sections, see: http://packages.debian.org/stable/

One important gotcha here is the Architecture clause. It can be any, all or a specific arch, such as amd64. Generally, you want to use either any or all, where any means that the final package will be compiled for a particular architecture, and all means that the same package will run on all architectures. Generally, anything that's compiled will use any while software written in interpreted languages will have Architecture set to all.

Dependencies, recommendations, suggestions and conflicts may be listed here as well. Note that you can even specify version: debhelper (>= 7). You can also use the Depends clause for your own packages. For example you could have trilug-simple depend on trilug-lib, and then when you run $ apt-get install trilug-simple, trilug-lib will be installed automatically.

copyright

This file is not particularly important, but building the package will not work without it. Here is the sample content for a commerical product (I am not a lawyer, this is not legal advice):

Format: http://dep.debian.net/deps/dep5
Upstream-Name: trilug-simple

Files: *
Copyright: 2011 Example Name <example@trilug.org>
License: Proprietary

License:

ALL RIGHTS RESERVED

(trilug-simple).cron.d:

This is an optional, but often very useful file which will let you define cron jobs that your package will need to run. It will get installed as /etc/cron.d/trilug-simple. If you are going to use this file, make sure to list cron as one of your dependencies.

docs

This file lists files/directories that will be installed into /usr/share/doc/trilug-simple/. For example:

doc/README
doc/LICENSE

init.d

This optional file gets installed as /etc/init.d/trilug-simple. This is the standard mechanism for starting up processes on system boot. This file is rather long, and in most cases the sample file is a very good start.

Note: most daemons do not need to run as root, but that is the default in the sample init.d file. To make sure the process does not run as root you will need to:

  • Make sure a package-specific user exists (we will do this below in preinst)
  • Make sure the package-specific user has write access to all the necessary directories, such as /var/logs/trilug-simple, /var/run/trilug-simple, /var/lib/trilug-simple, etc. This is best done in the start function of the init script.
  • Make sure to tell start-stop-daemon command that we want to run as a specific user using the --chuid parameter.

preinst, postinst, prerm and postrm

These files will be run before and after the package is installed and removed, respectively. If one of these files is not present, a default one will be generated. What each file should include:

  • preinst - creating package-specific users
  • postinst - installing the init script to the appropriate runlevels and starting up the daemons
  • prerm - stopping the daemons
  • postrm - removing the package-specific users, removing init script from all runlevels

rules

This file is an executable Makefile (separate from the one that comes with your codebase). It allows you to have a great deal of customization for how your package is built (see Debian New Maintainers' Guide for details). However, there is a shortcut:

#!/usr/bin/make -f
# -*- makefile -*-
# Sample debian/rules that uses debhelper.
# This file was originally written by Joey Hess and Craig Small.
# As a special exception, when this file is copied by dh-make into a
# dh-make output file, you may use that output file without restriction.
# This special exception was added by Craig Small in version 0.37 of dh-make.

# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1

%:
        dh $@

The above file will run all the default rules, which will use our trilug-simple/Makefile to do the building, installation, cleanup, etc.

Step 3: Building the package

Building the package is a simple matter of using the debuild(1) command:

$ debuild -us -uc

This command produces a lot of output that includes the result of building your source and the output of Lintian.

Lintian is a program that finds packaging errors in your Debian packages. It can be very helpful in debugging errors in your packages, and can help catch some bugs that you would not normally see. It produces warnings like: possibly-insecure-handling-of-tmp-files-in-maintainer-script postrm:25. You can search the Lintian website for the meaning of these messages, which should help you fix them.

Note: If you want to remove all the temporary files that debuild created run: $ debuild clean

Lyrical aside: Python packages

A lot of software is written in languages other than C. Python is one popular choice. Debian supports Python-based packages through the python-support framework. There are a few things that are different:

  • You can use a setup.py file (using setuptools) instead of a Makefile. The magic shortcut in the debian/rules file will automatically detect this.
  • Starting a Python-based daemon is different, since the name of the executable is going to be something like /usr/bin/python instead of the name you gave it. Adjust your init.d file accordingly.
  • Python is a semi-compiled language. Installing the source does not mean that you can import it. Test using "import trilug.hello" from the Python interpreter. To make sure this works run update-python-modules trilug-python in the postinst script and before you start the daemon in the init.d file.
  • You should include python-support and python as dependencies in your control file.

Step 4: Setting up a repository

After your software is built, you will want to set up an APT repository so all of your servers can install the packages via the normal apt-get/aptitude command.

An APT repository is simply a web service which hosts the .deb files, a GPG signature and a package list file. We want this repository to be secure from unauthorized access if we host it on the internet. To do this, we will use HTTP Basic Auth and only allow access to our repository over HTTPS.

Note: for the sake of brevity, we are skipping the SSL/TLS setup.

First let's get nginx set up:

$ sudo apt-get install nginx apache2-utils apt-utils

Update the /etc/nginx/sites-available/default:

server {
    listen   [::]:80;
    server_name localhost;

    location / {
        root /var/www;
        autoindex on;
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/apt-htpasswd;
    }
}

Add a Basic Auth user, using trilug as password:

$ sudo htpasswd -c /etc/nginx/apt-htpasswd testuser

Create a location where the repository will live:

sudo /etc/init.d/nginx restart
mkdir -p /var/www/testing
mkdir -p /var/www/stable

We also need to create an /etc/apt/apt-release.conf file:

APT::FTPArchive::Release::Origin "trilug";
APT::FTPArchive::Release::Label "trilug";
APT::FTPArchive::Release::Suite "custom";
APT::FTPArchive::Release::Codename "custom";
APT::FTPArchive::Release::Architectures "amd64 i386 source any all";
APT::FTPArchive::Release::Components "main";
APT::FTPArchive::Release::Description "Custom debian packages for TriLUG.";

Create an exectuable file /usr/local/bin/update-apt-repo:

#!/bin/sh

if [ $# -lt 1 ]; then
    echo "Usage: update-apt-repo path [gpg_key_id]"
    exit -1
fi

REPO_PATH=$1
KEY=$2

if [ -n $KEY ]; then
    KEY_COMMAND="--default-key $KEY"
fi

cd $REPO_PATH;

apt-ftparchive packages . > Packages
cat Packages | gzip -c9 > Packages.gz
apt-ftparchive -c /etc/apt/apt-release.conf release . > ./Release
rm -rf Release.gpg
gpg --armor --detach-sign --sign $KEY_COMMAND --output Release.gpg Release

This file will be run with two parameters: the path to the APT repository and the name of GPG key which will be used to sign the Release file.

Let's add the user and create the GPG key:

$ sudo useradd -m repomaster
$ sudo chown -R repomaster:repomaster /var/www
$ sudo su repomaster
$ gpg --gen-key

Make sure to use a blank passphrase for the last step. After answering all the questions GPG asks, figure out what key to use:

$ gpg --list-keys
/home/repomaster/.gnupg/pubring.gpg
-----------------------------------
pub   2048R/29DDDFA8 2011-09-05
uid                  Repo Master <repomaster@trilug.org>

The key name is 29DDDFA8. Now let's run the update-apt-repo script we wrote to test it:

$ cp /home/YOURUSER/buildroot/trilug-simple/*.deb /var/www/testing/
$ update-apt-repo /var/www/testing 29DDDFA8
$ ls /var/www/testing
Packages  Packages.gz  Release  Release.gpg  trilug-simple_1.0_amd64.deb`</pre>

If you look in the Packages file, you will find that our package is listed there.

We also need to export the GPG public key so that apt-get/aptitude know to trust our repository lists:

$ gpg --export > /var/www/public.key

As a final step, let's make sure that the update-apt-repo script runs automatically. To do so, simply add the following to your /etc/crontab file:

* * * * * repomaster /usr/bin/update-apt-repo /var/www/apt/stable 29DDDFA8
* * * * * repomaster /usr/bin/update-apt-repo /var/www/apt/testing 29DDDFA8

Step 5: Adding repositories to your machines and using them

HTTPS support is not available by default on older Debian distributions. To fix this we run:

$ sudo apt-get install apt-transport-https

Now, let's add our repository to /etc/apt/sources.list file:

deb http://testuser:trilug@localhost:80/testing /

We also need to tell APT that it can trust this repository. This is done by importing the repomaster's public key:

$ curl http://testuser:trilug@localhost/public.key | sudo apt-key add -

Note: if you are going to use a self-signed SSL certificate, make sure to add it to the list of trusted certificates on all your machines.

Now, we are ready to use the repository:

$ sudo apt-get update
$ apt-cache search trilug
trilug-simple - TriLUG greeter
$ sudo apt-get install trilug-simple

Lyrical aside #2: Your build process

The process of "debianizing" your software is a one-time investment. However, you will end up building it frequently. Things you may want to consider to make it easier on yourself:

  • If you are using a compiled language and the result is arch-dependent, set up a different build machine for each architecture (e.g.: amd64, i386, etc).
  • Write a build script that pulls changelog data from your VCS, updates debian/changelog, runs debuild on all your build hosts, uploads to your repo.
  • Have your build script run automated tests before debuild is run.
  • Have your VCS system trigger a build each time a developer creates a new tag.

Useful commands and tools

Commands you may find useful:

  • $ dpkg -l - List installed packages
  • $ dpkg -L trilug-simple - List files installed by a package
  • $ dpkg -S /usr/bin/hello - Find the package the file belongs to (great for figuring out your dependencies)
  • $ apt-cache policy trilug-simple - View available versions of packages
  • $ apt-get install trilug-simple=1.0 - Install a specific version of the package

Useful tools:

  • pbuilder - an automatic Debian Package Building system for personal development
  • reprepro - a tool to run local repositories of Debian packages

Further reading: External resources