Creating a python package

Posted on Fri 03 June 2016 in python

Python packaging

Over the last few years the process of creating a Python package has evolved. However, for historical reasons a lot of Python packages are still using old packaging mechanisms, and the laziness of developers, which are just copying and adapting existing packages, delays the convergence to recent approaches. However, no matter of the used approach the least common denominator is that all Python packages should be easily installable using PyPI, the Python Package Index.

Instead of sketching the history of possible old approaches and mistakes, in the following, a simple up-to-date approach is presented to create a Python package and publish it on PyPI.

Required tools

Starting from Python 2.7.9/ 3.4 pip should already be installed. If it is not installed yet, install it following PIP's documentation. Make sure to upgrade to the latest version:

pip install -U pip setuptools

To upload a Python package to PyPI it is strongly recommended to use twine since it provides a secure transmission to the webpage and it is easy to use. Install twine as follows:

pip install twine

Package structure

Each python package has a certain structure and must include several files. Here an overview of the required package files (here: package example):

example/
    example/
        __init.__.py
        example.py
        ...
    CHANGELOG.rst
    LICENSE.txt
    MANIFEST.in
    README.rst
    setup.cfg
    setup.py

Details on required files

In the following the required files and their meanings are discussed:

setup.py

setup.py is the most important file at the root level of the project. It serves two main functions:

  1. Set the basic configuration for the package via the setup() function
  2. Provide a CLI for package related tasks. Available commands can be listed by calling: python setup.py --help-commands

Example: setup.py

from setuptools import setup, find_packages
import codecs
import os

# get current directory
here = os.path.abspath(os.path.dirname(__file__))

def get_long_description():
    """
    get long description from README.rst file
    """
    with codecs.open(os.path.join(here, "README.rst"), "r", "utf-8") as f:
        return f.read()

setup(
    name='example',
    version='0.0.1',
    description='An example project',
    long_description=get_long_description(),
    url='https://myurl.net/example',
    author='John Doe',
    author_email='jdoe@foobar.com',
    license='MIT',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Topic :: Software Development :: Build Tools',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 2',
        'Programming Language :: Python :: 2.6',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3',
        'Programming Language :: Python :: 3.3',
        'Programming Language :: Python :: 3.4',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
    ],
    keywords='python packaging',
    packages=find_packages(
        exclude=['contrib', 'docs', 'tests']
    ),
    install_requires=[],
)
Parameter hints

The most important parameters are:

  • name: name of the project as it can be found on PyPI. Make sure that it follows the naming conventions as stated in PEP 426 and that the name is not used by another project yet.
  • version: the version of the project (each new version will be listed on PyPI). Make sure to use a valid version scheme as stated in PEP 440. It is recommended to store the version only at one place in the project. See the advance technique of single sourcing the version for ways to implement this.
  • description and long_description will be shown on the PyPI webpage on publishing
  • url: URL to the project's webpage
  • author and author_email: author's contact information
  • license: every project MUST have a license, otherwise people can not legally use it! Make sure that it corresponds to the licence file LICENSE.txt.
  • classifiers: a list of classifiers that categorize the project on PyPI. See the Classifiers List for available categories. Since packages with unknown categories are rejected, it is possible to add a category like Private :: Do Not Upload to avoid the upload of a package. Nevertheless, it is recommended to state at least the following categories for a package:
    • development status (alpha, beta, production)
    • intended audience and topic
    • license
    • supported programming language versions
  • keywords: list of keywords describing the project
  • packages: list of packages that should be included in the project. find_packages() can be used to automatically find these packages.
  • install_requires: dependencies that are required to run the project. Will be automatically installed when a package is installed via pip

Note that there are a couple of other parameters that are not listed here.

setup.cfg

setup.cfg is an ini file that contains default options for setup.py commands. For example, the universal flag can be set that states that the package works with Python 2 and 3.

Example: setup.cfg

[bdist_wheel]
universal=1

README.rst

Contains the description and goal of the project in the reStructuredText format. This file will also be parsed for the auto generated webpage, when the package is uploaded to PyPI.

LICENSE.txt

Contains the license of the project. This is very important because otherwise the project cannot be legally used by others. NEVER made up some custom license, but rather rely on one of the Open Source Licenses.

Example: LICENSE.txt

The MIT License (MIT)
Copyright (c) <year> <copyright holders>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

MANIFEST.in

The MANIFEST.in file contains a list of all additional files that should be included in the package, but that are not covered by the setup() call. A detailed explanation and a template can be found in the diskutils help.

Example: MANIFEST.in

# include additional documents such as readme files
include *.rst *.txt

The check-manifest package can be helpful to check its content and probably missing files.

CHANGELOG.rst

Though, it is not a strict requirement, it is recommended to add a CHANGELOG.rst file to highlight the changes between the versions.

The package itself (example/)

The source code of the package is normally put into a separate directory that has the same name as the package (here: example/).

Additional considerations

There are a couple of other things that may be interesting in the context of package creation:

Testing

For a real package it would make sense to create some automated tests. For example, there are a couple of approaches that use Continuous Integration. In this context tox provides a standardize way of testing Python code. It can be used in combination with Travis-CI a distributed CI server which builds tests for open source projects for free and which is well-integrated with GitHub.

Packaging the projects

Finally, a package, also referred to as distribution, can be created. There are different ways of creating a package:

Source distribution

The simplest way to create a package is to create a source distribution:

python setup.py sdist

The drawback of this approach is that it is a little bit slow on installation time, because the build step must be conducted each time the package is installed via pip.

Wheels

A wheel is a built package i.e. it can be installed without the build step on installation. As a result, the installation is much faster for the user.

python setup.py bdist_wheel

There are different types of wheels:

  • Universal wheels: can be created for pure python packages that support Python 2 and 3. For this, the --universal flag must be attached to the command or must be set in setup.cfg.
  • Pure python wheels: can be created from pure python packages that cannot run on both Python 2 and 3 without modification. However, often Python 2 code can be converted with 2to3 and then build again for Python 3.
  • Plattform wheels: are created when the package can only be built for a certain plattform due to the involvement of non-Python binary extensions. The platform dependency will automatically be detected so that the package name changes correspondingly.

Uploading the project to PyPI

After building the python package, it can now be uploaded to PyPI.

Create an account

First, create an account on the PyPI webpage. You will obtain a mail with a confirmation link that must be visited.

After the successful registration, create the ~/.pypirc file with the following content:

[distutils]
index-servers=pypi

[pypi]
repository = https://pypi.python.org/pypi
username = <username>
password = <password>

Since things are getting real, it is recommended to start with the testpypi environment, which can be used for testing purposes. A detailed setup description is provided here.

Register the project at PyPI

The simplest and most secure way to register a project at PyPI is to use twine:

twine register dist/*

The project's metadata are used to register the project at the webpage.

Upload the project to PyPI

Simply use twine to upload the project to PyPI.

twine upload dist/*

Now, your python package should be findable on PyPI.

Conclusion

Building a python package is still a little bit tricky, but by following the described steps it should be possible. The described steps do only cover the basics -- for a detailed description of possible options and parameters check the Python Packaging User Guide.

python packaging