Excited about your python project? Do you want to share Python open source projects for beginners? Packaging, publishing your python projects source code is a great way to get other people interesting python projects. Here is a simple step by step guide to build and publish your first python package

Excited about your python project? Do you want to share Python open source projects for beginners? Packaging, publishing your python projects source code is a great way to get other people interesting python projects. Here is a simple step by step guide to build and publish your first python package

We’ll describe all the steps we’ve been through to create Scitime, a package that gives an estimate of algorithm training times (you can learn more about Scitime here).

Note: We will assume that you already have a python projects source code in GitHub that you would like to package and publish.

Step 0: Have your project licensed

Before doing anything else, your project should have a license since it’s going to be open source. Depending on how you’d like your package to be used, many licenses are available. Some usual licenses for open source projects are MIT or BSD.

To add a license to your project, simply add a [LICENSE](https://github.com/nathan-toubiana/scitime/blob/master/LICENSE) file to the repository root by following the steps described here.

Step 1: make your code publish-ready

There are a few preliminary things you need to do to have your Python project package-able:

Your project structure needs to be in place. Usually, the root of the repository contains a folder with the name of the repository — this is where the core project code should be. What’s outside of this folder is the extra code that’s necessary to run and build the package (tests, documentation, etc.)

needs to be in place. Usually, the root of the repository contains a folder with the name of the repository — this is where the core project code should be. What’s outside of this folder is the extra code that’s necessary to run and build the package (tests, documentation, etc.) The core folder should include one (or more) module(s) and an [**__init__.py**](https://github.com/nathan-toubiana/scitime/blob/master/scitime/__init__.py) file which imports the classes/functions you’ll want the end user to have access to.

file which imports the classes/functions you’ll want the end user to have access to. Ideally, a proper logging system (instead of prints ) should be set by using the logging package.

(instead of ) should be set by using the package. Ideally, your core code should be organized in one or more classes.

from .estimate import Estimator

import logging class LogMixin(object): @property def logger(self): name = '.'.join([self.__module__, self.__class__.__name__]) FORMAT = '%(name)s:%(levelname)s:%(message)s' logging.basicConfig(format=FORMAT, level=logging.DEBUG) logger = logging.getLogger(name) return logger

Once your project has a set structure, you should then add a setup.py file at the root of the repository. This mostly helps to automate all the publishing and version maintaining processes. Here’s an example of what a setup.py should look like (source here).

from setuptools import setup from os import path DIR = path.dirname(path.abspath(__file__)) INSTALL_PACKAGES = open(path.join(DIR, 'requirements.txt')).read().splitlines() with open(path.join(DIR, 'README.md')) as f: README = f.read() setup( name='scitime', packages=['scitime'], description="Training time estimator for scikit-learn algorithms", long_description=README, long_description_content_type='text/markdown', install_requires=INSTALL_PACKAGES, version='0.0.2', url='http://github.com/nathan-toubiana/scitime', author='Gabriel Lerner & Nathan Toubiana', author_email='[email protected]', keywords=['machine-learning', 'scikit-learn', 'training-time'], tests_require=[ 'pytest', 'pytest-cov', 'pytest-sugar' ], package_data={ # include json and pkl files '': ['*.json', 'models/*.pkl', 'models/*.json'], }, include_package_data=True, python_requires='>=3' )

A few notes:

If your package has dependencies, a clean way to handle these is to add them in the setup file, through the install_requires argument (if the list is long you can always point to a requirement.txt file as done above)

argument (if the list is long you can always point to a requirement.txt file as done above) If you want metadata (from the repository) to be downloaded when anyone installs the package, you should add these through the package_data argument

argument More information about the setup() function can be found here

Note: steps 3 to 6 are optional (but highly recommended), however you can jump straight to step 7 if you want to publish your package right now.

Step 3: Set local tests and test coverage checks

If it is not already the case, at this point, your project should definitely have unit tests. Although there are many frameworks that can help you do that, one simple way of doing it is using pytest. All tests should be in a dedicated folder (named tests/ or testing/ for example). In that folder, put all the test files you need in order to cover as much of your core code as possible. Here’s an example of how to write a unit test. Here’s also one of our Scitime’s test files.

Once in place, you can run your tests locally by running python -m pytest from the repository root.

Once your tests are created, you should also be able to estimate the coverage. This is important as you want to maximize the amount of code that’s tested in your project (to reduce the amount of unexpected bugs).

A lot of frameworks are available for this as well, for Scitime we used codecov. You can decide on a threshold of minimum coverage allowed by creating a .codecov.yml file and also decide on what files to include in the coverage analysis by creating a .coveragerc file.

comment: false coverage: status: project: default: target: auto threshold: 10% patch: default: target: auto threshold: 10%

Example of a .codecov.yml file

[run] branch = True source = scitime include = */scitime/* omit = */_data.py */setup.py

Example of a .coveragerc file

Step 4: Standardize syntax and code style

You’ll also need to make sure that your code follows the PEP8 guidelines (i.e has a standard style and that the syntax is correct). Again, a lot of tools can help here. We used flake8.

Step 5: Create a proper documentation

Now that your project has tests and a good to structure, it should be a good time to add a proper documentation. First thing is to have a nice readme file that will show up on your Github repo root. Once this is done, here are a few additional “nice to have”:

Pull Request & issue templates : when a new PR or issue is created, these files permit you to template the descriptions as you wish. Follow these steps for PRs and these steps for issues to create them. Here are Scitime’s PR template and issue template.

: when a new PR or issue is created, these files permit you to template the descriptions as you wish. Follow these steps for PRs and these steps for issues to create them. Here are Scitime’s PR template and issue template. A contribution guide . This should be a short guide on how you want external users to contribute to your package. Here’s Scitime’s contribution guide.

. This should be a short guide on how you want external users to contribute to your package. Here’s Scitime’s contribution guide. A code of conduct (here’s Scitime’s).

(here’s Scitime’s). Tags and a description (see screenshot below).

(see screenshot below). Badges in the readme file (here’s a nice article on how to set these up).

As the readme file should be quite synthesized, it’s also pretty standard to have a more in-depth documentation. You can use sphinx to do so, and then host the documentation on readthedocs (here’s Scitime’s example). The documentation related files are usually in a docs/ folder (as done for Scitime). Here’s a nice sphinx & readthedocs tutorial.

Example of a repo that has tags and a description

Step 6: Set up continuous integration

At this point, your project is not far away from being ready to be published. However, it seems a little overwhelming to have to deal with updating the docs, running the tests and checking the style and coverage after each commit. Fortunately, Continuous Integration (CI) helps you with that. You can use webhooks to GitHub to automate all these things after each commit. Here’s the set of CI tools we used for Scitime:

For running tests, we used travis-ci and appveyor (for tests on windows platforms). For travis-ci, besides setting up the webhooks on the repository, you also have to create a .travis.yml file in which you can not only run tests but also upload updated coverage outputs and check style and format. Same goes for appveyor by creating a appveyor.yml file

Codecov and readthdocs also have dedicated webhooks

language: python python: - "3.6" # command to install dependencies install: - pip install -r requirements.txt - pip install flake8 - pip install pytest-cov - pip install codecov # command to run tests script: - python -m pytest --cov=scitime - ./build_tools/flake_diff.sh after_success: - codecov

Example of a .travis.yml file: note that here, for each commit, the tests are run along with the test coverage checks. But there’s also a flake8 check (the logic being defined in the flake_diff.sh file)

environment: matrix: - PYTHON: "C:\\Python36-x64" install: # We need wheel installed to build wheels - "%PYTHON%\\python.exe -m pip install -r requirements.txt" - "%PYTHON%\\python.exe -m pip install pytest==3.2.1" build: off test_script: - "%PYTHON%\\python.exe -m pytest"

Example of a appveyor.yml file: here, we only run the tests

This should make the whole process of updating the repository a lot easier.

Example of commit history with integrated webhooks

Step 7: Create your first release and publication

At this point, your soon-to-be package should look similar to this:

your_package/ __init__.py your_module.py docs/ tests/ setup.py travis.yml appveyor.yml .coveragerc .codecov.yml README.md LICENSE .github/ CODE_OF_CONDUCT.md CONTRIBUTING.md PULL_REQUEST_TEMPLATE.md ISSUE_TEMPLATE/

It’s now ready to be published! The first thing to do is to create your first release on GitHub — this is to keep track of the state of your project at a given point in time, a release should be created each time the version changes. You can follow this to create the release.

Once done, the only thing left to do is to publish your package. The most common platforms to publish a python package are PyPI and Conda. We will describe how to publish in both:

For PyPI , you first need to create an account, and then follow these steps using twine . This should be fairly straightforward, and PyPI also provides a test environment that can be used right before the actual deploy. The latter basically consists of creating a source distribution ( python setup.py sdist ) and uploading it with twine ( twine upload dist/* ). Once done, there should be a PyPI page corresponding to your package (here’s Scitime’s) and anyone should be able to install your package running the pip command.

, you first need to create an account, and then follow these steps using . This should be fairly straightforward, and PyPI also provides a test environment that can be used right before the actual deploy. The latter basically consists of creating a source distribution ( ) and uploading it with twine ( ). Once done, there should be a PyPI page corresponding to your package (here’s Scitime’s) and anyone should be able to install your package running the command. For conda, we recommend publishing your package through conda-forge which is a community that helps you publish and maintain your package through their conda channel. You can follow these steps to add your package to the community, you will then be added to the conda-forge GitHub organization and be able to maintain your package quite easily (here’s Scitime’s conda-forge page) — anyone should then be able to install your package running the conda command.

All done!

You should now have your package finally published and available to anyone! Although most of the work is done, you’ll still have to maintain your project as you need to make some updates: this basically means changing the versions each time you make significant changes, creating new releases and going through step 7 again.

Originally published on https://medium.freecodecamp.org

python