Setting up python development environment with buildout

Attn: Checkout Conda before trying this.

Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.
–buildout.org

I’ve documented the steps required to create a simple buildout based project.

1. Start by creating a project directory and initialise a virtual environment inside the project directory.

   $ mkdir word_count_buildout && cd word_count_buildout
   $ virtualenv --no-site-packages .env
   $ source .env/bin/activate
   

2. Fetch bootstrap-buildout.py. A common script required to create necessary directories and eggs (like setuptools, etc. )

    
   $ wget https://bootstrap.pypa.io/bootstrap-buildout.py
   

3. Create a buildout configuration file

   $ vi buildout.cfg
   

   copy & paste the following snippet

   [buildout]
   develop = .
   parts = job job-test scripts zipeggs
   # index = http://mypypicloud.example.com:6543/pypi/
   
   [job]
   recipe = zc.recipe.egg
   interpreter = python
   eggs = wordcount-job
   
   [job-test]
   recipe = pbp.recipe.noserunner
   eggs = pbp.recipe.noserunner
   working-directory = ${buildout:directory}
   
   [scripts]
   recipe = zc.recipe.egg:scripts
   eggs = dumbo
   
   [zipeggs]
   recipe = zipeggs:zipeggs
   target = dist 
   source = eggs
   

   basic config file structure explained here.

   • ln:4 index If using private PyPi, uncomment and replace the URL.
   • ln:21 recipe = zipeggs:zipeggs – a buildout recipe to zip all flattened/unzipped eggs, flattened/unzipped eggs are convenient while developing (for debugging purpose) and they load faster. Dumbo requires zipped eggs to be passed via -libegg param, zipeggs recipe can generate zipped eggs under target directory (dist). more details @tamizhgeek repo.
4. Execute the bootstrap file

   $ python bootstrap-buildout.py
   
   Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-18.1.zip
   Extracting in /tmp/tmpZ7ki33
   Now working in /tmp/tmpZ7ki33/setuptools-18.1
   Building a Setuptools egg in /tmp/bootstrap-q4Xfc5
   /tmp/bootstrap-q4Xfc5/setuptools-18.1-py2.7.egg
   Creating directory '/home/raj/workspace/word_count/eggs'.
   Creating directory '/home/raj/workspace/word_count/bin'.
   Creating directory '/home/raj/workspace/word_count/parts'.
   Creating directory '/home/raj/workspace/word_count/develop-eggs'.
   Generated script '/home/raj/workspace/word_count/bin/buildout'.
   

   The script has created few directories and a buildout script inside bin directory. Read more about the directory structure here

5. Create setup.py.

   $ vi setup.py
   

   from setuptools import setup, find_packages
   import os
   version = os.environ.get("PIPELINE_LABEL", "1.0")
   setup(
       name="wordcount-job",
       version=version,
       packages=find_packages(),
       zip_safe=True,
       install_requires=[
           'dumbo'
       ]
   )
   

   Read more about setuptools here.
   Any changes to setup.py and buildout.cfg should be followed by executing ./bin/buildout

6. Create a python module for a simple word-count dumbo job.

   $ mkdir wordcount-job && touch wordcount-job/__init__.py
   $ vi wordcount-job/wordcount.py
   

   copy & paste the following code.

   def mapper(key, value):
       for word in value.split():
           yield word, 1
   
   def reducer(key, values):
       yield key, sum(values)
   
   if __name__ == "__main__":
       import dumbo
       dumbo.run(mapper, reducer)
   

7. Finally, run the buildout script to fetch artifacts from private or central pypi

   $ ./bin/buildout
   

   All the dependencies (and its dependencies) mentioned in setup.py are collected under eggs/ directory. Zipped eggs are available under dist/ directory and executable scripts with dependencies wired are generated under bin directory. Try viewing the contents of bin/dumbo.

8. To run the job

   $ ./bin/dumbo start wordcount-job/wordcount.py -input /tmp/input -output /tmp/output 
   

   wordcount.py can access all the dependencies (under eggs/ directory) mentioned in setup.py.

9. To run tests.

   $ ./bin/job-test
   

   read more about pbp.recipe.noserunner recipe and nose

10. To build egg

    $ ./bin/buildout setup . bdist_egg
    

11. Private PyPi repos can also be used to distribute python eggs. To publish an egg to a private pypi, create config for pypicloud under home directory

    $ cd $HOME && vi .pypirc
    

    copy & paste the following

    [distutils]
    index-servers = my-pypi
     
    [my-pypi]
    repository: http://mypypicloud.example.com:6543/pypi/
    username: username
    password: password
    

    Under project working directory

    $ ./bin/buildout setup . bdist_egg upload -r my-pypi
    

12. Source code
13. Beer to bud @azhaguselvan for support and stuff.