Examples of using conda + poetry for cross-platform Python package portability

Summary

Examples in this note simulate the scenario of developing a geoparsing package that could work across MacOS and Linux systems (CentOS Linux7 docker image is used to emulate AWS sagemaker EC2’s RedHat OS; ). The cross-platform portability is achieved by using conda and poetry in the following ways:

• conda for python version + environment managements
• poetry for package managements (cross-platform dependency management + package templating, creation, build, and publishing)

Assumed Requirements

• Python version = 3.6.13 (same version used by sagemaker jupyter kernerl conda_pythone)
• Portable between MacOS (Kernel version Darwin 19.6.0) and Linux (CentOS Linux7). More specifically, “portable“ means:

1. The virtual environment can be created on both platforms
2. Dependencies can be installed on both platforms
3. Package wheel (generated via poetry build in the examples) can be installed on both platforms via pip install <path-to-.whl file>

Content of Dummy Package

1. Package structure

2. Core module content and expected behaviour

Upon successful installation within the virtual environment:

>>> from conda_poetry_tester import run
>>> run.get_location()
Flatiron Building, 175, 5th Avenue, Flatiron District, Manhattan, New York County, New York, 10010, United States

3. Core packages

• geopy: Work horse for geoparsing
• black (development only): auto-formatter
• poetry2setup (development only): For generating setup.py for editable install of the being-developed local package under virtual environment

usage of poetry2setup: poetry2setup > setup.py && python setup.py develop

Steps for Creating and Testing Package

1. Install miniconda in silent mode

Reference: Installing in silent mode — Anaconda documentation

2. Install poetry via installer

Reference: Poetry documentation

# The following command installs poetry via its official installer
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

3. Create python-version-specific virtual environment with conda

4. Export cross-platform portable environment.yml

This is where conda’s default behaviors are not ideal. Flag --from-historyand piping into grep -v is necessary to avoid platform incompatibility issues:

conda env export — from-history | grep -v "^prefix" > environment.yml

Content of environment.yml:

5. Install dependencies with poetry

During prompts of poetry init, specify python version to be 3.6.13. The resulting pyproject.toml file should have the following content:

6. Generate setup.py for editable local installation

7. Test installation

Steps for Testing Cross-Platform Portability

Both directions are tested:

1. Developed and tested on MacOS and installed on Linux
2. Developed and tested on Linux and installed on MacOS

The following example is based on direction 1:

1. Create and activate virtual environment on CentOS Linux 7 docker

Why CentOS Linux 7? To mimic AWS Sagemaker notebook instance Red Hat Linux 7

2. Install dependencies with pyproject.toml

poetry install && python setup.py develop

3. Validate editable installation

Supplement Information

CenOS Linux7 docker container creation