Converting poetry.lock to requirements.txt for Legacy CI/CD Pipelines
You’ve adopted Poetry for its superior dependency resolution and poetry.lock pinning, moving beyond pip’s unpredictability. But legacy CI/CD pipelines—like Jenkins, Travis CI, or older GitLab CI—still expect the traditional pip install -r requirements.txt. In this article, we’ll export your poetry.lock to a pip-compatible requirements.txt, maintaining reproducibility and adding security via hashes where needed.
Why Convert poetry.lock to requirements.txt?
Poetry’s lockfile captures a complete, resolved dependency tree—far superior to loose specifiers in requirements.txt. Exporting it lets you develop with Poetry’s power while deploying via pip in constrained environments.
Benefits:
- Reproducibility: Exact versions and sub-dependencies prevent environment drift—what installs today should install tomorrow.
- Broad compatibility: Pip works everywhere; skip Poetry installation in air-gapped or minimal CI runners.
- Security hardening: Optional hashes verify package integrity against supply-chain attacks.
- CI efficiency: No bootstrap overhead; faster builds by minutes.
Trade-offs:
- Duplicate lockfiles: requirements.txt must sync with poetry.lock—use hooks to automate.
- Hash fragility: Platform or index differences can break installs; omit for dev, include for prod.
- Misses Poetry features: No dev groups or plugins in basic exports.
Of course, upgrading CI to Poetry support avoids exports altogether—but that’s not always feasible.
Prerequisites
- Poetry 1.2+:
$ pipx install poetry(preferred) or$ curl -sSL https://install.python-poetry.org | python3 - - Project with pyproject.toml and poetry.lock.
Method 1: poetry export (Recommended)
Poetry’s built-in exporter handles groups, extras, and hashes.
$ poetry export -f requirements.txt --output requirements.txt --without-hashes # main group only
$ poetry export -f requirements.txt --output requirements.txt # with hashes (recommended for prod)
$ poetry export -f requirements.txt --output requirements-dev.txt --dev --without-hashes # dev dependencies
$ poetry export -f requirements.txt --output requirements-all.txt --all-extras # all groups and extras
Expected requirements.txt (example):
requests==2.32.3
click==8.1.7
(Versions/hashes from your lockfile; cat the file to inspect.)
Verify:
$ python -m venv tmp-venv
$ source tmp-venv/bin/activate # Linux/macOS; Windows: tmp-venv\\Scripts\\activate
$ pip install -r requirements.txt
$ pip list # Should match `poetry show --latest` output
$ deactivate
$ rm -rf tmp-venv
Tip: If lists differ, your export missed groups—use --dev or --all-extras.
Method 2: Scripted Export for Custom Needs
For automation or filtering:
# export_lock.py (basic version; for production, use `poetry export`)
import tomllib
from pathlib import Path
def poetry_to_reqs(lockfile="poetry.lock", include_dev=False):
with open(lockfile, "rb") as f:
data = tomllib.load(f)
reqs = []
for pkg in data["package"]:
if not include_dev and pkg.get("category") == "dev":
continue
name = pkg["name"] # Pip names use '-', no replacement needed
version = pkg["version"]
reqs.append(f'{name}=={version}')
Path("requirements.txt").write_text("\n".join(reqs) + "\n")
print("Exported to requirements.txt")
if __name__ == "__main__":
poetry_to_reqs()
Note: Skips full hashes/sources—use for dev prototyping. Handles deps by category; toggle include_dev=True.
Quick Alternatives Comparison
Before scripting, weigh options:
- poetry export (default): Full-featured, official ✓
- pip freeze > reqs.txt (post-poetry install): Easy but impure (env state)
- poetry2reqs pip: Third-party; check compatibility
Prefer official when possible.
$ python export_lock.py
Exported to requirements.txt
$ cat requirements.txt
requests==2.32.3
click==8.1.7
...
(Matches your lockfile’s resolved packages.)
Integrating in CI/CD Pipelines
GitHub Actions Example
# .github/workflows/ci.yml
name: CI
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: '3.12'}
- run: pip install poetry # Or pre-commit hook to generate
- run: poetry export -f requirements.txt --output requirements.txt
- run: pip install -r requirements.txt
- run: pytest
Jenkins Pipeline
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'pipx install poetry'
sh 'poetry export -f requirements.txt --output requirements.txt'
sh 'pip install -r requirements.txt'
}
}
}
}
Best practices:
- Commit
requirements.txtto repo for prod deploys. .gitignore: Keep both lockfiles; regenerate reqs on Poetry changes.- Automate: Pre-commit hook ensures sync:
# .pre-commit-config.yaml
repos:
- repo: local
hooks:
- id: poetry-export
name: 'Export Poetry lock to requirements.txt'
entry: poetry export -f requirements.txt --output requirements.txt --without-hashes
language: system
files: ^poetry\\.lock$
pass_filenames: false
Install: $ pre-commit install
- Security: Use
--hashfor prod;--without-hashesdev/CI caches. - Legacy limits: Older pipelines lack Python caching—exports add little overhead vs full Poetry.
Troubleshooting Common Issues
Here are pitfalls we’ve encountered and fixes:
- “No poetry.lock found”: Run
$ poetry lockfirst to generate/update it. - Hash verification fails on pip install: Platform/wheel mismatch—use
$ poetry export --without-hashesor export on CI target OS. - Missing dev/test deps: Specify
$ poetry export --devor--only=mainexplicitly. - requirements.txt too large/slow: Export groups separately:
--only=main,--dev; pip handles duplicates fine. - Version drift between exports: Lockfile unchanged? Use
$ poetry lock --no-updateto avoid resolutions. - TOML parse errors: Outdated Poetry—upgrade:
$ poetry self update. - Source repos (git/private): Hashes may miss; export includes URLs, but test pip install.
Pro tip: Always verify with the venv test from Method 1 before committing.
Wrapping Up
You’ve now got reliable ways to span Poetry and pip worlds. Here’s what to remember:
Key Takeaways:
poetry exporthandles 90% of cases—prioritize it over scripts.- Always verify: Isolated venv
pip listshould mirrorpoetry show. - Automate exports: Pre-commit hooks keep files in sync without manual toil.
- Hashes boost security but demand careful platform matching.
- Long-term: Migrate CI to Poetry for richer features like parallel installs.
Try it: Export your project’s lockfile, spin up a Docker container mimicking your CI (e.g., docker run python:3.12-slim), and test pip install.
Further Reading:
Questions or edge cases? Share in the comments—we learn together.
Sponsored by Durable Programming
Need help maintaining or upgrading your Python application? Durable Programming specializes in keeping Python apps secure, performant, and up-to-date.
Hire Durable Programming