Rui CarmoPython is one of my favorite programming languages due to its terseness and amazing flexibility, and after decades using it I find it quite amusing that it is now the language of choice for data science and machine learning.
My Stuff
| Year | Item | Notes |
|---|---|---|
| 2016 | Sushy | my current wiki engine |
| 2015 | rss2imap | The way I used read my news a while back |
| 2009 | PNG Canvas | a native Python PNG creation module, suitable for use in Google App Engine |
| 2007 | Yaki | my previous wiki engine |
| 2006 | MailArchive | a quick hack to file web pages as MIME compliant e-mail messages. |
| 2005 | PicoStats | a minimalist Apache log parser. |
| IMAP Backup | another quick hack that apparently filled a long-standing need. |
|
| The Python Grimoire | which I converted into TiddlyWiki format |
|
| 2004 | PicoRendezvous.py | a minimalist (and dumb) Bonjour / Rendezvous client |
| newspipe | which I used daily for a couple of years and occasionally contributed to |
|
| Growl native protocol implementation | this spawned numerous variants, including a proxy. |
Resources
| Category | Date | Link | Notes |
|---|---|---|---|
| Astronomy | 2021 | Skyfield | An astronomical library for Python |
| Audio | 2024 | pedalboard | a library for audio processing |
| Authentication | 2023 | authlib | OAuth, OpenID Connect and other niceties |
| 2021 | aioauth | Oauth 2.0 framework and provider for asyncio |
|
| Automation | 2024 | helium | a library for automating browsers such as Chrome and Firefox |
| CAD | build123d | a parametric BREP modeling framework for 2D and 3D CAD built on OpenCascade |
|
| 2023 | sdf | Generate 3D meshes based on SDFs (signed distance functions) |
|
| sdf (Gitlab) | Another version of the |
||
| CLI | 2021 | pqdm | parallel tqdm |
| 2020 | pyp | a clever shell tool to do inline Python |
|
| typer | A click-like library for building CLI applications |
||
| 2019 | tqdm | a configurable progress bar |
|
| 2018 | python-fire | a CLI wrapper for any Python object |
|
| click | A very nice way to do composable CLI tools (see also argh, defopt, docopt, python-fire, etc.) |
||
| Clint | A great module for developing CLI apps |
||
| pbs | A nice, simple and consistent generic command wrapper. |
||
| ptpython | The Python Prompt Toolkit |
||
| plumbum | Another command wrapper, with SSH support. |
||
| Cloud | 2020 | pyinfra | a fabric-like agentless automation tool |
| Compilers | 2022 | codon | an LLVM compiler for a Python subset |
| 2021 | mypyc | a compiler for a subset of type-annotated Python |
|
| Transcrypt | Another JavaScript transpiler |
||
| Compression | 2021 | zipfly | a library for doing on-the-fly zip compression with low memory footprint |
| Concurrency | 2024 | dramatiq | background queueing/task processing library |
| protean | framework for building event-driven applications using the CQRS pattern |
||
| asyncpal | a unified approach at providing elastic task pools |
||
| 2023 | mpire | a |
|
| 2020 | aiomultiprocess | an asyncio Pool implementation to manage multiple processes with independent event loops |
|
| Trio | a friendlier wrapper around asyncio for concurrent tasks |
||
| OpenReplica | A coordination service that provides replication and synchronization support for large-scale distributed systems |
||
| python-rq | A simpler queueing library |
||
| Pulsar | Another actor-based multiprocessing library with |
||
| Celery | My current go-to library for task queueing |
||
| Pykka | An implementation of the Actor model, with gevent support |
||
| offset | ports of the Go concurrency model |
||
| aioprocessing | A set of |
||
| ray | A distributed execution engine that implements a |
||
| goless | ports of the Go concurrency model |
||
| Littleworkers | Parallel job management without the hassles. |
||
| Console | 2020 | rich | a console library with various layout options |
| bpython | My favorite REPL |
||
| Colorama | Making ANSI color sequences work cross-platform (see also this gist) |
||
| fabulous | Prints images to console |
||
| urwid | A |
||
| Data Engineering | 2024 | polars-h3 | a Polars extension that adds support for the H3 discrete global grid system |
| Pathway | a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG |
||
| 2020 | Little Ball of Fur | a graph sampling library that works with NetworkX |
|
| riptable | a multithreaded array processing library |
||
| Happy | A Hadoop library to run map-reduce jobs via Jython |
||
| biplist | A library that can manage Mac binary |
||
| pyparsing | a handy, flexible parser library that does away with lex/yacc. |
||
| snakebite | A native HDFS client to access the Hadoop datastore. |
||
| xlwings | An Excel plugin |
||
| Bounter | A fast probabilistic counter |
||
| voluptuous | A data schema validation library. |
||
| fuzzywuzzy | A fuzzy string matching library |
||
| PyCascading | A better way to write Hadoop jobs. |
||
| xlwt | a library to parse Excel documents (cheatsheet. |
||
| Data Science | 2025 | kreuzberg | a text extraction library supporting PDFs, images, office documents and more |
| 2024 | sail | unifies stream and batch processing with a drop-in replacement for Spark SQL |
|
| cuDF | a GPU DataFrame library leveraging CUDA |
||
| 2023 | pygwalker | a Tableau-like dataframe exploration tool for Jupyter |
|
| datapane | a data and reporting and analisys framework |
||
| fugue | a unified interface for executing SQL, Python and Pandas code on Spark, Dask and Ray |
||
| 2021 | Kats | a time series analysis library |
|
| euporie | a text-based interface for Jupyter notebooks |
||
| xeus-sql | a Jupyter kernel for SQL databases |
||
| eztable | a simple table library |
||
| pyxelate | a pixelation/dithering algorithm that uses a mix of algorithmic and ML approaches |
||
| 2020 | TextHero | an NLP library for large-scale text processing |
|
| papermill | a way to parameterize and automate Jupyter notebooks |
||
| SciPy | scientific (including plotting) libraries for Python |
||
| Data Validation | 2021 | schema | another data validation library |
| pydantic | Data validation using type hinting |
||
| Schematics | A validation/schema handling library. |
||
| Database | 2023 | mongolite | another MongoDB-like database |
| 2021 | pottery | a Redis library with Pythonic syntactic sugar |
|
| dataset | trivial abstraction layer over SQLAlchemy |
||
| Mongita | a MongoDB-like database library for in-process use |
||
| 2020 | DuckDB | an embeddable SQL database with OLAP features |
|
| montydb | a tiny MongoDB implementation in pure Python |
||
| simple-graph | A simple graph database in SQLite |
||
| pymssql | SQL Server and Sybase driver |
||
| FreeTDS | |||
| SQLAlchemy | tutorial |
||
| Peewee | A small, very flexible ORM |
||
| tinydb | A (very) lightweight document oriented database |
||
| Goatfish | An even smaller schemaless ORM |
||
| Debugging | 2020 | Scalene | A high-performance CPU and memory profiler |
| PrettyErrors | an error handler that prettifies output |
||
| Eww | Insert a REPL into a running application |
||
| Pycallgraph | Generate a program flow graph Graphviz |
||
| Hypothesis | A |
||
| pudb | A console debugger |
||
| lptrace | A pretty amazing single-file strace-like tool for Python |
||
| assertEquals | a testing framework with a great screncast |
||
| Django | 2025 | django-qrauth | a Django library for QR code-based authentication |
| django-tastypie | a Django library for building RESTful APIs |
||
| django-rest-framework | another Django library for building RESTful APIs |
||
| django-admin-sortable | a library for adding drag-and-drop sorting to Django admin |
||
| 2024 | django-organizations | a Django library for managing organization accounts |
|
| 2023 | django-unicorn | a Reactive framework for Django that uses minimal JavaScript |
|
| 2022 | django-ninja | for building APIs with Django and Python 3.6+ type hints |
|
| 2021 | diskcache | a Django-compatible caching library |
|
| wid.gy | A Django-based CMS framework |
||
| Django | Probably the best app framework I’ve every used. |
||
| Wagtail | Another Django-based CMS framework |
||
| django-micro | A lightweight wrapper to use Django as you would Bottle |
||
| Electronics | 2024 | micropython-fusion | a sensor fusion library for MicroPython |
| Kalman-Filter-Python-for-mpu6050 | a Kalman filter for the MPU6050 IMU |
||
| 2023 | belay | a library and helper for developing MicroPython projects, including an on-device REPL |
|
| 2021 | pinout | Helps create SVG pin-out diagrams for electronic hardware |
|
| Enterprise | toil | A cross-platform pipeline management system with Common Workflow Language support |
|
| Environments | 2020 | snakeware | a Python userspace for Linux |
| Frameworks | 2024 | Platypush | a modular platform for home automation with easy MQTT event handling and a graphical editor |
| Functional Programming | Hy | An awesome LISP dialect that sits atop the runtime. |
|
| funcy | A nice, fairly complete functional library |
||
| Toolz | A functional complement to the standard library. |
||
| pyrsistent | A persistent data structure library |
||
| GIS | 2021 | prettymaps | A minimal Python library to draw customized maps from OpenStreetMap data |
| geopy | an amazingly flexible geocoding library. |
||
| reverse-geocoder | A fast offline reverse geocoder |
||
| GUI | 2024 | CustomTkinter | offers customizable widgets and supports system appearance modes across platforms |
| mesop | a rapid web development framework for Python |
||
| micropython-nano-gui | A lightweight MicroPython GUI library |
||
| nicegui | a library for creating Web UIs |
||
| 2022 | flet | creates realtime web, mobile and desktop apps using Flutter |
|
| 2021 | encodermenu | a MicroPython library for GUI creation, using uasyncio |
|
| 2020 | Gooey | a wxWidgets wrapper for quickly building GUI versions of CLI tools |
|
| DearPyGui | a Linux/Windows GUI library atop ImGUI (C++) |
||
| 2019 | fman build system | a complete build system for Qt applications |
|
| 2018 | Toga | a cross-platform GUI that works on the Mac |
|
| kivy | A sophisticated UI/UX library with Android support |
||
| PythonCard | A bit basic, but interesting. |
||
| VPython | 3D OpenGL libraries for quick visualization |
||
| winGuiAuto.py | for Windows automation |
||
| pyglet | a cross-platform windowing and multimedia library with OpenGL support |
||
| PySide | A recent Qt binding (up to and including 4.7) with QML support |
||
| PySimpleGUI | a simple tkinter-based GUI wrapper |
||
| Graphics | 2024 | renderer | rendering and rasterizing library for STL |
| pyrender | glTF 2.0-compliant OpenGL renderer for visualization of 3D scenes |
||
| pygfx | render engine targeting Vulkan/Metal/DX12 |
||
| 2023 | pyxel | A retro game engine for Python |
|
| 2022 | taichi | a high-performance graphics library |
|
| 2021 | image-blender | a Cython implementation of Adobe Photoshop blend modes |
|
| 2020 | blurhash | A way to encode image placeholders as strings |
|
| hiplot | high dimensional interactive plotting |
||
| diagrams | a library for drawing cloud architecture diagrams (Azure, AWS, k8s, etc.) |
||
| 2019 | douglas-quaid | a library for image matching and correlation |
|
| imagededup | uses several different algorithms (including NNs) to detect duplicate images |
||
| plottyprint | printable formatting for matplotlib |
||
| smc.freeimage | A wrapper for the freeimage library that can handle various TIFF and fax formats as well as ICC color profiles |
||
| Graphite | a real-time graphing system similar to |
||
| Skip's Python Bits | lots of useful snippets |
||
| Pygal | An SVG charting library with a variety of built-in formats. |
||
| Pyx | Oldie, but goodie. |
||
| pygame | a game-oriented library with SDL support |
||
| pyprocessing | A Processing clone in Python. |
||
| Vapory | A library to generate photorealistic renderings using POV-Ray |
||
| drawille | Uses Braille characters for TTY charting |
||
| Cropmap | Face-aware image cropping. |
||
| HTML | lxml | a different |
|
| cssutils | a CSS parser |
||
| Gnosis Utilities | all sorts of XML goodies |
||
| pygments | syntax highlighter able to handle multiple nested languages |
||
| pyquery | A jQuery-like library for manipulating documents. |
||
| BeautifulSoup | The most powerful HTML parser out there. |
||
| squeezeit | A CSS and JavaScript bundler/minifier |
||
| Interpreters | 2024 | RustPython | A Python Interpreter written in Rust |
| 2023 | pocketpy | an embeddable C-based interpreter in 5000 LOC |
|
| PyScript | a WASM runtime using Pyodide for web embedding |
||
| Large Language Models | pykoi | a unified interface for data and feedback collection, comparing models, and doing basic RAG |
|
| Libraries | 2024 | unsafe-python | A library to assist writing memory-unsafe code, without any imports |
| domscribe | converts HTML to semantic Markdown while preserving structure and meaning |
||
| foc | an intriguing collection of higher-level functions using operators for composability |
||
| BM25S | an ultrafast implementation of BM25 in pure Python |
||
| 2023 | fastkafka | A Kafka library inspired by fastapi |
|
| reader | A modern RSS/Atom/JSON feed reader library. |
||
| pgvector-python | for using Postgres as a vector database |
||
| 2022 | slack-machine | A Slack bot library that uses socket mode and |
|
| mintotp | a tiny TOTP library |
||
| 2021 | Sorted Containers | a fast C-backed collections library |
|
| seedir | A utility library to render filesystem tree diagrams |
||
| 2020 | minotaur | an asynchronous inotify interface |
|
| minibatch | a simple stream processing library |
||
| Machine Learning | 2021 | Perceptilabs | free visual ML model creation tool |
| 2020 | netron | a viewer for neural network, deep learning and machine learning models |
|
| discomll | A ML framework for Disco |
||
| Tensorflow | Usual staple, largely superseded by PyTorch |
||
| 2021 | imapautofiler | an IMAP mailbox sorter that I unwittingly replicated (with less features) |
|
| 2018 | aiosmtpd | an asyncio version of smtpd.py |
|
| Salmon | an SMTP mail server |
||
| slimta | a mail app framework for building MTAs |
||
| Messaging | 2023 | Propan | a declarative Python MQ framework inspired by |
| Music | musicpy | A domain-specific language for writing music |
|
| 2020 | pippi | an audio processing/music library |
|
| pymidi | a library for building MIDI over IP/RTP applications (compatible with Apple specs) |
||
| Networking | 2023 | p2pd | an async networking library with P2P and UDP support |
| 2019 | opendrop | a partial implementation of AirDrop |
|
| 2018 | asphalt | An |
|
| PyRendezvous | neat little Bonjour module I use a lot |
||
| telnetscript | a simple module to do scripted |
||
| IPy | an IP address manipulation library. |
||
| pyvnc | A |
||
| ircutils | A relatively modern async IRC framework |
||
| Eventlet | Amazing event-oriented framework |
||
| ftputil | a high-level interface to the |
||
| Python smtplib progress indication | Might come in useful some day. |
||
| gunicorn | A WSGI server that supports a number of different worker daemons and is highly customizable. |
||
| dpkt | packet creation and parsers |
||
| Pcapy | a packet manipulation library |
||
| uwsgi | A C application container that speaks WSGI and is seriously good. |
||
| PuSSH | SSH wrapper |
||
| RPyC | an awesome RPC library |
||
| SleekXMPP | A more modern XMPP library |
||
| Tftpy | a pure Python TFTP library, useful for implementing UDP file transfer testbeds. |
||
| ThriftPy | A pure Python implementation of Apache Thrift |
||
| pyxmpp2 | another one, probably more interesting. |
||
| pyrobuf | a Cython implementation of Google’s |
||
| pyapns | An Apple Push Notification Service provider based on Twisted. |
||
| OCR | 2024 | surya | OCR library supporting 90+ languages |
| marker | a PDF to markdown converter optimized for books and scientific papers, using surya |
||
| 2023 | peepdf | a Python tool to explore PDF files in order to find out if the file can be harmful or not. |
|
| 2021 | borb | a library for creating and manipulating PDF files |
|
| 2020 | camelot | a library to extract tabular data from PDFs |
|
| Kiva | a Display PDF library |
||
| PyPdf | a PDF toolkit |
||
| WeasyPrint | Alternatives to PrinceXML for generating formated PDF documentation. |
||
| xhtml2pdf | |||
| Packaging | 2021 | PyOxidizer | a better way to distribute Python bundled apps than pyinstaller |
| shiv | builds self-contained applications for easy deployment |
||
| 2019 | Poetry | a dependency management and packaging tool |
|
| Presentations | pydown | An HTML presentation generator |
|
| Reference | 2023 | python-cheatsheet | a modern reference, useful for kids |
| Reporting | Relatorio | A very neat reporting library with multiple output formats |
|
| ReportLab Toolkit | a pure Python PDF library that includes a presentation tool. |
||
| Resources | 2024 | awesome-micropython | another curated list of MicroPython resources |
| awesome-micropython | a curated list of MicroPython resources |
||
| 2023 | python-package-template | a template project for building packages |
|
| Robotics | kabuki | A declarative control library for MicroPython |
|
| Runtimes | 2022 | pikascript | an embedded Python runtime for the STM32 and other tiny MCUs |
| 2021 | Pyjion | Pyjion, a JIT extension for CPython that compiles your Python code into native CIL and executes it using the .NET 5 CLR. |
|
| 2020 | Pycopy | a fork of MicroPython |
|
| Nuitka | An interesting batch compiler. |
||
| Pyjamas | Some interesting twists on using Python on the way to JavaScript. |
||
| Brython | a fairly complete in-browser runtime |
||
| Skulpt | Some interesting twists on using Python on the way to JavaScript. |
||
| tinypy | Python in 64K of code. |
||
| Js2Py | A JavaScript interpreter written in pure Python |
||
| Hope | A specialized JIT compiler |
||
| Scraping | 2023 | trafilatura | a library and CLI tool for text extraction |
| 2020 | InvoiceNet | a neural network for extracting invoice information from PDFs |
|
| Mechanize | a programmatic web browser for screen scraping. |
||
| subliminal | A library for looking up movie subtitles in various languages. |
||
| pholcidae | A web crawler library. |
||
| urlwatch | A flexible page modification monitor |
||
| Site Generators | 2023 | django-distill | Minimal configuration static site generator for Django |
| 2020 | MiniConf | a lovely site generator for managing conferences |
|
| 2019 | portray | a static site generator for Python project documenntation |
|
| Speech | silero-models | Multi-language neural text-to-speech |
|
| TUI | 2023 | trogon | a genius way to automatically generate a TUI for |
| pyTermTk | an impressive text UI toolkit |
||
| 2021 | textual | Text user interface widgets and fields |
|
| Task management | doit | a task runner that can replace make for many use cases |
|
| Tools | 2025 | posting | an amazing terminal-based Postman replacement |
| 2024 | hancho | a simple, pleasant build system |
|
| berylliumOS | a Python-based shell for CircuitPython devices |
||
| 2023 | ssh-crypt | a clever way to manage secrets on remote servers |
|
| frogmouth | a TUI Markdown browser |
||
| pypipe | a Python command-line tool for pipeline processing that can export pipelines as standalone scripts |
||
| marcel | a shell that can pipe structured data (like PowerShell, but done right) |
||
| flameshow | a terminal flamegraph viewer |
||
| termgraph | simple terminal charting |
||
| reloadium | advanced hot reloading and profiling |
||
| 2022 | reloading | a decorator for hot reloading code |
|
| Doctool | a tool to manipulate .docx files |
||
| honcho | a process supervisor that uses |
||
| Tutorials | 2021 | ugit | a step-by-step git implementation |
| Unicode | Unidecode | Does character transliterations. |
|
| Video | 2021 | webcam-filters | a set of GStream filters for doing live webcam background segmentation and blurring |
| Visual Programming | 2024 | nodezator | a graphical node workflow editor based on PyGame |
| 2022 | PyFlow | a another (confusingly, identically named) visual programming environment |
|
| Ryven | a flow-based programming environment with a Qt front-end |
||
| PyFlow | a desktop-based visual programming environment |
||
| Web | newscatcher | a cute scraper database/fetcher combo |
|
| Web Frameworks | 2023 | socketify.py | another |
| litestar | another high-performance ASGI API framework with rigorous typing |
||
| robyn | an async Python framework with a Rust connection handler |
||
| sse-starlette | Server-Sent Events for Starlette and FastAPI |
||
| ucall | a JSON-RPC library that claims to be 100x faster than FastAPI |
||
| 2022 | pynecone | a full-stack Python web framework that automatically renders Next.js components |
|
| 2021 | FastAPI | decorator-based, runs under uvicorn |
|
| 2020 | blacksheep | an ASGI web framework that uses decorators |
|
| starlette | an ASGI web framework that runs under uvicorn |
||
| 2019 | appkernel | a microservices framework |
|
| 2018 | zatt | An |
|
| Bottle | Tiny, flexible, awesome. My favorite since 2011. |
||
| Draco | Old and busted, to various extents. |
||
| Falcon | A Cythonizable framework with little magic and good performance |
||
| Karrigell | Old and busted, to various extents. |
||
| pecan | Based on object dispatching |
||
| eve | Another RESTful foundation framework |
||
| wheezy.web | A high-performance framework that seems to get along well with PyPy |
||
| Snakelets | A minimalistic app server, my old web platform of choice. |
||
| Twisted | Old and busted, to various extents. |
||
| medusa | |||
| CherryPy | |||
| Tornado | non-blocking, which is pretty interesting. |
||
| Zope | Old and busted, to various extents. |
||
| sandman | A simple RESTful wrapper |
||
| Vibora | An |
||
| Moya | a new(ish), template-centric web framework with some interesting twists |
||
| Wikis | 2024 | silicon | another Markdown-focused wiki |
| otterwiki | a nice git-backed wiki |
||
| Writing | 2023 | proselint | a linter for prose |
Techniques
- daemon.py, an example daemon script.
- Pydoc - built-in Web help, anytime, anywhere
- How to Write a Spelling Corrector - an interesting technique that can come in handy to fix/suggest search terms, etc.
- Python Webcam Color Track
- Python webcam fun - motion detection
Notes:
Setting distutils paths in OS X:
Create a file named .pydistutils.cfg in your home directory with the
following contents and set $PATH accordingly:
[install]
install_lib = ~/Library/Python/$py_version_short/site-packages
install_scripts = ~/Library/Python/$py_version_short/bin
install_data = ~/Library/Python/$py_version/share
This will install packages to your Library folder rather than to the
system directory, saving you the trouble to use sudo or risking
messing with the system installation. Of course virtualenv does a
better job for keeping track of project-specific dependencies, but this
is great for common/complex dependencies and is sure to survive
operating system upgrades.
And then you can use easy_install or pip install --user with
impunity, although you might need to do something like:
$ ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future easy_install pandas
…for getting around some clang oddities.
Instant Web Server on port 8000 (2.x):
#!/bin/sh
python -c '__import__("SimpleHTTPServer").test()'