Python is one of my favorite programming languages due to its terseness and amazing flexibility.
See also Django for more.
My Stuff:
| Year | Item | Notes |
|---|---|---|
| 2016 | Sushy | my current wiki engine |
| 2015 | rss2imap | The way I used read my news a while back |
| 2009 | PNG Canvas | a native Python PNG creation module, suitable for use in Google App Engine |
| 2007 | Yaki | my previous wiki engine |
| 2006 | MailArchive | a quick hack to file web pages as MIME compliant e-mail messages. |
| 2005 | PicoStats | a minimalist Apache log parser. |
| IMAP Backup | another quick hack that apparently filled a long-standing need. | |
| The Python Grimoire | which I converted into TiddlyWiki format | |
| 2004 | PicoRendezvous.py | a minimalist (and dumb) Bonjour / Rendezvous client |
| newspipe | which I used daily for a couple of years and occasionally contributed to | |
| Growl native protocol implementation | this spawned numerous variants, including a proxy. |
Resources:
| Category | Link | Notes |
|---|---|---|
| Astronomy | Skyfield | An astronomical library for Python |
| Authentication | aioauth | Oauth 2.0 framework and provider for asyncio |
| CLI | click | A very nice way to do composable CLI tools (see also argh, defopt, docopt, python-fire, etc.) |
| Clint | A great module for developing CLI apps | |
| pbs | A nice, simple and consistent generic command wrapper. | |
| python-fire | a CLI wrapper for any Python object | |
| pyp | a clever shell tool to do inline Python | |
| ptpython | The Python Prompt Toolkit | |
| pqdm | parallel tqdm | |
| typer | A click-like library for building CLI applications | |
| plumbum | Another command wrapper, with SSH support. | |
| tqdm | a configurable progress bar | |
| CMS | Wagtail | Another Django-based CMS framework |
| wid.gy | A Django-based CMS framework | |
| MiniConf | a lovely site generator for managing conferences | |
| Caching | diskcache | a Django-compatible caching library |
| Cloud | pyinfra | a fabric-like agentless automation tool |
| Compilers | mypyc | a compiler for a subset of type-annotated Python |
| Transcrypt | Another JavaScript transpiler | |
| Compression | zipfly | a library for doing on-the-fly zip compression with low memory footprint |
| Concurrency | OpenReplica | A coordination service that provides replication and synchronization support for large-scale distributed systems |
| python-rq | A simpler queueing library | |
| Pulsar | Another actor-based multiprocessing library with asyncio
support and a fair degree of network transparency
|
|
| Celery | My current go-to library for task queueing | |
| Pykka | An implementation of the Actor model, with gevent support | |
| aiomultiprocess | an asyncio Pool implementation to manage multiple processes with independent event loops | |
| offset | ports of the Go concurrency model | |
| aioprocessing | asyncio equivalents of the multiprocessing library |
|
| Trio | a friendlier wrapper around asyncio for concurrent tasks | |
| ray | A distributed execution engine that implements a pandas clone |
|
| goless | ports of the Go concurrency model | |
| Littleworkers | Parallel job management without the hassles. | |
| Console | bpython | My favorite REPL |
| Colorama | Making ANSI color sequences work cross-platform (see also this gist). | |
| fabulous | Prints images to console | |
| urwid | A curses based library for building console UIs |
|
| rich | a console library with various layout options | |
| textual | Text user interface widgets and fields | |
| Data Handling | Happy | A Hadoop library to run map-reduce jobs via Jython |
| biplist | A library that can manage Mac
binary .plist files
|
|
| pyparsing | a handy, flexible parser library that does away with lex/yacc. | |
| snakebite | A native HDFS client to access the Hadoop datastore. | |
| xlwings | An Excel plugin | |
| Bounter | A fast probabilistic counter | |
| voluptuous | A data schema validation library. | |
| Little Ball of Fur | a graph sampling library that works with NetworkX | |
| riptable | a multithreaded array processing library | |
| fuzzywuzzy | A fuzzy string matching library | |
| PyCascading | A better way to write Hadoop jobs. | |
| xlwt | a library to parse Excel documents (cheatsheet). | |
| Data Science | SciPy | scientific (including plotting) libraries for Python |
| Kats | a time series analysis library | |
| TextHero | an NLP library for large-scale text processing | |
| euporie | a text-based interface for Jupyter notebooks | |
| xeus-sql | a Jupyter kernel for SQL databases | |
| papermill | a way to parameterize and automate Jupyter notebooks | |
| eztable | a simple table library | |
| pyxelate | a pixelation/dithering algorithm that uses a mix of algorithmic and ML approaches | |
| Data Validation | schema | another data validation library |
| pydantic | Data validation using type hinting | |
| Schematics | A validation/schema handling library. | |
| Database | pymssql | SQL Server and Sybase driver |
| FreeTDS | ||
| SQLAlchemy | tutorial | |
| pottery | a Redis library with Pythonic syntactic sugar | |
| Peewee | A small, very flexible ORM | |
| DuckDB | an embeddable SQL database with OLAP features | |
| montydb | a tiny MongoDB implementation in pure Python | |
| simple-graph | A simple graph database in SQLite | |
| tinydb | A (very) lightweight document oriented database | |
| dataset | trivial abstraction layer over SQLAlchemy | |
| Mongita | a MongoDB-like database library for in-process use | |
| Goatfish | An even smaller schemaless ORM | |
| Debugging | Eww | Insert a REPL into a running application |
| Pycallgraph | Generate a program flow graph via Graphviz | |
| Hypothesis | A very nice testing framework |
|
| Scalene | A high-performance CPU and memory profiler | |
| pudb | A console debugger | |
| lptrace | A pretty amazing single-file strace-like tool for Python | |
| PrettyErrors | an error handler that prettifies output | |
| assertEquals | a testing framework with a great screencast | |
| Electronics | pinout | Helps create SVG pin-out diagrams for electronic hardware |
| Enterprise | toil | A cross-platform pipeline management system with Common Workflow Language support |
| Environments | snakeware | a Python userspace for Linux |
| Functional Programming | Hy | An awesome LISP dialect that sits atop the runtime. |
| funcy | A nice, fairly complete functional library | |
| Toolz | A functional complement to the standard library. | |
| pyrsistent | A persistent data structure library | |
| GIS | geopy | an amazingly flexible geocoding library. |
| prettymaps | A minimal Python library to draw customized maps from OpenStreetMap data | |
| reverse-geocoder | A fast offline reverse geocoder | |
| GUI | kivy | A sophisticated UI/UX library with Android support |
| PythonCard | A bit basic, but interesting. | |
| VPython | 3D OpenGL libraries for quick visualization | |
| winGuiAuto.py | for Windows automation | |
| pyglet | a cross-platform windowing and multimedia library with OpenGL support | |
| PySide | A recent Qt binding (up to and including 4.7) with QML support | |
| fman build system | a complete build system for Qt applications | |
| PySimpleGUI | a simple tkinter-based GUI wrapper | |
| Gooey | a wxWidgets wrapper for quickly building GUI versions of CLI tools | |
| DearPyGui | a Linux/Windows GUI library atop ImGUI (C++) | |
| Toga | a cross-platform GUI that works on the Mac | |
| encodermenu | a MicroPython library for GUI creation, using uasyncio | |
| Graphics | smc.freeimage | A wrapper for the freeimage library that can handle various TIFF and fax formats as well as ICC color profiles |
| Graphite | a real-time graphing system similar to RRDTool |
|
| Skip's Python Bits | lots of useful snippets | |
| Pygal | An SVG charting library with a variety of built-in formats. | |
| Pyx | Oldie, but goodie. | |
| pygame | a game-oriented library with SDL support | |
| blurhash | A way to encode image placeholders as strings | |
| pyprocessing | A Processing clone in Python. | |
| douglas-quaid | a library for image matching and correlation | |
| Vapory | A library to generate photorealistic renderings using POV-Ray | |
| drawille | Uses Braille characters for TTY charting | |
| hiplot | high dimensional interactive plotting | |
| imagededup | uses several different algorithms (including NNs) to detect duplicate images | |
| diagrams | a library for drawing cloud architecture diagrams (Azure, AWS, k8s, etc.) | |
| plottyprint | printable formatting for matplotlib | |
| image-blender | a Cython implementation of Adobe Photoshop blend modes | |
| taichi | a high-performance graphics library | |
| Cropmap | Face-aware image cropping. | |
| HTML | lxml | a different libxml binding, with excellent HTML parsing performance |
| cssutils | a CSS parser | |
| Gnosis Utilities | all sorts of XML goodies | |
| pygments | syntax highlighter able to handle multiple nested languages | |
| pyquery | A jQuery-like library for manipulating documents. | |
| BeautifulSoup | The most powerful HTML parser out there. | |
| squeezeit | A CSS and JavaScript bundler/minifier | |
| Libraries | Sorted Containers | a fast C-backed collections library |
| seedir | A utility library to render filesystem tree diagrams | |
| minotaur | an asynchronous inotify interface | |
| minibatch | a simple stream processing library | |
| Machine Learning | Perceptilabls | free visual ML model creation tool |
| netron | a viewer for neural network, deep learning and machine learning models | |
| discomll | A ML framework for Disco | |
| Tensorflow | ||
| aiosmtpd | an asyncio version of smtpd.py | |
| imapautofiler | an IMAP mailbox sorter that I unwittingly replicated (with less features) | |
| Salmon | an SMTP mail server | |
| slimta | a mail app framework for building MTAs | |
| Music | pippi | an audio processing/music library |
| pymidi | a library for building MIDI over IP/RTP applications (compatible with Apple specs) | |
| Networking | PyRendezvous | neat little Bonjour module I use a lot |
| telnetscript | a simple module to do scripted telnet sessions. |
|
| IPy | an IP address manipulation library. | |
| pyvnc | ctypes interface to the VNC libraries |
|
| ircutils | A relatively modern async IRC framework | |
| Eventlet | Amazing event-oriented framework | |
| ftputil | a high-level interface to the ftplib module |
|
| Python smtplib progress indication | Might come in useful some day. | |
| gunicorn | A WSGI server that supports a number of different worker daemons and is highly customizable. | |
| dpkt | packet creation and parsers | |
| Pcapy | a packet manipulation library | |
| uwsgi | A C application container that speaks WSGI and is seriously good. | |
| PuSSH | SSH wrapper | |
| RPyC | an awesome RPC library | |
| SleekXMPP | A more modern XMPP library | |
| Tftpy | a pure Python TFTP library, useful for implementing UDP file transfer testbeds. | |
| ThriftPy | A pure Python implementation of Apache Thrift | |
| pyxmpp2 | another one, probably more interesting. | |
| pyrobuf | a Cython implementation of Google's protobuf library |
|
| asphalt | An asyncio framework for network oriented applications, with a considerable number of modules |
|
| pyapns | An Apple Push Notification Service provider based on Twisted. | |
| opendrop | a partial implementation of AirDrop | |
| Kiva | a Display PDF library | |
| PyPdf | a PDF toolkit | |
| WeasyPrint | Alternatives to PrinceXML for generating formated PDF documentation. | |
| camelot | a library to extract tabular data from PDFs | |
| xhtml2pdf | Alternatives to PrinceXML for generating formated PDF documentation. | |
| borb | a library for creating and manipulating PDF files | |
| Packaging | PyOxidizer | a better way to distribute Python bundled apps than pyinstaller |
| shiv | builds self-contained applications for easy deployment | |
| Poetry | a dependency management and packaging tool | |
| Presentations | pydown | An HTML presentation generator |
| Reporting | Relatorio | A very neat reporting library with multiple output formats |
| ReportLab Toolkit | a pure Python PDF library that includes a presentation tool. | |
| Robotics | kabuki | A declarative control library for MicroPython |
| Runtimes | Nuitka | An interesting batch compiler. |
| Pyjamas | Some interesting twists on using Python on the way to JavaScript. | |
| Brython | a fairly complete in-browser runtime | |
| Skulpt | Some interesting twists on using Python on the way to JavaScript. | |
| tinypy | Python in 64K of code. | |
| Js2Py | A JavaScript interpreter written in pure Python | |
| Hope | A specialized JIT compiler | |
| Pycopy | a fork of MicroPython | |
| Pyjion | Pyjion, a JIT extension for CPython that compiles your Python code into native CIL and executes it using the .NET 5 CLR. | |
| Scraping | Mechanize | a programmatic web browser for screen scraping. |
| subliminal | A library for looking up movie subtitles in various languages. | |
| pholcidae | A web crawler library. | |
| InvoiceNet | a neural network for extracting invoice information from PDFs | |
| urlwatch | A flexible page modification monitor | |
| Site Generators | portray | a static site generator for Python project documenntation |
| Task management | doit | a task runner that can replace make for many use cases |
| Tutorials | ugit | a step-by-step git implementation |
| Unicode | Unidecode | Does character transliterations. |
| Video | webcam-filters | a set of GStream filters for doing live webcam background segmentation and blurring |
| Visual Programming | PyFlow | a another (confusingly, identically named) visual programming environment |
| Ryven | a flow-based programming environment with a Qt front-end | |
| PyFlow | a desktop-based visual programming environment | |
| Web Frameworks | Bottle | Tiny, flexible, awesome. My favorite since 2011. |
| Draco | Old and busted, to various extents. | |
| Falcon | A Cythonizable framework with little magic and good performance | |
| Karrigell | Old and busted, to various extents. | |
| pecan | Based on object dispatching | |
| eve | Another RESTful foundation framework | |
| wheezy.web | A high-performance framework that seems to get along well with PyPy | |
| Snakelets | A minimalistic app server, my old web platform of choice. | |
| Twisted | Old and busted, to various extents. | |
| medusa | ||
| CherryPy | ||
| Django | The new hotness. | |
| Tornado | non-blocking, which is pretty interesting. | |
| Zope | Old and busted, to various extents. | |
| appkernel | a microservices framework | |
| blacksheel | an ASGI web framework that uses decorators | |
| sandman | A simple RESTful wrapper | |
| zatt | An asyncio implementation of the Raft consensus algorithm (some of the forks add significant enhancements) |
|
| FastAPI | decorator-based, runs under uvicorn | |
| Vibora | An asyncio framework that seems to beat Sanic in static performance |
|
| django-micro | A lightweight wrapper to use Django as you would Bottle | |
| Moya | a new(ish), template-centric web framework with some interesting twists | |
| starlette | an ASGI web framework that runs under uvicorn |
Techniques
- daemon.py, an example daemon script.
- Pydoc - built-in Web help, anytime, anywhere
- How to Write a Spelling Corrector - an interesting technique that can come in handy to fix/suggest search terms, etc.
- Python Webcam Color Track
- Python webcam fun - motion detection
Notes:
Setting distutils paths in OS X:
Create a file named .pydistutils.cfg in your home directory with the
following contents and set $PATH accordingly:
[install]
install_lib = ~/Library/Python/$py_version_short/site-packages
install_scripts = ~/Library/Python/$py_version_short/bin
install_data = ~/Library/Python/$py_version/share
This will install packages to your Library folder rather than to the
system directory, saving you the trouble to use sudo or risking
messing with the system installation. Of course virtualenv does a
better job for keeping track of project-specific dependencies, but this
is great for common/complex dependencies and is sure to survive
operating system upgrades.
And then you can use easy_install or pip install --user with
impunity, although you might need to do something like:
$ ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future easy_install pandas
…for getting around some clang oddities.
Instant Web Server on port 8000 (2.x):
#!/bin/sh
python -c '__import__("SimpleHTTPServer").test()'