The Tao of Mac
Python is one of my favorite programming languages due to its terseness and amazing flexibility.
See also Django for more.
My Stuff:
| Year | Item | Notes |
|---|---|---|
| 2016 | Sushy | my current wiki engine |
| 2015 | rss2imap | The way I used read my news a while back |
| 2009 | PNG Canvas | a native Python PNG creation module, suitable for use in Google App Engine |
| 2007 | Yaki | my previous wiki engine |
| 2006 | MailArchive | a quick hack to file web pages as MIME compliant e-mail messages. |
| 2005 | PicoStats | a minimalist Apache log parser. |
| IMAP Backup | another quick hack that apparently filled a long-standing need. | |
| The Python Grimoire | which I converted into TiddlyWiki format | |
| 2004 | PicoRendezvous.py | a minimalist (and dumb) Bonjour / Rendezvous client |
| newspipe | which I used daily for a couple of years and occasionally contributed to | |
| Growl native protocol implementation | this spawned numerous variants, including a proxy. |
Resources:
| Category | Link | Notes |
|---|---|---|
| Authentication | aioauth | Oauth 2.0 framework and provider for asyncio |
| CLI | bpython | My favorite REPL |
| click | A very nice way to do composable CLI tools (see also argh, defopt, docopt, python-fire, etc.) | |
| Clint | A great module for developing CLI apps | |
| fabulous | Prints images to console | |
| pbs | A nice, simple and consistent generic command wrapper. | |
| python-fire | a CLI wrapper for any Python object | |
| ptpython | The Python Prompt Toolkit | |
| typer | A CLI tool like click that uses Python type hinting | |
| plumbum | Another command wrapper, with SSH support. | |
| rich | a console library with various layout options | |
| CMS | Wagtail | Another Django-based CMS framework |
| wid.gy | A Django-based CMS framework | |
| MiniConf | a lovely site generator for managing conferences | |
| Compilers | mypyc | a compiler for a subset of type-annotated Python |
| Transcrypt | Another JavaScript transpiler | |
| Concurrency | OpenReplica | A coordination service that provides replication and synchronization support for large-scale distributed systems |
| python-rq | A simpler queueing library | |
| Pulsar | Another actor-based multiprocessing library with asyncio
support and a fair degree of network transparency
|
|
| Celery | My current go-to library for task queueing | |
| Pykka | An implementation of the Actor model, with gevent support | |
| aiomultiprocess | an asyncio Pool implementation to manage multiple processes with independent event loops | |
| offset | ports of the Go concurrency model | |
| aioprocessing | asyncio equivalents of the multiprocessing library |
|
| goless | ports of the Go concurrency model | |
| Littleworkers | Parallel job management without the hassles. | |
| Console | Colorama | Making ANSI color sequences work cross-platform (see also this gist). |
| pyp | a clever shell tool to do inline Python | |
| urwid | A curses based library for building console UIs |
|
| Data Handling | Happy | A Hadoop library to run map-reduce jobs via Jython |
| biplist | A library that can manage Mac
binary .plist files
|
|
| pyparsing | a handy, flexible parser library that does away with lex/yacc. | |
| snakebite | A native HDFS client to access the Hadoop datastore. | |
| xlwings | An Excel plugin | |
| Bounter | A fast probabilistic counter | |
| voluptuous | A data schema validation library. | |
| Little Ball of Fur | a graph sampling library that works with NetworkX | |
| ray | A distributed execution engine that implements a pandas clone |
|
| riptable | a multithreaded array processing library | |
| Schematics | A validation/schema handling library. | |
| fuzzywuzzy | A fuzzy string matching library | |
| PyCascading | A better way to write Hadoop jobs. | |
| xlwt | a library to parse Excel documents (cheatsheet). | |
| Data Science | SciPy | scientific (including plotting) libraries for Python |
| euporie | a text-based interface for Jupyter notebooks | |
| xeus-sql | a Jupyter kernel for SQL databases | |
| Database | pymssql | SQL Server and Sybase driver |
| FreeTDS | ||
| SQLAlchemy | tutorial | |
| Peewee | A small, very flexible ORM | |
| DuckDB | an embeddable SQL database with OLAP features | |
| montydb | a tiny MongoDB implementation in pure Python | |
| simple-graph | A simple graph database in SQLite | |
| tinydb | A (very) lightweight document oriented database | |
| Goatfish | An even smaller schemaless ORM | |
| Debugging | Eww | Insert a REPL into a running application |
| Pycallgraph | Generate a program flow graph via Graphviz | |
| Hypothesis | A very nice testing framework |
|
| pudb | A console debugger | |
| lptrace | A pretty amazing single-file strace-like tool for Python | |
| assertEquals | a testing framework with a great screencast | |
| Enterprise | toil | A cross-platform pipeline management system with Common Workflow Language support |
| Environments | snakeware | a Python userspace for Linux |
| Functional Programming | Hy | An awesome LISP dialect that sits atop the runtime. |
| funcy | A nice, fairly complete functional library | |
| Toolz | A functional complement to the standard library. | |
| pyrsistent | A persistent data structure library | |
| GIS | geopy | an amazingly flexible geocoding library. |
| reverse-geocoder | A fast offline reverse geocoder | |
| GUI | kivy | A sophisticated UI/UX library with Android support |
| PythonCard | A bit basic, but interesting. | |
| VPython | 3D OpenGL libraries for quick visualization | |
| winGuiAuto.py | for Windows automation | |
| pyglet | a cross-platform windowing and multimedia library with OpenGL support | |
| PySide | A recent Qt binding (up to and including 4.7) with QML support | |
| fman build system | a complete build system for Qt applications | |
| PySimpleGUI | a simple tkinter-based GUI wrapper | |
| Gooey | a wxWidgets wrapper for quickly building GUI versions of CLI tools | |
| DearPyGui | a Linux/Windows GUI library atop ImGUI (C++) | |
| Toga | a cross-platform GUI that works on the Mac | |
| Graphics | smc.freeimage | A wrapper for the freeimage library that can handle various TIFF and fax formats as well as ICC color profiles |
| Graphite | a real-time graphing system similar to RRDTool |
|
| Skip's Python Bits | lots of useful snippets | |
| Pygal | An SVG charting library with a variety of built-in formats. | |
| Pyx | Oldie, but goodie. | |
| pygame | a game-oriented library with SDL support | |
| blurhash | A way to encode image placeholders as strings | |
| pyprocessing | A Processing clone in Python. | |
| Vapory | A library to generate photorealistic renderings using POV-Ray | |
| drawille | Uses Braille characters for TTY charting | |
| hiplot | high dimensional interactive plotting | |
| diagrams | a library for drawing cloud architecture diagrams (Azure, AWS, k8s, etc.) | |
| Cropmap | Face-aware image cropping. | |
| HTML | lxml | a different libxml binding, with excellent HTML parsing performance |
| cssutils | a CSS parser | |
| Gnosis Utilities | all sorts of XML goodies | |
| pygments | syntax highlighter able to handle multiple nested languages | |
| pyquery | A jQuery-like library for manipulating documents. | |
| BeautifulSoup | The most powerful HTML parser out there. | |
| squeezeit | A CSS and JavaScript bundler/minifier | |
| Libraries | Sorted Containers | a fast C-backed collections library |
| zipfly | a library for doing on-the-fly zip compression with low memory footprint | |
| douglas-quaid | a library for image matching and correlation | |
| minotaur | an asynchronous inotify interface | |
| imagededup | uses several different algorithms (including NNs) to detect duplicate images | |
| TextHero | an NLP library for large-scale text processing | |
| pippi | an audio processing/music library | |
| minibatch | a simple stream processing library | |
| plottyprint | printable formatting for matplotlib | |
| Trio | a friendlier wrapper around asyncio for concurrent tasks | |
| Mongita | a MongoDB-like database library for in-process use | |
| Skyfield | An astronomical library for Python | |
| Machine Learning | netron | a viewer for neural network, deep learning and machine learning models |
| discomll | A ML framework for Disco | |
| Tensorflow | ||
| aiosmtpd | an asyncio version of smtpd.py | |
| Salmon | an SMTP mail server | |
| slimta | a mail app framework for building MTAs | |
| Music | pymidi | a library for building MIDI over IP/RTP applications (compatible with Apple specs) |
| Networking | PyRendezvous | neat little Bonjour module I use a lot |
| telnetscript | a simple module to do scripted telnet sessions. |
|
| IPy | an IP address manipulation library. | |
| pyvnc | ctypes interface to the VNC libraries |
|
| ircutils | A relatively modern async IRC framework | |
| Eventlet | Amazing event-oriented framework | |
| ftputil | a high-level interface to the ftplib module |
|
| Python smtplib progress indication | Might come in useful some day. | |
| gunicorn | A WSGI server that supports a number of different worker daemons and is highly customizable. | |
| dpkt | packet creation and parsers | |
| Pcapy | a packet manipulation library | |
| uwsgi | A C application container that speaks WSGI and is seriously good. | |
| PuSSH | SSH wrapper | |
| RPyC | an awesome RPC library | |
| SleekXMPP | A more modern XMPP library | |
| Tftpy | a pure Python TFTP library, useful for implementing UDP file transfer testbeds. | |
| ThriftPy | A pure Python implementation of Apache Thrift | |
| pyxmpp2 | another one, probably more interesting. | |
| pyrobuf | a Cython implementation of Google's protobuf library |
|
| asphalt | An asyncio framework for network oriented applications, with a considerable number of modules |
|
| pyapns | An Apple Push Notification Service provider based on Twisted. | |
| opendrop | a partial implementation of AirDrop | |
| Kiva | a Display PDF library | |
| PyPdf | a PDF toolkit | |
| WeasyPrint | Alternatives to PrinceXML for generating formated PDF documentation. | |
| camelot | a library to extract tabular data from PDFs | |
| xhtml2pdf | Alternatives to PrinceXML for generating formated PDF documentation. | |
| Reporting | Relatorio | A very neat reporting library with multiple output formats |
| ReportLab Toolkit | a pure Python PDF library that includes a presentation tool. | |
| Robotics | kabuki | A declarative control library for MicroPython |
| Runtimes | Nuitka | An interesting batch compiler. |
| Pyjamas | Some interesting twists on using Python on the way to JavaScript. | |
| Brython | a fairly complete in-browser runtime | |
| Skulpt | Some interesting twists on using Python on the way to JavaScript. | |
| tinypy | Python in 64K of code. | |
| Js2Py | A JavaScript interpreter written in pure Python | |
| Hope | A specialized JIT compiler | |
| Pycopy | a fork of MicroPython | |
| Pyjion | Pyjion, a JIT extension for CPython that compiles your Python code into native CIL and executes it using the .NET 5 CLR. | |
| Scraping | Mechanize | a programmatic web browser for screen scraping. |
| subliminal | A library for looking up movie subtitles in various languages. | |
| pholcidae | A web crawler library. | |
| InvoiceNet | a neural network for extracting invoice information from PDFs | |
| urlwatch | A flexible page modification monitor | |
| Tools | pyinfra | a fabric-like agentless automation tool |
| Scalene | A high-performance CPU and memory profiler | |
| imapautofiler | an IMAP mailbox sorter that I unwittingly replicated (with less features) | |
| pydown | An HTML presentation generator | |
| papermill | a way to parameterize and automate Jupyter notebooks | |
| PrettyErrors | an error handler that prettifies output | |
| portray | a static site generator for Python project documenntation | |
| Poetry | a dependency management and packaging tool | |
| Unicode | Unidecode | Does character transliterations. |
| Web Frameworks | Bottle | Tiny, flexible, awesome. My favorite since 2011. |
| Draco | Old and busted, to various extents. | |
| Falcon | A Cythonizable framework with little magic and good performance | |
| Karrigell | Old and busted, to various extents. | |
| pecan | Based on object dispatching | |
| eve | Another RESTful foundation framework | |
| wheezy.web | A high-performance framework that seems to get along well with PyPy | |
| Snakelets | A minimalistic app server, my old web platform of choice. | |
| Twisted | Old and busted, to various extents. | |
| medusa | ||
| CherryPy | ||
| Django | The new hotness. | |
| Tornado | non-blocking, which is pretty interesting. | |
| Zope | Old and busted, to various extents. | |
| appkernel | a microservices framework | |
| blacksheel | an ASGI web framework that uses decorators | |
| sandman | A simple RESTful wrapper | |
| zatt | An asyncio implementation of the Raft consensus algorithm (some of the forks add significant enhancements) |
|
| Vibora | An asyncio framework that seems to beat Sanic in static performance |
|
| django-micro | A lightweight wrapper to use Django as you would Bottle | |
| Moya | a new(ish), template-centric web framework with some interesting twists | |
| starlette | an ASGI web framework that runs under uvicorn |
Techniques
- daemon.py, an example daemon script.
- Pydoc - built-in Web help, anytime, anywhere
- How to Write a Spelling Corrector - an interesting technique that can come in handy to fix/suggest search terms, etc.
- Python Webcam Color Track
- Python webcam fun - motion detection
Notes:
Setting distutils paths in OS X:
Create a file named .pydistutils.cfg in your home directory with the
following contents and set $PATH accordingly:
[install]
install_lib = ~/Library/Python/$py_version_short/site-packages
install_scripts = ~/Library/Python/$py_version_short/bin
install_data = ~/Library/Python/$py_version/share
This will install packages to your Library folder rather than to the
system directory, saving you the trouble to use sudo or risking
messing with the system installation. Of course virtualenv does a
better job for keeping track of project-specific dependencies, but this
is great for common/complex dependencies and is sure to survive
operating system upgrades.
And then you can use easy_install or pip install --user with
impunity, although you might need to do something like:
$ ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future easy_install pandas
…for getting around some clang oddities.
Instant Web Server on port 8000 (2.x):
#!/bin/sh
python -c '__import__("SimpleHTTPServer").test()'