Posts

Showing posts from February, 2017

setup.cfg - a solution to python config file soup? A howto guide.

Sick of config file soup cluttering up your repo? Me too. However there is a way to at least clean it up for many python tools. Some of the tools you might use and the config files they support... flake8 - .flake8 , setup.cfg , tox.ini , and config/flake8 on Windows pytest - pytest.ini , tox.ini , setup.cfg coverage.py - .coveragerc , setup.cfg , tox.ini mypy - setup.cfg , mypy.ini tox - tox.ini  Can mypy use setup.cfg as well? OK, you've convinced me. -- Guido With that mypy now also supports setup.cfg, and we can all remove many more config files. The rules for precedence are easy: read --config-file option - if it's incorrect, exit read [tool].ini - if correct, stop read setup.cfg   How to config with setup.cfg? Here's a list to the configuration documentation for setup.cfg. coverage config pytest config flake8 config mypy config behave What does a setup.cfg look like now? Here's an example setup.cfg for you with various too...

Is Type Tracing for Python useful? Some experiments.

Type Tracing - as a program runs you trace it and record the types of variables coming in and out of functions, and being assigned to variables. Is Type Tracing useful for providing quality benefits, documentation benefits, porting benefits, and also speed benefits to real python programs? Python is now a gradually typed language , meaning that you can gradually apply types and along with type inference statically check your code is correct. Once you have added types to everything, you can catch quite a lot of errors. For several years I've been using the new type checking tools that have been popping up in the python ecosystem. I've given talks to user groups about them, and also trained people to use them. I think a lot of people are using these tools without even realizing it. They see in their IDE warnings about type issues, and methods are automatically completed for them. But I've always had some thoughts in the back of my head about recording types at run...

Is PostgreSQL good enough?

Image
tldr; you can do jobs, queues, real time change feeds, time series, object store, document store, full text search with PostgreSQL. How to, pros/cons, rough performance and complexity levels are all discussed. Many sources and relevant documentation is linked to. Your database is first. But can PostgreSQL be second? Web/app projects these days often have many distributed parts. It's not uncommon for groups to use the right tool for the job. The right tools are often something like the choice below. Redis for queuing, and caching. Elastic Search for searching, and log stash. Influxdb or RRD for timeseries. S3 for an object store. PostgreSQL for relational data with constraints, and validation via schemas. Celery for job queues. Kafka for a buffer of queues or stream processing. Exception logging with PostgreSQL (perhaps using Sentry) KDB for low latency analytics on your column oriented data. Mongo/ZODB for storing documents JSON (or mangodb for /dev/null replacemen...

Gradual Packaging, with python - Part two

Gradual Packaging - allow packaging simple python code with very little effort. Later on (if needed) you can create a more polished package. Follow conventions, support the simple cases. If you need to go out of the simple cases, allow a path towards better packaging. Evolution of your python code. in your ipython session messing around. paste it into badnamechoice.py -> ( everything starts with a bad name in a single .py file ) test_badnamechoice.py -> ( then you add a test. Or did you do this first? ) renamed.py -> ( now you rename the file to something better ) afolderoffiles/ -> ( now there is a folder of files ) add docs/ and .travisci CI files configure coverage testing configure an annoying mypy static type checking option add flakes testing config tweak support some weird debian derived distro appveyor CI added next to travisci pytest config ( change because... ARG reasons) . add a requirements.txt, and modify your setup file to have them too. rem...