tyler butler

Thanksgiving Tradition: Alice’s Restaurant Massacre

I have a tradition on Thanksgiving: I listen to Arlo Guthrie’s classic Alice’s Restaraunt Massacre; all 18 minutes and 37 seconds of it. I don’t remember exactly where I picked up this tradition, but it feels distinctly mine since it’s not something that my family did growing up.

Anyway, I found out yesterday that the song is available on Spotify in its full-length glory. So kick back, close your eyes for 20 minutes, and enjoy.

Documentation, Versions, and Read the Docs

Engineer is a side project for me right now. That means that while I am actively working on Engineer pretty regularly, releases themselves are not necessarily regular. I’ve adopted a repository/branch structure that’s influenced by git flow, so my main development branch is not master, it’s dev.

  • master: The most recent released version of the code
  • dev: The in-development version of the code

Up until Engineer v0.5.0, when you went to GitHub, you saw master by default.

When you go to GitHub, I want you to see the latest in-development version. The reason is pretty simple: Since official releases are fairly slow, but I actually make changes fairly often, I want to make sure that activity is shown on the GitHub homepage – via the ‘x days ago’ text that shows up on the far right of the code listing. My hypothesis is that people make some judgements based on the activity level of project. If people are searching for a static site generator, and they come across Engineer, I don’t want them to think that the project is abandoned and simply leave. If my default branch shown in GitHub is master, then it looks as though the project isn’t under active development at first glance, which clearly isn’t what I want. Thus, I want GitHub to display the dev branch by default, which is easy enough to change in the repository settings. I made this change along with the release of Engineer v0.5.0, so now when you go to the repository on GitHub, you’ll see the dev branch by default.

There’s still a problem, though, related to the fact that I host Engineer’s public documentation on Read the Docs (RTD). Imagine someone finds my project on GitHub, likes what they see, and installs the release version using pip:

pip install engineer

Seems to be the natural thing to do, right? If they click the link in Engineer’s README to visit the docs at RTD, then I want them to go to the version of the docs that corresponds to the version they just installed – the most recent released version.

Now this is also relatively easy to do. RTD actually has some smarts around multiple versions of documentation. RTD offers a few different settings that are relevant to my goals.

First, there is a baked in ‘version’ identifier called latest which is intended to point to the most recent version of your docs:

In the normal case, the latest version will always point to the most up to date development code. If you develop on a branch that is different than the default for your VCS, you should set the Default Branch to that branch.

Of course, in my case, development is done on the dev branch, so I want latest to point to that branch. Fortunately that’s easy to change, as the second sentence above alludes to. In the Advanced Settings section of the RTD dashboard, you’ll find a Default branch setting, in which I entered dev. Great; now latest points to dev.

The second setting of relevance in RTD is the default version. This controls what version of your docs / redirects to. By default this will be latest, but since I want / to always redirect to the most recent released version of Engineer, I changed this to master. Cool; now / simply redirects to the version of my docs from the master branch, which will always be the most recent released version.

There is, of course, still a problem. Ideally, I would like links that people follow to go to the version of the documentation that matches the version of the code they’re coming from and vice-versa. In other words, I would like the documentation link from the README file in the master branch to go to https://engineer.readthedocs.org/en/master/, and the link in the dev branch to go to https://engineer.readthedocs.org/en/latest/.

Unfortunately, that’s not really possible. Sure, I could build some intelligent redirector or something that would look at the referrer URL and redirect to the appropriate docs version, but that’s not something I want to build anytime soon. The best I can do is provide some notes in the documentation itself telling people that they may be looking for a version of documentation that is different from what they’re seeing. It’s not quite ideal from my perspective, but I think it helps.

So bottom line, this is what I’ve wound up with:

  • If you go directly to https://engineer.readthedocs.org/, you’ll get the latest released version of the docs, which will correspond to what most people will install using pip.
  • If you visit the GitHub repo, you’ll see the most recent in-development version of the code. The README links to https://engineer.readthedocs.org/, which as I mentioned earlier will redirect to the released version of the docs.
  • The docs themselves contain notes redirecting people to https://engineer.readthedocs.org/en/latest/ if they need the most recent version of the docs. RTD itself also contains links to all versions of the docs, but I don’t think most people know that and if they do, it may not be clear which version they want.

Installing Binary Python Packages on Windows

Updated December 31, 2014

In my Python Windows installation guide, I concluded with the following paragraph:

After all of that’s done you should be good to go! You can pop open a PowerShell window and create/switch to virtualenvs as needed and install packages using pip. At this point you should have most of what you need to follow the installation instructions for most Python packages (except those that require C extension compilation, but that’s a topic for another post).

Despite writing the initial version of that guide over two years ago, I never got around to writing that ‘other post’ to cover installing packages that require C extension compilation. I personally rarely run into this need, but when it comes up it’s incredibly annoying. And guess what? It came up recently when I tried to install Dulwich, a Python implementation of Git. Fortunately for you, I decided to take this opportunity to actually write the guide.

Do I Need This Guide?

You only need this guide if you try to install a Python package on Windows and you get an error like this:

building 'Crypto.Random.OSRNG.winrandom' extension
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
error: Unable to find vcvarsall.bat

Ahhh, the dreaded Unable to find vcvarsall.bat error… This error means that the package you’re installing has a C extension that needs to be compiled. Python itself is compiled using a specific version of the Visual Studio C++ compiler, and when you try to install packages that require C compilation, it goes looking for the compiler locally so it can compile the necessary stuff.

Of course, in your case, you probably don’t have Visual Studio, or if you do, it’s not the right version, or you don’t have the C++ compiler installed, or it’s installed in a non-standard location, or a specific environment variable isn’t set, or you forgot to reopen your PowerShell/cmd window after you set that environment variable… As you can see, there are many many reasons why this is painful. Don’t worry; you’re not alone. As this question on Stack Overflow indicates, lots of people run into this problem, and there are lots of ways to ‘solve’ it.1

Because of the relative complexity of the problem, and all the potential ways various solutions could be thwarted, this used to be annoyingly difficult to address, but now it’s pretty easy.

Installing the Microsoft C++ Compiler for Python 2.7

I am extremely glad I didn’t try to write this guide a few years ago, when I wrote my Python installation guide, because in the time since then, some smart person2 at Microsoft felt the collective pain and anguish of Python developers everywhere and made a package available directly on microsoft.com that “contains the compiler and set of system headers necessary for producing binary wheels for Python 2.7 packages.” Hooray!3

If you download that package and install it, you should be able to successfully install whatever package that was erroring out with Unable to find vcvarsall.bat before. Make sure you re-open any PowerShell or cmd windows you had open to make sure your environment variables are up to date. Oh, and in case you care, the compiler and all its supporting files can be found in the following directory after installation:

~\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0

If you’re still having problems, chances are your version of setuptools is out of date.

Update setuptools and pip

There have been a number of changes in the Python packaging/distribution world in the past few months. I’m not involved in any of the relevant projects, but since my installation guide is quite popular, I get emails from some folks every so often that are. One of the biggest changes is the reintegration of the Distribute fork of setuptools back into the main project. This also means that setuptools – the main project – is getting a lot more love, which means more updates.

The installation instructions for the Microsoft C++ Compiler for Python 2.7 package says that it requires setuptools 6.0 or later. I had a crusty old version from who knows when. Updating pip and setuptools is a little weird, but it’s not that difficult. I actually wrote a separate guide for that as part of the Engineer 0.5.0 documentation.

There are more details there, but it basically boils down to executing two commands: python -m pip install -U pip followed by pip install -U setuptools. When you do that, you should see some output like this:

C:\Users\Tyler\Code> python -m pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-6.0.2-py2.py3-none-any.whl#md5=26404d27a64a40d4c358a2405b16d043
Installing collected packages: pip
  Found existing installation: pip 1.5.2
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...

C:\Users\Tyler\Code> pip install -U setuptools
Collecting setuptools from https://pypi.python.org/packages/3.4/s/setuptools/setuptools-8.2.1-py2.py3-none-any.whl#md5=a0582adbe0c56b3945570049b8d7c953
  Downloading setuptools-8.2.1-py2.py3-none-any.whl (551kB)
  100% |################################| 552kB 975kB/s ta 0:00:01
Installing collected packages: setuptools
  Found existing installation: setuptools 2.2
    Uninstalling setuptools-2.2:
      Successfully uninstalled setuptools-2.2

Successfully installed setuptools-8.2.1

Congratulations, your pip and setuptools installations are now upgraded. As I note in the Engineer 0.5.0 upgrade guide, “if you’re using virtualenv, you may need to upgrade pip and setuptools in your virtualenv as well as the ‘global’ (outside the virtualenv) versions.” You should be able to avoid doing this for all new virtualenvs by upgrading virtualenv itself (pip install -U virtualenv – version 12.7.8 is the latest as of this writing). Once it was upgraded my new virtualenvs got the correct updated versions of pip and setuptools. If you don’t want to recreate your virtualenvs, then you can just upgrade the ones you need.

Once pip and setuptools are upgraded, try installing the previously failed package again. You should see a bunch of output like this:

C:\Users\Tyler\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\Libs /LIBPATH:C:\Users\Tyler\.virtualenvs\test\libs /LIBPATH:C:\Users\Tyler\.virtualenvs\test\PCbuild /EXPORT:init_diff_tree build\temp.win32-2.7\Release\dulwich/_diff_tree.obj /OUT:build\lib.win32-2.7\dulwich\_diff_tree.pyd /IMPLIB:build\temp.win32-2.7\Release\dulwich\_diff_tree.lib /MANIFESTFILE:build\temp.win32-2.7\Release\dulwich\_diff_tree.pyd.manifest
   Creating library build\temp.win32-2.7\Release\dulwich\_diff_tree.lib and object build\temp.win32-2.7\Release\dulwich\_diff_tree.exp
Successfully installed dulwich-0.9.8

Congratulations, you can now install source Python packages that include C extensions (like Dulwich)!

The Future: Binary Wheels

Now, the fact that you need to install some separate dependency on Windows in order to install some Python packages clearly sucks. Fortunately, there are ways that package distributors can remove this need. There is a new package format, called Wheel, which includes pre-compiled versions of C extensions. If the Dulwich package maintainer produced a Wheel in addition to the source distribution, then users wouldn’t all need to install the Microsoft C++ Compiler for Python 2.7.4 Wheels can be installed using pip version 1.4+.

If you release packages on PyPI, consider creating a Wheel if your package has a C extension. There’s a great guide for distributing your Python projects, including creating Wheels, in the Python Packaging User Guide.

Addendum: dulwich-windows

In the event you’re using Windows and need Dulwich, but don’t want to fool with following the steps above, I forked the repository and published a Wheel (my first!) – it’s dulwich-windows on PyPI. The only change to the code is a few minor changes to the setup file to differentiate it from the official Dulwich package. You can see those changes on the windows_wheel branch in my fork. Feel free to install it (pip install dulwich-windows). I may or may not keep it up to date, though, so I recommend using the official Dulwich releases if possible.

This guide was last updated December 31, 2014. It was tested on a Windows 8.1 machine. If you notice any errors or missing/out-of-date information, please let me know at: __tyler AT tylerbutler DOT com__.

  1. For the record, I don’t recommend following most of the answers on Stack Overflow at this point. Many are quite old, and as you’ll see when you read on, there’s a much simpler solution. 

  2. Well, since it’s Microsoft, it was likely a whole team of people! 

  3. If you happen to know who at Microsoft was responsible for this, please let me know, because I want to buy them something nice. 

  4. I do think that the Wheel would need to be built on a Windows box with the compiler installed, though I am not sure about that. If I’m right, this would certainly be a blocker since many package maintainers don’t have easy access to a Windows box.