Infrastructure suggestions, part2: builds

We recently went through a phase where the nightly builds were generally deemed more stable than the last release. This created issues with people running into known bugs with the release and some issues with ongoing development since breaking a build we were recommending people run isn't really acceptable. I'd like to avoid this in the future.

I can see at least two ways to fix this. One is to change the release process so a single person isn't a bottleneck. Appointing a couple of deputy release engineers who know the process, or making the process public and letting the currently working developers decide when it should happen would do it.

Another is to provide more automated builds. Most of the open source projects I've been associated with have three levels of "blessed" build, not just two. They all had the a "recommended for production" build even if it wasn't as stable as a release. Then there's something similar to or nightly build: could well be broken in major ways, undocumented changes, and no guarantee that any feature in it would actually be there in the future. But in between those was a build that had more extensive testing, features that were definitely going to stay and almost certainly with the current implementation, and hence documentation on them. Adding a level (or even two) of automated builds would help with our issues. In particular, adding a build that requires someone to document changes before they were added to it would make the release process easier, since taking releases from it would make preparing documentation for a release easier.

My own choice is both. The published plan included three sets of automated builds designed to be compatible with each other. There was no release listed simply because I knew it would be a while before one was ready, and wanted to take the extra time to think about how they should be done.

I looked at travis-CI at one point. It certainly had a time-based build, in that you could trigger a test for new commits at a fixed time every day.

Are the build logs going to be available? Or will a build failure trigger a notice to the developer that gets "blame" fingers as the author of the error (the CI systems I've worked with didn't trigger on every commit - they needed a period with no new code before that happened, so it wasn't uncommon to see multiple new commits in a single build).