What makes for a good software project, from an engineering standpoint? There are projects that are a joy to work on, and others that are a drag. The industry has accumulated enough wisdom to be able to name a few practices that can help us keep our projects in the former category. These are a handful of my favorite practices and tools for maintaining the joy.
If you'd like to check out an example project preconfigured for these practices and tools, help yourself!
Development and Production Parity
The developer's everyday working environment should be as close as possible to the environment in which the production application will run. The most subtle of differences between dependency versions or configuration can result in a feature that "works fine on my machine," but fails in production. To prevent this scenario, we can rely on tools to ensure that things match up.
Dependency Management
I'm a fan of the asdf version manager for its light weight, ease of use, cross-platform implementation, and wide range of supported tools. It allows us to install multiple isolated versions of programming languages, databases, and other tools. Then, for each project, we can list the desired version of each dependency in a single file. In the past, there have been several popular tools for managing any given language or tool, requiring knowledge, installation, and management of each one. It's great how asdf provides a single interface for managing many dependencies at once.
I have yet to work on a project where asdf would not be a good fit, but for projects where dependency management or deployment strategy are considerably complex, something more heavyweight such as Docker may be appropriate — although it brings additional complexity along with it.
Environment Configuration
Once we have our dependencies installed, we need to configure each environment in which our application will run, such as development, test, and production. This usually takes the form of specifying numeric and string values somewhere. Similarly to dependency management, configuration should be as similar as possible across environments, but things will naturally diverge due to the sensitive nature of production data.
Many programming languages have packages that will load configuration for use by that language's runtime. Each of these packages generally have their own conventional locations for storing configuration. They also often work only within the context of a running application or test suite, and are not loaded in shell sessions for use by ad hoc tooling.
Every test and production environment will have its preferred avenue of configuration, but for development, I like to use a tool called direnv. It's similar to asdf in that it's ultimately tool agnostic. direnv allows us to work efficiently in a multi-tool environment by providing a single interface to manage configuration. Variables are also automatically loaded into shell sessions without additional action on the part of the developer, and it's cross-platform.
Having dependency and configuration strategies across environments is great, but how do we set up a development environment to begin with? In a perfect world, the answer is: by running a single script at the command line.
Development Environment Setup
For someone new to a project — whether that's a developer, a quality engineer, or a project manager wanting to contribute some copy changes — the onboarding experience often goes as follows:
New person (Newb): I'm excited to get started!
Existing developer (Dev): Great, we sure could use your help! Everything you need to get set up is in the
[Five minutes later...]
Newb: The
Dev: Oh, right. OK, look up tool Y and install it. Then you should be good to go.
[Five minutes later...]
Newb: OK, I did that, but now I'm getting error Z.
Dev: Oh, right. To get around that you need to talk to so-and-so.
[Continue this process over the next several days, until the new person is set up.]
This common scenario hardly represents a pleasant onboarding experience, and while unfortunate, is avoidable. Setting up a project is a matter of providing a computer a series of instructions, and computers are happy to carry out those instructions, whether they come from a user typing step by step over days, or from a single pre-written script that takes a minute to run.
It takes work to write and maintain such a script, but it's well worth the effort for the ability to quickly bring new people onto a project, or to set up a new machine should someone's get taken out of commission. It also sometimes happens that work on a particular project is suspended for a time, only to resume months (or years) later. A well-maintained setup script serves as valuable project documentation, as well as insurance against being blocked from getting up and running again since so-and-so, who knew how to get around error Z, has in the interim moved to an off-the-grid yurt in Saskatchewan.
Testing
Testing is a deep topic, so here it will mostly suffice to say that testing is good, and we want to make it easy to do, and do often. However, testing is only as good as its return on investment. Some simple criteria can point us in the right direction toward that end.
It should be easy to test the application at every individual level, whether that's interaction with a database, the business logic that lives in a server application, or the user interface in a front-end web application. It's up to the developers to determine the nature and distribution of tests at any particular level.
Of critical importance is also the testing of the entire application, end-to-end. Teams will sometimes get in a routine of only writing tests at one or two isolated levels, such as database reads and writes, or tests that run against generated UI snapshots. However, unless end-to-end tests of an application's critical paths are written and maintained, the only way to be sure that all of the levels work together is by manually exercising their integration via the application's interface. Having to do so is much more costly than the work that goes into a test suite that can be run quickly during the development process, or just prior to merging new code. Such a test suite's ability to provide confidence in code and catch bugs makes it one of the most valuable things a team can have.
The more lightweight, fast, and portable an end-to-end test suite is, the more likely it is to be run, and thus deliver value. For this reason, whenever possible, the end-to-end suite should be able to run in an individual developer's working environment, with a minimum of dependence on external systems or network overhead (though some projects and tests will necessitate such added complexity for adequate testing confidence).
There are many tools to help us write and run tests at every level, including end-to-end. Some are focused directly on a particular programming language ecosystem, encouraging developers to write in the same language for both code and tests, although with others it's not necessary that this be the case.
Formatting
Writing code can be an aesthetic experience on many levels, and engineers naturally develop stylistic preferences for the layout of code. I certainly have my preferences, and can enjoy thoughtful conversation with others about theirs. However, it's easy to get carried away with these discussions, detracting from time that could be spent collaborating on more meaningful issues. Above all, the layout of code should serve to aid its readability. Thankfully, there are robust communities of people who spend time working out mutually agreeable ways to lay out the code of a particular language in service of readability and ease of understanding. These open source forums are the best places to have stylistic discussions. We should engage with and support these communities, while being mindful of working in the context of building products and services.
When all the code in a project follows the same established formatting rules, collaboration on code behavior can hang on an identical structural latticework across the whole team. For that reason, I'm a proponent of integrating automatic code formatting — with a minimum of custom configuration — into the development and continuous integration processes. There are tools available for this purpose for the majority of popular languages, and some tools, such as Prettier, can handle multiple languages and formats.
Commit History
A growing software project is constantly changing. As design decisions are made and implemented, a team builds a history of shared knowledge. At the code level, much of this knowledge can be stored in the metadata that accompanies source control. Teams that utilize this metadata well benefit from a readily accessible way to re-visit the moments and context in which important decisions were made. Rather than trying to coax the reasoning for a particular change out of memory, developers can instantly see messages from their past selves that describe the change, as well as see all related changes made across the codebase at the time. They can rewind and advance the evolution of the code step by step like time travellers.
This ability is only as valuable as the quality of the team's source control hygiene, which encompasses many factors. One of these is the care that developers take in the writing of their commit messages. To help preserve the quality of commit messages, we can use tools to encourage developers to follow conventions to that end. Conventional Commits is a specification that guides us toward a concise yet illustrative format for commit messages. Much like formatting code itself, using a common format for commit messages helps us to favor time spent on informational content over structure. Tools such as pre-commit can integrate the Conventional Commits spec directly into our source control workflow.
Continuous Integration and Deployment
The software we write can only deliver value if we can put it in front of users feature-complete and bug-free. The more seamless this process, the more often we can confidently deliver value. Continuous integration and deployment workflows support this goal.
I like to use the suite of services provided by Heroku as an integrated hosting environment for these workflows. It allows us to set up isolated testing environments, called [review apps], for each individual code change tied to a GitHub [pull request]. Using this feature, engineers can easily develop, test, and show their work to others, without interfering with the work of team members or having to coordinate the use of shared environments. When trying out the smallest change is only a pull request away, the friction of change is minimized and teams can iterate faster.
Heroku also provides a [testing service] that creates similarly isolated environments for running the project's test suite on every push to a GitHub branch. Together with its hosting platform for staging and production environments, including application servers and databases, it can serve as a one-stop-shop for continuous integration and deployment. Going this route avoids the need for a team to initially integrate and continually interface with several different service providers.
Really Though
Many books have been written about each of the topics we've discussed. The truth is that the main points and types of tooling contained herein are the easy things. They are the must-haves on a modern project. Once these are taken care of, what really concerns us in the making of a good project from an engineering perspective, is the skillful balancing of tradeoffs in the service of delivering value to people.