Refactoring Software: philosophical takes (scientific laws) to methodological approach (lean 5S)

By Adam McQuistan in DevOps  04/10/2024 Comment

Let’s unpack this term refactoring

The term software refactoring can be found as far back as the 1990s and refers to the act of changing software implementations with the goal of making it easier to understand as well as cheaper to maintain and evolve while still preserving the original intended behavior and exposed interfaces ^{[1, 2]}. However, the concept of software refactoring has matured significantly over the last 30-40 years due to the contributions of many thought leaders like Martin Fowler, Kent Beck, Robert Martin and Joshua Kerievsky to name just a few ^[1-3].

Software refactoring is an often used tool for paying down technical debt to reduce the cost of maintaining and modifying software. Here technical debt refers to the extra burden a development team experiences when introducing changes to a codebase due to prior prioritization tradeoffs favoring time-to-market over investments in modular, cohesive, reusable software design. In practice this often looks like kludgy quick fixes, workarounds, and one off patches over full-scale solutions.

My justifications for refactoring

I’ll give a few justifications for why I feel software refactoring is important, ranging from more philosophical (2nd law of thermodynamics) towards being backed by commonly accepted management principles (Lean management’s 5S). However, justifications for software refactoring, and more broadly addressing tech debt, have been fraught with discouragement and skepticisms. I see this largely due to scoping issues and insufficient clarity of intention, process, and opportunity costs of performing refactoring. In many cases refactoring efforts have left bitter tastes in the mouths of business leaders managing engineering investments who have previously been persuaded to bite off more than engineering resources can chew, which of course is a common source of most software failures. Any change in software whether enhancement, bug fix or refactoring, should always be small, well defined in scope, and introduced iteratively so it's no wonder when refactoring is proposed and undertaken in such big bang approaches the result is a big bang outcome in all the wrong ways. No matter the justification you settle on, be sure to limit scope and remember that practice is required to gain mastery. Practice is iterating until you get it right and this is what I refer to as the Practice of Software Engineering.

First justification for refactoring: 2nd law of thermodynamics predicts software decay

I’ll get the more idiosyncratic philosophical justification out of the way first which leans on the 2nd Law of Thermodynamics, perhaps with some liberties taken. The 2nd Law of Thermodynamics tells us energy must be expended to avoid the natural tendency of a system to decay towards disorder (aka, increased entropic randomness) overtime ^[4]. Essentially the scientific law states that without application of energy to impose order to the elements of a system it will decay overtime to disorder. I like to think of a software codebase as such a system. This is because my experience has shown that a codebase withheld from refactoring, which is an application of energy, will decay towards disorder. Refactoring, my friends, is all about imposing order on a codebase to reduce or avoid decay.

Signs and Symptoms of Software Decay

• Growing code duplication
• Growing complexity 
• Reduced understandability
• Reduced stability
• Reduced changeability

Causes of Software Decay

• Insufficient initial understanding of domain resulting in hard to maintain design lacking comprehensible boundaries of responsibility within the code
• Loss of conceptual integrity of design, domain boundaries, or responsibilities of code
• Insufficient establishment or enforcement of team standards or norms
• Sustained periods of rapid growth resulting in feature proliferation, new developers unfamiliar with domain or codebase, and attrition of existing developers resulting in loss of organizational knowledge pertaining to the domain and codebase

Second justification for refactoring: DevEx is Important to productivity

My second justification is centered on improving the developer experience by optimizing code for cognitive ease through refactoring which in turn has been shown to increase developer productivity, software quality, and reduce developer attrition ^[5-7]. The premise here is that given code is read many more times than its written then improving readability will naturally benefit the developer experience while also improving the code. This is a win win.

Final justification for refactoring: it aligns to lean management’s 5S 

My final justification is centered around a Lean Management method known as 5S which aims to optimize the workspace for productivity by making things easier to find, easier to access and easier to identify non-value adding items or movements by following a simple 5 step process ^[8]. If you consider the codebase to be the workspace of Software Engineers then these are powerful concepts to apply to the Practice of Software Engineering.  In fact, I’ve found the 5S method to be so useful I’ve incorporated it into a methodological approach or framework of thought for reading code and identifying what I like to call refactoring candidates.

Where does refactoring fit in Agile / SDLC?

My approach and advice to where refactoring fits into the planning and execution of software delivery is simple, I advise refactoring with purpose and refactoring continuously. This means every time you’re browsing code to implement a bug fix or new feature that should be an opportunity to identify potential code refactoring candidates. Any time when you have a reason to be reading code is an opportunity to identify refactoring candidates so, think of things like a PR review, code walk through, or pairing session as all great opportunities to start the refactoring dialog, internally or externally with other developers.

When I’m reading code I often tag refactoring candidate code blocks I observe with comments to keep track of them. Then when I have enough of an understanding of the changes I need to make to implement a bug or feature I take into account the potential refactorings I’ve identified as part of the design.  Which refactoring candidates are chosen to invest time and energy on depend on the following set of guidelines. 

1. isolate code investigation to the code I have a reason to be looking at to keep the area of scope small and bounded to the local code I’m working in
2. respect constraints by taking into account capacities like time, experience and skill
3. reach sufficient confidence identifying the functional scope within the codebase that minimizes and isolates a set of refactoring candidates while also maintaining overall behavior
4. reach sufficient confidence investments will add value to the codebase without introducing risk of unintended consequences

What exactly are refactoring candidates?

The “refactoring candidate” term I’ve been using is my preferred verbiage for the more common synonym of “code smell”.  While I do acknowledge that code smell is a clear descriptor for an area of code that indicates understandability challenges are present I’ve found the term refactoring candidate better facilitates team discussions especially where the author of said code small is likely to be present. In all my human experiences I’ve repeatedly observed that coachability and learning rate tend to track personal confidence which seems imminent to sink when one's intellectual work is being compared to a foul odor. Therefore, in the spirit of promoting a confidence building growth oriented culture I prefer speaking in terms of refactoring candidates over code smells. Readers can be comforted to find they can consult the catalog of refactorings for remediating many such code smells maintained by Martin Fowler representing the collective work of the earlier mentioned software thought amplifiers ^[10].

Identifying refactoring candidates using 5S principles

Engineering is primarily concerned with applying analytical methods to design solutions that increase productivity, or otherwise optimize processes and systems to be more efficient (9). Lean management similarly puts a focus on continuous improvement by applying analytical methods to eliminate waste from processes and systems thus it nicely aligns with Engineering. Having drawn this correlation between Engineering and Lean concepts I’ve learned to find ways to incorporate them symbiotically into my work habits. I’ve found that 5S can serve as a useful thought framework for both reading code and incorporating continuous refactoring into that process. 

1. Seiri (sort)
   • As you read software create an inventory of code elements and constructs around the area of investigation. This effectively sorts and categorized clean understandable code from lower value code that represent refactoring candidates
2. Seiton (set in order)
   • Look for code that could be reorganized or grouped into more cohesive units of shared or similar functionality that would improve flow readability by improving the order of code elements and constructs
3. Seiso (scrub or shine)
   • Look for elements of code that could be renamed to add clarity of intent effectively cleaning up the semantics of the software’s functionality
4. Seiketsu (standardize)
   • Look for patterns of functionality that emerge from the code which can be replaced by abstractions increasing modularity, consistency, and reusability in the form of standardized patterns
5. Shitsuke (sustain)
   • Ensure control measures for quality and architecture constraints are in place and automated through unit tests and architecture fitness functions
   • Prefer refactoring as a sustainable continuous improvement component of the Agile/SDLC delivery vehicle rather than infrequent big bang projects

References and resources for further learning

1. Martin Fowler. Refactoring: Improving the Design of Existing Code (2nd Ed.). Addison-Wesley Professional. 2018
2. Joshua Kerievsky. Refactoring to Patterns. Addison-Wesley Professional. 2004
3. Robert Martin. Clean Code: A handbook of Agile Software Craftsmanship. Pearson. 2008
4. “Laws of Thermodynamics” Wikipedia https://en.wikipedia.org/wiki/Laws_of_thermodynamics Accessed Apr 9, 2024
5. Forsgren, Nicole. Et al. DevEx in Action: A study of its tangible impacts. Amcqueue. January 2024. https://queue.acm.org/detail.cfm?id=3639443
6. Eirini Kalliamvakou. Yes, good DevEx increases productivity. Here is the data. GitHub Blog. January 2024. https://github.blog/2024-01-23-good-devex-increases-productivity/
7. Forsgren, Nicole. Et al. Accelerate: The science of Lean Software and DevOps; building and scaling high performing technology organizations. IT Revolution Press. 2018.
8. Masaaki Imai. Gemba Kaizen: A commonsense approach to a Continuous Improvement Strategy (2nd Ed.) McGraw Hill. 2012.
9. “Engineering” Wikipedia https://en.wikipedia.org/wiki/Engineering. Accessed Apr 9, 2024
10. Martin Fowler. Refactorings Catalog. https://refactoring.com/catalog/ Accessed April 9, 2024