5 Implementation

Now that you have a plan, it’s finally time to get started with the implementation.

From Design to Code: Fill in the Blanks

Armed with your design from the last chapter, you can now translate your sketch into a code skeleton. Start by outlining the functions, place calls to them where needed, and add comments for any steps you’ll figure out later. For example, the design from Figure 4.13 could result in the following draft:

import numpy as np
import pandas as pd

class MyModel:
    def __init__(self, param1):
        self.param1 = param1

    def fit(self, x, y):
        pass

    def predict(self, x):
        y = ...
        return y

def preprocess(df):
    df = ...
    return df

def load_data(file_name):
    df = pd.read_csv(file_name)
    df = preprocess(df)
    return df

def compute_r2(y, y_pred):
    r2 = ...
    return r2

def evaluate_plot(y, y_pred):
    r2 = compute_r2(y, y_pred)
    # ... create and save plot ...
    return r2

def main(model):
    df_train = load_data("train.csv")
    model.fit(df_train.x, df_train.y)
    df_test = load_data("test.csv")
    y_pred = model.predict(df_test.x)
    r2 = evaluate_plot(df_test.y, y_pred)

if __name__ == '__main__':
    # script is called as `python script.py seed param1 [param2]`
    seed, param1, param2 = ...
    np.random.seed(seed)
    # TODO: check which type of model should be created (mine or baseline)
    model = MyModel(param1)
    main(model)

Order of functions

Your script likely includes multiple functions, so you’ll need to decide their order from top to bottom. Since scripts typically start with imports (e.g., of libraries like numpy) and end with a main function, personally I prefer to put more general functions (i.e., the ones that are at the lower levels of abstraction in your call hierarchy and that only rely on external dependencies) towards the top of the file. This ensures that, as you read the script from top to bottom, each function depends only on what was defined before it. Maintaining this order avoids circular dependencies and encourages you to write reusable, modular functions that serve as building blocks for the code that follows.

Once your skeleton stands, you “only” need to fill in the details, which is a lot less intimidating than facing a blank page. Plus, since you started with a thoughtful design, your final program is more likely to be well-structured and easy to understand. Compare this to writing code on the fly, where decisions about functions are often made haphazardly—you’ll appreciate the difference.

Using AI code generators

AI assistants like ChatGPT, Claude, or GitHub Copilot can be helpful tools when writing code, especially at the level of individual functions. However, remember that these tools only reproduce patterns from their training data, which includes both good and bad code. As a result, the code they generate may not always be optimal. For instance, they might use inefficient for-loops instead of more elegant matrix operations. Similarly, support for less popular programming languages may be subpar.
To get better results, consider crafting prompts like: “You are a senior Python developer with 10 years of experience writing efficient, edge-case-aware code. Write a function …”

Breaking Code into Modules

Starting a new project often begins with all your code in a single script or notebook. This is fine for quick and small tasks, but as your project grows, keeping everything in one file becomes messy and overwhelming. To keep your code organized and easier to understand, it’s a good idea to move functionality into separate files, also called (sub)modules. Separating code into modules makes your project easier to navigate and can lay the foundation for your own library of reusable functions and classes, useful across multiple projects.

A typical first step is splitting the main logic of your analysis (main.py) from general-purpose helper functions (utils.py). Over time, as utils.py expands, you’ll notice clusters of related functionality that can be moved into their own files, such as data_utils.py, models.py, or results.py. This modular approach naturally leads to a clean directory structure, which might look like this for a larger Python project:¹

src/
├── main.py
└── my_library/
    ├── __init__.py
    ├── data_utils.py
    ├── models/
    │   ├── __init__.py
    │   ├── baseline_a.py
    │   ├── baseline_b.py
    │   ├── interface.py
    │   └── my_model.py
    └── results.py

In main.py, you can import the relevant classes and functions from these modules to keep the main script clean and focused:

from my_library.models.my_model import MyModel
from my_library.data_utils import load_data

if __name__ == '__main__':
    # steps that will be executed when running `python main.py`
    model = MyModel()

Keep helper functions separate

Always separate reusable helper functions from the main executable code. This also means that files like data_utils.py should not include a main function, as they are not standalone scripts. Instead, these modules provide functionality that can be imported by other scripts—just like external libraries such as numpy.

As you tackle more projects, you may develop a set of functions that are so versatile and useful that you find yourself reusing them across multiple projects. At that point, you might consider packaging them as your own open-source library, allowing other researchers to install and use them as well.

Keep It Compact

When writing code, aim to achieve your goals while using as little screen space as possible—this applies to both the number of lines and their length.

Tips to create compact, reusable code

Avoid duplication: Instead of copying and pasting code in multiple places, consolidate it into a reusable function to save lines (DRY principle).
Prefer ‘deep’ functions: Avoid extracting very short code fragments (1-2 lines) into a separate function, especially if this function would require many arguments. Such shallow functions with wide interfaces increase complexity without meaningfully reducing line count. Instead, strive for deep functions (spanning multiple lines) with narrow interfaces (e.g., only 1-3 input arguments, i.e., fewer arguments than the function has lines of code), which tend to be more general and reusable [6].
Address nesting: If your code becomes overly nested, this can be a sign that parts of the code should be moved into a separate function. This simplifies logic and shortens lines.
Use Guard Clauses: Deeply nested if-statements can make code harder to read. Instead, use guard clauses [1] to handle preconditions (e.g., checking for wrong user input) early, leaving the “happy path” clear and concise. For example:
```
if condition:
    if not other_condition:
        # do something
        return result
else:
    return None
```
Can be refactored into:
```
if not condition:
    return None
if other_condition:
    return None
# do something
return result
```
This approach reduces nesting and improves readability.

Documentation & Comments: A Note to Your Future Self

While you write it, everything seems obvious. However, when revisiting your code a few months later (e.g., to try a different experiment), you’re often left wondering what the heck you were doing. This is especially true when some external constraint (like a library quirk) forced you to create a workaround instead of opting for the straightforward solution. When returning to such code, you might be tempted to replace the awkward implementation with something more elegant, only to rediscover why you chose that approach in the first place. This is where comments can save you some trouble. And they are even more important when collaborating with others who need to understand your code.

We distinguish between documentation and comments: Documentation provides the general description of when and how to use your code, such as function docstrings explaining what the function computes, its input arguments, and return values. This is particularly important for open source libraries where you can’t personally explain the code’s purpose and usage to others. Comments help developers understand why your code was written in a certain way, like explaining that unintuitive workaround. Additionally, for scientific code, you may also need to document the origin of certain values or equations by referencing the corresponding paper in the comments.

Code should be self-documenting

Ideally, your code should be written so clearly that it’s self-explanatory. Comments shouldn’t explain what the code does, only why it does that (when not obvious). Comments and documentation, like code, need to be maintained—if you modify code, update the corresponding comments, or they become misleading and harmful rather than helpful. Using comments sparingly minimizes the risk of confusing, outdated comments.

Informative variable and function names are essential for self-explanatory code. When you’re tempted to write a comment that summarizes what the following block of code does (e.g., # preprocess data), consider moving these lines into a separate function with an informative name, especially if they contain significant, reusable logic.

Naming Is Hard

There are only two hard things in Computer Science: cache invalidation and naming things.
– Phil Karlton²

Finding informative names for variables, functions, and classes can be challenging, but good names are crucial to make the code easier to understand for you and your collaborators.

Tips for effective naming

Names should reveal intent. Longer names (consisting of multiple words in snake_case or camelCase, depending on the conventions of your chosen programming language) are usually better. However, stick to domain conventions—if everyone understands X and y as feature matrix and target vector, use these despite common advice denouncing single letter names.
Be consistent: similar names should indicate similar things.
Avoid reserved keywords (i.e., words your code editor colors differently, like Python’s input function).
Use verbs for functions, nouns for classes.
Use affirmative phrases for booleans (e.g., is_visible instead of is_invisible).
Use plurals for collections (e.g., cats instead of list_of_cats).
Avoid encoding types in names (e.g., sample_array), since if you decide to change the data type later, you either need to rename the variable everywhere or the name is now misleading.

Tests: Protect What You Love

We all want our code to be correct. During development, we often verify this manually by running the code with example inputs to check if the output matches our expectations. While this approach helps ensure correctness initially, it becomes cumbersome to recreate these test cases later when the code needs changes. The simple solution? Package your manual tests into a reusable test suite that you can run anytime to check your code for errors.

Tests typically use assert statements to confirm that the actual output matches the expected output. For example:

def add(x, y):
    return x + y

def test_add():
    # verify correctness with examples, including edge cases
    # syntax: assert (expression that should evaluate to True), "error message"
    assert add(2, 2) == 4, "2 + 2 should equal 4"
    assert add(5, -6) == -1, "5 - 6 should equal -1"
    assert add(-2, 10.6) == 8.6, "-2 + 10.6 should equal 8.6"
    assert add(0, 0) == 0, "0 + 0 should equal 0"

Testing in Python with pytest

Consider using the pytest framework for your Python tests. Organize all your test scripts in a dedicated tests/ folder to keep them separate from the main source code.

As discussed in Chapter 4, pure functions—those without side effects like reading or writing external files—are especially easy to test because you can directly supply the necessary inputs. Placing your main logic into pure functions therefore simplifies testing the critical parts of your code. For impure functions, such as those interacting with databases or APIs, you can use techniques like mocking to simulate external resources, ideally combined with dependency injection.

When designing your tests, focus on edge cases—unusual or extreme scenarios like values outside the normal range or invalid inputs (e.g., dividing by zero or passing an empty list). The more thorough your tests, the more confident you can be in your code. Each time you make significant changes, run all your tests to ensure the code still behaves as expected.

Some developers even adopt Test-Driven Development (TDD), where they write tests before the actual code. The process begins with writing tests that fail (or don’t even compile), then creating the code to make them pass. TDD can be highly motivating as it provides clear goals, but it requires discipline and may not always be practical in the early stages of development when function definitions are still evolving.

Testing at different levels

Ideally, you’ll test your software at all levels:

Unit Tests: Test individual units (e.g., single functions) to verify basic logic. These should make up the bulk of your test code.
Integration/System Tests: Check that different parts of the system work together as expected. These often require more complex setups, like running multiple services at the same time, to test the flow end-to-end.
Manual Testing: Identify unexpected behavior or overlooked edge cases. Whenever a bug is found, create an automated test to reproduce it and prevent regression.
User Testing: Evaluate the user interface (UI) with real users to ensure clarity and usability. UX designers often perform these tests using design mockups before coding begins.

Debugging

When your code doesn’t work as intended, you’ll need to debug—systematically identify and fix the problem. Debugging becomes easier if your code is organized into small, testable functions covered by unit tests. These tests often help narrow down the source of the issue. If your existing tests didn’t catch the bug, first write a new, failing test to reproduce it before fixing it. This confirms your fix works and prevents a regression—meaning the bug won’t sneak back into the code later.

To isolate the exact line causing the error:

Use print statements to log variable values at key points and understand the program’s flow.
Add assert statements to verify intermediate results.
Use a debugger, often integrated into your IDE, to set breakpoints where execution will pause, allowing you to step through the program manually and inspect variables.

Debugging is an essential skill, not only to identify the root cause of bugs, but also to improve your understanding of the code and its behavior.

Make It Fast

Make it run, make it right, make it fast.
– Kent Beck (or rather his dad, Douglas Kent Beck³)

Now that your code works and produces the right results (as you’ve dutifully confirmed with thorough testing), it’s time to think about performance.

Readability over performance

Always prioritize writing code that’s easy to understand. Performance optimizations should not come at the cost of readability. More time is spent by humans reading and maintaining code than machines executing it.

Find and fix the bottlenecks

Instead of randomly trying to speed up everything, focus on the parts of your code that are actually slow. A simple way to identify bottlenecks is to insert log statements with timestamps at key points in your code to measure the time elapsed between steps. And if you manually interrupt a (Python) script during a long run and it always stops in the same place, that’s also often an indication that this step could be the issue. For a more systematic approach, use a profiler. Profilers analyze your code and show you how much time each part takes, helping you decide where to focus your efforts.

Accessing files on disk or fetching data over the network is one of the slowest operations in most programs. Whenever possible, cache the results by storing the loaded data in memory to avoid repeated access to external resources. Just be mindful of how frequently the external data changes and invalidate the cache when the information becomes outdated.

Run it in the cloud

Working with large datasets may trigger Out of Memory errors as your computer runs out of RAM. While optimizing your code can help, sometimes the quickest solution is to run it on a larger machine in the cloud. Platforms like AWS, Google Cloud, Azure, or your institution’s own compute cluster make this cost-effective and accessible. That said, always look for simple performance improvements first!

Think About Big O

Some computations have unavoidable limits. For example, finding the maximum value in an unsorted list requires checking every item—there is no way around this. The “Big O” notation is used to describe these limits, helping you understand how your code scales as data grows (both in terms of execution time and required memory).

Constant time (\(\mathcal{O}(1)\)): Independent of dataset size (e.g., looking up a key in a dictionary).
Linear time (\(\mathcal{O}(n)\)): Grows proportionally to data size (e.g., finding the maximum in a list).
Problematic growth (e.g., \(\mathcal{O}(n^3)\) or \(\mathcal{O}(2^n)\)): Polynomial or exponential scaling can make algorithms impractical for large datasets.

When developing a novel algorithm, you should examine its scaling behavior both theoretically (e.g., using proofs) and empirically (e.g., timing it on datasets of different sizes). Designing a more efficient algorithm is a major achievement in computational research!

Divide & Conquer

If your code is too slow or your dataset too large, try splitting the work into smaller, independent chunks and combining the results. Such a “divide and conquer” approach is used in many algorithms, like the merge sort algorithm, and in big data frameworks like MapReduce.

Example: MapReduce

MapReduce [2] was one of the first frameworks developed to work with ‘big data’ that does not fit on a single computer anymore. The data is split into chunks and distributed across multiple machines, where each chunk is processed in parallel (map step), and then the results are combined into the final output (reduce step).

For instance, if you’re training a machine learning model on a very large dataset, you could train separate models on subsets of the data and then aggregate their predictions (e.g., by averaging them), thereby creating an ensemble model.

Replace For-Loops with Map/Filter/Reduce

Sequential for loops can often be replaced with map, filter, and reduce operations for better readability and potential parallelism:

map: Transform each element in a sequence.
filter: Keep elements that meet a condition.
reduce: Aggregate elements recursively (e.g., summing values).

For example:

from functools import reduce

### Simplify this loop:
result_sum = 0
result_max = -float('inf')
for i in range(10000):
    new_i = i**0.5
    # the modulo operator x % y gives the remainder when diving x by y
    # i.e., we're checking for even numbers, where the rest is == 0
    if (round(new_i) % 2) == 0:
        result_sum += new_i
        result_max = max(result_max, new_i)

### Using map/filter/reduce:
# map(function to apply, list of elements)
new_i_all = map(lambda x: x**0.5, range(10000))
# filter(function that returns true or false, list of elements)
new_i_filtered = filter(lambda x: (round(x) % 2) == 0, new_i_all)
# reduce(function to combine current result with next element, list of elements, initial value)
result_sum = reduce(lambda acc, x: acc + x, new_i_filtered, 0)
result_max = reduce(lambda acc, x: max(acc, x), new_i_filtered, -float('inf'))
# (of course, for these simple cases you could just use sum() and max() on the list directly)

In Python, list comprehensions also offer concise alternatives:

new_i_filtered = [i**0.5 for i in range(10000) if (round(i**0.5) % 2) == 0]

Exploit Parallelism

Many scientific computations are “embarrassingly parallelizable,” meaning tasks can run independently. For example, running simulations with different model configurations, initial conditions, or random seeds. Each of these experiments can be submitted as a separate job and run in parallel on a compute cluster. By identifying parts of your code that can be parallelized, you can save time and make full use of available resources.

Refactoring: Make Change Easy

Refactoring is the process of modifying existing code without altering its external behavior [4]. In other words, it preserves the “contract” (interface) between your code and its users while improving its internal structure.
Common refactoring tasks include:

Renaming: Giving (internally used) variables, functions, or classes more meaningful and descriptive names.
Extracting Functions: Breaking large functions into smaller, more focused ones (with a single responsibility).
Eliminating Duplication: Consolidating repeated code into reusable functions (DRY principle).
Simplifying Logic: Reducing deeply nested code structures or introducing guard clauses for clarity.
Reorganizing Code: Grouping related functions or classes into appropriate files or modules.

Why refactor?

Refactoring is typically done for two main reasons:

Addressing Technical Debt:
When code is written quickly—often to meet deadlines—it may include shortcuts that make future changes harder. This accumulation of compromises is called “technical debt.” Refactoring cleans up this debt, improving code quality and making the code easier to understand.
- Example: Revisiting old code can be like tidying up a messy campsite. Just as a good scout leaves the campground cleaner than they found it, a responsible developer leaves the codebase better for the next person (or themselves in the future).
Making Change Easier:
Sometimes, implementing a new feature in your existing code feels like forcing a square peg into a round hole. Instead of struggling with awkward workarounds, you should first refactor your code to align with the new requirements. The goal of software design isn’t to predict every possible future change (which is impossible) but to adapt gracefully when those changes arise. This promotes an evolutionary architecture, where you solve problems once you understand them better [3].
- Before adding a new feature, clean up your code so that the change feels natural and seamless. This not only simplifies the task at hand but also results in a more general, reusable functions and classes.

Tips when refactoring

Test as you refactor: Always run tests before and after refactoring to ensure no functionality is accidentally broken. Writing or expanding automated tests is often part of the process to safeguard against regressions.
Leverage IDE support: Modern IDEs like PyCharm or Visual Studio Code provide tools for automated refactoring, such as renaming, extracting functions, or moving files. These can save time and reduce errors.
Avoid over-refactoring: While cleaning up code is valuable, avoid making unnecessary changes that don’t improve functionality or clarity. Over-refactoring wastes time and can confuse collaborators.

Refactorings to simplify changes

For each desired change, make the change easy (warning: this may be hard), then make the easy change.
– Kent Beck⁴

Replace Magic Numbers with Constants: Magic numbers—values with unclear meaning—can make code harder to understand and maintain. By replacing them with constants, you create a single source of truth that’s easy to modify.

# before:
if status == 404:
    ...

# after:
ERROR_NOT_FOUND = 404
if status == ERROR_NOT_FOUND:
    ...

You can also use enumerations to specify a set of related constants. Enums can be especially helpful to make a function’s interface explicit. For example, by specifying that an input argument should be an HTTPStatus, users of the function know that they can’t just pass any arbitrary integer:

from enum import Enum

class HTTPStatus(Enum):
    OK = 200
    CREATED = 201
    FORBIDDEN = 403
    NOT_FOUND = 404
    INTERNAL_SERVER_ERROR = 500
    SERVICE_UNAVAILABLE = 503

def get_status_message(status_code: HTTPStatus):
    """Returns the HTTP status message for a given HTTPStatus enum value."""
    return f"{status_code.name.replace('_', ' ').title()} ({status_code.value})"

if __name__ == "__main__":
    print(get_status_message(HTTPStatus.OK))  # Output: Ok (200)
    print(get_status_message(HTTPStatus.NOT_FOUND))  # Output: Not Found (404)

Don’t Repeat Yourself (DRY): Copying and pasting code may seem like a quick fix, but it leads to problems later. If the logic changes, you’ll need to update it everywhere it’s duplicated, which is error-prone. Instead, move the logic into a reusable function or method.
```
# before:
if (model.a > 5) and (model.b == 3) and (model.c < 8):
    ...

# after:
class MyModel:
    def is_ready(self):
        return (self.a > 5) and (self.b == 3) and (self.c < 8)

if model.is_ready():
    ...
```
Implement Wrappers: When working with external libraries or APIs, their provided interface might not align with your needs, and adapting to it directly can lead to awkward implementations in your code. A better solution is to create a wrapper that implements the interface you wish you had, translating the external API’s inputs and outputs into the format that best suits your implementation. This approach keeps your code clean, consistent, and easier to maintain, while confining the less-than-ideal API interactions to a single location. Plus, if the external API changes, you only need to update the wrapper instead of changing your code everywhere.
Use Alternative Constructors: Similar to a wrapper, you can add a class method to create objects in a way that’s different from the regular constructor. This is useful when the input data doesn’t directly match what the constructor needs. For instance, imagine you have a configuration file that specifies settings for a simulation. If the names or structure of these settings don’t match the constructor’s parameters, you can create a from_config method to handle the translation and then call the constructor with the correct arguments. The advantage is that if the format of the configuration file changes in the future, you only need to update this one method, keeping the rest of your code the same.
```
class Date:
  def __init__(self, year, month, day):
      self.year = year
      self.month = month
      self.day = day

  @classmethod
  def from_str(cls, date_str):
      # parse the string and create a new instance
      year, month, day = map(int, date_str.split('-'))
      return cls(year, month, day)

# usage:
date1 = Date(2025, 01, 30)
date2 = Date.from_str("2025-01-30")
```
Organize for Coherence: Keep code elements that need to change together in the same file or module. Conversely, separate unrelated parts of your code to prevent unnecessary entanglement. This way, changes are localized, which reduces cognitive load.

In larger codebases shared by multiple teams, this is even more critical. When changes require excessive communication and coordination, it signals a need to reorganize the code. Clear ownership and reduced dependencies help teams work independently while keeping the system coherent through agreed upon interfaces.

Large-scale Refactoring

While smaller portions of code are often tidied up as you go [1], larger refactorings that span multiple files or repositories require more upfront planning [5].

The following steps will help keep larger refactorings on track and ensure they deliver real improvements:

Identify the problems this refactoring should address.

Don’t refactor just because the code is ugly—refactor because it’s causing problems. For example, the current structure may make it difficult to implement important new features or maintain the code efficiently. List all the problems the code creates and rank them by severity to ensure your refactoring tackles the most critical issues.
Envision the ideal state.

Code can become suboptimal for many reasons. Maybe a looming deadline forced developers to take shortcuts, leading to technical debt. Maybe an inexperienced developer made a design choice that no longer fits. But most likely, the requirements have evolved since the code was written, making what was once a good solution no longer suitable.

Try to break free from the existing structure and its limitations. If you were designing the system from scratch today, given everything you now know about current and future requirements, what would you build? What should the code look like once refactored?
Verify that the ideal state solves the most important issues.

Revisit your list of problems and ensure that your envisioned ideal state actually addresses them. If necessary, iterate on your vision until you have a solution that tackles the most critical challenges. This gives you a better understanding of which changes to the codebase are actually necessary to achieve your goal.
Make a realistic plan to get closer to your ideal state.

While it might be tempting to rewrite everything from scratch, this is rarely practical. A complete rewrite would likely take longer than expected and introduce new, unforeseen issues, as the existing code likely accounts for edge cases and hidden requirements you’ve forgotten.

Instead, translate your ideal state into small, targeted changes to the existing codebase that still provide significant benefits. Ideally, each change should:
- Be independent of the others, allowing for incremental progress.
- Deliver some immediate value on its own.
Create a prioritized list of independent changes, considering both:
- Impact: Which of the original problems does this change solve?
- Effort: How difficult would this be to implement? What dependencies or additional steps are required (e.g., database migrations, external system changes, or involvement from other teams)?
To accurately assess effort, detail your plan—outline which files and functions will be affected and identify any external dependencies. Once you’ve evaluated impact vs. effort, decide which changes are essential, which are nice-to-have, and which might not be worth the effort, while taking into account that some steps might depend on the successful completion of other changes.
Get feedback on your plan.

If possible, discuss your plan with collaborators and stakeholders—especially those affected by the changes. They might catch overlooked dependencies or identify potential blockers before you run into them during implementation.
Execute incrementally and merge frequently.

Instead of implementing all changes at once, work step by step, merging updates back into the codebase as quickly as possible. This minimizes risk, ensures early testing and validation, and helps maintain motivation—since every small change delivers immediate value.

By refactoring regularly and following these practices, you’ll create a cleaner, more maintainable codebase that is adaptable to future needs and enjoyable to work with.

Before you continue

At this point, you should have a clear understanding of:

How to transform your ideas into code.
Some best practices to write code that is easy to understand and maintain.

The __init__.py file is needed to turn a directory into a package from which other scripts can import functionality. Usually, the file is completely empty.↩︎
https://martinfowler.com/bliki/TwoHardThings.html↩︎
https://x.com/KentBeck/status/704385198301904896↩︎
https://x.com/KentBeck/status/250733358307500032↩︎