Abstraction - Art or Science

1. Introduction
2. The Essence of Abstraction
3. The Art of Abstraction
4. The Science of Abstraction
5. Avoiding Over-Abstraction and Under-Abstraction
6. Its role with (de-)coupling
7. Conclusion

Introduction

In the world of software development, the concept of abstraction holds a pivotal role in creating efficient and scalable solutions. It serves as a bridge between the complex inner workings of software systems and the practical needs of end-users. Abstraction can be seen as both an art and a science, requiring a delicate balance to achieve optimal results. In this article, we will delve into the significance of abstraction, its impact on software engineering, and how it distinguishes a software engineer from a software developer.

The Essence of Abstraction

Abstraction is the process of simplifying complex systems by focusing on the most essential aspects while hiding unnecessary details. It allows developers to create higher-level representations that are easier to understand, maintain, and reuse. By abstracting away intricate implementation details, software engineers can design robust and scalable architectures that stand the test of time.

The Art of Abstraction

Successful abstraction in software development requires a keen sense of artistry. Like an artist painting a masterpiece, a software engineer carefully selects the right level of abstraction to create a harmonious balance between simplicity and functionality. It involves identifying and extracting the core concepts, patterns, and behaviors that define a system, enabling developers to build flexible and adaptable software solutions.

The Science of Abstraction

Abstraction is not just an artistic endeavor but also a scientific approach. It relies on proven principles, methodologies, and best practices to ensure the abstraction is meaningful and effective. Software engineers leverage their technical expertise, domain knowledge, and analytical skills to abstract complex systems into manageable and reusable components. They strive to find the sweet spot where abstraction enhances productivity and maintainability without introducing unnecessary complexity.

Avoiding Over-Abstraction and Under-Abstraction

Finding the right level of abstraction is a continuous challenge for software engineers. Over-abstraction, also known as over-engineering, can lead to unnecessary complexity, reduced productivity, and difficulties in understanding and maintaining the codebase. Conversely, under-abstraction can result in code duplication, reduced reusability, and increased development effort.

The key lies in striking a balance, understanding the specific requirements of the project, and abstracting in a way that addresses the immediate needs while allowing room for future expansion and evolution. It requires experience, collaboration, and a deep understanding of the problem domain.

Its role with (de-)coupling

Abstraction plays a pivotal role in managing coupling and achieving maintainable code. However, it’s important to clarify that abstraction is not inherently linked to coupling. Coupling can exist independently, but meaningful decoupling requires a broader perspective on the software application and a forward-thinking mindset.
Some argue that abstraction introduces coupling and complexity. While the latter can be true, the former is not strictly accurate. Coupling already exists in a system before introducing a base class or an interface; it simply manifests at a different level.

Let’s consider a widely-accepted community mindset example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

class JsonFileStateSaver
{
  private string filename;
  public JsonFileStateSaver(string filename)
  {
    this.filename = filename;
  }

  public void Save(State state)
  {
    using (FileStream stream = new FileStream(filename, FileMode.Create))
    {
      //json write operations
    }
  }
}
class XmlFileStateSaver
{
  private string filename;
  public XmlFileStateSaver(string filename)
  {
    this.filename = filename;
  }

  public void Save(State state)
  {
    using (FileStream stream = new FileStream(filename, FileMode.Create))
    {
      //xml write operations
    }
  }
}
class StateHandler
{
  // ...
  void Save(Config config)
  {
    if (SaveMode.Json | SaveMode.Local == config.SaveMode)
    {
      var json = new JsonFileStateSaver("state.json");
      json.Save(state);
    }
    else if (SaveMode.Xml | SaveMode.Local == config.SaveMode)
    {
      var xml = new XmlFileStateSaver("state.xml");
      xml.Save(state);
    }
    else
    {
      throw new NotSupportedException($"Save mode is not supported! ({config.SaveMode})");
    }
  }
}

In this example, there is a debate about whether abstraction through a base class or interface is suitable. While I acknowledge the arguments against using a base class or interface in this specific scenario, I disagree with the notion that abstraction is entirely devoid of value here. In reality, the existence of two separate classes for file-savers already demonstrates a form of abstraction, as it aims to abstract the target file format while maintaining a close coupling with the underlying filesystem. It’s important to understand that coupling is an inherent aspect that cannot be completely eliminated.

But abstraction goes beyond creating base classes or interfaces; it involves logically separating components to hide intricate implementation details. It helps us create a cleaner design that focuses on essential aspects while abstracting away unnecessary complexities.

If we believe that neither a base class nor an interface makes sense in this use-case, we should question the design itself. If the filename instance argument or the Save(State state) method are not worth abstracting, then why abstract at all that way? In such cases, it may be more appropriate to refactor our savers as “helper” classes for the StateHandler to a single purpose: saving a desired format to a given target managed by the handler.

Consider the following approach:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

class JsonStateSaver
{
  void Save(Stream stream, State state)
  {
    //json write operations
  }
}
class XmlStateSaver
{
  void Save(Stream stream, State state)
  {
    //xml write operations
  }
}
class StateHandler
{
  // ...
  void Save(Config config)
  {
    if (SaveMode.Json | SaveMode.Local == config.SaveMode)
    {
      using (var stream = getFileStream("state.json"))
      var saver = new JsonStateSaver();
      saver.Save(stream, state);
    }
    else if (SaveMode.Xml | SaveMode.Local == config.SaveMode)
    {
      using (var stream = getFileStream("state.xml"))
      var saver = new XmlStateSaver();
      saver.Save(stream, state);
    }
    else
    {
      throw new NotSupportedException($"Save mode is not supported! ({config.SaveMode})");
    }
  }

  FileStream getFileStream(string filepath)
  {
    // do some common file stream preperations, pre-checks, etc..
    // return filestream
  }
}

Now, imagine introducing a new storage target, such as S3. With this level of abstraction, we can easily reuse existing components without resorting to copy-pasting code, or without now suddenly starting to implement a base class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

class StateHandler
{
  // ...
  void Save(Config config)
  {
    if (config.SaveMode.HasFlag(SaveMode.Json))
    {
      saveJson();
    }
    else if (config.SaveMode.HasFlag(SaveMode.Xml))
    {
      saveXml();
    }
    else 
    {
      throw new NotSupportedException($"Save mode is not supported! ({config.SaveMode})");
    }
  }

  void saveJson() 
  {
    if (config.SaveMode.HasFlag(SaveMode.S3))
    {
      using (var stream = new MemoryStream())
      var saver = new JsonStateSaver();
      saver.Save(stream, state);
      await fileTransferUtility.UploadAsync(stream, bucketName, keyName);
    }
    else
    {
      using (var stream = getFileStream("state.json"))
      var saver = new JsonStateSaver();
      saver.Save(stream, state);
    }
  }

  void saveXml()
  {
    // ...
  }
}

The benefits become apparent:

• Avoiding code duplication
• Increased reusability
• Hiding implementation details of state persisting for a file format
• Separation of concerns
• Improved unit testability

So the decision to couple our savers with filename as local filesystem file makes it impossible for us to use the Json and Xml format savers for a Cloud storage system like S3.
By also decoupling the target filesystem from our savers though, we can reuse those savers also for a S3 storage. This abstraction ensures that the file savers are solely responsible for saving data in a specific file format, without being tightly coupled to the target filesystem. It’s important to note that abstraction itself does not introduce coupling. The way abstraction is implemented determines whether coupling is introduced or not for a particular dependency in a component.

However, it is important to consider the potential challenges of this approach. As the number of save options increases, the Save() handler can become more complex and difficult to understand, making it harder to maintain code readability in the future. Nevertheless, it is possible to mitigate this issue by abstracting the Save() method into sub-procedures based on the target format, such as Xml, Json, and so on, as shown above.

When deciding on the appropriate level and focus of abstraction, it is crucial to take into account the future roadmap of the StateHandler and the development team constellation. Neglecting these considerations can lead to common pitfalls observed among dev teams. By anticipating the software project’s evolution and team dynamics, informed decisions can be made that result in meaningful abstractions.

When considering the introduction of a DbStateSaver, it is important to address the following questions to ensure correct abstraction:

When does it need to be available as a save option? If it is not in the roadmap any time soon, it can possibly be ignored in the abstraction process. However, if it is planned for implementation, it should be considered in the abstraction so that other developers can extend the already released Save() functionality without the need for refactoring and understanding every implementation detail later in the development cycle. Neglecting this consideration can lead to increased effort and a higher risk of introducing bugs in already functional code.
If the DbStateSaver should be implemented soon, the following question arises:

• How should the State object be serialized to a database?
• Should it be stored as Json, Xml, Protobuffer, or another format in a single column in a relational database?
• Should it be translated to a relational data representation?
• Should it be handled by native Json/Xml databases?

The answer to these questions depends on the requirements of the query/read usage of the State object.

To approach the implementation of a DbStateSaver without the need to refactor the existing design, it is recommended to start by drawing a simple visualization of all parts and dependencies. This helps to have a clear overview of the components involved and their relationships, facilitating the identification of the necessary abstractions and ensuring a smooth integration of the DbStateSaver regardless of the target format (relational, Xml, Json, etc.).

---
title: State Savers 
---
flowchart LR
  
  subgraph statefs["Local Filesystem State"]
    JSONfs[JSON]
    XMLfs[XML]
    BINfs[Binary]
  end

  subgraph statelfs["Remote Filesystem State"]
    JSONrfs[JSON]
    XMLrfs[XML]
    BINrfs[Binary]
  end

  subgraph statedb[Database State]
    JSONdb[JSON]
    XMLdb[XML]
    BINdb[Binary]
    Relational
  end

  StateHandler

In order to achieve independence and eliminate coupling between the State writers and target storage types, we need to design and implement them in a way that allows for reuse across all supported storage types. The key question we must address through proper design and abstraction is:

“How can we implement State writers that are decoupled from specific target storage types?”

---
title: State Savers Design (Abstraction)
---
flowchart LR
  
  subgraph statefs["LocalFsStateWriter"]
    JSONfs[JSON]
    XMLfs[XML]
    BINfs[Binary]
  end

  subgraph statelfs["S3StateWriter"]
    JSONrfs[JSON]
    XMLrfs[XML]
    BINrfs[Binary]
  end

  subgraph statedb[DatabaseStateWriter]
    JSONdb[JSON]
    XMLdb[XML]
    BINdb[Binary]
    Rdb[Relational]
  end

  StateHandler --> save[["Save()"]] -.-> statedb
  save ---> statelfs & statefs

  jsonW[JsonStateWriter]
  JSONfs & JSONrfs & JSONdb <--> jsonW

  xmlW[XmlStateWriter]
  XMLfs & XMLrfs & XMLdb <---> xmlW

  binW[BinaryStateWriter]
  BINfs & BINrfs & BINdb <----> binW

  dbW[RelationalStateWriter]
  Rdb <----> dbW

With a usage like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class StateHandler
{
  // ...
  void Save(Config config)
  {
    if (config.SaveMode.HasFlag(SaveMode.Local))
    {
      var saver = new LocalFsStateWriter(config);
      saver.Save(state);
    }
    else if (config.SaveMode.HasFlag(SaveMode.S3))
    {
      var saver = new S3StateWriter(config);
      saver.Save(state);
    }
    else if (config.SaveMode.HasFlag(SaveMode.Database))
    {
      var saver = new DatabaseStateWriter(config);
      saver.Save(state);
    }
    else 
    {
      throw new NotSupportedException($"Save mode is not supported! ({config.SaveMode})");
    }
  }
}

The DatabaseStateWriter can also utilize the same JsonStateWriter, XmlStateWriter, BinaryStateWriter like LocalFsStateWriter or S3StateWriter.

Just to give the idea:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

class DatabaseStateWriter
{
  // ...
  void Save(State state)
  {
    if (config.SaveMode.HasFlag(SaveMode.Json))
    {
      using (var ms = new MemoryStream())
      var json = new JsonStateWriter(ms);
      js.Write(state);
      var utf8 = new UTF8Encoding(false);
      var jsonString = utf8.GetString(ms.ToArray());
      updateStateColumn(jsonString);
    }
    else if (config.SaveMode.HasFlag(SaveMode.Xml))
    {
      //..
      updateStateColumn(xmlString);
    }
    else if (config.SaveMode.HasFlag(SaveMode.Binary))
    {
      //..
      updateStateBinaryColumn(ms);
    }
    else if (config.SaveMode.HasFlag(SaveMode.Relational))
    {
      var relational = new RelationalStateWriter(dbCon);
      relational.Write(state);
    }
    else 
    {
      throw new NotSupportedException($"Save mode is not supported! ({config.SaveMode})");
    }
  }
}

If you aim to enhance the decoupling between your StateHandler implementation and possible StateWriter implementations, you can achieve this by introducing an abstract base class and/or interface. Additionally, employing a proper Config design and utilizing dependency injection can help in achieving a more flexible and modular solution. By following this approach, you can use the StateWriter implementations in the StateHandler through dependency injection, allowing for easier maintenance and future extensibility.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class StateHandler
{
  private readonly IEnumerable<IStateWriter> availableSavers;

  public StateHandler(IEnumerable<IStateWriter> availableSavers)
  {
    this.availableSavers = availableSavers;
  }

  void Save(Config config)
  {
    var saver = availableSavers.FirstOrDefault(w => w.IsModeSupported(config.SaveMode))
      ?? throw new NotSupportedException($"No Saver available for SaveMode! ({config.SaveMode})");

    saver.Save(state);
  }
}

It is worth noting that by coupling the IStateWriter implementations to the Save(State state) method, you have effectively decoupled the StateHandler from the logic of the StateWriter. This allows for easier unit testing and separation of concerns by using interfaces.

However, it is essential to consider the broader context and goals of your project, such as the desired save mode capabilities, testability, modularity, and the composition of your development team. If your save mode capabilities are relatively simple, unit testability is not a primary concern, and modularity is not a high priority, abstracting for a more isolated and decoupled component design may not be necessary.

It is crucial to understand that poorly implemented abstractions can have negative consequences. They can introduce bugs when adding new features and may require extensive refactoring of functional aspects instead of building upon existing functionality.

In summary, the correct application of abstractions strikes a balance between excessive over-engineering and disregarding abstraction entirely. By strategically utilizing abstraction at the appropriate level and focal point, you can create systems that exhibit flexibility, maintainability, and scalability, ultimately enhancing the overall quality of the software architecture. This approach ensures that abstraction is a valuable tool rather than a burden, saving time, resources, and minimizing financial costs.

Conclusion

Abstraction in software development is a delicate fusion of art and science. It requires the mastery of striking the right balance between simplicity and functionality, while adhering to proven principles and methodologies. Software engineers are adept at navigating this realm, recognizing the pivotal role of meaningful abstraction in constructing scalable, maintainable, and adaptable software systems.

Through the symbiotic interplay of artistry and scientific precision, software engineers elevate their craft and contribute to the growth and success of their projects. As we delve further into the depths of abstraction, we uncover new possibilities and push the boundaries of what can be achieved in the ever-evolving world of software development.

While this blog post provides a glimpse into the vast realm of abstraction, there is much more to explore. We invite you to embark on this ongoing journey with us, as we delve into further topics and share valuable insights in the realm of software development. Stay tuned for more engaging discussions and practical knowledge that can shape the future of your software endeavors.