Embrace Modular Technology and Agile Process to Deliver Business Impact

Why Read This Green Paper

Is your enterprise technology built in a modular way? Can you modify or replace a component without affecting other parts of the technology architecture? Is your technology platform built with plug and play elements to allow for rapid change and adaptation to business and customer forces? Do you employ Agile processes to make calculated changes incrementally?

Technology architecture and implementation governed by a coherent platform strategy that prioritizes flexibility and component and service independence will deliver business impact. 

In this paper, we articulate technology platform and architecture requirements to support modern ways of delivering iterative value, increasing the velocity, productivity, and performance of the organization, and reducing product and service time to market.

Executive Summary

Modular technology platform architecture and Agile development are core elements of a successful digital transformation in all organizations. Organizations need to focus on continuous improvement and the currency of technologies to deliver business results for the enterprise while inspiring a culture of learning for employees.

Digital transformation is the use of technology to radically improve the performance or reach of an organization. In a truly digital business that is digitally enabled, new technologies enable process improvement, talent engagement, and new business models.

-The Nine Elements of Digital Transformation, MIT Sloan Management Review

3 Core Components: Modularity of Platform, Technically Governed Agile, and Organizational Empowerment

1.    Efficiencies are derived from the technical platform. In technical teams, this means iteratively improving productivity, performance and speed through modernization of the technology utilized. Currency and modularity of the strategic asset of technology will enable growth and the ability of the enterprise to compete and thrive. Modularity is the backbone to an extensible architecture.

2.    Architectural governance belongs inside Agile delivery. Architectural soundness is an operational need, not a philosophical stance. The ability to incrementally improve the whole without diluting the architecture allows Agile development practice in the short term to align with technical goals in the long term. By adhering to higher architectural constraints, Agile delivery can approach problems in a way that better supports the long-term goals within feature focused sprints.

3.    Organizational strategy is reflective of the need to support Agile with technical readiness. Just as Agile adoption needed to be accompanied by an organizational acceptance of the approach and the changes it brings, it is the organization as a whole which needs to help make sure the technology is at a coherence, modernity, and maturity where it can serve as the backdrop to productivity.

Insights

·      Architecture is a defined and documented understanding that encompasses future change. The highest technical strategy must be planned and mindful, keeping its shape despite Agile day-to-day delivery rather than being informed by it.

·      Change does not happen without leadership. Agile requires both organizational and technical readiness to enable it to operate efficiently.

Introduction | Modularization Driving Agile Productivity 

Companies that are investing in technology architecture today are creating a strategic advantage and a competitive differentiator for their businesses.

Enterprise architecture represents the vanguard

of modernization and digital transformation.

Organizations have largely moved to Agile methodologies. Agile productivity frameworks have a track record of helping drive iterative value, as well as helping product development remain focused on desirable product behavior.

However, the practice of sprint-based deliverables can be challenged by many commonplace aspects of enterprise software, such as:

·      large, complex codebases,

·      older or fragmented technology stacks,

·      differing customer needs,

·      complex data needs, and

·      evolving understanding of product direction.

Compromised architecture and big technical problems are not the environment in which Agile delivery thrives. Rather, they represent impediments which can reduce productivity and the quality of the final product.

Modularization presents itself as a way forward through large-scale technical encumbrances of a system. It can represent a cleaner delivery of functionality and help deployment options. As a step to technical modernization, modularization can also be a pathway to help reduce the risk of bringing portions of a product codebase forward in time while maintaining a complete working system. And, modularity can help to enable the shift toward increased leverage of cloud services.

The goal of modularity at the broadest level may be allied with a product strategy, wherein individual components or portions of a platform may be combined in ways that make sense to users. Formalizing the boundaries of service components within existing products may help to increase the opportunities in the same way that outward facing APIs have allowed customers to derive more diverse functionality from product offerings.

The acceptance of Agile methodology has come from the strategic and financial need to accelerate product development to provide differentiation and excellence in experience. Removing the obstacles to that delivery is a priority which must be taken seriously at the organizational level.

Modularization for Productivity Gains 

Modularization may be described as breaking down a system into smaller component pieces that are expected to carry out independent operations.

Modularization is a technique to divide a software system into multiple discrete and independent modules, which are expected to be capable of carrying out task(s) independently. -Tutorialpoint

 

Identify the Purpose

Modularity can improve various technical situations, and in its different ways may suit one goal over another. If the modularity is to allow different versions of a framework or differing frameworks, the underlying purpose of their co-existence will help to rationalize where boundaries are split. If differing domain models make sense to divide, then again, a functional division may be identified which is at odds with layering. On the other hand, if the modularity is driven purely by a wish to remain scalable and to reduce the coupling between pieces to allow easier upgrades, then infrastructural boundaries will likely be indicated.

In any event, the equation keeps the decision of modularizing in the language of the big picture of productivity and planning for efficiencies. In turn, this helps modularity claim its place in business level discussions and decision-making, and not be thought of as a technical subtlety.

Types of Modularization

Language Level Modularization & Patterns

Modularization has helped drive programming language design since early generations of language. The ability to break up methods into smaller units, and most significantly the developments of object orientation and encapsulation behind interfaces, has helped to isolate specifics in ways that allow for multiple implementations and later replacements to fit in neatly. Type systems, which may complicate the modularization of builds, give this isolation a semantically rich and meaningful contract. Microsoft’s .NET Common Language Runtime allowed this strategy to be shared between multiple languages, as did later Java Virtual Machine languages which took the ecosystem wider than just Java itself.

Beyond the languages, patterns emerged taking the benefits of modularity further, from SOLID principles helping build stronger object orientation, to practices of layered architecture which kept data access in its own place. Already these concerns tie in with some of the business goals of modularization. Strong encapsulation and layering allow for platforms to build up with a degree of isolation from feature-supporting code which relies on them. Translated to Agile processes, this can mean the difference between an Agile team staying safely inside its lanes while leveraging the best a platform can offer, and a team either having to tread cautiously because edits will inevitably affect many product areas at once, or compromising layering altogether and creating new duplicative infrastructural code in various places. In turn, compromises to layers will make the work exponentially harder when it comes to upgrading or replacing an infrastructural concern since there may not be a single incision point in the codebase.

Software build processes are also a target of focused modularity. A module depending on another module in a build should not, in turn, be required by that module, as cyclic dependencies at a minimum point toward undesirable coupling of module code. Build engineering would seem a more localized concern than other topics of this paper, but in an enterprise codebase, it is entirely common to find a compromised build where one module will not build in isolation, even though the project is laid out in a modular way. In the day-to-day work of a small, Agile sprint, this can translate to how long it takes for each single developer to run tests against changes they make. Combined with modern source control flows, with merges of many branches being a daily activity, the ability to test modular portions of code can be critical to efficiency.

Layering with Software Frameworks

Frameworks grew in importance as enterprise code increasingly leveraged more off-the-shelf code to handle functionality shared by all applications: database access, messaging, view layer logic, serialization and so on. Frameworks play a big role in productivity, and good experiences with various frameworks and languages caused a shift towards more framework-oriented development. Depending on the underlying language as well as the design and ecosystem of these, enterprise code would to a greater or lesser degree be decoupled from the framework assumptions. As with language considerations, this is where some nuances can end up playing a divisive role later.

An example is dependency injection. Since early in its history, the most popular Java framework – Spring - provided excellent support for dependency injection. At the same time, Spring was also careful to follow Java community standards such that annotations applied to a codebase to be acted on by Spring could also be understood by different frameworks should an organization choose to go a different way. Likewise, the Java community arrived at standards for persistence annotation, so that different Object Relational Mappers could be plugged in without changing application code.

Furthering the example of dependency injection is Microsoft’s .NET offering. For a long time, .NET has supported dependency injection through 3rd party tools like AutoFac, Unity, and Ninject, among others. Nonetheless, more recently with the release of .NET Core, dependency injection has become a first-class citizen no longer requiring 3rd party tools, although they are still supported. The limit of frameworks that can be integrated, scoped, and tested using dependency injection in .NET is limited only by the developer’s imagination and the project’s architectural needs. The abstractions now built into .NET make tasks such as Test-Driven Development (TDD), changes in business logic at runtime, or injecting logic through middleware easy to develop, integrate and plug-and-play using well understood best practices.

Not all frameworks are alike in this, and some software frameworks perceived as supportive of productivity may provide poorly for upgrades, demanding many changes and targeting new greenfield projects ahead of existing ones. Patterns adopted in application logic to work well with a framework may translate clumsily into alternatives if they later need dropping in. Stronger typed systems may make boundaries and encapsulations more obvious, while in some dynamically typed code, replacing libraries or pieces of infrastructure may not present their incompatibilities and complexities until the code is run.

Informed trade-offs need to be made, between maximizing the off-the-shelf efficiency of a framework, and adding enough layered code to future proof change should a dependency become legacy and unsupported. Project code complexity can be thought of as a measure of how maintainable a codebase will be, so the adoption of software frameworks, while often a practical necessity with benefits, has serious long-term implications.

Message Based | Microservices

With message-based systems and microservices, modularization entered a new phase. Enterprise Service Bus architectures connected code via growing patterns of message queues, adapters and listeners. Where this approach lived in a more prescriptive, specific sector of software development, a broader definition of microservices became commonplace across all types of server-side software development.

In place of language types, the encapsulation comes in the form of RESTful service endpoints or in the defined semantics of the messages being published to persistent queues. A particular note of caution is to be clear of the difference between data and the interface, as complicated query parameters may cover validly variable shapes of data or equally may hide a lack of clear API definition. The modularity gains from separated services can be skin deep if detailed entrails of services are leaked to other services and opportunities missed for stricter contracts.

One appeal of these systems is to reduce complexity within each service. Another benefit is to allow for greater code differences between the various parts. For domains involving specific code approaches such as machine learning, this polyglot approach can have appeal. Even so, where the justifications are weak, deliberate fragmentation of a codebase can make it harder to understand. Perhaps less glamorously, separation of code between services allows different frameworks, and just as crucially, different versions of frameworks, to exist at once in the same codebase. This will be discussed further in the Agile and Productivity section.

User Interfaces | Separate Module or Modules

User Interfaces may best be thought of as themselves being a good target of modularity. Commonalities between the modeled concepts may be strong between the back end and front end. Notwithstanding, this reliance is a big risk for both sides of the equation. User experience has unique variables and considerations, along with changes in expectation and best practice, and must be free to evolve outside the rest of the system.

Microservices on the back end may be split by user feature areas, by more generic layering of infrastructure needs, or a combination of the two. Front end code does not need to follow the same chosen separations. A single area of a single page application may communicate data with various parts of the system whole; whether they do so via one or multiple entry-points is a choice of implementation with nuanced implications.

Sometimes the choice of service boundaries will point towards data bottlenecks which will affect user experience, and consideration will need to be given to whether the obstruction will be soluble via some relaxation of the need for real-time reflection of data.

Outside of this, recognize that the UI will likely need refactoring and restructuring as users of the application want different things, when UI or device trends change, or when frameworks evolve in unexpected ways. The ability of a system to have a new front end or front ends talking to the existing RESTful APIs represents perhaps the highest ROI of modularity in a user-facing system.

Modularity may serve an important role in separating parts of the front end itself. Redux is an example of layering working effectively in the front end, giving data its own space orthogonal to page logic and layout code. Higher order components or complex services can be examples of separating cross cutting concerns in JavaScript code in a way which helps a clean approach. Nevertheless, the front end tends to be a domain where code organization and clean sharing of functionality is still more challenging, and the benefits and costs of approaches must be weighed carefully.

Agile Productivity Supported by Modularization

Agile methodologies prescribe a culture of continuous improvement, and Agile teams generally grow in efficiency as they have the chance to refine how they work as a unit. Velocity measurement can be a useful tool to help people understand when things have gone better or worse. Even so, it should not be used as a goal or a metric by which the team’s performance is measured.

In the above discussion of microservice and message-based modularity, the notion of services allowing differing frameworks or versions of frameworks existing side-by-side is explored. The significance of this to productivity and unhindered Agile sprints is that it allows the isolation of legacy portions of application code which are more difficult to work with, as well as to enable new features or versions of features to be built in more ideal, up to date technology. The most outdated portions of an application may become a focus area of specific developers, while other teams work outside of its constraints. By moving with the times in a piecemeal way, there is the opportunity to limit accretion of outdated code, something which often grows in anticipation of a later big-bang system upgrade.

One other advantage of Agile methodologies is that frequent delivery is encouraged. Both technical and non-technical audiences alike – be it the development teams or key business stakeholders - benefit from the iterative delivery approach which fosters frequent communication and confidently results in regular production releases. These periodic releases reduce risk to the end user and provide a steady stream of improvement while simultaneously returning valuable user data to the development team and business.

Agile Process Delivered in Harmony with Architecture

In this paper, the ways in which Modularization presents a number of different approaches to escaping technical debt and empowering the Agile development process in ongoing feature work have been discussed. There is a counter reality to this as well, namely that it is the role of sound development processes to maintain that same extensible architecture as much as possible.

Enlightened product owners will allow for some portion of sprints to undo architectural damage caused by compromises of previous sprints, although this may often be less effective than knowing how to avoid the compromise from the onset. There are times when speed of delivery dictates a well understood and controlled technical debt accrual, yet often the detrimental processes that undermine architecture are more hidden.

Consider the following scenario.

Sprint 1.

Product Owner: “I am including a story around an upcoming feature in this sprint. We’re not delivering this yet, but I know that there is an architectural problem, and I want to give the team a chance to think about it.”

Team: “We discussed this, but the person who knows the most about this is out this week, and there seems to be a solution so hopefully it is the right one.” 

Sprint 2.

Product Owner: “The spike was not conclusive last sprint, so I need us to deliver this functionality this sprint.”

Team: “The person who knows the specifics of that area is back, but has limited time. We chatted about it in a pull request. I am not sure we have time to change how we have done this.”

Sprint 3.

(A different team): “It is a shame that other feature got implemented in the way it did. We need to do something similar, but cannot follow the same pattern. And, now the codebase is going to suffer from some confusion between the two approaches.”

Reading the above realistic scenario assembled in summary form makes clear the need for an architecture document. The very nature of Agile delivery is that it relies on team members being empowered, having overlapping skill sets which allow work to continue without becoming blocked by specific individual’s expertise in a specific area. Formal statements of architecture and guiding principles act to widen the pool of developers able to make decisions that contribute to and maintain architectural integrity. Where Agile delivers small bites of functionality, architecture is by its nature a big picture pillar.

While architectural plans and documents are of significant importance, the documents are not necessarily the creation of a single or select group of developers. Rather, they represent critical elements derived from decisions taken from a vantage point contemplating important future considerations. The strategic questions which have recognized answers can be more easily moved past in the course of ordinary sprint work once determined. Where questions are reached for the first time, the architectural plan can grow and become refined.

By recognizing the strategic lines which different implementation details cross, it is easier for a development team and product owner to more successfully choose when to take short-cuts in the interest of immediate gain. In this way, the purely technical requirements needed to keep the development machine running efficiently can be given a voice to compete with pressing needs of functionality being delivered.

Organization Strategy and Modularity 

The ways in which informed choices around modularity can play a vital role in helping the effectiveness of software development has been explored. From an organizational viewpoint, there are some ways in which teams can support the maintenance of well kept, modular code, namely:

·      Let the vocabulary of the most strategic elements of architecture permeate conversations at high levels. Where costs must be incurred to arrive at an efficient base level of technology, technology stakeholders need to understand the nature of the investment.

·      Give recognition to infrastructural wins as well as to the more tangible and user-facing achievements.

·      Product owners and platform owners must work in partnership. A product excelling in its development cycles yet leaving the technology platform weaker as it does so, is a short-term gain but a longer-term risk.

A healthy technical platform which has embraced service level modularity may also become supportive of a mix-and-match product offering approach. Organizations may wish to deliver offerings in new and innovative ways either to better support existing customers, or to appeal to new categories of customers. Adapting technology to support such a requirement in the short term can be limited in its success, and will often involve compromises in functionality, design, maintainability, and speed of deployment. A technical platform whose boundaries are readily understood should already provide a semantic backbone to allow modular solutions to be designed and built.

Security and Modularity

It is a common experience in the enterprise for security concerns to be a prioritization which waxes and wanes as threat awareness grows either internally or in the media. Beyond quick fixes, software development can only promote security through the longer-term application of best practices.

Modularization can help by encouraging the use of up-to-date technology, hiding older technologies where necessary behind tightly confined boundaries, and keeping software complexity in check. Still, the security strategy of a microservices system must be kept in mind during all development, and its assumptions validated or invalidated as RESTful services are designed. Simple definition of internal versus external facing endpoints, while useful, should not preclude an application of defense-in-depth as it relates to modular components.

Development Operations and Agile 

DevOps is an integral partner in enabling software frameworks, Agile methodologies, and software modularity concepts to become a reality. DevOps enables improved security by being the nexus between infrastructure and software teams enabling software to be delivered reliably and automatically for the user of the application or system.

Further, DevOps is a key partner in modular software, providing the tools and components to deliver fit for purpose software components securely and at scale in an automated fashion. When applied together Agile and DevOps better empower organizations to deliver technology with greater speed and reliability, while simultaneously driving customer needs to the top of the priority list. 

Modern DevOps practices, in the cloud as well as in on-premise environments, help make service-based modularity a practical reality. Tooling such as docker allows developers to explore and build complex multi-service environments within a single machine. In turn, modularity which is well-thought out and documented can help to contain the complexity and stability of the DevOps processes needed to support the platform.

Conclusion

Modular technology platform architecture and Agile development are core elements of a successful digital transformation in all organizations. Organizations need to focus on continuous improvement and currency of technologies to deliver business results for the enterprise while inspiring a culture of learning for employees.

Digital transformation is the use of technology to radically improve the performance or reach of an organization. In a truly digital business that is digitally enabled, new technologies enable process improvement, talent engagement, and new business models.

-The Nine Elements of Digital Transformation, MIT Sloan Management Review 

Organizations that are successfully negotiating the needed modernization and technology reengineering have leadership who recognize the need for growth in organizational architecture, Agile adoption, and understanding across the business.

With this shift, enterprise architects and technologists are becoming more accountable for solution outcomes and the need to increase technology and software development productivity. They are also driving Agile adoption and speed to market to assist the enterprise to grow, and mitigate future risks.

Take Aways

·      Technical architecture and development productivity have a symbiotic relationship. The technical readiness of the platform is a force which has enormous impact on Agile teams. The ability of those teams to achieve short term objectives without hampering the effectiveness of the technical strategy is crucial.

·      Stakeholders in the output of Agile development are stakeholders in the platform. The voice of the customer and product needs must be heard and will naturally be a priority, while development with a view of the long-term extensibility of the application will also enable the meeting of future customer needs.

Investing in technical readiness requires its own organization focus. The rewards of a strong architecture and up-to-date technology are many, and will be supportive of Agile efforts. In contrast to the more easily determined ROI of the development of outward facing product functionality, the case for technical investment must ultimately come from an organizational understanding of its importance.

Authors

Julian Flaks – CTO

Julian is a relentless problem solver and hoarder of full stack expertise. Having thrown himself headlong into Internet technology when best practices had barely begun to emerge, Julian is happiest putting his experience to use unlocking business value. Julian holds a Bachelor’s of Laws from The University of Wolverhampton, England and a Master of Science in Software Engineering from The University of Westminster. 

Russ Harding – Microsoft Practice Leader | Principal Consultant/Technical Architect

Russ specializes in providing technology leadership that inspires while bringing new ideas, and technologies that result in high performance, high availability enterprise products. His primary goal is always to improve efficiency and productivity for clients and bridge the communication gap with leadership, team leads, project managers, and internal partners to guarantee reliable and regular software delivery, and ensure an approach to development that contributes to organizational goals and measurable business results.

Mark Hewitt – President & CEO

Mark is a driven leader that thinks strategically and isn’t afraid to roll up his sleeves and get to work. He believes collaboration, communication, and unwavering ethics are the cornerstones of building and evolving leading teams. Prior to joining EQengineered, Mark worked in various management and sales leadership capacities at companies including Forrester Research, Collaborative Consulting, Cantina Consulting and Molecular | Isobar. Mark is a graduate of the United States Military Academy and served in the US Army.