EQengineered's 2023 Green Paper
Green Paper Versus White Paper
The term white paper originated with the British government, and many point to the Churchill White Paper of 1922 as the earliest well-known example under this name.
White papers are a way the government can present policy preferences before it introduces legislation. Publishing a white paper tests public opinion on controversial policy issues and helps the government gauge its probable impact.
By contrast, green papers, which are issued much more frequently, are more open-ended. Also known as consultation documents, green papers may merely propose a strategy to implement in the details of other legislation, or they may set out proposals on which the government wishes to obtain public views and opinion.
Why Read This Green Paper?
Today’s organizations are adjusting to the new normal of digital business. Leading organizations have pivoted toward a digital first mindset and data-driven decision-making. Fast followers are advancing forward by mimicking patterns and learning from market leaders. Strategies designed to modernize the enterprise are themselves disrupted by perennial forces of technology change.
Digital organizations need to embrace and master specific competencies including:
1. Process Delivery (Agile - Kanban, Scrum)
2. Customer Experience (CX) and Design
3. Data Architecture and Engineering
4. Software Architecture and Development
5. Cloud | Infrastructure | DevOps
Bringing these disciplines together to effect change is at the heart of digital transformation. In this green paper, we examine ways in which disciplined organizations can negotiate change and evolve. Key questions that all digital businesses must answer include:
· Do we have the talent and skills required to become a digital organization?
· How can the organization employ agile methodology to maintain focus?
· What can CX and design disciplines do to anticipate customer needs?
· What data do we have? Is it usable? What data do we need?
· What partnerships should we consider to catalyze our journey?
This green paper explores enterprise modernization, competencies, and resilience strategies to help catalyze enterprise business growth in today’s digital environment.
Executive Summary
Leading enterprise organizations are fully engaged in their digital transformation journeys. The same forces of change which have created technical debt and legacy systems continue to exert challenges on the digital enterprise. Anticipating organizational challenges and adhering to the digital-first roadmap requires executive commitment and effective use of:
1. Agile processes and a PMO to advance toward objectives
2. CX focus and a design system to maintain fidelity of digital experiences
3. Robust technical architecture to ensure a sustainable digital platform
4. Forward-thinking data architecture
As a matter of survival, most organizations have focused on demonstrating results through tactical strategies that reduce the cost of operations or eliminate legacy technology.
Leading organizations have been more strategic, leapfrogging the competition by setting a foundation for the digital and data-driven enterprise to evolve alongside the changing landscape of technological innovation and greater customer expectations.
In this green paper, learn how the C-Suite and digital leadership can galvanize the team toward a proactive business direction that meets customers’ demands faster, delivers sustainable technology solutions, and defines data-driven ways of identifying critical decisions about new business opportunities to create differentiation and competitive advantage.
Introduction
Change is an accepted, pervasive business condition. No longer is the notion of flexibility to the forces of market change able to be postponed to next quarter, next half, or next year. Customers are fickle, brand loyalty is re-earned with each interaction, and the demand for frictionless, contextualized, and personalized experience is mandatory.
Enterprises that have begun their digital transformation journeys have accepted the new digital-first reality and are focused on developing disciplined, sustainable competencies, partnerships, and regimens. Infusion of digital DNA has penetrated leading organizations and radical transparency about leadership shortcomings and employee skills deficiencies is no longer an avoided topic; it is rather a conversational and planning norm.
To catalyze, measure, and achieve effective digital transformation, enterprises must prioritize modern, customer-focused, data-driven solutions underpinned by institutionalized Agile process. These foci require significant evolution to be responsive to both business and customer needs, and be prepared for the future.
To address the competing demands of speed, customer experience, and digital currency while successfully staying flexible, it is essential that enterprises think in terms of multiple dimensions: agile processes, human experience, and business architecture. When achieved, enterprise organizations will be in the practice of delivering the anticipated, iterative value.
Agile Practice
With the manifestation of continued change in the workforce focused on delivering agile transformations, it is important to define the transformation success not simply by whether velocity has been maintained. Within the past year, organizations moving forward with their transformation plans have been met with workforce challenges that are focused on the quality of the environment that the teams work in, and less about the products and services that they are tasked with executing. What this means for executive leadership teams trying to maintain growth and provide value for customers is that the teams on the ground perform at a higher level when their needs are met. Stability within the organization comes in many forms – communication excellence, process improvement, and executive support.
Communication excellence is the primary core component of agile methodology, and should be incorporated throughout the organization. Value-driven execution and delivery require consistent, reciprocal feedback that is clear and aligned to the overall strategic goals. Just as an agile team receives information regularly and uses it to improve their process and execution, organizations should adopt this cycle of feedback to keep focus on the teams working towards the broader goals. Executive leadership should build communication loops throughout their teams, reaching down to the individual contributors to ensure alignment.
The alignment of communication and overall effort in delivery can be bolstered by a solid standardized process that allows for flexibility and iteration. While a PMO may not function in the same roles that were previously held in traditional project management methods, the PMO will still heavily support the administration of a process and adhere to strategic goals. There is room in an organization for the PMO and agile teams to collaborate around processes that recognize and mitigate risks surfaced within the teams while fostering communication throughout the organization. Scrum masters as servant leaders are keenly in tune with the teams with which they work, and are valuable assets for the PMO to leverage when crafting solutions for any changes to goals and roadmap fluctuations. Successful organizations encourage close communication between agile teams which stay focused on both short-term and long-term goals.
Process and communication alignment should inform and focus any executive measurement reporting tactics. Metrics should be carefully curated to pinpoint progress towards target goals, opportunities, and team success. Organizations with ongoing data initiatives should consider leveraging specific team metrics to include in their analysis, which will provide a better view of the progress overall, and will shed light on areas that may need additional support or improvement. Executive teams can provide more consistent support by integrating the team data into their strategic discussions about goals, showing all teams a unified management approach rooted in empathy and positive motivation via transparent communication of goals and strategy.
The PMO should provide visibility into delivery and execution through open forums of feedback and process standardization. Alignment comes from a holistic understanding of the needs of an organization at all levels, translating down to executable tactics that can be maintained as the teams mature. The agile methodology at its core provides a foundation for a tailored project management approach that meets the team members where they are and also provides important feedback for executive teams making critical business decisions.
As an organization goes through its agile transformation, it is important to champion the process and dispel any preconceived notions about what agile methodology means to the teams using it. Agile processes are an established answer to attaining productivity and achieving the highest quality results in the face of non-stop change.
Design Systems and Adaptability
Design systems are an organization’s strongest defense against CX strategy being disrupted by change. Design is affected by product evolution, external trends, emerging capabilities and best practices, and also by technology change. By focusing on high-level design principles and guidelines, design systems are able to anticipate and provide coherence between existing elements and entirely new additions to a platform. Design systems also provide a way of thinking about visual layout which should be able to extend to cover new use cases without throwing away a platform’s look and feel.
Design systems which have emerged more haphazardly and serve more as a compendium or superset of what various applications and use cases have done in the past are more prone to disruption when new challenges arise. Similarly, component libraries may bring their own changes from look and feel shifts to micro-interaction updates. If the design system can answer the questions of how to incorporate the changes, the result should be harmonious. If a component library must be updated or switched out due to a technology change, it will be the design system’s integrity which will pave the way.
At the application level, technological change can disrupt CX in a number of ways: new capabilities in the platform try to find their place in already feature-laden screens, data source changes affect what is presented to the customer, and new channels (such as mobile) force interactions to be modernized.
Application interfaces which are focused on what a customer is trying to accomplish will be more easily evolved at a controlled pace. Interfaces that are too closely tied to technical implementation details, for example ones presenting information in a way that is not filtered by user needs, will likely be subject to a succession of changes that can distort the CX in unhelpful ways.
Architecture Anticipating Change
Technology architecture can provide a buffer against technological change, but it can also be disrupted by it. Industry trends can emerge which challenge previous designs and approaches. Cloud technologies, especially in AWS and Azure, are a compelling reason to couple code bases to offerings rather than front-load potentially unnecessary or futile abstraction efforts.
As with the smaller scale power of abstractions, keeping business logic and concerns in ordered places helps the whole from being intractably tied to one vision of the technology landscape. If business entities are well conceived and true to the real-world domain, persistence concerns may adapt and be refactored without challenging underlying intentions. Where code gets coupled to cloud offerings, for example in the use of serverless computing, well-conceived and maintained code libraries can still offer an important role in future proofing.
In server-side programming, frameworks and libraries offer risks as well as obvious rewards. Maintaining a sustainable framework needs an ongoing commitment from corporations. Teams utilizing under-supported frameworks risk suffering from poor documentation and the fickleness of flavor-of-the-month open-source communities. Where teams end up among a micro-community of users of a legacy framework, they can spend their time fixing the problems the framework was meant to handle, and the documentation limitations become a deficit in the organization’s own software documentation.
As with API coupling, coupling to frameworks should be done with a sideways glance to the possibility of having to de-couple later. Frameworks provide more lasting benefit when they set out concerns in layered architecture which allows project code to be well structured, because that project code can be lifted and shifted if needed. Very idiosyncratic representations of business logic forced by an inflexible framework will make a ground-up rewrite more likely later on.
A safeguard against the changing landscape could be summarized as keeping portable, clear separated representation of:
· Business logic
· Business object representations beyond immediate persistence needs
· Data transport representations
If the view logic and persistence layers can be maintained such that they lend themselves to being ported to different platforms later, all the better. However, keeping the business logic safely out of the front-end, and keeping business logic out of persistent entities, means that the pain of necessary migrations will be limited. Modularity across service boundaries can represent an even stricter adherence to layering; however, not all architectures are created equal. Inaccurately identified service boundaries can lead to a conflation of responsibilities in microservices as challenging as in traditional monoliths.
Architecture in its documented form represents another piece of the durability puzzle: the ongoing maintenance of strong internal knowledge of code and designs. Adapting to change will always be easier for an organization that has placed an emphasis on these considerations:
· Understandability of the solution and code
· Understanding of the solution and product within the organization
· Documentation of the solution
· Test coverage via understandable tests
A final mitigation of the perils of change is to stay ahead of updates on technology already employed. Small changes between framework versions can often be absorbed easily, while a multitude of stacked breaking changes can become migration efforts in themselves. Currency of knowledge helps to equip teams with awareness of challenges ahead, and to give a more accurate sense of the ongoing level of maintenance, support, and sustainability of technologies.
The enterprise's biggest hurdle to resilience is legacy solutions still in play. That which has been deprecated or has hit its end of life presents bigger risks the longer it is utilized. Coping for years with such problems should not be taken as a guarantee of ongoing resilience. Community knowledge and availability of expertise diminish as technologies move further into historical mainstream usage. Adapting to modern development or cloud offerings can prove the final straw for solutions that have already been stretched beyond their designs.
Data Resilience
Modern architectural and organizational thinking has been fundamentally changed by Eric Evans’s book Domain-Driven Design (2003). The idea of domain-oriented decomposition and ownership has become pervasive in implementing applications using a microservices architecture. The domain-oriented design techniques move away from centrally planned monolithic architectures, towards loosely coupled autonomous applications.
Splitting application code into discrete chunks is a core theme of modern architecture. A strongly modular architecture increases responsiveness to business needs while reducing the need for global changes. It focuses on separation of concerns, with clean, separate layers front to back, containing user and service interactions, business logic, storage, and infrastructure.
In the data world, however, these ideas have been slower to percolate. Most organizations are still structured around a centralized ownership and governance model across the enterprise, with a data architecture consisting of operational data stores feeding into a central data warehouse and fronted by BI and reporting tools.
To build adaptation to change in its data architecture, an organization can think along the dimensions of modularization, governance, and infrastructure.
Data Modularization. A modular approach, guided by principles of Domain Driven Design (DDD) and data mesh architecture allows an organization to easily add or remove components from its data architecture as needed. The key tasks for implementing a modular data architecture are:
· Identify the domains involved in the organization based on the needs of the business. These domains should be as independent as possible with clear lines of communication and well-defined input and output.
· For each domain, identify the core business level entities, business services, and the interactions between them. These domains and services should be designed to be independent and self-contained, with clear inputs and outputs.
· Identify what data needs to be collected, stored, and processed for the domains.
· Create a virtual data layer on top of the various data sources, to better align with domain models identified earlier.
· Integrate the domains cross-functionally using interfaces appropriate to a domain, which may use API-based architectures or other methods of data integration like ETL or ELT.
· Continue to iterate and modify the domains and their interactions guided by the changing needs of the business.
Data Governance. A clear set of policies, procedures, and practices that can provide guidance on how data should be managed, protected, and used can help a business adapt to changes in its data architecture.
Below are the steps to setup data governance for a modular data architecture:
· Define roles and responsibilities for data stewardship.
· For the domains identified earlier, identify owners who are responsible for their management and use.
· Implement data governance controls like data quality control, data cleansing, data standardization, and data security across the various domains.
· Monitor and enforce compliance with data governance policies on an ongoing basis.
Data Infrastructure. A modern data architecture for a flexible data infrastructure, primarily using cloud technologies, enables an organization to adapt to change. The approaches to a modern data architecture can be:
· Using a cloud-based data warehouse and data lake for flexibility and on-demand resourcing like compute and storage.
· Using modern ETL/ELT tools and processes for handling large amounts of data in different formats.
· Using “DataOps” practices like test-driven development, continuous verification, continuous deployment and integration, and monitoring.
The traditional approaches to data architecture and management—on-prem centralized data warehouse, centralized data governance, and very manual operational control of the infrastructure—are no longer suitable for the needs of the modern, resilient enterprise. The principles and techniques described herein can enable transformative data-driven decision-making on a large scale, and improve operational efficiency, customer experience, or create new revenue streams.
Insights | Take Aways
Forward-leaning organizations have embraced the opportunity to accelerate digital transformation to advance beyond their competitors.
Transformations and modernization take years to complete, and must follow long term strategies despite the distractions of change.
PMOs and agile were born from the need to achieve the most important objectives while accommodating change. Organizations leveraging what they have to offer have a head start in staying flexible.
Design systems and CX thinking allow customer experience to be emphasized and optimized while accommodating technical and feature disruptions.
Technical architecture has always sought ideals of decoupled technical neutrality and unbounded foresight of potential change. The reality is that unexpected changes will always confound plans; however, strong architectural strategy coupled with adaptability and a culture of documentation will help changes be accommodated gracefully.
4. Data engineering offers the possibility of building in modularity to insulate one part of an organization’s data from changes in another.
Leading enterprises have already changed in meaningful ways, and have learned the techniques to keep doing so. All organizations will follow course to meet the forces of change and disruption, or will find their digital transformation initiatives hindered.
References
Domain Driven Design. URL: https://www.domainlanguage.com/ddd/
Modern Data Architecture - A Data Modernization Green Paper. URL: https://www.eqengineered.com/insights/modern-data-architecture
Embrace Modular Technology & Agile Process to Deliver Business Impact (Parts 1—5). URL: https://www.eqengineered.com/insights/embrace-modular-technology-and-agile-process-to-deliver-business-impact-part-1
Building Software with Empathy. URL: https://www.eqengineered.com/insights/building-software-with-empathy
Authors
Mark Hewitt – President & CEO
Mark is a driven leader that thinks strategically and isn’t afraid to roll up his sleeves and get to work. He believes collaboration, communication, and unwavering ethics are the cornerstones of building and evolving leading teams. Prior to joining EQengineered, Mark worked in various management and sales leadership capacities at companies including Forrester Research, Collaborative Consulting, Cantina Consulting and Molecular | Isobar. Mark is a graduate of the United States Military Academy and served in the US Army.
Julian Flaks– Chief Technology Officer
Julian is a relentless problem solver and hoarder of full stack expertise. Having thrown himself headlong into Internet technology when best practices had barely begun to emerge, Julian is happiest putting his experience to use unlocking business value. Julian holds a Bachelor’s of Laws from The University of Wolverhampton, England and a Master of Science in Software Engineering from The University of Westminster.
Ranjan Bhattacharya – Chief Digital Officer
Ranjan is passionate about building technology solutions aligned with business needs, intersecting data, platform, and cloud. He is a believer in delivering value incrementally through agile processes incorporating early user feedback. Outside of technology, Ranjan loves to read widely, listen to music, and travel. Ranjan has a BS in Electrical Engineering, and an MS in Computer Science from the Indian Institute of Technology, Kharagpur, India.
Margo Rabchenuk – Principal Consultant | Project & Program Management Practice Lead
With a background in technical project management, agency project delivery, and a Certified ScrumMaster®, Margo is focused on software program and project management practice for application software and web development. Margo is an adjunct faculty member at Rochester Institute of Technology, developing and teaching courses for agile certifications in the School of Individualized Study.
Anne Lewson – Director | Program Management and Consulting Services
Anne blends technical with practical approaches to deliver projects ranging from large data mergers to more detailed, analytical solutions for a wide array of internal and external stakeholders. Anne leads the project management practice by using an applied Agile methodology suited to our clients' requirements. Anne has a B.S. in Computer Technology/Computer Systems Technology from University of Nantes and has her PMP and PMI-Agile certification.