Monday 21 October 2013

Using Enterprise Architecture at a Media Company -- Part three: TOGAF

What is TOGAF?

TOGAF stands for "The Open Group Architecture Framework". But, I suppose that really doesn't tell you much about what it is. I like to think of TOGAF as analogous to a big box home improvement store, such as Home Depot, Lowes or Rona (Canada only). You go to a home improvement store when you need tools or materials for a building something or a home improvement project.  You might not get everything you need there and you may need to adapt some of the materials and tools to work properly in your home. 

Similarly, if you are trying to construct or improve an Enterprise Architecture, you can go to the TOGAF "store"  and get at least some of the materials and especially the guidance that you need. You don't need to get everything there, as you may be using other tools such as the Zachman framework. You will also likely need to adapt what you get from TOGAF to your needs, just like you adapt the materials you buy at a home improvement store to your home.  

Big box home improvement stores can be somewhat daunting places.  It is sometimes hard to find what you need, because the stores contain so much merchandise. In my opinion (and the opinion of others that I have talked to) TOGAF is similar.  It contains megabytes of text that makes suggestions and recommendations about how to do an Enterprise Architecture. In my opinion, there are a lot of good ideas/guidance there, but it can be a bit overwhelming, especially at first. From talking to others, some people give up on it because TOGAF seems like far too much effort, especially for a small to medium size business. Also, many Enterprise Architecture groups simply cannot get the necessary corporate buy-in to implement a large part of TOGAF as a single "big bang" type project. 

Luckily, just like you don't need to buy everything in your local Home Depot in order to fix your house, you also don't need to use every word of TOGAF to build your enterprise architecture.  TOGAF actually mentions this and spends quite a bit of time talking about how it must be integrated into existing initiatives (like project and portfolio management). 

In this blog entry, I will give a really high level view of TOGAF, and then, in the next entry, discuss how it was used in "the Sphere", a fictitious medium-sized media company, that has a small architecture team: one enterprise architect and five solutions/information architects (some of which are only part time architects).

A Fairly Quick, High Level View of TOGAF

I like to think of TOGAF as having two main parts: the architectural development method (ADM), and the enterprise continuum.  The ADM is a recommended set of phases for the development process of an Enterprise Architecture.  The enterprise continuum is divided into two smaller parts: the architecture and solutions continuum. Both of these contains building blocks (for architecture and solutions) respectively ordered from the most generic to the most specific (the four categories, from the most generic to the most specific, are: foundation, common, industry and organization). You can populate the architecture and solutions continuum with your own architecture and solutions building blocks, essentially creating a customized library of re-usable components.  This is a great way to encourage consistency between projects. You can also get building blocks from industry reference models, and you can use the TOGAF technical reference model as an extremely generic starting point. TOGAF recommends that you use some sort of repository tool to store all the artifacts/documents/models that you produce.

Keep in mind that there are megabytes and megabytes of documents describing TOGAF, so we really are just scratching the surface in this blog entry. Much of the TOGAF documentation contains suggested ways to use the components of TOGAF. The user of TOGAF is free to use as much of this advice as he or she wishes.

The Architecture Development Method (ADM)

The TOGAF phases (source: Stephen Marley/NASA/Sci)


TOGAF is divided into ten phases, each of which is further divided into steps and sub-steps. With the possible exception of the Preliminary phase (at the top of the diagram), TOGAF assumes that we constantly iterate through the steps in order to produce architectures. Although the iteration looks like it starts in phase A and continues sequentially to phase H, new requirements may force us to go back one or more phases and revise our work (which is why there is a "requirements" circle in the middle of the diagram). Realistically, we may also need to revisit a phase because we forgot something. After revisiting a phase, we may need to revisit the subsequent phases to deal with the effects of the changes we made.

Preliminary and Architecture Vision Phases

The preliminary phase is essentially the initiation of the Enterprise Architecture effort within an organization. The organization needs to decide to what extent they are going to use Enterprise Architecture, how it impacts the org chart and how it impacts business planning, operations management (technical systems and development) and project management. The organization needs to create and agree on basic architecture principles (such as "only one authoritative source for each data element", "avoiding overlap and duplication of applications", and "systems should be designed to operate in spite of component failures") as well as a customized version of TOGAF, which will likely evolve over time.  Identifying a place to store architecture documents (such as Sharepoint, Confluence, Wikis, etc) is also desirable and some architecture tools/methodologies (ERDs, UML, etc) may be chosen.

The Architecture Vision phase is the start of an iteration of the architecture process. Depending on how one sets up TOGAF, it may be triggered by a "request for architecture", which can be part of the project management/project initiation process in an organization. The architecture vision phase needs to first identify the stakeholders and should ensure that their concerns and requirements will be addressed. With this information, and the architecture principles defined in the previous iteration (realistically, you might need to go back and add some principles), TOGAF recommends that you attempt to define the scope of the required architecture activities and create  rough business, data, application and technical architectures. Although this may seem obvious to some, you should also make sure that the project isn't beyond the capabilities of the architecture team and that any necessary business transformations are possible. TOGAF recommends that you attempt to achieve consensus among various stakeholders before proceeding in order to avoid the (not unknown) scenario where you finish the architecture project and it is never accepted.  Finally, you should produce an architecture vision document which contains at least the preliminary architectures and requirements, and may contain a resource plan, KPIs, milestones, etc.  This should be approved by a sponsor and/or architecture board before proceeding to the next phase.

Architecture Phases

Common steps for all architecture phases

In TOGAF, the three architecture phases are broken down into the same series of steps (which are actually done somewhat iteratively):

  1. Select reference models, viewpoints, and tools
  2. Develop baseline architecture
  3. Develop target architecture
  4. Perform gap analysis
  5. Define candidate roadmap components
  6. Resolve impacts across the architecture
  7. Conduct formal stakeholder review
  8. Formalize the architecture
  9. Create architecture definition document
The first step is basically the preparation step for the architecture phase. It asks us to consider whether there are any well known generalized models for what we are doing. For example, if you are architecting a call centre, you may wish to use the One-VA technical reference model. You should also determine the relevant stakeholder's viewpoints for the architecture you are doing, the models you need to cover them and if any of the architectural principles apply.  You also need to decide what tools you want to use (e.g. Visio, Rational Rose, etc).

The first step is usually broken into the following sub steps, some of which you may not need to do in detail or may wish to skip:

  1. Determine the overall modelling process, selecting models needed to support all the views/viewpoints.
  2. Identify possible building blocks from the architecture repository
  3. Identify the matrices you want to build (or are required).
  4. Identify the diagrams that you will need
  5. Identify the types of requirements that you will need to collect.
  6. Select appropriate services, ideally using combinations of services from the TOGAF TRM.



The second step is to describe the existing architecture, at least to the extent necessary to show how the target architecture will change things. Where possible, existing building blocks should be used for this.  It's possible that all or part of the existing architecture was done as part of a previous architectural iteration.

The third step is build enough of a target architecture to show how things will change in order to accomplish the architecture vision. Sometimes the third step is done before the second. Where possible, existing building blocks should be used for the step, but it is quite possible that you may need to define new building blocks, which will need to be added to the architecture repository and placed in the architecture continuum.

The fourth step is to perform a gap analysis.  This means that you look for missing parts in the baseline and target architectures (and presumably fill them in) as well as look for the gaps between the baseline and target. You also look for conflicts between views of the system and resolve them by examining tradeoffs.

The fifth step is to create a roadmap that takes you from the baseline to the target architecture. The roadmaps built in the architecture phases can often be used to in the migration planning phase.

The sixth step is to consider whether the target architecture will impact any other architectures. You may removing building blocks that are used in other architectures, or perhaps adding capabilities that will be useful elsewhere.

The seventh step is to review the architecture with the stakeholders (possibly the ones whose viewpoints you considered in step 1) and make sure they agree you are accomplishing the architecture vision and they approve of how you are doing it.

The eighth step is to finalize the architecture, essentially filling in any necessary details in architectural deliverables (such as building blocks) and ensuring requirements traceability for what you've proposed.

The ninth and last step is to create the architecture definition document, using the deliverables and make sure it goes into the architectural repository.

Business Architecture Phase

Like all architecture phases, the business architecture phase goes through the sub steps given above. Given that you are working on a business architecture, it is important to keep the business drivers in mind. As with all architectures, you also need to make sure you are considering all stakeholders, including business owners and users.

A key activity in the business architecture phase is modelling the baseline and target architectures.  You can use a lot of different models for this, including data flow diagrams (where the processes are business processes), entity-relationship diagrams (or class diagrams in UML -- the entities or classes would represent high level business entities), information exchange matrices (and node connectivity diagrams) showing what information is exchanged between business groups, use cases and structured analysis (which breaks down key business functions and assigns them to organizational units). It's also possible to use UML sequence diagrams to document business processes, so you might want to use them as one of your business architecture models.

One of the activities in the first step in any architecture phase is investigating whether there are any industry reference models for what you are doing.  The U.S. Veterans administration has many such models available as do industry groups like the telemanagement forum.  You should also look internally for re-usuable building blocks (essentially little bits of business architecture). For example, you might be able to re-use architectures for things like invoice processing and approval.

Information Systems Architecture Phase

The Information Systems Architecture phase includes data architecture and applications architecture. Data architecture is often done first, but not always.  In some situations, the two are almost done concurrently because changes in one tend to result in changes in the other.  As with the business architecture phase, the basic architecture steps (given above) are followed.

Data Architecture 


As with all architecture phases, we need to start by considering the business drivers for architectural change, determine who the stakeholders are (don't forget the operations team or the internal/external auditors). We also need to consider how any new data we introduce will be created, distributed, migrated, secured, and archived. If the gap between the baseline and target data architectures is very large, we will need to make sure we plan a data migration as part of the implementation effort.

As with all architecture phases, we will use models to represent the target and baseline architectures. These models should represent the viewpoints of all the stakeholders. The changes in the target models should ideally be traceable back to the business architecture and/or the business requirements. Where possible, we should start by considering any industry reference architectures that may exist (such as ARTS for retail, and Energistics for the Petroleum industry). Probably the most basic data model is the list of data elements. From there, we can model the data using an ERD or UML class diagram. ERDs and class diagrams can be done at various levels of detail, including conceptual,  logical and physical.  It may make sense to do all three in this phase, although a really high level ERD or class diagram (which may essentially count as the conceptual model) may have been done in the business architecture phase. There are a number of other models which may be useful to cover viewpoints, including data flow diagrams (logical and physical), You may also wish to write documents relating data elements to business entities, business functions, access restrictions, and cover any interoperability requirements for data (such as data formats for interchange between components).

Application Architecture

The application architecture follows the steps given above for all architecture phases, and considers the baseline applications and how they must change, based on the business and usually the data target architectures in order to arrive at a target application architecture.


You can use a UML component diagram to model all applications and their flows (for both baseline and target architectures). You may also wish to create application portfolio catalogs and application migration documentation if the target architecture requires large changes to the set of applications.

Technology Architecture Phase

Technology architecture covers the hardware and physical infrastructure and may also deal with "software infrastructure" such databases and software which implements services from the TOGAF Technical Reference Model (TRM).  Like the other architecture phases, the technology architecture phase follows the steps common to all architecture phases, which are outlined above. The starting reference architecture for this phase is generally the aforementioned TRM.

It's possible to use a UML component diagram or a Visio network diagram to draw the hardware/software infrastructure for the baseline and target technology architectures, being sure to remember to document locations and communications/network requirements between components.  It may also be useful to produce documentation that ties the abstract services of the physical and technical infrastructure to those of the TOGAF TRM (see above). When you need to add new technology, you should ideally follow existing technical standards, where they are applicable. For any new technology that you wish to add, quite a few factors need to be considered (and of course, should ideally be put into the target architecture) including: performance, maintainability, location (and resulting network latency), requirements for availability and you will need to do sizing, costing, capacity planning, and migration planning. New technology might also be subject to your organization's technical governance processes (for example, PCI, audit compliance, etc).

Solutions and Opportunities Phase

In this phase, we go through the baseline and target architectures that we previously created, decide what needs to be done to bridge the gaps (and if it can in fact be done), and then create a preliminary plan which is finished in the next phase.

When we examine the gaps in the various architectures, we will need to look at whether business priorities will impose constraints on what we would like to do. For example, if we would like to replace a call centre, but opening new stores has a higher business priority, we may need to come up with an alternative to replacing the call centre. We will take this sort of thing into account as we consolidate the gaps from the architecture phases and then examine each one in order to make decisions on how the gap should be addressed. These decisions will then be documented (possibly into an "implementatin factor assessement and deduction matrix") as we go.

As we are examining the gaps, we may notice common requirements that span several business functions. These should be consolidated (or "factored out") to make sure we only address them once. As we determine solutions, we may end up considering applications that will need to interoperate. We need to make sure that we keep track of these interoperability requirements and deal with them by changing input/output specifications or introducing adaptor components.

We next need to look for dependencies between our gap-bridging solutions so that we can determine the order in which the solutions can be built, determine possible deliverable dates, and so that we can start grouping the solutions into work packages. As we continue to look more closely at the solutions, we always need to ask whether it is within the organization's capabilities to implement them.  If not, we will need to find other solutions. We also need to decide if a solution is completely new ("greenfield"), directly obsoletes existing systems ("revolutionary"), or gradually changes existing systems ("evolutionary"). It's sometimes also valuable to identify the "quick win" solutions and distinguish them from the ones achievable in the middle and longer term. Quick win solutions can be helpful to show that you are making progress in an implementation, although not everything can be a quick win. Besides classifying solutions, we also need to determine whether the components of our baseline architecture will continue to exist in the new architecture, will be gradually phased out or will be replaced as a result of the current effort.

Once we know what we wish to do (aka the work packages) we need to determine whether we will move from baseline to the target architecture in a single step or whether we will need to plan intermediate steps or transition architectures.  Ideally, transition architectures should deliver some (business) value or else they may be hard to justify.

Once we know what our transition architectures will be (or if we are going to use them at all), then we create the initial versions of the migration documents that we will refine in the next phase: the architecture roadmap, and the migration and implementation plan. We also update the architecture vision document, the architecture definition document and the architecture requirements as necessary.

Migration Planning Phase

In the migration planning phase, we finalize the migration documents we started in the previous phase. The migration phase essentially completes the architecture activities for the current iteration of the ADM.

The first step of the migration phase is to determine how/if the changes necessary to implement the target architectures will affect project/portfolio management, business planning and operations management. It may make sense to have one of these three management frameworks deal with some of the changes rather than the enterprise architecture process. It's also possible that the people who govern these frameworks may want modifications to the work packages, so it is best to find out.

Next, we try to assign a business value to each work package, considering ROI, strategic fit, or ultimate value to the business (value chain analysis). This analysis should ideally help get the work packages approved and is a good double check on whether the architecture is aligned with business objectives. Critical success factors can also be defined so that the success in implementing the work package can be measured.

The third step is to figure out what resources (such as people) are required to do each work package, how long it will take and determine whether or not the resources required can be made available for the required time.

We then try to prioritize the work packages based on cost-benefit and risk and then get the stakeholders to agree to the prioritization.

With all of the above information, we can finalize the architecture roadmap, update the architecture definition document (basically the baseline and target architectures) and generate the implementation and migration plan.

At this point, we are done the architecture activities of the current iteration of the ADM

Implementation Governance Phase

Often, we move from the baseline to target architectures in a series of intermediate steps, called transition architectures. In this phase, we monitor the implementation work that takes us from baselines to target architectures, possibly monitoring each transition architecture implementation as a separate step. Generally, there is some sort of formal or informal review process (such as a steering committee that meets periodically) to make sure the implementation is proceeding as planned and conforms to the transition or target architectures. During the implementation it is important to prepare any necessary changes to business processes or operations processes and make sure these are in place when the implementation is complete. Once the implementation is complete, it makes sense to do a "lessons learned" session.

Architecture Change Management Phase

Architectures will need to change -- that is a given. Technology is always changing and so are business priorities. The former means that sometimes a better solution comes along, while the latter can mean that the solution you architected is longer needed or needs to change significantly. Architecture change requests may be received through the architecture governance process and may possibly originate from operations management (possibly because a solution is not performing as it should) or from business process management. We might need to make changes because of the need to reduce costs, a certain technology becoming unsupported or because we have decided to standardize. Sometimes the required change may be small and doable very quickly without significant (or perhaps any) re-architecture work.  It's also possible that the requested change is already accounted for in a transition architecture that hasn't yet been implemented. Other times, it may trigger a "request for architecture" and we may need to iterate through the ADM.

Assuming that an enterprise architecture is being used to try and realize value in a organization, we need to monitor how well it is meeting business and operational objectives and make changes where there are problems or gaps between what is desired and what is being delivered. We also need to consider the effects of new technologies, which may make it possible to better meet requirements or meet them more cheaply. We should watch carefully for changes in business strategies to make sure that the enterprise architecture continues to meet the needs of the organization, otherwise, we risk having an Enterprise Architecture that was perfect for the business strategy two years ago, but not now.

When we or someone else discovers a problem or gap, we or they need to prepare an architectural change request. This request needs to be analyzed to determine the risks of implementing it and to determine how well a solution will fit with the current enterprise architecture. It's also important to determine if any service level agreements or whether the business value currently being delivered by the existing system will be affected. We need to propose changes to the change request if necessary in order to mitigate risks, ensure that SLAs are met and to ensure that systems continue to provide business value.

Once all of this is done, we need to hold a meeting with the architectural council (or the appropriate governing body, depending on how the change needs to be handled) to get their feedback, buy-in, and hopefully approval. Assuming we get approval, then we need to initiate the process to implement the change, possibly starting another ADM iteration, if the change warrants it.


Requirements Management 

Requirements management sits in the centre of the TOGAF ADM diagram because it operates continuously during all the phases. Essentially, it contains all the activities that collect, organize and get approval for requirements. The changes that result from the requirements are done in the other (outer) phases. Because requirements management runs continuously, it is hard to describe it as a phase.  Therefore we will refer to its steps as belonging to the "requirements management activity".  We will refer to the outer circles in the ADM diagram as the "outer ADM phases".

The outer phases in the ADM (especially the architecture phases) identify new requirements and then these are conceptually passed to the requirements management activity, where they are prioritized, approved by the necessary stakeholders and then put into a repository. At the same time that this is occurring, the ADM phase that identified the requirement(s) will modify or add to the requirements it is considering, noting the priorities determined in the requirements management phase and possibly changing them again. The requirements management activity then updates its repository as necessary, communicates the changes to stakeholders and architects, gets their buy-in, deals with conflicts with other requirements and prepares a requirements impact assessment. The current outer ADM phase is then responsible for determining the impact of the requirements on its activities and deciding whether to revisit earlier outer phases or to defer the requirements to a later outer ADM phase. As always, if the requirements change, the requirements management phase needs to update the requirements repository and try to get stakeholder and architect buy-in.

The requirements management activity also needs to note any requirements changes that occur during the architectural change management phase. These requirements will often (if significant enough) be resolved in a subsequent iteration of the ADM (i.e. by starting at the preliminary phase and going through all the phases again).






Tuesday 8 October 2013

Enterprise Architecture and Conflict Resolution

It happens to all Enterprise Architects sooner or later.  Someone doesn't agree with your architecture and they would rather implement a solution in a way that doesn't match the current technical road map.  Sometimes the resulting discussions can be quite civilized and profitable for both parties.  However, to make that more likely, and to prevent a disagreement from escalating beyond where it should, here are some ideas that I have found useful.

When you are first told that someone wants to deviate from your carefully thought out architecture, don't get defensive. I think this is quite natural when the objection to your ideas comes from someone more senior than you in the company and it may happen regardless. You don't need to immediately defend your work.  It's best to say something like, "That's an interesting idea, let me think about it a bit" and then go somewhere else and do just that.

Likewise, back away if you feel yourself getting angry.  Showing anger at someone else's ideas when they happen to conflict with your own is not going to help you.

In some organizations, an Enterprise Architect can "pull rank" and simply inform people that disagree with them that they will follow the Enterprise Architecture. Giving people orders is probably not the best way of dealing with the situation.  You may get public compliance but they could be resentful and may look for a way to circumvent you at the first opportunity.

Let the people who disagree with you know that you value their input and that you are glad that they feel strongly enough about enterprise architecture to come talk to you. Set up a time (usually not right away -- the next day or later would be good if the matter is of utmost urgency) to discuss their concerns in more detail.

When the meeting starts, let them do most of the talking.  You should stick to asking questions, pointing out areas of actual agreement (sometimes this works really well) and admitting to any mistakes that you might have made that actually led to the disagreement.  I know the second thing is hard to do, but sometimes when one side in a disagreement admits mistakes, the other side makes concessions as well.  The key thing is that you are not trying to "win" the argument.  You are trying to bring the sides together.

When they are done presenting their points, don't argue with them.  Thank them again for caring enough to come to you and tell them that you will get back to them as soon as you can.  Then, go away and think carefully about what they said.

You need to think carefully about the degree of disagreement.  Is this an issue where you need to go to the wall (i.e. the CIO/CTO) and insist on getting your way?  Keep in mind that could be damaging in the long run, but it is sometimes necessary. Is there any middle ground that is reasonable? Is it possible to use some sort of objective test to decide between the alternatives? Is this an issue that should go to the Architecture Review Board?  Normally, if you don't have enough support at the Architecture Review Board on the issue in question, then perhaps you should consider accommodating the request.

If you do have to oppose the request, try to refer to the organization's architecture principles when doing so. You might also want to give the person who disagrees with you a chance to address the architecture review board and make their case. You should find out if it is possible for them to do a proof of concept for their idea. The key thing is that you need to make sure that they do not suffer anything resembling humiliation as a result of disagreeing with your architecture.  Many times, the person opposing you will be a valued member of the organization.  Keep in mind that they may be more valued than you are :-) and treat them the way you would expect to be treated in their shoes.