Technology Architecture & Projects

Agile Meets The Recession

It's no secret to anyone about the state of the economy, and the issues organizations face in keeping their businesses stable, including cutting costs. This also leads to the "do more with less" syndrome that project managers are facing on their projects - assuming that they aren't being canceled wholesale along with other initiatives.

Lately, I have been hearing flack from clients and in my Intro to Agile course that I teach at the University of Wisconsin-Milwaukee about two agile methods, pair programming and refactoring with respect to cost and time.

Pair programming is germane to Extreme Programming (XP), but can be used in almost any situation or methodology as desired. The gist of the technique is that while one developer sits at the computer and codes, another developer is looking over his shoulder catching errors and brainstorming with the first developer as the code is being written. They switch roles after a time, and members of the project team pair up with other members after a few days or a week (roughly).

While I am not personally a fan of the practice, I have seen it used fairly successfully in the past on specific projects. The argument against it now, which has come through loud and clear in the past few months is simply this: "We cannot afford to pay two programmers to do the work of one." That pretty much sums it up - the budget is cut to the bone, and the project cannot pay for additional staff to adopt or continue the practice. Not having (or refusing to spend) the cash to support this technique makes all technical arguments in its favor moot.

Refactoring code is being scrutinized heavily as well, particularly if it's "do-over" code that optimizes the system, but provides no new functionality or capabilities for the business or end-users. The argument here is "We cannot afford to refactor working code to optimize it unless it can be shown that there are distinct and immediate advantages/benefits to the business in doing so." Can't say I blame the business and product owners here as well if there is only enough budget to take one pass (or maybe two) at development in each iteration.

There are a number of methods to mitigate such risks, and if you're told the team cannot pair program and/or to keep technical (not feature-related) refactoring to a minimum, you have some choices to make regarding the project.

The primary method to mitigate these risks is to do some design up front. Not Big Design Up Front (BDUF), where artifacts use lots of paper and diagrams to describe a system and take many months to produce (as well as having dubious value to the programming teams). Rather, we do enough design to enable single programmers to write code that requires minimal refactoring or tweaking down the road for purely technical reasons. The thought here is that the time spent on design pays for itself in two ways: reduced staffing cost; and the reduction of technical debt to a point that significant amounts of refactoring are not necessary.

The business may complain that time spent in useful (that is, useful to the development team) design inhibits getting started on development, and we're agile, and we need to get a system up and running quickly. However, they cannot have it both ways - we either spend some time and money on intelligent, useful design, or we spend them on refactoring and higher developer head count during development iterations. I think that some up-front design is a better investment of scarce time and funding than doubling programmer headcount and/or rewriting code to 'optimize' the system without much benefit to the business users.

This line of thought also doesn't address changing requirements, timeframes, and other project uncertainties that are faced every day by development teams and their sponsors. That is a topic for another day, but suffice it to say that as good as Agile methods are in addressing uncertainty through iterative development and small, focused teams, we don't get those benefits for free, and never did.

Architects and Standup Meetings

Many organizations I consult to or are intimately familiar with have embraced certain concepts or 'features' of Agile development while eschewing others - in particular, the daily "standup" meeting where team members transmit and share information with leadership and each other. As I have remarked in the past, these meetings are highly useful when properly facilitated, but are not useful (and perhaps damaging) to projects when not run well with a controlled format.

Oftentimes, an IT architect is faced with supporting multiple iterative projects simultaneously - providing the overall architecture/technical direction to multiple teams working in concert, but very focused on their specific portion - their project - of the overall approach. Of course, leaders for each team have their separate daily standup meetings, and here is where trouble starts for architects in this type of role: they can spend almost their entire workday, every day, attending standup meetings.

I have been in the architect's role on large projects managed in this fashion utilizing multiple project leads/managers/scrum masters, etc. In each case, these folks wanted me to attend their daily standup, and while its great that they consider me part of their respective teams, I decline the invites because once these teams grow past a quantity of say, three, a substantial part of my work day will be consumed in standups - that leave little room to get "real work" done or attend to other important issues.

This also takes into account the fact that a lot of standups go well past the allotted 15-30 minutes for various reasons, are held in numerous locations including different worksites, and generally deal with granularity and minutae that the specific teams are dealing with - code, testing, etc., that don't always need architectural input and direction.

Architects still need successful interactions with these teams on a regular basis - so what to do about it? When talking with project leads and facilitators in such situations, I offer the following:

1. I usually attend specific project standup meetings once per week, and generally speaking, Wednesday through Friday is better so the team has some work completed or in process as the week progresses. Mondays are not real effective because everybody is usually settling in from the weekend.

2. If there are architecture-related impediments or issues that need quick resolution, I'm always available to meet with teams or individual project leads and members as such needs dictate. This is not a standup meeting, but one to resolve issues or impediments, and are focused on those topics alone.

3. When there are architectural work products or issues to discuss, a separate meeting for such topics is required with the necessary attendees invited. From my experience, these types of meetings do not work well in a 'standup' format.

What I am not advocating here is that architects assume an "ivory tower" approach to these issues - becoming pompous that they somehow "grace" a meeting with their attendance and inputs. The fact of the matter is that in the scenario I'm describing, architects' time and efforts must be used prudently, and spending 1/2 to 3/4 (or more) of every day in various project standups is not going to be very productive for the architect spending such time, and the teams trying to gauge where they are on projects and resolve issues as quickly as possible.

Support your iterative/Agile project teams, but know your specific role and apply your time and effort wisely.

14th Carnival of Enterprise Architecture Coming January 15, 2009

The 14th Carnival of Enterprise Architecture that I host through Blog Carnival is coming up on January 15, 2009, with submissions due by January 14, 2009. If you have an EA- or IT-related post that you'd like featured through the Carnival, you can submit the Permalink URL and related information here.

This is a very easy and effective way for EA and IT bloggers to get their messages across to a wider audience, and I hope that EA and IT bloggers will consider submitting their best pieces to the Carnival. Please note that you do not have to be the author of the material to submit it, as long as it: a) meets the guidelines below; and b) honors the author's copyright and fair use guidelines.

As always, here are a few pointers for article submissions to the carnival:

* Off-topic material, sales pitches (direct or disguised) for products & services, and content spam will be not be published. Prior to every carnival episode, I delete a number of pieces of obvious spam, vendor ads, and off-topic material prior to publication - and it's a complete pain in the rear, believe me. Please don't waste your time or mine and thanks in advance.

* Feel free to nominate blog posts of others for the Carnival, but make sure to identify the author!

* Material posted in previous editions of the Carnival will not be re-published, regardless of who submitted it and when it was originally submitted.

Again, the deadline for submissions is January 14, 2009, and the carnival will be published on January 15, 2009. I look forward to all worthy contributions!

The 14th Carnival of Enterprise Architecture that I host through Blog Carnival is coming up on January 15, 2009, with submissions due by January 14, 2009. If you have an EA- or IT-related post that you'd like featured through the Carnival, you can submit the Permalink URL and related information here.

This is a very easy and effective way for EA and IT bloggers to get their messages across to a wider audience, and I hope that EA and IT bloggers will consider submitting their best pieces to the Carnival. Please note that you do not have to be the author of the material to submit it, as long as it: a) meets the guidelines below; and b) honors the author's copyright and fair use guidelines.

As always, here are a few pointers for article submissions to the carnival:

* Off-topic material, sales pitches (direct or disguised) for products & services, and content spam will be not be published. Prior to every carnival episode, I delete a number of pieces of obvious spam, vendor ads, and off-topic material prior to publication - and it's a complete pain in the rear, believe me. Please don't waste your time or mine and thanks in advance.

* Feel free to nominate blog posts of others for the Carnival, but make sure to identify the author!

* Material posted in previous editions of the Carnival will not be re-published, regardless of who submitted it and when it was originally submitted.

Again, the deadline for submissions is January 14, 2009, and the carnival will be published on January 15, 2009. I look forward to all worthy contributions!

One Reason We're In A Recession...

Looks like these idiots thought that my "Measuring Technical Debt" post yesterday merited attention due to the "great money management tips" I offered....

There are myriad ways people lose money in the financial markets, and this is obviously one web-related way to do so.

Disclaimer: I am NOT an investment advisor. Not federally insured either. Some investors can and do lose money. Your mileage may vary. Specific investment performance is not guaranteed. LOL

Measuring Technical Debt

A number of bloggers including yours truly have been revisiting the concept of technical debt over the past month or two, given the parallels between it and the financial kind making headlines every 10 minutes; and also the fact that projects are being reined-in across numerous organizations due to budget, economic, and headcount issues caused by difficult current economic conditions.

Martin Fowler recently posted an interesting piece about how one would go about measuring technical debt, and I'll add a few other items to consider.

One thing about technical debt that often gets overlooked is that it cannot be reasonably measured or estimated until it has been incurred - that is, after the fact. Financial debt scenarios are estimated in a more straightforward way because certain critical factors such as interest rates and payback periods are known up front, reducing the uncertainty, but not eliminating it.

Martin also mentions that measurements of technical debt are subjective because the resulting system is measured against a system in, as he put it, "an imaginary state." While this is true in a substantial number of cases, I believe that some (not all) of the subjectivity in such measurements can be mitigated if an overall design/architecture exists up front. How much design and architecture? That's also quite subjective, but such artifacts would establish a baseline that facilitate objectivity in measuring and controlling technical debt.

Then we come to the debt measurements themselves. The most straightforward metric with technical debt is time - how much time will/did it take to build/refactor assuming that the overall system was "clean?" Those estimates come in units of time as that what the project team utilizes daily and reports in standups or project meetings. However, time as a form of 'debt' usually doesn't resonate with non-technical managers - but money does. For example, I can tell a project sponsor or product owner something along the lines of Martin's example - 'if the system was 'clean,' we incurred 2 days of interest on our technical debt." While that usually gets acknowledged, the ramifications aren't clear.

What does get their attention is when that 2 days of interest is converted into monetary terms. The straightforward way of doing that is: (Cost of involved team members per day) X (Number of days of interest). For this to work, we must know how much 1 day of labor per team member (hours work also) is costing the project and multiply it by the number of team members involved. Then multiply it by the days (or hours) of interest paid. Now, we have an approximation of how much the technical debt cost in financial terms, which will definitely get management attention if its high enough (and hopefully not high enough to get the messenger shot).

Again, keep in mind that these measurements are made after the technical debt is incurred, and are rough approximations, not cold hard facts. Having a design/architecture as a baseline improves these measurements, otherwise the 'imaginary state" (or design/architecture after the fact) as Martin describes must suffice.

If anyone has better methods or optimizations on this, I'm all ears and eyes.

The Most Critical P-Word Of All

Nick Malik opines that measuring customer/sponsor/stakeholder behavior on projects should allow more of them to succeed. Nice thoughts, but I don't think it will fly, at least not in a lot of organizations.

Why? We focus primarily on what I call P-words: People, Process, Projects, Programs...and yet, there is one P-word missing; and it's the one that trumps all of the others almost every time:

Politics.

While Nick looks at projects as a partnership between business and IT - which it should be - it more often isn't, and the model most frequently used is the customer-contractor model. The business "hires" IT to develop systems and manage projects just like people hire general contractors to build or renovate their homes.When and if things on the project go awry, whether caused by the customer's actions or not, the contractor is much more likely to be blamed, take the fall and suffer the consequences - whatever those are.

Other project situations that may become politically-charged are when there are opposing forces/viewpoints against the project from other powerful people in the organization - including non-sponsor stakeholders that have competing interests or don't want things to 'change' in the manner in which they operate their portions of the organization. You can try and 'measure' that behavior all you'd like, but in the end, it won't much matter because a political crusade is being waged, not some rational discourse that leads to logical, obvious, and optimized conclusions and action.

Nick asks, "The project scorecard can be used to demonstrate that the right behaviors are happening on both sides.  After all, if a project fails because the business sponsor was unwilling to buy in to the approach, or wouldn't sign off on the interface design, or because the business users wouldn't participate in the test process, why should the IT team take the rap for missing the dates or overruns in cost?"

Because in the culture of a number of organizations, IT always takes the rap for project budget overruns and missed dates, whether it was responsible for them or not. That's largely the result of politics at work, not process or maintaining scorecards on unhealthy behaviors on the part of project sponsors and business stakeholders. Granted, sponsors and stakeholders who develop track records for sponsoring projects that repeatedly fail or cost too much often fall on their swords - eventually - but it usually takes more than one discrete project failure to have that happen. Often it takes several and in some cases, it never happens at all.

The best one can do in these types of situations is the following:

1. Perform the appropriate level of risk management on and in all phases of the project - or in every iteration if it's an Agile effort.
2. All risks should be communicated to the business, not only the consequences of the risk events, but what the remedies (if any) will be.
3. If risks are triggered for any reason, document it. Share it with sponsors and major stakeholders. If you encounter resistance or objection along the lines that Nick described above, there are reasons for those attitudes/behaviors (whether rational or not), and you must find out what those are and hope there are remedies short of project failure/cancellation.
4. Deal appropriately with the consequences of customer-triggered risk events. Business sponsors certainly have managers looking over their shoulders, but going over a sponsor's head with 'scorecards' and 'metrics' may backfire on you huge. Do that repeatedly, and let me put it this way: it's the wrong time in economic history to be unemployed.
5. Project managers must have an intimate business relationship with business sponsors and major stakeholders. There must be trust developed or already present on both sides, and when there is, that's a prime reason why successful project managers make the statements that Nick cites in his post about their sponsors. In such cases, there is no need to document business sponsor behavior on a 'scorecard.'

Projects are a lot like life - scorecards are useful for some things, but relationships matter far more in the end.

Discipline, not Dogma

I received a few interesting comments to my "Bastardization of Agile" post, in particular this one from Pawel Brodzinski:

'While I agree with most of your points I strongly disagree with that one:
"Picking what parts of Scrum or XP are appealing to the PM, Scrum Master, or product owner, and ignoring the rest or superimposing waterfall as related above. Disaster often lurks right around the corner."

Actually picking what parts of Scrum/XP/any other nice methodology suits your process fine and applying only those parts is wise. If one of my projects would benefit from stand-ups why not to start them even if we don't keep 2-week iterations? If I find (sic) alue in short iterations while I don't have designated scrum master what keeps me from implementing it?

My goal isn't to be agile or have pure-agile processes in software development. My goal is to improve my processes. You can always do it as they do it in the books - 100% implementation of some methodology - and it should work. Unfortunately it's quite an effort to go this way. On the other hand you can choose wisely different techniques from different methodologies and craft your own software development and project management process. Sure, I won't call it agile but as far as I can call it good I don't care.'

Therein lies the problem: in organizations that are very new to and/or inexperienced with Agile methods, picking and choosing usually isn't good, as I related in my post. If it were, I would have no stories to relate and nothing to write about...

The point of my post isn't about dogma, it's about discipline. The problems I related occurred primarily due to two factors:

• Inexperience with the methods used
• Advancement of specific political agendas and/or weak overall management

This goes against Pawel's assertion of "choosing wisely" from different techniques - because of inexperience, grasping for silver bullets, or competing political agendas cause substantial project risk to be (perhaps unwittingly) introduced or perpetuated regardless of original intent.

In my opinion, the better move would be to manage a few projects "following the recipe" with whatever 'new' methodology - Agile or non-Agile - the organization wishes to embrace to gain valuable experience with the selected methods. And, staying the course through project completion in order to evaluate how well the methods worked - and will work in the future. That's the point of retrospectives, lessons-learned meetings, and the old, somewhat politically incorrect term "post-mortem" - do the project following the roadmap, keep/optimize what works, and drop or replace what doesn't. 

The only project managers that can do what Pawel suggests successfully are those that have substantial and real-world experience in the methods under consideration. Those that don't have such experience are limited to what they read in the books and blogs or hear from others second-hand - making critical project choices from that data or opinions alone.  The risk of major time/cost/scope problems increases when approaches are combined without thought or realization about the implications, or worse, as my post describes, are executed poorly because of a lack of discipline, not dogma.

Dogmatic approaches to anything never served anyone well - but lack of discipline in approach has ramifications that are most often just as bad.

The Bastardization of Agile

It's always easy to tell when something is resonating within organizations - and Agile is now firmly in that camp based on my experiences over the past two years. There is a tremendous amount of interest in Agile methods and many organizations are taking their first or second steps with it. On the surface, this looks great, but some of my experiences have led me to believe that Agile is thought of not as a different, solid approach to software design and development, but more (as I originally feared) as a silver bullet, completely misunderstood, or license-to-hack with little or no accountability for outcomes.

Herewith are a number of 'incidents' that I've recently observed or participated in:

• Regular loss-of-control-and-context in standup meetings, which at times make them drag on for hours. Scrum masters and PMs/facilitators must be vigilant in staying to the script, and taking issues off-line when necessary - and it very often is necessary. The result of not doing so is completely wasting uninvolved team members' time. Think of it this way: if standups are routinely going for an hour instead of 15 minutes, usually for one team member's issues, you are wasting 45 minutes of other people's time X the number of uninvolved team members. If this occurs on a daily or near-daily basis, the amount of wasted time is huge. And, since you're operating under iteration and timeboxing constraints, wasting team members' time like this will cause iteration goals to be missed repeatedly, and perhaps kill your project. Take very-specific issues brought up in standup that don't involve the entire team OFFLINE!
• "We're going to 'organizationalize' Agile." Which translated means "we're going to superimpose our culture and previous practices into whatever Agile methods (or parts) that we've selected. Translated further, this means that what will be superimposed is a lot of politics and waterfall/SDLC when Agile doesn't jibe with the existing organizational scheme - bad, bad move. Every time I've seen this happen, the projects involved are in trouble, and some have failed outright
• Picking what parts of Scrum or XP are appealing to the PM, Scrum Master, or product owner, and ignoring the rest or superimposing waterfall as related above. Disaster often lurks right around the corner.
• PMs, Scrum Masters, or Product/Business owners reverting to the term "Agile" when making decisions or taking action that are not, by any stretch, Agile. Examples: ignoring the self-directed team concept in Scrum by timeboxing work without consulting the team - which in reality is "make my date" project management - pick the date, work backwards to today, and tell the team when the work they are responsible for is due, regardless of team inputs, risks, or other schedules. Another example: assuming 100% dedicated team members and scheduling work when, in reality, the team members involved are not 100% dedicated because of other responsibilities they cannot get out of or pass to others. Finally, there are multiple product owners (ouch) that must come to consensus on stories and other direction - often quickly - and cannot or will not. The team is then left spinning their wheels waiting or worse, attempting to get some work completed by guessing what the stories/intent are - and the risk of being wrong is often high enough to cause a great deal of problems.

Based on these experiences and what I've been told by my students and colleagues, I get the strong impression that Agile is often used as an 'excuse' to advance a particular agenda. It's usually mentioned as a 'selling point' for projects - particularly to management and project sponsors/stakeholders to get projects approved and funded. It is also used for behaviors that are not Agile at all - when the PM who played the 'make my date' card on the project schedule was challenged in a meeting for not consulting with the people actually doing the work about the proposed dates, she replied, "Look, these are week-long iterations. It's Agile!" No, it isn't when the team isn't involved; it's 'make my date' and the consequent rectally-generated analysis that always follows.

My advice here remains the same as always: if you're going to do Agile development - do it. Completely and without compromise. When I am challenged in my Agile classes about modifying Scrum or XP or FDD to 'fit' some organizational political or budget scenario, my answer is always the same: "If you do that, then you are not doing Scrum/XP/FDD/etc. You are doing something, but not Agile methods."

And its always that "something" that leads organizations and groups who rationalize projects in this manner to fail.

 

Carnival of Enterprise Architecture #13 Coming December 2, 2008

The (Lucky) 13th Carnival of Enterprise Architecture that I host through Blog Carnival is coming up on December 2, 2008, with submissions due by December 1, 2008. If you have an EA- or IT-related post that you'd like featured through the Carnival, you can submit the Permalink URL and related information here.

This is a very easy and effective way for EA and IT bloggers to get their messages across to a wider audience, and I hope that EA and IT bloggers will consider submitting their best pieces to the Carnival. Please note that you do not have to be the author of the material to submit it, as long as it: a) meets the guidelines below; and b) honors the author's copyright and fair use guidelines.

As always, here are a few pointers for article submissions to the carnival:

* Off-topic material, sales pitches (direct or disguised) for products & services, and content spam will be not be published. Prior to every carnival episode, I delete a number of pieces of obvious spam, vendor ads, and off-topic material prior to publication - and it's a complete pain in the rear, believe me. Please don't waste your time or mine and thanks in advance.

* Feel free to nominate blog posts of others for the Carnival, but make sure to identify the author!

* Material posted in previous editions of the Carnival will not be re-published, regardless of who submitted it and when it was originally submitted.

Again, the deadline for submissions is December 1, 2008, and the carnival will be published on December 2, 2008. I look forward to all worthy contributions!

Book Review: SOA Governance by Todd Biske

My colleague and friend Todd Biske recently authored SOA Governance - The Key to Successful SOA Adoption in Your Organization, which was published earlier in October by Packt Publishing. I offer a review of this work with the disclaimer that not only is Todd a friend, I was given a review copy of his book gratis by his publisher. I also recommend Todd's blog highly for his views and tips on SOA, governance, and architecture in general.

While the book provides some technical details regarding SOA implementation, it focuses more on SOA adoption from a business and techno-functional perspective - that is, how enterprise architects, project managers, IT management, and business analysts would comprehend and address SOA as opposed to developers. In explaining SOA and governance concepts, Todd chose to use the "business fable," or "story" style where real-world scenarios and dialog are added within chapters to make his points. This style is similar to that used by Patrick Lencioni in his works ("The Five Dysfunctions of a Team:  A Leadership Fable") and Timothy Johnson (" Race Through the Forest: A Project Management Fable "). This style of writing, while rarely seen discussing technical/tactical topics such as SOA, is highly effective if used properly, and Todd's stories superbly reinforces the material at numerous points.

The book is comprised of the following topics:

• The definition and concepts of governance (with extensions to IT governance), SOA, and linkage to project management and the business
• Avoiding "Just A Bunch of Services" (JBOS) - Enterprise SOA Governance
• Service Versioning
• Business Analysis Governance
• Design-time and Run-time SOA Governance
• Roadmap to establishing and running SOA governance in organizations
  

A key strength in this book are the many links of proper governance and SOA concepts to the everyday problems and issues faced by IT architects/managers, project managers, and business analysts. In particular, Todd deftly develops and explains the tensions that normally exist between architects (taking a holistic view of IT infrastructure and SOA) and project managers who are under time, cost, and risk constraints to get their projects in production as they were originally planned. Most SOA texts I read come from a purely technical and technology-based perspective, and don't discuss the impacts on the people using, building, or guiding SOA. Thus, Todd's take on these issues is a very welcome addition to the literature.

As with most books that I read, I do have a couple of quibbles. The main one is the book pays little to no attention to data architecture and governance, which are key components of most SOA implementations since its data that's getting moved around almost all services implementations. Some discussion about this linkage would have been beneficial. Another issue that is not addressed, but probably should be, is the role that vendors play in these processes, because in most large organizations, they are a significant factor in most phases of inception, development, and operations. Finally, while the figures and diagrams in the text are numerous and very helpful, it would help the text greatly if they were labeled/tagged as opposed to simply inserted into the text.

This book is a must-have for all IT managers, architects, PMs and business analysts dealing with SOA issues, be they implementation, governance, or both. I also highly recommend this book for those who are starting or facing IT governance issues in general, even if they aren't contemplating or building-out an SOA at present - the governance principles, techniques, and advice Todd gives apply to much more than SOA.