How To Do IT Assessments (8 Practices for SMBs)


Nobody likes assessments. And why should we? No matter what we\’re told, assessments are really designed to show up our weaknesses — whether in skills, work habits or processes and practices. Yet, companies need to be able to do assessments to evaluate what they have in terms of strengths and weaknesses. Taos Mountain, started in 1989 and based in Santa Clara, CA, is a consulting company that specializes in the art of technical and staff assessment. Companies call on Taos to provide a third-party perspective on how well they\’re doing things, whether their staff has the smarts to tackle an upcoming project and to get a handle on processes when considering whether to outsource. Brandon Nutter, former technical director for Taos, shared seven of his previous company\’s practices when doing technical, process and staff assessments. (Yes, he was still employed at Taos when we interviewed him.)


–> PRACTICE #1: Develop tables of ratings and then apply them — to processes, technology areas or people skills.

Common sense, right? The Taos rating system ranks from one to five. As Mr. Nutter described it, ÒA five would be a best practices and very cutting edge in that area. Four would be best practices. Three would be more average or typical. Two would be showing some deficiencies. And one would be severe deficiencies.Ó

For technology areas, Taos has a set of criteria for each area, data management, disaster recovery, security, email and so on.

The same with personnel. Do it by type of job. In the case of systems administration, for example, Taos uses a set of 10 skills assessment categories and works through each during the evaluation. Where do you get the categories? Taos uses a set based on recommendations developed by SAGE, a technical interest group of the USENIX Association that focuses on systems administrators. (The categories appear in a booklet SAGE publishes on job descriptions for sys admins. Copies are free to members, who pay $40 a year for their memberships.)

The point is, no matter what the position in the IT crew, some professional organization out there has come up with a job description that\’s worth borrowing to develop the criteria for your evaluations.

In the sys admin area, Taos has modified the various job duties and created four levels of positions. Number four is the senior advisory level, three is the server-side system administrator, two is a desktop client system administrator, and one is more of a help-desk level. The goal, Mr. Nutter explained, is to figure out what an individual should be doing. ÒIs this person most suited to desktop support or great technical support and understanding, or can this person design your bills and your SMS layout? What is their core skill set, and where are they best used? We may have individuals that have a little bit of skill in multiple areas and we are trying to identify where their real strengths are.Ó

–> PRACTICE #2: Bring in an outside evaluator to help out with the assessments.

You\’d expect this one, right? But give it some thought. Bringing in an experienced third-party can help out by allowing you to compare your ratings with other organizations that have similarities, either in size, industry or resources. At the same time, an outside consultant can introduce you to analysis tools that you may not be aware of or just not know how to use.

A typical situation, according to Mr. Nutter, is a company that wants a security audit. ÒWe have a set of tools that we can use to investigate their environment. We can do scans and analysis from servers to look at if they\’re at the right patch level, if they have security holes or misconfigurations. We\’ve taken a lot of open source tools and tried to make those the basis of how we do the analysis. More important, we have a process for how we handle the data from these tools. We have a method where we can go through and find out some key issues that that data is telling us. It\’s pretty easy to get a slew of tools to dump out more data than anyone could possibly know what to do with. We start off using open source tools then we put wrapper scripts and process around that.

Pricey and time-consuming? You bet. But not so much that you won\’t have money left over to keep lights on in the IT offices. In the case of Taos, assessments take about a month and cost in the range of $20,000. (Of course, this varies, depending on the size of your company and what it is you want done.)

–> PRACTICE #3: Although most companies don\’t have one yet, learn a framework and apply it — even if you\’re in a small firm.

It hardly matters whether it\’s ITIL, Cobit, Six Sigma, CMM or something else newer. These frameworks really do work for managing, measuring and improving the delivery of IT services.

Taos has its own version of the Information Technology Infrastructure Library (ITIL), which is particularly relevant to smaller companies.

Said Mr. Nutter, ÒWe find that some smaller companies have a rough process, but there are a lot of things that they haven\’t given a lot of attention to and they have kind of given up on because of this feeling that the large systems in Cobit and ITIL are just too complicated and won’t really work. There is kind of a compromise, — a basic amount of a process that you can put in that is practical, that will work.

–> PRACTICE #4: When evaluating staff, look at how they\’re spending their time, but don\’t start by pulling out the stopwatch or make people record how they spend every minute of their day.

Taos just sits down with individual IT staff members and asks them where they\’re spending their time. ÒWe get pretty straightforward answers a lot of the time,Ó said Mr. Nutter.

Taos will also look at ticket reports to determine what percentage of work is actually being recorded in the ticketing system. They\’ll classify each of the issues by what level of resource would be required to resolve it and who could handle each type of ticket.

ÒIt is a common phenomenon especially in small to medium sized companies, that because they need to hire to get the skill set, they will hire senior Windows, sys admins, senior network. But they don’t really always have enough work to fully utilize the senior level resource,Ó he said. ÒSo we find that often, senior level resource managers are working on some mid-level tasks. We can help identify how that is happening.Ó

The same information can come out of the face-to-face interviews, which typically take about 90 minutes. ÒWe don’t necessarily just say, ÔWhat percentage of time do you spend firefighting?\’ We ask them to tell us more in detail — to break down how they spend their time during typical days, and maybe even keep track of their time on some days that they feel are typical. We will classify which of those activities we think will go in which bucket.Ó

–> PRACTICE #5: Staff has outsourcing on its mind. Be upfront about your intentions.

The first part of any interview should put the staff member at ease by expressing that nobody knows everything and that the intent is to find out what areas he or she has strength in and also what areas he or she would like to pursue more.

Although staff may believe the exercise is intended for the employer to downsize or outsource, typically, the Taos evaluation team has to reassure them that it\’s a matter of enabling the organization to reorganize and put the right skills on the right projects — and also to come up with training tracks to be pursued by individuals. Said Mr. Nutter: ÒWhat we actually find is that companies are pretty loyal to their employees. A lot of managers want to go through this practice as a best practice that they are providing kind of a learning path for their employees as well.Ó

Here\’s where the managers need to step in and be upfront — to acknowledge the downsizing or outsourcing elephant in the room. ÒMost of the people that we have talked to are diligent in making sure they are keeping their staff up to date and trained. Really, the motivation was in their best interest and making sure they are trained to do their jobs. This is an objective third-party way to evaluate on the technical side.Ó

Interestingly, soft skills — project management abilities, professionalism, leadership — don\’t get much attention by Taos during such assessments. That\’s because it\’s tough in a short amount of time to assess and be objective about. Those, Mr. Nutter said, are skills best left to the individual manager to assess.

–> PRACTICE #6: When evaluating IT skills, have technical people evaluate technical people.

The fact is, if your tech skills are out of date as a manager, you\’re not the best person to evaluate how good your staff\’s tech skills are. So what do you do? Here\’s what Taos does. They pull together other senior people to act as the evaluators. Get everybody together in a room for an hour or two with a whiteboard and no interruptions. (Also, no surprises. The individual being assessed should know what to expect.) The person on the Òhot seatÓ needs to describe a project he or she is currently working on or has worked on that involves creation, design and implementation. That person needs to explain what the project was and why he or she made some of the technical decisions that were made — as well as what the alternatives were and why they were discarded. The purpose of the discussion is to understand the person\’s ability to translate technical decisions into business motivations. It should only take about an hour or two.

As Mr. Nutter explained, ÒWe will ask why they didn’t do it this way. Well, that would cost a lot compared to this. How did you go about finding out if the customer wanted to spend that or not?… We tend to keep it fairly friendly. The goal is not to intimidate, but to see whether or not they can translate their technical decisions to business motivation.Ó

For example, he said, if somebody\’s designing a disaster recovery solution for email, there are lots of different levels they can shoot for in terms of availability — everything from one standby server built from scratch stashed in a closet to a couple of servers in an high availability pair, where they\’re sharing storage, to synchronous remote data replication. As he points out, the difference between doing synchronous remote data replication and doing an HA or the difference between a downtime of eight hours vs. eight minutes can easily mean an order of a magnitude in cost. Can the person being assessed get at what\’s meant when somebody asks for Òdisaster recoveryÓ? ÒSome people mean, ÔI want it backed up on tape and stored off-site.\’ And other people mean, ÔI want synchronous data replication remotely.\’ Were you able to identify what it is they are really saying and try to get at what their needs were, rather than saying that disaster recovery must mean this…Ó

Often, senior people will understand the ideas and consequences of their technical decisions, but they haven\’t had experience in mapping those to the business needs. You need senior tech people on board (whether internally or as provided by a service provider) who can do that.

–> PRACTICE #7: When the assessment is done, put together a list of recommendations and pick a project to get moving on.

Companies that bring Taos in typically get a scorecard to know how they rate in individual areas of the matrix, as well as a list of projects that should be done to mesh with company goals.

Said Mr. Nutter, ÒFor instance, if they say that security is a big issue and they really want to solve some of these problems they have been having, then in our recommendation, we will focus on some [suggestions] that will help them make some progress in the security area. We will also give an idea of some of the most critical holes we have found and an idea for a project that we think they should initiate in that area.Ó

As part of that Taos will also come up with a ballpark estimate for how long the project might take in dedicated people-weeks.

–> PRACTICE #8: Outsource those functions where you don\’t want to reinvent the wheel.

Mr. Nutter said that 80% of the time, when he visited customer sites and asked what projects they were working on, it involved the same chores being undertaken at every other company — choosing a spam solution, for instance. The value of using the right service provider is that that kind of research has already been done and can simply be re-applied. After all, that\’s how providers like Taos make their money: by developing processes to be as good as they can and repeating them over and over for multiple clients, constantly seeking out new efficiencies. There\’s no reason internal IT staffs can\’t do the same thing, but they often don\’t.

Useful Links:

Taos Mountain, Inc.

A sample scorecard from Taos