How to block ads

I got tired of looking at all the internet advertising and just install some ad-blocking software. Link to the article that inspired me, here. The plugins are for Firefox. Updating the hosts file was simple. I just searched for the file name "hosts" and then added the big list of advertisers to both files that were found. No more ads, but Yahoo looks weird.

Tips for implementing a BI project

I am speaking at AICPA in Las Vegas on Business Intelligence. My talk is supposed to be a "lessons learned" kind of case study on using BI. I developed 11 tips when rolling out a BI solution. Some of these may look very familiar:

Tip 1: When deciding what to show in your BI tools, use a balanced score card approach.
Balanced scorecards provide a nice framework for thinking about developing useful metrics

Tip 2 : Select right hardware.
We needed a “Data Appliance” like Netezza. Feel free to overbuy. Your future self will thank you.

Tip 3: Take your time building requirements.
Figure out who is going to use the data and for what. What are needs going to be a year from now? Three years from now?

Tip 4: Conduct a data quality audit.
Check for missing data, unusual variability, unusual stability

Tip 5: Make your data warehouse responsible for fixing data quality problems.
Don’t try to build in work-arounds. You will have bought yourself a bunch of maintenance headaches. Let the guys who are supposed to give you clean data do their jobs.

Tip 6: Provide some context for each metric.
Show trends, goals, and/or min-max for each metric. This will allow the exec to decide if some metric is worth further attention.

Tip 7: Enable drill down on your charts (but don’t overdo it).
When an exec sees something “anomalous” they are going to want to see if they can figure out what is going on. Computer users are trained to clicking on things they are curious about. Leverage this behavior.

Tip 8: Avoid being “flashy”and cool.
Keep your charts simple and redundant. Allow your audience to learn how to consume your BI quickly, not be impressed with your teams programming skills.

Tip 9: Conduct 1-on-1 sessions with senior execs to ensure that they found the BI tools useful and informative. Adoption of these things is much harder than technical implementation. Do anything you can do to drive adoption

Tip 10: Choose a software package for ease of integration.
Time spent integrating is not worth the loss of strategic use of the data. Remember, the time you take to get things working right has a cost to the business.
All of the major BI vendors have very similar functionality and differences are not likely to have any impact on business decision making

Tip 11: Be ruthlessly simple about what you metrics you show. Complexity is your enemy.
Strive for few, but very meaningful metrics. Too often, you are going to want to create complex reports. Fight the urge. They will never be looked at. In this context, I will always sacrifice completeness for simplicity.

What is the secret sauce in direct marketing?

I am a big fan what I call tribal wisdom. Kevin Hillstrom put together a post of database marketing tribal wisdom. The item that most resonates for me? Segmentation and treating segments differently for marketing purposes. This was one of the first things I took from my McKinsey experience. Much of what he talks about are just specific instances of differential treatment of segments. Nice post.

How should you manage your relationships with recruiters?

As the old joke says, carefully.

I get calls from recruiters at a fairly constant rate and have 3 principles that I use when talking with them. Note that most of the folks I speak with are retained search recruiters, they are paid, in advance, to fill a position. Companies tend to use retained search firms for more senior jobs and the search firm has an exclusive arrangement with the firm. I occasionally get a call from a contingency search firm. These firms are paid when they fill a job and are not exclusive. Most of what I am going to say is applicable to retained search folks who have a more relationship based business. Contingency folks are more transactional, so relationship building may not be as critical. Even still, making friends is always worthwhile. On to the principles.

First, always take the call. I speak with every recruiter that calls, even if they are recruiting for a position that I am not appropriate for. Someone gave me the advice that you should cultivate a recruiter network. It was good advice. In order to build that network, you actually need to speak to them. So, even if I am not right for the position, I chat with the recruiter. Often, especially for analytic jobs, they don't know the space, help them understand the job rec (truly!) or refer them to someone else. I always try to make the calls a positive experience for both of us. Even if we just chat about raising kids.

Second, if you are not interested in a job really try to pass on a referral. I almost always pass on a name. This means that I need to spend a couple of minutes looking through my contacts and see who might be appropriate for their job. One of my former direct reports is my go-to guy for referrals. If I am not interested in a job, he gets the referral. This helps him build his network and helps me deepen my relationship with the recruiter.

Third, be honest in your assessment of your interest level. If you are not the right person for the job or the job is too small, tell the recruiter. Don't try to get the interview for the practice. You'll mess up your relationship with the recruiter. Having said that, I have let a recruiter talk me into interviewing at a company, even though the scope was too small. The company agreed and then built a job around my skills. I wound up not taking the job, but I was up front about my concerns and they decided to proceed with the process, anyway.

I consider my recruiter network a real asset. Every job offer I received (I had 3) was through a recruiter. A good recruiter network will make your job search much easier.

Skip level meetings

For part 2 of the Dafacto meeting series I present: Skip level meetings.

I love having regular skip level meetings. For those who have never heard of a skip level, the term refers to the direct reports of your direct reports. I try to hold a 1-1 meeting with each of my skips every 4-6 weeks. In my last organization, I had regular skip level meetings with about 9 people (all of my US folks. I did not hold regular skips with my overseas staff, though I met with almost all of them when I went to visit). Including the weekly 1-1’s with my direct reports, I typically had 7 hours of meetings with various staff members. Obviously, this was an enormous investment in time. Was it worthwhile? Absolutely. Otherwise you are dependent on your directs for information regarding things like staff morale, organizational issues, project progress, etc. I know staff found them valuable.

What did we talk about? Though the skip owned the meeting agenda (notice a theme), the skips were really focused on two things. First, staff development. I wanted to learn the staffs’ career and personal goals. We would use the time to talk about if they were making progress against those goals and what could I do to help.The time was very focused on their careers. In fact, after my first meeting, each person had to put together a development plan so we would have a structured conversation about their development.

Second, we talked about their projects. We talked about what was going well and what was not. People knew that I was fair and that if a project wasn’t going well, they would often tell me in those meetings. There were a couple of times where one of my directs was not living up to his commitment to his staff and the folks on his team needed a channel to be able to voice their concerns. Also, I would get to learn what was going well and use that information to give folks special recognition (I gave bottles of wine and gift cards) and visibility both in the team and to more senior executives.

One last thing. As a manager, it is very easy to move or cancel these meetings. After all, these folks are on your staff and are going to understand that stuff comes up. Right? As a leader, it is a terrible idea. If I had no choice, I would sometime postpone, knowing that this was sending a bad message to the staff member. I tried would make sure that they knew I did not want to cancel the meeting, and reschedule as soon as practical. The worst thing is blowing off the meetings. You wind up alienating the staff instead of helping them.

Staff Meetings

I am amazed at how many managers don't have staff meetings. Do they think they are not necessary? I think regular staff meetings are a critical management practice for, well, good management and leadership. And don't get me started on weekly meetings with direct reports or skip meetings. First things first. Staff meetings.

When I first started managing multiple groups, I found that I was repeating the same news over and over again. Also, some of the groups were working on complimentary projects or were interdependent and I was increasingly acting as the communication bridge between groups. So, I started to have staff meetings. I have always taken the same general approach to my staff meetings. First some general practices.

First, who owns the agenda and runs the meeting? Not me. Never me. Typically, one of my direct reports who I am starting to think about promoting. I want to give them the experience of running the meetings. They are going to need to do it themselves soon enough. Also, I don't see any reason to control the agenda. If I want to talk about something, I'll ask to have it on the agenda or just bring it up in the meeting.

Second, who attends? This depends on the organization and the needs. If you are often talking sharing confidential information, then a small, senior staff meeting is the way to go. If you want to use the meeting for sharing information across groups, then invite the senior folks and maybe their direct reports. I have seen people invite their directs on odd weeks and include the skip level folks on even weeks. I tend to go with inviting the larger group and make it clear that what is discussed does not go outside the family.

Third, how long? Between an hour and an hour and a half.

Typical agenda?

1. Company Updates. I use this time to talk any big company or departmental news that is relevant. Typically, this time was spent explaining why the company, my boss, or myself was doing something that did not seem to make sense to the staff. Some senior folks are pretty command and control. like to have a pretty tight reign on the discussions. I would rather use the time to share information.

2. Ken Rona updates. I give the team a sense of what I am working on. I do this so the staff can act as effective agents on my behalf and bring up any items that would materially affect my work. In this way, the people on the team can proactively participate in helping me solve my problems. Also, I really liked discussing my work in front of the team. Not only were folks helpful and pushed my thinking, it is good for morale. people like having the transparency.

3. Direct Reports update. My directs share their project lists. The agenda keeper is responsible for putting together an update project list for the group for every meeting. Mostly I focused these discussions on time lines. Are we going to meet this commitment we made. I think having to affirm the commitments in public, every week, keeps people focused.

4. Information sharing. We share interesting team outputs. I am a big fan of sharing information across silos. I often find that someone would do an analysis and share with the team, only to find that either there was a better way to do the analysis or that we could reuse the analysis for another internal client.

Oh, another tip. Have someone bring food. We did not use catering. We rotated this responsibility and reimbursed the cost of the food. It would have been easier (and more expensive) to have it catered, but I liked that people could bring their own style to the catering.

Kenny's 1 rule of segmentation

I was reading the segmentation post on Analytic Engine and it reminded me that I wanted to post my standard segmentation rant. First a story. If you don't know anything about Marketing Segmentation, check out the Wikipedia post.

When I first got to A fOrmer empoLoyer (think a large web portal), I found that we were big users of Prizm clustering, by Claritas. For those of you who have not seen the Prizm product, it is cool. The folks at Claritas have taken the whole US population and ran a cluster analysis on, well, us. They have classified each household into 66 unique segments. The segments group households that have similar lifestyle and socio-economic traits. They are a really neat way to do market to a target a pre-defined demographic and get some basic insight into who your customers are.

One problem, though, A fOrmer empoLoyer's targeting efforts were not based on simple demographics. We built fairly sophisticaed models to predict who would accept a given offer. In our case, Prizim clusters were rarely predictive, over an above the other variables we had available. People at the company had tried to use Prizm (and some of the other Claritas products) for all kinds of marketing-y things and they just did not provide value.

So, some bright person at A fOrmer empoLoyer said "we need to develop our own segmentation scheme. We'll provide a Universal Segmentation" (that is really what it was called) that can be used for any marketing or targeting activity." Strike two. The Universal Segmentation was a worse solution than Prizm and never made it out of the lab. Actually, A fOrmer empoLoyer took several runs at building a custom segmentation scheme using cluster analysis. None of them were found to be useful. As an aside, I mentioned the Universal Segmentation project to an expert in segmentation and he laughed and laughed. It was a bonding moment.

So why did A fOrmer empoLoyer have some much difficulty in using a tool that is in use by marketers everywhere? In both cases, the segments were not built with A fOrmer empoLoyer's needs in mind. They tried to do too much.

To finish the story. A fOrmer empoLoyer had been sending out a mass email to the whole customer base as a way of stimulating engagement and generating page views. The program was, uh, less than effective. I convinced the Customer Engagement folks to let us build a custom segmentation scheme for their email newsletter program that categorized each customer on the basis of the content they visited. In the spirit of transparency, Omniture did most of the work, they had the data. The thought was that the content someone viewed was a good proxy for their interests.

From the segmentation, we found that A fOrmer empoLoyer had less than 20 unique customer segments, but that very few segments described most of our customers. The Member Engagement staff started to create newsletters for each of the large segments. We had just started to use the segmentation scheme before A fOrmer empoLoyer stopped marketing, but the first campaign had much higher open rates than the mass mailing approach.

The lesson here is that when you initiate a segmentation project, you need to be really thoughtful about what you are trying to accomplish. Don't use every variable that you have available and just start cranking the k-means. Your segments will not be interpretable, and thus (you never see thus any more) won't be actionable. You need focus.

Instead, think about what you are trying to accomplish (e.g., be able to classify your customers into demographic groups) and build datasets that only include variables that are actionable or would give you insight about your customers. Build a segmentation scheme for one purpose. And when the guys at Claritas say "we have been doing this for 20 years. We have this nut cracked. Use our segmentation scheme", ask yourself, why do they have several segmentation products?

I could say some other stuff about creating segments, but if you focus on what you are trying to accomplish and build your dataset accordingly, you should have good results.

Break Down The Wall!

The other day, someone asked me what I do to break down silos between functional teams. First, let me say that I don't think breaking down silos is an academic exercise; you almost always generate value by talking to folks outside your group, department, business unit, whatever. I can think of at least 10 examples of really impactful projects that have come out of "Silo Busting." Second, creating an environment for silo busting is not hard. It just requires some time. However, the way you attack your silos varies depending on the construction material.

Just keep in mind, some silos are harder to bust than others. If structural conflicts (e.g., working together means that a rational person would do worse off by cooperating) or long standing animosity between silos exist, the problem becomes a lot tougher.

Set expectations.
When I first take responsibility for a team, I set the expectation that we are going to make things better. Every day. And that means innovating. And that in my experience, innovation often comes when two people who are very different talk to each other about their areas of expertise and their problems. So, there is just an incentive to getting to know your colleagues and what they do; Opportunity for improvement often results.

As an aside, I actually have an expectations document that I share with all of my staff; innovation in the service of continuous improvement, etc is one the core values listed in the document. Also, I set an example. I am often going out to lunch to meet with folks whom I have no obvious business connection and I encourage my staff to do the same.

Information sharing
In most cases, people are working in silos not becuase they enjoy it but becuase they are too busy to share information. There are some easy tactics that you can employ to facilitate information sharing. A couple of things I do is have whole team staff meetings where and monthly round tables. During these meetings, we discuss the projects we are working on if there are ways that the group can help. Barriers fall very naturally. Also, as I mentioned above, I try to go out to lunch with people outside my business unit and encourage my staff to take their clients (and anyone else they think they should get to know better) out to lunch. To provide an incentive, I offer to pay for it.

Personality or Competence
Other times, it is a personality or competence issue. Here, I run at the problem head on and really set the expectation that the staff has to work together effectively. I then put the staff members in situations where they have to work together to be successful (e.g., some special business initiative.) The expectation of being an effective collaborator also becomes part of their development plans and makes sure the problem gets fixed. If we get to the development plan stage, then I am going to be paying close attention and actively trying to help the staff member be an effective collaborator. Typically, by working together, people reach some accommodation, build some level of relationship, and break down the silos.

If it turns out that the problem is due to competence, well, no amount of relationship building is going to matter. If the staff member on my team is causing the problem, then a performance plan is going to be put in place. If it is a staff member outside my influence, I will likely have a conversation with their boss about their performance. Good luck.

The Analytic Value Chain - Understand the relationships in your data

One the dataset is together and the data is QA'ed, New analysts want to dive right in and start mining the data or building statistical models . More experienced hands just putter. The run descriptives, they check to see relationships that they thought they would see actually exist (age and income are positively related, for example), creating some histograms to look at the frequency distribution of the variables.

Why all the putterage? An effective analyst needs to develop an intuition about the data. Without that intuition, they won't have the common sense required to make some of the judgment calls they are going to need to make when they get down to modeling. Data analysis requires judgement (which is why you hire experts to do it) and without a good intuitive understanding of how the data behaves, the judgement calls to come are going to be on an unstable foundation. And you get to understand the business dynamics (by understanding what variables drive outcomes) in a way that the people who are running the business can not.

Another point, you are going to find relationships that were unexpected. These unexpected relationships are things you should take note of, you are going to want to follow-up on them and understand if they are real. These unexpected relationships are the beginning of finding what I call "Game Changing Analyses", the home run of data analytics. What do I mean by game changing? Analyses that lead your business partners into changing their business strategy, that indicate that activities that are fundamental to the business need to be reassessed. A good analyst should be able to have this kind of impact multiple times a year.

For those data analysts who read the blog, developing the intuition is critical for your ability to have impact and your reputation. Imagine that you are presenting some work to a senior executive and they ask a question that you can't answer based on your analysis. If you have a good understanding of the data you can say "I can't answer your question definitively, but given what I know of the data, I believe this to be the answer to your question. Of course, I will check the answer as soon as I get back to my desk." To the extent that you consistently get these kinds of questions right, you be come a trusted resource for the executive. I know more than one analyst whose job was saved because they proved over and over again that they were an expert in the dynamics of a business unit.

Upshot, spend the time getting to know your data . If you are a manager, build time to explore data into your project plans. Understanding the relationships in the data is a critical part of the data analysts and managers job.

The Analytic Value Chain - QA the data

Related posts
The Analytic Value Chain introduced
Defining the problem
Determine Data Requirements
Locate Relevant Data
Extract, Transform, and Load the data

Sorry for the long delay between posts. I have been focused on my job search and just got my head above water.

I won't do a big song and dance on this part of the value chain, I already did a post on QA'ing your data, here.

A fast story on how the prevalence of data quality problems . I was speaking with someone whom is an expert in the direct marketing industry about a brief sojourn in consulting. She went into consulting because she likes doing novel things and she thought that consulting was going to provide that variety. She was lamenting that all of the engagements she worked on were similar, with "correcting data hygiene" as how she spent most of her time. The problem is endemic.

No matter where you get your data, a third party or an internal data mart, you have to check the quality of the data. And regularly. Assume data is guilty until proven innocent.

"Growing" a SAS analyst

The other day, someone asked me how to “grow” a SAS business analyst. My first thought was “Let Capital One do it for you”. I then got to thinking about what it means to “grow” someone. I think the question was really “what skills does a SAS analyst need?” I was talking to David Ye, who is a senior manager on my stats team about this problem and he noted that there is often a difference between SAS programmers and SAS business analyst. Just to be clear, I am talking about a business analyst role.

Putting aside things that make a good analyst (being a voracious and tenacious problem solver, have a good understanding of your business dynamics, good written communication skills, etc), an analyst who relies on SAS as their primary analytic tool needs to:

• To be able to pull their own data (SQL skills and Proc SQL)
• Know how to use SAS efficiently (can’t overstate the importance of this, think temp tables…)
• Have a good understand the analytic procedures available (and options) they need for their job (I like Proc Means, Anova, Reg, Corr, Cluster, Factor, Chart, Plot, and Tabulate. Also, if you are doing serious experimental work, you need GLM and Mixed)
• How to leverage the various programming options (Macro, SAS Code)

Some of these are easier than others to develop. Most of these skills are hard won, so if you are trying to train someone up on anything but the most basic Procs, my advice would be to hire an experienced person and have them train new staff. Someone who is new to SAS, in my opinion, needs someone close by to answer questions.

The Analytic Value Chain - Extract, Transform, and Load the data

Related posts
The Analytic Value Chain introduced
Defining the problem
Determine Data Requirements
Locate Relevant Data

Extract, Transform, and Load (or ETL) is database administrator talk for getting data ready to analyze. For those who want a more formal definition, check out Wikipedia. I'll talk a little bit about each, but most of the action is in Transform.

From the data analyst perspective, there is not much to say on extraction; you need to get the data out of your systems and you need people who have the requisite skills. Think SQL. If you are hiring data analysts make sure they can write SQL and can architect a simple database. It will be a big help when they need to merge data sets or give requirements to your IT staff.

As I said above, transformation is much more interesting, involving both cleaning the data and creating new variables for analysis. Lets talk about data cleansing first. By data cleansing, I mean how your team handles missing values and outliers.

Though unglamorous, data cleansing is a critical task. Without having a consistent method for data cleaning, everyone will invent their own method. Once people start sharing data sets, a lack of consistency leads to bad analysis. Further, you want to ensure that your vendors and your data warehouse play by the same rules as your analysts. You might think it is obvious that missing values should be coded as missing, but your DBAs are not analysts or statisticians and they have different priorities.

True story on handling of missing data: We had one data set that tracked values over of something over time, call it Revenue per Week. If Revenue per Week for Time2 was missing, the DBA pulled the values from Time1 and plugged it into Time2. In our case, we found data from t1 was being used in t71. As a result, the data set was unnaturally stable over a very long period of time. Because of our data quality checking efforts, we found the problem and now missing data is coded as missing.

The second part of transformation involves creating variables. As I have said before, I like a focused approach to data analysis. If you start transforming variables for analysis (taking the square and cubes, etc.) you add variables. Seems to run counter to taking a focused approach, I know. I am not suggesting creating every possible transformation, but you are going to need a few to help you capture non-linear effects or normalize your data.

In my experience, if you take take the square, the log (base 10), and the inverse of your variables you are going to get most of the value out of your transformations. You are going to need to use some common sense (what is the square of a categorical variable?) but in general, you are not going to need to go crazy creating new variables. However, your data analysts are going to want to create every possible transformation that they can think of; it is easy and they might need them later. Serious emphasis on might. In my opinion, the time would be better spent thinking about what transformations are actually useful. Also, each of those transformations creates another column of data that has to be processed, slowing down your analyses. My recommendation is to use the big three and if you can logically think through other variables that may need to be transformed, tackle those on an one-off basis.

Another type of transformation is creating interaction terms. I had written a long and involved descriptions of how interactions work and why you should care, but I deleted it; I think interactions deserve deserves its own post. Interactions capture additional impact of two variables combines that is not captured by considering each variable separately. The short version is that you can (and should) create interaction variables by multiplying variables together. The challenge: it is difficult to know which interactions to create. You could guess by now that I am not in favor of creating every possible interaction terms, you would create an enormous data set that would not be useful. I am a fan of creating interactions that, due to your understanding of your business, you think exist and making them a regular part of your transformation process.

Last piece, Load. I have nothing to say here. In most analytic platforms, once you have extracted the data, it is loaded. Really, the process for data analyst should be called ELT.

Take aways? You want to be thoughtful about your transformation process. People often create tons of variables as a proxy for thinking deeply about the problem they are trying to solve. Creating big, dumb, datasets have real costs associated with them. Not least of which is that they are hard to analyze. Also, create a standard method for data cleaning and make sure everyone knows the standard.

The Analytic Value Chain - Locate relevant data

Related posts
The Analytic Value Chain introduced
Defining the problem
Determine Data Requirements

This post overlaps with the previous Value Chain post on determining the data requirements, but is meant to be a bit more tactical. If you have followed my advice from the previous post, you have a good sense of what data you are going to need. Now you need to find it. In most organizations I have worked with, the data to solve any given business problem exists. The challenge is that the data often exists in a place that is not accessible to most of the folks in the organization. The data may not be in a production environment (it is sitting on a server under someone's desk) and if it is in production, the data warehouse might be so large that no one really knows what is in there (this is not an uncommon problem in real warehouses. I had an expert in physical warehousing once tell me that a really good warehouse knows where a specific pallet can be found 80% of the time). I once had both problems at the same time. I found two data sets that answered a critical business problem when combined, but were running on different desktops in two different parts of my client's organization. I found the data by luck, but wound up doing a very impactful analysis. My value was in carrying the data sets, on floppy disks, back to my PC. Obviously, you can't analyze data that you can't find.

So, what to do? I don't have a ton of advice here, but I have found somethings work pretty well in identifying the data that is out there and making it accessible. First, treasure your staff that really know your data infrastructure because they have hard fought knowledge (for those in my organization, you know who you are. And you know how much I value your contributions. You also know that I am understating.) There is no replacement for just having experience in your data infrastructure. Having said that we rely on people power, metadata helps. And even the best staff are not going to be able to find useful data if you don't have data dictionaries for every dataset.

Second, create tables that aggregate your most useful data. We do this and can get our hands on useful data sets pretty quickly; in minutes. We evaluate the variables in those tables about once a year (or after any major strategy shift) to ensure that the data set maintains its usefulness. This data set has an additional value in that it can be shared with your entire organization and forms a common "fact base" for the organization.

Third, try to think ahead and ask your team to be on the lookout for certain types of data. There are business questions that I know I am going to want to take a look at and by communicating to the team my topics of interest, they can make serendipitous discoveries.

The Analytic Value Chain - Determine Data Requirements

Related posts
The Analytic Value Chain introduced
Defining the problem

So, once you have a good understanding, you should probably grab every piece of data you can find, transform the variables every way you can think of and start analyzing to figure out the variables that impact your dependant variable, right?

This way lies madness and spurious results. And is an approach that a naive data analyst would take (experts in data-mining might have a different take, but I was trained as a research scientist and I don't like the kitchen sink approach.) Before you start grabbing data, you should spend some time thinking about the behavior you are trying to predict and what phenomena might drive that behavior. Let me go a step further, I recommend that you generate hypotheses on what relationships you expect to find. Write 'em down. I go so far as to create something I call a conceptual map that graphically shows how I think the all of the variables, including the dependant, impact each other. Once I see the relationships, I can quickly generate hypotheses. Maybe you are thinking, what does this have to do with determining your data requirements? Hang on, I am getting to it.

Once you have the hypotheses, then you can start thinking about what data you need to test those hypotheses. In my world, it is important to have a good idea of what kinds of analyses we are going to be doing because the most time we have access to many more variables than we can possibly look at. We can't go on fishing expedition. Some other reasons to think hard about your data requirements:

• In rapidly changing businesses, you'll spend more time finding data than analyzing the data. So, parsimony is key, you don't want to spend time getting data that is not going to be useful.
  
• Most large enterprises have real restrictions on who can access specific types of data. So, even if you find the data, you'll have to figure out who owns the data.
• Merging and analyzing larger datasets takes processor time. The smaller the datasets, the less time the analysts spend waiting for results.
• The world is complicated. The more variables you have, the greater the chance of finding a spurious correlation. Also, if you have a good set of hypotheses and a conceptual map, you'll have a better sense if a relationship makes sense.
  

In sum, planning for your data needs, though it takes time up front, saves you time in the end.

Principles of Innovation

Over the past couple of years, I have been thinking about innovation. Specifically, what (and should ) I do with my teams to encourage innovations. Someone brought to my attention a video from Google's Vice President, Search Products & User Experience, Marisa Mayer.



Marisa talks about nine guiding principles that she believes are key enablers of innovation in her business. I would not argue that we all should adopt her principles, just that it was an interesting discussion and that having such a set of guiding principles is probably something worth thinking about. Mine are (somewhat redundantly with Marisa's):

1. Transparency is your friend.
2. Don't be afraid to iterate and experiment. Always test. Without it, you will not learn.
3. Noble failure is ok. The corollary is that stupid failure is not ok.
5. Share the credit. There is plenty to go around and you will likely have more than 1 good idea in your life.
6. Try to work around people you would be honored to hang with - and recruit someone when you find them! I am fortunate in this regard.
7. Don't have an opinion. Have facts.
8. You can build a business on users, but you need to run it on money. So, while you are innovative, think about how you are going to make money with the idea.
9. Be brave (but not disrespectful) in the presence of senior executives. That is how most of them got to be in their position.
10. Encourage dissent in your organization.
11. Be dissatisfied, but don't be a jerk. It makes you want to make things better.

The Analytic Value Chain - Determine data requirements

Defining the analytic value chain post

This year, I have had three conversations with vendors who are trying to sell my company some piece of third party data. We currently have well over a thousand variables available to us for modeling and insight, before transformations; our problem is not a lack of data. Our problem is that we have too much of it. On a recent data mining project, we included over 6000 variables in the dataset. So, why do I even mention determining data requirements as part of the analytic value chain? If you can include every variable under the sun, just grab 'em all? This way, you don't have to make any hard choices about what to include and what to leave out of the analysis.

I wish that things were that simple. For well understood problems,

The Analytic Value Chain - Defining the Problem

Related posts
The Analytic Value Chain introduced

One mistake analysts make when starting analytic work is to just jump right in with the data. I have had staff waste weeks of time learning that they (and me) misunderstood the problem being asked by our business partners. I am not proud of this, but it is a lesson learned. What have we done to mitigate the problem? We try to have kick off meetings for every large scale analysis so that the person doing the analysis (and their manager) can get a good bead on the problem the business is trying to solve. In these conversations, we spend a lot of time trying to understand why our internal clients think something is a problem, try to get a sense of root causes (that drive our data needs and analytic choices), and get agreement as to the deliverable (as well as timing.) We also try to set some ground rules around changing the problem in mid-stream. Once we kick off an analysis, any change in the request resets the timeline that we provided for the analysis. Another best practice here is for the manager to check in with the analyst a week or so after the analysts starts the work.

So, the first way that an analytic staff can add value is to help their internal clients define the problem.

The Analytic Value Chain

I am currently hiring for a Director for my statistics team and it is a hard position to hire for (if you think you might be qualified, please send me a resume.) A perfect candidate has to have skills all across the data analysis value chain. I imagine you are thinking "consultant speak"; maybe so, but I think of analyses as a product that needs to run through an analytic "factory." You can manufacture impactful analyses, just like GM manufactures cars. And I think of the value chain as having 9 steps.

1. Define the problem
2. Determine data requirements
3. Locate relevant data
4. Extract, Transform, and Load the data
5. QA data
6. Understand relationships in data
7. Do the analysis
8. Create presentation materials
9. Share results with business

It is hard to find someone who has the business skills to understand the problem faced by my internal clients, has the technical skill to manage the analysis, has the process re-engineering skills to improve the process (it is a factory, remember?), and the communication skills to present the results back to the business.

In the next couple of weeks, I'll discuss each of the steps in the value chain. Hopefully this set of posts will help folks as they set up their own analytic shops.

Best of Breed or One Vendor?

Analytics folks need a good set of tools to do their work. I won't get into a SAS vs. SPSS discussion (different tools for different situations. Ok, I got into it) but I did want to talk really briefly about choosing an analytic "stack." By stack, I mean the set of entire tools that enable high quality analytic work; The tools that you need for data extractions, transformation, and loading, data QA, analysis, reporting and data mining, etc. In my organization the stack includes Netezza and Sybase as our database platform, SAS for our data ETL, QA and access, SAS for our ad hoc analysis/data mining, and Business Objects for standard reporting.

if you look not too closely, you can see a common theme. SAS is the dominant vendor. Do we use a lot of SAS because they make the best products for each part of our stack? No, but their products are good and they are well integrated. By concentrating our purchases on one vendor we reduce the integration costs; we are assured that the products are going to work well together. Or at least that we can hold someone accountable if the disparate products don't play nicely. In fact, SAS wrote a Netezza database connector for us when we needed it.

The other option is to take a best of breed approach and buy your products from multiple vendors. The advantage in going best of breed is that you can really tailor your stack to your needs. For example, you may need a lot of control over the charts created by your analysis tools and only one vendor allows you to have that control (I once worked for a company whose tick marks on charts had to point away from the chart. You can actually select this option in Excel, but most vendors don't allow such fine control. Having such a fine level of control over the look of the output was considered a "requirement" for our business intelligence platform.)

So which approach should you take? My advice is to fight the logic of best of breed. I know it is difficult to pick products that are not the best fit for your situation. Fight it. In my experience, most of these products are more similar then different. Think about word processors. Most of us use Word, but would not shy away from trying Open Office (actually, I am writing this post using Google Documents). Word processors all have the same basic functionality. Word may be very full featured, but most of us just use it as a text editor with a touch of formatting. Similarly, the stats packages are more similar than different. Those little features or very specialized functions that seemed so important when you were making your purchase decisions won't make a lick of difference in the quality of your team's analytics. What does make a difference to your team's effectiveness is their ability to deploy and exchange data between applications. There are times when you need to select a "best of breed" application; when you to use proprietary analytic techniques that are offered by a single vendor. But in general, you are better off giving up the perfect analytical app for one that is well integrated into an analytic suite. My advice is to stick with one vendor for your analytic stack or at least minimize the number of times you change vendors as data moves up your stack.

Helpful tips for off-shoring

I may do a longer post about my experiences with my team in India, but for now, here are somethings you should keep in mind if you are thinking about off-shoring reporting and to a lesser extent, analytics. Here are some practical things you can do to help the effort be a success.

1. Put process in place. By this I mean that you need to have a documented process (and process maps) for each report that is being run. Documentation includes things like data sources, business owners for the reports, how to run a report, what to do in cases of failure (with each type of failure enumerated), etc. This should be a living document that gets updated as reports find new ways to fail to run (we keep ours on a wiki). And the process of updating should be part of the process documentation. I have a recommended book, Business Process Management: Practical Guidelines to Successful Implementations that is a good reference. By putting process in place, you make the responsibilities of both the US and off-shoring side very clear. This clarity is critical. In fact, we won't start producing a report in India until the US side does documentation for an existing report. Trying to do off-shore reporting on an ad hoc basis will be very difficult.
2. Be very careful about checking skills. We have not had a hard time finding qualified candidates on paper. We have had a very hard time finding qualified candidates in life. We have had a number of instances where candidates have grossly misrepresented their skills. Worse than anything I have seen in the US. Each candidate now has to pass through multiple screening tests of their skills. The tests are both oral, given during the phone screens and interviews, as well as written (given on site for candidates who have made it to an in-person interview). The tests are not hard, but you can't fake knowing what a proc freq is during an interview. We tried to give a pretty comprehensive written test to prospective candidates after the phone screen to be completed before they came in for an in-person interview, but we found significant cheating. Lesson learned.
3. Check the references. Nuff said.
4. Don't hire job hoppers. In the India market, anyway, there is some job hopping going on. It is not uncommon to find candidates who have taken several jobs and moved on. Don't think that they are going to treat you any different. We invest a significant amount in hiring and training and we need to make that training pay off. Also, I want people to become part of the culture. We won't even look at hoppers resumes.
   
5. Make sure that the US side is invested in success of the off-shoring efforts. In my case, we track utilization and report quality of the team at our weekly staff meetings. I am the person on the US side who is ultimately responsible for the India's team success and sharing metrics with the rest of the leadership team ensures that both me and my staff stay focused. Also, if I find that someone is not using their India resource effectively, I will take the resource away.
6. Use the off-shored staff for project based work where they can be fairly self sufficient. The original vision for the India staff was to be the equivalent of a US analyst. These were unrealistic expectations. The US staff works with their clients everyday and are much more able to solve problems both proactively and on an ad hoc basis. The time differences makes it much more difficult for the off-shore staff to find the people they need to speak with and they are, by the nature of the distance, more removed from the day to day needs of the business. In our case, production reporting was a perfect thing to move. Reports are produced on a regular basis, allowing the analyst time to get familiar with the infrastructure needed to run the reports and learn what the results mean. Processes can be documented and, in the case of staff turn over, be easily transitioned to someone else We kept ad hoc reporting stays in the US. Maybe over time it will move to India, but for now, we are staying put.
7. Travel! Both you and your senior off-shore staff need to travel to each others locations, at bare minimum, a couple of times a year for a week. Also, think about bringing your more junior folks over once a year for 5-6 weeks. That will given them an opportunity to meet their US counterparts and build relationships that they need to do their job effectively. I went to India recently and found the experience invaluable. the trip gave me a first hand appreciation on how difficult managing the time zone differences are. I also got to play cricket. Make sure you get your shots and carry a small pharmacy with you. I got a very small cut that turned into a bad infection in about 8 hours. Thankfully, I had Cipro and Bacitracin with me. I treated the cut myself and it turned out fine, but it was touch and go for a while. Just to reiterate, the trip was invaluable.
8. Meet the staff regularly and use video. I have a weekly pull up with my manager in India and a monthly round table with the whole team. I also hold a bi-weekly "office hours:" where folks in the US can stop by and give the manager and I feedback on how things are going. I just got tired of all the complaining about things not working right and now people not addressing their challenges in a forthright way. By having these forums, people have no excuses. Another tings. I found that when I was in India it was impossible for the off-shored staff to understand what was being said on speaker phone. The phones cut in and out. And no one said anything. We now try to use video for meetings whenever possible.
9. Ask other folks what has worked for them. I got good counsel from a number of sources. Some of what I have listed above is redundant with that advice, but I agree with their advice.
   

I would not say that we have off-shoring nailed, but I think we are making good progress. Our next step is to actually offshore advanced analytic work, but we have just started. Once we get our feet wet, I'll put up a lessons learned for advanced analytics. Are their other things people have learned that should get added to the list?