Monday, August 6, 2007

Getting a pet project off the ground - Part 2

Link to Step 1.
Step 2
I don’t control the web properties for any of my company’s web sites, so if I wanted to introduce the organization to the benefits of site testing, I would need partners. So, the next step was to start evangelizing site-testing though the organization (BTW, the vendor was Optimost). This step has multiple sub-stages; create a sound-bite, chat the project up, and educate senior business leaders and others.

Sound Bite
I think that a common mistake with junior folks who want their organization to try a new technology, software, whatever, is to lead discussions with the technology. “Look at how cool this is!”

The problem is that a business leader is not going to care about the technology. They care about the problem. You need to make the problem come to life. So, I gave a lot of thought to being able to quickly explain the business problem I was trying to solve using as little jargon as possible...

You are going to use this sound bite in front of senior execs, so it is worth getting it tight and getting it right. Once I had someone interested in the problem, I could get them interested in the solution. In my case, my problem statement is: “We make site design decisions based on opinion, and not fact. In order to know the facts, we need to be able to rapidly test our site designs. We can use a “site testing” vendor to test billions of site design combinations in a matter of weeks. This will let us build a web page that optimizes for the things we care about, like generating page views, increasing the number of unique visitors, or contribution value.” We can argue metrics later, at this point; I was just trying to get some folks interested.

Chat it up!
So I had my sound bite, my next step was to start using it incessantly; in my meetings with my manager, her manager, my colleagues, you get the idea. I wanted as many people as possible to have heard the sound bite. This is really about laying the groundwork, getting people familiar with your problem and agreeing that this is a problem that needs to be solved. The reality is that big companies have any number of big problems needing to be solved. I was trying to get agreement that site design was an important one, one worthy of solving.

Over the course of the next month, I had a couple of meeting with Very Senior folks (regarding other projects) and worked my problem statement into the discussions (I really was shameless). Both of them agreed that our site design process did not take business needs into account, and that the organization really had no way ensuring that the site of gathering those facts; facts that we needed to optimize our page design. I suggested to both of them that I bring in a vendor and invite each of their senior staffs to learn of the benefits of site testing.

This was an easy sell. Most Very Senior folks want their staff to be more innovative and are happy to give a little push. If you try to go from the bottom of the organization up, well, I have tried that method and have had little success. I am sure it is possible, but in my very big company, people are busy doing their regular jobs and need that push to take on additional responsibilities. I had actually tried to get the organization interested in site testing about 6 months previously. One of my colleagues and I invited a number of junior staff to an information session and nothing came of it. While they appreciated the session, no one felt empowered to kick off a pilot.

One last piece here, you better have become an expert in step 1. One of the business leaders had used site testing in a previous organization and knew his stuff. So, I needed to be able to have a pretty detailed conversation with him in order for him to be confident that I was the right guy to push this project forward. Almost home!

Educate senior business leaders and others

As mentioned above, I asked the Very Senior folks who should be invited to the educational sessions and they both suggested inviting all of their direct reports. I then put together an email that invited the directs to a meeting. The email explained site testing and offered to have a second session for their direct reports. This is a critical point. We actually had 2 meetings. One for folks who could reasonably sponsor a pilot and one for the folks who would be responsible for pulling it off. The types of discussions are different in those meetings and I wanted the leadership to be excited by the potential while I wanted their staff to be interested in the execution. I actually had at least 3 execution level meetings for various groups in the organization, but one Very Senior meeting.

It worked out well, though truthfully, I was trying to generate as much support as I could. If one of the more junior folks would have expressed interest in implementing site testing, I am sure we could have worked something out. Once again, I don’t know if my approach would work in every situation, but I was trying to plant 100 flowers and watched to see which one would bloom. The nice thing about having the Very Senior folks engaged was that their influence could help break log jams.

Next step: Conduct a successful pilot

2 comments:

John Aitchison said...

I was interested in your "sound bite"

“We make site design decisions based on opinion, and not fact. In order to know the facts, we need to be able to rapidly test our site designs. We can use a “site testing” vendor to test billions of site design combinations in a matter of weeks. This will let us build a web page that optimizes for the things we care about, like generating page views, increasing the number of unique visitors, or contribution value.”



I understand where you are coming from, and indeed I wrote a short post on this at http://dsanalytics.com/dsblog/website-optimization-and-the-design-of-experiments_101
and another longer post at
http://dsanalytics.com/dsblog/website-optimization-and-data-analytics_90

But I worry about testing "billions of site design combinations in a matter of weeks".

What happens with fractional factorial designs is that only a few (maybe 16, maybe 64, maybe 128) combinations are ACTUALLY tested, and from the results of that experiment we can fit a model that allows us to INFER what the response to other combinations might be. Optimum seeking from such a model is certainly theoretically feasible but only with some simplifying (and possibly heroic) assumptions.

I would prefer to see such an approach as identifying an approximate REGION (as in, a combination of settings of variables) where the optimum might lie, then do another round of experimentation focussed on that region... and so on

Unknown said...

We are actually taking a regional approach. At least. thats how I understand the methodology. They use "optimal" experimental design. They learn as they go and test into areas that are more productive.