Managing a data project – Some shortcuts

Here are some things that should make your data project go smoother whether its a data warehouse migration, report migration, or some other data project such as master data automation.

Use a project management tool of some sort to to track progress, blockers and issues in a meaningful and reportable way. I like to use Team Foundation Server or Visual Team Services to track all problems, iterations and store my code. I’ve also used Microsoft Project*, GIT, Huddle, OnTime and Visual Source Safe an eon ago.

Ask yourself and then your shareholders these questions.

  • Who really uses this data?
  • Who is the final consumer?
  • Who is a filter / formatter?
  • Why do you need this data?
  • Who or what positions should be able to view this data?
  • How does this data relate to the data you’re combining it with?
    • This could possibly create a unique key between records or at least a set of dimensions that could be shared between multiple data sets.
  • Who is skilled enough to read and understand what the data is telling us?
  • Who will be testing this when it’s complete?
  • When the requirements are gathered, who will be signing off on the scope  of work?
    • Many times shareholders don’t sign off on requirements and then the scope of the work (report, transfomation or process) never ends because they keep finding new faults that weren’t built into the original request.
      • I would rather have many test points going in to the project than less.
      • It’s better to ask too many questions then not enough.
      • Generalized requirements produce general results.
  • What is the expected turn around for this work?
    • Is that reasonable given the amount of resources and unknown information about the project?
    • Can you manage an expectation if the unknown variables are too high to weigh out?

Break work into work sized pieces. Project planning is WORK. Creating a good looking backlog that has completely unusable tasks with unrealistic deadlines will only frustrate the developers. Developers are the only ones with skin in the game. Most of the time developers get the brunt of the work. They have to deal with, requirements gathering, permission and access issues, automation difficulties and political meandering. Good project management always includes understanding the amount of work required to do the tasks a PM is doling out. A PM is only as good as the team functions.

If you can use and agile scrum methodology. These groups use small skillful teams, regular updates, are project guided, usually without a project manager with a major shareholder as their leader. This will keep things moving along and everyone will be briefed daily on what’s going on with the project.

When there is a difficult road blocker, STOP! Do an all hands work session on the problem with as many skilled people as possible. You’ll find that many geniuses do better when they are working with their peers in an uninterrupted and open setting.

Communicate with end users. Don’t let a milestone be a surprise to anyone. Announce, announce, announce! Use a newsletter if you need to. Put release notes out with ever newsletter. Time their release to coincide with a sprint release. Get your testers buy in by attacking the bugs they find and then thanking them publicly. Make this an inclusive process where people feel like they’ve contributed.


These are just some of the things that make projects go well. I look forward to working with you in the future. Even if you only need a project setup for success we can guide you in that direction.

Always a pleasure.

Myles Yamada


Items marked with * an asterisk are either copywrited or registered trade marks that do not belong to LPUData, Inc.