Managing a data project – Some shortcuts

Here are some things that should make your data project go smoother whether its a data warehouse migration, report migration, or some other data project such as master data automation.

Use a project management tool of some sort to to track progress, blockers and issues in a meaningful and reportable way. I like to use Team Foundation Server or Visual Team Services to track all problems, iterations and store my code. I’ve also used Microsoft Project*, GIT, Huddle, OnTime and Visual Source Safe an eon ago.

Ask yourself and then your shareholders these questions.

  • Who really uses this data?
  • Who is the final consumer?
  • Who is a filter / formatter?
  • Why do you need this data?
  • Who or what positions should be able to view this data?
  • How does this data relate to the data you’re combining it with?
    • This could possibly create a unique key between records or at least a set of dimensions that could be shared between multiple data sets.
  • Who is skilled enough to read and understand what the data is telling us?
  • Who will be testing this when it’s complete?
  • When the requirements are gathered, who will be signing off on the scope  of work?
    • Many times shareholders don’t sign off on requirements and then the scope of the work (report, transfomation or process) never ends because they keep finding new faults that weren’t built into the original request.
      • I would rather have many test points going in to the project than less.
      • It’s better to ask too many questions then not enough.
      • Generalized requirements produce general results.
  • What is the expected turn around for this work?
    • Is that reasonable given the amount of resources and unknown information about the project?
    • Can you manage an expectation if the unknown variables are too high to weigh out?

Break work into work sized pieces. Project planning is WORK. Creating a good looking backlog that has completely unusable tasks with unrealistic deadlines will only frustrate the developers. Developers are the only ones with skin in the game. Most of the time developers get the brunt of the work. They have to deal with, requirements gathering, permission and access issues, automation difficulties and political meandering. Good project management always includes understanding the amount of work required to do the tasks a PM is doling out. A PM is only as good as the team functions.

If you can use and agile scrum methodology. These groups use small skillful teams, regular updates, are project guided, usually without a project manager with a major shareholder as their leader. This will keep things moving along and everyone will be briefed daily on what’s going on with the project.

When there is a difficult road blocker, STOP! Do an all hands work session on the problem with as many skilled people as possible. You’ll find that many geniuses do better when they are working with their peers in an uninterrupted and open setting.

Communicate with end users. Don’t let a milestone be a surprise to anyone. Announce, announce, announce! Use a newsletter if you need to. Put release notes out with ever newsletter. Time their release to coincide with a sprint release. Get your testers buy in by attacking the bugs they find and then thanking them publicly. Make this an inclusive process where people feel like they’ve contributed.


These are just some of the things that make projects go well. I look forward to working with you in the future. Even if you only need a project setup for success we can guide you in that direction.

Always a pleasure.

Myles Yamada


Items marked with * an asterisk are either copywrited or registered trade marks that do not belong to LPUData, Inc.

Author: Myles Yamada

Owner of LPU Data, Inc. - Data warehousing and Reporting. With over twenty years as a consultant in the IT industry, programming databases, websites and ERP systems. I found my self in a very unique position to extract data from many sources and provide meaningful results in a data warehouse. My first real data mart was to find patterns of abuse in schedule I drugs from pharmacy data and hospital claims forms. This challenge was exciting and provided interesting insights into the abuse by patients and questionable dispensing patterns of doctors. When this same model was applied to blood thinners, this became a life saving tool to help prevent seniors from stroking out after having been in a hospital recently. Hospital blood thinners are stronger than the ones that doctors prescribe for patients. Medical personnel are always present in the hospital either in person or by telemetry. When the patient is on the high dose of blood thinners so they can detect any symptoms that may lead to complications. During the transition between hospital, nursing home or patient home, a patient may still be holding an active prescription for a high dose of blood thinners. When the patients refills at the pharmacy they may have been ordered a lower dose. Not knowing that they should have dumped the previous medication some patients end up taking both low and high dosages which creates a very high dosage of blood thinners. This possibly leads to a stroke and puts the patient at grave risk. It is my intention to use data responsibly and for the good of the people. Myles Yamada