The Export-to-Excel Culture
As a data professional, I’ve held many roles. These roles have spanned everything from report & dashboard developer to the business analyst gathering requirements and translating them to technical needs. I’ve written code, I’ve written user stories, and I’ve written Powerpoints explaining where we are in all of the above. Towards the end of my data career I found myself doing software demonstrations, either as a consultant or as a Sales / Solution Engineer. A lot of my experience is from both sides of the “wall”. I know what it’s like to be an end user of a platform. I know what it’s like to be on the inside of the software, working with Product and other experts. It’s with that I bring you the classic “Export to Excel” discussion (all opinion, but from a grounded experience).
Examples
I’d say when I explore the requirement “Export to Excel” more with end-users, the use-cases can be boiled down to two main categories, probably split 50/50.
Tactical Operational Needs
When I say “Tactical Operational Needs” I’m referring to:
Environments with more than one software / solution powering someone’s job; somewhat disjointed
Teams working in raw data
Teams without an operational database solution at all
Specifically, if you’re working with a team that analyzes their operational lists in one place (i.e. they’re using a dashboard to filter their data down to 100 contacts / leads / issues) but then handles the operational day-to-day elsewhere (i.e. either nowhere or in some disconnected database that requires manual effort to import data into it), chances are that team is asking if they can “export to Excel.”
I saw this a lot with sales (and certain internal operations teams) where they used a dashboard or some special data list to narrow down their 1000’s of leads to the top 100. They’d then take that list and either manually run through a Salesforce instance, or import that Excel into somewhere in Salesforce, to then perform their day-to-day tasks of calling, emailing, etc.
The root cause of this type of example is technical debt in the form of an operational team needing more of a cohesive technical solution than is currently available to them. Might be the legacy systems hanging around for dear life. Might be IT is too busy with bigger, higher priority issues. Regardless, technical debt rears its ugly head and in the meantime operational teams, the business units, your end users have a job that doesn’t stop, so they do what they need to to keep going, typically in an easily accessible spreadsheet program.
Ad Hoc Analyses
Ad hoc analyses are an unstoppable force. In any business a question will arise and a team of people will do whatever they can and need to in an attempt to answer it in a “data-driven” way. A business can’t reasonably expect a company to be data-driven and not have ad hoc analyses.
The root cause for needing to export to Excel for an ad hoc analysis is technical debt.
Excel does a great job of being easily used across multiple systems, data sources, and is ubiquitous amongst citizen data analysts. When an organization has data stored across multiple [legacy] solutions (technical debt) and an answer needs to be generated and worked on before an IT team can be slated to develop an “official solution” (technical debt), then the business unit expert / analyst will do what they do best. Pull out their “Let’s get scrappy” tee and start exporting data to Excel tables, joining in whatever solution they could get their hands on a license for (technical debt), and answer their team’s biggest questions.
Excel, and other spreadsheet options, is readily accessible with minimal extra licensing, and readily usable by most users. Introducing new software options or built-in analytics isn’t always feasible in ad-hoc analyses because:
Users don’t know how / are not confident in the tools
Built-in analytics may not have external data sources as options
Users don’t feel they have the time to train / learn a whole new program when it’s not part of their main job duties (i.e. Sales team member who needs to analyze pipe but without slowing down their Sales job)
Results
In short: more technical debt. Having Excel files moving about on various Desktops / shared network drives / cloud drive solutions can become a real problem for any organization.
Data security. It’s impossible for IT Security to protect Desktop level data points in a random Excel file the way they can a proper database system.
Data discrepancies. By the time an analysis is rolled up and shown to a VP-level team, depending on the source and all of the transformations between the source and that analysis, the VP’s are likely getting different results from different teams / analysts.
Data Lineage. When data is transformed in a spreadsheet program, it’s all transactional and typically overwrites, erasing any lineage of where the data started and how it ended up in its final state, unless detailed notes are taken (typically external to the workbook / file). This puts the burden on the analyst to maintain notes / procedures / repeatability, and could be a major failure point in ongoing analyses.
Data redundancies. Along the lines of discrepancies and lack of lineage, data is being copied all over, increasing hardware storage issues, but more importantly, diluting trust. What is the single-source-of-truth? If each department has a slightly different version of this, and it’s in an Excel file in their locked down folder, then IT is going to have a hard time, as will VP’s, in identifying what is really happening in the business.
What Drives It
Technical debt. This article explains it nicely, but my favorite analogy is where Justin Brodley, VP cloud operations & engineering at Ellie Mae and co-host of The Cloud Pod calls it “Human Spackle.”
Technical debt, in my experience, can be generated rather quickly and easily through design decisioning. In my own business it was directly related to budget, but also I was accruing technical debt from the technical debt of my vendors. I, myself, wasn’t building the database that supports my online store and site, so with my own operations, I found my technical debt was directly related to that of my website host(s), merchant services vendor(s), and Etsy store. This is what I mean when I say quickly. I’m 1 person and was in business for a mere months before I had already lost control of my technical debt. I quickly found myself working around my systems, which is one of my biggest pet peeves.
In a perfect solution:
Disparate systems would be replaced or be forced to work together (in real-time)
User interfaces would be designed to fit the actual workflow of a business unit instead of the business unit fitting their workflow to a system
Users would feel confident using built-in, interconnected analytical tools
Users wouldn’t need stacks of pages on procedures and hours of training. Systems would be intuitive.
Not many software platforms play nicely with others. Often times (not all of the times) their business plan is built around complete buy-in. This isn’t a practical decision.
Depending on age / size of a company, different managers / teams will make different solutioning decisions, choosing softwares from different, often competing “stacks” (i.e. Microsoft v. Google).
Business Continuity must be considered. What if there’s an outage / bankruptcy / security breach? If all processes / data are stored in one “stack”, this could lead to serious problems down the road.
I’ve yet to see a truly “all-in-one” solution. There are gaps in any platform / solution. Being able to plug those holes with other platforms is imperative, but requires they are somewhat compatible.
Technical debt is inevitable. Mitigating it is a project of its own. I think a culture-first discussion can help move the needle on it.
Addressing & Driving Change
Curious Acceptance
I put this first because I think it is THE MOST IMPORTANT. I’m the first one to cringe inside, make jokes, and trash talk exporting to Excel. It’s one of my most dreaded questions on calls. However, when I’m on these calls, I answer the question with “yes, but to understand more on the design around how that would work, can you please explain to me why you export to Excel and how your team uses the Excels once you have them?”
Curious. Accepting.
Business units will rebel against solutions that won’t let them work in the ways they know and can support themselves. As soon as you tell them they need to work with another group / developer / etc, they will begin an internal panic. Management needs to know that if they need something done, they can set the priority and get their people on it immediately, minimizing interdependencies with groups / teams that might have conflicting priorities.
With that - they know they have people and licenses for Excel. They know they can do what they need to, even if it’s ugly and manual, in Excel on their desktops, without relying on an IT developer or data scientist to run python or R code.
It’s okay to let them feel seen, feel understood, and ask THEM what they need. This builds trust with them but just as importantly, gives everyone an opportunity to document further challenges and needs a business unit might have.
Requirements Gathering
Along those lines, and the second step after Curious Acceptance, is gathering requirements. Essentially answering the question:
“What would it take to stop exporting to Excel?”
This is a wildly loaded question. Answers might shift, solutions may feel impossible or so far out it’s not worth even considering as a possibility, and the whole conversation might as well be in a fantasy novel, but try it out.
Document it.
What answering this question does:
Helps a company understand what might be leading to technical debt.
Drives prioritizations for helping re-develop / solve to mitigate technical debt. If use cases are documented, they can be tracked / prioritized against a backlog to move the process along.
Provides visibility into what is otherwise lurking in the shadows (in the form of the dreaded “shadow IT”)
Again, builds trust with the end users. The more the technical analysts & decision makers understand the more they’ll feel seen and understood.
Backlogs will pile up and it’ll all feel daunting, but at the very least having the conversation (in a curious and accepting way) and then documenting possible solutions / requirements will still be a move in a more positive direction.
User Group Discussions
Data needs, ad hoc analyses, and all the things surrounding anything database related will all shift and change over time. Depending on the industry, size, and age of the business this could be monthly, this could be weekly, this could be annually.
Talk to the end-users and keep talking to them on some pre-determined cadence. Audit the solutions they have in place, update requirements and notes, and stay in touch. I’ve worked on so many projects where years later I find out end-users are doing this bizarre work-around to complete some form of business that would have been a short email, 2 lines of code, and a quick re-release. Staying in touch with them and building a relationship of openness to criticism and feedback helped me help them. They’d complain more openly about something not working right, I’d dig into it and realize it was something simple, and then everyone would be very excited to have that small thing fixed. If it wasn’t small, I’d explain our options, and let them choose the next step. If it wasn’t possible, I’d explain why, make a note for a “someday when there’s less technical debt”, and move on. The point here - it was always a conversation.
Shutting down the conversation is where technical debt grows, not subsides. It’s how systems end up irrelevant, unused, and out-of-date. It’s how data ends up messy, incomplete, and useless.
Data work is partnering with the people generating the data, making the data-entry process seamless, and mitigating the friction wherever possible. The only way to do that is to talk to the front-line people.
This might be:
Regular one-on-ones with SME’s
User group meetings / sessions on a regular cadence with pre-determined homework / agendas for effective use
General networking - grabbing a drink and being casual about it
Some of my best work, even if I couldn’t solve every problem, was listening, documenting, and working with the end-users.
Conclusion
It’s difficult being on either side of the “Export to Excel” discussion. I’ve run into it in so many different companies, use cases, and involving various software platforms. The key to having a productive discussion around these points is to understand and empathize with these business units. Documenting their pain points and helping solve where possible will likely mean giving them a button or two that say “Export to Excel". This will at least open the door for conversations and future improvement.
Shop
As a former data professional, I’ve created a Zazzle shop, and some Etsy items, just for the #datafam. Check out my full list of designs in this blog here which I’ll keep updated regularly. Or go directly to my full-on Zazzle collection “Inside Data Jokes” here.
