€500K Big Data Platform
Every AI project at the company hit the same ceiling: the data infrastructure. Pipelines were slow, fragmented across tools, and had a heavy dependency on a single expensive vendor that was becoming both a cost problem and a technical bottleneck. Data engineers were spending most of their time firefighting pipeline failures rather than building new capabilities. AI teams were queuing for data access. Experimentation velocity had effectively stalled. This wasn't a problem that one team could solve. It required C-suite commitment to a significant infrastructure investment, coordinated delivery across 15+ engineers from multiple teams, and a clear technical direction that everyone could align on.
I started by making the problem legible to leadership. Not in technical terms. In business terms. I calculated the cost of the current vendor over three years, projected it against growth, and quantified the opportunity cost of blocked AI teams in terms of delayed product launches. The executive summary I wrote put a €500K investment against potential losses of double that over the same period. Then I ran a vendor evaluation. Three shortlisted options, structured scoring against criteria that the technical leads defined, a proof-of-concept with real production data volumes for the top two. The recommendation came with a risk register, a migration plan, and a timeline. Not just a technology preference. Once the investment was approved, I became the coordination layer across 15+ engineers from different teams who'd never worked together on a shared infrastructure project. I set the delivery milestones, ran the weekly sync, unblocked dependencies, and made the call on trade-offs when teams disagreed.
Three months into the build, we hit a significant incompatibility between our existing data lake structure and the new platform's ingestion model. Two of the engineering leads had completely different views on how to resolve it. One wanted to rebuild the data lake schema, the other wanted to build a translation layer. Neither was clearly wrong. I made the call: translation layer, because rebuilding the schema would require all dependent teams to update their queries simultaneously, which was a coordination risk we couldn't absorb mid-project. The translation layer was technically inelegant but delivered on time. Looking back, it was the right call.
The migration timeline was optimistic. We'd accounted for the technical migration but underestimated the time needed for teams to update their dependent pipelines and validate outputs. The final month of the project was more intense than planned because of this. I should have built a longer parallel-run period into the timeline from the start.
Build the acceptance criteria for each dependent team into the project plan from day one. Each team had their own definition of "done". We discovered most of them during the final month rather than upfront. A structured handoff checklist per team, agreed before the build started, would have distributed that work across the timeline instead of compressing it at the end.
A technical case for infrastructure investment rarely lands with C-suite. Translating "our pipelines are slow" into "we're paying €X/year for a vendor that will cost €Y by 2027, and our AI roadmap is running 6 months late because of infrastructure waits" is a different conversation. Leadership approved the full €500K in one meeting.
Vendor claims and real-world performance with production-scale data volumes don't always match. We put the top two vendors through a PoC with actual data. One significantly underperformed its benchmarks. That's not a discovery you want to make six months into a migration.
The schema rebuild was the cleaner technical solution. The translation layer was the deliverable solution. With 15+ engineers from different teams and a grant deadline, coordinating a simultaneous schema migration across all dependent systems was too high a risk. Pragmatism over elegance.