* Conway's law causing multiple different data science toolchains, different philosophies on model training, data handling, schema and protocol, data retention policies, etc.
* Coming up with tech solutions to try to mitigate the impact of multiple silos insisting on doing things their own way while also insisting that other silos do it their way because they need to access other silos' data.
And the reason standardization won't happen: the feudal lords of each of those branches of the hierarchy strongly believe their way is the only way that can meet their business/tech needs. As someone who gets to see all of those approaches - most of their approaches are both valid and flawed and often not in the way their leaders think. A few are "it's not going to work" levels of flawed as a result of an architect or leadership lacking operating experience.
So yeah, it might look like technical problems on the surface, but it's really people problems.
- Requirements are rarely clear from the beginning;
- We (DE) are not enabling self-service and automation so we are drowned in small requests (add this column for example;
- Upstream rarely notify us about the changes so we only know when downstream alerts us. We end up building expensive pipelines to scan and send alerts. Sometimes the cost of alerts > cost of pipeline itself;
- We have so many ad-hoc requests that sprint is meaningless. If I were the manager I'd abolish sprint completely;
- Shadow knowledge that no one bothered to write down. I tried to write down as much as possible, but there are always more unknowns than knowns;
Working in DE definitely gives me enough motivation to teach myself about lower level CS.