ETL Development using Scrum
Hi,
I'm an ETL developer and team leader (also known as data engineers), we are asked to start managing our work using Scrum.
I've tried to understand how can I fit an ETL Development to Scrum, here is my scenario:
- we get a requirement for data we need to gather - usually from sources outside our organization
- the project starts by describing the required data to our sources (it might take several months until we see sample data, which is usually not what we asked)
- than we need to design the QA / DQA for the whole process
- than we simulate the required data and start developing the ETL implementing the DQA within the process (might take more than a month it should be synced with QA team)
- starting integration with the consumers (usually BI)
- starting integration with our sources and QA (usually requires lots of back and forth - might take months until we see response)
I guess my questions are
- How can I manage ETL development that might take several months in sprints, it an entire process and can't be broken into sub processes or tested until completion
- How would you manage a project when your progress is limited by resources outside of your control, for example we needed to supply data by the end of April, We've had an obligation from our source that the data will be available by the end of march, in the end we've got the data at the end of April and the structure and granularity are wrong, and we need to reDevelop some of the process
Thanks,
Eyal
Who is asking the team to use Scrum, why, and what do they aim to achieve?
It's not clear to me if you need or can benefit from Scrum other than someone outside the team asking for it. Trying to shoehorn Scrum (or any framework) into a place where it doesn't belong won't lead to a successful team. If you're not operating in a complex environment and can get key stakeholder involvement to iterate and plan on the next steps, I'm not sure that Scrum would be useful to the team.
we are asked to start managing our work using Scrum.
I've tried to understand how can I fit an ETL Development to Scrum
Why? Presumably different outcomes are expected, and a different way-of-working will be needed. So why try and fit the process you have described, which would produce the same old outcomes, into something where it doesn't fit?
Scrum is about innovation, and learning to build the right thing at the right time. The conversation you need to have is not about fitting an old process in, but about the appetite in the organization for deep and pervasive change.
- How can I manage ETL development that might take several months in sprints, it an entire process and can't be broken into sub processes or tested until completion
With this constraint, You can only count the sprints but nothing will change.
in the end we've got the data at the end of April and the structure and granularity are wrong, and we need to reDevelop some of the process
if the expectation is to address this challenge through Scrum, it would require stakeholder involvement as well.