Software Engineer - Azure data lake migration

*Please note: the service contract for this position will not be concluded with Henkel AG & Co. KGaA but with an external party”.

Projektname /project name

Migration of Azure Data Lake (M026525)

Projektbeschreibung /project description

The service is requested as part of the project above. The project has the purpose to

• stabilize and enhance data pipelines loading data into our central data lake;
• migrate the solution from Azure Gen 1 Data Lake to Azure Gen 2 Data Lake;
• plan a proposal of a handover the solution to maintenance.

Leistungsbeschreibung / task description

The service of the contractor is delivered using an agile working method. External resources are needed as there is no internal staff with the required expertise in the following areas:

• Azure Data Factory regarding pipeline implementations;
• Azure Databricks regarding pipeline implementations
• Azure Data Lake 1 Authorization.

Therefore, the external contractor is in a unique position and performs significantly different tasks than the internal employees.

One sprint consists of 2 weeks and there is a daily jour fixe. During these meetings, the team discusses the current requirements and the contractor independently performs the following tasks:

• Documentation of the current data processing pipelines which is subject to approval by Henkel;
• Source code will be provided to the contractor in advance;
• Independently conduct interviews with stakeholders and developers to understand requirements from business and from technical point of view
• Enhancement and maintenance of existing pipelines developed with Azure Data Factory and Azure Data Bricks based on the documentation above;
• Developing a concept to migrate the pipelines to our Gen2 data lake which is also subject to approval by Henkel.

The service provision of the contractor has the goal to prepare the migration from Azure Data Lake 1 to Data Lake 2.

Timelines

The following timelines are to be adhered to by the contractor during the provision of the service:

• Current deadline is 30/09/2021