Consulting Engineer - Henkel Data foundation enhancements

*Please note: the service contract for this position will not be concluded with Henkel AG & Co. KGaA but with an external party”.

Projektname /project name

M024250 - T-AA: Analytics Foundation

Projektbeschreibung /project description

The service is requested as part of the project T-AA: Analytics Foundation. The project has the purpose to implement and automate processes in the environment and within the HDF. The project is running in DevOps mode and implementing the MVP 2.0.

Leistungsbeschreibung / task description

The service of the contractor is delivered using an agile working method. External resources are needed as there is no internal staff with the required expertise in the following areas:

• Programming language Python
• Process automation for Azure Cloud services: Databricks, Datafactory, Log Analytics

Therefore, the external consultant is in a unique position and performs significantly different tasks than the internal employees.

One sprint consists of 2 weeks and there is a daily jour fixe. During these meetings, the team discusses the current requirements and the contractor independently performs the following tasks:

• Technical consultation for and implementation of enhancements of the data distributor for the Data Foundation Platform (Python) inclusive process automation
• Technical consultation for and implementation & integration of further Azure services: Databricks, Datafactory, Log Analytics for the Data Foundation Platform (Python)
• Technical consultation for and implementation of Azure Databricks transformations (SQL or Python) of data stored in Azure Data Lake (from several data sources: IoT, master data and process data) so that is can be accessed efficiently
• Technical consultation for implementation of “delta transformations”, that means, transformation of only new data since the last transformation (per minute, daily, hourly)
• Set up Databricks jobs for the above transformations considering dependencies between them (that means, Job A must be finished before Job B can start).
• Creation of (automated) tests for the above solutions
• Creation of hand-over documents (operational task list, operational handbook, infrastructure sheet) for maintenance team, that will be provided to Henkel for verification and approval
• Remote training of the maintenance team about implemented transformations (probably 1-3 sessions, depending on the complexity)

The service of the contractor has the goal to further enhance and automate the processes of the Henkel Data Foundation.

Timelines

The in the sprint planning defined timelines and assigned backlog items are to be adhered to by the contractor during the provision of the service.