Data Engineer

We are building a modern Databricks-based lakehouse from the ground up and are now looking for a data engineer to support the ingestion and stabilization phase of the platform. The focus of this role is to help operationalize, debug, and harden data pipelines so they are production-ready, reliable, and well-monitored.

You will work in an established engineering setup with infrastructure already provisioned, CI/CD in place, and clear data engineering patterns defined. You will work closely with a senior data engineer who designs the ingestion patterns and prototypes each data source. Your role is to implement, debug, test, and stabilize the pipelines following these established patterns, with ongoing guidance and code reviews.

Scope of assignment

Build and maintain ingestion pipelines (Bronze layer) from internal and external sources (BigQuery, SharePoint, Dynamics 365, files in cloud storage, APIs)

Build standardized Delta tables from raw data (Silver layer)

Debug, stabilize, and redeploy existing pipelines following established patterns

Implement basic data quality checks and tests (row counts, schema validation, null checks, duplicates) using pytest and SQL

Monitor and troubleshoot Databricks pipelines and jobs

Assist with operational monitoring and cost/usage dashboards in Databricks

Work with Git-based workflows and deploy changes across different environments

Assist in migrating existing data transformation notebooks from legacy platforms (such as Classic Databricks and Azure Synapse) into the new Databricks lakehouse. This includes adapting and refactoring the code to follow clearly defined patterns, standards, and best practices, with guidance and support.

Technical requirements

Hands-on experience with Databricks:

Delta tables

Notebooks

Jobs & Workflows

SQL endpoints and Spark clusters

Databricks Dashboards

Unity Catalog

Databricks Asset Bundles

Programming languages:

PySpark

SQL

Familiar with Git and pull-request–based workflows

Comfortable working with existing infrastructure and patterns

Basic understanding of cloud concepts (storage accounts, permissions, service principals)

Experience of AI-assisted development tools for increased productivity is a plus.

Databricks Associate Data Engineer certification is a plus