As a Senior Data Engineer, you will play a critical role in designing, implementing, and supporting secure, scalable, and high-performing enterprise data platforms.
As a trusted technical consultant, you'll leverage technologies including Databricks, Apache Spark (PySpark), Azure Data Factory, Azure Data Lake, Kafka, Python, SQL, Scala, and cloud-native services to build high-performance data pipelines, modern integrations, and scalable analytics platforms.
This is an excellent opportunity for an engineer who enjoys combining technical expertise with business problem-solving while working in a collaborative Agile environment focused on innovation, quality, and continuous improvement.
Responsibilities :
Design, develop, test, deploy, and maintain enterprise data engineering solutions using modern cloud and big data technologies
Design and implement scalable ETL/ELT pipelines utilizing Databricks, Apache Spark (PySpark), Delta Lake, and Azure Data Factory
Build high-performance data ingestion, transformation, and integration frameworks supporting enterprise analytics and reporting
Develop, optimize, and maintain complex SQL queries, stored procedures, and data transformation processes
Design and implement scalable data models supporting business intelligence, analytics, and AI initiatives
Build and integrate RESTful APIs, event-driven architectures, and legacy SOAP services to facilitate seamless enterprise data exchange
Develop and maintain streaming and messaging solutions using Apache Kafka
Monitor, troubleshoot, and optimize production data pipelines while performing root cause analysis and implementing long-term solutions
Implement data quality, governance, security, and performance best practices across enterprise platforms
Manage source code using Git while following CI/CD and enterprise DevOps best practices
Collaborate with Solution Architects, Product Owners, Business Analysts, and cross-functional engineering teams to translate business requirements into technical solutions
Participate in Agile ceremonies including sprint planning, backlog refinement, architecture discussions, code reviews, and retrospectives
Create technical documentation, deployment artifacts, testing documentation, and operational runbooks
Mentor junior engineers and contribute to engineering standards, best practices, and continuous improvement initiatives
Qualifications :
Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical discipline (or equivalent professional experience)
6+ years** of experience designing, developing, and supporting enterprise data platforms, cloud analytics solutions, or large-scale data engineering initiatives
Strong hands-on experience with:
Databricks
Apache Spark (PySpark)
Python
SQL
Scala
Experience working with the Databricks ecosystem, including:
Databricks Workspaces
Delta Lake
Unity Catalog
Delta Live Tables (DLT)
Databricks SQL
MLflow
Databricks Jobs
Strong SQL development experience, including query optimization and performance tuning
Experience designing and implementing ETL/ELT solutions using Azure Data Factory or comparable cloud integration platforms
Experience with Apache Kafka or other event streaming technologies
Experience integrating enterprise applications using REST APIs, JSON, XML, and SOAP web services
Experience with Azure Data Lake Storage (ADLS Gen2) or comparable cloud storage platforms
Experience using Git for source code management and collaborative software development
Experience working within Linux environments, including shell scripti
ng and command-line utilities
Familiarity with Infrastructure as Code (IaC) tools such as Terraform is preferred
Experience implementing CI/CD pipelines using Azure DevOps, GitHub Actions,or similar DevOps platforms is preferred
Strong analytical, troubleshooting, and problem-solving skills
Experience working in Agile environments utilizing Scrum, Kanban, or SAFe methodologies
Demonstrated ability to manage multiple priorities while delivering high-quality solutions
Proven ability to work independently while mentoring teammates and contributing to technical leadership
Desired Skills :
Microsoft Azure cloud services
Azure Synapse Analytics
Microsoft Fabric
Power BI
Data governance and metadata management
DataOps and MLOps practices
Enterprise data warehousing
Master Data Management (MDM)
Financial services or public sector data environments
Communication & Collaboration
