Senior Data Engineer – IoT

Location: Fort Mill, SC
Schedule: Monday–Friday | Primarily On-Site with flexibility for 1–2 days remote
Compensation: $185,000–$190,000 base salary + 7.5%–15% bonus

Position Overview

Our client is seeking a Senior Data Engineer – IoT to take a leading role in the development and long-term direction of its IoT data platform. This is a newly created position for a highly hands-on engineer who brings significant experience working with large-scale IoT, machine, or telemetric data.

This individual will help drive the continued migration of IoT data into Databricks, while designing and building a scalable data platform that supports real-time and near-real-time reporting, analytics, data science, AI/ML, customer-facing applications, and sustainability initiatives.

This is an opportunity to have meaningful ownership over the organization's IoT data strategy—not simply implement an existing specification. The ideal candidate will be comfortable moving between hands-on engineering, architecture, technical guidance, and business-facing discussions.

Key Responsibilities

Design, build, and maintain end-to-end IoT data pipelines from edge devices into Databricks using Bronze/Silver/Gold medallion architecture.
Lead the migration of IoT data from a third-party application into a scalable Databricks environment.
Develop and optimize streaming ingestion pipelines using technologies such as Apache Kafka, Azure Event Hubs, and Databricks Structured Streaming.
Establish data engineering best practices, including pipeline design patterns, data quality frameworks, schema governance, and scalable development standards.
Architect and optimize Delta Lake tables, including partitioning, Z-ordering, and data lifecycle management for performance and cost efficiency.
Collaborate with infrastructure and cloud teams on Databricks clusters, Unity Catalog, Delta Live Tables, and related cloud technologies.
Develop solutions capable of supporting multi-terabyte, high-volume telemetry and time-series data.
Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
Ensure IoT data is structured and optimized for downstream analytics, AI/ML, reporting, and customer-facing use cases.
Mentor junior and mid-level data engineers through code reviews, pair programming, technical guidance, and architectural direction.
Troubleshoot production pipeline issues, perform root-cause analysis, and implement long-term solutions.
Evaluate emerging technologies and recommend improvements to the organization's IoT and data engineering platform.
Represent the data engineering team in technical and business discussions and help shape the organization's long-term IoT data strategy.

Required Qualifications

5+ years of hands-on experience working with IoT, machine, telemetric, or similarly large-scale device data.
Strong senior-level data engineering experience, preferably in environments involving high-volume or real-time data.
Strong proficiency in Python and SQL, with significant experience developing production-grade Spark/PySpark solutions.
Experience designing and implementing data pipelines for high-frequency telemetry, time-series data, or device-generated data.
Experience with Databricks and/or other large-scale cloud data platforms.
Strong understanding of data architecture, pipeline design, data quality, and schema governance.
Experience working with streaming or near-real-time data architectures.
Ability to lead technical discussions and translate business requirements into scalable technical solutions.
Strong communication and interpersonal skills, with the ability to work effectively with engineers, data scientists, analysts, and business stakeholders.
Ability to provide technical guidance and mentorship to other engineers.

Preferred Qualifications

Hands-on experience with Databricks, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
Experience with Apache Kafka, Azure Event Hubs, or similar streaming technologies.
Experience with fleet telematics platforms such as Geotab or similar IoT/connected-vehicle platforms.
Experience working with Azure cloud infrastructure and infrastructure-as-code concepts.
Familiarity with Azure Key Vault and Azure Data Factory.
Experience with dbt or other modern data transformation tools.
Experience supporting multi-terabyte data environments.
Databricks Certified Data Engineer Associate or Professional certification.
Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field, or equivalent professional experience.
Experience within the rental, transportation, fleet, logistics, automotive, manufacturing, or connected-device industries is a plus.

What You'll Bring

The successful candidate will be more than a data pipeline developer. You'll bring the technical depth to work hands-on with complex IoT data while also providing architectural direction and helping shape how the organization approaches IoT data for years to come.

You are a hands-on technical leader who enjoys solving complex data challenges, mentoring engineers, collaborating with stakeholders, and turning large volumes of machine-generated data into solutions that drive meaningful business outcomes.

Why This Opportunity?

Own the direction of IoT data engineering within a growing enterprise environment.
Help design a scalable, modern IoT data platform rather than simply implementing an established specification.
Play a key role in the continued migration and expansion of IoT data within Databricks.
Work at the intersection of IoT, data engineering, analytics, AI/ML, customer-facing solutions, and sustainability.
Provide technical leadership while remaining hands-on with the technology.
Join a collaborative data organization with opportunities to influence architecture, engineering standards, and future technology decisions.

Category Code: JN008
#LI-DM1
#CRGSearchJobs
#LI-Onsite
Seniority Level: Mid-Senior Level