Role: Data Engineer
Experience: 5-10
Description:
We are seeking a skilled Databricks Data Engineer with expertise in Scala to design, implement, and maintain scalable data solutions on the Databricks platform. The ideal candidate will have a strong background in data engineering, distributed computing, and cloud technologies, with a focus on leveraging Databricks and Apache Spark for data processing and analytics.

The project in which the person will be assigned is a data engineering effort to support one of clients with timeseries data migration.
Responsibilities:
Design and Implementation : Develop and maintain robust data pipelines using Databricks Scala, ensuring efficient data processing and transformation.
Data Modeling : Create and optimize data models to support analytical workloads and reporting needs.
Collaboration : Work closely with cross-functional teams, including analysts, and business stakeholders, to deliver data-driven solutions that meet business objectives.
Performance Optimization : Monitor and enhance the performance of existing data pipelines and applications, implementing best practices for scalability and efficiency.
ETL Development : Design and implement ETL processes to extract, transform, and load data from various sources into Databricks.
Automation : Utilize Dev Ops practices to automate deployment processes and monitor data pipelines for reliability.
Documentation : Maintain clear documentation of data architecture, processes, and workflows to ensure knowledge sharing within the team.
Skills:
Programming Languages : Proficiency in Scala is required; experience with Python or SQL is a plus.
Databricks Experience : Hands-on experience with the Databricks platform and Apache Spark for big data processing.
Cloud Platforms : Strong knowledge of Azure in relation to data services.
Distributed Computing : Understanding of distributed computing principles and big data technologies.
Data Warehousing : Familiarity with data warehousing concepts and tools.
Timeseries DB : Knowledge of timeseries DB like influx db
Columnar Data store : Knowledge of columnar datastore like Apache Druid or Imply Polaris