Key Responsibilities : Data Pipeline Development : Design, implement, and manage scalable data pipelines on Azure Data Services (Azure Data Factory, Azure Synapse, Databricks) to support real-time and batch data processing.
Palantir Integration : Develop and maintain data integrations between Palantir Foundry and Azure-based data storage and processing services.
Ensure seamless data flow across platforms.
Data Modeling & Transformation : Work with data scientists and business analysts to design efficient data models and transform data into formats that meet business and analytical needs.
Big Data Management : Handle large datasets using Azure Data Lake , Azure SQL Database , Databricks , and other big data tools, ensuring performance and scalability.
Collaboration : Work closely with cross-functional teams (data science, engineering, business intelligence) to understand requirements and implement efficient data-driven solutions.
Optimization & Automation : Continuously improve data pipelines by optimizing processing time, automating manual tasks, and ensuring high data quality and consistency.
Security & Compliance : Ensure proper data security measures are in place, including encryption, role-based access control (RBAC), and compliance with industry standards.
Technical Support & Troubleshooting : Provide expertise and support for the Azure and Palantir platforms, resolving data-related issues as they arise.
Required Skills & Experience : Bachelor’s or Master’s Degree in Computer Science, Engineering, Information Technology, or a related field.3+ years of hands-on experience as an Azure Data Engineer, working with Azure Data Services (Azure Data Factory, Azure Synapse, Azure Databricks).
Strong experience with Palantir Foundry , including data integration, modeling, and transformation within the platform.
Strong proficiency in SQL and experience with big data tools (Apache Spark, Databricks).
Experience in managing large datasets in Azure Data Lake or similar storage solutions.
Familiarity with Azure security features (RBAC, identity management, encryption).
Solid understanding of data architecture, data pipelines, ETL processes, and data warehousing concepts.
Experience with scripting and automation using Python , Shell , or Power Shell.
Knowledge of Azure Synapse Analytics , Stream Analytics , and Data Factory.
Familiarity with cloud computing , data processing, and cloud-native technologies.
