Big Data Engineer with Pyspark Full Time

Big Data Engineer
Location: Irving, TX
Job Type: Full-Time (Hybrid)
Experience: 7+ Years
F2F Interview
Job Summary
We are seeking an experienced Big Data Engineer with 7+ years of professional experience in designing, developing, and implementing scalable data solutions. The ideal candidate will have strong hands-on experience with Hadoop, Spark, PySpark, Kafka, Hive, SQL, and cloud-based data platforms.
The candidate will be responsible for building high-performance data pipelines, processing large and complex datasets, and working closely with data scientists, analysts, application developers, and business stakeholders.
Key Responsibilities
Design, develop, and maintain scalable Big Data pipelines and data processing solutions.
Develop batch and real-time data processing applications using Apache Spark/PySpark.
Build and optimize ETL/ELT pipelines for large-scale data processing.
Work with Hadoop, HDFS, Hive, Spark, Kafka, and other Big Data technologies.
Develop complex SQL queries and optimize data processing jobs for performance.
Design data ingestion and transformation processes from multiple structured and unstructured sources.
Develop real-time streaming pipelines using Kafka and Spark Streaming.
Perform data cleansing, transformation, validation, and quality checks.
Troubleshoot data pipeline failures and optimize jobs for scalability and reliability.
Work with cloud-based data platforms and storage services.
Collaborate with data architects, data scientists, analysts, and engineering teams.
Implement data security, governance, and access-control best practices.
Participate in code reviews, testing, deployment, and production support.
Contribute to Agile/Scrum development activities and technical documentation.
Required Skills
7+ years of experience in Big Data Engineering
Strong hands-on experience with Apache Spark and PySpark
Strong programming experience with Python and/or Scala
Strong SQL skills and experience with relational databases
Hands-on experience with Hadoop, HDFS, Hive, and MapReduce
Experience with Apache Kafka and real-time data processing
Strong experience building ETL/ELT data pipelines
Experience with data ingestion, transformation, and processing of large datasets
Knowledge of distributed computing and Big Data architecture
Experience with data warehouse/data lake concepts
Experience with Git and CI/CD
Strong troubleshooting and performance-tuning skills
Cloud & Modern Data Technologies
Experience with one or more of the following is highly preferred:
AWS: S3, EMR, Glue, Redshift, Lambda
Azure: ADLS, Databricks, Azure Data Factory, Synapse
GCP: BigQuery, Dataproc, Cloud Storage
Databricks
Snowflake
Preferred Qualifications
Experience developing both batch and streaming data pipelines.
Experience with Apache Airflow or similar workflow orchestration tools.
Knowledge of data modeling and data warehousing concepts.
Experience with Docker/Kubernetes is a plus.
Understanding of data governance, security, and data quality frameworks.
Experience working in Agile/Scrum environments.
Strong communication and analytical problem-solving skills. Full-Time

Tagged as: Data, engineer