Overview
As a big data lead, you will need to lead the effort to design, build and configure applications and serve as the primary point of contact for teams. For this role, you will work on collecting, storing, processing and analyzing large sets of data. Your primary focus will be to choose & develop optimal solutions along with maintaining, implementing and monitoring them.
Responsibilities
- Ability to interpret and map business, functional and non functional requirements to technical specifications
- Interact with diverse stakeholders like clients, project manager/scrum master, business analysts, testing and
- other cross-functional teams as part of application development
- Develop solutions following established technical design, application development standards and quality
- processes in projects to deliver efficient, reusable and reliable code with complete ownership
- Follow best practices to ensure the best possible performance, quality, and responsiveness of the applications
- Assess the impacts on technical design because of the changes in functional requirements.
- Provide support specific to application bugs or issues within defined SLAs
- Support architect in developing artifacts such as high level design, technical design, etc.
- Help project manager/lead/pre sales with precise estimates keeping in mind risks and dependencies and
- recommend big data technologies to solve problems and create solutions
- Proactively identify and communicate technical risks, issues, and challenges with mitigations
- Build real time and batch data pipelines and optimize big data pipelines
- Integrate data from a variety of data sources either on premise or on cloud
- Build, deploy and optimize data lake and/other big data solutions
- Create and execute testing strategies including unit, integration, and full end-to-end tests of data pipelines
- Perform independent code reviews and guide junior team members for correction
- Actively mentor and coach the team to help them figure out best solution
- Adapt and learn new technologies in a fast manner
Skills Required
- Mandatory (Minimum 4 years of relevant experience
- Proven hands-on experience in Hadoop, Sqoop, Hive, Yarn, Pig, Impala
- Experienced in working with NoSQL databases, such as HBase, Cassandra, MongoDB
- Experienced in working with streaming data with technologies like Kafka, Spark
- Experience working with different kinds of structured and unstructured data formats (Parquet/Delta
- Lake/Avro/XML/JSON/YAML/CSV/Zip/Xlsx/Text)
- Solid programming experience in Java/ Python/ Scala
- Possess strong analytic skills related to working with unstructured and structured datasets
- Experienced in working with distributed (multi-tiered) systems and real time systems
- Experience of architecting big data pipeline on cloud (AWS, Azure or GCP)
- Ability to create architectures and tune to scale to handle 10x data, is highly available and cost effective.
- Experienced in data modelling, architecture, and data Governance on Hadoop clusters
- Expertise in ETL and good understanding on Joins, Partitions and optimizing queries
- Well versed with SDLC methodologies and practices including Agile
- Good To Have (1+ years of working experience)
- Experience working with workflow managers like Airflow, Prefect, Luigi, Oozie
- Experience working with data governance tools like Apache Atlas, Apache Sentry, Apache Ranger
- Knowledge of Docker, Kubernetes
- Experience using ElasticSearch / Apache Solr, and AWS Redshift/Google BigQuery
