Sr. Data Engineer, Hadoop/Spark
The Walt Disney Studios
The Senior Data Engineer works in a small team of multi-disciplined engineers on creating the next generation video consumption platforms. We expect you to be up to date on the happenings in the data community, passionate about what you do and connected to the open source community. You will participate in overall system design, come up with webtailored solutions that emphasize reuse and good design patterns
Responsibilities
- Build and optimize performance of Hadoop/Spark batch jobs.
- Build and optimize performance of Spark, Kafka, Cassandra, ELK, and whatever else makes sense for realtime pipelines.
- Design and architect high quality data-lake, data-warehouse, and data-marts data models.
- Enable and implement Data Science workflows and advanced machine learning althorithms.
- Build and optimize performance of ElasticSearch cluster and relevance
- Build data pipelines orchestration.
- Become and stay an expert in current and emerging technologies and tools
- Contribute to Open Source solutions and communities wherever you can
- Collaborate with other software engineers and crossfunctional teams
- Evangelize technologies, solutions, and best practices
- Contribute new ideas to a larger community of highcaliber professionals
- Balance resources, requirements, and complexity
Basic Qualifications
- Very passionate about coding (If you have a Github profile, that’s awesome! Would love to check it out! Otherwise we will ask you to code before or during the interview).
- String understanding of distributed systems and distributed computation.
- Strong working knowledge in at least 2 of: Scala, Java, Python, or Go-Lang
- Demonstrated working knowledge with data Apache Hadoop / Spark ecosystem like Spark, Hive, Presto, Oozie, Pig, Hue, Zeppelin
- Demonstrated working knowledge of data modeling
- Unit, Integration, and Load testing
- Developing REST APIs.
- Git
- Ant, Maven, SBT, and/or Gradle
- Unix/Linux
- Docker containers building and deployment.
- Excellent communication and collaboration skills
Preferred Qualifications
- GraphQL knowledge
- Kubernetes knowledge
- Apache Spark MLlib
- Apache Spark GraphX
- R
- Amazon AWS or other cloud Services
- Jenkins
Required Education
- BS in Computer Science or related field with 5+ years of experience
Company Overview
Movies Anywhere is the next generation of in-home entertainment, providing an unparalleled digital entertainment experience. Leveraging cutting edge technology, unique partnerships, and a talented team, Movies Anywhere is an exclusive, cross-platform, cloud-based movie service that enables consumers to seamlessly discover, grow, access, and enjoy their personal digital movie collection across a variety of studios, retailers, and platforms all in one convenient app and/or website.
Additional Information
This position is a legal entity of The Walt Disney Studios, an equal opportunity employer.
Full Time