V
2 months ago
Job Summary
We are looking for a highly motivated Senior C++ Storage Engineer to join Voltron Data’s team. On the team, you’ll have the opportunity to help support and grow the Voltron Data and Apache Arrow ecosystems. You will work closely with Voltron Data development teams to implement performant storage and I/O functions targeting a wide variety of networked, cloud, and local storage solutions.
- Minimum Qualification: Degree
- Experience Level: Senior level
- Experience Length: 5 years
Job Description/Requirements
Responsibilities:
Upon joining:
- Spending time learning about the Apache Arrow memory layout, compute primitives, and APIs.
- Familiarizing yourself with the different partners for compute kernels and the query execution engine on Apache Arrow.
- Learning and embracing the Apache development process.
Within a month:
- Implementing new high-performance storage and I/O primitives.
- Benchmarking existing I/O library functions to determine where there are bottlenecks.
- Discovering and implementing optimizations in data reads and writes.
- Participating in peer code review of all PRs related to file storage and interacting with different filesystems.
- Contributing to technical discussions and technical design documents.
Within 6 months:
- Developing a comprehensive set of low level benchmarks for I/O functions targeting various local, networked and cloud storage technologies to enable monitoring for performance regressions.
- Ensuring that all filesystems interactions are compatible and performant across platforms (Linux, MacOS, and Windows).
- Identifying and building reusable software components to ensure a high quality and maintainable codebase.
Within 12 months:
- Analyzing I/O throughput in a massively parallel and distributed query engine to identify inefficiencies and crafting solutions to tackle those inefficiencies.
- Ensuring that the everything related to storage is built as high quality as possible, balancing performance, usability, and maintainability across the Voltron Data and Apache Arrow ecosystems.
Previous experience that could be helpful:
- Strong experience developing in C++, especially using Modern C++.
- Experience developing and using various data lake storage technologies as: S3, Google Compute Storage, Azure Blob Storage.
- Building and using distributed networked file systems such as HDFS or Ceph.
- Experience working with technologies such as io_uring, DMA, RDMA, or GPUDirect Storage.
- Experience with different data storage file formats such as ORC, Parquet, and Avro.
- Experience with data lake table formats such as Iceberg, Delta Lake, and Hudi.
Important Safety Tips
- Do not make any payment without confirming with the Jobberman Customer Support Team.
- If you think this advert is not genuine, please report it via the Report Job link below.
