<p>As a Software Engineer within the Autopilot AI Infrastructure team, you will work on reinforcing, optimizing, and scaling our infrastructure components supporting AI research activities for Autopilot and the Optimus.</p> <p>At the core of our autonomy capabilities are neural networks that the research team is designing to train on very large amounts of data, across large-scale GPU clusters and our supercomputer Dojo. Robustly training these models at scale and in the shortest amount of timeis critical to our mission.</p> <p>We are building and improving the in-house distributed training framework used by the research team to train production models, ensuring good ergonomics and flexibility for experimentation while providing good stability and performance.</p> <ul> <li>Write robust Python software code in our machine learning training repository while applying best software practices to support the research team </li><li>Increase the reliability of our training jobs by debugging and root causing failures across thousands of nodes and implementing fixes to prevent future failures </li><li>Improve our training framework to support new training paradigms and experimentation methods </li><li>Build and improve our monitoring/observability infra to quickly debug cluster and training application issues </li><li>Profile and identifyperformance bottlenecks of training software in our training cluster </li><li>Coordinate with the supercomputing team managing the training cluster to maintain high availability and job throughput </li><li>Members of the Autopilot AI Infrastructure team are expected to beadaptable tothe dynamic requirements of AI research and capable of contributing across all parts of the AI training software stack </li><li>Practical programming experience in Python and/or C/C++ </li><li>Experience working with ML training frameworks (ideally PyTorch) </li><li>Demonstrated experience scaling neural network training jobs across many GPUs </li><li>Experience with parallel programming concepts and primitives </li><li>Experience profiling and optimizing CPU-GPU interactions (pipelining computation with data transfers, etc.) </li><li>Proficient in system-level software, in particular hardware-software interactions and resource utilization </li><li>Understanding of state-of-the-art deep learning concepts </li><li>Experience programming in CUDA/Triton and/or NCCL internals </li></ul>