Awesome Software Engineering for Machine Learning

Software Engineering for Machine Learning are techniques and guidelines for building ML applications that do not concern the core ML problem -- e.g. the development of new algorithms -- but rather the surrounding activities like data ingestion, coding, testing, versioning, deployment, quality control, and team collaboration. Good software engineering practices enhance development, deployment and maintenance of production level applications using machine learning components.

⭐ Must-read

🎓 Scientific publication

Feel free to take and share the survey!

Contents

• Broad Overviews
• Data Management
• Model Training
• Deployment and Operation
• Social Aspects
• Governance
• Tooling

Broad Overviews

These resources cover all aspects.

• Best Practices for Machine Learning Applications
• Hidden Technical Debt in Machine Learning Systems 🎓⭐
• Machine Learning Engineering Best Practices ⭐
• Rules of Machine Learning: Best Practices for ML Engineering ⭐
• Software Engineering for Machine Learning: A Case Study 🎓⭐

Data Management

How to manage the data sets you use in machine learning.

• A Survey on Data Collection for Machine Learning A Big Data - AI Integration Perspective_2019 🎓
• Automating Large-Scale Data Quality Verification 🎓
• Data management challenges in production machine learning
• How to organize data labelling for ML
• The curse of big data labeling and three ways to solve it
• The Data Linter: Lightweight, Automated Sanity Checking for ML Data Sets 🎓
• The ultimate guide to data labeling for ML

Model Training

How to organize your model training experiments.

• 10 Best Practices for Deep Learning
• Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement 🎓
• How do you manage your Machine Learning Experiments?
• Machine Learning Testing: Survey, Landscapes and Horizons 🎓
• On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach 🎓⭐
• On human intellect and machine failures: Troubleshooting integrative machine learning systems 🎓
• Pitfalls and Best Practices in Algorithm Configuration 🎓
• Pitfalls of supervised feature selection 🎓
• Preparing and Architecting for Machine Learning
• Preliminary Systematic Literature Review of Machine Learning System Development Process 🎓
• Software development best practices in a deep learning environment
• Testing and Debugging in Machine Learning
• What Went Wrong and Why? Diagnosing Situated Interaction Failures in the Wild 🎓

Deployment and Operation

How to deploy and operate your models in a production environment.

• Best Practices in Machine Learning Infrastructure
• Continuous Delivery for Machine Learning ⭐
• Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform 🎓
• Fairness Indicators: Scalable Infrastructure for Fair ML Systems 🎓
• Machine Learning Logistics
• Machine learning: Moving from experiments to production
• ML Ops: Machine Learning as an engineered disciplined
• Model Governance Reducing the Anarchy of Production 🎓
• ModelOps: Cloud-based lifecycle management for reliable and trusted AI
• Operational Machine Learning
• Scaling Machine Learning as a Service🎓
• TFX: A tensorflow-based Production-Scale ML Platform 🎓
• The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction 🎓
• Versioning for end-to-end machine learning pipelines 🎓

Social Aspects

How to organize teams and projects to ensure effective collaboration and accountability.

• Data Scientists in Software Teams: State of the Art and Challenges 🎓
• Machine Learning Interviews
• Managing Machine Learning Projects
• Principled Machine Learning: Practices and Tools for Efficient Collaboration

Governance

• Responsible AI practices ⭐
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims

Tooling

Tooling can make your life easier.

We only share open source tools, or commercial platforms that offer substantial free packages for research.

• Airflow
• Auto-PyTorch
• Ax
• Data Version Control (DVC)
• Facets Overview / Facets Dive
• Git Large File System (LFS)
• HParams
• Kubeflow
• Label Studio
• MLFlow
• Neptune.ai
• Neuraxle
• OpenML
• Spark Machine Learning
• Sacred
• TensorBoard
• Tensorflow Extended (TFX)
• Weights & Biases

Contribute

Contributions welcomed! Read the contribution guidelines first