Update mlops content (#7210)

pull/7253/head
dsh 2 months ago committed by GitHub
parent 1090545b82
commit a5f043b4e9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
  1. 7
      src/data/roadmaps/mlops/content/bash@mMzqJF2KQ49TDEk5F3VAI.md
  2. 11
      src/data/roadmaps/mlops/content/cicd@a6vawajw7BpL6plH_nuAz.md
  3. 5
      src/data/roadmaps/mlops/content/cloud-computing@00GZcwe25QYi7rDzaOoMt.md
  4. 8
      src/data/roadmaps/mlops/content/cloud-native-ml-services@kbfucfIO5KCsuv3jKbHTa.md
  5. 2
      src/data/roadmaps/mlops/content/containerization@tKeejLv8Q7QX40UtOjpav.md
  6. 7
      src/data/roadmaps/mlops/content/data-engineering-fundamentals@VykbCu7LWIx8fQpqKzoA7.md
  7. 7
      src/data/roadmaps/mlops/content/data-ingestion-architecture@Berd78HvnulNEGOsHCf8n.md
  8. 8
      src/data/roadmaps/mlops/content/data-lakes--warehouses@wOogVDV4FIDLXVPwFqJ8C.md
  9. 7
      src/data/roadmaps/mlops/content/data-lineage--feature-stores@6XgP_2NLuiw654zvTyueT.md
  10. 7
      src/data/roadmaps/mlops/content/data-pipelines@cOg3ejZRYE-u-M0c89IjM.md
  11. 6
      src/data/roadmaps/mlops/content/experiment-tracking--model-registry@fGGWKmAJ50Ke6wWJBEgby.md
  12. 3
      src/data/roadmaps/mlops/content/git@06T5CbZAGJU6fJhCmqCC8.md
  13. 10
      src/data/roadmaps/mlops/content/github@7t7jSb3YgyWlhgCe8Se1I.md
  14. 7
      src/data/roadmaps/mlops/content/go@vdVq3RQvQF3mF8PQc6DMg.md
  15. 8
      src/data/roadmaps/mlops/content/infrastructure-as-code@sf67bSL7HAx6iN7S6MYKs.md
  16. 6
      src/data/roadmaps/mlops/content/kubernetes@XQoK9l-xtN2J8ZV8dw53X.md
  17. 8
      src/data/roadmaps/mlops/content/machine-learning-fundamentals@ulka7VEVjz6ls5SnI6a6z.md
  18. 2
      src/data/roadmaps/mlops/content/mlops-components@l1xasxQy2vAY34NWaqKEe.md
  19. 22
      src/data/roadmaps/mlops/content/mlops-principles@iTsEHVCo6KGq7H2HMgy5S.md
  20. 7
      src/data/roadmaps/mlops/content/monitoring--observability@r4fbUwD83uYumEO1X8f09.md
  21. 9
      src/data/roadmaps/mlops/content/orchestration@fes7M--Y8i08_zeP98tVV.md
  22. 2
      src/data/roadmaps/mlops/content/programming-fundamentals@_7uvOebQUI4xaSwtMjpEd.md
  23. 11
      src/data/roadmaps/mlops/content/python@Vh81GnOUOZvDOlOyI5PwT.md

@ -1,5 +1,8 @@
# Bash
Understanding bash is essential for MLOps tasks.
Bash (Bourne Again Shell) is a Unix shell and command language used for interacting with the operating system through a terminal. It allows users to execute commands, automate tasks via scripting, and manage system operations. As the default shell for many Linux distributions, it supports command-line utilities, file manipulation, process control, and text processing. Bash scripts can include loops, conditionals, and functions, making it a powerful tool for system administration, automation, and task scheduling.
- **Book Suggestion:** _The Linux Command Line, 2nd Edition_ by William E. Shotts
Learn more from the following resources:
- [@opensource@bash-guide](https://github.com/Idnan/bash-guide)
- [@video@Bash Scripting Course](https://www.youtube.com/watch?v=tK9Oc6AEnR4)

@ -1,9 +1,8 @@
# CI / CD
Critical for traceable and reproducible ML model deployments.
CI/CD (Continuous Integration and Continuous Deployment/Delivery) is a software development practice that automates the process of integrating code changes, running tests, and deploying updates. Continuous Integration focuses on regularly merging code changes into a shared repository, followed by automated testing to ensure code quality. Continuous Deployment extends this by automatically releasing every validated change to production, while Continuous Delivery ensures code is always in a deployable state, but requires manual approval for production releases. CI/CD pipelines improve code reliability, reduce integration risks, and speed up the development lifecycle.
- **Books:**
- _Learning GitHub Actions_ by Brent Laster
- _Learning Git_ by Anna Skoulikari
- **Tutorials & Courses:** [Git & GitHub for beginners](https://www.youtube.com/watch?v=RGOj5yH7evk), [Python to Production guide](https://www.udemy.com/course/setting-up-the-linux-terminal-for-software-development/), [Version Control Missing Semester](https://missing.csail.mit.edu/2020/version-control/), https://learngitbranching.js.org/
- **Tool:** [Pre-commit hooks](https://marvelousmlops.substack.com/p/welcome-to-pre-commit-heaven)
Learn more from the following resources:
- [@article@What is CI/CD?](https://www.redhat.com/en/topics/devops/what-is-ci-cd)
- [@video@CI/CD In 5 Minutes](https://www.youtube.com/watch?v=42UP1fxi2SY)

@ -1,3 +1,8 @@
# Cloud Computing
**Cloud Computing** refers to the delivery of computing services over the internet rather than using local servers or personal devices. These services include servers, storage, databases, networking, software, analytics, and intelligence. Cloud Computing enables faster innovation, flexible resources, and economies of scale. There are various types of cloud computing such as public clouds, private clouds, and hybrids clouds. Furthermore, it's divided into different services like Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). These services differ mainly in the level of control an organization has over their data and infrastructures.
Learn more from the following resources:
- [@article@What is cloud computing?](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-cloud-computing)
- [@video@What is Cloud Computing? | Amazon Web Services](https://www.youtube.com/watch?v=mxT233EdY5c)

@ -2,8 +2,8 @@
Most of the cloud providers offer managed services for machine learning. These services are designed to help data scientists and machine learning engineers to build, train, and deploy machine learning models at scale. These services are designed to be cloud-native, meaning they are designed to work with other cloud services and are optimized for the cloud environment.
Here are the services offered by the major cloud providers:
Learn more from the following resources:
- **Amazon Web Services (AWS)**: SageMaker
- **Google Cloud Platform (GCP)**: AI Platform
- **Microsoft Azure**: Azure Machine Learning
- [@official@AWS Sage Maker](https://aws.amazon.com/sagemaker/)
- [@official@Azure ML](https://azure.microsoft.com/en-gb/products/machine-learning)
- [@video@What is Cloud Native?](https://www.youtube.com/watch?v=fp9_ubiKqFU)

@ -1,6 +1,6 @@
# Containers
Containers are a construct in which [cgroups](https://en.wikipedia.org/wiki/Cgroups), [namespaces](https://en.wikipedia.org/wiki/Linux_namespaces), and [chroot](https://en.wikipedia.org/wiki/Chroot) are used to fully encapsulate and isolate a process. This encapsulated process, called a container image, shares the kernel of the host with other containers, allowing containers to be significantly smaller and faster than virtual machines.
Containers are a construct in which cgroups, namespaces, and chroot are used to fully encapsulate and isolate a process. This encapsulated process, called a container image, shares the kernel of the host with other containers, allowing containers to be significantly smaller and faster than virtual machines.
These images are designed for portability, allowing for full local testing of a static image, and easy deployment to a container management platform.

@ -1,3 +1,8 @@
# Data Engineering Fundamentals
Data Engineering is essentially dealing with the collection, validation, storage, transformation, and processing of data. The objective is to provide reliable, efficient, and scalable data pipelines and infrastructure that allow data scientists to convert data into actionable insights. It involves steps like data ingestion, data storage, data processing, and data provisioning. Important concepts include designing, building, and maintaining data architecture, databases, processing systems, and large-scale processing systems. It is crucial to have extensive technical knowledge in various tools and programming languages like SQL, Python, Hadoop, and more.
Data Engineering is essentially dealing with the collection, validation, storage, transformation, and processing of data. The objective is to provide reliable, efficient, and scalable data pipelines and infrastructure that allow data scientists to convert data into actionable insights. It involves steps like data ingestion, data storage, data processing, and data provisioning. Important concepts include designing, building, and maintaining data architecture, databases, processing systems, and large-scale processing systems. It is crucial to have extensive technical knowledge in various tools and programming languages like SQL, Python, Hadoop, and more.
Learn more from the following resources:
- [@article@Data Engineering 101](https://www.redpanda.com/guides/fundamentals-of-data-engineering)
- [@video@Fundamentals of Data Engineering](https://www.youtube.com/watch?v=mPSzL8Lurs0)

@ -2,4 +2,9 @@
Data ingestion is the process of collecting, transferring, and loading data from various sources to a destination where it can be stored and analyzed. There are several data ingestion architectures that can be used to collect data from different sources and load it into a data warehouse, data lake, or other storage systems. These architectures can be broadly classified into two categories: batch processing and real-time processing. How you choose to ingest data will depend on the volume, velocity, and variety of data you are working with, as well as the latency requirements of your use case.
Lambda and Kappa architectures are two popular data ingestion architectures that combine batch and real-time processing to handle large volumes of data efficiently.
Lambda and Kappa architectures are two popular data ingestion architectures that combine batch and real-time processing to handle large volumes of data efficiently.
Learn more from the following resources:
- [@article@Data Ingestion Patterns](https://docs.aws.amazon.com/whitepapers/latest/aws-cloud-data-ingestion-patterns-practices/data-ingestion-patterns.html)
- [@video@What is a data pipeline?](https://www.youtube.com/watch?v=kGT4PcTEPP8)

@ -1,3 +1,9 @@
# Data lakes & Warehouses
"**Data Lakes** are large-scale data repository systems that store raw, untransformed data, in various formats, from multiple sources. They're often used for big data and real-time analytics requirements. Data lakes preserve the original data format and schema which can be modified as necessary. On the other hand, **Data Warehouses** are data storage systems which are designed for analyzing, reporting and integrating with transactional systems. The data in a warehouse is clean, consistent, and often transformed to meet wide-range of business requirements. Hence, data warehouses provide structured data but require more processing and management compared to data lakes."
"**Data Lakes** are large-scale data repository systems that store raw, untransformed data, in various formats, from multiple sources. They're often used for big data and real-time analytics requirements. Data lakes preserve the original data format and schema which can be modified as necessary. On the other hand, **Data Warehouses** are data storage systems which are designed for analyzing, reporting and integrating with transactional systems. The data in a warehouse is clean, consistent, and often transformed to meet wide-range of business requirements. Hence, data warehouses provide structured data but require more processing and management compared to data lakes."
Learn more from the following resources:
- [@article@Data lake definition](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-a-data-lake)
- [@video@What is a data lake?](https://www.youtube.com/watch?v=LxcH6z8TFpI)
- [@video@@hat is a data warehouse?](https://www.youtube.com/watch?v=k4tK2ttdSDg)

@ -2,8 +2,7 @@
**Data Lineage** refers to the life-cycle of data, including its origins, movements, characteristics and quality. It's a critical component in MLOps for tracking the journey of data through every process in a pipeline, from raw input to model output. Data lineage helps in maintaining transparency, ensuring compliance, and facilitating data debugging or tracing data related bugs. It provides a clear representation of data sources, transformations, and dependencies thereby aiding in audits, governance, or reproduction of machine learning models.
Feature stores are a crucial component of MLOps infrastructure.
Learn more from the following resources:
- **Tutorial:** Creating a feature store with Feast [Part 1](https://kedion.medium.com/creating-a-feature-store-with-feast-part-1-37c380223e2f) [Part 2](https://kedion.medium.com/feature-storage-for-ml-with-feast-part-2-34df1971a8d3) [Part 3](https://kedion.medium.com/feature-storage-for-ml-with-feast-a061899fc4a2)
- **Tool:** DVC for data tracking
- **Course:** [End-to-end machine learning (DVC piece)](https://www.udemy.com/course/sustainable-and-scalable-machine-learning-project-development/)
- [@article@What is Data Lineage?](https://www.ibm.com/topics/data-lineage)
- [@article@What is a feature store](https://www.snowflake.com/guides/what-feature-store-machine-learning/)

@ -1,5 +1,8 @@
# Data Pipelines
Data pipelines refer to a set of processes that involve moving data from one system to another, for purposes such as data integration, data migration, data transformation, or data synchronization. These processes can involve a variety of data sources and destinations, and may often require data to be cleaned, enriched, or otherwise transformed along the way. It's a key concept in data engineering to ensure that data is appropriately processed from its source to the location where it will be used, typically a data warehouse, data mart, or a data lake. As such, data pipelines play a crucial part in building an effective and efficient data analytics setup, enabling the flow of data to be processed for insights.
Data pipelines are a series of automated processes that transport and transform data from various sources to a destination for analysis or storage. They typically involve steps like data extraction, cleaning, transformation, and loading (ETL) into databases, data lakes, or warehouses. Pipelines can handle batch or real-time data, ensuring that large-scale datasets are processed efficiently and consistently. They play a crucial role in ensuring data integrity and enabling businesses to derive insights from raw data for reporting, analytics, or machine learning.
It is important to understand the difference between ELT and ETL pipelines. ELT stands for Extract, Load, Transform, and refers to a process where data is first extracted from source systems, then loaded into a target system, and finally transformed within the target system. ETL, on the other hand, stands for Extract, Transform, Load, and refers to a process where data is first extracted from source systems, then transformed, and finally loaded into a target system. The choice between ELT and ETL pipelines depends on the specific requirements of the data processing tasks at hand, and the capabilities of the systems involved.
Learn more from the following resources:
- [@article@What is a data pipeline?](https://www.ibm.com/topics/data-pipeline)
- [@video@What are data pipelines?](https://www.youtube.com/watch?v=oKixNpz6jNo)

@ -2,7 +2,7 @@
**Experiment Tracking** is an essential part of MLOps, providing a system to monitor and record the different experiments conducted during the machine learning model development process. This involves capturing, organizing and visualizing the metadata associated with each experiment, such as hyperparameters used, models produced, metrics like accuracy or loss, and other information about the computational environment. This tracking allows for reproducibility of experiments, comparison across different experiment runs, and helps in identifying the best models.
Logging metadata, parameters, and artifacts of training runs.
Learn more from the following resources:
- **Tool:** MLflow
- **Courses:** [MLflow Udemy course](https://www.udemy.com/course/mlflow-course/), [End-to-end machine learning (MLflow piece)](https://www.udemy.com/course/sustainable-and-scalable-machine-learning-project-development/)
- [@article@Experiment Tracking](https://madewithml.com/courses/mlops/experiment-tracking/#dashboard)
- [@article@ML Flow Model Registry](https://mlflow.org/docs/latest/model-registry.html)

@ -1,9 +1,10 @@
# Git
[Git](https://git-scm.com/) is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.
Git is a distributed version control system used to track changes in source code during software development. It enables multiple developers to collaborate on a project by managing versions of code, allowing for branching, merging, and tracking of revisions. Git ensures that changes are recorded with a complete history, enabling rollback to previous versions if necessary. It supports distributed workflows, meaning each developer has a complete local copy of the project’s history, facilitating seamless collaboration, conflict resolution, and efficient management of code across different teams or environments.
Visit the following resources to learn more:
- [@roadmap@Learn Git & GitHub](https://roadmap.sh/git-github)
- [@video@Git & GitHub Crash Course For Beginners](https://www.youtube.com/watch?v=SWYqp7iY_Tc)
- [@article@Learn Git with Tutorials, News and Tips - Atlassian](https://www.atlassian.com/git)
- [@article@Git Cheat Sheet](https://cs.fyi/guide/git-cheatsheet)

@ -1,15 +1,11 @@
# GitHub
GitHub is a provider of Internet hosting for software development and version control using Git. It offers the distributed version control and source code management functionality of Git, plus its own features.
GitHub is a web-based platform built on top of Git that provides version control, collaboration tools, and project management features for software development. It enables developers to host Git repositories, collaborate on code through pull requests, and review and track changes. GitHub also offers additional features like issue tracking, continuous integration, automated workflows, and documentation hosting. With its social coding environment, GitHub fosters open-source contributions and team collaboration, making it a central hub for many software development projects.
Visit the following resources to learn more:
- [@opensource@GitHub Website](https://github.com)
- [@roadmap@Learn Git & GitHub](https://roadmap.sh/git-github)
- [@official@GitHub Website](https://github.com)
- [@article@GitHub Documentation](https://docs.github.com/en/get-started/quickstart)
- [@article@How to Use Git in a Professional Dev Team](https://ooloo.io/project/github-flow)
- [@video@What is GitHub?](https://www.youtube.com/watch?v=w3jLJU7DT5E)
- [@video@Git vs. GitHub: Whats the difference?](https://www.youtube.com/watch?v=wpISo9TNjfU)
- [@video@Git and GitHub for Beginners](https://www.youtube.com/watch?v=RGOj5yH7evk)
- [@video@Git and GitHub - CS50 Beyond 2019](https://www.youtube.com/watch?v=eulnSXkhE7I)
- [@article@Learn Git Branching](https://learngitbranching.js.org/?locale=en_us)
- [@feed@Explore top posts about GitHub](https://app.daily.dev/tags/github?ref=roadmapsh)

@ -1,15 +1,12 @@
# Go
Go is an open source programming language supported by Google. Go can be used to write cloud services, CLI tools, used for API development, and much more.
Go, also known as Golang, is an open-source programming language developed by Google that emphasizes simplicity, efficiency, and strong concurrency support. Designed for modern software development, Go features a clean syntax, garbage collection, and built-in support for concurrent programming through goroutines and channels, making it well-suited for building scalable, high-performance applications, especially in cloud computing and microservices architectures. Go's robust standard library and tooling ecosystem, including a powerful package manager and testing framework, further streamline development processes, promoting rapid application development and deployment.
Visit the following resources to learn more:
- [@roadmap@Visit Dedicated Go Roadmap](https://roadmap.sh/golang)
- [@official@A Tour of Go – Go Basics](https://go.dev/tour/welcome/1)
- [@official@Go Reference Documentation](https://go.dev/doc/)
- [@article@Go by Example - annotated example programs](https://gobyexample.com/)
- [@article@W3Schools Go Tutorial ](https://www.w3schools.com/go/)
- [@video@Go Programming Course](https://www.youtube.com/watch?v=un6ZyFkqFKo)
- [@article@Making a RESTful JSON API in Go](https://thenewstack.io/make-a-restful-json-api-go/)
- [@article@Go, the Programming Language of the Cloud](https://thenewstack.io/go-the-programming-language-of-the-cloud/)
- [@video@Go Class by Matt](https://www.youtube.com/playlist?list=PLoILbKo9rG3skRCj37Kn5Zj803hhiuRK6)
- [@feed@Explore top posts about Golang](https://app.daily.dev/tags/golang?ref=roadmapsh)

@ -1,7 +1,7 @@
# Infrastructure as Code
Essential for a reproducible MLOps framework.
Infrastructure as Code (IaC) is a modern approach to managing and provisioning IT infrastructure through machine-readable configuration files, rather than manual processes. It enables developers and operations teams to define and manage infrastructure resources—such as servers, networks, and databases—using code, which can be versioned, tested, and deployed like application code. IaC tools, like Terraform and AWS CloudFormation, allow for automated, repeatable deployments, reducing human error and increasing consistency across environments. This practice facilitates agile development, enhances collaboration between teams, and supports scalable and efficient infrastructure management.
- **Course:** [Terraform course for beginners](https://www.youtube.com/watch?v=SLB_c_ayRMo)
- **Video:** [8 Terraform best practices by Techworld by Nana](https://www.youtube.com/watch?v=gxPykhPxRW0)
- **Book Suggestion:** _Terraform: Up and Running, 3rd Edition_ by Yevgeniy Brikman
- [@article@What is Infrastructure as Code?](https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac)
- [@video@Terraform course for beginners](https://www.youtube.com/watch?v=SLB_c_ayRMo)
- [@video@8 Terraform best practices](https://www.youtube.com/watch?v=gxPykhPxRW0)

@ -1,14 +1,12 @@
# Kubernetes
Kubernetes is an [open source](https://github.com/kubernetes/kubernetes) container management platform, and the dominant product in this space. Using Kubernetes, teams can deploy images across multiple underlying hosts, defining their desired availability, deployment logic, and scaling logic in YAML. Kubernetes evolved from Borg, an internal Google platform used to provision and allocate compute resources (similar to the Autopilot and Aquaman systems of Microsoft Azure).
The popularity of Kubernetes has made it an increasingly important skill for the DevOps Engineer and has triggered the creation of Platform teams across the industry. These Platform engineering teams often exist with the sole purpose of making Kubernetes approachable and usable for their product development colleagues.
Kubernetes is an open source container management platform, and the dominant product in this space. Using Kubernetes, teams can deploy images across multiple underlying hosts, defining their desired availability, deployment logic, and scaling logic in YAML. Kubernetes evolved from Borg, an internal Google platform used to provision and allocate compute resources (similar to the Autopilot and Aquaman systems of Microsoft Azure). The popularity of Kubernetes has made it an increasingly important skill for the DevOps Engineer and has triggered the creation of Platform teams across the industry. These Platform engineering teams often exist with the sole purpose of making Kubernetes approachable and usable for their product development colleagues.
Visit the following resources to learn more:
- [@roadmap@Kubernetes Roadmap](https://roadmap.sh/kubernetes)
- [@official@Kubernetes Website](https://kubernetes.io/)
- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/)
- [@video@Kubernetes Crash Course for Absolute Beginners](https://www.youtube.com/watch?v=s_o8dwzRlu4)
- [@article@Primer: How Kubernetes Came to Be, What It Is, and Why You Should Care](https://thenewstack.io/primer-how-kubernetes-came-to-be-what-it-is-and-why-you-should-care/)
- [@article@Kubernetes: An Overview](https://thenewstack.io/kubernetes-an-overview/)
- [@feed@Explore top posts about Kubernetes](https://app.daily.dev/tags/kubernetes?ref=roadmapsh)

@ -1,6 +1,8 @@
# Machine Learning Fundamentals
An MLOps engineer should have a basic understanding of machine learning models.
Machine learning fundamentals encompass the key concepts and techniques that enable systems to learn from data and make predictions or decisions without being explicitly programmed. At its core, machine learning involves algorithms that can identify patterns in data and improve over time with experience. Key areas include supervised learning (where models are trained on labeled data), unsupervised learning (where models identify patterns in unlabeled data), and reinforcement learning (where agents learn to make decisions based on feedback from their actions). Essential components also include data preprocessing, feature selection, model training, evaluation metrics, and the importance of avoiding overfitting. Understanding these fundamentals is crucial for developing effective machine learning applications across various domains.
Learn more from the following resources:
- **Courses:** [MLCourse.ai](https://mlcourse.ai/), [Fast.ai](https://course.fast.ai)
- **Book Suggestion:** _Applied Machine Learning and AI for Engineers_ by Jeff Prosise
- [@course@Fundamentals of Machine Learning - Microsoft](https://learn.microsoft.com/en-us/training/modules/fundamentals-machine-learning/)
- [@course@MLCourse.ai](https://mlcourse.ai/)
- [@course@Fast.ai](https://course.fast.ai)

@ -1,3 +1,3 @@
# MLOps Components
MLOps components can be broadly classified into three major categories: Development, Operations and Governance. The **Development** components include everything involved in the creation of machine learning models, such as data extraction, data analysis, feature engineering, and machine learning model training. The **Operations** category includes components involved in deploying, monitoring, and maintaining machine learning models in production. This may include release management, model serving, and performance monitoring. Lastly, the **Governance** category encompasses the policies and regulations related to machine learning models. This includes model audit and tracking, model explainability, and security & compliance regulations.
MLOps components can be broadly classified into three major categories: Development, Operations and Governance. The **Development** components include everything involved in the creation of machine learning models, such as data extraction, data analysis, feature engineering, and machine learning model training. The **Operations** category includes components involved in deploying, monitoring, and maintaining machine learning models in production. This may include release management, model serving, and performance monitoring. Lastly, the **Governance** category encompasses the policies and regulations related to machine learning models. This includes model audit and tracking, model explainability, and security & compliance regulations.

@ -1,9 +1,19 @@
# MLOps Principles
Awareness of MLOps principles and maturity factors is required.
MLOps (Machine Learning Operations) principles focus on streamlining the deployment, monitoring, and management of machine learning models in production environments. Key principles include:
- **Books:**
- _Designing Machine Learning Systems_ by Chip Huyen
- _Introducing MLOps_ by Mark Treveil and Dataiku
- **Assessment:** [MLOps maturity assessment](https://marvelousmlops.substack.com/p/mlops-maturity-assessment)
- **Great resource on MLOps:** [ml-ops.org](https://ml-ops.org)
1. **Collaboration**: Foster collaboration between data scientists, developers, and operations teams to ensure alignment on model goals, performance, and lifecycle management.
2. **Automation**: Automate workflows for model training, testing, deployment, and monitoring to enhance efficiency, reduce errors, and speed up the development lifecycle.
3. **Version Control**: Implement version control for both code and data to track changes, reproduce experiments, and maintain model lineage.
4. **Continuous Integration and Deployment (CI/CD)**: Establish CI/CD pipelines tailored for machine learning to facilitate rapid model iteration and deployment.
5. **Monitoring and Governance**: Continuously monitor model performance and data drift in production to ensure models remain effective and compliant with regulatory requirements.
6. **Scalability**: Design systems that can scale to handle varying workloads and accommodate changes in data volume and complexity.
7. **Reproducibility**: Ensure that experiments can be reliably reproduced by standardizing environments and workflows, making it easier to validate and iterate on models.
These principles help organizations efficiently manage the lifecycle of machine learning models, from development to deployment and beyond.

@ -2,6 +2,7 @@
**Monitoring** in MLOps primarily involves tracking the performance of machine learning (ML) models in production to ensure that they continually deliver accurate and reliable results. Such monitoring is necessary because the real-world data that these models handle may change over time, a scenario known as data drift. These changes can adversely affect model performance. Monitoring helps to detect any anomalies in the model’s behaviour or performance and such alerts can trigger the retraining of models with new data. From a broader perspective, monitoring also involves tracking resources and workflows to detect and rectify any operational issues in the MLOps pipeline.
- [@article@**ML Monitoring vs Observability article**](https://marvelousmlops.substack.com/p/ml-monitoring-vs-ml-observability)
- **Course:** [Machine learning monitoring concepts](https://app.datacamp.com/learn/courses/machine-learning-monitoring-concepts), [Monitoring ML in Python](https://app.datacamp.com/learn/courses/monitoring-machine-learning-in-python)
- **Tools:** [Prometheus, Grafana](https://www.udemy.com/course/mastering-prometheus-and-grafana/)
Learn more from the following resources:
- [@article@ML Monitoring vs ML Observability](https://medium.com/marvelous-mlops/ml-monitoring-vs-ml-observability-understanding-the-differences-fff574a8974f)
- [@video@ML Observability vs ML Monitoring: What's the difference?](https://www.youtube.com/watch?v=k1Reed3QIYE)

@ -1,4 +1,7 @@
Systems like Airflow and Mage are important in ML engineering.
# Orchestration
- **Course:** [Introduction to Airflow in Python](https://app.datacamp.com/learn/courses/introduction-to-airflow-in-python)
- **Note:** Airflow is also featured in the _ML Engineering with Python_ book and [_The Full Stack 7-Steps MLOps Framework_](https://www.pauliusztin.me/courses/the-full-stack-7-steps-mlops-framework).
ML orchestration refers to the process of managing and coordinating the various tasks and workflows involved in the machine learning lifecycle, from data preparation and model training to deployment and monitoring. It involves integrating multiple tools and platforms to streamline operations, automate repetitive tasks, and ensure seamless collaboration among data scientists, engineers, and operations teams. By using orchestration frameworks, organizations can enhance reproducibility, scalability, and efficiency, enabling them to manage complex machine learning pipelines and improve the overall quality of models in production. This ensures that models are consistently updated and maintained, facilitating rapid iteration and adaptation to changing data and business needs.
Learn more from the following resources:
- [@article@ML Observability: what, why, how](https://ubuntu.com/blog/ml-observability)

@ -1,3 +1,3 @@
# Programming Fundamentals
Programming is the key requirement for MLOps. You need to be proficient in atleast one programming language. Python is the most popular language for MLOps.
ML programming fundamentals encompass the essential skills and concepts needed to develop machine learning models effectively. Key aspects include understanding data structures and algorithms, as well as proficiency in programming languages commonly used in ML, such as Python and R. Familiarity with libraries and frameworks like TensorFlow, PyTorch, and scikit-learn is crucial for implementing machine learning algorithms and building models. Additionally, concepts such as data preprocessing, feature engineering, model evaluation, and hyperparameter tuning are vital for optimizing performance. A solid grasp of statistics and linear algebra is also important, as these mathematical foundations underpin many ML techniques, enabling practitioners to analyze data and interpret model results accurately.

@ -2,11 +2,10 @@
Python is an interpreted high-level general-purpose programming language. Its design philosophy emphasizes code readability with its significant use of indentation. Its language constructs as well as its object-oriented approach aim to help programmers write clear, logical code for small and large-scale projects. Python is dynamically-typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly, procedural), object-oriented and functional programming. Python is often described as a "batteries included" language due to its comprehensive standard library.
To start learning Python, here are some useful resources:
Learn more from the following resources:
- [@official@Python.org](https://www.python.org/) - The official website offers extensive documentation and tutorials for beginners as well as advanced users.
- [@article@Real Python](https://realpython.com/) - Offers a variety of Python tutorials, articles, and courses that cater to different experience levels.
- [@article@Automate the Boring Stuff with Python](https://automatetheboringstuff.com/) - A beginner-friendly book that teaches Python by guiding you through practical tasks and automation examples.
- [@roadmap@Python Roadmap](https://roadmap.sh/python)
- [@official@Python.org](https://www.python.org/)
- [@article@Real Python](https://realpython.com/)
- [@article@Automate the Boring Stuff with Python](https://automatetheboringstuff.com/)
- [@feed@Explore top posts about Python](https://app.daily.dev/tags/python?ref=roadmapsh)
Remember, practice is key, and the more you work with Python, the more you'll appreciate its utility in the world of cyber security.

Loading…
Cancel
Save