fix: refactor and optimize resources (#8257)

pull/8389/head
Vedansh 2 weeks ago committed by GitHub
parent 4268ce4346
commit f077f354d7
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
  1. 3
      src/data/roadmaps/mlops/content/airflow@pVSlVHXIap0unFxLGM-lQ.md
  2. 9
      src/data/roadmaps/mlops/content/aws--azure--gcp@u3E7FGW4Iwdsu61KYFxCX.md
  3. 1
      src/data/roadmaps/mlops/content/bash@mMzqJF2KQ49TDEk5F3VAI.md
  4. 3
      src/data/roadmaps/mlops/content/cicd@a6vawajw7BpL6plH_nuAz.md
  5. 5
      src/data/roadmaps/mlops/content/cloud-computing@00GZcwe25QYi7rDzaOoMt.md
  6. 4
      src/data/roadmaps/mlops/content/containerization@tKeejLv8Q7QX40UtOjpav.md
  7. 2
      src/data/roadmaps/mlops/content/data-ingestion-architecture@Berd78HvnulNEGOsHCf8n.md
  8. 8
      src/data/roadmaps/mlops/content/data-lakes--warehouses@wOogVDV4FIDLXVPwFqJ8C.md
  9. 2
      src/data/roadmaps/mlops/content/data-lineage--feature-stores@6XgP_2NLuiw654zvTyueT.md
  10. 4
      src/data/roadmaps/mlops/content/data-pipelines@cOg3ejZRYE-u-M0c89IjM.md
  11. 5
      src/data/roadmaps/mlops/content/docker@XIdCvT-4HyyglHJLRrHlz.md
  12. 4
      src/data/roadmaps/mlops/content/flink@o6GQ3-8DgDtHzdX6yeg1w.md
  13. 2
      src/data/roadmaps/mlops/content/git@06T5CbZAGJU6fJhCmqCC8.md
  14. 6
      src/data/roadmaps/mlops/content/github@7t7jSb3YgyWlhgCe8Se1I.md
  15. 1
      src/data/roadmaps/mlops/content/go@vdVq3RQvQF3mF8PQc6DMg.md
  16. 7
      src/data/roadmaps/mlops/content/infrastructure-as-code@sf67bSL7HAx6iN7S6MYKs.md
  17. 2
      src/data/roadmaps/mlops/content/kafka@fMNwzhgLgHlAZJ9NvKikR.md
  18. 4
      src/data/roadmaps/mlops/content/kubernetes@XQoK9l-xtN2J8ZV8dw53X.md
  19. 1
      src/data/roadmaps/mlops/content/machine-learning-fundamentals@ulka7VEVjz6ls5SnI6a6z.md
  20. 5
      src/data/roadmaps/mlops/content/mlops-components@l1xasxQy2vAY34NWaqKEe.md
  21. 6
      src/data/roadmaps/mlops/content/mlops-principles@iTsEHVCo6KGq7H2HMgy5S.md
  22. 10
      src/data/roadmaps/mlops/content/model-training--serving@zsW1NRb0dMgS-KzWsI0QU.md
  23. 4
      src/data/roadmaps/mlops/content/python@Vh81GnOUOZvDOlOyI5PwT.md
  24. 3
      src/data/roadmaps/mlops/content/version-control-systems@oUhlUoWQQ1txx_sepD5ev.md
  25. 3
      src/data/roadmaps/mlops/content/version-control@kHDSwlSq8WkLey4EJIQSR.md

@ -4,5 +4,6 @@ Airflow is a platform to programmatically author, schedule and monitor workflows
Visit the following resources to learn more:
- [@article@Airflow website](https://airflow.apache.org/)
- [@official@Airflow](https://airflow.apache.org/)
- [@official@Airflow Documentation](https://airflow.apache.org/docs)
- [@feed@Explore top posts about Apache Airflow](https://app.daily.dev/tags/apache-airflow?ref=roadmapsh)

@ -1,10 +1,11 @@
# AWS / Azure / GCP
AWS (Amazon Web Services) Azure and GCP (Google Cloud Platform) are three leading providers of cloud computing services. AWS by Amazon is the oldest and the most established among the three, providing a breadth and depth of solutions ranging from infrastructure services like compute, storage, and databases to the machine and deep learning. Azure, by Microsoft, has integrated tools for DevOps, supports a large number of programming languages, and offers seamless integration with on-prem servers and Microsoft’s software. Google's GCP has strength in cost-effectiveness, live migration of virtual machines, and flexible computing options. All three have introduced various MLOps tools and services to boost capabilities for machine learning development and operations.
AWS (Amazon Web Services), Azure and GCP (Google Cloud Platform) are three leading providers of cloud computing services. AWS by Amazon is the oldest and the most established among the three, providing a breadth and depth of solutions ranging from infrastructure services like compute, storage, and databases to the machine and deep learning. Azure, by Microsoft, has integrated tools for DevOps, supports a large number of programming languages, and offers seamless integration with on-prem servers and Microsoft’s software. Google's GCP has strength in cost-effectiveness, live migration of virtual machines, and flexible computing options. All three have introduced various MLOps tools and services to boost capabilities for machine learning development and operations.
Visit the following resources to learn more about AWS, Azure, and GCP:
- [@roadmap.sh@AWS Roadmap](https://roadmap.sh/aws)
- [@article@Azure Tutorials](https://docs.microsoft.com/en-us/learn/azure/)
- [@article@GCP Learning Resources](https://cloud.google.com/training)
- [@roadmap.sh@Visit Dedicated AWS Roadmap](https://roadmap.sh/aws)
- [@official@Microsoft Azure](https://docs.microsoft.com/en-us/learn/azure/)
- [@official@Google Cloud Platform](https://cloud.google.com/)
- [@official@GCP Learning Resources](https://cloud.google.com/training)
- [@feed@Explore top posts about AWS](https://app.daily.dev/tags/aws?ref=roadmapsh)

@ -4,5 +4,6 @@ Bash (Bourne Again Shell) is a Unix shell and command language used for interact
Learn more from the following resources:
- [@article@Bash Reference Manual](https://www.gnu.org/software/bash/manual/bashref.html)
- [@opensource@bash-guide](https://github.com/Idnan/bash-guide)
- [@video@Bash Scripting Course](https://www.youtube.com/watch?v=tK9Oc6AEnR4)

@ -4,5 +4,6 @@ CI/CD (Continuous Integration and Continuous Deployment/Delivery) is a software
Learn more from the following resources:
- [@article@What is CI/CD?](https://www.redhat.com/en/topics/devops/what-is-ci-cd)
- [@article@What is CI/CD? - Gitlab](https://about.gitlab.com/topics/ci-cd/)
- [@article@What is CI/CD? - Redhat](https://www.redhat.com/en/topics/devops/what-is-ci-cd)
- [@video@CI/CD In 5 Minutes](https://www.youtube.com/watch?v=42UP1fxi2SY)

@ -4,5 +4,6 @@
Learn more from the following resources:
- [@article@What is cloud computing?](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-cloud-computing)
- [@video@What is Cloud Computing? | Amazon Web Services](https://www.youtube.com/watch?v=mxT233EdY5c)
- [@article@Cloud Computing - IBM](https://www.ibm.com/think/topics/cloud-computing)
- [@article@What is Cloud Computing? - Azure](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-cloud-computing)
- [@video@What is Cloud Computing? - Amazon Web Services](https://www.youtube.com/watch?v=mxT233EdY5c)

@ -6,8 +6,8 @@ These images are designed for portability, allowing for full local testing of a
Visit the following resources to learn more:
- [@article@What are Containers?](https://cloud.google.com/learn/what-are-containers)
- [@article@What is a Container?](https://www.docker.com/resources/what-container/)
- [@article@What are Containers? - Google Cloud](https://cloud.google.com/learn/what-are-containers)
- [@article@What is a Container? - Docker](https://www.docker.com/resources/what-container/)
- [@video@What are Containers?](https://www.youtube.com/playlist?list=PLawsLZMfND4nz-WDBZIj8-nbzGFD4S9oz)
- [@article@Articles about Containers - The New Stack](https://thenewstack.io/category/containers/)
- [@feed@Explore top posts about Containers](https://app.daily.dev/tags/containers?ref=roadmapsh)

@ -1,6 +1,6 @@
# Data Ingestion Architectures
Data ingestion is the process of collecting, transferring, and loading data from various sources to a destination where it can be stored and analyzed. There are several data ingestion architectures that can be used to collect data from different sources and load it into a data warehouse, data lake, or other storage systems. These architectures can be broadly classified into two categories: batch processing and real-time processing. How you choose to ingest data will depend on the volume, velocity, and variety of data you are working with, as well as the latency requirements of your use case.
Data ingestion is the process of collecting, transferring, and loading data from various sources to a destination where it can be stored and analyzed. There are several data ingestion architectures that can be used to collect data from different sources and load it into a data warehouse, data lake, or other storage systems. These architectures can be broadly classified into two categories: batch processing and real-time processing. How you choose to ingest data will depend on the volume, velocity, and variety of data you are working with, as well as the latency requirements of your use case.
Lambda and Kappa architectures are two popular data ingestion architectures that combine batch and real-time processing to handle large volumes of data efficiently.

@ -1,9 +1,9 @@
# Data lakes & Warehouses
"**Data Lakes** are large-scale data repository systems that store raw, untransformed data, in various formats, from multiple sources. They're often used for big data and real-time analytics requirements. Data lakes preserve the original data format and schema which can be modified as necessary. On the other hand, **Data Warehouses** are data storage systems which are designed for analyzing, reporting and integrating with transactional systems. The data in a warehouse is clean, consistent, and often transformed to meet wide-range of business requirements. Hence, data warehouses provide structured data but require more processing and management compared to data lakes."
**Data Lakes** are large-scale data repository systems that store raw, untransformed data, in various formats, from multiple sources. They're often used for big data and real-time analytics requirements. Data lakes preserve the original data format and schema which can be modified as necessary. On the other hand, **Data Warehouses** are data storage systems which are designed for analyzing, reporting and integrating with transactional systems. The data in a warehouse is clean, consistent, and often transformed to meet wide-range of business requirements. Hence, data warehouses provide structured data but require more processing and management compared to data lakes.
Learn more from the following resources:
- [@article@Data lake definition](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-a-data-lake)
- [@video@What is a data lake?](https://www.youtube.com/watch?v=LxcH6z8TFpI)
- [@video@@hat is a data warehouse?](https://www.youtube.com/watch?v=k4tK2ttdSDg)
- [@article@Data Lake Definition](https://azure.microsoft.com/en-gb/resources/cloud-computing-dictionary/what-is-a-data-lake)
- [@video@What is a Data Lake?](https://www.youtube.com/watch?v=LxcH6z8TFpI)
- [@video@@hat is a Data Warehouse?](https://www.youtube.com/watch?v=k4tK2ttdSDg)

@ -5,4 +5,4 @@
Learn more from the following resources:
- [@article@What is Data Lineage?](https://www.ibm.com/topics/data-lineage)
- [@article@What is a feature store](https://www.snowflake.com/guides/what-feature-store-machine-learning/)
- [@article@What is a Feature Store](https://www.snowflake.com/guides/what-feature-store-machine-learning/)

@ -4,5 +4,5 @@ Data pipelines are a series of automated processes that transport and transform
Learn more from the following resources:
- [@article@What is a data pipeline?](https://www.ibm.com/topics/data-pipeline)
- [@video@What are data pipelines?](https://www.youtube.com/watch?v=oKixNpz6jNo)
- [@article@What is a Data Pipeline? - IBM](https://www.ibm.com/topics/data-pipeline)
- [@video@What are Data Pipelines?](https://www.youtube.com/watch?v=oKixNpz6jNo)

@ -4,7 +4,8 @@ Docker is a platform for working with containerized applications. Among its feat
Visit the following resources to learn more:
- [@article@Docker Documentation](https://docs.docker.com/)
- [@roadmap@Visit Dedicated Docker Roadmap](https://roadmap.sh/docker)
- [@official@Docker Documentation](https://docs.docker.com/)
- [@video@Docker Tutorial](https://www.youtube.com/watch?v=RqTEHSBrYFw)
- [@video@Docker simplified in 55 seconds](https://youtu.be/vP_4DlOH1G4)
- [@video@Docker Simplified in 55 Seconds](https://youtu.be/vP_4DlOH1G4)
- [@feed@Explore top posts about Docker](https://app.daily.dev/tags/docker?ref=roadmapsh)

@ -4,6 +4,6 @@ Apache Flink is an open-source stream processing framework designed for real-tim
Visit the following resources to learn more:
- [@article@Apache Flink Documentation](https://flink.apache.org/)
- [@official@Apache Flink Documentation](https://flink.apache.org/)
- [@article@Apache Flink](https://www.tutorialspoint.com/apache_flink/apache_flink_introduction.htm)
- [@feed@Explore top posts about Apache Flink](https://app.daily.dev/tags/apache-flink?ref=roadmapsh)
-[@reference@Apache Flink Tutorialpoint](https://www.tutorialspoint.com/apache_flink/apache_flink_introduction.htm)

@ -4,7 +4,7 @@ Git is a distributed version control system used to track changes in source code
Visit the following resources to learn more:
- [@roadmap@Learn Git & GitHub](https://roadmap.sh/git-github)
- [@roadmap@Visit Dedicated Git & GitHub Roadmap](https://roadmap.sh/git-github)
- [@video@Git & GitHub Crash Course For Beginners](https://www.youtube.com/watch?v=SWYqp7iY_Tc)
- [@article@Learn Git with Tutorials, News and Tips - Atlassian](https://www.atlassian.com/git)
- [@article@Git Cheat Sheet](https://cs.fyi/guide/git-cheatsheet)

@ -4,8 +4,8 @@ GitHub is a web-based platform built on top of Git that provides version control
Visit the following resources to learn more:
- [@roadmap@Learn Git & GitHub](https://roadmap.sh/git-github)
- [@official@GitHub Website](https://github.com)
- [@article@GitHub Documentation](https://docs.github.com/en/get-started/quickstart)
- [@roadmap@Visit Dedicated Git & GitHub Roadmap](https://roadmap.sh/git-github)
- [@official@GitHub](https://github.com)
- [@official@GitHub Documentation](https://docs.github.com/en/get-started/quickstart)
- [@video@What is GitHub?](https://www.youtube.com/watch?v=w3jLJU7DT5E)
- [@feed@Explore top posts about GitHub](https://app.daily.dev/tags/github?ref=roadmapsh)

@ -1,6 +1,7 @@
# Go
Go, also known as Golang, is an open-source programming language developed by Google that emphasizes simplicity, efficiency, and strong concurrency support. Designed for modern software development, Go features a clean syntax, garbage collection, and built-in support for concurrent programming through goroutines and channels, making it well-suited for building scalable, high-performance applications, especially in cloud computing and microservices architectures. Go's robust standard library and tooling ecosystem, including a powerful package manager and testing framework, further streamline development processes, promoting rapid application development and deployment.
Visit the following resources to learn more:
- [@roadmap@Visit Dedicated Go Roadmap](https://roadmap.sh/golang)

@ -2,6 +2,9 @@
Infrastructure as Code (IaC) is a modern approach to managing and provisioning IT infrastructure through machine-readable configuration files, rather than manual processes. It enables developers and operations teams to define and manage infrastructure resources—such as servers, networks, and databases—using code, which can be versioned, tested, and deployed like application code. IaC tools, like Terraform and AWS CloudFormation, allow for automated, repeatable deployments, reducing human error and increasing consistency across environments. This practice facilitates agile development, enhances collaboration between teams, and supports scalable and efficient infrastructure management.
Visit the following resources to learn more:
- [@roadmap@Visit Dedicated Terraform Roadmap](https://roadmap.sh/terraform)
- [@article@What is Infrastructure as Code?](https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac)
- [@video@Terraform course for beginners](https://www.youtube.com/watch?v=SLB_c_ayRMo)
- [@video@8 Terraform best practices](https://www.youtube.com/watch?v=gxPykhPxRW0)
- [@video@Terraform Course for Beginners](https://www.youtube.com/watch?v=SLB_c_ayRMo)
- [@video@8 Terraform Best Practices](https://www.youtube.com/watch?v=gxPykhPxRW0)

@ -4,6 +4,6 @@ Apache Kafka is an open-source distributed event streaming platform used by thou
Visit the following resources to learn more:
- [@article@Apache Kafka quickstart](https://kafka.apache.org/quickstart)
- [@official@Apache Kafka Quickstart](https://kafka.apache.org/quickstart)
- [@video@Apache Kafka Fundamentals](https://www.youtube.com/watch?v=B5j3uNBH8X4)
- [@feed@Explore top posts about Kafka](https://app.daily.dev/tags/kafka?ref=roadmapsh)

@ -4,8 +4,8 @@ Kubernetes is an open source container management platform, and the dominant pro
Visit the following resources to learn more:
- [@roadmap@Kubernetes Roadmap](https://roadmap.sh/kubernetes)
- [@official@Kubernetes Website](https://kubernetes.io/)
- [@roadmap@Visit Dedicated Kubernetes Roadmap](https://roadmap.sh/kubernetes)
- [@official@Kubernetes](https://kubernetes.io/)
- [@official@Kubernetes Documentation](https://kubernetes.io/docs/home/)
- [@video@Kubernetes Crash Course for Absolute Beginners](https://www.youtube.com/watch?v=s_o8dwzRlu4)
- [@article@Kubernetes: An Overview](https://thenewstack.io/kubernetes-an-overview/)

@ -1,6 +1,7 @@
# Machine Learning Fundamentals
Machine learning fundamentals encompass the key concepts and techniques that enable systems to learn from data and make predictions or decisions without being explicitly programmed. At its core, machine learning involves algorithms that can identify patterns in data and improve over time with experience. Key areas include supervised learning (where models are trained on labeled data), unsupervised learning (where models identify patterns in unlabeled data), and reinforcement learning (where agents learn to make decisions based on feedback from their actions). Essential components also include data preprocessing, feature selection, model training, evaluation metrics, and the importance of avoiding overfitting. Understanding these fundamentals is crucial for developing effective machine learning applications across various domains.
Learn more from the following resources:
- [@course@Fundamentals of Machine Learning - Microsoft](https://learn.microsoft.com/en-us/training/modules/fundamentals-machine-learning/)

@ -1,3 +1,8 @@
# MLOps Components
MLOps components can be broadly classified into three major categories: Development, Operations and Governance. The **Development** components include everything involved in the creation of machine learning models, such as data extraction, data analysis, feature engineering, and machine learning model training. The **Operations** category includes components involved in deploying, monitoring, and maintaining machine learning models in production. This may include release management, model serving, and performance monitoring. Lastly, the **Governance** category encompasses the policies and regulations related to machine learning models. This includes model audit and tracking, model explainability, and security & compliance regulations.
Learn more from the following resources:
- [@article@MLOps Workflow, Components, and Key Practices](https://mlops.tv/p/understanding-ml-pipelines-through)
- [@article@MLOps Lifecycle](https://www.moontechnolabs.com/blog/mlops-lifecycle/)

@ -16,4 +16,8 @@ MLOps (Machine Learning Operations) principles focus on streamlining the deploym
7. **Reproducibility**: Ensure that experiments can be reliably reproduced by standardizing environments and workflows, making it easier to validate and iterate on models.
These principles help organizations efficiently manage the lifecycle of machine learning models, from development to deployment and beyond.
These principles help organizations efficiently manage the lifecycle of machine learning models, from development to deployment and beyond.
Visit the following resources to learn more:
- [@article@MLOps Principles](https://ml-ops.org/content/mlops-principles)

@ -1,8 +1,10 @@
# Model Training and Serving
"Model Training" refers to the phase in the Machine Learning (ML) pipeline where we teach a machine learning model how to make predictions by providing it with data. This process begins with feeding the model a training dataset, which it uses to learn and understand patterns or perform computations. The model's performance is then evaluated by comparing its prediction outputs with the actual results. Various algorithms can be used in the model training process. The choice of algorithm usually depends on the task, the data available, and the requirements of the project. It is worth noting that the model training stage can be computationally expensive particularly when dealing with large datasets or complex models.
Model Training refers to the phase in the Machine Learning (ML) pipeline where we teach a machine learning model how to make predictions by providing it with data. This process begins with feeding the model a training dataset, which it uses to learn and understand patterns or perform computations. The model's performance is then evaluated by comparing its prediction outputs with the actual results. Various algorithms can be used in the model training process. The choice of algorithm usually depends on the task, the data available, and the requirements of the project. It is worth noting that the model training stage can be computationally expensive particularly when dealing with large datasets or complex models.
Decisions depend on the organization's infrastructure.
Visit the following resources to learn more:
- **Repository Suggestion:** [ML Deployment k8s Fast API](https://github.com/sayakpaul/ml-deployment-k8s-fastapi/tree/main)
- **Tutorial Suggestions:** [ML deployment with k8s FastAPI, Building an ML app with FastAPI](https://dev.to/bravinsimiyu/beginner-guide-on-how-to-build-a-machine-learning-app-with-fastapi-part-ii-deploying-the-fastapi-application-to-kubernetes-4j6g), [Basic Kubeflow pipeline](https://towardsdatascience.com/tutorial-basic-kubeflow-pipeline-from-scratch-5f0350dc1905), [Building and deploying ML pipelines](https://www.datacamp.com/tutorial/kubeflow-tutorial-building-and-deploying-machine-learning-pipelines?utm_source=google&utm_medium=paid_search&utm_campaignid=19589720818&utm_adgroupid=157156373991&utm_device=c&utm_keyword=&utm_matchtype=&utm_network=g&utm_adpostion=&utm_creative=683184494153&utm_targetid=dsa-2218886984380&utm_loc_interest_ms=&utm_loc_physical_ms=9064564&utm_content=&utm_campaign=230119_1-sea~dsa~tofu_2-b2c_3-eu_4-prc_5-na_6-na_7-le_8-pdsh-go_9-na_10-na_11-na-dec23&gad_source=1&gclid=Cj0KCQiA4Y-sBhC6ARIsAGXF1g7iSih9h2RGL27LwWY6dlPLhEss-e5Af8pnaBvdDynRh7IHIKi8sGgaApD-EALw_wcB), [KServe tutorial](https://towardsdatascience.com/kserve-highly-scalable-machine-learning-deployment-with-kubernetes-aa7af0b71202)
- [@article@MLOps Principles](https://ml-ops.org/content/mlops-principles)
- [@opensource@ML Deployment k8s Fast API](https://github.com/sayakpaul/ml-deployment-k8s-fastapi/)
- [@article@ML deployment with k8s FastAPI, Building an ML app with FastAPI](https://dev.to/bravinsimiyu/beginner-guide-on-how-to-build-a-machine-learning-app-with-fastapi-part-ii-deploying-the-fastapi-application-to-kubernetes-4j6g)
- [@article@KServe Tutorial](https://towardsdatascience.com/kserve-highly-scalable-machine-learning-deployment-with-kubernetes-aa7af0b71202)

@ -4,8 +4,8 @@ Python is an interpreted high-level general-purpose programming language. Its de
Learn more from the following resources:
- [@roadmap@Python Roadmap](https://roadmap.sh/python)
- [@official@Python.org](https://www.python.org/)
- [@roadmap@Visit Dedicated Python Roadmap](https://roadmap.sh/python)
- [@official@Python](https://www.python.org/)
- [@article@Real Python](https://realpython.com/)
- [@article@Automate the Boring Stuff with Python](https://automatetheboringstuff.com/)
- [@feed@Explore top posts about Python](https://app.daily.dev/tags/python?ref=roadmapsh)

@ -4,5 +4,6 @@ Version control/source control systems allow developers to track and control cha
Visit the following resources to learn more:
- [@article@Git](https://git-scm.com/)
- [@official@Git](https://git-scm.com/)
- [@article@What is Version Control?](https://www.atlassian.com/git/tutorials/what-is-version-control)
- [@feed@Explore top posts about Version Control](https://app.daily.dev/tags/version-control?ref=roadmapsh)

@ -4,6 +4,7 @@ Version control/source control systems allow developers to track and control cha
Visit the following resources to learn more:
- [@article@Git](https://git-scm.com/)
- [@official@Git](https://git-scm.com/)
- [@official@Git Documentation](https://git-scm.com/docs)
- [@article@What is Version Control?](https://www.atlassian.com/git/tutorials/what-is-version-control)
- [@feed@Explore top posts about Version Control](https://app.daily.dev/tags/version-control?ref=roadmapsh)

Loading…
Cancel
Save