Pipeline Automation - Learn Data Science with Travis

Travis chat

Filters

Pipeline Automation

Pipeline automation refers to the process of automating the various stages of a software development pipeline, which includes building, testing, and deploying applications. This automation is crucial for enhancing efficiency, reducing human error, and speeding up the development lifecycle.

One of the key components of pipeline automation is the use of REST APIs. These APIs can significantly streamline processes by allowing different tools and services to communicate with each other seamlessly. For instance, in a CI/CD pipeline, REST APIs can automate tasks such as project creation in tools like SonarQube, enabling developers to focus on coding rather than manual setup tasks ².

Additionally, building an automated development pipeline involves several steps, including setting up servers, containerizing applications, and automating the build process. Utilizing technologies like Docker can simplify server configuration and enhance the portability of applications across different environments ⁴.

In summary, pipeline automation is essential for modern software development, leveraging tools and APIs to create efficient workflows.

How to use the magic of pipelines

Towards Data Science

Surely you have heard of pipelines or ETL (Extract Transform Load), or seen some method in a library, or even heard of any tool to create pipelines. However, you aren’t using it yet. So, let me…

REST APIs — The Silver Bullet in Pipeline Automation

Better Programming

As we know, there is no silver bullet in IT. Crowning REST APIs as the silver bullet in pipeline automation is only meant to emphasize the great potential REST APIs can bring for the whole automation…...

The Prefect Way to Automate & Orchestrate Data Pipelines

Towards Data Science

We used Apache Airflow to manage tasks on a data science project. But, with Prefect, you can manage tasks conveniently.

How to Build an Automated Development Pipeline

Towards Data Science

I have led many development teams in the past years. One thing that I found common among them is the need to build an automated development pipeline. You don’t necessarily need a sophisticated…

Building Data Pipelines Without a Single Line of Code

Towards Data Science

A post about the steps to create ETL data pipeline without writing a line of code using Google Cloud Dataprep and BigQuery.

Automating Machine Learning Pipelines

Towards Data Science

Creating a Machine Learning model is a difficult task because we need to write a lot of code to try different models and find out the performing model for that particular problem. There are different…...

ML Pipeline

Analytics Vidhya

In this post we will see what is pipeline, why it is essential and what are the versions of pipelines that are available. For any machine learning models it is necessary to maintain the workflow and…

Automating Data Pipelines with Python & GitHub Actions

Towards Data Science

A simple (and free) way to run data workflows This is the 4th article in a larger series on Full Stack Data Science (FSDS). In the last post, I shared a concrete example of how to build data pipeline...

A Brief Introduction to SciKit Pipelines

Towards Data Science

And why you should start using them. Photo by Sigmund on Unsplash Have you ever trained a machine learning model and your predictions looked too good to be true? But then you realized that you had so...

MLOps Level 0 — Manual Pipelines

Towards Data Science

In my previous article I went over the steps that make up a machine learning process (https://towardsdatascience.com/steps-of-a-machine-learning-process-7acc43973385). The level of automation of…

Clean Data Science workflow with Sklearn Pipeline

Analytics Vidhya

Pipelines are a container of steps, they are used to package workflow and fit a model into a single object. Pipelines are stacked on top of one another, taking input from one block sending output to…

Heron: A Hybrid Approach to Data Pipelines in Python

Python in Plain English

I am an experimenter and a data analyst of neuroscientific data. That means I have two disparate sets of needs when it comes to software tools and a very hard time finding the appropriate tooling to…