Continuous Integration and Continuous Delivery (CI/CD) pipelines are essential components of modern software development, especially in the world of Kubernetes and containerized applications. To facilitate these pipelines, many organizations use Prow, a CI/CD system built specifically for Kubernetes. While Prow offers a rich set of features out of the box, you may need to develop your own plugins to tailor the system to your organization’s requirements. In this guide, we’ll explore the world of Prow plugin development and show you how to get started.
Prerequisites
Before diving into Prow plugin development, ensure you have the following prerequisites:
- Basic Knowledge of Kubernetes and CI/CD Concepts: Familiarity with Kubernetes concepts such as Pods, Deployments, and Services, as well as understanding CI/CD principles, will be beneficial for understanding Prow plugin development.
- Access to a Kubernetes Cluster: You’ll need access to a Kubernetes cluster for testing your plugins. If you don’t have one already, you can set up a local cluster using tools like Minikube or use a cloud provider’s managed Kubernetes service.
- Prow Setup: Install and configure Prow in your Kubernetes cluster. You can visit Velotio Technologies – Getting Started with Prow: A Kubernetes-Native CI/CD Framework
- Development Environment Setup: Ensure you have Git, Go, and Docker installed on your local machine for developing and testing Prow plugins. You’ll also need to configure your environment to interact with your organization’s Prow setup.
The Need for Custom Prow Plugins
While Prow provides a wide range of built-in plugins, your organization’s Kubernetes workflow may have specific requirements that aren’t covered by these defaults. This is where developing custom Prow plugins comes into play. Custom plugins allow you to extend Prow’s functionality to cater to your needs. Whether automating workflows, integrating with other tools, or enforcing custom policies, developing your own Prow plugins gives you the power to tailor your CI/CD pipeline precisely.
Getting Started with Prow Plugin Development
Developing a custom Prow plugin may seem daunting, but with the right approach and tools, it can be a rewarding experience. Here’s a step-by-step guide to get you started:
1. Set Up Your Development Environment
Before diving into plugin development, you need to set up your development environment. You will need Git, Go, and access to a Kubernetes cluster for testing your plugins. Ensure you have the necessary permissions to make changes to your organization’s Prow setup.
2. Choose a Plugin Type
Prow supports various plugin types, including postsubmits, presubmits, triggers, and utilities. Choose the type that best fits your use case.
- Postsubmits: These plugins are executed after the code is merged and are often used for tasks like publishing artifacts or creating release notes.
- Presubmits: Presubmit plugins run before code is merged, typically used for running tests and ensuring code quality.
- Triggers: Trigger plugins allow you to trigger custom jobs based on specific events or criteria.
- Utilities: Utility plugins offer reusable functions and utilities for other plugins.
3. Create Your Plugin
Once you’ve chosen a plugin type, it’s time to create it. Below is an example of a simple Prow plugin written in Go, named comment-plugin.go. It will create a comment on a pull request each time an event is received.
This code sets up a basic HTTP server that listens for GitHub events and handles them by creating a comment using the GitHub API. Customize this code to fit your specific use case.
package main
import (
"encoding/json"
"flag"
"net/http"
"os"
"strconv"
"time"
"github.com/sirupsen/logrus"
"k8s.io/test-infra/pkg/flagutil"
"k8s.io/test-infra/prow/config"
"k8s.io/test-infra/prow/config/secret"
prowflagutil "k8s.io/test-infra/prow/flagutil"
configflagutil "k8s.io/test-infra/prow/flagutil/config"
"k8s.io/test-infra/prow/github"
"k8s.io/test-infra/prow/interrupts"
"k8s.io/test-infra/prow/logrusutil"
"k8s.io/test-infra/prow/pjutil"
"k8s.io/test-infra/prow/pluginhelp"
"k8s.io/test-infra/prow/pluginhelp/externalplugins"
)
const pluginName = "comment-plugin"
type options struct {
port int
config configflagutil.ConfigOptions
dryRun bool
github prowflagutil.GitHubOptions
instrumentationOptions prowflagutil.InstrumentationOptions
webhookSecretFile string
}
type server struct {
tokenGenerator func() []byte
botUser *github.UserData
email string
ghc github.Client
log *logrus.Entry
repos []github.Repo
}
func helpProvider(_ []config.OrgRepo) (*pluginhelp.PluginHelp, error) {
pluginHelp := &pluginhelp.PluginHelp{
Description: `The sample plugin`,
}
return pluginHelp, nil
}
func (o *options) Validate() error {
return nil
}
func gatherOptions() options {
o := options{config: configflagutil.ConfigOptions{ConfigPath: "./config.yaml"}}
fs := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
fs.IntVar(&o.port, "port", 8888, "Port to listen on.")
fs.BoolVar(&o.dryRun, "dry-run", false, "Dry run for testing. Uses API tokens but does not mutate.")
fs.StringVar(&o.webhookSecretFile, "hmac-secret-file", "/etc/hmac", "Path to the file containing GitHub HMAC secret.")
for _, group := range []flagutil.OptionGroup{&o.github} {
group.AddFlags(fs)
}
fs.Parse(os.Args[1:])
return o
}
func main() {
o := gatherOptions()
if err := o.Validate(); err != nil {
logrus.Fatalf("Invalid options: %v", err)
}
logrusutil.ComponentInit()
log := logrus.StandardLogger().WithField("plugin", pluginName)
if err := secret.Add(o.webhookSecretFile); err != nil {
logrus.WithError(err).Fatal("Error starting secrets agent.")
}
gitHubClient, err := o.github.GitHubClient(o.dryRun)
if err != nil {
logrus.WithError(err).Fatal("Error getting GitHub client.")
}
email, err := gitHubClient.Email()
if err != nil {
log.WithError(err).Fatal("Error getting bot e-mail.")
}
botUser, err := gitHubClient.BotUser()
if err != nil {
logrus.WithError(err).Fatal("Error getting bot name.")
}
repos, err := gitHubClient.GetRepos(botUser.Login, true)
if err != nil {
log.WithError(err).Fatal("Error listing bot repositories.")
}
serv := &server{
tokenGenerator: secret.GetTokenGenerator(o.webhookSecretFile),
botUser: botUser,
email: email,
ghc: gitHubClient,
log: log,
repos: repos,
}
health := pjutil.NewHealthOnPort(o.instrumentationOptions.HealthPort)
health.ServeReady()
mux := http.NewServeMux()
mux.Handle("/", serv)
externalplugins.ServeExternalPluginHelp(mux, log, helpProvider)
logrus.Info("starting server " + strconv.Itoa(o.port))
httpServer := &http.Server{Addr: ":" + strconv.Itoa(o.port), Handler: mux}
defer interrupts.WaitForGracefulShutdown()
interrupts.ListenAndServe(httpServer, 5*time.Second)
}
func (s *server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
logrus.Info("inside http server")
_, _, payload, ok, _ := github.ValidateWebhook(w, r, s.tokenGenerator)
logrus.Info(string(payload))
if !ok {
return
}
logrus.Info(w, "Event received. Have a nice day.")
if err := s.handleEvent(payload); err != nil {
logrus.WithError(err).Error("Error parsing event.")
}
}
func (s *server) handleEvent(payload []byte) error {
logrus.Info("inside handler")
var pr github.PullRequestEvent
if err := json.Unmarshal(payload, &pr); err != nil {
return err
}
logrus.Info(pr.Number)
if err := s.ghc.CreateComment(pr.PullRequest.Base.Repo.Owner.Login, pr.PullRequest.Base.Repo.Name, pr.Number, "comment from smaple-plugin"); err != nil {
return err
}
return nil
}4. Deploy Your Plugin
To deploy your custom Prow plugin, you will need to create a Docker image and deploy it into your Prow cluster.
FROM golang as app-builder
WORKDIR /app
RUN apt update
RUN apt-get install git
COPY . .
RUN CGO_ENABLED=0 go build -o main
FROM alpine:3.9
RUN apk add ca-certificates git
COPY --from=app-builder /app/main /app/custom-plugin
ENTRYPOINT ["/app/custom-plugin"]docker build -t jainbhavya65/custom-plugin:v1 .
docker push jainbhavya65/custom-plugin:v1
Deploy the Docker image using Kubernetes deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: comment-plugin
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: comment-plugin
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: comment-plugin
spec:
containers:
- args:
- --github-token-path=/etc/github/oauth
- --hmac-secret-file=/etc/hmac-token/hmac
- --port=80
image: <IMAGE>
imagePullPolicy: Always
name: comment-plugin
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /etc/github
name: oauth
readOnly: true
- mountPath: /etc/hmac-token
name: hmac
readOnly: true
volumes:
- name: oauth
secret:
defaultMode: 420
secretName: oauth-token
- name: hmac
secret:
defaultMode: 420
secretName: hmac-tokenCreate a service for deployment:
apiVersion: v1
kind: Service
metadata:
name: comment-plugin
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: comment-plugin
sessionAffinity: None
type: ClusterIP
view rawAfter creating the deployment and service, integrate it into your organization’s Prow configuration. This involves updating your Prow plugin.yaml files to include your plugin and specify when it should run.
external_plugins:
- name: comment-plugin
# No endpoint specified implies "http://{{name}}". // as we deploy plugin into same cluster
# if plugin is not deployed in same cluster then you can give endpoint
events:
# only pull request and issue comment events are send to our plugin
- pull_request
- issue_comment
Conclusion
Mastering Prow plugin development opens up a world of possibilities for tailoring your Kubernetes CI/CD workflow to meet your organization’s needs. While the initial learning curve may be steep, the benefits of custom plugins in terms of automation, efficiency, and control are well worth the effort.
Remember that the key to successful Prow plugin development lies in clear documentation, thorough testing, and collaboration with your team to ensure that your custom plugins enhance your CI/CD pipeline’s functionality and reliability. As Kubernetes and containerized applications continue to evolve, Prow will remain a valuable tool for managing your CI/CD processes, and your custom plugins will be the secret sauce that sets your workflow apart from the rest.
