Introduction:
Custom resources definition (CRD) is a powerful feature introduced in Kubernetes 1.7 which enables users to add their own/custom objects to the Kubernetes cluster and use it like any other native Kubernetes objects. In this blog post, we will see how we can add a custom resource to a Kubernetes cluster using the command line as well as using the Golang client library thus also learning how to programmatically interact with a Kubernetes cluster.
What is a Custom Resource Definition (CRD)?
In the Kubernetes API, every resource is an endpoint to store API objects of certain kind. For example, the built-in service resource contains a collection of service objects. The standard Kubernetes distribution ships with many inbuilt API objects/resources. CRD comes into picture when we want to introduce our own object into the Kubernetes cluster to full fill our requirements. Once we create a CRD in Kubernetes we can use it like any other native Kubernetes object thus leveraging all the features of Kubernetes like its CLI, security, API services, RBAC etc.
The custom resource created is also stored in the etcd cluster with proper replication and lifecycle management. CRD allows us to use all the functionalities provided by a Kubernetes cluster for our custom objects and saves us the overhead of implementing them on our own.
How to register a CRD using command line interface (CLI)
Step-1: Create a CRD definition file sslconfig-crd.yaml
apiVersion: "apiextensions.k8s.io/v1beta1"
kind: "CustomResourceDefinition"
metadata:
name: "sslconfigs.blog.velotio.com"
spec:
group: "blog.velotio.com"
version: "v1alpha1"
scope: "Namespaced"
names:
plural: "sslconfigs"
singular: "sslconfig"
kind: "SslConfig"
validation:
openAPIV3Schema:
required: ["spec"]
properties:
spec:
required: ["cert","key","domain"]
properties:
cert:
type: "string"
minimum: 1
key:
type: "string"
minimum: 1
domain:
type: "string"
minimum: 1 Here we are creating a custom resource definition for an object of kind SslConfig. This object allows us to store the SSL configuration information for a domain. As we can see under the validation section specifying the cert, key and the domain are mandatory for creating objects of this kind, along with this we can store other information like the provider of the certificate etc. The name metadata that we specify must be spec.names.plural+”.”+spec.group.
An API group (blog.velotio.com here) is a collection of API objects which are logically related to each other. We have also specified version for our custom objects (spec.version), as the definition of the object is expected to change/evolve in future so it’s better to start with alpha so that the users of the object knows that the definition might change later. In the scope, we have specified Namespaced, by default a custom resource name is clustered scoped.
# kubectl create -f crd.yaml
# kubectl get crd NAME AGE sslconfigs.blog.velotio.com 5sStep-2: Create objects using the definition we created above
apiVersion: "blog.velotio.com/v1alpha1"
kind: "SslConfig"
metadata:
name: "sslconfig-velotio.com"
spec:
cert: "my cert file"
key : "my private key"
domain: "*.velotio.com"
provider: "digicert"# kubectl create -f crd-obj.yaml
# kubectl get sslconfig NAME AGE sslconfig-velotio.com 12sAlong with the mandatory fields cert, key and domain, we have also stored the information of the provider ( certifying authority ) of the cert.
How to register a CRD programmatically using client-go
Client-go project provides us with packages using which we can easily create go client and access the Kubernetes cluster. For creating a client first we need to create a connection with the API server.
How we connect to the API server depends on whether we will be accessing it from within the cluster (our code running in the Kubernetes cluster itself) or if our code is running outside the cluster (locally)
If the code is running outside the cluster then we need to provide either the path of the config file or URL of the Kubernetes proxy server running on the cluster.
kubeconfig := filepath.Join(
os.Getenv("HOME"), ".kube", "config",
)
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
log.Fatal(err)
}OR
var (
// Set during build
version string
proxyURL = flag.String("proxy", "",
`If specified, it is assumed that a kubctl proxy server is running on the
given url and creates a proxy client. In case it is not given InCluster
kubernetes setup will be used`)
)
if *proxyURL != "" {
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
&clientcmd.ClientConfigLoadingRules{},
&clientcmd.ConfigOverrides{
ClusterInfo: clientcmdapi.Cluster{
Server: *proxyURL,
},
}).ClientConfig()
if err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}When the code is to be run as a part of the cluster then we can simply use
import "k8s.io/client-go/rest" ... rest.InClusterConfig() Once the connection is established we can use it to create clientset. For accessing Kubernetes objects, generally the clientset from the client-go project is used, but for CRD related operations we need to use the clientset from apiextensions-apiserver project.
apiextension “k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset”
kubeClient, err := apiextension.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v.", err)
}Now we can use the client to make the API call which will create the CRD for us.
package v1alpha1
import (
"reflect"
apiextensionv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
apierrors "k8s.io/apimachinery/pkg/api/errors"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
CRDPlural string = "sslconfigs"
CRDGroup string = "blog.velotio.com"
CRDVersion string = "v1alpha1"
FullCRDName string = CRDPlural + "." + CRDGroup
)
func CreateCRD(clientset apiextension.Interface) error {
crd := &apiextensionv1beta1.CustomResourceDefinition{
ObjectMeta: meta_v1.ObjectMeta{Name: FullCRDName},
Spec: apiextensionv1beta1.CustomResourceDefinitionSpec{
Group: CRDGroup,
Version: CRDVersion,
Scope: apiextensionv1beta1.NamespaceScoped,
Names: apiextensionv1beta1.CustomResourceDefinitionNames{
Plural: CRDPlural,
Kind: reflect.TypeOf(SslConfig{}).Name(),
},
},
}
_, err := clientset.ApiextensionsV1beta1().CustomResourceDefinitions().Create(crd)
if err != nil && apierrors.IsAlreadyExists(err) {
return nil
}
return err
}In the create CRD function, we first create the definition of our custom object and then pass it to the create method which creates it in our cluster. Just like we did while creating our definition using CLI, here also we set the parameters like version, group, kind etc.
Once our definition is ready we can create objects of its type just like we did earlier using the CLI. First we need to define our object.
package v1alpha1
import meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
type SslConfig struct {
meta_v1.TypeMeta `json:",inline"`
meta_v1.ObjectMeta `json:"metadata"`
Spec SslConfigSpec `json:"spec"`
Status SslConfigStatus `json:"status,omitempty"`
}
type SslConfigSpec struct {
Cert string `json:"cert"`
Key string `json:"key"`
Domain string `json:"domain"`
}
type SslConfigStatus struct {
State string `json:"state,omitempty"`
Message string `json:"message,omitempty"`
}
type SslConfigList struct {
meta_v1.TypeMeta `json:",inline"`
meta_v1.ListMeta `json:"metadata"`
Items []SslConfig `json:"items"`
}Kubernetes API conventions suggests that each object must have two nested object fields that govern the object’s configuration: the object spec and the object status. Objects must also have metadata associated with them. The custom objects that we define here comply with these standards. It is also recommended to create a list type for every type thus we have also created a SslConfigList struct.
Now we need to write a function which will create a custom client which is aware of the new resource that we have created.
package v1alpha1
import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/runtime/serializer"
"k8s.io/client-go/rest"
)
var SchemeGroupVersion = schema.GroupVersion{Group: CRDGroup, Version: CRDVersion}
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&SslConfig{},
&SslConfigList{},
)
meta_v1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}
func NewClient(cfg *rest.Config) (*SslConfigV1Alpha1Client, error) {
scheme := runtime.NewScheme()
SchemeBuilder := runtime.NewSchemeBuilder(addKnownTypes)
if err := SchemeBuilder.AddToScheme(scheme); err != nil {
return nil, err
}
config := *cfg
config.GroupVersion = &SchemeGroupVersion
config.APIPath = "/apis"
config.ContentType = runtime.ContentTypeJSON
config.NegotiatedSerializer = serializer.DirectCodecFactory{CodecFactory: serializer.NewCodecFactory(scheme)}
client, err := rest.RESTClientFor(&config)
if err != nil {
return nil, err
}
return &SslConfigV1Alpha1Client{restClient: client}, nil
}Building the custom client library
Once we have registered our custom resource definition with the Kubernetes cluster we can create objects of its type using the Kubernetes cli as we did earlier but for creating controllers for these objects or for developing some custom functionalities around them we need to build a client library also using which we can access them from go API. For native Kubernetes objects, this type of library is provided for each object.
package v1alpha1
import (
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
)
func (c *SslConfigV1Alpha1Client) SslConfigs(namespace string) SslConfigInterface {
return &sslConfigclient{
client: c.restClient,
ns: namespace,
}
}
type SslConfigV1Alpha1Client struct {
restClient rest.Interface
}
type SslConfigInterface interface {
Create(obj *SslConfig) (*SslConfig, error)
Update(obj *SslConfig) (*SslConfig, error)
Delete(name string, options *meta_v1.DeleteOptions) error
Get(name string) (*SslConfig, error)
}
type sslConfigclient struct {
client rest.Interface
ns string
}
func (c *sslConfigclient) Create(obj *SslConfig) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Post().
Namespace(c.ns).Resource("sslconfigs").
Body(obj).Do().Into(result)
return result, err
}
func (c *sslConfigclient) Update(obj *SslConfig) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Put().
Namespace(c.ns).Resource("sslconfigs").
Body(obj).Do().Into(result)
return result, err
}
func (c *sslConfigclient) Delete(name string, options *meta_v1.DeleteOptions) error {
return c.client.Delete().
Namespace(c.ns).Resource("sslconfigs").
Name(name).Body(options).Do().
Error()
}
func (c *sslConfigclient) Get(name string) (*SslConfig, error) {
result := &SslConfig{}
err := c.client.Get().
Namespace(c.ns).Resource("sslconfigs").
Name(name).Do().Into(result)
return result, err
}We can add more methods like watch, update status etc. Their implementation will also be similar to the methods we have defined above. For looking at the methods available for various Kubernetes objects like pod, node etc. we can refer to the v1 package.
Putting all things together
Now in our main function we will get all the things together.
package main
import (
"flag"
"fmt"
"time"
"blog.velotio.com/crd-blog/v1alpha1"
"github.com/golang/glog"
apiextension "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
clientcmdapi "k8s.io/client-go/tools/clientcmd/api"
)
var (
// Set during build
version string
proxyURL = flag.String("proxy", "",
`If specified, it is assumed that a kubctl proxy server is running on the
given url and creates a proxy client. In case it is not given InCluster
kubernetes setup will be used`)
)
func main() {
flag.Parse()
var err error
var config *rest.Config
if *proxyURL != "" {
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
&clientcmd.ClientConfigLoadingRules{},
&clientcmd.ConfigOverrides{
ClusterInfo: clientcmdapi.Cluster{
Server: *proxyURL,
},
}).ClientConfig()
if err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
} else {
if config, err = rest.InClusterConfig(); err != nil {
glog.Fatalf("error creating client configuration: %v", err)
}
}
kubeClient, err := apiextension.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v", err)
}
// Create the CRD
err = v1alpha1.CreateCRD(kubeClient)
if err != nil {
glog.Fatalf("Failed to create crd: %v", err)
}
// Wait for the CRD to be created before we use it.
time.Sleep(5 * time.Second)
// Create a new clientset which include our CRD schema
crdclient, err := v1alpha1.NewClient(config)
if err != nil {
panic(err)
}
// Create a new SslConfig object
SslConfig := &v1alpha1.SslConfig{
ObjectMeta: meta_v1.ObjectMeta{
Name: "sslconfigobj",
Labels: map[string]string{"mylabel": "test"},
},
Spec: v1alpha1.SslConfigSpec{
Cert: "my-cert",
Key: "my-key",
Domain: "*.velotio.com",
},
Status: v1alpha1.SslConfigStatus{
State: "created",
Message: "Created, not processed yet",
},
}
// Create the SslConfig object we create above in the k8s cluster
resp, err := crdclient.SslConfigs("default").Create(SslConfig)
if err != nil {
fmt.Printf("error while creating object: %vn", err)
} else {
fmt.Printf("object created: %vn", resp)
}
obj, err := crdclient.SslConfigs("default").Get(SslConfig.ObjectMeta.Name)
if err != nil {
glog.Infof("error while getting the object %vn", err)
}
fmt.Printf("SslConfig Objects Found: n%vn", obj)
select {}
}Now if we run our code then our custom resource definition will get created in the Kubernetes cluster and also an object of its type will be there just like with the cli. The docker image akash125/crdblog is build using the code discussed above it can be directly pulled from docker hub and run in a Kubernetes cluster. After the image is run successfully, the CRD definition that we discussed above will get created in the cluster along with an object of its type. We can verify the same using the CLI the way we did earlier, we can also check the logs of the pod running the docker image to verify it. The complete code is available here.
Conclusion and future work
We learned how to create a custom resource definition and objects using Kubernetes command line interface as well as the Golang client. We also learned how to programmatically access a Kubernetes cluster, using which we can build some really cool stuff on Kubernetes, we can now also create custom controllers for our resources which continuously watches the cluster for various life cycle events of our object and takes desired action accordingly. To read more about CRD refer the following links:
- https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
- https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
- https://blog.openshift.com/kubernetes-custom-resources-grow-up-in-v1-10/
- https://medium.com/@trstringer/create-kubernetes-controllers-for-core-and-custom-resources-62fc35ad64a3
