騰訊云：一文讀懂Kubernetes APIServer原理（下）

·APIExtensionServer負(fù)責(zé)CustomResourceDefinition（CRD）apiResources以及apiVersions的注冊(cè)，同時(shí)處理CRD以及相應(yīng)CustomResource（CR）的REST請(qǐng)求(如果對(duì)應(yīng)CR不能被處理的話(huà)則會(huì)返回404)，也是apiserver Delegation的最后一環(huán)

·crdRegistrationController負(fù)責(zé)將CRD GroupVersions自動(dòng)注冊(cè)到APIServices中。具體邏輯為：枚舉所有CRDs，然后根據(jù)CRD定義的crd.Spec.Group以及crd.Spec.Versions字段構(gòu)建APIService，并添加到autoRegisterController.apiServicesToSync中，由autoRegisterController進(jìn)行創(chuàng)建以及維護(hù)操作。這也是為什么創(chuàng)建完CRD后會(huì)產(chǎn)生對(duì)應(yīng)的APIService對(duì)象

·APIExtensionServer包含的controller以及功能如下所示：

openapiController：將crd資源的變化同步至提供的OpenAPI文檔，可通過(guò)訪(fǎng)問(wèn)/openapi/v2進(jìn)行查看；

crdController：負(fù)責(zé)將crd信息注冊(cè)到apiVersions和apiResources中，兩者的信息可通過(guò)kubectl api-versions和kubectl api-resources查看；

kubectl api-versions命令返回所有Kubernetes集群資源的版本信息（實(shí)際發(fā)出了兩個(gè)請(qǐng)求，分別是https://127.0.0.1:6443/api以及https://127.0.0.1:6443/apis，并在最后將兩個(gè)請(qǐng)求的返回結(jié)果進(jìn)行了合并）

$ kubectl -v=8 api-versions 

I1211 11:44:50.276446   22493 loader.go:375] Config loaded from file:  /root/.kube/config

I1211 11:44:50.277005   22493 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s

...

I1211 11:44:50.290265   22493 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}

I1211 11:44:50.293673   22493 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s

...

I1211 11:44:50.298360   22493 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]

apiextensions.k8s.io/v1

apiextensions.k8s.io/v1beta1

apiregistration.k8s.io/v1

apiregistration.k8s.io/v1beta1

apps/v1

authentication.k8s.io/v1beta1

...

storage.k8s.io/v1

storage.k8s.io/v1beta1

v1

kubectl api-resources命令就是先獲取所有API版本信息，然后對(duì)每一個(gè)API版本調(diào)用接口獲取該版本下的所有API資源類(lèi)型

$ kubectl -v=8 api-resources

 5077 loader.go:375] Config loaded from file:  /root/.kube/config

 I1211 15:19:47.593450   15077 round_trippers.go:420] GET https://127.0.0.1:6443/api?timeout=32s

 I1211 15:19:47.602273   15077 request.go:1068] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"x.x.x.x:6443"}]}

 I1211 15:19:47.606279   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis?timeout=32s

 I1211 15:19:47.610333   15077 request.go:1068] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1"," [truncated 4985 chars]

 I1211 15:19:47.614700   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s

 I1211 15:19:47.614804   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s

 I1211 15:19:47.615687   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/auth.tkestack.io/v1?timeout=32s

 https://127.0.0.1:6443/apis/authentication.k8s.io/v1beta1?timeout=32s

 I1211 15:19:47.616794   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s

 I1211 15:19:47.616863   15077 round_trippers.go:420] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s

 ...

 NAME                              SHORTNAMES   APIGROUP                  NAMESPACED       KIND

 bindings                                                                                         true                       Binding

 endpoints                         ep                                                          true                      Endpoints

 events                               ev                                                          true                      Event

 limitranges                       limits                                                      true                      LimitRange

 namespaces                     ns                                                           false                     Namespace

 nodes                               no                                                           false                    Node

 ...

namingController：檢查crd obj中是否有命名沖突，可在crd.status.conditions中查看；

establishingController：檢查crd是否處于正常狀態(tài)，可在crd.status.conditions中查看；

nonStructuralSchemaController：檢查crd obj結(jié)構(gòu)是否正常，可在crd.status.conditions中查看；

apiApprovalController：檢查crd是否遵循Kubernetes API聲明策略，可在crd.status.conditions中查看；

finalizingController：類(lèi)似于finalizes的功能，與CRs的刪除有關(guān)；

總結(jié)CR CRUD APIServer處理邏輯如下：

// New returns a new instance of CustomResourceDefinitions from the given config.

func (c completedConfig) New(delegationTarget genericapiserver.DelegationTarget) (*CustomResourceDefinitions, error) {

  ...

    crdHandler, err := NewCustomResourceDefinitionHandler(

      versionDiscoveryHandler,

        groupDiscoveryHandler,

      s.Informers.Apiextensions().V1().CustomResourceDefinitions(),

        delegateHandler,

      c.ExtraConfig.CRDRESTOptionsGetter,

        c.GenericConfig.AdmissionControl,

      establishingController,

        c.ExtraConfig.ServiceResolver,

      c.ExtraConfig.AuthResolverWrapper,

        c.ExtraConfig.MasterCount,

        s.GenericAPIServer.Authorizer,

        c.GenericConfig.RequestTimeout,

        time.Duration(c.GenericConfig.MinRequestTimeout)*time.Second,

        apiGroupInfo.StaticOpenAPISpec,

        c.GenericConfig.MaxRequestBodyBytes,

    )

    if err != nil {

        return nil, err

    }

    s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", crdHandler)

    s.GenericAPIServer.Handler.NonGoRestfulMux.HandlePrefix("/apis/", crdHandler)

    ...

    return s, nil

}

·crdHandler處理邏輯如下：

解析req(GET/apis/duyanghao.example.com/v1/namespaces/default/students)，根據(jù)請(qǐng)求路徑中的group(duyanghao.example.com)，version(v1)，以及resource字段(students)獲取對(duì)應(yīng)CRD內(nèi)容(crd,err:=r.crdLister.Get(crdName))

通過(guò)crd.UID以及crd.Name獲取crdInfo，若不存在則創(chuàng)建對(duì)應(yīng)的crdInfo(crdInfo,err:=r.getOrCreateServingInfoFor(crd.UID,crd.Name))。crdInfo中包含了CRD定義以及該CRD對(duì)應(yīng)Custom Resource的customresource.REST storage

customresource.REST storage由CR對(duì)應(yīng)的Group(duyanghao.example.com)，Version(v1)，Kind(Student)，Resource(students)等創(chuàng)建完成，由于CR在Kubernetes代碼中并沒(méi)有具體結(jié)構(gòu)體定義，所以這里會(huì)先初始化一個(gè)范型結(jié)構(gòu)體Unstructured(用于保存所有類(lèi)型的Custom Resource)，并對(duì)該結(jié)構(gòu)體進(jìn)行SetGroupVersionKind操作(設(shè)置具體Custom Resource Type)

從customresource.REST storage獲取Unstructured結(jié)構(gòu)體后會(huì)對(duì)其進(jìn)行相應(yīng)轉(zhuǎn)換然后返回

// k8s.io/kubernetes/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:223

func (r *crdHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {

  ctx := req.Context()

  requestInfo, ok := apirequest.RequestInfoFrom(ctx)

  ...

  crdName := requestInfo.Resource + "." + requestInfo.APIGroup

  crd, err := r.crdLister.Get(crdName)

  ...

  crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name)

  verb := strings.ToUpper(requestInfo.Verb)

  resource := requestInfo.Resource

  subresource := requestInfo.Subresource

  scope := metrics.CleanScope(requestInfo)

  ...

  switch {

  case subresource == "status" && subresources != nil && subresources.Status != nil:

      handlerFunc = r.serveStatus(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  case subresource == "scale" && subresources != nil && subresources.Scale != nil:

      handlerFunc = r.serveScale(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  case len(subresource) == 0:

      handlerFunc = r.serveResource(w, req, requestInfo, crdInfo, terminating, supportedTypes)

  default:

      responsewriters.ErrorNegotiated(

          apierrors.NewNotFound(schema.GroupResource{Group: requestInfo.APIGroup, Resource: requestInfo.Resource}, requestInfo.Name),

          Codecs, schema.GroupVersion{Group: requestInfo.APIGroup, Version: requestInfo.APIVersion}, w, req,

      )

  }

  if handlerFunc != nil {

      handlerFunc = metrics.InstrumentHandlerFunc(verb, requestInfo.APIGroup, requestInfo.APIVersion, resource, subresource, scope, metrics.APIServerComponent, handlerFunc)

      handler := genericfilters.WithWaitGroup(handlerFunc, longRunningFilter, crdInfo.waitGroup)

      handler.ServeHTTP(w, req)

      return

  }

}

更多代碼原理詳情，參考kubernetes-reading-notes[1]。

Conclusion

本文從源碼層面對(duì)Kubernetes apiserver進(jìn)行了一個(gè)概覽性總結(jié)，包括：aggregatorServer，kubeAPIServer，apiExtensionsServer以及bootstrap-controller等。通過(guò)閱讀本文可以對(duì)apiserver內(nèi)部原理有一個(gè)大致的理解，另外也有助于后續(xù)深入研究。

參考資料

[1]kubernetes-reading-notes:https://github.com/duyanghao/kubernetes-reading-notes/tree/master/core/api-ser