您好,登錄后才能下訂單哦!
本篇內容介紹了“Kubernetes StatefulSet源碼是什么”的有關知識,在實際案例的操作過程中,不少人都會遇到這樣的困境,接下來就讓小編帶領大家學習一下如何處理這些情況吧!希望大家仔細閱讀,能夠學有所成!
下面是簡單的StatefulSet Controller工作的內部結構圖。
同其他Controller一樣,StatefulSet Controller也是由ControllerManager初始化時負責啟動。
// NewStatefulSetController creates a new statefulset controller. func NewStatefulSetController( podInformer coreinformers.PodInformer, setInformer appsinformers.StatefulSetInformer, pvcInformer coreinformers.PersistentVolumeClaimInformer, revInformer appsinformers.ControllerRevisionInformer, kubeClient clientset.Interface, ) *StatefulSetController { ... ssc := &StatefulSetController{ kubeClient: kubeClient, control: NewDefaultStatefulSetControl( NewRealStatefulPodControl( kubeClient, setInformer.Lister(), podInformer.Lister(), pvcInformer.Lister(), recorder), NewRealStatefulSetStatusUpdater(kubeClient, setInformer.Lister()), history.NewHistory(kubeClient, revInformer.Lister()), ), pvcListerSynced: pvcInformer.Informer().HasSynced, queue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "statefulset"), podControl: controller.RealPodControl{KubeClient: kubeClient, Recorder: recorder}, revListerSynced: revInformer.Informer().HasSynced, } podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ // lookup the statefulset and enqueue AddFunc: ssc.addPod, // lookup current and old statefulset if labels changed UpdateFunc: ssc.updatePod, // lookup statefulset accounting for deletion tombstones DeleteFunc: ssc.deletePod, }) ssc.podLister = podInformer.Lister() ssc.podListerSynced = podInformer.Informer().HasSynced setInformer.Informer().AddEventHandlerWithResyncPeriod( cache.ResourceEventHandlerFuncs{ AddFunc: ssc.enqueueStatefulSet, UpdateFunc: func(old, cur interface{}) { oldPS := old.(*apps.StatefulSet) curPS := cur.(*apps.StatefulSet) if oldPS.Status.Replicas != curPS.Status.Replicas { glog.V(4).Infof("Observed updated replica count for StatefulSet: %v, %d->%d", curPS.Name, oldPS.Status.Replicas, curPS.Status.Replicas) } ssc.enqueueStatefulSet(cur) }, DeleteFunc: ssc.enqueueStatefulSet, }, statefulSetResyncPeriod, ) ssc.setLister = setInformer.Lister() ssc.setListerSynced = setInformer.Informer().HasSynced // TODO: Watch volumes return ssc }
很熟悉的代碼風格,也是創建對應的eventBroadcaster,然后給對應的objectInformer注冊對應的eventHandler:
StatefulSetController主要ListWatch Pod和StatefulSet對象;
Pod Informer注冊了add/update/delete EventHandler,這三個EventHandler都會將Pod對應的StatefulSet加入到StatefulSet Queue中。
StatefulSet Informer同樣注冊了add/update/event EventHandler,也都會將StatefulSet加入到StatefulSet Queue中。
目前StatefulSetController還未感知PVC Informer的EventHandler,這里繼續按照PVC Controller全部處理。在StatefulSet Controller創建和刪除Pod時,會調用apiserver創建和刪除對應的PVC。
RevisionController類似,在StatefulSet Controller Reconcile時會創建或者刪除對應的Revision。
接下來,會進入StatefulSetController的worker(只有一個worker,也就是只一個go routine),worker會從StatefulSet Queue中pop out一個StatefulSet對象,然后執行sync進行Reconcile操作。
// sync syncs the given statefulset. func (ssc *StatefulSetController) sync(key string) error { startTime := time.Now() defer func() { glog.V(4).Infof("Finished syncing statefulset %q (%v)", key, time.Now().Sub(startTime)) }() namespace, name, err := cache.SplitMetaNamespaceKey(key) if err != nil { return err } set, err := ssc.setLister.StatefulSets(namespace).Get(name) if errors.IsNotFound(err) { glog.Infof("StatefulSet has been deleted %v", key) return nil } if err != nil { utilruntime.HandleError(fmt.Errorf("unable to retrieve StatefulSet %v from store: %v", key, err)) return err } selector, err := metav1.LabelSelectorAsSelector(set.Spec.Selector) if err != nil { utilruntime.HandleError(fmt.Errorf("error converting StatefulSet %v selector: %v", key, err)) // This is a non-transient error, so don't retry. return nil } if err := ssc.adoptOrphanRevisions(set); err != nil { return err } pods, err := ssc.getPodsForStatefulSet(set, selector) if err != nil { return err } return ssc.syncStatefulSet(set, pods) }
sync中根據setLabel匹配出所有revisions、然后檢查這些revisions中是否有OwnerReference為空的,如果有,那說明存在Orphaned的Revisions。
注意:只要檢查到有一個History Revision就會觸發給所有的Resivions打上Patch:
{"metadata":{"ownerReferences":[{"apiVersion":"%s","kind":"%s","name":"%s","uid":"%s","controller":true,"blockOwnerDeletion":true}],"uid":"%s"}}
調用getPodsForStatefulSet獲取這個StatefulSet應該管理的Pods。
獲取該StatefulSet對應Namesapce下所有的Pods;
執行ClaimPods操作:檢查set和pod的Label是否匹配上,如果Label不匹配,那么需要release這個Pod,然后檢查pod的name和StatefulSet name的格式是否能匹配上。對于都匹配上的,并且ControllerRef UID也相同的,則不需要處理。
如果Selector和ControllerRef都匹配不上,則執行ReleasePod操作,給Pod打Patch: {“metadata":{"ownerReferences":[{"$patch":"delete","uid":"%s"}],"uid":"%s"}}
對于Label和name格式能匹配上的,但是controllerRef為空的Pods,就執行AdoptPod,給Pod打上Patch: {“metadata":{"ownerReferences":[{"apiVersion":"%s","kind":"%s","name":"%s","uid":"%s","controller":true,"blockOwnerDeletion":true}],"uid":"%s"}}
syncStatefulSet的實現只是調用UpdateStatefulSet。
func (ssc *defaultStatefulSetControl) UpdateStatefulSet(set *apps.StatefulSet, pods []*v1.Pod) error { // list all revisions and sort them revisions, err := ssc.ListRevisions(set) if err != nil { return err } history.SortControllerRevisions(revisions) // get the current, and update revisions currentRevision, updateRevision, collisionCount, err := ssc.getStatefulSetRevisions(set, revisions) if err != nil { return err } // perform the main update function and get the status status, err := ssc.updateStatefulSet(set, currentRevision, updateRevision, collisionCount, pods) if err != nil { return err } // update the set's status err = ssc.updateStatefulSetStatus(set, status) if err != nil { return err } glog.V(4).Infof("StatefulSet %s/%s pod status replicas=%d ready=%d current=%d updated=%d", set.Namespace, set.Name, status.Replicas, status.ReadyReplicas, status.CurrentReplicas, status.UpdatedReplicas) glog.V(4).Infof("StatefulSet %s/%s revisions current=%s update=%s", set.Namespace, set.Name, status.CurrentRevision, status.UpdateRevision) // maintain the set's revision history limit return ssc.truncateHistory(set, pods, revisions, currentRevision, updateRevision) }
UpdateStatefulSet主要流程為:
ListRevisions獲取該StatefulSet的所有Revisions,并按照Revision從小到大進行排序。
getStatefulSetRevisions獲取currentRevison和UpdateRevision。
只有當RollingUpdate策略時Partition不為0時,才會有部分Pods是updateRevision。
其他情況,所有Pods都得維持currentRevision。
updateStatefulSet是StatefulSet Controller的核心邏輯,負責創建、更新、刪除Pods,使得聲明式target得以維護:
使得target state始終有Spec.Replicas個Running And Ready的Pods。
如果更新策略是RollingUpdate,并且Partition為0,則保證所有Pods都對應Status.CurrentRevision。
如果更新策略是RollingUpdate,并且Partition不為0,則ordinal小于Partition的Pods保持Status.CurrentRevision,而ordinal大于等于Partition的Pods更新到Status.UpdateRevision。
如果更新策略是OnDelete,則只有刪除Pods時才會觸發對應Pods的更新,也就是說與Revisions不關聯。
truncateHistory維護History Revision個數不超過.Spec.RevisionHistoryLimit
。
updateStatefulSet是整個StatefulSetController的核心。
func (ssc *defaultStatefulSetControl) updateStatefulSet( set *apps.StatefulSet, currentRevision *apps.ControllerRevision, updateRevision *apps.ControllerRevision, collisionCount int32, pods []*v1.Pod) (*apps.StatefulSetStatus, error) { // get the current and update revisions of the set. currentSet, err := ApplyRevision(set, currentRevision) if err != nil { return nil, err } updateSet, err := ApplyRevision(set, updateRevision) if err != nil { return nil, err } // set the generation, and revisions in the returned status status := apps.StatefulSetStatus{} status.ObservedGeneration = new(int64) *status.ObservedGeneration = set.Generation status.CurrentRevision = currentRevision.Name status.UpdateRevision = updateRevision.Name status.CollisionCount = new(int32) *status.CollisionCount = collisionCount replicaCount := int(*set.Spec.Replicas) // slice that will contain all Pods such that 0 <= getOrdinal(pod) < set.Spec.Replicas replicas := make([]*v1.Pod, replicaCount) // slice that will contain all Pods such that set.Spec.Replicas <= getOrdinal(pod) condemned := make([]*v1.Pod, 0, len(pods)) unhealthy := 0 firstUnhealthyOrdinal := math.MaxInt32 var firstUnhealthyPod *v1.Pod // First we partition pods into two lists valid replicas and condemned Pods for i := range pods { status.Replicas++ // count the number of running and ready replicas if isRunningAndReady(pods[i]) { status.ReadyReplicas++ } // count the number of current and update replicas if isCreated(pods[i]) && !isTerminating(pods[i]) { if getPodRevision(pods[i]) == currentRevision.Name { status.CurrentReplicas++ } else if getPodRevision(pods[i]) == updateRevision.Name { status.UpdatedReplicas++ } } if ord := getOrdinal(pods[i]); 0 <= ord && ord < replicaCount { // if the ordinal of the pod is within the range of the current number of replicas, // insert it at the indirection of its ordinal replicas[ord] = pods[i] } else if ord >= replicaCount { // if the ordinal is greater than the number of replicas add it to the condemned list condemned = append(condemned, pods[i]) } // If the ordinal could not be parsed (ord < 0), ignore the Pod. } // for any empty indices in the sequence [0,set.Spec.Replicas) create a new Pod at the correct revision for ord := 0; ord < replicaCount; ord++ { if replicas[ord] == nil { replicas[ord] = newVersionedStatefulSetPod( currentSet, updateSet, currentRevision.Name, updateRevision.Name, ord) } } // sort the condemned Pods by their ordinals sort.Sort(ascendingOrdinal(condemned)) // find the first unhealthy Pod for i := range replicas { if !isHealthy(replicas[i]) { unhealthy++ if ord := getOrdinal(replicas[i]); ord < firstUnhealthyOrdinal { firstUnhealthyOrdinal = ord firstUnhealthyPod = replicas[i] } } } for i := range condemned { if !isHealthy(condemned[i]) { unhealthy++ if ord := getOrdinal(condemned[i]); ord < firstUnhealthyOrdinal { firstUnhealthyOrdinal = ord firstUnhealthyPod = condemned[i] } } } if unhealthy > 0 { glog.V(4).Infof("StatefulSet %s/%s has %d unhealthy Pods starting with %s", set.Namespace, set.Name, unhealthy, firstUnhealthyPod.Name) } // If the StatefulSet is being deleted, don't do anything other than updating // status. if set.DeletionTimestamp != nil { return &status, nil } monotonic := !allowsBurst(set) // Examine each replica with respect to its ordinal for i := range replicas { // delete and recreate failed pods if isFailed(replicas[i]) { glog.V(4).Infof("StatefulSet %s/%s is recreating failed Pod %s", set.Namespace, set.Name, replicas[i].Name) if err := ssc.podControl.DeleteStatefulPod(set, replicas[i]); err != nil { return &status, err } if getPodRevision(replicas[i]) == currentRevision.Name { status.CurrentReplicas-- } else if getPodRevision(replicas[i]) == updateRevision.Name { status.UpdatedReplicas-- } status.Replicas-- replicas[i] = newVersionedStatefulSetPod( currentSet, updateSet, currentRevision.Name, updateRevision.Name, i) } // If we find a Pod that has not been created we create the Pod if !isCreated(replicas[i]) { if err := ssc.podControl.CreateStatefulPod(set, replicas[i]); err != nil { return &status, err } status.Replicas++ if getPodRevision(replicas[i]) == currentRevision.Name { status.CurrentReplicas++ } else if getPodRevision(replicas[i]) == updateRevision.Name { status.UpdatedReplicas++ } // if the set does not allow bursting, return immediately if monotonic { return &status, nil } // pod created, no more work possible for this round continue } // If we find a Pod that is currently terminating, we must wait until graceful deletion // completes before we continue to make progress. if isTerminating(replicas[i]) && monotonic { glog.V(4).Infof( "StatefulSet %s/%s is waiting for Pod %s to Terminate", set.Namespace, set.Name, replicas[i].Name) return &status, nil } // If we have a Pod that has been created but is not running and ready we can not make progress. // We must ensure that all for each Pod, when we create it, all of its predecessors, with respect to its // ordinal, are Running and Ready. if !isRunningAndReady(replicas[i]) && monotonic { glog.V(4).Infof( "StatefulSet %s/%s is waiting for Pod %s to be Running and Ready", set.Namespace, set.Name, replicas[i].Name) return &status, nil } // Enforce the StatefulSet invariants if identityMatches(set, replicas[i]) && storageMatches(set, replicas[i]) { continue } // Make a deep copy so we don't mutate the shared cache replica := replicas[i].DeepCopy() if err := ssc.podControl.UpdateStatefulPod(updateSet, replica); err != nil { return &status, err } } // At this point, all of the current Replicas are Running and Ready, we can consider termination. // We will wait for all predecessors to be Running and Ready prior to attempting a deletion. // We will terminate Pods in a monotonically decreasing order over [len(pods),set.Spec.Replicas). // Note that we do not resurrect Pods in this interval. Also not that scaling will take precedence over // updates. for target := len(condemned) - 1; target >= 0; target-- { // wait for terminating pods to expire if isTerminating(condemned[target]) { glog.V(4).Infof( "StatefulSet %s/%s is waiting for Pod %s to Terminate prior to scale down", set.Namespace, set.Name, condemned[target].Name) // block if we are in monotonic mode if monotonic { return &status, nil } continue } // if we are in monotonic mode and the condemned target is not the first unhealthy Pod block if !isRunningAndReady(condemned[target]) && monotonic && condemned[target] != firstUnhealthyPod { glog.V(4).Infof( "StatefulSet %s/%s is waiting for Pod %s to be Running and Ready prior to scale down", set.Namespace, set.Name, firstUnhealthyPod.Name) return &status, nil } glog.V(4).Infof("StatefulSet %s/%s terminating Pod %s for scale dowm", set.Namespace, set.Name, condemned[target].Name) if err := ssc.podControl.DeleteStatefulPod(set, condemned[target]); err != nil { return &status, err } if getPodRevision(condemned[target]) == currentRevision.Name { status.CurrentReplicas-- } else if getPodRevision(condemned[target]) == updateRevision.Name { status.UpdatedReplicas-- } if monotonic { return &status, nil } } // for the OnDelete strategy we short circuit. Pods will be updated when they are manually deleted. if set.Spec.UpdateStrategy.Type == apps.OnDeleteStatefulSetStrategyType { return &status, nil } // we compute the minimum ordinal of the target sequence for a destructive update based on the strategy. updateMin := 0 if set.Spec.UpdateStrategy.RollingUpdate != nil { updateMin = int(*set.Spec.UpdateStrategy.RollingUpdate.Partition) } // we terminate the Pod with the largest ordinal that does not match the update revision. for target := len(replicas) - 1; target >= updateMin; target-- { // delete the Pod if it is not already terminating and does not match the update revision. if getPodRevision(replicas[target]) != updateRevision.Name && !isTerminating(replicas[target]) { glog.V(4).Infof("StatefulSet %s/%s terminating Pod %s for update", set.Namespace, set.Name, replicas[target].Name) err := ssc.podControl.DeleteStatefulPod(set, replicas[target]) status.CurrentReplicas-- return &status, err } // wait for unhealthy Pods on update if !isHealthy(replicas[target]) { glog.V(4).Infof( "StatefulSet %s/%s is waiting for Pod %s to update", set.Namespace, set.Name, replicas[target].Name) return &status, nil } } return &status, nil }
主要流程:
獲取currentRevision和updateRevision對應的StatefulSet Object,并設置generation,currentRevision, updateRevision等信息到StatefulSet status。
將前面getPodsForStatefulSet獲取到的pods分成兩個slice:
valid replicas slice: : 0 <= getOrdinal(pod) < set.Spec.Replicas
condemned pods slice: set.Spec.Replicas <= getOrdinal(pod)
如果valid replicas中存在某些ordinal沒有對應的Pod,則創建對應Revision的Pods Object,后面會檢測到該Pod沒有真實創建就會去創建對應的Pod實例:
如果更新策略是RollingUpdate且Partition為0或者ordinal < Partition,則使用currentRevision創建該Pod Object。
如果更新策略時RollingUpdate且Partition不為0且ordinal >= Partition,則使用updateRevision創建該Pod Object。
從valid repilcas和condemned pods兩個slices中找出第一個unhealthy的Pod。(ordinal最小的unhealth pod)
healthy pods means:pods is running and ready, and not terminating.
對于正在刪除(DeletionTimestamp非空)的StatefulSet,不做任何操作,直接返回當前status。
遍歷valid replicas中pods,保證valid replicas中index在[0,spec.replicas)的pod都是Running And Ready的:
Pod is Running and Ready means:
pod.Status.Phase = Runnin,
pod.Status.Condition = Ready
如果檢測到某個pod Failed (pod.Status.Phase = Failed), 則刪除這個Pod,并重新new這個pod object(注意revisions匹配)
如果這個pod還沒有recreate,則Create it。
如果ParallelPodManagement = "OrderedReady”,則直接返回當前status。否則ParallelPodManagement = "Parallel”,則循環檢測下一個。
如果pod正在刪除并且ParallelPodManagement = "OrderedReady”,則返回status結束。
如果pod不是RunningAndReady狀態,并且ParallelPodManagement = "OrderedReady”,則返回status結束。
檢測該pod與statefulset的identity和storage是否匹配,如果有一個不匹配,則調用apiserver Update Stateful Pod進行updateIdentity和updateStorage(并創建對應的PVC),返回status,結束。
遍歷condemned replicas中pods,index由大到小的順序,確保這些pods最終都被刪除:
如果這個Pod正在刪除(DeletionTimestamp),并且Pod Management是OrderedReady,則進行Block住,返回status,流程結束。
如果是OrderedReady策略,Pod不是處于Running and Ready狀態,且該pod不是first unhealthy pod,則返回status,流程結束。
其他情況,則刪除該statefulset pod。
根據該pod的controller-revision-hash Label獲取Revision,如果等于currentRevision,則更新status.CurrentReplicas;如果等于updateRevision,則更新status.UpdatedReplicas;
如果是OrderedReady策略,則返回status,流程結束。
OnDelete更新策略:刪除Pod才會觸發更新這個ordinal的更新 如果UpdateStrategy Type是OnDelete, 意味著只有當對應的Pods被手動刪除后,才會觸發Recreate,因此直接返回status,流程結束。
RollingUpdate更新策略:(Partition不設置就相當于0,意味著全部pods進行滾動更新) 如果UpdateStrategy Type是RollingUpdate, 根據RollingUpdate中Partition
配置得到updateMin作為update replicas index區間最小值,遍歷valid replicas,index從最大值到updateMin遞減的順序:
如果pod revision不是updateRevision,并且不是正在刪除的,則刪除這個pod,并更新status.CurrentReplicas,然后返回status,流程結束。
如果pod不是healthy的,那么將等待它變成healthy,因此這里就直接返回status,流程結束。
updateStatefulSet Reconcile中,會檢查identity match的情況,具體包含哪些?
StatefulSetPodNameLabel = "statefulset.kubernetes.io/pod-name" // identityMatches returns true if pod has a valid identity and network identity for a member of set. func identityMatches(set *apps.StatefulSet, pod *v1.Pod) bool { parent, ordinal := getParentNameAndOrdinal(pod) return ordinal >= 0 && set.Name == parent && pod.Name == getPodName(set, ordinal) && pod.Namespace == set.Namespace && pod.Labels[apps.StatefulSetPodNameLabel] == pod.Name }
pod name和statefulset name內容匹配。
namespace匹配。
Pod的Label:statefulset.kubernetes.io/pod-name
與Pod name真實匹配。
updateStatefulSet Reconcile中,會檢查Storage match的情況,具體怎么匹配的呢?
// storageMatches returns true if pod's Volumes cover the set of PersistentVolumeClaims func storageMatches(set *apps.StatefulSet, pod *v1.Pod) bool { ordinal := getOrdinal(pod) if ordinal < 0 { return false } volumes := make(map[string]v1.Volume, len(pod.Spec.Volumes)) for _, volume := range pod.Spec.Volumes { volumes[volume.Name] = volume } for _, claim := range set.Spec.VolumeClaimTemplates { volume, found := volumes[claim.Name] if !found || volume.VolumeSource.PersistentVolumeClaim == nil || volume.VolumeSource.PersistentVolumeClaim.ClaimName != getPersistentVolumeClaimName(set, &claim, ordinal) { return false } } return true }
基于上述分析,下面是一個相對完整的StatefulSetController的代碼邏輯圖。 (不支持大于2MB的圖片,所以不太清晰,不過基本在前面內容都提到了。)
在上一篇博文淺析Kubernetes StatefulSet中遺留了一個問題:StatefulSet滾動更新時,如果某個Pod更新失敗,會怎么辦呢?
通過上面源碼分析中滾動更新部分的分析,我們知道:
如果UpdateStrategy Type是RollingUpdate, 根據RollingUpdate中Partition
(Partition不設置就相當于0,意味著全部pods進行滾動更新)配置得到updateMin作為update replicas index區間最小值,遍歷valid replicas,index從最大值到updateMin遞減的順序:
如果pod revision不是updateRevision,并且不是正在刪除的,則刪除這個pod,并更新status.CurrentReplicas,然后返回status,流程結束。
如果pod不是healthy的,那么將等待它變成healthy,因此這里就直接返回status,流程結束。
知道這一點后,就能回答這個問題了,答案很簡單:
如果更新策略是RollingUpdate,則逐個滾動更新過程中,如果在更新某個ordinal replica時這個Pod一直無法達到Running and Ready狀態,那么整個滾動更新流程將Block在這里。還沒有更新的replicas將不會觸發更新,已經更新成功的replicas就保持更新后的版本,并不存在什么自動回滾的機制。在下一次sync時,檢測到這個Pod isFailed(pod.Status.Phase = Failed),會delete and recreate這個failed pod。
問題:podManagementPolicy: "Parallel"體現在什么時候呢?Scale的時候?RollingUpdate的時候?
在前面代碼分析中updateStatefulSet中那段-"遍歷valid replicas中pods,保證valid replicas中index在[0,spec.replicas)的pod都是Running And Ready的":如果發現某個ordinal replica應該創建但是還沒被創建,則會觸發create。如果podManagementPolicy設置為Parallel,則會繼續delete then create
其他應該創建的replicas,而不會等待前面創建的replicas成為Running and Ready。
在前面代碼分析中updateStatefulSet中那段-”遍歷condemned replicas中pods,index由大到小的順序,確保這些pods最終都被刪除":podManagementPolicy設置為Parallel,如果發現某個ordinal replica正在刪除,則繼續刪除其他應該刪除的replicas,而不會等待之前刪除的replica重建并成為Running and Ready狀態。
因此Parallel體現在以下場景:
初始化部署StatefulSet時,并行create pods。
級聯刪除StatefulSet時,并行delete pods。
Scale up時,并行create pods。
Scale down時,并行delete pods。
而在滾動更新時,是不會受podManagementPolicy的配置影響的,都是按照逐個地、ordinal從大到小的的順序,保證前者Running and Ready的原則,進行RollingUpdate。
“Kubernetes StatefulSet源碼是什么”的內容就介紹到這里了,感謝大家的閱讀。如果想了解更多行業相關的知識可以關注億速云網站,小編將為大家輸出更多高質量的實用文章!
免責聲明:本站發布的內容(圖片、視頻和文字)以原創、轉載和分享為主,文章觀點不代表本網站立場,如果涉及侵權請聯系站長郵箱:is@yisu.com進行舉報,并提供相關證據,一經查實,將立刻刪除涉嫌侵權內容。