make scheduler metrics visible

[sanposhiho]

It would be nice if we could see the time used in scheduling for each plugin.

• Aggregate the time used for each plugin.
  • It can be achieved in the same way as scheduling results. We can aggregate it in simulatorPlugin and send used time to resultStore.
  • EDIT: Not need to aggregate the time on simulator. We can use plugin_execution_duration_seconds to see each plugins time. Potentially other metrics can also be utilized.
• [optional] show them on WebUI in table or graph..? should be something nice.
• update doc. (include https://github.com/kubernetes-sigs/kube-scheduler-simulator/blob/master/docs/how-it-works.md)

/assign
/kind feature

[196Ikuchil]

Hi @sanposhiho
Is there any progress on this? Are you looking for a worker?

[sanposhiho]

I haven't worked on this. So, feel free to pick this issue.
But, I think we should work on the following issue before this feature.
#125

[matthewygf]

Hi @196Ikuchil @sanposhiho , I am planning to work on this feature, additionally, my thoughts are to do something similar to Kube-Scheduler, where these performance metrics are emitted in prometheus metrics format. Something like this ? WDYT ?

---

EDIT: turns out it is already implemented here. Maybe we could implement a metrics endpoint for the simulator server ?

[sanposhiho]

EDIT: turns out it is already implemented here. Maybe we could implement a metrics endpoint for the simulator server ?

Oh, yes. You are right. I misunderstood about the metrics plugin_execution_duration_seconds 😓
Change this issue to make scheduler metrics visible.

/retitle make scheduler metrics visible

[sanposhiho]

One problem is pluginMetricsSamplePercent is const and we cannot change the value. Ideally, I would like to see metrics recorded for all pod schedules. (Because unlike real clusters, users do not always create many pods on the simulator)

I will make a feature request issue to k/k later. We will not know if the request will be approved until discussing it, but consider a work-around or another way if it is not approved then.

[sanposhiho]

kubernetes/kubernetes#108903

[matthewygf]

Regarding the metrics available, should we add a metric handler to the simulator server ? WDYT about having our own frontend charts to show these metrics like dashboard (probably need to investigate some of the frontend charts) or using prometheus to show them ?

[sanposhiho]

@matthewygf

Sorry, I missed your comment.

having our own frontend charts to show these metrics like dashboard (probably need to investigate some of the frontend charts) or using prometheus to show them ?

Yeah, hmm, I prefer the former. But, charts in frontend.. It could be a very tough road because at least I'm not very familiar with frontend... 😓 It would be so much easier if there were a public component or a public library to show charts from data in the form of grafana.

[matthewygf]

@matthewygf

Sorry, I missed your comment.

having our own frontend charts to show these metrics like dashboard (probably need to investigate some of the frontend charts) or using prometheus to show them ?

Yeah, hmm, I prefer the former. But, charts in frontend.. It could be a very tough road because at least I'm not very familiar with frontend... 😓 It would be so much easier if there were a public component or a public library to show charts from data in the form of grafana.

No problem @sanposhiho , I will have a look at it ~

[sanposhiho]

One approach is to process the grafana data on the simulator api into a form that is easy to display on the front end, and then display it on the front end using some library or component.

It seems vuetify has the component to write a simple graph. (But, not sure whether the component is sufficiently expressive)
https://vuetifyjs.com/en/components/sparklines/