Notes: k8s prow jobs testgrid walkthrough, k8s enhancement tracking spreadsheet, bi-weekly structured-logging workgroup meeting #19

kubernetes

May 27, 2021

Quick notes from today:

k8s prow jobs testgrid walkthrough

[The meeting notes are in this google document]

• During today’s 1:1 Thursday meeting with dims, he very kindly walked me through the testGrid for different prow ci job under the k8s project.
• Below are most of the things (I guess all, if I’m really not missing any more) that I learnt:
  • How to point to the prow job results dashboard from a PR.
    • Look for the list of the prow tests at the bottom of the PR, pick one prow job & click on the Details button next to it. For ex, this one goes to prow job pull-kubernetes-e2e-kind outcomes for this PR.
  • From there, I walked through all the following tabs on the prow job dashboard:
    • Job History: lists all the previous runs of a particular job across PRs under different k8s projects, with their statuses, whether failed or passed or skipped or else.
    • Prow Job YAML: shows the k8s yaml manifest for a particular prow ci job
    • PR History: Lists outcome of all the prow jobs/tests runs (not for a particular one job) on a particular PR across it’s different commits
    • PR: takes back to the PR from where we started
    • Artifacts: directory containing a rich compilation of all the logs & artifacts that were created during a particular prow job run

    • Testgrid: looks like the screenshot below. I need to play around it more to write a better minimal explanation for this, but what I understand right now is, it is same as Job History but with a tile (or a graph) representation distributed across differnet time periods (if it’s a periodic jobs) for when the job was executed & wha were the results of those runs (success, failure or skipped). The job results are in a compact grid structure, so it becomes easier to do retrospective on the performance of a particular job for a really long period of time in one screen itself.

      

  • Followed by that, I learnt how I could utilize cs.k8s.io (or Hound) to fetch both the job configuration & corresponding testgrid configuration for a particular prow job (for ex, here is for this job ~ ci-cri-containerd-node-e2e
  • And lastly, I learnt about different types of prow jobs under k8s projects. The ones I learnt today (not sure if these are the only ones) are:
    • Periodic jobs ~ runs over a defined period of time, say over every 5 hours, etc.
    • Pre-submit jobs ~ the CI jobs which are required to be green for a PR to be merged. Pre-submit here means this job is a pre-submit check for a PR.
    • `On-demand jobs`

      Post-submit jobs ~ the CI jobs that are expensive to run periodically & are not common to most of the PRs. Can be initiated using a prow test flag in the PR commaents, for ex ~ /test pull-kubernetes-e2e-kind.

• Exercise for the coming week (Testgrid exercise)
  • Pick up one PR -> Pick up one prow job
  • Investigate how that prow job is defined, whether it is periodic or pre-submit or an `on-demand` post-submit job
  • Investiage the results of a prow job run, so check & understand the following:
    • the artifacts created by the job run
    • logs it produced across various components

---

k8s enhancement tracking spreadsheet

• Find the enhancements tracking spreadsheet for all the previous & current kubernetes project release version milestone here

---

bi-weekly structured-logging workgroup meeting

• Today was the second meeting for the structured-logging working group.
• It was a wonderful discussion, touching progress on all following areas:
  • How the work under the working group would be organised/managed?
    • new label under all k8s projects ~ wg/structured-logging-label (apply it as /wg structured-logging)
    • teams ~ interested folks could add themselves as either member or reviewer to the project, based on their level of understanding. member is open to everyone, reviewer is for folks having prior skills & experience.
      • I’ve added myself as a member currently.
  • Updates on the status of work in areas:
    • Retrospection of kubelet migration (a survery is passed to the teams/people who participated in the kubelet structured logging migration process as either code-contributots, or reviewers, or approvers, etc)
    • Structured logging is well tested to check correctness and prevent regressions (related PRs were discussed)
    • Verification of log volumes (discussed as one of the major concerns. Issues will be created to track progress in this space. One of the suggestion was to add a periodic prow test that would check if the logs generated by a PR under structured logging is generating more than 10% logs)
    • Graduate JSON format to Beta (related PRs were discussed)
  • As part of the retrospective on the kubelet logs migration, the following point was reflected:
    • Lack of plural support in case of structured logging key names
    • The proposed solution is to implement the plural imitations of the existing klog.KObj, klog.KRef methods. I’ve picked up to do the same.
    • I’ve current not much idea on how it would be implemented. I’ll start by looking at the previous singular implementation here.
    • Will also read through the discussion that reflected the issue

---

That’s all for the day!

PS: Find here, the links to all the kubernetes-contributions for the month of May, 2021.