GitLab CI and Code Quality
The last few days after New Year (Happy New Year!!) I have spent some time on a CI flow for GitLab. I will be involved in a new course on OOP in Java and there will likely be quite a lot of students (i.e. 150-200). This quickly becomes a challenge for the examination of assignments where things tend to take a lot of time and effort. We run a GitLab instance at our department and it has been put to good use in a few courses mainly in introductory courses in programming using JavaScript.
Anyway using a CI flow would give the benefit of streamlining the process of examination by automation. The students would be presented with some base code with accompanying unit tests as the passing criteria. This code would be copied to the student’s own assignment specific project on GitLab and some suitable branches would be created. When the student pushes to the branch the CI flow would run and tests would be executed and progress would immediately be visible. The student could of course run the test in their local environment too for an even tighter feedback loop. To add some spice we can also add code quality requirements such as formatting of source code and avoidance of known bug/issue patterns.
There are three key challenges in such a setup. First, tools and integration must work well. More tools can add to the complexity and pose problems that distract from the learning goals. Second, the construction of the unit tests should present a problem as they can reveal too much about the solution to the problem at hand. There is a risk of over-specification that removes some of the more creative aspects of solving a problem. Thirdly the feedback from tests should be good so that the student can understand and learn from it.
The tools we use are quite mature i.e. git/GitLab and for building/running tests etc we will use Gradle. They are tools used in industry and students will benefit from being familiar with them. This can at least motivate the use of tools when/if you run into tool problems.
The construction of unit tests that cover the assignment task can always be debated. In this course, I don’t think it is a big issue as it is more of an introductory OOP course. I think more time and effort needs to be spent the more freedom the student is expected to have in solving a problem. Also in this case the streamlining and automation are worth the drawback of not allowing total freedom.
The third problem is where my time was spent. Getting a well-formatted output for code quality issues proved to be more challenging than I first thought. Basically, GitLab provides a mechanism to save the results of a pipeline stage as a downloadable artifact. It also provides a few reports. The reports are very good as they provide feedback in the GUI directly, i.e. you get some visualization of unit test results when you run a pipeline or when you issue a merge request. This actually works quite well for unit tests, but not so for the code quality report.
GitLab supports the CodeClimate format. It is basically a JSON file with a specific format. Using Gradle we can generate some reports of the code quality, for example checkstyle and spotbugs/findbugs. These generate their own specific XML formats, but there the violations plugin to Gradle which is excellent and can parse and consolidate many formats to CodeClimate. So far so good. So we can run Gradle with checkstyleMain and spotbugsMain and then we run violations to get the final code-climate report. Here GitLab poses two problems. First, the formatting of code quality issues is not very good. Basically, you get an unformatted list of all issues and this simply does not work. Secondly, the code-quality report is used in merge requests in a somewhat strange way: it reports a relative number i.e. a code-quality increase or decrease by some value. This could be fine but the problem is that you must merge with the branch you branched off from AND that branch must not have advanced. Basically, if someone else has made a commit to the branch you try to merge to… the comparison fails and you get no report of code quality change in the merge request.
I could live with the branch issue as the students would work on their own projects and probably just work in one branch at a time they will likely stay in synch. However, the second problem is worse. It is simply not a good idea to present a big blob of problems for a student. I tried to get the formatting by changing the resulting JSON but it was not the way to go and compared to the information you get for the JUnit reports in GitLab I finally decided to just parse the XML file myself and output the result as JUnit XML files. This proved to be a much better approach with the added benefit that I can allow different levels of code quality problems before the test fails. The student can also inspect the problems on a per source code file basis. I.e. one test case per file for checkstyle and one test case per file for findbugs. Indeed the feedback in findbugs is excellent (if properly formated).
💬 Comments
Post comment