Optimal code metric values for P4 programs

[stan-leetasi]

Recently, new functionality was added to the compiler, which allows for collecting code metrics from compiled programs, in PR #5250. Implementing this functionality was part of my bachelor's thesis.

As a part of my thesis, I also conducted an empirical analysis of code metric values from public P4 programs sourced from the P4Guide and P4Tutorials repositories. The goal was to determine optimal metric values and the upper acceptable thresholds for metric values of P4 programs. The final results are in the tables below.

Optimal values

Metric	Mean	Median
Lines of Code	164.20	154.00
Cyclomatic Complexity	21.22	18.00
Halstead Metrics
Unique Operators	28.61	25.00
Unique Operands	60.78	53.00
Total Operators	192.33	159.00
Total Operands	204.37	176.00
Vocabulary	89.39	89.00
Length	396.71	327.00
Difficulty	47.53	35.58
Volume	2714.79	2090.29
Effort	216974.81	75213.00
Estimated Bugs	0.90	0.70

Maximum acceptable values

Metric	Threshold
Cyclomatic Complexity	42.00
Halstead Metrics
Unique Operators	40.00
Unique Operands	113.00
Total Operators	446.00
Total Operands	443.00
Vocabulary	137.00
Length	928.00
Difficulty	90.46
Volume	6760.85
Effort	527913.00
Estimated Bugs	2.25

The detailed explanation of how the analysis was conducted, and graphs visualising the data can be found in my thesis.

I am opening this issue because I think that this data is an useful reference point for utilizing the code metric collection functionality. If you have any suggestions about where I should post this, please let me know.

I am also including some of the graphs which visualize the collected data.

[fruffy]

Any particular insights from this data? E.g., what types of programs cross this threshold and how the estimated bugs correlate with actual bugs.

@jafingerhut Do we have a place for "comment issues" like this one? Maybe we should enable Github discussions...

[stan-leetasi]

Any particular insights from this data? E.g., what types of programs cross this threshold and how the estimated bugs correlate with actual bugs.

I didn't dive that deep in this analysis, it was meant more as a simple reference point for the metric values, or a starting point for future projects.

[jafingerhut]

Any particular insights from this data? E.g., what types of programs cross this threshold and how the estimated bugs correlate with actual bugs.

@jafingerhut Do we have a place for "comment issues" like this one? Maybe we should enable Github discussions...

I do not know, nor am I familiar enough with Github discussions to comment on that idea. I tend to create articles in "docs" directories for notes that I think are worth saving for a while, updating over time, etc. and linking to them from elsewhere, but understood that something that doesn't require a review cycle to add can be nice.

@stan-leetasi I think all of the P4 programs in the https://github.com/p4lang/tutorials and https://github.com/jafingerhut/p4-guide repositories are small examples intended for emphasizing one feature of P4, or implementing one or two features of a packet processing device.

A couple of longer more substantive published P4 examples come to mind:

(1) It is P4_14, but can be auto-translated into P4_16 source code (let me know if you need the command to do so, but it can be found in top level README of this repo): https://github.com/p4lang/p4c/tree/main/testdata/p4_14_samples/switch_20160512

(2) https://github.com/p4lang/p4c/tree/main/testdata/p4_16_samples/fabric_20190420

[vgurevich]

@stan-leetasi -- it is a very good idea to collect the statistics and other data about P4 programs.

In the natural language research there is a special branch, called corpus linguistics, that studies text corpora -- the collections of text which are specially marked so that useful information can be extracted. So, for example, when you open a dictionary, like Merriam-Webster and see that the word "bamboozle" first appeared in 1703 and that Jonathan Swift disliked this "new" word, know that all that info came from the big corpora of English language.

Now, there are a couple of things that a required to make a corpus (or corpora) useful:

1. It goes without saying, but the texts should be in the same language, otherwise you'll be really surprised, right :) In case of P4, it is thus very important to separate the programs per P4 architecture, per target or both. Programs written for v1model architecture are quite different from the ones, written for (or in) PSA, PNA, TNA or other architectures. I am not sure if you can access to the programs written for other proprietary targets, like Intel DPUs, AMD Pensando NICs, Cisco's Silicon One, etc., but I can bet anything that they are all quite different. Note also that even though Tofino allegedly supports v1model architecture, it is markedly different from the one used on BMv2 and should not be even called that. Similarly, P4_14 and P4_16 are two different languages.
2. You definitely need to get as much code as you can get your hands on, but you also need to be selective. For example, if you try to build a language corpus by only collecting newspaper classifieds (or only collecting computer user manuals, etc.), your view of the language will become very skewed. You really need real code.

To that end, I think I can make a couple of recommendations:

1. Be very careful about the testcases. 99% of them do not even use any particular architecture and are minimal examples that illustrate a language feature (or a compiler bug). In fact, I would not recommend using any test code that is not written for a specific target architecture.
2. There are at least two examples of a "big" switch program available publicly:
   a. switch.p4_16 (multiple variants ("profiles")) written for TNA or T2NA.
   b. A much earlier version of that program written for BMv2 in p4_14.
3. There are a lot of public repositories on Github that contain original code.
4. There is an archived copy of a page I used to maintain back then that has pointers to many papers written using P4 for Tofino and many of them include pointers to the source code.

Good luck with your research.

Happy hacking,
Vladimir

[stan-leetasi]

Thank you all for your suggestions. I will look into this more and report back if I come to any useful conclusions.