RQ2

The links in the tables below contain data in the form of a JSON list. Each element of the list contains a JSON object which represents a test case. Here is an example:

[
  {
    "init": "int_list=[]",
    "assertion": "assert count == 0"
  },
  .
  .
  .
]  

The key “init” contains a piece of code to initialise the input variables. The key “assertion” contains assertion statements to validate the correctness of the variant.

Please refer Table 1 to find the details of the CPAT Numbers mentioned below.

GPT-3.5

CPAT	num-tests	invalid_format_count	syntax_err_count	uninitialised_vars_count	invalid_test_count
1	342	0	2	0	54
2	266	1	13	1	160
3	230	2	1	1	25
4	177	1	2	0	119
5	166	1	0	10	56
6	256	0	0	0	12
7	117	2	0	0	104
8	71	1	0	0	88
9	119	1	1	0	39
10	186	0	2	10	41
11	26	2	1	39	102
12	152	5	0	0	40
13	0	1	0	0	146
14	240	0	2	5	26
15	76	0	0	0	158
16	156	0	3	1	71
17	222	1	4	4	71
18	184	0	8	1	222
19	105	0	0	0	159
20	271	1	0	48	183

GPT-4

CPAT	num-tests	invalid_format_count	syntax_err_count	uninitialised_vars_count	invalid_test_count
1	119	0	1	0	18
2	326	1	1	2	3
3	215	0	2	0	21
4	210	0	5	0	67
5	281	1	4	3	39
6	266	0	0	0	27
7	145	1	0	0	21
8	184	0	0	0	86
9	136	0	1	0	44
10	268	1	3	0	21
11	93	0	2	0	275
12	155	2	1	0	9
13	0	0	0	0	172
14	222	0	3	11	3
15	209	1	0	0	5
16	224	1	4	6	41
17	239	0	1	0	8
18	359	1	12	3	4
19	184	1	1	1	4
20	372	2	28	2	8

PALM

CPAT	num-tests	invalid_format_count	syntax_err_count	uninitialised_vars_count	invalid_test_count
1	112	5	1	0	118
2	59	20	0	2	46
3	61	13	0	0	11
4	64	15	20	0	26
5	41	11	1	0	46
6	108	9	2	0	48
7	106	3	1	0	127
8	145	5	42	0	72
9	125	6	19	0	59
10	127	13	0	1	73
11	0	17	0	0	9
12	133	1	0	0	122
13	8	7	0	0	76
14	74	9	12	9	31
15	39	5	2	0	54
16	113	1	26	0	74
17	22	2	1	0	157
18	90	1	10	0	206
19	62	4	15	2	189
20	122	1	14	0	142