RQ1

Each number inside the following tables links to a JSON list containing the corresponding data. Each element in the JSON list is a piece of code, in the form of a string.

Here is a sample JSON list:

[
 "count = 0\nfor i in int_list:\n    count += i",
 "import numpy as np\ncount = np.sum(int_list)",
 "count = sum(int_list)",
 .
 .
 .
]

Please refer Table 1 to find the details of the CPAT Numbers mentioned below.

GPT-3.5

CPAT Number	correct	syntax-errors	type-errors	import-errors	invalid_format	incorrect
1	40	0	286	47	0	100
2	102	0	147	14	1	168
3	39	0	0	34	0	327
4	11	0	279	20	1	145
5	100	0	161	32	2	215
6	71	0	242	12	0	169
7	61	0	204	78	0	193
8	18	0	203	62	0	96
9	9	0	158	27	2	281
10	100	0	45	43	0	295
11	150	0	63	81	0	230
12	97	0	254	67	0	64
13	0	0	0	316	2	123
14	11	0	0	83	0	343
15	77	0	0	32	1	276
16	69	0	0	10	1	327
17	109	0	0	9	0	333
18	39	0	0	57	0	369
19	20	0	127	11	0	407
20	136	0	0	2	2	296

GPT-4

CPAT Number	correct	syntax-errors	type-errors	import-errors	invalid_format	incorrect
1	67	0	159	27	0	116
2	187	0	45	15	0	162
3	58	0	0	18	1	268
4	101	0	150	18	0	114
5	132	0	107	26	2	192
6	161	0	113	27	0	136
7	53	0	229	10	3	103
8	21	0	210	27	0	89
9	15	0	165	31	1	135
10	184	0	30	23	0	196
11	115	0	37	31	0	252
12	98	0	274	20	0	11
13	0	0	0	312	0	106
14	272	0	0	16	0	152
15	132	0	0	20	3	293
16	224	0	0	7	20	119
17	101	0	0	6	0	271
18	166	0	0	9	3	219
19	144	0	52	16	0	153
20	153	0	0	23	0	247

PALM

CPAT Number	correct	syntax-errors	type-errors	import-errors	invalid_format	incorrect
1	27	123	154	61	0	133
2	36	68	177	29	0	195
3	18	11	0	116	0	370
4	3	26	172	10	0	233
5	34	38	166	115	0	159
6	40	35	41	141	0	256
7	21	36	94	121	0	135
8	12	114	130	111	0	128
9	13	43	137	65	0	139
10	32	3	82	81	0	332
11	7	66	36	254	0	164
12	29	92	132	184	0	96
13	0	27	0	357	0	91
14	18	42	0	104	0	329
15	10	86	0	36	0	253
16	34	53	0	56	0	324
17	31	31	0	38	0	439
18	71	49	0	26	0	341
19	0	107	225	169	0	70
20	25	39	0	4	0	422