PyCraft Replication Package

This site contains information about the tool, data and plots for the PyCraft tool submitted to FSE 2024.

Additional Plots

Figure 5

The figures below describe how the total number of variants generated changes by querying the LLM repeatedly with the same prompt. The x-axis shows the number of prompts, and the y-axis shows the percentage of variants generated.

The figure seen below is the one present in the paper. It shows the average behaviour across all chosen temperatures.

Average of all temperatures(Fig 5)

The figures below are supplementary, they depict the behaviour for each of the chosen temperature values.

Temperature 0

Temperature 0.3

Temperature 0.5

Temperature 0.7

Temperature 0.9

Temperature 1.2

Figure 7

The figures below describe how the number of hallucinations increases with the increase in number of fix-point iterations.

The figure seen below is the one present in the paper. It shows the behavior for temperature 0.

Temperature 0 (Fig 7)

The figures below are supplementary, they depict the behaviour for the remaining chosen temperature values.

Temperature 0.3

Temperature 0.5

Temperature 0.7

Temperature 0.9

Temperature 1.2