reportinator

< Back Home

Objective

Noel Boetie used ChatNPT to write a pentest report. He needs help to understand which report findings are legitimate and which are hallucinations. Read through the report carefully and select either legitimate finding or hallucinated finding for each report element.

Conversations

Noel Boetie

Initial conversation:

• Hey there, Noel Boetie speaking! I recently tried using ChatNPT to generate my penetration testing report.
• It’s a pretty nifty tool, but there are a few issues in the output that I’ve noticed.
• I need some guidance in finding any errors in the way it generated the content, especially those odd hallucinations in the LLM output.
• I know it’s not perfect, but I’d really appreciate the extra eyes on this one.
• Some of the issues might be subtle, so don’t be afraid to dig deep and ask for further clarification if you’re unsure.
• I’ve heard that you folks are experts about LLM outputs and their common issues, so I trust you can help me with this.
• Your input will be invaluable to me, so please feel free to share any insights or findings you may have.
• I’m looking forward to working with you all and improving the quality of the ChatNPT-generated penetration testing report.
• Thanks in advance for your help! I truly appreciate it! Let’s make this report the best it can be!

After solving challenge:

• Great job on completing that challenge! Ever thought about how your newfound skills might come into play later on? Keep that mind sharp, and remember, today’s victories are tomorrow’s strategies!

Hints

• I know AI sometimes can get specifics wrong unless the prompts are well written. Maybe chatNPT made some mistakes here.

Approach

For the approach, I spent some cycles reading through the report, running the report through chat GPT and Bing chat enterprise, but found a brute force approach will be most efficient. The application uses client side java to interact with 9 report findings and post the values to /check, so there is an opportunity to inject some code into the console and brute force until we get a successful response. With only 9 binary report findings, there is a total of 9 bits we need to iterate through which means we only need to iterate through a total of 512 combinations max.

To begin, I modified the main.js file slightly to stop having to click on an alert every time an iteration fails, and creating a new var “error” to establish something to stop iterating if we find a successful combination of values.

Existing code in main.js (starting at line 83):

                .then(data => {
                    if (data.error && data.error === 'FAILURE') {
                        alert("Failure, please try again.");
                    } else {
                        updateWebContent();
                        __POST_RESULTS__(data);
                    }

Modified code:

                .then(data => {
                    if (data.error && data.error === 'FAILURE') {
                        //alert("Failure, please try again.");
                    } else {
                        updateWebContent();
                        __POST_RESULTS__(data);
                        var error = false; //to check during our iterations if we found the answer
                    }

Next, I created some helper functions to get us through modifying the values on the html document elements. This is the code you will want to run through the console.

//create a change selection function which takes an int and creates a binary array and uses that to make a selection array
function changeSelections(iter) {
    var selection = iter.toString(2).split('');
    for (let i = 0; i < 9; i++) {
        if (selection[i] == undefined) {
        selection.unshift('0');
        }
    }
    //now that selection is fully populated, we must iterate through one more time to change all the values on the html document
    for (let i = 0; i < 9; i++) {
      var inputNum = i+1;
      document.getElementById('input-'+inputNum).value = selection[i];
    }
    console.log(selection);
    document.getElementsByClassName("submit-button")[0].click();
};

//delay iterations while we wait for post response
function loop() { 
    setTimeout(function() {
        changeSelections(iteration);
        console.log(iteration);
        iteration ++;
        if (iteration < 512 && error) {
            loop();
        }
    }, 1000);
};

var iteration = 0;
var error = true;

loop();

Running the above code in the console successfully finds a binary combination at iteration 73 of 001001001. A value of 0 means finding is legit and value of 1 means finding is a hallucination so the following is the answer:

1. Legit
2. Legit
3. Hallucination
4. Legit
5. Legit
6. Hallucination
7. Legit
8. Legit
9. Hallucination