Exclude Raw Request Payloads #3710
Conversation
There was a problem hiding this comment.
lgtm !
$ ./nuclei -u oast.site -id "robots-txt-endpoint" -je results.json -erp && cat results.json| jq . 3 ↵
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v2.9.4-dev
projectdiscovery.io
[WRN] Found 24 templates loaded with deprecated paths, update before v2.9.5 for continued support.
[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v2.9.5 for continued support.
[INF] Current nuclei version: v2.9.4-dev (outdated)
[INF] Current nuclei-templates version: v9.5.0 (latest)
[INF] New templates added in latest release: 62
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[robots-txt-endpoint] [http] [info] https://oast.site/robots.txt
[
{
"template": "http/miscellaneous/robots-txt-endpoint.yaml",
"template-url": "https://github.com/projectdiscovery/nuclei-templates/blob/main/http/miscellaneous/robots-txt-endpoint.yaml",
"template-id": "robots-txt-endpoint",
"template-path": "/Users/tarun/nuclei-templates/http/miscellaneous/robots-txt-endpoint.yaml",
"info": {
"name": "robots.txt endpoint prober",
"author": [
"caspergn",
"pdteam"
],
"tags": null,
"reference": null,
"severity": "info",
"metadata": {
"max-request": 2
}
},
"type": "http",
"host": "https://oast.site",
"matched-at": "https://oast.site/robots.txt",
"ip": "178.128.16.97",
"timestamp": "2023-05-23T00:11:10.354518+05:30",
"curl-command": "curl -X 'GET' -d '' -H 'Accept: */*' -H 'Accept-Language: en' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36' 'https://oast.site/robots.txt'",
"matcher-status": true,
"matched-line": null
}
]
|
@ehsandeep , i think we can improve flag description to explicitly tell that |
|
Is there a change requested on this to emphasize that it's the reports only and not the console formatted JSON (which I don't believe it includes anyway), or is that under review? |
There was a problem hiding this comment.
@kchason @tarunKoyalwar nuclei already has -irr option i.e
-irr, -include-rr include request/response pairs in the JSONL output (for findings only)
so instead of adding a new option to exclude, we can default to exclude raw requests/responses and only include when -irr option is used (similar to -j option) for export feature as well.
…to exclude-raw-request-payloads # Conflicts: # README.md
|
Should be all set now. I changed the internal variable name to better match what it's doing as well, it doesn't change anyone's integrations since the flag is the same |
tarunKoyalwar
requested review from
Mzack9999
and removed request for
tarunKoyalwar
There was a problem hiding this comment.
I think the default behavior should be to include them and skin them optionally. What do you think @ehsandeep ?
|
@Mzack9999 that wouldn't match the existing functionality of the |
|
Is this not accepted as a concept? Or are there changes requested on this? |
|
@kchason was closed by mistake. |
There was a problem hiding this comment.
@kchason as mzack suggested, it would be nice (also to align with our default behavior across project) to include all the information as default and optionally exclude things, also to keep the uniform behavior across all the output events which is not the case for now, for example -irr is enabled as default for reporting and markdown but not for jsonl.
Can you adapt the PR as per #3891? let me know if you have any questions.
ehsandeep
added
Status: Revision Needed
|
The PR has been updated with:
|
There was a problem hiding this comment.
lgtm !
$ ./nuclei -u oast.site -id "robots-txt-endpoint" -or -je results.json && cat results.json| jq .
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v2.9.8
projectdiscovery.io
[WRN] Found 3 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.8 (latest)
[INF] Current nuclei-templates version: v9.5.4 (latest)
[INF] New templates added in latest release: 51
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[robots-txt-endpoint] [http] [info] https://oast.site/robots.txt
[
{
"template": "http/miscellaneous/robots-txt-endpoint.yaml",
"template-url": "https://github.com/projectdiscovery/nuclei-templates/blob/main/http/miscellaneous/robots-txt-endpoint.yaml",
"template-id": "robots-txt-endpoint",
"template-path": "/Users/tarun/nuclei-templates/http/miscellaneous/robots-txt-endpoint.yaml",
"info": {
"name": "robots.txt endpoint prober",
"author": [
"caspergn",
"pdteam"
],
"tags": [
"misc",
"generic"
],
"reference": null,
"severity": "info",
"metadata": {
"max-request": 2
}
},
"type": "http",
"host": "https://oast.site",
"matched-at": "https://oast.site/robots.txt",
"ip": "178.128.16.97",
"timestamp": "2023-07-05T01:20:09.794878+05:30",
"curl-command": "curl -X 'GET' -d '' -H 'Accept: */*' -H 'Accept-Language: en' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36' 'https://oast.site/robots.txt'",
"matcher-status": true,
"matched-line": null
}
]
ehsandeep
removed
the
Status: Revision Needed
|
Thank you @kchason for adding this. |
Proposed changes
Addresses #3709
Checklist