Adding Creawler to POPS

Our security work today was to add a Python crawler to the POPS toolbelt. The crawler is a simple script that will be used to gather information from a target website. The script is written in Python and uses the requests library.

What is a Creawler?

In general terms, a crawler is a script that will navigate through a website and gather information. This information can be used for various purposes. In this case, the crawler lists all links present in the webpage and, if they match the target domain, it will check them recursively.