Unlocking Insights: An Extensible Framework for Automated Metadata Extraction from Online Documents

Information Gathering is a fundamental stage in a typical Penetration Testing (PT) process, in which penetration testers collect as much information as possible regarding a target system to uncover vulnerabilities, threats, and security issues. Metadata extraction plays an important role in this stage since it can reveal significant details about the target system, such as used technologies, software versions, user information, and network data, which can expose potential attack vectors. This paper introduces a novel framework for automated metadata extraction from documents linked within a specified web page. The framework is designed to streamline Information Gathering processes by offering an easy-to-use, integrated, extensible, and flexible solution. Our proposal can be effective in uncovering information that is not immediately visible to a penetration tester, giving them a greater chance of success in identifying the most fruitful attack patterns.