Cyber Security
Time for Open Source Intelligence and the ‘Deep Web’ By Tyson Johnson Vice President of Business Development, BrightPlanet
Introduction “Deep Web” is a vague description of the Internet not typically accessible by search engines. The Deep Web is often misinterpreted as the “Dark Web” and the two terms get frequently interchanged in media. While browsing the Internet, the Deep Web is usually right in front of you, you may just not notice it yet. Whether you are looking through thousands of unstructured Web pages or trying to answer narrowly targeted questions, the Deep Web and Surface Web co-exist and can help you answer some of your toughest security questions from, “Where is the next protest taking place?” to “Whose selling my companies goods online fraudulently?”. To understand how to leverage Open Source Intelligence (OSINT) from both the Surface Web and the Deep Web, it’s important to understand first where they are and what you can find there. Deep Web vs Surface Web: The difference The Deep Web is a part of the Internet not accessible to link-crawling search engines like Google. The only way a
36 | Australian Security Magazine
user can access this portion of the Internet is by typing a query into a Web search form, thereby retrieving content within a database that is not linked by standard Web pages. In layman’s terms, the only way to access the Deep Web is by conducting a search within a particular website. The Surface Web is the portion of the Internet that can be found via link-crawling techniques. Link-crawling means connecting via an HTML hyperlink from one page to another. Google can find this Surface Web data very easily. Surface Web search engines (Google/Bing/Yahoo!) can lead you to websites that have unstructured Deep Web content. Think of searching for Government Court Cases at the Common Wealth Courts Portal (https://www.comcourts. gov.au/public/esearch). Google can take you to the portal page, but it can’t find the results of your searches within the Courts Portal. By entering a search query into this database, you are completing a Deep Web search and finding Deep Web content. There are millions of disparate sources online today that contain Deep Web information; anything from government databases, travel sites, Web pages requiring logins, and even some social media pages.