Understanding the Differences Between the Surface Web, Deep Web and Darknet

UNDERSTANDING THE DIFFERENCE BETWEEN THE SURFACE WEB, DEEP WEB, AND DARKNET DarkOwl Analysts break down the various layers of the internet, from “regular” search engine-compatible websites to complex hidden networks.

Copyright © 2022 DarkOwl, LLC All rights reserved. TLP: WHITE

Intro The internet, social media, and mobile devices are the fundamental requirements for conducting business and engaging in society. Whether checking email, catching up on industry news or accessing customer information, most of us use the internet (and the deep web) throughout the day, every day, in a variety of capacities. But, do we understand how it works – technically – even at a basic level? Do we understand the differences between the internet and the deep web or what it means to go even darker into the decentralized anonymous networks like the darknet?

Below is a breakdown of the various layers of the internet, from “regular” search enginecompatible websites to complex hidden networks.

The Internet The term internet is short for internetwork, which is a system created by connecting any number of computer networks together. An internet allows for communication between devices that are a part of that internetwork.

The internet is the most well-known example of an internetwork. This is the internet that we find indispensable to our daily lives, and it links billions of devices across the world through a network of networks using standardized procedures or protocol. The traditional server client1 architecture and HTTP protocol is the backbone of the internet and used extensively in websites and mobile applications.

Browsing websites on the web is not the only way in which information is shared via the internet. Email, instant messaging, and file transfer protocol (FTP) are other ways to share information like emails, messages, and files.

To clarify, the web is not synonymous with the internet and should not be confused with it. The “world wide web2” is simply a way of accessing websites over the medium of the internet.

Copyright © 2022 DarkOwl, LLC All rights reserved. TLP: WHITE

1

The Surface Web The websites we browse each day make up only a small percentage of the internet.

These sites, collectively known as the surface web (or “clearnet”), are visible and accessible to common search engines such as Google and Yahoo. Youtube videos, blogs, Instagram are all examples of surface web content most interact with every single day. While estimates vary, many experts agree that the surface web comprises roughly 4% of all online content. For more reading on how search engines crawl and index web content, there are several articles that describe systems like Google3 in detail.

High Risk Surface Web High risk surface web consists of areas of the surface web that have a high degree of hosting criminal or illicit content. Many of the users of the high risk surface web also maintain access to other, darker networks and communities.

This includes some “chan”-type imageboards4, transient paste sites, and other select non-authenticated forums that mirror dark web sites with surface web top level domains5 (TLDs).

While .com domains are the most common website domain, DarkOwl regularly tracks various TLDs that are popular with criminals. Our analysts have observed an increase in .top, .ru, and .cc TLDs. Many high risk surface websites popular with Chinese threat actors end in the TLD .cn.

To learn more about Chan Imageboards, check out our blog “Chan Imageboards Proliferate on the Darknet”

Below the Surface Beyond the surface web, an estimated 96% of online publicly accessible content is hosted in the deep web and the darknet.

The Deep Web

The deep web consists of website THE MAJORITY OF content that cannot be found or CONTENT RESIDES directly accessed via surface web IN THE DEEP WEB... search engines such as Google AN ESTIMATED and DuckDuckGo. Examples of 96%. deep web sites include websites that require any authentication credentials, such as email address and password, unlinked sites that require the direct URL to access, sites that are purposefully designed to keep search crawlers out, and databases. The majority of content resides in the deep web.

Copyright © 2022 DarkOwl, LLC All rights reserved. TLP: WHITE

2

Deep web databases commonly have their own search functionality which allows users to access the data contained within them. Government databases, patient medical records, and library catalogs are just a few examples of deep web databases. While these databases do not always require login credentials, many of them do.

Banking website portals for accessing account holder data and credit card statements are technically in the deep web because most banking websites will not allow access to their sensitive servers without authorization. Most social media is technically deep web content.

A specific example is the Denver Property Taxation and Assessment System6 website which allows users to search property assessment and tax data by entering a Denver-based address into the system. However, if you enter this same Denver-based address into a Google search (and even include terms such as ‘property assessment’ or ‘tax data’), you will not find any documents or URL results from the Denver Property Taxation and Assessment System website. This database and its search functionality are one example of a deep web database that is hidden from surface web search engines and technically resides in the deep web.

The Darknet and The Dark Web Beyond the deep web is the darknet.

The darknet is any anonymous network, built on top of the internet, that is purposefully hidden, meaning it has been designed specifically for anonymity.

Unlike the deep web, the darknet is only accessible with specialized tools and software – browsers and other protocols beyond direct links or credentials. You cannot technically directly access the darknet by simply typing a darknet address into your web browser, even though browsers like Brave7 offer private tabs with Tor for enhanced privacy.

Most people associate the darknet with Tor, but Tor is one of many darknets available. Let’s explore some of these darknets in more detail:

Tor, or The Onion Router8: an overlay network comprised of volunteer-operated servers that help route internet client-server traffic to provide obfuscation to users accessing the network. Theoretically, Tor is like a virtual private network (VPN) but with a VPN the servers are centralized to the VPN provider. With Tor, users connect through a series of virtual tunnels rather than making a direct connection between a client and server. The identity of the client IP address is unknown to the requesting website server, and the IP address of the server is only known to the exit node but not the originating client.

I2P, or the Invisible Internet Project9: an anonymous overlay network – using the distributed peer-to-peer10 (p2p) model – intended to protect communication from surveillance and monitoring. It was designed as a self-contained internet and behaves much like an ‘internet’ inside the internet. The I2P router relies heavily on Java to I2P browser, but i2pd is built on C++ for those adverse to Java. I2P routing is described as “garlic routing” where Tor is “onion routing.” It originated in 2003 as a ‘fork’ of Freenet.

Copyright © 2022 DarkOwl, LLC All rights reserved. TLP: WHITE

3

ZeroNet11: another example of a decentralized peer to peer network built on the Blockchain that functions as a darknet. Zeronet relies on BitTorrent network trackers to resolve network handshaking between peers. Instead of IP addresses in the network, nodes are assigned their own public key, more specifically a Namecoin12 (Bitcoin) cryptography address. The entry IP address of the network user is not technically private and Zeronet developers offer bundling Zeronet with Tor13 for additional anonymity.

Freenet14: distributed, peer to peer anonymous network which allows users to anonymously share files, browse and publish “freesites” (web sites accessible only through Freenet) and chat on forums. It is a distributed ‘data store’ allowing the content to be available on the network even though the originator or publisher is no longer on the network. Communications by Freenet nodes are encrypted and are routed through other nodes to make it extremely difficult to determine who is requesting the information and what its content is. The distributed data store nature of the Freenet environment is ideal for microblogging and media sharing, but also puts the users of the network at risk of unknowingly hosting illicit or CSAM content as encrypted fragments of media are stored on the hard drive of every user in the network.

Lokinet15: another example of an overlay decentralized network built on the Blockchain that serves as a darknet providing enhanced anonymity and privacy. Lokinet relies on a multi-hop low-latency onion routing protocol16 (LLARP) for routing traffic but is not limited to TCP traffic for serving HTTP requests. Since Lokinet sits in the network layer, it can handle and IP-based protocol like UDP and ICMP making it a highly secure option for web-based video and voice conferencing applications. The Loki network is developed atop the Oxen17 Blockchain which also hosts the end-to-end encrypted chat application, Session.

Yggdrasil18: an IPv6 overlay fully encrypted mesh network19 where each node in the network is assigned a cryptographic public key (like Zeronet) and routing is highly adaptable, using spanning trees for synchronization. Keys are assigned in alignment with the most optimized routes between all network participants. Yggdrasil nodes serve as routers and paths are built automatically without any data store or shared address book of the network. Yggdrasil is still considered ‘proof of concept’ and is designed as a more scalable version of CJDNS.20

Navigating these networks can be frustrating and challenging for any OSINT/Darknet investigator and the public often incorrectly uses the terminology associated with these different layers of the internet.

Any website that hosts or serves illicit content whether it is in the surface web, deep web, or darknet is technically a segment of the “dark web.” Dark web and darknet are often used interchangeably by us and other information security researchers.

The darknet is home to a diverse group of users with complex lexicons that often overlap with the hacking, gaming, software development, law enforcement communities, and more. DarkOwl’s Glossary of Darknet Terms is a continually evolving resource that defines the common vernacular, slang terms, and acronyms that our analysts find in places like underground forums, instant Check out our Glossary of messaging platforms (such as Telegram), as well as in information security research pertaining to the darknet.

Darknet Terms >> Copyright © 2022 DarkOwl, LLC All rights reserved. TLP: WHITE

4

SOURCES & RESOURCES

DARKOWL DATA SOURCES

https://toolsqa.com/client-server/client-serverarchitecture-and-model 2 https://en.wikipedia.org/wiki/World_Wide_Web 3 https://www.google.com/search/howsearchworks/ 4 https://www.darkowl.com/blog-content/chan-blog/ 5 https://www.techopedia.com/definition/1348/top-leveldomain-tld 6 https://www.denvergov.org/property 7 https://brave.com/new-onion-service/ 8 https://www.torproject.org/ 9 https://geti2p.net/en/ 10 https://www.blockchain-council.org/blockchain/peer-topeer-network/ 11 https://zeronet.io/ 12 https://www.namecoin.org/ 13 https://zeronet.io/docs/faq/#how-to-use-zeronet-with-tor 14 https://freenetproject.org/ 15 https://docs.oxen.io/products-built-on-oxen/lokinet 16 https://github.com/majestrate/llarp/tree/dev/docs 17 https://docs.oxen.io/ 18 https://yggdrasil-network.github.io/about.html/ 19 https://en.wikipedia.org/wiki/Mesh_networking/ 20 https://crypto.pkt.cash/updates/cjdns-how-compactsource-routing-is-more-efficient/

Tor, I2P, ZeroNet, authenticated

1

Copyright © 2022 DarkOwl, LLC All rights reserved. TLP: WHITE

forums, darknet marketplaces, IRC, high-risk paste sites, encrypted chat services, and open FTP servers.

5

ABOUT DARKOWL DarkOwl uses machine learning to automatically, continuously, and anonymously collect, index and rank darknet, deep web, and high-risk surface net data that allows for simplicity in searching.

Our platform collects and stores data in near realtime, allowing darknet sites that frequently change location and availability, be queried in a safe and secure manner without having to access the darknet itself.

For more information, visit www.darkowl.com

Copyright © 2022 DarkOwl, LLC All rights reserved.