vendredi 13 juillet 2012

Surfing "the deep web"

Hello folks,

I've seen quite some articles on "the deep web" recently and I've read that it's like 550 times bigger than the actual "public web" so it kind of tinkled my curiosity.

The deep web is basically the web that is not referenced by search engines or not easily attainable.
There are multiples ways to hide a website from the eyes of search engines, TOR hidden services, Freenet, dynamic content, etc.

I mainly played with Freenet and here is the summary of my (short) exploration.

Surfing the deep web


Since there are no search engines for the deep web. It is quite particular as how  you surf on it.
The following has been experimented on Freenet but I'm sure you could apply some of it to other deep web services (TOR, etc).

Indexes


The common way to surf on Freenet is to make use of indexes such, some a re unmaintained and other are maintained:
Linkageddon
In The Raw
Another Index
The Public Index
AFK index
Active Link Index
Uncensored Index
Entry Point

Google hacking


Then you can of course find freesite link through Google hacking:
intext:"http://127.0.0.1:8888/usk@" OR intext:"freenet:usk@"

Pastebin


Same on pastebin:
http://pastebin.com/search?q=127.0.0.1%3A8888%2FUSK

Other ways of finding sites


Use what is at your disposal on Freenet:
Sone (social network), boards, etc.
Then you can write custom spiders for instance.

There are also wikis, forums, irc, etc.

What can we find there?


Anything since Freenet is a network distributed file system.
What is really interesting about Freenet though is that it is censor free, so even though you divulge information about something (like vulns, etc), it can't be taken down but by you.

Freesites/Flogs/etc


At first, I was quite disappointed since a lot of those freesites is shit, scumbags, paedophilia, etc. After searching a bit, you get to find good stuff:

XKCD
Fravia's archive pages of reverse engineering
Freenet Social Networking Guide
TI Calculator Signing Keys
AKG: Open Source Intelligence
Laputa Ebooks
Randomizer : Lockpicking
Without A Trace - A Forensics Manual for You and Me

Then there are emulation, abandonware, ebooks, etc, freesites.
I didn't find much freesites about hacking/reversing yet. If you have any interesting freesites, could be great to share ;).

Filesharing


Through boards and newsgroup, you can share files in Freenet. You could also find
long lost or forbidden software such as DeCSS (without this first DVD hack, actual software on Linux would still be unable to play DVD ;)), etc.


Conclusion


We can see that surfing the deep web is not as convenient and easy as the "public web" but it is possible with some patience.
In my usage, I found for now that there isn't anything on Freenet that I can't find on the internet.

The deepweb allow ways of expression, and in particular true for Freenet, that allows non censorship, complete freedom of information. This complete freedom of information comes with drawbacks for sure but you have the choice to ignore those weird/illegal/non ethical sites, nobody told you to go there in the first place.
Thus, I would mainly see Freenet as a mean to express and share information freely without censorship. It would be useful against repressing governments (China, etc) or in the current race for stupid laws (ACTA, SOPA, PIPA, HADOPI, etc, ... just die die die and kill those laws once and for all).

References

What is the Deep Web? A first trip into the abyss
The Deep Web
Wikipedia: Deep Web