Searching is one of the core activities people do online. We’ve been posting about it since our introductory post about Google and searching. But in spite of the fact that most people use Google to search there are actually plenty of choices, including some specialized choices. We saw that in our previous post about searching the Internet of Things, and again in the next post when we followed up by discussing a couple viable decentralized search engines.
In this post and in future ones we’ll talk about an area of growing interest in search: Open Source Intelligence (OSINT) These are the tools with which people try to discover information about companies, people, domains, etc. from open source tools that are freely available. Let’s begin with domains and websites.
In the previous post, in case you missed it, we covered a variety of privacy focused search engines. There are more, but we omitted many of them because they seemed less interested in privacy than in simply cashing in on the trend. By their very nature these posts have lots of links. While they are not endorsements, they do represent a curated set of sites that we consider to be interesting or useful. Here again there are an overwhelming number of similar websites so we’ve tried to curate a small number of the most useful sites.
For those who aren’t sure the Domain Name System (DNS) is that plumbing that allows us to assign memorable names to Internet hosts instead of remembering the IP addresses. This is useful information in the security industry, as what servers and services a company exposes to the Internet is telling. One of the main items people look for in DNS records are the WHOIS records, which are a database of registrant information.
A longtime favorite resource of ours for DNS information is the DNSDumpster.com site, that provides detailed DNS information well beyond what a simple dig query would. Another classic is DNSstuff.com which has a variety of DNS and IP lookup tools.
Another nice set of tools can be found at ViewDNS.info where a complete set of utilities is available plus an easy way to check if a site is visible from a variety of locations inside strictly firewalled countries like China and Iran, as well as convenient tools like checking if a site is configured for DNSSEC.
It’s often useful to use programs to help enumerate subdomains, to find more content, other websites and more. There are websites that will do this as well, including SecurityTrails site, which seems reliable and has an API that can be used in programs for paid accounts.
If you’re setting up a phishing site as part of training, pen testing or red teaming exercises, you’ll want a convincing domain name. Domize will let you search for available domain names using regular expressions and finding words that are synonyms. Also worth checking out is dnstwister, which will find similar domains.
Some allow users to search through historical data like DNS registration and WHOIS records, along with historical changes to the WHOIS records, the transfer history and other domain names associated with the same registrant or web server. Before GDPR, ICANN pushed hard for making domain ownership details like contact information public, and it is still available for those willing to pay a fee.
ICANN has it’s own WHOIS search that includes a link to the relevant registry server, but Domaintools remains the classic choice with extensive WHOIS history and lots of data to drill down and discover. Another great resource is domainbigdata.com which makes it easy to drill down and find other domains with same owner or other sites hosted on the same IP.
OSINT searches often aim to discover information about company websites like the technologies they use for development, what kind of server software they use to answer HTTP requests, how they divide their assets among servers, how they load balance, what database technologies they use, Which CDN providers they use, what IP address(es) they use and if it’s shared, what areas of the website might be not linked from the main site, admin areas, what type of resources are hosted there and so on.
Old versions and deleted websites can be searched using archive.org’s Wayback Machine. This can provide insights and even let you find resources that are still online, but no longer linked to. Also check with Archive.is if you can’t find the site/version you’re interested in.
Website technology is always useful to know if you’re planning a pen testing exercise, and an oldie but goodie is the Netcraft website. For years they’ve surveyed the Internet compiling statistics about servers and server software. A similar site is at BuiltWith.com where they provide a variety of server information.
There are of course, so many more topics to discuss in the world of searching and OSINT, we can’t possibly do them all justice. Yet we’re going to continue to try, so stay tuned!