Specialized Search Engines

Monday , 26, August 2019

We would be remiss if we did not mention specialized search sites in one of these posts. There are so many websites that index links in a particular industry or links to specific types of documents like patent applications. They don’t fit neatly into other categories but are incredibly useful for people with specific research needs and niche industry players.

There are of course specialized job search websites aplenty, but they’re annoying and chocked full of trackers and their entire goal often seems to be to collect personal information that is specific and current for the sole purpose of resale to data brokers. Same can be said for various shopping comparison and coupon sites. You know who they are already if you care, so we’re just going to move on.

So herein we have a loose collection of varied resources similar to our privacy-focused search engines post, making this one more like a listing or a directory than others we’ve done like posts on Google search, or the post dedicated specifically to decentralized search but it’s important to note that none of the links in this post are afilliate links. The next couple of posts will be more along the lines of one of my favorites in the series, the Shodan-heavy post about Internet of Things searching, and the previous post about various search resources to use in gathering OSINT website and DNS information. Let’s get started with the rundown of various content rich services.

Directory of Open Access Journals (DOAJ) is a directory indexing over 13,000 journals, where you can search through millions of articles and find topical journals. According to their website, DOAJ “indexes and provides access to high quality, open access, peer-reviewed journals” about Agriculture, Medicine, Law, Technology, Education, Fine Arts and much more.

In fact there are numerous search portals specializing in locating publications like white papers and peer reviewed journal content. Google Scholar is one such resource, bolstered by the sheer horsepower of Google search. Specializing in technical and IT topics, FindWhitePapers.com is one such resource to keep bookmarked. For whitepapers published by various cryptocurrency projects try Whitepaper.io – but don’t neccessarily invest in them! On the other hand, billing themselves as a resource for HR, IT, finance and marketing whitepapers is Whitepapers Online.

In more general searches for files, there are sites like faganfinder.com that allow searching for specific filetypes. This is useful for finding specific type of files like PDFs, Word docs, etc. on company websites. They use Google dorks to find these files, and if nothing else you can learn the specific file extensions to search for here. In fact they have a number of specialized searches on the main site, including quotations and legal searches.

Takedown notices, on the other hand sould probably be searched for in a dedicated site like Lumen, that was founded by EFF and Harvard University Law School. The biggest takeaway for me after doing some sample searches on the site was just how many takedown notices Google receives; it is a staggering total, and leaves no doubt that they have collected millions of these notices to remove content from the web.

Airfare search sites are their own niche, different from others partly due to the realtime nature of pricing data for flight reservation systems. There are booking engines, airlines (Southwest) and what amounts to basically travel agent sites that scrape the others. Beware of bait and switch tactics too, as some seem to have different ideas about what is a non-stop flight or may even show you a slightly lower price before you choose.

Expedia is one of the perennial favorites, according to Alexa but that may or may not translate into finding better rates. They bought their main competitors Travelocity and Orbitz if that matters to you. HipMonk is a smaller entry in this space that has a nice layout. Shown above is one of the screens from SeatGuru site, which has loads of these plane interior layout images. There are other top-tier sites like Hotwire and Kayak, as well as numerous others that seem competitive like CheapOAir, SkyScanner and Momondo. Unfortunately we can’t dive into detail about them all or it would consume an entire post on its own.

Real Estate search websites hav carved out territory all their own, so let’s take a look. Zillow, Trulia, and Realtor.com are probably the best known of these, but many others rank highly and get a lot of traffic. I don’t want to get into compaisons here, since like airlines and hotel sites, there is more than enough to say about them to fill a lengthy blog post. Some of the other heavy hitters include Homes.com, Redfin, Yahoo Homes, Curbed, ApatrmentGuide, ReMax, HotPods, ZipRealty, HomeFinder.com, Apartments.com, Rent.com, ForRent.com, and Zip Realty. These sites had the highest traffic totals as of August 2019 according to Alexa – which may not be extremely accurate but should be in the ballpark.

I decided not to even post links to all those sites above because it’s just too many, and not useful. They promote themselves well in this competitive space, and you can easily search for one of them when need be. Having mentioned all those sites, I think it’s worth mentioning that LoopNet has long been a leading venue for commercial real estate searches.

Let’s move on to something unique instead of a category with a zillion competitors. Wolfram Alpha offers an intelligent search like no other. Ask it questions that you might pose to a teacher and it will often give terrific, detailed answers. Try it out with math, science and engineering questions, or art, history and health questions. It’s an amazing resource unique among search engines.

How about a completely different category like searching for maps of phone coverage dead spots? We found great results at this site for the US market, and here for the UK. Of course some of the cellular providers have maps of their own coverage on their sites but this is a convenient way to see those areas where their coverage doesn’t reach reliably.

It should be mentioned also that search preferences vary greatly based on location. Baidu is the leading search engine in China for websites and images, for example. Here is a listing of some popular search engines that specialize in local (and local language) content and websites.

Search EngineCountry
ChinaBaidu, Yam, Youdao
RussiaYandex, Rambler
GermanyDas Oertliche
JapanGoo, Biglobe
ItalyVirgilio
South KoreaNaver, Daum

Since this post is getting a bit long, I’ll admit to getting a bit hungry, so let’s mention a nice foodie search engine called Yummly that does a bit of handholding to help you find recipes. It’ll ask you some questions to narrow down the result set or to help you make decisions about how to cook something, while showing you the number of results there currently are. Be ready with a disposible email address though – they walk you through the entire process before asking for an address before letting you see the results.

Well there are plenty of other categories, but in the interest of preventing this from becoming the longest post ever, we’ll just leave it here. Besides, some of the other sites can be dealt with in more detail, since some of those include OSINT search resources for finding information about people and companies. We will definitely be covering those in greater depth in future posts, so stay tuned!