Sourcing with Semantic Search: The Holy Grail of Recruiting Researchers

Right now, we're only ankle deep into the Deep Web.  When a sourcer uses a search engine, we're only seeing a fraction of the content that exists on the public Internet, because much content is dynamically-generated pages from online databases that web crawlers / bots are unable to spider.  While search engines and various other software companies work on that problem, there's another huge opportunity right under our noses that is being developed in parallel: semantic search.

Semantics is the field of study that focuses on meaning that's inherent in symbols, words, phrases, sentences and larger blocks of text.  A perfect semantic search engine would instantly and automatically take into consideration the meaning behind your question or "search string." For example, if you type Apple, it would know if you meant the fruit, the company, its products or the Beatles' former music recording entity, and deliver the appropriate results.  In a slightly more complex example, searching for a combination of job titles, companies and skill-related keywords would deliver only job listings to job-seekers and only resumes or people profiles to recruiters.

Search engines like Google will show different results over time if you stay logged in, but it's using statistically-based search driven by popularity of results (also how Google's "page rank" works), but that's not semantic search at all. Popular pages are not necessarily credible, and credible sources are not always popular.

One of the main problems with implementing true semantic search has been that it is difficult for the computer to know who you are (or in the example above, if you are a job seeker or a recruiter) unless the search engine can learn from your behavior and your previous selections.  Alternatively, you could manually indicate a category for it to narrow down your results into categories of meaning. To do this today, so-called semantic search engines ask you to tag, catalog, sort and otherwise try to "train" the search engine, which is far too time-consuming for the average user who, like me, is very lazy.

So why is this important to you?

Well, if a computer knows what you mean right away, without having to learn from you or be trained by you, it would give you only relevant results and omit all that other junk you have to manually sift through today. We're getting there, but we're not close enough yet.

As a human, if you read "stair well," you don't imagine an oil well inside a stair, or if the stair is feeling well, or all the variant noun, adverb and idiom forms -- you immediately know what it means. Computers, on the other hand, have to calculate dozens of variations and probabilities in order to arrive at a best guess. I'm sure you've looked up words in the dictionary only to find they have a few completely different definitions, sometimes even more!  Disambiguating them is easy for us humans because of context and subtext, but not very easy for computers.  In an old blogpost, I mentioned a few tools that help with this a bit.  But if it were really easy for computers to disambiguate subtext then they would be able to create original humor, which we humans frequently do by altering the "meaning" around what would ordinarily be a straightforward comment. Check this out, for example!

Someday, search engines may be able to infer meaning from the pages they index. I'm waiting, with bated breath, for a solution that approaches the artificial intelligence needed to successfully extract this from pages.  In a future white paper, I will discuss many of the promising public and private solutions that are starting to deliver parts of the semantic holy grail.