AI and Discovery: Semantic Search, Vector Search, Neural Search and Beyond

Artificial Intelligence (AI) has been one of the most rapidly growing fields in the technology industry in recent years. Within this realm, the emergence of powerful models like the ChatGPT (Generative Pre-trained Transformer) has been a game-changer in natural language processing. The GPT-3 model, developed by OpenAI, is a neural network that can generate human-like responses to textual prompts. However, as with any emerging technology, there are limitations and concerns that need to be addressed.

At Klevu’s Discovered London event, Niraj Aswani, one of the co-founders and the CTO of Klevu, spoke about the current state of AI and where it’s heading.

Niraj, who has been working in AI since 2002, stressed the importance of understanding how we have reached this moment with AI and what it means to run these models in a large cloud infrastructure. He also emphasized the importance of understanding some of the vocabulary that is used in AI, such as chat boards, vectors, and embeddings.

Different types of search

Niraj highlighted some important vocabulary related to search. He explained that while most search engines use keyword-based search, which means search engine will go through its index and whichever document has the exact keywords from the query, they will come out as a result.

On the other hand, we have neural and semantic search which do not rely on keywords, but they rather mainly focus on the meaning of those keywords to find similar results in the index.

For example, if someone searches for “Italian food,” a neural search would suggest Italian restaurants based on the user’s location, while a semantic search would take into account the user’s preferences for the type of Italian food they enjoy. Simply put, intention of the semantic search is to make user’s wish come true. And depending on that, it will try to get the most relevant results.

Chatbots and conversational AI

Niraj then continued on with chatbots and conversational AI, which both try to make it possible for humans to interact with computers in a conversational manner.

Chatbots produce more static responses, while conversational AI is more advanced as it understands human language and produces natural responses. The conversational feel is very natural, and that’s why we can all look at the ChatGPT as a wonderful example of conversational AI agent.

Generative AI

Generative AI and transformers are becoming increasingly popular in the field of artificial intelligence. Generative AI, as the name suggests, is a type of AI that generates new things such as text, images, and videos.

The technology behind it helps in creating content that is unique and not seen before. Transformers are one of the neural network technologies that help in generating such content. They have a special architecture that understands the context around the task, which helps in producing more relevant and meaningful content.

Niraj stated that this technology has proven to be very useful in various industries like gaming, creative content, and marketing. With the advancement in technology, generative AI and transformers are becoming more accessible and easier to use. These developments have opened up a world of possibilities and are changing the way we interact with AI.

ChatGPT and Beyond

One of the main benefits of the ChatGPT model is its ability to learn from user feedback. During the development phase, the OpenAI team interacted with the machine for six months, giving feedback on correct and incorrect responses. This feedback helped the machine learn and improve over time. The prompts used to train the model varied from how to write an essay to how to solve a problem. With time, the machine learned how to understand these questions and provide accurate answers.

Despite the incredible power of the ChatGPT model, there are limitations to be aware of. One of the main limitations is the possibility of feedback bias. The individuals who gave feedback to the machine may have had their own biases or limitations in their understanding. Additionally, the machine does not know what it is doing, which means that its responses should always be taken with caution.

One promising application of ChatGPT and other AI models is in e-commerce. By using these powerful algorithms, businesses can enrich their catalogues and provide better recommendations to customers. Another exciting development is the ability of AI models like DALL-E 2 to generate images from textual prompts.

While ChatGPT is one of the most well-known AI models, there are hundreds of other models available, each with its own unique strengths and applications. The Hugging Face company has over 100,000 models in their repository built by the open-source community. With so many models available, the future of AI is likely to involve combining these models to create even more powerful systems.

The Future of AI

AI and models like Chat GPT have already had a significant impact on the technology industry. As more businesses adopt these technologies, we may see a future where there are only two types of companies: those that have adopted AI and those that do not exist. While there are limitations and concerns to address, the future of AI is exciting and promising. By learning how to ask questions to these machines, we can make the most of their incredible capabilities.

As we move forward, the focus will be on combining different AI models to create something truly extraordinary. Imagine a robot that not only has the power of ChatGPT but also understands how to analyze images and combine that knowledge with textual interpretation. This is already being demonstrated in the latest release of GPT4. However, as we continue to advance, there will be a greater need for investment in AI security to ensure that the information being produced is secure.

Behind the scenes, companies like OpenAI are helping to speed up the development of these technologies. Today, we are already seeing the use of conversational AI agents in shopping, as demonstrated by the launch of conversational commerce chatbot MOI. The future of AI begins here, and it’s up to us to leverage these technologies to create a better world.