How Uber is Leading AI Race with GenAI Gateway?

👋 Hey, SK here!  Welcome to XTEN10X. I write about stories on product management , career growth and startups based on my observations and experiences.

Today, let’s talk about the story of Uber’s game changing AI milestone.

How many teams in your organization are using LLMs? I’m sure almost all the teams are in this AI race.

For every use case, one LLM vendor might not be the best option. Choosing the right LLM based on organizational needs is very crucial and it depends on factors like:

- Intended Use Case: Define core tasks (e.g., content generation, classification).

- Data Domain: Select models pre-trained on relevant data (e.g., healthcare, finance).

- Accuracy : Larger models offer higher precision.

- Inference Speed: Smaller models infer faster for real-time needs.

- Scalability: High query volumes can increase costs.

- Cloud vs. On-Premise: Cloud offers convenience; on-premise offers control.

- Budget Constraints: Balance cost and capabilities.

- Ethical Considerations: Assess model biases, safety, and misuse risks.

Well, some teams prefer to work with OpenAI, while others use Vertex, and there are many more.

Now, imagine this: Rather than having each team in your organization manage their own LLM credentials, how about having centralized access to multiple LLMs with guardrails?

Add to this: seamless integration and onboarding of new LLMs for easy accessibility across teams to develop new applications.

This is exactly what the GenAI Gateway does!

Uber has now introduced the GenAI Gateway, which offers many advantages beyond just being a serving component. This gateway helps Uber integrate large language models (LLMs), offering a streamlined interface that caters to various teams within the company.

Key Features of the GenAI Gateway

1. Multi-Vendor Compatibility:

By supporting multiple LLM vendors, the GenAI Gateway offers flexibility and choice, ensuring that teams can select the most suitable models for their specific needs.

2. Security Measures:

GenAI Gateway includes a robust Personally Identifiable Information (PII) redactor, enhancing data security and compliance with privacy regulations.

The PII redactor anonymizes sensitive information within requests before sending them to third-party vendors. Once responses are received from these external LLMs, the redacted entities are restored through an un-redaction process. This redaction/un-redaction method aims to minimize the risk of exposing sensitive data.

3. Developer-Friendly:

With support for several programming languages, including Go, Java, and Python, the GenAI Gateway is designed to be accessible and easy to use for developers across various disciplines.

4. Seamless Tool Integration:

The platform’s ability to work in harmony with existing tools like Uber’s Michelangelo machine learning platform ensures a smooth workflow and operational efficiency.

Real-World Use Case:

Uber enhances customer support efficiency by using LLMs. These models help summarize user issues and suggest resolutions quickly, improving the response time and overall user experience. LLMs are used to provide agents with crucial background information and user sentiments for empathetic support.

The outcomes are clear: The implementation of LLMs has significantly improved efficiency, with 97% of generated summaries effectively resolving customer issues. Agents report substantial time savings and a productivity boost, responding to users 6 seconds faster. Currently, around 20 million summaries are generated weekly, with plans to expand usage to more regions and contact types.

Before GenAI Gateway:

Uber expanded the CO Inference Gateway to support text, chat, and embedding generation through a new Generation ML Task. This service extension connects to OpenAI and Google Vertex AI models, enhancing flexibility and adaptability by integrating various machine learning tasks and abstracting different ML model hosts for internal services.

After GenAI Gateway:

To address challenges such as PII redaction, cost attribution, and the need for a centralized service to connect with external Language Models (LLMs), Uber chose to use the GenAI Gateway. This strategic decision ensures robust and secure integration, effectively navigating complexities and optimizing AI-powered solutions, aligning with Uber’s commitment to innovation and excellence in customer support.

Looking Ahead

The GenAI Gateway is not just a technological advancement; it represents a strategic move towards a more integrated and intelligent operational framework. While this is a good move, there are still some challenges which Uber is facing:

1. Latency : Nature of scanning and anonymizing entire requests incurs added latency while serving the requests.

2. Quality : The PII redactor can also impact the quality of results. Its process of anonymizing sensitive data, while safeguarding user information, sometimes strips away crucial context, thereby affecting the response quality of LLMs.

3. Outage : In case of vendor outage events, there can be cases of misattributions of customer issues. Uber is incorporating the possibility of Uber-hostedLLMs as a fallback option.

References:

For a deeper dive into Uber's GenAI Gateway and its transformative impact, you can read the full article here.

That’s all for today, see you again.

Cheers,
SK