If 2023 is the year of generative AI-powered chatbots and search, 2024 is about AI agents. What started from Devin earlier this year has become a full-fledged phenomenon, offering businesses and individuals a way to change how they work at different levels, from programming and development to personal tasks. such as planning and booking tickets for a holiday.

Among this wide range of applications, we have also seen the rise of data agents this year – AI-powered agents that manage different types of tasks across the data infrastructure stack. Some do basic data integration work while others handle downstream tasks, such as analysis and pipeline management, making things simpler and easier for business users.

The benefits are improved efficiency and cost savings, leading many to wonder: How will things change for data teams in the coming years?

Gen AI Agents take over data tasks

While agent capabilities have been around for a long time, allowing businesses to automate some basic tasks, the rise of generative AI took things completely to the next level.

With gen AI’s natural language processing and tool usage capabilities, agents can go beyond simple reasoning and response to actually plan multi-step actions, independently interacting with digital devices. system to complete actions while cooperating with other agents and people at the same time. They will also learn to improve their performance over time.

The Devin of Cognition AI is the first major agent offering, enabling engineering operations at scale. Then, the big players started to provide more targeted business and personal agents powered by their models.

In a conversation with VentureBeat earlier this year, Gerrit Kazmaier of Google Cloud said he’s heard from customers that their data practitioners often face challenges including automating manual work for data teams, reducing the cycle time of data pipelines and analysis and simplifying data management. In fact, teams are not short on ideas for how they can create value from their data, but they lack the time to implement those ideas.

To fix this, Kazmaier explained, Google is revamping BigQuery, its core data infrastructure offering, with Gemini AI. The resulting agent capabilities not only give businesses the ability to discover, clean and prepare data for downstream applications – breaking down data silos and ensuring quality and consistency – but also support management. and pipeline analysis, freeing up teams to focus on higher-value tasks.

Many businesses are now using Gemini’s agentic BigQuery capabilities, including fintech companies More thanwhich taps into Gemini’s ability to understand complex data structures to automate the query creation process. Japanese IT firm Unery also leveraged BigQuery’s Gemini SQL generation capabilities to help its data teams deliver insights faster.

However, discovery, preparation and analysis assistance are only the beginning. As the underlying models evolve, even granular data operations – pioneered by startups specializing in their respective domains – are being targeted by deeper agent-driven automation.

For example, AirByte and SPEEDILY creates category headings data integration. The former launches a helper that creates data connectors from API documentation links in seconds. Meanwhile, the latter improves the wider offering of application development agents that create business-grade APIs – whether it is for reading or writing information on any subject – using only natural language description.

Based in San Francisco Ultimate AIfor its part, it focuses on various data operations including documentation, testing and transformation, with the new DataMates technology, which uses the AI ​​agent to extract context from the entire data stack. Many other startups, incl Red bird and RapidCanvasis also working in the same direction, claiming to offer AI agents that can handle up to 90% of the data tasks required in AI and analytics pipelines.

Agents that activate RAG and more

Beyond broad data operations, agent capabilities are also explored in areas such as retrieval-augmented generation (RAG) and downstream workflow automation. For example, the team behind the vector database Weaviate recently discussed the idea of agent RAGa process that allows AI agents to access a wide range of tools – such as a web search, calculator or a software API (such as Slack/Gmail/CRM) – to retrieve and -validate data from multiple sources to improve the accuracy of answers.

In addition, at the end of the year, Snowflake Wisdom appears, giving enterprises the option to set up data agents that can tap not only the business intelligence data stored in their Snowflake instance, but also the structured and unstructured data in siled third-party tools – such as sales transactions in a database, documents in knowledge bases such as SharePoint and information in productivity tools such as Slack, Salesforce and Google Workspace.

With this additional context, agents display relevant insights in response to natural language queries and take specific actions around the generated insights. For example, a user can ask their data agent to enter emerging insights into an editable form and upload the file to their Google Drive. They can even be prompted to write Snowflake tables and make changes to the data as needed.

More to come

Although we haven’t covered every application of data agents seen or announced this year, one thing is very clear: The technology is here to stay. As gen AI models continue to evolve, the adoption of AI agents will move at full steam, with most organizations, regardless of their sector or size, choosing to hand over the repetitive tasks of specialized agents. This directly translates into efficiency.

As evidence of this, in a recent survey of 1,100 tech executives conducted by Capgemini82% of respondents said they intend to integrate AI-based agents into their stacks within the next 3 years – up from the current 10%. More importantly, as many as 70 to 75% of respondents said they would trust an AI agent to analyze and synthesize data for them, as well as handle tasks such as creating and again improving the code.

This agent-driven shift also means significant changes in how data teams operate. At the moment, the results of the agents are not production-grade, which means that a person has to replace at some point to make the work better for their needs. However, with a few more advances in the coming years, this gap is likely to disappear – making AI agents faster, more accurate and less prone to the mistakes that humans often make.

So, to sum up, the roles of data scientists and analysts as we see them today are likely to change, with users possibly moving into the AI ​​oversight domain (where they can monitor AI actions) or higher value ones. task that the system may struggle to perform.