ETR Insights presents an interview with the Head of Data Science for a large technology and business services enterprise that adopted generative AI early, though despite initial industry enthusiasm, there is a growing realization that generative AI cannot address deep-seated structural issues; there is a particular discrepancy between executive expectations and the actual capabilities of AI. This executive is skeptical of “prompt engineering” as a distinct discipline, seeing the skills as essentially those of a business analyst. They use both open-source models from OpenAI and select tools like Salesforce Einstein Analytics for specific use cases, within an otherwise generically unified Microsoft tech stack. Pinecone is praised for RAG, a quickly evolving but temperamental alternative to semantic search. Read on to learn more about why this firm still uses both Databricks and Snowflake; the coming disintermediation of ETL and data visualization vendors; and why companies still tend to favor managed solutions like Confluent over developing directly in Apache Kafka.
Vendors Mentioned: Amazon (AWS) / Confluent / Databricks / Elastic / Google / IBM / Meta / Microsoft / MongoDB / OpenAI / Oracle / Pinecone / Salesforce / SAP / Snowflake
Generative AI. Some firms are using open-source models from OpenAI and Anthropic, while others prefer to wait for their established vendors like Microsoft, Adobe, and Salesforce to introduce AI enhancements. Our guest uses OpenAI and Salesforce Einstein Analytics only for specific use cases with limited deployment. “If I want to do some use case that is across the entire organization, building something that's going to rival Copilot, then integrating it is going to be a nightmare. Microsoft 365 Copilot is perfect for that use case.” They describe one practical use case that involves processing structured email data to pre-populate entries using a general LLM model. “You would say, you could do it beforehand as well, right? But the problem was, sometimes the structure wasn't fully completely followed. It was a more natural language text, and it was very hard to extract data.”
Salesforce Einstein Analytics. Companies must weigh the cost of using tools like Salesforce against the scope of use, and expense of developing similar capabilities in-house. “Einstein Analytics is going to work in Salesforce. If the interaction is not going to be in Salesforce, Einstein Analytics isn't going to be that much of a use case.”
RAG. This company is currently deploying retrieval augmented generation (RAG), albeit through external providers. “We are not a tech company and we don't envision to be one, so we don't want to spend our time where it’s not our core function.” The RAG space is “exploding”; they reference Elastic, MongoDB, and Pinecone in particular, on top of existing Microsoft. “Pinecone, because we felt they had quite a great amount to offer considering what we wanted, and they were one of the very few who started RAG databases initially, so they have a lot of expertise there.” Our guest suggests that MongoDB might be a good match for companies already using them for unstructured data storage, but this organization doesn’t use them.
RAG offers quick insights and can sometimes outperform semantic search, yet its value proposition is complex and often difficult to justify. “It's a very niche tool. Not everybody has experience with building the integrations and APIs and managing the entire tech stack. The skill set out there is limited. Semantic search is way older; it's easier to build it.” This company’s unstructured internal data sets add to the complexity; benchmarks standards are constantly shifting; the interaction between different foundational models further complicates.
RPA. Our guest questions RPA’s ongoing value, given its dependency on structured systems and high maintenance needs when business processes evolve. “The value saving is there if you have something that just stays as is, but as in any business, nothing stays as is. You need something that's fluid, that stays fixed only for a specific amount of time - and that's what the new gen AI solutions capabilities are.”
ETR Data: In ETR's April 2024 State of Generative AI Survey, those indicating that their Generative AI budget was redirected from RPA spend has increased by 2 percentage-points to 9%. Polling for the July iteration of this survey is currently in the field; we will continue to monitor this metric and publish the full results in the coming weeks. Contact the service team if you would like access to the upcoming State of Generative AI report.
Data architecture. This company prioritizes a unified tech stack, primarily Microsoft, Databricks, and some Snowflake, while phasing out Oracle. Our guest admits it is unusual to have both Databricks and Snowflake on board; competition from the Big Three cloud providers has pushed both to innovate and expand their services. “Snowflake was requested based on preferences of certain segments of businesses that had experience, and then it's too much of a challenge to upgrade them to another set of tech stacks.” Our guest is generally satisfied with a single cloud platform. “We don't have anything that's so critical that it can't go down even for a minute. For us, it's fine.”
Open source. ETR data indicates Confluent has been gaining significant traction, particularly among Global 2000 enterprises. Users appear to favor Confluent over Apache Kafka itself. “[It’s like], why would somebody use Databricks and not Spark itself? It's a similar question. The answer is, open source is good sometimes, but the support or the proliferation and the way it's handled is not built for enterprises.” Open source also presents integration challenges; not every enterprise has the capacity or interest to learn and implement these technologies independently. “And enterprises feel better if they use something where they can have another partner to take the risk if things go bad.”
Disintermediate vendors. Niche ETL tools like Informatica are being subsumed by cloud providers offering integrated solutions. While Power BI remains embedded in Microsoft, and Streamlit well-adapted to open source, other visualization tools like Tableau are losing ground. Databricks already offers an integrated dashboard. “What's the use case [for Tableau] if my provider already gives me a dashboard at no extra cost?”
Economic Overview
Only a limited number of firms have transitioned from experimentation to live production with generative AI; success hinges on organizational maturity and leadership support at multiple levels. Our guest’s firm brought rudimentary AI-powered services online relatively early this year. “You might not have the best tech stack like a data science company or a mature tech company—you would expect we don't have that, we're not a tech company—however, we did jump on the gen AI bandwagon, because we saw how quickly it can deliver value without having to go through the entire tech stack itself.” They do, however, note some waning enthusiasm, as if “coming down off a high of a drug.” Companies are realizing generative AI is not a panacea for deep-seated structural issues. “It can mask a full person when it's responding to some parts, but it can't fix or understand the entire org structure, understand how to get Salesforce data from one point, and then to understand why the order is late, etc. All those details, it can't do that.”
While large enterprises typically have fixed budgets based on a percentage of their revenue, the rise of generative AI has forced many to reallocate funds, particularly from business applications, software, and productivity tools. “It’s not just IT having the budget itself, but it's also sometimes business is saying, ‘I recognize the market is shifting. If we don't put money into this right now, we're going to be playing catch up, and it might not be relevant here by that point in time.’” However, companies struggle with uncertain return on investment and the difficulty of integrating new technologies into an existing system, making long-tail investments impractical for some. “We've tried to pick use cases for gen AI which either give value immediately, or we say gen AI might not be the answer for us right now, because it's too new as a technology.” There remain multiple barriers to full-scale production, including legal, ethical, and security concerns, especially in markets with strict AI regulations.
ETR Data: The clear leading reason why organizations have not moved generative AI / LLM evaluation into production is that they are still in the evaluation phase (83%). Data privacy or security concerns (38%) and legal, compliance, or regulatory concerns (37%) are the next most common reasons, and they have grown since last survey, with more than a third of respondents citing these issues. Notably, too, the proportion of respondents citing a lack of usefulness of generative AI for business tripled since last survey (from 2% to 6%).
Lastly, our guest is skeptical of the development title “prompt engineer.” “To me, it's a rebranded business analyst or data analyst, or whatever person you couldn't figure out what they should be doing. At this point, prompt engineering should be straightforward.” That said, there remains a disconnect between expected and actual outcomes, particularly at the C-level. “Executives expect a standard, definitive answer, while with gen AI it's hard to get a very consistent response.”
The ETR Transcripts Library is filled with hundreds of interviews with experts in our ITDM community just like this one. If you don't have access to ETR Inisghts, please reach out to our service team or start your own free trial today.
Enterprise Technology Research (ETR) is a technology market research firm that leverages proprietary data from our targeted IT decision maker (ITDM) community to provide actionable insights about spending intentions and industry trends. Since 2010, we have worked diligently at achieving one goal: eliminating the need for opinions in enterprise research, which are often formed from incomplete, biased, and statistically insignificant data. Our community of ITDMs represents $1+ trillion in annual IT spend and is positioned to provide best-in-class customer/evaluator perspectives. ETR’s proprietary data and insights from this community empower institutional investors, technology companies, and ITDMs to navigate the complex enterprise technology landscape amid an expanding marketplace. Discover what ETR can do for you at www.etr.ai