Amidst the media blitz around ChatGPT, ETR has repeatedly been asked to provide their opinion on the runaway technology, including recent coverage in The Wall Street Journal. Luckily for our clients, we don’t have opinions; instead, we have data and evaluation commentary from our IT decision maker community. When relying on these two sources, ETR sees two distinct levels of ChatGPT adoption to analyze. One is in a testing manner and the other is in real, external-facing use cases. The sentiment around the two is starkly opposite.
As background, ETR recently completed its quarterly Emerging Technology Survey (ETS) of 1421 enterprise IT decision makers, in which we added OpenAI for the first time. At 496 citations, OpenAi was the 15th most cited vendor (among 488 vendors) behind much bigger, more pervasive vendors/technologies such as Kubernetes, 1Password, PostgreSQL, and Apache Kafka. Within the ML/AI sector, OpenAI was a clear leader in Mind Share, Net Sentiment, and overall evaluations. 41% of those 496 respondents indicated their organizations were currently or planning on evaluating OpenAI. On an absolute basis, that is 7x more evaluations than another vendor in the history of ETR’s Emerging Tech Survey series.
ETR Data: According to our February 2023 Emerging Technology Survey (ETS), OpenAI has a stellar showing in its debut ETS with leading evaluation rates and Mind Share. With sector-leading evaluation rates and citations, this is a promising initial pass at OpenAI’s potential traction in the enterprise. The ability to convert those evaluations into paying clients utilizing their products will be the ultimate litmus test if the reality will exceed the promise (hype).
Therefore, CIOs/CTOs are not waiting for the enterprise IT giants to embed ChatGPT into their services to evaluate the potential use-cases in their organizations. It is still early to determine if OpenAI is able to convert those evaluations into paying customers or if the value accretes to those embedding the technology as an additional function (ie SlackGPT, etc). Only 12% of those respondents said they plan to utilize or are allocating further resources to OpenAI post-evaluation.
ETR Data: OpenAI’s utilization rate of 12% is respectable for its first survey, placing it higher than most peers in ML/AI, lagging only TensorFlow, Databricks, and Anaconda. In the next installment of the ETS (MAY23), we can assess how successful the company is in converting evaluations into customers utilizing their products.
The technology is exciting but also so new that the development of real value-generating use cases will take time. The early adopters will likely come from startups and more agile companies but for the most part, larger (and regulated) enterprises will continue to keep this contained in the testing phase until it's integrated into larger systems.
In this specific instance, the source from where the solution is coming is less relevant because the technology itself is so new. Therefore, it doesn’t matter if this is coming from a long-time trusted vendor or an AI startup. Until some guardrails, controls, and security are wrapped around the technology, this will be relegated to safer internal use cases like knowledge sharing instead of external facing like virtual agents or customer support.
ETR Data: OpenAI leads all ML/AI peers in evaluation rates by a wide margin, a positive sign for the vendor’s customer pipeline. Since the first ETS in NOV18, only 19 other vendors have had a higher evaluation rate than OpenAI (31,817 total observations), however, on a substantially lower respondent citation base. Open AI received 396 respondent citations compared to an average of 60 across the 19 other vendors. On absolute level, the 205 respondents that are evaluating OpenAI is over 7x higher than the other 19 vendors mentioned.
The majority of CIO/CTOs would be cautious about exposing internal private data to ChatGPT or any other type of AI companies without an enterprise-level contract with guardrails built around it. However, for non-sensitive data, a lot of organizations across industries are very excited about the technology and are already testing it themselves.
IT Decision Maker Commentary on ChatGPT and Generative AI
In recent ETR Insights interviews, IT decision makers have reacted to the buzz around ChatGPT and potential varied enterprise use cases for generative AI technology:
I think that everyone in technology right now is trying to figure out how they're going to manage and integrate AI because our expectations of what AI is, what it could be, and what it should be has completely changed with ChatGPT, unless if that was something you were really honed in on. Any dinner meeting, lunch– heck, having dinner with family who know nothing about technology – the conversation is this AI capability with ChatGPT. I now have my analysts running all their code through ChatGPT to comment on code and document it.
My take, at least as it goes for my team, is that expectations for productivity and throughput should increase. I've set a target of 20% because if I can have those who are writing code debugging, documenting, commenting, and sharing information more efficiently, then I should see – the expectation is that they do less, and now my expectation is that I get more done. And I think there are a lot of leaders out there who are looking at it the same way. How do I integrate this thing into what we do?
-VP of BI & Analytics, Midsize Financials Enterprise | ETR Insights 335, 2/3/23
If I can add one more point on RPA, juxtaposed with machine learning and AI. When you think “classical” RPA, it's user and operational experience of the types of work benches and also the way you build workflows. I think what we've seen primarily – and I'm sure this is probably kicking a dead horse – but what OpenAI has shown with ChatGPT is that you can get rid of a lot of what I would call “overhead,” complicated artifact building, or user actors around typical RPA. I see that as a very interesting value proposition, to be able to supplant some of these work benches, like in “classical” RPA, that take quite a while to master and quite a while to get any value past the regular use cases.
A typical RPA use case is in the finance or back office area, but to be able to spread that out now into cyber, as I've mentioned, but in other operational areas, I think the barrier to entry is going to be a lot lower. And I see the amount of innovation rapidly improving from a machine learning perspective to the point where it's much more tangible now. So, it's an interesting way I'm hearing about both terms. It’s going to be very interesting to see how vendors position their messaging around this, as well – ChatGPT is the opening salvo. Obviously, there are going to be more generative AI solutions out there that are going to be niche or incremental to what we've seen with ChatGPT. A very interesting dynamic there that I’ll be watching.
-CIO & CISO, Nonprofit Medical Research Institute| ETR Insights 331, 1/31/23
I think that NLP [technology is transformative] - the ability for a computer to read human language and get it. ChatGPT is the famous example, but email security is transformative when you apply that technology to it. Because now instead of looking for IP addresses and weak signals, you can read the email at light speed with a computer and tell that it's phishing. So, I do think we've had a breakthrough in that space. There will always be phishing, because even humans can be fooled, but even Mimecast and Proofpoint are adding those types of capabilities to their platforms. So, I think those tools will get a lot better, but more generally I think that that natural language processing angle is going to become pervasive in a lot of security tools and a lot of enterprise tools.
-CISO, Large Hospitality Enterprise | ETR Insights 326, 12/13/22
From my security angle, ChatGPT is worrisome to me because the whole model is based on what words should come after previous words. It’s a language model, it doesn’t have any intelligence, and it can be very convincingly wrong around a wide breadth of topics, as it’s been demonstrated in the news. From another security angle, I worry about how quickly we make it accessible and how quickly people will start putting things that shouldn’t go into those training models into it. There are PII and HIPAA concerns, along with many more things of that nature. From purely an IT perspective or from a cogeneration perspective, I think this is very, very interesting just to be able to quickly parse out some Python code, some C#, or something that’s well documented like Python libraries and that nature. That is very neat but it still needs a lot of guardrails, and even then you shouldn’t approach it with a ton of confidence. I would be very, very wary of huge investments into it because it’s still emerging.
Microsoft has a huge investment in it, and I like how they’re trying to integrate it into Edge in particular. Google’s bread and butter is their search but more and more I find google search to be not useful because it’s just a ton of ads. You’ve got to get through page two, maybe page three, to find what you’re looking for. There’s so much that Alphabet is doing these days that it seems like corporate ADD at times. To see how Microsoft weasels this technology into Edge to try to promote people leaving Google Chrome is clever.
-CISO, Large Hospitality Enterprise | ETR Insights 338, 3/7/23
Conclusion
Ultimately, there is tremendous interest, and it’s only a matter of time before big companies provide enterprise offerings and it gets adapted. Once that happens, companies will be more easily persuaded to move past testing and towards real use case implementations by leveraging the larger and more established IT vendors. With that said, it could be years until using this type of technology is widely implemented in the mainstream due to the careful approach and languid pace at which large companies are able to integrate new technology. In addition, many companies aren’t positioned to take advantage of it due to their legacy data infrastructure.
One cannot deny the buzz around the technology, although others would correctly point out that advanced ML/AI has been around for years, and this could possibly just be riding a wave of marketing hype. Either way, ChatGPT has the potential (and danger) to become the ultimate Shadow IT that has strong interest and application from both the commercial and enterprise side; it will be difficult to keep this new technology contained in the enterprise. Therefore, within the enterprise, a CIO/CTO would rather trust a well-established IT vendor to help scope and implement the utilization of this technology.
Interested in more industry-leading market research? Start your own free trial here.
Enterprise Technology Research (ETR) is a technology market research firm that leverages proprietary data from our targeted IT decision maker (ITDM) community to provide actionable insights about spending intentions and industry trends. Since 2010, we have worked diligently at achieving one goal: eliminating the need for opinions in enterprise research, which are often formed from incomplete, biased, and statistically insignificant data. Our community of ITDMs represents $1+ trillion in annual IT spend and is positioned to provide best-in-class customer/evaluator perspectives. ETR’s proprietary data and insights from this community empower institutional investors, technology companies, and ITDMs to navigate the complex enterprise technology landscape amid an expanding marketplace. Discover what ETR can do for you at www.etr.ai