“So most of the world is made up of full stack application developers who build software for anything from HRIS to accounting, to supply chain, and what have you? For those people to easily add generative AI capabilities into their applications while remaining in compliance with the security, trust, and safety requirements that enterprises have, well that’s a fairly difficult challenge.”
Vivek Sriram– Co-founder of Bookend.ai
Summary:
In this episode of Science 4-Hire, I welcome my old friend and partner in crime Vivek Sriram, co-founder of Bookend AI, a start up that provides secure infrastructure that supports the efficient spin up and fine tuning of open source LLMs
We waste no time delving into the fascinating, confusing, and intricate world of Large Language Models (LLMs) and their burgeoning role within enterprise solutions, with a special focus on HR and hiring applications. We sure do agree on the transformative potential of LLMs to revolutionize enterprise software, enhancing functionalities such as candidate screening, resume parsing, and even generating interview questions—tasks pivotal to modern HR departments.
Vivek brings me back down to earth a bit as he provides words of caution about the considerable challenges that come with integrating LLMs into enterprise systems, especially within the HR sector. Concerns around data privacy, the risk of perpetuating biases, and maintaining compliance with labor laws are significant when deploying AI in hiring. Vivek emphasizes the critical need for enterprises to navigate these challenges carefully, ensuring that LLM integration respects ethical guidelines and regulatory requirements, thus preventing potential adverse impacts on candidates and the hiring process.
The good news is that Vivek outlines strategies for implementing LLMs in a manner that balances innovation with responsibility. Approaches such as utilizing open-source models for greater control and customization, and employing platforms that offer secure, compliant AI integration, are discussed as viable solutions. The idea of fine-tuning LLMs with proprietary data to better align with specific HR needs provides additional levels of confidence for those looking to use LLMs securely.
Listeners will gain a nuanced understanding of the dual-edged nature of LLMs in HR and hiring contexts—recognizing their potential to significantly improve efficiency and decision-making in talent acquisition, while also grappling with the ethical, privacy, and compliance issues inherent in their use. This episode underscores the importance of thoughtful AI integration in HR practices, aiming for a future where technology serves to augment human judgment rather than supplant it.
Take Aways:
- We must recognize that integrating open-source LLMs into enterprise applications comes with complex challenges, including navigating licensing, data usage permissions, output control, and auditing requirements.
- Enterprises must address strict compliance and security standards, especially in regulated industries, when implementing open-source LLMs. This includes ensuring data privacy, adhering to industry-specific regulations, and maintaining the integrity of sensitive information.
- Tailoring open-source LLMs to specific enterprise needs requires significant customization. Enterprises need to modify these models to align with their unique operational, compliance, and ethical standards.
- Potential solutions to the issues inherent with the use of LLMs involve employing strategies for effective management of open-source LLMs, include:
-
- Selective Model Adoption: Carefully selecting open-source models that best fit the enterprise’s technical and compliance requirements.
-
- Data Management and Security: Implementing robust data management practices to ensure that the use of LLMs complies with data privacy laws and enterprise security policies.
-
- Model Customization and Fine-Tuning: Customizing and fine-tuning open-source LLMs with enterprise-specific data to improve relevance and performance while adhering to ethical guidelines.
- It is essential to stay informed about evolving regulatory landscapes related to AI and machine learning technologies to ensure ongoing compliance and adapt strategies as needed.
Full transcript:
Speaker 0: Welcome to Science for Hyre. With your host doctor Charles Handler. Science for Hire provides thirty minutes of enlightenment on best practices and news from the front lines of the improvement testing universe.
Speaker 1: Hello, and welcome to the latest edition of Science for Hire I your host doctor Charles Handler. And I have a guest today. A guest back in the day, we’ve known each other probably like twenty five years. I don’t know, be sitting around having a beer twenty five years ago and said, you wanna be on my podcast about generative AI. You would have thought I’ve probably been doing a lot of shots because nobody would know what the hell I’m talking about. So anyway, my guests today, a good old friend and someone now I’m excited to be collaborate in a little bit with a completely different field as me, which is awesome and hopefully expose our listeners to some new ideas and thoughts. About large language models and all that fun stuff. So my guess, Vivek Shiram, who is a cofounder of Bookend, AI welcome.
Speaker 2: Well, thanks for having me, Charles. It’s a it’s a pleasure to be on your podcast.
Speaker 1: Yeah. So I always let the audience I don’t think I’m lazy, but you know, kind of part of me always says, well, are you lazy? And this is why you always have your guests introduce themselves. But it’s really because who knows you better than you? So just tell tell our audience here, you know, who you are, what you do.
Man, you’ve had a pretty illustrious history, I will say, before I interrupt you. I’ll interrupt myself. You know, I lived in San Francisco around two thousand. I came out there to be part of the I call it the first Internet revolution there, and it was pretty exciting times. I went out there to work for the first company putting testing online, but I hadn’t known Vivek from LSU. He’s more in the realm of of IT development, exciting stuff like that he’ll tell us about. And so we were we were you know, hanging out a lot back then. It was really interesting. Again, it it the world at that time, twenty three years ago or whatever, it you know, we were we were really like, wow, this is such new stuff. It’s gonna change everything, which did it seems very flat and simplistic compared to where we are now.
So anyway
Speaker 2: All all that is true. I mean, you know, I’ve I’ve known you for probably more than half my life now going going back to college from from from LSU. And you’re right. Twenty five years ago, I’ve been sitting around drinking beer, playing pool, you know, whatever trouble we’re getting up to. I don’t think any of us had any inkling about about podcasts or generated AI or any of this stuff.
Exactly. It is it certainly is a new world. Quickly by way of background, I’ve been in enterprise software, basically my entire professional career. Which also runs about twenty five years now. Most recently, I was chief before starting Bookend a year ago, I was the chief product officer for a company called LucidWorks.
We LucidWorks makes an AI powered search engine, which powers a large number of consumer facing websites. If you ever go to Costco and, you know, look for a five gallon jar mayonnaise. You put stuff in the search box, you know, that’s that’s what Musa Gorax does. Before that, I built a couple of search engines at Amazon Web Services also to been in and around commercializing various kinds of things whether we called it AI or not. The idea behind bookend was that we wanted to make it easy for for non AI native developers.
So most of the world is made up of full stack application developers who build software for anything from HRIS to accounting, supply chain. What have you? For those people to easily add generative AI capabilities into their applications while remaining in compliance with the security, trust, and safety requirements that enterprises have, that’s a fairly difficult challenge. That’s that’s kind of what we’re trying to solve at Booking. The rest of the team, just real quick, is is is also made up of a number of people.
They’re ten we just had ten people, so we’re not really big. But everybody’s got quite a bit of experience in building managed services at companies like Google, Amazon, Dropbox, IBM, etcetera. So that’s that’s that’s what we’re up to.
Speaker 1: Cool. Yeah. And, you know, why we’re here today, really, it’s it’s funny because we’ve been friends a while. And when I was in San Fran, you know, we talked about a lot of times, well, how can we work together? What where do our things intersect?
They never really did. And that’s okay because that doesn’t happen with a lot of my friends. But on a recent trip to San Francisco. I think, again, beer is involved and a lot of bullshitting is involved. But but Vivek’s like, you know, I think there there might be something here that would be valuable to you.
And, you know, we’re always doing that and hadn’t really found it yet, but I don’t think there’s much of a force fit here. I kinda heard what what bookend is doing and looking at my own self. And I think this applies to any industry, any person or company who is really looking to take advantage of this, you know, revolutionary times that we have now in the tools and technology that really seem otherworldly, quite honestly, but are real in in terms of what they’re doing in in advancing their fields and, you know, I’m really thinking a lot about this stuff lately. And once I heard a little bit about, you know, what y’all do, I’m like, wow. Okay.
That makes a lot of sense as a foundational building block of a product or a program or whatever in a lot of different ways and haven’t learned more about it, you know, it’s it’s exciting to to think about working with you all a little bit in the in the HR technology space. And as I’ve done that, as
I’ve gotten kind of more knowledgeable about your product. It’s it’s really opened up my mind. Like, I’ve been thinking small ball, I still feel like as as far as, like, We can use generative AI to do x, y, or z. But you talk to people who are building out architecture to support it and you’re just listening to some of the use cases that you all had.
I’m like, wow. Okay. There’s a lot more here. It seems like you are helping democratize some hard things. So if you had to give your I don’t know.
I would say elevator pitch, but we have more time than that. So, you know, even elevators to really tall buildings like the Empire State building, they only take, like, thirty seconds these days. So we have more time than that. So what are you guys all about? You know? Give me a sales pitch. Give our audience a sales pitch. It’s okay. To save space here
Speaker 2: for that? Yeah. I’ll I’ll I’ll I’ll give you the the the simple version of it. By now, I mean, everybody’s familiar with with chat, GPT, and open all out of everybody. Virtually everybody’s gone around and and played with it.
And, you know, our image generators or or what have you. You know, that’s a that’s a a a closed source piece of software. Right? So you can go to open AI. You can you can use it either for enterprise.
Purposes or for just playing around with it on on tragedy, Peter, or what have you. But it’s a it’s a effectively, it’s a black box. You’re you’re gonna you’re gonna get the service Open AI dictates
that that the terms of how you use it, and then you can if if your use case fits within those circumstances, then Right. For a a lot of enterprise use cases that that may not necessarily be the best fit. Their options they have other you know, one of the primary ones is is taking an open source model, a pre trained model that can do any number of things from text summarization or image generation or, you know, natural language processing or what have you. And then using that inside of their own use case. The challenge with that is that the the most popular place to go and get that from is a is a a website called hugging face. Your maybe your listeners are familiar with that as well. I’m gonna just go there right now and go and see. I think hugging face has four hundred and twenty three thousand seven hundred and seventy three models available open source models available.
Speaker 1: That’s not many.
Speaker 2: That’s not many at all. So, you know, most people I’m sure can keep that in their minds and and just know exactly what to do. The problem is that it’s difficult to know where to get started. And enterprises typically have a bunch of rules and regulations about what’s allowed to be done and what’s not allowed to be done. So licensing, what data you’re allowed to use into it?
How do you control the outputs? The auditing that goes along with it? Who’s allowed access to it? All of those kinds of things are are frequently a a difficult set of challenges. So imagine, you know, if you wanna go and take Meta’s open source model, Lama two.
And and you wanted to go and and build a generative AI application in HR. That parses resumes and identifies relevant skills. Right? So simple usage. Yep.
And and you’re at a and you’re doing this in in the context working in a in a in a financial institution, in a large bank.
Speaker 1: Right?
Speaker 2: You’re gonna have all kinds of of restrictions about what you can and cannot do with it. What Bookend does is to offer a platform for those developers that make it a little bit more risk free to take an open source model and run it, sanitize it, run it, use it kind of in the context of an enterprise. This isn’t different from how enterprises adopt to open source technologies in any number of other things, databases, search engines, web servers, what have you. In all of those cases, you know, you take an open source piece of software, license it, sanitize it, benchmark it, and make it available so that the IT leaders who are responsible for the running and maintenance of enterprise software can do so with a controlled amount of risk and and know how about what’s actually going on. So that’s that’s basically what Bookend does.
Speaker 1: Gotcha. So one thing that, you know, I’m learning a lot because I I get general ideas. Like I said, I’m teaching myself as much about this stuff, which right now I’m probably in kindergarten, maybe first grade, you know, some some parts of it I’ll never get because I I don’t need to, but which is the whole point of some of this stuff like GPT. But, you know, what I’m curious about is I I kept thinking that conceptualizing what you all do that there’s there’s like a outbound reach to say, chat GTP, let’s say. Right?
Because I you can’t take or maybe you can, like, you can’t take chatGTP and put a copy of it on your own server. Like, you have to reach out to it, right, and then get a turn, right, which is what exposes all your stuff. But on the hugging face site, are there ones that you can just like say, I’m
gonna download this thing and put it on my own server? Right? So do you conceptually serve both those use cases?
Do you have enterprise software or enterprise applications that people are building, you know, internally? That would need to reach out to say, chat GTP. Maybe there’s a a late a title for something like a chat GTP that’s Maybe it’s not even open source and the open source ones are the one on hugging face. So explain a little bit about the difference between a model like chat g g p or Bard or what’s the one my Microsoft has, which is now embedded in every thing. You know, you
Speaker 2: know Yeah. Microsoft uses open AI, which is, you know, Jeff’s right. Speaker 1: That’s right.
Speaker 2: That’s part. Right. Yeah. So Bard is Google’s version of it. They’re all farmer models. They all they all are trained on I’m kind of about the same kinds of datasets too, which is Right. Buying large the Internet.
Speaker 1: Yeah. Everything. Everything. Yeah.
Speaker 2: Yeah. So you can you can use these services like Bard or or OpenAI and and and interact with it. And they have OpenAI also has a commercial version of it and an enterprise version of it. Right. And use it to to, you know, use to to to power your own applications. But they don’t give you huge amounts of control over what can be done with it. You can’t you can’t download the model, for instance. You can’t run it on your hardware. So all there there are limitations to that.
Speaker 1: Right.
Speaker 2: There are also some constraints that that chief information security officers, particularly in regulated industries, are still wrapping their heads about what they’re comfortable doing and not mode.
Speaker 1: Right. Right.
Speaker 2: So, you know, it’s a I’ll give you an example. You know, one of the one of the companies that we’re working with is a little hospital chain. For them if they wanted to go build a chatbot or a kind of an assistant that helps physicians summarize patient notes, for instance, the raw data, and it is electronic patient records. There’s a sensitive data. It’s about people’s, you know, infirmities and all these kinds of things.
There’s a huge amount of sensitivity about from within the IT leaders, within hospitals, for instance, about taking that data and putting that into AI or bard or any any any of these kind of AI services. So the option for them is is instead is is a a frequently popular one is take a model, an open source model, from hugging base, like, you know, maybe Open Hermes or LAMA or what have you. Run it on their own hardware. That could be, you know, it could be they could have a cloud account. They but they manage it.
It’s their data.
Speaker 1: Right?
Speaker 2: Or they could use a service like ours, which give them control over what happens with that model and, you know, with the data and all that. It becomes their own thing then. So it’s kind of you know, there there are maybe two or three different alternatives like does. Right? So open AI barter are on one one end.
The hugging face kind of the open source wild west is at the other end. And in the middle, are are companies like AnthroPIC. Right? Which which is kind of the they’re branding themselves as a safe alternative to open AI. Right.
And tropic has taken on a a huge investment from Amazon Web Services.
Speaker 1: Right. So you
Speaker 2: can use those capabilities on AWS if you, you know, if you wanna more control and and assure about safety.
Speaker 1: Uh-huh.
Speaker 2: But all of these things that involve certain certain trade offs in terms of control and cost and what you get out of it. Yeah. With with open AI, you get the least amount of control. With the logging phase, you get the most amount of control. But you gotta do the most amount of work to make it work there.
Speaker 1: Yeah. Interesting. That’s the exact parallel. You know, I bet that that pair that is a metaverse of all every different industry. Right?
Like in my industry, Same thing. You got more control, less control, more danger, less danger, more effort, less effort. Those are all the tradeoffs we’re constantly working with and when tools come along that kinda help you manage those trade offs or make make maybe make accessible what you’d like to do most but have limitations on, that’s always good. So I got a couple questions here too just one thing I’ve never looked up, I could ask Chad GGP this, which I’ve done a lot of education that way. But what exactly of two two questions?
The first one, what exactly is a transformer model? What does that actually mean? Because I read that a lot and, you know, I know it’s not Optimus Prime or or any of the stuff my kid plays with. But
Speaker 2: Yeah. So there was a, you know, a paper that that Google had put out a a while. Back that, you know, that in a in a this is, like, the the the simplest version of it. You can go and you you can go and ask chat, GPT. Know, that might be an interesting test to go and try right now. Yes, sir. You know, how do you Well, let’s just ask you. Let’s just ask for it.
Speaker 1: I’ll tell you what. You are given our listeners and viewers a prequel because I’m working now to figure out how to have a chatGTP as a co host. I had it as a guest you can I got a little plug in? You could talk to it. It’ll talk back, which is basically a preview of what’s gonna happen when we have our own agents and we’re just telling it to do stuff and it’s talking back to us.
Right now, it’s it’s spread out a little, but, you know, we all know that Chad TTP’s agenda is to is
to integrate this all just like now I’m enjoying using dolly from the same interface. If you’ve never played with dolly, it’s It is awesome.
Speaker 2: It’s pretty cool.
Speaker 1: Professionally, I’ve done stuff, but I’ve also just done goofy stuff with my son when we type in, like, make me you know, one thing for that I did with it as an aside. So we’re looking at, like, Halloween costumes for my kids. So I I asked it, could I upload a picture of my kid? You know? And then transform them into different stuff.
It wouldn’t let you do that because it said I can’t recognize. So I described him and I’m just like curly blonde hair, you know, really don’t know. Good looking. I said, which is true. And and it it gave me four versions and I swear to God, one of them was like a spitting image of this kid. I I strive it a little bit, you know, more specifically. And then I was turning them into nosferatu and turning them. We turned them into a bird. And, I mean, you just you never know what you’re gonna get. I think the first time I played around, I was like, I have a car, a Lotus. I’m like, I’m always, like, thinking about Iron Maiden when I’m driving that car even though it’s fiberglass.
Speaker 2: No. That that I think everybody should think about it. I don’t need more of that.
Speaker 1: I know. So so I said give me an image of Eddie, the mascot of Iron Maiden, you know, driving a nineteen ninety five load of a spree with a union jack. And I mean, I’m telling you, dude, I went through like fifteen versions. Sometimes it put two heads on Eddie, sometimes it gave them two flags. Like, it does a lot of random stuff like that, but
Speaker 2: but it was
Speaker 1: sitting there going, holy shit. This is crazy. Yeah. I mean, it it was so fun. So it’s a fun little diversion.
Anyway, I don’t even remember what oh, I was talking about, you know, the integration of stuff. So, yeah, what did ChatGTP say?
Speaker 2: So I I actually went and asked about
Speaker 1: it and
Speaker 2: said explain explain what a transformer model is and and the and the and the first sentence first two sentences are very interesting. To transformer models or a type of neural network architecture that are particularly well suited for natural language processing task. They’re introduced in the paper attention is all you need in twenty seventeen. You may become the dominant architecture for natural language processing.
Speaker 1: Gotcha.
Speaker 2: Okay. That’s sorta helpful, but not really helpful. Yeah. What what what they’re what they’re the simplistic way to to that I I might explain it to to, you know, to my dad or something is that so that if you if you train a huge amount of data on on or if you if you if you train a
transformer model with a huge amount of data, they tend to get really good at predicting what tokens. The token is a part of a war.
In what what order they come in. Right? And so then then you can you can use that capability in order to to to put it to work to do things like generating time. Right? So this question that I asked explain the transformer model has generated an answer for me, which is very different from from maybe two years ago or even last year pre open AI, if you’d gone to Google and look for what’s a transformer model, you get back a a set of search results.
Right? And those things point back to the various websites that they load in and then, you know, it’s it’s up to you to go look at they’re not generating the summaries on the fly. They’re not gonna generating a response on the fly. Right? But, you know, these things are able to capture the dependencies between words And they also are able to to process these these inputs in in parallel, which, you know, make some do certain things really fast.
Like, for instance, joining this picture that you’re talking about. When it comes to pictures, they kinda work in the same way, you know, that what they’re doing is it’s creating a vector representation of of of these images and then be able to combine them and generate them on the flat. Issues, though, is that, you know, like, it it’s a cool example that you mentioned with Eddie, I mean, like, I’m a big made in fan too, as you know, and it’s a Yeah. That’s fine. I don’t know if Eddie’s copyrighted or not.
Like, you know, and what what Bruce and the boys might have to say about, you know, what’s going on with it. So maybe nothing. Right? So, you know, if you if you if you did the same thing with the Grateful Dead still your face, you know, which I don’t think that the Grateful Dead ever copyrighted, you could probably get away with that. Nobody really gives a shit. But, you know, when in inside of the enterprise, like, these are some of the situations that come up and that can be potentially problematic. Yeah.
Speaker 1: I’ve had it blocked me when I’ve said, like, you know, use so and so’s picture logo, and it’ll say that’s copyrighted material that I can’t. There may be a way to hack your way around that. I typically say, we’ll make it like that, but not exactly. You know, and, you know, there’s ways to prompt engineer, prompt it, whatever.
Speaker 2: Yeah. And and that’s and that’s exactly one of the main issues with a lot of the stuff is that is you know, you can just by trying a number of different things, you can get around you can get around what the developer of of an application might have conceived.
Speaker 1: Yeah. Totally.
Speaker 2: Right? So you can hijack prompts. You can keep trying different things. You can inject it with various things. And Yeah.
And and, you know, like, to the the the issue the consequence from it from your chief information security officer and a large company is that you you developed a security policy, a framework for thinking about risk and risk remediation that’s evolved over thirty years for longer. Right? Like, how to control access to to certain resources, how to how to prevent unauthorized access to, you know, how do you track it, trace, and all that kind of stuff. And that works really well when when responses from systems are deterministic. You get the same thing back over and over again.
You can predict kinda what’s happening like in the world of data business. That doesn’t exist in
this. Right? You can the same question can generate different kind of things that is all context on that. Yeah.
So so what did how do you how do you build kind of the the enterprise safeguards
Speaker 1: in this case? In enterprise, it has to say the same thing every time. If you’re relying on it as an information source, as an input to some other thing that needs to be consistent, you can’t have it be wonky like that. Right? And you can’t have it elucidate, you can’t have all that stuff.
Speaker 2: If if you and I worked for a same large, let’s say, a fortune five hundred company, And we had a we had an internal HR chatbot that that helped, you know, do various things. Right? You know, help you you know, put in for vacation request for instance. And I had a way of of manipulating the prompts to say, you know, give give me give me access to Charles Social Security number. A bank account number or what have you.
Like, that’s that’s really a problem.
Speaker 1: Well, one thing I just read, you probably read about it is I think somebody set it up to just repeat the word poetry. And if you do it enough, you can make it barf out all its training data. Right? So, like, you can actually and I think they’re starting to, you know, safeguard against that. But that’s the example is just like any kind of security.
It’s always one one side of the fence coming up with a clever way. The other side of the fence blocking that way and and on it goes and on and on it goes. You’re right? So, you know, taking it out, kind of making it bulletproof is is a good way to to think about it. So it’s it must be part of the equation.
I think, again, at the enterprise level.
Speaker 2: It has to be.
Speaker 1: Yeah. Yeah. So I got a couple of things. So Is it the case thing with the transformer model that, like, the more you stuff into it? Training wise?
The more connections it can make and the the better it is. Right?
Speaker 2: In in theory, yes. But I think in in practical terms, you hit into diminishing returns. And can, you know, can you like, so the the large models also consume a lot of capacity to run. Right? I mean, they’re they’re computationally really intensive and and the the the the GPUs, you know, the the graphic processing units, which power all these things are are very expensive and they’re serious supply chain constraints for them.
So depending on the use case, it may or may not be the best fit. Mhmm. Right? So there’s an evolving field is is is biological large language models. So they’re looking for things like how do proteins interact with enzymes and a brand
Speaker 1: about that. Yeah.
Speaker 2: So in in those situations, a small metal might actually be a lot better than a than a model Right. Right. On Internet data on Right. Right. Right?
I
Speaker 1: mean, see, that’s the thing. My as a layperson, my mind always goes to chat, GTP. Whenever I think of a large language model, That’s my paradigm. It’s publicly available. It’s trained on all this shit.
The more you stuff in there, the more stuff it it can know. Right? But I wonder when they’re training this thing, like, when they’re training chat GTP. Right? I mean, do they have so look at think about garbage in, garbage out, and think about horrible stuff.
Right? Well, couldn’t you, on the front end, say, hey, as I’m training this thing, let’s leave all the horrible stuff out. Therefore, it won’t come out on the other end, but it seems like it’s not like that, that it it knows the horrible stuff. It’s in there. They try to put these safeguards around it, but clever people can pull it out.
And so in there is are the instructions for making a nuclear weapon or, you know, methamphetamine or, you know, I remember, I don’t know if you ever saw this
Speaker 2: I I remember when we were when we were young men, you know, the the most insidious that it got was that, you know, we’d have, like, a photocopy couple of pages out of the anarchist cookbook and, you know, we all think we’re really cool. You know?
Speaker 1: Dude, are you kidding me? I was just about to say The only thing we had was the Anika’s cookbook, and everybody said, oh, man, if you buy that, the FBI opens a file on you and then Yeah. There was another there was another set of books called getting even by this guy George. Hey, dude. Did you ever see those?
Speaker 2: Yeah. Yeah.
Speaker 1: That those were horrible. Like, how to how to mantle a balloon full of pee to somebody in a box, so when they open it, it’s flashing over him. And it would always say, in order to get your mark they call the person. So that’s crazy. Well, we just dated ourselves and it goes to show you that there was nothing really besides that or make
Speaker 2: this this world is a whole lot scarier.
Speaker 1: Oh my god. The anarchist cookbooks like a nursery rhyme book at this point. Right? So Anyway,
Speaker 2: but do you know go go on back to your question though, the the the p in the GPT is pre trained.
Speaker 1: Gotcha. Yeah. Yeah. Yeah.
Speaker 2: So it’s generative pre trained transformers. So most of these things, like, are are trained on public data. So on on common crawl for instance, which, you know, but they’re they’re you might you probably have seen, like, a lot of the authors and, you know, Steven King and and who’s the Harry Potter Lord?
Speaker 1: Yeah. Yeah. JK Rowling or something?
Speaker 2: JK Rowling. Yeah. You know, they’ve they’re I think rightfully so, up in arms about Yeah. Their their their creative work being used with Yeah.
Speaker 1: Music too. Right? In art images, it’s all like, that’s a whole another set of a million podcasts. So Another quick question. I write these questions down because I’m so curious, and then I got I got some some mills to transition to.
But which is more energy intensive, mining Bitcoin or running to chat GTP. Right? Because you still hear about people like taking fifteen thousand playstations and rigging them up, you know, somewhere somewhere in a faraway world so they can mine bitcoin. But we know it takes a lot of energy
Speaker 2: I I don’t I don’t actually know, but the the the value the usefulness is is fairly obvious to me. Yeah. You know, one one of them is far more useful than the other, but also that, you know, to the question you’re asking earlier about about small models and large models. Like, they’re going the opposite ways. Right?
So Bitcoin, mining a new the marginal Bitcoin will always be more expensive than the previous one. It’ll be computationally more expensive because that’s how the algorithm is interesting. The the trend in in in GPTs and and AI is going the other way. Right? The trend is towards smaller models.
So you you you the the amount of work that it would have taking you to to to get to an answer a year ago is going to be far more expensive, computationally expensive than what it will be a year from now. So that that’s going the other way. So, you know,
Speaker 1: that’s interesting interesting interesting. So here’s the thing too that I’m learning. I’m just starting to understand this. When you first describe it to me, I didn’t really get it. But there’s also this comp concept or reality of a of a hybrid model like a like an RAG. Right? So in other words, you could take a large language model that’s open source or whatever, put it somewhere and then take your own corpus of information and say, I wanna inject this stuff into your relational capabilities model. That’s pretty cool.
Speaker 2: Yeah. It’s it’s related to this concept called fine tuning. Right? Which is which is you take a a pre trained model, you know, let let’s say, you know, the LAMA tube from matter. And and you wanna use it for your own situation, which which, you know, maybe it’s it’s a it’s a chatbot for an internal HR chatbot in a in a large company.
That is going to get more and more useful. The the more it’s it’s applied, it’s tuned on with your own data. Right? So so RAG is a is a technique it stands for retrieval augmented generation. What it does is kind of a hybrid approach that combines retrieval, right, which is search with with generative models to to to improve the the quality and relevance.
And those things end up being particularly useful for things like question answer, kind of use cases, or or natural language generation, you know, stuff like that.
Speaker 1: Gotcha. So that that mean, you guys you guys accommodate that as well. Right? Like, you can do a hybrid where I might take a a pre trained model on a big one or a small one, whatever we’re talking about. And then let’s say, I have a just a old dictionary of medical coding things or something.
You want more control over that, you know, the open source model or whatever may have that in there. But you gotta have more control over it. You can’t to your point about getting different
results. I think if you inject your own model that’s very clearly labeled and structured, that’ll allow you a whole new set of insights and capabilities. Right?
Speaker 2: Absolutely. A hundred percent. And, you know, and all of the stuff is particularly useful when you think about, you know, like manufacturing, for instance. Right? You might you might have part numbers, part descriptions, things that are that are very unique to your own organization.
Which nobody else is really going to know. So in order to find real usefulness out of some of this some of this generative AI capabilities requires fine tuning with with with your own proprietary data and using techniques like rad.
Speaker 1: So let’s talk now about product. Right? So building product. I’m curious, you know, you’ve got enterprise applications. Right?
So I look at it as b to b. You’ve got enterprises that you know, are basically doing things for consumers with their economic business models, etcetera. And they they there’s a lot of ways they could use that natively, like with their own developers and their own stuff. But they also use a lot of products. Right?
Enterprise grade products that that solve problems or create functionality. You know, I’m starting to think in that direction. Right? Like, in my industry, I’m looking at what are what are people doing in terms of predictive hiring? How can we automate the the compliance based high touch stuff that I’ve been doing for three decades in order to help at scale and and, you know, it’s it’s the same questions everybody in every industry is asking.
You know, as I think about that, I’m like, so I’m a product person and I want to use this is where, again, I don’t understand, like, would I be using just the giant capability of chat, GTP, and wanna build that into my product? And you know, make sure that I could can leverage that within the protected walls of my product that I’m gonna sell to an enterprise. Would I ever even have a use case to do that or would I more likely be saying, alright, I’m either gonna train my own model or take one of these basic things because the way I understand it And this this opened my eyes too. I I attended a webinar from some database company and he was talking about I was I was actually a a friend of mine forwarded it to me, and it was all about using chatGTP to match resumes. So I’m like, oh, man, that’s like, right up my alley.
There’s some really dubious stuff maybe happening there, scary stuff. So I attended the webinar, and it was from a database company, and I was really surprised they were talking about going out to chat GTP to work with the resume and then sending back a match. And I’m I’m in the chat there on the thing going, and this is highly technical. I had no idea at most of what’s going on. I said, well, why the hell would you go out to chatGTP to do this.
There’s tons of people who use natural language processing to match resumes to job descriptions. They were like, well, the relational capabilities of chat, ETP, are so strong that there it’s gonna do a much better job of this. When you feed it information going in one way and it gives you information coming out the other way, that was a whole way. I hadn’t even thought about it. So you know, that made me think, well, if I’m building a product to do that, would I would I just have to reach out to GTP just for its relational capabilities?
Or could I use one of these open source models from a hugging face to do that? So So just think I’m designing a product in my industry. It’s gonna use a large language model to provide all kind of crazy efficiencies and new things how how am I doing that? Am I using GTT? Am I using it? You know, and how would that plug into your product? So that’s a lot.
Speaker 2: Yeah. So So I mean, you you can I’ll tell you the reasons why you might make a decision to not use OpenAI chat GPT. Right? And that that might that and and I think there’s plenty of places where it might be the obvious choice. For in the situations where the developers or the product itself requires more control than an open source alternative is probably a a a better better decision.
It’s also going to typically be cheaper to do. You know, there’s a open AI and and others like coherent charge on a on a token based model. So you gotta kinda figure out how many token based it it’s going to take for for powering your use case.
Speaker 1: Gonna get super expensive? Like like crazy expensive?
Speaker 2: Yeah. It can. So that’s another reason. In regulated industries, it’s especially when you’re dealing with sensitive data like like HR or or legal documents or what have you this isn’t this isn’t to say that OpenAI is insecure or anything, but rather that the regulatory environment is still moving around a whole bunch. So if you’re, you know, if you’re a global organization and you’re dealing with people who are who are in multiple geographies, let’s say, Japan, Germany, and in the United States, moving the data around and doing things on it, you know, or whatever if you have an application that that has to satisfy people in multiple geographies, there’s a there’s a deeper regulatory burden that comes along with it Yeah.
Externally. Internally, for companies, you know, they have varying policies about what’s what they will allow and not allow for their own users to do. So if you’re an application developer, you’re building a piece of you know, piece of software for running as a SaaS opportunity for whatever you like, these are all the things you gotta consider. Right? And And there’s it’s one possibility you might end up saying that, like, actually, I don’t none of these things are a big deal. I don’t care about the cost. I don’t really need any control over it. And I’m you know, and I don’t have any geographic or regulatory issues. Hoping I might be perfectly good answer. What we find though is that in in particularly regulated industries or or companies with global footprints, those concerns tend to be pretty severe.
You know, they want control over the data. They want control over the model so that they can track what’s going on. They can authorize and and deauthorize access to it and all of that. So and and frequently, you know, for especially for application developers, you know, who are kinda comfortable with with running cloud workloads and stuff. Open source models are are great. Right? I mean, it’s not it’s not too hard. A piece of a a platform like Bookend makes a a bunch of the operational headaches go away. So you can you can take a model, you know, run it on on on multiple clouds if you need. You know, you got a bunch of the the the security, auditing, governance kind of capabilities that just come out of the box, so you don’t have to build all that stuff.
So there’s a there’s a there’s a bunch of bunch of things that you kinda just get free from using a platform like ours and and we’re pretty cheap. Right? It’s not which it’s quite cost effective to run it.
Speaker 1: Cool. That’s very cool. So so I’m a product part, you know, I work for a company that serves up you know, pre hire evaluations, predictive hiring. And, you know, I’ve got some ideas to to build because I sell the enterprise, and I’ve got some ideas to to leverage large language models for various things. Right?
It it could be a lexicon of information about what it takes to do jobs like a taxonomy. It could be
generating interview questions or scenarios. Right? And I I could have my own model my own data for over the years that I wanna incorporate in that. But again, I wanna use some of the various things.
How do I use you guys. Like, wrap my stuff around yours, etcetera. Like, I I don’t a hundred percent get that. So talk to me about how product builder for enterprise?
Speaker 2: Our ours ours is fairly easy to use. I mean, it’s it’s kinda it’s kinda designed for use by by, you know, kind of full stack application developers. So it’s it’s a it’s a REST API. So any any developers who’s used to, you know, using using an API through popular languages. You know, we support client libraries for Python as well as Java.
So, you know, any developer who’s familiar with using those kind of languages can can call all of the the different things that Bookend does through an API. And and all of the models that that we make available on our platform are are curated, sanitized, benchmark, licensed all suitable for enterprise use. Right? So then we kinda run that entire process. So from we don’t have four hundred and twenty three thousand models on our platform.
We only have a dozen. Right? But they’re all right now and, you know, we’ll have about a hundred or so in the next couple of months. But, you know, they perform a number of different tasks from summarization to question answered to generating embeddings and various kinds of common things that developers want. But they’re all kind of, you know, really they’re suitable for enterprise use.
And and so from the models on down, we make available a whole bunch of security governance and and trust and safety related things out of the box, including, you know, things like data loss prevention. Right? So keeping your, you know, sensitive information, like Social Security numbers or account information and stuff like that from turning up and Right.
Speaker 1: Right. And returns Oh, gotcha. So what about what about if I wanted to say, well, you know what? There’s some stuff I wanna use from chatGTP in what I’m doing. Can I use that securely through Booking too?
Or is it is it only models that, like, I would download completely?
Speaker 2: We support you can you can use your own model on you know, like, we have a we have a customer who’s in a designed their own model. Right? This is a this is another biotech. So they’re, you know, they’re kind of interested in in doing some protein analysis. There’s not a suitable open source model for that, so they run it on our platform.
But we’re almost exclusively open source models. So it’s not a, you know, you wouldn’t you might have an application that that that you know, that kind of incorporates both OpenAI and some open source models, but you can’t use Open Source. You can’t use OpenAI with Bookend.
Speaker 1: Gotcha. Gotcha. Right. But both most people who are building secure applications for enterprise aren’t aren’t injecting GPT in there. I think the use case more in enterprise is people who work there going, okay, how can I make my job easier?
How can I find out information etcetera, which is a great use case for it? But as you start prompt engineering it to do stuff, so, like, what if I prompt engineer it, to be able to run like a dataset or something, and then I wanna build that dataset running into my, you know, my product, I guess, that simplistic thing of running a data set, you could code Python or something else to do that.
You don’t necessarily need it. It’s from people like me that don’t know how to do that that coding
who can give it an Excel file and say, analyze these different trends or whatever. That’s the magic of it.
It’s not a you know, enterprise has stuff that does the same thing. Would that be a a correct app? Yeah.
Speaker 2: Yeah. I I think I think that’s fair. Look, we we we tend to do well in in situations. Like, we really back off this stuff. Right?
Like, dealing with sensitive information, in in we’re very comfortable in regulated industries and and, you know, kind of the the decidedly non sexy stuff. Supply chain HRIS. Like, so all the the platforms that make a bunch of organizations work well. But it’s not, you know, it’s not like the the the cool, like, you know, image generation for your, you know, dating profile of the time or whatever like that. Yeah.
It’s not
Speaker 1: I always think of I don’t know if you remember as a movie from I think probably the eighties, but down and out in Beverly Hills, And what I always remember about that is, you know, they were super rich. And the guy, he had a coat hanger factory, you know, makes the most boring ass thing, a code hanger. Right? But people need it and, you know, super lucrative. So I always think about that, like, there’s there’s a lot of money to be made in the unsexy stuff too. Okay. So so, you know, in my world and it’s getting that way, everywhere. But, you know, vendors of predictive hiring tools are gonna need to have it’s not the case yet. But there’s gonna be a mandate probably eventually from the feds, but most likely from California and, you know, the EU is working on something. So a mandate of being able to have a a third party audit from a to z of what they’re doing.
Speaker 2: Yeah.
Speaker 1: So if someone were as a product come company were utilizing, you know, a large language model, and they hadn’t wrapped up in something like yours. I’m assuming it’d make it a lot easier for them to pass some kind of security audit, almost like when I tell when I tell someone on our little platform that it’s hosted on AWS, I get to skip skip over crap load of IT Yeah. Security questions. Thank goodness. Right?
So you got that same capability. Right?
Speaker 2: Yep. So so things like algorithmic bias and, you know, to the big deal for for the kind of stuff that you’re you guys are doing. Like, so we make all of that a bunch of that kinda stuff is available out of the box and auditing for it, tracking what’s going on, explaining what’s happened, and stuff like that.
Speaker 1: Interesting. Good stuff. Well, there’s there’s I’m looking forward to, you know, the journey here. I mean, you’ve been a big part of educating me on this stuff and it’s so accessible, yet difficult to conceptualize. If you if you really dig into it, I mean, I call chat TTP, you know, supernatural math.
Because all it is is math equations, predictions, but geez, you know, it’s ones at zeros, ones and zeros,
Speaker 2: but
Speaker 1: but for God’s sake, it it it writes books and generates crazy images and answers all your questions and gives you recipes and I’m taking a prompt engineering class right now, like online. And I’m wondering a lot from that too. So in exploration mode, there’s lots to learn, and
I’m looking forward to, you know, learning more about what you guys are doing. So I appreciate your time and and and hanging out. Where can people keep up with you and you guys and what you’re doing?
I mean, I say that all the time, and it’s always, well, we’re on LinkedIn. I’m on LinkedIn. Is there anything else, you know?
Speaker 2: Yeah. Well, we’re we’re we’re we’re bookend dot AI. Hopefully, easy to to to not spell wrong. Yeah. We’re we’re on all all the usual LinkedIn discord and all that stuff. Yeah. No. It was a it was a pleasure pleasure speaking with you today. Yeah.
Speaker 1: Thank you
Speaker 2: for having that. Absolutely. Hopefully hopefully we do it again.
Speaker 1: Yeah. For sure. As we wind down today’s episode to your listeners, I want to remind you to check out our website rockethire dot com and learn more about our latest line of business, which is auditing and advising on AI based hiring tools and talent assessment tools. Take a look
at the site. There’s a really all some FAQs document around New York City local law one forty four that should answer all your questions about that complex and untested piece of legislation. And guess what? There’s gonna be more to come. So check us out. We’re here to help.
The post LLAMA, LLAMA- How to Avoid Generative AI Drama! appeared first on Rocket-Hire.