I get some version of this question every semester, usually from a student who just watched a ChatGPT demo and wants to know if they can build “something like that” for their capstone project. And I get the founder version of the same question from people who assume that building a custom LLM app requires a data science team and a pile of venture money. Neither is true. I built my first version of one of these tools with a single PDF, a handful of Python scripts, and a shaky understanding of vector stores. It worked well enough to become the foundation for tools I still use in the classroom and in startup work today. Here’s the updated version of that story, with the parts that are now outdated stripped out.

Start with the problem, not the technology

This part hasn’t changed and never will. In a classroom, you probably want a tool that lets students ask questions against a textbook or a set of lecture notes instead of Googling around. In a startup, you probably want something that onboards new hires, drafts reports, or fields the same ten customer questions your team answers by hand every day. Pick one problem. One dataset. Resist the urge to build the everything-app on day one. I’ve watched more student projects die from scope creep than from bad code.

The core workflow is still retrieval, but the tools underneath it have moved on

The basic shape of a custom LLM app hasn’t changed: you ingest your content, chunk it, embed it, and let the model retrieve from it instead of guessing from what it learned in training. What has changed is what you’d actually reach for at each step.

  • Chunking: Fixed-size chunking, splitting every document into the same-sized blocks regardless of what’s in them, is showing its age. Semantic chunking, where you let the model find natural breakpoints in the text, holds context together a lot better, especially for anything with real structure like a syllabus or a policy manual.
  • Embeddings: I used to point people at text-embedding-ada-002 by default. That’s dated now. OpenAI’s newer text-embedding-3-large is a fine generalist, and if you’d rather not depend on a single vendor, open alternatives like BGE and E5 have closed the gap and give you more control over where your data actually lives, which matters more than people realize once you’re handling student records or internal company data.
  • Storage and retrieval: FAISS and Chroma are still perfectly good starting points for a prototype. For anything you intend to actually run in production, look at Qdrant or Weaviate. If your use case involves questions that need multiple hops of reasoning across documents (think: “how does this policy interact with that one”), plain vector similarity search starts to strain, and that’s where pairing it with a lightweight knowledge graph earns its keep.

The bigger shift: from “answer engine” to “teammate with tools”

When I wrote about this a year ago, the goal was a chatbot that could answer questions about your content. That’s table stakes now. The more useful pattern in 2026 is giving your custom application actual tools, not just retrieval, but the ability to look things up, call an API, or take an action, through something like the Model Context Protocol. For a classroom app, that might mean the assistant can check a student’s actual grade in the LMS before answering “how am I doing in this class.” For a startup, it might mean the onboarding assistant can actually file the IT ticket instead of just telling the new hire how to file it. The line between “chatbot” and “agent” has mostly disappeared, and it’s worth designing for that from the start rather than bolting it on later.

The problems worth worrying about haven’t changed

Privacy, data security, and hallucination are still the three things that’ll bite you if you ignore them. None of that has gotten easier just because the tools got better. If you’re working with student data or internal company records, know where that data lives and who can see it before you go live, not after. And keep a human in the loop for anything with real consequences. The models are good. They are not infallible, and treating them like they are is how you end up explaining yourself to a dean or a general counsel.

None of this requires a PhD

It requires a real problem, a willingness to start small, and the grit to keep iterating when the first version doesn’t work, because it won’t. When it finally does click, you haven’t just shipped an app. You’ve built something that works around the clock, scales without complaining, and gets better every time you feed it more of your own material. That part hasn’t changed either, and I don’t expect it to.