Llm on Marian Zeis Blog

How I Use AI for Development and Why Context Matters

Mon, 20 Apr 2026 20:00:00 +0200

Series note: This post is part of my AI ABAP development series, where I go from AI development in general, to ABAP-specific problems, and then to ARC-1.

When it comes to AI, I do not really think about magic.

I think more about another colleague helping me out.

Sometimes that means architecture. Sometimes implementation. Sometimes testing, review, CI/CD, or deployment.

But this only works with trust, and trust needs communication.

If I do not know something, I ask. If code looks strange, I ask. If tests fail, I ask.

The less I need to ask, the more likely it is because I already gave enough context up front: architecture, documentation, constraints, examples, and what I actually want. Even then I still review the result.

That is why using AI well is a skill you build over time.

My first larger projects with AI were much less smooth than newer ones. Over time I learned that better context, better tooling, and better guardrails lead to much better results. AI can speed things up a lot, but it does not remove engineering judgement.

My workflow changed over the last year

Last year I was mostly just using ChatGPT directly, before tools like Claude Code became more normal. I asked something, copied the generated code into my project, checked if it runs, copied the error back if it failed, and repeated that loop.

The error message was context. The result of trying it out was context. That manual loop slowly improved the output.

What changed over time was not only the model. More and more the AI can get context itself.

Before I had to spoon feed much more manually. Now tools can search, read docs, inspect files, and in some cases get the needed context on their own. For me that is one of the biggest improvements, because the AI does not only answer, it can first understand more.

That is also why I liked How I write software with LLMs by Stavros. Not because I use exactly the same workflow, but because it shows quite well that this is not only about the model. The workflow around the model matters a lot as well.

Context matters more than people first think

A lot of the time the biggest value is not even that the model writes code for me.

Very often it helps with understanding. It can explain a codebase, compare options, write down a plan, summarize documentation, help with tests and refactoring, or point me to the next thing I should check. For a lot of developers that alone is already useful, without going fully agentic.

In many projects the important information is not only in the source code. It is in documentation, issues, diagrams, product docs, logs, dumps, table data, screenshots, pipeline output, and things people in the team just know. If the model only sees one file, it often gives you an answer that sounds nice but misses the actual problem.

That is why I really liked the post Context is the Difference Between Slop and Shipping. I think the core point is right.

If you do not understand the system first, or if the model does not get the right grounding, then you just get plausible looking nonsense faster. It looks like momentum, but often it is just wrong output with confidence.

One of the first places where I really noticed this was UI5, especially with controls like the UI5 Wizard. I had to explain the API again and again, and without that extra context the output was often just wrong. That was one of the first moments where I noticed more clearly that in SAP development good context is not optional.

Especially in SAP I noticed pretty quickly that you need some kind of grounding. You want to be able to say: use this documentation, use this guideline, use this code reference, trust this as the source of truth and do not just invent something.

It still hallucinates sometimes, that does not fully go away, but it becomes less. Especially weaker models benefit a lot from having the right information at the right time.

Why I care so much about tooling around the model

That is also one reason why last year I started building my own MCP server for context. It can search GitHub repositories, SAP Help, and related sources, so I can bring better grounding into the workflow.

For SAP development especially this helped, because so much is not directly visible in one code file. If the model can pull in SAP documentation, guidelines, and related references, it gets pushed much more into the right direction.

I now almost always have that SAP docs setup with me. At the beginning it felt even more important, because the models were weaker and hallucinations were stronger. Today it is a bit better with the stronger frontier models.

I also noticed this with BTP and Cloud Foundry work. There the models can often already use the CLI and public documentation quite well. So the models are improving.

But the more special something gets, the more hidden it is, or the more SAP specific it is, the more important that extra context becomes again.

That is also why I do not find the current AI discussion fully convincing sometimes. People talk a lot about models, but much less about the system around the model. To me, context and tooling matter more in real projects than benchmark discussions alone.

Model quality still matters, but it is not the whole story

I already wrote two separate blog posts about ABAP LLM benchmark results, one more around code generation and one around understanding.

My takeaway there was that strong general models are currently still the more practical choice for me, and that tooling and grounding are often the bigger lever than hoping for one SAP specific model to solve everything by itself.

What also helps a lot is that my MCP SAP docs setup is not only pure API documentation. There are also style guides, DSAG guidelines, and other recommendations in there. These are often not hard rules, but they still push you in a certain direction. So even if I do not know the exact recommendation by heart, I can ask the model to look for guidelines and it gets nudged more automatically towards a better direction.

And context is not only documentation. If I say change this class, then the context is not just that class. It is also the tables that are used, the other classes that are called, similar implementations, internal guidelines, what this system can even do, and what is already there in working code. That broader context improves quality a lot more than just throwing one file at the model.

I am still skeptical of full agentic hype

There is also a very practical spectrum in how much AI help you even want. You can just ask questions about context. You can ask it to find docs and let you read them. You can use it for analysis and planning. Or you can let it implement things more autonomously. That is not one binary choice. It is more a slider, and depending on the project, the team, and the risk, you move that slider differently.

At the same time I am still not sure that going more agentic automatically makes you faster. Pure AI assistance already helps me a lot. But once it becomes more agentic, other things get harder again. The coding itself is very fast, but then you spend more time on plans, instructions, guardrails, review, and correction.

This also matches quite well with the METR study on experienced open-source developers. What I found interesting there is not some simple “AI bad” takeaway, but more the gap between feeling faster and actually being faster.

If less time goes into typing code, but more time goes into prompting, reviewing, waiting, and correcting, then it can still feel fast while the total time says something else. That matches quite well with what I notice myself.

At the same time I also do not want to overcorrect too much into pure skepticism. The post Eight years of wanting, three months of building with AI is a good counterexample. AI helped a lot there, but the first more maximalist agentic approach also created a messy codebase, and things only got better again with tighter ownership and stronger review. That feels much closer to reality than either extreme.

For me there is still a big difference between strong AI assistance and just letting the agent fully take over. Writing the code is now often the easiest part. I spend more time before coding on research, testing, planning, and architecture. And I spend more time after coding on review, rework, manual testing, and documentation.

It is still faster in my opinion, but the time moved.

Process helps, but process is also overhead

This also fits well to How I teach Claude Code to work my way. What I like there is that the focus is not on one magical prompt, but on process. Explore first, think first, plan first, then execute.

I do not use exactly the same process everywhere, but the general idea fits very well to what I learned myself. If you give the AI a process, it usually follows one. If you do not, it drifts more easily. At the same time, process also creates overhead, so it is only worth it when the task is big enough.

That is also why I often want the model to first find a reference before it gives me a strong answer. Search for the SAP documentation. Find the guideline. Show me a similar implementation. Do not just make something up because it sounds right.

Why this gets harder in SAP

Security is also part of that trust question. How much trust do you put in the llm itself, how much in the model provider, and how much in the tools around it? How much security are you willing to give up to move faster? For toy projects people often accept a lot of risk. In enterprise development that looks very different.

In SAP this matters even more, because so much important context is outside the code itself. The actual code change can be very small, but understanding why you need that change can require docs, config, logs, dumps, table entries, business process context, authorizations, and knowledge about how the landscape is set up.

At the same time SAP is also moving. I was pleasantly surprised when SAP released MCP servers for UI5, CAP, and Fiori.

In that world it already feels much more natural, because a lot of that development is off stack anyway. You work locally, use git, run pipelines, get peer review, and if something goes wrong you can usually revert it. That is much easier territory for AI assisted development.

So the interesting question for me is not if AI can help in SAP development. It already can. The more interesting question is where it helps today, where it becomes risky, and why especially ABAP development is still much harder than frontend or CAP development.

In frontend and CAP development, AI can already fit quite naturally into workflows many teams already have. In ABAP, things look very different.

The next post is where I want to look more concrete at that gap: what is already possible today, which patterns exist, and why enterprise guardrails matter much more there.

Discuss this post

References & links

SAP’s ABAP-1 Loses Every ABAP Benchmark, Even “Explaining”

Tue, 03 Mar 2026 22:00:00 +0100

Previous post (code generation benchmark): Benchmarking LLMs for ABAP

Live benchmark results (old + new): abap-llm-benchmark.marianzeis.de

In my first evaluation (based on the TH Köln benchmark paper), I extended the original setup with additional models and focused on a very concrete question: how well can LLMs generate ABAP code that actually compiles and passes ABAP Unit tests?

I also tested SAP’s model ABAP-1, and it performed very poorly for code generation. To be fair: SAP also states this in the documentation. ABAP-1 is primarily meant for explaining ABAP code not for reliably generating full working implementations.

So I built a second test that targets exactly that: understanding and explaining ABAP.

The “Understanding” test

In the “Understanding” benchmark, the model gets existing ABAP code plus the corresponding ABAP Unit tests (so this is not a generation task). From that, it must extract concrete facts as structured JSON, for example:

which classes and methods are relevant
expected behavior and validations
inputs and outputs

The JSON output is then scored automatically against a reference, so we can measure in a reproducible way how well a model understands ABAP code and describes it correctly. This works without running SAP or ADT.

Here is the key result as a chart (cumulative success across feedback rounds):

What this says about ABAP-1

As expected, ABAP-1 is clearly better in this “Understanding” setup than in code generation. But it still does not keep up with current general-purpose models.

Even the current Anthropic Haiku model (optimized for speed and cost) beats ABAP-1 on this benchmark.

That makes the practical conclusion pretty simple: for almost any ABAP use case, ABAP-1 is not a good default choice. It is priced closer to premium models, but does not deliver premium results. So the “maybe it is cheaper” argument also does not hold.

New models

After the first post, people asked for more models to be added. I’m tracking those requests as GitHub issues here: LLM-Benchmark-ABAP-Code-Generation issues

I added the following models:

DeepSeek Reasoner
Mistral Large 2512
GPT-5.3 Codex
Claude Haiku 4.5
Gemini 3.1 Flash Preview

To put the “new models” into context, here is the updated code generation overview (this is the original benchmark: generate ABAP, then iterate with compiler/test feedback up to 5 rounds). The plot already includes the new lineup:

GPT-5.3 Codex produced very strong results and even beat Opus 4.5 on some metrics. My guess is: if we compare against Opus 4.6, the gap will likely be small (or they might end up roughly equal).

Depending on the metric, Opus 4.5 is still excellent: it has the highest first-try success rate (Round 0) and the best AUC across all feedback rounds.

Unfortunately, Mistral’s current “best” model performed worse than I expected and does not look like a great choice for ABAP.

For price/performance, DeepSeek Reasoner is the clear winner in this lineup.

Using the public list prices (at the time of writing), Codex is at around $1.75 per 1M input tokens, while DeepSeek Reasoner is at $0.28 per 1M input tokens (more than 6x cheaper). The bigger difference is output: DeepSeek output is $0.42 per 1M tokens, while Codex output is around $14.00 per 1M tokens (about 33x).

Gemini 3.1 Flash Preview

Gemini 3.1 Flash Preview also surprised me with strong results. It is cheap and in my runs it was even slightly ahead of GPT-5.3 Codex, while being significantly cheaper.

At the time of writing, Gemini Flash was roughly $0.50 per 1M input tokens and $3.00 per 1M output tokens, which is a big difference compared to Codex pricing.

Unfortunately, I could not test Gemini 3.1 Pro: the rate limit was effectively capped at ~25 requests per minute and I repeatedly hit 503 errors (“system overloaded”). With several thousand requests, that made a full benchmark run impractical. My guess is that Gemini 3.1 Pro would be similar to, or better than, GPT-5.3 Codex, but Flash already delivered very good results.

Why Haiku 4.5 did so badly here

Haiku 4.5 failed systematically because it kept using CDS-style type notation like abap.char(20) in classic ABAP implementations (syntax errors). Worse, misleading parser error messages made it hard for the model to identify the real root cause, so it did not correct itself even across multiple feedback rounds.

A practical takeaway

For me, this is now a pretty solid baseline to decide which models are worth considering for ABAP, and which ones are not, depending on whether you care more about API calls (cost) or daily development (quality + speed).

One important reminder: these models acted only on what they already “know”. Results can be improved a lot with tooling and better context, for example:

ABAP MCP Server for ABAP best practices and keyword documentation: marianfoo/abap-mcp-server
abaplint for static checks and fast feedback loops: abaplint.org

With that kind of tooling and feedback loop, you can generate very good ABAP code even if you “only” use a strong general model.

And yes, my conclusion from the first post still stands: I would not recommend ABAP-1. If SAP wants better outcomes, I think the bigger lever is not “yet another model”, but better tools and better integration so frontier models (OpenAI, Anthropic, Google) can reliably produce correct ABAP in real-world projects.

You can still suggest models I have not tested yet, but benchmarks like this are not cheap. For now I will pause adding more models unless there is a very strong reason to do another run.

References & links

TH Köln paper: Benchmarking Large Language Models for ABAP Code Generation
Previous post (code generation benchmark): Benchmarking LLMs for ABAP
Live results website: abap-llm-benchmark.marianzeis.de
Original benchmark repo (TH Köln): timkoehne/LLM-Benchmark-ABAP-Code-Generation
Dataset (prompts + ABAP Unit tests): dataset folder
Benchmark repo (my fork with updates): marianfoo/LLM-Benchmark-ABAP-Code-Generation
Model requests / discussion: GitHub issues
SAP ABAP-1 documentation: SAP ABAP-1
ABAP Cloud Developer Trial (Docker image used by the benchmark): SAP-docs/abap-platform-trial-image
Batch APIs (used for running large benchmarks cheaper): OpenAI Batch API, Anthropic batch processing
Pricing references:
- OpenAI: API pricing
- DeepSeek: Pricing (official docs)
- Google Gemini: generative AI pricing
- Anthropic: Claude API pricing

Benchmarking LLMs for ABAP: Why ABAP-1 Isn't a Code Generator (Yet)

Mon, 09 Feb 2026 09:30:00 +0100

Live benchmark results: abap-llm-benchmark.marianzeis.de

In a lot of SAP webcasts and webinars, especially around AI, the question comes up very early: which model are you using, and which one do you recommend?

For CAP and UI5 the answer is usually pretty simple: use the current best model from Anthropic. If you add good context via MCP servers from the community or SAP, you are basically fine. There is just a lot of public knowledge available, and most of it is in JavaScript/TypeScript, which LLMs handle extremely well.

With ABAP it’s different.

ABAP is not the framework, it’s the language. And compared to JavaScript/TypeScript, far less ABAP knowledge and code is publicly accessible, so models simply know much less about it.

Sure, you can help with extra context and tooling, but if you want this to work well, the model should already have a solid ABAP base. A good foundation makes it much easier to build on top.

So how do you figure out which models are actually best for ABAP? Anthropic, Google, OpenAI, or maybe ABAP-1 from SAP?

Luckily, TH Köln recently published a paper that answers exactly this question: Benchmarking Large Language Models for ABAP Code Generation.

The benchmark

The paper builds a benchmark so different LLMs can be compared fairly for ABAP code generation.

It includes 180 tasks (HumanEval adapted to ABAP plus a set of ABAP/SAP practical scenarios). The flow is fully automated: the model generates ABAP code, the code is created and activated in a fixed SAP environment (ABAP Cloud Developer Trial image documentation), and then ABAP Unit tests run against it.

If syntax checks or tests fail, the model gets the error messages as feedback and can try again for up to 5 rounds. In the end, it’s not about whether the code “looks plausible”, but whether it actually passes the tests (and how quickly it gets there).

The nice part: the authors published everything.

Original code: LLM-Benchmark-ABAP-Code-Generation
Dataset (prompts, unit tests, expected classes): dataset folder
My updated benchmark repo (additional models, speed-ups, plots, data, webpage): marianfoo/LLM-Benchmark-ABAP-Code-Generation

Updating the benchmark

One small problem: the evaluation in the paper is from mid-2025 (for example using GPT-5 and Claude Sonnet 4). The model landscape moves fast, and now we already have things like GPT-5.3 and Claude Opus 4.6.

So I wanted a way to update the results and include additional models that are relevant for ABAP.

I built on the original benchmark and added runs for additional models, so I can compare them against the original GPT-5 and Sonnet numbers:

Anthropic Claude Opus 4.5 (4.6 will be next)
OpenAI GPT-5.2
SAP ABAP-1

I also made a few changes to speed up execution:

For OpenAI and Anthropic I used the vendor batch APIs like in the original code (OpenAI Batch API, Anthropic batch processing). GPT-5.2 Codex was not available via batch for me yet, but that model will be next.
ABAP-1 has no batch API, so I implemented a parallel execution with multiple threads.
I also parallelized the execution on the ABAP Trial system itself to reduce overall runtime. I’m happy I have a well-equipped MacBook, but during the runs RAM usage and fan speed were definitely not idle.
Since there is a lot of Python code and I don’t know Python well enough, all the coding was done with AI help.

The next step is a small website with an overview of the current results and an easy model comparison.

You can find the current results here: abap-llm-benchmark.marianzeis.de

So what is the best model for ABAP?

Not too surprising: the best results came from GPT-5 and Claude Opus 4.5, with a small edge for Opus.

To make it more concrete, here are the cumulative success rates from my current comparison (Round 0 = first attempt, Round 5 = after up to 5 feedback iterations with compiler/test errors):

Model	Round 0	Round 5
Claude Opus 4.5	31.61%	78.72%
GPT-5	19.28%	77.11%
GPT-5.2	16.33%	64.00%
SAP ABAP-1	10.67%	19.89%

Here is the same as a chart:

Cost (rough numbers)

One more thing people ask quickly: cost.

For my run, I ended up at about $20.26 for GPT-5.2, $39.76 for Claude Opus 4.5 (8,973,159 tokens), and EUR 98.80 for ABAP-1 (17,983,990 tokens).

To be fair: for OpenAI and Anthropic I used batch processing, so these numbers are already discounted (batch is usually cheaper, sometimes roughly half compared to the regular API). ABAP-1 has no batch API, so there is no discount like that.

Why was ABAP-1 so much more expensive? The benchmark is iterative: when activation or ABAP Unit tests fail, you send the error back to the model and try again. Since ABAP-1 produced errors much more often, many prompts did not get “finished” early and had to be sent again and again, which drives up tokens and cost.

What I find interesting here is not only the final number, but also how the models behave across the feedback loops.

Why is GPT-5.2 worse than GPT-5 (in this benchmark)?

I can’t prove the exact reason (this is a black box), but based on the patterns in the benchmark it looks like GPT-5 is simply better at using compiler feedback to dig itself out of ABAP syntax and declaration issues.

GPT-5.2 still does fine, but compared to GPT-5 it is:

less consistent across reruns (about 41% of prompts solved 10/10 vs about 59% for GPT-5)
less likely to recover from certain ABAP-specific errors within a few feedback rounds

My best guess is that GPT-5.2 is tuned differently (for example more general, faster, cheaper, or optimized for other coding workflows), and that does not translate 1:1 to “generate a full ABAP class that compiles and passes ABAP Unit tests”. Maybe when using the codex model, the results are better.

Claude Opus 4.5

Opus starts very strong in Round 0. That usually means less back and forth for basic ABAP class structure and syntax, and more time spent on the actual logic.

In practice, this is what you want: you want to reach the unit test stage quickly, because that feedback is much more actionable than “your code does not activate”.

SAP ABAP-1

ABAP-1 is clearly not competitive for code generation in this benchmark, and it also benefits much less from the feedback loops.

This matches what SAP writes in the docs: ABAP-1 is mainly meant to explain ABAP code, while generation is experimental. For me, ABAP-1 is still useful as an “ABAP explainer”, but if you want to generate correct ABAP code fast, you still want a strong general model plus good tooling and feedback loops.

What this means in practice

This matters most when you work with APIs and actually build things. If you only call APIs, the model choice matters more. If you actively develop, it matters a bit less because we can add a lot of context and feedback loops.

When developing ABAP, we can provide much more context (for example via my ABAP MCP server), catch syntax errors early, run tools like abaplint, and then feed that back into the LLM to improve the code, basically like in TypeScript. That usually gets you to pretty good ABAP code that also follows your rules.

Depending on cost, I’ll try to keep this up to date with newer models. If you have a model you want to see in the comparison, just open an issue in the benchmark repo: GitHub issues.

Big thanks to Stephan Wallraven, Tim Köhne, Hartmut Westenberger, and Andreas Moser for creating the foundation with their benchmark and paper.

Supercharged Local Open Source Joule for Consultants: Search Your Data & Code

Wed, 29 Oct 2025 09:30:00 +0100

This post is a mirrored copy of my LinkedIn article, kept here so it remains searchable and independent from external platforms. You can still find the original on LinkedIn: LinkedIn Pulse article.

Joule for Consultants costs 250 € and mainly searches SAP Help, SAP Community and SAP for Me (Notes). That is useful, but it stops where real projects start: your own systems, codebase and sensitive data. Even though public assistants like ChatGPT and Claude can already reach SAP Help and SAP Community, the real gap is internal/authenticated data and code.

For this case I created a LibreChat fork that includes and auto-starts SAP MCP servers (Docs, Notes, S/4 OData and ABAP ADT) via Docker. So the MCP Server are local inside your Docker and no external MCP Server is used. LibreChat is an open‑source local chat UI with MCP support and built‑in Ollama integration, and it also supports many other model providers. You get a private assistant that can read official docs, inspect code and data from your systems without sending customer context to third‑party clouds. The fork’s documentation is here: sap‑mcp‑docs.

Running the assistant locally means the client, tools and, if you choose, the model stay on your machine (LibreChat supports Ollama), so you avoid pasting sensitive context into public chats. By adding S/4 OData and ABAP ADT, you bring real metadata, code and data into the loop.

Which MCP servers are used here?

The SAP Docs MCP provides unified search and fetch across SAPUI5/OpenUI5, CAP, ABAP keyword docs, SAP Help and SAP Community. The SAP Notes MCP is for searching and fetching SAP Notes when you have an error, OSS number or concrete symptom; it requires an S‑User certificate (Passport). The ABAP ADT MCP reads ABAP sources, DDIC structures and tables content via ADT APIs so you never paste code into chat and still get precise context. The S/4 OData MCP discovers services and exposes safe CRUD tools so your assistant can read and write via approved APIs.

How this works in practice

In my example I used o4‑mini from OpenAI, but LibreChat supports many model providers including OpenAI, Anthropic, Google, Azure AI Foundry and even SAP AI Core (not tested by me). The key point: you can also use a local model via Ollama. Depending on your use case, a local model is smart enough to call the right tools and MCP servers. I also use an ABAP Cloud Developer Trial instance connected via the S/4 OData and ADT MCP servers.

SAP Notes lookup + ABAP table verification

This first example flow combines the SAP Notes MCP and ABAP ADT MCP. First the assistant searches and fetches a specific SAP Note (sap_note_search, sap_note_get), then verifies the relevant DDIC table and fields via ADT. This confirms the symptom and the fix steps before proposing a safe, auditable corrective action (for example, an OData action). This is the pattern for many “error → note → verify → fix” tasks.

Service metadata + ValueList guidance

The assistant uses the S/4 MCP to discover the Football service and inspect entity schema (search-sap-services, discover-service-entities, get-entity-schema). It then pivots to the SAP Docs MCP to sap_help_search and sap_help_get the ValueList documentation and shows exactly where to add annotations. This marries live service metadata with the official docs so consultants can implement confidently, then validate in a Fiori app.

CDS view Value Help research

For CDS‑centric cases, the assistant focuses on SAP Docs MCP to collect the canonical examples for Common.ValueList and relevant ABAP/CDS annotations. If needed, ABAP ADT MCP can retrieve the current CDS definition (GetStructure/GetInclude) to compare the as‑is state and generate the minimal edit plan.

Other options and clients

There are other options and clients to use these MCP servers. If you prefer the Microsoft ecosystem, Visual Studio Code’s GitHub Copilot Chat supports MCP servers so you can connect the same SAP MCP endpoints there, as described in the VS Code Copilot Chat MCP guide. This is a simpler setup compared to the LibreChat fork. For a more governed setup you can use Copilot Studio together with the hosted S/4 OData MCP server from Nessi NV and the SAP Docs MCP Server.

I talked about these MCP servers and showed a live example of Copilot Studio and other LLM clients at SAP Inside Track Munich 2025. The recording is here: SAP Inside Track Munich 2025.

You can also build your own LLM client and use the MCP servers directly. I tried a quick PoC with the current libraries and tools. It worked, but turning it into a production‑ready assistant with robust MCP support is non‑trivial; I would not recommend rolling your own for production. The PoC is here: Custom LLM Client.

Getting started

To run this on your own machine, clone the repository, configure the .env and librechat.yaml files and start the Docker container. The docs are in this folder and the setup guide is here: MCP_SETUP_GUIDE.md.

Minimal path (see guides for details):

# Clone and configure
git clone https://github.com/marianfoo/local-llm-client-for-sap-consultants-librechat
cd local-llm-client-for-sap-consultants-librechat
cp .env.example .env
cp librechat.example.yaml librechat.yaml

# Configure the .env and librechat.yaml files

# Start in Docker
docker compose up -d

Security: running AI and MCP locally

Because the client, MCP server and (optionally) the model run on your machine, prompts, retrieved docs and tool results stay inside your environment. When you do connect to external sources, do it intentionally and stick to trusted endpoints. Operationally keep it simple and start read-only by default and enable write tools only when you require it. The short version: a local, open-source stack is the safer option here; unlike cloud assistants, your information never leaves your machine unless you choose to send it.

Conclusion

With MCP servers, LLM clients become smarter and more useful in real projects. LLMs simply cannot know everything, especially internal context that lives on enterprise networks and changes quickly. Because the Model Context Protocol is still relatively new in the enterprise, treat adoption with care and integrate it securely. Keeping information on your own machine and using open-source libraries lets you see exactly what runs and how.

SAP clearly wants to position Joule for Consultants as a unique product; today it still lags behind community projects for these use cases. At TechEd next week we’ll see whether community MCP servers are embraced and whether SAP’s own products improve enough to justify the price.

References

Custom LLM Client PoC: https://github.com/marianfoo/custom-llm-client
SAP Inside Track Munich 2025 session: https://www.youtube.com/watch?v=vTZCeaxsOwU
SAP partner listing (S/4 integration via OAuth): https://www.sap.com/products/artificial-intelligence/partners/nessi-nv-ai-data-enabler.html?countryCode=US
MCP clients overview (Claude Desktop etc.): https://modelcontextprotocol.io/clients/
Copilot Studio: https://adoption.microsoft.com/ai-agents/copilot-studio/
VS Code Copilot Chat MCP docs: https://code.visualstudio.com/docs/copilot/chat/mcp-servers
CAP MCP server (official): https://github.com/cap-js/mcp-server
CAP MCP plugin (community): https://github.com/gavdilabs/cap-mcp-plugin
S/4 OData MCP: https://github.com/lemaiwo/btp-sap-odata-to-mcp-server
ABAP ADT MCP: https://github.com/mario-andreschak/mcp-abap-adt
SAP Notes MCP: https://github.com/marianfoo/mcp-sap-notes
SAP Docs MCP: https://github.com/marianfoo/mcp-sap-docs
LibreChat Ollama guide: https://www.librechat.ai/blog/2024-03-02_ollama#getting-started-with-ollama
LibreChat model providers: https://www.librechat.ai/docs/configuration/pre_configured_ai
LibreChat website: https://www.librechat.ai/
LibreChat fork setup guide: https://github.com/marianfoo/local-llm-client-for-sap-consultants-librechat/blob/main/sap-mcp-docs/MCP_SETUP_GUIDE.md
LibreChat fork docs folder: https://github.com/marianfoo/local-llm-client-for-sap-consultants-librechat/tree/main/sap-mcp-docs
LibreChat fork (SAP consultants, with MCP servers): https://github.com/marianfoo/local-llm-client-for-sap-consultants-librechat