Think Wider And Help Kinder

Exploring how deterministic document processing can work alongside local models, cloud AI APIs, and existing AI applications.

• Document intelligence and AI
• Local AI with llama.cpp
• Using the OpenAI API
• Using an AI application directly
• A hybrid architecture
• Thinkwider's position

1. Document Intelligence and Generative AI Are Different Layers

AI has created new possibilities for document processing. A document engine can be combined with different AI technologies, but document processing and generative AI do not have to be implemented by the same component.

2. Local AI with llama.cpp

llama.cpp is an open-source inference project for running supported large language models on local hardware. Its server provides an OpenAI-compatible API, which can make it easier for developers to connect applications using familiar API patterns.

Document
    |
    v
DocAnax / LibDocAnax
    |
    +-- extracted text
    +-- document structure
    +-- tables and paragraphs
    +-- search and comparison
    +-- source information
    |
    v
Local application / retrieval
    |
    v
llama.cpp server
    |
    v
Local language model

This architecture can keep document processing and model inference on the user's machine. It may be suitable when offline operation, local data control, or a self-managed environment is important.

Trade-offs of local AI

• Hardware and performance: Inference speed and usable model size depend on memory, processor, GPU acceleration, and other hardware resources.
• Model capability: Smaller or more heavily quantized models may not match leading cloud models on every reasoning, multilingual, or instruction-following task.
• Model freshness: A locally installed model does not automatically acquire new knowledge. It must be updated or connected to suitable external information sources.
• Reliability: Local models can also hallucinate. Keeping data on the device does not guarantee that generated answers are correct.

3. Combining Local Document Processing with the OpenAI API

Another approach is to process documents locally and send selected information to a cloud AI service through an API. This avoids requiring the user's computer to run a large language model and provides access to the models offered by the cloud provider.

Local device                         Cloud service

Document
   |
   v
DocAnax / LibDocAnax
   |
   v
Selected, structured context ------> OpenAI API
                                         |
                                         v
                                    AI response
   <-------------------------------------|

This can be useful when an application needs cloud-model capabilities but still wants document extraction and preprocessing to remain local. Developers should nevertheless consider what information is sent to the service, applicable privacy requirements, network dependency, and API costs.

Conversation history is an application-design problem

Modern APIs can provide mechanisms for maintaining conversation state. However, an API does not automatically manage every part of an application's state. Developers may still need to manage:

• Conversation history and context-window limits
• Document identity and version changes
• Retrieved passages and supporting evidence
• User corrections, decisions, and application state
• Local storage, synchronization, and recovery

A chat transcript alone is not necessarily a reliable record of the current situation. If an application omits relevant history, uses outdated document content, or fails to preserve important decisions, the AI may respond with incomplete or out-of-sync information.

4. Sometimes, Use an AI Application Directly

If the goal is simply to ask questions about a document, building a custom AI application may be unnecessary. Using an established AI application through its web, desktop, or mobile interface may be the more practical choice.

An existing application already provides a user interface and a managed service environment. Depending on the product, it may also provide file handling, conversation management, tools, and model updates.

Building through an API makes more sense when there is a specific need for a custom workflow, integration, automation, user experience, or application-level control.

5. A Hybrid Architecture

These approaches are not mutually exclusive. A system can keep document processing local and select an AI option according to the task, user, or deployment environment.

                    Document
                        |
                        v
               DocAnax / LibDocAnax
                        |
                        v
              Structured document data
                        |
             +----------+----------+
             |          |          |
             v          v          v
          Local AI   Cloud API   AI application

For example, deterministic extraction, structure analysis, indexing, hashing, comparison, and source tracing can remain in the local document layer. A local model, cloud API, or existing AI application can then provide natural-language interaction and generative analysis.

The appropriate choice depends on requirements such as privacy, accuracy, hardware, cost, connectivity, maintenance, and the complexity of the workflow.

6. Thinkwider's Position

Thinkwider will continue to develop LibDocAnax and DocAnax as offline, local-first document technologies. This does not mean that cloud AI is unsuitable, or that every user needs a local language model.

Our approach is to keep the document layer independent from the generative AI layer:

References

• llama.cpp — project repository and server documentation
• OpenAI API documentation