A server with 640 GB of GPU memory: which models can it run, and what can a company gain?

A server with eight accelerators and 640 GB of GPU memory may look like a research-laboratory purchase. Yet a company can have practical uses for it: internal document search, developer assistance, meeting transcription, scanned-document processing and large request queues without mandatory transfers to an external AI service.

This article was prompted by a ServerFlow video published on 17 February 2026. Its creator presents eight NVIDIA A100 80 GB GPUs, two AMD EPYC 9354 processors, 768 GB of DDR5 RAM and four 1600 W power supplies. The quoted RUB 15 million is the creator's estimate at publication, not a current AI Office offer. ServerFlow · What can you do with 640 GB of GPU memory? ↗

We examine candidate models, useful workflows and the limits of a single server. Specifications and model cards were checked on 12 September 2026. We have not benchmarked the particular build shown in the video.

GPU memory primarily holds model weights and data needed for current computations. More capacity allows larger models, additional room for long requests or several independent services. These options compete for the same resources: the largest model and the highest number of concurrent users do not automatically come together.

System RAM serves different purposes, including document preparation, loading weights, databases and application processes. AMD lists 32 cores and DDR5 support for the EPYC 9354; two processors provide the stated 64 physical cores. However, 768 GB of system RAM is not another 768 GB of equally fast GPU memory. Moving weights or computation to the CPU changes performance. AMD · EPYC 9354 specifications ↗

Storage is another project component. It holds original files, indexes, model versions, outputs and backups. A terabyte-scale archive does not mean that the entire archive fits inside a model's context or can be processed within a known time.

Crucially, these are eight separate 80 GB GPUs. An inference server—the software executing model requests—must distribute a large model across them. vLLM documents multiple parallelism approaches, with GPU interconnects among the factors affecting the choice. vLLM · Parallelism and Scaling ↗

Eight A100 cards alone do not establish a fast all-to-all connection. Procurement needs the exact GPU variant, PCIe topology, NVLink/NVSwitch details and test results. A short video cannot replace that specification.

NVIDIA specifies 32 GB for the RTX 5090. Thus, 20 × 32 = 640 matches the aggregate capacity advertised for eight A100s. This is a memory-capacity comparison, not evidence of equivalent model speed. NVIDIA · GeForce RTX 5090 specifications ↗

NVIDIA lists different specifications for A100 80 GB PCIe and SXM variants: memory bandwidth of 1,935 and 2,039 GB/s, and maximum TDP of 300 and 400 W respectively. NVIDIA · A100 80GB specifications ↗

The video's power comparison also needs context. Twenty RTX 5090s at 575 W total 11.5 kW on their specifications; eight A100 PCIe cards at 300 W total 2.4 kW. The ratio is close to five, but it compares accelerator ratings rather than two complete systems performing the same task. CPUs, RAM, fans, power conversion and execution time are absent.

A more useful business measure is the cost of processing a thousand comparable requests at the required quality and latency. It comes from testing the selected model and configuration, not dividing gigabytes or advertised compute figures.

The following models have published weights and documentation and illustrate the range of possibilities. They are candidates for selection and testing, not installations we have verified on the video's server. Each deployment should pin the model version, weight format, context limit and software environment.

Qwen3-32B has 32.8 billion parameters and is a candidate for a text assistant. Qwen3-Coder-30B-A3B-Instruct targets code, with 30.5 billion total parameters and roughly 3.3 billion active. These options allow a project to begin with employee tasks without assigning the entire node to the largest available model. Qwen · Qwen3-32B model card ↗ Qwen · Qwen3-Coder-30B-A3B-Instruct ↗

Llama 4 Scout and Maverick differ considerably: approximately 109 billion and 400 billion total parameters, with 17 billion active in each. Both accept text and images. Russian is absent from Meta's listed supported languages for Llama 4, so suitability for Russian-language company documents needs separate evaluation. Meta · Llama 4 Scout model card ↗ Meta · Llama 4 Maverick model card ↗

The video mentions Gemini. For self-hosting, this article considers Gemma, Google's separate family with published weights. Installing Gemma does not create a local Gemini copy. Gemma 3 27B, for example, accepts text and images and generates text responses. Google · Gemma 3 27B model card ↗

Each model's licence must be checked against the intended use. Downloadable weights do not imply identical terms across model families, jurisdictions or distribution methods. This article makes no promise of unrestricted use.

DeepSeek is a family containing substantially different models. For a concrete example, consider DeepSeek-R1 with 671 billion total parameters and about 37 billion active per token. Smaller distilled models are also published; running a 32B version is not the same as running the complete 671B model. DeepSeek · DeepSeek-R1 model card ↗

A mixture-of-experts, or MoE, architecture reduces active computation but does not turn the full weight set into a compact 37B model. Keeping all weights resident requires accounting for all parameters. Offloading experts to system RAM introduces additional transfer and latency considerations.

An initial weight estimate is parameter count × bits per parameter / 8. The table uses decimal GB—billions of bytes. It estimates the raw weight payload, not the packaged file size or the memory consumed by a running service.

Weights also need room alongside the KV cache: intermediate data retained while generating an answer. Cache demand depends on model architecture, request length and concurrent conversations. Successfully loading a model therefore does not establish a production service for the whole company.

At 16 bits, full DeepSeek-R1 needs around 1.34 TB for weights alone, exceeding 640 GB of GPU memory. The approximately 336 GB estimate at 4 bits makes this configuration a candidate for further testing. Quantisation metadata, buffers and request memory add to that total. At 8 bits, headroom is particularly problematic: calculations must use actual bytes and available capacity, accounting for GB versus GiB.

Quantisation reduces the precision used to represent weights. It saves memory but requires quality and compatibility checks. vLLM's Ampere compatibility table includes AWQ and GPTQ while ordinary FP8 W8A8 is unsupported. Some FP8 storage modes use other compute paths; an FP8 label alone does not establish A100 compatibility or speed. vLLM · Quantization and supported hardware ↗

Full DeepSeek in a suitable compressed format is therefore a potential engineering project. Memory capacity alone cannot establish its performance, maximum context or user count. The selected checkpoint, format and engine need testing on the intended accelerators before purchase.

Several specialised models may provide more value than one very large chatbot. Examples include:

• Qwen3-Embedding-4B converts text into numerical representations for semantic retrieval. It can form part of RAG: finding relevant passages before generating an answer. Qwen · Qwen3-Embedding-4B ↗
• Qwen3-VL-32B-Instruct is a candidate for evaluating document images, diagrams and photographs. Its developer describes OCR capabilities; extracted totals, identifiers and technical labels still require checks. Qwen · Qwen3-VL-32B-Instruct ↗
• Whisper large-v3 provides speech recognition for transcripts. Speaker identification, meeting minutes and task creation need additional components. OpenAI · Whisper large-v3 model card ↗
• FLUX.1-schnell generates images from text. It can be considered for sketches and illustrations, with output quality and input-use rights evaluated. Black Forest Labs · FLUX.1-schnell ↗
• Wan2.2 provides video-generation models. Its official project includes an I2V-A14B workflow requiring a GPU with at least 80 GB of VRAM and offloading to system RAM, but generation time and concurrency depend on the selected configuration. Wan-Video · Wan2.2 official project ↗

These services need not all run simultaneously. Video generation, archive processing and interactive answers compete for GPUs. Predictable operation requires queues, priorities and scheduling for heavy jobs. Analysing an image also does not establish an accurate engineering measurement or automatically confirm workmanship.

The first scenario is an internal knowledge base. An employee asks which requirements apply to a project or how two contract versions differ. The system retrieves authorised sources, provides evidence and drafts a response. Value depends on retrieval speed and citation accuracy; a larger model cannot fix outdated documents or incorrect access rights.

The second is commercial document preparation. A request can be parsed into items, matched against a catalogue and turned into a draft proposal. Software should calculate prices, discounts and totals using approved rules. A person reviews the result before sending it. The objective is less repetitive entry and faster preparation while retaining control over company commitments.

The third is assistance for an internal development team. A local model can explain code, propose changes and tests, and help analyse error logs. Repositories remain within the chosen infrastructure, while suggestions still require tests, review and restricted access to production systems.

The fourth is archive processing. Authorised meeting recordings, scans and technical documents can enter an overnight queue, producing transcripts and retrieval-ready material for the next day. Throughput should be measured in pages, audio minutes or defined document types, not abstract “AI terabytes”.

For construction and building operations, the same infrastructure could support report analysis, instruction retrieval and issue summaries. The AI Office Field concept links such capabilities to assignments and acceptance. The server supplies compute; review rules and the industry-specific module remain separate implementation work.

A node can host one large model distributed across GPUs, or several smaller models and replicas. The latter approach can support different workflows, separate workloads and allow one service to be updated without replacing every other service at once.

An illustrative test layout might assign four GPUs to a primary text model, two to document processing, one to queued speech and indexing tasks, and one to experiments. This is only an example allocation of eight accelerators. It does not apply if the selected model needs the entire node; compatibility and memory headroom must be checked for each group.

Employee count is not the same as concurrent request count. A hundred people asking occasional short questions create a different workload from twenty people uploading long contracts simultaneously. Tests should record time to first response, total processing time, input and output lengths, queue depth and error rates.

A server is useful when it delivers the required work result with acceptable latency and verifiable quality. Model size is only one factor in that outcome.

Three activities should be separated. RAG connects current documents to an existing model through retrieval. Fine-tuning changes behaviour using prepared examples. Training from scratch creates a model from a large dataset and requires a separate resource assessment.

LoRA or QLoRA are reasonable adaptation methods to investigate on a node of this class. They train a relatively small set of additional parameters. PEFT documentation describes combining quantisation with LoRA to reduce memory requirements. Model size, example length, training steps and available time still constrain the project. Hugging Face PEFT · Quantization ↗

The ability to load a 671B model for inference does not establish the ability to train all its parameters on the same machine. Training requires additional memory for gradients, optimiser states and intermediate computations. Changing prices and current policies are usually better maintained in a controlled knowledge base than repeatedly embedded in weights.

This equipment belongs in a prepared server room or suitable data centre. Power, cooling, noise, weight, rack depth, networking and maintenance are part of the project alongside GPUs. Four 1600 W power supplies do not imply a constant 6.4 kW draw; allowable load after a supply failure depends on the redundancy design.

Energy costs should use whole-system measurements. An illustrative average draw of 3.5 kW × 24 hours × 30 days equals 2,520 kWh per month, excluding separately accounted room cooling. This is not a measurement of the video's build. The final cost depends on tariffs and hosting conditions.

Ownership costs also include support, backups, storage, updates, spare components and engineering time. A single node with redundant power supplies can still fail elsewhere. If company workflows depend on AI, define acceptable downtime and test recovery or an alternative execution route.

Local inference can reduce mandatory calls to external AI APIs. Offline operation nevertheless requires prepared weights, dependencies and local data sources. Telemetry, cloud integrations and external tools need separate checks. Data protection depends on permissions, networks, logging and backups as well as the model's physical location.

For AI Office, such a server is a possible compute layer. Models run on it while the platform manages documents, assignments, permissions and approvals. A larger model can assist analysis, while a smaller one handles a routine operation. Routing and workload allocation must be designed for the company.

The current prototype supports Ollama and a compatible HTTP provider, retrieval from authorised sources, tasks and action approval. It has not been installed or validated on an eight-A100 server. An HTTP connection does not establish compatibility with every model, proven multi-user performance or high availability.

OCR, audio and video processing, and email, CRM and accounting integrations require separate implementation and validation. Hardware discussions should therefore begin with the workflow: incoming data, required result, responsible reviewer and acceptable latency.

A server of this class is worth evaluating when a company has sustained workloads, a justified need for larger models or multiple services, and the resources to operate it. For a few employees making occasional requests, start by testing a smaller node or renting a comparable configuration. Idle expensive accelerators remain part of the cost.

Select two or three workflows and prepare representative real tasks. Compare suitable models on quality, latency and memory; then test simultaneous requests, long documents and recovery after failure. Performance must be assessed in the operating conditions employees will actually create.

Potential benefit can be expressed as released staff capacity. If 80 employees save 15 minutes on each of 22 working days, that totals 440 hours per month. This is an illustrative calculation requiring pilot confirmation, including answer review and error correction. Released hours do not automatically become payroll savings.

Eight A100s give a company substantial choice in local models and workload allocation. Prepared data, a functioning workflow and measurable results create the actual value. With AI Office, we propose starting there: define a pilot, evaluate models and then decide whether the company needs a 640 GB node or a more compact configuration.