Fine-Tune Neural Chat with Ertas

Overview

Neural Chat is a conversational language model developed by Intel Labs, fine-tuned from Mistral 7B with a focus on delivering high-quality chat performance with efficient inference on Intel hardware. Released in November 2023, Neural Chat 7B v3.3 achieved the top position on the Hugging Face Open LLM Leaderboard at the time of its release, demonstrating Intel's growing expertise in language model development.

The model was fine-tuned on a curated mixture of open-source conversational datasets using Intel's Neural Compressor and Intel Extension for PyTorch (IPEX) frameworks. The training process emphasized instruction-following, helpful responses, and conversational coherence. Intel also developed optimized inference kernels specifically for Neural Chat, enabling efficient execution on Intel Xeon processors, Intel Arc GPUs, and Intel Core Ultra processors with NPUs.

Architecturally, Neural Chat inherits Mistral 7B's features: sliding window attention, grouped-query attention, 32K token context window, and a 32K vocabulary. The model uses the standard Mistral chat template for multi-turn conversations. Intel provides quantized variants optimized for their hardware, including INT4 and INT8 configurations tuned for Intel AMX (Advanced Matrix Extensions) instructions.

Neural Chat is released under the Apache 2.0 license. While many open-source models focus on GPU inference, Neural Chat's optimization for Intel hardware makes it uniquely relevant for organizations deploying on Intel-based infrastructure, which represents the majority of enterprise server hardware worldwide.

Key Features

Intel hardware optimization is Neural Chat's primary differentiator. Intel developed custom inference kernels using IPEX (Intel Extension for PyTorch) and OpenVINO that take advantage of Intel-specific instructions including AMX (Advanced Matrix Extensions) on 4th and 5th generation Xeon processors, VNNI (Vector Neural Network Instructions), and AVX-512. These optimizations deliver significantly faster CPU inference on Intel hardware compared to generic implementations.

Neural Chat includes optimized quantization profiles for Intel hardware. INT4 quantization using Intel's Neural Compressor achieves minimal quality loss while enabling efficient execution on Xeon CPUs with AMX support. This is particularly valuable for enterprise environments where GPU availability is limited but Intel Xeon servers are abundant.

The model demonstrates strong performance on conversational benchmarks relative to its 7B parameter count. Intel's fine-tuning process incorporated careful dataset curation, including rejection sampling where multiple candidate responses were generated and the best selected for training. This approach improves response quality without requiring expensive human preference annotation.

Fine-Tuning with Ertas

Neural Chat is fully compatible with Ertas Studio's fine-tuning pipeline since it uses the standard Mistral 7B architecture. QLoRA fine-tuning requires 8-10GB VRAM, making it accessible on consumer GPUs. For organizations with Intel GPU hardware (Arc A770 16GB, for example), Ertas Studio can leverage IPEX for training acceleration.

Fine-tuning Neural Chat is recommended for organizations that will deploy on Intel hardware infrastructure. Start with Neural Chat's Intel-optimized base, fine-tune on your domain-specific data in Ertas Studio, and then deploy using Intel's optimized inference stack. This end-to-end Intel optimization path delivers the best possible performance on Xeon-based servers and Intel GPU systems.

After fine-tuning, Ertas Studio exports to GGUF format. For Intel hardware deployment, the model can also be exported in OpenVINO IR format for maximum Intel hardware utilization. Standard GGUF deployment through Ollama and llama.cpp works well and benefits from AVX-512 optimizations on Intel CPUs, with llama.cpp automatically detecting and using available Intel instruction sets.

Use Cases

Neural Chat is the natural choice for organizations with significant Intel hardware deployments looking to run AI inference on existing infrastructure. Enterprise data centers running Intel Xeon servers can deploy Neural Chat for internal chatbots, document processing, and employee assistance without purchasing dedicated GPU hardware. The optimized CPU inference path delivers practical speeds for interactive applications.

The model is well-suited for edge deployment on Intel-based devices: industrial PCs, point-of-sale systems, kiosks, and embedded systems running Intel processors. The INT4 quantized variant runs efficiently on Intel Core Ultra processors with NPU acceleration, enabling on-device AI in client-side applications.

Neural Chat also serves as a useful baseline for evaluating the performance characteristics of LLMs on CPU versus GPU inference. Organizations planning their AI infrastructure can use Neural Chat to benchmark Intel Xeon throughput against GPU-based alternatives, informing hardware purchasing decisions based on actual workload performance.

Hardware Requirements

Neural Chat 7B at Q4_K_M requires approximately 4.4GB of RAM, identical to Mistral 7B. The model runs on any system with 8GB+ RAM, but Intel hardware provides optimized performance. On Intel Xeon 4th Gen (Sapphire Rapids) with AMX, expect 15-25 tokens per second for CPU inference with INT4 quantization — significantly faster than non-optimized CPU inference.

On consumer Intel hardware, the model runs on Intel Core Ultra processors with NPU acceleration and Intel Arc GPUs (Arc A770 16GB provides 20-35 tokens per second). Standard non-Intel CPUs and NVIDIA GPUs also work well through llama.cpp and Ollama with standard GGUF quantization.

For fine-tuning in Ertas Studio, 8-10GB VRAM with QLoRA is sufficient on any supported GPU. Intel Arc A770 16GB can be used for fine-tuning via IPEX, though NVIDIA GPUs remain the most streamlined option. The 7B model size ensures fast training regardless of hardware platform.