GPT4All: Chat with Local LLMs on Any Device

A programming framework for agentic AI. Discord:https://aka.ms/autogen-dc.Roadmap:https://aka.ms/autogen-roadmap

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

Official inference library for Mistral models

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Bổn hạng mục chỉ ở chia sẻ đại mô hình tương quan kỹ thuật nguyên lý cùng với thực chiến kinh nghiệm.

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

The easiest way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Multi-model Inference Graph/Pipelines, LLM/RAG apps, and more!

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference

Superduper: Bring AI to your database! Integrate AI models and workflows with your database to implement custom AI applications, without moving your data. Including streaming inference, scalable model hosting, training and vector search.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Sparsity-aware deep learning inference runtime for CPUs

Code examples and resources for DBRX, a large language model developed by Databricks

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

AICI: Prompts as (Wasm) Programs