Data ingestion overview

Importing from object storage is the most efficient and cost-effective method to load large numbers of records or documents into an index. You store your data in object storage (Parquet for vector indexes, JSON Lines (JSONL) for document indexes), integrate your object storage with Pinecone, and then start an asynchronous, long-running operation that imports and indexes your data.

For ongoing ingestion into an index, one record or document at a time, or in batches, use the upsert operation. Batch upserting can improve throughput performance and is a good option for larger numbers of records or documents if you can't work around import's current limitations.

Import and upsert move vectors into Pinecone. For workflows where you only need vectors from hosted models (for example, to embed offline and upsert later), use the Inference API as follows:

You can call the embed operation through Pinecone Inference to turn text into vectors without writing to an index. That differs from upsert_records on an index with integrated embedding, where each request embeds and stores records in one step. To see how embedding consumption appears in billing and usage reports, see Embedding tokens.

• To understand how cost is calculated for imports, see Import cost.
• To understand how cost is calculated for upserts, see Write unit pricing.
• For up-to-date pricing information, see Pricing.

Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries. You can view index stats to check data freshness.