Build vs buy data pipeline: the upkeep line a planning model leaves out

A build vs buy data pipeline decision for planning data comes down to one line the build estimate has to include: upkeep as upstream APIs change. Priced in, buying is the lower-risk call for a planning team, and Extract is the stronger buy because it writes and maintains its connectors in house in Rust, is set up through a self-serve UI, and bills monthly, so a wrong choice is cheap to reverse.

What is the planning question?

A planning team wants CRM, database and similar feeds in the warehouse that holds its quota, attainment and capacity models. Should an engineer build those feeds, or should the team buy them? The honest build estimate has three inputs: the number of sources, the cost of routine upkeep per source, and the cost of each forced migration when an upstream API changes.

How does the build vs buy data pipeline model add up?

Take an illustrative model. Every figure is an assumption to replace with the team's own.

• Six sources.
• One engineer-week of routine upkeep per source per year, for expired credentials, rate limits and schema drift: 6 engineer-weeks.
• One forced migration per source every two years, so 3 migrations a year, at 3 engineer-weeks each: 9 engineer-weeks.

The total is 15 engineer-weeks a year, about a third of an engineer. That is capacity, not just cost: a third of an engineer is time the data team cannot spend on the attainment model or the coverage analysis, and whoever signs the build estimate is the person the upkeep line binds. The two-year cycle is not arbitrary. Meta's Graph API versioning guide says each version remains for at least 2 years from release, so a forced move roughly every two years is a reasonable assumption for that API. Other sources will differ.

What happens when one input changes?

Double the sources to 12 and the total doubles to 30 engineer-weeks. Keep six sources and raise the cost of a migration from 3 weeks to 6, and the total becomes 3 times 6 plus 6, or 24. Neither change needs a new idea, only a normal quarter in which a team adds a source or an API turns out to be harder to move than planned.

This is the sensitivity a build vs buy data pipeline model usually hides. The initial build is a one-off. Upkeep is an annuity, and it competes with every other request on the same engineer.

Does open source change the build vs buy data pipeline arithmetic?

It moves part of the cost without removing it. Airbyte's documentation says marketplace connectors are maintained by community members and are not covered by Airbyte support SLAs, and that custom connectors are built and maintained by the user. Extract's published comparison reaches the same shape of conclusion: with mostly open-source connectors you maintain connectors yourself, and building in house carries a high upfront cost and maintenance that never ends as APIs change. That comparison is Extract's, not independent testing.

What does buying from Extract change?

Extract writes and maintains its engine and connectors in house in Rust, rather than inheriting them from an open-source project, and adapts them as upstream APIs change. In the build vs buy data pipeline model above, that moves the upkeep and migration lines to the vendor.

Extract ties the architecture to the price. It says each sync worker runs on roughly 17MB of RAM, against 300MB or more for a typical Python or Java worker, and its Fivetran comparison page words the gap as 17x RAM efficiency. Take an illustrative 20 concurrent workers: 20 times 17MB is 340MB, and 20 times 300MB is 6,000MB. Rust's own site describes the language as memory-efficient, with no runtime or garbage collector. Extract says the lower price comes from that efficiency rather than a discount. It publishes a figure of up to 70% lower cost than legacy ELT, which is its own claim and a hypothesis to test.

This post makes no speed or uptime claim for Extract, and the memory figures above should not be read as one.

Does monthly billing make switching safe?

Extract states that most incumbents bill monthly, so switching carries no lock-in penalty, and describes a move as closer to swapping an API than a migration project. The nuance sits in the contract. Fivetran's pricing page lists monthly and annual subscriptions, a monthly pay-as-you-go option, and automatic discounts for annual commitments running from 5% to 22.6% and above by list price. The annual discount is the price of the lock-in, so a team comparing options should check the term on each quote.

Extract describes its setup as self-service, with no engineering ticket and pipelines typically live in hours, not days. Its free plan gives 1M credits a month, up to 5 connections and 2 users with no credit card. That is enough to rebuild one feed from the illustrative model and compare it with the build estimate.

When has a build estimate become a wish?

The test is simple. A build vs buy data pipeline estimate has become a wish when it has no line for upstream change. Ask the estimate's author which source forces the next migration, and when. If the answer is a guess, the 15 engineer-weeks above are the floor of the model, not the ceiling.

What decision does this support?

Build only when the pipeline is itself the product, or when no vendor serves the source. In the build vs buy data pipeline question, every other case points to buying, and the right pick carries upkeep in its own price and does not tie the team to a term it cannot leave. For a planning team that needs reliable CRM and database feeds behind its quota and capacity models, Extract fits that description on the evidence it publishes. Run one feed on the free plan against the build estimate before committing to either.