If every dog has its day, well, maybe the same goes for every database. Judging by the news of late, SQLite has his day in the sun. In recent weeks, several companies have announced that they are building or supporting new projects built around the venerable open source database.
Is SQLite one of the foundations for the next-generation Internet? Some think so. cloudflare announced that they were going to implement a new database service built around the backend tool. In the meantime, Fly announced that it has hired one of the developers of Litestream, an open source project that improves upon the base version of SQLite by adding the ability to replicate the data to increase performance and survivability.
At first glance, the idea seems a bit strange. The project is over 20 years old and written in simple, old C. It’s less of a standalone app worthy of its name, but rather a library that can be linked to your code. It’s not so much a front-of-the-house, selective software option as a forgotten servant doing thankless work. Many developers may start using the code when they sketch a project or build a prototype, but they often move to other full-featured options such as Oracle or PostgreSQL.
But the announcements suggest the companies are seeing something more. For example, Cloudflare is rolling out a new database service called D1 to give developers another way to store data generated by their serverless Workers apps. They already offer a key-value store and bucket product (R2), but developers often want to rely on the structure and power of SQLite to simplify their workload.
“It’s a database for the edge model,” said Rita Kozlov, senior director of product at Cloudflare. “Its embedded nature also made a lot of sense, with the whole purpose of D1 being to make it really easy for our developers to run a database right next to their computers.”
Cloudflare is rapidly expanding beyond its beginnings as a global static cache. They have built hundreds of data centers close to end users for rapid response. Lately, they’ve added products like Workers or Pages that provide a serverless model for app deployment. Developers can write a few basic functions, pay only for the time the functions run, and also deliver lightning-fast responses because the code runs on machines that are close to the user.
Adding SQLite allows developers to deliver more advanced applications. Data can be stored locally on an edge node and eventually replicated around the world. Developers who are more ambitious and need to track more user statuses can adopt the platform.
“We’ve asked a lot of our in-house developers, ‘How do we make sure you can build more with Workers?’” Kozlov says. “Their response was, ‘Give me a database. That’s the tool I’m used to.’ I can probably figure out how to do what to do with [Key-Value] but it’s just not where people are today. We always want to meet developers where they are.”
Fly jumps in for the same purpose. They have announced that they are supporting work on Litestream, an open source project that adds background processing to SQLite. It will stream updates to various object stores and FTP sites, so developers can be confident that SQLite’s data is still available and recoverable after trauma.
It’s easy to see where Fly got the idea to revise and extend an open-source database. One of their main products is PostgreSQL fully supported clusters. Developers can build a scalable, resilient version of PostgreSQL in just a few clicks.
Many other companies do the same with open source databases. Companies like PlanetScale, Yugabyte, Amazon, Oracle, and Google start with MySQL or PostgreSQL, then add additional layers of features to improve reliability, scalability, and more.
Last week, Google announced AlloyDB, their version of PostgreSQL that offers full compatibility with some additional improvements, such as column storage that can dramatically improve some workloads.
Single-threaded, but multidimensional
However, there are several differences between SQLite and the other projects. SQLite is a simple, single-threaded system. Other databases are designed with multiple threads to juggle more complex constellations of users.
For many smaller projects, this isn’t really a limitation and some developers see it as a feature.
“I had a database company before this and I think what people like me never want to talk about is that almost everyone has a few sub-10 gigabyte databases.” said Kurt Mackey, Fly’s CEO. “If you’re really in that category, you know this is super interesting because it’s SQL and it’s great for 10 gig databases.”
Developers can often get a lot of what they want from the basic core features without the complexities of supporting an entire database.
“The documentation for Postgres 14 is almost 3,000 pages† said Ben Johnson, one of the developers at Fly. “And if you don’t need the Postgres features, they’re a must. For example, even if you don’t use multiple user accounts, you must configure and debug host-based authentication. You need to disable your Postgres server’s firewall.”
The Litestream open source project supported by Fly enhances SQLite by adding the option to add more resilience to hardware failures while adding more concurrency. It solves the biggest concerns developers have about using the tool with more serious, server-side projects.
“It’s really fun during development.” said Kent Dodds, a developer who regularly uses SQLite in projects. †[There’s] no need to deploy a database server (or a docker container). It’s just a file. You can even send the database file to a colleague if you need help with something.”
The cons and what awaits us
While many Fly customers successfully use SQLite to store data for some simple apps that run on the service, Fly’s Mackey reports that there are some rough edges. The software runs very fast with no performance issues, but there are not the same number of tools that can support it.
“I think the biggest complication for us is that there’s no tooling for it.” said Mackey. “We have people implementing apps. They say, ‘How do I connect to my database and do I like to query things? How do I import data?’”
Many rely on folklore and third-party tools that are common. The code has been widely adopted over the years and many have written their tools that can support it. Even though the tools are not directly targeted at the new server-side operations, they can still be modified.
“One of the things I like about the product is that it’s very stable.” said David Crawshaw, Tailscale’s technical director. The company uses SQL to support much of its network operations. “The things it did 15 years ago it still does. That means when I come back to it, the things I learned are still useful”
Another wrinkle is that it’s not really open source. The original developer of SQLite, Dwan Richard Hipp, placed it in the public domain. In general, this means that there are no legal restrictions on using the code at all, although there are some questions about whether each country recognizes this view.
It’s a good stepping stone or starting point for developers, and sometimes that’s all it needs. This freedom has encouraged many developers in the past. It is common to find SQLite on many devices. Many smartphones and tablets use it for the default storage.
Still, some wonder how much this is a real trend and how much is just a stepping stone for the companies. This was further underlined by Kozlov, who noted that the project name “D1” at Cloudflare is called that for a number of reasons: It’s easy to increase the number, saying, “I don’t think this is — or rather, “I know this isn’t our last last stop in the database space. I think we’ll find ways to expand or we’ll expand our offering.”
The mission of VentureBeat is a digital city square for technical decision makers to gain knowledge about transformative business technology and transactions. Learn more about membership.