Today, we’re delighted to announce that we’ve doubled the concurrent connections limit for the Firebase Realtime Database from 100k to 200k. This takes effect today, for all existing databases and new projects. And while this level of scalability will be enough for the majority of projects out there, some of you might want to support even more concurrent users. For that, you might want to consider sharding your database.
Many kinds of apps can scale much higher by sharding their data across multiple Realtime Database instances in a single project. The 200k concurrent user limit applies to each individual database, so the total number of concurrent connections increases linearly as you add more instances.
Sharding is a good strategy to employ when each client interacts with isolated parts of the database. Imagine a virtual whiteboard app, which is the sort of high-frequency, low-latency collaboration app for which the Realtime Database really excels. Whiteboard sessions don’t interact with each other -- just with the handful of users that are drawing together. So they can be sharded across an unlimited number of instances. When multiple users create a session, your app could assign that session to a random shard; then the clients only need to be connected to that database in order to receive their realtime updates.
Most IoT apps can also take advantage of sharding. If you want to have a gigantic number of tiny sensors sending periodic updates, they probably don’t all need to write to the same RTDB instance. You can create lots of shards (we support up to 1000), and assign each sensor to a shard. If you're interested in working with multiple versions of the Realtime Database in the same project, make sure to check out our documentation.
Of course, if you're looking for a powerful realtime database that scales without sharding, we still recommend our newer database, Cloud Firestore, for most new projects. It has the same magical realtime, offline, serverless functionality as the Realtime Database, but has been architected for higher reliability, has more powerful queries, and locations around the world.
Nonetheless, there are some use cases for which the Realtime Database is the right choice, even for new applications. That’s why we’re continuing to invest in making it even better for these purposes.
When should you pick the Realtime Database for a new project? The short answer is that if you are building an application that will have lots of tiny operations, the Realtime Database may be less expensive and more performant. For the whiteboarding app, you’d want to send lots of frequent little updates as the users draw on the whiteboard, and you’d want latency to be as low as possible. In the IoT case, performance may not be as important, but the Realtime Database may end up cheaper than Cloud Firestore if you are sending a massive stream of tiny writes.
Of course, you can use both Cloud Firestore and the Realtime Database together in the same project — we encourage it! For instance, you can use the Realtime Database to power the live whiteboard feature, but then persist the whiteboard contents to Firestore periodically to take advantage of its 99.999% availability and less expensive storage.
We hope you are excited about the increased scaling capabilities of the Realtime Database. And we also hope this guidance helps you decide between Cloud Firestore and the Realtime Database for new projects. As always if you have any questions, feel free to reach out on StackOverflow, or the firebase-talk discussion group.
Author's Note: This post was updated on January 31st 2019 to reflect the fact that Cloud Firestore has graduated from Beta to General Availability
Hey, did you hear the big news? We just announced the release of Cloud Firestore -- the new database that lets you easily store and sync app data to the cloud, even in realtime!
Now if you're experiencing some deja vu, you're not alone. We realize this sounds awfully similar to another product you might already be using -- the Firebase Realtime Database. So if you're experiencing some deja vu, you're not alone.
So why did we build another database? And when would you choose one over another? Well, let's talk about what's new and different with Cloud Firestore, and why you might want to use it for your next app.
What's different with Cloud Firestore?
While our documentation covers all of the differences between the Realtime Database and Cloud Firestore in much more detail, let's look at the main differences between the two products. And we'll start with...
Better querying and more structured data
While the Firebase Realtime Database is basically a giant JSON tree where anything goes and lawlessness rules the land1, Cloud Firestore is more structured. Cloud Firestore is a document-model database, which means that all of your data is stored in objects called documents that consist of key-value pairs -- and these values can contain any number of things, from strings to floats to binary data to JSON-y looking objects the team likes to call maps. These documents, in turn, are grouped into collections.
Your Cloud Firestore database will probably consist of a few collections that contain documents that point to subcollections. These subcollections will contain documents that point to other subcollections, and so on.
This new structure gives you several important advantages in being able to query your data.
For starters, all queries are shallow, meaning that you can simply fetch a document without having to fetch all of the data contained in any of the linked subcollections. This means you can store your data hierarchically in a way that makes sense logically without worrying about downloading tons of unnecessary data.
In this example, the document at the top can be fetched without grabbing any of the documents in the subcollections below
Second, Cloud Firestore has more powerful querying capabilities than the Realtime Database. In the Realtime Database, trying to create a query across multiple fields was a lot of work and usually involved denormalizing your data.
For example, imagine you had a list of cities, and you wanted to find a list of all cities in California with a population greater than 500k.
Cities, stored in the Realtime Database
In the Realtime Database, you'd need to conduct this search by creating an explicit "states plus population" field and then running a query sorted on that field.
Creating a combined state_and_population field, just for queries
With Cloud Firestore, this work is no longer necessary. In some cases, Cloud Firestore can automatically search across multiple fields. In other cases, like our cities example, Cloud Firestore will guide you towards automatically building an index required to make these kinds of queries possible…
...and then you can simply search across multiple fields.
Cloud Firestore will automatically maintain this index for you throughout the lifetime of your app. No combo fields required!
Designed to Scale
While the Realtime Database does scale to meet the needs of many apps, things can start to get difficult when your app becomes really popular, or your dataset gets truly massive.
Cloud Firestore, on the other hand, is built on top of the same Google Cloud infrastructure that powers some pretty popular apps. So it will be able to scale much more easily and to a much greater capacity than the Realtime Database can. While the Realtime Database tops out at about 100,000 concurrent connections, for instance, Cloud Firestore will accept up to 1,000,000 concurrent client connections per database. For a complete list of Cloud Firestore's limits, be sure to visit the documentation.
Cloud Firestore also has a more robust Service Level Agreement than the Realtime Database. Cloud Firestore guarantees 99.999% uptime in multi-region instances (more on that below) and 99.99% uptime in regional instances. The Realtime Database, by contrast, guarantees 99.95% uptime.
And with the new querying structure, all Cloud Firestore queries scale to the size of your result set -- not the size of your data. This means that a search for the top 10 restaurants in Chicago for a restaurant review app will take the same amount of time whether your database has 300 restaurants, 300 thousand or 30 million. As one engineer here likes to put it, "It's basically impossible to create a slow query in Cloud Firstore."
Easier manual fetching of data
While some developers appreciated the real-time nature of the Realtime Database for building features like chat or enabling magical collaborative experiences, we found that many of our developers simply wanted to use the Realtime Database as a traditional, "Just fetch data when I ask for it" kind of service.
Although the Realtime Database does support this with .once calls, they can sometimes feel a bit unnatural to use and often play second-fiddle to the streaming methods within the SDKs. With Cloud Firestore, making simple one-time fetch queries is much more natural and is built as a primary use case within the Firestore API.
Of course, you can still add support for listeners, so that your clients have the ability to receive updates whenever you data changes in the database. But now you have the flexibility to retrieve your data however you'd like.
Multi-Region support for better reliability and more locations
Cloud Firestore has support for multi-region locations. This means that your data is automatically copied to multiple geographically separate regions at once. So if some unforeseen disaster were to render a data center -- or even an entire region -- offline, you can rest assured that your data will continue to be served
And for you database aficionados out there, we should point out that Cloud Firestore offers strong consistency (just like Cloud Spanner!), which means that you get the benefits of multi-region support, while also knowing that you'll be getting the latest version of your data whenever you perform a read.
There are currently two different multi-region locations you could use to host your data (one in North America, the other in Europe). And for those of you who don't need the heavy-duty capabilities of a multi-region database, Cloud Firestore also offers several regional instances in locations around the world, most of which will be available at a lower price tier starting in early March, 2019. By contrast, the Realtime Database is only hosted in North America.
Different pricing model
The two databases have fairly different pricing models: The Realtime Database primarily determines cost based on the amount of data that's downloaded, as well as the amount of data you have stored on the database.
While Cloud Firestore does charge for these things as well, they are significantly lower than what you would see in the Realtime Database2. Instead, Cloud Firestore's pricing is primarily driven by the number of reads or writes that you perform.
What this means is that if you have a more traditional mobile app where your client is occasionally requesting larger chunks of data from your database -- think something like a news app, a dating app, or a turn-based multiplayer game, for instance -- you will find that Cloud Firestore's pricing model might be more favorable than if you ran the same app on the Realtime Database.
On the other hand, if you have an app that's making very large numbers of reads and writes per second per client (for instance, a group whiteboarding app, where you might be broadcasting everybody's drawing updates to everybody else several times a second), Cloud Firestore will probably be more expensive.3 At least for that portion of your app -- you can always use both databases together, and that's fine, too.
Of course these are just rough guidelines, make sure you check out the Pricing section of our documentation for all the details on Cloud Firestore pricing.
Why you still might want to use the Realtime Database
With this list of changes, you might come away with the impression that Cloud Firestore is simply better than the Realtime Database. And while Cloud Firestore does have a fair number of improvements over the Realtime Database, there are still a few situations where you might want to consider using the Realtime Database for some of your data. Specifically…
The Realtime Database has native support for presence -- that is, being able to tell when a user has come online or gone offline. While we do have a solution for Cloud Firestore, it's not quite as elegant.
The Realtime Database has slightly better latency. Usually not by much, but it's something to keep in mind if you have a special need for very low-latency operations.
As we noted above, Cloud Firestore's pricing model means that applications that perform very large numbers of small reads and writes per second per client could be significantly more expensive than a similarly performing app in the Realtime Database.
The tl;dr: Just tell me what to use!
In general, we recommend that new applications start with Cloud Firestore, unless you think that your app has unique needs, like those we outlined above, that make it more suitable for the Realtime Database.
On the other hand, if you have an existing database that's already running just fine on the Realtime Database, go ahead and keep it there! If you find you're running up against issues where Cloud Firestore could really help you out, then you could consider switching, or moving part of your data to Cloud Firestore and using both databases together. But don't switch just for the sake of switching.
And if you're looking for a magic, "Please convert my database from the Realtime Database to Cloud Firestore" button, there isn't one4! And, frankly, we don't know if there ever will be. Given how different the database, querying, and pricing structures are between the two, blindly converting a database that's been optimized for the Realtime Database over to Cloud Firestore wouldn't necessarily be a great experience. We want you to be more thoughtful about making this kind of change.
There's a lot we think you'll be able to do with Cloud Firestore and we're excited to see what kinds of apps you're able to build with it. As always, if you have questions, you can hit us up on any of our support channels, or post questions on Stack Overflow with the google-cloud-firestore and firebase tags. Good luck, and have fun!