Back in 2019 we launched Firebase Extensions - pre-packaged solutions that save you time by providing extended functionality to your Firebase apps, without the need to research, write, or debug code on your own. Since then, a ton of extensions have been added to the platform covering a wide range of features, from email triggers and text messaging, to image resizing, translation, and much more.
With the Google Pay Firebase Extension, your app can accept payments from Google Pay users, using one or more of the many supported Payment Service Providers (or PSPs), without the need to invoke their individual APIs.
With the extension installed, your app can pass a payment token from the Google Pay API to your Cloud Firestore database. The extension will listen for a request written to the path defined during installation, and then send the request to the PSP's API. It will then write the response back to the same Firestore node, which you can listen and respond to in real time.
Open Source
Like all Firebase Extensions, the Google Pay Firebase Extension is entirely open source, so you can modify the code yourself to change the functionality as you see fit, or even contribute your changes back via pull requests - the sky's the limit.
Summing it up
Whether you're new to Google Pay or Firebase, or an existing user of either, the new Google Pay extension is designed to save you even more time and effort when integrating Google Pay and any number of Payment Service Providers with your application.
The Cloud Firestore C++ SDK uses the firebase::firestore::FieldValue class to represent document fields. A FieldValue is a union type that may contain a primitive (like a boolean or a double), a container (e.g. an array), some simple structures (such as a Timestamp) or some Firestore-specific sentinel values (e.g. ServerTimestamp) and is used to write data to and read data from a Firestore database.
Other Firebase C++ SDKs use firebase::Variant for similar purposes. A Variant is also a union type that may contain primitives or containers of nested Variants; it is used, for example, to write data to and read data from the Realtime Database, to represent values read from Remote Config, and to represent the results of calling Cloud Functions using the Firebase SDK. If your application is migrating from Realtime Database to Firestore (or uses both side-by-side), or, for example, uses Firestore to store the results of Cloud Functions, you might need to convert between Variants and FieldValues.
In many ways, FieldValue and Variant are similar. However, it is important to understand that for all their similarities, neither is a subset of the other; rather, they can be seen as overlapping sets, each reflecting its domain. These differences make it impossible to write a general-purpose converter between them that would cover each and every case -- instead, handling each instance where one type doesn’t readily map to the other would by necessity have to be application-specific.
With that in mind, let’s go through some sample code that provides one approach to conversion; if your application needs to convert between FieldValue and Variant, it should be possible to adapt this code to your needs. Full sample code is available here.
Converting primitives
The one area where FieldValues and Variants correspond to each other exactly are the primitive values. Both FieldValue and Variant support the exact same set of primitives, and conversion between them is straightforward:
FieldValue ConvertVariantToFieldValue(const Variant& from) {
switch (from.type()) {
case Variant::Type::kTypeNull:
return FieldValue::Null();
case Variant::Type::kTypeBool:
return FieldValue::Boolean(from.bool_value());
case Variant::Type::kTypeInt64:
return FieldValue::Integer(from.int64_value());
case Variant::Type::kTypeDouble:
return FieldValue::Double(from.double_value());
}
}
Variant Convert(const FieldValue& from) {
switch (from.type()) {
case FieldValue::Type::kNull:
return Variant::Null();
case FieldValue::Type::kBoolean:
return Variant(from.boolean_value());
case FieldValue::Type::kInteger:
return Variant(from.integer_value());
case FieldValue::Type::kDouble:
return Variant(from.double_value());
}
}
Strings and blobs
Variant distinguishes between mutable and static strings and blobs: a mutable string (or blob) is owned by the Variant and can be modified through its interface, whereas a static string (or blob) is not owned by the Variant (so the application needs to ensure it stays valid as long as the Variant’s lifetime hasn’t ended; typically, this is only used for static strings) and cannot be modified.
Firestore does not have this distinction -- the strings and blobs held by FieldValue are always immutable (like static strings or blobs in Variant) but owned by the FieldValue (like mutable strings or blobs in Variant). Because ownership is the more important concern here, Firestore strings and blobs should be converted to mutable strings and blobs in Variant:
// `FieldValue` -> `Variant`
case FieldValue::Type::kString:
return Variant(from.string_value());
case FieldValue::Type::kBlob:
return Variant::FromMutableBlob(from.blob_value(), from.blob_size());
// `Variant` -> `FieldValue`
case Variant::Type::kTypeStaticString:
case Variant::Type::kTypeMutableString:
return FieldValue::String(from.string_value());
case Variant::Type::kTypeStaticBlob:
case Variant::Type::kTypeMutableBlob:
return FieldValue::Blob(from.blob_data(), from.blob_size());
Arrays and maps
Both FieldValues and Variants support arrays (called “vectors” by Variant) and maps, so for the most part, converting between them is straightforward:
// `FieldValue` -> `Variant`
case FieldValue::Type::kArray:
return ConvertArray(from.array_value());
case FieldValue::Type::kMap:
return ConvertMap(from.map_value());
}
// `Variant` -> `FieldValue`
case Variant::Type::kTypeVector:
return ConvertArray(from.vector());
case Variant::Type::kTypeMap:
return ConvertMap(from.map());
}
// ...
FieldValue ConvertArray(const std::vector<Variant>& from) {
std::vector<FieldValue> result;
result.reserve(from.size());
for (const auto& v : from) {
result.push_back(Convert(v));
}
return FieldValue::Array(std::move(result));
}
FieldValue ConvertMap(const std::map<Variant, Variant>& from) {
MapFieldValue result;
for (const auto& kv : from) {
// Note: Firestore only supports string keys. If it's possible
// for the map to contain non-string keys, you would have to
// convert them to a string representation or skip them.
assert(kv.first.is_string());
result[kv.first.string_value()] = Convert(kv.second);
}
return FieldValue::Map(std::move(result));
}
Variant ConvertArray(const std::vector<FieldValue>& from) {
std::vector<Variant> result;
result.reserve(from.size());
for (const auto& v : from) {
result.push_back(Convert(v));
}
return Variant(result);
}
Variant ConvertMap(const MapFieldValue& from) {
std::map<Variant, Variant> result;
for (const auto& kv : from) {
result[Variant(kv.first)] = Convert(kv.second);
}
return Variant(result);
}
Nested arrays
Firestore does not support nested arrays (that is, one array being a direct member of another array). FieldValue itself would not reject a nested array, though -- it will only be rejected by Firestore’s input validation when passed to a Firestore instance (like in a call to DocumentReference::Set).
The approach to handling this case would have to be application-specific. For example, you might simply omit nested arrays, perhaps logging a warning upon encountering them; on the other extreme, you may want to terminate the application:
FieldValue ConvertArray(const std::vector<Variant>& from) {
std::vector<FieldValue> result;
result.reserve(from.size());
for (const auto& v : from) {
if (v.type() == Variant::Type::kTypeVector) {
// Potential approach 1: log and forget
LogWarning("Skipping nested array");
continue;
// Potential approach 2: terminate
assert(false && "Encountered a nested array");
std::terminate();
}
result.push_back(Convert(v));
}
return FieldValue::Array(std::move(result));
}
Yet another approach might be to leave the nested arrays in place and rely on Firestore input validation to reject them (this approach is mostly applicable if you don’t expect your data to contain any nested arrays).
Translating nested arrays
One possible workaround if you need to pass a nested array to Firestore might be to represent arrays as maps:
case Variant::Type::kTypeVector: {
MapFieldValue result;
const std::vector<Variant>& array = from.vector();
for (int i = 0; i != array.size(); ++i) {
result[std::to_string(i)] = Convert(array[i]);
}
return FieldValue::Map(std::move(result));
}
Another approach, which has the nice property of being generalizable to other cases, is to automatically translate the structure of “array-array” into “array-map-array” when converting to FieldValue.
If you decide to use this approach, you will need to ensure that the translated structure roundtrips properly (assuming your application needs bidirectional conversion). That is, an “array-map-array” structure within a FieldValue converts back to an “array-array” structure in Variant. To achieve this, the artificial map would have to be somehow marked to indicate that it does not represent an actual value in the database.
Once again, the implementation for this would be application-specific. You could add a boolean field called “special” with its value set to true and establish a convention that a map that contains a “special” field never represents user input. If this is not true for your application, you might use a more distinct name than “special” or come up with a different convention altogether.
These next two examples use “special” as a marker, but please keep in mind that it’s just one possible approach:
// `Variant` -> `FieldValue`
FieldValue Convert(const Variant& from, bool within_array = false) {
switch (from.type()) {
// ...
case Variant::Type::kTypeVector:
if (!within_array) {
return ConvertArray(from.vector());
} else {
// Firestore doesn't support nested arrays. As a workaround, create an
// intermediate map to contain the nested array.
return FieldValue::Map({
{"special", FieldValue::Boolean(true)},
{"type", FieldValue::String("nested_array")},
{"value", ConvertArray(from.vector())},
});
}
}
}
FieldValue ConvertArray(const std::vector<Variant>& from) {
std::vector<FieldValue> result;
result.reserve(from.size());
for (const auto& v : from) {
result.push_back(Convert(v, /*within_array=*/true));
}
return FieldValue::Array(std::move(result));
}
// `FieldValue` -> `Variant`
Variant Convert(const FieldValue& from) {
switch (from.type()) {
// ...
case FieldValue::Type::kArray:
return ConvertArray(from.array_value());
case FieldValue::Type::kMap: {
const auto& m = from.map_value();
// Firestore doesn't support nested arrays, so nested arrays are instead
// encoded as an "array-map-array" structure. Make sure nested arrays
// round-trip.
// Note: `TryGet*` functions are helpers to simplify getting values
// out of maps. See their definitions in the full sample code.
bool is_special = TryGetBoolean(m, "special");
if (is_special) {
return ConvertSpecialValue(m);
} else {
return ConvertMap(from.map_value());
}
}
}
Variant ConvertSpecialValue(const MapFieldValue& from) {
// Note: in production code, you would have to handle
// the case where the value is not in the map.
// Note: `TryGet*` functions are helpers to simplify getting values
// out of maps. See their definitions in the full sample code.
std::string type = TryGetString(from, "type");
if (type == "nested_array") {
// Unnest the array.
return ConvertArray(TryGetArray(from, "value"));
}
// ...
}
Firestore structs
Finally, there are several kinds of entities supported by FieldValue that have no direct equivalent in Variant:
Timestamp
GeoPoint
DocumentReference
Sentinel values (see below).
Similarly to nested arrays, your application could omit these values, issue errors upon encountering them, or else convert them into some representation supported by Variant. The exact representation would depend on the needs of your application and on whether the conversion is bidirectional or not (that is, whether it should be possible to convert the representation back into the original Firestore type).
An approach that is general (if somewhat heavyweight) and allows bidirectional conversion is to convert such structs into “special” maps. It could look like this:
// `FieldValue` -> `Variant`
case FieldValue::Type::kTimestamp: {
Timestamp ts = from.timestamp_value();
MapFieldValue as_map = {
{"special", FieldValue::Boolean(true)},
{"type", FieldValue::String("timestamp")},
{"seconds", FieldValue::Integer(ts.seconds())},
{"nanoseconds", FieldValue::Integer(ts.nanoseconds())}};
return ConvertMap(as_map);
}
case FieldValue::Type::kGeoPoint: {
GeoPoint gp = from.geo_point_value();
MapFieldValue as_map = {
{"special", FieldValue::Boolean(true)},
{"type", FieldValue::String("geo_point")},
{"latitude", FieldValue::Double(gp.latitude())},
{"longitude", FieldValue::Double(gp.longitude())}};
return ConvertMap(as_map);
}
case FieldValue::Type::kReference: {
DocumentReference ref = from.reference_value();
std::string path = ref.path();
MapFieldValue as_map = {
{"special", FieldValue::Boolean(true)},
{"type", FieldValue::String("document_reference")},
{"document_path", FieldValue::String(path)}};
return ConvertMap(as_map);
}
FieldValue ConvertSpecialValue(const std::map<Variant, Variant>& from) {
// Special values are Firestore entities encoded as maps because they are not
// directly supported by `Variant`. The assumption is that the map contains
// a boolean field "special" set to true and a string field "type" indicating
// which kind of an entity it contains.
std::string type = TryGetString(from, "type");
if (type == "timestamp") {
Timestamp result(TryGetInteger(from, "seconds"),
TryGetInteger(from, "nanoseconds"));
return FieldValue::Timestamp(result);
} else if (type == "geo_point") {
GeoPoint result(TryGetDouble(from, "latitude"),
TryGetDouble(from, "longitude"));
return FieldValue::GeoPoint(result);
}
// ...
The only complication here is that to convert a “special” map back to a DocumentReference, you would need a pointer to a Firestore instance so that you may call Firestore::Document. If your application always uses the default Firestore instance, you might simply call Firestore::GetInstance. Otherwise, you can pass Firestore* as an argument to Convert or make Convert a member function of a class, say, Converter, that acquires a pointer to a Firestore instance in its constructor.
} else if (type == "document_reference") {
DocumentReference result =
firestore->Document(TryGetString(from, "document_path"));
return FieldValue::Reference(result);
}
One more thing to note is that Realtime Database represents timestamps as the number of milliseconds since the epoch in UTC. If you intend to use the resulting Variant in the Realtime Database, a more natural representation for a Timestamp might thus be an integer field. However, you would have to provide some way to distinguish between numbers and timestamps in the Realtime Database -- a possible solution is to simply add a _timestamp suffix to the field name, but of course other alternatives are possible. In that case, the conversion from FieldValue to Variant might look like:
If bidirectional conversion is required, you would also have to somehow distinguish between numbers and timestamps when converting back to a FieldValue. If you're using the solution of adding _timestamp suffix to the field name, you would have to pass the field name to the converter. Another approach might be to use heuristics and presume that a very large number that readily converts to a reasonably recent date must be a timestamp.
Firestore sentinel values
Finally, there are some unique values in Firestore that represent a transformation to be applied to an existing value or a placeholder for a value to be supplied by the backend:
Delete
ServerTimestamp
ArrayUnion
ArrayRemove
IncrementInteger
IncrementDouble
Some of these values are only meaningful in Firestore, so most likely it wouldn’t make sense to try to convert them in your application. Otherwise, Delete and ServerTimestamp, being stateless, can be straightforwardly converted to maps using the approach outlined above. If you’re using Variant with the Realtime Database, you might want to represent a ServerTimestamp in the Realtime Database-specific format ( a map that contains a single element: {".sv" : "timestamp"}):
case FieldValue::Type::kServerTimestamp:
return ConvertMap({{".sv", FieldValue::String("timestamp")}});
Similarly, you may represent Delete as a null in the Realtime Database:
case FieldValue::Type::kDelete:
return Variant::Null();
However, other than Delete and ServerTimestamp, the rest of the sentinel values are stateful and there is no way to get their underlying value from a FieldValue, so lossless conversion is not possible. Likely the best thing to do is just to ensure these values are never passed to the converter and assert if they do.
The Firestore Emulator Requests Monitor allows you to see requests to your local Firestore Emulator in real-time, and drill down to the details of each request, such as method, path, and Firebase Security Rules evaluation. You can access the Requests Monitor right now from the Emulator UI if you have the latest Firebase CLI running. (If not, it's never too late to upgrade!)
Requests Monitor helps you understand your request traffic in detail, and puts Firebase Security Rules front and center (just like security should be in your production app). Ever wonder what collections are pulled in for the amazing pizza tracker feature in your app? Forgot about how that cute button is backed by 400 lines of code changes Firestore? Worried about changes to security rules breaking production apps? The Requests Monitor has answers to all of these questions, plus more!
A New Requests View
First, start the Emulator Suite, then navigate to the Firestore tab in the Emulator UI and you'll be greeted with two views: the default "Data" view is the familiar data viewer and editor you know and love, and the "Requests" view is just a click away.
Each client request to the Firestore Emulator will be added to the table as a new row. For example, if you connect your app to the Firestore Emulator and create a new document, it will show as a CREATE request on the table in real time. This works regardless of which platform your app is on -- be it Android, iOS, web, or anything else. And if you ever forget to open this page before you make those requests, don't worry -- we've got you covered. The last few requests will be kept for you to review when you navigate to the Requests page.
The Requests View is a great help in developing security rules. A checkmark indicates if the request has been allowed by your security rules; denials and errors are also displayed. If you follow best practices and keep your rules locked down as much as possible, you'll certainly see some denials when you develop new features, and those are the perfect opportunity to learn! To take the guesswork out of updating security rules, just click on any request to see the evaluation details view.
On this page, you'll see your current local security rules on the left, and some information about the specific request on the right. Statements that allow, deny, or error will be highlighted. You may also run into situations where the request is not yet covered by any of the allow statements and is thus denied by default. Ever wonder why a request is denied where it should be allowed, or the reverse? The panel on the right can help. You can see the existing resource, the would-be version of the document (i.e. request.resource) and other important fields in the request. If your goal is to modify the security rules to align with changes to your app, you can also use those fields as inspiration for new conditions that your rules should gate on.
While we're at it, you'll notice the security rules on the evaluation details page are not modifiable -- that's because they are snapshots at the time when the request happened. To make rules changes, just directly modify the firebase.rules file with your favorite editor or IDE and the Firebase CLI will automatically apply the changes to the Firestore Emulator. And if you make another request, it will show up in the table view as a different row with new rules and results. Sometimes, it may be helpful to compare the old and new rules and see differences in how they were evaluated.
For those of you who are familiar with the Security Rules Playground in Firebase Console, you may miss the simulated requests feature here. But in the emulator world, there's no need to guess what a request should look like -- you can just simply make that request from your app using the Firestore SDK in your favorite programming language. The Request Monitor always shows you faithfully how that request is represented in Security Rules and the actual decision of your rules. Any client request is fair game -- even lists and queries that are hard to simulate in production. We think you will eventually get used to it and love this new interactive development workflow as much as we do.
Limitations
While you enjoy the new Requests Monitor, just keep in mind that only client requests are shown in the table. Admin requests bypass any security rules and therefore don't appear in the list. Similarly, if your rules depend on other documents in the Firestore (e.g. get(...) or exists(...)), only the main request is shown but not the dependent fetches, even though those dependent fetches count towards your quotas and billing in production. Just remember emulated benchmarks are not an indicator of production in terms of performance or cost estimation.
We've already heard some developers asking if this feature will also be available in Firebase Console. While we cannot say for sure, recording production requests will certainly create a huge challenge to your app's Firestore performance and security. Aaaand, well, you know, production is not the best place to test out changes, especially security-related changes. We recommend developing and testing locally before rolling out to production, and the Firebase Emulator Suite is always seeking ways to help.
Enjoy your new view into Firestore Emulator
With the Firestore Emulator and Requests Monitor, you can see your prototyping path more clearly. In fact, you have a better view into unit and integration testing as well: just make sure to keep the Monitor open and run your tests against the same (emulated) Project ID that your app connects to. You only need to deploy when you feel comfortable with your changes.
Feel free to play around with the Firestore Emulator Requests Monitor, and let us know what you think!
Hi, Firestore developers. We are pleased to announce that with the latest version of the client SDKs, you are able to use Firestore data bundles in your mobile and web applications! Data bundles can make certain types of Firestore applications much faster or less expensive.
That's great! Just one tiny little follow-up question: What are data bundles?
So essentially, data bundles are serialized groups of documents -- either individual documents, or a number of documents that you've retrieved using a specific query.
You can save these data bundles onto a CDN or your favorite object storage service, and then load them from your client applications. Once they've been loaded on to your client, your clients can read them in from your local cache like any other locally-cached Firestore data.
Well that just sounds like a database query with extra steps. Why would I want to do this?
The biggest reason is that by having your clients read common queries from a bundle on a CDN and then querying specifically against that cached data, you can avoid making extra calls against the Firestore database. If your application has a substantial number of users, this can lead to some cost savings, and could potentially be faster, too.
But to be clear, data bundles are an advanced feature and probably not something you should be considering until your application has a pretty sizable user base.
So what kinds of documents make sense to put into bundles?
The best kinds of bundles are ones where a majority of your users will be reading in all of the documents in that bundle, the number of documents is on the smaller side, and the contents of that data doesn't change too frequently.
Some good examples for this might be:
You're storing configuration data across several documents that every client needs to read in upon startup. Since you might have millions of clients reading in the same handful of documents, this would be a great opportunity to put that data into a bundle and save a few million document reads every day.
You have a news app, blog, or similar application, and you know that every client will be reading in the same top 10 stories every day.
You want to load in some "starter" data for users who aren't signed in, and you suspect the majority of your users will be ones who aren't signed in -- this is particularly useful in web applications, where you're more likely to encounter users who aren't signed in.
Data bundles are not good for:
Database queries where you expect each of your users will be making different queries or want different pieces of information.
Any data that contains private information -- because data bundles are created by the server SDK, they can bypass security rules. So you need to be careful that any data bundles you load onto a client application is something that's intended to be read by the general public.
I just had an idea: What if I were to stuff my entire database into a bundle, load that bundle onto my client, and then just query my entire database using only my cache?
That's a terrible idea.
Remember, the Firestore cache is not particularly fast when it comes to searching through large amounts of data. Overloading your cache by asking it to store a lot of documents that your users won't ever use is a good way of slowing your application to an unusable state. You should really only be leveraging data bundles to load up documents that most, if not all, of your users will be reading in.
Also, keep in mind that with bundles, you have to load in the entire bundle of documents when you load up a bundle, whereas with a normal document query, you're only downloading the documents that have changed from your local cache. So from a data usage perspective, stuffing too many documents into a bundle can be quite heavy.
Gotcha. So how would I implement these?
You can read the documentation for all the details, but in general, the process will work something like this:
Using the server SDKs, you would generate a bundle that consists of either individual documents, or documents generated using a query.
You can also associate these queries with names.
You can then store these bundles onto your favorite CDN, web hosting service, or binary storage service like Cloud Storage.
When your client starts up, you will tell it to load in your bundle using your favorite networking library.
Once your bundle has been properly loaded, the Firestore SDK can merge it with any cached data you might already have. This process is asynchronous, so you should wait until everything has been merged in before trying to use your bundled data.
So, once they're loaded, I can query them like normal data?
Yes. They'll be merged in with your locally cached data and you can make use of them like you would any other cached data. The data bundle also remembers the names of the queries you used to generate the bundle, so you can refer to these queries by name instead of re-creating them in code.
And how do I make sure I'm querying bundled data without incurring any additional Firestore costs?
The best way is to force your client to use the cached data. When you make a document request or database query, you're able to add an option like {source: 'cache'} that directs your client to only use the cached data, even if the network is available.
Tell me more about these "named queries" -- are these different from regular queries?
Not really -- the trick to using bundles effectively is that you want to make sure the query you're requesting on the client is exactly the same as the query that generated the bundle you've loaded. You could certainly do this in code, but by using a named query, you're ensuring that the client will always make the same query that generated the bundle in the first place. There's less room for error that way.
This also means that you can modify the server query that generates the bundle and, assuming you're still using the same name, the client will also use this new query without your needing to update any of the client code.
Notice that the version number has changed, but the client can continue to run the "seed-data" named query
What if I wanted to include data bundles alongside the rest of my local application data when I publish it to app stores? That way, if a user opens my app for the first time and is offline, I can read in the bundle and use that as starter data for my application.
Yes; that can be another great use of bundles. The process is the same, except you'll be reading in the bundle using a local call. Just be aware of the warnings above -- Firestore wasn't designed to be an "offline-first" database, so try to only load in as much data as you'll need to make sure your application is functional. If you overload your cache with too much information, you'll slow it down too much.
Okay, I think I'm ready to start using data bundles in my application.
Great! As always, feel free to ask questions on Stack Overflow if you need help, or reach out to us on the Cloud Firestore Discussion list if you have any other suggestions. Happy coding!
Firebase Security Rules gate your user's access to and enforce validations for Firestore, Firebase Storage, and the Realtime Database. It's important to code review the Security Rules, just like you code review the application code. Because these rules are written in a domain-specific language, that puts some people in the position of code reviewing something they don't feel like they understand.
If you're finding yourself in that position, don't worry! This post will walk through how to approach reviewing and giving good feedback on Security Rules. The examples will be from Firestore Security Rules, and are mostly applicable to Storage as well. If you're reviewing Realtime Database Security Rules, although these principles apply, the rules use a different language, so the examples will be different.
Watch out for global rules
When you're looking at Security Rules, check first for any top level rules that apply to everything. A document can match multiple rules, and if any rule grants access, access is granted. In the example below, there's a specific rule that only grants authors access to post documents, but the global rule lets anyone on the internet read or write to any place in your database:
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
match /{document=**} {
allow read, write: if true;
}
match /posts/{post} {
allow read, write: if request.resource.data.authorUID == request.auth.uid ;
}
}
}
It's important to look for the global match statement match /{document=**} throughout the entire rules file, not just at the top. If there are thorough rules but also a global match statement, universal access will still be granted through the global match statement.
There are a very small number of valid use cases for global match statements. For example, granting access to admin users:
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
match /{document=**} {
allow read, write: if isAdmin();
}
}
}
If you find a global rule that is read, write: if true, you can stop your review and ask them to fix it. If you find any other condition on a global rule, make sure it makes sense for your application.
Personally Identifiable Information should be separated
The next thing to do is to look at the data that you're storing. What is Personally Identifiable Information (PII) that should only be accessed by that particular user?
Security Rules apply to entire documents, not just fields. You should be able to look at each kind of document that you have and describe which kind of users should be able to read it, create it, update it, and delete it.
If you find documents where it's fine for most of the document to be read by anyone, but a few fields need to remain private, break those fields into their own document; the easiest way to do this is usually to create a subcollection off the existing document.
Now check the Security Rules for each of the documents that you identified as containing PII. The best practice for PII is that it is keyed by the user's ID, and only that user is allowed access:
match /secrets/{uid} {
allow create, update: if request.auth.uid == uid;
}
This pattern only works if there's a max of one document in the collection for each user. For cases of multiple documents per user, you could create subcollections of a user document:
match /users/{uid}/secrets/{secret} {
allow create, update: if request.auth.uid == uid;
}
Or you could include the id of the user who should have access as an attribute, and restrict access to that user:
match /secrets/{secret} {
allow create, update: if request.auth.uid == request.resource.data.uid;
}
Ideally, you would check that each document is as locked down as possible, but if you have limited time, the most important thing to check is that PII lives in separate documents and those PII documents are only accessible to that user.
Check the unit tests for the Security Rules
Just like application changes should come with test changes, so should Security Rules. Security Rules tests run against the Firebase Emulator Suite, and can be included in your CI setup.
If you're not yet testing Security Rules, check out this documentation to get started, this video for an overview of testing, and this blog post and video on adding your tests to CI.
If the Security Rules are fully tested, there should be a lot of tests. For each kind of document, there are usually four different kinds of access: read, create, update, and delete. (These are just the most common permissions to grant; create, update, and delete can be rolled up into write, and read can be broken into get and list.) For each permission, there should be a test of the happy path, granting permission, and a test for each situation that should deny access.
Say I have one rule:
// firestore.rules
allow update: if
// User is the author
resource.data.authorUID == request.auth.uid &&
// `authorUID` and `createdAt` are unchanged
request.resource.data.diff(request.resource.data).unchangedKeys().hasAll([
"authorUID",
"createdAt"
]) &&
// Title must be < 50 characters long
request.resource.data.title.size < 50;
To completely test this, I would write tests around granting access and each way that access could be denied:
// test.js
const firebase = require("@firebase/rules-unit-testing");
const dbAuthorAuth = firebase.initializeTestApp({
projectId: TEST_FIREBASE_PROJECT_ID,
auth: {
uid: "author",
email: "alice@example.com"
}
}).firestore();
const dbOtherAuth = firebase.initializeTestApp({
projectId: TEST_FIREBASE_PROJECT_ID,
auth: {
uid: "other",
email: "otto@example.com"
}
}).firestore();
describe("blog posts", () => {
before(async () => {
dbAuthorAuth.doc("drafts/12345").set({
authorUID: "author",
createdAt: Date.now(),
title: "Make an apple",
content: "TODO!"
});
});
it("can be updated by author if immutable fields are unchanged", async () => {
await firebase.assertSucceeds(dbAuthorAuth.doc("drafts/12345").update({
title: "Make an app",
content: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
}));
});
it("cannot be updated by anyone other than the author", async () => {
await firebase.assertFails(dbOtherAuth.doc("drafts/12345").update({
title: "Make an app",
content: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
}));
});
it("cannot be updated by author if the author ID is changed", async () => {
await firebase.assertFails(authorDb.doc("drafts/12345").update({
authorUID: "New Person"
title: "Make an app",
content: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
}));
});
it("cannot be updated by author if the created date is changed", async () => {
await firebase.assertFails(authorDb.doc("drafts/12345").update({
createdAt: Date.now(),
title: "Make an app",
content: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
}));
});
it("cannot be updated if the title is over 50 characters", async () => {
await firebase.assertFails(authorDb.doc("drafts/12345").update({
title: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.",
content: "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."
}));
});
As the reviewer, look through their test cases, and make sure they have tested each kind of document. If they've written good test descriptions, that's the easiest place to understand what the Security Rules are doing. Similarly, if they've written code comments in their rules, that can also be a good entry point to understanding the Security Rules.
Firestore only: Pay attention to subcollections
Subcollections don't inherit the rules from the parent document, so make sure they're specifically covered in the Security Rules. Look at the subcollections you're storing in Firestore; if all documents should have the same access that a parent document has, then that parent document should use the glob syntax: match /post/{id=**}.
What's more common is that a subcollection has been broken out into a subcollection because a different person needs to read it, or someone else is allowed to write to it. In that case, check the Security Rules to make sure it has its own match statement. Nesting is only a stylistic difference; whether you nest your match statements or not, make sure this is a match statement for the documents in the subcollections:
// Nested match statements are fine
match /post/{postID} {
allow read: if ...
match /comments/{commentID} {
allow read: if ...
}
}
// Unnested match statements are fine
match /post/{postID} {
allow read: if ...
}
match /comments/{commentID} {
allow read: if ...
}
If you're reviewing Realtime Database Security Rules, child nodes inherit from parent nodes, the opposite of the behavior in Firestore.
Check that validations are enforced
Security Rules can enforce type and data validations for specific fields in addition to preventing unwanted access. If you know that some documents have required fields, immutable fields, fields that must be a timestamp, or a field that must contain data in a specific range, check that those are enforced in the Security Rules for those documents. For example, in this rule, I'm checking that the immutable fields of authorUID, publishedAt, and url aren't changed by an update, and that the required fields of content, title, and visible are still present:
allow update: if
// Immutable fields are unchanged
request.resource.data.diff(request.resource.data).unchangedKeys().hasAll([
"authorUID",
"publishedAt",
"url"
]) &&
// Required fields are present
request.resource.data.keys().hasAll([
"content",
"title",
"visible"
]);
If you use the more granular permissions of create, update, and delete in place of write, make sure that validations apply to both create and update.
Admin SDKs bypass Security Rules
Because the Admin SDK authorizes using service account credentials, all requests from the Admin SDK, including Cloud Functions for Firebase, bypass Security Rules. To give a thorough security review for a Firebase app, it's also necessary to look at any other writes that are happening to your backend via the Admin SDK.
For example, if I have great Firestore Security Rules, but a Cloud Function exports data to a storage bucket that has no Security Rules, that would be a terrible way to treat my user's data. The flip side of this is that Cloud Functions can be used in situations where Security Rules don't work, for example, returning individual fields from documents. See this blog post for tips to use Cloud Functions in conjunction with Security Rules.
If you're using Cloud Firestore or Cloud Storage for Firebase, you're also using Security Rules. (If you're using the default rules instead of tailoring them to your app, this is where to start!) We're excited to announce that in the last few months we've released some substantial improvements to the tools for writing and debugging Rules, improvements to the Rules language itself, and increases to the size limits for Rules!. These are a few of the great new features. Check out the Security Rules Release Notes for a comprehensive list of everything we've released.
Sets
We've released several improvements to make the rules language more expressive and succinct. One particularly verbose pattern was comparing the new values of a document to existing values. The new Set type available in Rules is purpose-built for these comparisons, and also has methods for functionality you'd expect for a Set, like getting the intersection, union, or difference between Sets. For example:
Allow a user to create a document if the document has required and optional fields, but not others:
allow create: if (request.resource.data.keys().toSet()
.hasOnly(["required","and","optional","keys"])
Sets come with == and in operators and hasAll, hasAny, hasOnly, difference, intersection, union, and size methods.
New Map methods
Sets are most useful in conjunction with the Map class, and because the request and resource objects are both structured as maps, you're probably already familiar with it. Map recently got a few new methods, diff and get, that will hopefully open the door to more concise rules for everyone. Here's how they work:
Map.diff() is called on one map, and takes the second map as an argument: map1.diff(map2). It returns a MapDiff object, and all of the MapDiff methods, like addedKeys, changedKeys, or affectedKeys return a Set object.
Map.diff() can solve some verbose patterns like checking which fields changed before and after a request. For example, this rule allows an update if the "maxLevel" field was the only field changed:
allow update: if
request.resource.data.diff(resource.data).changedKeys().hasOnly(["maxLevel"]);
In the next example, posts have a field indicating the user role required to modify the post. We'll use Map.get() to get the "roleToEdit" field. If the document doesn't have the field, it will default to the "admin" role. Then we'll compare that to the role that's on the user's custom claims:
allow update, delete: if resource.data.get("roleToEdit", "admin") == request.auth.token.role;
Keep in mind that because Sets are not ordered but Lists are. You can convert a List to a Set, but you can't convert a Set to a List.
Local variables
Local variables have been one of the most requested features in Rules, and they're now available within functions. You can declare a variable using the keyword let, and you can have up to 10 local variables per function.
Say you're commonly checking that a user meets the same three conditions before granting access: that they're an owner of the product or an admin user, that they successfully answered a challenge question, and that they meet the karma threshold.
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
match /products/{product} {
allow read: if true;
allow write: if (exists(/databases/$(database)/documents/admins/$(request.auth.uid))
|| exists(/databases/$(database)/documents/product/owner/$(request.auth.uid)))
&& get(/databases/$(database)/documents/users/$(request.auth.uid))
.data.passChallenge == true
&& get(/databases/$(database)/documents/users/$(request.auth.uid))
.data.karma > 5;
}
match /categories/{category} {
allow read: if true;
allow write: if (exists(/databases/$(database)/documents/admins/$(request.auth.uid))
|| exists(/databases/$(database)/documents/product/owner/$(request.auth.uid)))
&& get(/databases/$(database)/documents/users/$(request.auth.uid))
.data.passChallenge == true
&& get(/databases/$(database)/documents/users/$(request.auth.uid))
.data.karma > 5;
}
match /brands/{brand} {
allow read, write: if (exists(/databases/$(database)/documents/admins/$(request.auth.uid))
|| exists(/databases/$(database)/documents/product/owner/$(request.auth.uid)))
&& get(/databases/$(database)/documents/users/$(request.auth.uid))
.data.passChallenge == true
&& get(/databases/$(database)/documents/users/$(request.auth.uid))
.data.karma > 5;
}
}
}
Those conditions, along with the paths I'm using for lookups can all now become variables in a function, which creates more readable rules:
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
function privilegedAccess(uid, product) {
let adminDatabasePath = /databases/$(database)/documents/admins/$(uid);
let userDatabasePath = /databases/$(database)/documents/users/$(uid);
let ownerDatabasePath = /databases/$(database)/documents/$(product)/owner/$(uid);
let isOwnerOrAdmin = exists(adminDatabasePath) || exists(ownerDatabasePath);
let meetsChallenge = get(userDatabasePath).data.get("passChallenge", false) == true;
let meetsKarmaThreshold = get(userDatabasePath).data.get("karma", 1) > 5;
return isOwnerOrAdmin && meetsChallenge && meetsKarmaThreshold;
}
match /products/{product} {
allow read: if true;
allow write: if privilegedAccess();
}
match /categories/{category} {
allow read: if true;
allow write: if privilegedAccess();
}
match /brands/{brand} {
allow read, write: if privilegedAccess();
}
}
}
You can see at a glance that the same conditions grant access to write to documents in the three different collections.
The updated version also uses map.get() to fetch the karma and passChallenge fields from the user data, which helps keep the new function concise. In this example, if there is no karma field for a user, then the get returns false. Keep in mind that Map.get() fetches a specific field, and is separate from the DocumentReference.get() that fetches a document.
Ternary operator
This is the first time we've introduced an if/else control flow, and we hope it will make rules smoother and more powerful.
Here's an example of using a ternary operator to specify complex conditions for a write. A user can update a document in two cases: first, if they're an admin user, they need to either set the field overrideReason or approvedBy. Second, if they're not an admin user, then the update must include all the required fields:
It was possible to express this before the ternary, but this is a much more concise expression.
The size limit for rules files is now 256 KB!
And finally, here's a feature for those of you with longer rules. Until now, rules files had to be smaller than 64 KB. (To be more specific, the compiled AST of the rules file had to be smaller than 64 KB, and you wouldn't know you were within the limit until you tried to deploy the rules.) This limit was holding some developers back, and once you reached the limit, you had to start making tradeoffs in your rules. We definitely wanted to fix this.
Since this is one of the limits that helps rules return a decision in nanoseconds, we wanted to find a way to increase the limit without sacrificing performance. We optimized how we compile and store the Rules file, and we were able to quadruple the limit to 256 KB!
The limits on rules are in place to keep rules fast enough to return a decision in nanoseconds, but we work hard to keep them workable. Let us know if you start to outgrow any of them
All of these features are informed by the feedback we hear from you about what's great, what's hard, and what's confusing about Firestore Security Rules, so keep letting us know what you think!
Hey, there Firebase Developers! We wanted to let you know of a new feature we added to Cloud Firestore, and that's the ability to increment (or decrement) numeric values directly in the database! Why is this a big deal? Because previously, if you wanted to increment a value in the database, you would need to use a transaction to keep it safe from concurrent writes.
For instance, imagine that your fitness app has a feature that allows groups of friends to pool their steps together for a "team step counter". Then, imagine Bob and Alice both have their current team's total steps recorded locally and try to add their values to it…
If both clients send down these new values at around the same time, it's possible for Alice's changes to be written first...
...and then for Bob's changes to come in and overwrite Alice's changes.
In the past, if you wanted to prevent this from happening, you would have to use a transaction. And while transactions are still a fine solution, they are a little more difficult to write, they don't support cases where the user is offline, and frankly, they seem a bit heavy-handed for something as simple as adding two numbers together.
So now, if you want Bob to record his 500 steps in the app, you can simply do that by asking the server to increment the step_counter value. In an iOS app, for instance, you would write code that looks a little something like this.
With this call, the database would instantly make the change based on whatever value it has. So even if Alice sneaks in a change before Bob's request reaches the server, everything still works.
Now, there are two things to keep in mind when it comes to performing these operations:
First of all, if you do want to add some logic to your operation (like, for instance, making sure this new value doesn't go over or under a certain limit), you'll still need to use a transaction. Luckily, we will soon release a fantastic video all about transactions that I can heartily recommend as a completely unbiased observer.
Second, don't forget that documents are still limited to a sustained write limit of 1 QPS per document. I know it's tempting to look at these numeric functions and start making counters left and right in your database, but if you think these are values you're going to be updating more than once per second, you're still going to want to rely on something like our distributed counter solution.
At Firebase, we're committed to transparency and a thriving developer community, which is why we started open sourcing our SDKs last year. Today, we're continuing on that mission by open sourcing our first Firebase Android SDKs.
For this initial release, we are open sourcing our Cloud Firestore, Cloud Functions, Realtime Database, Storage, and FirebaseCommon SDKs. We intend to release more SDKs going forward, so don't forget to star or watch the repo.
For the SDKs included in the repository, GitHub is the source of truth, though you can also find our project in the Google Open Source directory and on firebaseopensource.com. On GitHub, you'll be able to observe the progress of new features and bug fixes and build a local copy of the SDK on your development machine to preview upcoming releases. Our GitHub README provides more details on how you build, test, and contribute to our Android SDK.
As with our already available open-source iOS, Javascript, Node.js, Java, Python, Go and .NET SDKs, should you find issues in our code you can report them through the standard GitHub issue tracker. Your feedback is vital in shaping the future of Firebase and we look forward to hearing from you on GitHub!
In this edition: Cloud Firestore, Flutter, search with Algolia, Phone Auth, and the brand new ML Kit!
Hey there, Firebase developers! The weather is really starting to warm up here in California as we approach summer, and if you're like me, it's the perfect time of year to go outside, find a nice shady tree, and cozy up with a programming project.
Ok, maybe some of you prefer going to the beach when it's nice out, but I can't seem to figure out how to keep the sand out of my computer! Anyway, when I'm feeling inspired to start a new project, I love having a tutorial to help out. Luckily, the Firebase community is creating new material all the time to keep me busy! If you're looking to find out more about a Firebase product, there's a good chance someone has tried it out and shared their experiences in a blog or video.
Since content can be scattered all over, I figured I'd do the work of scouring the Internet to bring you some of the best Firebase tutorials released over the past couple months. While I'd love to highlight every article and video about Firebase, there simply isn't time or space for them all, so I'm going to showcase a few that I'm most excited about.
Video Tutorials
Cloud Firestore Video Series
Author: The Net Ninja
In this series, Shaun walks you step by step through incorporating Cloud Firestore into a simple web app. He starts with an intro to Cloud Firestore, and continues on through reading data, creating queries, writing data, and real-time updates. The illustrations really helped me visualize the content. The videos also include links to the code on GitHub.
Authenticating Users with Google in Firebase and Firestore inside of Flutter
Author: Tensor Programming
As the title suggests, this tutorial shows developers how to get users logged into your Flutter app with Firebase Authentication. This video picks up in the middle of a series, so you may want to go back to the beginning to learn about incorporating Cloud Firestore with Flutter. I've never worked with Dart before, but the instructor does such a great job of explaining each step, it was still very easy to follow along!
Algolia Firestore Instant Search
Author: Angular Firebase
The Angular Firebase channel creates a myriad of rich tutorials that use different features of Firebase. I'm highlighting this one in particular because so many developers want to know how to implement full text search in Firestore. I also like the use of Cloud Functions, especially the demonstration on how to hide secret API keys in environment variables. Even if you don't use Angular, you'll still learn a lot from this video. If you do use Angular, even better!
Written Tutorials
Firebase Phone Authentication Android
Don't want to choose between written and video tutorials?! Check out this Firebase Phone Authentication tutorial from Belal Khan, that also includes a video of the content. Khan's writing style includes larger blocks of code with explanations in between rather than adding and explaining line-by-line. This is great for those of you who like to learn by examining the code as a whole. I know when I'm learning a new feature, I like to look at a complete sample to see how the pieces fit together, so this was really cool to me.
Making Sense of Google Analytics for Firebase BQ Data
Author: Anže Kravanja
I love this tutorial because, quite frankly, I know little about Google Analytics and nothing about Big Query. This is a great resource for developers who want to get more out of their data stored in Google Analytics for Firebase. The tutorial explains how sessions are tracked in Google Analytics for Firebase and demonstrates step by step how to group Firebase events into user sessions.
Exploring Text Recognition and Face Detection with Google's ML Kit for Firebase on iOS
Adam Talcott looks at a couple of fun and useful cases for ML Kit for Firebase. As a fellow Swift developer, I love to see how other iOS folks are using Firebase.
I hope that these resources inspire you to build something great! I know after checking them out, I'm ready to slather on the sunblock and enjoy some UV rays while I code my next Firebase project!
Thanks to all of the developers who shared their knowledge with us through their tutorials! Firebase wouldn't be what it is without you. I'm going to feature tutorials here every couple of months, so when you find a tutorial you love, be sure to tell me about it! Find me on Twitter at @ThatJenPerson.
We all know that securing your database from malicious or misguided clients is critical. And with Security Rules for Cloud Firestore, you can create a very powerful access control system with simple matching syntax and logic.
Until now, however, testing your rules was difficult. You couldn't test your rules before deploying them, and running arbitrary tests against these rules (to ensure they worked the way you expected) wasn't easy, either.
Today, we're pleased to announce our first big step in making it easier to secure your Cloud Firestore database with the release of the Rules Simulator.
Rules Simulator
With the new Rules Simulator in the Firebase console, you can test your rules as you write them — and before you deploy!
The simulator lets you test document reads, writes, and deletes against any part of your Cloud Firestore database. It will also let you simulate being signed in with a particular userID, so you can better test user-based access control. And for more sophisticated auth-based security, the simulator will help you build authentication tokens for various providers, giving you a preview of what that token will look like and allowing you to directly map the shape of the token to the rules you are writing.
The simulator tests against the rules as they are currently drafted in your editor, not as they are in deployment, allowing you to rapidly test different rules with different types of requests. This means that next time you click the publish button, you can have more confidence that your rules are protecting your data and doing exactly what you expect them to do!
We've also significantly increased the number of get(), exists() and getAfter() calls you can make in each security rule. For a single document request, you can now make 10 document access calls (up from 3). For multi-resource requests, such as batched writes, you'll be able to make a total of 20 document access calls for all documents in that request.
Check out our documentation for more information and examples.
Improved reference documentation
Firestore Security Rules may look like JavaScript, but they're actually a purpose-built language with its own unique syntax and behavior. While we've always provided guides to help you write security rules, many developers mentioned to us that it is too hard to discover all of the functions, types, and edge-cases that you need to learn in order to write complex rules.
That's why we published comprehensive reference documentation on the security rules language and the built-in types and functions that it provides. We hope this will enable you to be more confident when writing advanced rules conditions.
The Cloud Firestore data viewer in the console is a great place to view and update data. You can watch in real time as documents and fields update.
We all know that Cloud Firestore scales to huge amounts of data automatically -- but what about the data viewer? Until now, it was hard to navigate through a big dataset.
To solve the problem, we added a new feature that lets you order and filter right in the data viewer.
We think this will be especially useful in two scenarios:
Sorting by a field. Let's say you have a field last_updated on all of your documents in a collection users, and you want to see the documents that were updated most recently. Just open the menu, choose the field last_updated, select Descending and click apply.
Finding a specific document. Perhaps you have a collection of users which has the fields email and last_updated, and someone tells you they are having a problem with their account. Using the filter menu, input the field email and add a condition (email == "test@gmail.com") to instantly find that user's document.
These are just a few ways that you can use the new menu. We hope it helps you browse large datasets with ease.