REST users rejoice! Conditional requests are here!

How would you increment an integer in the Realtime Database? It's not always as easy as just grabbing a value, adding one, and writing it back again. When multiple users are simultaneously reading, updating and writing the same value, some users' writes could get "lost" due to race conditions.

Let's say you're using the Realtime Database to store upvotes on videos, and two of your users want to upvote the "Silly Cat Video" which currently stands at ten upvotes. Each user reads the value '10' from the database. They then independently increment the value locally and write '11' back to the database. "Silly Cat Video" now only has eleven upvotes when it should have twelve.

How do we prevent this? On a mobile or web client, you could use transactions, which have long been supported in our realtime SDKs. But what if you're using the REST API?

Good news for you: we've just added conditional requests to the REST API! While conditional requests work differently than the realtime transactions, we can still use them to safely perform atomic operations.

Here's how you upvote that cat video the right way:
Step 1: Request Data and ETag

Request:

# -i returns headers in the response, -H sets a header
curl -i \
-H 'X-Firebase-ETag: true' \
'https://<your-database>.firebaseio.com/videos/silly_cat_video/upvotes.json'

Response:

HTTP/1.1 200 OK
...
ETag: ViJFJowpbyRvgGNPzPJdGeN+mCY=
...

10  // Value at the specified location

Step 2: Write the New Value

Request:

# -X sets the method (defaults to GET), -d provides data to the method
curl -i \
-X PUT \
-d '11' \
-H 'if-match: ViJFJowpbyRvgGNPzPJdGeN+mCY=' \
'https://<your-database>.firebaseio.com/videos/silly_cat_video/upvotes.json'

If no other users have incremented upvotes.json the PUT request will succeed. If another user has, the request will fail and the response will contain the new ETag and data. Use these to calculate a new upvotes value and retry the PUT request.

But wait, what does all this ETag and if-match stuff mean, and how do they work?

The Realtime Database's conditional REST requests use ETags and the if-match header from the HTTP conditional requests standard. An ETag is a short unique identifier for data at a location: if the data changes, so does its ETag. if-match, when set to an ETag value, tells the database to only complete a request's operation if the data the request would overwrite matches the provided ETag.
Together, these two features enable a Compare and Swap (CAS) paradigm. Compare and Swap uses the two step process we saw in our video upvote example:
Step 1:
Fetch the location you wish to update along with its ETag.
Step 2:
Issue a PUT request with new data in the body and the ETag of the old data in the if-match header.
If the data at the location changes between Step 1 and Step 2, the database will not complete the operation and will return an error containing the new ETag and data. You can use these values to construct a new PUT request and retry from Step 2.

For more details and to get started check out our REST conditional requests guide.

Sometimes the worst bugs to track down are the ones that seem to be impossible to reproduce. Or worse, inconsistent performance problems that can cause "Application Not Responding" errors in Android apps. No matter how much test code you write, these types of errors seem to have a way of sneaking into your app and causing problems for your users. However, with some clever use of Android's StrictMode API, alongside Firebase Test Lab for Android, you can find out about these problems before they reach production!

Over the years since Android was first available, a number of best practices have been observed for writing solid apps. For example, you shouldn't be performing blocking I/O on the main thread, and you shouldn't store references to Activity objects that are held after the Activity is destroyed. While it's likely that no one will force you to observe these practices (and you may never see a problem during development), enabling StrictMode in your app will let you know where you've made a mistake. These are called "policy violations", and you configure these policies with code in your app.

Personally, I don't want any of my code to cause any StrictMode violations. In fact, I'd like to consider them bugs that are just as serious as a regular crash, because they could crash my app in some cases! And, in fact, I can configure all StrictMode violations to crash the app. The Java code for that looks like this:

    
private static final StrictMode.ThreadPolicy FATAL_THREAD_POLICY =
    new StrictMode.ThreadPolicy.Builder()
        .detectAll()
        .penaltyLog()
        .penaltyDeath()
        .build();

private static final StrictMode.VmPolicy FATAL_VM_POLICY =
    new StrictMode.VmPolicy.Builder()
        .detectAll()
        .penaltyLog()
        .penaltyDeath()
        .build();

public static void enableFatalStrictMode() {
    if (shouldEnableFatalStrictMode()) {
        StrictMode.setThreadPolicy(FATAL_THREAD_POLICY);
        StrictMode.setVmPolicy(FATAL_VM_POLICY);
    }
}

Here I have both a ThreadPolicy (for the main thread) and a VmPolicy (for the entire app) that look for all known violations, and will both log that violation and crash the app. Take a look through the javadoc for those builders to learn about all the situations they can catch.

Also notice that I'm enabling the policies conditionally. You probably don't want an accidental violation to crash on your users in production, so the method shouldEnableFatalStrictMode() typically looks like this, which activates it for your debug builds only:

    
private static boolean shouldEnableFatalStrictMode() {
    return BuildConfig.DEBUG;
}

What I'd like to do is take this further and have StrictMode crash my app also when it's running in Firebase Test Lab. This is handy because the crash will fail the test, and I'll get a report about that in the test results. For that to happen, I can change the method to query a system setting on the device that's unique to devices in Test Lab. Note that this requires an Android Context to obtain a ContentResolver:

    
private static boolean shouldEnableFatalStrictMode() {
    ContentResolver resolver = context.getContentResolver();
    String isInTestLab = Settings.System.getString(resolver, "firebase.test.lab");
    return BuildConfig.DEBUG || "true".equals(isInTestLab);
}

Now, my instrumented tests (and the automated Robo test) will crash with any StrictMode violations, and I'll see those very clearly in my Test Lab report. Here's an app that crashed in Test Lab with a StrictMode violation during a Robo test:

This doesn't tell you exactly what happened, though. Digging into the logs in the test report, I see the details of the violation right before the crash:

D/StrictMode(6996): StrictMode policy violation; ~duration=80 ms: android.os.StrictMode$StrictModeDiskReadViolation: policy=327743 violation=2
D/StrictMode(6996):  at android.os.StrictMode$AndroidBlockGuardPolicy.onReadFromDisk(StrictMode.java:1415)
D/StrictMode(6996):  at java.io.UnixFileSystem.checkAccess(UnixFileSystem.java:251)
D/StrictMode(6996):  at java.io.File.exists(File.java:807)
D/StrictMode(6996):  at android.app.ContextImpl.getDataDir(ContextImpl.java:2167)
D/StrictMode(6996):  at android.app.ContextImpl.getPreferencesDir(ContextImpl.java:498)
D/StrictMode(6996):  at android.app.ContextImpl.getSharedPreferencesPath(ContextImpl.java:692)
D/StrictMode(6996):  at android.app.ContextImpl.getSharedPreferences(ContextImpl.java:360)
D/StrictMode(6996):  at android.content.ContextWrapper.getSharedPreferences(ContextWrapper.java:167)
D/StrictMode(6996):  at com.google.firebasesandbox.MainActivity.onCreate(MainActivity.java:25)
D/StrictMode(6996):  at android.app.Activity.performCreate(Activity.java:6982)
D/StrictMode(6996):  at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1213)
D/StrictMode(6996):  at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2770)
D/StrictMode(6996):  at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892)
D/StrictMode(6996):  at android.app.ActivityThread.-wrap11(Unknown Source:0)
D/StrictMode(6996):  at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1593)
D/StrictMode(6996):  at android.os.Handler.dispatchMessage(Handler.java:105)
D/StrictMode(6996):  at android.os.Looper.loop(Looper.java:164)
D/StrictMode(6996):  at android.app.ActivityThread.main(ActivityThread.java:6541)
D/StrictMode(6996):  at java.lang.reflect.Method.invoke(Native Method)
D/StrictMode(6996):  at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
D/StrictMode(6996):  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)

So we've discovered here that an activity's onCreate() called getSharedPreferences(), and this method ends up reading the device filesystem. Blocking the main thread like this could be the source of some jank in the app, so it's worth figuring out a good way to move that I/O to another thread.

Now we have a way to check for StrictMode violations during a test, but there's still a few more details to work out.

When should I enable StrictMode?

If you do this immediately when the app launches (in an Application subclass or a ContentProvider), there is a subtle bug in some versions of Android that may need to be worked around in order to avoid losing your StrictMode policies when the first Activity appears.

When should I disable StrictMode?

There might be some times when you know there is a violation, but you can't fix the code right away for whatever reason (for example, it's in a library you don't control). In that case, you might want to temporarily suspend the crashing behavior and simply log it instead. To temporarily suspend the crash, you can define a couple other non-fatal policies and swap them in where you know there's an issue to ignore:

    
private static final StrictMode.ThreadPolicy NONFATAL_THREAD_POLICY =
    new StrictMode.ThreadPolicy.Builder()
        .detectAll()
        .penaltyLog()
        .build();

private static final StrictMode.VmPolicy NONFATAL_VM_POLICY =
    new StrictMode.VmPolicy.Builder()
        .detectAll()
        .penaltyLog()
        .build();

public static void disableStrictModeFatality() {
    StrictMode.setThreadPolicy(NONFATAL_THREAD_POLICY);
    StrictMode.setVmPolicy(NONFATAL_VM_POLICY);
}

Similarly, you may not want to track all the possible violations all the time. In that case, you might want to instruct the policy builder to only detect a subset of the potential issues.

Get out there and test your app!

While StrictMode violations are not always the worst thing for your users, I've always found it educational to enable StrictMode in order to find out where my app might be causing problems. In combination with Firebase Test lab, it's a great tool for maximizing the quality of your app.

One of my favorite parts about working with the Cloud Functions for Firebase team is helping developers move logic from their mobile apps to a fully managed backend hosted by Firebase. With just a few lines of JavaScript code, they're able to unify logic that automatically takes action on changes to the contents of their Realtime Database. It's really fun to see what people build!

When Cloud Functions was first announced at Cloud Next 2017, there was just one trigger available for all types of changes to the database. This trigger is specified using the onWrite() callback, and it was the code author's responsibility to figure out what sort of change occurred. For example, imagine your app has a chat room, and you want to use Firebase Cloud Messaging to send a notification to users in that room when a new message arrives. To implement that, you might write some code that looks like this:

exports.sendNotification = functions.database
        .ref("/messages/{messageId}").onWrite(event => {
    const dsnap = event.data  // a DeltaSnapshot that describes the write
    if (dsnap.exists() && !dsnap.previous.exists()) {
        // This is a new message, not a change or a delete.
        // Send notifications with FCM...
    }
})

Note that you have to check the DeltaSnapshot from the event object to see if new data now exists at the location of the write, and also if there was no prior data at that location. Why is this necessary? Because onWrite() will trigger on all changes to data under the matched location, including new messages, updated messages, and deleted messages. However, this function is only interested in new messages, so it has to make sure the write is not an update or a delete. If there was an update or delete, this function would still get triggered and return immediately. This extra execution costs money, and we know you'd rather not be billed for a function that doesn't do useful work.

Today, it gets better

The good news is that these extra checks are no longer necessary with Cloud Functions database triggers! Starting with the firebase-functions module version 0.5.9, there are now three new types of database triggers you can write: onCreate(), onUpdate(), and onDelete(). These triggers are aware of the type of change that was made, and only run in response to the type of change you desire. So, now you can write the above function like this:

exports.sendNotification = functions.database
        .ref("/messages/{messageId}").onCreate(event => {
    // Send notifications with FCM...
})

Here, you don't have to worry about this function getting triggered again for any subsequent updates to the data at the same location. Not only is this easier to read, it costs you less to operate.

Note that onWrite() hasn't gone away. You can still keep using it with your functions for as long as you like.

Avoid infinite loops with onWrite() and onUpdate()

If you've previously used onWrite() to add or change data at a location, you're aware that the changes you made to that data during onWrite() would trigger a second invocation of onWrite() (because all writes count as writes, am I right?).

Consider this function that updates the lastUpdated property of a message after it's written:

exports.lastUpdate = functions.database
        .ref("/messages/{messageId}").onWrite(event => {
    const msg = event.data.val()
    msg.lastUpdated = new Date().getTime()
    return event.data.adminRef.set(msg)
})

This seems OK at first, but there's something very important missing. Since this function writes back to the same location where it was triggered, it will effectively trigger another call to onWrite(). You can see here that this will cause an infinite loop of writes, so it needs some way to bail out of the second invocation.

It turns out that onUpdate() function implementations share this concern. One solution in this specific case could be to check the existing value of lastUpdated and bail out if it's not more than 30 seconds older than the current date. So, if we want to rewrite this function using onUpdate(), it could look like this:

exports.lastUpdate = functions.database
        .ref("/messages/{messageId}").onUpdate(event => {
    const msg = event.data.val()
    const now = new Date().getTime()
    if (msg.lastUpdated > now - (30*1000)) {
        return
    }
    msg.lastUpdated = now
    return event.data.adminRef.set(msg)
})

This function now defends against infinite loops with a little extra logic.

To start using these new triggers, be sure to update your firebase-functions module to 0.5.9. And for more information about these updates, check out the documentation right here.

Building a great game is tough. You have to make it easy to pick up, challenging, a little bit addicting, and, of course, fun. Not to mention the technical effort involved in designing all the infrastructure that will power your blockbuster game. Our goal with Firebase is to take care of that last piece, so that you can focus on all of the elements that make your game unique.

At Google I/O this year, we decided to put our platform to the test and build a game with Firebase. The result was AppShip3000 and we're pretty happy with how it turned out.

How it works

AppShip3000 is a 3 person, collaborative, rocketship-flying, trivia game. The players work together to control a rocket ship that is flying upwards and quickly running out of fuel. Asteroids rain down from the top of the screen, while the crew answers Firebase and Google Cloud trivia to reach higher heights.

The catch is that players can only see either the question or one of the multiple choice answers, never both. They have to talk (or yell) to each other to figure out which player has the right answer before time runs out. Similarly, one player sees asteroids before the others and has to shout out where they'll be, so the entire team can move the rocket. All players have to move the joystick in that same direction to successfully avoid losing fuel from an asteroid collision.

Building with Firebase

Now, let's take a look at how Firebase powers AppShip3000. First off, we use Firebase Authentication for the sign-in experience. A player opens a progressive web app (PWA) on her phone and logs in using her Google account. Creation of an account is an Authentication Trigger for a Cloud Function that generates a custom button sequence (or launch code). This launch code is used to associate the account with the game controller for the duration of the game, so that when the game finishes, all stats are automatically associated with her account.

When a game finishes, Realtime Database is used to immediately update the leaderboard in the PWA. At I/O, we also had giant screens reflecting the leaderboards. Realtime Database is also used to manage the trivia questions in the game. This allows for on-the-fly updates to questions that can be written after one game and pushed to production in time for the next game.

We also generate an image of the rocket's flight path after each game and store it in Cloud Storage for Firebase. This action triggers another Cloud Function that sends a notification to the player, using Firebase Cloud Messaging. The notification lets the player know about the souvenir image and prompts her to share it with friends.

Getting Started

We had a blast building AppShip3000. Using Firebase to handle most of the infrastructure allowed us to focus on building a unique game that (we think) turned out to be a lot of fun to play. If you're interested in using Firebase in your own game, check our our C++ and Unity documentation. And to learn more about how Firebase and Google Cloud were used to manage the data pipeline of AppShip3000, check out our I/O talk. Can't wait to see what you build!

Last April, we announced the general availability of the Firebase Admin SDK for Python. The initial release of this SDK supported two important features related to Firebase Authentication: minting custom tokens, and verifying ID tokens. Now, we are excited to announce that database support is available in the Firebase Admin SDK for Python starting from version 2.1.0.

Due to the way it's implemented, there are several notable differences between this API and the database APIs found in our other Admin SDKs (Node.js and Java). The most prominent of these differences is the lack of support for realtime event listeners. The Python Admin SDK currently does not provide a way to add event listeners to a database reference in order to receive realtime update notifications. Instead, all data retrieval operations are provided as blocking methods. However, despite these differences there's a lot that can be achieved using this API.

The database module of the Python Admin SDK facilitates both basic data manipulation operations and advanced queries. To begin interacting with the database from a Python environment, initialize the SDK with the Realtime Database URL:

import firebase_admin
from firebase_admin import credentials

cred = credentials.Cert('path/to/serviceKey.json')
firebase_admin.initialize_app(cred, {
    'databaseURL' : 'https://my-db.firebaseio.com'
})

Then obtain a database reference from the db module of the SDK. Database references expose common database operations as Python methods (get(), set(), push(), update() and delete()):

from firebase_admin import db

root = db.reference()
# Add a new user under /users.
new_user = root.child('users').push({
    'name' : 'Mary Anning', 
    'since' : 1700
})

# Update a child attribute of the new user.
new_user.update({'since' : 1799})

# Obtain a new reference to the user, and retrieve child data.
# Result will be made available as a Python dict.
mary = db.reference('users/{0}'.format(new_user.key)).get()
print 'Name:', mary['name']
print 'Since:', mary['since']

In the Firebase Realtime Database, all data values are stored as JSON. Note how the Python Admin SDK seamlessly converts between JSON and Python's native data types.

To execute an advanced query, call one of the order_by_* methods available on the database reference. This returns a query object, which can be used to specify additional parameters. You can use this API to execute limit queries and range queries against your data, and retrieve sorted results.

from firebase_admin import db

dinos = db.reference('dinosaurs')

# Retrieve the five tallest dinosaurs in the database sorted by height.
# 'result' will be a sorted data structure (list or OrderedDict).
result = dinos.order_by_child('height').limit_to_last(5).get()

# Retrieve the 5 shortest dinosaurs that are taller than 2m.
result = dinos.order_by_child('height').start_at(2).limit_to_first(5).get()

# Retrieve the score entries whose values are between 50 and 60.
result = db.reference('scores').order_by_value() \
    .start_at(50).end_at(60).get()

Take a look at the Admin SDK documentation for more information about this new API. Also check out our Github repo, and help us further improve the Admin SDK by reporting issues and contributing patches. In fact, it was your continuing feedback that motivated us to build and release this API in such a short period. Happy coding with Firebase!

Testing your application is a great way to help maximize its quality, and many of you know that Firebase Test Lab for Android has some useful tools for testing Android apps. If you're the type of engineer who likes to maximize test coverage by writing instrumented tests (and regular unit tests), you can send those to Test Lab for execution. Even if you don't like writing tests, you have some options. You can record instrumented tests by interacting with your app using Espresso Test Recorder in Android Studio. And there's almost no effort required at all to run a Robo test that automatically crawls your app.

These tests are helpful for data-driven apps because there are robust test frameworks that understand how to navigate the Android platform widgets used to receive input and display data on screen. However, most games don't work with platform widgets. Games typically take over the screen using their own UI elements, and provide their own touch controls. As a result, it's extremely difficult to write instrumented tests for testing games, and Robo test won't know how to navigate the game at all.

To help deal with these challenges with testing games, the Test Lab team has come up with a way for game developers to test their games effectively across many of the devices that Test Lab offers. It's a new type of test called a Game Loop Test, and it's available today in beta.

How does a Game Loop test work?

If you've seen arcade video games operate, you know that they're always showing something on screen, typically some automated demo of the game, called "attract mode". With Firebase Test Lab, game developers can now use this concept of attract mode to construct test scenarios, and Test Lab will arrange for those scenarios to be invoked in sequence. This gives developers the opportunity to test a wide variety of game levels and situations in a single test run on all the devices provided by Test Lab. If there are any problems with a scenario, you'll find out which ones are problematic, so you can focus your efforts on improving that specific case.

Even better, Test Lab now provides performance data with every report. Of particular interest to game developers is a graph of the rendered frame rate over time throughout the test, tied to a video of the device display. This helps you quickly identify the parts of your game that aren't rendering at an acceptable FPS.

In addition to FPS, there are other performance metrics available for all apps. You'll be able to see your app's CPU utilization, memory usage, and network ingress and egress. Notice how you can jump directly to the point in the video where you're observing performance problems.

If you're a game developer, now is a great time to check out Firebase Test Lab for Android to see what it can do to help the quality of your game.