Compose Database-as-a-Service Help and Documentation

Everything you need to know about Compose, Hosted or Enterprise, is here in our help system. Whether you run one database for your businesses' sole application or six different databases to support an entire corporation, we've got the information you need.

MongoDB Classic FAQ

Frequent questions from those running MongoDB on Compose

🚧

MongoDB Classic Retirement

All Compose MongoDB Classic databases must be migrated to a current IBM/Compose MongoDB offering by June 30, 2020. Read more here: https://help.compose.com/docs/mongodb-classic-retirement

How can I tell how much disk my database is using

When logged into Compose, when you click on Monitoring and look at the Status tab of one of your provisioned Compose databases, as well as the db.stats() method available in the MongoDB command line interface, you will see output that looks like this:

{
  "db" : "amusement_parks",
  "collections" : 5,
  "objects" : 80614,
  "avgObjSize" : 364.53333333333336,
  "dataSize" : 21069700,
  "storageSize" : 39845376,
  "numExtents" : 5,
  "indexes" : 4,
  "indexSize" : 6012928,
  "fileSize" : 50331648,
  "nsSizeMB" : 16,
  "ok" : 1
}

As you can see, there are a few different storage numbers available. While it can look a bit confusing, understanding these numbers is straight-forward. First, it is important to note that Compose uses the fileSize number to calculate your storage consumption/size. We use this number because it reflects how much space in permanent storage (disk or SSD) that your MongoDB database is consuming at any given time. This includes the fact that MongoDB pre-allocates spaces for some files and does not auto-compact data files.

Here is a quick breakdown of the other storage metics that MongoDB displays:

dataSize

The dataSize value is simply the size of the data contained in your Compose database, regardless of file allocation.

indexSize

Similar to dataSize, the indexSize value is the size of all the indexes contained in your Compose database.

storageSize

The storageSize value is the combination of the dataSize and indexSize values. This is what we use to determine how much storage you are using on the Compose hosting platform.

fileSize

The fileSize value is the size of the space that MongoDB has allocated to contain your database. MongoDB, by default, preallocates files as your database hits certain sizes (to ensure performant writes).

My database is inaccessible!

From time to time, your database could become inaccessible and the reasons often differ. We will use this article to list out some of the most common reasons. Please scan this list to see if any match.

"Don't know about a mongo process"

If you see this error in the web interface, your database is still functioning properly, but the software responsible for reporting information about your database to the web interface is having issues.

Building an index in the foreground

If you start a long-running index build and leave off the {background:true} option, it will block operations until the index has finished building. For more information, see Indexing as a Background Operation.

Mismatch in database and database driver versions

Due to the rapid release schedule from MongoDB, many drivers become stale after only a few releases (which can be as short as 3 to 6 months). Make sure you know which version of MongoDB you are using and if the driver that you are using in your app supports that version of MongoDB. Currently, driver management is one of the issues developers must stay on top of to ensure connectivity.

Your environment is seeing issues

Although rare, from time to time, the environment that hosts your Compose / MongoDB database may be having issues. If you think this is the case, then visit the Compose Status page to review the latest updates.

How do I debug MongoDB connections?

Many people are using different types of servers and platform-as-a-service offerings to connect to Compose servers. Thus, this document will provide a generic debugging process for MongoDB. The process will walk through each step in the connection stack.

  1. Access to Host & Port
  2. Access to Mongo
  3. Using Authentication
  4. Language Driver Assitance

It is important to differentiate the points of contact with your Compose database. The three primary points are:

Localhost (i.e. development machine)

A local computer used for development. These machines connect to Compose over the Internet. Connections over the Internet often result in higher latency and slower connection rates. Compose environments can exist in different physical locations on the Internet -- depending on your location relative to your deployment, latency and speed can be variable.

Application server or PaaS provider

This server runs the application code for production or staging environments. Customers use a diverse set of products, such as virtual servers on AWS, DigitalOcean, Rackspace, Softlayer, Joyent, or Linode (among many others) for their application hosts, or PaaS providers like Heroku, Cloudbees, or EngineYard. As with localhost connectivity, the location of your application server or PaaS provider relative to your Compose deployment can influence performance between the application server(s) and your Compose deployment.

Application code and driver

The code executing connections to the Compose database. When networks are performing as expected, and you can connect to your MongoDB database, the issues are usually in the code itself.

MongoDB drivers are built to handle MongoDB replica sets and shards with high availability and scalability. With MongoDB drivers, development is active and versions change quickly. Most issues can be resolved by updating to the latest version of the driver. Please ensure you are running the latest version of your language's driver, and the driver has been updated in the past 6 - 9 months (e.g. in active development).

If you are comfortable with the above distinctions, please see the following debugging steps:

Access to Host & Port

From both your localhost and application server, run the following:

bash> telnet host port

i.e. if your MongoDB URL is mongodb://rose.compose.io:11000/mydb, then run:

bash> telnet rose.compose.io 11000

If you do not have access to the command line, you can also open your
web browser to your MongoDB host and port. For the above example, go to
http://host:port in your web browser.

Success

If successful using telnet, you should see: Escape character is '^]'.. On success, move to Access to Mongo

If successful using the browser, you will see:

You are trying to access MongoDB on the native driver port.
For http diagnostic access, add 1000 to the port number.

On success, move to Access to Mongo

Failure

If the telnet fails, check the following:

  • Double check the host:port in Compose on the Database's "Admin" page
  • If connecting from a corporate network, check with the firewall/gateway maintainer to confirm outbound traffic is allowed to the host and port found within the Compose UI.
  • If connected via public wifi, the owner of the firewall may block the non-typical outbound ports used by MongoDB. Please try on a different network.
  • Run this test from a different server on AWS or another public cloud. If you get different results than your local machine, debug your connection issues for your local environment.
  • Flush your DNS cache

Access to Mongo

From your localhost and application server (if you have mongo installed), run:

bash> mongo host:port/dbname
mongo> db.isMaster()

Success


For a successful response, the command will print: "ismaster": true.

The isMaster() command is accessible without authentication on MongoDB, and provides a good check to see if MongoDB is responding.

If successful, continue to Using Authentication, else see the "Failure" below.

Failure

If isMaster() fails, check the following:

  • If using a replica set, run the same commands above against the other member fo your replica set
  • View realtime logs in the Compose UI for error messages regarding connection failures

Using Authentication

In the Compose UI, reset your username and password for your database. Access the Compose web interface, navigate to your database, delete your current user, and re-add your user. After resetting your username / password, run the following:

bash> mongo host:port/dbname
mongo> db.auth("yourusername", "yourpassword")

Your MongoDB username / password is not your web Compose username / password, but a different username / password set on the database.

Success

On success, you will see: 1 (yes, that is it). On success, attempt to rerun or debug your application code.

Failure

An authentication failure responds with: { ok: 0.0, errmsg: "auth fails" }

  • Reset your username / password again, and retry authentication
  • Create a new, different another username / password, and retry authentication with new username / password
  • Confirm host:port in Compose UI against used values

Language Driver Assistance

For debugging issues with drivers, the above 3 topics must be verified. Drivers are only as successful as the underlying connection and authentication.

When testing a driver, test with the simplest form of code and build on top of success. By doing this, you will understand the failure point.

Success

A successful connection means you can interact with your database from your programming language.

Failure

Typically, when diagnosing any failures, we do the following:

  • Strip the code down to the essentials
  • In the Compose UI, look at MongoDB log files for errors
  • Insert more error logging into the database to determine cause
  • Double check driver documentation and syntax against the driver version

Still not connecting?

Please consider walking through the steps again, or contact our [email protected].

From the command line, how do I access historical logs to view slow queries?

You can investigate your own logs for anything that might be of interest to you in a number of ways. First you can check the deployment's Live Logs tab under the Monitoring section. This feature is available for all deployment types, even legacy deployments.

You can access logs through our API: https://help.compose.io/docs/classic-api-historical-logs

What you can do is either search your logs using the ?grep="string" option in our API, or you can simply pull down your logs sequentially based on date, reconstruct them into a text file, and then parse them locally with your search tools of choice. Some good things to search for in your logs include:

  • High nscanned values indicating the need for an index. Search example: "nscanned:[0-9]{6,}"
  • High read lock times that indicate poor performance overall. Search example: "r:[0-9]{5,}"
  • High write lock times that also indicate poor performance overall. Search example: "w:[0-9]{5,}"
  • Anything that takes a long time to complete. Search example "[0-9]{4,}ms"
  • Anything that caused MongoDB to not be able to log the query because of sheer length. Search example: "log line attempted"
  • Anything that caused a collection scan. Search example: "COLLSCAN"
  • Anything that caused an index scan that indicates a query isn't complete covered by indexes. Search example: "IXSCAN"

Once you know the queries that are taking the longest, you can start slaying those giants with the help of MongoDB's documentation with how it prefers to have data shaped and queries formed.

How do I upgrade MongoDB Classic?

How do I upgrade?

When the version has been made available, deployments can be immediately upgraded on-demand. Please be aware that due to changes between new and earlier versions, upgrades are irreversible.

Because upgrades to MongoDB are one-way, we recommend heavily testing your application before fully committing to this upgrade. The only way to return to using an older version is to perform a mongodump and restore to a new MongoDB Classic deployment.

If you are ready to upgrade a current deployment that's running an older version, you can follow these steps to upgrade your database automatically. In Settings, under the Change MongoDB Version heading, use the drop-down menu to select the version you would like to upgrade to. If you do not see this section, then you are already on the latest version available for MongoDB Classic. Clicking on the Change Version will start the upgrade process. You can monitor the status of the upgrade from Jobs. The upgrade is performed in place and should only take a few minutes.

MongoDB 3.2 and beyond

MongoDB Classic deployments are not eligible to be upgraded past 3.0.11. All future versions will take place on our latest generation of MongoDB Deployments. If you wish to update to the new MongoDB, you can create a new MongoDB deployment and use its import tool to move your data into the new deployment. More information on using the import tool is here.

Does my driver support the upgrade?

We encourage all customers to verify their driver can support the latest updates available in MongoDB. If you have specific questions about your driver, please reach out to its development team or contact our team at [email protected].

I reduced the size of my database, why am I still being charged?

The way MongoDB works, unless you perform a resync, size on disk is not reclaimed even if data is removed from a database. This is a technical limit imposed by the database engine itself. As a result, if you drastically reduce your data size you'll need to perform a resync through our UI.

As a result of how MongoDB handles permissions, you'll not be able to perform a resync through the application layer or the mongo shell; only through our UI. If you plan to have large data sets that scale down in size frequently, you may want to look into using capped or TTL collections to avoid this scenario.

For advice on how to do that... see below!

How do I use the 'resync' button?

You can resync your replica set from within the UI by going to a deployment, clicking the "settings" tab, and looking for the "Resync Secondary" section. The one caveat is that after you resync a secondary, you have to trigger a stepdown for your replica set which means that your new primary has no oplog.

Our safety mechanism will not allow a resync to be triggered on a secondary through the UI if the oplog has less than 12 hours of data. As a result you'll need to wait 12 hours or a little more for the oplog on the new primary to regain enough play-time before triggering the final resync.

Here's the outline for the steps to resync your replica set:

  • Click resync, and your current secondary pulls fresh data from the primary. You can watch your live logs on the secondary to see how that process is progressing.

  • When the secondary is done syncing, click stepdown to elect a new primary. You can check server status in the UI to see if the secondary is in a startup / recovering state or is in a healthy secondary state to determine if you can perform a stepdown.

  • Wait 12+ hours for the oplog on the new primary to fill with data.

  • Click the resync button again and your new secondary (the former primary) will be resynced and your entire deployment will be reduced to its smallest possible size on disk.

Let us know if we can help with anything else. Remember, it is not enough to resync once. You'll need to resync -> stepdown -> resync to completely reduce your deployment's billable amount of space.

For more information on resync, checkout our blog post: https://www.compose.io/articles/resync-for-mongodb-elastic-deployments/

How do I delete my MongoDB database?

To remove a MongoDB database that you host on Compose, follow these steps:

  1. Log into Compose and click on the database you want to remove.

  2. Click the Admin tab for the selected database, scroll down to the Delete Database area.

  3. Click the Delete Database button to remove the database.

  4. You will be presented with a confirmation message asking to confirm that you want to really delete the database. If you are sure you want to remove the database from your account (and the data contained within the database) then click the Delete button.

❗️

Please Note

This will remove all data. There is no undo.

How do I Access the Oplog?

Adding a user with oplog access.

Customers using our Elastic Deployments can benefit from easy oplog access. The process is as simple as adding a new user to your existing database with oplog access specifically enabled.

Adding a User with Oplog Access

Simply sign in to Compose and select the database where you want to add a user with oplog access. Click Admin in the sidebar, select the Users tab and proceed to fill out the username and password.

Be sure to tick the oplogAccess button and then add the user and they will be able to access your Elastic Deployment's oplog.

How to use Oplog Access

A user with oplog access will have access to a capped collection which contains a constantly updated set of all the changes taking place to the database. By following updates to this collection, an application can track changes as they happen.

An application which can use the oplog in this way is ElasticSearch MongoRiver – most other tools are libraries that aid the construction of applications, like Mongo-Watch for Node.js, MongoRiver for Ruby, or complete frameworks which make use of the oplog to increase their efficiency​, such as Meteor.

Where do I locate the name of my Replica Set?

Many of the official and unofficial MongoDB drivers require that the replica set name be specified in order to connect properly.

When you deploy a new replica set with Compose, that name is assigned for you and can be found under your deployments' Status tab in the Monitoring panel under rs.status()

The name will start with set- followed by a random numerical string.

How the replica set name appears in the Compose UI.

The rs.status() output reflects the current status of the replica set, using data derived from the heartbeat packets sent by the other members of the replica set.

How does Compose work with AWS?

For our single-server shared MongoDB hosting environments, the database environments are spread across multiple availability zones in the US East region.

Can my Compose database be in the same availability zone as my app?

In short, you can't. If you are hosting your application on top of the Amazon EC2 platform and in the US East region, then latency will be very low, as all traffic is internal to that region. Amazon purposefully tries to abstract away the ability to focus on one zone in an effort to prevent customers from flocking to a specific data centre in a region.

Therefore, surprisingly enough, your Zone A is not necessarily someone else's Zone A.

In general, being in the same region works perfectly. All intra-regional traffic has low latency and is not pushed across the internet.

What about your shared replica sets?

The nodes of Compose shared replica sets are purposefully spread across multiple data centers within a region to ensure that, in the event of a single data center within a region showing issues, the replica set will remain active.

How do I use 'mongodump' and 'mongorestore'?

Besides the various ways that Compose provides to backup and restore MongoDB databases, you can also use some of the built-in tools that MongoDB provides. We will cover mongodump and mongorestore in this article.

Things you will need

MongoDB Command Line Tools

If you have MongoDB running locally, you should have these tools installed on your computer already. However, if you don't, you can follow these instructions: MongoDB installation instructions.

Your Compose database credentials

For the Compose database that you want to work with, you will need:

  • Mongo database name
  • The host/environment that your database runs on (and host/environment for your backup DB)
  • Database port
  • Database username
  • Database password

Use mongodump to export data

MongoDB Command Line Tools ship with a utility named "mongodump", which can export data from a running MongoDB instance to a local file system. You may run mongodump against your backup database with a command like this (files will be stored in the dump/ directory relative to your current working path):

> mongodump --host hostname --port portnumber --db database_name -u username -ppassword
> ls dump/*

Use mongorestore to restore data

The mongorestore utility uses the dump files to send a series of inserts to the target database, but ignoring duplicate records. If your data is largely "write once", this is a good way to merge backup data with production data. Caveat: This process will not apply updates that may happened since the backup was made, and will reinsert any since deleted documents.

You may run mongorestore against your target database. The following example makes use of a few assumptions that you'll need to modify for your specific situation:

  • The host in the example is `candidate.40.mongolayer.com`. It will most likely be different for you.
  • The port in the example is `10000`, but will most likely be higher for your deployment.
  • The working path for the following command assumes that `dump/` exists.

With the above in mind, let's look at an example of how to use mongorestore:

mongorestore --host candidate.40.compose.io:10000 --db database_name -u username -ppassword dump/database_name

Dump and restore speed will vary based on the total size of your data, the number of documents, and indexing schema on your database. For the best speed, you should run your commands from a server near your Compose database (an Amazon EC2 instance, for example).

How do I backup my database locally?

Even though Compose provides a mechanism for doing regular backups to Amazon S3 and other cloud-storage solutions, sometimes it is good to pull down a local copy of your database for running tests or, if you are about to run a large migration, to keep a safe copy before anything changes. To do this is straight-forward, using the mongodump tool provided along with MongoDB.

TL;DR

  1. Use mongodump to export the data from your Compose database.

Requirements

  1. MongoDB Command Line Tools -- if you have MongoDB running locally,
    you should have these tools installed.

  2. Compose credentials:

  • Mongo database name
  • The host/environment that your database runs on
  • Database port
  • Database username
  • Database password

You can get this information out of Compose easily. Once logged into Compose,
simply click your database, then click on "Admin", and you will see the connection string for your database.

Perform the Mongodump

Now that you have all the information you need, creating the necessary mongodump is easy. Head to your most commonly used terminal window and type the following command:

mongodump --host <hostname>.compose.io:<portname> --db <database_name> -u <username> -p<password> -o /path/to/local/file

Given the above example, your command may look like this:

mongodump --host hatch.compose.io:10040 --db trees -u mhquser -p123456 -o ~/Desktop

Please Note: The lack of a space between the -p and your database password is significant.

That is it ... if you need to update the information, you can run a mongorestore (covered above) to push data from your local computer (or from another server) up to your Compose database. From there, you can use Compose for all your Mongo hosting.

How are resources allocated to my database?

Let's say your current MongoDB Classic Deployment is only taking 25GB of space. Our standard resource provisioning ratio is:

1GB of storage
0.1GB of RAM
60 disk IOPs

... so you've got 2.5GB of RAM and r/w IOPS. Our MongoDB Classic Deployments can potentially scale to 1TB and even larger sizes and can grow to 100,000+ IOPS.

How do I take advantage of a Replica Set?

The benefits of a replica set don't come entirely without some work on the application and driver side. First you'll need to get the replica set URI from the Compose Admin section of your database. It will include your username and password as well as the hostnames of both hosts, their ports, and the database name and will look something like this:

mongodb://youruser:[email protected]:10017,candidate.19.mongolayer.com:10017/yourdatabase

You'll also need to take into consideration things like read preference and using some form of try / catch mechanism to handle errors including if your replica set has a step-down event and switches primaries. Remember not to directly address one MongoDB server, but use the replica set URI which will discover all members.

If you write to your primary directly, then when a step-down happens in the case of a degraded host, instance, or networking issues you'll lose all the benefits of a highly available replica set.

We wrote a short, introductory blog post about the topic here: https://www.compose.io/articles/support-desk-keeping-mongodb-live/

For the specifics of how to handle all the complexities and best practices for your driver and language, you'll need to consult the documentation and communities around the tools that you're using. However, once you get those elements in place, you'll have a pretty solid interface to your replica set that will be able to withstand conditions that most other databases can't.

What does the error 'not master and slaveOk=false' mean?

The error "not master and slaveOk=false" indicates that your application is attempting to write to a secondary. Drivers typically have the ability to determine which MongoDB server is a primary even after a stepdown, however if the drivers aren't configured right they'll still attempt to write to a server even after it became a secondary.

That error can also indicate that the application is attempting to read from a secondary and doesn't have the "slaveOk" option set to true. It all depends on the driver or tools that you're using to connect to the database, so you'll need to consult with your driver's documentation and/or community to figure out how to detect the primary, utilize the replica set URI, and/or set slaveOk to true.

Can I encrypt my deployment with SSL?

MongoDB Classic does not offer SSL. If you need SSL enabled, our current standard MongoDB deployments offer it.

To learn more about SSL and MongoDB: https://www.compose.io/articles/bringing-ssl-and-more-to-compose-mongodb/

To learn how enabling SSL on a new standard deployment affects how you connect: https://help.compose.com/docs/connecting-to-mongodb

Can you tell me about stepdown events?

MongoDB replica sets are made to be highly availably, but only insofar as the application drivers that access the replica set are set up to detect and recover from a step-down, which your replica set experienced.

Step-down events aren't entirely uncommon, so handling them is fairly important. By themselves, step-downs aren't necessarily a sign that something is broken with the replica set itself. A brief list of the things that we've seen trigger step-downs would be: heavy usage causing missed heartbeats, very brief undocumented cloud network outages from providers, and queries or evaluated JavaScript that trigger a bug in that version of MongoDB which causes a restart and thus an election.

You'll need to consult with the documentation and community for the driver that you're using to determine how best to handle replica sets and the step-down of any primary and election of a new primary. Each driver and language poses some of its own special challenges when dealing with replica sets. Some things you'll want to know about your replica set include the replica set name as well as the replica set URI.

You can find your replica set URI by going into your MongoDB deployment, clicking into the specific database that you're working with, clicking the "Admin" tab and looking under the Overview sub-tab for your database's specific replica set URI.

How are MongoDB back-ups done?

We've written a blog post that should answer this question well: https://www.compose.io/articles/how-we-do-it-backups/

Why would I use Compose over another MongoDB-as-a-service provider?

It is difficult to compare Compose deployments to others you may find out in the wild; there may be some similarities with other providers, but there aren't any direct equivalents. We like to begin by talking to you about what your project actually entails. We'd love to know the answers to some questions, such as:

  • What is the prospective size of your database?
  • What are your growth expectations over the next 6-12 months?
  • Which region you're planning to host in?
  • Any specific performance needs for RAM or IOPs?
  • Need any security features such as SSL or VPC?
  • Do you need to tie into other databases?

Once we know exactly how to help you, we'll be able to give you the information you need to compare to other services.


Still Need Help?

If this article didn't solve things, summon a human and get some help!

Updated 9 months ago

MongoDB Classic FAQ


Frequent questions from those running MongoDB on Compose

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.