Just enough code to be dangerous

Part One describes the manual process for setting up a MongoDB replica set using EC2 instances. This part presents the program that automates the operation. It’s written in Java in what I’d describe as a scripting style, such as you might use to write a Ruby or Python program that performs the same sort of operation. It’s not designed for completely robust unattended operation, it doesn’t need to be for its use cases. The complete source code for the MongoDB on AWS project, including pom etc. is available on github. Rather than write a post describing the program I’ve just added lots of comments and posted the significant parts here, it should be easy enough to follow along just reading from the top.

p.s. Some interesting and useful information from Foursquare describing how they use replica sets. Extra configuration wrinkles on top of what I’ve done in this experiment.

package com.zanthan.aws.scripting;

// imports removed

/**
 * Create a new MongoDB replica set using two Amazon EC2
 * instances. This program has to be run from the correct
 * directory because it uses relative paths for some files.
 * It is deliberately written in a scripting style, the
 * sort of program you might think of writing in Ruby or
 * Python.
 *
 * @author amoffat Alex Moffat
 */
public class StartMongoDBReplicaSet {

    /**
     * Id of the image to use for each of the instances.
     */
    private static final String IMAGE_ID =
            "ami-74f0061d";

    /**
     * Type of instance to use.
     */
    private static final String INSTANCE_TYPE =
            "t1.micro";

    /**
     * Availability zone to start the instances in.
     */
    private static final String AVAILABILITY_ZONE =
            "us-east-1a";

    /**
     * Security group that defines firewall rules for
     * the MongoDB instances.
     */
    private static final String SECURITY_GROUP =
            "MongoDB";

    /**
     * Name of the key pair to use for the instances.
     */
    private static final String KEY_NAME =
            "KeyPair20110224";

    /**
     * Location of the file containing the private key from
     * the key pair used for launching instances.
     */
    private static final String KEY_FILE =
            "keys/KeyPair20110224.pem";

    /**
     * Id of the EBS snapshot containing the MongoDB executables.
     * This will be used to create the disks mounted by the AWS
     * instances.
     */
    private static final String SNAPSHOT_ID =
            "snap-a9c5e0c6";

    /**
     * Name of the device the disks created from the snapshot will
     * be attached to. This is referenced in the MongoDBInit.txt file.
     */
    private static final String DEVICE_NAME =
            "/dev/sdf";

    /**
     * File of CloudInit instructions.
     */
    private static final String LOCAL_USER_DATA_FILE =
            "MongoDBInit.txt";

    /**
     * Command that is executed on each Amazon EC2 instance to start
     * MongoDB. The private ip address of the other machine in the
     * replica set is appended to the command.
     */
    private static final String REMOTE_MONGO_DB_START_CMD =
            "/mongodb/mongodb-linux-x86_64-1.8.1/bin/mongod " +
                    "--dbpath /home/ec2-user/data/db " +
                    "--logpath /home/ec2-user/mongodb.log " +
                    "--nohttpinterface " +
                    "--fork " +
                    "--replSet logSet/";

    /**
     * Path to the local MongoDB mongo executable. This is used
     * to issue the commands to configure the replica set.
     */
    private static final String LOCAL_MONGO_EXECUTABLE =
            "mongodb-osx-x86_64-1.8.1/bin/mongo";

    /**
     * The command to configure the replica set. Need additional
     * single quotes because of Java MessageFormat formatting rules.
     * {0} and {1} are substituted with the private ip addresses
     * of the two machines in the set.
     */
    private static final String START_REPLICASET_CMD =
            "'db.runCommand({\"replSetInitiate\" : {\n" +
                    "\"_id\" : \"logSet\",\n" +
                    "\"members\" : [\n" +
                    "{\n" +
                    "\"_id\" : 1,\n" +
                    "\"host\" : \"'{0}'\"\n" +
                    "},\n" +
                    "{\n" +
                    "\"_id\" : 2,\n" +
                    "\"host\" : \"'{1}'\"\n" +
                    "}\n" +
                    "]}})'";

    /**
     * Start the whole process.
     *
     * @param args Arguments passed to the program are ignored.
     * @throws IOException In the event of an error.
     */
    public static void main(String[] args)
            throws IOException {

        // Create the AmazonEC2 client to communicate with EC2
        AmazonEC2 ec2 = createEC2Client();

        // Create object to execute ssh commands on the
        // EC2 instances started by the program.
        SshCommandExecutor sshCommandExecutor =
                createSSHCommandExecutor();

        // Create the replica set.
       new StartMongoDBReplicaSet(ec2, sshCommandExecutor).run();
    }

    /**
     * Create the object that will use SSH to execute commands
     * on the EC2 instances.
     *
     * @return New executor.
     * @throws IOException If errors occur.
     */
    private static SshCommandExecutor createSSHCommandExecutor()
            throws IOException {

        // Check that the key file we want to use exists.
        File keyFile = new File(KEY_FILE);
        if (!keyFile.exists()) {
            throw new IllegalStateException("Key file " +
                    keyFile.getPath() + " does not exist.");
        }

        // Create a temporary hosts file. The ssh command will
        // automatically add hosts to this file. A new one is
        // used instead of the default one. EC2 reuses IP addresses
        // so you may find you get the different host fingerprints
        // for the same IP over the course of multiple executions.
        // There is no way to tell SSH not to blow up in this case
        // but using a new host file for each replica set guarantees
        // it won't happen.
        File hostsFile = File.createTempFile("Hosts", ".txt");
        hostsFile.deleteOnExit();

        return new SshCommandExecutorImpl(keyFile, hostsFile);
    }

    /**
     * Create Amazon EC2 client to use to communicate with AWS. The
     * AWS_CREDENTIAL_FILE environment variable that is used by the
     * Amazon command line tools is used here to find the access
     * and secret key we need.
     *
     * @return The client.
     * @throws IOException If things go wrong.
     */
    private static AmazonEC2 createEC2Client()
            throws IOException {

        // Find the name of the credentials file and make sure
        // it exists.
        String credentialFileName =
                System.getenv("AWS_CREDENTIAL_FILE");
        if (credentialFileName == null) {
            throw new IllegalStateException("No value for environment " +
                    "variable AWS_CREDENTIAL_FILE");
        }
        File propertiesFile = new File(credentialFileName);
        if (!propertiesFile.exists()) {
            throw new IllegalStateException("Properties file " +
                    propertiesFile.getPath() + " does not exist.");
        }

        // Load the credentials file as a properties file.
        FileReader propertiesReader = new FileReader(propertiesFile);
        Properties awsKeys = new Properties();
        awsKeys.load(propertiesReader);

        // Pull out the correct properties to create the basic
        // credentials we need.
        BasicAWSCredentials credentials =
                new BasicAWSCredentials(awsKeys.getProperty("AWSAccessKeyId"),
                        awsKeys.getProperty("AWSSecretKey"));

        return new AmazonEC2Client(credentials);
    }

    /**
     * Communication with EC2.
     */
    private final AmazonEC2 ec2;

    /**
     * Execute commands on remote hosts using SSH.
     */
    private final SshCommandExecutor sshCommandExecutor;

    /**
     * Create an new object.
     *
     * @param ec2 Communication with EC2.
     * @param sshCommandExecutor Remote command execution
     */
    public StartMongoDBReplicaSet(AmazonEC2 ec2,
                                  SshCommandExecutor sshCommandExecutor) {
        this.ec2 = ec2;
        this.sshCommandExecutor = sshCommandExecutor;
    }

    /**
     * The top level entry point. This is the basic script that sequences
     * the necessary operations.
     *
     * @throws IOException In the event of an error.
     */
    private void run()
            throws IOException {

        // Start the instances.
        List instances = startInstances();

        // Even after the instances have started we need
        // to wait a little while for the CloudInit
        // processing to complete so that we can connect
        // to them with SSH.
        waitABit(45, "for initialization to complete.");

        // Get the public and private ip addresses of
        // the two instances.
        String machineOnePublicIp =
                instances.get(0).getPublicIpAddress();
        String machineOnePrivateIp =
                instances.get(0).getPrivateIpAddress();

        String machineTwoPublicIp =
                instances.get(1).getPublicIpAddress();
        String machineTwoPrivateIp =
                instances.get(1).getPrivateIpAddress();

        // Start MongoDB on the first machine.
        startMongoDb(machineOnePublicIp,
                machineTwoPrivateIp);

        // Start MongoDB on the second machine.
        startMongoDb(machineTwoPublicIp,
                machineOnePrivateIp);

        // Wait till everything is ready.
        waitABit(30, "for MongoDB to become available.");

        // Start the replica set. We'll contact machineOne.
        startReplicaSet(machineOnePublicIp,
                machineOnePrivateIp,
                machineTwoPrivateIp);

    }

    /**
     * Start the 2 instances needed for the replica set.
     *
     * @return List of the instances. All will be in the
     * running state.
     * @throws IOException If things go wrong.
     */
    private List startInstances()
            throws IOException {

        // New block device to create from snapshot
        // containing MongoDB executables.
        EbsBlockDevice blockDevice =
                new EbsBlockDevice()
                        .withSnapshotId(SNAPSHOT_ID)
                        .withDeleteOnTermination(true);
        // Mapping for the new block device to map it
        // to /dev/sdf
        BlockDeviceMapping blockDeviceMapping =
                new BlockDeviceMapping()
                        .withDeviceName(DEVICE_NAME)
                        .withEbs(blockDevice);
        // Date for CloudInit to use when configuring
        // the started instance.
        String userData = readUserData();
        // Configure the request using the block device
        // mapping, user data and other parameters.
        RunInstancesRequest runRequest =
                new RunInstancesRequest(IMAGE_ID, 2, 2)
                        .withSecurityGroups(SECURITY_GROUP)
                        .withKeyName(KEY_NAME)
                        .withInstanceType(INSTANCE_TYPE)
                        .withPlacement(new Placement(AVAILABILITY_ZONE))
                        .withBlockDeviceMappings(blockDeviceMapping)
                        .withUserData(userData);

        // Ask AWS to start the instances.
        RunInstancesResult runResult = ec2.runInstances(runRequest);

        // Wait for them to start.
        List instances =
                waitForInstancesToStart(runResult.getReservation()
                        .getInstances());

        System.out.println("Started instances.");

        return instances;
    }

    /**
     * Read the CloudInit configuration data from a file and return it.
     *
     * @return The config data, base64 encoded.
     * @throws IOException If there are problems.
     */
    private String readUserData() throws IOException {
        // Make sure the file exists.
        File dataFile = new File(LOCAL_USER_DATA_FILE);
        if (!dataFile.exists()) {
            throw new IllegalStateException("Can not find ec2 init " +
                    "data file " + dataFile.getPath());
        }
        // Read and convert to base64
        Base64InputStream inputStream =
                new Base64InputStream(new FileInputStream(dataFile),
                        true);
        String userData =
                IOUtils.toString(inputStream);
        inputStream.close();
        return userData;
    }

    /**
     * Wait until all of the instances in the list report their state
     * as running, or 1.5 minutes have elapsed. If they aren't all
     * running after 1.5 minutes and IllegalStateException is thrown.
     *
     * @param instances The instances to check.
     * @return The instances, all in the running state.
     */
    private List waitForInstancesToStart(List instances) {
        // Number of times we've checked
        int count = 0;
        // Start by checking, just in case.
        boolean allRunning = checkIfAllRunning(instances);
        while (!allRunning && count < 6) {
            // Wait 15 seconds.
            waitABit(15, "for instances to start.");
            // Get the status of the instances by providing
            // a list of their ids.
            DescribeInstancesRequest describeRequest =
                    new DescribeInstancesRequest()
                            .withInstanceIds(extractIds(instances));
            DescribeInstancesResult describeResult =
                    ec2.describeInstances(describeRequest);
            // We know there will be a single reservation.
            instances = describeResult.getReservations()
                    .get(0).getInstances();
            allRunning = checkIfAllRunning(instances);
            ++count;
        }

        if (!allRunning) {
            throw new IllegalStateException("All instances did not start.");
        }
        return instances;
    }

    /**
     * Wait some number of seconds.
     *
     * @param timeInSeconds Number of seconds to wait.
     * @param reason The reason we're waiting.
     */
    private void waitABit(long timeInSeconds, String reason) {
        System.out.println("Waiting " + timeInSeconds + " seconds " +
                reason);
        synchronized (this) {
            try {
                wait(timeInSeconds * 1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }

    /**
     * From a list of instances extract a list of their ids.
     *
     * @param instances The instances.
     * @return The ids.
     */
    private List extractIds(List instances) {
        List instanceIds =
                new ArrayList(instances.size());
        for (Instance instance : instances) {
            instanceIds.add(instance.getInstanceId());
        }
        return instanceIds;
    }

    /**
     * If all of the instances in the list have a state of "running"
     * return true, otherwise return false.
     *
     * @param instances The instances to check.
     * @return True or false.
     */
    private boolean checkIfAllRunning(List instances) {
        System.out.println("Checking that all instances are running.");
        boolean allRunning;
        allRunning = true;
        for (Instance instance : instances) {
            System.out.println("Instance " + instance.getInstanceId() +
                    " is " + instance.getState().getName() + ".");
            if (!"running".equals(instance.getState().getName())) {
                allRunning = false;
            }
        }
        return allRunning;
    }

    /**
     * Start a MongoDB server on first machine telling it that it's part
     * of a replica set that includes second machine. Communication
     * between the members of the replica set is done via their private
     * IP addresses.
     *
     * @param firstMachinePublicIp Public IP of the first machine.
     * @param secondMachinePrivateIp Private IP of the second machine.
     * @throws IOException If errors occur.
     */
    private void startMongoDb(String firstMachinePublicIp,
                              String secondMachinePrivateIp)
            throws IOException {
        // Need to create an address to log on to the first
        // machine.
        String firstMachineAddress = "ec2-user@" +
                firstMachinePublicIp;
        System.out.println("Going to start MongoDB as " +
                firstMachineAddress);

        // Variables to collect results starting MongoDB.
        final String[] processNumber = new String[] {null};
        final StringBuilder sb = new StringBuilder();

        // Number of times we've tried to start MongoDB.
        int count = 0;

        // It may take a while for the instance to be fully initialized
        // and reachable via ssh so try several times.
        while (processNumber[0] == null && count < 4) {
            // Use SSH to issue the command to start MongoDB on
            // the remote machine. Look at the output and if
            // we see "forked process: " assume success.
            sshCommandExecutor.execute(firstMachineAddress,
                    REMOTE_MONGO_DB_START_CMD + secondMachinePrivateIp,
                    new SshCommandExecutor.OutputHandler() {
                        public void handle(String line) {
                            sb.append(line);
                            sb.append('\n');
                            if (line.startsWith("forked process: ")) {
                                processNumber[0] =
                                        line.substring("forked process: ".length()).trim();
                            }
                        }
                    });
            ++count;
            if (processNumber[0] == null) {
                waitABit(30, "for instance to be reachable via ssh.");
            }
        }

        // If we didn't find a process then print out all of
        // the output from SSH and throw an exception.
        if (processNumber[0] == null) {
            System.out.println("Response from starting MongoDB was:");
            System.out.println(sb.toString());
            throw new IllegalStateException("Could not start MongoDB as " +
                    firstMachineAddress);
        }

        System.out.println("MongoDB started process " + processNumber[0] +
                " as " + firstMachineAddress);
    }

    /**
     * Start a MongoDB replica set by contacting the admin database on machine
     * one.
     *
     * @param firstMachinePublicIp Public IP of first machine.
     * @param firstMachinePrivateIp Private IP of first machine.
     * @param secondMachinePrivateIp Private IP of second machine.
     * @throws IOException In the event of error.
     */
    private void startReplicaSet(String firstMachinePublicIp,
                                 String firstMachinePrivateIp,
                                 String secondMachinePrivateIp)
            throws IOException {

        // Substitute the two private ip addresses into the command
        // to start the replica set.
        String cmd =
                MessageFormat.format(START_REPLICASET_CMD,
                        firstMachinePrivateIp, secondMachinePrivateIp);

        // Write the start command into a file with the
        // correct extension.
        File jsFile = File.createTempFile("replicaset", ".js");
        jsFile.deleteOnExit();
        FileUtils.writeStringToFile(jsFile, cmd);

        // Run the command and write out the results.
        ProcessBuilder pb = new ProcessBuilder(
                LOCAL_MONGO_EXECUTABLE,
                firstMachinePublicIp + "/admin",
                jsFile.getPath()
        );
        pb.redirectErrorStream(true);

        Process p = pb.start();

        BufferedReader rdr =
                new BufferedReader(new InputStreamReader(p.getInputStream()));
        try {
            String line;
            while ((line = rdr.readLine()) != null) {
                System.out.println(line);
            }
        } finally {
            rdr.close();
        }
    }
}

This describes how to run MongoDB on Amazon Web Services, in particular using Amazon Elastic Compute Cloud (EC2) to support a MongoDB replica set.

Choice of machine image

A 64bit machine is needed if a mongod is to manage more than 2 GB of data. It will also need medium or high IO performance which indicates a Large instance as the minimum realistic starting point. The 850 GB of instance storage would be used for the MongoDB data storage. This will cost $0.34 per hour (pricing for US East region) so a replica set with two machines (one master and one slave) will cost $0.68 an hour or about $5,590 a year. That’s the on-demand pricing, because we want to run 24×7 reserved instances may be cheaper. Using reserved instances with a one year term in the same region is $0.12 per hour with a $910 down payment each giving a total of about $3,922.

Location

For reliability in the event of the failure of an availability zone there should be a second slave in a different availability zone in the same region or even in a different region. This will incur data transfer charges between the master and the second slave, much cheaper ($0.01 per GB transferred) between availability zones in the same region than between regions ($0.10 per GB transferred in plus between from $0.15 per GB transferred out).
If the load is low it would be nice to be able to use the machines for other purposes as well, for instance primary web server with slave MongoDB server and backup web server with master MongoDB server.

Experimental

To start with I want to experiment with using Micro instances, even though these have slower EBS storage, to see what the throughput is.

Cookbook

This describes the actual steps I took to get a MongoDB replica set working using manual commands. The next post will build on this to show how to automate the process of starting and stopping a replica set. You have to have automation to work successfully with cloud services.

Security Group

Create a security group to restrict connections into the instances running MongoDB. Both master and slave instances will use this same group because a slave could need to take over from a master so no distinction should exist between them.

#
ec2-authorize MongoDB -P tcp -p 22
ec2-authorize MongoDB -P tcp -p 27017 -u 844613644011 -o MongoDB
ec2-authorize MongoDB -P tcp -p 27017 -u 844613644011 -o AppServer
ec2-authorize MongoDB -P tcp -p 27017 -s 24.153.207.123/32

Install MongoDB

Create a volume for the MongoDB executables. This isn’t going to hold the data so it doesn’t have to be very big. Create the smallest one we can, which is 1GB. Record the volume it.

#
ec2-create-volume --size 1 --availability-zone us-east-1a

Start a micro instance in same availability zone. KeyPair20110224 is the key I’m going to use to log on with SSH later.

#
ec2-run-instances ami-74f0061d -g MongoDB -k KeyPair20110224 -t t1.micro --availability-zone us-east-1a

Wait till it’s running.

#
ec2-describe-instances

Attach the new volume to the instance. Instance id from the output of ec2-describe-instances above, volume id from when the volume was created. You could also use ec2-describe-volumes.

#
ec2-attach-volume vol-77815b1c --instance i-73c2221d --device /dev/sdf

Logon to the instance. The address comes from ec2-describe-instances output.

#
ssh -i keys/KeyPair20110224.pem ec2-user@ec2-174-129-155-243.compute-1.amazonaws.com

Format the volume and mount it.

#
sudo mkfs -t ext3 /dev/sdf
sudo mkdir /mongodb
sudo mount /dev/sdf /mongodb

Download and unpack mongodb package onto the volume we mounted. Using the 64 bit linux package.

#
curl http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-1.8.1.tgz > mongodb.tgz
sudo mv mongodb.tgz /mongodb/
cd /mongodb
sudo tar tzf mongodb.tgz
sudo rm mongodb.tgz

Create a data directory under the ec2-user home directory.

#
cd
mkdir -p data/db

Start mongod.

#
cd /mongodb/mongodb-linux-x86_64-1.8.1
bin/mongod --dbpath /home/ec2-user/data/db --logpath /home/ec2-user/mongodb.log --fork

Test that mongod is working and can be reached locally using the shell. Start mongo and try some commands.

#
bin/mongo
MongoDB shell version: 1.8.1
connecting to: test
> db
test
> post = {"title" : "My Test Post", "content" : "Some stuff I wrote"}
{ "title" : "My Test Post", "content" : "Some stuff I wrote" }
> db.blog.insert(post)
> db.blog.find()
{ "_id" : ObjectId("4dcc9ed4052e568e1f58cb00"), "title" : "My Test Post", "content" : "Some stuff I wrote" }
> exit
bye

Stop mongod. Look in the log to get the process id or record it when the instance starts.

#
head /home/ec2-user/mongodb.log
sudo kill -2 1032
tail /home/ec2-user/mongodb.log

Create a snapshot

Unmount the EBS volume. This is so we can take a snapshot to use to create volumes when running instances.

#
cd
sudo umount -d /mongodb

Logoff the instance and detach the volume. Use ec2-describe-volumes to check that the detach has completed.

#
ec2-detach-volume vol-77815b1c
ec2-describe-volumes vol-77815b1c

Terminate the instance.

#
ec2-terminate-instances i-73c2221d

Take a snapshot of the volume. This is so we can launch other instances using the snapshot to create and attach a volume.

#
ec2-create-snapshot vol-77815b1c -d 'MongoDB installation'

Test the snapshot

Create a CloudInit configuration file called MongoDBInit.txt to start an instance, contents are shown below. For this experiment I’ll use the space on the default drive associated with the instance to store the data. This configuration mounts the volume with the mongodb executables, creates the data directory and starts mongod.

#cloud-config
mounts:
 - [ /dev/sdf, /mongodb, “auto”, “defaults”, “0”, “2” ]
runcmd:
 - cd /home/ec2-user
 - mkdir -p data/db
 - cd /mongodb/mongodb-linux-x86_64-1.8.1
 - bin/mongod --dbpath /home/ec2-user/data/db --logpath /home/ec2-user/mongodb.log --nohttpinterface --fork

Start an ec2 micro instance using the snapshot and CloudInit configuration file. This is to let us test that the automatic mounting works and the installed mongod will run successfully.

#
ec2-run-instances ami-74f0061d -g MongoDB -k KeyPair20110224 -t t1.micro --availability-zone us-east-1a -b "/dev/sdf=snap-a9c5e0c6::true" --user-data-file MongoDBInit.txt

Wait till the new instance is running

#
ec2-describe-instances

Check that it’s possible to connect to mongod on the new instance using the mongo shell from your local machine. Address of the machine comes from ec2-describe-instances.

#
mongodb-osx-x86_64-1.8.1/bin/mongo ec2-67-202-13-208.compute-1.amazonaws.com

Terminate the instance.

#
ec2-terminate-instance i-8fc02be1

Modify CloudInit configuration to support replica sets

To create a replica set we need to know the ip addresses of the machines in the set so we’ll have to start the instances and then use ssh to start mongod. The commands run by ssh will be run as ec2-user so the directories that MongoDBInit.txt creates will have to have their owner changed. Modify MongoDB.init.txt by removing the last two lines so that mongod isn’t automatically started and adding “chown -R ec2-user data” as the last line.

Start another instance with the new MongoDBInit.txt

#
ec2-run-instances ami-74f0061d -g MongoDB -k KeyPair20110224 -t t1.micro --availability-zone us-east-1a -b "/dev/sdf=snap-a9c5e0c6::true" --user-data-file MongoDBInit.txt

Use SSH to start mongod.

#
ssh -i keys/KeyPair20110224.pem ec2-user@ec2-204-236-247-165.compute-1.amazonaws.com "/mongodb/mongodb-linux-x86_64-1.8.1/bin/mongod --dbpath /home/ec2-user/data/db --logpath /home/ec2-user/mongodb.log --nohttpinterface --fork"

Test that mongod is running by reaching it from the admin machine.

#
mongodb-osx-x86_64-1.8.1/bin/mongo ec2-204-236-247-165.compute-1.amazonaws.com

Shutdown mongod using ssh and the process id reported when it was started with ssh.

#
ssh -i keys/KeyPari20110224.pem ec2-user@ec2-204-236-247-165.compute-1.amazonaws.com “kill 1049”

Terminate the instance.

#
ec2-terminate-instances i-8fc02be1

Starting a replica set.

Start two instances to host the two servers of the replica set.

#
ec2-run-instances -n 2 ami-74f0061d -g MongoDB -k KeyPair20110224 -t t1.micro --availability-zone us-east-1a -b "/dev/sdf=snap-a9c5e0c6::true" --user-data-file MongoDBInit.txt

Start mongodb on both machines. Use the internal ip addresses to communicate between the instances. So the first command tells machine one to talk to machine two and the second tells two to talk to one. The replica set is called logSet.

#
ssh -i keys/KeyPair20110224.pem ec2-user@ec2-50-17-110-242.compute-1.amazonaws.com "/mongodb/mongodb-linux-x86_64-1.8.1/bin/mongod --dbpath /home/ec2-user/data/db --logpath /home/ec2-user/mongodb.log --nohttpinterface --fork --replSet logSet/domU-12-31-39-04-0C-AF.compute-1.internal"

ssh -i keys/KeyPair20110224.pem ec2-user@ec2-50-19-70-14.compute-1.amazonaws.com "/mongodb/mongodb-linux-x86_64-1.8.1/bin/mongod --dbpath /home/ec2-user/data/db --logpath /home/ec2-user/mongodb.log --nohttpinterface --fork --replSet logSet/domU-12-31-39-09-88-8B.compute-1.internal"

Use mongo to configure the repl set. Until this is done the replica set isn’t created. Connect to one of the two instances. The internal ip addresses of the two instances are used in the configuration. It takes a short while before the set is configured and the response appears.

#
mongodb-osx-x86_64-1.8.1/bin/mongo ec2-50-19-70-14.compute-1.amazonaws.com/admin
MongoDB shell version: 1.8.1
connecting to: ec2-50-19-70-14.compute-1.amazonaws.com/admin
> db.runCommand({"replSetInitiate" : {
... "_id" : "logSet",
... "members" : [
... {
... "_id" : 1,
... "host" : "domU-12-31-39-04-0C-AF.compute-1.internal"
... },
... {
... "_id" : 2,
... "host" : "domU-12-31-39-09-88-8B.compute-1.internal"
... }
... ]}})
{
	"info" : "Config now saved locally.  Should come online in about a minute.",
	"ok" : 1
}

Look at the mongodb.log files on the two machines. You can see which has become the primary and which the secondary. Also, when you connect to an instance you can see if it is the primary or not.

#
mongodb-osx-x86_64-1.8.1/bin/mongo ec2-50-19-70-14.compute-1.amazonaws.com/adminMongoDB shell version: 1.8.1
connecting to: ec2-50-19-70-14.compute-1.amazonaws.com/admin
logSet:PRIMARY>

Now shutdown the slave and then the master using the process numbers we recorded when we started them.

#
ssh -i keys/KeyPair20110224.pem ec2-user@ec2-50-17-110-242.compute-1.amazonaws.com "kill 1044"
ssh -i keys/KeyPair20110224.pem ec2-user@ec2-50-19-70-14.compute-1.amazonaws.com "kill 1065"

Terminate the instances

#
ec2-terminate-instances i-bb7780d5 i-b97780d7

Clearly executing all of these commands manually isn’t going to work for more than this initial setup, it’s just too complex. The next step is to try and automate the process of bringing up and shutting down a replica set.