Thursday, May 28, 2020

How To Install WiredTiger Command Line Tool To Fix MongoDB Collections

If you use MongoDB database and have had issue with corrupted WiredTiger collection (table) you may need to download and install WiredTiger command line tool to repair / salvage / verify etc... your MongoDB Collection file.

This guide will help you install 'wt' (wiredtiger) command line utility tool for using it with MongoDB database.

STEP 1 - UPDATE YOUR APT

sudo apt-get update

STEP 2 - DOWNLOAD SOURCE FILE + EXTRACT IT

wget http://source.wiredtiger.com/releases/wiredtiger-2.7.0.tar.bz2
tar xvf wiredtiger-2.7.0.tar.bz2

STEP 3 - INSTALL GOOGLE SNAPPY (LIBRARY FILES)

apt-get install libsnappy-dev build-essential

STEP 4 - COMPILE AND MAKE 'wt' 

./configure --enable-snappy && make


STEP 5 - TEST 'wt'

after you are done with all the steps, you should be able to just execute 'wt' by typing the following command and should see results similar to below:

root@testserver:/usr/local/src/wiredtiger-2.7.0# ./wt
WiredTiger Data Engine (version 2.7) global options:
 -C wiredtiger_open configuration
 -h database directory
 -L turn logging off for debug-mode
 -R run recovery if configured
 -V display library version and exit
 -v verbose
commands:
 backup   database backup
 compact   compact an object
 copyright copyright information
 create   create an object
 drop   drop an object
 dump   dump an object
 list   list database objects
 load   load an object
 loadtext   load an object from a text file
 printlog  display the database log
 read   read values from an object
 rename   rename an object
 salvage   salvage a file
 stat   display statistics for an object
 upgrade   upgrade an object
 verify   verify an object
 write   write values to an object





[SOLVED] MongoDB will not start - invariant failure - database has corrupted collection files

One of my MongoDB server suddenly failed to start after abnormal / abrupt shutdown (lost power abruptly too many times).

When it tried to start I noticed errors like these in the log.

"invariant failure"

Then I tried to repair the database using the following command (it does not really help - but provide more clues what is wrong):

(your dbpath may be different)

mongod --repair --dbpath=/data_local/mongodb

The above command attempts to repair many bad indexes, collections, etc... but eventually it stopped at repairing one collection with an error message like this:

MongoDB wiredTiger Error: collection-<blah-blah>.wt does not appear to be a WiredTiger file

So, one of my collection file has been corrupted. My MongoDb version is 3.2 and it does not have a 'wt' tool (wiredtiger command line utility to repair, salvage, etc... wiredtiger files).  I have to install 'wt' tool from wiredtiger.com.  Please follow my guide How To Install WiredTiger Command Line Tool To Fix MongoDB Collections.

After I have 'wt' command line tool installed, I then proceed to try to fix (salvage) my collection file. Before I do that I prepare a working/salvage directory to provide a separate directory (I used /data_local/mongo_salvage).  I put my collection file (*.wt) and a few other needed files there:

(note my collection file name is collection-1217--8628481310102098565.wt, change to whatever your collection file name is)

-rw-r--r-- 1 root      root      4738772992 Feb  9 14:06 collection-1217--8628481310102098565.wt
-rw-r--r-- 1 root      root         1155072 Feb  9 14:05 _mdb_catalog.wt
-rw-r--r-- 1 root      root        26935296 Feb  9 14:05 sizeStorer.wt
-rw-r--r-- 1 root      root              95 Feb  9 14:05 storage.bson
-rw-r--r-- 1 root      root              46 Feb  9 14:04 WiredTiger
-rw-r--r-- 1 root      root              21 Feb  9 14:04 WiredTiger.lock
-rw-r--r-- 1 root      root             916 Feb  9 14:04 WiredTiger.turtle
-rw-r--r-- 1 root      root        10436608 Feb  9 14:04 WiredTiger.wt


Then I go to that working salvage directory and tried to salvage the collection file:

(note my collection file name is collection-1217--8628481310102098565.wt, change to whatever your collection file name is)


cd /data_local/mongo_salvage
./wt -v -h /data_local/mongo_salvage -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage -F collection-1217--8628481310102098565.wt

The 'wt' tool did not return any error, which mean it worked.

Now, all I have to do is copy it (overwrite) my previously (original) corrupted collection file. BUT before I do that, lets be safe and make a copy of that corrupted file first.

cp /data_local/mongodb/collection-1217--8628481310102098565.wt /data_local/mongo_salvage/collection-1217--8628481310102098565-ORIGINAL-CORRUPTED.wt

Finally we are ready to overwrite it with the good salvaged version:

cp /data_local/mongo_salvage/collection-1217--8628481310102098565.wt /data_local/mongodb/

Then, we re-execute the same repair command:

mongod --repair --dbpath=/data_local/mongodb


In my case, I had 1 additional collection that was also corrupted. But I was relieved to see the above procedure worked (made progress - woohoo!).  So, I just repeat the process again for another corrupted collection file.

SUCCESS!  finally after salvaging 2 collection files and copying them over to overwrite the original corrupted .wt file, I was able to fully run the mongod --repair without it borking out midway. It executed all the way through and I was able to start my MongoDB system!

I hope this instruction is helpful for somebody. If so, please kindly let me know via comment.


Tuesday, February 7, 2017

Installing MongoDB 3.x on Ubuntu Xenial 16.04 LTS - using official MongoDB repository

This is a quick and dirty step by step guide (just list of commands) to install MongoDB into Ubuntu Xenial 16.04 LTS.

Pre-requisites:


apt install software-properties-common

apt-get update




Add MongoDB repository:


apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
apt-get update




Install MongoDB Server Files:


apt-get install -y mongodb-org


Setup start-up script:


nano /etc/systemd/system/mongodb.service

Insert the following lines of content:

[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target

[Service]
User=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf

[Install]
WantedBy=multi-user.target


Enable start-up script:


systemctl enable mongodb


DONE - how to use your new MongoDB:

Configuration file is located here:
/etc/mongodb.conf

To Start service:
systemctl start mongodb

To Stop service:
systemctl stop mongodb

To Check Status of service:
systemctl status mongodb



MongoDB will start automatically on your next server boot process.


Enjoy! -  if you experience any issue - pls leave me a comment. Thanks

Thursday, August 25, 2016

Ubuntu init.d start-up script for MongoDB Community Edition 3.x

I have just installed MongoDB community edition via apt. It installed without issue, but did not install the start-up script in /etc/init.d for some reason. So I had to install it manually:

create the following content in the file /etc/init.d/mongod

----------------------------------

#!/bin/sh
#
# init.d script with LSB support.
#
# Copyright (c) 2007 Javier Fernandez-Sanguino <jfs@debian.org>
#
# This is free software; you may redistribute it and/or modify
# it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2,
# or (at your option) any later version.
#
# This is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License with
# the Debian operating system, in /usr/share/common-licenses/GPL;  if
# not, write to the Free Software Foundation, Inc., 59 Temple Place,
# Suite 330, Boston, MA 02111-1307 USA
#
### BEGIN INIT INFO
# Provides:          mongod
# Required-Start:    $network $local_fs $remote_fs
# Required-Stop:     $network $local_fs $remote_fs
# Should-Start:      $named
# Should-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: An object/document-oriented database
# Description:       MongoDB is a high-performance, open source, schema-free
#                    document-oriented data store that's easy to deploy, manage
#                    and use. It's network accessible, written in C++ and offers
#                    the following features:
#
#                       * Collection oriented storage - easy storage of object-
#                         style data
#                       * Full index support, including on inner objects
#                       * Query profiling
#                       * Replication and fail-over support
#                       * Efficient storage of binary data including large
#                         objects (e.g. videos)
#                       * Automatic partitioning for cloud-level scalability
#
#                    High performance, scalability, and reasonable depth of
#                    functionality are the goals for the project.
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/mongod
DESC=database

NAME=mongod
# Defaults.  Can be overridden by the /etc/default/$NAME
# Other configuration options are located in $CONF file. See here for more:
# http://dochub.mongodb.org/core/configurationoptions
CONF=/etc/mongod.conf
PIDFILE=/var/run/$NAME.pid
ENABLE_MONGOD=yes

# Include mongodb defaults if available.
# All variables set before this point can be overridden by users, by
# setting them directly in the defaults file. Use this to explicitly
# override these values, at your own risk.
if [ -f /etc/default/$NAME ] ; then
        . /etc/default/$NAME
fi

# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
    NUMACTL="`which numactl` -- $NUMACTL_ARGS"
    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}
else
    NUMACTL=""
    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}
fi


if test ! -x $DAEMON; then
    echo "Could not find $DAEMON"
    exit 0
fi

if test "x$ENABLE_MONGOD" != "xyes"; then
    exit 0
fi

. /lib/lsb/init-functions

STARTTIME=1
DIETIME=10                  # Time to wait for the server to die, in seconds
                            # If this value is set too low you might not
                            # let some servers to die gracefully and
                            # 'restart' will not work

DAEMONUSER=${DAEMONUSER:-mongodb}
DAEMONGROUP=${DAEMONGROUP:-mongodb}

set -e

running_pid() {
# Check if a given process pid's cmdline matches a given name
    pid=$1
    name=$2
    [ -z "$pid" ] && return 1
    [ ! -d /proc/$pid ] &&  return 1
    cmd=`cat /proc/$pid/cmdline | tr "\000" "\n"|head -n 1 |cut -d : -f 1`
    # Is this the expected server
    [ "$cmd" != "$name" ] &&  return 1
    return 0
}

running() {
# Check if the process is running looking at /proc
# (works for all users)

    # No pidfile, probably no daemon present
    [ ! -f "$PIDFILE" ] && return 1
    pid=`cat $PIDFILE`
    running_pid $pid $DAEMON || return 1
    return 0
}

start_server() {
            # Recommended ulimit values for mongod or mongos
            # See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
            #
            ulimit -f unlimited
            ulimit -t unlimited
            ulimit -v unlimited
            ulimit -n 64000
            ulimit -m unlimited

            # In dash, ulimit takes -p for maximum user processes
            # In bash, it's -u
            if readlink /proc/$$/exe | grep -q dash
            then
                    ulimit -p 64000
            else
                    ulimit -u 64000
            fi

            # Start the process using the wrapper
            start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
                        --make-pidfile --chuid $DAEMONUSER:$DAEMONGROUP \
                        --exec $NUMACTL $DAEMON $DAEMON_OPTS
            errcode=$?
        return $errcode
}

stop_server() {
# Stop the process using the wrapper
            start-stop-daemon --stop --quiet --pidfile $PIDFILE \
                        --retry 300 \
                        --user $DAEMONUSER \
                        --exec $DAEMON
            errcode=$?
        return $errcode
}

force_stop() {
# Force the process to die killing it manually
        [ ! -e "$PIDFILE" ] && return
        if running ; then
                kill -15 $pid
        # Is it really dead?
                sleep "$DIETIME"s
                if running ; then
                        kill -9 $pid
                        sleep "$DIETIME"s
                        if running ; then
                                echo "Cannot kill $NAME (pid=$pid)!"
                                exit 1
                        fi
                fi
        fi
        rm -f $PIDFILE
}


case "$1" in
  start)
        log_daemon_msg "Starting $DESC" "$NAME"
        # Check if it's running first
        if running ;  then
            log_progress_msg "apparently already running"
            log_end_msg 0
            exit 0
        fi
        if start_server ; then
            # NOTE: Some servers might die some time after they start,
            # this code will detect this issue if STARTTIME is set
            # to a reasonable value
            [ -n "$STARTTIME" ] && sleep $STARTTIME # Wait some time
            if  running ;  then
                # It's ok, the server started and is running
                log_end_msg 0
            else
                # It is not running after we did start
                log_end_msg 1
            fi
        else
            # Either we could not start it
            log_end_msg 1
        fi
        ;;
  stop)
        log_daemon_msg "Stopping $DESC" "$NAME"
        if running ; then
            # Only stop the server if we see it running
                        errcode=0
            stop_server || errcode=$?
            log_end_msg $errcode
        else
            # If it's not running don't do anything
            log_progress_msg "apparently not running"
            log_end_msg 0
            exit 0
        fi
        ;;
  force-stop)
        # First try to stop gracefully the program
        $0 stop
        if running; then
            # If it's still running try to kill it more forcefully
            log_daemon_msg "Stopping (force) $DESC" "$NAME"
                        errcode=0
            force_stop || errcode=$?
            log_end_msg $errcode
        fi
        ;;
  restart|force-reload)
        log_daemon_msg "Restarting $DESC" "$NAME"
                errcode=0
        stop_server || errcode=$?
        # Wait some sensible amount, some server need this
        [ -n "$DIETIME" ] && sleep $DIETIME
        start_server || errcode=$?
        [ -n "$STARTTIME" ] && sleep $STARTTIME
        running || errcode=$?
        log_end_msg $errcode
        ;;
  status)

        log_daemon_msg "Checking status of $DESC" "$NAME"
        if running ;  then
            log_progress_msg "running"
            log_end_msg 0
        else
            log_progress_msg "apparently not running"
            log_end_msg 1
            exit 1
        fi
        ;;
  # MongoDB can't reload its configuration.
  reload)
        log_warning_msg "Reloading $NAME daemon: not implemented, as the daemon"
        log_warning_msg "cannot re-read the config file (use restart)."
        ;;

  *)
        N=/etc/init.d/$NAME
        echo "Usage: $N {start|stop|force-stop|restart|force-reload|status}" >&2
        exit 1
        ;;
esac

exit 0

--------------------------------

then you can start it using:

service mongod start

or

/etc/init.d/mongod start

Tuesday, July 5, 2016

Fatal error: exception 'Error' with message 'Invalid class method "createIndex"' in .... .php

I found this error this morning after upgrading my PHP MongoDB extension from 1.5.x to 1.6.x.

Fatal error: exception 'Error' with message 'Invalid class method "createIndex"' in .... .php

This was cause by the function ensureIndex() being deprecated/remove in favor of createIndex()

Took me about 30 min to figure out (a lot longer) because there was no result on Google about this, so I thought this blog article hopefully helps somebody in future.

Monday, November 23, 2015

RoboMongo - A great MongoDB GUI for Mac OS X

I have been looking for a good enough MongoDB GUI to replace RockMongo which is one of the best web based GUI I have used for many years.

MongoDB is a great No SQL database, but surprisingly I have not seen any good quality GUI for Mac OS X, the last project that came close to becoming a replacement for RockMongo was Mongo Hub.  Mongo Hub had limited functionality and the project seems to have gotten stuck (in-active).

Today, I want to introduce RoboMongo.  It has minimal GUI, but it is easy to use.



You simply just need to create connection profiles and RoboMongo supports SSH tunneling which makes it easy to use and saves time.

RoboMongo's concept is simple, it is a command / script base GUI concept, the called it shell-centric, which means you need to know some basic MongoDB javascript like commands to be able to query and perform database actions.

I have been using RoboMongo since only about 2 weeks ago.  Since then I have rarely return back to use RockMongo.

Try RoboMongo, leave a comment of how you like it or tell me if you have found something better.

Thanks

-Hadi


Saturday, August 1, 2015

Rename Database using Copy then delete method in MongoDB

Simplest way to rename a MongoDB database using MongoDB CLI:

db.copyDatabase('old_name', 'new_name');
use old_name
db.dropDatabase();