codem - blog

Posts Tagged ‘mysql’

MySQL – dealing correctly with date diffs & timezones

A recent project had us dealing with the differences in whole days between two dates stored in a MySQL table. As a first effort, you may turn to the trusty DATEFIFF() function to return the interval. This works well when the dates you have to diff are in the same timezone as the dates you want to display but breaks down when, for instance, your dates are stored in UTC (as they should be) and are displayed in a local timezone.

Take this case of a datetime stored as UTC in a table – ’2009-10-24 23:00:00′ compared against the current UTC_TIMESTAMP() of, say, ’2009-10-23 02:00:00′.
A simple DATEFIFF() will return 1 as the day difference between the two datetimes -which is correct.

Now let’s take a look at those dates when displayed, for instance as UTC + 10. The first datetime becomes ’2009-10-25 09:00:00′ and the second becomes ’2009-10-23 12:00:00′.  If we’re displaying those local datetimes as dates on a website along with the day difference, we’ll get 23/10/2009 – 25/10/2009 (1 day)  – as an example, which is  plainly wrong. We could do the diff in scriptland but then MySQL comes with excellent date mathematics functions already.

The issue is that DATEDIFF ignores the time part of the timestamp, only working on the date part.

MySQL provides another method called TIMEDIFF, returning a time value of the datetime difference. In this example, the time value returned is 45:00:00 (i.e 45 hours). This is great, as you can now write a query something like

SELECT ROUND(EXTRACT(HOUR FROM TIMEDIFF('2009-10-24 23:00:00', '2009-10-23 02:00:00')) / 24)

Which is a bit of a mess, but gives the required result of 2 days. Solved? yes, until you hit something like the following:

SELECT ROUND(EXTRACT(HOUR FROM TIMEDIFF('2009-10-23 02:00:00', '2008-05-29 23:00:00'))/ 24)

… and suddenly the day difference becomes 35! The problem here is that “TIME values may range from '-838:59:59' to '838:59:59'" (quoting the MySQL TIME documentation), which happens to be about 35 days.

You may tear your hair out here unless you look at the TIMESTAMPDIFF function, which will do exactly what you want, regardless of timezone:

SELECT ROUND(TIMESTAMPDIFF(HOUR, '2009-10-23 02:00:00', '2009-10-24 23:00:00') / 24)

Noting that the query uses HOUR as the interval in order to get a fractional day that can be rounded to the nearest integer day after division.

Building mongodb on Ubuntu

I’ve been looking at mongodb as a solution for some projects. It looks like a decent solution that sits between a standard RDBMS like MySQL or Postgres and schema-less solutions like CouchDB. Result: I need to do some testing…

As the mongodb website says: A key goal of MongoDB is to bridge the gap between key/value stores (which are fast and highly scalable) and traditional RDBMS systems (which are deep in functionality).

It’s also the name of the planet in Flash Gordon, the big guy’s name in Blazing Saddles and the name of the giant Gingerbread man in Shrek, in case you were wondering.
According to the mongodb site, it draws it’s name from “humongous” – although I always thought that was spelt humungous but then we’d have ended up with MungoDB which doesn’t have the same pop culture links. Digression.

Update for 10.10

For users of Ubuntu + derivatives 10.10 and up, mongodb is available via APT, the Ubuntu package management system.
Simply:

sudo aptitude install mongodb

Which will install the core and any dependencies, along with setting up a mongodb user and providing an upstart job to manage mongod. To stop, start and restart mongodb, issue the commands sudo service mongodb stop|start|restart or alternatively sudo stop|start|restart mongodb.
On Ubuntu, the mongodb config file lives at /etc/mongodb.conf. Unfortunately it doesn’t look like we have a conf.d system to allow core config overrides just yet.

For those using older versions of Ubuntu, read on. If you have 10.10+ installed, then skip to the Install a PHP Driver part below.

10.04 and earlier…

I do most testing on a Kubuntu (9.04) system, so the instructions for building on Ubuntu can be used. (Kubuntu is just Ubuntu with KDE as the desktop environment).

Here’s the process (with some changes based on some problems I encountered):

Install the related packages

You can substitute aptitude for apt-get if you want. I  prefer aptitude as it covers all the APT commands.

sudo aptitude install tcsh git-core scons g++
sudo aptitude install libpcre++-dev libboost1.37-dev libreadline-dev

There are two changes here from the instructions:

  1. libmozjs-dev in Ubuntu 9.04 conflicts with xulrunner-1.9. This means the package manager will try to remove anything that depends on xulrunner-1.9, namely Firefox 3.5. In my case aptitude tried to downgrade Firefox to 3.5b4. To work around this install it from source (see below). This won’t be a problem if you don’t need xulrunner-1.9 or its dependencies (or if you aren’t installing in a desktop environment that uses Firefox 3.5)
  2. You will need libboost1.37-dev rather than libboost-dev, otherwise the mongodb install process will complain about an old version of libboost.

Finally, I’m not sure why tcsh is needed given it’s just a shell and I have bash installed – maybe scons needs it?

SpiderMonkey

This process will install SpiderMonkey in place of the libmozjs-dev packages, which MongoDB uses to parse data reduction functions. The MongoDB site provides the background to this.
I’m installing in a tmp directory in my home directory

cd ~
mkdir tmp && cd tmp
wget ftp://ftp.mozilla.org/pub/mozilla.org/js/js-1.7.0.tar.gz
tar -zxvf js-1.7.0.tar.gz
cd js/src
export CFLAGS="-DJS_C_STRINGS_ARE_UTF8"
make -f Makefile.ref
sudo JS_DIST=/usr make -f Makefile.ref export

Download and install MongoDB

Finally, get MongoDB using Git and install it using scons:

cd ~/tmp
git clone git://github.com/mongodb/mongo.git
cd mongo
scons all
sudo scons --prefix=/opt/mongo install

If all goes well, MongoDB should install to /opt/mongo. In /opt/mongo/bin you’ll see a mongod executable, which when executed starts the mongo daemon.

Create a location to store the mongodb database(s). I’m just testing so I created it in my home directory. You can create it anywhere you want, providing the user starting mongod can read and write to it.

mkdir -p ~/testing/mongo/data/db

Start mongo, as your user

/opt/mongo/bin/mongod --dbpath ~/testing/mongo/data/db

Mongodb should spit out some startup information, telling you there is a web interface listening on port 28017 (browse to http://localhost:28017, or the IP of the system, for the results). Hit Ctrl-c to kill mongod when done.

Help and keeping up-to-date

To get some basic mongod help:

/opt/mongo/bin/mongod --help

To keep up to date, switch to your mongo source directory (~/tmp/mongo in my case) and:

git pull

Then rebuild.

Install a PHP driver

The PHP Mongo driver can be installed using PECL:

sudo pecl install mongo

and follow the instructions.

Now for some testing!