The robots are taking over!

Read on if you’ve ever wondered about the logistical challenges behind a system like MasterVision – collecting, integrating and analysing millions of rows of data every week. Sadly we don’t employ an army of Oompa-Loompas, but we do have an impressive set of software robots doing almost all of the hard work:

1. Collecting files. It sounds mundane, but it’s quite a challenge to manage a constant stream of new and updated data files – delivered at all times of the week to various different FTP sites. These sites are all ‘watched’ by our systems, so whenever an updated file appears it is automatically collected and logged within hours. On receipt, our robot also helpfully unzips and decrypts source files as necessary, so the entire process is fully automated.

2. Checking email domains. In order to validate the deliverability of email addresses, we maintain our own database of over 250k email domains (i.e. the part of an email after the @ symbol) so that we can help our clients to predict the majority of emails which will ‘bounce’. Keeping this list updated is a major ongoing task, and one of our robots does the job of re-testing around 5,000 domains every night. New domains are also automatically added as we encounter them.

3. Data cleaning and integration. Frequent data builds are run to keep our clients’ systems up-to-date, and again the robots do all of the hard work. This includes validating emails, cleaning up ‘free text’ for countries etc. into standard names, and joining different data sources into single customer records. Once finished, reports and customised lists are automatically supplied back to each client via FTP or as email attachments, providing full visibility of the process.

4. Web site monitoring. We offer a fully-managed and hosted service with a 100% service level, and in order to track this a ‘robot web browser’ tests the availability of our sites every 5 minutes around the clock, and logs the results into daily and monthly uptime reports. So far we’ve managed over a year of uptime, but in the event of any problems our robot helper will send us an SMS message so we’ll be the first to know about it, wherever we are!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s