Apollo Mission

Background

DoneDeal grew from a simple classified ads website to become one of Ireland’s hottest internet properties. DoneDeal is Ireland’s biggest and most successful classifieds website and today claims a market leadership position four times as big as its nearest competitor.

From a standing start in 2005, the website has experienced an average of 10-15pc growth in visitor traffic and ad placements per month. By 2014, company turnover reached over €8 million and active ads totalled more than 230,000.

DoneDeal is now the number one place to buy and sell things in Ireland, according to Google Trends.

The Challenge

DoneDeal’s objectives were as follows:

  1. Build a future proof data analytics platform that will scale with the company over the next 5 years
  2. Take ownership of their data. Collect more data
  3. Replace existing reporting tool
  4. Provide a holistic view of our users (buyers and sellers), ads, products
  5. Use their data in a smarter manner and provide recommendations in a timely fashion

DoneDeal wanted to look at technologies such as Machine Learning to make it easy to obtain predictions based on event analysis about how items would sell on their website. In doing this – they aimed to improve on the services that they provide to customers and develop a set of premium services for corporate clients etc.

They were also looking at predictability in terms of the AWS “Big Data” platform that is secure and can be easily extended to deliver future business requirements. Today, the technologies in AWS such as EMR (Hadoop) – enable analysis of vast amounts of data, Kinesis – for real time data processing of large data streams, and Data Pipeline – a web service that you can use to automate the movement and transformation of data within an AWS architecture such as archive your web servers logs to Amazon Simple Storage Service (Amazon S3)

The Solution

DNM provided an iterative approach to the delivery of the data platform, delivering an end to end solution (data source to dashboard) for a single subject area as part of some services that they wanted to pilot – the concept was to use machine learning to evaluate the behaviours of end users and actions that are carried out on their sites and to test the concepts to see how they could be applied to:

  • Current and future value of cars
  • Time to sell prediction (based on price, make, colour, region)
  • Ad performance (bump / spotlight / search / click through metrics)
  • Supply / Demand information
  • Providing the sales team with value-add information to retain and increase the number of Motor Dealers using DoneDeal.

The solution is inline with AWS Data and analytics reference architecture.

Data Movement  – The Web Application database (PostgreSQL) – an extract of the data is provided in JSON format. We use Kinesis for stream-based data to take data from the source data sources and storing it in the initial “raw” data lake for ongoing consumption.

Data Lake – Data stored by Kinesis will was delivered to the raw data lake storage (S3) using the Kinesis Connector Library (KCL) deployed as an application on an AWS EC2 server.   S3 Raw Data Lake – this is the output from the initial ingestion from Kinesis and the S3 file staging area, ready for the remaining processing.  S3 Staging area for Processed Data – this is the output from the processing layer, ready for input into the Redshift data warehouse.

Analytics -Donedeal have a comprehensive set of services for data analysis, Data Pipeline will be used to orchestrate (sequence, schedule, run, and manage recurring data processing workloads) many of the data movement functions of the solution while the “heavy lifting” of the ETL and data enrichment processing will be carried out using AWS Elastic Map Reduce (EMR).  Data visualisation in this instance a Tableau server provided the dashboard / data visualisation solution. While other AWS services such as Amazon Quicksight would be looked at in the future.

Machine Learning

A number of Machine learning (ML) algorithms have been derived from their external and internal systems. Applying these are a core capability to enhance the Customer experience for DoneDeal

Impact

Using the AWS platform, DoneDeal were able to transform their business model. The AWS technologies meant that they could use machine learning cost effectively to simplify the predictability of how customers could better use their services without making a huge Capex investment.

They were able to avail of quicker time to market and a faster time to realise value in the services that they were providing.