Use case and methodology for data science in medium-size companies with big data open source technologies

Big data technologies represent an enabler of crucial importance for medium-size companies. They create new opportunities for data analytics by allowing companies to analyze large volumes of data. Retail companies, for instance, wish to use their sales data to obtain insights on the effectiveness of their promotions. Nevertheless, open source big data technology is not mature and there is a need for building an ad hoc architecture to support ad hoc data analytics methodologies. In the context of understanding the profitability and effectiveness of promotions we propose an architecture for the computation of new indicators. This has involved the definition of not only an architecture integrating different open source components, but also a new methodology for the assessment of the benefits from promotions. These benefits have been rarely measured before due to a lack of affordable technologies for data analytics. Apache Hadoop and Apache Hive represent two examples of such technologies. However, Hadoop lacks integration, making it difficult for company managers to run analytics by themselves. It is thus necessary to build an architecture integrating Hadoop and Hive with an ad hoc software to run the analysis automatically. In the context of promotions, a particular type of pricing strategy for their products that firms are apprehensive about is loss leader pricing, which consists in selling selected products at or below their marginal cost. Selling these products results in an economic loss for the firm which should be compensated by an increase in store traffic and, therefore in the number of products sold at a regular price. The goal of this thesis is creating a software tool capable of analyzing loss leader promotions. In order to do that, it's first necessary to understand the impact and profitability of these promotions. Using data from a retail food store and some tools for big data mining such as Apache Hadoop and Apache Hive, we propose methodologies, indicators and techniques to analyze loss leader promotions and we integrate these methodologies into a software tool to be used together with Hadoop. This tool can be used by store managers to analyze the impact of loss leader promotions and can also be used as a blueprint for future software tools in the field.