Contact us

“We built a really cool game, “marketer vs. machine,” where we showed marketers a couple of different versions of a landing page and asked which version is going to convert best, and we let all the marketers at the conference answer, and then we let the machine answer. What we saw was about a 50/50 split. So, you basically had an equal chance of flipping a coin as the marketer making an educated, not informed, decision about which version of the landing page is going to convert best. Meanwhile, the machine had predicted, I think, 80% of the time, so we really saw the power machine learning had.”

– Tamara Grominsky, CSO at Unbounce, How to Win podcast

LPP, with over 2000 stores and operating in 30 countries, stands out as one of the largest retailers in Central and Eastern Europe. The group includes brands such as Reserved, Sinsay, House, and others.


One of the signs of its rapid growth is its constant expansion into new countries. That raises the question: "Is it better to open a local shop or invest in online marketing?" Or perhaps a mix of both?

The problem

   The problem is that the volume of LPP data sets is so large that it's difficult to find valuable information without drowning in a sea of non-task data. Then, when you factor in the fact that last-click data provides
a skewed perspective on attribution, the process becomes even more complicated.

 

  So far, expansion decisions have been based on the experience of the leadership team, and that has proven to drive growth. However, if there's a way to streamline this process and maximize ROI at this scale, it's worth looking into.

 

  Making informed scaling decisions could accelerate expansion by uncovering places with high potential and make the whole process less burdensome for decision-makers. That being said, understanding complex operations while working with ROI required building a solid foundation, and that's exactly what LPP did as a first step.

image (36)

"Our current data sets are so large that it takes a lot of time to find new insights.
If we don’t know where to look, the process might be consuming too much time for data analysts."

– Marek Pełszyński, Head of Analytics from LPP

The true influence of a channel

 

MMM (marketing mix modeling) has been around for a while. It’s a time-tested method to determine the impact of the different elements of your business. With this type of analysis, you’re able to find answers to questions like: 

Not actual clients data
image contains sample numbers to explain the idea

What is the channel attribution?

What’s the actual impact of each marketing channel? If the lead was generated by a TV ad and closed by SEO, what’s the contribution of each channel?

What’s the halo effect between channels?

The halo effect shows the interaction of the channels on one another that’s impossible to measure with the last click. Because of this, you can see, for example, how a campaign from Facebook affects organic sales from Google.

 

As it turns out, channels overlap more than we'd like. Investing in flyers could have a bigger impact on your online channels than online ads.

Not actual clients data
image contains sample numbers to explain the idea

Not actual clients data
image contains sample numbers to explain the idea

What's the most effective way to allocate a budget?

If we know the attribution and penetration of each channel, we should be able to calculate the optimal allocation of our budget.

How accurate can we get?

Which is better, to invest in an offline business or in online advertising?

 

If we know how to allocate the budget for each market, we can easily determine if and when to open a store. The problem is that MMM analysis is done by big consultancies like Nielsen, costs a five-figure sum, and is just an analysis of a single document. If you wanted to react dynamically, you would have to do an analysis every month or even every week. That would defeat the purpose of optimization.

 

So, what are the other options?

In addition to pandemic restrictions, 2021 also brought privacy changes to big giants like Apple and Google. The so-called "cookie apocalypse" has begun. Measuring and predicting consumer behavior in the forms previously used has become ineffective and unreliable. In response to these changes, Facebook has launched an experimental open-source algorithm, Robyn, to democratize access to modern MMM and enable data analysis despite the apocalypse.

 

As a pioneer in the use of modern solutions, LPP was interested in using Robyn. There were many things to suggest that this tool could be the right choice to get answers. However, it turned out that there was a big challenge. As usual in fast-moving companies, there was a lack of resources to learn new technology and implement it quickly.

div_sap-spartacus_lp_icons_architects-1

“Effective marketing is built on understanding what drives results. That’s true especially in a fast-changing environment as we are in today. For many companies undergoing fast transformations from a more traditional way of operating into digital models, it may be a real challenge. In that context, it’s crucial to evaluate every investment and compare the profitability of it. Robyn might be a good starting point here as it democratizes access to modern MMM.”

                                                                  - Gabriel Matwiejczyk, Measurement Lead at Facebook.

Robyn and Predicty

predicty logo raster-1
Robyn is powerful, but it requires expertise and resources to make the most of it. LPP did not have spare resources in-house to handle it. Also, large amounts of data from multiple sources made it even more difficult to process and draw business conclusions. 

In their search for solutions, they came across Predicty. Predicty is an overlay for Robyn that simplifies implementation by providing integrations and customer support. It also offers additional features like a live data feed and visual dashboards. LPP wanted to find out if the models fed by their data would accurately reflect what was happening in their business. To that end, they turned to Divante, the creators of Predicty. 

LPP could try to implement raw Robyn on their own, and keep the project in-house, or choose to work with Divante and delegate the implementation of Predicty. It wasn’t an easy decision because both projects had their benefits and risks. However, we suggested doing a small exercise with Predicty to see what we could do together. 

"Due to the workload and lack of free resources, it was not possible to get Robyn up and running quickly. That's why we decided to look for an alternative."

 

- Marek Pełszyński, Head of Analytics from LPP

   We chose a database with data through the end of June, and our goal was to predict results for July and August to check the overlap of the model and business data. If this task was successful and the data was correct, LPP would work on implementing Predicty. The plan for collaboration emphasized not only Robyn itself but also the implementation support we provided.

This greatly expands the scope of Robyn.

  The whole area related to forecasting and planning activities and budgets was extended to include the possibility of rapid implementation and handling, even by non-technical people. The vision of making decisions based on reliable data was extremely tempting, but we didn’t know if the models could handle LPP datasets.

  The results of the test were presented at a meeting attended by the LPP team as well as representatives from Divante and Facebook.
It was a beta prediction result and the final number was within LPP's stated range. This showed LPP a glimpse of the possibilities and helped them decide on further modeling efforts with Divante.

“Robyn is open source by design and community contribution is extremely valuable. It’s great to see how Divante contribute to that common community journey”  
– Gabriel Matwiejczyk, Measurement Lead at Facebook.

   In the next phase, we ran a 30-day sprint. During the sprint, we deployed a proof of concept (POC) to gain initial actionable insights based on real data.

 
div_sap-spartacus_lp_icons_developers

Architecture

int1

When we started working on the PoC together, Marek Pełszyński, the Head of the Analytics Team, said to us,

“I wanted to work with someone who has fresh thoughts on this topic. Let us see what your approach will be.”

Before moving to modeling, we wanted to make sure our architecture would be able to handle the massive datasets LPP has. That's why we decided to use Fast API to get data from clients BigQuery to Elasticsearch in the form of Dat

After proper optimization, we managed to speed up the operational process from days to hours. Even though this was already an impressive outcome, we needed to optimize this process even further because we planned to make hundreds of calculations.

We needed something that would allow us to iterate fast without the need to constantly reload the data. That’s why we decided to use a Redis in-memory data structure. By doing this, operations could be stored on our devices’ RAM and enable incredibly fast access.

int2
submod3

Moreover, this infrastructure allowed us to manipulate the data while still keeping all the databases in sync. This reduced the possibility of a data mismatch. After that, we were ready to start the modeling phase. In order to get as much value from Robyn as possible, we decided to take its subcomponents, like Prophet or Nevergrad, and use them as standalone engines.


Our idea was that we could feed the Robyn models with outcomes from other sub-models and use it as a cross-validation mechanism. For example, you could use Robyn to calculate eCommerce traffic and conversion separately to check if the outcomes of doing so would match with the Robyn models of online revenue.

Also, by doing so, we were able to break down the revenue trend line to traffic and conversion lines. While the effects of the pandemic weren’t so clear on the revenue trend line, traffic and conversions showed much more clear market behavior.

The final step in assuring the quality of the results was the use of Prophet to lay the groundwork for KPI predictions. After various experiments, we were able to understand the complexity of forecasting in the case of the dynamic LPP structure.


As we found out later, this cross-validation approach was an innovation on Robyn-using companies' market. It also pushed us towards the direction of considering additional cross-validation engines in further stages.




Future plans


Right now, we’re starting phase two of our project with LPP. Our ultimate goal is to figure out whether they should invest in online marketing or offline businesses. Along the way, we need to fully understand the influence of each marketing channel and each campaign, but also parameters like the weather or the influence of the competition.

The goal of phase two is to have a detailed understanding of online channels so that we can leverage the huge potential we found earlier. To this end, we have the delivery of:

  • C1 cross-validation engine
    C1 is a dedicated module developed in collaboration with a data science agency Iine. It’s used to calculate attribution, like Robyn does, but with different mathematical formulas. This approach allows for the cross-validation of Robyn and C1 results. It also ensures that the basic assumptions are correct before further analysis steps are taken, speeding up the process and adding credibility.


  • Higher granularity
    While MMM is a great tool, it also provides high-level information that requires additional work to turn into operational tasks. To simplify the flow from insights to action, we decided to break down online marketing channels and perform all analyses per campaign group.



"Thanks to the additional granulation at the campaign level, we’ll be able to turn strategic insights into operational tasks"
– Marek Pełszyński, Head of Analytics from LPP


We see growth potential for LPP and are eager to help them achieve it. See you in the case study from phase two.

Discover how you can use Predicty in your business

If you'd like to see how to use this solution in your business - such as budget allocation,
marketing channel optimization, the impact of eCommerce elements on revenue,
and more - sign up.

 

Predicty Team

A special shoutout to the LPP team engaged in this project. Thanks to your fast responses and high engagement, we managed to find our way through the massive datasets of LPP. Special thanks to Marek Pełszyński, Michał Borowczyk, Agnieszka Budzisz, Rafał Krzynowek, and Jakub Kozak.

We also greatly appreciate the support of Gabriel Matwiejczyk, from Facebook, and our consultant from Nielsen, Monika Stezewska-Kruk.

Thank you for the support from iine team, whose analytical experience helps us move faster and better in Predicty: Jakub Slusarczyk CEO, Katarzyna Sekścińska PhD Business Psychology, and Piotr Wnuk-Lipiński PhD Computational Intelligence.

Lastly, thanks to the Divante team for your effort and agile approach to problem-solving. Project delivery wouldn't be possible without Artur Wala, Kamil Janik, Agata Czapla, Agata Semenowicz, and Stanley Swiatkiewicz.