Smartspider is a company that can provide insights into, for example, the news of product prices across the market by means of data collection and the processing of the data.
In the past six months, we have built an application for the company Smartspider that consists of 3 different main components:
- Smartspider Collector The collection application that ensures that data is found, changed and stored correctly in the database.The biggest challenges in this application that we came across was to find the right data on the websites, and to ensure that the data was always properly picked up. One of these challenges we faced were cookie banners. It was no problem when searching for an article, but when reading an article or a product, for example, the fact that it might have a cookie banner was not taken into account. If this was the case, only the text of the cookie banner returned. We have solved this challenge by clicking on all possible buttons on the first visit of the website and saving all resulting cookies, than sending them with every request in the future. This solution has ensured that the data is no longer clouded by cookie banner statements. Links have also been made with Facebook and Twitter to easily retrieve extra data from these channels and include it in the data collector.
- Smartspider API the API to ensures that the data can be integrated in the web version that we have also built. The main challenges we faced was sending the collected data more securely, and make it easy to find via the search functionality. In addition, an integration has been made with Stripe to make it possible to retrieve the data with a subscription.
- Smartspider Web To bring all these components together, a web application has been created to provide easy insight into what is happening with the API and the Collector. It is made easy here to search, request and view the detailed information.
These 3 components provide the total product with which Smartspider can help its customers to gain insight into processes with data. For example, this tool is currently used by Achmea to automatically obtain news about files with the file.