How to build a scalable and automated web scraping workflow using AI and N8N

Access to information has emerged as one of the crucial resources for business decision-making processes. Companies are continually inundated with information from various sources, yet they struggle to process and convert that information into actionable insights. Firms that can harness technology and integrate it with their strategic plans are the ones that create impact and surpass their competitors. Automation tools and AI are part of these key technologies that businesses need to master to improve their data-driven decision-making processes. Using automation tools such as N8N to articulate and coordinate the data extraction and loading process while leveraging AI to generate insightful analysis on the extracted data, transforms a manual logic-lacking business process into a powerful ally for upcoming businesses.

The tools mentioned in this post are:

• N8N
• Claude AI
• Hookdeck
• Sevalla Cloud Server
• Snowflake
• Python – Beautiful Soup/Selenium

Our ELT process will integrate several sub-processes involving the data extraction through a cloud server-based Python code, loading data into Snowflake, and generating a basic analysis with Claude AI to extract relevant information from the extracted data.

Step 1 - Implementing N8N as a process orchestrator

N8N has become one of the most popular automation tools. With the ability to integrate multiple apps, platforms, and make API calls, N8N has become a major orchestrator for these kinds of processes. The first step for unlocking N8N orchestration is to create several credentials for our connections.

• Create N8N credentials for Snowflake and Claude (Anthropic).

• Select the desired platform/app/service.

• Complete Credential’s fields to connect your N8N instance with the service.

• Repeat this step for Anthropic AI and any other service needed.

Step 2 – Automated process creation in N8N

For this step is important to understand how our data will flow from the raw extraction from to source, into our database created in Snowflake. Not only what data we are going to ingest, but the reason why we are ingesting that data. A clear understanding of the process’s purpose and demands will help avoid unnecessary data flowing from our sources to our database.

1. Create a scheduled trigger for the daily process run

Using the “schedule trigger” node in N8N, we can schedule our workflow to run automatically every minute, hour, day, or week.

2. Sevalla Server API call and Python App

As mentioned before, the webscraping will be done by a Python script running on a cloud server called Sevalla. This will help our workflow run Python cloud outside our local servers.

For this action, it is necessary to set an HTTP Request node that will call the Python code through a POST API call and start the web scraping process.

It’s important to understand the API composition according to the data we need to input to our Python app to work.

3. Receiving the scraped data from Sevalla

Since the web scraping process might take more than 60 seconds, N8N is not able to wait for the scraped data (N8N is programmed to close the HTTP request node if no data is received within 60000ms). That is the reason why we use Hookdeck as an intermediary. Hookdeck will receive the output data generated by the Python app and send it directly to our N8N’ webhook trigger.

4. Receiving the scraped data through our webhook

As mentioned in the previous step, we will be receiving the output data from our Python app in a trigger called “webhook”. The webhook acts as a 24/7 open gate ready to receive data from API calls.

The webhook trigger has different instances: the TEST instance, for webhook and workflow testing, and the PRODUCTION URL, used for the actual workflow running process. Our Hookdeck connection will be sending data from the platform to the webhook URL specified in our Webhook trigger.

5. Processing and loading scraped data into Snowflake

Once the data is ingested in our N8N workflow, the loading magic begins. By analysing the incoming data with the available data in our database, we can identify new data coming from the web.

It is always important to modify the data structure coming from the webhook trigger since a lot of API call information is included. This is the main reason why we use a “Split” and “Code” nodes to extract only the relevant information from the API call.

6. Loading data into our Snowflake DB using Snowflake’s nodes

For the data loading, we will need to use our Snowflake credentials mentioned in Step 1. Each Snowflake node enables the user to create a SQL query to interact with Snowflake. In this case, the three SQL queries will be INSERT INTO “database”.

N8N allows us to call data and use expressions from previous nodes to reference previous nodes in our data processing.

7. Finalise the extraction subprocess

Once all our extracted data is loaded, creating data transformation processes inside the database environment is important. This will help correct data types, null values, structures, and any possible outliers.

Step 3 – Using AI to analyse and process our loaded data

AI has become a major discussion topic in the last few years, with AI implementation as a must for businesses. However, trying to implement AI and failing to understand its usage and implications can be counterproductive.

For this part of the process, we will be using AI (Anthropic Claude AI) to analyse text strings related to our scraped input and compare them to our business logic input. The AI node in N8N will focus on comparing our business description and logic and determine if it matches our scraped input. This way, the AI will return a “relevant” or “not_relevant” flag.

1. Setting up Claude AI node

As mentioned before, it’s important to set the AI node through our credentials. For this operation, we will be using an LLM Chain node. This N8N helps the workflow interact with an LLM model through a specified prompt.

The node will help us define a source for our prompt (chat message or a defined input), and also a specific prompt. As seen at the bottom of the previous picture, the connected mode (through our credentials) is the Anthropic model.

The node will help us define a source for our prompt (chat message or a defined input) and a specific prompt. As seen at the bottom of the previous picture, the connected model (through our credentials) is the Anthropic model.

It is possible to specify even more system behaviour for our AI node by adding prompts in the “Add prompt” section.

Input business logic to AI prompt.

Here is where the magic begins. As we know, the response we get from AI is only as good as the prompt we create and the input data. It's important to structure the input data clearly, focusing on the main business drivers and in an appropriate format. Is possible to specify even more system behaviour for our AI node by adding prompts with the “Add prompt” section.

1. Input business logic to AI prompt

Here is where the magic begins. As we know, the response we get from AI is only as good as the prompt we make and the input data. It is important to structure the input data in a clear way, focusing on the main business drivers, and in an appropriate format.

2. Loop through not-analysed records in our DB

By implementing an IF loop in N8N, we can find all the “not-analysed” records in our status table and input them into our AI node for analysis. By comparing the business input with the data from our database, the AI node will generate a relevance analysis that outputs a “relevant” or “not_relevant” flag.

After this process, our N8N workflow will update that record in our database and update the analysis flag to an “analysed” status, and update the relevance_flag status to either “relevant” or “not_relevant”.

Conclusion

Automating the ELT process through an orchestrating platform like N8N can be a powerful ally for small businesses that lack the resources of larger companies. By leveraging N8N and its integrations with various platforms, along with AI, businesses can ingest data from multiple sources, load it into their preferred database platform, and process it to gain powerful insights by using their business inputs. Automating this process enables a more up-to-date, data-driven decision-making processunlocking better results.