Learn How to INSTALL n8n in QUEUE Mode with Docker Compose - COMPLETE 2023

This article provides a comprehensive guide on deploying N8n in queue mode using Docker Compose, detailing the various components, configurations, and best practices. It covers setting up the editor, webhook, and worker services, along with essential environment variables like database connections, encryption keys, and base URLs. The guide also emphasizes resource allocation for each service, demonstrating how to optimize CPU and memory usage to ensure smooth operation. Special attention is given to integrating with Traefik for SSL and URL routing, and managing DNS records with Cloudflare. The process of creating the necessary PostgreSQL database and verifying service health through Docker commands and service logs is also thoroughly explained. This detailed walkthrough aims to equip users with the knowledge to implement a robust N8n queue mode setup, even on resource-constrained servers, highlighting the importance of testing and incremental resource adjustments.

Understanding N8n Queue Mode Deployment

Deploying N8n in queue mode involves setting up multiple services that work together to handle workflows efficiently. This mode is particularly useful for managing a high volume of requests, where a central message queue facilitates asynchronous processing. The core components of this setup include an editor, a webhook handler, and one or more workers.

The speaker highlighted the importance of starting with adequate server resources, ideally 2GB of RAM, although 1GB can suffice for initial setups without other applications. The deployment relies on specific prerequisites: Redis for caching and message brokering, and PostgreSQL (version 13 or higher) for persistent storage and queue management.

You must have a PostgreSQL version 13 or higher. It must also have the execution mode set to queue mode. So, it will send to the database, and you must have these two variables in the configuration file, better speaking, and others that we will see shortly. Redis is also needed.

The speaker noted that the official N8n documentation for queue mode can be somewhat confusing, especially for newcomers. The current guide aims to simplify this process, providing a clear and practical approach based on extensive research and aggregation of information.

Service Components in Docker Compose

The Docker Compose setup orchestrates three distinct services, each with a specific role:

Editor Service

The editor service is the primary interface for users to access the N8n web application and design their workflows. It is configured to run the latest N8n image and connects to two crucial networks:

• Public Network (with Traefik): This network allows Traefik to expose the editor service to the internet and handle SSL certificates automatically.
• App Network: This network enables the editor to communicate with the PostgreSQL database.

The configuration ensures that the editor can reach Traefik for secure access and the database for data storage and queue interactions.

Webhook Service

The webhook service is dedicated to receiving incoming requests via webhooks. Although it uses the same N8n image and network configurations as the editor, its command is specifically set to `webhook`. This ensures that the container solely focuses on webhook handling, receiving requests, and pushing them to the database queue for processing by the workers.

When you put here 'webhook,' this container that we are going to spin up will simply use the webhook resource. It will simply wait to receive through this container.

It requires a predefined webhook URL, which must be configured in the environment variables and pointed to via Cloudflare DNS.

Worker Service

The worker service is responsible for processing tasks from the queue. It also uses the same N8n image but has a different command: `worker`. A key configuration for the worker is its concurrency level, which determines how many tasks it can process simultaneously. The speaker suggested starting with a concurrency of 3, which can be adjusted based on available resources and workload. The default concurrency in N8n is 10, but lowering it is beneficial for servers with limited resources.

Each service shares the same core image and network setup, with the primary differentiation being the specific command executed, which dictates the role of each container.

Environment Variables Configuration

Environment variables play a critical role in configuring N8n services. These variables are consistently applied across the editor, webhook, and worker services, ensuring uniform behavior and connectivity. Key environment variables include:

The speaker mentioned that all environment variables are mostly going to be the same across all services, making it simpler to manage, as you just need to configure them once and then replicate them.

Resource Allocation and Deployment

Proper resource allocation is crucial for the stability and performance of N8n services, especially on servers with limited resources. The speaker emphasized the importance of starting with a base resource allocation and gradually increasing it based on monitoring and actual consumption.

Editor Service Resources

The editor service, while critical for UI access, typically has lighter resource requirements compared to workers. The speaker suggested an initial configuration with a replica count of 1. For resource limits, an example setup includes 30% CPU and 512MB of memory. These values serve as a starting point and should be adjusted according to the server's capabilities and observed performance.

The editor will always run in replication mode, the replica will always be one, and the placement will always be where you run it. In my case, it's in 'mana'. The resources here I will limit to 30% CPU, remembering it's just an example, do not take this and put it in your scenario. See what you have, define the initial point, and increase according to your needs. So, 30% and 512MB memory.

The editor also integrates with Traefik for URL routing and SSL certificates, ensuring secure and accessible UI and API endpoints. The labels within the Docker Compose file configure Traefik to enable this. The speaker recommends using the same name for the Traefik route as for the service name for better organization.

Webhook Service Resources

The webhook service manages incoming requests and requires careful resource allocation. While the speaker initially stated that webhooks typically use more memory, the example limits provided were more balanced: 50% CPU and a flexible memory allocation. Unlike the editor, the webhook service can benefit from multiple replicas to handle a large influx of requests, allowing for load balancing by Traefik across these replicas.

Similar to the editor, the webhook service's URL is configured via Traefik labels, pointing to its specific endpoint (e.g., `webhook.n8n.mydomain.com`). It's essential to ensure the service name and the Traefik route name match for consistency.

Worker Service Resources

Workers are the most resource-intensive components as they execute workflows. The initial resource allocation for a worker in the example was 50% CPU and a slightly higher memory allocation than the webhook service. Workers can also be replicated to increase processing capacity, with each replica independently processing tasks from the queue with its defined concurrency level.

For better user experience, it makes sense that the editor will be more lightweight, because it just handle the UI for the user. But the important part is on the webhook and worker services as them will handle all the automation.

Monitoring resource usage through tools like Portainer or Docker stats is crucial. The speaker demonstrated how to check CPU and memory consumption for running containers, advising users to increase resources incrementally if they notice performance bottlenecks, especially when a service reaches its defined limits. This iterative approach ensures optimal performance without over-allocating resources.

Database Creation and DNS Configuration

Before deploying the N8n Docker stack, two critical steps are necessary: creating the PostgreSQL database and configuring DNS records to resolve the N8n and webhook URLs.

PostgreSQL Database Creation

The N8n queue mode requires a dedicated PostgreSQL database. The speaker demonstrated how to create this database using the Portainer web interface, specifically by accessing the console of the running PostgreSQL container. From the PostgreSQL prompt, a new database named `n8n_queue` is created using the simple command:

CREATE DATABASE N8N_QUEUE;

This ensures that N8n has a dedicated schema to store its queue-related data and workflow information. The speaker advised checking the database creation by listing all databases using `\l`.

Cloudflare DNS Configuration

To make the N8n editor and webhook services accessible via custom domain names, DNS records must be configured in Cloudflare. The speaker created two CNAME records:

• Editor URL: `n8n.mydomain.com` pointing to the server's primary domain (e.g., `serve.learnweb.com.br`).
• Webhook URL: `webhook.n8n.mydomain.com` also pointing to the server's primary domain.

These CNAME records resolve the custom URLs to the server's IP address, allowing Traefik to intercept the requests and route them to the correct N8n service. Disabling the Cloudflare proxy (setting it to "DNS only") is also recommended to allow Traefik to handle SSL certificate issuance.

Takeaways

1. N8n Queue Mode Essentials: Deploying N8n in queue mode requires Redi for messaging and PostgreSQL (v13+) for the database.
2. Docker Compose Structure: The setup uses three distinct Docker services: `editor`, `webhook`, and `worker`, each with specific commands and functionalities.
3. Environment Variable Consistency: Key environment variables, including database connection details, encryption keys, and URLs, must be consistently applied across all services.
4. Strategic Resource Allocation: Start with minimal CPU and memory limits (e.g., 30% CPU, 512MB RAM for editor) and scale incrementally based on monitoring, as webhook and worker services are generally more resource-intensive.
5. Database and DNS Prerequisites: Before deployment, create the N8n queue database in PostgreSQL and configure CNAME DNS records (e.g., with Cloudflare) to point to your N8n and webhook URLs.

References

• N8n queue mode documentation
• Docker Compose deployment guide
• PostgreSQL database creation
• Cloudflare DNS CNAME records