Receive real-time HTTP callbacks the moment a scrape event completes — no polling, no delays. This guide walks you through setup, verification, event handling, and production best practices.
When you make an API scrape request, ResilientLink processes it and — once complete — fires an HTTP POST to the endpoint URL you have configured in your dashboard. Your server receives the event payload, processes it, and returns 200 OK.
This is entirely asynchronous. You do not wait for the webhook inside your original API call — the scrape response comes back immediately, and the webhook fires in the background once the job settles.
All webhook settings live in your ResilientLink dashboard under Webhooks. No code deployment needed to change them.
https:// and point to a real host. http:// and localhost are not accepted.X-ResilientLink-Signature on every delivery so your server can verify the request is genuine.403. Leave blank to allow all IPs.localhost will not receive deliveries. Use a tunnel like ngrok during local development.ResilientLink fires webhooks for three event types. Every payload shares the same outer envelope — only the data object differs.
| Event | When it fires | Frequency |
|---|---|---|
| scrape.completed | Metadata extraction succeeded and structured data was returned. | Every successful scrape |
| scrape.failed | The scrape request errored, timed out, or the target returned an error status. | Every failed scrape |
| quota.warning | Your monthly request usage crossed 80% or 95% of your effective limit. | Once per threshold per calendar month |
All webhook requests are HTTP POST with Content-Type: application/json. The body always follows this structure:
{
"event": "scrape.completed",
"timestamp": "2026-05-19T12:00:00.000Z",
"data": {
"url": "https://example.com/article",
"status": "success",
"status_code": 200,
"response_time": 312, // milliseconds
"metadata": {
"title": "Example Domain",
"description":"This domain is for illustrative examples.",
"image": "https://example.com/og.png",
"domain": "example.com",
"word_count": 142,
"scraped_at": "2026-05-19T12:00:00.000Z"
}
}
}
{
"event": "scrape.failed",
"timestamp": "2026-05-19T12:01:00.000Z",
"data": {
"url": "https://example.com/blocked-page",
"status": "failed",
"status_code":403, // HTTP status from target, null if network error
"error": "Target returned 403 Forbidden",
"metadata": {}
}
}
status_code in a scrape.failed payload is the HTTP status returned by the target website, not by ResilientLink. It may be null if the failure was a network-level error (timeout, DNS failure, etc.).| Field | Type | Description |
|---|---|---|
event | string | Event type identifier. One of scrape.completed, scrape.failed, quota.warning. |
timestamp | string (ISO 8601) | UTC time the event was fired. |
data | object | Event-specific payload. Shape varies by event type — see below. |
Your webhook endpoint is publicly accessible, so anyone could POST to it. If you set a Webhook Secret in your dashboard, ResilientLink includes it verbatim in the X-ResilientLink-Signature header on every delivery. Compare this value server-side to confirm the request is genuine.
401 if missing or mismatched.// On every incoming webhook request: incomingSignature = request.headers["x-resilientlink-signature"] expectedSecret = ENV["RESILIENTLINK_WEBHOOK_SECRET"] if incomingSignature != expectedSecret: return HTTP 401 // reject — not from ResilientLink // safe to process payload = parse_json(request.body) handle(payload.event, payload.data)
Copy the example for your stack, replace the secret env var, and deploy. Your handler must return 200 OK — the response body is ignored entirely.
const express = require('express'); const app = express(); app.use(express.json()); app.post('/webhook', (req, res) => { // 1. Verify signature const sig = req.headers['x-resilientlink-signature']; const secret = process.env.RESILIENTLINK_WEBHOOK_SECRET; if (!sig || sig !== secret) { return res.status(401).end(); } // 2. Parse event const { event, timestamp, data } = req.body; console.log(`[webhook] Received: ${event}`); // 3. Route by event type if (event === 'scrape.completed') { // data.url, data.metadata.title, etc. handleScrapeCompleted(data); } else if (event === 'scrape.failed') { handleScrapeFailed(data); } else if (event === 'quota.warning') { handleQuotaWarning(data); // data.percent_used, data.threshold } // 4. MUST return 200 — body is ignored res.status(200).end(); }); app.listen(3000);
from flask import Flask, request, abort import os app = Flask(__name__) @app.route('/webhook', methods=['POST']) def webhook(): # 1. Verify signature sig = request.headers.get('X-ResilientLink-Signature') secret = os.environ['RESILIENTLINK_WEBHOOK_SECRET'] if sig != secret: abort(401) # 2. Parse event payload = request.get_json() event = payload['event'] data = payload['data'] timestamp = payload['timestamp'] # 3. Route by event type if event == 'scrape.completed': handle_completed(data) elif event == 'scrape.failed': handle_failed(data) elif event == 'quota.warning': handle_quota(data) # 4. MUST return 200 return '', 200 if __name__ == '__main__': app.run(port=3000)
<?php // webhook.php // 1. Verify signature $sig = $_SERVER['HTTP_X_RESILIENTLINK_SIGNATURE'] ?? ''; $secret = getenv('RESILIENTLINK_WEBHOOK_SECRET'); if ($sig !== $secret) { http_response_code(401); exit(); } // 2. Parse event $payload = json_decode(file_get_contents('php://input'), true); $event = $payload['event'] ?? ''; $data = $payload['data'] ?? []; // 3. Route by event type match($event) { 'scrape.completed' => handle_completed($data), 'scrape.failed' => handle_failed($data), 'quota.warning' => handle_quota($data), default => null, }; // 4. MUST return 200 http_response_code(200); exit();
200, then dispatch the work to a background queue or job processor.The quota.warning event fires when your usage crosses 80% or 95% of your effective monthly limit. It fires at most once per threshold per calendar month — you will not be spammed.
{
"event": "quota.warning",
"timestamp": "2026-05-19T18:42:00.000Z",
"data": {
"percent_used": 80, // 80 or 95
"requests_used": 800, // requests consumed this month
"requests_limit": 1000, // effective limit (plan + bonus)
"plan_limit": 1000, // base plan allowance
"bonus_requests": 0, // remaining one-time bonus (referral)
"threshold": 80, // which threshold triggered this
"reset": "Resets on the 1st of next month."
}
}
if (event === 'quota.warning') { const { percent_used, threshold, requests_used, requests_limit } = data; if (threshold === 95) { // Critical — switch to cached fallback or pause non-essential scrapes enableFallbackMode(); alertOpsTeam(`CRITICAL: ${percent_used}% quota used`); } else { // Warning — notify team, consider upgrading sendSlackAlert(`Quota at ${percent_used}%: ${requests_used}/${requests_limit} used`); } }
| Rule | Detail |
|---|---|
Must return 200 |
Any other status (201, 204, 4xx, 5xx) is treated as a failed delivery. The response body is never read — only the status code matters. |
| No retries | ResilientLink does not automatically retry failed webhook deliveries. Implement idempotency in your handler in case you re-process events manually. |
| Inactive status skips delivery | If your webhook status is set to Inactive in the dashboard, no events are fired at all — even if a URL is saved. |
| IP restriction applies at API & Webhook level | The Allowed IPs setting blocks API requests — incoming webhooks . Your webhook endpoint can be on any IP or the specific IP you specified when enabled. |
| Secret header is optional | If no secret is set, the X-ResilientLink-Signature header is omitted entirely. You can still receive events — but you lose the ability to verify origin. |
| Quota warnings are deduplicated | The 80% and 95% thresholds each fire at most once per calendar month. The same threshold will not fire again until the counter resets on the 1st. |
If your webhook endpoint returns a non-200 response, or is completely unreachable, ResilientLink sends an email notification to your account email explaining the failure. To avoid inbox flooding, these emails are rate-limited to at most once per hour per failure type.
| Type | Cause | How to fix |
|---|---|---|
| Non-200 response | Your server is reachable but returned a status other than 200 (e.g. 500, 404, 204). |
Update your handler to always return exactly 200 OK, even if you don't process the event. |
| Unreachable endpoint | The URL is invalid, the server is offline, DNS fails, or the connection is refused before a response is sent. | Verify the URL is publicly accessible. Check server uptime, firewall rules, and DNS resolution. |
204 No Content thinking it means "success with no body." ResilientLink requires exactly 200. Always call res.status(200).end() explicitly — do not rely on framework defaults.200 — for every event type, including ones you don't handle yet. Use a catch-all that returns 200 for unknown events.X-ResilientLink-Signature header on every request.timestamp and data.url as a natural deduplication key.scrape.completed event and logs it correctly.