> For the complete documentation index, see [llms.txt](https://script-ware.gitbook.io/cyberdrop-dl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://script-ware.gitbook.io/cyberdrop-dl/reference/config/crawlers.md).

# Crawlers

## `disabled`

| Type                | Default | Additional Info                                                                                                                         |
| ------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `list[NonEmptyStr]` | `[]`    | This is an [`AdditiveArg`](https://github.com/Cyberdrop-DL/cyberdrop-dl/tree/main/docs/reference/special_setting_types.md#additiveargs) |

You can supply a list of crawlers to disable for the current run. This will make CDL completely ignore the crawler, as if the site was not supported. However, links from the site will still be processed by Real-Debrid (if enabled), JDownloader (If enabled) and the Generic crawler (If enabled), in that order.

The list should be valid crawlers names. The name of the crawler is the name of the primary site they support. ex: `4Chan`, `Mega.nz`, `Dropbox`

You can get the crawler' name from the [supported sites page](https://script-ware.gitbook.io/cyberdrop-dl/reference/supported-websites#supported-sites) The name of the crawer is the title of their section in the page (in bold).

```yaml
crawlers:
  disabled: []
```

## `generic`

Generic crawlers are designed to work on any site that uses a **specific** framework. Users can supply a list of sites to map to these crawlers, and CDL will then be able to download from them. The URL in the list should be the primary URL of the site. ex: `https://forums.docker.com/`

Supported generic crawlers:

* `chevereto`: This works on any site that uses [Chevereto](https://chevereto.com//).
* `discourse`: This works on any forum that uses [Discourse](https://www.discourse.org/).
* `kvs`: Works on any tube site using [Kernel Video Sharing](https://www.kernel-video-sharing.com/en/). Basically, any site that looks like one of these: <https://www.kernel-video-sharing.com/en/themes/>, ex: <https://www.kvs-demo.com/>
* `wordpress_media`: This crawler should work on any [WordPress](https://wordpress.com/) site where content primarily consists of images or galleries. The images need to be hosted on the site itself. It requires sites to have a public WordPress REST API.
* `wordpress_html`: This works on any WordPress site. It scrapes the actual HTML of the site, which means it works on sites that have embedded third-party media like videos or links to hosting sites. It is always slower than `wordpress_media`.

### `chevereto`

| Type            | Default |
| --------------- | ------- |
| `list[HttpURL]` | `[]`    |

### `discourse`

| Type            | Default |
| --------------- | ------- |
| `list[HttpURL]` | `[]`    |

### `kvs`

| Type            | Default |
| --------------- | ------- |
| `list[HttpURL]` | `[]`    |

### `wordpress_media`

| Type            | Default |
| --------------- | ------- |
| `list[HttpURL]` | `[]`    |

### `wordpress_html`

| Type            | Default |
| --------------- | ------- |
| `list[HttpURL]` | `[]`    |

```yaml
crawlers:
  generic:
    chevereto: []
    discourse: []
    kvs: []
    wordpress_html: []
    wordpress_media: []
```

## Bandcamp

### `formats`

| Type                                                                          | Default                                                                |
| ----------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| List of `mp3-320`, `mp3`, `aac-hi`, `wav`, `flac`, `vorbis`, `aiff` or `alas` | \[`mp3-320`, `mp3`, `aac-hi`, `wav`, `flac`, `vorbis`, `aiff`, `alas`] |

Format to choose for downloads (if available), ordered by preference.

```yaml
crawlers:
  bandcamp:
    formats:
      - mp3-320
      - mp3
      - aac-hi
      - wav
      - flac
      - vorbis
      - aiff
      - alas
```

## Clyp.it

### `prefer_mp3`

| Type   | Default |
| ------ | ------- |
| `Bool` | `false` |

Download audios as `.mp3` files even if WAV (high quality) versions are available

## OnePace

### `prefer_dub`

| Type   | Default |
| ------ | ------- |
| `Bool` | `false` |

Download episodes with english audio tracks instead of japanese (if available)

## Pawchive

### `file`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Download the main file in a post (if any)

### `attachments`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Download all attachments in a post (may or may not include `file`)

### `content_urls`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Scan the description (text) in a post and download any URL found

{% hint style="warning" %}
This option slows down scraping significally. The default response from a search query does not return the content of each post. `cyberdrop-dl` needs to make an additional request for each post to get its content (50x requests in total).
{% endhint %}

### `embed`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Download the embedded file from third party sites (if any)(mega.nz, pcloud, dropbox, etc..)

Embedded files show up as clickable boxes on the website:

![embed preview](https://raw.githubusercontent.com/Cyberdrop-DL/cyberdrop-dl/refs/heads/main/docs/assets/pawchive_embed.png)

```yaml
crawlers:
  pawchive:
    attachments: true
    content_urls: true
    embed: true
    file: true
```

## Tiktok

```yaml
crawlers:
  tiktok:
    original: false
```

### `original`

| Type   | Default |
| ------ | ------- |
| `Bool` | `false` |

By default, CDL will download the "optimized for streaming" version of tiktok videos. Setting this option to `True` will download videos in original (source) quality.

`_original` will be added as a suffix to their filename.

{% hint style="warning" %}
This will make video downloads several times slower

When it is set to `false` (the default) CDL can download 50 videos with a single request. When it is set to `true` , CDL needs to make at least 3 requests *per* video to download them.

There's also a daily limit of the API CDL uses: 5000 requests per day per IP

Setting this option to `true` will consume the daily limit faster
{% endhint %}

## Twitter

```yaml
crawlers:
  twitter:
    articles:
      cover: true
      media: true
    cards: true
    content_urls: true
    image_size: orig
    retweets: false
    threads: true
```

### `cards`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Parse and download cards in a post (embeds from third-party sites)

### `content_urls`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Parse and try to download any URL found inside the text of a tweet

### `image_size`

| Type                                                     | Default |
| -------------------------------------------------------- | ------- |
| `orig`, `4096x4096`, `large`, `medium`, `small`, `thumb` | `orig`  |

Resolution used for image downloads

{% hint style="info" %}
`orig` is original quality but it's not always available. You may get `404 - Not Found` or `403 - Forbidden` errors. The same applies to `4096x4096`.

`large`, `medium`, or `small` are always available.
{% endhint %}

### `threads`

| Type   | Default |
| ------ | ------- |
| `Bool` | `true`  |

Downloads media from all direct replies of the tweet's author to their own tweet

### `retweets`

| Type   | Default |
| ------ | ------- |
| `Bool` | `false` |

Download media from retweets in the user's timeline

### `articles`

Controls which content to download when CDL finds an article (longer tweets from premium users)

{% hint style="info" %}
Scanning the content (text) of articles for URLs is not supported
{% endhint %}

#### `cover`

Download the cover image of articles

#### `media`

Download media files (attachments) in the article