Integrating CrowdSec with Traefik & Discord

Update

After chatting with one of the guys from CrowdSec I've updated this post - turns out you can do string manipulation, they're just using a different library to the one I was expecting to be there.

Also I fixed the Discord embed colours - turns out they're int not hex. Oops.

Also also, I've found a bug with the Traefik bouncer where it doesn't process X-Forwarded-For headers correctly so if you've got an upstream proxy like Cloudflare it won't apply the bans as expected.

Introduction

CrowdSec is a free, open-source and collaborative IPS; it's like Fail2Ban but you share your bans with all of the other users to try and preemptively block malicious hosts.

As is something of a habit with me, I decided to try out a product only to realise it has awful documentation - in this instance lacking a lot of information around the docker implementation. Hopefully this post will make things easier for you than they were for me.

Getting Started

You can install CrowdSec natively on your host but that's not very on-brand so let's start with a simple compose. Mine is going to cover Traefik and sshd:

services:
  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - GID=1000
    volumes:
      - config:/etc/crowdsec
      - db:/var/lib/crowdsec/data/
      - /var/lib/docker/volumes/ssh/_data/logs/openssh/:/var/log/opensshd/:ro
      - /var/lib/docker/volumes/traefik/_data/logs/:/var/log/traefik/:ro
    networks:
      - proxy
    restart: unless-stopped
    
networks:
  proxy:
    external: true

volumes:
  config:
  db:    

Nothing out of the ordinary, I'm using docker volumes but you can use bind mounts if you prefer. Make sure the GID has permissions to read all the logs you want to mount, and mount them all into /var/log/<some folder>. Spin up the container and it should create all the basics for you.

Go find acquis.yml in your config mount and edit it. Here you need to label your logs so CrowdSec knows what it's looking at. Something like:

filenames:
  - /var/log/opensshd/*
labels:
  type: sshd
---
filenames: 
  - /var/log/traefik/*
labels:
  type: traefik

Restart the container afterwards to have it pick up the new config.

Now by default CrowdSec only ships with a couple of rules (what they call Scenarios) for ssh along with log parsers (Parsers) for a few basic types and nothing else. New components can be installed from their Hub either individually or as Collections - and it's a Collection we're going to start with.

The basic interface to everything in the container is their cscli tool, which you'll see if you run docker exec crowdsec cscli scenarios list or docker exec crowdsec cscli parsers list or, indeed, docker exec crowdsec cscli collections list

We'll go ahead and install the Traefik Collection, which includes a Parser for the logs and a set of common http Scenarios. docker exec crowdsec cscli collections install crowdsecurity/traefik

You can also install Collections, Scenarios, and Parsers using environment variables in your compose. For example

      - COLLECTIONS="crowdsecurity/linux crowdsecurity/traefik crowdsecurity/wordpress crowdsecurity/sshd"

Would install the linux, traefik, wordpress, and sshd collections on startup.

Side Note On Updates

While updating the container will update the core CrowdSec application, it won't update the Parsers/Scenarios/etc. To do that you need to periodically run docker exec crowdsec cscli hub update and then docker exec crowdsec cscli hub upgrade. As far as I can tell so far, there's no way to have this run automatically.

At this point CrowdSec should be reading your logs and processing them, you can have a look at what's going on with 2 separate commands. docker exec crowdsec cscli metrics will give you summary details on everything that CrowdSec is doing, and docker exec crowdsec cscli decisions list will show you any bans that have been created.

The most important bit of the metrics output initially is the Acquisition Metrics which will hopefully look something like:

time="11-01-2022 10:10:26 AM" level=info msg="Acquisition Metrics:"
+----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE              | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+----------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/opensshd/current   |        805 |           89 |            716 |                    249 |
| file:/var/log/traefik/access.log |      42941 |        42937 |              4 |                   8665 |
+----------------------------------+------------+--------------+----------------+------------------------+

If you don't see it at all, it means no logs have been parsed from any source and you may need to check the container logs to see if anything stands out. If you're seeing logs being parsed, buckets filling, and decisions being made, then everything is working as intended.

If Your Name's Not Down

OK, so CrowdSec is reading your logs and adding suspect IPs to its ban list. Cool. But that's not actually doing anything to stop them connecting at this point. CrowdSec uses Bouncers to do this. There are lots of them. There's a Firewall Bouncer for iptables/nftables, there's a Cloudflare Bouncer for the Cloudflare Firewall, and there's a Traefik Bouncer which is what we're interested in right now.

So, let's add the bouncer to our compose:

  bouncer-traefik:
    image: docker.io/fbonalair/traefik-crowdsec-bouncer:latest
    container_name: crowdsec-bouncer-traefik
    environment:
      CROWDSEC_BOUNCER_API_KEY: ${TRAEFIK_BOUNCER_KEY}
      CROWDSEC_AGENT_HOST: crowdsec:8080
    networks:
      - proxy
    depends_on:
      - crowdsec
    restart: unless-stopped

This needs to be on a common network with both your CrowdSec container and your Traefik container.

Register the bouncer with CrowdSec to get an API key with docker exec crowdsec cscli bouncers add bouncer-traefikand add the key to your bouncer container compose/.env file - don't lose it as you can't get it again without deleting and re-adding the bouncer.

Now you can do the blocking per-container but we're going to take the nuclear option and apply it across the board because we don't want any of these suspect users hitting our services.

Create a new yml file (or add to an existing one) where ever you keep your dynamic configs and define a new middleware:

http:
  middlewares:
    middleware-crowdsec-bouncer:
      forwardauth:
        address: http://crowdsec-bouncer-traefik:8080/api/v1/forwardAuth
        trustForwardHeader: true  

And then in your static config, add the middleware to your entryPoints

entryPoints:
  http:
    address: ":80"
    http:
      middlewares:
        - [email protected]

  https:
    address: ":443"
    http:    
      middlewares:
        - [email protected]        
      tls: {}

Restart Traefik and now any inbound request will get sent to the Bouncer to check if it's on the banlist or not. Anything naughty gets a 403 and told to go away.

Alerting

Now we've got our detection and blocking in place there's one more thing to do and that's alerting. CrowdSec offers a few alerting configs out of the box but Discord isn't one of them. You can just use the Slack notification and append /slack to the end of your Discord webhook but it doesn't look very pretty. If you want a "proper" Discord notifier you need to work for it.

Head to your CrowdSec config directory and edit profiles.yaml. Uncomment the notifications: and - http_default lines and sort out the indentation so it looks something like this:

notifications:
 - http_default

And then find the http.yaml under the notifications subdirectory. This needs a few changes making to get it working the way we want. First up, uncomment the headers: section and add

headers:
  Content-Type: application/json

Then set the url: to your Discord webhook. Finally, replace the format: section with something like this:

format: |  
  {
    "content": null,
    "embeds": [
      {{range . -}}
      {{$alert := . -}}
      {{range .Decisions -}}
      {
        "title": "{{.Scenario}}",
        "description": ":no_entry_sign: {{$alert.Source.IP}} will get {{.Type}} for next {{.Duration}}. <https://www.shodan.io/host/{{$alert.Source.IP}}>",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": 16711680,
        "image": {
          "url": "https://www.mapquestapi.com/staticmap/v5/map?center={{$alert.Source.Latitude}},{{$alert.Source.Longitude}}&size=500,300&key=<API_KEY>"
        }
      }
      {{end -}}
      {{end -}}
    ]
  }

Again, you need to restart the container for CrowdSec to pick up the updated config.

This will create a Discord embed message with details of the IP that's been banned, a link to the db-ip.com page for the IP, a link to the Shodan page on the IP and an image from MapQuest showing you where in the world your malicious actor is. If you want to use the MapQuest integration you need to register for an API key and add it to the config; they give you 15k free hits a month which should be enough most of the time.

I would like to be able to have :flag_{{$alert.Source.Cn}}: in the embed but unfortunately the country codes output by CrowdSec are in CAPITALS and Discord only accepts them in lower case for the emoji and the Go plugin running the notifications doesn't have the strings module so you can't do string conversions. It may end up annoying me enough that I submit a PR to add it.

You can do flags, it turns out, I was just trying to use the wrong strings library. So if you want pretty country flags in your notifications try something like:

format: |
  {
    "content": null,
    "embeds": [
      {{range . -}}
      {{$alert := . -}}
      {{range .Decisions -}}
      {{if $alert.Source.Cn -}}
      {
        "title": "{{.Scenario}}",
        "description": ":flag_{{ $alert.Source.Cn | lower }}: {{$alert.Source.IP}} will get {{.Type}} for next {{.Duration}}. <https://www.shodan.io/host/{{$alert.Source.IP}}>",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": 16711680,
        "image": {
          "url": "https://www.mapquestapi.com/staticmap/v5/map?center={{$alert.Source.Latitude}},{{$alert.Source.Longitude}}&size=500,300&key=<API_KEY>"
        }
      }
      {{end}}
      {{if not $alert.Source.Cn -}}
      {
        "title": "{{.Scenario}}",
        "description": ":pirate_flag: {{$alert.Source.IP}} will get {{.Type}} for next {{.Duration}}. <https://www.shodan.io/host/{{$alert.Source.IP}}>",
        "url": "https://db-ip.com/{{$alert.Source.IP}}",
        "color": 16711680
      }
      {{end}}
      {{end -}}
      {{end -}}
    ]
  }

This is basically the same as the first example only with an IF statement that says if there's a country value for the IP then use that country's flag emoji, otherwise Pirate Flag. Also no map in the latter instance as if there's no country data there's no lat/lon data to use for a map either.

If all goes to plan you should start getting Discord notifications a bit like this (this particular screenshot is from when I was still messing around trying to get flags working):

disc

CrowdSec do also offer a Metabase Dashboard for keeping track of what it's up to. Personally I wouldn't bother as it uses an absurd amount of CPU and RAM to show you some pretty graphs. If you really want pretties there's a Prometheus endpoint you can use to grab data and throw it at a Grafana Dashboard. They even provide some though they're not great, I'll be honest.

CrowdSec also have a hosted dashboard offering that's currently in beta, you can register for it at https://app.crowdsec.net. It's definitely a good option if you're not allergic to The Cloud.

Conclusion

CrowdSec is an interesting product. It positions itself as a straight upgrade to Fail2Ban, giving you collaborative, community blocking of malicious hosts without having to write a load of regex. In reality though its multi-part nature with collectors and bouncers and alerting plugins makes it considerably more complex to set up the way you want to get a basically functional product.

For now I'm taking a hybrid approach, using Fail2Ban for failed auth blocking at a host level and then running CrowdSec against Traefik to watch for more specific exploit attempts against things like Wordpress or popular vulnerabilities like log4j. Whether I stick with it, or move to it entirely, remains to be seen.