Component Overview
The scanners aim to provide a cloud-native and platform-agnostic solution that can be extended easily and scaled to handle any workload or traffic volume.
When the "Scan Queue" Knative service receives an inbound request from the API containing a valid payload, the request will be parsed, enqueued, and finally dispatched to the designated scanner(s) corresponding to the scan request. In short, it is the Scan Queue's job to determine where to direct scan requests.
After successful scanning, each individual "Scanner" Knative service will then send a request to the Result Queue containing the raw scan data. This result processing request is then enqueued before being dispatched to the Result Processor service, which is responsible for the parsing and evaluation of scan results in order to generate all applicable guidance tags. These guidance tags will link to related corrective guidance on improving/ensuring policy compliance when inspected from the scan detail page.
The technology stack in use consists of:
Knative Serving - Cloud-native scanning services with configurable autoscaling
Gunicorn - Python WSGI server running on scanner deployments
Uvicorn - ASGI worker class providing asynchronous computation ability for scanner services
Flask - WSGI framework in use by queue services
Redis - In-memory data structure store
RQ - Python library supporting the worker-handled queueing and processing of jobs
HTTPS scanning functionality adapted and extended from pshtt
sslyze - TLS/SSL scanning library
checkdmarc - SPF/DMARC DNS record validator
dkimpy - DKIM (DomainKeys Identified Mail) validation
Logging stack consists of 4 components:
Prometheus - Metric exporting
Promtail - Log collection
Loki - Log aggregation and storage
Grafana - Fully configurable monitoring dashboard
The Tracker API is exclusively focused on serving data, rather than HTML. It is a GraphQL API, chosen because of its composability, legibility and for the way it enables both security and security automation. It is built with the Express webserver using the express-graphql middleware, and follows the Relay specifications for pagination.
