Skip to content

Backend & crawler for the OSS catalog of Developers Italia

License

Notifications You must be signed in to change notification settings

bfabio/publiccode-crawler

 
 

Repository files navigation

publiccode.yml crawler for the software catalog of Developers Italia

Go Report Card Join the #publiccode channel Get invited

Description

Developers Italia provides a catalog of Free and Open Source software aimed to Public Administrations.

This crawler retrieves the publiccode.yml files from the repositories of publishers found in the Developers Italia API.

Setup and deployment processes

The crawler can either run manually on the target machine or it can be deployed from a Docker container.

Manually configure and build the crawler

  1. Rename config.toml.example to config.toml and set the variables

    NOTE: The application also supports environment variables in substitution to config.toml file. Remember: "environment variables get higher priority than the ones in configuration file"

  2. Build the binary with go build

Docker

You can build the Docker image using

docker build .

or use the image published to DockerHub:

docker run -it italia/publiccode-crawler

Commands

crawler crawl

Gets the list of publishers from https://api.developers.italia.it/v1/publishers and starts to crawl their repositories.

crawler crawl publishers*.yml

Gets the list of publishers in publishers*.yml and starts to crawl their repositories.

Other commands

  • crawler download-publishers downloads organizations and repositories from the onboarding portal repository and saves them to a publishers YAML file.

See also

Authors

Developers Italia is a project by AgID and the Italian Digital Team, which developed the crawler and maintains this repository.

About

Backend & crawler for the OSS catalog of Developers Italia

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 99.3%
  • Dockerfile 0.7%