Tools to compute multiple indexes on SWH's raw contents:
- content:
- mimetype
- fossology-license
- metadata
- origin:
- metadata (intrinsic, using the content indexer; and extrinsic)
An indexer is in charge of:
- looking up objects
- extracting information from those objects
- store those information in the swh-indexer db
There are multiple indexers working on different object types:
- content indexer: works with content sha1 hashes
- revision indexer: works with revision sha1 hashes
- origin indexer: works with origin identifiers
Indexation procedure:
- receive batch of ids
- retrieve the associated data depending on object type
- compute for that object some index
- store the result to swh's storage
Current content indexers:
- mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype
- fossology-license (queue swh_indexer_fossology_license): compute the license
- metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta vocabulary)
Current origin indexers:
- metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta and ForgeFed vocabularies)
Custom indexers and metadata mappings can be added as plugins.