Apache ManifoldCF

• Open-source framework for content crawling and ingestion.
• Provides a connector-based architecture for integrating multiple repositories.
• Supports crawling and indexing content from filesystems, websites, databases, and other repositories.
• Provides a web-based user interface for configuring connections and crawling jobs.
• Supports configurable crawling schedules and job priorities.
• Provides repository and output connector support for flexible content workflows.
• Supports document metadata and content processing during ingestion.
• Provides authority services for managing document access and permissions.
• Supports multiple deployment configurations, including single-process and multiprocess environments.
• Can use databases such as PostgreSQL and MySQL for production deployments.
• Includes Jetty-based example deployment for quick testing and evaluation.
• Provides logging and administrative tools for monitoring crawler activity.
• Supports extensibility through custom connectors and Java-based components.
• Designed for scalable enterprise content integration and search indexing workflows.

Apache ManifoldCF Usage:

# Navigate to the Apache ManifoldCF installation directory

$ cd /opt/manifoldcf/app

# Verify Java installation

$ java -version

# Check the Apache ManifoldCF release configured in the example

$ grep "org.apache.manifoldcf.versionstring" /opt/manifoldcf/app/example/properties.xml

# Expected output

value="2.30"

# Navigate to the example deployment

$ cd /opt/manifoldcf/app/example

# Make the example scripts executable

$ chmod +x *.sh

# Start Apache ManifoldCF

$ ./start.sh

# Expected messages include:

Jetty started.

Starting crawler...

# Check whether ManifoldCF is listening on port 8345

# Run this from another terminal session

$ ss -lntp | grep 8345

# Test the ManifoldCF server locally

$ curl -I http://127.0.0.1:8345

# Open the ManifoldCF crawler interface

http://YOUR_SERVER_IP:8345/mcf-crawler-ui/

# API service

http://YOUR_SERVER_IP:8345/mcf-api-service/

# Authority service

http://YOUR_SERVER_IP:8345/mcf-authority-service/

# Check running Java/ManifoldCF processes

$ ps aux | grep '[j]ava'

$ ps aux | grep '[m]anifold'

# Check listening ports

$ ss -lntp | grep 8345

# Stop Apache ManifoldCF

$ cd /opt/manifoldcf/app/example

$ ./stop.sh

# Check the status after stopping

$ ss -lntp | grep 8345

# View available ManifoldCF startup scripts

$ find /opt/manifoldcf/app -type f -name "*.sh" -print

# Check the configured database implementation

$ grep "databaseimplementationclass" /opt/manifoldcf/app/example/properties.xml

# The example deployment uses HSQLDB

# Check HSQLDB database files

$ ls -lah /opt/manifoldcf/app/example

# Check the configured ManifoldCF version

$ grep "org.apache.manifoldcf.versionstring" /opt/manifoldcf/app/example/properties.xml

Apache ManifoldCF Configuration:

# Main example configuration

/opt/manifoldcf/app/example/properties.xml

# Database configuration

/opt/manifoldcf/app/example/dbname.properties

# Jetty configuration

/opt/manifoldcf/app/example/jetty.xml

# Logging configuration

/opt/manifoldcf/app/example/logging.xml

# Connector configuration

/opt/manifoldcf/app/connectors.xml

# Web applications

/opt/manifoldcf/app/web/war/

Apache ManifoldCF Web Services:

# Crawler UI

http://YOUR_SERVER_IP:8345/mcf-crawler-ui/

# General API service

http://YOUR_SERVER_IP:8345/mcf-api-service/

# Authority service

http://YOUR_SERVER_IP:8345/mcf-authority-service/

Firewall Configuration:

# Allow ManifoldCF port through UFW

$ sudo ufw allow 8345/tcp

# Check firewall status

$ sudo ufw status

Disclaimer:

Apache ManifoldCF is an open-source content crawling and repository integration framework developed and maintained by the Apache Software Foundation and its community. It is not affiliated with or endorsed by any third-party search engine, cloud provider, database vendor, content management system, or software service mentioned for compatibility, integration, crawling, or indexing purposes.