Skip to main content

Repository Metadata System - Complete Guide

Overview​

The Repository Metadata System provides comprehensive tracking and management of repository metadata including read/write permissions, clone status, version information, synchronization dates, and much more. This system creates a structured database of all repository information for intelligent management and automation.

Features​

🏗️ Comprehensive Metadata Tracking​

  • Access & Permissions: Read/write status, admin rights, private/public status
  • Clone & Sync Status: Clone dates, sync status, local repository information
  • Version Information: Branches, tags, releases, commit tracking
  • Statistics: Stars, forks, watchers, issues, repository size, activity
  • Local Repository Info: Git status, uncommitted changes, branch information
  • Custom Fields: Tags, priority, category, notes for organization

🔧 GitHub API Integration​

  • Automatic Data Fetching: Real-time repository information from GitHub
  • Permission Detection: Automatic access level determination
  • Statistics Updates: Live stats including stars, forks, issues
  • Language Analysis: Repository language breakdown
  • Activity Tracking: Last activity and update timestamps

💻 Command Line Interface​

  • Metadata Management: Update, show, and manage repository metadata
  • Comprehensive Reporting: Detailed reports with filtering and analysis
  • Sync Status Monitoring: Track repository synchronization status
  • Cleanup Tools: Remove metadata for non-existent repositories

Metadata Structure​

Core Repository Information​

{
  "full_name": "docxology/codomyrmex",
  "owner": "docxology",
  "name": "codomyrmex",
  "repo_type": "OWN",
  "url": "https://github.com/docxology/codomyrmex.git",
  "clone_url": "https://github.com/docxology/codomyrmex.git",
  "description": "Repository description"
}

Access & Permissions​

{
  "access_level": "read_write",
  "is_private": false,
  "is_fork": false,
  "can_push": true,
  "can_admin": false
}

Clone & Sync Information​

{
  "clone_status": "cloned",
  "sync_status": "up_to_date",
  "local_path": "/Users/user/repos/project",
  "clone_date": "2025-08-29T14:30:00Z",
  "last_sync_date": "2025-08-29T14:35:00Z",
  "last_fetch_date": "2025-08-29T14:35:00Z"
}

Version Information​

{
  "default_branch": "main",
  "current_local_branch": "feature/new-feature",
  "latest_remote_commit": "abc123...",
  "latest_local_commit": "def456...",
  "version_tags": ["v1.0.0", "v1.1.0"],
  "latest_release": "v1.1.0"
}

Repository Statistics​

{
  "total_commits": 150,
  "contributors": 5,
  "stars": 25,
  "forks": 8,
  "watchers": 12,
  "issues": 3,
  "pull_requests": 2,
  "size_kb": 1755,
  "languages": {"Python": 85, "JavaScript": 15},
  "last_activity": "2025-08-29T14:14:15Z"
}

Local Repository Information​

{
  "path": "/Users/user/repos/project",
  "exists": true,
  "is_git_repo": true,
  "current_branch": "main",
  "uncommitted_changes": true,
  "untracked_files": ["new_file.py"],
  "modified_files": ["existing_file.py"],
  "staged_files": ["ready_file.py"],
  "last_commit_hash": "c70676d4",
  "last_commit_date": "2025-08-29T07:14:09",
  "last_commit_message": "Add new feature",
  "total_local_commits": 150
}

Custom Fields​

{
  "tags": ["important", "active-development"],
  "notes": "Primary development repository",
  "priority": 1,
  "category": "core-project"
}

Command Line Interface​

Installation & Setup​

# The metadata CLI is available at:
python src/codomyrmex/git_operations/metadata_cli.py

# Or create an alias for convenience:
alias metadata-cli="python /path/to/codomyrmex/src/codomyrmex/git_operations/metadata_cli.py"

Basic Commands​

1. Update Metadata​

# Update single repository
python metadata_cli.py update --repository docxology/codomyrmex --type OWN --path /local/path

# Update from repository library
python metadata_cli.py update --from-library repository_library.txt

# Update with GitHub token for enhanced data
python metadata_cli.py --token YOUR_GITHUB_TOKEN update --repository docxology/codomyrmex

2. Show Metadata​

# Show specific repository
python metadata_cli.py show --repository docxology/codomyrmex

# Show all repositories summary
python metadata_cli.py show

# Show with verbose details
python metadata_cli.py -v show

3. Generate Reports​

# Basic report
python metadata_cli.py report

# Detailed report with analysis
python metadata_cli.py report --detailed

# Export report to JSON
python metadata_cli.py report --detailed --export report.json

4. Check Sync Status​

# Check synchronization status
python metadata_cli.py sync-status

# Verbose sync status with details
python metadata_cli.py -v sync-status

5. Cleanup Metadata​

# Dry run to see what would be removed
python metadata_cli.py cleanup --dry-run

# Actually remove metadata for non-existent repositories
python metadata_cli.py cleanup

Python API Usage​

Basic Metadata Management​

from codomyrmex.git_operations.core.metadata import (
    RepositoryMetadataManager, AccessLevel, CloneStatus
)

# Initialize manager
manager = RepositoryMetadataManager(
    metadata_file="custom_metadata.json",
    github_token="your_github_token"
)

# Create or update metadata
metadata = manager.create_or_update_metadata(
    full_name="docxology/codomyrmex",
    owner="docxology",
    name="codomyrmex",
    repo_type="OWN",
    url="https://github.com/docxology/codomyrmex.git",
    description="Main project repository",
    local_path="/Users/user/repos/codomyrmex"
)

# Save metadata
manager.save_metadata()

Querying Metadata​

# Get specific repository metadata
metadata = manager.get_repository_metadata("docxology/codomyrmex")
if metadata:
    print(f"Clone Status: {metadata.clone_status.value}")
    print(f"Stars: {metadata.stats.stars}")
    print(f"Last Activity: {metadata.stats.last_activity}")

# Get repositories by status
cloned_repos = manager.get_repositories_by_status(CloneStatus.CLONED)
print(f"Cloned repositories: {len(cloned_repos)}")

# Get repositories by access level
writable_repos = manager.get_repositories_by_access(AccessLevel.READ_WRITE)
print(f"Writable repositories: {len(writable_repos)}")

# Get outdated repositories
outdated = manager.get_outdated_repositories(days=30)
print(f"Repositories not synced in 30 days: {len(outdated)}")

Bulk Operations​

# Bulk update from repository list
repositories = [
    {
        'owner': 'docxology',
        'name': 'codomyrmex',
        'type': 'OWN',
        'url': 'https://github.com/docxology/codomyrmex.git',
        'description': 'Main project',
        'local_path': 'docxology/codomyrmex'
    },
    # ... more repositories
]

results = manager.bulk_update_metadata(repositories, "/Users/user/repos")
successful = sum(1 for success in results.values() if success)
print(f"Updated {successful}/{len(results)} repositories")

Generating Reports​

# Generate comprehensive report
report = manager.generate_metadata_report()

print(f"Total repositories: {report['total_repositories']}")
print(f"Clone status breakdown: {report['status_breakdown']}")
print(f"Access level breakdown: {report['access_breakdown']}")
print(f"Total stars: {report['total_stars']}")
print(f"Outdated repositories: {report['outdated_repositories']}")

Integration with Repository Manager​

Enhanced Repository Manager​

from codomyrmex.git_operations.core.repository import RepositoryManager

# Initialize with metadata support
manager = RepositoryManager(
    library_file="repository_library.txt",
    base_path="/Users/user/repos",
    metadata_file="metadata.json",
    github_token="your_github_token"
)

# Clone repository with automatic metadata tracking
success = manager.clone_repository("docxology/codomyrmex")
# Metadata is automatically created and updated with:
# - Clone date and status
# - Local repository information
# - GitHub statistics
# - Access permissions

Automatic Metadata Updates​

# The repository manager automatically:
# 1. Creates metadata when cloning repositories
# 2. Updates clone status and dates
# 3. Tracks local repository information
# 4. Fetches GitHub statistics and permissions
# 5. Saves metadata after operations

# Example workflow
repos_to_clone = ["docxology/codomyrmex", "docxology/docxology"]
for repo_name in repos_to_clone:
    success = manager.clone_repository(repo_name)
    if success:
        # Metadata automatically updated with:
        # - clone_date
        # - clone_status = CLONED
        # - local_info (branch, commits, etc.)
        # - GitHub stats (stars, forks, etc.)
        print(f"✅ Cloned {repo_name} with metadata tracking")

Metadata Fields Reference​

Repository Types​

  • OWN: Your original repositories for development
  • FORK: Forked repositories for contributions
  • USE: External repositories for reference/usage

Access Levels​

  • READ_ONLY: Can read repository content
  • READ_WRITE: Can read and push changes
  • ADMIN: Full administrative access
  • UNKNOWN: Access level not determined

Clone Status​

  • NOT_CLONED: Repository not cloned locally
  • CLONED: Repository successfully cloned
  • OUTDATED: Local copy needs updating
  • ERROR: Error during clone/sync operations
  • UNKNOWN: Status not determined

Sync Status​

  • UP_TO_DATE: Local and remote are synchronized
  • AHEAD: Local has commits not pushed to remote
  • BEHIND: Remote has commits not pulled locally
  • DIVERGED: Local and remote have different commits
  • UNKNOWN: Sync status not determined

Advanced Usage Examples​

Repository Health Monitoring​

def monitor_repository_health(manager):
    """Monitor repository health and generate alerts."""

    # Check for repositories with uncommitted changes
    all_repos = list(manager.metadata.values())
    uncommitted = [r for r in all_repos if r.local_info.uncommitted_changes]

    if uncommitted:
        print("⚠️ Repositories with uncommitted changes:")
        for repo in uncommitted:
            print(f"   {repo.full_name}: {len(repo.local_info.modified_files)} modified files")

    # Check for outdated repositories
    outdated = manager.get_outdated_repositories(7)  # 7 days
    if outdated:
        print("📅 Repositories not synced in 7 days:")
        for repo in outdated:
            print(f"   {repo.full_name}: {repo.last_sync_date or 'Never'}")

    # Check for repositories with issues
    issues = [r for r in all_repos if r.stats.issues > 0]
    if issues:
        print("🐛 Repositories with open issues:")
        for repo in issues:
            print(f"   {repo.full_name}: {repo.stats.issues} issues")

# Usage
monitor_repository_health(manager)

Development Workflow Integration​

def development_workflow_status(manager):
    """Check development workflow status."""

    # Get development repositories
    dev_repos = [r for r in manager.metadata.values() if r.repo_type == "OWN"]

    print(f"📊 Development Repository Status ({len(dev_repos)} repos)")
    print("-" * 50)

    for repo in dev_repos:
        status_icon = "✅" if repo.clone_status.value == "cloned" else "❌"
        changes_icon = "📝" if repo.local_info.uncommitted_changes else "🔒"

        print(f"{status_icon} {changes_icon} {repo.full_name}")
        print(f"   Branch: {repo.local_info.current_branch}")
        print(f"   Last Commit: {repo.local_info.last_commit_date}")
        print(f"   Stars: {repo.stats.stars}, Forks: {repo.stats.forks}")

        if repo.local_info.uncommitted_changes:
            print(f"   ⚠️ {len(repo.local_info.modified_files)} modified, "
                  f"{len(repo.local_info.untracked_files)} untracked files")
        print()

# Usage
development_workflow_status(manager)

Automated Reporting​

def generate_weekly_report(manager):
    """Generate weekly repository activity report."""

    from datetime import datetime, timedelta

    # Get repositories active in the last week
    week_ago = datetime.now() - timedelta(days=7)

    active_repos = []
    for repo in manager.metadata.values():
        if repo.stats.last_activity:
            try:
                last_activity = datetime.fromisoformat(
                    repo.stats.last_activity.replace('Z', '+00:00')
                )
                if last_activity > week_ago:
                    active_repos.append((repo, last_activity))
            except ValueError:
                continue

    # Sort by activity
    active_repos.sort(key=lambda x: x[1], reverse=True)

    print("📊 WEEKLY REPOSITORY ACTIVITY REPORT")
    print("=" * 50)
    print(f"Report Period: {week_ago.strftime('%Y-%m-%d')} to {datetime.now().strftime('%Y-%m-%d')}")
    print(f"Active Repositories: {len(active_repos)}")
    print()

    for repo, activity_date in active_repos[:10]:  # Top 10
        print(f"🔥 {repo.full_name}")
        print(f"   Last Activity: {activity_date.strftime('%Y-%m-%d %H:%M')}")
        print(f"   Type: {repo.repo_type}")
        print(f"   Stars: {repo.stats.stars}, Issues: {repo.stats.issues}")
        print()

# Usage
generate_weekly_report(manager)

Best Practices​

1. Regular Metadata Updates​

# Set up a cron job or scheduled task to update metadata regularly
# Daily update of all repositories
python metadata_cli.py update --from-library repository_library.txt

# Weekly detailed report
python metadata_cli.py report --detailed --export weekly_report.json

2. GitHub Token Usage​

# Use GitHub token for enhanced metadata and higher API limits
export GITHUB_TOKEN="your_personal_access_token"
python metadata_cli.py --token $GITHUB_TOKEN update --from-library library.txt

3. Metadata Organization​

# Use custom fields for organization
metadata.tags = ["important", "active-development", "python"]
metadata.priority = 1  # 0=normal, 1=high, 2=critical
metadata.category = "core-project"
metadata.notes = "Primary development repository with CI/CD"

4. Monitoring and Alerts​

# Set up monitoring for important repositories
def check_critical_repos(manager):
    critical_repos = [r for r in manager.metadata.values() if r.priority >= 1]

    for repo in critical_repos:
        if repo.local_info.uncommitted_changes:
            print(f"🚨 CRITICAL: {repo.full_name} has uncommitted changes")

        if not repo.last_sync_date:
            print(f"⚠️ WARNING: {repo.full_name} never synced")

5. Backup and Recovery​

# Regular backup of metadata
cp repository_metadata.json repository_metadata.backup.$(date +%Y%m%d)

# Recovery from backup
cp repository_metadata.backup.20250829 repository_metadata.json

Troubleshooting​

Common Issues​

1. Metadata Not Saving​

# Check if metadata manager is properly initialized
manager = RepositoryMetadataManager()
print(f"Metadata file: {manager.metadata_file}")
print(f"File exists: {manager.metadata_file.exists()}")

# Manually save metadata
success = manager.save_metadata()
print(f"Save successful: {success}")

2. GitHub API Rate Limits​

# Use GitHub token to increase rate limits
python metadata_cli.py --token YOUR_TOKEN update --repository owner/repo

# Check rate limit status
curl -H "Authorization: token YOUR_TOKEN" https://api.github.com/rate_limit

3. Local Repository Detection Issues​

# Check local repository information
metadata = manager.get_repository_metadata("owner/repo")
if metadata:
    print(f"Local path: {metadata.local_path}")
    print(f"Path exists: {Path(metadata.local_path).exists()}")
    print(f"Is git repo: {metadata.local_info.is_git_repo}")

4. Permission Issues​

# Check file permissions
ls -la repository_metadata.json

# Fix permissions if needed
chmod 644 repository_metadata.json

Integration Examples​

With CI/CD Pipelines​

# GitHub Actions example
name: Update Repository Metadata
on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM

jobs:
  update-metadata:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Update Metadata
        run: |
          python metadata_cli.py --token ${{ secrets.GITHUB_TOKEN }} \
            update --from-library repository_library.txt
          python metadata_cli.py report --export metadata_report.json
      - name: Upload Report
        uses: actions/upload-artifact@v2
        with:
          name: metadata-report
          path: metadata_report.json

With Development Scripts​

#!/usr/bin/env python3
"""Development environment status checker."""

from codomyrmex.git_operations.core.metadata import RepositoryMetadataManager

def main():
    manager = RepositoryMetadataManager()

    # Check development environment status
    dev_repos = [r for r in manager.metadata.values() if r.repo_type == "OWN"]

    print("🔧 Development Environment Status")
    print("=" * 40)

    for repo in dev_repos:
        if repo.clone_status.value == "cloned":
            if repo.local_info.uncommitted_changes:
                print(f"⚠️ {repo.full_name}: Has uncommitted changes")
            else:
                print(f"✅ {repo.full_name}: Clean")
        else:
            print(f"❌ {repo.full_name}: Not cloned")

if __name__ == "__main__":
    main()

This comprehensive metadata system provides complete visibility and control over your repository ecosystem, enabling intelligent automation and informed decision-making for your development workflow!