Liga Magic Scraper

A Ruby gem for scraping product data from Liga Magic's card listings using Capybara and Selenium.

• 💎 Can be used as a gem or standalone CLI tool
• 🖥️ Supports headed (visible) and headless browser modes
• 📝 Saves results to JSON files in the scrapped/ directory

• Global Ligamagic product scrapping
• Store specific scrapping
• Alert System

• Extracts product name, ID, and pricing information
• Automatically clicks "Load More" to paginate through all products
• Stops when unavailable products are encountered
• Uses a global-search flag (-g or --global)

• Extracts product name and pricing information from any Liga Magic store
• Accepts store domain name only (e.g., test-store)
• Store search with search term support (-u STORE -s TERM)
• Store listing with max pages limit (-u STORE -p N)
• Automatic pagination with safety limits
• Automatically orders by price (most expensive first)
• Filters only in-stock items automatically
• Automatically builds clean URLs with only necessary parameters
• Uses --store or -u flag
• Extracts store name and ID automatically
• Corrected selectors for card-desc structure
• [!] Limitation: Store searches with search terms cannot extract price/qty due to CSS obfuscation (see docs)

• Detects changes between scrapes (new products, removed products, price changes, availability changes)
• Compares current scrape with most recent previous scrape
• Multiple alert types supported: file, telegram
• CLI flag to enable alerts: -a or --alerts
• Configurable alert types via --alert-types
• File alert fully implemented with organized directory structure
• Telegram alert action - currently stub only

• Ruby 2.7 or higher
• Chrome browser installed
• ChromeDriver (automatically managed by selenium-webdriver)

Install the gem globally to use the ligamagic-scraper command anywhere:

gem install ligamagic-scraper

Add this line to your application's Gemfile:

gem 'ligamagic-scraper'

And then execute:

bundle install

After installing the gem globally, you can use the ligamagic-scraper command from anywhere:

ligamagic-scraper -s "booster box"

Search Options:

• -s, --search SEARCH - Search term (global search or store search with -u)
• -u, --store DOMAIN - Store domain (e.g., test-store)
• -p, --pages N - Pages to scrape (max: 5, required with -u if no -s)

Options:

• -g, --global - Use global Liga Magic search (default, optional)
• -b, --browser-mode MODE - Browser mode: headed (default, visible browser) or headless (no UI)
• -a, --alerts - Enable alert system (detects changes from previous scrapes)
• --alert-types TYPES - Alert types: file, email, telegram, webhook (comma-separated, default: file)
• -h, --help - Show help message
• --version - Show version

Usage Examples:

Search Modes:

The gem supports three search modes:

Notes:

• Maximum 5 pages allowed for store listings (values above 5 are capped)
  • -p 2 uses 2 pages
  • -p 5 uses 5 pages
  • -p 10 is capped to 5 pages
• Store searches with search term have unlimited pagination
• Automatic filtering: Store results are automatically:
  • Ordered by price (most expensive first)
  • Filtered to show only in-stock items
• Limitation: Store searches with search terms (-u STORE -s TERM) cannot extract price/qty
  • Liga Magic uses CSS sprite obfuscation for anti-scraping protection
  • Only card name and ID are extracted when searching
  • Store listings (-u STORE -p N) work normally with full price/qty data
• Use -b headless for background scraping without browser UI

You can also import and use the scraper in your Ruby projects:

require 'ligamagic_scraper'

# Create a global scraper instance
scraper = LigaMagicScraper::GlobalScraper.new(
  search_term: "booster box",
  browser_mode: 'headless'
)

# Run the scrape
results = scraper.scrape

# Save results to JSON
scraper.save_to_json(results)

# Access the results
results.each do |product|
  puts "#{product[:name]}: R$ #{product[:min_price]}"
end

# Access execution logs
scraper.formatted_logs.each { |log| puts log }

require 'ligamagic_scraper'

# Create a store scraper instance for listing (with max pages)
scraper = LigaMagicScraper::StoreScraper.new(
  store_domain: "test-store",
  max_pages: 5,
  browser_mode: 'headless'
)

# Run the scrape
results = scraper.scrape

# Save results to JSON
scraper.save_to_json(results)

# Access the results
results.each do |product|
  puts "[#{product[:card_id]}] #{product[:name]}: R$ #{product[:price]} (#{product[:qtd]} available)"
end

# Access execution logs
scraper.formatted_logs.each { |log| puts log }

If you're developing or testing, you can run the executable directly from the project:

# With visible browser (default)
./bin/ligamagic-scraper -s "booster box"

# With headless mode
./bin/ligamagic-scraper -s "commander deck" -b headless

# Or with bundle exec
bundle exec ./bin/ligamagic-scraper -s "commander deck" -b headless

All scrapers collect execution logs that can be accessed programmatically:

scraper = LigaMagicScraper::GlobalScraper.new(search_term: "test")
results = scraper.scrape

# Access full log entries (with timestamp, level, source)
scraper.logs
# => [
#   { timestamp: 2025-10-27 12:00:00, level: :info, message: "🚀 Starting...", source: "LigaMagicScraper::GlobalScraper" },
#   ...
# ]

# Access just the messages (for display)
scraper.formatted_logs
# => ["🚀 Starting Liga Magic global search scraper...", "🔍 Search term: test", ...]

# Filter by level
error_logs = scraper.logs.select { |l| l[:level] == :error }

# Display logs
scraper.formatted_logs.each { |msg| puts msg }

Log Levels:

• :info - Major milestones (starting, completing, saving)
• :debug - Detailed progress (pagination clicks, individual product extraction)
• :warning - Non-fatal issues (skipped items, limits reached)
• :error - Errors and exceptions

Note: The CLI automatically displays all logs during execution. When using as a library, logs are collected silently and can be accessed via scraper.logs or scraper.formatted_logs

The scraper will:

1. Visit the Liga Magic search page with your search term
2. Automatically click "Load More" until all available products are loaded
3. Extract product names, IDs, slugs, and prices
4. Display results in a formatted output
5. Save results to a JSON file in scrapped/YYYYMMDD_search_slug.json
6. Return a Ruby array of hashes with the data

The global scraper returns an array of hashes with product details:

[
  {
    id: "48851",
    slug: "caixa_de_booster_amonkhet",
    name: "Caixa de Booster - Amonkhet",
    min_price: 1499.0,
    avg_price: 1499.0,
    max_price: 1499.0
  },
  {
    id: "80494",
    slug: "caixa_de_booster_dominaria",
    name: "Caixa de Booster - Dominaria",
    min_price: 1399.99,
    avg_price: 1482.66,
    max_price: 1499.95
  },
  # ...
]

Global Search Fields:

• id: Unique product code from Liga Magic (pcode), e.g., "48851" (can be null)
• slug: URL-friendly version of product name (normalized, no accents, always present)
• name: Full product name in Portuguese
• min_price: Minimum price across all stores (float)
• avg_price: Average price across all stores (float)
• max_price: Maximum price across all stores (float)

The store scraper returns an array with different fields:

[
  {
    card_id: "16149",
    name: "Volcanic Island",
    slug: "volcanic_island",
    price: 2500.0,
    qtd: 1,
    available: true
  },
  # ...
]

Store-Specific Fields:

• card_id: Unique card identifier from Liga Magic (extracted from link URL)
• name: Full product name
• slug: URL-friendly version of product name
• price: Product price at this store (float)
• qtd: Quantity available (integer)
• available: Whether the product is available (boolean based on qtd > 0)

Results are automatically saved to organized directories:

Global Searches:

• Directory: scrapped/global/
• Format: YYYYMMDD_HHMMSS__search_slug.json
• Examples:
  • scrapped/global/20251027_143022__booster_box.json
  • scrapped/global/20251027_151530__commander_deck.json

Store Searches:

• Directory: scrapped/stores/{store_domain}/
• Format (without search): YYYYMMDD_HHMMSS.json
• Format (with search): YYYYMMDD_HHMMSS__search_slug.json
• Examples:
  • scrapped/stores/test-store/20251027_143022.json (store listing)
  • scrapped/stores/test-store/20251027_151530__volcanic_island.json (store search)

The datetime format allows multiple scrapes per day, which is useful for price tracking and change detection.

JSON structure:

{
  "search_term": "booster box",
  "scraped_at": "2025-10-26T14:30:00-03:00",
  "total_products": 42,
  "products": [
    {
      "id": "48851",
      "slug": "caixa_de_booster_amonkhet",
      "name": "Caixa de Booster - Amonkhet",
      "min_price": 1499.0,
      "avg_price": 1499.0,
      "max_price": 1499.0
    }
  ]
}

The alert system automatically detects changes between scrapes and can notify you through multiple channels.

1. Enable alerts with the -a or --alerts flag
2. The system automatically finds the most recent previous scrape file
3. Compares the current scrape with the previous one
4. Detects changes:
   • 🆕 New products added
   • ❌ Products removed
   • 💰 Price changes (with percentage change)
   • 📊 Quantity changes (for store scraper)
   • 📦 Availability changes
5. Sends notifications via configured alert types

• file: Save alert to JSON file in alerts_json/ directory (implemented)
• telegram: Send Telegram message (stub - not implemented yet)

# Basic alert (uses file alert)
ligamagic-scraper -s "commander deck" -a

# Multiple alert types
ligamagic-scraper -u test-store -s "volcanic" -a --alert-types file,email,telegram

# Alerts with headless mode
ligamagic-scraper -s "booster box" -a -b headless

require 'ligamagic_scraper'

# Configure alerts
alert_config = {
  enabled: true,
  alert_types: [:file, :email],
  compare_previous: true
}

# Create scraper with alerts
scraper = LigaMagicScraper::GlobalScraper.new(
  search_term: "commander deck",
  alert_config: alert_config
)

results = scraper.scrape
scraper.save_to_json(results)  # Automatically processes alerts

{
  has_changes: true,
  timestamp: "2025-10-27T...",
  search_type: "global",
  search_term: "commander deck",
  new_products: [...],           # Products that weren't in previous scrape
  removed_products: [...],       # Products that are no longer available
  price_changes: [               # Products with price changes
    {
      id: "12345",
      name: "Product Name",
      previous_min: 100.0,
      current_min: 85.0,
      change: -15.0,
      change_percent: -15.0
    }
  ],
  quantity_changes: [            # Quantity changes (store scraper only)
    {
      id: "16149",
      name: "Volcanic Island",
      previous_qtd: 3,
      current_qtd: 1,
      change: -2
    }
  ],
  availability_changes: [...]    # Products that became available/unavailable
}

File Alert saves changes to organized directories:

Global Searches:

• Directory: alerts_json/global/
• Example: alerts_json/global/20251027_143022.json

Store Searches:

• Directory: alerts_json/stores/{store_domain}/
• Example: alerts_json/stores/kamm-store/20251027_143022.json

Each alert file contains full change details as JSON.

Example Alert File:

{
  "has_changes": true,
  "timestamp": "2025-10-27T14:30:22-03:00",
  "search_type": "global",
  "search_term": "commander deck",
  "new_products": [...],
  "removed_products": [...],
  "price_changes": [...],
  "quantity_changes": [...],
  "availability_changes": [...]
}

The scraper uses:

• Driver: Selenium Chrome (headed mode by default, headless mode available)
• Browser Modes:
  • headed (default): Visible browser window for monitoring
  • headless: No UI, ideal for servers and automation
• Wait Time: 10 seconds for dynamic content
• Base URL: https://www.ligamagic.com.br/?view=cards%2Fsearch&tipo=1
• Max Pagination: 50 clicks maximum (configurable via MAX_CLICKS constant)
• Output Directory: scrapped/ (created automatically)

After checking out the repo, run bundle install to install dependencies.

To build and install the gem manually from source:

gem build ligamagic-scraper.gemspec
gem install ./ligamagic-scraper-0.6.0.gem

To uninstall:

gem uninstall ligamagic-scraper

The gem includes automated Rake tasks for version management following Semantic Versioning.

The easiest way to release a new version is using the automated release tasks:

# For bug fixes (0.1.0 -> 0.1.1)
rake release_patch

# For new features (0.1.0 -> 0.2.0)
rake release_minor

# For breaking changes (0.1.0 -> 1.0.0)
rake release_major

These commands will automatically:

1. Bump the version number
2. Clean old gem files
3. Build the new gem
4. Install it locally

rake version
# Output: Current version: 0.1.0

If you just want to bump the version without building:

# Bump patch version (0.1.0 -> 0.1.1)
rake bump_patch

# Bump minor version (0.1.0 -> 0.2.0)
rake bump_minor

# Bump major version (0.1.0 -> 1.0.0)
rake bump_major

# Build and install locally (recommended)
rake install_local

# Just build (without installing)
rake build

# Clean old gem files
rake clean

# Uninstall the gem
rake uninstall

• Patch (X.X.1): Bug fixes, small tweaks, no new features
• Minor (X.1.0): New features, backward compatible changes
• Major (1.0.0): Breaking changes, major redesigns

# 1. Make your code changes
# 2. Update CHANGELOG.md with your changes
# 3. Release the new version
rake release_patch

# Output:
# 📈 Version bumped: 0.1.0 -> 0.1.1
# 🧹 Cleaning old gem files...
# 🔨 Building gem...
# 📦 Installing ligamagic-scraper-0.1.1.gem...
# ✅ Gem installed successfully!

To see all available tasks:

rake -T

The gem is available as open source under the terms of the MIT License.

Bug reports and pull requests are welcome!