Features

• Web spider: browser rendering, ajax data crawling
• Auto extract: extract valuable data from pages using advanced AI
• Performance: highly optimized, rendering hundreds of pages in parallel on a single machine without be blocked
• Data quantity assurance: smart retry, accurate scheduling, web data lifetime management
• Large scale: fully distributed, designed for large scale crawling
• Simple API: single line of code to scrape, or single SQL to turn a website into a table
• X-SQL: extended SQL to manage web data: Web crawling, scraping, Web content mining, Web BI
• Bot stealth: IP rotation, web driver stealth, never get banned
• RPA: simulating human behaviors, SPA crawling, or do something else awesome
• Big data: various backend storage support: MongoDB/HBase/Gora
• Logs & metrics: monitored closely and every event is recorded
• Web UI: create powerful crawlers without code skills