Implement a more robust web scrapper

[BrataBuilds]

Web Scrappers in knowledge_manager.py, extract_wiki_content() and extract_forum_content() sometimes gets blocked by certain websites, (namely the ones present in minecraft.csv)

Recommended Solutions :

• Use an existing tool such as scrappy or playwright.
• Mask the scrappers to behave to more like human by introducing delays between different searches.
• Make the scrapper behave like a browser as much as possible
• Use Proxies to circumvent IP bans.
• Rotate a list of User Agents and headers.
• Make it asynchronous using asyncio + httpx

[metadroix35]

let me work on this

[BrataBuilds]

let me work on this

You have been asssigned