Minethings Screen Scraper

Minethings is a brower-based persistent MMOG.  Visit the site to find out more.  Very simple interface and the “game” itself is not much to sneeze at – mainly an economic simulation, IMHO.  But that makes it kind of interesting … all items in the game can be sold via auction to other players, therefore the going rate for a given item is solely determined by the supply/demand.

I thought it would be interesting to track how the prices in Minethings changes over time.  I wondered if there were any patterns that could bring success in the game, if known and exploited.  Hence the screen scraper.

Mtscraper.py has two usage options.  First is “census mode”, which you run with “mtscraper.py -c”.  This will loop through all the miners currently registered with Minethings.  Names, # of melds, profession, and home city will be printed to a file called “mtscrape_census_YYYYMMDDHHSS.txt” that will be created in the same directory where mtscraper.py is located.  Second is “prices mode”, run with “mtscraper.py” (no command line option).  This will loop through all items of each mine type and record the sales history as well as current listings and bids (comment in the script says average sales price and high bid, low listing, but it is a bit off.  It actually lists all of them – all the sales, all the listings, and all the bids.)  Oh, and it is set up as is to just get the prices for one city, Harmond.  To change cities, or get prices for ALL cities, look at the “cities” variable in the script.  The script takes quite a while (about a minute) to query all the items for sale in just one city; so doing all of the cities or doing it frequently will probably hammer the servers and cause the admin no end of headaches … sorry!

Usage : need a valid Minethings account, then put your username and password into the USERNAME and PASSWORD fields of the script.  You may need a proxy server (I did) so there is a slot for that as well.  Getting a webpage to load is accomplished through mechanize; you’ll need to install that.  (If the way I connected isn’t working, consult the mechanize library and try to figure it out.)  Python required as well, of course.  Developed and tested with 2.6.

The script is heavily dependent on the format of the Minethings internal webpages.  If the site were redesigned, then likely mtscraper.py would need to be tweaked.  This is a serious negative quality of a good screen scraper; one shared by my earlier Captionator project.  Next time I’ll try to see if I can avoid it; Beautiful Soup may help ….?

Anyway, copy the output of mtscraper.py into Excel and you can make some charts.  This one uses “census” data from July 21 and shows the average number of melds per job type:  (click on image for higher quality)

With the prices mode, it’s interesting to copy the data for multiple days into a spreadsheet to see the prices change over time.  (This can generate a very large data set very quickly – I hit Excel’s 65536 row limit with only 9 days worth of data, and that’s just for Harmond!)  Here’s the average bid/listing/sale price for a Starter Mine in Harmond:


One response

  1. […] done a few projects now involving screen scraping web data with Python.  I thought I would write a post on my […]

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: