Crawl #2 Coming Soon! (and a request)

Crawl #2 is a 2-parter: grab all profiles & then grab all portfolios. Mind you, that's not the full process. The pages still then need to be run through the processing code that pulls the data from the pages & loads it all into a database. This is NOT a short process, folks, with each half of the crawl taking 1-2 hours. Overall, you're looking at about a 5-6 hour process (currently). A retooling of the code will help shorten that timespan, but for now, it is what it is.

I'll be using this second crawl to produce member lists for all personal communities, as well as a list of private community names. I won't be publishing member lists of private communities, though I will provide them to the admins of those communities upon request. Please, though, don't request a member list if you don't have over 50 members; that's just lazy.

And now for the request: what data do you want me to compile here? Different statistics about communities? Perhaps connections between followers/follow-backs/investors? The sky's the limit, so let's hear it! You'll be helping to actively contribute to the game because I'll be using your suggestions to identify ways that the EA developers can change their code to better enable the retrieval of the necessary data!

Zod's Army

Below is a veritable cheat sheet for winning investments in Empire Avenue. This is the full member list of the Zod's Army community, the #1 personal community in the game. Almost everyone in this list falls into one of these categories:

A) Worth investing serious eavage in

B) Working within the community to help boost every other member's stock by buying 10 shares from each

C) Both

Since this is one of the communities I belong to & we're trying to get it so that all members follow each other AND purchase 10 shares from each other, it gets special treatment in terms of getting its member list published. But don't worry...all the others will be posted in the next day or two!

Zod's Army on Google Docs (no login necessary): http://bit.ly/zodsarmy

First Community Member Lists (Q&D)

I'm still compiling the lists for personal/private communities, but below are the complete member lists (accurate as of June 9-10, 2010) for every city/interest community. There are 0 other sites that have this data. Use it however you like, even if it means repackaging it & posting it on your own site, just as long as it's ok if I use your formatting in future versions of the compilation.

For ease of use, I decided to host the lists in Google Docs. This may change in the future, but it's easy to grab the docs in multiple useful formats this way & you don't have to have an account with Google to access them. Each new version of the master list will get it's own spreadsheet and they'll all be contained within a shared folder.

The folder: http://bit.ly/eacommunitylists
Crawl #1: http://bit.ly/eacml1

The Empire Avenue Crawl

The other night, I downloaded the profile pages of the first 3235 EA users. Tonight, I'm finishing up the code to pull user data out of the pages & load it into a database. As soon as that's done, I'll start working on my next post: a complete member list of every community! So be prepared!