Comments on: Released: Twitter Timeline Export (TweetDumpr)

By: Adam Franco

Adam Franco — Mon, 13 Oct 2008 16:24:02 +0000

Using the script I mentioned I was able to retrieve all 34 pages of my Tweets. Maybe HTML scraping the twitter site is the only place where the page limitation exists…

By: Adam Franco

Adam Franco — Mon, 13 Oct 2008 16:11:14 +0000

Oops, I guess you are already doing this. My bad.

By: Adam Franco

Adam Franco — Mon, 13 Oct 2008 16:09:54 +0000

Hi Brad,

Just wanted to let you know that you can access more than 200 Tweets if you load them by page. I’ve posted a small export script here that shows how to do this:
http://www.adamfranco.com/archives/88

For now this script just dumps the raw XML and is not set up as an end-user service, but it can be copied and run by anyone who wants to back up all of their time-line.

By: Adam Franco.com » Blog Archive » Twitter Export Script

Adam Franco.com » Blog Archive » Twitter Export Script — Mon, 13 Oct 2008 15:59:41 +0000

[...] doings and wished to export my time-line for reformatting into a calender format. Unfortunately TweetDumpr just retrieves the list of Tweets using a single fetch request which is limited by the Twitter API [...]

By: Morten Blaabjerg

Morten Blaabjerg — Fri, 05 Sep 2008 20:05:49 +0000

I think I’ve now waited about 30 mins at least for an email to arrive, especially since the TweetDumpr page says it ought to take just 10 mins to dump a five figure timeline. My timeline is not nearly as big, and yet still no response after half an hour or more. How long should I expect to wait to get an email response?

By: Morten Blaabjerg

Morten Blaabjerg — Fri, 05 Sep 2008 19:51:56 +0000

Well, now you have the sentiments of another (perhaps former) member of the Twitter user base

By: Brad Kellett

Brad Kellett — Fri, 05 Sep 2008 19:48:31 +0000

To be clear, I have no privacy concerns. To me, what is public is public. I put in the password requirement based on feedback from many other Twitter users. I am trying to respect the Twitter user base and respond to their concerns.

By: Morten Blaabjerg

Morten Blaabjerg — Fri, 05 Sep 2008 19:43:13 +0000

Well, for one thing it raises my suspicions when a web tool such as yours asks for my account details as well as my email adress for something which is basically public data. There’s no real reason for this.

What people use Twitter for or would or would not like others to do with their data does not concern me. If you don’t like someone accessing something you post in public, it’s really easy not to post it in public, if it means a lot to you.

What concerns me is the usability of the tools I have at hand - what they can do for me. And when you superimpose privacy concerns into what is basically mining public data, you get in my way, even if you want to help me. Great, your tool helps me export my own stuff, which is of high priority to myself right now. I’m happy you put your tool out here, and I found it.

But it makes your own tool less useful, because you think people shouldn’t use it in ways you think violates your privacy concerns. Twitter is, in fact, a great way to analyze markets, if you can access all that data buried in it’s archives. I wonder if you can see what kind of value your tool may have, if you can develop and make it accessible in ways which makes it easy for users to use your tool for such purposes. But right now you’re blinded (as I see it) by privacy concerns for something which is completely publically accessible. And if you don’t create a tool which allows for versatile mining of this data, others will. In fact, I am surprised that Twitter do not seem to see this very clearly. If they do, why would they impose obscure, randomly picked hard limits on our accessibility to the gold in our own archives?

By: Morten Blaabjerg

Morten Blaabjerg — Fri, 05 Sep 2008 19:20:50 +0000

Btw, the 200-tweet archive limit is no longer enforced by Twitter (apparently). My personal archive of Tweets have been released from it’s unwilling hostage status on Twitter’s servers, and I am grateful of a tool such as yours which allow me to retrieve ALL my data from Twitter, in whatever form possible, to prevent this from happening again.

See this thread on more about the archives inaccessibility problems many users have experienced and on Biz Stone’s “lifting” of this severe limitation of Twitter : http://getsatisfaction.com/twitter/topics/why_cant_i_view_all_of_my_updates?

By: Brad Kellett

Brad Kellett — Fri, 05 Sep 2008 19:19:08 +0000

I don’t personally have concerns over it, but the Twitter users I talked to had an issue with making it so easy to mine that data. I think there is a difference to people between posting something publicly and making it really easy to get a full archive to use however.

To me, they are the same thing, but people don’t necessarily see posting to Twitter in public as the same as posting downloadable text.