Jerry Krinock
Admin
Offline
Posts: 1666
Location: San Jose, California, USA
Joined: Sep 29th, 2008
|
Re: Delicious Upload -> Crash
Reply #8 - Aug 20th, 2010 at 12:08pm
|
Print Post
|
Regarding being "kicked out" or "banned" by Delicious during an extended Export…
According to Delicious' documentation, they should not ban our IP address as long as we don't send more than one request (one bookmark) per second. When I looked at this a couple years ago, I found that we would be banned after 1000 bookmarks. So we set our algorithm in Bookdog, and in BookMacster 1.0, to send a request every 1.2 seconds, up to 900 bookmarks, and then back off to once every 5.0 seconds, and that seemed to work.
However, based on my own experience recently, I agree with you that this no longer works. Actually, in two tests I found that I get banned after about 5 minutes and 35 seconds when sending a request every 0.8 seconds (a little fast, but seen similar results at 1.2). Also, instead of banning you with an HTTP 503 error, now they sending this undefined HTTP 999 error. The good news, however, is that instead of banning your IP address for a couple hours, now it seems to be much less. I've seen it as little as a few minutes, but up to 50 minutes if I kept hitting it once every five minutes. Clearly, some group within Yahoo! (who does not know or chooses not to use HTTP protocol specifications) decided to make some unilateral adjustments, but I sure wish they would publish their actual policies so we wouldn't need to reverse-engineer.
BookMacster 1.1.1 Beta (just published on /beta path) addresses this issue with an improved algorithm which is adaptive and user-adjustable. More details are in the Release Notes. Also, of couse, it no longer crashes.
|