Page Index Toggle Pages: 1 Send TopicPrint
Hot Topic (More than 5 Replies) Delicious Upload -> Crash (Read 11394 times)
Jerry Krinock
Admin
Offline
Posts: 1666
Location: San Jose, California, USA
Joined: Sep 29th, 2008
Re: Delicious Upload -> Crash
Reply #8 - Aug 20th, 2010 at 12:08pm
Print Post  
Regarding being "kicked out" or "banned" by Delicious during an extended Export…

According to Delicious' documentation, they should not ban our IP address as long as we don't send more than one request (one bookmark) per second.  When I looked at this a couple years ago, I found that we would be banned after 1000 bookmarks.  So we set our algorithm in Bookdog, and in BookMacster 1.0, to send a request every 1.2 seconds, up to 900 bookmarks, and then back off to once every 5.0 seconds, and that seemed to work.

However, based on my own experience recently, I agree with you that this no longer works.  Actually, in two tests I found that I get banned after about 5 minutes and 35 seconds when sending a request every 0.8 seconds (a little fast, but seen similar results at 1.2).  Also, instead of banning you with an HTTP 503 error, now they sending this undefined HTTP 999 error.  The good news, however, is that instead of banning your IP address for a couple hours, now it seems to be much less.  I've seen it as little as a few minutes, but up to 50 minutes if I kept hitting it once every five minutes.  Clearly, some group within Yahoo! (who does not know or chooses not to use HTTP protocol specifications) decided to make some unilateral adjustments, but I sure wish they would publish their actual policies so we wouldn't need to reverse-engineer.

BookMacster 1.1.1 Beta (just published on /beta path) addresses this issue with an improved algorithm which is adaptive and user-adjustable.  More details are in the Release Notes.  Also, of couse, it no longer crashes.
  

Jerry Krinock
Back to top
IP Logged
 
Jerry Krinock
Admin
Offline
Posts: 1666
Location: San Jose, California, USA
Joined: Sep 29th, 2008
Re: Delicious Upload -> Crash
Reply #7 - Aug 18th, 2010 at 7:34pm
Print Post  
Charltan wrote on Aug 18th, 2010 at 7:15pm:
But still it tries to sync unchanged bookmarks. Also the Log tells me there were 68 bookmarks deleted but I have exactly 68 bookmarks more in delicious than in BM.  Maybe the uploading bookmarks are the one that should be deleted. As they can't be deleted it tries uploading them each time?
Nice observation!  When BookMacster is deleting bookmarks, in the status bar at the bottom of the window it should say "Deleting Whatever…"  It does all the deletions first, then the additions.  But I've never seen BookMacster mix up adding and deleting, nor have I never seen Delicious silently fail to delete a bookmark after acking the request.  Anyhow, yesterday we built some tools into BookMacster 1.1 to help us get data on issues like this, and right now I'm working on the speed control algorithm, as you suggested, to better handle the current de facto policies of Delicious.

Charltan wrote on Aug 18th, 2010 at 7:15pm:
Btw: I just purchased a license. Not because the app is working without any problems but for your great user support and because I know you will make the best out of this application!
Thanks.  Will get back to you soon.
  

Jerry Krinock
Back to top
IP Logged
 
Charltan
Junior Member
Offline
Posts: 8
Joined: Aug 17th, 2010
Re: Delicious Upload -> Crash
Reply #6 - Aug 18th, 2010 at 7:15pm
Print Post  
Now I successfully synced my bookmarks three times with delicious without a crash Smiley
But still it tries to sync unchanged bookmarks. Also the Log tells me there were 68 bookmarks deleted but I have exactly 68 bookmarks more in delicious than in BM. Maybe the uploading bookmarks are the one that should be deleted. As they can't be deleted it tries uploading them each time?

Btw: I just purchased a license. Not because the app is working without any problems but for your great user support and because I know you will make the best out of this application!
  
Back to top
WWW  
IP Logged
 
Jerry Krinock
Admin
Offline
Posts: 1666
Location: San Jose, California, USA
Joined: Sep 29th, 2008
Re: Delicious Upload -> Crash
Reply #5 - Aug 18th, 2010 at 3:57pm
Print Post  
Charltan wrote on Aug 18th, 2010 at 1:58pm:
I needed to re-authenticate BM with delicious and received an error. So the problem is delicious.
Well, it's both.  When Delicious "kicks you out", as I say, they send an error code 999 which is indicated in the dialog.  This error code is not defined in internet protocols; it's something they cooked up on their own.  The current version of BookMacster rolls down a sheet to re-authenticate.  The next version of BookMacster does not do this.  In the current version, when you get a 999 error, click "Cancel" in the sheet.  Don't re-authenticate; it's a waste of time.
Charltan wrote on Aug 18th, 2010 at 1:58pm:
It just would be nice if the app could … pause the upload when kicked out and resume later automatically).
That's a good idea.  I'd been hoping that Delicious would improve their behavior (and have filed bugs with them), but users don't have much leverage over a free service.
Charltan wrote on Aug 18th, 2010 at 1:58pm:
Another thing I found: I have about 400 Bookmarks in all browsers. On delicious there are over 500. Any reason why the amount isn't the same, even I told to overwrite delicious (clean slate)?
No.  If things are working properly Delicious should always have the same or fewer than Bookmarkshelf, since there are some bookmark types, and also duplicates, which they do not allow.  I'll get back to you on this.
  

Jerry Krinock
Back to top
IP Logged
 
Charltan
Junior Member
Offline
Posts: 8
Joined: Aug 17th, 2010
Re: Delicious Upload -> Crash
Reply #4 - Aug 18th, 2010 at 1:58pm
Print Post  
OK, I deleted the sql file and tried it again. When uploading was almost finished BM crashed again, so this didn't help. I needed to re-authenticate BM with delicious and received an error. So the problem is delicious. It just would be nice if the app could work better with those errors (e.g. No re-authentication, pause the upload when kicked out and resume later automatically).
Another thing I found: I have about 400 Bookmarks in all browsers. On delicious there are over 500. Any reason why the amount isn't the same, even I told to overwrite delicious (clean slate)?
  
Back to top
WWW  
IP Logged
 
Jerry Krinock
Admin
Offline
Posts: 1666
Location: San Jose, California, USA
Joined: Sep 29th, 2008
Re: Delicious Upload -> Crash
Reply #3 - Aug 18th, 2010 at 12:25pm
Print Post  
Charltan wrote on Aug 18th, 2010 at 10:44am:
Thanks Jerry
Of course I don't know how BookMacster works together with delicious. What I see is that there is a local sql file for delicious.
Yes, Delicious requires that.
Charltan wrote on Aug 18th, 2010 at 10:44am:
Do you think it is worth deleting this and start from scratch?
Sometimes.  There is actually a menu command for doing that.  BookMacster ▸ Empty Cache.
Charltan wrote on Aug 18th, 2010 at 10:44am:
What I do now is uploading the bookmarks, after a while aborting it and start over again a few minutes later. If timeout is the problem maybe this can help.
Yes, that would avoid being timed out / kicked out by Delicious and crashing
Charltan wrote on Aug 18th, 2010 at 10:44am:
I just found that yesterday I successfully uploaded all bookmarks and today many changes are reuploaded again. I can't understand why as I didn't made any changes until then.
When you get a chance, click in Reports ▸ Logs and it may say if attributes were deemed to have changed (Δ) or if bookmarks were not recognized and deemed to have been new (+, -).  It's not much info, but better than nothing.
  

Jerry Krinock
Back to top
IP Logged
 
Charltan
Junior Member
Offline
Posts: 8
Joined: Aug 17th, 2010
Re: Delicious Upload -> Crash
Reply #2 - Aug 18th, 2010 at 10:44am
Print Post  
Thanks Jerry
Of course I don't know how BookMacster works together with delicious. What I see is that there is a local sql file for delicious. Do you think it is worth deleting this and start from scratch? Or is it a general problem and deleting the sql file doesn't help?
What I do now is uploading the bookmarks, after a while aborting it and start over again a few minutes later. If timeout is the problem maybe this can help.
I just found that yesterday I successfully uploaded all bookmarks and today many changes are reuploaded again. I can't understand why as I didn't made any changes until then.
Can't wait to test the new version where BM won't crash Wink

Chris
  
Back to top
WWW  
IP Logged
 
Jerry Krinock
Admin
Offline
Posts: 1666
Location: San Jose, California, USA
Joined: Sep 29th, 2008
Re: Delicious Upload -> Crash
Reply #1 - Aug 17th, 2010 at 10:23pm
Print Post  
Hello Charltan,

While retesting that 1Password issue you mentioned in the other thread, I too noticed that Delicious seems to be kicking me out faster than usual today.  (BookMacster operates within their guidelines … that's why it uploads bookmarks so slowly, but some days they kick you out anyhow.)

Then, there is a bug which causes BookMacster to crash instead of just presenting the error.  That bug is fixed in BookMacster 1.1 which we're trying hard to get out the door ASAP.  So, don't bother with the crash log.

However, after the crash, I've never seen BookMacster re-upload (or re-update or re-delete) bookmarks which it had already uploaded, updated or deleted before the crash.  I almost want to say it's "impossible", because of the way that BookMacster and Delicious work.  I just deleted 1000 bookmarks, and it took two retries today, but it did get all of them.  But I've been in this business too long to say "impossible".

I'll send you a link privately when BookMacster 1.1.1 is ready "for new users".  (It will take a little longer for existing users because we need to add database migration.)
  

Jerry Krinock
Back to top
IP Logged
 
Charltan
Junior Member
Offline
Posts: 8
Joined: Aug 17th, 2010
Delicious Upload -> Crash
Aug 17th, 2010 at 9:30pm
Print Post  
I don't think this is normal behavior but I try uploading (syncing) with delicious and each time the application crashes after a while. I first thought it is a special bookmark that can't be synced but as it seems deleting this bookmark didn't help. Also I have to re-authenticate delicious many times and it seems even many bookmarks are uploaded before the application crashes, BM syncs the same bookmarks Over again whenever I start a new sync (and not only changes). I tried to get it to work for hours now without any success. Would it help to send you the crash log? If this problem can't be fixed I can't find a reason to purchase the app - even I really would like to use it.
  
Back to top
WWW  
IP Logged
 
Page Index Toggle Pages: 1
Send TopicPrint