Duplicate visit_token - ahoy_matey (1.0.2)

[pctj101]

FYI, still getting this type of error after moving to v1.0.2

Mysql2::Error: Duplicate entry 'ecf37acb-ef0d-4f5f-b536-093c3a899868' for key 'index_visits_on_visit_token': INSERT INTO visits (browser, cf_connecting_ip, country, created_at, device_type, ip, landing_page, os, platform, user_agent, utm_campaign, utm_medium, utm_source, visit_token, visitor_token) VALUES ('IE', '1.2.3.4', 'US', '2014-07-31 06:26:27', 'Desktop', '1.2.3.4', 'https://example.com/';, 'Windows 7', 'Web', 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko', 'xx', 'yy-under', 'zz', 'ecf37acb-ef0d-4f5f-b536-093c3a899868', 'd57022c6-7748-490d-825f-0cbda451b845')

[ankane]

Does this only happen occasionally? Also, what version of Rails are you on? I believe Rails now throws an ActiveRecord::RecordNotUnique error rather than a Mysql2::Error, but we obviously want to handle this case as well if possible.

[pctj101]

Rails 4.1.4
FYI Haven't seen this error for the last 2 days, but I did switch my logging system, so perhaps I'm totally missing it in the logs. :( Will let you know if I see it again. For the moment, since newrelic and my logs don't show this particular error, maybe it was some old javascript that was cached in client browsers.

[ckmorris]

With postgres I get PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "visits_pkey" DETAIL: Key (id)=(the-key) already exists.

[ankane]

Probably related to #32 - can you tell if it's trying to insert a nil id?

[rarshonsky]

Any solution to this? I've got thousands of these errors a day right now.

[ankane]

Still need more info to debug (not seeing it in any of my setups). Do you have the parameters passed to the /ahoy/visits endpoint?

[sheetalank]

I am also getting a similar error (primary key is duplicate). It happens when I refresh a page for which the visit duration is over. Was wondering why that is happening. Visit id is not nil but one that is already in the table.
Any help would be appreciated.
I am using Rails 3.2.15.

[ckmorris]

It's because he's rescuing the creation of the visit and then doing nothing in order to skip the duplicate object error.

[sheetalank]

But the visit duration is over. Shouldn't it create a new visit ?

[mchristen]

I'm experiencing this too. It seems that if the visit duration is exceeded the next updates sent back to the server cause duplicate key errors on the id(UUID) field on the events table. I can't find any instances of this happening recently for the visits table.

[sheetalank]

I figured this out - the migration for visits creates a binary(16) field for visit_id (and visitor_id) in sql server. This field is not big enough to accommodate the 36 char long UUID (thus losing it's uniqueness). Changed the field to varbinary(max) and it is working now.

[mchristen]

Well using PostgreSQL and the generated migrations my column types are uuid, so I don't think that's the same issue I am seeing.

[ckmorris]

Same here on PostgreSQL and the column types being uuid.

[mchristen]

I'm thinking that it is some problem with the JS tracking code.

I managed to get this to re-produce on my local development machine and when I set a breakpoint in the client side JS I can see that all the events that are causing the duplicate ID's are still in the queue client side, even though a matching event already exists in the database.

So perhaps something isn't clearing them out of the queue after they are sent to the server?

[ankane]

@sheetalank Interesting, 16 bytes is all that's needed to store a UUID.

@mchristen Events aren't removed from the queue until the server gives a successful response, so it's expected that events can be sent multiple times. The store should gracefully handle this situation, but maybe this line needs updated?

[mchristen]

Well I guess I am wondering how the events can be populated already in the database without the client side receiving a success code indicating it should remove the event from the queue.

So either the event is never being removed the queue after being successfully added to the database server side, or the same event is being re-created and re-queued client side after it has already been sent to the server and processed(and removed from the client side queue).

[nicolasgarnil]

I'm having the same problem.

PG::Error: ERROR:  duplicate key value violates unique constraint "visits_pkey"
DETAIL:  Key (id)=(b9f96737-a1b5-4990-ba4f-6956c7e04e24) already exists.
: INSERT INTO "visits" ("browser", "country", "device_type", "id", "ip", "landing_page", "os", "started_at", "user_agent", "visitor_id") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING "id"

[Aerlinger]

Having this problem as well. Has anyone found a reasonable solution?

[bilby91]

Same issue here with pg 9.3.5 and rails 4.1.8

[honeybadaguas]

Having this same issue. Is there any way we can solve it?

[juno]

Same issue here with ahoy_matey 1.2.0, rails 4.2.3, pg 9.4.2.

@ankane
Thank you for great gem. Our service heavily relies on ahoy. ✨

I dug into the source code and noticed that generateId() JavaScript function is a little naive implementation.
https://github.com/ankane/ahoy/blob/master/vendor/assets/javascripts/ahoy.js#L85-L91

You borrowed the generateId() logic from StackOverflow post.
The original author updated the comment at 2015-06-02 and he saids:

Update, 2015-06-02: Be aware that UUID uniqueness relies heavily on the underlying random number generator (RNG). The solution above uses Math.random() for brevity, however Math.random() is not guaranteed to be a high-quality RNG. See Adam Hyland's excellent writeup on Math.random() for details. For a more robust solution, consider something like node-uuid.js[Disclaimer: I'm the author], which uses higher quality RNG APIs where available.]

http://stackoverflow.com/questions/105034/create-guid-uuid-in-javascript/2117523#2117523

I suppose we need a more robust UUID generator in client-side. Or generate UUID on server-side.
What do you think?

[miguelpeniche]

Same issue here:

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "visits_pkey"
DETAIL:  Key (id)=(143b6187-7868-42ae-b34f-06814e7ed45b) already exists.
INSERT INTO "visits" ("browser", "city", "country", "device_type", "id", "ip", "landing_page", "os", "referrer", "referring_domain", "region", "screen_height", "screen_width", "started_at", "user_agent", "user_id", "visitor_id") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17) RETURNING "id"

Do anyone find any solution?

[mikeghen]

I am having a similar issue, any new development? What's the deal with PR #44?

[ethankang]

Same issue here, it happens when tracks visits across subdomains.

seems like in ahoy.js, destroyCookie("ahoy_track") does not destroy the cookie, that causes to re-post visit.

in https://github.com/ankane/ahoy/blob/master/vendor/assets/javascripts/ahoy.js#L146

if (visitId && visitorId && !track) {
    // TODO keep visit alive?
    log("Active visit");
    setReady();
  } else {
    if (track) {
      destroyCookie("ahoy_track");
    }

...

  $.post(visitsUrl, data, setReady, "json");
}

[ankane]

Cleaning up issues. Feel free to reopen or create a new issue if someone is still seeing this on the latest version and interested in helping to debug.

[GlombSt]

Just ran into this issue for tracking_events.
For a base of 10 million events we see about 50 duplicate UUIDs over 24 hours.

[ankane]

@GlombSt Can you create a gist with the full error and relevant backtrace?