Fabric: "mouse:down" not returning object in event.target

In this jsfiddle I coded with Javascript/Fabric a polyline with lines that can be resized by moving the intersections (i.e. not stretched proportionally as a whole object). Click several times on the canvas and you'll construct the polyline, press Esc to end. Then you can select the polyline by clicking on any of the lines, and unselect by either clicking on the canvas or on a different object.

This is the issue: selecting/unselecting the polyline is unstable. Sometimes, "mouse:down" returns e.target = undefined even though I click on a line, and the opposite, it returns e.target = the line object when I click outside the line (but somehow close to the line).

What's wrong with this code? I apologize if the code is too long, but couldn't find a way to make it shorter (and could motivate some ideas if someone is interested in achieving the same result)

Relevant javascript:

canvas.observe("mouse:down", function (e) {

    if (drawing)
        drawPoly(e);
    else
        seePolyline(e);
});


var seePolyline = function(e) {

    deselectAll();

    var obj = e.target;  // e.target contains undefined when it should 
                         // contain the clicked object, and the opposite
    if (typeof obj !== 'undefined') {
        var type = obj.get('type');
        if (type === 'rect' && (obj.line1 || obj.line2) || 
            type === 'line' && obj.rect2 ) { 
                var rect = searchFirstRect(obj);
                selectPolyline(rect);
        }
    }

    canvas.renderAll();

};

UPDATE

I believe one of the problems happen when the line is moved; if you click on the area where the line was before (now empty canvas), the polyline is selected.

UPDATE 2

I'm getting closer; when you change coordinates you need to call setCoords() for example:

if (rect.line1) {
   rect.line1.set({ 'x2': rect.left, 'y2': rect.top });
   rect.line1.setCoords();
}

Updated jsfiddle

The problem that persist is that when I'm close to the line (not on the line) a click is interpreted as if the click is on the line itself. Any ideas will be appreciated.