There are several ways to solve this, easy and hard. The information you want to learn is tension physics. 1 2 3
I've drawn a diagram of the expected forces.

There are couple issues:
- First, we're using a rigid system while conserving the energy, assuming the asteroid has infinite mass (meaning we don't affect it by moving it).
- Second, position is always updated by velocity. (I removed your your else statement and moved the code down)
When your player reaches the edge of the chain, in this case, it is similar to a collision. I usually rotate the system so the Y axis/component is along the tension vector and the X axis/component is the remaining component.

Some Pseudocode based on yours:
Player.prototype.update = function()
{
// ...
if (this.grabbed)
{
var asteroid = this.grabbedAsteroid();
if (this.distance(asteroid) > this.chain.length)
{
// Begin pseudocode
var chainVector:Vector = asteroid.position - this.position; // Vector pointing to asteroid
var playerRelativeVector:Vector = this.velocity; // Temporary vector that we are going to rotate forward and backward.
// align component vectors along the tangent of the tension (i.e., remove rotation and align components to axis)
playerRelativeVector.angle -= chainVector.angle; // (2)
// Reverse Y component because chain is 'pulling' on it
playerRelativeVector.Y *= -1;
// Rotate the vector back into world coordinates
playerRelativeVector.angle += chainVector.angle; // (3)
// Update our velocity to reflect changes to system
this.velocity = playerRelativeVector;
}
}
this.position += this.velocity;
}
Note, this method will not work in high-energy or precision simulations as you are calculating the position after you have traveled past the length of the chain. If you were to anticipate the position, you would be calculating the position prior to reaching the length of the chain. The only way to perform this precisely is to use a sweep-test and calculate the moment the player reaches the maximum length, and modify the vector at that moment. (Extremely high energy systems would require a recursive sweep test >.<)
Finally, I've implemented one recursive sweep-test physics collision system and I absolutely despise it. I would suggest using Box2D as Cameron Fredman suggested.