Advanced Lane Finding Project
The goals / steps of this project are the following:
- Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
- Apply a distortion correction to raw images.
- Use color transforms, gradients, etc., to create a thresholded binary image.
- Apply a perspective transform to rectify binary image ("birds-eye view").
- Detect lane pixels and fit to find the lane boundary.
- Determine the curvature of the lane and vehicle position with respect to center.
- Warp the detected lane boundaries back onto the original image.
- Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.
Determining the camera calibration matrix and distortion coefficients is the first step toward correcting distortion in images. To obtain these we use a set of chessboard images and OpenCV's calibrateCamera() and findChessboardCorners() functions. Chessboard corners are detected in each image using findChessboardCorners() and are stored in the img_points array. For each set of img_points found a corresponding set of 3D points representing the position of the corner in the world coordinate space is kept in obj_points. These obj_points and img_points are then fed to calibrateCamera() to obtain our calibration matrix and distortion coefficients. Given an image, the matrix, and the distortion coefficients, OpenCV's undistort() can then be used to correct for distortion on any image taken with that camera and lens.
Camera calibration code can be found in camera.py. The second code cell in P2.ipynp uses this code to calibrate the camera and apply distortion correction to two test images.
The next step after distortion correction is thresholding. Finding the correct thresholding to handle all different colors, shadows, reflections, and other noise was a tricky problem.
The color spaces I eventually settled on are the L Channel of the LUV color space for picking up white lines and the B Channel of the Lab color space for picking up yellow lines. I used different threshold values for each video but I found that for the project_video.mp4 L values between 199 and 255 and B values between 159 and 255 worked fairly well in the general case. Shadows in the challenge videos were a particularly difficult problem and I attempted to do an adaptive threshold where sections of the image with lower brightness could be thresholded with different values but I only saw modest gains. I finally settled on breaking the image up into 9 sections in the Y dimension and 20 sections in the X dimension. Lower brightness values for project_video.mp4 were between 118 and 255 for the L channel and 159 and 255 for the B Channel. All project thresholding values can be found in project_data.py
Below is an example of thresholding an undistored test image:
Thresholding code can be found in ./image_utils.py. See the function threshold_image() at the top of the file. To make deterimining these numbers easier I wrote a tool so that I could see the changes in threshold interactively. The tool can be found at ./tools/find_threshold.py
The next step is to warp the image into a bird's-eye view. The perspective matrix for performing this transformation is obtained by a call to the openCV function getPerspectiveTransform() To calculate this getPerspectiveTransform() needs points from the source image mapped to locations in the destination image. In an attempt to handle differing image sizes I kept source points normalized between 0 and 1. Four source points were mapped to destination points in the following manner:
- (bottom_left_x, bottom_left_y) in the source mapped to (bottom_right_x, height_of_image) in the destination
- (top_left_x, top_left_y) in the source mapped to (bottom_left_x, 0) in the destination
- (top_right_x, top_right_y) in the source mapped to (bottom_right_x, 0) in the destination
- (bottom_right_x, bottom_right_y) in the source mapped to (bottom_right_x, height_of_image) in the destination
Once the proper source and destintion points are mapped all it takes is a call to cv2.warpPerspective() to take the image to/from a top-down view. warp_image() in the file image_utils.py (lines 58-71) takes care of both obtaining the perspective matrix and warping the image.
The example below shows the original and the warped image.
The bulk of the work to recognize the lane lines is done in the function find_lane_lines().
This function takes one of two approaches to finding the lane lines:
- If this is the first detection or the previous line was not been successfully detected:
- Take a histogram of the bottom half of the warped and thresholded binary image.
- Take the histogram peak on each half of the image as the starting point of each line.
- Using a 'sliding window approach' move upwards in the image adjusting the window position as necessary to follow the area with the largest number of pixels.
- use Numpy's
polyfit()to fit the pixels to a polynomial.
OR: 2. If a previous line has been successfully detected and mapped to a polynomial:
- Since the lane lines do not move much from frame to frame, use the previous frame's polynomial to search around for pixels in the current frame.
To facilitate the holding of individual line data and properties I created a Line class. The class implementation is in the file p2.py from line 80 to line 142. This class holds the pixels detected, the line coefficients obtained from polyfit(), the line curvature, and derivatives taken at 3 points along the line. A method is also provided to detect if the line is valid.
Several properties of the line are checked for validity. The isValid() function of the Line class handles this determination. Line slopes, coefficients, the number of pixels detected, and curvature factor into validation.
A visualization of the process can be seen in the lower right quadrant of this video frame:
To measure curvature I used the following formula
A Pixel space to world space conversion was made using standard lane widths and a polynomial was refit to the new world space values. The curvature equation was then evaluated at the lowest point of the line(bottom of screen).
Offset was calculated under the assumption that the camera was recording from the center point of the car. The difference of the center of the image from the center point between the detected lane lines was calculated and converted from pixel space to world space.
The final step is the fill the lane area detected and render it back over the original image. This is done by the function fillLane()( file p2.py, lines 304 through 341).
OpenCV's fillpoly() was used to fill the lane boundaries and warp_image() was used in reverse to warp the image back into a first-person perspective.
Both the project_video and the challenge_video went well. The harder_challenge_video exposed many weaknesses in my approach. The many shadows and bright spots on the road confused my algorithm, as did the bright grass off the right shoulder of the road. Even so, the lane detection does recover up until the last very brightly lit hairpin turn at the end.
Here's a link to my video result for the project video
Here's a link to my video result for the challenge video
Here's a link to my video result for the fiendishly difficult challenge video
This was a very engaging and fun project but posed many challenges. I wish I had more time to work on it. Determining the proper thresholding initially took a lot of time. I still feel that there could be some fine tuning there to better handle shadows and over-exposed areas of the road. The pipeline fails when taking a very sharp turn (for example, in the harder challenge video at the end) and one of the lane lines goes off the side of the video. Perhaps an assumption as to the position of the line would be necessary here. Also, more time could be spent on the determination of lane lines when the initial methods fail. If the polynomial or sliding window approach fails a fallback to a hough transform on a warped, unthresholded image might provide more line data here. In addition, some time spent on removal of shadows could help overall.