Cat & Mouse Drones

Introduction.

For this project, we are to develop the code (Python) for a drone –the cat– to follow another drone –the mouse– which will be flying randomly around de scene.

Our drone is equipped with two cameras –one front and one ventral– through which it will receive visual input from its surroundings. We don't have access to the Python script for the drone role-playing the mouse, and its movement is apparently random, so we must follow it based solely on our visual input.

Just as we did for our previous project, our script will be reactive-control-method based. This means for every iteration of the loop our drone will get an image from a camera, process it and make a decision based on it. Since there are many concepts already covered in the previous entry of this blog I'll try to focus on the new ones.

Perception and filtering.

What does an image captured by our front camera look like, you may wonder? Well, nothing odd. There's a grass-like-painted flat floor, some trees in the distance and –if we are lucky– the mouse drone. Some of this information will be really useful, essential actually, and some won't be of use at all.

We need to pick out the mouse drone from all the other elements in our image. Good news: it is painted red on its top half, so we just need to filter the image to obtain these red pixels. Easy. Get image --> Translate to HSV --> Filter red values

See what it looks like:

After filtering the red values, I deem it necessary to apply Gaussian blur so we get a more solid view of the drone from far away. The cv2 function GaussianBlur() returns a non-binary image, so we must threshold it afterwards.

Keeping the mouse within view.

There are a bunch of different commands we can give our drone, such as moving forwards, sideways or backwards, or rotating on the Z axis. But, in order to make any decision, the mouse drone must be within view of our camera. To achieve this, it will rotate in the Z axis according to the position of the mouse in the image. If the mouse is to its right, the cat will turn clockwise, and if the mouse is to its left, the cat will turn counterclockwise. The speed of its turning will be proportional to how deviated from the centre of the image the mouse is.

The mouse drone will be moving up and down as well, and our cat must respond to such movements. Just like we did to get the mouse centered horizontally in the image, we can compare the vertical position of the mouse with the ideal position we seek for. Our drone will correct the error by going up or down accordingly. The ideal position we want the mouse to be might be slightly lower than the centre of the image so we ensure the red part of it is always within our view.

Useful cv2 methods and math.

We've already used a few methods from this module –cv2.cvtColor() and cv2.inRange()– which were useful for filtering the image. Should there be a method to get the values for the horizontal and vertical position of the mouse drone, it would make things much easier for us. Fortunately, the methods findContours(), contourArea(), boundingRect() and rectangle() from the cv2 module will provide us with such information and more. By calling them with their corresponding parametres –according to the cv2 online manual– we can obtain the coordinates of the rectangle that bounds the visible shape of the mouse drone.

What the boundingRect() method actually returns is the coordinates of the top-left pixel of the rectangle (x, y) and its height and width (h, w). We can calculate its centre like:

(x + w/2, y + h/2)

In order to get the other three corners of the rectangle it would be:

(x + w, y), (x, y + h) and (x + w, y + h),

but we won't be needing these.

Actually following the mouse.

It seems like we've got all the information we needed, so now it's time to put it to use. We want our cat drone to follow the mouse from a reasonable distance so it has enough time to react to variations in speed, height or direction without colliding. There are three possible scenarios: the cat is far from the mouse, the cat is at an acceptable distance from the mouse, or the cat is too close to the mouse. The solution is pretty straightforward. Whenever the cat is too far from the mouse, go forwards. If the cat is too close, go backwards. If it's within a reasonable distance, just stay there. But how can we tell from our filtered image how close the cat is from the mouse? Well, it's simple, really. The bigger an object is to you, the bigger you see it. So if the cat is closer to the mouse it will see a bigger white surface in the filtered image:

Knowing that, we can just set a couple of thresholds that determine in which of the situations we are at an specific moment. As I said before, we can decide on the area of the white surface, but the value of the area not only depends on how far the mouse is but also on its tilt, and it isn't a subtle difference –the area could easily double just from the tilt of the mouse–. Hence, instead of evaluating the area, we will decide on the horizontal length of the mouse's bounding rectangle.