Hi! To achieve this you definitely need a custom algorithm. I think the first step is to define the variables you are going to use for your classifiers (wings shape, wingspan, color and patterns in the wings). If the pattern in the wings are similar by species you can use some algorithms like fisherface or eigenface to classify your images due is for indoor use I assume the lighting conditions will be controlled. This is more a recognition feature than augmented reality but I can figure out how to create a detector for scenarios like this. I suggest you to try some OpenCV demos (available in C++, Python, or Java) about FisherFaces classifier, is widely used to detect ages or glasses on the faces, I think it should work for butterflies.