Ground will be faster and you won't need as much annatto seed to get the color. You will probably want to filter it with some fine mesh to get the grains out. But, it is entirely possible you wont be able to get all the ground annatto out of the oil. But if that doesn't bother you no biggie.