Heat the oil in a large pot over medium-high heat. Add the onion, bell pepper, garlic, celery, oregano, thyme and cayenne pepper. Cook for about 5 minutes or until vegetables are softened.
Add the rice and stir to coat with the oil and vegetables. Pour in the chicken broth or stock and add the tomatoes with their juice. Add the bay leaves.
Bring to a boil then reduce heat to low. Cover pot and simmer for 20 minutes or until all of the liquid is absorbed.
Add the shrimp to the pot and stir to combine. Cover and simmer for an additional 10 minutes or until shrimp is cooked through.
Remove from heat and let stand for 5 minutes before serving.