In mathematics if someone wants to write their function as being infinitely recursive... well... they can't.. the best they can do is the sort of thing Newton did where he wrote for his root-finding method, which is not written in it's entirety (where _ denotes a subtext)...

N_(n+1)=h(N_n)...
or maybe another way would be...
f(n+1)=f(f(n))...

and then ending both by saying that whatever value is equal to f(infinite) or N_infinity...

well, I think there should be a notation for both finite and infinite recursion (something analogous to the notations for sum and product)...

I would like a capital Omicron to be used (with the number of recursions above and the first value inputted into the function below - like with the upper limits and lower limits on series) because I think it slightly resembles the notation for composite functions...

I think this would be especially useful when working with things like fractal equations and the like?

Mostly baked, and I'm not sure there's a need for it in general:

n! is an example, of course. f^{(n)}(x) is often used (e.g., in Taylor and Maclaurin series definitions) to describe the nth derivative of f(x). A more general instance is that of most recurrence relations; these are often described using the same subscript notation as for sequences, where a_n, for instance, would denote the nth iteration.

In essence, you want a notation to capture the idea of "the nth member of a sequence" where the sequence happens to have a recursive definition. But there are often recursive and non-recursive ways to derive the same expression, and we have several forms of general nth-member notation already.

And if you want to express the result of an infinite number of iterations, isn't that what "lim" is for?

Feel free to correct me if I'm missing your point. I should mention in any case that certain Greek letters, including omicron, are not generally used in mathematical notation because they're hard (or impossible, depending on the typeface) to distinguish from Roman letters.

I'm having trouble seeing how you would do this. To find the (n + 1)th term, don't you have to find the nth term (assuming there's no equivalent non-recursive expression)? Or is this just to give a name to a term that you're not even figuring out?

Sadly this is going to look really confusing, but assume (circle) is a little circle and ^x^ is superscript x. (The final paragraph might be clearer.)

(f(circle)f)(x) is f(f(x)), and (f(circle)f(circle)f)(x) is f(f(f(x))), therefore you sometimes see application of a function n times written with a superscript as f^n^(x). This is not ambiguous with exponentiation if you remember that functions and numbers are different types of things and you restrict exponentiation to numbers.

This might look slightly better, although the circles are a bit funny if you see them at all:

(f º f)(x) is f(f(x)), and (f º f º f)(x) is f(f(f(x))), therefore you sometimes see application of a function n times written as f^n^(x). This is not ambiguous with exponentiation if you remember that functions and numbers are different types of things and you restrict exponentiation to numbers.

// if you want to express the result of an infinite number of iterations, isn't that what "lim" is for? //

No it is most certainly not! It's to find the limit as a number approaches a certain value such that...

lim(x->y) f(x) = L,
|L - f(x)| < E,
|y - x| < d,
d = z(E), where y and L are both finite...

it is only used to express an infinite number of iterations when used in conjuction with either a recursive notation like that I suggest or discussed by you, or a variable series(a good example being the theory of definite integration)/product expression...

I believe a good example of lim being used for something commonly is in differentiation?...

I had a very similar idea during my earlier college years except I used an upside-down uppercase Greek "Delta." I say it's still a great idea and you should go for it!

I also developed some mathematical proofs related to that during those years. Maybe I should dig those up and annotate them here.