If you're new to programming with mathematica, take a look at the virtual book which will give you some background on how Mathematica is used.

You're looking to solve a functional / Integral equation. So we're not going to use the notation for defining a function, but instead use the notation for a symbolic equality:

b[x] == 1 + InverseFunction[b][x] b[5]

If possible, remove the InverseFunction using some algebra which I hopefully did correctly:

b[(b[x]-1)/b[5]]==x

There's a functional equation you want to solve under the assumption that b has an inverse. That's tough to do. RSolve can manage some functional equations, but not this one. There aren't easy answers for functional equations and I don't know of any methods that'll solve that one.

Taking a look at the integral equation,

b[x]==1+Integrate[InverseFunction[b][a],{a,0,x}] b[5] 

The first thing you do with integral equations is to turn them into differential equations if possible.

b'[x]==b[5] InverseFunction[b][x]

Using the inverse function theorem.

b''[x]==b[5]/b'[b[x]] 

Assuming I didn't mess anything up, we get the differential equation above. This equation isn't easy to solve and DSolve won't be able to do it. If you make some assumptions maybe you can estimate it with a numerical method. Looking online, I don't see any information about solving this kind of equation, but some examples of disproving that a solution exists .