Hello there!
First, I am assuming that, whatever method is used, you want the function to remain 100% in Matlab language and not use MEX files (with C/C++ code). Otherwise, you could get very big gains in performance, at the expense of making it difficult to edit the code later (due to compilation needed etc.),
Leaving this option out of the way, after having a look I can tell you that your problem is the 'for' loops. Unfortunatelly, Matlab is OK if you are doing computations with matrix/vector calculations, but is very slow if you do the work 'manually' with loops. There are ways to vectorize the code, but it improves performance for actual matrices/arrays. However, as far I can see you are using cell arrays, where each cell is a matrix. Calculations on cell arrays can still be vectorized with the 'cellfun' function, but it doesn't seem from experience to really improve performance. So, first I would look inside the functions that do work on single cells/matrices and see if they are vectorized (e.g. Q_matrix_layer).
Another option one has to improve performance is to parallelize the code with 'parfor'. It is not as simple as putting a 'par' in front of a few 'for's (due to needing to 'slice' cell arrays and some other stuff), but you can get a good speedup usually.
Best Regards
Alex