"Subodh Paudel" <subodhpaudel@gmail.com> wrote in message <kfu4se$6s6$1@newscl01ah.mathworks.com>...
> Hai Greg,
> Thank You for your previous suggestion. Accordingly, i changed my training structure and still the building energy forecast is not good.
> I have some queries, further,:
> 1) All the time i run, i have the R-value i.e. in the regression plot in the neural network changing? I could not figure it out...is the weight changed all the time...
> 2) Regarding MSE00 i.e. initial condition, MSE00=mean(var(target)), it has a large value in the range of exp.. does it should have a normalize value?
> 3) How can i read the R2 value of the test, validation and training value although matlab describes the value for all the cases:
>
> Here is My Rest Code:
>
> data=xlsread('C:\Users\Dell\Desktop\testdata.xlsx','Sheet1');
> [dN dI]=size(data);
>
> %Input and Target Data
> P= data(1:dN,1:2)';
> T=data(1:dN,3)';
>
> %Input and Hidden Layer
> [I N]=size(P); %[2 1826]
> [O N]=size(T); %[1 1826]
> % MSE00=mean(var(T));
>
> %Normalization of Input and Output Data
> [pn,ps]=mapstd(P);
> [tn,ts]=mapstd(T);
>
> H=8; %(Iterative Optimization)
> Ntrneq=N*O; %1826 No of Training Equations
> Nw = (I+1)*H+(H+1)*O; % No. of unknown weights
> %This yields an upperbound for H
> Hub = -1 + ceil( (Ntrneq-O)/(I+O+1)); % 319
> %Hub = -1 + ceil( ( 0.7*1826-1)/(2+1+1) ) = 319
> %Hmax ~ round(Hub/8) = 40
>
> %MLP (2-H-1)
> net=newff(minmax(pn),[H O]);
>
> %Divide Samples into Training, Validation and Test
> valPercent=0.015;
> testPercent=0.015;
> [trainV, valV, testV]=dividevec(pn,tn,valPercent, testPercent);
>
> %Parameters Setting for Learning and Training
> % net.trainParam.goal=0.01*MSE00;
> net.trainParam.epochs=1000;
>
> %Train the Network
> [net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);
>
> %Simulate the Network
> normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
> normValidateOutput=sim(net,valV.P,[],[],valV.T);
> normTestOutput=sim(net,testV.P,[],[],testV.T);
>
> %Reverse Normalize Outputs
> trainOutput=mapstd('reverse',normTrainOutput,ts);
> validateOutput=mapstd('reverse',normValidateOutput,ts);
> testOutput=mapstd('reverse',normTestOutput,ts);
>
> %Plot the Predicted Values to the Actual Values
> [s sTr]=size(trainOutput);
> [s sVa]=size(validateOutput);
> [s sTe]=size(testOutput);
>
> figure(1)
> plot(testOutput,'b')
> hold on
> plot(T((sTr+sVa+1):N),'k')
> title('comparision between Training Actual and Predictions')
>
> Again, Thank you for reading and feedback.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> subodhpaudel@gmail.com wrote in message <b3df16d2-4414-4ce1-aadd-528cb7f0a7d0@googlegroups.com>...
> > On Friday, February 15, 2013 12:40:07 PM UTC+1, Greg Heath wrote:
> > > "Subodh Paudel" <subodhpaudel@gmail.com> wrote in message
> > >
> > > <kfitfe$8hi$1@newscl01ah.mathworks.com>...
> > >
> > > > Hai All,
> > >
> > > > I want to forecast the building energy demand: input parameters:
> > >
> > > outside temperature and solar radiation, and output parameter: energy
> > >
> > > consumption
> > >
> > > >
> > >
> > > > I am choosing 2 layered network. I have a 1826 samples. My R2
> > >
> > > value is in negative and i am not finding the solution of it? Please,
> > >
> > > could you help me.
> > >
> > > >
> > >
> > > > The data size consists all inputs and targets in column vector.
> > >
> > >
> > >
> > > [ I N ] = size( input ) % [ 2 1826 ]
> > >
> > > [ O N ] = size( target ) % [ 1 1826 ]
> > >
> > >
> > >
> > > >I have used 70% samples for the generalization and 30% for prediction.
> > >
> > >
> > >
> > > Terminology:
> > >
> > >
> > >
> > > total = design + test
> > >
> > > design = train + validation
> > >
> > >
> > >
> > > > The souce code is given:
> > >
> > > >
> > >
> > > > data=xlsread('z:\testdata.xlsx','Sheet1')
> > >
> > > > %Size of input and Target Layers I:Input Layer, O:Output Layer,
> > >
> > >
> > >
> > > Official Terminology: 2 layers of Hidden and Output neurons. Input
> > >
> > > elements are fan-in units and NOT neurons. The unmodifed term
> > >
> > > 'layer' does not apply to inputs. The typical MLP is a 2-layer FFNN.
> > >
> > >
> > >
> > > I don't like this at all. Therefore I only refer to (3) layers of nodes (NOT
> > >
> > > neurons!). Consequently, since the typical MLP always has a layer of
> > >
> > > input nodes and a layer of output nodes, I refer to the typical topology
> > >
> > > as a MLP or FFNN with one hidden layer (of nodes or neurons).
> > >
> > >
> > >
> > > dN=size of
> > >
> > > > %the data
> > >
> > > > [dN I]=size(data);
> > >
> > >
> > >
> > > Please remove the "d" it is unnecessary and confusing.
> > >
> > >
> > >
> > > > dNt=round(0.7*dN); %70% of Data Used for the Training and
> > >
> > > Generalization
> > >
> > >
> > >
> > > Not standard terminology or proceedure
> > >
> > > Design = Training and Validation
> > >
> > > Use Validation set iteratively to determine training parameters and
> > >
> > > choose best of multiple designs.
> > >
> > > Need a nondesign subset (Test) to predict generalization error.
> > >
> > > If results are unsatisfactory, need to repeat with a new trn/val/tst
> > >
> > > data split (i.e., Test set is only used ONCE!)
> > >
> > >
> > >
> > > > input_sample= data((1:dNt),1:2)';
> > >
> > > > target_sample=data((1:dNt),3)';
> > >
> > > > validation_inputsample=data(((dNt+1):dN),1:2)';
> > >
> > > > validation_targetsample=data(((dNt+1):dN),3)';
> > >
> > > >
> > >
> > > > %DataProcessing of Training Sample (normalization with
> > >
> > > mean 0.1 and 0.9)
> > >
> > > > [pn,ps]=mapstd(input_sample,0.1,0.9);
> > >
> > > > [tn,ts]=mapstd(target_sample,0.1,0.9);
> > >
> > >
> > >
> > > No! There is absolutely no good reason to do that!
> > >
> > > Standarization means zero-mean/unit-variance.
> > >
> > >
> > >
> > > > %Apply Normalization to the Validation Input Sample
> > >
> > > > an=mapstd('apply',validation_inputsample,ps);
> > >
> > > >
> > >
> > > > %Iterative Optimization wrt to RMSE to determine Hidden
> > >
> > > Neurons:hidden_node,
> > >
> > >
> > >
> > > Use dimensionless NMSE = MSE/MSE00 (MSE00 =
> > >
> > > mean(var(target',1)) ) instead of scale dependent RMSE.
> > >
> > > Then coefficient of determination (i.e., R^2) is given by
> > >
> > > R2 =1-NMSE.
> > >
> > >
> > >
> > > > %Learning Rate:lr_rate and Momentum:mc_rate
> > >
> > > > sbmin=99999999;
> > >
> > > > hidden_node=0;
> > >
> > > > mc_rate=0;
> > >
> > > > lr_rate=0;
> > >
> > > > mm=1;
> > >
> > >
> > >
> > > ??
> > >
> > > Always begin with the MATLAB defaults.
> > >
> > >
> > >
> > > > %MLP (2-1-1 Network)with tangent hyperbolic transfer function in
> > >
> > > the hidden layer and
> > >
> > > > %Linear Activation Function in the Output Layer
> > >
> > >
> > >
> > > TERMINOLOGY: 2-H-1 node topology
> > >
> > >
> > >
> > > > %Initial Hidden Neuron Size Varies from 5 to 10
> > >
> > >
> > >
> > > Ntrneq = Ntrn*O % No. of training equations
> > >
> > > Nw = (I+1)*H+(H+1)*O % No. of unknown weights
> > >
> > >
> > >
> > > To mitigate noise and measurement error when not
> > >
> > > using regularization(msereg) or validation stopping,
> > >
> > > require Ntrneq >= Nw but desire Ntrneq >> Nw.
> > >
> > > This yields an upperbound for H
> > >
> > >
> > >
> > > Hub = -1 + ceil( (Ntrneq-O)/(I+O+1)) % 319
> > >
> > >
> > >
> > > %Hub = -1 + ceil( ( 0.7*1826-1)/(2+1+1) ) = 319
> > >
> > > %Hmax ~ round(Hub/8) = 40
> > >
> > >
> > >
> > > > for i=1:1:40;
> > >
> > >
> > >
> > > How about starting with a coarse search 1:5:40
> > >
> > >
> > >
> > > > net=newff(minmax(pn),[i 1],{'tansig','purelin'},'trainlm')
> > >
> > > > net.performFcn='mse';
> > >
> > > > net.performParam=[];
> > >
> > > > net.trainParam.goal=0.01*var(tn);
> > >
> > > > net.trainParam.epochs=1000;
> > >
> > > > net.trainParam.show=30;
> > >
> > > > net.trainParam.mu_max=1e10;
> > >
> > > > net.trainParam.min_grad=0.000001;
> > >
> > >
> > >
> > > Too complicated. To start, use as many defaults as possible
> > >
> > > For example
> > >
> > >
> > >
> > > net=newff(minmax(pn),[i 1]);
> > >
> > > net.trainParam.goal=0.01*var(tn);
> > >
> > > net.trainParam.show=30;
> > >
> > >
> > >
> > > > net.trainParam.max_fail=20;
> > >
> > >
> > >
> > > Too high for val stopping. Just delete and accept the default of 6.
> > >
> > >
> > >
> > > > for k=1:1:4 %Learning Rate Varies from 0.25 to 1
> > >
> > > > lr=0.25+(k-1)*.25;
> > >
> > > > net.trainParam.lr=lr;
> > >
> > > > for m=1:1:4 %Momentum Varies from 0.25 to 1
> > >
> > > > mc=0.25+(m-1)*0.25;
> > >
> > > > net.trainParam.m=mc;
> > >
> > > > net=init(net);%Assume 0 weight
> > >
> > > > net=train(net,pn,tn);
> > >
> > > > yn=sim(net,pn);
> > >
> > > > y=mapstd('reverse',yn,ts);
> > >
> > > > error=y-target_sample;
> > >
> > > > RMSE(i,k,m)=sqrt(mse(error));%RMSE Value Check for
> > >
> > > the Minimum Root Mean Square
> > >
> > > > if RMSE(i,k,m)<sbmin
> > >
> > > > sbmin=RMSE(i,k,m);
> > >
> > > > hidden_node=i;
> > >
> > > > mc_rate=mc;
> > >
> > > > lr_rate=lr;
> > >
> > > > IW_initial=net.IW{1,1};
> > >
> > > > LW_initial=net.IW{2,1};
> > >
> > > > hidden_bias_initial=net.b{1};
> > >
> > > > output_bias_initial=net.b{2};
> > >
> > > > end
> > >
> > > > end
> > >
> > > > end
> > >
> > > > end
> > >
> > >
> > >
> > > ??? Life is already hard enough, why not accept most defaults
> > >
> > > and just search over H and initial weights?
> > >
> > >
> > >
> > > NOTE: minmax version of newff automatically configures the
> > >
> > > initial weights.
> > >
> > >
> > >
> > > I'll stop here. Basically, start simple and gradually make
> > >
> > > improvements. Your current code will probably take hours and
> > >
> > > not give you any insight.
> > >
> > >
> > >
> > > Hope this helps.
> > >
> > >
> > >
> > > Greg
> > >
> > >
> > >
> > > > parameters =[hidden_node mc_rate lr_rate]
> > >
> > > > net=newff(minmax(pn),[hidden_node 1],{'tansig','purelin'},'trainlm')
> > >
> > > > net.performFcn='mse';
> > >
> > > > net.performParam=[];
> > >
> > > > net.trainParam.goal=0.01*var(tn);
> > >
> > > > net.trainParam.epochs=1000;
> > >
> > > > net.trainParam.show=30;
> > >
> > > > net.trainParam.mu_max=1e10;
> > >
> > > > net.trainParam.max_fail=20;
> > >
> > > > net.trainParam.min_grad=0.000001;
> > >
> > > > net.trainParam.lr=lr_rate;
> > >
> > > > net.trainParam.m=mc_rate;
> > >
> > > >
> > >
> > > > MSE_initial=mean(var(target_sample)); %Approximate MSE for
> > >
> > > a constant output model
> > >
> > > > net.IW{1,1}=IW_initial;
> > >
> > > > net.IW{2,1}=LW_initial;
> > >
> > > > net.b{1}=hidden_bias_initial;
> > >
> > > > net.b{2}=output_bias_initial;
> > >
> > > >
> > >
> > > > net=train(net,pn,tn);
> > >
> > > > yn=sim(net,an);
> > >
> > > > y=mapstd('reverse',yn,ts);
> > >
> > > > %Error Calculation
> > >
> > > > error=y-validation_targetsample;
> > >
> > > > percentage_error=abs(error)./y*100; %Percentage of Error
> > >
> > > > RMSE=sqrt(mse(error)); %Root Mean Square Error
> > >
> > > > NMSE= mse(error)/MSE_initial; % Normalized Mean Square Error
> > >
> > > > R2=1-NMSE; %R^2 Statistics for the Regression Analysis
> > >
> > > > MAE=mean(abs(error)); %Mean Absolute Error
> > >
> > > > MAPE = mean(percentage_error); % Mean Absolute Percentage
> > >
> > > Error
> > >
> > > > %Analysis of Predicted and Target Values
> > >
> > > > performance_error=[RMSE R2]
> > >
> > > > pp=[validation_targetsample y];
> > >
> > > > figure(1)
> > >
> > > > plot(y,'g')
> > >
> > > > hold
> > >
> > > > plot(validation_targetsample,'r')
> > >
> > > > title('comparision between actual targets and predictions'
> >
> >
> > Thank you very much Greg. I will correct my code from your suggestions and came to you.
Dear Sir Greg,
Thank you very much for your previous suggestion and the your precious time given to me. I have forget to address some of the queries also. Accordingly, I changed my training structure. Here is the code i changed, but still, i found many problems. the prediction pattern follows, however, R2 value for training is 0.62 but R2 value is negative for the training and validation phase. Please, could you address my some questionaries below:
Here is the code:
Here is My Rest Code:
%Input and Hidden Layer
[I N]=size(P); %[2 1826]
[O N]=size(T); %[1 1826]
MSE00=mean(var(T)); % (1) Is this correct ?
%Normalization of Input and Output Data
[pn,ps]=mapstd(P); %Already Transposed
[tn,ts]=mapstd(T);
H=18; %(Iterative Optimization)
%MLP (2-H-1)
net=newff(minmax(pn),[H O]);
%Divide Samples into Training, Validation and Test
valPercent=0.015;
testPercent=0.015;
[trainV, valV, testV]=dividevec(pn,tn,valPercent, testPercent);
%Parameters Setting for Learning and Training
net.trainParam.goal=0.01*MSE00;
net.trainParam.epochs=1000;
%Train the Network
[net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);
%Simulate the Network
normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
normValidateOutput=sim(net,valV.P,[],[],valV.T);
normTestOutput=sim(net,testV.P,[],[],testV.T);
%Reverse Normalize Outputs
trainOutput=mapstd('reverse',normTrainOutput,ts);
validateOutput=mapstd('reverse',normValidateOutput,ts);
testOutput=mapstd('reverse',normTestOutput,ts);
% (2) Is this a correct way to calculate MSE and R2 values for training, testing and validation. I am Using Matlab 2009a??? Because MSE00 is the global for all, how to compute MSE00 for train, validatation and test??
[s sTr]=size(trainOutput);
[s sVa]=size(validateOutput);
[s sTe]=size(testOutput);
errorTrain=(T(1:sTr))-trainOutput;
MSETrain=mse(errorTrain);
R2Train=1-MSETrain/MSE00; % (3) Is this should be divided by MSE00 ?? %0.624
errorValidate=(T((sTr+1):(sTr+sVa)))-validateOutput;
MSEValidate=mse(errorValidate);
R2Validate=1-MSEValidate/MSE00; %-3.245
errorTest=(T((sTr+sVa+1):N))-testOutput;
MSETest=mse(errorTest);
R2Test=1-MSETest/MSE00; %-1.245
%(4) What is the differences between R2 train/val/test this correlation value compare to the neural network shown shown R value for train/val/test? Because in the neural network, i have this value 0.89/0.86/0.84 for train/val/test? Is the Neural network shown is Naive Linear Model?
%(5) Although i achieved even this value, my R value is always changing when i simulate? Is this due to change of weight. Assume, i get satisfactory value, how can i generalize further to predict other new values??
figure(1)
plot(testOutput,'b')
hold on
plot(T((sTr+sVa+1):N),'k')
title('comparision between Training Actual and Predictions')
Again, Thank you for reading and feedback.![]()
> Hai Greg,
> Thank You for your previous suggestion. Accordingly, i changed my training structure and still the building energy forecast is not good.
> I have some queries, further,:
> 1) All the time i run, i have the R-value i.e. in the regression plot in the neural network changing? I could not figure it out...is the weight changed all the time...
> 2) Regarding MSE00 i.e. initial condition, MSE00=mean(var(target)), it has a large value in the range of exp.. does it should have a normalize value?
> 3) How can i read the R2 value of the test, validation and training value although matlab describes the value for all the cases:
>
> Here is My Rest Code:
>
> data=xlsread('C:\Users\Dell\Desktop\testdata.xlsx','Sheet1');
> [dN dI]=size(data);
>
> %Input and Target Data
> P= data(1:dN,1:2)';
> T=data(1:dN,3)';
>
> %Input and Hidden Layer
> [I N]=size(P); %[2 1826]
> [O N]=size(T); %[1 1826]
> % MSE00=mean(var(T));
>
> %Normalization of Input and Output Data
> [pn,ps]=mapstd(P);
> [tn,ts]=mapstd(T);
>
> H=8; %(Iterative Optimization)
> Ntrneq=N*O; %1826 No of Training Equations
> Nw = (I+1)*H+(H+1)*O; % No. of unknown weights
> %This yields an upperbound for H
> Hub = -1 + ceil( (Ntrneq-O)/(I+O+1)); % 319
> %Hub = -1 + ceil( ( 0.7*1826-1)/(2+1+1) ) = 319
> %Hmax ~ round(Hub/8) = 40
>
> %MLP (2-H-1)
> net=newff(minmax(pn),[H O]);
>
> %Divide Samples into Training, Validation and Test
> valPercent=0.015;
> testPercent=0.015;
> [trainV, valV, testV]=dividevec(pn,tn,valPercent, testPercent);
>
> %Parameters Setting for Learning and Training
> % net.trainParam.goal=0.01*MSE00;
> net.trainParam.epochs=1000;
>
> %Train the Network
> [net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);
>
> %Simulate the Network
> normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
> normValidateOutput=sim(net,valV.P,[],[],valV.T);
> normTestOutput=sim(net,testV.P,[],[],testV.T);
>
> %Reverse Normalize Outputs
> trainOutput=mapstd('reverse',normTrainOutput,ts);
> validateOutput=mapstd('reverse',normValidateOutput,ts);
> testOutput=mapstd('reverse',normTestOutput,ts);
>
> %Plot the Predicted Values to the Actual Values
> [s sTr]=size(trainOutput);
> [s sVa]=size(validateOutput);
> [s sTe]=size(testOutput);
>
> figure(1)
> plot(testOutput,'b')
> hold on
> plot(T((sTr+sVa+1):N),'k')
> title('comparision between Training Actual and Predictions')
>
> Again, Thank you for reading and feedback.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> subodhpaudel@gmail.com wrote in message <b3df16d2-4414-4ce1-aadd-528cb7f0a7d0@googlegroups.com>...
> > On Friday, February 15, 2013 12:40:07 PM UTC+1, Greg Heath wrote:
> > > "Subodh Paudel" <subodhpaudel@gmail.com> wrote in message
> > >
> > > <kfitfe$8hi$1@newscl01ah.mathworks.com>...
> > >
> > > > Hai All,
> > >
> > > > I want to forecast the building energy demand: input parameters:
> > >
> > > outside temperature and solar radiation, and output parameter: energy
> > >
> > > consumption
> > >
> > > >
> > >
> > > > I am choosing 2 layered network. I have a 1826 samples. My R2
> > >
> > > value is in negative and i am not finding the solution of it? Please,
> > >
> > > could you help me.
> > >
> > > >
> > >
> > > > The data size consists all inputs and targets in column vector.
> > >
> > >
> > >
> > > [ I N ] = size( input ) % [ 2 1826 ]
> > >
> > > [ O N ] = size( target ) % [ 1 1826 ]
> > >
> > >
> > >
> > > >I have used 70% samples for the generalization and 30% for prediction.
> > >
> > >
> > >
> > > Terminology:
> > >
> > >
> > >
> > > total = design + test
> > >
> > > design = train + validation
> > >
> > >
> > >
> > > > The souce code is given:
> > >
> > > >
> > >
> > > > data=xlsread('z:\testdata.xlsx','Sheet1')
> > >
> > > > %Size of input and Target Layers I:Input Layer, O:Output Layer,
> > >
> > >
> > >
> > > Official Terminology: 2 layers of Hidden and Output neurons. Input
> > >
> > > elements are fan-in units and NOT neurons. The unmodifed term
> > >
> > > 'layer' does not apply to inputs. The typical MLP is a 2-layer FFNN.
> > >
> > >
> > >
> > > I don't like this at all. Therefore I only refer to (3) layers of nodes (NOT
> > >
> > > neurons!). Consequently, since the typical MLP always has a layer of
> > >
> > > input nodes and a layer of output nodes, I refer to the typical topology
> > >
> > > as a MLP or FFNN with one hidden layer (of nodes or neurons).
> > >
> > >
> > >
> > > dN=size of
> > >
> > > > %the data
> > >
> > > > [dN I]=size(data);
> > >
> > >
> > >
> > > Please remove the "d" it is unnecessary and confusing.
> > >
> > >
> > >
> > > > dNt=round(0.7*dN); %70% of Data Used for the Training and
> > >
> > > Generalization
> > >
> > >
> > >
> > > Not standard terminology or proceedure
> > >
> > > Design = Training and Validation
> > >
> > > Use Validation set iteratively to determine training parameters and
> > >
> > > choose best of multiple designs.
> > >
> > > Need a nondesign subset (Test) to predict generalization error.
> > >
> > > If results are unsatisfactory, need to repeat with a new trn/val/tst
> > >
> > > data split (i.e., Test set is only used ONCE!)
> > >
> > >
> > >
> > > > input_sample= data((1:dNt),1:2)';
> > >
> > > > target_sample=data((1:dNt),3)';
> > >
> > > > validation_inputsample=data(((dNt+1):dN),1:2)';
> > >
> > > > validation_targetsample=data(((dNt+1):dN),3)';
> > >
> > > >
> > >
> > > > %DataProcessing of Training Sample (normalization with
> > >
> > > mean 0.1 and 0.9)
> > >
> > > > [pn,ps]=mapstd(input_sample,0.1,0.9);
> > >
> > > > [tn,ts]=mapstd(target_sample,0.1,0.9);
> > >
> > >
> > >
> > > No! There is absolutely no good reason to do that!
> > >
> > > Standarization means zero-mean/unit-variance.
> > >
> > >
> > >
> > > > %Apply Normalization to the Validation Input Sample
> > >
> > > > an=mapstd('apply',validation_inputsample,ps);
> > >
> > > >
> > >
> > > > %Iterative Optimization wrt to RMSE to determine Hidden
> > >
> > > Neurons:hidden_node,
> > >
> > >
> > >
> > > Use dimensionless NMSE = MSE/MSE00 (MSE00 =
> > >
> > > mean(var(target',1)) ) instead of scale dependent RMSE.
> > >
> > > Then coefficient of determination (i.e., R^2) is given by
> > >
> > > R2 =1-NMSE.
> > >
> > >
> > >
> > > > %Learning Rate:lr_rate and Momentum:mc_rate
> > >
> > > > sbmin=99999999;
> > >
> > > > hidden_node=0;
> > >
> > > > mc_rate=0;
> > >
> > > > lr_rate=0;
> > >
> > > > mm=1;
> > >
> > >
> > >
> > > ??
> > >
> > > Always begin with the MATLAB defaults.
> > >
> > >
> > >
> > > > %MLP (2-1-1 Network)with tangent hyperbolic transfer function in
> > >
> > > the hidden layer and
> > >
> > > > %Linear Activation Function in the Output Layer
> > >
> > >
> > >
> > > TERMINOLOGY: 2-H-1 node topology
> > >
> > >
> > >
> > > > %Initial Hidden Neuron Size Varies from 5 to 10
> > >
> > >
> > >
> > > Ntrneq = Ntrn*O % No. of training equations
> > >
> > > Nw = (I+1)*H+(H+1)*O % No. of unknown weights
> > >
> > >
> > >
> > > To mitigate noise and measurement error when not
> > >
> > > using regularization(msereg) or validation stopping,
> > >
> > > require Ntrneq >= Nw but desire Ntrneq >> Nw.
> > >
> > > This yields an upperbound for H
> > >
> > >
> > >
> > > Hub = -1 + ceil( (Ntrneq-O)/(I+O+1)) % 319
> > >
> > >
> > >
> > > %Hub = -1 + ceil( ( 0.7*1826-1)/(2+1+1) ) = 319
> > >
> > > %Hmax ~ round(Hub/8) = 40
> > >
> > >
> > >
> > > > for i=1:1:40;
> > >
> > >
> > >
> > > How about starting with a coarse search 1:5:40
> > >
> > >
> > >
> > > > net=newff(minmax(pn),[i 1],{'tansig','purelin'},'trainlm')
> > >
> > > > net.performFcn='mse';
> > >
> > > > net.performParam=[];
> > >
> > > > net.trainParam.goal=0.01*var(tn);
> > >
> > > > net.trainParam.epochs=1000;
> > >
> > > > net.trainParam.show=30;
> > >
> > > > net.trainParam.mu_max=1e10;
> > >
> > > > net.trainParam.min_grad=0.000001;
> > >
> > >
> > >
> > > Too complicated. To start, use as many defaults as possible
> > >
> > > For example
> > >
> > >
> > >
> > > net=newff(minmax(pn),[i 1]);
> > >
> > > net.trainParam.goal=0.01*var(tn);
> > >
> > > net.trainParam.show=30;
> > >
> > >
> > >
> > > > net.trainParam.max_fail=20;
> > >
> > >
> > >
> > > Too high for val stopping. Just delete and accept the default of 6.
> > >
> > >
> > >
> > > > for k=1:1:4 %Learning Rate Varies from 0.25 to 1
> > >
> > > > lr=0.25+(k-1)*.25;
> > >
> > > > net.trainParam.lr=lr;
> > >
> > > > for m=1:1:4 %Momentum Varies from 0.25 to 1
> > >
> > > > mc=0.25+(m-1)*0.25;
> > >
> > > > net.trainParam.m=mc;
> > >
> > > > net=init(net);%Assume 0 weight
> > >
> > > > net=train(net,pn,tn);
> > >
> > > > yn=sim(net,pn);
> > >
> > > > y=mapstd('reverse',yn,ts);
> > >
> > > > error=y-target_sample;
> > >
> > > > RMSE(i,k,m)=sqrt(mse(error));%RMSE Value Check for
> > >
> > > the Minimum Root Mean Square
> > >
> > > > if RMSE(i,k,m)<sbmin
> > >
> > > > sbmin=RMSE(i,k,m);
> > >
> > > > hidden_node=i;
> > >
> > > > mc_rate=mc;
> > >
> > > > lr_rate=lr;
> > >
> > > > IW_initial=net.IW{1,1};
> > >
> > > > LW_initial=net.IW{2,1};
> > >
> > > > hidden_bias_initial=net.b{1};
> > >
> > > > output_bias_initial=net.b{2};
> > >
> > > > end
> > >
> > > > end
> > >
> > > > end
> > >
> > > > end
> > >
> > >
> > >
> > > ??? Life is already hard enough, why not accept most defaults
> > >
> > > and just search over H and initial weights?
> > >
> > >
> > >
> > > NOTE: minmax version of newff automatically configures the
> > >
> > > initial weights.
> > >
> > >
> > >
> > > I'll stop here. Basically, start simple and gradually make
> > >
> > > improvements. Your current code will probably take hours and
> > >
> > > not give you any insight.
> > >
> > >
> > >
> > > Hope this helps.
> > >
> > >
> > >
> > > Greg
> > >
> > >
> > >
> > > > parameters =[hidden_node mc_rate lr_rate]
> > >
> > > > net=newff(minmax(pn),[hidden_node 1],{'tansig','purelin'},'trainlm')
> > >
> > > > net.performFcn='mse';
> > >
> > > > net.performParam=[];
> > >
> > > > net.trainParam.goal=0.01*var(tn);
> > >
> > > > net.trainParam.epochs=1000;
> > >
> > > > net.trainParam.show=30;
> > >
> > > > net.trainParam.mu_max=1e10;
> > >
> > > > net.trainParam.max_fail=20;
> > >
> > > > net.trainParam.min_grad=0.000001;
> > >
> > > > net.trainParam.lr=lr_rate;
> > >
> > > > net.trainParam.m=mc_rate;
> > >
> > > >
> > >
> > > > MSE_initial=mean(var(target_sample)); %Approximate MSE for
> > >
> > > a constant output model
> > >
> > > > net.IW{1,1}=IW_initial;
> > >
> > > > net.IW{2,1}=LW_initial;
> > >
> > > > net.b{1}=hidden_bias_initial;
> > >
> > > > net.b{2}=output_bias_initial;
> > >
> > > >
> > >
> > > > net=train(net,pn,tn);
> > >
> > > > yn=sim(net,an);
> > >
> > > > y=mapstd('reverse',yn,ts);
> > >
> > > > %Error Calculation
> > >
> > > > error=y-validation_targetsample;
> > >
> > > > percentage_error=abs(error)./y*100; %Percentage of Error
> > >
> > > > RMSE=sqrt(mse(error)); %Root Mean Square Error
> > >
> > > > NMSE= mse(error)/MSE_initial; % Normalized Mean Square Error
> > >
> > > > R2=1-NMSE; %R^2 Statistics for the Regression Analysis
> > >
> > > > MAE=mean(abs(error)); %Mean Absolute Error
> > >
> > > > MAPE = mean(percentage_error); % Mean Absolute Percentage
> > >
> > > Error
> > >
> > > > %Analysis of Predicted and Target Values
> > >
> > > > performance_error=[RMSE R2]
> > >
> > > > pp=[validation_targetsample y];
> > >
> > > > figure(1)
> > >
> > > > plot(y,'g')
> > >
> > > > hold
> > >
> > > > plot(validation_targetsample,'r')
> > >
> > > > title('comparision between actual targets and predictions'
> >
> >
> > Thank you very much Greg. I will correct my code from your suggestions and came to you.
Dear Sir Greg,
Thank you very much for your previous suggestion and the your precious time given to me. I have forget to address some of the queries also. Accordingly, I changed my training structure. Here is the code i changed, but still, i found many problems. the prediction pattern follows, however, R2 value for training is 0.62 but R2 value is negative for the training and validation phase. Please, could you address my some questionaries below:
Here is the code:
Here is My Rest Code:
%Input and Hidden Layer
[I N]=size(P); %[2 1826]
[O N]=size(T); %[1 1826]
MSE00=mean(var(T)); % (1) Is this correct ?
%Normalization of Input and Output Data
[pn,ps]=mapstd(P); %Already Transposed
[tn,ts]=mapstd(T);
H=18; %(Iterative Optimization)
%MLP (2-H-1)
net=newff(minmax(pn),[H O]);
%Divide Samples into Training, Validation and Test
valPercent=0.015;
testPercent=0.015;
[trainV, valV, testV]=dividevec(pn,tn,valPercent, testPercent);
%Parameters Setting for Learning and Training
net.trainParam.goal=0.01*MSE00;
net.trainParam.epochs=1000;
%Train the Network
[net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);
%Simulate the Network
normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
normValidateOutput=sim(net,valV.P,[],[],valV.T);
normTestOutput=sim(net,testV.P,[],[],testV.T);
%Reverse Normalize Outputs
trainOutput=mapstd('reverse',normTrainOutput,ts);
validateOutput=mapstd('reverse',normValidateOutput,ts);
testOutput=mapstd('reverse',normTestOutput,ts);
% (2) Is this a correct way to calculate MSE and R2 values for training, testing and validation. I am Using Matlab 2009a??? Because MSE00 is the global for all, how to compute MSE00 for train, validatation and test??
[s sTr]=size(trainOutput);
[s sVa]=size(validateOutput);
[s sTe]=size(testOutput);
errorTrain=(T(1:sTr))-trainOutput;
MSETrain=mse(errorTrain);
R2Train=1-MSETrain/MSE00; % (3) Is this should be divided by MSE00 ?? %0.624
errorValidate=(T((sTr+1):(sTr+sVa)))-validateOutput;
MSEValidate=mse(errorValidate);
R2Validate=1-MSEValidate/MSE00; %-3.245
errorTest=(T((sTr+sVa+1):N))-testOutput;
MSETest=mse(errorTest);
R2Test=1-MSETest/MSE00; %-1.245
%(4) What is the differences between R2 train/val/test this correlation value compare to the neural network shown shown R value for train/val/test? Because in the neural network, i have this value 0.89/0.86/0.84 for train/val/test? Is the Neural network shown is Naive Linear Model?
%(5) Although i achieved even this value, my R value is always changing when i simulate? Is this due to change of weight. Assume, i get satisfactory value, how can i generalize further to predict other new values??
figure(1)
plot(testOutput,'b')
hold on
plot(T((sTr+sVa+1):N),'k')
title('comparision between Training Actual and Predictions')
Again, Thank you for reading and feedback.